Random comma inserted at character 8192 in python "json" result called from node.js
<p>I'm a JS developer just learning python. This is my first time trying to use node (v6.7.0) and python (v2.7.1) together. I'm using restify with python-runner as a bridge to my python virtualenv. My python script uses a RAKE NLP keyword-extraction package.</p>

<p>I can't figure out for the life of me why my return data in <strong>server.js</strong> inserts a random comma at character 8192 and roughly multiples of. There's no pattern except the location; Sometimes it's in the middle of the object key string other times in the value, othertimes after the comma separating the object pairs. This completely breaks the JSON.parse() on the return data. Example outputs below. When I run the script from a python shell, this doesn't happen.</p>

<p>I seriously can't figure out why this is happening, any experienced devs have any ideas?</p>

<p><em>Sample output in browser</em></p>

<pre><code>[..., {...ate': 1.0, 'intended recipient': 4.,0, 'correc...}, ...]
</code></pre>

<p><em>Sample output in python shell</em></p>

<pre><code>[..., {...ate': 1.0, 'intended recipient': 4.0, 'correc...}, ...]
</code></pre>

<p><strong>DISREGARD ANY DISCREPANCIES REGARDING OBJECT CONVERSION AND HANDLING IN THE FILES BELOW. THE CODE HAS BEEN SIMPLIFIED TO SHOWCASE THE ISSUE</strong></p>

<p><strong>server.js</strong></p>

<pre><code>var restify = require('restify');
var py = require('python-runner');

var server = restify.createServer({...});

server.get('/keyword-extraction', function( req, res, next ) {

    py.execScript(__dirname + '/keyword-extraction.py', {
        bin: '.py/bin/python'
    })
    .then( function( data ) {
        fData = JSON.parse(data); &lt;---- ERROR
        res.json(fData);
    })
    .catch( function( err ) {...});

    return next();
});

server.listen(8001, 'localhost', function() {...});
</code></pre>

<p><strong>keyword-extraction.py</strong></p>

<pre><code>import csv
import json
import RAKE

f = open( 'emails.csv', 'rb' )
f.readline() # skip line containing col names

outputData = []

try:
    reader = csv.reader(f)

    for row in reader:
        email = {}
        emailBody = row[7]

        Rake = RAKE.Rake('SmartStoplist.txt')

        rakeOutput = Rake.run(emailBody)        

        for tuple in rakeOutput:

            email[tuple[0]] = tuple[1]

        outputData.append(email)

finally:
    file.close()

    print( json.dumps(outputData))
</code></pre>

<p>This looks suspiciously like a bug related to size of some buffer, since 8192 is a power of two.</p>

<p>The main thing here is to isolate exactly where the failure is occurring. If I were debugging this, I would </p>

<ol>
<li><p>Take a closer look at the output from <code>json.dumps</code>, by printing several characters on either side of position 8191, ideally the integer character code (unicode, ASCII, or whatever). </p></li>
<li><p>If that looks OK, I would try capturing the output from the python script as a file and read that directly in the node server (i.e. don't run a python script). </p></li>
<li><p>If that works, then create a python script that takes that file and outputs it without manipulation and have your node server execute that python script instead of the one it is using now.</p></li>
</ol>

<p>That should help you figure out where the problem is occurring. From comments, I suspect that this is essentially a bug that you cannot control, unless you can increase the python buffer size enough to guarantee your data will never blow the buffer. 8K is pretty small, so that might be a realistic solution.</p>

<p>If that is inadequate, then you might consider processing the data on the the node server, to remove every character at <code>n * 8192</code>, if you can consistently rely on that.  Good luck. </p>

javascript python json node.js restify