Collect data in chunks from stdin: Python
<p>I have the following Python code where I collect data from standard input into a list and run syntaxnet on it. The data is in the form of json objects from which I will extract the text field and feed it to syntaxnet.</p>

<pre><code>data = []
for line in sys.stdin:
    data.append(line)
run_syntaxnet(data)    ##This is a function##
</code></pre>

<p>I am doing this because I do not want Syntaxnet to run for every single tweet since it will take a very long time and hence decrease performance. </p>

<p>Also, when I run this code on very large data, I do not want to keep collecting it forever and run out of memory. So I want to collect data in chunks- may be like 10000 tweets at a time and run Syntaxnet on them. Can someone help me how to do this?</p>

<p>Also, I want to understand what can be the maximum length of the list <code>data</code> so that I do not run out of memory.</p>

<p>I think this is all you need:</p>

<pre><code>data = []
for line in sys.stdin:
    data.append(line)
    if len(data) == 10000:
        run_syntaxnet(data)    ##This is a function##
        data = []
</code></pre>

<p>once the list get to 10000, then run the function and reset your data list. Also the maximum size of the list will vary from machine to machine, depending on how much memory you have, so it will probably be best to try it out with different lengths and find out what is optimum.</p>

python python-2.7 stdin