How to set the "chunk size" of read lines from file read with Python subprocess.Popen() or open()?
<p>I have a fairly large text file which I would like to run in chunks. In order to do this with the <code>subprocess</code> library, one would execute following shell command:</p>

<pre><code>"cat hugefile.log"
</code></pre>

<p>with the code:</p>

<pre><code>import subprocess
task = subprocess.Popen("cat hugefile.log", shell=True,  stdout=subprocess.PIPE)
data = task.stdout.read()
</code></pre>

<p>Using <code>print(data)</code> will spit out the entire contents of the file at once. How can I present the number of chunks, and then access the contents of this file by the chunk size (e.g. chunk = three lines at a time). </p>

<p>It must be something like: </p>

<pre><code>chunksize = 1000   # break up hugefile.log into 1000 chunks

for chunk in data:
    print(chunk)
</code></pre>

<p>The equivalent question with Python <code>open()</code> of course uses the code</p>

<pre><code>with open('hugefile.log', 'r') as f:
     read_data = f.read()
</code></pre>

<p>How would you <code>read_data</code> in chunks? </p>

<p>Using a file, you can iterate on the file handle (no need for subprocess to open <code>cat</code>):</p>

<pre><code>with open('hugefile.log', 'r') as f:
     for read_line in f:
        print(read_line)
</code></pre>

<p>Python reads a line by reading all the chars up to <code>\n</code>. To simulate the line-by-line I/O, just call it 3 times. or read and count 3 <code>\n</code> chars but you have to handle the end of file, etc... not very useful and you won't gain any speed by doing that.</p>

<pre><code>with open('hugefile.log', 'r') as f:
     while True:
        read_3_lines = ""
        try:
           for i in range(3):
               read_3_lines += next(f)
        # process read_3_lines
        except StopIteration:  # end of file
            # process read_3_lines if nb lines not divisible by 3
            break
</code></pre>

<p>With <code>Popen</code> you can do exactly the same, as a bonus add <code>poll</code> to monitor the process (no need with <code>cat</code> but I suppose that your process is different and that's only for the question's purpose)</p>

<pre><code>import subprocess
task = subprocess.Popen("cat hugefile.log", shell=True,  stdout=subprocess.PIPE)
while True:
    line = task.stdout.readline()
    if line == '' and task.poll() != None: break

rc = task.wait()   # wait for completion and get return code of the command
</code></pre>

<p>Python 3 compliant code supporting encoding:</p>

<pre><code>    line = task.stdout.readline().decode("latin-1")
    if len(line) == 0 and task.poll() != None: break
</code></pre>

<p>Now, if you want to split the file into a given number of chunks:</p>

<ul>
<li>you cannot use <code>Popen</code> for obvious reasons: you would have to know the size of the output first</li>
<li>if you have a file as input you can do as follows:</li>
</ul>

<p>code:</p>

<pre><code>import os,sys
filename = "hugefile.log"
filesize = os.path.getsize(filename)
nb_chunks = 1000
chunksize = filesize // nb_chunks

with open(filename,"r") as f:
   while True:
      chunk = f.read(chunksize)
      if chunk=="":
          break
      # do something useful with the chunk
      sys.stdout.write(chunk)
</code></pre>

python bash shell subprocess chunking