Read multiple lines from a file batch by batch
<p>I would like to know is there a method that can read multiple lines from a file batch by batch. For example:</p>

<pre><code>with open(filename, 'rb') as f:
    for n_lines in f:
        process(n_lines)
</code></pre>

<p>In this function, what I would like to do is: for every iteration, next n lines will be read from the file, batch by batch. </p>

<p>Because one single file is too big. What I want to do is to read it part by part.</p>

<p><a href="https://docs.python.org/3/library/itertools.html#itertools.islice" rel="nofollow"><code>itertools.islice</code></a> and two arg <code>iter</code> can be used to accomplish this, but it's a little funny:</p>

<pre><code>from itertools import islice

n = 5  # Or whatever chunk size you want
with open(filename, 'rb') as f:
    for n_lines in iter(lambda: tuple(islice(f, n)), ()):
        process(n_lines)
</code></pre>

<p>This will keep <code>islice</code>ing off <code>n</code> lines at a time (using <code>tuple</code> to actually force the whole chunk to be read in) until the <code>f</code> is exhausted, at which point it will stop. The final chunk will be less than <code>n</code> lines if the number of lines in the file isn't an even multiple of <code>n</code>. If you want all the lines to be a single string, change the <code>for</code> loop to be:</p>

<pre><code>    # The b prefixes are ignored on 2.7, and necessary on 3.x since you opened
    # the file in binary mode
    for n_lines in iter(lambda: b''.join(islice(f, n)), b''):
</code></pre>

<p>Another approach is to use <code>izip_longest</code> for the purpose, which avoids <code>lambda</code> functions:</p>

<pre><code>from future_builtins import map  # Only on Py2
from itertools import izip_longest  # zip_longest on Py3

    # gets tuples possibly padded with empty strings at end of file
    for n_lines in izip_longest(*[f]*n, fillvalue=b''):

    # Or to combine into a single string:
    for n_lines in map(b''.join, izip_longest(*[f]*n, fillvalue=b'')):
</code></pre>

python python-2.7 io readfile