Read multiple lines from a file batch by batch
<p>I would like to know is there a method that can read multiple lines from a file batch by batch. For example:</p>

<pre><code>with open(filename, 'rb') as f:
    for n_lines in f:
        process(n_lines)
</code></pre>

<p>In this function, what I would like to do is: for every iteration, next n lines will be read from the file, batch by batch. </p>

<p>Because one single file is too big. What I want to do is to read it part by part.</p>

<p>You can actually just iterate over lines in a file (see <a href="https://docs.python.org/2/library/stdtypes.html#file.next" rel="nofollow">file.next</a> docs - this also works on Python 3) like</p>

<pre><code>with open(filename) as f:
    for line in f:
        something(line)
</code></pre>

<p>so your code can be rewritten to</p>

<pre><code>n=5 # your chunk size
with open(filename) as f:
    batch=[]
    for line in f:
        chunk.append(line)
        if len(batch)==n:
            process(batch)
            chunk=[]
process(batch) #this batch might be smaller or even empty
</code></pre>

<p>but normally just processing line-by-line is more convenient (first example)</p>

<p>If you dont care about how many lines are read exactly for each batch but just that it is not too much memory then use <a href="https://docs.python.org/2/library/stdtypes.html#file.readlines" rel="nofollow">file.readlines</a> with <code>sizehint</code> like</p>

<pre><code>size_hint=2&lt;&lt;24 # 16MB
with open(filename) as f:
    while f: # not sure if this check works
        process(f.readlines(size_hint)
</code></pre>

python python-2.7 io readfile