python: efficient way to check only a couple of rows in a csvreader iterator?
<p>I have a (very large) CSV file that looks something like this:</p>

<pre><code>header1,header2,header3
name0,rank0,serial0
name1,rank1,serial1
name2,rank2,serial2
</code></pre>

<p>I've written some code that processes the file, and writes it out (using csvwriter) modified as such, with some information I compute appended to the end of each row:</p>

<pre><code>header1,header2,header3,new_hdr4,new_hdr5
name0,rank0,serial0,salary0,base0
name1,rank1,serial1,salary1,base1
name2,rank2,serial2,salary2,base2
</code></pre>

<p>What I'm trying to do is structure the script so that it auto-detects whether or not the CSV file it's reading has <strong><em>already been processed</em></strong>.  If it has been processed, I can skip a lot of expensive calculations later.  I'm trying to understand whether there is a reasonable way of doing this within the reader loop.  I <em>could</em> just open the file once, read in enough to do the detection, and then close and reopen it with a flag set, but this seems hackish.</p>

<p>Is there a way to do this within the same reader?  The logic is something like:</p>

<pre><code>read first N lines   ###(N is small)
   if (some condition)
      already_processed = TRUE
      read_all_csv_without_processing
   else
      read_all_csv_WITH_processing
</code></pre>

<p>I can't just use the iterator that reader gives me, because by the time I've gotten enough lines to do my conditional check, I don't have any good way to go back to the beginning of the CSV.  Is closing and reopening it really the most elegant way to do this?</p>

<p>I suppose if you do not want to just test the first few lines of the file, you could create a single iterator from a  list and then the continuation of the csv reader.</p>

<p>Given:</p>

<pre><code>header1,header2,header3
name0,rank0,serial0
name1,rank1,serial1
name2,rank2,serial2
</code></pre>

<p>You can do:</p>

<pre><code>import csv
from itertools import chain

with open(fn) as f:
    reader=csv.reader(f)
    header=next(reader)
    N=2
    p_list=[]
    for i in range(N):   # N is however many you need to set processed flag
        p_list.append(next(reader))
    print("p_list:", p_list)
    # now use p_list to determine if processed however processed=check(p_list)

    for row in chain(iter(p_list), reader):  # chain creates a single csv reader...
        # handler processed or not here from the stream of rows...
        # if not processed:
        #     process
        # else:
        #     handle already processed...
        # print row just to show csv data is complete:
        print(row)
</code></pre>

<p>Prints:</p>

<pre><code>p_list: [['name0', 'rank0', 'serial0'], ['name1', 'rank1', 'serial1']]
['name0', 'rank0', 'serial0']
['name1', 'rank1', 'serial1']
['name2', 'rank2', 'serial2']
</code></pre>

python csv