python: efficient way to check only a couple of rows in a csvreader iterator?
<p>I have a (very large) CSV file that looks something like this:</p>

<pre><code>header1,header2,header3
name0,rank0,serial0
name1,rank1,serial1
name2,rank2,serial2
</code></pre>

<p>I've written some code that processes the file, and writes it out (using csvwriter) modified as such, with some information I compute appended to the end of each row:</p>

<pre><code>header1,header2,header3,new_hdr4,new_hdr5
name0,rank0,serial0,salary0,base0
name1,rank1,serial1,salary1,base1
name2,rank2,serial2,salary2,base2
</code></pre>

<p>What I'm trying to do is structure the script so that it auto-detects whether or not the CSV file it's reading has <strong><em>already been processed</em></strong>.  If it has been processed, I can skip a lot of expensive calculations later.  I'm trying to understand whether there is a reasonable way of doing this within the reader loop.  I <em>could</em> just open the file once, read in enough to do the detection, and then close and reopen it with a flag set, but this seems hackish.</p>

<p>Is there a way to do this within the same reader?  The logic is something like:</p>

<pre><code>read first N lines   ###(N is small)
   if (some condition)
      already_processed = TRUE
      read_all_csv_without_processing
   else
      read_all_csv_WITH_processing
</code></pre>

<p>I can't just use the iterator that reader gives me, because by the time I've gotten enough lines to do my conditional check, I don't have any good way to go back to the beginning of the CSV.  Is closing and reopening it really the most elegant way to do this?</p>

<p>I think what you're trying to achieve is to use the first lines to decide between the type of processing, then re-use those lines for your read_all_csv_WITH_processing or read_all_csv_without_processing, while still not loading the full csv file in-memory. To achieve that you can load the first lines in a list and concatenate that with the rest of the file with itertools.chain, like this:</p>

<pre><code>import itertools

top_lines = []
reader_iterator = csv.reader(fil)
do_heavy_processing = True
while True:
    # Can't use "for line in reader_iterator" directly, otherwise we're
    # going to close the iterator when going out of the loop after the first
    # N iterations
    line = reader_iterator.__next__()
    top_lines.append(line)
    if some_condition(line):
        do_heavy_processing = False
        break
    elif not_worth_going_further(line)
        break
full_file = itertools.chain(top_lines, reader_iterator)

if do_heavy_processing:
    read_all_csv_WITH_processing(full_file)
else:
    read_all_csv_without_processing(full_file)
</code></pre>

python csv