python: efficient way to check only a couple of rows in a csvreader iterator?
<p>I have a (very large) CSV file that looks something like this:</p>

<pre><code>header1,header2,header3
name0,rank0,serial0
name1,rank1,serial1
name2,rank2,serial2
</code></pre>

<p>I've written some code that processes the file, and writes it out (using csvwriter) modified as such, with some information I compute appended to the end of each row:</p>

<pre><code>header1,header2,header3,new_hdr4,new_hdr5
name0,rank0,serial0,salary0,base0
name1,rank1,serial1,salary1,base1
name2,rank2,serial2,salary2,base2
</code></pre>

<p>What I'm trying to do is structure the script so that it auto-detects whether or not the CSV file it's reading has <strong><em>already been processed</em></strong>.  If it has been processed, I can skip a lot of expensive calculations later.  I'm trying to understand whether there is a reasonable way of doing this within the reader loop.  I <em>could</em> just open the file once, read in enough to do the detection, and then close and reopen it with a flag set, but this seems hackish.</p>

<p>Is there a way to do this within the same reader?  The logic is something like:</p>

<pre><code>read first N lines   ###(N is small)
   if (some condition)
      already_processed = TRUE
      read_all_csv_without_processing
   else
      read_all_csv_WITH_processing
</code></pre>

<p>I can't just use the iterator that reader gives me, because by the time I've gotten enough lines to do my conditional check, I don't have any good way to go back to the beginning of the CSV.  Is closing and reopening it really the most elegant way to do this?</p>

<p>I will outline what I consider a much better approach. I presume this is happening over various runs. What you need to do is persist the files seen between runs and only process what has not been seen:</p>

<pre><code>import pickle
import glob


def process(fle):
    # read_all_csv_with_processing

def already_process(fle):
    # read_all_csv_without_processing
try:
    # if it exists, we ran the code previously.
  with open("seen.pkl", "rb") as f:
     seen = pickle.load(f)
except IOError as  e:
    # Else first run, so just create the set.
    print(e)
    seen = set()

for file in glob.iglob("path_where_files_are/*.csv"):
    # if not seen before, just process
    if file not in seen:
        process(file)
    else:
        # already processed so just do whatever
        already_process(file)
    seen.add(file)

# persist the set.
with open("seen.pkl", "w") as f:
    pickle.dumps(seen, f)
</code></pre>

<p>Even if for some strange reason you somehow process the same files in the same run, all you need to do then is implement the seen set logic.</p>

<p>Another alternative would be to use a unique marker in the file that you add at the start if processed. </p>

<pre><code># something here
header1,header2,header3,new_hdr4,new_hdr5
name0,rank0,serial0,salary0,base0
name1,rank1,serial1,salary1,base1
name2,rank2,serial2,salary2,base2
</code></pre>

<p>Then all you would need to process is the very first line. Also if you want to get the first n lines from a file even if you wanted to <a href="http://stackoverflow.com/questions/38064374/iterate-from-a-certain-row-of-a-csv-file-in-python/38064407#38064407">start from a certain row</a>, use <a href="https://docs.python.org/2/library/itertools.html" rel="nofollow">itertools.islce</a></p>

<p>To be robust you might want to wrap your code in a try/finally in case it errors so you don't end up going over the same files already processed on the next run:</p>

<pre><code>try:
    for file in glob.iglob("path_where_files_are/*.csv"):
        # if not seen before, just process
        if file not in seen:
            process(file)
        else:
            # already processed so just do whatever
            already_process(file)
        seen.add(file)
finally: 
    # persist the set.
    with open("seen.pkl", "wb") as f:
        pickle.dump(seen, f)
</code></pre>

python csv