Read CSV file and only keep some lines according to values in list (Python)
<p>I will reformulate the problem I had previously stated:</p>

<p>I am currently trying to only read about 26 million rows from a file that has about 600 million. I currently have a list with those 26 million rows that I am interested. </p>

<p>My solution is as follows:</p>

<pre><code>## list_ is a list of indices with the number of the 26MM rows 

# First, open the output file where i want to copy the 26MM rows
with open(output_file,'w') as g:
# Open the source file with 600MM rows
  with open(source_file,'r') as f:
    for i,line in enumerate(f):
      if i in list_:
        g.write(line)
</code></pre>

<p>Given the size of the list and the size of the original file, I am afraid it might take too long to process this file. I am aware that this topic has been covered in other questions, but I don't think other posts have asked when the text files are very large.</p>

<p>Thank you and apologies for the previous confusing post,</p>

<p>If you only want to check whether a value is in that list, the best time is O(1) for each check. You probably want to use hashset instead of a list. You can google hashset in python to see some example, like <a href="http://stackoverflow.com/questions/26724002/contains-of-hashsetinteger-in-python">this</a> or see this doc, <a href="https://docs.python.org/2/library/sets.html" rel="nofollow">sets</a>.  </p>

python