read data from a huge CSV file efficiently
<p>I was trying to process my huge CSV file (more than 20G), but the process was killed when reading the whole CSV file into memory. To avoid this issue, I am trying to read the second column line by line. </p>

<p>For example, the 2nd column contains data like</p>

<ol>
<li>xxx, computer is good</li>
<li><p>xxx, build algorithm</p>

<pre><code>import collections

wordcount = collections.Counter()

with open('desc.csv', 'rb') as infile:
    for line in infile:
         wordcount.update(line.split())
</code></pre></li>
</ol>

<p>My code is working for the whole columns, how to only read the second column without using CSV reader?</p>

<p>As far as I know, calling <code>csv.reader(infile)</code> opens and reads the whole file...which is where your problem lies.</p>

<p>You can just read line-by-line and parse manually:   </p>

<pre><code>X=[]

with open('desc.csv', 'r') as infile:    
   for line in infile:
      # Split on comma first
      cols = [x.strip() for x in line.split(',')]

      # Grab 2nd "column"
      col2 = cols[1]

      # Split on spaces
      words = [x.strip() for x in col2.split(' ')]
      for word in words:     
         if word not in X:
            X.append(word)

for w in X:
   print w
</code></pre>

<p>That will keep a smaller chunk of the file in memory at a given time (one line).  However, you may still potentially have problems with variable <code>X</code> increasing to quite a large size, such that the program will error out due to memory limits.  Depends how many unique words are in your "vocabulary" list</p>

python csv