Read in csv file faster
<p>I am currently reading in a large csv file (around 100 million lines), using command along the lines of that described in <a href="https://docs.python.org/2/library/csv.html" rel="nofollow">https://docs.python.org/2/library/csv.html</a> e.g. :</p>

<pre><code>import csv
with open('eggs.csv', 'rb') as csvfile:
     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
     for row in spamreader:
          process_row(row)
</code></pre>

<p>This is proving rather slow, I suspect because each line is read in individually (requiring lots of read calls to the hard drive). Is there any way of reading the whole csv file in at once, and then iterating over it? Although the file itself is large in size (e.g. 5Gb), my machine has sufficient ram to hold that in memory.</p>

<pre><code>import pandas as pd
df =pd.DataFrame.from_csv('filename.csv')
</code></pre>

<p>This will read it in as a pandas dataframe so you can do all sorts of fun things with it</p>

python csv