How to resolve memory issue of pandas while reading big csv files
<p>I have a 100GB csv file with millions of rows. I need to read, say, 10,000 rows at a time in pandas dataframe and write that to the SQL server in chunks. </p>

<p>I used chunksize as well as iteartor as suggested on <a href="http://pandas-docs.github.io/pandas-docs-travis/io.html#iterating-through-files-chunk-by-chunk" rel="nofollow">http://pandas-docs.github.io/pandas-docs-travis/io.html#iterating-through-files-chunk-by-chunk</a>, and have gone through many similar questions,but I am still getting the out of memory error. </p>

<p>Can you suggest a code to read very big csv files in pandas dataframe iteratively?</p>

<p>Demo:</p>

<pre><code>for chunk in pd.read_csv(filename, chunksize=10**5):
    chunk.to_sql('table_name', conn, if_exists='append')
</code></pre>

<p>where <code>conn</code> is a SQLAlchemy engine (created by <code>sqlalchemy.create_engine(...)</code>)</p>

python csv pandas dataframe iterator