How to read data in chunks in Python dataframe?
<p>I want to read the file f in chunks to a dataframe. Here is part of a code that I used.</p>

<pre><code>for i in range(0, maxline, chunksize):
df = pandas.read_csv(f,sep=',', nrows=chunksize, skiprows=i)
df.to_sql(member, engine, if_exists='append',index= False, index_label=None, chunksize=chunksize)
</code></pre>

<p>I get the error:</p>

<blockquote>
  <p>pandas.io.common.EmptyDataError: No columns to parse from file</p>
</blockquote>

<p>The code works only when the chunksize >= maxline (which is total lines in file f). However, in my case, the chunksize&lt;=maxline.</p>

<p>Please advise the fix.</p>

<p>I think it is better to use the parameter <code>chunksize</code> in <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html" rel="nofollow"><code>read_csv</code></a>. Also, use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html" rel="nofollow"><code>concat</code></a> with the parameter <code>ignore_index</code>, because of the need to avoid duplicates in <code>index</code>:</p>

<pre><code>chunksize = 5
TextFileReader = pd.read_csv(f, chunksize=chunksize)

df = pd.concat(TextFileReader, ignore_index=True)
</code></pre>

<p>See pandas <a href="http://pandas.pydata.org/pandas-docs/stable/io.html#iterating-through-files-chunk-by-chunk" rel="nofollow">docs</a>.</p>

python csv pandas dataframe chunks