Pandas - is it possible to "rewind" read_csv with chunk= argument?
<p>I am dealing with a big dataset, therefore to read it in pandas I use <code>read_csv</code> with <code>chunk=</code> option.</p>

<pre><code>data = pd.read_csv("dataset.csv", chunksize=2e5)
</code></pre>

<p>then I operate on the chunked DataFrame in the following way</p>

<pre><code>any_na_cols = [chunk.do_something() for chunk in data]
</code></pre>

<p>the problem is, when I want to do something else in the same way as above, I will get an empty result because I have iterated over chunked DataFrame already. Therefore I would have to call <code>data = pd.read_csv("dataset.csv", chunksize=2e5)</code> again to perform next operation.</p>

<p>Most likely there is no problem with that, but for some reason I feel that this approach is inelegant in some way. Isn't there a method like <code>data.rewind()</code> or something similar that would enable me to iterate through the chunks again? I could not find anything like that in the Documentation. Or maybe I am comitting some design mistake with that approach?</p>

<p>I don't think it's a good idea to read your CSV <strong>again</strong> - you will double the number of IOs. It's better to "do something else" during the same iteration:</p>

<pre><code>any_na_cols = pd.DataFrame()

for chunk in pd.read_csv("dataset.csv", chunksize=2e5)
    any_na_cols = pd.concat([any_na_cols, chunk.do_something()], ignore_index=True)
    # do something else
</code></pre>

python pandas chunks chunking