Python Pandas: Retrieve id of data from a chunk
<p>The dataset is read chunk by chunk, because it is to big. The ids are the first column and I would like to store them in data structure like array. So far it is not working. It looks like this</p>

<pre><code>tf = pd.read_csv('data.csv', chunksize=chunksize)
for chunk in tf:
    here I wanna store the ids chunk["Id"] in an array
</code></pre>

<p>How do I do that?</p>

<p>IIUC you can do it this way:</p>

<pre><code>ids = pd.DataFrame()
tf = pd.read_csv('data.csv', chunksize=chunksize)
for chunk in tf:
    ids = pd.concat([ids, chunk['Id']], ignore_index=True)
</code></pre>

<p>you can always access <code>ids</code> Series as NumPy array:</p>

<pre><code>ids.values 
</code></pre>

python python-3.x pandas