Pandas Split 9GB CSV into 2 5GB CSVs
<p>I have a 9GB CSV and need to split it into 2 5GB CSVs.
I started out doing this:</p>

<pre><code>for i, chunk in enumerate(pd.read_csv('csv_big_file2.csv',chunksize=100000)):
    chunk.drop('Unnamed: 0',axis=1,inplace=True)
    chunk.to_csv('chunk{}.csv'.format(i),index=False)
</code></pre>

<p>What I need to do is somehow tell pandas to write the chunk to a CSV until that CSV reaches a size of 6,250,000,000 (or a filesize of 5GB) then start a new CSV file with the rest of the data (without starting again from the beginning of the data from the big CSV file).</p>

<p>Can this be done?</p>

<p>Thanks in advance!</p>

<p>Give this a try. </p>

<pre><code>for i, chunk in enumerate(pd.read_csv('csv_big_file2.csv',chunksize=312500)):
    if i&lt;11:
        chunk.to_csv(file_name, chunksize = 312500)
    else 
        chunk.to_csv(file_name_2, chunksize = 312500)
</code></pre>

python python-3.x csv pandas