Pandas Split 9GB CSV into 2 5GB CSVs
<p>I have a 9GB CSV and need to split it into 2 5GB CSVs.
I started out doing this:</p>

<pre><code>for i, chunk in enumerate(pd.read_csv('csv_big_file2.csv',chunksize=100000)):
    chunk.drop('Unnamed: 0',axis=1,inplace=True)
    chunk.to_csv('chunk{}.csv'.format(i),index=False)
</code></pre>

<p>What I need to do is somehow tell pandas to write the chunk to a CSV until that CSV reaches a size of 6,250,000,000 (or a filesize of 5GB) then start a new CSV file with the rest of the data (without starting again from the beginning of the data from the big CSV file).</p>

<p>Can this be done?</p>

<p>Thanks in advance!</p>

<p>Solution is a little messy. But this should split the data based on the ~6 billion row threshold you mentioned. </p>

<pre><code>import pandas as pd  
from __future__ import division

numrows = 6250000000 #number of rows threshold to be 5 GB
count = 0 #keep track of chunks
chunkrows = 100000 #read 100k rows at a time
df = pd.read_csv('csv_big_file2.csv', iterator=True, chunksize=chunkrows) 
for chunk in df: #for each 100k rows
    if count &lt;= numrows/chunkrows: #if 5GB threshold has not been reached 
        outname = "csv_big_file2_1stHalf.csv"
    else:
        outname = "csv_big_file2_2ndHalf.csv"
    #append each output to same csv, using no header
    chunk.to_csv(outname, mode='a', header=None, index=None)
    count+=1
</code></pre>

python python-3.x csv pandas