How to use `.split()` for importing tab-delimited text from a large `gzip` file? Chunks?
<p>I have huge <code>gzip</code> file (several GB) of tab-delimited text which I would like to parse into a pandas dataframe. </p>

<p>If the contents of this file were text, one would simply use <code>.split()</code>, e.g. </p>

<pre><code>file_text = """abc   123   cat   456   dog   678   bird   111   fish   ...
moon   1969    revolution    1789   war   1927   reformation    1517    maxwell   ..."""

data = [line.split() for line in file_text.split('\n')]
</code></pre>

<p>and then you could put the data into a pandas dataframe using</p>

<pre><code>import pandas as pd
df = pd.DataFrame(data)
</code></pre>

<p>However, this isn't a text document. It is a tab-delimited file in a gzip, with several GB of data. What is the most efficient way to parse this data into a dataframe, using <code>.split()</code>? </p>

<p>I guess the first step would be to use </p>

<pre><code>import gzip
with gzip.open(filename, 'r') as f:
    file_content = f.read()
</code></pre>

<p>and use <code>.split()</code> on <code>file_content</code>, but saving all GB to a single variable and then splitting would be inefficient. Is it possible to do this in "chunks"? </p>

<p><code>read_csv()</code> supports <code>GZIP</code>ped files, so you can simply do the following:</p>

<pre><code>for chunk in pd.read_csv('/path/to/file.csv.gz', sep='\s*', chunksize=10**5):
    # process chunk DF
</code></pre>

<p>if you are sure that you have a TSV (<strong>TAB</strong> separated file), you can use <code>sep='\t'</code></p>

python pandas dataframe split gzip