How to read a compressed (gz) CSV file inro a dask Dataframe?
<p>Is there a way to read a .csv file that is compressed via gz into a dask dataframe?</p>

<p>I've tried it directly with</p>

<pre><code>import dask.dataframe as dd
df = dd.read_csv("Data.gz" )
</code></pre>

<p>but get an unicode error (probably because it is interpreting the compressed bytes) There is a <code>"compression"</code> parameter but <code>compression = "gz"</code> won't work and I can't find any documentation so far.</p>

<p>With pandas I can read the file directly without a problem other than the result blowing up my memory ;-) but if I restrict the number of lines it works fine.</p>

<pre><code>import pandas.Dataframe as pd
df = pd.read_csv("Data.gz", ncols=100)
</code></pre>

<p>Without the file it's difficult to say. what if you set the encoding <code>like # -*- coding: latin-1 -*-</code>? or since <code>read_csv</code> is based off of Pandas, you may even <code>dd.read_csv('Data.gz', encoding='utf-8')</code>. Here's the list of Python encodings: <a href="https://docs.python.org/3/library/codecs.html#standard-encodings" rel="nofollow">https://docs.python.org/3/library/codecs.html#standard-encodings</a></p>

python csv pandas dask