Can dask parralelize reading fom a csv file?
<p>I'm converting a large textfile to a hdf storage in hopes of a faster data access. The conversion works allright, however reading from the csv file is not done in parallel. It is really slow (takes about 30min for a 1GB textfile on an SSD, so my guess is that it is not IO-bound). </p>

<p>Is there a way to have it read in multiple threads in parralel?
Sice it might be important, I'm currently forced to run under Windows -- just in case that makes any difference.</p>

<pre><code>from dask import dataframe as ddf
df = ddf.read_csv("data/Measurements*.csv",
             sep=';', 
             parse_dates=["DATETIME"], 
             blocksize=1000000,
             )

df.categorize([ 'Type',
                'Condition',               
          ])

df.to_hdf("data/data.hdf", "Measurements", 'w')
</code></pre>

<p>Yes, dask.dataframe can read in parallel.  However you're running into two problems:</p>

<h3>Pandas.read_csv only partially releases the GIL</h3>

<p>By default dask.dataframe parallelizes with threads because most of Pandas can run in parallel in multiple threads (releases the GIL).  Pandas.read_csv is an exception, especially if your resulting dataframes use object dtypes for text</p>

<h3>dask.dataframe.to_hdf(filename) forces sequential computation</h3>

<p>Writing to a single HDF file will force sequential computation (it's very hard to write to a single file in parallel.)</p>

<h3>Solution</h3>

<p>Because your dataset likely fits in memory, use dask.dataframe.read_csv to load in parallel with multiple processes, then switch immediately to Pandas.</p>

<pre><code>import dask.dataframe as dd
import dask.multiprocessing

df = df = ddf.read_csv("data/Measurements*.csv",  # read in parallel
             sep=';', 
             parse_dates=["DATETIME"], 
             blocksize=1000000,
             )

df = df.compute(get=dask.multiprocessing.get)     # convert to pandas

df['Type'] = df['Type'].astype('category')
df['Condition'] = df['Condition'].astype('category')

df.to_hdf('data/data.hdf', 'Measurements', format='table', mode='w')
</code></pre>

python csv pandas dask