dask / pandas categorical transformation differences
<p>I am managing larger than memory csv files of mostly categorical data. Initially I used to create a large csv file, then read it via Pandas read_csv, convert to categorical and save into hdf5. Once into categorical format, it nicely fits in memory.</p>

<p>Files are growing and I moved to Dask. Same process though.</p>

<p>However, in empty fields, Pandas seems to use np.nan and the category is not included in the cat.categories listing.</p>

<p>With Dask, empty values are filled with NaN, it is included as a separate category and whence saved into HDF I get future compatibility warning. </p>

<p>Is this a bug or am I missing any steps ? Behaviour seems to differ between pandas and dask.</p>

<p>Thanks</p>

<p>JC</p>

<p>This is solved in dask ver 0.11.1</p>

<p>See <a href="https://github.com/dask/dask/pull/1578" rel="nofollow">https://github.com/dask/dask/pull/1578</a></p>

python csv pandas dask