Is it possible to use DictVectorizer on chunked data?
<p>I am trying to import chunked data using python pandas csv reader,to overcome memory error, and use DicVectorizer to transform string to float dtypes. But I could see two different strings are having same codes after transformation. Do we have alternative/option to do the data type transformation on chunked data?</p>

<p>In Pandas 0.19, you can declare columns as Categorial in read_csv. See <a href="http://pandas.pydata.org/pandas-docs/version/0.19.0/whatsnew.html#whatsnew-0190-enhancements-read-csv-categorical" rel="nofollow">documentaion</a>.</p>

<p>So as an example for the doc, you can type a column named <code>col1</code> in your csv like this and reduce memory footprint:</p>

<pre><code>pd.read_csv(StringIO(data), dtype={'col1': 'category'})
</code></pre>

python pandas