DictVectorizer with a large dataset
<p>I have a large dataset with categorical values and tried to encode them using <code>DictVectorizer</code>. The following is a snippet of the code I tried.</p>

<pre><code>dv = DictVectorizer(sparse=True)
_dicts = []
for line in fp:
    _dict = create_dict_feature(line)
    _dicts.append(_dict)
dv.fit_transform(_dicts)
</code></pre>

<p>But, <code>MemoryError</code> occurs in <code>_dicts.append(_dict)</code>. I am wondering what would be an efficient way of getting around this problem. </p>

<p>According to the docs, <code>fit_transform</code> can take an iterable. If the memory issue is coming from the size of the list, consider using a generator instead of a <code>list</code>, which will yield your <code>dict</code>s one at a time as it is iterated.
</p>

<pre><code>_dicts = (create_dict_feature(line) for line in fp)
dv = DictVectorizer(sparse=True)
dv.fit_transform(_dicts)
</code></pre>

<p>This won't help much if <code>fit_transform</code> accumulates the <code>dict</code>s or <code>Mapping</code>s just as you were doing before.</p>

python scikit-learn