sklearn FeatureHasher parallelized
<p>sklearn's <code>Featurehasher</code> feature extractor has several advantages compared to its <code>DictVectorizer</code> counter-part, thanks to using the hashing trick.</p>

<p>One advantage which seems harder to tap is its ability to run in parallel.</p>

<p>My question is, how can I easily make <code>FeatureHasher</code> run in parallel?</p>

<p>You could implement a parallel version of <code>FeatureHasher.transform</code> using <code>joblib</code> (the library favoured by scikit-learn for parallel processing):</p>

<pre><code>from sklearn.externals.joblib import Parallel, delayed
import numpy as np
import scipy.sparse as sp

def transform_parallel(self, X, n_jobs):
    transform_splits = Parallel(n_jobs=n_jobs, backend="threading")(
        delayed(self.transform)(X_split)
        for X_split in np.array_split(X, n_jobs))

    return sp.vstack(transform_splits)

FeatureHasher.transform_parallel = transform_parallel
f = FeatureHasher()
f.transform_parallel(np.array([{'a':3,'b':2}]*10), n_jobs=5)

&lt;10x1048576 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
    with 20 stored elements in Compressed Sparse Row format&gt;
</code></pre>

python machine-learning scikit-learn feature-extraction