sklearn DictVectorizer(sparse=False) with a different default value, Impute a constant
<p>I'm building a pipeline that starts with a <code>DictVectorizer</code> that produces a sparse matrix. Specifying <code>sparse=True</code> changes the output from a scipy sparse matrix to a numpy dense matrix which is good, but the next stages in the pipeline complain about <code>NaN</code> values, which our obvious outcome of using the <code>DictVectorizer</code> in my case. I'd like the pipeline to consider missing dictionary values not as not available but as zero.</p>

<p><code>Imputer</code> doesn't help me as far as I can see, because I want to "impute" with a constant value and not a statistical value dependant of other values of the column.</p>

<p>Following is the code I've been using:</p>

<pre><code>vectorize = skl.feature_extraction.DictVectorizer(sparse=False)
variance = skl.feature_selection.VarianceThreshold()
knn = skl.neighbors.KNeighborsClassifier(4, weights='distance', p=1)

pipe = skl.pipeline.Pipeline([('vectorize', vectorize),
                            # here be dragons ('fillna', ),
                            ('variance', variance),
                            ('knn', knn)])
pipe.fit(dict_data, labels)
</code></pre>

<p>And some mocked dictionaries:</p>

<pre><code>dict_data = [{'city': 'Dubai', 'temperature': 33., 'assume_zero_when_missing': 7},
             {'city': 'London', 'temperature': 12.},
             {'city': 'San Fransisco', 'temperature': 18.}]
</code></pre>

<p>Notiec that in this example, <code>assume_zero_when_missing</code> is missing from most dictionaries, which will lead later estimators to complain about <code>NaN</code> values:</p>

<blockquote>
  <p>ValueError: Input contains NaN, infinity or a value too large for dtype('float64').</p>
</blockquote>

<p>While the result I'm hoping for is that <code>NaN</code> values will be replaced with <code>0</code>.</p>

<p>You could fill the <code>NaNs</code> with 0's after converting your <code>list</code> of <code>dictionaries</code> to a pandas <code>dataframe</code> using <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html" rel="nofollow"><code>DF.fillna</code></a> as shown:</p>

<pre><code>df = pd.DataFrame(dict_data)
df.fillna(0, inplace=True)
</code></pre>

<hr>

<p>Inorder to use it as steps inside the pipeline estimator, you could write a custom class implementing the <code>fit</code> and <code>transform</code> methods yourself as shown:</p>

<pre><code>class FillingNans(object):
    '''
    Custom function for assembling into the pipeline object 
    '''
    def transform(self, X):
        nans_replaced = X.fillna(0)
        return nans_replaced

    def fit(self, X, y=None):
        return self
</code></pre>

<p>Then, you could modify the manual feature selection steps in pipeline as shown:</p>

<pre><code>pipe = skl.pipeline.Pipeline([('vectorize', vectorize),
                             ('fill_nans', FillingNans()),
                             ('variance', variance),
                             ('knn', knn)])
</code></pre>

python pandas dataframe machine-learning scikit-learn