imputing missing values using a predictive model
<p>I am trying to impute missing values in Python and <code>sklearn</code> does not appear to have a method beyond average (mean, median, or mode) imputation. <a href="http://docs.orange.biolab.si/2/reference/rst/Orange.feature.imputation.html#Orange.feature.imputation.Model" rel="nofollow">Orange imputation model</a> seems to provide a viable option. However, it appears <code>Orange.data.Table</code> is not recognizing <code>np.nan</code> or somehow the imputation is failing.</p>

<pre><code>import Orange
import numpy as np

tmp = np.array([[1, 2, np.nan, 5, 8, np.nan], [40, 4, 8, 1, 0.2, 9]])
data = Orange.data.Table(tmp)
imputer = Orange.feature.imputation.ModelConstructor()
imputer.learner_continuous = Orange.classification.tree.TreeLearner(min_subset=20)
imputer = imputer(data )
impdata = imputer(data)
for i in range(0, len(tmp)):
    print impdata[i]
</code></pre>

<p>Output is </p>

<pre><code>[1.000, 2.000, 1.#QO, 5.000, 8.000, 1.#QO]
[40.000, 4.000, 8.000, 1.000, 0.200, 9.000]
</code></pre>

<p>Any idea what I am missing? Thanks!</p>

<p>It seems the issue is that a missing value in Orange is represented as <code>?</code> or <code>~</code>. Oddly enough, the <code>Orange.data.Table(numpy.ndarray)</code> constructor does not infer that <code>numpy.nan</code> should be converted to <code>?</code> or <code>~</code> and instead converts them to <code>1.#QO</code>. The custom function below, <code>pandas_to_orange()</code>, addresses this problem. </p>

<pre><code>import Orange
import numpy as np
import pandas as pd

from collections import OrderedDict

# Adapted from https://github.com/biolab/orange3/issues/68

def construct_domain(df):
    columns = OrderedDict(df.dtypes)

    def create_variable(col):
        if col[1].__str__().startswith('float'):
            return Orange.feature.Continuous(col[0])
        if col[1].__str__().startswith('int') and len(df[col[0]].unique()) &gt; 50:
            return Orange.feature.Continuous(col[0])
        if col[1].__str__().startswith('date'):
            df[col[0]] = df[col[0]].values.astype(np.str)
        if col[1].__str__() == 'object':
            df[col[0]] = df[col[0]].astype(type(""))
        return Orange.feature.Discrete(col[0], values = df[col[0]].unique().tolist())
    return Orange.data.Domain(list(map(create_variable, columns.items())))

def pandas_to_orange(df):
    domain = construct_domain(df)
    df[pd.isnull(df)]='?'
    return Orange.data.Table(Orange.data.Domain(domain), df.values.tolist())

df = pd.DataFrame({'col1':[1, 2, np.nan, 4, 5, 6, 7, 8, 9, np.nan, 11], 
                    'col2': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110.]}) 

tmp = pandas_to_orange(df)
for i in range(0, len(tmp)):
    print tmp[i]
</code></pre>

<p>The output is:</p>

<pre><code>[1.000, 10.000]
[2.000, 20.000]
[?, 30.000]
[4.000, 40.000]
[5.000, 50.000]
[6.000, 60.000]
[7.000, 70.000]
[8.000, 80.000]
[9.000, 90.000]
[?, 100.000]
[11.000, 110.000]
</code></pre>

<p>The reason I wanted to properly encode the missing values is so I can use the Orange imputation library. However, it appears that the predictive tree model in the library does not do much more than simple mean imputation. Specifically, it imputes the same value for all missing values. </p>

<pre><code>imputer = Orange.feature.imputation.ModelConstructor()
imputer.learner_continuous = Orange.classification.tree.TreeLearner(min_subset=20)
imputer = imputer(tmp )
impdata = imputer(tmp)
for i in range(0, len(tmp)):
    print impdata[i]
</code></pre>

<p>Here's the output:</p>

<pre><code>[1.000, 10.000]
[2.000, 20.000]
[5.889, 30.000]
[4.000, 40.000]
[5.000, 50.000]
[6.000, 60.000]
[7.000, 70.000]
[8.000, 80.000]
[9.000, 90.000]
[5.889, 100.000]
[11.000, 110.000]
</code></pre>

<p>I was looking for something that will fit a model, say kNN, on the complete cases and use the fitted model to predict the missing cases. <a href="https://pypi.python.org/pypi/fancyimpute" rel="nofollow"><code>fancyimpute</code> (a Python 3 package)</a> does this but throws <code>MemoryError</code> on my 300K+ input.</p>

<pre><code>from fancyimpute import KNN

df = pd.DataFrame({'col1':[1, 2, np.nan, 4, 5, 6, 7, 8, 9, np.nan, 11], 
                    'col2': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110.]}) 

X_filled_knn = KNN(k=3).complete(df)
X_filled_knn
</code></pre>

<p>Output is:</p>

<pre><code>array([[   1.        ,   10.        ],
       [   2.        ,   20.        ],
       [   2.77777784,   30.        ],
       [   4.        ,   40.        ],
       [   5.        ,   50.        ],
       [   6.        ,   60.        ],
       [   7.        ,   70.        ],
       [   8.        ,   80.        ],
       [   9.        ,   90.        ],
       [   9.77777798,  100.        ],
       [  11.        ,  110.        ]])
</code></pre>

<p>I can probably find a workaround or split the dataset into chunks (not ideal).</p>

python python-2.7 scikit-learn orange imputation