How to impute each categorical column in numpy array
<p>There are good solutions to impute panda dataframe. But since I am working mainly with numpy arrays, I have to create new panda DataFrame object, impute and then convert back to numpy array as follows:</p>

<pre><code>nomDF=pd.DataFrame(x_nominal) #Convert np.array to pd.DataFrame
nomDF=nomDF.apply(lambda x:x.fillna(x.value_counts().index[0])) #replace NaN with most frequent in each column
x_nominal=nomDF.values #convert back pd.DataFrame to np.array
</code></pre>

<p>Is there a way to directly impute in numpy array?</p>

<p>We could use <a href="http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.stats.mstats.mode.html" rel="nofollow"><code>Scipy's mode</code></a> to get the highest value in each column. Leftover work would be to get the <code>NaN</code> indices and replace those in input array with the <code>mode</code> values by indexing. </p>

<p>So, the implementation would look something like this -</p>

<pre><code>from scipy.stats import mode

R,C = np.where(np.isnan(x_nominal))
vals = mode(x_nominal,axis=0)[0].ravel()
x_nominal[R,C] = vals[C]
</code></pre>

<p>Please note that for <code>pandas</code>, with <code>value_counts</code>, we would be choosing the highest value in case of many categories/elements with the same highest count. i.e. in tie situations. With <code>Scipy's mode</code>, it would be lowest one for such tie cases.</p>

<p>If you are dealing with such mixed dtype of <code>strings</code> and <code>NaNs</code>, I would suggest few modifications, keeping the last step unchanged to make it work -</p>

<pre><code>x_nominal_U3 = x_nominal.astype('U3')
R,C = np.where(x_nominal_U3=='nan')
vals = mode(x_nominal_U3,axis=0)[0].ravel()
</code></pre>

<p>This throws a warning for the mode calculation : <code>RuntimeWarning: The input array could not be properly checked for nan values. nan values will be ignored.
  "values. nan values will be ignored.", RuntimeWarning)</code>. But since, we actually want to ignore <code>NaNs</code> for that mode calculation, we should be okay there.</p>

python pandas numpy