Pandas sklearn one-hot encoding dataframe or numpy?
<p>How can I transform a pandas data frame to sklearn one-hot-encoded (dataframe / numpy array) where some columns do not require encoding?</p>

<pre><code>mydf = pd.DataFrame({'Target':[0,1,0,0,1, 1,1],
                   'GroupFoo':[1,1,2,2,3,1,2],
                    'GroupBar':[2,1,1,0,3,1,2],
                    'GroupBar2':[2,1,1,0,3,1,2],
                    'SomeOtherShouldBeUnaffected':[2,1,1,0,3,1,2]})
columnsToEncode = ['GroupFoo', 'GroupBar']
</code></pre>

<p>Is an already label encoded data frame and I would like to only encode the columns marked by <code>columnsToEncode</code>? </p>

<p>My problem is that I am unsure if a <code>pd.Dataframe</code> or the <code>numpy</code> array representation are better and how to re-merge the encoded part with the other one.</p>

<p>My attempts so far:</p>

<pre><code>myEncoder = OneHotEncoder(sparse=False, handle_unknown='ignore')
myEncoder.fit(X_train)
df = pd.concat([
         df[~columnsToEncode], # select all other / numeric
        # select category to one-hot encode
         pd.Dataframe(encoder.transform(X_train[columnsToEncode]))#.toarray() # not sure what this is for
        ], axis=1).reindex_axis(X_train.columns, axis=1)
</code></pre>

<p>Notice: I am aware of <a href="http://stackoverflow.com/questions/36285155/pandas-get-dummies">Pandas: Get Dummies</a> / <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html" rel="nofollow">http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html</a> but that does not play well in a train / test split where I require such an encoding per fold.</p>

<p>This library provides several categorical encoders which make sklearn / numpy play nicely with pandas <a href="https://github.com/wdm0006/categorical_encoding" rel="nofollow">https://github.com/wdm0006/categorical_encoding</a></p>

<p>However, they do not yet support "handle unknown category"</p>

<p>for now I will use</p>

<pre><code>myEncoder = OneHotEncoder(sparse=False, handle_unknown='ignore')
myEncoder.fit(df[columnsToEncode])

pd.concat([df.drop(columnsToEncode, 1),
          pd.DataFrame(myEncoder.transform(df[columnsToEncode]))], axis=1).reindex()
</code></pre>

<p>As this supports unknown datasets. For now, I will stick with half-pandas half-numpy because of the nice pandas labels. for the numeric columns.</p>

python pandas numpy scikit-learn dummy-variable