Dummy creation in pipeline with different levels in train and test set
<p>I'm currently exploring the scikit learn pipelines.
I also want to preprocess the data with a pipeline.
However, my train and test data have different levels of the categorical variable.
Example:
Consider:</p>

<pre><code>import pandas as pd
train = pd.Series(list('abbaa'))
test = pd.Series(list('abcd'))
</code></pre>

<p>I wrote a TransformerMixinClass using pandas</p>

<pre><code>class CreateDummies(TransformerMixin):

def transform(self, X, **transformparams):
    return pd.get_dummies(X).copy()

def fit(self, X, y=None, **fitparams):
    return self
</code></pre>

<p>fit_transform yields for the train data 2 columns and for the test data 4 columns. So no surprise here, but not suitable for a pipeline</p>

<p>Similary, I tried to import the label encoder (and OneHotEncoder for the potential next steps):</p>

<pre><code>from sklearn.preprocessing import LabelEncoder, OneHotEncoder
le = LabelEncoder()
le.fit_transform(train)
le.transform(test)
</code></pre>

<p>which yields, not surprisingly, an error.</p>

<p>So the problem here is that I need some information contained in the test set.
Is there a good way to include this in a pipeline?</p>

<p>You can use categoricals as explained in <a href="http://stackoverflow.com/a/37451867/2285236">this answer</a>:</p>

<pre><code>categories = np.union1d(train, test)
train = train.astype('category', categories=categories)
test = test.astype('category', categories=categories)

pd.get_dummies(train)
Out: 
   a  b  c  d
0  1  0  0  0
1  0  1  0  0
2  0  1  0  0
3  1  0  0  0
4  1  0  0  0

pd.get_dummies(test)
Out: 
   a  b  c  d
0  1  0  0  0
1  0  1  0  0
2  0  0  1  0
3  0  0  0  1
</code></pre>

python pandas scikit-learn