CountTokenizing a field, turning into columns
<p>I'm working with data that look something like this:</p>

<pre><code>ID      PATH         GROUP  
11937   MM-YT-UJ-OO  GT  
11938   YT-RY-LM     TQ  
11939   XX-XX-OT     DX  
</code></pre>

<p>I'd like to tokenize the PATH column into n-grams and then one-hot encode those into their own columns so I'd end up with something like:</p>

<pre><code>ID     GROUP   MM  YT  UJ  OO  RY  LM  XX  OT  MM-YT  YT-UH ...  
11937  GT      1   1   1   1   0   0   0   0   1      1
</code></pre>

<p>I could also use counted tokens rather than one-hot, so 11939 would have a 2 in the XX column instead of a 1, but I can work with either.</p>

<p>I can tokenize the column quite easily with scikitlearn CountVectorizer, but then I have to cbind the <code>ID</code> and <code>GROUP</code> fields.  Is there a standard way to do this or a best practice that anyone has discovered?</p>

<p>Maybe you can try something like that.</p>

<pre><code># Test data
df = DataFrame({'GROUP': ['GT', 'TQ', 'DX'],
 'ID': [11937, 11938, 11939],
 'PATH': ['MM-YT-UJ-OO', 'YT-RY-LM', 'XX-XX-OT']})

# Expanding data and creating on column by token
tmp = pd.concat([df.loc[:,['GROUP', 'ID']], 
                 df['PATH'].str.split('-', expand=True)], axis=1)
# Converting wide to long format
tmp = pd.melt(tmp, id_vars=['ID', 'GROUP'])
# Now grouping and counting
tmp.groupby(['ID', 'GROUP', 'value']).count().unstack().fillna(0)

#             variable                                   
# value             LM   MM   OO   OT   RY   UJ   XX   YT
# ID    GROUP                                            
# 11937 GT         0.0  1.0  1.0  0.0  0.0  1.0  0.0  1.0
# 11938 TQ         1.0  0.0  0.0  0.0  1.0  0.0  0.0  1.0
# 11939 DX         0.0  0.0  0.0  1.0  0.0  0.0  2.0  0.0
</code></pre>

python pandas numpy scikit-learn