Generate features from "comments" column in dataframe
<p>I have a dataset with a column that has comments. This comments are words separated by commas.</p>

<p>df_pat['reason'] = </p>

<ol>
<li>chest pain</li>
<li>chest pain, dyspnea</li>
<li>chest pain, hypertrophic obstructive cariomyop...</li>
<li>chest pain </li>
<li>chest pain</li>
<li>cad, rca stents</li>
<li>non-ischemic cardiomyopathy, chest pain, dyspnea</li>
</ol>

<p>I would like to generate separated columns in the dataframe so that a column represent each word from all the set of words, and then have 1 or 0 to the rows where I initially had that word in the comment.</p>

<p>For example:
df_pat['chest_pain'] =
  1
  1
  1
  1
  1
  1
  0
  1</p>

<p>df_pat['dyspnea'] = 
  0
  1
  0
  0
  0
  0
  1</p>

<p>And so on...</p>

<p>Thank you!</p>

<p><code>sklearn.feature_extraction.text</code> has something for you! It looks like you may be trying to predict something. If so - and if you're planning to use sci-kit learn at some point, then you can bypass making a dataframe with len(set(words)) number of columns and just use <code>CountVectorizer</code>. This method will return a matrix with dimensions (rows, columns) = (number of rows in dataframe, number of unique words in entire <code>'reason'</code> column). </p>

<pre><code>from sklearn.feature_extraction.text import CountVectorizer

df = pd.DataFrame({'reason': ['chest pain', 'chest pain, dyspnea', 'chest pain, hypertrophic obstructive cariomyop', 'chest pain', 'chest pain', 'cad, rca stents', 'non-ischemic cardiomyopathy, chest pain, dyspnea']})

# turns body of text into a matrix of features
# split string on commas instead of spaces
vectorizer = CountVectorizer(tokenizer = lambda x: x.split(","))

# X is now a n_documents by n_distinct_words-dimensioned matrix of features
X = vectorizer.fit_transform(df['reason'])
</code></pre>

<p><code>pandas</code> plays really nicely with <code>sklearn</code>.</p>

<p>Or, a strict <code>pandas</code> solution that should probably be vectorized, but if you don't have that much data, should work:</p>

<pre><code># split on the comma instead of spaces to get "chest pain" instead of "chest" and "pain"
reasons = [reason for case in df['reason'] for reason in case.split(",")]

for reason in reasons:
    for idx in df.index:
        if reason in df.loc[idx, 'reason']:
            df.loc[idx, reason] = 1
        else:
            df.loc[idx, reason] = 0
</code></pre>

python text dataframe comments