Unable to extract unique words from a CSV
<p>I have a <code>csv</code> which looks like below:</p>

<pre><code>    Description
0   ['boy']
1   ['boy', 'jumped', 'roof']
2   ['paris']
3   ['paris', 'beautiful', 'new', 'york']
4   ['lets', 'go', 'party']
5   ['refused', 'come', 'party']
</code></pre>

<p>I need to find out unique words from this data. So output would be:</p>

<pre><code>    Unique Words
0   boy
1   jumped
2   roof
3   paris
4   beautiful
5   new
6   york
</code></pre>

<p>as so on. I am trying to do this using Pandas and Python and unable to achieve it. My code is:</p>

<pre><code>df = pd.read_csv('output.csv')
list(set(df.Description))
g = list(df['Description'].unique())
print(g)
</code></pre>

<p>This throws wrong output, it just throws the original csv dataframe.</p>

<p>You can first need convert <code>string</code> column to <code>list</code>, I use <code>ast.literal_eval</code>. Then make flat list of lists by list comprehension, use <code>set</code> and last create new <code>DataFrame</code> by constructor:</p>

<pre><code>import ast

print (type(df.ix[0, 'Description']))
&lt;class 'str'&gt;

df.Description = df.Description.apply(ast.literal_eval)

print (type(df.ix[0, 'Description']))
&lt;class 'list'&gt;

#http://stackoverflow.com/q/952914/2901002
unique_data = list(set([item for sublist in df.Description.tolist() for item in sublist]))
print (unique_data)
['refused', 'jumped', 'go', 'roof', 'come', 'beautiful',
 'paris', 'york', 'lets', 'new', 'boy', 'party']

print (pd.DataFrame({'Unique Words': unique_data}))
   Unique Words
0       refused
1        jumped
2            go
3          roof
4          come
5     beautiful
6         paris
7          york
8          lets
9           new
10          boy
11        party
</code></pre>

<p>Another solution without <code>ast</code>:</p>

<pre><code>df.Description = df.Description.str.strip('[]').str.split(',')
print (df)
                                Description
0                                   ['boy']
1               ['boy',  'jumped',  'roof']
2                                 ['paris']
3  ['paris',  'beautiful',  'new',  'york']
4                 ['lets',  'go',  'party']
5            ['refused',  'come',  'party']

unique_data = list(set([item.strip().strip("'") for sublist in df.Description.tolist() for item in sublist]))
print (unique_data)
['refused', 'jumped', 'go', 'roof', 'come', 'beautiful', 
'paris', 'york', 'lets', 'new', 'boy', 'party']

print (pd.DataFrame({'Unique Words': unique_data}))
   Unique Words
0       refused
1        jumped
2            go
3          roof
4          come
5     beautiful
6         paris
7          york
8          lets
9           new
10          boy
11        party
</code></pre>

python list csv pandas unique