Unable to extract unique words from a CSV
<p>I have a <code>csv</code> which looks like below:</p>

<pre><code>    Description
0   ['boy']
1   ['boy', 'jumped', 'roof']
2   ['paris']
3   ['paris', 'beautiful', 'new', 'york']
4   ['lets', 'go', 'party']
5   ['refused', 'come', 'party']
</code></pre>

<p>I need to find out unique words from this data. So output would be:</p>

<pre><code>    Unique Words
0   boy
1   jumped
2   roof
3   paris
4   beautiful
5   new
6   york
</code></pre>

<p>as so on. I am trying to do this using Pandas and Python and unable to achieve it. My code is:</p>

<pre><code>df = pd.read_csv('output.csv')
list(set(df.Description))
g = list(df['Description'].unique())
print(g)
</code></pre>

<p>This throws wrong output, it just throws the original csv dataframe.</p>

<p>This approach works:</p>

<pre><code>import pandas as pd
import ast

test = {'Description':["['boy']","['boy', 'jumped', 'roof']","['paris']",\
 "['paris', 'beautiful', 'new', 'york']","['lets', 'go', 'party']",\
 "['refused', 'come', 'party']"]}

tt = pd.DataFrame(test)

listOfWords = []
for i,row in tt.iterrows():
  listOfWords.extend(ast.literal_eval(tt.ix[i,'Description']))
uniqueWords = pd.DataFrame(listOfWords,columns=['Unique Words']).drop_duplicates()
</code></pre>

<p>If you want it sorted:</p>

<pre><code>uniqueWords = uniqueWords.sort_values('Unique Words')
</code></pre>

<p>You iterate over all rows, convert your strings to lists and gather all those lists into one long list with <code>extend</code>. Then just make a new DataFrame from that list and drop the duplicates.</p>

<p>EDIT: Thanks to jezrael for correcting my solution, I borrowed the <code>ast.literal_eval</code> approach from his solution.</p>

<p>I tried to compare our solutions using the <code>%timeit</code> command but got <code>ValueError: malformed string</code> on <code>ast.literal_eval</code> in both solutions.</p>

<p>EDIT2: jezrael's solution is twice as fast for the small data example we have here.</p>

<p>EDIT3: I can't test with a large data example (multiply the given one by some number) because <code>timeit</code> keeps throwing <code>malformed string</code> errors for reasons unclear to me.</p>

<p>EDIT4: Made it work somehow. For a larger dataset (6000 rows) jezrael's solution is over 8 times faster. Guess even iterating with <code>iterrows</code> is rather slow compared to list comprehensions. Also I tested jezrael's second solution without <code>ast</code>. It's more than twice as fast as his first solution.</p>

python list csv pandas unique