Python unique values per column in csv file row
<p>Crunching on this for a long time. Is there an easy way using Numpy or Pandas or fixing my code to get the unique values for the column in a row separated by "|"</p>

<p>I.e the data:</p>

<pre><code>"id","fname","lname","education","gradyear","attributes"
"1","john","smith","mit|harvard|ft|ft|ft","2003|207|212|212|212","qa|admin,co|master|NULL|NULL"
"2","john","doe","htw","2000","dev"
</code></pre>

<p>Output should be:</p>

<pre><code>"id","fname","lname","education","gradyear","attributes"
"1","john","smith","mit|harvard|ft","2003|207|212","qa|admin,co|master|NULL"
"2","john","doe","htw","2000","dev"
</code></pre>

<p>My broken code:</p>

<pre><code>import csv
import pprint

your_list = csv.reader(open('out.csv'))
your_list = list(your_list)

#pprint.pprint(your_list)
string = "|"
cols_no=6
for line in your_list:
    i=0
    for col in line:
      if i==cols_no:
        print "\n" 
        i=0
      if string in col:
        values = col.split("|")
        myset = set(values)
        items = list()
        for item in myset:
          items.append(item)
        print items
      else:
        print col+",",
      i=i+1
</code></pre>

<p>It outputs:</p>

<pre><code>id, fname, lname, education, gradyear, attributes, 1, john, smith, ['harvard', 'ft', 'mit']
['2003', '212', '207']
['qa', 'admin,co', 'NULL', 'master']
2, john, doe, htw, 2000, dev,
</code></pre>

<p>Thanks in advance!</p>

<p><code>numpy</code>/<code>pandas</code> is a bit overkill for what you can achieve with <code>csv.DictReader</code> and <code>csv.DictWriter</code> with a <code>collections.OrderedDict</code>, eg:</p>

<pre><code>import csv
from collections import OrderedDict

# If using Python 2.x - use `open('output.csv', 'wb') instead
with open('input.csv') as fin, open('output.csv', 'w') as fout:
    csvin = csv.DictReader(fin)
    csvout = csv.DictWriter(fout, fieldnames=csvin.fieldnames, quoting=csv.QUOTE_ALL)
    csvout.writeheader()
    for row in csvin:
        for k, v in row.items():
            row[k] = '|'.join(OrderedDict.fromkeys(v.split('|')))
        csvout.writerow(row)
</code></pre>

<p>Gives you:</p>

<pre><code>"id","fname","lname","education","gradyear","attributes"
"1","john","smith","mit|harvard|ft","2003|207|212","qa|admin,co|master|NULL"
"2","john","doe","htw","2000","dev"
</code></pre>

python pandas numpy