remove duplicate values from items in a dictionary in Python
<p>How can I check and remove duplicate values from items in a dictionary?
I have a large data set so I'm looking for an efficient method. The following is an example of values in a dictionary that contains a duplicate:</p>

<pre><code>'word': [('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])]
</code></pre>

<p>needs to become </p>

<pre><code>'word': [('769817', [6]), ('769819', [4, 10])]
</code></pre>

<p>You can uniqify the items based on the hash they generate. Hash could be anything, a sorted <code>json.dumps</code>, or <code>cPickle.dumps</code>.
This one liner can uniqify your dict as required.</p>

<pre><code>&gt;&gt;&gt; d =  {'word': [('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])]}
&gt;&gt;&gt; import json
&gt;&gt;&gt; { k: { json.dumps(x,sort_keys = True):x for x in v}.values() for k,v in d.iteritems()}
{'word': [('769817', [6]), ('769819', [4, 10])]}
</code></pre>

python dictionary