remove duplicate values from items in a dictionary in Python
<p>How can I check and remove duplicate values from items in a dictionary?
I have a large data set so I'm looking for an efficient method. The following is an example of values in a dictionary that contains a duplicate:</p>

<pre><code>'word': [('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])]
</code></pre>

<p>needs to become </p>

<pre><code>'word': [('769817', [6]), ('769819', [4, 10])]
</code></pre>

<p>Strikethrough applied to original question before edits, left for posterity:
<s>You're not using a <code>dict</code> at all, just a <code>list</code> of two-<code>tuple</code>s, where the second element in each <code>tuple</code> is itself a <code>list</code>. If you actually want a <code>dict</code>, </p>

<pre><code>dict([('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])])
</code></pre>

<p>will convert it, and uniquify by key (so you'd end up with <code>{'769817': [6], '769819': [4, 10]}</code>, though it loses order, and doesn't pay attention to whether the values (the sub-<code>list</code>s) are unique or not (it just keeps the last pairing for a given key).</s></p>

<p>If you need to uniquify adjacent duplicates (where the values are important to uniqueness) while preserving order, and don't want/need a real <code>dict</code>, use <code>itertools.groupby</code>:</p>

<pre><code>import itertools
nonuniq = [('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])]
uniq = [k for k, g in itertools.groupby(nonuniq)]
# uniq is [('769817', [6]), ('769819', [4, 10])]
# but it wouldn't work if the input was
# [('769819', [4, 10]), ('769817', [6]), ('769819', [4, 10])]
# because the duplicates aren't adjacent
</code></pre>

<p>If you need to collapse non-adjacent duplicates, and don't need to preserve order (or sorted order is fine), you can use <code>groupby</code> to get a <code>O(n log n)</code> solution (as opposed to naive solutions that create a new list and avoid duplicates by checking for presence in the new list at <code>O(n^2)</code> complexity, or <code>set</code> based solutions that would be <code>O(n)</code> but require you to convert sub-<code>list</code>s in your data to <code>tuple</code>s to make them hashable):</p>

<pre><code># Only difference is sorting nonuniq before grouping
uniq = [k for k, g in itertools.groupby(sorted(nonuniq))]
# uniq is [('769817', [6]), ('769819', [4, 10])]
</code></pre>

python dictionary