remove duplicate values from items in a dictionary in Python
<p>How can I check and remove duplicate values from items in a dictionary?
I have a large data set so I'm looking for an efficient method. The following is an example of values in a dictionary that contains a duplicate:</p>

<pre><code>'word': [('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])]
</code></pre>

<p>needs to become </p>

<pre><code>'word': [('769817', [6]), ('769819', [4, 10])]
</code></pre>

<p>This problem essentially boils down to removing duplicates from a list of <strong>unhashable</strong> types, for which converting to a set does not possible.</p>

<p>One possible method is to check for membership in the current value while building up a new list value.</p>

<pre><code>d = {'word': [('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])]}
for k, v in d.items():
    new_list = []
    for item in v:
        if item not in new_list:
            new_list.append(item)
    d[k] = new_list
</code></pre>

<p><em>Alternatively</em>, use <a href="https://docs.python.org/2/library/itertools.html#itertools.groupby" rel="nofollow"><code>groupby()</code></a> for a more concise answer, although <strong>potentially</strong> slower (<em>the list must be sorted first, if it is, then it is faster than doing a membership check</em>).</p>

<pre><code>import itertools

d = {'word': [('769817', [6]), ('769819', [4, 10]), ('769819', [4, 10])]}
for k, v in d.items():
    v.sort()
    d[k] = [item for item, _ in itertools.groupby(v)]
</code></pre>

<p><strong>Output</strong> -> <code>{'word': [('769817', [6]), ('769819', [4, 10])]}</code></p>

python dictionary