Remove duplicates from a list of list based on a subset of each list
<p>I wrote a function to remove "duplicates" from a list of list.</p>

<p>The elements of my list are: </p>

<pre><code>[ip, email, phone number].
</code></pre>

<p>I would like to remove the sublists that got the same EMAIL and PHONE NUMBER, I don't really care about the IP address.</p>

<p>The solution that I currently use is :</p>

<pre><code>def remove_duplicate_email_phone(data):
    for i in range(len(data)):
        for j in reversed(range(i+1,len(data))):
            if data[i][1] == data[j][1] and data[i][2] == data[j][2] :
                data.pop(j)
    return data
</code></pre>

<p>I would like to optimize this. It took more than 30 minutes to get the result.</p>

<p>Another solution might be to use groupby.</p>

<pre><code>from itertools import groupby
from operator import itemgetter

deduped = []

data.sort(key=itemgetter(1,2))
for k, v in groupby(data, key=itemgetter(1,2):
    deduped.append(list(v)[0])
</code></pre>

<p>or using a list comprehension:</p>

<pre><code>deduped = [next(v) for k, v in groupby(data, key=itemgetter(1,2))]
</code></pre>

python performance list python-3.x