Remove duplicates from a list of list based on a subset of each list
<p>I wrote a function to remove "duplicates" from a list of list.</p>

<p>The elements of my list are: </p>

<pre><code>[ip, email, phone number].
</code></pre>

<p>I would like to remove the sublists that got the same EMAIL and PHONE NUMBER, I don't really care about the IP address.</p>

<p>The solution that I currently use is :</p>

<pre><code>def remove_duplicate_email_phone(data):
    for i in range(len(data)):
        for j in reversed(range(i+1,len(data))):
            if data[i][1] == data[j][1] and data[i][2] == data[j][2] :
                data.pop(j)
    return data
</code></pre>

<p>I would like to optimize this. It took more than 30 minutes to get the result.</p>

<p>Your approach does a full scan for each and every element in the list, making it take O(N**2) (quadratic) time. The <code>list.pop(index)</code> is also expensive as everything following <code>index</code> is moved up, making your solution approach O(N**3) cubic time.</p>

<p>Use a set and add <code>(email, phonenumber)</code> tuples to it to check if you already have seen that pair; testing containment against a set takes O(1) constant time, so you can clean out dupes in O(N) total time:</p>

<pre><code>def remove_duplicate_email_phone(data):
    seen = set()
    cleaned = []
    for ip, email, phone in data:
        if (email, phone) in seen:
            continue
        cleaned.append([ip, email, phone])
        seen.add((email, phone))
    return cleaned
</code></pre>

<p>This produces a <em>new</em> list, the old list is left untouched.</p>

python performance list python-3.x