pandas drop_duplicates using comparison function
<p>Is it somehow possible to use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html" rel="nofollow">pandas.drop_duplicates</a> with a comparison operator which compares two objects in a particular column in order to identify duplicates? If not, what is the alternative?</p>

<p>Here is an example where it could be used:</p>

<p>I have a pandas DataFrame which has lists as values in a particular column and I would like to have duplicates removed based on column <code>A</code></p>

<pre><code>import pandas as pd

df = pd.DataFrame( {'A': [[1,2],[2,3],[1,2]]} )
print df
</code></pre>

<p>giving me</p>

<pre><code>        A
0  [1, 2]
1  [2, 3]
2  [1, 2]
</code></pre>

<p>Using <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html" rel="nofollow">pandas.drop_duplicates</a></p>

<pre><code>df.drop_duplicates( 'A' )
</code></pre>

<p>gives me a <code>TypeError</code></p>

<pre><code>[...]
TypeError: type object argument after * must be a sequence, not itertools.imap
</code></pre>

<p>However, my desired result is</p>

<pre><code>        A
0  [1, 2]
1  [2, 3]
</code></pre>

<p>My comparison function would be here:</p>

<pre><code>def cmp(x,y):
    return x==y
</code></pre>

<p>But in principle it could be something else, e.g.,</p>

<pre><code>def cmp(x,y):
    return x==y and len(x)&gt;1
</code></pre>

<p>How can I remove duplicates based on the comparison function in a efficient way?</p>

<p>Even more, what could I do if I had more columns to compare using a different comparison function, respectively?</p>

<p><code>Lists</code> are unhashable in nature. Try converting them to hashable types such as <code>tuples</code> and then you can continue to use <code>drop_duplicates</code>:</p>

<pre><code>df['A'] = df['A'].map(tuple)
df.drop_duplicates('A').applymap(list)
</code></pre>

<p><a href="http://i.stack.imgur.com/Ge4NF.png" rel="nofollow"><img src="http://i.stack.imgur.com/Ge4NF.png" alt="Image"></a></p>

<hr>

<p>One way of implementing it using a function would be based on computing <code>value_counts</code> of the series object, as duplicated values get aggregated and we are interested in only the <code>index</code> part (which by the way is unique) and not the actual count part.</p>

<pre><code>def series_dups(col_name):
    ser = df[col_name].map(tuple).value_counts(sort=False)
    return (pd.Series(data=ser.index.values, name=col_name)).map(list)

series_dups('A')

0    [1, 2]
1    [2, 3]
Name: A, dtype: object
</code></pre>

<hr>

<p>If you do not want to convert the values to <code>tuple</code> but rather process the values as they are, you could do:</p>

<p><strong>Toy data:</strong></p>

<pre><code>df = pd.DataFrame({'A': [[1,2], [2,3], [1,2], [3,4]], 
                   'B': [[10,11,12], [11,12], [11,12,13], [10,11,12]]})
df
</code></pre>

<p><a href="http://i.stack.imgur.com/Al7FS.png" rel="nofollow"><img src="http://i.stack.imgur.com/Al7FS.png" alt="Image"></a></p>

<pre><code>def series_dups_hashable(frame, col_names):
    for col in col_names:
        ser, indx = np.unique(frame[col].values, return_index=True)
        frame[col] = pd.Series(data=ser, index=indx, name=col)
    return frame.dropna(how='all')

series_dups_hashable(df, ['A', 'B'])   # Apply to subset/all columns you want to check
</code></pre>

<p><a href="http://i.stack.imgur.com/MTzqn.png" rel="nofollow"><img src="http://i.stack.imgur.com/MTzqn.png" alt="Image"></a></p>

python pandas