pandas drop_duplicates using comparison function
<p>Is it somehow possible to use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html" rel="nofollow">pandas.drop_duplicates</a> with a comparison operator which compares two objects in a particular column in order to identify duplicates? If not, what is the alternative?</p>

<p>Here is an example where it could be used:</p>

<p>I have a pandas DataFrame which has lists as values in a particular column and I would like to have duplicates removed based on column <code>A</code></p>

<pre><code>import pandas as pd

df = pd.DataFrame( {'A': [[1,2],[2,3],[1,2]]} )
print df
</code></pre>

<p>giving me</p>

<pre><code>        A
0  [1, 2]
1  [2, 3]
2  [1, 2]
</code></pre>

<p>Using <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html" rel="nofollow">pandas.drop_duplicates</a></p>

<pre><code>df.drop_duplicates( 'A' )
</code></pre>

<p>gives me a <code>TypeError</code></p>

<pre><code>[...]
TypeError: type object argument after * must be a sequence, not itertools.imap
</code></pre>

<p>However, my desired result is</p>

<pre><code>        A
0  [1, 2]
1  [2, 3]
</code></pre>

<p>My comparison function would be here:</p>

<pre><code>def cmp(x,y):
    return x==y
</code></pre>

<p>But in principle it could be something else, e.g.,</p>

<pre><code>def cmp(x,y):
    return x==y and len(x)&gt;1
</code></pre>

<p>How can I remove duplicates based on the comparison function in a efficient way?</p>

<p>Even more, what could I do if I had more columns to compare using a different comparison function, respectively?</p>

<p><strong><em>Option 1</em></strong></p>

<pre><code>df[~pd.DataFrame(df.A.values.tolist()).duplicated()]
</code></pre>

<p><a href="http://i.stack.imgur.com/pQ4BD.png" rel="nofollow"><img src="http://i.stack.imgur.com/pQ4BD.png" alt="enter image description here"></a></p>

<p><strong><em>Option 2</em></strong></p>

<pre><code>df[~df.A.apply(pd.Series).duplicated()]
</code></pre>

python pandas