pandas drop_duplicates using comparison function
<p>Is it somehow possible to use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html" rel="nofollow">pandas.drop_duplicates</a> with a comparison operator which compares two objects in a particular column in order to identify duplicates? If not, what is the alternative?</p>

<p>Here is an example where it could be used:</p>

<p>I have a pandas DataFrame which has lists as values in a particular column and I would like to have duplicates removed based on column <code>A</code></p>

<pre><code>import pandas as pd

df = pd.DataFrame( {'A': [[1,2],[2,3],[1,2]]} )
print df
</code></pre>

<p>giving me</p>

<pre><code>        A
0  [1, 2]
1  [2, 3]
2  [1, 2]
</code></pre>

<p>Using <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop_duplicates.html" rel="nofollow">pandas.drop_duplicates</a></p>

<pre><code>df.drop_duplicates( 'A' )
</code></pre>

<p>gives me a <code>TypeError</code></p>

<pre><code>[...]
TypeError: type object argument after * must be a sequence, not itertools.imap
</code></pre>

<p>However, my desired result is</p>

<pre><code>        A
0  [1, 2]
1  [2, 3]
</code></pre>

<p>My comparison function would be here:</p>

<pre><code>def cmp(x,y):
    return x==y
</code></pre>

<p>But in principle it could be something else, e.g.,</p>

<pre><code>def cmp(x,y):
    return x==y and len(x)&gt;1
</code></pre>

<p>How can I remove duplicates based on the comparison function in a efficient way?</p>

<p>Even more, what could I do if I had more columns to compare using a different comparison function, respectively?</p>

<p>IIUC, your question is how to use an <em>arbitrary</em> function to determine what is a duplicate. To emphasize this, let's say that two lists are duplicates if the sum of the first item, plus the square of the second item, is the same in each case</p>

<pre><code>In [59]: In [118]: df = pd.DataFrame( {'A': [[1,2],[4,1],[2,3]]} )
</code></pre>

<p>(Note that the first and second lists are equivalent, although not same.)</p>

<p>Python typically <a href="https://wiki.python.org/moin/HowTo/Sorting" rel="nofollow">prefers key functions to comparison functions</a>, so here we need a function to say what is the key of a list; in this case, it is <code>lambda l: l[0] + l[1]**2</code>.</p>

<p>We can use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.core.groupby.GroupBy.first.html" rel="nofollow"><code>groupby</code> + <code>first</code></a> to group by the values of the key function, then take the first of each group:</p>

<pre><code>In [119]: df.groupby(df.A.apply(lambda l: l[0] + l[1]**2)).first()
Out[119]: 
         A
A         
5   [1, 2]
11  [2, 3]
</code></pre>

<hr>

<p><strong>Edit</strong></p>

<p>Following further edits in the question, here are a few more examples using</p>

<pre><code>df = pd.DataFrame( {'A': [[1,2],[2,3],[1,2], [1], [1], [2]]} )
</code></pre>

<p>Then for</p>

<pre><code>def cmp(x,y):
    return x==y
</code></pre>

<p>this could be</p>

<pre><code>In [158]: df.groupby(df.A.apply(tuple)).first()
Out[158]: 
             A
A             
(1,)       [1]
(1, 2)  [1, 2]
(2,)       [2]
(2, 3)  [2, 3]
</code></pre>

<p>for</p>

<pre><code>def cmp(x,y):
     return x==y and len(x)&gt;1
</code></pre>

<p>this could be </p>

<pre><code>In [184]: class Key(object):
   .....:     def __init__(self):
   .....:         self._c = 0
   .....:     def __call__(self, l):
   .....:         if len(l) &lt; 2:
   .....:             self._c += 1
   .....:             return self._c
   .....:         return tuple(l)
   .....:     

In [187]: df.groupby(df.A.apply(Key())).first()
Out[187]: 
             A
A             
1          [1]
2          [1]
3          [2]
(1, 2)  [1, 2]
(2, 3)  [2, 3]
</code></pre>

<p>Alternatively, this could also be done much more succinctly via</p>

<pre><code>In [190]: df.groupby(df.A.apply(lambda l: np.random.rand() if len(l) &lt; 2 else tuple(l))).first()
Out[190]: 
                     A
A                     
0.112012068449     [2]
0.822889598152     [1]
0.842630848774     [1]
(1, 2)          [1, 2]
(2, 3)          [2, 3]
</code></pre>

<p>but some people don't like these Monte-Carlo things.</p>

python pandas