Pandas does not drop nan
<p>I have the following group</p>

<pre><code>              unemp   nobsRel  measure   rank  nobsRel2  nobsCumSummed
year foo                                                              
2000 8010  0.000024  0.000167      1.0  348.0  0.000167       0.980176
     4950  0.000264  0.003630      1.0  349.0  0.003630       0.983806
     2540  0.000438  0.011027      1.0  350.0  0.011027       0.994833
     9830  0.000118  0.000118      NaN    NaN  0.000118       0.994952
     9920  0.005048  0.005048      NaN    NaN  0.005048       1.000000
</code></pre>

<p>Here's what happens when I try to drop duplicates:</p>

<p>In[104]: test.drop_duplicates(subset='rank')
Out[104]: </p>

<pre><code>              unemp   nobsRel  measure   rank  nobsRel2  nobsCumSummed
year foo                                                              
2000 8010  0.000024  0.000167      1.0  348.0  0.000167       0.980176
     4950  0.000264  0.003630      1.0  349.0  0.003630       0.983806
     2540  0.000438  0.011027      1.0  350.0  0.011027       0.994833
     9830  0.000118  0.000118      NaN    NaN  0.000118       0.994952
</code></pre>

<p>Why is the last row still there? It is missing!</p>

<pre><code>In[105]: np.isnan(test.loc[(2000, 9830), 'rank'])
Out[105]: 
True
</code></pre>

<p>I'm on <code>pandas</code> 0.19.0</p>

<p>Sadly it appears that <code>drop_duplicates</code> doesn't do what you want. It removes duplicates, but not the original row that the duplicates are a duplicate of ...</p>

<p>Fortunately that can be overridden with a <code>keep=False</code> option to the call.</p>

python pandas