Removing Duplicates conditional in Pandas based on all but on column?
<p>So I have a Pandas DataFrame loaded with a bunch of data, yet, there are some duplicates in the data. The way duplicates exist, make it hard to remove them. Imagine this:</p>

<pre><code>1         |a          |b         |c          |1232
2         |           |b         |c          |1232
3         |           |as        |ac         |89231
</code></pre>

<p>Now I want the code to be able to remove row 2, because it's basically same as row 1, but its second column is empty (the second column has some empty artifacts based on the way data was scraped from the web), yet I don't want the code to remove 3 and 1. </p>

<p>Any ideas?</p>

<p>You can use <a href="http://pandas.pydata.org/pandas-docs/version/0.17.1/generated/pandas.DataFrame.drop_duplicates.html" rel="nofollow">drop_duplicates</a>. If your column names are, let's say: <code>['A', 'B', 'C', 'D', 'E']</code> and your data frame is <code>df</code> and <code>row 0</code> and <code>row 1</code> are not duplicated in column 'A' and column 'B', then you can do this:</p>

<pre><code>df.drop_duplicates(['C', 'D','E'], keep='first')
</code></pre>

python database algorithm csv pandas