Finding duplicate rows python
<p>I have <code>timestamp</code> and <code>id</code> variables in my dataframe (<code>df</code>)</p>

<pre><code>timestamp                    id
2016-06-09 8:33:37           a1  
2016-06-09 8:33:37           a1  
2016-06-09 8:33:38           a1
2016-06-09 8:33:39           a1
2016-06-09 8:33:39           a1
2016-06-09 8:33:37           b1
2016-06-09 8:33:38           b1
</code></pre>

<p>Each <code>id</code> can't have two timestamps. I have to print these duplicate timestamps for each <code>id</code>. In my above case, the output should be for rows 1,2,4,5</p>

<p>The following code will give the duplicate <code>timestamp</code></p>

<pre><code>set([x for x in df['timestamp'] if df['timestamp'].count(x) &gt; 1])
</code></pre>

<p>How to consider <code>id</code> along with <code>timestamp</code> to have the duplicate rows?</p>

<p>If you wish to find all duplicates then use the <code>duplicated</code> method. It only works on the columns. On the other hand <code>df.index.duplicated</code> works on the index. Therefore we do a quick <code>reset_index</code> to bring the index into the columns.</p>

<pre><code>df = df.reset_index()
df.ix[df.duplicated(keep=False)]

    index               id
0   2016-06-09 8:33:37  a1
1   2016-06-09 8:33:37  a1
3   2016-06-09 8:33:39  a1
4   2016-06-09 8:33:39  a1
</code></pre>

<p>If you just wish to remove duplicates there is the DataFrame method <code>drop_duplicates</code>.</p>

<pre><code>df = df.reset_index()
df = df.drop_duplicates() # keep='first' by default.

    index               id
0   2016-06-09 8:33:37  a1
2   2016-06-09 8:33:38  a1
3   2016-06-09 8:33:39  a1
5   2016-06-09 8:33:37  b1
6   2016-06-09 8:33:38  b1
</code></pre>

<p>If you wish to get back your old index then set_index using the default columns name 'index' for either of the above then simply use.</p>

<pre><code>df.set_index('index')

                    id
index               
2016-06-09 8:33:37  a1
2016-06-09 8:33:38  a1
2016-06-09 8:33:39  a1
2016-06-09 8:33:37  b1
2016-06-09 8:33:38  b1
</code></pre>

<p>The above methods allow you choose whether to keep the first, last or none of the duplicates by setting the <code>keep</code> attribute to <code>'first'</code>, <code>'last'</code> or <code>False</code>. So to remove all duplicates in <code>df</code> then use <code>keep=False</code>.</p>

python pandas timestamp duplicates