python - Removing duplicate rows under conditions in Pandas
<p>I have a DataFrame like this:</p>

<pre><code>  NoDemande   NoUsager  Sens  IdVehiculeUtilise  Fait  HeurePrevue  HeureDebutTrajet
0 42191000823  001208    +         246VÃ©h         1    08:20:04     08:22:26 
1 42191000822  001208    +         246VÃ©h         1    08:20:04     08:18:56 
2 42191000822  001208    -         246VÃ©h        -99   09:05:03     08:56:26 
3 42191000823  001208    -         246VÃ©h         1    09:05:03     08:56:26 
4 42191000834  001208    +         246VÃ©h         1    16:50:04     16:39:26 
5 42191000834  001208    -         246VÃ©h         1    17:45:03     17:25:10 
6 42192000761  001208    +         246VÃ©h        -1    08:20:04     08:15:07 
7 42192000762  001208    +         246VÃ©h         1    08:20:04     08:18:27 
8 42192000762  001208    -         246VÃ©h        -99   09:05:03     08:58:29 
9 42192000761  001208    -         246VÃ©h        -11   09:05:03     08:58:29 
</code></pre>

<p>I get this data frame from<code>df[df.duplicated(['NoUsager','NoDemande'],keep=False)]</code>which ensure my rows being in pair. I want to drop a pair of rows when <code>NoDemande</code> are continuous numbers (like 42191000822 and 42191000823, 42192000761 and 42192000762) and the column <code>HeurePrevue</code> are the same, which means the records are recorded twice. I have to delete a pair and I'd like to preseve the one with more positive numbers in column <code>Fait</code>(at least one greater than 0)</p>

<p>So my result should look like:</p>

<pre><code>  NoDemande   NoUsager  Sens  IdVehiculeUtilise  Fait  HeurePrevue  HeureDebutTrajet
0 42191000823  001208    +         246VÃ©h         1    08:20:04     08:22:26 
3 42191000823  001208    -         246VÃ©h         1    09:05:03     08:56:26 
4 42191000834  001208    +         246VÃ©h         1    16:50:04     16:39:26 
5 42191000834  001208    -         246VÃ©h         1    17:45:03     17:25:10 
7 42192000762  001208    +         246VÃ©h         1    08:20:04     08:18:27 
8 42192000762  001208    -         246VÃ©h        -99   09:05:03     08:58:29 
</code></pre>

<p>I know it's something about <code>OR</code> logic but I have no idea how to realize it.</p>

<p>Any help will be appreciated~</p>

<p>My approach on this problem was to make two columns which contain the conditions for a check (same heure and continuous increasing NoDemande). Then iterate over the dataframe dropping the pairs you do not want based on the Fait columns.</p>

<p>It's a bit of a hacky code but this seems to do the trick:</p>

<pre><code># Recreate DataFrame
df = pd.DataFrame({
    'NoDemande': [23, 22, 22, 23, 34, 34, 61, 62, 62, 61],
    'HeurePrevue': [84, 84, 93, 93, 64, 73, 84, 84, 93, 93],
    'Fait': [1, 1, -99, 1, 1, 1, -1, 1, -99, -11]
    }, columns=['NoDemande', 'Fait', 'HeurePrevue'])

# Make columns which contain conditions for inspection
df['sameHeure'] = df.HeurePrevue.iloc[1:] == df.HeurePrevue.iloc[:-1]
df['cont'] = df.NoDemande.diff()

# Cycle over rows
for prev_row, row in zip(df.iloc[:-1].itertuples(), df.iloc[1:].itertuples()):
    if row.sameHeure and (row.cont == 1):  # If rows are continuous and have the same Heure delete a pair
        pair_1 = df.loc[df.NoDemande == row.NoDemande]
        pair_2 = df.loc[df.NoDemande == prev_row.NoDemande]
        if sum(pair_1.Fait &gt; 0) &lt; sum(pair_2.Fait &gt; 0):  # Find which pair to delete
            df.drop(pair_1.index, inplace=True)
        else:
            df.drop(pair_2.index, inplace=True)

df.drop(['cont', 'sameHeure'], 1, inplace=True)  # Throw away the added columns
</code></pre>

<p>result:</p>

<pre><code>print(df)

   NoDemande  Fait  HeurePrevue
0         23     1           84
3         23     1           93
4         34     1           64
5         34     1           73
7         62     1           84
8         62   -99           93
</code></pre>

python pandas dataframe duplicates