How to remove a subset of a data frame in Python?
<p>My dataframe df is 3020x4.
I'd like to remove a subset df1 20x4 out of the original. In other words, I just want to get the difference whose shape is 3000x4. I tried the below but it did not work. It returned exactly df. Would you please help? Thanks.</p>

<pre><code>new_df = df.drop(df1)
</code></pre>

<p>As you seem to be unable to post a representative example I will demonstrate one approach using <code>merge</code> with param <code>indicator=True</code>:</p>

<p>So generate some data:</p>

<pre><code>In [116]:
df = pd.DataFrame(np.random.randn(5,3), columns=list('abc'))
df

Out[116]:
          a         b         c
0 -0.134933 -0.664799 -1.611790
1  1.457741  0.652709 -1.154430
2  0.534560 -0.781352  1.978084
3  0.844243 -0.234208 -2.415347
4 -0.118761 -0.287092  1.179237
</code></pre>

<p>take a subset:</p>

<pre><code>In [118]:
df_subset=df.iloc[2:3]
df_subset

Out[118]:
         a         b         c
2  0.53456 -0.781352  1.978084
</code></pre>

<p>now perform a left <code>merge</code> with param <code>indicator=True</code> this will add <code>_merge</code> column which indicates whether the row is <code>left_only</code>, <code>both</code> or <code>right_only</code> (the latter won't appear in this example) and we filter the merged df to show only <code>left_only</code>:</p>

<pre><code>In [121]:
df_new = df.merge(df_subset, how='left', indicator=True)
df_new = df_new[df_new['_merge'] == 'left_only']
df_new

Out[121]:
          a         b         c     _merge
0 -0.134933 -0.664799 -1.611790  left_only
1  1.457741  0.652709 -1.154430  left_only
3  0.844243 -0.234208 -2.415347  left_only
4 -0.118761 -0.287092  1.179237  left_only
</code></pre>

<p>here is the original merged df:</p>

<pre><code>In [122]:
df.merge(df_subset, how='left', indicator=True)

Out[122]:
          a         b         c     _merge
0 -0.134933 -0.664799 -1.611790  left_only
1  1.457741  0.652709 -1.154430  left_only
2  0.534560 -0.781352  1.978084       both
3  0.844243 -0.234208 -2.415347  left_only
4 -0.118761 -0.287092  1.179237  left_only
</code></pre>

python pandas subset