Changing the value of groups with few members in a pandas data frame
<p>I have a data frame which represent different classes with their values. for example:</p>

<pre><code>df=pd.DataFrame(
 {'label':['a','a','b','a','b','b','a','c','c','d','e','c'],
'date':[1,2,3,4,3,7,12,18,11,2,5,3],'value':np.random.randn(12)})
</code></pre>

<p>I want to choose the labels with values_counts less than a specific threshold and then put them into one class i.e. label them as for example 'zero'. </p>

<p>This is my attemp:</p>

<pre><code>value_count=df.label.value_counts()
threshold = 3
for index in value_count[value_count.values&lt;=threshold].index:
    df.label[df.label==index]='zero'
</code></pre>

<p>Is there a better way to do this?</p>

<p>You could do</p>

<pre><code>In [59]: df.loc[df['label'].isin(value_count[value_count.values&lt;=threshold].index),
 'label'] = 'zero'

In [60]: df
Out[60]:
    date label     value
0      1     a -0.132887
1      2     a -1.306601
2      3  zero -1.431952
3      4     a  0.928743
4      3  zero  0.278955
5      7  zero  0.128430
6     12     a  0.200825
7     18  zero -0.560548
8     11  zero -2.925706
9      2  zero -0.061373
10     5  zero -0.632036
11     3  zero -1.061894
</code></pre>

<p>Timings</p>

<pre><code>In [87]: df = pd.concat([df]*10**4, ignore_index=True)

In [88]: %timeit df['label'].isin(value_count[value_count.values&lt;=threshold].index)
100 loops, best of 3: 7.1 ms per loop

In [89]: %timeit df.groupby('label')['label'].transform('count') &lt;= threshold
100 loops, best of 3: 11.7 ms per loop

In [90]: df.shape
Out[90]: (120000, 3)
</code></pre>

<p>You may want to benchmark with larger dataset. And, this may not be aaccurate to compare, since you're precomuting <code>value_count</code></p>

python pandas