Assigning indicators based on observation quantile
<p>I am working with a pandas DataFrame. I would like to assign a column indicator variable to 1 when a particular condition is met. I compute quantiles for particular groups. If the value is outside the quantile, I want to assign the column indicator variable to 1. For example, the following code prints the quantiles for each group:</p>

<pre><code>df[df['LENGTH'] &gt; 1].groupby(['CLIMATE', 'TEMP'])['LENGTH'].quantile(.95)]
</code></pre>

<p>Now for all observations in my dataframe which are larger than the grouped value I would like to set </p>

<pre><code>df['INDICATOR'] = 1
</code></pre>

<p>I tried using the following if statement:</p>

<pre><code>if df.groupby(['CLIMATE','BIN'])['LENGTH'] &gt; df[df['LENGTH'] &gt; 1].groupby(['CLIMATE','BIN'])['LENGTH'].quantile(.95):
    df['INDICATOR'] = 1
</code></pre>

<p>This gives me the error: "ValueError: operands could not be broadcast together with shapes (269,) (269,2)". Any help would be appreciated!   </p>

<p>you want to use <a href="http://pandas.pydata.org/pandas-docs/stable/groupby.html#transformation" rel="nofollow"><code>transform</code></a> after your <code>groupby</code> to get an equivalently sized array.  <code>gt</code> is greater than.  <code>mul</code> is multiply.  I multiply by <code>1</code> to get the boolean results from <code>gt</code> to <code>0</code> or <code>1</code>.  </p>

<p>You can see other examples here 
<a class='doc-link' href="http://stackoverflow.com/documentation/pandas/1822/grouping-data/14011/using-transform-to-get-group-level-statistics-while-preserving-the-original-data#t=201609262324317180894">using transform to get group-level statistics while preserving the original dataframe</a></p>

<p>Consider the dataframe <code>df</code></p>

<pre><code>df = pd.DataFrame(dict(labels=np.random.choice(list('abcde'), 100),
                       A=np.random.randn(100)))
</code></pre>

<p>I'd get the indicator like this</p>

<pre><code>df.A.gt(df.groupby('labels').A.transform(pd.Series.quantile, q=.95)).mul(1)
</code></pre>

<hr>

<p>In your case, I'd do</p>

<pre><code>df['INDICATOR'] = df['LENGTH'].gt(df.groupby(['CLIMATE','BIN'])['LENGTH'] \
                                    .transform(pd.Series.quantile, q=.95)).mul(1)
</code></pre>

python pandas numpy dataframe