Calculating percentile of bins from numpy digitize?
<p>I have a set of data, and a set of thresholds for creating bins:</p>

<pre><code>data = np.array([0.01, 0.02, 1, 1, 1, 2, 2, 8, 8, 4.5, 6.6])
thresholds = np.array([0,5,10])
bins = np.digitize(data, thresholds, right=True)
</code></pre>

<p>For each of the elements in <code>bins</code>, I want to know the base percentile. For example, in <code>bins</code>, the smallest bin should start at the 0th percentile. Then the next bin, for example, the 20th percentile. So that if a value in <code>data</code> falls between the 0th and 20th percentile of <code>data</code>, it belongs in the first <code>bin</code>.</p>

<p>I've looked into pandas <code>rank(pct=True)</code> but can't seem to get this done correctly.</p>

<p>Suggestions?</p>

<p>You can calculate the percentile for each element in your data array as described in a previous StackOverflow question (<a href="http://stackoverflow.com/questions/12414043/map-each-list-value-to-its-corresponding-percentile#answer-28577101">Map each list value to its corresponding percentile</a>).</p>

<pre><code>import numpy as np
from scipy import stats
data = np.array([0.01, 0.02, 1, 1, 1, 2, 2, 8, 8, 4.5, 6.6])
</code></pre>

<p>Method 1: Using <a href="http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.percentileofscore.html" rel="nofollow">scipy.stats.percentileofscore</a> :</p>

<pre><code>data_percentile = np.array([stats.percentileofscore(data, a) for a in data])
data_percentile
Out[1]:
array([  9.09090909,  18.18181818,  36.36363636,  36.36363636,
        36.36363636,  59.09090909,  59.09090909,  95.45454545,
        95.45454545,  72.72727273,  81.81818182])
</code></pre>

<p>Method 2: Using <a href="http://docs.scipy.org/doc/scipy-0.16.0/reference/generated/scipy.stats.rankdata.html" rel="nofollow">scipy.stats.rankdata</a> and normalising to 100 (faster) :</p>

<pre><code>ranked = stats.rankdata(data)
data_percentile = ranked/len(data)*100
data_percentile
Out[2]:
array([  9.09090909,  18.18181818,  36.36363636,  36.36363636,
        36.36363636,  59.09090909,  59.09090909,  95.45454545,
        95.45454545,  72.72727273,  81.81818182])
</code></pre>

<p>Now that you have a list of percentiles, you can bin them as before using <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.digitize.html" rel="nofollow">numpy.digitize</a> :</p>

<pre><code>bins_percentile = [0,20,40,60,80,100]
data_binned_indices = np.digitize(data_percentile, bins_percentile, right=True)
data_binned_indices
Out[3]:
array([1, 1, 2, 2, 2, 3, 3, 5, 5, 4, 5], dtype=int64)
</code></pre>

<p>This gives you the data binned according to the indices of your chosen list of percentiles. If desired, you could also return the actual (upper) percentiles using <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.take.html" rel="nofollow">numpy.take</a> : </p>

<pre><code>data_binned_percentiles = np.take(bins_percentile, data_binned_indices)
data_binned_percentiles
Out[4]:
array([ 20,  20,  40,  40,  40,  60,  60, 100, 100,  80, 100])
</code></pre>

python pandas numpy histogram percentage