How to calculate a partial Area Under the Curve (AUC)
<p>In scikit learn you can compute the area under the curve for a binary classifier with</p>

<pre><code>roc_auc_score( Y, clf.predict_proba(X)[:,1] )
</code></pre>

<p>I am only interested in the part of the curve where the false positive rate is less than 0.1.</p>

<blockquote>
  <p>Given such a threshold false positive rate, how can I compute the AUC
  only for the part of the curve up the threshold?</p>
</blockquote>

<p>Here is an example with several ROC-curves, for illustration:</p>

<p><a href="http://i.stack.imgur.com/Xg4uc.png" rel="nofollow"><img src="http://i.stack.imgur.com/Xg4uc.png" alt="Illustration of ROC-curves plot for several types of a classifier."></a></p>

<p>The scikit learn docs show how to use roc_curve</p>

<pre><code>&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; from sklearn import metrics
&gt;&gt;&gt; y = np.array([1, 1, 2, 2])
&gt;&gt;&gt; scores = np.array([0.1, 0.4, 0.35, 0.8])
&gt;&gt;&gt; fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2)
&gt;&gt;&gt; fpr
array([ 0. ,  0.5,  0.5,  1. ])
&gt;&gt;&gt; tpr
array([ 0.5,  0.5,  1. ,  1. ])
&gt;&gt;&gt; thresholds
array([ 0.8 ,  0.4 ,  0.35,  0.1 ]
</code></pre>

<p>Is there a simple way to go from this to the partial AUC?</p>

<hr>

<p>It seems the only problem is how to compute the tpr value at fpr = 0.1 as roc_curve doesn't necessarily give you that.</p>

<p>Say we start with</p>

<pre><code>import numpy as np
from sklearn import  metrics
</code></pre>

<p>Now we set the true <code>y</code> and predicted <code>scores</code>:</p>

<pre><code>y = np.array([0, 0, 1, 1])

scores = np.array([0.1, 0.4, 0.35, 0.8])
</code></pre>

<p>(Note that <code>y</code> has shifted down by 1 from your problem. This is inconsequential: the exact same results (fpr, tpr, thresholds, etc.) are obtained whether predicting 1, 2 or 0, 1, but some <code>sklearn.metrics</code> functions are a drag if not using 0, 1.)</p>

<p>Let's see the AUC here:</p>

<pre><code>&gt;&gt;&gt; metrics.roc_auc_score(y, scores)
0.75
</code></pre>

<p>As in your example:</p>

<pre><code>fpr, tpr, thresholds = metrics.roc_curve(y, scores)
&gt;&gt;&gt; fpr, tpr
(array([ 0. ,  0.5,  0.5,  1. ]), array([ 0.5,  0.5,  1. ,  1. ]))
</code></pre>

<p>This gives the following plot:</p>

<pre><code>plot([0, 0.5], [0.5, 0.5], [0.5, 0.5], [0.5, 1], [0.5, 1], [1, 1]);
</code></pre>

<p><a href="http://i.stack.imgur.com/A142Q.png" rel="nofollow"><img src="http://i.stack.imgur.com/A142Q.png" alt="enter image description here"></a></p>

<p>By construction, the ROC for a finite-length <em>y</em> will be composed of rectangles:</p>

<ul>
<li><p>For low enough threshold, everything will be classified as negative.</p></li>
<li><p>As the threshold increases continuously, at <em>discrete points</em>, some negative classifications will be changed to positive.</p></li>
</ul>

<p>So, for a finite <em>y</em>, the ROC will always be characterized by a sequence of connected horizontal and vertical lines leading from <em>(0, 0)</em> to <em>(1, 1)</em>. </p>

<p>The AUC is the sum of these rectangles. Here, as shown above, the AUC is 0.75, as the rectangles have areas 0.5 * 0.5 + 0.5 * 1 = 0.75. </p>

<p>In some cases, people choose to calculate the AUC by linear interpolation. Say the length of <em>y</em> is much larger than the actual number of points calculated for the FPR and TPR. Then, in this case, a linear interpolation is an approximation of what the points in between <em>might</em> have been. In some cases people also follow the <em>conjecture</em> that, had <em>y</em> been large enough, the points in between would be interpolated linearly. <code>sklearn.metrics</code> does not use this conjecture, and to get results consistent with <code>sklearn.metrics</code>, it is necessary to use rectangle, not trapezoidal, summation.</p>

<p>Let's write our own function to calculate the AUC directly from <code>fpr</code> and <code>tpr</code>:</p>

<pre><code>import itertools
import operator

def auc_from_fpr_tpr(fpr, tpr, trapezoid=False):
    inds = [i for (i, (s, e)) in enumerate(zip(fpr[: -1], fpr[1: ])) if s != e] + [len(fpr) - 1]
    fpr, tpr = fpr[inds], tpr[inds]
    area = 0
    ft = zip(fpr, tpr)
    for p0, p1 in zip(ft[: -1], ft[1: ]):
        area += (p1[0] - p0[0]) * ((p1[1] + p0[1]) / 2 if trapezoid else p0[1])
    return area
</code></pre>

<p>This function takes the FPR and TPR, and an optional parameter stating whether to use trapezoidal summation. Running it, we get:</p>

<pre><code>&gt;&gt;&gt; auc_from_fpr_tpr(fpr, tpr), auc_from_fpr_tpr(fpr, tpr, True)
(0.75, 0.875)
</code></pre>

<p>We get the same result as <code>sklearn.metrics</code> for the rectangle summation, and a different, higher, result for trapezoid summation.</p>

<p>So, now we just need to see what would happen to the FPR/TPR points if we would terminate at an FPR of 0.1. We can do this with the <a href="https://docs.python.org/2/library/bisect.html" rel="nofollow"><code>bisect</code> module</a></p>

<pre><code>import bisect

def get_fpr_tpr_for_thresh(fpr, tpr, thresh):
    p = bisect.bisect_left(fpr, thresh)
    fpr = fpr.copy()
    fpr[p] = thresh
    return fpr[: p + 1], tpr[: p + 1]
</code></pre>

<p>How does this work? It simply checks where would be the insertion point of <code>thresh</code> in <code>fpr</code>. Given the properties of the FPR (it must start at 0), the insertion point must be in a horizontal line. Thus all rectangles before this one should be unaffected, all rectangles after this one should be removed, and this one should be possibly shortened. </p>

<p>Let's apply it:</p>

<pre><code>fpr_thresh, tpr_thresh = get_fpr_tpr_for_thresh(fpr, tpr, 0.1)
&gt;&gt;&gt; fpr_thresh, tpr_thresh
(array([ 0. ,  0.1]), array([ 0.5,  0.5]))
</code></pre>

<p>Finally, we just need to calculate the AUC from the updated versions:</p>

<pre><code>&gt;&gt;&gt; auc_from_fpr_tpr(fpr, tpr), auc_from_fpr_tpr(fpr, tpr, True)
0.050000000000000003, 0.050000000000000003)
</code></pre>

<p>In this case, both the rectangle and trapezoid summations give the same results. Note that in general, they will not. For consistency with <code>sklearn.metrics</code>, the first one should be used.</p>

python machine-learning statistics scikit-learn