Faster AUC in sklearn or python
<p>I have over half a million pairs of true labels and predicted scores (the length of each 1d array varies and can be between 10,000-30,000 in length) that I need to calculate the AUC for. Right now, I have a for-loop that calls:</p>

<pre><code># Simple Example with two pairs of true/predicted values instead of 500,000
from sklearn import metrics
import numpy as np

pred = [None] * 2
pred[0] = np.array([3,2,1])
pred[1] = np.array([15,12,14,11,13])

true = [None] * 2
true[0] = np.array([1,0,0])
true[1] = np.array([1,1,1,0,0])

for i in range(2):
    fpr, tpr, thresholds = metrics.roc_curve(true[i], pred[i])
    print metrics.auc(fpr, tpr)
</code></pre>

<p>However, it takes about 1-1.5 hours to process the entire dataset and calculate the AUC for each true/prediction pair. Is there a faster/better way to do this?</p>

<p><strong>Update</strong></p>

<p>Each of the 500k entries can have shape (1, 10k+). I understand that I could parallelize it but I'm stuck on a machine with only two processors and so my time can really only be effectively cut down to say, 30-45, minutes which is still too long. I've identified that the AUC calculation itself is slow and was hoping to find a faster AUC algorithm than what is available in sklearn. Or, at least, find a better way to vectorize the AUC calculation so that it can be broadcasted across multiple rows.</p>

<blockquote>
  <p>find a better way to vectorize the AUC calculation so that it can be broadcasted across multiple rows</p>
</blockquote>

<p>Probably not - sklearn already uses efficient numpy operations for its calculation of relevant parts:</p>

<pre><code>#Â -- calculate tps, fps, thresholds
#Â sklearn.metrics.ranking:_binary_clf_curve()
(...)
distinct_value_indices = np.where(np.logical_not(isclose(
        np.diff(y_score), 0)))[0]
threshold_idxs = np.r_[distinct_value_indices, y_true.size - 1]
# accumulate the true positives with decreasing threshold
tps = (y_true * weight).cumsum()[threshold_idxs]
if sample_weight is not None:
    fps = weight.cumsum()[threshold_idxs] - tps
else:
    fps = 1 + threshold_idxs - tps
return fps, tps, y_score[threshold_idxs]

# -- calculate auc
# sklearn.metrics.ranking:auc()
...
area = direction * np.trapz(y, x)
...
</code></pre>

<p>You might be able to optimize this by profiling these functions and removing operations that you can apply more efficiently beforehand. A quick profiling of your example input scaled to 5M rows reveals a few potential bottlenecks (marked <code>&gt;&gt;&gt;</code>):</p>

<pre><code># your for ... loop wrapped in function roc()
%prun -s cumulative roc
722 function calls (718 primitive calls) in 5.005 seconds
Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    5.005    5.005 &lt;string&gt;:1(&lt;module&gt;)
        1    0.000    0.000    5.005    5.005 &lt;ipython-input-51-27e30c04d997&gt;:1(roc)
        2    0.050    0.025    5.004    2.502 ranking.py:417(roc_curve)
        2    0.694    0.347    4.954    2.477 ranking.py:256(_binary_clf_curve)
     &gt;&gt;&gt;2    0.000    0.000    2.356    1.178 fromnumeric.py:823(argsort)
     &gt;&gt;&gt;2    2.356    1.178    2.356    1.178 {method 'argsort' of 'numpy.ndarray' objects}
        6    0.062    0.010    0.961    0.160 arraysetops.py:96(unique)
     &gt;&gt;&gt;6    0.750    0.125    0.750    0.125 {method 'sort' of 'numpy.ndarray' objects}
     &gt;&gt;&gt;2    0.181    0.090    0.570    0.285 numeric.py:2281(isclose)
        2    0.244    0.122    0.386    0.193 numeric.py:2340(within_tol)
        2    0.214    0.107    0.214    0.107 {method 'cumsum' of 'numpy.ndarray' objects}
</code></pre>

python scikit-learn data-science auc