Faster AUC in sklearn or python
<p>I have over half a million pairs of true labels and predicted scores (the length of each 1d array varies and can be between 10,000-30,000 in length) that I need to calculate the AUC for. Right now, I have a for-loop that calls:</p>

<pre><code># Simple Example with two pairs of true/predicted values instead of 500,000
from sklearn import metrics
import numpy as np

pred = [None] * 2
pred[0] = np.array([3,2,1])
pred[1] = np.array([15,12,14,11,13])

true = [None] * 2
true[0] = np.array([1,0,0])
true[1] = np.array([1,1,1,0,0])

for i in range(2):
    fpr, tpr, thresholds = metrics.roc_curve(true[i], pred[i])
    print metrics.auc(fpr, tpr)
</code></pre>

<p>However, it takes about 1-1.5 hours to process the entire dataset and calculate the AUC for each true/prediction pair. Is there a faster/better way to do this?</p>

<p><strong>Update</strong></p>

<p>Each of the 500k entries can have shape (1, 10k+). I understand that I could parallelize it but I'm stuck on a machine with only two processors and so my time can really only be effectively cut down to say, 30-45, minutes which is still too long. I've identified that the AUC calculation itself is slow and was hoping to find a faster AUC algorithm than what is available in sklearn. Or, at least, find a better way to vectorize the AUC calculation so that it can be broadcasted across multiple rows.</p>

<blockquote>
  <p>Is there a faster/better way to do this?</p>
</blockquote>

<p>Since the calculation of each true/pred pair is independent (if I understood your setup), you should be able to reduce total processing time by using <a href="https://docs.python.org/2/library/multiprocessing.html" rel="nofollow"><code>multiprocessing</code></a>, effectively parallelizing the calculations:</p>

<pre><code>import multiprocessing as mp

def roc(v):
    """ calculate one pair, return (index, auc) """
    i, true, pred = v
    fpr, tpr, thresholds = metrics.roc_curve(true, pred, drop_intermediate=True)
    auc = metrics.auc(fpr, tpr)
    return i, auc

pool = mp.Pool(3) 
result = pool.map_async(roc, ((i, true[i], pred[i]) for i in range(2)))
pool.close()
pool.join()
print result.get()
=&gt;
[(0, 1.0), (1, 0.83333333333333326)]
</code></pre>

<p>Here <code>Pool(3)</code> creates a pool of 3 processes, <code>.map_async</code> maps all true/pred pairs and calls the <code>roc</code> function, passing one pair at a time. The index is sent along to map back results. </p>

<p>If the true/pred pairs are too large to serialize and send to the processes, you might need to write the data into some external data structure before calling <code>roc</code>, passing it just the reference <code>i</code> and read the data for each pair <code>true[i]/pred[i]</code> from within <code>roc</code> before processing.</p>

<p>A <a href="https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing.pool" rel="nofollow">Pool</a> automatically manages the scheduling of processes. To reduce the risk of a memory hog, you might need to pass the <code>Pool(...., maxtasksperchild=1)</code> parameter which would start a new process for each true/pred pair (choose any other number as you see fit).</p>

<p><strong>Update</strong></p>

<blockquote>
  <p>I'm stuck on a machine with only two processors</p>
</blockquote>

<p>naturally this is a limiting factor. However considering the availability of cloud computing resources at very reasonable cost that you only pay for the time you actually need it, you might want to consider alternatives in hardware before you spend eons of hours optimizing a calculation that can be so effectively parallelized. That's a luxury in its own right, really.</p>

python scikit-learn data-science auc