For loop to evaluate accuracy doesn't execute
<p>So I've the following numpy arrays.</p>

<ul>
<li>X validation set, X_val: (47151, 32, 32, 1)</li>
<li>y validation set (labels), y_val_dummy: (47151, 5, 10) </li>
<li>y validation prediction set, y_pred: (47151, 5, 10)</li>
</ul>

<p>When I run the code, it seems to take forever. Can someone suggest why? I  believe it's a code efficiency problem. I can't seem to complete the process.</p>

<pre><code>y_pred_list = model.predict(X_val)
correct_preds = 0
# Iterate over sample dimension
for i in range(X_val.shape[0]):         
    pred_list_i = [y_pred_array[i] for y_pred in y_pred_array]
    val_list_i  = [y_val_dummy[i] for y_val in y_val_dummy]
    matching_preds = [pred.argmax(-1) == val.argmax(-1) for pred, val in zip(pred_list_i, val_list_i)]
    correct_preds = int(np.all(matching_preds))

total_acc = correct_preds / float(x_val.shape[0])
</code></pre>

<p>You're main problem is that you're generating a massive number of very large lists for no real reason</p>

<pre><code>for i in range(X_val.shape[0]):
    # this line generates a 47151 x 5 x 10 array every time          
    pred_list_i = [y_pred_array[i] for y_pred in y_pred_array]
</code></pre>

<p>What's happening is that iterating over an nd numpy array iterates over the slowest varying index (i.e. the leftmost), so every list comprehension is running operating on 47K entries.</p>

<p>Marginally better would be </p>

<pre><code>for i in range(X_val.shape[0]):         
    pred_list_i = [y_pred for y_pred in y_pred_array[i]]
    val_list_i  = [y_val for y_val in y_val_dummy[i]]
    matching_preds = [pred.argmax(-1) == val.argmax(-1) for pred, val in zip(pred_list_i, val_list_i)]
    correct_preds = int(np.all(matching_preds))
</code></pre>

<p>But you're still copying a lot of arrays for no real purpose. The following code should do the same, without the useless copying.</p>

<pre><code>correct_preds = 0.0
for pred, val in zip(y_pred_array, y_val_dummy):
    correct_preds += all(p.argmax(-1) == v.argmax(-1) 
                         for p, v in zip(pred, val))
total_accuracy = correct_preds / x_val.shape[0]
</code></pre>

<p>This assumes that your criteria for a correct prediction is accurate.
You can probably avoid the explicit loop entirely with a couple of calls to <code>np.argmax</code>, but you'll have to work that out on your own.</p>

python numpy