Speeding up Numpy Masking
<p>I'm still an amature when it comes to thinking about how to optimize.  I have this section of code that takes in a list of found peaks and finds where these peaks,+/- some value, are located in a multidimensional array.  It then adds +1 to their indices of a zeros array.  The code works well, but it takes a long time to execute.  For instance it is taking close to 45min to run if <code>ind</code> has 270 values  and <code>refVals</code> has a shape of (3050,3130,80). I understand that its a lot of data to churn through, but is there a more efficient way of going about this? </p>

<pre><code>maskData = np.zeros_like(refVals).astype(np.int16)

for peak in ind:
        tmpArr = np.ma.masked_outside(refVals,x[peak]-2,x[peak]+2).astype(np.int16)
        maskData[tmpArr.mask == False  ] += 1
        tmpArr = None

maskData = np.sum(maskData,axis=2)
</code></pre>

<p><strong>Approach #1 :</strong> Memory permitting, here's a vectorized approach using <a href="https://docs.scipy.org/doc/numpy/user/basics.broadcasting.html" rel="nofollow"><code>broadcasting</code></a> -</p>

<pre><code># Craate +,-2 limits usind ind
r = x[ind[:,None]] + [-2,2]

# Use limits to get inside matches and sum over the iterative and last dim
mask = (refVals &gt;= r[:,None,None,None,0]) &amp; (refVals &lt;= r[:,None,None,None,1])
out = mask.sum(axis=(0,3))
</code></pre>

<hr>

<p><strong>Approach #2 :</strong> If running out of memory with the previous one, we could use a loop and use NumPy boolean arrays and that could be more efficient than masked arrays. Also, we would perform one more level of <code>sum-reduction</code>, so that we would be dragging less data with us when moving across iterations. Thus, the alternative implementation would look something like this -</p>

<pre><code>out = np.zeros(refVals.shape[:2]).astype(np.int16)
x_ind = x[ind]
for i in x_ind:
    out += ((refVals &gt;= i-2) &amp; (refVals &lt;= i+2)).sum(-1)
</code></pre>

<p><strong>Approach #3 :</strong> Alternatively, we could replace that limit based comparison with <a href="http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.isclose.html" rel="nofollow"><code>np.isclose</code></a> in approach #2. Thus, the only step inside the loop would become -</p>

<pre><code>out += np.isclose(refVals,i,atol=2).sum(-1)
</code></pre>

python arrays performance numpy masking