Numpy in1D multiple evaluation statements
<p>I'm trying to use numpy as opposed to nested for loops and trying to find if a value is within a particular tolerance.</p>

<p>The code in python using the nested loops works fine and I do get the results I'm looking for but unfortunately is not scalable and takes a couple of hours when the size of the list is 200k plus items.</p>

<p>What I have now as a second iteration of the process is:</p>

<pre><code>import numpy as np
import numpy.ma as ma
from numpy import newaxis

#some data provided as an example
a= np.array([['id1', 8988, 7997, 210.0, 240.0, 180, 300, 7000.0, 9038, 8938, 8047, 7947, 231.0, 189.0, 8400.0, 5600.0],
['id2', 7314, 5613, 210.0, 240.0, 180, 300, 7000.0, 7364, 7264, 5663, 5563, 231.0, 189.0, 8400.0, 5600.0],
['id3', 5520, 9888, 35.0, 55.0, -125, 235, 7000.0, 5570, 5470, 9938, 9838, 38.5, 31.5, 8400.0, 5600.0],
['id4', 6270, 4270, 0.0, 90.0, -90, 270, 7000.0, 6320, 6220, 4320, 4220, 0.0, 0.0, 8400.0, 5600.0]])

print(a)

validation = np.ma.MaskedArray(((a[:, 1:2] &lt;= a[:, 8:9]) &amp; (a[:, 1:2] &gt;= a[:, 9:10])) \
    &amp; ((a[:, 2:3] &lt;= a[:, 10:11]) &amp; (a[:, 2:3] &gt;= a[:, 11:12])) \
    &amp; ((a[:, 3:4] &lt;= a[:, 12:13]) &amp; (a[:, 3:4] &gt;= a[:, 13:14])) \
    &amp; ((a[:, 7:8] &lt;= a[:, 14:15]) &amp; (a[:, 7:8] &gt;= a[:, 15:])))

e = np.in1d(a[:, 1:2], a[validation]) &lt;-- this is were I try to apply the check for tolerances
e1 = np.where(e[:, newaxis], a[:, :1], np.zeros(1, dtype=int))
ef = e1[~np.all(e1 == 0, axis=0)]

print('Final array', ef)
</code></pre>

<p>On the first attempt using numpy's meshgrid to create all combinations, one for each comparison, and then doing a numpy.where on the results it works but when using 100k plus items, the total amount of RAM required is more than 150GB of RAM.</p>

<p>Any help, advice, comment is appreciated.</p>

<p>As hpaulj says, first get rid of the <code>id</code>'s.</p>

<p>Second, why are your tolerances and your values in the same array? If you had <code>min_tol</code> and <code>max_tol</code> in separate arrays, you could do this much more easily.</p>

<p>You probably need something like (after removing the <code>id</code>'s):</p>

<pre><code>min_tol = a[:, 8:15:2]
max_tol = a[:, 7:14:2]
a_val = np.c_[a[:, :3], a[:, 6]]
validation = (a_val &gt;= min_tol) &amp; (a_val &lt;= max_tol)
</code></pre>

<p>Although I'm really not sure at this stage...</p>

python numpy evaluation