Numpy in1D multiple evaluation statements
<p>I'm trying to use numpy as opposed to nested for loops and trying to find if a value is within a particular tolerance.</p>

<p>The code in python using the nested loops works fine and I do get the results I'm looking for but unfortunately is not scalable and takes a couple of hours when the size of the list is 200k plus items.</p>

<p>What I have now as a second iteration of the process is:</p>

<pre><code>import numpy as np
import numpy.ma as ma
from numpy import newaxis

#some data provided as an example
a= np.array([['id1', 8988, 7997, 210.0, 240.0, 180, 300, 7000.0, 9038, 8938, 8047, 7947, 231.0, 189.0, 8400.0, 5600.0],
['id2', 7314, 5613, 210.0, 240.0, 180, 300, 7000.0, 7364, 7264, 5663, 5563, 231.0, 189.0, 8400.0, 5600.0],
['id3', 5520, 9888, 35.0, 55.0, -125, 235, 7000.0, 5570, 5470, 9938, 9838, 38.5, 31.5, 8400.0, 5600.0],
['id4', 6270, 4270, 0.0, 90.0, -90, 270, 7000.0, 6320, 6220, 4320, 4220, 0.0, 0.0, 8400.0, 5600.0]])

print(a)

validation = np.ma.MaskedArray(((a[:, 1:2] &lt;= a[:, 8:9]) &amp; (a[:, 1:2] &gt;= a[:, 9:10])) \
    &amp; ((a[:, 2:3] &lt;= a[:, 10:11]) &amp; (a[:, 2:3] &gt;= a[:, 11:12])) \
    &amp; ((a[:, 3:4] &lt;= a[:, 12:13]) &amp; (a[:, 3:4] &gt;= a[:, 13:14])) \
    &amp; ((a[:, 7:8] &lt;= a[:, 14:15]) &amp; (a[:, 7:8] &gt;= a[:, 15:])))

e = np.in1d(a[:, 1:2], a[validation]) &lt;-- this is were I try to apply the check for tolerances
e1 = np.where(e[:, newaxis], a[:, :1], np.zeros(1, dtype=int))
ef = e1[~np.all(e1 == 0, axis=0)]

print('Final array', ef)
</code></pre>

<p>On the first attempt using numpy's meshgrid to create all combinations, one for each comparison, and then doing a numpy.where on the results it works but when using 100k plus items, the total amount of RAM required is more than 150GB of RAM.</p>

<p>Any help, advice, comment is appreciated.</p>

<p>If I copy-n-paste your <code>a</code> I get a 4x16 array of strings</p>

<pre><code>In [37]: a
Out[37]: 
array([['id1', '8988', '7997', '210.0', '240.0', '180', '300', '7000.0',
        '9038', '8938', '8047', '7947', '231.0', '189.0', '8400.0',
        '5600.0'],
      ....
      dtype='&lt;U6')
</code></pre>

<p>Applying your <code>validation</code> expression to that produces (forget about the Maskedarray bit).  Of course it is trying to do a comparison of strings.</p>

<pre><code>array([[ True],
       [ True],
       [ True],
       [ True]], dtype=bool)
</code></pre>

<p>If I remove the <code>id</code> column I get a 4x15 of floats</p>

<pre><code>In [39]: a
Out[39]: 
array([[ 8988. ,  7997. ,   210. ,   240. ,   180. ,   300. ,  7000. ,
         9038. ,  8938. ,  8047. ,  7947. ,   231. ,   189. ,  8400. ,
         5600. ],
...]])
</code></pre>

<p>WIth that I think the <code>validation</code> test can be simplified to:</p>

<pre><code>In [41]: ((a[:, 0] &lt;= a[:, 7]) &amp; (a[:, 0] &gt;= a[:, 8])) \
    ...:     &amp; ((a[:, 1] &lt;= a[:, 9]) &amp; (a[:, 1] &gt;= a[:, 10])) \
    ...:     &amp; ((a[:, 2] &lt;= a[:, 11]) &amp; (a[:, 2] &gt;= a[:, 12])) \
    ...:     &amp; ((a[:, 6] &lt;= a[:, 13]) &amp; (a[:, 6] &gt;= a[:, 14]))
Out[41]: array([ True,  True,  True,  True], dtype=bool)
</code></pre>

<p>What is this doing?</p>

<pre><code>e = np.in1d(a[:, 1:2], a[validation])
</code></pre>

<p><code>a[validation]</code> is all the <code>ok</code> rows of <code>a</code>; <code>a[:,0]</code> is the first value of each row.  But <code>np.in1d</code> is meant to check the contents of one 1d array against the contents of another 1d.  As you wrote it, it is using 2 2d arrays.</p>

<p>At this point, I'm going to give up.  </p>

<p>Construct a simpler test case, and make sure it works at each step.  Show the intermediate values.  Then we can discuss the step(s) where it does not work.</p>

python numpy evaluation