python filter 2d array by a chunk of data
<pre><code>import numpy as np

data = np.array([
    [20,  0,  5,  1],
    [20,  0,  5,  1],
    [20,  0,  5,  0],
    [20,  1,  5,  0],
    [20,  1,  5,  0],
    [20,  2,  5,  1],
    [20,  3,  5,  0],
    [20,  3,  5,  0],
    [20,  3,  5,  1],
    [20,  4,  5,  0],
    [20,  4,  5,  0],
    [20,  4,  5,  0]
])
</code></pre>

<p>I have the following 2d array. lets called the fields <code>a, b, c, d</code> in the above order where column <code>b</code> is like <code>id</code>. I wish to delete all cells that doesnt have atlist 1 appearance of the number "1" in column <code>d</code> for all cells with the same number in column <code>b</code> (same id) so after filtering i will have the following results:</p>

<pre><code>[[20  0  5  1]
 [20  0  5  1]
 [20  0  5  0]
 [20  2  5  1]
 [20  3  5  0]
 [20  3  5  0]
 [20  3  5  1]]
</code></pre>

<p>all rows with <code>b = 1</code> and <code>b = 4</code> have been deleted from the data</p>

<p>to sum up because I see answers that doesnt fit. we look at chunks of data by the <code>b</code> column. if a complete chunk of data doesnt have even one appearance of the number "1" in column <code>d</code> we delete all the rows of that <code>b</code> item. in the following example we can see a chunk of data with <code>b = 1</code> and <code>b = 4</code> ("id" = 1 and "id" = 4) that have 0 appearances of the number "1" in column <code>d</code>. thats why it gets deleted from the data </p>

<p>Let's assume the following:</p>

<ul>
<li><code>b &gt;= 0</code></li>
<li><code>b</code> is an integer</li>
<li><code>b</code> is fairly dense, ie <code>max(b) ~= len(unique(b))</code></li>
</ul>

<p>Here's a solution using <a href="https://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.ufunc.at.html" rel="nofollow"><code>np.ufunc.at</code></a>:</p>

<pre><code># unpack for clarity - this costs nothing in numpy
b_vals = data[:,1]
d_vals = data[:,3]

# build an array indexed by b values
is_ok = np.zeros(np.max(b_vals) + 1, dtype=np.bool_)
np.logical_or.at(is_ok, b_vals, d_vals)
# is_ok == array([ True, False,  True,  True, False], dtype=bool)

# take the rows which have a b value that was deemed OK
result = data[is_ok[b_vals]]
</code></pre>

<hr>

<p><code>np.logical_or.at(is_ok, b_vals, d_vals)</code> is a more efficient version of:</p>

<pre><code>for idx, val in zip(b_vals, d_vals):
    is_ok[idx] = np.logical_or(is_ok[idx], val)
</code></pre>

python arrays numpy