Identification of rows containing column median in numpy matrix of cum percentiles
<p>Consider the matrix <code>quantiles</code> that's a subset <code>[:8,:3,0]</code> of a 3D matrix with shape <code>(10,355,8)</code>. </p>

<pre><code>quantiles = np.array([
       [ 1.        ,  1.        ,  1.        ],
       [ 0.63763978,  0.61848863,  0.75348137],
       [ 0.43439645,  0.42485407,  0.5341457 ],
       [ 0.22682343,  0.18878366,  0.25253915],
       [ 0.16229408,  0.12541476,  0.15263742],
       [ 0.12306046,  0.10372971,  0.09832783],
       [ 0.09271845,  0.08209844,  0.05982584],
       [ 0.06363636,  0.05471266,  0.03855727]])
</code></pre>

<p>I want a boolean output of the same shape as the <code>quantiles</code> matrix where <code>True</code> marks the row in which the median is located:</p>

<pre><code>In [21]: medians
Out[21]:
array([[False, False, False],
       [ True, True,  False],
       [False,  False, True],
       [False, False, False],
       [False, False, False],
       [False, False, False],
       [False, False, False],
       [False, False, False]], dtype=bool)
</code></pre>

<p>To achieve this, I have the following algorithm in mind:</p>

<p>1) Identify the entries that are greater than <code>.5</code>:</p>

<pre><code>In [22]: quantiles&gt;.5
Out[22]:
array([[ True,  True,  True],
       [ True,  True,  True],
       [False, False,  True],
       [False, False, False],
       [False, False, False],
       [False, False, False],
       [False, False, False],
       [False, False, False]], dtype=bool)
</code></pre>

<p>2) Considering only the values subset by the <code>quantiles&gt;.5</code> operation, mark the row that minimizes the <code>np.abs</code> distance between the entry and <code>.5</code>. Torturing the terminology a bit, I wish to intersect the two matrices of <code>np.argmin(np.abs(quantiles-.5),axis=0)</code> and <code>quantiles&gt;.5</code> to get the above result. However, I cannot for my life figure out a way to perform the <code>np.argmin</code> on the subset and retain the shape of the <code>quantile</code> matrix. </p>

<p>PS. Yes, there is a similar question <a href="http://stackoverflow.com/questions/33658580/find-the-median-of-each-row-of-a-2-dimensional-array-in-python">here</a> but it doesn't implement my algorithm which could be, I think, more efficient on a larger scale</p>

<p>Bumping into the old <code>mask</code> operation in <code>Numpy</code>, I found the following solution</p>

<pre><code>#mask quantities that are less than .5
masked_quantiles = ma.masked_where(quantiles&lt;.5,quantiles)

#identify the minimum in column of the masked array
median_idx = np.where(masked_quantiles == masked_quantiles.min(axis=0))

#make a matrix of all False values
median_mat = np.zeros(quantiles.shape, dtype=bool)

#assign True value to corresponding rows
In [86]: median_mat[medians] = True

In [87]: median_mat
Out[87]:
array([[False, False, False],
       [ True,  True, False],
       [False, False,  True],
       [False, False, False],
       [False, False, False],
       [False, False, False],
       [False, False, False],
       [False, False, False]], dtype=bool)
</code></pre>

<h1>Update: comparison of my answer to that of Divakar's:</h1>

<p>I ran two comparisons, one on the sample 2D matrix provided for this question and one on my 3D <code>(10,380,8)</code> dataset (not large data by any means).</p>

<h3>Sample dataset:</h3>

<p>My code</p>

<pre><code>%%timeit
masked_quantiles = ma.masked_where(quantiles&lt;=.5,quantiles)
median_idx = masked_quantiles.argmin(0)

10000 loops, best of 3: 65.1 Âµs per loop
</code></pre>

<p>Divakar's code</p>

<pre><code>%%timeit
mask1 = quantiles&lt;=0.5
min_idx = (quantiles+mask1).argmin(0)

The slowest run took 17.49 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 5.92 Âµs per loop
</code></pre>

<h3>Full dataset</h3>

<p>My code:</p>

<pre><code>%%timeit
masked_quantiles = ma.masked_where(quantiles&lt;=.5,quantiles)
median_idx = masked_quantiles.argmin(0)

1000 loops, best of 3: 490 Âµs per loop
</code></pre>

<p>Divakar's code:</p>

<pre><code>%%timeit
mask1 = quantiles&lt;=0.5
min_idx = (quantiles+mask1).argmin(0)

10000 loops, best of 3: 172 Âµs per loop
</code></pre>

<h3>Conclusion:</h3>

<p>Divakar's answer seems about 3-12 times faster than mine. I presume that the <code>np.ma.where</code> masking operation takes longer than matrix addition. However, the addition operation needs to be stored whereas masking may be more efficient on larger datasets. I wonder how it would compare on something that doesn't or nearly doesn't fit into memory. </p>

python arrays numpy boolean median