How to get unique rows and their occurrences for 2D array?
<p>I have a 2D array, and it has some duplicate columns. I would like to be able to see which unique columns there are, and where the duplicates are.</p>

<p>My own array is too large to put here, but here is an example:</p>

<pre><code>a = np.array([[ 1.,  0.,  0.,  0.,  0.],[ 2.,  0.,  4.,  3.,  0.],])
</code></pre>

<p>This has the unique column vectors <code>[1.,2.]</code>, <code>[0.,0.]</code>, <code>[0.,4.]</code> and <code>[0.,3.]</code>. There is one duplicate: <code>[0.,0.]</code> appears twice.</p>

<p>Now I found a way to get the unique vectors and their indices <a href="http://stackoverflow.com/questions/16970982/find-unique-rows-in-numpy-array">here</a> but it is not clear to me how I would get the occurences of duplicates as well. I have tried several naive ways (with <code>np.where</code> and list comps) but those are all very very slow. Surely there has to be a numpythonic way?</p>

<p>In matlab it's just the <code>unique</code> function but <code>np.unique</code> flattens arrays.</p>

<p>The <a href="https://github.com/EelcoHoogendoorn/Numpy_arraysetops_EP" rel="nofollow">numpy_indexed</a> package (disclaimer: I am its author) contains efficient functionality for computing these kind of things:</p>

<pre><code>import numpy_indexed as npi
unique_columns = npi.unique(a, axis=1)
non_unique_column_idx = npi.multiplicity(a, axis=1) &gt; 1
</code></pre>

<p>Or alternatively:</p>

<pre><code>unique_columns, column_count = npi.count(a, axis=1)
duplicate_columns = unique_columns[:, column_count &gt; 1]
</code></pre>

python arrays numpy unique