Efficient converting of numpy int array with shape (M, N, P) array to 2D object array with (N, P) shape
<p>From a 3D array with the shape (M, N, P) of data type <code>int</code>, I would like to get a 2D array of shape (N, P) of data type <code>object</code> and have this done with reasonable efficiency.</p>

<p>I'm happy with the objects being of either <code>tuple</code>, <code>list</code> or <code>numpy.ndarray</code> types.</p>

<p>I have a working hack of a solution where I have to go via a list. So it feels like I'm missing something:</p>

<pre><code>import numpy as np

m = np.mgrid[:8, :12]

l = zip(*(v.ravel() for v in m))
a2 = np.empty(m.shape[1:], dtype=np.object)
a2.ravel()[:] = l
</code></pre>

<p>The final array <code>a2</code>, in this example, should have the property that <code>a2[(x, y)] == (x, y)</code></p>

<p>It feels like it should have been possible to transpose <code>m</code> and make <code>a2</code> like this:</p>

<p><code>a2 = m.transpose(1,2,0).astype(np.object).reshape(m.shape[1:])</code></p>

<p>since numpy doesn't really care about what's inside the objects or alternatively when creating a numpy-array of type <code>np.object</code> be able to tell how many dimensions there should be:</p>

<pre><code>a2 = np.array(m.transpose(1,2,0), astype=object, ndim=2)
</code></pre>

<p>Numpy knows to stop before the final depth of nested iterables if they have different shape at the third dimension (in this example), but since <code>m</code> doesn't have irregularities, this seems impossible.</p>

<p>Or create <code>a2</code> and fill it with the transposed:</p>

<pre><code>a2 = np.empty(m.shape[1:], dtype=np.object)
a2[...] = m.transpose(1, 2, 0)
</code></pre>

<p>In this case e.g. <code>m.transpose(1, 2, 0)[2, 4]</code> is <code>np.array([2, 4])</code> and assigning it to <code>a2[2, 4]</code> would have been perfectly legal. However, none of these three more reasonable attempts work.</p>

<p>So for a smaller <code>m</code>:</p>

<pre><code>In [513]: m = np.mgrid[:3,:4]
In [514]: m.shape
Out[514]: (2, 3, 4)
In [515]: m
Out[515]: 
array([[[0, 0, 0, 0],
        [1, 1, 1, 1],
        [2, 2, 2, 2]],

       [[0, 1, 2, 3],
        [0, 1, 2, 3],
        [0, 1, 2, 3]]])
In [516]: ll = list(zip(*(v.ravel() for v in m)))
In [517]: ll
Out[517]: 
[(0, 0),
 (0, 1),
 (0, 2),
 ...
 (2, 3)]
In [518]: a2=np.empty(m.shape[1:], dtype=object)
In [519]: a2.ravel()[:] = ll
In [520]: a2
Out[520]: 
array([[(0, 0), (0, 1), (0, 2), (0, 3)],
       [(1, 0), (1, 1), (1, 2), (1, 3)],
       [(2, 0), (2, 1), (2, 2), (2, 3)]], dtype=object)
</code></pre>

<p>Making an empty of the right shape, and filling it via <code>[:]=</code> is the best way of controlling the <code>object</code> depth of such an array.  <code>np.array(...)</code> defaults to the highest possible dimension, which in this case would 3d. </p>

<p>So the main question is - is there a better way of constructing that <code>ll</code> list of tuples. </p>

<pre><code> a2.ravel()[:] = np.array(ll)
</code></pre>

<p>does not work, complaining <code>(12,2) into shape (12)</code>.</p>

<p>Working backwards, if I start with an array like <code>ll</code>, turn it into a nested list, the assignment works, except elements of <code>a2</code> are lists, not tuples:</p>

<pre><code>In [533]: a2.ravel()[:] = np.array(ll).tolist()
In [534]: a2
Out[534]: 
array([[[0, 0], [0, 1], [0, 2], [0, 3]],
       [[1, 0], [1, 1], [1, 2], [1, 3]],
       [[2, 0], [2, 1], [2, 2], [2, 3]]], dtype=object)
</code></pre>

<p><code>m</code> shape is (2,3,4)<code>and</code>np.array(ll)<code>shape is (12,2), then</code>m.reshape(2,-1).T` produces the same thing.</p>

<pre><code>a2.ravel()[:] = m.reshape(2,-1).T.tolist()
</code></pre>

<p>I could have transposed first, and then reshaped, <code>m.transpose(1,2,0).reshape(-1,2)</code>.</p>

<p>To get tuples I need to pass the reshaped array through a comprehension:</p>

<pre><code>a2.ravel()[:] = [tuple(l) for l in m.reshape(2,-1).T]
</code></pre>

<p>===============</p>

<p><code>m.transpose(1,2,0).astype(object)</code> is still 3d; it's just changed the integers with pointers to integers.  There's a 'wall' between the array dimensions and the dtype.  Things like reshape and transpose only operate on the dimensions, and don't penetrate that wall, or move it.  Lists are pointers all the way down. Object arrays use pointers only at the <code>dtype</code> level.</p>

<p>Don't be afraid of the <code>a2.ravel()[:]=</code> expression.  <code>ravel</code> is a cheap reshape, and assignment to a flatten version of an array may actually be faster than assignment to 2d version.  After all, the data (in this case pointers) is stored in a flat data buffer.</p>

<p>But (after playing around a bit) I can do the assignment without the ravel or reshape (still need the <code>tolist</code> to move the <code>object</code> boundary).  The list nesting has to match the <code>a2</code> shape down to 'object' level.</p>

<pre><code>a2[...] = m.transpose(1,2,0).tolist()   # even a2[:] works
</code></pre>

<p>(This brings to mind a discussion about giving <code>np.array</code> a <code>maxdim</code> parameter - <a href="http://stackoverflow.com/questions/38774922/prevent-numpy-from-creating-a-multidimensional-array">Prevent numpy from creating a multidimensional array</a>).</p>

<p>The use of <code>tolist</code> seems like an inefficiency.  But if the elements of <code>a2</code> are tuples (or rather pointers to tuples), those tuples have to be created some how.  The <code>c</code> databuffer of the <code>m</code> cannot be viewed as a set of tuples.  <code>tolist</code> (with the <code>[tuple...]</code> comprehension) might well be the most efficient way of creating such objects.</p>

<p>==============</p>

<p>Did I note that the transpose can be indexed, producing 2 element arrays with the right numbers?</p>

<pre><code>In [592]: m.transpose(1,2,0)[1,2]
Out[592]: array([1, 2])
In [593]: m.transpose(1,2,0)[0,1]
Out[593]: array([0, 1])
</code></pre>

<p>==================</p>

<p>Since the <code>tolist</code> for a structured array uses tuples, I could do:</p>

<pre><code>In [598]: a2[:]=m.transpose(1,2,0).copy().view('i,i').reshape(a2.shape).tolist()

In [599]: a2
Out[599]: 
array([[(0, 0), (0, 1), (0, 2), (0, 3)],
       [(1, 0), (1, 1), (1, 2), (1, 3)],
       [(2, 0), (2, 1), (2, 2), (2, 3)]], dtype=object)
</code></pre>

<p>and thus avoid the list comprehension.  It's not necessarily simpler or faster.</p>

python numpy