From scatter plot to 2D array
<p>My mind has gone completely blank on this one.</p>

<p>I want to do what I think is very simple.</p>

<p>Suppose I have some test data:</p>

<pre><code>import pandas as pd
import numpy as np
k=10
df = pd.DataFrame(np.array([range(k), 
                           [x + 1 for x in range(k)],
                           [x + 4 for x in range(k)], 
                           [x + 9 for x in range(k)]]).T,columns=list('abcd'))
</code></pre>

<p>where rows correspond to time and columns to angles, and it looks like this:</p>

<pre><code>   a   b   c   d
0  0   1   4   9
1  1   2   5  10
2  2   3   6  11
3  3   4   7  12
4  4   5   8  13
5  5   6   9  14
6  6   7  10  15
7  7   8  11  16
8  8   9  12  17
9  9  10  13  18
</code></pre>

<p>Then for reasons I convert it to and ordered dictionary:</p>

<pre><code>def highDimDF2Array(df):
    from collections import OrderedDict # Need to preserve order

    vels = [1.42,1.11,0.81,0.50]

    # Get dataframe shapes
    cols = df.columns

    trajectories = OrderedDict()
    for i,j in enumerate(cols):
        x = df[j].values
        x = x[~np.isnan(x)]

        maxTimeSteps = len(x)
        tmpTraj = np.empty((maxTimeSteps,3))
        # This should be fast
        tmpTraj[:,0] = range(maxTimeSteps) 
        # Remove construction nans
        tmpTraj[:,1] = x
        tmpTraj[:,2].fill(vels[i])

        trajectories[j] = tmpTraj

    return trajectories
</code></pre>

<p>Then I plot it all</p>

<pre><code>import matplotlib.pyplot as plt
m = highDimDF2Array(df)
M = np.vstack(m.values())
plt.scatter(M[:,0],M[:,1],15,M[:,2])
plt.title('Angle $[^\circ]$ vs. Time $[s]$')
plt.colorbar()
plt.show()
</code></pre>

<p><a href="http://i.stack.imgur.com/vtEcv.png" rel="nofollow"><img src="http://i.stack.imgur.com/vtEcv.png" alt="enter image description here"></a></p>

<p>Now all I want to do is to put all of that into a 2D numpy array with the properties:</p>

<ul>
<li>Time is mapped to the x-axis (or y doesn't matter)</li>
<li>Angle is mapped to the y-axis</li>
<li>The entries in the matrix correspond to the values of the coloured dots in the scatter plot</li>
<li>All other entries are treated as <code>NaNs</code> (i.e. those that are undefined by a point in the scatter plot)</li>
</ul>

<p>In 3D the colour would correspond to the height.</p>

<p>I was thinking of using something like this: <a href="http://stackoverflow.com/questions/13990465/3d-numpy-array-to-2d">3d Numpy array to 2d</a> but am not quite sure how.</p>

<p>I don't use pandas, so I cannot really follow what your function does. But from the description of your array M and what you want, I think the funktion np.histogram2d is what you want. It bins the range of your independent values in equidistant steps and sums all the occurrences. You can apply weighting with your 3rd column to get the proper height. You have to choose the number of bins:</p>

<pre><code>z, x, y   = np.histogram2d(M[:,0], M[:,1], weights=M[:,2], bins=50)
num, x, y = np.histogram2d(M[:,0], M[:,1], bins=50)

z /= num # proper averaging, it also gives you NaN where num==0

plt.pcolor(x, y, z) #visualization
</code></pre>

<p>Also <code>plt.hist2d</code> could be interesting</p>

<p><strong>edit:</strong>
The histogram2d yields the 2D array which was asked for in the question. The visualization, however, should be done with imshow, since pcolor doesn't skip NaN values (is there some way to teach it?)</p>

<p>The advantage of this method is that the x,y values can be float and of arbitrary order. Further, by defining the number of bins, one can choose the resolution of the resulting image. 
Nevertheless, to get exactly the result which was asked for, one should do:</p>

<pre><code>binx = np.arange(M[:,0].min()-0.5, M[:,0].max()+1.5) # edges of the bins. 0.5 is the half width
biny = np.arange(M[:,1].min()-0.5, M[:,1].max()+1.5)

z,   x, y   = np.histogram2d(M[:,0], M[:,1], weights=M[:,2], bins=(binx,biny))
num, x, y   = np.histogram2d(M[:,0], M[:,1], bins=(binx,biny))

z /= num


plt.imshow(z.T, interpolation='none', origin = 'lower')
</code></pre>

<p><a href="http://i.stack.imgur.com/z7oXZ.png" rel="nofollow"><img src="http://i.stack.imgur.com/z7oXZ.png" alt="enter image description here"></a></p>

<p>the output of pcolor doesn't leave out the nans but therefore takes also x and y values into account:</p>

<pre><code>plt.pcolormesh(x, y, z.T, vmin=0, vmax=2)
</code></pre>

<p><a href="http://i.stack.imgur.com/vSLVm.png" rel="nofollow"><img src="http://i.stack.imgur.com/vSLVm.png" alt="enter image description here"></a></p>

python arrays numpy matplotlib