Quick method to make 2 dimensional sparse arrays from a master data array
<p>I will simplify my problem with an example of what I would like done (the actual data I am working with is massive).</p>

<p>The solution to my problem would be simple if there was a 3 dimensional sparse array object that could be fed the x,y,z coordinates along with the data to be stored. Note that I will have memory issues if I do not use a sparse array. Lets say I have an array:</p>

<pre><code>    import numpy as np
    from scipy.sparse import coo_matrix
    arr=np.array([1,3,4,5],[0,6,7,8],[1,7,8,7]])
</code></pre>

<p>The first 3 entries are the x,y,z. Ideally I would want a sparse matrix
that would yield B[1,3,4]=5 and B[0,6,7]=8, etc. Right now I am using a dictionary where the x coordinate is the key. y and z will be the sparse array coordinates. So its something like this:</p>

<pre><code>    row=arr[:,1]
    col=arr[:,2]
    data=arr[:,3]
    dic={}
    ### x here goes from only 0 to 1 in this simple example
    for x in range(2):
        bool=arr[:,0][arr[:,0]==x] ####### this step takes too long
        ###### now create sparse matrix for all data with 
        ###### x value equal to the iterator
        dic[x]=coo_matrix((data[bool], (row[bool],     
        col[bool])), shape=(1536, 32768)) ### this part ok
</code></pre>

<p>So the bool test that I marked down is the step that is taking up 90 percent of the time. This is due to the fact that my array is massive.
Ideally I want to just index my way into everything all at once. </p>

<p>Solutions?</p>

<p>Thanks!</p>

<p>Thanks!       </p>

<p>Here's an alternative approach -</p>

<pre><code>out_shp = (10,10)  # (1536, 32768) for your actual case
out_dic = {}
for i in np.unique(arr[:,0]):
    RCD = arr[arr[:,0] == i,1:]
    out_dic[i] = coo_matrix((RCD[:,2],(RCD[:,0],RCD[:,1])), shape=out_shp)
</code></pre>

<p>Sample input, output -</p>

<pre><code>In [87]: arr
Out[87]: 
array([[1, 3, 4, 5],
       [0, 6, 7, 8],
       [1, 7, 8, 7]])

In [88]: out_dic[0].toarray()
Out[88]: 
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 8, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])

In [89]: out_dic[1].toarray()
Out[89]: 
array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 5, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 7, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
</code></pre>

<hr>

<p>Here's another approach that sorts the array based on the first column before going into the loop and simply slices inside the loop. This is a meant as an approach with focus on performance as we would be doing less computations inside the loop as that seems to be the bottleneck. So, we would have another implementation like so -</p>

<pre><code>s_arr = arr[arr[:,0].argsort()] # Sorted array
unq,start_idx = np.unique(s_arr[:,0],return_index=1)
stop_idx = np.append(start_idx[1:],arr.shape[0])
out_shp = (10,10)  # (1536, 32768) for your actual case
out_dic = {}
for i,u in enumerate(unq):
    RCD = s_arr[start_idx[i]:stop_idx[i],1:] # Row-col-data
    out_dic[u] = coo_matrix((RCD[:,2],(RCD[:,0],RCD[:,1])), shape=out_shp)
</code></pre>

python numpy indexing sparse-matrix