Group by sparse matrix in scipy and return a matrix
<p>There are a few questions on SO dealing with using <code>groupby</code> with sparse matrices. However the output seem to be lists, <a href="http://stackoverflow.com/questions/35410839/group-by-on-scipy-sparse-matrix">dictionaries</a>, <a href="http://stackoverflow.duapp.com/questions/30295570/groupby-sum-sparse-matrix-in-pandas-or-scipy-looking-for-performance?rq=1" rel="nofollow">dataframes</a> and other objects. </p>

<p>I'm working on an NLP problem and would like to keep all the data in sparse scipy matrices during processing to prevent memory errors.</p>

<p>Here's the context:</p>

<p>I have vectorized some documents (<a href="http://andrewbrown.ca/data/groupbysparsematrix%20.csv" rel="nofollow">sample data here</a>): </p>

<pre><code>import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

df = pd.read_csv('groupbysparsematrix.csv')
docs = df['Text'].tolist()

vectorizer = CountVectorizer()
train_X = vectorizer.fit_transform(docs)

print("Dimensions of training set: {0}".format(train_X.shape))
print type(train_X)

Dimensions of training set: (8, 180)
&lt;class 'scipy.sparse.csr.csr_matrix'&gt;
</code></pre>

<p>From the original dataframe I use the date, in a day of the year format, to create the groups I would like to sum over:</p>

<pre><code>from scipy import sparse, hstack    

df['Date'] = pd.to_datetime(df['Date'])
groups = df['Date'].apply(lambda x: x.strftime('%j'))
groups_X = sparse.csr_matrix(groups.astype(float)).T
train_X_all = sparse.hstack((train_X, groups_X))

print("Dimensions of concatenated set: {0}".format(train_X_all.shape))

Dimensions of concatenated set: (8, 181)
</code></pre>

<p>Now I'd like to apply <code>groupby</code> (or a similar function) to find the sum of each token per day. I would like the output to be another sparse scipy matrix.   </p>

<p>The output matrix would be 3 x 181 and look something like this:</p>

<pre><code> 1, 1, 1, ..., 2, 1, 3
 2, 1, 3, ..., 1, 1, 4
 0, 0, 0, ..., 1, 2, 5
</code></pre>

<p>Where the columns 1 to 180 represent the tokens and column 181 represents the day of the year.</p>

<p>The best way of calculating the sum of selected columns (or rows) of a <code>csr</code> sparse matrix is a matrix product with another sparse matrix that has 1's where you want to sum.  In fact <code>csr</code> sum (for a whole row or column) works by matrix product, and index rows (or columns) is also done with a product (<a href="http://stackoverflow.com/a/39500986/901925">http://stackoverflow.com/a/39500986/901925</a>)</p>

<p>So I'd group the dates array, and use that information to construct the summing 'mask'.</p>

<p>For sake of discussion, consider this dense array:</p>

<pre><code>In [117]: A
Out[117]: 
array([[0, 2, 7, 5, 0, 7, 0, 8, 0, 7],
       [0, 0, 3, 0, 0, 1, 2, 6, 0, 0],
       [0, 0, 0, 0, 2, 0, 5, 0, 0, 0],
       [4, 0, 6, 0, 0, 5, 0, 0, 1, 4],
       [0, 0, 0, 0, 0, 1, 0, 0, 0, 0],
       [0, 7, 0, 8, 1, 0, 9, 0, 2, 4],
       [9, 0, 8, 4, 0, 0, 0, 0, 9, 7],
       [0, 0, 0, 1, 2, 0, 2, 0, 4, 7],
       [3, 0, 1, 0, 0, 0, 0, 0, 0, 2],
       [0, 0, 1, 8, 5, 0, 0, 0, 8, 0]])
</code></pre>

<p>Make a sparse copy:</p>

<pre><code>In [118]: M=sparse.csr_matrix(A)
</code></pre>

<p>generate some groups, based on the last column; <code>collections.defaultdict</code> is a convenient tool to do this:</p>

<pre><code>In [119]: grps=defaultdict(list)
In [120]: for i,v in enumerate(A[:,-1]):
     ...:     grps[v].append(i)

In [121]: grps
Out[121]: defaultdict(list, {0: [1, 2, 4, 9], 2: [8], 4: [3, 5], 7: [0, 6, 7]})
</code></pre>

<p>I can iterate on those groups, collect rows of <code>M</code>, sum across those rows and produce:</p>

<pre><code>In [122]: {k:M[v,:].sum(axis=0) for k, v in grps.items()}
Out[122]: 
{0: matrix([[0, 0, 4, 8, 7, 2, 7, 6, 8, 0]], dtype=int32),
 2: matrix([[3, 0, 1, 0, 0, 0, 0, 0, 0, 2]], dtype=int32),
 4: matrix([[4, 7, 6, 8, 1, 5, 9, 0, 3, 8]], dtype=int32),
 7: matrix([[ 9,  2, 15, 10,  2,  7,  2,  8, 13, 21]], dtype=int32)}
</code></pre>

<p>In the last column, values include 2*4, and 3*7</p>

<p>So there are 2 tasks - collecting the groups, whether with this defaultdict, or <code>itertools.groupby</code> (which in this case would require sorting), or <code>pandas</code> groupby.  And secondly this collection of rows and summing.  This dictionary iteration is conceptually simple. </p>

<p>A masking matrix might work like this:</p>

<pre><code>In [141]: mask=np.zeros((10,10),int)
In [142]: for i,v in enumerate(A[:,-1]): # same sort of iteration
     ...:     mask[v,i]=1
     ...:     
In [143]: Mask=sparse.csr_matrix(mask)
...
In [145]: Mask.A
Out[145]: 
array([[0, 1, 1, 0, 1, 0, 0, 0, 0, 1],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 1, 0],
       ....
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], dtype=int32)
In [146]: (Mask*M).A
Out[146]: 
array([[ 0,  0,  4,  8,  7,  2,  7,  6,  8,  0],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
       [ 3,  0,  1,  0,  0,  0,  0,  0,  0,  2],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
       [ 4,  7,  6,  8,  1,  5,  9,  0,  3,  8],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
       [ 9,  2, 15, 10,  2,  7,  2,  8, 13, 21],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0]], dtype=int32)
</code></pre>

<p>This <code>Mask*M</code> has the same values as the dictionary row, but with the extra 0s.  I can isolate the nonzero values with the <code>lil</code> format:</p>

<pre><code>In [147]: (Mask*M).tolil().data
Out[147]: 
array([[4, 8, 7, 2, 7, 6, 8], [], [3, 1, 2], [],
       [4, 7, 6, 8, 1, 5, 9, 3, 8], [], [],
       [9, 2, 15, 10, 2, 7, 2, 8, 13, 21], [], []], dtype=object)
</code></pre>

<p>I can construct the <code>Mask</code> matrix directly using the <code>coo</code> sparse style of input:</p>

<pre><code>Mask = sparse.csr_matrix((np.ones(A.shape[0],int),
    (A[:,-1], np.arange(A.shape[0]))), shape=(A.shape))
</code></pre>

<p>That should be faster and avoid the memory error (no loop or large dense array).</p>

python matrix scipy nlp