Sparse matrix slicing using list of int
<p>I'm writing a machine learning algorithm on huge &amp; sparse data (my matrix is of shape (347, 5 416 812 801) but very sparse, only 0.13% of the data is non zero. </p>

<p>My sparse matrix's size is 105 000 bytes (&lt;1Mbytes) and is of <code>csr</code> type.</p>

<p>I'm trying to separate  train/test sets by choosing a list of examples indices for each.
So I want to split my dataset in two using :</p>

<pre><code>training_set = matrix[train_indices]
</code></pre>

<p>of shape <code>(len(training_indices), 5 416 812 801)</code>, still sparse</p>

<pre><code>testing_set = matrix[test_indices]
</code></pre>

<p>of shape <code>(347-len(training_indices), 5 416 812 801)</code> also sparse</p>

<p>With <code>training_indices</code> and <code>testing_indices</code> two <code>list</code> of <code>int</code></p>

<p>But <code>training_set = matrix[train_indices]</code> seems to fail and return a <code>Segmentation fault (core dumped)</code></p>

<p>It might not be a problem of memory, as I'm running this code on a server with 64Gbytes of RAM.</p>

<p>Any clue on what could be the cause ?</p>

<p>I think I've recreated the <code>csr</code> row indexing with:</p>

<pre><code>def extractor(indices, N):
   indptr=np.arange(len(indices)+1)
   data=np.ones(len(indices))
   shape=(len(indices),N)
   return sparse.csr_matrix((data,indices,indptr), shape=shape)
</code></pre>

<p>Testing on a <code>csr</code> I had hanging around:</p>

<pre><code>In [185]: M
Out[185]: 
&lt;30x40 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
    with 76 stored elements in Compressed Sparse Row format&gt;

In [186]: indices=np.r_[0:20]

In [187]: M[indices,:]
Out[187]: 
&lt;20x40 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
    with 57 stored elements in Compressed Sparse Row format&gt;

In [188]: extractor(indices, M.shape[0])*M
Out[188]: 
&lt;20x40 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
    with 57 stored elements in Compressed Sparse Row format&gt;
</code></pre>

<p>As with a number of other <code>csr</code> methods, it uses matrix multiplication to produce the final value.  In this case with a sparse matrix with 1 in selected rows.  Time is actually a bit better.</p>

<pre><code>In [189]: timeit M[indices,:]
1000 loops, best of 3: 515 Âµs per loop
In [190]: timeit extractor(indices, M.shape[0])*M
1000 loops, best of 3: 399 Âµs per loop
</code></pre>

<p>In your case the extractor matrix is (len(training_indices),347) in shape, with only <code>len(training_indices)</code> values.  So it is not big.</p>

<p>But if the <code>matrix</code> is so large (or at least the 2nd dimension so big) that it produces some error in the matrix multiplication routines, it could give rise to segmentation fault without python/numpy trapping it.</p>

<p>Does <code>matrix.sum(axis=1)</code> work.  That too uses a matrix multiplication, though with a dense matrix of 1s.   Or <code>sparse.eye(347)*M</code>, a similar size matrix multiplication?</p>

python scipy segmentation-fault sparse-matrix