Remove nan rows in a scipy sparse matrix
<p>I am given a (normalized) sparse adjacency matrix and a list of labels for the respective matrix rows. Because some nodes have been removed by another sanitization function, there are some rows containing NaNs in the matrix. I want to find these rows and remove them <em>as well as their respective labels</em>. Here is the function I wrote:</p>

<pre class="lang-py prettyprint-override"><code>def sanitize_nan_rows(adj, labels):
    # convert to numpy array and keep dimension
    adj = np.array(adj, ndmin=2)

    for i, row in enumerate(adj):
        # check if row all nans
        if np.all(np.isnan(row)):
            # print("Removing nan row label in %s" % i)
            # remove row index from labels
            del labels[i]
    # remove all nan rows
    adj = adj[~np.all(np.isnan(adj), axis=1)]
    # return sanitized adj and labels_clean
    return adj, labels
</code></pre>

<p><code>labels</code> is a simple Python list and <code>adj</code> has the type <code>&lt;class 'scipy.sparse.lil.lil_matrix'&gt;</code> (containing elements of type <code>&lt;class 'numpy.float64'&gt;</code>), which are both the result of</p>

<pre><code>adj, labels = nx.attr_sparse_matrix(infected, normalized=True)
</code></pre>

<p>On execution I get the following error:</p>

<pre><code>---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-503-8a404b58eaa9&gt; in &lt;module&gt;()
----&gt; 1 adj, labels = sanitize_nans(adj, labels)

&lt;ipython-input-502-ead99efec677&gt; in sanitize_nans(adj, labels)
      6     for i, row in enumerate(adj):
      7         # check if row all nans
----&gt; 8         if np.all(np.isnan(row)):
      9             print("Removing nan row label in %s" % i)
     10             # remove row index from labels

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
</code></pre>

<p>So I thought that SciPy NaNs were different from numpy NaNs. After that I tried to convert the sparse matrix into a numpy array (taking the risk of flooding my RAM, because the matrix has about 40k rows and columns). When running it, the error stays the same however. It seems that the <code>np.array()</code> call just wrapped the sparse matrix and didn't convert it, as <code>type(row)</code> inside the for loop still outputs <code>&lt;class 'scipy.sparse.lil.lil_matrix'&gt;</code></p>

<p>So my question is how to resolve this issue and whether there is a better approach that gets the job done. I am fairly new to numpy and scipy (as used in networkx), so I'd appreciate an explanation. Thank you!</p>

<p>EDIT: After changing the conversion to what <a href="http://stackoverflow.com/users/901925/hpaulj">hpaulj</a> proposed, I'm getting a MemoryError:</p>

<pre><code>---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
&lt;ipython-input-519-8a404b58eaa9&gt; in &lt;module&gt;()
----&gt; 1 adj, labels = sanitize_nans(adj, labels)

&lt;ipython-input-518-44201f4ff35c&gt; in sanitize_nans(adj, labels)
      1 def sanitize_nans(adj, labels):
----&gt; 2     adj = adj.toarray()
      3 
      4     for i, row in enumerate(adj):
      5         # check if row all nans

/usr/lib/python3/dist-packages/scipy/sparse/lil.py in toarray(self, order, out)
    348     def toarray(self, order=None, out=None):
    349         """See the docstring for `spmatrix.toarray`."""
--&gt; 350         d = self._process_toarray_args(order, out)
    351         for i, row in enumerate(self.rows):
    352             for pos, j in enumerate(row):

    /usr/lib/python3/dist-packages/scipy/sparse/base.py in_process_toarray_args(self, order, out)
    697             return out
    698         else:
--&gt; 699             return np.zeros(self.shape, dtype=self.dtype, order=order)
    700 
    701 

MemoryError: 
</code></pre>

<p>So apparently I'll have to stick with the sparse matrix to save RAM.</p>

<p>If I make a sample array:</p>

<pre><code>In [328]: A=np.array([[1,0,0,np.nan],[0,np.nan,np.nan,0],[1,0,1,0]])
In [329]: A
Out[329]: 
array([[  1.,   0.,   0.,  nan],
       [  0.,  nan,  nan,   0.],
       [  1.,   0.,   1.,   0.]])

In [331]: M=sparse.lil_matrix(A)
</code></pre>

<p>This lil sparse matrix is stored in 2 arrays:</p>

<pre><code>In [332]: M.data
Out[332]: array([[1.0, nan], [nan, nan], [1.0, 1.0]], dtype=object)
In [333]: M.rows
Out[333]: array([[0, 3], [1, 2], [0, 2]], dtype=object)
</code></pre>

<p>With your function, no rows will be removed, even though the middle row of the sparse matrix only contains <code>nan</code>.</p>

<pre><code>In [334]: A[~np.all(np.isnan(A), axis=1)]
Out[334]: 
array([[  1.,   0.,   0.,  nan],
       [  0.,  nan,  nan,   0.],
       [  1.,   0.,   1.,   0.]])
</code></pre>

<p>I could test the rows of <code>M</code> for <code>nan</code>, and identify the ones that only contain <code>nan</code> (besides 0s).  But it's probably easier to collect the ones that we want to keep. </p>

<pre><code>In [346]: ll = [i for i,row in enumerate(M.data) if not np.all(np.isnan(row))]
In [347]: ll
Out[347]: [0, 2]
In [348]: M[ll,:]
Out[348]: 
&lt;2x4 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
    with 4 stored elements in LInked List format&gt;
In [349]: _.A
Out[349]: 
array([[  1.,   0.,   0.,  nan],
       [  1.,   0.,   1.,   0.]])
</code></pre>

<p>A row of <code>M</code> is a list, but <code>np.isnan(row)</code> will convert it to an array and do it's array test.</p>

python numpy scipy sparse-matrix networkx