Scipy sparse matrix as DataFrame column
<p>I'm developing tooling based on pandas DataFrame objects. I would like to keep scipy sparse matrices around as column of a DataFrame without converting it row-wise to a list / numpy array of dtype('O').</p>

<p>The snippet below doesn't work as pandas treats the matrix as a scalar, and suggests to add an index. When providing a pd.RangeIndex over the row indices in the matrix, the matrix gets repeated for every row in the dataframe (as pandas thinks it is a scalar).</p>

<pre><code>ma = scipy.sparse.rand(10, 100, 0.1, 'csr', dtype=np.float64)
df = pd.DataFrame(dict(X=ma))
</code></pre>

<p>This does work:</p>

<pre><code>df = pd.DataFrame(dict(X=list(ma)))
</code></pre>

<p>However, this cuts up the matrix row-wise into CSR matrices each of 1 row. Which I would then need to vstack everytime I'd want to work on the original matrix.</p>

<p>Any pointers? I tried wrapping the CSR matrix into a pd.Series object, pretending it has dtype('O'), but I run into a lot of assumptions on the underlying data being numpy arrays and such.</p>

<p>There is a sparse dataframe or dataseries feature.  It is still experimental.  I've answered a few SO questions about converting back and forth between that and <code>scipy</code> sparse matrices.</p>

<p>From the sidebar:</p>

<p><a href="http://stackoverflow.com/questions/34181494/populate-a-pandas-sparsedataframe-from-a-scipy-sparse-coo-matrix">Populate a Pandas SparseDataFrame from a SciPy Sparse Coo Matrix</a></p>

<p>Without such a specialized pandas structure I don't see how a sparse matrix could be added to a pandas frame.  The internal structure of a sparse matrix is too different.  For a start it is not a subclass of numpy array.</p>

<p>A <code>csr</code> matrix is an object with data contained in 3 arrays, <code>ma.data</code> and <code>ma.indices</code> are 1d arrays with one value for each non-zero element of the array.  <code>ma.indptr</code> has a value for each row of the matrix.</p>

<p><code>list(ma)</code> is meaningless.  <code>ma.toarray()</code> produces a 2d array with the same data, and will all those zeros filled in as well.</p>

<p>Other sparse matrix formats store their data in other structures - 3 equal length arrays for <code>coo</code>, two lists of lists for <code>lil</code>, and a dictionary of <code>dok</code>.</p>

python pandas dataframe scipy sparse-matrix