Pandas: Summing arrays as as an aggregation with multiple groupby columns
<p>I'm using Python 3.5.1 and Pandas 0.18.0.</p>

<p>Let's say I have a Pandas dataframe with multiple columns.  The dataframe has one column that includes a numpy array. Here is an example:</p>



<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; import pandas as pd
&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt; df = pd.DataFrame([{'A': 'Label1', 'B': 'yellow', 'C': np.array([0,0,0]), 'D': 1},
                       {'A': 'Label2', 'B': 'yellow', 'C': np.array([1,1,1]), 'D': 4},
                       {'A': 'Label1', 'B': 'yellow', 'C': np.array([1,0,1]), 'D': 2},
                       {'A': 'Label2', 'B': 'green', 'C': np.array([1,1,0]), 'D': 3}])
&gt;&gt;&gt; df
        A       B          C  D
0  Label1  yellow  [0, 1, 0]  1
1  Label2  yellow  [1, 1, 1]  4
2  Label1  yellow  [1, 0, 1]  2
3  Label2   green  [1, 1, 0]  3
</code></pre>

<p>I want to create a dataframe that groups by columns A and B and aggregates columns C and D with a sum. <strong>Like this:</strong></p>

<pre class="lang-py prettyprint-override"><code>               C         D
A      B
Label1 yellow  [1, 1, 1] 3
Label2 green   [1, 1, 0] 3
       yellow  [1, 1, 1] 4
</code></pre>

<p>When I try and do the aggregation using the entire dataframe, column C (the one with the numpy arrays) is not returned:</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; df.groupby(['A','B']).sum()
               D
A      B
Label1 yellow  3
Label2 green   3
       yellow  4
</code></pre>

<p>If I ignore column D and only attempt to output column C, I get an error:</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; df[['A','B','C']].groupby(['A','B']).sum()
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\pandas\core\groupby.py", line 96, in f
    return self._cython_agg_general(alias, numeric_only=numeric_only)
  File "C:\Anaconda3\lib\site-packages\pandas\core\groupby.py", line 3038, in _cython_agg_general
    how, numeric_only=numeric_only)
  File "C:\Anaconda3\lib\site-packages\pandas\core\groupby.py", line 3084, in _cython_agg_blocks
    raise DataError('No numeric types to aggregate')
pandas.core.base.DataError: No numeric types to aggregate
</code></pre>

<p>If I group by only a single column and only output my array column, the arrays sum correctly:</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; df[['A','C']].groupby(['A']).sum()
                C
A
Label1  [1, 1, 1]
Label2  [2, 2, 1]
</code></pre>

<p>But if I try to include the scalar column as an aggregate as well, my array column again is not returned:</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; df[['A','C','D']].groupby(['A']).sum()
        D
A
Label1  3
Label2  7
</code></pre>

<p>Also, if I try and include column B (contains strings) in the aggregate function, columns B and C return but column D does not:</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; df[['A','B','C']].groupby(['A']).sum()
               B          C
A
Label1  yellowyellow  [1, 1, 1]
Label2   yellowgreen  [2, 2, 1]
</code></pre>

<p>Can anyone explain why this is happening?  I know I could create a [A+B] column and then group by that, sum my array column, and then merge the result it back in with the rest of my data on column [A+B], but it seems like there should be a much simpler way. Any ideas?</p>

<p><code>pd.concat</code> separate groupbys is a workaround</p>

<pre><code>g = df.groupby(['A', 'B'])
pd.concat([g.C.apply(np.sum), g.D.sum()], axis=1)
</code></pre>

<p><a href="http://i.stack.imgur.com/HzryQ.png" rel="nofollow"><img src="http://i.stack.imgur.com/HzryQ.png" alt="enter image description here"></a></p>

python pandas numpy