Working with the output of groupby and groupby.size()
<p>I have a pandas dataframe containing a row for each object manipulated by participants during a user study. Each participant participates in the study 3 times, one in each of 3 conditions (<code>a</code>,<code>b</code>,<code>c</code>), working with around 300-700 objects in each condition.</p>

<p>When I report the number of objects worked with I want to make sure that this didn't vary significantly by condition (I don't expect it to have done, but need to confirm this statistically).</p>

<p>I think I want to run an ANOVA to compare the 3 conditions, but I can't figure out how to get the data I need for the ANOVA.</p>

<p>I currently have some pandas code to group the data and count the number of rows per participant per condition (so I can then use mean() and similar to summarise the data). An example with a subset of the data follows:</p>

<pre><code>&gt;&gt;&gt; tmp = df.groupby([FIELD_PARTICIPANT, FIELD_CONDITION]).size()
&gt;&gt;&gt; tmp
participant_id  condition
1               a            576
2               b            367
3               a            703
4               c            309
dtype: int64
</code></pre>

<p>To calculate the ANOVA I would normally just filter these by the condition  column, e.g.</p>

<pre><code>cond1 = tmp[tmp[FIELD_CONDITION] == CONDITION_A] 
cond2 = tmp[tmp[FIELD_CONDITION] == CONDITION_B] 
cond3 = tmp[tmp[FIELD_CONDITION] == CONDITION_C]
f_val, p_val = scipy.stats.f_oneway(cond1, cond2, cond3)
</code></pre>

<p>However, since <code>tmp</code> is a <code>Series</code> rather than the <code>DataFrame</code> I'm used to, I can't figure out how to achieve this in the normal way.</p>

<pre><code>&gt;&gt;&gt; tmp[FIELD_CONDITION]
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/Library/Python/2.7/site-packages/pandas/core/series.py", line 583, in __getitem__
    result = self.index.get_value(self, key)
  File "/Library/Python/2.7/site-packages/pandas/indexes/multi.py", line 626, in get_value
    raise e1
KeyError: 'condition'
&gt;&gt;&gt; type(tmp)
&lt;class 'pandas.core.series.Series'&gt;
&gt;&gt;&gt; tmp.index
MultiIndex(levels=[[u'1', u'2', u'3', u'4'], [u'd', u's']],
           labels=[[0, 1, 2, 3], [0, 0, 0, 1]],
           names=[u'participant_id', u'condition'])
</code></pre>

<p>I feel sure this is a straightforward problem to solve, but I can't seem to get there without some help :) </p>

<p>I think you need <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.reset_index.html" rel="nofollow"><code>reset_index</code></a> and then output is <code>DataFrame</code>:</p>

<pre><code>tmp = df.groupby([FIELD_PARTICIPANT, FIELD_CONDITION]).size().reset_index(name='count')
</code></pre>

<p>Sample:</p>

<pre><code>import pandas as pd

df = pd.DataFrame({'participant_id': {0: 1, 1: 1, 2: 1, 3: 1, 4: 2, 5: 2, 6: 2, 7: 3, 8: 4, 9: 4},
                   'condition': {0: 'a', 1: 'a', 2: 'a', 3: 'a', 4: 'b', 5: 'b', 6: 'b', 7: 'a', 8: 'c', 9: 'c'}})
print (df)
  condition  participant_id
0         a               1
1         a               1
2         a               1
3         a               1
4         b               2
5         b               2
6         b               2
7         a               3
8         c               4
9         c               4

tmp = df.groupby(['participant_id', 'condition']).size().reset_index(name='count')
print (tmp)
   participant_id condition  count
0               1         a      4
1               2         b      3
2               3         a      1
3               4         c      2
</code></pre>

<p>If need working with <code>Series</code> you can use condition where select values of level <code>condition</code> of <code>Multiindex</code> by <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Index.get_level_values.html" rel="nofollow"><code>get_level_values</code></a>:</p>

<pre><code>tmp = df.groupby(['participant_id', 'condition']).size()
print (tmp)
participant_id  condition
1               a            4
2               b            3
3               a            1
4               c            2
dtype: int64

print (tmp.index.get_level_values('condition'))
Index(['a', 'b', 'a', 'c'], dtype='object', name='condition')

print (tmp.index.get_level_values('condition') == 'a')
[ True False  True False]

print (tmp[tmp.index.get_level_values('condition') == 'a'])
participant_id  condition
1               a            4
3               a            1
dtype: int64
</code></pre>

python pandas group-by scipy condition