covariance between two columns in pandas groupby pandas
<p>I am trying to calculate the covariance between two columns by group. I am doing doing the following:</p>

<pre><code>A = pd.DataFrame({'group':['A','A','A','A','B','B','B'],
                  'value1':[1,2,3,4,5,6,7],
                  'value2':[8,5,4,3,7,8,8]})

B = A.groupby('group')

B['value1'].cov(B['value2'])
</code></pre>

<p>Ideally, I would like to get the covariance between X and Y and not the whole variance-covariance matrix, since I only have two columns.</p>

<p>Thank you,</p>

<p>You are almost there, only that you do not clear understand the groupby object, see <a href="http://pandas.pydata.org/pandas-docs/stable/groupby.html" rel="nofollow">Pandas-GroupBy</a> for more details.</p>

<p>For your problem, if I understand correctly, you would like to calculate cov between two columns in same group.</p>

<p>The simplest one is to use <code>groupeby.cov</code> function, which gives pairwise cov between groups.</p>

<pre><code>A.groupby('group').cov()

                value1    value2
group                           
A     value1  1.666667 -2.666667
      value2 -2.666667  4.666667
B     value1  1.000000  0.500000
      value2  0.500000  0.333333
</code></pre>

<p>If you only need cov(grouped_v1, grouped_v2)</p>

<pre><code>grouped = A.groupby('group')
grouped.apply(lambda x: x['value1'].cov(x['value2']))

group
A   -2.666667
B    0.500000
</code></pre>

<p>In which, <code>grouped</code> is a <code>groupby</code> object. For <code>grouped.apply</code> function, it need a callback function as argument and each group will be the argument for the callback function. Here, the callback function is a <code>lambda</code> function, and the argument <code>x</code> is a group (a DataFrame).</p>

<p>Hope this will be helpful for your understanding of groupby.</p>

python pandas