Calculate correlation for discrete-like values from two columns of DataFrame in Pandas
<p>Here's the code snippet:</p>

<pre><code>df = pd.DataFrame(data=[1,1,2,2,3,3,3], columns =list('A'))

def m(x):
    if x == 1:
        return 2
    if x == 2:
        return 3
    if x == 3:
        return 1
    return -1
df['B'] = df['A'].map(m)
print df.head(n=10)

   A  B
0  1  2
1  1  2
2  2  3
3  2  3
4  3  1
5  3  1
6  3  1
</code></pre>

<p>As we can see, column B is created by mapping value from column A, thus they should have correlation of value 1, but what I got from below is all not satisfying. Could anyone give me some idea on how to calculate correlation of discrete data for two columns? Great thanks!</p>

<pre><code>df['A'].cov(df['B'])
-0.47619047619047611
df['A'].corr(df['B'], method='spearman')
-0.68000000000000016
df['A'].corr(df['B'], method='kendall')
-0.50000000000000011
df['A'].corr(df['B'])
-0.58823529411764708
</code></pre>

<p>The values in the 5th row move in the opposite direction, that's why you  get a  correlation of <code>-0.58823529411764708</code>. You can see that in column A the 4th value is 2 and then the 5th value is 3 so your series is increasing in this column. Instead in column B the 4th value is 3 and then the fifth value is 1 so your series is decreasing.  There is no problem with your calculation. If you calculate the correlation til the 4th row you will get a correlation coefficient = 1 because values in both columns move to the same direction.</p>

<p>You can find a nice explanation of correlation in this post: <a href="http://stats.stackexchange.com/questions/29713/what-is-covariance-in-plain-language">http://stats.stackexchange.com/questions/29713/what-is-covariance-in-plain-language</a></p>

python pandas dataframe statistics correlation