Hierarchical Clustering using Python on Correlation Coefficient
<p>I have the data in 50 by 50 Matrix that represents the 50 Journals with their correlation. Now, I am trying to plot the graph showing on which clusters those 50 Journals fall based on the data. </p>

<p>1) I prefer to use complete-linkage or Ward's method to do the clusters.
2) I am stuck at where to begin the clustering as the documentation in scikit-learn is too technical for me
3) Could you please help me to give a kick-start? </p>

<p>Thank you very much in advance...</p>

<p>My all data falls between 0 and 1 as it is correlation coefficients.</p>

<p>Example of Data Sample (50*50):</p>

<p>data = [[ 1.          0.49319094  0.58838586 ...,  0.11433441  0.6450184
   0.60842821]</p>

<p>[ 0.49319094  1.          0.39311674 ..., -0.00795401  0.42944597
   0.68855177]</p>

<p>[ 0.58838586  0.39311674  1.         ...,  0.39785574  0.864322
   0.68910632]</p>

<p>..., </p>

<p>[ 0.11433441 -0.00795401  0.39785574 ...,  1.          0.38623474
   0.34228516]</p>

<p>[ 0.6450184   0.42944597  0.864322   ...,  0.38623474  1.          0.65408474]</p>

<p>[ 0.60842821  0.68855177  0.68910632 ...,  0.34228516  0.65408474  1.        ]]</p>

<p>Python expects <em>distances</em>, i.e. low values are better.</p>

<p>Ward is designed for squared Euclidean, so while it may work with correlation, the support from theory may be weak. Complete linkage will be supported.</p>

<p>What about negative correlations - how do you want to treat them?</p>

<p>I believe I know three popular transformations:</p>

<ol>
<li><code>1 - p**2</code> (depending on the implementation, this may be a good choice with Ward because of the square)</li>
<li><code>1 - abs(p)</code></li>
<li><code>1 - p</code> (This will treat negative correlations as bad!)</li>
</ol>

<p>Make sure to set your metric to <code>precomputed</code>. And get used to reading sklearn documentation. It is one of the least technical you will find, so you better become more technical yourself then.</p>

python scikit-learn cluster-analysis correlation hierarchical-clustering