how to analyze the value of tfidf matrix in sklearn?
<p>I am using sklearn KMeans algorithm for document clustering as guided in 
<a href="http://brandonrose.org/clustering" rel="nofollow">http://brandonrose.org/clustering</a></p>

<p>Here There is a calculation of TFIDF matrix. I have understood the concept behind the TFIDF technique.
But, when I printed this Matrix in the given program, The matrix is like this,</p>

<pre><code>  (0, 11)   0.238317554822
  (0, 34)   0.355850989305
  (0, 7)    0.355850989305
  (0, 21)   0.238317554822
  (0, 16)   0.355850989305
  (0, 35)   0.355850989305
  (0, 8)    0.355850989305
  (0, 17)   0.355850989305
  (0, 36)   0.355850989305
  (1, 11)   0.238317554822
  (1, 21)   0.238317554822
  (1, 23)   0.355850989305
  (1, 0)    0.355850989305
  (1, 24)   0.355850989305
  (1, 12)   0.355850989305
  (1, 22)   0.355850989305
  (1, 25)   0.355850989305
  (1, 13)   0.355850989305
  (2, 2)    0.27430356415
  (2, 18)   0.339992197465
  (2, 26)   0.339992197465
  (2, 39)   0.339992197465
  (2, 3)    0.339992197465
  (2, 19)   0.339992197465
  (2, 27)   0.339992197465
  (2, 4)    0.339992197465
  (2, 20)   0.339992197465
  (3, 2)    0.27430356415
  (3, 40)   0.339992197465
  (3, 9)    0.339992197465
  (3, 1)    0.339992197465
  (3, 5)    0.339992197465
  (3, 41)   0.339992197465
  (3, 10)   0.339992197465
  (3, 6)    0.339992197465
  (3, 42)   0.339992197465
  (4, 11)   0.202877476983
  (4, 21)   0.202877476983
  (4, 28)   0.302932576437
  (4, 31)   0.302932576437
  (4, 37)   0.302932576437
  (4, 14)   0.302932576437
  (4, 29)   0.302932576437
  (4, 32)   0.302932576437
  (4, 38)   0.302932576437
  (4, 15)   0.302932576437
  (4, 30)   0.302932576437
  (4, 33)   0.302932576437
</code></pre>

<p>What values this matrix is representing. ? can anybody worked on this can help me to understand this ?</p>

<p>The first column contains the tuples <code>(ind_document, ind_word)</code> where <code>ind_document</code> is the index of your document (in your case a <code>string</code>) contained in your data set, and <code>ind_word</code> the index of the word in the dictionary of words generated by the <code>TfidfVectorizer</code> object.</p>

<p>The second column contains the TF-IDF value of your given <code>word</code> (the word corresponding to <code>(ind_document, ind_word)</code>.</p>

<hr>

<p><strong>UPDATE</strong></p>

<p>If you look closer to the implementation of <code>TfidfVectorizer</code><a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html" rel="nofollow">here</a>, you can see that there is a parameter called <code>norm</code>. By <strong>default</strong> this parameter is set to <code>l2</code> which is the L2-norm used to normalize the data obtained.</p>

<p>If you don't want to normalize your data and compare it to the results obtained manually <strong>change this parameter</strong> to <code>norm = None</code></p>

python scikit-learn nltk k-means tf-idf