TF-IDF score from TfIdfTransformer in sklearn on same word in two sentences with same frequency
<p>If I have two sentences containing the same word, and this word appears with the same counts (frequency) in both sentences, why is the Tf-Idf score I get different for them?</p>

<p>Consider this list of texts:</p>

<pre><code>data = [
    'Jumper in knit.',
    'A sweater in knit, black sweater.',
]
</code></pre>

<p>and consider that I fit and transform a <code>CountVectorizer</code> and a <code>TfIdfTransformer</code> on it, as in </p>

<pre><code>count_vec = CountVectorizer(stop_words='english')
tf_transformer = TfidfTransformer(use_idf=True)

X_counts = count_vec.fit_transform(x_data_manual)
X_tfidf = tf_transformer.fit_transform(X_counts)
</code></pre>

<p>Then I print the features with their IDFs scores:</p>

<pre><code>print zip(count_vec.get_feature_names(), tf_transformer.idf_)
</code></pre>

<p>obtaining</p>

<blockquote>
  <p>[(u'black', 1.4054651081081644), (u'jumper', 1.4054651081081644), (u'knit', 1.0), (u'sweater', 1.4054651081081644)]</p>
</blockquote>

<p>so we can see that all tokens have the same IDF except 'knit', all legit.</p>

<p>If I now ask for printing both the counts matrix and the TF-IDFs matrix, </p>

<pre><code>print X_counts.todense()
print X_tfidf.todense()
</code></pre>

<p>what I get is, respectively </p>

<blockquote>
  <p>[[0 1 1 0]<br>
   [1 0 1 2]]</p>
</blockquote>

<p>(which is legit), and</p>

<blockquote>
  <p>[[ 0.     0.815  0.58   0.   ] </p>
  
  <p>[ 0.426  0.     0.303  0.852]]</p>
</blockquote>

<p>Now, I thought the TF-IDF score would be a multiplication of the term frequency (however calculated from the count/raw frequency) and the IDF, but I'm seeing 'knit' having a different score in the two sentences despite the frequencies are the same. </p>

<p>So what I'm missing/misunderstanding?</p>

<p>Reason is, after some more digging, that in the default configuration of the <code>TfIdfTransformer</code>, the scores get normalised in L2 norm at the end, per row.</p>

<p>In fact, the kwarg <code>norm</code> is documents in the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html" rel="nofollow">docs</a>, but when reading I didn't properly understand what was being normalised, maybe a few more words would help.</p>

<p>A step by step explanation is reported <a href="http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction" rel="nofollow">here</a>.</p>

python scikit-learn tf-idf