How to classify new documents with tf-idf?
<p>If I use the <code>TfidfVectorizer</code> from <code>sklearn</code> to generate feature vectors as:</p>

<p><code>features = TfidfVectorizer(min_df=0.2, ngram_range=(1,3)).fit_transform(myDocuments)</code></p>

<p>How would I then generate feature vectors to classify a new document? Since you cant calculate the tf-idf for a single document. </p>

<p>Would it be a correct approach, to extract the feature names with:</p>

<p><code>feature_names = TfidfVectorizer.get_feature_names()</code></p>

<p>and then count the term frequency for the new document according to the <code>feature_names</code> ?</p>

<p>But then I wont get the weights that has the information of a words importnance.</p>

<p>You need to save the instance of the TfidfVectorizer, it will remember the term frequencies and vocabulary that was used to fit it. It may make things clearer sense if rather than using <code>fit_transform</code>, you use <code>fit</code> and <code>transform</code> separately:</p>

<pre><code>vec = TfidfVectorizer(min_df=0.2, ngram_range=(1,3))
vec.fit(myDocuments)
features = vec.transform(myDocuments)
new_features = fec.transform(myNewDocuments)
</code></pre>

python scikit-learn text-mining tf-idf text-analysis