Python - tf-idf predict a new document similarity
<p>Inspired by <strong><a href="http://stackoverflow.com/questions/12118720/python-tf-idf-cosine-to-find-document-similarity">this</a></strong> answer, I'm trying to find cosine similarity between a trained trained tf-idf vectorizer and a new document, and return the similar documents.</p>

<p>The code below finds the cosine similarity of the <strong>first vector</strong> and not a new query</p>

<pre><code>&gt;&gt;&gt; from sklearn.metrics.pairwise import linear_kernel
&gt;&gt;&gt; cosine_similarities = linear_kernel(tfidf[0:1], tfidf).flatten()
&gt;&gt;&gt; cosine_similarities
array([ 1.        ,  0.04405952,  0.11016969, ...,  0.04433602,
    0.04457106,  0.03293218])
</code></pre>

<p>Since my train data is huge, looping through the entire trained vectorizer sounds like a bad idea.
How can I infer the vector of a new document, and find the related docs, same as the code below?</p>

<pre><code>&gt;&gt;&gt; related_docs_indices = cosine_similarities.argsort()[:-5:-1]
&gt;&gt;&gt; related_docs_indices
array([    0,   958, 10576,  3277])
&gt;&gt;&gt; cosine_similarities[related_docs_indices]
array([ 1.        ,  0.54967926,  0.32902194,  0.2825788 ])
</code></pre>

<p>For huge data sets, there is a solution called <strong>Text Clustering By Concept</strong>. search engines use this Technic,</p>

<p>At first step, you cluster your documents to some groups(e.g 50 cluster), then each cluster has a representative document(that contain some words that have  some useful information about it's cluster)<br />
At second step, for calculating cosine similarity between New Document and you data set, you loop through all representative(50 numbers) and find top near representatives(e.g 2 representative)<br />
At final step, you can loop through all documents in selected representative and find nearest cosine similarity</p>

<p>With this Technic, you can reduce the number of loops and improve performace,
You can read more tecnincs in some chapter of this book: <a href="http://nlp.stanford.edu/IR-book/html/htmledition/irbook.html" rel="nofollow">http://nlp.stanford.edu/IR-book/html/htmledition/irbook.html</a></p>

python machine-learning scikit-learn tf-idf document-classification