Python - tf-idf predict a new document similarity
<p>Inspired by <strong><a href="http://stackoverflow.com/questions/12118720/python-tf-idf-cosine-to-find-document-similarity">this</a></strong> answer, I'm trying to find cosine similarity between a trained trained tf-idf vectorizer and a new document, and return the similar documents.</p>

<p>The code below finds the cosine similarity of the <strong>first vector</strong> and not a new query</p>

<pre><code>&gt;&gt;&gt; from sklearn.metrics.pairwise import linear_kernel
&gt;&gt;&gt; cosine_similarities = linear_kernel(tfidf[0:1], tfidf).flatten()
&gt;&gt;&gt; cosine_similarities
array([ 1.        ,  0.04405952,  0.11016969, ...,  0.04433602,
    0.04457106,  0.03293218])
</code></pre>

<p>Since my train data is huge, looping through the entire trained vectorizer sounds like a bad idea.
How can I infer the vector of a new document, and find the related docs, same as the code below?</p>

<pre><code>&gt;&gt;&gt; related_docs_indices = cosine_similarities.argsort()[:-5:-1]
&gt;&gt;&gt; related_docs_indices
array([    0,   958, 10576,  3277])
&gt;&gt;&gt; cosine_similarities[related_docs_indices]
array([ 1.        ,  0.54967926,  0.32902194,  0.2825788 ])
</code></pre>

<p>This problem can be partially addressed by combining the <strong>vector space model</strong> (which is the tf-idf &amp; cosine similarity) together with the <strong>boolean model</strong>. These are concepts of information theory and they are used (and nicely explained) in <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/scoring-theory.html" rel="nofollow">ElasticSearch</a>-  a pretty good search engine.</p>

<p>The idea is simple: you store your documents as inverted indices. Which is comparable to the words present at the end of a book, which hold a reference to the pages (documents) they were mentioned in.</p>

<p>Instead of calculating the tf-idf vector for all document it will only calculate it for documents that have at least one (or specify a threshold) of the words in common. This can be simply done by looping over the words in the queried document, finding the documents that also have this word using the inverted index and calculate the similarity for those.</p>

python machine-learning scikit-learn tf-idf document-classification