Python - tf-idf predict a new document similarity
<p>Inspired by <strong><a href="http://stackoverflow.com/questions/12118720/python-tf-idf-cosine-to-find-document-similarity">this</a></strong> answer, I'm trying to find cosine similarity between a trained trained tf-idf vectorizer and a new document, and return the similar documents.</p>

<p>The code below finds the cosine similarity of the <strong>first vector</strong> and not a new query</p>

<pre><code>&gt;&gt;&gt; from sklearn.metrics.pairwise import linear_kernel
&gt;&gt;&gt; cosine_similarities = linear_kernel(tfidf[0:1], tfidf).flatten()
&gt;&gt;&gt; cosine_similarities
array([ 1.        ,  0.04405952,  0.11016969, ...,  0.04433602,
    0.04457106,  0.03293218])
</code></pre>

<p>Since my train data is huge, looping through the entire trained vectorizer sounds like a bad idea.
How can I infer the vector of a new document, and find the related docs, same as the code below?</p>

<pre><code>&gt;&gt;&gt; related_docs_indices = cosine_similarities.argsort()[:-5:-1]
&gt;&gt;&gt; related_docs_indices
array([    0,   958, 10576,  3277])
&gt;&gt;&gt; cosine_similarities[related_docs_indices]
array([ 1.        ,  0.54967926,  0.32902194,  0.2825788 ])
</code></pre>

<p>You should take a look at <a href="https://radimrehurek.com/gensim/tutorial.html" rel="nofollow">gensim</a>. Example starting code looks like this:</p>

<pre><code>from gensim import corpora, models, similarities

dictionary = corpora.Dictionary(line.lower().split() for line in open('corpus.txt'))
corpus = [dictionary.doc2bow(line.lower().split()) for line in open('corpus.txt')]

tfidf = models.TfidfModel(corpus)
index = similarities.SparseMatrixSimilarity(tfidf[corpus], num_features=12)
</code></pre>

<p>At prediction time you first get the vector for the new doc:</p>

<pre><code>doc = "Human computer interaction"
vec_bow = dictionary.doc2bow(doc.lower().split())
vec_tfidf = tfidf[vec_bow]
</code></pre>

<p>Then get the similarities (sorted by most similar):</p>

<pre><code>sims = index[vec_tfidf] # perform a similarity query against the corpus
print(list(enumerate(sims))) # print (document_number, document_similarity) 2-tuples
</code></pre>

<p>This does a linear scan like you wanted to do but they have a more optimized implementation. If the speed is not enough then you can look into approximate similarity search (Annoy, Falconn, NMSLIB).</p>

python machine-learning scikit-learn tf-idf document-classification