Python Spacy beginner : similarities function
<p>In the tutorial example of spaCy in Python the results of <code>apples.similarity(oranges)</code> is
<code>0.39289959293092641</code>
instead of  <code>0.7857989796519943</code></p>

<p>Any reasons for that?
Original docs of the tutorial 
<a href="https://spacy.io/docs/" rel="nofollow">https://spacy.io/docs/</a>
A tutorial with a different answer to the one I get: 
<a href="http://textminingonline.com/getting-started-with-spacy" rel="nofollow">http://textminingonline.com/getting-started-with-spacy</a></p>

<p>Thanks</p>

<p>That appears to be a bug in spacy.</p>

<p>Somehow <code>vector_norm</code> is incorrectly calculated. </p>

<pre><code>import spacy
import numpy as np
nlp = spacy.load("en")
# using u"apples" just as an example
apples = nlp.vocab[u"apples"]
print apples.vector_norm
# prints 1.4142135381698608, or sqrt(2)
print np.sqrt(np.dot(apples.vector, apples.vector))
# prints 1.0
</code></pre>

<p>Then <code>vector_norm</code> is used in <code>similarity</code>, which always returns a value that is always half of the correct value.</p>

<pre><code>def similarity(self, other):
    if self.vector_norm == 0 or other.vector_norm == 0:
        return 0.0
    return numpy.dot(self.vector, other.vector) / (self.vector_norm * other.vector_norm)
</code></pre>

<p>If you are ranking similarity scores for synonyms, this might be OK. But if you need the correct cosine similarity score, then the result is incorrect.</p>

<p>I submitted the issue <a href="https://github.com/explosion/spaCy/issues/522" rel="nofollow">here</a>. Hopefully it will get fixed soon.</p>

python nlp spacy