score_cbow_pair in word2vec (gensim)
<p>I wanted to output the log-probability during learning of the word and doc vectors in gensim. I have taken a look at the implementation of the score function in the "slow plain numpy" version.</p>

<pre class="lang-py prettyprint-override"><code>def score_cbow_pair(model, word, word2_indices, l1):
    l2a = model.syn1[word.point]  # 2d matrix, codelen x layer1_size
    sgn = (-1.0)**word.code  # ch function, 0-&gt; 1, 1 -&gt; -1
    lprob = -log(1.0 + exp(-sgn*dot(l1, l2a.T)))
    return sum(lprob)
</code></pre>

<p>The score function should make use of the parameters learned during hierarchical softmax training. But in the calculation of the log-probability there is supposed to be a sigmoid function( <a href="http://www-personal.umich.edu/~ronxin/pdf/w2vexp.pdf" rel="nofollow">word2vec Parameter Learning Explained equation (45)</a>).
So does gensim really calculate the log-probability in <code>lprob</code> or is it just a score for comparison purposes.</p>

<p>I would have calculated the log-probability as follows:
<code>-log(1.0/(1.0+exp(-sgn*dot(l1, l2a.T))))</code></p>

<p>Is this equation not used because it explodes for values close to zero or is it in general wrong?</p>

<p>I've overlooked that the logarithm of the sigmoid function can be rewritten: <code>log(1.0/(1.0+exp(-sgn*dot(l1, l2a.T)))) = log(1)-log(1.0+exp(-sgn*dot(l1, l2a.T))) = -log(1.0+exp(-sgn*dot(l1, l2a.T)))</code></p>

<p>So the code does compute the log-likelihood.</p>

python numpy probability gensim word2vec