NaN from sparse_softmax_cross_entropy_with_logits in Tensorflow
<p>I am getting NaN when I attempt to use the sparse_softmax_cross_entropy_with_logits loss function in tensorflow. I have a simple network, something like:</p>

<pre><code>layer = tf.nn.relu(tf.matmul(inputs, W1) + b1)
layer = tf.nn.relu(tf.matmul(inputs, W2) + b2)
logits = tf.matmul(inputs, W3) + b3
loss = tf.sparse_softmax_cross_entropy_with_logits(logits, labels)
</code></pre>

<p>I have many classes (~10000), so I imagine I am getting NaN because the logit corresponding to correct class in at least one of my examples got truncated to zero. Is there a way to avoid this?</p>

<p>The <code>NaN</code> error probably occurs when one of the softmaxed logits gets truncated to 0, as you have said, and then it performs log(0) to compute the cross-entropy error.</p>

<p>To avoid this, as it is suggested in <a href="http://stackoverflow.com/questions/33712178/tensorflow-nan-bug/33713196#33713196">this other answer</a>, you could clip the values of the softmax output so that they are never zero.</p>

<pre><code>out = tf.clip_by_value(out,1e-10,100.0)
</code></pre>

<p>Or you could add a small constant to avoid having zeros:</p>

<pre><code>out = out + 1e-10
</code></pre>

<p>The problem with it is that the softmax function is applied on the logits internally by <code>sparse_softmax_cross_entropy_with_logits()</code> so you can not change its behavior.</p>

<p>To overcome this, code the cross entropy error yourself and add the constant <code>1e-10</code> to the output of the softmax, not to the logits.</p>

<pre><code>loss = -tf.reduce_sum(labels*tf.log(tf.nn.softmax(logits) + 1e-10))
</code></pre>

<p>Be aware that with the <code>sparse_softmax_cross_entropy_with_logits()</code> function the variable <code>labels</code> was the numeric value of the label, but if you implement the cross-entropy loss yourself, <code>labels</code> have to be the one-hot encoding of these numeric labels.</p>

<p><strong>Update:</strong> I have corrected the answer thanks to the comment by <a href="http://stackoverflow.com/users/997378/mdaoust">@mdaoust</a>. As he said the zeros are only relevant after the softmax function has been applied to the logits, not before.</p>

python tensorflow