NaN from sparse_softmax_cross_entropy_with_logits in Tensorflow
<p>I am getting NaN when I attempt to use the sparse_softmax_cross_entropy_with_logits loss function in tensorflow. I have a simple network, something like:</p>

<pre><code>layer = tf.nn.relu(tf.matmul(inputs, W1) + b1)
layer = tf.nn.relu(tf.matmul(inputs, W2) + b2)
logits = tf.matmul(inputs, W3) + b3
loss = tf.sparse_softmax_cross_entropy_with_logits(logits, labels)
</code></pre>

<p>I have many classes (~10000), so I imagine I am getting NaN because the logit corresponding to correct class in at least one of my examples got truncated to zero. Is there a way to avoid this?</p>

<p><code>tf.sparse_softmax_cross_entropy_with_logits</code> handles the case of <code>log(0)</code> for you, you don't have to worry about it.</p>

<p>Usually a <code>NaN</code> is due to a high learning rate of your optimization algorithm. Try to lower it until <code>NaN</code> errors disappear and the loss starts to decrease </p>

python tensorflow