In Tensorflow, how can you check if gradients are correct for your custom operation?
<p>My question is, how backpropatation paths are determined e.g., when using <code>tf.slice</code>? </p>

<p>Let me take an example. Let's say, I have a K-classification problem. I can do this in a standard way like</p>

<pre><code>conv1 = # conv1+relu1+lrm1+pool1
conv2 = # from conv1
fc1 = # from conv2 to 128D fully connected + relu
fc2 = # from fc2 to K-D fully connected
batch_loss = tf.softmax_cross_entropy(fc2, labels)
loss = tf.reduce_mean(batch_loss)
... minimize(loss)
</code></pre>

<p>In this case, the gradient of <code>loss</code> will backpropagate to each weights.</p>

<p>Let's say, I calcuated a custom loss value by slicing <code>labels</code> and <code>fc2</code> output (maybe since I think a certain class is more important?)</p>

<pre><code>label_sub = tf.slice(labels, ..)
output_sub =  tf.slice(fc2, ..)
batch_loss_sub = tf.softmax_cross_entropy(output_sub, label_sub)
loss = tf.reduce_mean(batch_loss + batch_loss_sub)
... minimize(loss)
</code></pre>

<p>In this case, I am not getting how back-propagation would work. From the "slicing", didn't we lose backprob paths?</p>

<p>This might be a weird pseudo-code, but my question is "when using tf.slice, how backpropagation work?" </p>

<p>There is nothing magical about it. You extract a part of the tensor and use for computations, so all the partial derivatives that "flow through" this slice are well defined. From math perspective it is something among the lines of having</p>

<pre><code>f([x1,x2,x3,x4]) = f(x) = 2 * sum(slice(x, 2, 2)) + 1 = 2 * (x2 + x3) + 1
</code></pre>

<p>and you can compute the gradient directly</p>

<pre><code>grad f(x) = [df / dx1, df / dx2, df / dx3, df / dx4] = [0, 2, 2, 0]
</code></pre>

<p>And now when you <strong>add</strong> this f your original loss g, from properties of the gradients, it is added as well. So </p>

<pre><code>grad (f + g)(x) = grad f(x) + grad g(x) = [0, 2, 2, 0] + grad g(x)
</code></pre>

<p>Everything works fine.</p>

<p>In particular, you can always <strong>visualize your graph</strong> in the TensorBoard, double click on "gradient" node and you will see exactly each operation used to compute your gradients.</p>

python tensorflow deep-learning