Tensorflow: how it trains the model?
<p>Working on Tensorflow, the first step is build a data graph and use session to run it. While, during my practice, such as the <a href="https://www.tensorflow.org/versions/r0.9/tutorials/mnist/beginners/index.html#Training" rel="nofollow">MNIST tutorial</a>. It firstly defines <em>loss</em> function and the <em>optimizer</em>, with the following codes (and the MLP model is defined before that):</p>

<pre><code>cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1])) #define cross entropy error function

loss = tf.reduce_mean(cross_entropy, name='xentropy_mean') #define loss

optimizer = tf.train.GradientDescentOptimizer(learning_rate) #define optimizer

global_step = tf.Variable(0, name='global_step', trainable=False) #learning rate

train_op = optimizer.minimize(loss, global_step=global_step) #train operation in the graph
</code></pre>

<p><strong>The training process:</strong></p>

<pre><code>train_step =tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)


for i in range(1000):
  batch_xs, batch_ys = mnist.train.next_batch(100)
  sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
</code></pre>

<p>That's how Tensorflow did training in this case. But my question is, how did Tensorflow know which weight it needs to train and update? I mean, in the training codes, we only pass output <code>y</code> to <code>cross_entropy</code>, but for <code>optimizer</code> or <code>loss</code>, we didn't pass any information about the structure <strong>directly</strong>. In addition, we use dictionary to feed batch data to <code>train_step</code>, but <code>train_step</code> didn't directly use the data. <strong>How did Tensorflow know where to use these data as input?</strong></p>

<p>To my question, I thought it might be all those variables or constants are stored in <strong>Tensor</strong>. Operations such as <code>tf.matmul()</code> should a "subclass" of Tensorflow's operation class(I haven't check the code yet). There might be some mechanism for Tensorflow to recognise relations among tensors (<code>tf.Variable()</code>, <code>tf.constant()</code>) and operations (<code>tf.mul()</code>, <code>tf.div()</code>...). I guess, it could check the <code>tf.xxxx()</code>'s super class to find out whether it is a tensor or operation. This assumption raises <strong>my second question</strong>: should I use Tensorflow's 'tf.xxx' function as possible to ensure tensorflow could build correct data flow graph, even sometimes it is more complicated than normal Python methods or some functions are supported better in Numpy than Tensorflow? </p>

<p><strong>My last question</strong> is: Is there any link between Tensorflow and C++? I heard someone said Tensorflow is faster than normal Python since it uses C or C++ as backend. Is there any transform mechanism to transfer Tensorflow Python codes to C/C++?</p>

<p>I'd also be graceful if someone could share some debugging habits in coding with Tensorflow, since currently I just set up some terminals (Ubuntu) to test each part/functions of my codes.</p>

<p>You do pass information about your structure to Tensorflow when you define your loss with:</p>

<pre><code>loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
</code></pre>

<p>Notice that with Tensorflow you build a graph of operations, and every operation you use in your code is a node in the graph.</p>

<p>When you define your <code>loss</code> you are passing the operation stored in <code>cross_entropy</code>, which depends on <code>y_</code> and <code>y</code>. <code>y_</code> is a placeholder for your input whereas <code>y</code> is the result of <code>y = tf.nn.softmax(tf.matmul(x, W) + b)</code>. See where I am going? The operation <code>loss</code> contains all the information it needs to build the model an process the input, because it depends on the operation <code>cross_entropy</code>, which depends on <code>y_</code> and <code>y</code>, which depends on the input <code>x</code> and the model weights <code>W</code>.</p>

<p>So when you call</p>

<pre><code>sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
</code></pre>

<p>Tensorflow knows perfectly which operations should be computed when you run <code>train_step</code>, and it knows exactly where to put in the operations graph the data you are passing through <code>feed_dict</code>.</p>

<p>As for how does Tensorflow know which variables should be trained, the answer is easy. It trains any <code>tf.Variable()</code> in the operations graph which is trainable. Notice how when you define the <code>global_step</code> you set <code>trainable=False</code> because you don't want to compute gradients w.r.t that variable.</p>

python tensorflow