Tensorflow MNIST (Weight and bias variables)
<p>I'm learning how to use Tensorflow with the MNIST tutorial, but I'm blocking on a point of the tutorial.</p>

<p>Here is the code provided : </p>

<pre><code>from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
y_ = tf.placeholder(tf.float32, [None, 10])

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
saver = tf.train.Saver()
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for i in range(1000):
  batch_xs, batch_ys = mnist.train.next_batch(100)
  sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
</code></pre>

<p>But I actually don't understand at all how the variables "W" (The weight) and "b" (The bias) are changed while the computing ?
On each batch, they are initialized at zero, but after ?
I don't see at all where in the code they're going to change ?</p>

<p>Thanks you very much in advance!</p>

<p>TensorFlow <a href="https://www.tensorflow.org/versions/r0.11/how_tos/variables/index.html" rel="nofollow">variables</a> maintain their state from one <code>run()</code> call to the next. In your program they will be initialized to zero, and then progressively updated in the training loop.</p>

<p>The code that changes the values of the variables is created, implicitly, by this line:</p>

<pre><code>train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
</code></pre>

<p>In TensorFlow, a <a href="https://www.tensorflow.org/versions/r0.11/api_docs/python/train.html#optimizers" rel="nofollow"><code>tf.train.Optimizer</code></a> is a class that creates operations for updating variables, typically based on the gradient of some tensor (e.g. a loss) with respect to those variables. By default, when you
call <a href="https://www.tensorflow.org/versions/r0.11/api_docs/python/train.html#Optimizer.minimize" rel="nofollow"><code>Optimizer.minimize()</code></a>, TensorFlow creates operations to update <strong>all variables</strong> on which the given tensor (in this case <code>cross_entropy</code>) depends.</p>

<p>When you call <code>sess.run(train_step)</code>, this runs a graph that includes those update operations, and therefore instructs TensorFlow to update the values of the variables.</p>

python python-3.x machine-learning tensorflow