Extracting Adam update rates in Tensorflow
<p>I want to find out and keep track of Adam update rate of</p>

<pre><code>lr_t &lt;- learning_rate * sqrt(1 - beta2^t) / (1 - beta1^t)
</code></pre>

<p>I followed another post and got the nodes of a graph by </p>

<pre><code>[n.name for n in tf.get_default_graph().as_graph_def().node]
</code></pre>

<p>In which I found </p>

<pre><code>u'Adam/update',
u'Adam/value',
u'Adam',
</code></pre>

<p>I am assume that either of these three could be the updated learning rate I'm looking for but I don't know how to extract it. Please help </p>

<p>Suppose you've defined an optimizer:</p>

<pre><code>optm = tf.train.AdamOptimizer(learning_rate = 0.01)
train_step = optm.minimize(L)
</code></pre>

<p>Now you could reach v_t and m_t with the following code:</p>

<pre><code>optm.get_slot(W, 'v')
optm.get_slot(W, 'm')
</code></pre>

<p>These values are tensors so they are to be evaluated by running a session.</p>

<p>W is some previously defined variable. </p>

<p>Explore more by examing <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adam.py" rel="nofollow">the source</a>.</p>

python machine-learning tensorflow deep-learning