TensorFlow: How to write multistep decay
<p>There is multistep decay in Caffe. It is calculated as <code>base_lr * gamma ^ (floor(step))</code> where <code>step</code> is incremented after each of your decay steps. For example with <code>[100, 200]</code> decay steps and <code>global step=101</code> I want get <code>base_lr * gamma ^ 1</code>, for <code>global step=201</code> and more I want get <code>base_lr * gamma ^ 2</code> and so on.</p>

<p>I tried to implement it based on exponential decay sources but I can do nothing. Here is code of exponential decay (<a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/learning_rate_decay.py#L27" rel="nofollow">https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/learning_rate_decay.py#L27</a> ):</p>

<pre><code>def exponential_decay(learning_rate, global_step, decay_steps, decay_rate,
staircase=False, name=None):
  with ops.name_scope(name, "ExponentialDecay",
                      [learning_rate, global_step,
                       decay_steps, decay_rate]) as name:
    learning_rate = ops.convert_to_tensor(learning_rate, name="learning_rate")
    dtype = learning_rate.dtype
    global_step = math_ops.cast(global_step, dtype)
    decay_steps = math_ops.cast(decay_steps, dtype)
    decay_rate = math_ops.cast(decay_rate, dtype)
    p = global_step / decay_steps
    if staircase:
      p = math_ops.floor(p)
return math_ops.mul(learning_rate, math_ops.pow(decay_rate, p), name=name)
</code></pre>

<p>I must pass <code>decay_steps</code> as some sort of array - python array or Tensor. Also I must(?) pass <code>current_decay_step</code>  (<code>step</code> in above formula).</p>

<p><strong>First option:</strong> In pure python without tensors it is very simple:</p>

<pre><code>decay_steps.append(global_step)
p = sorted(decay_steps).index(global_step) # may be there must be `+1` or `-1`. I hope that main idea is clear
</code></pre>

<p>I cant' do it because there is no sort in TF. I don't know how many time it take to implement it.</p>

<p><strong>Second option</strong>: something like code below. It doesn't work for many reasons. Firstly I don't know how to pass args to funtion in <code>tf.cond</code>. Secondly, it may not work even if I will pass args: <a href="http://stackoverflow.com/questions/34931121/can-cond-support-tf-ops-with-side-effects">Can cond support TF ops with side effects?</a></p>

<pre><code>def new_decay_step(decay_steps):
        decay_steps = decay_steps[1:]
        current_decay_step.assign(current_decay_step + 1)
        return tf.no_op()

tf.cond(tf.greater(tf.shape(decay_steps)[0], 0),
                             tf.cond(tf.greater(global_step, decay_steps[0]), new_decay_step, tf.no_op()),
tf.no_op())

p = current_decay_step
</code></pre>

<p><strong>Third option:</strong> It will not work because I can't get element with <code>tensor[another_tensor]</code>.</p>

<pre><code>    # if len(decay_steps) &gt; (current_step + 1):
    #    if global_step &gt; decay_steps[current_step + 1]:
    #        current_step += 1


    current_decay_step = tf.cond(tf.greater(tf.shape(current_decay_step)[0], tf.add(current_decay_step,1)),
                                 tf.cond(tf.greater(global_step, decay_steps[tf.add(current_decay_step + 1]), tf.add(current_decay_step,1), tf.add(current_decay_step,0)),
                                 tf.add(current_decay_step, 0)
</code></pre>

<p>What can I do?</p>

<p><strong>UPD:</strong> I almost can make it with second option. </p>

<p>I can make </p>

<pre><code>   def nothing: return tf.no_op()
   tf.cond(tf.greater(global_step, decay_steps[0]),
                    functools.partial(new_decay_step, decay_steps),
                    nothing)
</code></pre>

<p>But for some reason inner <code>tf.cond</code> doesn't work</p>

<p>For this code I get error <code>fn1 must be callable</code></p>

<pre><code>   def nothing: return tf.no_op()
   tf.cond(tf.greater(tf.shape(decay_steps)[0], 0),
            tf.cond(tf.greater(global_step, decay_steps[0]),
                    functools.partial(new_decay_step, decay_steps),
                    nothing),
            nothing)
</code></pre>

<p><strong>UPD2:</strong> Inner <code>tf.cond</code> will not work because they return tensor and args must be functions. </p>

<p>I didn't check it but seems like it works (at least it doesn't crash with errors):</p>

<pre><code> tf.cond(tf.logical_and(tf.greater(tf.shape(decay_steps)[0], 0),  tf.greater(global_step, decay_steps[0])),
                    functools.partial(new_decay_step, decay_steps),
                    nothing)
</code></pre>

<p><strong>UPD3:</strong> I realized that code in <strong>UPD2</strong> wil not work because I can't change list inside the function.</p>

<p>Also I don't know what parts of <code>tf.logical_and</code> are really executed.</p>

<p>I made following code:</p>

<pre><code>class ohmy:
    def __init__(self, decay_steps):
        self.decay_steps = decay_steps

    def multistep_decay(self, learning_rate, global_step, current_decay_step, decay_steps, decay_rate,
                    staircase=False, name=None):

        learning_rate = tf.convert_to_tensor(learning_rate, name="learning_rate")
        dtype = learning_rate.dtype
        global_step = tf.cast(global_step, dtype)

        decay_rate = tf.cast(decay_rate, dtype)

        def new_step():
            self.decay_steps = self.decay_steps[1:]
            current_decay_step.assign(current_decay_step + 1)
            return current_decay_step

        def curr_step():
            return current_decay_step

        current_decay_step = tf.cond(tf.logical_and(tf.greater(tf.shape(self.decay_steps)[0], 0),  tf.greater(global_step, self.decay_steps[0])),
                new_step,
                curr_step)

        a = tf.Print(global_step, [global_step], "global")
        b = tf.Print(self.decay_steps, [self.decay_steps], "decay_steps")
        c = tf.Print(current_decay_step, [current_decay_step], "step")

        with tf.control_dependencies([a, b, c, current_decay_step]):
            p = current_decay_step

            if staircase:
                p = tf.floor(p)

            return tf.mul(learning_rate, tf.pow(decay_rate, p), name=name)


decay_steps = [3,4,5,6,7]
decay_steps = tf.convert_to_tensor(decay_steps, dtype=tf.float32)
current_decay_step = tf.Variable(0.0, trainable=False)
global_step = tf.Variable(0, trainable=False)
decay_rate = 0.5

c=ohmy(decay_steps)
lr = ohmy.multistep_decay(c, 0.010, global_step, current_decay_step, decay_steps, decay_rate)
#lr = tf.train.exponential_decay(0.001, global_step=global_step, decay_steps=2, decay_rate=0.5, staircase=True)
tf.scalar_summary('learning_rate', lr)

opt = tf.train.AdamOptimizer(lr)
#...train loop and so on
</code></pre>

<p>It doesn't work at all. Here is output :</p>

<pre><code>I tensorflow/core/kernels/logging_ops.cc:79] step[0]
I tensorflow/core/kernels/logging_ops.cc:79] global[0]
E tensorflow/core/client/tensor_c_api.cc:485] The tensor returned for MergeSummary/MergeSummary:0 was not valid.
Traceback (most recent call last):
  File "flownet_new.py", line 528, in &lt;module&gt;
    summary_str = sess.run(summary_op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 382, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 655, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 723, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 743, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: The tensor returned for MergeSummary/MergeSummary:0 was not valid.
</code></pre>

<p>As you can see there is no output of decay steps. I can't even debug it!</p>

<p>Now I definitely don't know how to make it with one function. 
Btw, either I do something wrong, or <code>tf.contrib.slim</code> doesn't work with learning rate decay. </p>

<p>For now most simple solution is make what you want in train loop as <a href="http://stackoverflow.com/users/4385912/cleros">cleros</a> said.</p>

<p>Use <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/train.html#exponential_decay" rel="nofollow"><code>tf.train.exponential_decay()</code></a>, it's exactly what you're looking for. The decayed learning rate is computed as follows:</p>

<pre><code>decayed_learning_rate = learning_rate *
                    decay_rate ^ (global_step / decay_steps)
</code></pre>

<p>Note that <code>decay_steps</code> parameter is an integer (not an array nor tensor) holding the period of iterations in which the learning rate changes. In your example <code>decay_steps=100</code>.</p>

python tensorflow