Tensorflow: parallel for loop results in out-of-memory
<p>My tensorflow code is like these:</p>

<pre><code>for i in range(100):
    Zk = Zk + function_call(...)
</code></pre>

<p>It seems that tensorflow runs these 100 iterations in parallel, and keeps many temp vectors which have same size with Zk. However, since Zk is a very long vector, this leads to out-of-memory error immediately. </p>

<p>Can anyone give some suggestion on how to force <code>Tensorflow</code> perform loop sequentially. Thanks a lot.</p>

<p>The simplest way to solve this problem is to use a <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/framework.html#control_dependencies" rel="nofollow"><code>with tf.control_dependencies():</code></a> block:</p>

<pre><code>Zk = ...
for i in range(100):
    with tf.control_dependencies([Zk.op]):
        Zk = Zk + function_call(...)
</code></pre>

<p>The <code>with tf.control_dependencies():</code> block ensures that the ops created by <code>function_call()</code> do not run until the previous value for <code>Zk</code> has been computed. This effectively causes the loop iterations to run sequentially.</p>

<hr>

<p>An alternative solution involves using a <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/control_flow_ops.html#while_loop" rel="nofollow"><code>tf.while_loop()</code></a> to define the iteration, rather than a Python loop. The <code>tf.while_loop()</code> function has a <code>parallel_iterations</code> optional argument that allows you to reduce the amount of parallelism between independent iterations of the loop. One advantage of this approach is that it keeps your graph small: the loop would use O(1) nodes rather than O(N) to perform N iterations. However, it can be slightly tricky to re-cast an arbitrary Python <code>function_call()</code> as a TensorFlow loop body, so you'll probably find it easier to use <code>with tf.control_dependencies():</code> at first.</p>

python parallel-processing tensorflow