How to use tf.cond in combination with batching operations / queue runners
<h1>Situation</h1>

<p>I want to train a specific network architecture (a <a href="https://arxiv.org/abs/1406.2661" rel="nofollow">GAN</a>) that needs inputs from different sources during training.</p>

<p>One input source is examples loaded from disk. The other source is a generator sub-network creating examples.</p>

<p>To choose which kind of input to feed to the network I use <code>tf.cond</code>. There is one caveat though that <a href="http://stackoverflow.com/a/34934543/850264">has already been explained</a>: <code>tf.cond</code> evaluates the inputs to both conditional branches even though only one of those will ultimately be used.</p>

<p>Enough setup, here is a minimal working example:</p>

<pre><code>import numpy as np
import tensorflow as tf

BATCH_SIZE = 32

def load_input_data():
  # Normally this data would be read from disk
  data = tf.reshape(np.arange(10 * BATCH_SIZE, dtype=np.float32), shape=(10 * BATCH_SIZE, 1))
  return tf.train.batch([data], BATCH_SIZE, enqueue_many=True)

def generate_input_data():
  # Normally this data would be generated by a much bigger sub-network
  return tf.random_uniform(shape=[BATCH_SIZE, 1])

def main():
  # A bool to choose between loaded or generated inputs
  load_inputs_pred = tf.placeholder(dtype=tf.bool, shape=[])

  # Variant 1: Call "load_input_data" inside tf.cond
  data_batch = tf.cond(load_inputs_pred, load_input_data, generate_input_data)
  # Variant 2: Call "load_input_data" outside tf.cond
  #loaded_data = load_input_data()
  #data_batch = tf.cond(load_inputs_pred, lambda: loaded_data, generate_input_data)

  init_op = tf.initialize_all_variables()

  with tf.Session() as sess:
    sess.run(init_op)

    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess, coord=coord)

    print(threads)

    # Get generated input data
    data_batch_values = sess.run(data_batch, feed_dict={load_inputs_pred: False})
    print(data_batch_values)

    # Get input data loaded from disk
    data_batch_values = sess.run(data_batch, feed_dict={load_inputs_pred: True})
    print(data_batch_values)

if __name__ == '__main__':
  main()
</code></pre>

<h1>Problem</h1>

<p>Variant 1 does not work at all since the queue runner threads don't seem to run. <code>print(threads)</code> outputs something like <code>[&lt;Thread(Thread-1, stopped daemon 140165838264064)&gt;, ...]</code>.</p>

<p>Variant 2 does work and <code>print(threads)</code> outputs something like <code>[&lt;Thread(Thread-1, started daemon 140361854863104)&gt;, ...]</code>. But since <code>load_input_data()</code> has been called outside of <code>tf.cond</code>, batches of data will be loaded from disk even when <code>load_inputs_pred</code> is <code>False</code>.</p>

<p>Is it possible to make Variant 1 work, so that input data is only loaded when <code>load_inputs_pred</code> is <code>True</code> and not for every call to <code>session.run()</code>?</p>

<p>If you're using a queue when loading your data and follow it up with a batch input then this shouldn't be a problem as you can specify the max amount to have loaded or stored in the queue.</p>

<pre><code>input = tf.WholeFileReader(somefilelist) # or another way to load data
return tf.train.batch(input,batch_size=10,capacity=100) 
</code></pre>

<p>See here for more details:
<a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/io_ops.html#batch" rel="nofollow">https://www.tensorflow.org/versions/r0.10/api_docs/python/io_ops.html#batch</a></p>

<hr>

<p>Also there's an alternative approach that skips the tf.cond completely.  Just define two losses one that follows the data through the autoencoder and discrimator and the other that follows the data through just the discriminator.</p>

<p>Then it just becomes a matter of calling</p>

<pre><code>sess.run(auto_loss,feed_dict)
</code></pre>

<p>or</p>

<pre><code>sess.run(real_img_loss,feed_dict)
</code></pre>

<p>In this way the graph will only run through which ever loss was called upon. Let me know if this needs more explanation.</p>

<hr>

<p>Lastly I think to make variant one work you need to do something like this  if you're using preloaded data.</p>

<p><a href="https://www.tensorflow.org/versions/r0.10/how_tos/reading_data/index.html#preloaded-data" rel="nofollow">https://www.tensorflow.org/versions/r0.10/how_tos/reading_data/index.html#preloaded-data</a></p>

<p>Otherwise I'm not sure what the issue is to be honest.</p>

python tensorflow