tensorflow: efficient feeding of eval/train data using queue runners
<p>I'm trying to run a tensorflow graph to train a model and periodically evaluate using a separate evaluation dataset. Both training and evaluation data is implemented using queue runners.</p>

<p>My current solution is to create both inputs in the same graph and use a <code>tf.cond</code> dependent on an <code>is_training</code> placeholder. My issue is highlighted by the following code:</p>

<pre><code>import tensorflow as tf
from tensorflow.models.image.cifar10 import cifar10
from time import time


def get_train_inputs(is_training):
    return cifar10.inputs(False)


def get_eval_inputs(is_training):
    return cifar10.inputs(True)


def get_mixed_inputs(is_training):
    train_inputs = get_train_inputs(None)
    eval_inputs = get_eval_inputs(None)

    return tf.cond(is_training, lambda: train_inputs, lambda: eval_inputs)


def time_inputs(inputs_fn, n_runs=10):
    graph = tf.Graph()
    with graph.as_default():
        is_training = tf.placeholder(dtype=tf.bool, shape=(),
                                     name='is_training')
        images, labels = inputs_fn(is_training)

    with tf.Session(graph=graph) as sess:
        coordinator = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(sess=sess, coord=coordinator)
        t = time()
        for i in range(n_runs):
            im, l = sess.run([images, labels], feed_dict={is_training: True})
        dt = time() - t
        coordinator.request_stop()
        coordinator.join(threads)

    return dt / n_runs

print('Train inputs: %.3f' % time_inputs(get_train_inputs))
print('Eval inputs: %.3f' % time_inputs(get_eval_inputs))
print('Mixed inputs: %.3f' % time_inputs(get_mixed_inputs))
</code></pre>

<p>I also had to comment out the <code>image_summary</code> line <code>133</code> of <code>tensorflow/models/image/cifar10/cifar10_inputs.py</code>.</p>

<p>This yielded the following results:</p>

<pre><code>Train inputs: 0.055
Eval inputs: 0.050
Mixed inputs: 0.105
</code></pre>

<p>It would seem in the mixed case both inputs are being read/parsed, even though only 1 is used. Is there a way of avoiding this redundant computation? Or is there a nicer way of switching between training/evaluation data that still leverages the queue-runner setup?</p>

<p>After some experimentation, my current best solution is to have a main graph featuring training inputs and a separate graph with just evaluation data operations. I open a separate session to get evaluation data and feed this to the training graph when I want to evaluate. Highly inelegant (and evaluation runs take longer than they ideally would as they have to come ot of one session only to be fed to another), but assuming evaluation runs are rare compared to training runs, this seems preferable to the original version...</p>

<pre><code>import tensorflow as tf
from tensorflow.models.image.cifar10 import cifar10
from time import time


class DataSupplier:
    def __init__(self, tensor_fn):
        graph = tf.Graph()
        with graph.as_default():
            with graph.device('/cpu:0'):
                self.tensor = tensor_fn()
        self.sess = tf.Session(graph=graph)
        self.coord = tf.train.Coordinator()
        self.threads = tf.train.start_queue_runners(sess=self.sess,
                                                    coord=self.coord)

    def get_tensor_val(self):
        return self.sess.run(self.tensor)

    def clean_up(self):
        self.coord.request_stop()
        self.coord.join(self.threads)


eval_batcher = DataSupplier(lambda: cifar10.inputs(True))

graph = tf.Graph()
with graph.as_default():
    images, labels = cifar10.inputs(False)

    out_images = tf.identity(images)
    out_labels = tf.identity(labels)

n_runs = 100

with tf.Session(graph=graph) as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess, coord)
    for i in range(n_runs):
        sess.run([out_images, out_labels])
    t = time()
    for i in range(n_runs):
        sess.run([out_images, out_labels])
    dt = (time() - t)/n_runs
    print('Train time: %.3f' % dt)
    t = time()
    for i in range(n_runs):
        eval_images, eval_labels = eval_batcher.get_tensor_val()
        sess.run([out_images, out_labels],
                 feed_dict={images: eval_images, labels: eval_labels})
    dt = (time() - t)/n_runs
    print('Eval time: %.3f' % dt)
    coord.request_stop()
    coord.join(threads)

eval_batcher.clean_up()
</code></pre>

<p>Results:</p>

<pre><code>Train time: 0.050
Eval time: 0.064
</code></pre>

<p>Update: when using this approach in training problems with tf.contrib.layers and regularization, I find the regularization losses go to infinity if the DataSupplier graph is on the same device as the training graph. I cannot for the life of me explain why this is the case, but explicitly setting the device of the DataSupplier to the CPU (given the training graph is on my GPU) seems to work...</p>

python tensorflow