How to train TensorFlow network using a generator to produce inputs?
<p>The TensorFlow <a href="https://www.tensorflow.org/versions/r0.10/how_tos/reading_data/index.html" rel="nofollow">docs</a> describe a bunch of ways to read data using TFRecordReader, TextLineReader, QueueRunner etc and queues.</p>

<p>What I would like to do is much, much simpler:  I have a python generator function that produces an infinite sequence of training data as (X, y) tuples (both are numpy arrays, and the first dimension is the batch size).  I just want to train a network using that data as inputs.  </p>

<p>Is there a simple self-contained example of training a TensorFlow network using a generator which produces the data?  (along the lines of the MNIST or CIFAR examples)</p>

<p>Suppose you have a function that generates data: </p>



<pre><code> def generator(data): 
    ...
    yield (X, y)
</code></pre>

<p>Now you need some function that describes your network architecture. It could be any network that processes X and has to predict y as output. </p>

<p>Suppose your function receives X and y as inputs, computes prediction for y from X in some way and returns loss function (e.g. cross-entropy or MSE in the case of regression) between y and predicted y:</p>



<pre><code> def NN(X, y): 
    #computation of prediction for y using X
    return loss(y, y_pred)
</code></pre>

<p>To make your model work, you need to define placeholders for both X and y and then run a session:</p>



<pre><code> X = tf.placeholder(tf.float32, shape = [batch_size, X_dim])
 y = tf.placeholder(tf.float32, shape = [batch_size, y_dim])
</code></pre>

<p>Placeholders are something like "free variables" which you need to specify when running the session by feed_dict:</p>



<pre><code> sess = tf.Session()
 sess.run(init)

 batch_gen = generator(data)
 batch = batch_gen.next()
 #note here batch is a tuple with X for first components and y for second
 sess.run([optm, loss, ...], feed_dict = {X: batch[0], y: batch[1]})
 #optm here stands for optimizer you have defined and loss for loss function (return value of NN function)
</code></pre>

<p>Here I demonstrated how to run a session for one batch, but you of course could use for loop to train over multiple batches. </p>

<p>Suppose you would find it useful. However, bear in mind this is not full working implementation but rather a pseudocode as you specified almost none details.</p>

python tensorflow