Keras - Generator for large dataset of Images and Masks
<p>I'm trying to build a Model that has Images for both its Inputs and Outputs (masks).
Because of the size of the Dataset and my limited Memory, I tried using <a href="https://keras.io/preprocessing/image/" rel="nofollow">the Generator Approach introduced in the Keras Documentation</a>:</p>

<pre><code># Provide the same seed and keyword arguments to the fit and flow methods
seed = 1

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

model.fit_generator(
    train_generator,
    samples_per_epoch=2000,
    nb_epoch=50)
</code></pre>

<p>Everything seems to work except when the code gets to this line:</p>

<pre><code>train_generator = zip(image_generator, mask_generator)
</code></pre>

<p>it seems the process of zipping the two lists explicitly makes them generate their content and the system starts consuming lots of RAM until it runs out of Memory. </p>

<p>The point of using Generators is to avoid running out of RAM while this piece of code is exactly doing the opposite.</p>

<p>Is there any way to fix this problem?</p>

<p>You can user <code>itertools.izip()</code> to return an iterator instead of a list.</p>

<pre><code>itertools.izip(*iterables)

Make an iterator that aggregates elements from each of the iterables. Like zip() except that it returns an iterator instead of a list. Used for lock-step iteration over several iterables at a time.
</code></pre>

python machine-learning computer-vision theano keras