My loss with fit_generator is 0.0000e+00 (using Keras)
<p>I am trying to use Keras on a âlargeâ dataset for my GPU. To do so, I make use of fit_generator, the problem is that my loss is 0.0000e+00 every time.</p>

<p>My print class and generator function:</p>

<pre><code>class printbatch(callbacks.Callback):
    def on_batch_end(self, batch, logs={}):
        if batch%10 == 0:
            print "Batch " + str(batch) + " ends"
    def on_epoch_begin(self, epoch, logs={}):
        print(logs)
    def on_epoch_end(self, epoch, logs={}):
        print(logs)

def simpleGenerator():
    X_train = f.get('X_train')
    y_train = f.get('y_train')
    total_examples = len(X_train)
    examples_at_a_time = 6
    range_examples = int(total_examples/examples_at_a_time)

    while 1:
        for i in range(range_examples): # samples
            yield X_train[i*examples_at_a_time:(i+1)*examples_at_a_time], y_train[i*examples_at_a_time:(i+1)*examples_at_a_time]
</code></pre>

<p>This is how I use them:</p>

<pre><code>f = h5py.File(cache_file, 'r')

pb = printbatch()
sg = simpleGenerator()

class_weighting = [0.2595, 0.1826, 4.5640, 0.1417, 0.5051, 0.3826, 9.6446, 1.8418, 6.6823, 6.2478, 3.0, 7.3614]

history = autoencoder.fit_generator(sg, samples_per_epoch=366, nb_epoch=10, verbose=2, show_accuracy=True, callbacks=[pb], validation_data=None, class_weight=class_weighting)
</code></pre>

<p>This is (a part of) my output:</p>

<pre><code>{}
Epoch 1/100
Batch 0 ends
Batch 10 ends
Batch 20 ends
Batch 30 ends
Batch 40 ends
Batch 50 ends
Batch 60 ends
{'loss': 0.0}
120s - loss: 0.0000e+00
[â¦]
{}
Epoch 9/10
Batch 0 ends
Batch 10 ends
Batch 20 ends
Batch 30 ends
Batch 40 ends
Batch 50 ends
Batch 60 ends
{'loss': 0.0}
124s - loss: 0.0000e+00
{}
Epoch 10/10
Batch 0 ends
Batch 10 ends
Batch 20 ends
Batch 30 ends
Batch 40 ends
Batch 50 ends
Batch 60 ends
{'loss': 0.0}
127s - loss: 0.0000e+00
Training completed in 1263.76883411 seconds
</code></pre>

<p>X_train and y_train shapes are:</p>

<pre><code>X_train.shape
Out[5]: (366, 3, 360, 480)
y_train.shape
Out[6]: (366, 172800, 12)
</code></pre>

<p>So my question is, how could I solve the 'loss: 0.0000e+00' issue?</p>

<p>Thank you for your time.</p>

<p>Edit: the model, the original comes from  pradyu1993.github.io/2016/03/08/segnet-post.html by Pradyumna.</p>

<pre><code>class UnPooling2D(Layer):
"""A 2D Repeat layer"""
def __init__(self, poolsize=(2, 2)):
    super(UnPooling2D, self).__init__()
    self.poolsize = poolsize

@property
def output_shape(self):
    input_shape = self.input_shape
    return (input_shape[0], input_shape[1],
            self.poolsize[0] * input_shape[2],
            self.poolsize[1] * input_shape[3])

def get_output(self, train):
    X = self.get_input(train)
    s1 = self.poolsize[0]
    s2 = self.poolsize[1]
    output = X.repeat(s1, axis=2).repeat(s2, axis=3)
    return output

def get_config(self):
    return {"name":self.__class__.__name__,
        "poolsize":self.poolsize}


def create_encoding_layers():
    kernel = 3
    filter_size = 64
    pad = 1
    pool_size = 2
    return [
    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(filter_size, kernel, kernel,     border_mode='valid'),
    BatchNormalization(),
    Activation('relu'),
    MaxPooling2D(pool_size=(pool_size, pool_size)),

    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(128, kernel, kernel, border_mode='valid'),
    BatchNormalization(),
    Activation('relu'),
    MaxPooling2D(pool_size=(pool_size, pool_size)),

    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(256, kernel, kernel, border_mode='valid'),
    BatchNormalization(),
    Activation('relu'),
    MaxPooling2D(pool_size=(pool_size, pool_size)),

    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(512, kernel, kernel, border_mode='valid'),
    BatchNormalization(),
    Activation('relu'),
]


def create_decoding_layers():
    kernel = 3
    filter_size = 64
    pad = 1
    pool_size = 2
    return[
    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(512, kernel, kernel, border_mode='valid'),
    BatchNormalization(),

    UpSampling2D(size=(pool_size,pool_size)),
    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(256, kernel, kernel, border_mode='valid'),
    BatchNormalization(),

    UpSampling2D(size=(pool_size,pool_size)),
    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(128, kernel, kernel, border_mode='valid'),
    BatchNormalization(),

    UpSampling2D(size=(pool_size,pool_size)),
    ZeroPadding2D(padding=(pad,pad)),
    Convolution2D(filter_size, kernel, kernel, border_mode='valid'),
    BatchNormalization(),
]
</code></pre>

<p>And:</p>

<pre><code>autoencoder = models.Sequential()
autoencoder.add(Layer(input_shape=(3, img_rows, img_cols)))
autoencoder.encoding_layers = create_encoding_layers()
autoencoder.decoding_layers = create_decoding_layers()
for l in autoencoder.encoding_layers:
    autoencoder.add(l)
for l in autoencoder.decoding_layers:
    autoencoder.add(l)

autoencoder.add(Convolution2D(12, 1, 1, border_mode='valid',))
autoencoder.add(Reshape((12,img_rows*img_cols), input_shape=(12,img_rows,img_cols)))
autoencoder.add(Permute((2, 1)))
autoencoder.add(Activation('softmax'))
autoencoder.compile(loss="categorical_crossentropy", optimizer='adadelta')
</code></pre>

<p>I solved this issue. The problem was that in '.theanorc' I had float16: this is not enough, so I changed it to float64 and now it works. </p>

<p>This is my '.theanorc' at the moment:</p>

<pre><code>[global]
device = gpu
floatX = float64
optimizer_including=cudnn

[lib]
cnmem=0.90

[blas]
ldflags = -L/usr/local/lib -lopenblas

[nvcc]
fastmath = True

[cuda]
root = /usr/local/cuda/
</code></pre>

python deep-learning keras autoencoder