Replicating results with different artificial neural network frameworks (ffnet, tensorflow)
<p>I'm trying to model a technical process (a number of nonlinear equations) with artificial neural networks. The function has a number of inputs and a number of outputs (e.g. 50 inputs, 150 outputs - all floats).</p>

<p>I have tried the <a href="http://ffnet.sourceforge.net/overview.html" rel="nofollow">python library ffnet</a> (wrapper for a fortran library) with great success. The errors for a certain dataset are well below 0.2%.</p>

<p>It is using a fully connected graph and these additional parameters. </p>

<pre><code>Basic assumptions and limitations:
    Network has feed-forward architecture.
    Input units have identity activation function, all other units have sigmoid activation function.
    Provided data are automatically normalized, both input and output, with a linear mapping to the range (0.15, 0.85). Each input and output is treated separately (i.e. linear map is unique for each input and output).
    Function minimized during training is a sum of squared errors of each output for each training pattern.
</code></pre>

<p>I am using one input layer, one hidden layer (size: 2/3 of input vector + size of output vector) and an output layer. I'm using the scipy conjugate gradient optimizer.</p>

<p>The downside of ffnet is the long training time and the lack of functionality to use GPUs. Therefore i want to switch to a different framework and have chosen <strong>keras with TensorFlow</strong> as the backend.</p>

<p>I have tried to model the previous configuration:</p>

<pre><code>model = Sequential()
model.add(Dense(n_hidden, input_dim=n_in))
model.add(BatchNormalization())
model.add(Dense(n_hidden))
model.add(Activation('sigmoid'))
model.add(Dense(n_out))
model.add(Activation('sigmoid'))
model.summary()
model.compile(loss='mean_squared_error',
              optimizer='Adamax',
              metrics=['accuracy'])
</code></pre>

<p>However the results are far worse, the error is up to 0.5% with a few thousand (!) epochs of training. The ffnet training was automatically canceled at 292 epochs. Furthermore the differences between the network response and the validation target are not centered around 0, but mostly negative. 
I have tried all optimizers and different loss functions. I have also skipped the BatchNormalization and normalized the data manually in the same way that ffnet does it. Nothing helps.</p>

<p><strong>Does anyone have a suggestion to obtain better results with keras?</strong></p>

<p>I understand you are trying to re-train the same architecture from scratch, with a different library. The first fundamental issue to keep in mind here is that neural nets <a href="http://stackoverflow.com/q/24161525/777285">are not necessarily reproducible</a>, when weights are initialized randomly.</p>

<p>For example, here is the default constructor parameter for <code>Dense</code> in Keras:</p>

<pre><code>init='glorot_uniform'
</code></pre>

<p>But even before trying to evaluate the convergence of Keras optimizations, I would recommend trying to port the weights for which you got good results, from ffnet, into your Keras model. You can do so either with the <em>kwarg</em> <code>Dense(..., weights=</code>) of each layer, or globally at the end <code>model.set_weights(...)</code></p>

<p>Using the same weights <strong>must</strong> yield the exact same result between the two libs. Unless you run into some floating point rounding issues. I believe that as long as porting the weights is not consistent, working on the optimization is unlikely to help.</p>

python neural-network tensorflow keras