Using Sigmoid instead of Tanh activation function fails - Neural Networks
<p>I'm looking at the following <a href="http://www.bogotobogo.com/python/files/NeuralNetworks/nn3.py" rel="nofollow">code</a> from <a href="http://www.bogotobogo.com/python/python_Neural_Networks_Backpropagation_for_XOR_using_one_hidden_layer.php" rel="nofollow">this blog</a></p>

<p>It gives the option to use both the <code>sigmoid</code> and the <code>tanh</code> activation function.</p>

<p>The XOR test seems to work fine with the <code>tanh</code> function yielding ~ <code>(0,1,1,0)</code></p>

<p>But upon changing to <code>sigmoid</code> I get the wrong output ~ <code>(0.5,0.5,0.5,0.5)</code></p>

<p>I've tried this with <a href="http://arctrix.com/nas/python/bpnn.py" rel="nofollow">another piece of code</a> I found online and the exact same problem occurs.</p>

<p>It seems the only thing changing is the activation function (and its derivative). Does changing this require other changes, say in backpropogation?</p>

<p>Thanks a lot for any help!</p>

<p>Looks like the model you use doesn't train biases.
The only difference between <code>tanh</code> and <code>sigmoid</code> is scaling and offset. Learning the new scaling will be done through the weights, but you'll also need to learn to compensate for the new offset, which should be done by learning the biases as well.</p>

python neural-network backpropagation sigmoid