Prediction of sinusoidal function using pybrain
<p>In an attempt to understand neural networks and pybrain, I tried to predict a sinusoidal function in noise using only the time index as input. The premise therefore is that a simple NN structure can mimic <em>y(t) = sin(t)</em>.</p>

<p>The design is: 1 input layer (linear), 1 hidden layer (tanh) and 1 output layer (linear). The number of nodes is: 1,10,1 for each respective layer.</p>

<p>The input (time variable <em>t</em>) is scaled such that its range is <em>[0;1]</em>. The target is scaled either to have range <em>[0;1]</em> or <em>[-1;1]</em> with different results (given below).</p>

<p>Here is my Python 2.7 code:</p>

<pre><code>#!/usr/bin/python
from __future__ import division
import numpy as np
import pylab as pl

from pybrain.structure import TanhLayer, LinearLayer #SoftmaxLayer, SigmoidLayer
from pybrain.datasets import SupervisedDataSet 
from pybrain.supervised.trainers import BackpropTrainer 
from pybrain.structure import FeedForwardNetwork
from pybrain.structure import FullConnection 

np.random.seed(0)
pl.close('all')

#create NN structure:
net = FeedForwardNetwork()
inLayer = LinearLayer(1)
hiddenLayer = TanhLayer(10)
outLayer = LinearLayer(1)

#add classes of layers to network, specify IO:
net.addInputModule(inLayer)
net.addModule(hiddenLayer)
net.addOutputModule(outLayer)

#specify how neurons are to be connected:
in_to_hidden = FullConnection(inLayer, hiddenLayer)
hidden_to_out = FullConnection(hiddenLayer, outLayer)

#add connections to network:
net.addConnection(in_to_hidden)
net.addConnection(hidden_to_out)

#perform internal initialisation:
net.sortModules()

#construct target signal:
T = 1
Ts = T/10
f = 1/T
fs = 1/Ts

#NN input signal:
t0 = np.arange(0,10*T,Ts)
L = len(t0)

#NN target signal:
x0 = 10*np.cos(2*np.pi*f*t0) + 10 + np.random.randn(L)

#normalise input signal:
t = t0/np.max(t0)

#normalise target signal to fit in range [0,1] (min) or [-1,1] (mean):
dcx = np.min(x0) #np.min(x0) #np.mean(x0) 
x = x0-dcx
sclf = np.max(np.abs(x))
x /= sclf

#add samples and train NN:
ds = SupervisedDataSet(1, 1)
for c in range(L):
 ds.addSample(t[c], x[c])

trainer = BackpropTrainer(net, ds, learningrate=0.01, momentum=0.1)

for c in range(20):
 e1 = trainer.train()
 print 'Epoch %d Error: %f'%(c,e1)

y=np.zeros(L)
for c in range(L):
 #y[c] = net.activate([x[c]])
 y[c] = net.activate([t[c]])

yout = y*sclf  
yout = yout + dcx

fig1 = pl.figure(1)
pl.ion()

fsize=8
pl.subplot(211)
pl.plot(t0,x0,'r.-',label='input')
pl.plot(t0,yout,'bx-',label='predicted')
pl.xlabel('Time',fontsize=fsize)
pl.ylabel('Amplitude',fontsize=fsize)
pl.grid()
pl.legend(loc='lower right',ncol=2,fontsize=fsize)
pl.title('Target range = [0,1]',fontsize=fsize)

fig1name = './sin_min.png'
print 'Saving Fig. 1 to:', fig1name
fig1.savefig(fig1name, bbox_inches='tight')
</code></pre>

<p>The output figures are given below.</p>

<p><a href="http://i.stack.imgur.com/CpOYx.png" rel="nofollow"><img src="http://i.stack.imgur.com/CpOYx.png" alt="Output when target is scaled to fit range *[0;1]*"></a></p>

<p><a href="http://i.stack.imgur.com/zwxm7.png" rel="nofollow"><img src="http://i.stack.imgur.com/zwxm7.png" alt="Output when target is scaled to fit range *[-1;1]*"></a></p>

<p>Although the first figure shows better results, both outputs are unsatisfactory. Am I missing some fundamental neural network principle or is my code defective? I know there are easier statistical methods of estimating the target signal in this case, but the aim is to use a simple NN structure here.</p>

<p>The issue is the range of input and output values. By <a href="https://en.wikipedia.org/wiki/Feature_scaling#Standardization" rel="nofollow">standardizing</a> these signals, the problem is solved.</p>

<p><a href="http://i.stack.imgur.com/HmFIi.png" rel="nofollow"><img src="http://i.stack.imgur.com/HmFIi.png" alt="NN input and output signal"></a></p>

<p>This can be explained by considering the <em>sigmoid</em> and <em>tanh</em> activation functions, displayed below.
<a href="http://i.stack.imgur.com/X2gwQ.png" rel="nofollow"><img src="http://i.stack.imgur.com/X2gwQ.png" alt="sigmoid and tanh activation functions"></a></p>

<p>The results will depend on the specific application. Also, changing the activation function (see <a href="http://stackoverflow.com/questions/13632976/neural-network-with-tanh-wrong-saturation-with-normalized-data">this answer</a>) will probably also affect the optimal scaling values of the input and output signals.</p>

python neural-network time-series prediction pybrain