Why isn't my RNN learning?
<p>I'm trying to implement a simple RNN using numpy (based on <a href="https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/" rel="nofollow">this article</a>), and I'm training it to do binary addition where it adds two 8-bit unsigned integers one bit at a time (starting from the end) with the purpose of having it learn to "carry the one" during addition when necessary. However, it doesn't seem to be learning. For training, I'm choosing two random numbers, forward propagating 8 steps with one bit from a and b as input and storing the outputs and hidden layer values at each timestep, and backpropagating 8 steps where I calculate the hidden layer error (<code>output_error.dot(weights_hidden_to_output.T)) * sigmoid_to_derivative(hidden) + future_hidden_error.dot(weights_hidden_to_hidden.T)</code>) and the updates for each of the weight matrices by matrix multiplying the parent layer by the error of the child layer. Is this the correct method?</p>

<p>Here's my code if it will make it more clear. I noticed that for some reason, the weights start increasing like crazy all of a sudden every time I train it, and they cause an overflow in the sigmoid function which causes the training to fail. Any idea what could cause this?</p>

<pre><code>import numpy as np
np.random.seed(0)

def sigmoid(x):
    return np.atleast_2d(1/(1+np.exp(-x)))
    #return np.atleast_2d(np.max(x, 0.01))
def sig_deriv(x):
    return x*(1-x)
def add_bias(x):
    return np.hstack([np.ones((len(x), 1)), x])
def dec_to_bin(dec):
    return np.array(map(int, list(format(dec, '#010b'))[2:]))
def bin_to_dec(b):
    out = 0
    for bit in b:
        out = (out &lt;&lt; 1) | bit
    return out


batch_size = 8
learning_rate = .1

input_size = 2
hidden_size = 16
output_size = 1

weights_xh = 2 * np.random.random((input_size+1, hidden_size)) - 1
weights_hh = 2 * np.random.random((hidden_size+1, hidden_size)) - 1
weights_hy = 2 * np.random.random((hidden_size+1, output_size)) - 1

xh_update = np.zeros_like(weights_xh)
hh_update = np.zeros_like(weights_hh)
hy_update = np.zeros_like(weights_hy)

for i in xrange(10000):
    a = np.random.randint(0, 2**batch_size/2)
    b = np.random.randint(0, 2**batch_size/2)
    sum_ = a+b
    X = add_bias(np.hstack([np.atleast_2d(dec_to_bin(a)).T, np.atleast_2d(dec_to_bin(b)).T]))
    y = np.atleast_2d(dec_to_bin(sum_)).T

    error = 0

    output_errors = []
    outputs = []
    hiddens = [add_bias(np.zeros((1, hidden_size)))]
    #forward propagation through time
    for j in xrange(batch_size):
        hidden = sigmoid(X[-j-1].dot(weights_xh) + hiddens[-1].dot(weights_hh))
        hidden = add_bias(hidden)
        hiddens.append(hidden)
        output = sigmoid(hidden.dot(weights_hy))
        outputs.append(output[0][0])
        output_error = (y[-j-1] - output)
        error += np.abs(output_error[0])
        output_errors.append((output_error * sig_deriv(output)))

    future_hidden_error = np.zeros((1,hidden_size))
    #backward ppropagation through time
    for j in xrange(batch_size):
        output_error = output_errors[-j-1]
        hidden = hiddens[-j-1]
        prev_hidden = hiddens[-j-2]

        hidden_error = (output_error.dot(weights_hy.T) * sig_deriv(hidden)) + future_hidden_error.dot(weights_hh.T)
        hidden_error = np.delete(hidden_error, 0, 1) #delete bias error

        xh_update += np.atleast_2d(X[j]).T.dot(hidden_error)
        hh_update += prev_hidden.T.dot(hidden_error)
        hy_update += hidden.T.dot(output_error)

        future_hidden_error = hidden_error

    weights_xh += (xh_update * learning_rate)/batch_size
    weights_hh += (hh_update * learning_rate)/batch_size
    weights_hy += (hy_update * learning_rate)/batch_size

    xh_update *= 0
    hh_update *= 0
    hy_update *= 0

    if i%1000==0:
        guess = map(int, map(round, outputs[::-1]))
        print "Iteration {}".format(i)
        print "Error: {}".format(error)
        print "Problem: {} + {} = {}".format(a, b, sum_)
        print "a:        {}".format(list(dec_to_bin(a)))
        print "+ b:      {}".format(list(dec_to_bin(b)))
        print "Solution: {}".format(map(int, y))
        print "Guess:    {} ({})".format(guess, bin_to_dec(guess))
        print
</code></pre>

<p>I figured it out. In case anyone wants to know why it wasn't working, it was because I was only multiplying one part of the hidden error (the part that came from the output error) by the derivative of the hidden layer activation. Now it's easily learning the addition problem within a few thousand iterations.</p>

python numpy machine-learning neural-network recurrent-neural-network