Theano learning AND gate
<p>I wrote a simple neural network to learn an AND gate. I'm trying to understand why my cost never decreases and the predictors are always 0.5:</p>

<pre><code>import numpy as np
import theano
import theano.tensor as T

inputs = [[0,0], [1,1], [0,1], [1,0]]
outputs = [[0], [1], [0], [0]]

x = theano.shared(value=np.asarray(inputs), name='x')
y = theano.shared(value=np.asarray(outputs), name='y')

alpha = 0.1

w_array = np.asarray(np.random.uniform(low=-1, high=1, size=(2, 1)), dtype=theano.config.floatX)
w = theano.shared(value=w_array, name='w', borrow=True)

output = T.nnet.sigmoid(T.dot(x, w))
cost = T.sum((y - output) ** 2)
updates = [(w, w - alpha * T.grad(cost, w))]

train = theano.function(inputs=[], outputs=[], updates=updates)
test = theano.function(inputs=[], outputs=[output])
calc_cost = theano.function(inputs=[], outputs=[cost])

for i in range(60000):
    if (i+1) % 10000 == 0:
        print(i+1)
        print(calc_cost())
    train()    

print(test())
</code></pre>

<p>The output is always the same:</p>

<pre><code>10000
[array(1.0)]
20000
[array(1.0)]
30000
[array(1.0)]
40000
[array(1.0)]
50000
[array(1.0)]
60000
[array(1.0)]

[array([[ 0.5],
       [ 0.5],
       [ 0.5],
       [ 0.5]])]
</code></pre>

<p>It always seems to predict 0.5 regardless of the input because the cost is not deviating from 1 during learning</p>

<p>If I switch the outputs to <code>[[0], [1], [1], [1]]</code> for learning an OR gate, I get the correct predictions, and correctly decreasing cost</p>

<p>Your model is of form</p>

<pre><code>&lt;w, x&gt;
</code></pre>

<p>thus it cannot build any separation which <strong>does not cross the origin</strong>. Such equation can only express lines going through point (0,0), and obviously line separating AND gate ((1, 1) from anything else) does not cross the origin. You have to add <strong>bias</strong> term, so your model is</p>

<pre><code>&lt;w, x&gt; + b
</code></pre>

python machine-learning neural-network theano