Theano - SGD with 1 hidden layer
<p>I was using <a href="https://github.com/Newmu/Theano-Tutorials/blob/master/2_logistic_regression.py" rel="nofollow">Newmu tutorial</a> for logistic regression from his github. Wanted to add one hidden layer to his model, so I divided weights variable into two arrays h_w and o_w.
The problem is - when I'm trying to make an update its impossible to operate on the list (w = [h_w, o_w])</p>

<pre><code> "File "C:/Users/Dis/PycharmProjects/untitled/MNISTnet.py",
 line 32, in &lt;module&gt;
     **update = [[w, w - gradient * 0.05]] TypeError: can't multiply sequence by non-int of type 'float'**"
</code></pre>

<p>I'm beginner in theano and numpy and theano documentation couldn't help me. I've found stack() function, but when combining <code>w = T.stack([h_w, o_w], axis=1)</code> theano gives me error:</p>

<pre><code>Traceback (most recent call last):
  File "C:\Users\Dis\PycharmProjects\untitled\MNISTnet.py", line 35, in &lt;module&gt;
    gradient = T.grad(cost=cost, wrt=w)
  File "C:\Program Files\Anaconda2\lib\site-packages\theano-0.9.0.dev1-py2.7.egg\theano\gradient.py", line 533, in grad
    handle_disconnected(elem)
  File "C:\Program Files\Anaconda2\lib\site-packages\theano-0.9.0.dev1-py2.7.egg\theano\gradient.py", line 520, in handle_disconnected
    raise DisconnectedInputError(message)
theano.gradient.DisconnectedInputError:  
Backtrace when that variable is created:

  File "C:\Users\Dis\PycharmProjects\untitled\MNISTnet.py", line 30, in &lt;module&gt;
    w = T.stack([h_w, o_w], axis=1)
</code></pre>

<p>So, my question: how can I convert that list <code>[&lt;TensorType(float64, matrix)&gt;, &lt;TensorType(float64, matrix)&gt;]</code> to variable <code>&lt;TensorType(float64, matrix)&gt;</code>?</p>

<p>My full code below:</p>

<pre><code>import theano
from theano import tensor as T
import numpy as np
from load import mnist

def floatX(X):
    return np.asarray(X, dtype=theano.config.floatX)

def init_weights(shape):
    return theano.shared(floatX(np.random.randn(*shape) * 0.01))

def model(X, o_w, h_w):
    hid = T.nnet.sigmoid(T.dot(X, h_w))
    out = T.nnet.softmax(T.dot(hid, o_w))
    return out

trX, teX, trY, teY = mnist(onehot=True)

X = T.fmatrix()
Y = T.fmatrix()

h_w = init_weights((784, 625))
o_w = init_weights((625, 10))

py_x = model(X, o_w, h_w)
y_pred = T.argmax(py_x, axis=1)
w = [o_w, h_w]

cost = T.mean(T.nnet.categorical_crossentropy(py_x, Y))
gradient = T.grad(cost=cost, wrt=w)
print type(gradient)
update = [[w, w - gradient * 0.05]]
</code></pre>

<p><code>T.grad(..)</code> returns <a href="http://deeplearning.net/software/theano/tutorial/gradients.html" rel="nofollow">gradient</a> w.r.t to each parameter, so you cannot do <code>[w, w - gradient * 0.05]</code>, you have to specify which gradient[*] parameter you are referring to. Also it's not a good idea to use stack for multiple parameters, simple list is good enough, check this <a href="http://deeplearning.net/tutorial/logreg.html" rel="nofollow">tutorial</a>.
This should work:</p>

<pre><code>import theano
from theano import tensor as T
import numpy as np
from load import mnist

def floatX(X):
    return np.asarray(X, dtype=theano.config.floatX)

def init_weights(shape):
    return theano.shared(floatX(np.random.randn(*shape) * 0.01))

def model(X, o_w, h_w):
    hid = T.nnet.sigmoid(T.dot(X, h_w))
    out = T.nnet.softmax(T.dot(hid, o_w))
    return out

trX, teX, trY, teY = mnist(onehot=True)

X = T.fmatrix()
Y = T.fmatrix()

h_w = init_weights((784, 625))
o_w = init_weights((625, 10))

py_x = model(X, o_w, h_w)
y_pred = T.argmax(py_x, axis=1)
w = [o_w, h_w]

cost = T.mean(T.nnet.categorical_crossentropy(py_x, Y))
gradient = T.grad(cost=cost, wrt=w)
print type(gradient)
update = [[o_w, o_w - gradient[0] * 0.05],
          [h_w, h_w - gradient[1] * 0.05]]
</code></pre>

<p>I suggest going through Theano <a href="http://deeplearning.net/software/theano/tutorial/index.html" rel="nofollow">tutorials</a> to get started.</p>

python numpy theano