Nonlinear regression on tensorflow
<p>What activation and cost functions on <code>tensorflow</code> could be suitable below for <strong>tf.nn</strong> to <strong>learn</strong> a simple single variate nonlinear relationship <code>f(x) = x * x</code> that is a priori unknown? </p>

<p>Certainly, this impractical model is used for the sole purpose of understanding <code>tf.nn mechanics 101</code>.</p>



<pre><code>import numpy as np
import tensorflow as tf

x = tf.placeholder(tf.float32, [None, 1]) 
W = tf.Variable(tf.zeros([1,1]))
b = tf.Variable(tf.zeros([1]))
y = some_nonlinear_activation_function_HERE(tf.matmul(x,W) + b) 
y_ = tf.placeholder(tf.float32, [None, 1]) 
cost = tf.reduce_mean(some_related_cost_function_HERE(y, y_)) 
learning_rate = 0.001
optimize = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

sess = tf.Session()
sess.run(tf.initialize_all_variables())
steps = 1000
for i in range(steps):
  sess.run(optimize, 
    feed_dict={x: np.array([[i]])), y_: np.array([[i*i]])})   
print("prediction: %f" % sess.run(y, 
  feed_dict={x: np.array([[1100]])})) 
# expected output near 1210000
</code></pre>

<p>The COST that is often used is simply the squared difference:</p>

<pre><code>def squared_error(y1,y2):
  return tf.square(y1-y2)
</code></pre>

<p>Plus an L1 or L2 penalty if you feel like it.  </p>

<p>However it seems to me that you need a hidden layer in your neural network if you want something remotely interesting. Plus if you squash your output and your target is the squared function you might not be able to do much.
I would do:</p>

<pre><code>x = tf.placeholder(tf.float32, [None, 1]) 
#Hidden layer with ten neurons
W1 = tf.Variable(tf.zeros([1,10]))
b1 = tf.Variable(tf.zeros([10]))
h1 = some_nonlinear_activation_function(tf.matmul(x,W) + b)
W2 = tf.Variable(tf.zeros([10,1]))
b2 = tf.Variable(tf.zeros([1]))
#I am not squashing the output
y=tf.matmul(h1,W2)+b
cost = tf.reduce_mean(squared_error(y, y_))
</code></pre>

<p>Also I would not use 0 weights but a more clever initialization scheme like Xavier's or He's which really come down to starting with practically zero weights but not exactly zeros for various reasons.
For activations you might use tanh, sigmoid or ReLU or anything really.</p>

python machine-learning neural-network tensorflow non-linear-regression