Why does my linear regression get nan values instead of learning?
<p>I'm running the following code:</p>

<pre><code>import tensorflow as tf

# data set
x_data = [10., 20., 30., 40.]
y_data = [20., 40., 60., 80.]

# try to find values for w and b that compute y_data = W * x_data + b
# range is -100 ~ 100
W = tf.Variable(tf.random_uniform([1], -1000., 1000.))
b = tf.Variable(tf.random_uniform([1], -1000., 1000.))

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

# my hypothesis
hypothesis = W * X + b

# Simplified cost function
cost = tf.reduce_mean(tf.square(hypothesis - Y))

# minimize
a = tf.Variable(0.1)  # learning rate, alpha
optimizer = tf.train.GradientDescentOptimizer(a)
train = optimizer.minimize(cost)  # goal is minimize cost

# before starting, initialize the variables
init = tf.initialize_all_variables()

# launch
sess = tf.Session()
sess.run(init)

# fit the line
for step in xrange(2001):
    sess.run(train, feed_dict={X: x_data, Y: y_data})
    if step % 100 == 0:
        print step, sess.run(cost, feed_dict={X: x_data, Y: y_data}), sess.run(W), sess.run(b)

print sess.run(hypothesis, feed_dict={X: 5})
print sess.run(hypothesis, feed_dict={X: 2.5})
</code></pre>

<p>and that is result follow  </p>

<pre><code>0 1.60368e+10 [ 4612.54003906] [ 406.81304932]
100 nan [ nan] [ nan]
200 nan [ nan] [ nan]
300 nan [ nan] [ nan]
400 nan [ nan] [ nan]
500 nan [ nan] [ nan]
600 nan [ nan] [ nan]
700 nan [ nan] [ nan]
800 nan [ nan] [ nan]
900 nan [ nan] [ nan]
1000 nan [ nan] [ nan]
1100 nan [ nan] [ nan]
1200 nan [ nan] [ nan]
1300 nan [ nan] [ nan]
1400 nan [ nan] [ nan]
1500 nan [ nan] [ nan]
1600 nan [ nan] [ nan]
1700 nan [ nan] [ nan]
1800 nan [ nan] [ nan]
1900 nan [ nan] [ nan]
2000 nan [ nan] [ nan]
[ nan]
[ nan]
</code></pre>

<p>I don't understand why this result is <code>nan</code>?</p>

<p>If i change the initial data to this</p>

<pre><code>x_data = [1., 2., 3., 4.]
y_data = [2., 4., 6., 8.]
</code></pre>

<p>Then it was working no problem. Why is that?</p>

<p>You are overflowing float32 because the learning rate is too high for your problem, and instead of converging the weight variable (<code>W</code>) is oscillating towards larger and larger magnitudes on each step of gradient descent.</p>

<p>If you change</p>

<pre><code>a = tf.Variable(0.1)
</code></pre>

<p>to </p>

<pre><code>a = tf.Variable(0.001)
</code></pre>

<p>the weights should converge better. You will probably want to increase number of iterations (to ~ 50000) too.</p>

<p>Picking a good learning rate is often the first challenge when implementing or using a machine learning algorithm. Getting increased loss values instead of converging to a minimum is usually a sign that learning rate is too high.</p>

<p>In your case the specific problem of fitting to the line is made more vulnerable to diverging weights when you use larger magnitudes in the training data. This is one reason why it is common to normalise data prior to training in e.g. neural networks.</p>

<p>In addition your starting weight and bias are given a very large range, which means they can be very far from ideal values and have very large loss values and gradients at the start. Picking a good range for initial values is another critical thing to get right as you look at more advanced learning algorithms.</p>

python tensorflow