Python gradient descent - cost keeps increasing
<p>I'm trying to implement gradient descent in python and my loss/cost keeps increasing with every iteration.</p>

<p>I've seen a few people post about this, and saw an answer here: <a href="http://stackoverflow.com/questions/17784587/gradient-descent-using-python-and-numpy">gradient descent using python and numpy</a></p>

<p>I believe my implementation is similar, but cant see what I'm doing wrong to get an exploding cost value:</p>

<pre><code>Iteration: 1 | Cost: 697361.660000
Iteration: 2 | Cost: 42325117406694536.000000
Iteration: 3 | Cost: 2582619233752172973298548736.000000
Iteration: 4 | Cost: 157587870187822131053636619678439702528.000000
Iteration: 5 | Cost: 9615794890267613993157742129590663647488278265856.000000
</code></pre>

<p>I'm testing this on a dataset I found online (LA Heart Data): <a href="http://www.umass.edu/statdata/statdata/stat-corr.html" rel="nofollow">http://www.umass.edu/statdata/statdata/stat-corr.html</a></p>

<p>Import code:</p>

<pre><code>dataset = np.genfromtxt('heart.csv', delimiter=",")

x = dataset[:]
x = np.insert(x,0,1,axis=1)  # Add 1's for bias
y = dataset[:,6]
y = np.reshape(y, (y.shape[0],1))
</code></pre>

<p>Gradient descent:</p>

<pre><code>def gradientDescent(weights, X, Y, iterations = 1000, alpha = 0.01):
    theta = weights
    m = Y.shape[0]
    cost_history = []

    for i in xrange(iterations):
        residuals, cost = calculateCost(theta, X, Y)
        gradient = (float(1)/m) * np.dot(residuals.T, X).T
        theta = theta - (alpha * gradient)

        # Store the cost for this iteration
        cost_history.append(cost)
        print "Iteration: %d | Cost: %f" % (i+1, cost)
</code></pre>

<p>Calculate cost:</p>

<pre><code>def calculateCost(weights, X, Y):
    m = Y.shape[0]
    residuals = h(weights, X) - Y
    squared_error = np.dot(residuals.T, residuals)

    return residuals, float(1)/(2*m) * squared_error
</code></pre>

<p>Calculate hypothesis:</p>

<pre><code>def h(weights, X):   
    return np.dot(X, weights)
</code></pre>

<p>To actually run it:</p>

<pre><code>gradientDescent(np.ones((x.shape[1],1)), x, y, 5)
</code></pre>

<p>In general, if your cost is increasing, then the very first thing you should check is to see if your learning rate is too large. In such cases, the rate is causing the cost function to jump over the optimal value and increase upwards to infinity. Try different small values of your learning rate. When I face the problem that you describe, I usually repeatedly try 1/10 of the learning rate until I can find a rate where J(w) decreases.</p>

<p>Another problem might be a bug in your derivative implementation. A good way to debug is to do a gradient check to compare the analytic gradient versus the numeric gradient.</p>

python numpy machine-learning regression gradient-descent