python linear regression implementation
<p>I've been trying to do my own implementation of a simple linear regression algorithm, but I'm having some trouble with the gradient descent.</p>

<p>Here's how I coded it:</p>

<pre><code>def gradientDescentVector(data, alpha, iterations):
    a = 0.0
    b = 0.0
    X = data[:,0]
    y = data[:,1]
    m = data.shape[0]
    it = np.ones(shape=(m,2))
    for i in range(iterations):
        predictions = X.dot(a).flatten() + b

        errors_b = (predictions - y)
        errors_a = (predictions - y) * X

        a = a - alpha * (1.0 / m) * errors_a.sum()
        b = b - alpha * (1.0 / m) * errors_b.sum()
    return a, b
</code></pre>

<p>Now, I know this won't scale well with more variables, but I was just trying with the simple version first, and follow up from there.</p>

<p>I was following the gradient descent algorithm from the machine learning course at coursera:</p>

<p><a href="http://i.stack.imgur.com/w1BRm.png" rel="nofollow"><img src="http://i.stack.imgur.com/w1BRm.png" alt="enter image description here"></a></p>

<p>But I'm getting infinite values after ~90 iterations (on a specific dataset), and haven't been able to wrap my head around this so far.</p>

<p>I've tried iterating over each value before I learned about numpy's broadcasting and was getting the same results.</p>

<p>If anyone could shed some light on what could be the problem here, it would be great.</p>

<p>It is clear that the parameters are diverging from the optimum ones. One possible reason may be that you are using too large a value for the learning rate ("alpha"). Try decreasing the learning rate. Here is a rule of thumb. Start always from a small value like 0.001. Then try increasing the learning rate by taking a three time higher learning rate than previous. If it gives less MSE error (or whatever error function you are using), then its fine. If not try taking a value between 0.001 and 0.003. Next if the latter hold true, then try this recursively until you reach a satisfying MSE.</p>

python numpy machine-learning linear-regression