How to use `scipy.optimize.leastsq` to optimize in the joint least squares direction?
<p>I want to be able to move along a gradient in the <em>joint least squares direction</em>.</p>

<p>I thought I could do this using <code>scipy.optimize.leastsq</code> (<a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html" rel="nofollow">http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.leastsq.html</a>).  (Perhaps I'm wrong, maybe there's an easier way to do this?).</p>

<p>I'm having difficulty understanding what to use, and how to move in the joint least squares direction, while still increasing the parameters.  </p>

<p>What I need to do is input something like this:</p>

<pre><code>[1,0]
</code></pre>

<p>And, have it move along the least squares direction, which would mean, increasing either or both values <code>1</code> and <code>0</code>, but doing so such that the sum of the squared values is as small as possible.  </p>

<p>This would mean <code>[1,0]</code> would increase to <code>[1, &lt;something barely greater than 0&gt;]</code>, and eventually would reach <code>[1,1]</code>.  At which point, both <code>1</code>'s would increase at the same rate.</p>

<p>How would I program this?  It seems to me like <code>scipy.optimize.leastsq</code> would be of use here, but I cannot figure out how to use it?  </p>

<p>Thankyou.</p>

<p>I don't think you need <code>scipy.optimize.leastsq</code> because your problem can be solved analytically. At any moment, the gradient of the fuction <code>np.sum(x)</code> where <code>x</code> is an array, is <code>2*x</code>. So, if you want to obtain the smallest increase, then you have to increase the smallest component of the gradient, which you can find with <code>np.argmin</code>. Here is a simple solution:</p>

<pre><code>def g(x):
    return np.array(2*x)

x = np.array([1.,0.])
for _ in range(200):
    eps = np.zeros_like(x)
    index = np.argmin(g(x))
    eps[index] = 0.01 #or whatever
    x += eps
    print(x)
</code></pre>

<p>When multiple indices have the same value, <code>np.argmin</code> returns the first occurrence, so you will encounter certain oscillations  that you can minimize reducing <code>eps</code></p>

python numpy optimization scipy least-squares