fmin_ncg not returning an optimized result
<p>I am trying to use fmin_ncg for minimizing my cost function. But, the results that I get back are not minimized. I get the same result I would get without advanced optimization. I know for a fact that it can further be minimized. </p>

<p>PS. I am trying to code assignment 2 of the <a href="https://www.coursera.org/learn/machine-learning" rel="nofollow">Coursera's ML course</a>. </p>

<p>My cost fn:</p>

<pre><code>def costFn(theta, X, y, m, lam):
    h = sigmoid(X.dot(theta))
    theta0 = theta
    J = 1 / m * np.sum((-(y * np.log(h))) - ((1-y) * np.log(1-h))) + (lam/(2*m) * theta0.T.dot(theta0))
    return J.flatten()
</code></pre>

<p>X would look something like this:</p>

<pre><code>[[  1.00000000e+00   5.12670000e-02   6.99560000e-01 ...,   6.29470940e-04
8.58939846e-03   1.17205992e-01]
 [  1.00000000e+00  -9.27420000e-02   6.84940000e-01 ...,   1.89305413e-03
   -1.39810280e-02   1.03255971e-01]
 [  1.00000000e+00  -2.13710000e-01   6.92250000e-01 ...,   1.04882142e-02
   -3.39734512e-02   1.10046893e-01]
 ..., 
 [  1.00000000e+00  -4.84450000e-01   9.99270000e-01 ...,   2.34007252e-01
   -4.82684337e-01   9.95627986e-01]
 ....
</code></pre>

<p>Y is a bunch of 0s and 1s</p>

<pre><code>[[1]
[1]
[1]
[1]
...
[0]
[0]]

X.shape = (118, 28)
y.shape = (118, 1)
</code></pre>

<p>My grad function:</p>

<pre><code>def grad(theta, X, y, m, lam):
    h = sigmoid(X.dot(theta))
    theta0 = initial_theta
    gg = 1.0 / m * ((X.T.dot(h-y)) + (lam * theta0))
    return gg.flatten()
</code></pre>

<p>Using just my costFn and grad, I get the following:</p>

<pre><code>Cost at initial theta (zeros): 0.69314718056
</code></pre>

<p>With fmin_ncg:</p>

<pre><code>xopt = fmin_ncg(costFn, fprime=grad, x0=initial_theta, args=(X, y, m, lam), maxiter=400, disp=True, full_output=True )
</code></pre>

<p>I get:</p>

<pre><code>Optimization terminated successfully.
     Current function value: 0.693147
     Iterations: 1
     Function evaluations: 2
     Gradient evaluations: 4
     Hessian evaluations: 0
</code></pre>

<p>Using octave, my J after advanced optimization should be:</p>

<pre><code> 0.52900
</code></pre>

<p>What am I doing wrong?</p>

<hr>

<p>EDIT:
I got my optimization to work:</p>

<pre><code>y1 = y.flatten()
Result = op.minimize(fun = costFn, 
                x0 = initial_theta, 
                args = (X, y1, m, lam),
                method = 'CG',
                options={'disp': True})
</code></pre>

<p>I get the costFn to be 0.52900, which is what I expected.</p>

<p>But the values of 'theta' are a bit off that the accuracy is only 42%. It's supposed to be 83%.</p>

<p>The values of theta I got:</p>

<pre><code>[ 1.14227089  0.60130664  1.16707559 -1.87187892 -0.91534354 -1.26956697
0.12663015 -0.36875537 -0.34522652 -0.17363325 -1.42401493 -0.04872243
-0.60650726 -0.269242   -1.1631064  -0.24319088 -0.20711764 -0.04333854
-0.28026111 -0.28693582 -0.46918892 -1.03640373  0.02909611 -0.29266766
 0.01725324 -0.32899144 -0.13795701 -0.93215664]
</code></pre>

<p>The actual values:</p>

<pre><code>[1.273005 0.624876 1.177376 -2.020142 -0.912616 -1.429907 0.125668 -0.368551
-0.360033 -0.171068 -1.460894 -0.052499 -0.618889 -0.273745 -1.192301 
-0.240993 -0.207934 -0.047224 -0.278327 -0.296602 -0.453957 -1.045511 
0.026463 -0.294330 0.014381 -0.328703 -0.143796 -0.924883]
</code></pre>

<p>First of all your gradient is invalid</p>

<pre><code>def grad(theta, X, y, m, lam):
    h = sigmoid(X.dot(initial_theta))
    theta0 = initial_theta
    gg = 1 / m * ((X.T.dot(h-y)) + (lam * theta0))
    return gg.flatten()
</code></pre>

<p>this function <strong>never uses theta</strong>, you put <code>initial_theta</code> instead, which is incorrect.</p>

<p>Similar error in the cost</p>

<pre><code>def costFn(theta, X, y, m, lam):
    h = sigmoid(X.dot(initial_theta))
    theta0 = theta
    J = 1 / m * np.sum((-(y * np.log(h))) - ((1-y) * np.log(1-h))) + (lam/(2*m) * theta0.T.dot(theta0))
    return J.flatten()
</code></pre>

<p>you have some odd mix of <code>theta</code> and <code>initial_theta</code>, which also does not make sense, there should be only <code>theta</code> inside. As a side note - there should be no need for flattening, your cost function should be <strong>a scalar</strong>, thus if you have to flatten - something is wrong in your computations.</p>

<p>Also worth checking - what is your <code>m</code>? If it is an integer, and you are using python 2.X, then <code>1 / m</code> equals <strong>zero</strong>, since it is <strong>integer division</strong>. You should do <code>1.0 / m</code> instead. (in both functions)</p>

python machine-learning octave jupyter logistic-regression