How to vectorize hinge loss gradient computation
<p>I'm computing thousands of gradients and would like to vectorize the computations in Python. The context is SVM and the loss function is Hinge Loss. Y is Mx1, X is MxN and w is Nx1.</p>

<pre><code> L(w) = lam/2 * ||w||^2 + 1/m Sum i=1:m ( max(0, 1-y[i]X[i]w) )
</code></pre>

<p>The gradient of this is</p>

<pre><code>grad = lam*w + 1/m Sum i=1:m {-y[i]X[i].T if y[i]*X[i]*w &lt; 1, else 0}
</code></pre>

<p>Instead of looping through each element of the sum and evaluating the max function, is it possible to vectorize this? I want to use something like np.where like the following</p>

<pre><code>grad = np.where(y*X.dot(w) &lt; 1, -X.T.dot(y), 0)
</code></pre>

<p>This does not work because where the condition is true, -X.T*y is the wrong dimension. </p>

<p>edit: list comprehension version, would like to know if there's a cleaner or more optimal way</p>

<pre><code>def grad(X,y,w,lam):
    # cache y[i]*X[i].dot(w), each row of Xw is multiplied by a single element of y
    yXw = y*X.dot(w)
    # cache y[i]*X[i], note each row of X is multiplied by a single element of y
    yX = X*y[:,np.newaxis]
    # return the average of this max function
    return lam*w + np.mean( [-yX[i] if yXw[i] &lt; 1 else 0 for i in range(len(y))] )
</code></pre>

<p>you have two vectors A and B, and you want to return array C, such that C[i] = A[i] if B[i] &lt; 1 and 0 else, consequently all you need to do is</p>

<pre><code>C := A * sign(max(0, 1-B)) # suprisingly similar to the original hinge loss, right?:)
</code></pre>

<p>since</p>

<ul>
<li>if B &lt; 1 then 1-B > 0, thus max(0, 1-B) > 0 and sign(max(0, 1-B)) == 1</li>
<li>if B >= 1 then 1-B &lt;= 0, thus max(0, 1-B) = 0 and sign(max(0, 1-B)) == 0</li>
</ul>

<p>so in your code it will be something like</p>

<pre><code>A = (y*X.dot(w)).ravel()
B = (X*y[:,np.newaxis]).ravel()
C = A * np.sign(np.maximum(0, 1-B))
</code></pre>

python optimization machine-learning