Tensorflow - adding L2 regularization loss simple example
<p>I am familiar with machine learning, but I am learning Tensorflow on my own by reading some slides from universities. Below I'm setting up the loss function for linear regression with only one feature. I'm adding an L2 loss to the total loss, but I am not sure if I'm doing it correctly:</p>

<pre><code># Regularization
reg_strength = 0.01

# Create the loss function.
with tf.variable_scope("linear-regression"):
    W    = tf.get_variable("W", shape=(1, 1), initializer=tf.contrib.layers.xavier_initializer())
    b    = tf.get_variable("b", shape=(1,), initializer=tf.constant_initializer(0.0))
    yhat = tf.matmul(X, W) + b

    error_loss = tf.reduce_sum(((y - yhat)**2)/number_of_examples)
    #reg_loss   = reg_strength * tf.nn.l2_loss(W)   # reg 1
    reg_loss   = reg_strength * tf.reduce_sum(W**2) # reg 2
    loss       = error_loss + reg_loss

# Set up the optimizer.
opt_operation = tf.train.GradientDescentOptimizer(0.001).minimize(loss)
</code></pre>

<p>My specific questions are:</p>

<ol>
<li><p>I have two lines (commented as <code>reg 1</code> and <code>reg 2</code>) that compute the L2 loss of the weight <code>W</code>. The line marked with <code>reg 1</code> uses the Tensorflow built-in function. Are these two L2 implementations equivalent?</p></li>
<li><p>Am I adding the regularization loss <code>reg_loss</code> correctly to the final loss function?</p></li>
</ol>

<p>Almost </p>

<p><a href="https://github.com/tensorflow/tensorflow/blob/d42facc3cc9611f0c9722c81551a7404a0bd3f6b/tensorflow/core/kernels/l2loss_op.h#L32" rel="nofollow">According to the L2Loss operation code</a></p>

<pre><code>output.device(d) = (input.square() * static_cast&lt;T&gt;(0.5)).sum();
</code></pre>

<p>It multiplies also for 0.5  (or in other words it divides by <code>2</code>)</p>

python machine-learning tensorflow