Tensorflow - adding L2 regularization loss simple example
<p>I am familiar with machine learning, but I am learning Tensorflow on my own by reading some slides from universities. Below I'm setting up the loss function for linear regression with only one feature. I'm adding an L2 loss to the total loss, but I am not sure if I'm doing it correctly:</p>

<pre><code># Regularization
reg_strength = 0.01

# Create the loss function.
with tf.variable_scope("linear-regression"):
    W    = tf.get_variable("W", shape=(1, 1), initializer=tf.contrib.layers.xavier_initializer())
    b    = tf.get_variable("b", shape=(1,), initializer=tf.constant_initializer(0.0))
    yhat = tf.matmul(X, W) + b

    error_loss = tf.reduce_sum(((y - yhat)**2)/number_of_examples)
    #reg_loss   = reg_strength * tf.nn.l2_loss(W)   # reg 1
    reg_loss   = reg_strength * tf.reduce_sum(W**2) # reg 2
    loss       = error_loss + reg_loss

# Set up the optimizer.
opt_operation = tf.train.GradientDescentOptimizer(0.001).minimize(loss)
</code></pre>

<p>My specific questions are:</p>

<ol>
<li><p>I have two lines (commented as <code>reg 1</code> and <code>reg 2</code>) that compute the L2 loss of the weight <code>W</code>. The line marked with <code>reg 1</code> uses the Tensorflow built-in function. Are these two L2 implementations equivalent?</p></li>
<li><p>Am I adding the regularization loss <code>reg_loss</code> correctly to the final loss function?</p></li>
</ol>

<blockquote>
  <p>Are these two L2 implementations equivalent?</p>
</blockquote>

<p>Almost, as @fabrizioM pointed out, you can see <a href="https://www.tensorflow.org/versions/r0.10/api_docs/python/nn.html#l2_loss" rel="nofollow">here</a> for the introduction to the l2_loss in TensorFlow docs. </p>

<blockquote>
  <p>Am I adding the regularization loss reg_loss correctly to the final loss function?</p>
</blockquote>

<p>So far so good : )</p>

python machine-learning tensorflow