Getting low test accuracy using Tensorflow batch_norm function
<p>I am using the official Batch Normalization (BN) function (<a href="https://github.com/tensorflow/tensorflow/blob/b826b79718e3e93148c3545e7aa3f90891744cc0/tensorflow/contrib/layers/python/layers/layers.py#L100" rel="nofollow">tf.contrib.layers.batch_norm()</a>) of Tensorflow on the MNIST data. I use the following code for adding BN:</p>

<pre><code>local4_bn = tf.contrib.layers.batch_norm(local4, is_training=True)
</code></pre>

<p>During testing, I change "is_training=False" in the above line of code and observe only 20% accuracy. However, it gives ~99% accuracy if I use the  above code also for testing (i.e., keeping is_training=True) with a batch size of 100 images. This observation indicates that the <em>exponential moving average and variance</em> computed by <a href="https://github.com/tensorflow/tensorflow/blob/b826b79718e3e93148c3545e7aa3f90891744cc0/tensorflow/contrib/layers/python/layers/layers.py#L100" rel="nofollow">batch_norm()</a> are probably incorrect or I am missing something in my code.</p>

<p>Can anyone please answer about the solution of the above problem.</p>

<p>You get ~99% accuracy when you test you model with <code>is_training=True</code> only because of the batch size of 100.
If you change the batch size to 1 your accuracy will decrease.</p>

<p>This is due to the fact that you're computing the exponential moving average and variance for the input batch and than you're (batch-)normalizing the layers output using these values.</p>

<p>The <code>batch_norm</code> function have the parameter <code>variables_collections</code> that helps you to store the computed moving average and variance during the train phase and reuse them during the test phase.</p>

<p>If you define a collection for these variables, then the <code>batch_norm</code> layer will use them during the testing phase, instead of calculating new values.</p>

<p>Therefore, if you change you batch normalization layer definition to</p>

<pre><code>local4_bn = tf.contrib.layers.batch_norm(local4, is_training=True, variables_collections=["batch_norm_non_trainable_variables_collection"])
</code></pre>

<p>The layer will store the computed variables into the <code>"batch_norm_non_trainable_variables_collection"</code> collection.</p>

<p>In the test phase, when you pass the <code>is_training=False</code> parameters, the layer will re-use the computed value that it find in the collection.</p>

<p>Note that the moving average and the variance are not trainable parameters and therefore, if you save only your model trainable parameters in the checkpoint files, you have to manually add the non-trainable variables stored into the previously defined collection.</p>

<p>You can do it when you create the <code>Saver</code> object:</p>

<pre><code>saver = tf.train.Saver(tf.get_trainable_variables() + tf.get_collection_ref("batch_norm_non_trainable_variables_coââllection") + otherlistofvariables)
</code></pre>

<p>In addiction, since batch normalization can limit the expressive power of the layer which is applied to (because it restricts the range of the values), you should enable the network to learn the parameters <code>gamma</code> and <code>beta</code> (the affine transformation coefficients described in the <a href="https://arxiv.org/abs/1502.03167" rel="nofollow">paper</a>) that allows the network to learn, thus, an affine transformation that increase the representation power of the layer.</p>

<p>You can enable the learning of these parameters setting to <code>True</code> the parameter of the <code>batch_norm</code> function, in this way:</p>

<pre><code>local4_bn = tf.contrib.layers.batch_norm(
    local4,
    is_training=True,
    center=True, # beta
    scale=True, # gamma
    variables_collections=["batch_norm_non_trainable_variables_collection"])
</code></pre>

python tensorflow