Python Minibatch Dictionary Learning
<p>I'd like to implement error tracking with dictionary learning in python, using sklearn's <a href="http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.MiniBatchDictionaryLearning.html" rel="nofollow">MiniBatchDictionaryLearning</a>, so that I can record how the error decreases over the iterations. I have two methods to do it, neither of which really worked. Set up:</p>

<ul>
<li><strong>Input data X</strong>, numpy array shape (n_samples, n_features) = (298143, 300). These are patches of shape (10, 10), generated from an image of shape (642, 480, 3).</li>
<li><strong>Dictionary learning parameters</strong>: No. of columns (or atoms) = 100, alpha = 2, transform algorithm = OMP, total no. of iterations = 500 (keep it small first, just as a test case)</li>
<li><p><strong>Calculating error</strong>: After learning the dictionary, I encode the original image again based on the learnt dictionary. Since both the encoding and the original are numpy arrays of the same shape (642, 480, 3), I'm just doing elementwise Euclidean distance for now: </p>

<p>err = np.sqrt(np.sum(reconstruction - original)**2))</p></li>
</ul>

<p>I did a test run with these parameters, and the full fit was able to produce a pretty good reconstruction with a low error, so that's good.Now on to the two methods:</p>

<p><strong>Method 1:</strong> Save the learnt dictionary every 100 iterations, and record the error. For 500 iterations, this gives us 5 runs of 100 iterations each. After each run, I compute the error, then use the currently learnt dictionary as an initialization for the next run. </p>

<pre><code># Fit an initial dictionary, V, as a first run
dico = MiniBatchDictionaryLearning(n_components = 100,
                                   alpha = 2,
                                   n_iter = 100,
                                   transform_algorithm='omp')
dl = dico.fit(patches)
V = dl.components_

# Now do another 4 runs.
# Note the warm restart parameter, dict_init = V.
for i in range(n_runs):
    print("Run %s..." % i, end = "")
    dico = MiniBatchDictionaryLearning(n_components = 100,
                                       alpha = 2,
                                       n_iter = n_iterations,
                                       transform_algorithm='omp',
                                       dict_init = V)
    dl = dico.fit(patches)
    V = dl.components_

    img_r = reconstruct_image(dico, V, patches)
    err = np.sqrt(np.sum((img - img_r)**2))
    print("Err = %s" % err)
</code></pre>

<p>Problem: The error isn't decreasing, and was pretty high. The dictionary wasn't learnt very well either. </p>

<p><strong>Method 2</strong>: Cut the input data X into, say, 500 batches, and do partial fitting, using the <code>partial_fit()</code> method.</p>

<pre><code>batch_size = 500
n_batches = X.shape[0] // batch_size
print(n_batches) # 596

for iternum in range(n_batches):
    batch = patches[iternum*batch_size : (iternum+1)*batch_size]
    V = dico.partial_fit(batch)
</code></pre>

<p>Problem: this seems to take about 5000 times longer. </p>

<p>I'd like to know if there's a way to retrieve the error over the fitting process?</p>

<p>Each call to <code>fit</code> re-initializes the model and forgets any previous call to <code>fit</code>: this is the expected behavior of all estimators in scikit-learn.</p>

<p>I think using <code>partial_fit</code> in a loop is the right solution but you should call it on small batches (as done in the fit method method, the default batch_size value is just 3) and then only compute the cost every 100 or 1000 calls to <code>partial_fit</code> for instance:</p>

<pre><code>batch_size = 3
n_epochs = 20
n_batches = X.shape[0] // batch_size
print(n_batches) # 596


n_updates = 0
for epoch in range(n_epochs):
    for i in range(n_batches):
        batch = patches[i * batch_size:(i + 1) * batch_size]
        dico.partial_fit(batch)
        n_updates += 1
        if n_updates % 100 == 0:
            img_r = reconstruct_image(dico, dico.components_, patches)
            err = np.sqrt(np.sum((img - img_r)**2))
            print("[epoch #%02d] Err = %s" % (epoch, err))
</code></pre>

python numpy scikit-learn