Caffe: Loading images and labels within python for fine-tuning
<p>I am having some problems when I try to load a batch of images + labels within python and try use it to train a network.
I am working with pair of images, that I transform into one (for example, by averaging equivalent pixels from both images) and then I feed it to the network. As the number of pairs is too big (combination of the individual images two by two) to store all in memory, I am creating each batch in each iteration and I would like to give it to the network.</p>

<p>I am instantiating the network this way:</p>

<pre><code>solver = caffe.get_solver(path_to_solver_file)
</code></pre>

<p>where the input network layers are defined with:</p>

<pre><code>layer {
  name: "data"
  type: "ImageData"
  top: "data"
  top: "label"
  include {
    phase: TRAIN
  }
  image_data_param { 
    batch_size: 1
    new_height: 227
    new_width: 227
  }
}
layer {
  name: "data"
  type: "ImageData"
  top: "data"
  top: "label"
  include {
    phase: TEST
  }
  image_data_param {
    batch_size: 1
    new_height: 227
    new_width: 227
  }
}
</code></pre>

<p>In my earlier experiments, I had a <strong>source</strong> parameter inside <strong>image_data_param</strong>, where I would pass a file with my images and labels for both training and test.
However, as I want to load them within python, I removed the source parameter, but got the following error:</p>

<pre><code>0830 17:01:49.014819 1967923200 layer_factory.hpp:77] Creating layer data

I0830 17:01:49.014858 1967923200 net.cpp:91] Creating Layer data

I0830 17:01:49.014868 1967923200 net.cpp:399] data -&gt; data

I0830 17:01:49.014890 1967923200 net.cpp:399] data -&gt; label

I0830 17:01:49.014910 1967923200 image_data_layer.cpp:38] Opening file 

I0830 17:01:49.014935 1967923200 image_data_layer.cpp:53] A total of 0 images.

Segmentation fault: 11
</code></pre>

<p>I haven't got to this point yet, but after I'm able to instantiate the network, I was going to load the batch and perform one step of the SGD optimization using:</p>

<pre><code>net.blobs["data"].data[...] = images
net.blobs["label"].data[...] = labels
net.step(1)
</code></pre>

<p>I have searched for examples and tutorials (for example, <a href="http://nbviewer.jupyter.org/github/BVLC/caffe/blob/tutorial/examples/03-fine-tuning.ipynb" rel="nofollow">here</a> and <a href="http://christopher5106.github.io/deep/learning/2015/09/04/Deep-learning-tutorial-on-Caffe-Technology.html" rel="nofollow">here</a>) that perform fine-tuning and testing using python, however the great majority only discuss the forward passes during a test phase, whereas the ones that fine-tune a network define the training data (labels and images) using the source parameter, and not by loading directly from the python interface.</p>

<p>The layer <code>ImageData</code> can only read images from a text file which holds the address of images on each line. You cannot use an <code>ImageData</code> layer and feed in images on the fly.</p>

<p>If you are interested in loading images on the fly, checkout this <a href="http://stackoverflow.com/a/39097123/5465000">link</a>. I suggest using the deploy version (3rd option explained in that post), which is way handier and more flexible.</p>

<p>When using the deploy version, you can set the input data for your network like this:</p>

<pre><code>x = data;
y = labels;
solver.net.blobs['data'].data[...] = x
net.blobs['label'].data[...] = y
</code></pre>

<p>Then you can either call <code>solver.net.forward()</code> to only see the outputs; 
or you can call <code>solver.net.step(1)</code> to run a forward and a backward (train for 1 iteration).</p>

python input batch-processing caffe pycaffe