Creating multilabel HDF5 file for caffe
<p>I created HDF5 data using the following python script and placed HDF5 data layer. However, when I tried to train caffe using this data it keeps complaining </p>

<blockquote>
<pre><code>Check failed: num_spatial_axes_ == 2 (0 vs. 2) kernel_h &amp; kernel_w can only be used for 2D convolution
</code></pre>
</blockquote>

<p>Here is how my data looks like:</p>

<ol>
<li><p>Data(1x3253), label(1x128) binary. I sliced the 128 into 16 bytes and translated that to dec to use it as a mulitlabel. So a typical key would look like, (20, 38, 123, 345,...) 1x16. and I have 1,000,000 of data like (1). For now I am just using the first byte, so I will have one integer as a label.</p>

<pre><code>    DIR ="/x/"
    h5_fn= os.path.join('/x/h5Data_train.h5')
    from numpy import genfromtxt  

    dim=64000 
    InputData=np.arange(3253)
    data=np.arange(dim*3253)
    data.shape=(dim,3253)

    fileList=[os.path.join(i) for folder, subdir,files in os.walk(DIR) for i in files]
    for i in range(0,len(fileList)):
         InputData=np.genfromtxt(DIR+fileList[i], delimiter=',',skip_header=24)
         data[i]=InputData

    label=np.arange(dim)
    labelData=np.genfromtxt(DIR+'label_file',comments='\t',dtype=None)

    for i in range(0,dim):
        label[i]=int(labelData[i][0:2],16)

    print "Creating HDF5..."

    with h5py.File(h5_fn,'w') as f:
       f['InputData']=data
       f['label']=label

    text_fn=os.path.join('/x/hdf5.txt')
    with open(text_fn,'w') as f:
       f.write('h5_fn')
</code></pre></li>
</ol>

<p>This script creates the HDF5, but I am suspecting that the error from caffe is related to how I created my HDF5 file. Can someone tell me if there is anything wrong on how I created the HDF5. Also, is there anyway one can check if the HDF5 file created is as you want? Thanks!</p>

<h3>The problem:</h3>

<p>Caffe, by default, expects its data to be 4D: batch_size -by- channel -by- height -by- width.<br>
In your <a href="http://stackoverflow.com/q/39888054/1714410">model</a> you assume each sample is of shape 1-by-1-by-3253, that is: your data is 1D with only non-singleton width dimension. This is an important detail since you apply convolution along the width dimension.<br>
On the other hand, your HDF5 data is only 2D, and caffe interpret it as <code>dim</code> examples with 3253 <em>channels</em> of width and height 1.<br>
Now you can understand the error message you get: you have a convolution layer with <code>kernel_width</code> and <code>kernel_height</code> params, but the data (as far as caffe understands it) has width and height of 1.</p>

<h3>A solution:</h3>

<p>You simply need to <code>reshape</code> your <code>data</code>:</p>

<pre><code>data.shape=(dim,1,1,3253)
</code></pre>

<p>Now <code>data</code> has 1 channel and height 1 for each sample and width of 3253.</p>

<hr>

<p>PS,<br>
You are writting to <code>'/x/hdf5.txt'</code> the actual <em>string</em> <code>'h5_fn'</code> instead of the string stroed in the variable <code>h5_fn</code>...</p>

python machine-learning deep-learning caffe hdf5