Excluding a set of indices from an array whilst keeping its original order
<p>I have an HDF5 dataset which I read as a numpy array: </p>

<pre><code>my_file = h5py.File(h5filename, 'r')
file_image = my_file['/image']
</code></pre>

<p>and a list of indices called <code>in</code>. </p>

<p>I want to split the <code>image</code> dataset into two separate <code>np.array</code>s: one containing images corresponding to the <code>in</code> indices, and the other containing images whose indices are not in <code>in</code>. The order of images is very important - I want to split the dataset such that the original order of the images is preserved within each subset. How can I achieve this? </p>

<p>I tried the following:</p>

<pre><code>labeled_image_dataset = list(file_image[in])
</code></pre>

<p>However, h5py gave me an error saying that the indices must be in increasing order. I can't change the order of indices, since I need the images to stay in their original order.</p>

<p>My code:</p>

<pre><code>my_file = h5py.File(h5filename, 'r')
file_image = my_file['/imagePatches']
li = dataframe["label"]
temporary_list = file_image

# select images whose indices exist in "in"
labeled_dataset = list(temporary_list[in])

# select images whose indices don't exist in "in"
unlabeled_dataset = np.delete(temporary_list, in, 0)
</code></pre>

<p>I'm not following why or how you are choosing the indexes, but as the error indicates, when you index an array on the h5 file, then indexes have to be sorted.  Remember files are serial storage, so it is easier and faster to read straight through rather than go back and forth.  Regardless of where the constraint lies, in <code>h5py</code> or the <code>h5</code> back end, <code>idx</code> has to ordered.</p>

<p>But if you load the whold array into memory (or some contiguous chunk), that may require a <code>copy</code>, then you can use an unsorted, or even repetitious, <code>idx</code> list.</p>

<p>In other words, <code>h5py</code> arrays can be indexed like <code>numpy</code> arrays, but with some limits.</p>

<p><a href="http://docs.h5py.org/en/latest/high/dataset.html#fancy-indexing" rel="nofollow">http://docs.h5py.org/en/latest/high/dataset.html#fancy-indexing</a></p>

python numpy indexing h5py