Excluding a set of indices from an array whilst keeping its original order
<p>I have an HDF5 dataset which I read as a numpy array: </p>

<pre><code>my_file = h5py.File(h5filename, 'r')
file_image = my_file['/image']
</code></pre>

<p>and a list of indices called <code>in</code>. </p>

<p>I want to split the <code>image</code> dataset into two separate <code>np.array</code>s: one containing images corresponding to the <code>in</code> indices, and the other containing images whose indices are not in <code>in</code>. The order of images is very important - I want to split the dataset such that the original order of the images is preserved within each subset. How can I achieve this? </p>

<p>I tried the following:</p>

<pre><code>labeled_image_dataset = list(file_image[in])
</code></pre>

<p>However, h5py gave me an error saying that the indices must be in increasing order. I can't change the order of indices, since I need the images to stay in their original order.</p>

<p>My code:</p>

<pre><code>my_file = h5py.File(h5filename, 'r')
file_image = my_file['/imagePatches']
li = dataframe["label"]
temporary_list = file_image

# select images whose indices exist in "in"
labeled_dataset = list(temporary_list[in])

# select images whose indices don't exist in "in"
unlabeled_dataset = np.delete(temporary_list, in, 0)
</code></pre>

<p>Using <code>in</code> as a variable name is not a good idea, since <code>in</code> is a Python keyword (e.g. see the list comprehensions below). For clarity I've renamed it as <code>idx</code></p>

<p>One straightforward solution would be to simply iterate over your set of indices in a standard Python <code>for</code> loop or list comprehension:</p>

<pre><code>labelled_dataset = [file_image[ii] for ii in idx]
unlabelled_dataset = [file_image[ii] for ii in range(len(file_image)) 
                      if ii not in idx]
</code></pre>

<p>It might be faster to use vectorized indexing, e.g. if you have a very large number of small image patches to load. In that case you could use <a href="http://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html" rel="nofollow"><code>np.argsort</code></a> to find the set of indices that will sort <code>idx</code> in ascending order, then index into <code>file_image</code> with the sorted indices. Afterwards you can "undo" the effect of sorting <code>idx</code> by indexing into the resulting array with the same set of indices that were used to sort <code>idx</code>.</p>

<p>Here's a simplified example to illustrate:</p>

<pre><code>target = np.array(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'])
idx = np.array([5, 1, 9, 0, 3])

# find a set of indices that will sort `idx` in ascending order
order = np.argsort(idx)
ascending_idx = idx[order]

# use the sorted indices to index into the target array
ascending_result = target[ascending_idx]

# "undo" the effect of sorting `idx` by indexing the result with `order`
result = ascending_result[order]

print(np.all(result == target[idx]))
# True
</code></pre>

python numpy indexing h5py