Dask/hdf5: Read by group?
<p>I must read in and operate independently over many chunks of a large dataframe/numpy array. However, these chunks are chosen in a specific, non-uniform manner and are broken naturally into groups within a hdf5 file. Each group is small enough to fit into memory (though even without restriction, I suppose the standard chunking procedure should suffice.) </p>

<p>Specifically, instead of </p>

<pre><code> f = h5py.File('myfile.hdf5')
 x = da.from_array(f['/data'], chunks=(1000, 1000))
</code></pre>

<p>I want something closer to (pseudocode):</p>

<pre><code> f = h5py.File('myfile.hdf5')
 x = da.from_array(f, chunks=(f['/data1'], f['/data2'], ...,))
</code></pre>

<p><a href="http://dask.pydata.org/en/latest/delayed-collections.html" rel="nofollow">http://dask.pydata.org/en/latest/delayed-collections.html</a> I believe hints this is possible but I am still reading into and understanding dask/hdf5.</p>

<p>My previous implementation uses a number of CSV files and reads them in as needed with its own multi-processing logic. I would like to collapse all this functionality into dask with hdf5.</p>

<p>Is chunking by hdf5 group/read possible and my line of thought ok?</p>

<p>I would read many dask.arrays from many groups as single-chunk dask.arrays and then concatenate or stack those groups.</p>

<h3>Read many dask.arrays</h3>

<pre><code>f = h5py.File(...)
dsets = [f[dset] for dset in datasets]
arrays = [da.from_array(dset, chunks=dset.shape) for dset in dsets]
</code></pre>

<h3>Alternatively, use a lock to defend HDF5</h3>

<p>HDF5 is not threadsafe, so lets use a lock to defend it from parallel reads. I haven't actually checked to see if this is necessary or not when reading across different groups.</p>

<pre><code>from threading import Lock
lock = Lock()

arrays = [da.from_array(dset, chunks=dset.shape, lock=lock) 
           for dset in dsets]
</code></pre>

<h3>Stack or Concatenate arrays together</h3>

<pre><code>array = da.concatenate(arrays, axis=0)
</code></pre>

<p>See <a href="http://dask.pydata.org/en/latest/array-stack.html" rel="nofollow">http://dask.pydata.org/en/latest/array-stack.html</a></p>

<h3>Or use dask.delayed</h3>

<p>You could also, as you suggest, use <a href="http://dask.pydata.org/en/latest/delayed.html" rel="nofollow">dask.delayed</a> to do the first step in reading single-chunk dask.arrays</p>

python hdf5 h5py dask