how can I save super large array into many small files?
<p>In linux 64bit environment, I have very big float64 array (single one will be 500GB to 1TB). I would like to access these arrays in numpy with uniform way: a[x:y]. So I do not want to access the array as segments file by file. Is there any tools that I can create memmap over many different files? Can hdf5 or pytables store a single CArray into many small files? Maybe something similar to the fileInput? Or Can I do something with the file system to simulate a single file? </p>

<p>In matlab I've been using H5P.set_external to do this. Then I can create a raw dataset and access it as a big raw file. But I do not know if I can create numpy.ndarray over these dataset in python. Or can I spread a single dataset over many small hdf5 files?</p>

<p>and unfortunately the H5P.set_chunk does not work with H5P.set_external, because set_external only work with continuous data type not chunked data type.</p>

<p>some related topics:
<a href="http://stackoverflow.com/questions/35534294/chain-datasets-from-multiple-hdf5-files-datasets">Chain datasets from multiple HDF5 files/datasets</a></p>

<p>I would use hdf5. In h5py, you can specify a chunk size which makes retrieving small pieces of the array efficient:</p>

<p><a href="http://docs.h5py.org/en/latest/high/dataset.html?#chunked-storage" rel="nofollow">http://docs.h5py.org/en/latest/high/dataset.html?#chunked-storage</a></p>

python numpy filesystems hdf5 pytables