Append 1D numpy array to file with new element in new line
<p>I have a number of numpy arrays which I generate iteratively. I want to save each array to a file. I then generate the next array and append it to the file and so forth (if I did it in one go I would use too much memory). How do I best do that? Is there a way of making us of numpy functions such as e.g. <code>numpy.savetxt</code>? (I couldn't find an append option for that function.)</p>

<p>My current code is:</p>

<pre><code>with open('paths.dat','w') as output:
    for i in range(len(hist[0])):
        amount = hist[0][i].astype(int)
        array = hist[1][i] * np.ones(amount)
        for value in array:
            output.write(str(value)+'\n')
</code></pre>

<p>I would recommend using HDF5.  They are very fast for IO.
Here is how you write your data:</p>

<pre><code>import numpy as np
import tables

fname = 'myOutput.h5'
length = 100  # your data length
my_data_generator = xrange(length) # Your data comes here instead of the xrange

filters = tables.Filters(complib='blosc', complevel=5)  # you could change these
h5file = tables.open_file(fname, mode='w', title='yourTitle', filters=filters)
group = h5file.create_group(h5file.root, 'MyData', 'MyData')
x_atom = tables.Float32Atom()

x = h5file.create_carray(group, 'X', atom=x_atom, title='myTitle',
                         shape=(length,), filters=filters)

# this is a basic example.  It will be faster if you write it in larger chunks in your real code
# like x[start1:end1] = elements[start2:end2]
for element_i, element in enumerate(my_data_generator):
    x[element_i] = element
    h5file.flush()

h5file.close()
</code></pre>

<p>For reading it use:</p>

<pre><code>h5file = tables.open_file(fname, mode='r')
x = h5file.get_node('/MyData/X')
print x[:10]
</code></pre>

<p>The result:</p>

<pre><code>marray([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.], dtype=float32)
</code></pre>

python arrays numpy file-io