How should I read/write a data structure containing large arrays?
<p>I fetch a big array of data from a server. I store it in a combination of a dictionary and multi-dimensional array, and it will be used for a simple plot. It looks like:</p>

<pre><code>&gt;&gt; print(data)
{'intensity_b2': [array([  1.46562588e+09,   1.46562588e+09,   1.46562588e+09, ...,
     1.46566369e+09,   1.46566369e+09,   1.46566369e+09]), array([ 0.,  0.,  0., ...,  0.,  0.,  0.])]}
&gt;&gt; print(len(data['intensity_b2'][0]))
37071
</code></pre>

<p>To avoid fetching the data every time I run the script I want to save this data structure to a file. I try to store the data as</p>

<pre><code>with open("data.dat", 'w') as f:
    f.write(str(data))
</code></pre>

<p>and read it with</p>

<pre><code>with open(data_store, 'r') as f:
    data = ast.literal_eval(f.read())
</code></pre>

<p>as suggested <a href="http://stackoverflow.com/questions/11026959/python-writing-dict-to-txt-file-and-reading-dict-from-txt-file">here</a>. However, I get an error</p>

<blockquote>
  <p>ValueError: malformed node or string: &lt;_ast.Call object at 0x108fce5f8></p>
</blockquote>

<p>which I suspect is due to the fact that the data gets stored with the <code>...</code> as was shown in the first printout (i.e. the first <code>print(data)</code> above is literally how the data looks in the file). How do I write a dictionary with a big array to a file and read it subsequently?</p>

<p>You can use <code>pickle</code> to handle serialization properly:</p>

<pre><code>In [23]: a
Out[23]: 
{'intensity_b2': [array('f', [1465625856.0, 1465625856.0, 1465625856.0]),
  array('f', [1465663744.0, 1465663744.0, 1465663744.0])]}

In [24]: pickle.dump(a, open('foo.p', 'wb'))

In [25]: aa = pickle.load(open('foo.p', 'rb'))

In [26]: aa
Out[26]: 
{'intensity_b2': [array('f', [1465625856.0, 1465625856.0, 1465625856.0]),
  array('f', [1465663744.0, 1465663744.0, 1465663744.0])]}
</code></pre>

<p>This does exactly what you want to do: saves your data structure to a file, and then reads it from the file.</p>

<p>However, it looks like you're reinventing the wheel here. You may want to have a look at <code>numpy</code> and <code>pandas</code>.</p>

python arrays dictionary file-io