Medium Pandas Panel to hdf5
<p>I'm struggling with stupid problem; until now I was working with pandas panel that fits in to the memory. Now I've to scale up the process and I run out of the memory.
The solution proposed on the panda's documentation is to save everything on a hdf5 file. Well it works if your panel is smaller than the memory... otherwise it's impossible.
Here the code</p>

<pre><code>import pandas as pd

store = pd.HDFStore('test.h5')

yrs = list(range(1999, 2014))
xsize = 15680
ysize = 40320

slPanel = pd.Panel(np.nan, items=yrs, major_axis=list(range(0, xsize)), minor_axis=list(range(0, ysize)))
store['SP'] = slPanel
</code></pre>

<p>Have you any ideas how to workaround this problem?<br>
By the way I'm using python 3.5 x64 
Thank you </p>

<p>I <em>think</em> this should work</p>

<p>To write to the file:</p>

<pre><code>store = pd.HDFStore(âtest.h5â, âwâ, append = True)
for partofpanel in panelparts:
  store.append('SP', partofpanel, data_columns = True, index = True)
</code></pre>

<p>To read from file:</p>

<pre><code>x = pd.HDFStore(âtest.h5â, ârâ)
for df in x.select(âSPâ, chunksize = 10):
   #Do some stuff
</code></pre>

<p>Separately, <a href="http://dask.pydata.org/en/latest/" rel="nofollow" title="dask">dask</a> may be your friend.</p>

python pandas panel hdf5