How to limit the size of pandas queries on HDF5 so it doesn't go over RAM limit?
<p>Let's say I have a pandas Dataframe</p>

<pre><code>import pandas as pd

df = pd.DataFrame()

df

   Column1    Column2
0  0.189086 -0.093137
1  0.621479  1.551653
2  1.631438 -1.635403
3  0.473935  1.941249
4  1.904851 -0.195161
5  0.236945 -0.288274
6 -0.473348  0.403882
7  0.953940  1.718043
8 -0.289416  0.790983
9 -0.884789 -1.584088
........
</code></pre>

<p>An example of a query is <code>df.query('Column1 &gt; Column2')</code></p>

<p>Let's say you wanted to limit the save of this query, so the object wasn't so large. Is there "pandas" way to accomplish this? </p>

<p>My question is primarily for querying at HDF5 object with pandas. An HDF5 object could be far larger than RAM, and therefore queries could be larger than RAM. </p>

<pre><code># file1.h5 contains only one field_table/key/HDF5 group called 'df'
store = pd.HDFStore('file1.h5')

# the following query could be too large 
df = store.select('df',columns=['column1', 'column2'], where=['column1==5'])
</code></pre>

<p>Is there a pandas/Pythonic way to stop users for executing queries that surpass a certain size? </p>

<p>Here is a small demonstration of how to use the <code>chunksize</code> parameter when calling <code>HDFStore.select()</code>:</p>

<pre><code>for chunk in store.select('df', columns=['column1', 'column2'],
                          where='column1==5', chunksize=10**6):
    # process `chunk` DF
</code></pre>

python pandas dataframe hdf5 pytables