Most efficient way to store list of integers
<p>I have recently been doing a project in which one of the aims is to use as little memory as possible to store a series of files using Python 3. Almost all of the files take up very little space, apart from one list of integers that is roughly <code>333,000</code> integers long and has integers up to about <code>8000</code> in size. </p>

<p>I'm currently using <code>pickle</code> to store the list, which takes up around <code>7mb</code>, but I feel like there must be a more memory efficient way to do this.</p>

<p>I have tried storing it as a text file and <code>csv</code>, bur both of these used in excess of <code>10mb</code> of space.</p>

<p>Here is a small demo, which uses Pandas module:</p>

<pre><code>import numpy as np
import pandas as pd
import feather

# let's generate an array of 1M int64 elements...
df = pd.DataFrame({'num_col':np.random.randint(0, 10**9, 10**6)}, dtype=np.int64)
df.info()

%timeit -n 1 -r 1 df.to_pickle('d:/temp/a.pickle')

%timeit -n 1 -r 1 df.to_hdf('d:/temp/a.h5', 'df_key', complib='blosc', complevel=5)
%timeit -n 1 -r 1 df.to_hdf('d:/temp/a_blosc.h5', 'df_key', complib='blosc', complevel=5)
%timeit -n 1 -r 1 df.to_hdf('d:/temp/a_zlib.h5', 'df_key', complib='zlib', complevel=5)
%timeit -n 1 -r 1 df.to_hdf('d:/temp/a_bzip2.h5', 'df_key', complib='bzip2', complevel=5)
%timeit -n 1 -r 1 df.to_hdf('d:/temp/a_lzo.h5', 'df_key', complib='lzo', complevel=5)

%timeit -n 1 -r 1 feather.write_dataframe(df, 'd:/temp/a.feather')
</code></pre>

<p><strong>DataFrame info:</strong></p>

<pre><code>In [56]: df.info()
&lt;class 'pandas.core.frame.DataFrame'&gt;
RangeIndex: 1000000 entries, 0 to 999999
Data columns (total 1 columns):
num_col    1000000 non-null int64
dtypes: int64(1)
memory usage: 7.6 MB
</code></pre>

<p><strong>Results (speed):</strong></p>

<pre><code>In [49]: %timeit -n 1 -r 1 df.to_pickle('d:/temp/a.pickle')
1 loop, best of 1: 16.2 ms per loop

In [50]: %timeit -n 1 -r 1 df.to_hdf('d:/temp/a.h5', 'df_key', complib='blosc', complevel=5)
1 loop, best of 1: 39.7 ms per loop

In [51]: %timeit -n 1 -r 1 df.to_hdf('d:/temp/a_blosc.h5', 'df_key', complib='blosc', complevel=5)
1 loop, best of 1: 40.6 ms per loop

In [52]: %timeit -n 1 -r 1 df.to_hdf('d:/temp/a_zlib.h5', 'df_key', complib='zlib', complevel=5)
1 loop, best of 1: 213 ms per loop

In [53]: %timeit -n 1 -r 1 df.to_hdf('d:/temp/a_bzip2.h5', 'df_key', complib='bzip2', complevel=5)
1 loop, best of 1: 1.09 s per loop

In [54]: %timeit -n 1 -r 1 df.to_hdf('d:/temp/a_lzo.h5', 'df_key', complib='lzo', complevel=5)
1 loop, best of 1: 32.1 ms per loop

In [55]: %timeit -n 1 -r 1 feather.write_dataframe(df, 'd:/temp/a.feather')
1 loop, best of 1: 3.49 ms per loop
</code></pre>

<p><strong>Results (size):</strong></p>

<pre><code>{ temp }  Â» ls -lh a*                                                                                         /d/temp
-rw-r--r-- 1 Max None 7.7M Sep 20 23:15 a.feather
-rw-r--r-- 1 Max None 4.1M Sep 20 23:15 a.h5
-rw-r--r-- 1 Max None 7.7M Sep 20 23:15 a.pickle
-rw-r--r-- 1 Max None 4.1M Sep 20 23:15 a_blosc.h5
-rw-r--r-- 1 Max None 4.0M Sep 20 23:15 a_bzip2.h5
-rw-r--r-- 1 Max None 4.1M Sep 20 23:15 a_lzo.h5
-rw-r--r-- 1 Max None 3.9M Sep 20 23:15 a_zlib.h5
</code></pre>

<p><strong>Conclusion:</strong> pay attention at HDF5 (+ <code>blosc</code> or <code>lzo</code> compression) if you need both speed and a reasonable size or at <a href="https://blog.rstudio.org/2016/03/29/feather/" rel="nofollow">Feather-format</a> if you only care of speed - it's 4 times faster compared to Pickle!</p>

python list python-3.x memory integer