Most efficient way to store list of integers
<p>I have recently been doing a project in which one of the aims is to use as little memory as possible to store a series of files using Python 3. Almost all of the files take up very little space, apart from one list of integers that is roughly <code>333,000</code> integers long and has integers up to about <code>8000</code> in size. </p>

<p>I'm currently using <code>pickle</code> to store the list, which takes up around <code>7mb</code>, but I feel like there must be a more memory efficient way to do this.</p>

<p>I have tried storing it as a text file and <code>csv</code>, bur both of these used in excess of <code>10mb</code> of space.</p>

<p>I like <a href="http://stackoverflow.com/a/39603768/1392132">Jim's suggestion</a> of using the <a href="https://docs.python.org/3/library/array.html" rel="nofollow"><code>array</code></a> module. If your numeric values are small enough to fit into the machine's native <code>int</code> type, then this is a fine solution. (I'd prefer to serialize the array with the <code>array.tofile</code> method instead of using <code>pickle</code>, though.) If an <code>int</code> is 32 bits, then this uses 4 bytes per number.</p>

<p>I would like to question how you did your text file, though. If I create a file with 333&#x202f;000 integers in the range [0, 8&#x202f;000] with one number per line,</p>

<pre><code>import random

with open('numbers.txt', 'w') as ostr:
    for i in range(333000):
        r = random.randint(0, 8000)
        print(r, file=ostr)
</code></pre>

<p>it comes out to a size of only 1.6&#x202f;MiB which isn't all that bad compared to the 1.3&#x202f;MiB that the binary representation would use. And if you do happen to have a value outside the range of the native <code>int</code> type one day, the text file will handle it happily without overflow.</p>

<p>Furthermore, if I <em>compress</em> the file using gzip, the file size shrinks down to 686&#x202f;KiB. That's better than gzipping the binary data! When using bzip2, the file size is only 562&#x202f;KiB. Python's standard library has support for both <a href="https://docs.python.org/3/library/gzip.html" rel="nofollow"><code>gzip</code></a> and <a href="https://docs.python.org/3/library/bz2.html" rel="nofollow"><code>bz2</code></a> so you might want to give the plain-text format plus compression another try.</p>

python list python-3.x memory integer