How to overcome MemoryError of numpy.unique
<p>I am using Numpy version 1.11.1 and have to deal with an two-dimensional array of</p>

<pre><code>my_arr.shape = (25000, 25000)
</code></pre>

<p>All values are integer, and I need a unique list of the arrays values. When using <code>lst = np.unique(my_arr)</code> I am getting:</p>

<pre><code>Traceback (most recent call last):
  File "&lt;pyshell#38&gt;", line 1, in &lt;module&gt;
    palette = np.unique(arr)
  File "c:\Python27\lib\site-packages\numpy\lib\arraysetops.py", line 176, in unique
    ar = np.asanyarray(ar).flatten()
MemoryError
</code></pre>

<p>My machine has only 8 GB RAM, but I tried it with another machine with 16 GB RAM, and the result is the same. Monitoring the memory and CPU usage doesn't show that the problems are related to RAM or CPU.</p>

<p>In principle, I know the values the array consists of, but what if the input changes... Also, if I want to replace values of the array by another (let's say all 2 by 0), will it need a lot of RAM as well?</p>

<p>Python 32-bit can't access more than 4 GiB RAM (often ~2.5 GiB). The obvious answer would be to use the 64-bit version. If that doesn't work, another solution would be to use <code>numpy.memmap</code> and memory-map the array into a file stored on disk.</p>

python arrays numpy