PIL OverflowError on loading PIL.Image.fromArray
<p>I am trying to store large images using pillow 3.3.1 on python 3.4. These images tend to be in the range from 1 to 4 GB, as uint8 RGB pixels. Linux and OSX give me the same result.</p>

<pre><code>from PIL import Image
import numpy as np

imgArray = np.random.randint(255, size=(39000, 35000, 3)).astype(np.uint8)
print("buffer size:", imgArray.size)
print("image max bytes:", 2**32)
pilImage = Image.fromarray(imgArray)
</code></pre>

<p>I get the following output</p>

<pre><code>buffer size: 4095000000
image max bytes: 4294967296

Traceback (most recent call last):
  File "storeLargeImage.py", line 6, in &lt;module&gt;
    pilImage = Image.fromarray(imgArray)
  File "/home/mpesavento/miniconda3/lib/python3.4/site-packages/PIL/Image.py", line 2189, in fromarray
    return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
  File "/home/mpesavento/miniconda3/lib/python3.4/site-packages/PIL/Image.py", line 2139, in frombuffer
    return frombytes(mode, size, data, decoder_name, args)
  File "/home/mpesavento/miniconda3/lib/python3.4/site-packages/PIL/Image.py", line 2074, in frombytes
    im.frombytes(data, decoder_name, args)
  File "/home/mpesavento/miniconda3/lib/python3.4/site-packages/PIL/Image.py", line 736, in frombytes
    s = d.decode(data)
OverflowError: size does not fit in an int
</code></pre>

<p>The buffer is smaller than the max I think PIL uses in python 3, which I thought used a uint32 for the buffer length. PIL in python 2 uses an int32, making the max 2**31-1.</p>

<p>This bug comes up before we determine the codec for storing. For example, I would like to either store lossless png or tif, via<br>
<code>pilImage.save(BytesIO(), format="png")</code><br>
or<br>
<code>pilImage.save(BytesIO(), format="tiff")</code>  </p>

<p>How can one go about saving an image larger than 2 GB (2147483647 bytes)?</p>

<p>Edit:
It looks like it should have been <a href="https://github.com/python-pillow/Pillow/issues/436" rel="nofollow">fixed a while ago</a>. Not sure why the problem still shows up.</p>

<p>I realize you asked about PIL, but you could try <a href="http://www.vips.ecs.soton.ac.uk/index.php?title=VIPS" rel="nofollow">libvips</a>. It specializes in large images (images larger than your available RAM) and should have no problems with your 4gb files. There are some <a href="http://www.vips.ecs.soton.ac.uk/index.php?title=Speed_and_Memory_Use#Results" rel="nofollow">speed and memory use benchmarks on the vips website</a>.</p>

<p>For example, I just did:</p>

<pre><code>$ python
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; from gi.repository import Vips
&gt;&gt;&gt; x = Vips.Image.perlin(100000, 100000, uchar = True)
&gt;&gt;&gt; x.write_to_file("x.tif", bigtiff = True)
&gt;&gt;&gt;
$ ls -l x.tif
-rw-rw-r-- 1 john john 10000012844 Oct  7 11:43 x.tif
</code></pre>

<p>Makes an 10gb, 8 bit, 100,000 x 100,000 pixel image of Perlin noise. It takes a little while, of course --- the final write is about three minutes on my laptop and needs 100MB of RAM. The Python binding is documented <a href="http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/using-from-python.html" rel="nofollow">here</a>. </p>

<p>You can <a href="http://stackoverflow.com/questions/31632265/extract-vips-image-and-save-it-to-numpy-array-same-as-with-pil">move images between numpy and vips</a>, though it's done by allocating and copying huge strings, sadly, so you need plenty of memory for that. You might be able to do what you want with just the vips operations, it depends on your needs. </p>

python pillow