Memory error in python with numpy.pad function
<p>I read a csv file in python and create an  4664605 x 4 array. I want a matrix. So, I use the numpy.pad (wit constant value  = 0) function in order to create 4664605 x 4664605 matrix. But I have the following error messaage:</p>

<blockquote>
  <p>Traceback (most recent call last):   File
  "C:\Users\Angelika\Eclipse\Projects\vonNeumann\vonNeumann.py", line 7,
  in 
      A_new = np.pad(A, ((0,0),(0,4664601)), 'constant',constant_values=(0))   File
  "C:\Anaconda\lib\site-packages\numpy\lib\arraypad.py", line 1394, in
  pad
      newmat = _append_const(newmat, pad_after, after_val, axis)   File "C:\Anaconda\lib\site-packages\numpy\lib\arraypad.py", line 138, in
  _append_const
      return np.concatenate((arr, np.zeros(padshape, dtype=arr.dtype)), MemoryError</p>
</blockquote>

<p>I have checked the maximum size of my system in case of overflowing but it is ok. More specifically, sys.maxsize = 9223372036854775807 and matrix size = 21758539806025.
The issue is that when append rows everything is ok. That is, the result is a 9329210 x 4 array. But I can't add 4664601 columns in order to have a matrix.
I don't know what to do.</p>

<p>Thank you very much,
Angelika</p>

<p>This is more of a question than an answer.  But it's too long for comment lines.</p>

<p>The distinction between a <code>4664605 x 4 array</code> and <code>4664605 x 4664605 matrix</code> doesn't make much sense.  Squareness does not define a <code>matrix</code>, at least not in most contexts.</p>

<p>What's the purpose of adding many 0 filled columns to this array?  Even if you had memory to create one that big, would you have enough memory to hold several copies (as needed for math and many other operations)?</p>

<p>The error line:</p>

<p>return np.concatenate((arr, np.zeros(padshape, dtype=arr.dtype))</p>

<p><code>arr</code> must be (4664605,4) in shape, and <code>padshape</code> <code>(4664605, 466401)</code>. So it is trying make an <code>zeros</code> array of <code>padshape</code> size, and then make a new array of the final size.  So simply constructing this requires space for 2 very larger arrays.</p>

<p>You might save a bit of space by doing the <code>pad</code> directly</p>

<pre><code>res = np.zeros((4664605, 4664605), dtype=arg.dtype)
res[:,:4] = arr
</code></pre>

<p>But still - why make such a big array that is mostly zero?</p>

python arrays numpy out-of-memory anaconda