Python MemoryError when 'stacking' arrays
<p>I am writing code to add data along the length of a numpy array (for combining satellite data records). In order to do this my code reads two arrays and then uses the function </p>

<pre><code>def swath_stack(array1, array2):
    """Takes two arrays of swath data and compares their dimensions.
    The arrays should be equally sized in dimension 1. If they aren't the 
    function appends the smallest along dimension 0 until they are.
    The arrays are then stacked on top of each other."""
    if array1.shape[1] &gt; array2.shape[1]:
        no_lines = array1.shape[1] - array2.shape[1]
        a = np.zeros((array2.shape[0], no_lines))
        a.fill(-999.)
        new_array = np.hstack((array2, a))
        mask = np.zeros(new_array.shape, dtype=int)
        mask[np.where(new_array==-999.)] = 1
        array2 = ma.masked_array(new_array, mask=mask)
    elif array1.shape[1] &lt; array2.shape[1]:
        no_lines = array2.shape[1] - array1.shape[1]
        a = np.zeros((array1.shape[0], no_lines))
        a.fill(-999.)
        new_array = np.hstack((array1, a))
        mask = np.zeros(new_array.shape, dtype=int)
        mask[np.where(new_array==-999.)] = 1
        array1 = ma.masked_array(new_array, mask=mask)
    return np.vstack((array1, array2))
</code></pre>

<p>to make one array of the two in the line</p>

<pre><code>window_data = swath_stack(window_data, stack_data)
</code></pre>

<p>In the event that the arrays under consideration are equal in width the swath_stack() function reduces to np.vstack(). My problem is that I keep encountering <code>MemoryError</code>during this stage. I know that in the case of arithmetic operators it is more memory efficient to do the arithmetic in place (i.e. <code>array1 += array2</code> as opposed to <code>array1 = array1 + array2</code>) but I don't know how to avoid this kind of memory issue whilst using my swath_stack() function. </p>

<p>Can anyone please help?</p>

<p>I changed your last line to <code>np.ma.vstack</code>, and got</p>

<pre><code>In [474]: swath_stack(np.ones((3,4)),np.zeros((3,6)))
Out[474]: 
masked_array(data =
 [[1.0 1.0 1.0 1.0 -- --]
 [1.0 1.0 1.0 1.0 -- --]
 [1.0 1.0 1.0 1.0 -- --]
 [0.0 0.0 0.0 0.0 0.0 0.0]
 [0.0 0.0 0.0 0.0 0.0 0.0]
 [0.0 0.0 0.0 0.0 0.0 0.0]],
             mask =
 [[False False False False  True  True]
 [False False False False  True  True]
 [False False False False  True  True]
 [False False False False False False]
 [False False False False False False]
 [False False False False False False]],
       fill_value = 1e+20)
</code></pre>

<p>This preserves the masking that you created during the padding.</p>

<p>The masked padding doubles the memory use of the intermediate array.</p>

<p>Do you get memory errors when using 2 equal size arrays?  I.e. with just the plain <code>vstack</code>?  There's no way of doing in-place stacking.  It must create one or more new arrays.  Arrays have a fixed size, so can't grow in-place.  And the final array must have a contiguous data buffer, so can't use the buffers of the originals.</p>

<p>It won't use masking, but <code>np.pad</code> might make padding a bit easier.</p>

python arrays numpy memory satellite-image