Speed up numpy summation between big arrays
<p>I have a code running operations on <code>numpy</code> arrays.
While linear algebra operations seem fast, I now am finding a bottleneck in a different issue: the summation of two distinct arrays. 
In the example below <code>WE3</code> and <code>T1</code> are two <code>1000X1000X1000</code> arrays.
First I calculate <code>WE3</code> using a numpy operation, then I sum those arrays.</p>

<pre><code>import numpy as np
import scipy as sp
import time
N = 100
n = 1000
X = np.random.uniform(size = (N,n))
wE = np.mean(X,0)
wE3 = np.einsum('i,j,k-&gt;ijk', wE, wE, wE) #22 secs
T1 = np.random.uniform(size = (n,n,n))
a = wE3 + T1 #115 secs
</code></pre>

<p>The calculation of <code>wE3</code> takes like 22 seconds, while the addition between <code>WE3</code> and <code>T1</code> takes 115 seconds.</p>

<p>Is there any known reason why the summation of those arrays is such slower than the calculation of WE3? They should have more or less the same complexity..</p>

<p>Is there a way to speed up that code? </p>

<p>You should really use Numba's Jit (just in time compiler) for this.  It is a purely numpy pipeline, which is perfect for Numba.</p>

<p>All you have to do is throw that above code into a function, and put an @jit decorater on top.  It gets speedups close to Cython. </p>

<p>However, as others have pointed out, it appears you're trying to work with data too large for your local machine, and numba would not solve your problems</p>

python arrays numpy