Multiprocessing python function for numerical calculations
<p>Hoping to get some help here with parallelising my python code, I've been struggling with it for a while and come up with several errors in whichever way I try, currently running the code will take about 2-3 hours to complete, The code is given below; </p>

<pre><code>import numpy as np
from scipy.constants import Boltzmann, elementary_charge as kb, e
import multiprocessing
from functools import partial 
Tc = 9.2
x = []
g= []
def Delta(T):
'''
Delta(T) takes a temperature as an input and calculates a
temperature dependent variable based on Tc which is defined as a
global parameter  
'''
    d0 = (pi/1.78)*kb*Tc
    D0 = d0*(np.sqrt(1-(T**2/Tc**2)))
    return D0

def element_in_sum(T, n, phi):
    D = Delta(T)
    matsubara_frequency = (np.pi * kb * T) * (2*n + 1)
    factor_d = np.sqrt((D**2 * cos(phi/2)**2) + matsubara_frequency**2)
    element = ((2 * D * np.cos(phi/2))/ factor_d) * np.arctan((D * np.sin(phi/2))/factor_d)
    return element

def sum_elements(T, M, phi):
'''
  sum_elements(T,M,phi) is the most computationally heavy part
  of the calculations, the larger the M value the more accurate the 
  results are.
  T: temperature
  M: number of steps for matrix calculation the larger the more accurate the calculation
 phi: The phase of the system can be between 0- pi 
'''
    X = list(np.arange(0,M,1))
    Y = [element_in_sum(T, n, phi) for n in X]
    return sum(Y)

def KO_1(M, T, phi):
    Iko1Rn = (2 * np.pi * kb * T /e) * sum_elements(T, M, phi)
    return Iko1Rn

def main():
    for j in range(1, 92):
        T = 0.1*j
        for i in range(1, 314):
            phi = 0.01*i
            pool = multiprocessing.Pool()
            result = pool.apply_async(KO_1,args=(26000, T, phi,))
            g.append(result)
            pool.close()
            pool.join()    
     A = max(g);
     x.append(A)
     del g[:]       
</code></pre>

<p>My approach was to try and send the KO1 function into a multiprocessing pool but I either get a <code>Pickling</code> error or a <code>too many files open</code>, Any help is greatly appreciated, and if multiprocessing is the wrong approach I would love any guide.</p>

<p>This is not an answer to the question, but if I may, I would propose how to speed up the code using simple numpy array operations. Have a look at the following code:</p>

<pre><code>import numpy as np
from scipy.constants import Boltzmann, elementary_charge as kb, e
import time
Tc = 9.2
RAM = 4*1024**2 # 4GB

def Delta(T):
    '''
    Delta(T) takes a temperature as an input and calculates a
    temperature dependent variable based on Tc which is defined as a
    global parameter  
    '''
    d0 = (np.pi/1.78)*kb*Tc
    D0 = d0*(np.sqrt(1-(T**2/Tc**2)))
    return D0

def element_in_sum(T, n, phi):
    D = Delta(T)
    matsubara_frequency = (np.pi * kb * T) * (2*n + 1)
    factor_d = np.sqrt((D**2 * np.cos(phi/2)**2) + matsubara_frequency**2)
    element = ((2 * D * np.cos(phi/2))/ factor_d) * np.arctan((D * np.sin(phi/2))/factor_d)
    return element


def KO_1(M, T, phi):
    X = np.arange(M)[:,np.newaxis,np.newaxis]
    sizeX = int((float(RAM) / sum(T.shape))/sum(phi.shape)/8) #8byte
    i0 = 0
    Iko1Rn = 0. * T * phi
    while (i0+sizeX) &lt;= M:
        print "X = %i"%i0
        indices = slice(i0, i0+sizeX)
        Iko1Rn += (2 * np.pi * kb * T /e) * element_in_sum(T, X[indices], phi).sum(0)
        i0 += sizeX
    return Iko1Rn

def main():
    T = np.arange(0.1,9.2,0.1)[:,np.newaxis]
    phi = np.linspace(0,np.pi, 361)
    M = 26000
    result = KO_1(M, T, phi)
    return result, result.max()

T0 = time.time()
r, rmax = main()
print time.time() - T0
</code></pre>

<p>It runs a bit more than 20sec on my PC. One has to be careful not to use too much memory, that is why there is still a loop with a bit complicated construction to use only pieces of X. If enough memory is present, then it is not necessary.</p>

<p>One should also note that this is just the first step of speeding up. Much improvement could be reached still using e.g. just in time compilation or cython.</p>

python multithreading numpy multiprocessing