weird behavior of multiprocessing scipy optimization inside of a function
<p>Here is a simple code that runs well. Even if the function minimize wraps the scipy.optimize.minimize it does not complain about pickling
</p>

<pre><code>import numpy as np
from scipy import optimize
from multiprocessing import Pool

def square(x):
    return np.sum(x**2+ 2*x)

def minimize(args):
    f,x = args
    res = optimize.minimize(f, x, method = 'L-BFGS-B')
    return res.x

x = np.random.rand(8,10)

args = [(square,x[i]) for i in range(8)]
p = Pool(8)
p.map(minimize,args)
</code></pre>

<p>However, if try the following it fails with the pickling error</p>

<p><code>PicklingError: Can't pickle &lt;type 'function'&gt;: attribute lookup __builtin__.function failed</code></p>

<pre class="lang-py prettyprint-override"><code>def run():
    def square(x):
        return np.sum(x**2+ 2*x)

    def minimize(args):
        f,x = args
        res = optimize.minimize(f, x, method = 'L-BFGS-B')
        return res.x

    x = np.random.rand(8,10)

    args = [(square,x[i]) for i in range(8)]
    p = Pool(8)
    p.map(minimize,args)

run()
</code></pre>

<p>I want to make a module to use scipy minimize in parallel with many population of initial guesses. However, as shown in the example, when I make it a module, it fails.</p>

<p>The problem is that Python cannot pickle nested functions, and in your second example, you're trying to pass the nested <code>minimize</code> and <code>square</code> functions to your child process, which requires pickling. </p>

<p>If there's no reason that you must nest those two functions, moving them to the top-level of the module will fix the issue. You can also see <a href="http://stackoverflow.com/questions/12019961/python-pickling-nested-functions">this question</a> for techniques to pickle nested functions.</p>

python optimization parallel-processing scipy multiprocessing