Python: How to use Value and Array in Multiprocessing pool
<p>For <code>multiprocessing</code> with <code>Process</code>, I can use <code>Value, Array</code> by setting <code>args</code> param.  </p>

<p>With <code>multiprocessing</code> with <code>Pool</code>, how can I use <code>Value, Array.</code> There is nothing in the docs on how to do this. </p>

<pre><code>from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])
</code></pre>

<p>I am trying to use <code>Value, Array</code> within the code snippet below. </p>

<pre><code>import multiprocessing


def do_calc(data):
    #  access num or 
    #  work to update arr
    newdata =data * 2
    return newdata

def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':
    num             = Value('d', 0.0)
    arr             = Array('i', range(10))  
    inputs          = list(range(10))
    print 'Input   :', inputs

    pool_size       = multiprocessing.cpu_count() * 2
    pool            = multiprocessing.Pool(processes=pool_size,initializer=start_process, )
    pool_outputs    = pool.map(do_calc, inputs)
    pool.close() # no more tasks
    pool.join()  # wrap up current tasks

    print 'Pool    :', pool_outputs
</code></pre>

<p>I never knew "the reason" for this, but <code>multiprocessing</code> (<code>mp</code>) uses different pickler/unpickler mechanisms for functions passed to most <code>Pool</code> methods.  It's a consequence that objects created by things like <code>mp.Value</code>, <code>mp.Array</code>, <code>mp.Lock</code>, ..., can't be passed as arguments to such methods, although they <em>can</em> be passed as arguments to <code>mp.Process</code> <em>and</em> to the optional <code>initializer</code> function of <code>mp.Pool()</code>.  Because of the latter, this works:</p>

<pre><code>import multiprocessing as mp

def init(aa, vv):
    global a, v
    a = aa
    v = vv

def worker(i):
    a[i] = v.value * i

if __name__ == "__main__":
    N = 10
    a = mp.Array('i', [0]*N)
    v = mp.Value('i', 3)
    p = mp.Pool(initializer=init, initargs=(a, v))
    p.map(worker, range(N))
    print(a[:])
</code></pre>

<p>and that prints</p>

<pre><code>[0, 3, 6, 9, 12, 15, 18, 21, 24, 27]
</code></pre>

<p>That's the only way I know of to get this to work across platforms.</p>

<p>On Linux-y platforms (where <code>mp</code> creates new processes via <code>fork()</code>), you can instead create your <code>mp.Array</code> and <code>mp.Value</code> (etc) objects as module globals any time <em>before</em> you do <code>mp.Pool()</code>.  Processes created by <code>fork()</code> inherit whatever is in the module global address space at the time <code>mp.Pool()</code> executes.</p>

<p>But that doesn't work at all on platforms (read "Windows") that don't support <code>fork()</code>.</p>

python arrays multiprocessing pool