How to pass keywords list to pathos.multiprocessing?
<p>I am using pathos.multiprocessing to parallelize a program that requires using instance methods. Here is a minimum working example:</p>

<pre><code>import time
import numpy as np
from pathos.multiprocessing import Pool, ProcessingPool, ThreadingPool

class dummy(object):
    def __init__(self, arg, key1=None, key2=-11):

        np.random.seed(arg)

        randnum = np.random.randint(0, 5)

        print 'Sleeping {} seconds'.format(randnum)
        time.sleep(randnum)

        self.value = arg
        self.more1 = key1
        self.more2 = key2

args = [0, 10, 20, 33, 82] 
keys = ['key1', 'key2']
k1val = ['car', 'borg', 'syria', 'aurora', 'libera']
k2val = ['a', 'b', 'c', 'd', 'e']
allks = [dict(zip(keys, [k1val[i], k2val[i]])) for i in range(5)]

pool = ThreadingPool(4)
result = pool.map(dummy, args, k1val, k2val)

print [[r.value, r.more1, r.more2] for r in result]
</code></pre>

<p>The result printed is (as expected):</p>

<pre><code>Sleeping 4 seconds
Sleeping 1 seconds
Sleeping 3 seconds
Sleeping 4 seconds
Sleeping 3 seconds
[[0, 'car', 'a'], [10, 'borg', 'b'], [20, 'syria', 'c'], [33, 'aurora', 'd'], [82, 'libera', 'e']]
</code></pre>

<p>However, in this call to <code>map</code> the order of the last two arguments matters, and if I do:</p>

<pre><code>result2 = pool.map(dummy, args, k2val, k1val)
</code></pre>

<p>I obtain:</p>

<pre><code>[[0, 'a', 'car'], [10, 'b', 'borg'], [20, 'c', 'syria'], [33, 'd', 'aurora'], [82, 'e', 'libera']]
</code></pre>

<p>whereas I would like to obtain the same as the first result. The behaviour would be the same as what <code>apply_async</code> <code>kwds</code> can do in the standard module <code>multiprocessing</code>, i.e. pass a list of dictionaries, where in each dictionary the keys are the keyword names and the items are the keyword arguments (see <code>allks</code>). Notice that the standard module <code>multiprocessing</code> cannot use instance methods, and therefore does not meet even the minimum requirements.</p>

<p>Tentatively this would be:
    result = pool.map(dummy, args, kwds=allks) # This does not work</p>

<p>I'm the <code>pathos</code> author.  Yeah, you hit on something that I have known needs a little work.  Currently, the <code>map</code> and <code>pipe</code> (i.e. <code>apply</code>) methods from <code>ProcessPool</code>, <code>ThreadPool</code>, and <code>ParallelPool</code> cannot take <code>kwds</code> -- you have to pass them as <code>args</code>. However, if you use <code>_ProcessPool</code> or <code>_ThreadPool</code>, then you can pass <code>kwds</code> to their <code>map</code> and <code>apply</code> methods.
The pools in <code>pathos.pools</code> that start with an underscore actually come directly from <code>multiprocess</code>, so they have identical APIs to those in <code>multiprocessing</code> (but with better serialization, so can pass class methods etc).</p>

<pre><code>&gt;&gt;&gt; from pathos.pools import _ProcessPool
&gt;&gt;&gt; from multiprocess.pool import Pool
&gt;&gt;&gt; Pool is _ProcessPool
True
</code></pre>

<p>So, for the edits to the original code would look something like this (<strong>from the OP's suggested edit</strong>):</p>

<pre><code>&gt;&gt;&gt; from pathos.pools import _ThreadPool
&gt;&gt;&gt; pool = _ThreadPool(4)
&gt;&gt;&gt; 
[â¦]
&gt;&gt;&gt; result = []
&gt;&gt;&gt; def callback(x):
&gt;&gt;&gt;    result.append(x)
&gt;&gt;&gt;
&gt;&gt;&gt; for a, k in zip(args, allks):
&gt;&gt;&gt;     pool.apply_async(dummy, args=(a,), kwds=k, callback=callback)
&gt;&gt;&gt; 
&gt;&gt;&gt; pool.close()
&gt;&gt;&gt; pool.join()
</code></pre>

python dictionary multiprocessing keyword-argument pathos