Python multiprocessing pool with shared data
<p>I'm attempting to speed up a multivariate fixed-point iteration algorithm using multiprocessing however, I'm running issues dealing with shared data. My solution vector is actually a named dictionary rather than a vector of numbers. Each element of the vector is actually computed using a different formula. At a high level, I have an algorithm like this:</p>

<pre><code>current_estimate = previous_estimate
while True:
for state in all_states:
    current_estimate[state] = state.getValue(previous_estimate)
if norm(current_estimate, previous_estimate) &lt; tolerance:
    break
else:
    previous_estimate, current_estimate = current_estimate, previous_estimate
</code></pre>

<p>I'm trying to parallelize the for-loop part with multiprocessing. The <code>previous_estimate</code> variable is read-only and each process only needs to write to one element of <code>current_estimate</code>. My current attempt at rewriting the for-loop is as follows:</p>

<pre><code># Class and function definitions
class A(object):
    def __init__(self,val):
        self.val = val

    # representative getValue function
    def getValue(self, est):
        return est[self] + self.val

def worker(state, in_est, out_est):
    out_est[state] = state.getValue(in_est)

def worker_star(a_b_c):
    """ Allow multiple arguments for a pool
        Taken from http://stackoverflow.com/a/5443941/3865495
    """
    return worker(*a_b_c)

# Initialize test environment
manager = Manager()
estimates = manager.dict()
all_states = []
for i in range(5):
     a = A(i)
     all_states.append(a)
     estimates[a] = 0

pool = Pool(process = 2)
prev_est = estimates
curr_est = estimates
pool.map(worker_star, itertools.izip(all_states, itertools.repeat(prev_est), itertools.repreat(curr_est)))
</code></pre>

<p>The issue I'm currently running into is that the elements added to the <code>all_states</code> array are not the same as those added to the <code>manager.dict()</code>. I keep getting <code>key value</code> errors when trying to access elements of the dictionary using elements of the array. And debugging, I found that none of the elements are the same.</p>

<pre><code>print map(id, estimates.keys())
&gt;&gt;&gt; [19558864, 19558928, 19558992, 19559056, 19559120]
print map(id, all_states)
&gt;&gt;&gt; [19416144, 19416208, 19416272, 19416336, 19416400]
</code></pre>

<p>This is happening because the objects you're putting into the <code>estimates</code> <code>DictProxy</code> aren't actually the same objects as those that live in the regular dict. The <code>manager.dict()</code> call returns a <code>DictProxy</code>, which is proxying access to a <code>dict</code> that actually lives in a completely separate manager process. When you insert things into it, they're really being copied and sent to a remote process, which means they're going to have a different identity.</p>

<p>To work around this, you can define your own <code>__eq__</code> and <code>__hash__</code> functions on <code>A</code>, as <a href="http://stackoverflow.com/questions/2909106/python-whats-a-correct-and-good-way-to-implement-hash">described in this question</a>:</p>

<pre><code>class A(object):
    def __init__(self,val):
        self.val = val

    # representative getValue function
    def getValue(self, est):
        return est[self] + self.val

    def __hash__(self):
        return hash(self.__key())

    def __key(self):
        return (self.val,)

    def __eq__(x, y):
        return x.__key() == y.__key()
</code></pre>

<p>This means the key look ups for items in the <code>estimates</code> will just use the value of the <code>val</code> attribute to establish identity and equality, rather than the <code>id</code> assigned by Python.</p>

python dictionary multiprocessing shared-memory