Dictionary multiprocessing
<p>I want to parallelize the processing of a dictionary using the multiprocessing library.</p>

<p>My problem can be reduced to this code:</p>

<pre><code>from multiprocessing import Manager,Pool

def modify_dictionary(dictionary):
    if((3,3) not in dictionary):
        dictionary[(3,3)]=0.
    for i in range(100):
        dictionary[(3,3)] = dictionary[(3,3)]+1
    return 0

if __name__ == "__main__":

    manager = Manager()

    dictionary = manager.dict(lock=True)
    jobargs = [(dictionary) for i in range(5)]

    p = Pool(5)
    t = p.map(modify_dictionary,jobargs)
    p.close()
    p.join()

    print dictionary[(3,3)]
</code></pre>

<p>I create a pool of 5 workers, and each worker should increment dictionary[(3,3)] 100 times. So, if the locking process works correctly, I expect dictionary[(3,3)] to be 500 at the end of the script. </p>

<p>However; something in my code must be wrong, because this is not what I get: the locking process does not seem to be "activated"  and dictionary[(3,3)] always have a valuer &lt;500 at the end of the script.</p>

<p>Could you help me?</p>

<p>The problem is with this line:</p>

<pre><code>dictionary[(3,3)] = dictionary[(3,3)]+1
</code></pre>

<p>Three things happen on that line:</p>

<ul>
<li>Read the value of the dictionary key (3,3)</li>
<li>Increment the value by 1</li>
<li>Write the value back again</li>
</ul>

<p>But the increment part is happening outside of any locking.</p>

<p>The whole sequence must be atomic, and must be synchronized across all processes. Otherwise the processes will interleave giving you a lower than expected total. </p>

<p>Holding a lock whist incrementing the value ensures that you get the total of 500 you expect:</p>

<pre><code>from multiprocessing import Manager,Pool,Lock

lock = Lock()

def modify_array(dictionary):
    if((3,3) not in dictionary):
        dictionary[(3,3)]=0.
    for i in range(100):
        with lock:
            dictionary[(3,3)] = dictionary[(3,3)]+1
    return 0

if __name__ == "__main__":

    manager = Manager()

    dictionary = manager.dict(lock=True)
    jobargs = [(dictionary) for i in range(5)]

    p = Pool(5)
    t = p.map(modify_array,jobargs)
    p.close()
    p.join()

    print dictionary[(3,3)]
</code></pre>

python dictionary python-multiprocessing