Python Multiprocessing, Pool map - cancel all running processes if one, returns the desired result
<p>given the following Python code:</p>

<pre><code>import multiprocessing

def unique(somelist):
    return len(set(somelist)) == len(somelist)


if __name__ == '__main__':
    somelist = [[1,2,3,4,5,6,7,8,9,10,11,12,13,2], [1,2,3,4,5], [1,2,3,4,5,6,7,8,9,1], [0,1,5,1]]

    pool = multiprocessing.Pool()
    reslist = pool.map(unique, somelist)
    pool.close()
    pool.join()
    print "Done!"

    print reslist
</code></pre>

<p>Now imagine, that the lists with integers in this toy example are extremely long, and what I'd like to achieve here is the following: if one of the lists in somelist returns True, kill all running processes. </p>

<p>This leads to two questions (and probably more which I haven't come up with):</p>

<ul>
<li><p>How can I 'read'/'listen' from a finished process the result, while other processes are running? If e.g. a process is dealing with [1,2,3,4,5] from somelist, and is finished before all other processes, how can I read out the result from that process in this very moment?</p></li>
<li><p>Given the case that it is possible to 'read' out the result of a finished process while other are running: how can I use this result as a condition to terminate all other running processes?</p></li>
</ul>

<p>e.g. if one process has finished and returned True, how I can use this as a condition to terminate all other (still) running processes?</p>

<p>Thank you in advance for any hints
Dan</p>

<p>Without fancy IPC (inter-process communication) tricks, easiest is to use a <code>Pool</code> method with a callback function instead.  The callback runs in the main program (in a thread created by <code>multiprocessing</code>), and consumes each result as it becomes available.  When the callback sees a result you like, it can terminate the <code>Pool</code>.  For example,</p>

<pre><code>import multiprocessing as mp

def worker(i):
    from time import sleep
    sleep(i)
    return i, (i == 5)

def callback(t):
    i, quit = t
    result[i] = quit
    if quit:
        pool.terminate()

if __name__ == "__main__":
    N = 50
    pool = mp.Pool()
    result = [None] * N
    for i in range(N):
        pool.apply_async(func=worker, args=(i,), callback=callback)
    pool.close()
    pool.join()
    print(result)
</code></pre>

<p>Which will almost certainly display the following (OS scheduling vagaries <em>may</em> allow another input or two to be consumed):</p>

<pre><code>[False, False, False, False, False, True, None, None, None, None,
 None, None, None, None, None, None, None, None, None, None,
 None, None, None, None, None, None, None, None, None, None,
 None, None, None, None, None, None, None, None, None, None,
 None, None, None, None, None, None, None, None, None, None]
</code></pre>

python dictionary multiprocessing pool