Exit a multiprocessing script
<p>I am trying to exit a multiprocessing script when an error is thrown by the target function, but instead of quitting, the parent process just hangs.</p>

<p>This is the test script I use to replicate the problem:</p>

<pre><code>#!/usr/bin/python3.5

import time, multiprocessing as mp

def myWait(wait, resultQueue):
    startedAt = time.strftime("%H:%M:%S", time.localtime())
    time.sleep(wait)
    endedAt = time.strftime("%H:%M:%S", time.localtime())
    name = mp.current_process().name
    resultQueue.put((name, wait, startedAt, endedAt))

# queue initialisation
resultQueue = mp.Queue()

# process creation arg: (process number, sleep time, queue)
proc =  [
    mp.Process(target=myWait, name = ' _One_', args=(2, resultQueue,)),
    mp.Process(target=myWait, name = ' _Two_', args=(2, resultQueue,))
    ]

# starting processes
for p in proc:
    p.start()

for p in proc:
    p.join()

# print results
results = {}
for p in proc:
    name, wait, startedAt, endedAt = resultQueue.get()
    print('Process %s started at %s wait %s ended at %s' % (name, startedAt, wait, endedAt))
</code></pre>

<p>This works perfectly, I can see the parent script spawning two child processes in <code>htop</code> but when I want to force the parent script to exit if an error is thrown in the <code>myWait</code> target function the parent process just hangs and doesn't even spawn any child process. I have to <code>ctrl-c</code> to kill it.</p>

<pre><code>def myWait(wait, resultQueue):
    try:
        # do something wrong
    except:
        raise SystemExit
</code></pre>

<p>I have tried every way to exit the function (e.g. <code>exit()</code>, <code>sys.exit()</code>, <code>os._exit()</code>...) to no avail.</p>

<p>You should use <code>multiprocessing.Pool</code> to manage your processes for you. And then use <code>Pool.imap_unordered</code> to iterate over the results in the order they are completed. As soon as you get the first exception, you can stop the pool and its child processes (this is automatically done when you exit the <code>with Pool() as pool</code> block). eg</p>

<pre><code>from multiprocessing import Pool
import time

def my_wait(args):
    name, wait = args
    if wait == 2:
        raise ValueError("error!")
    else:
        startedAt = time.strftime("%H:%M:%S", time.localtime())
        time.sleep(wait)
        endedAt = time.strftime("%H:%M:%S", time.localtime())
        return name, wait, startedAt, endedAt

if __name__ == "__main__":
    try:
        with Pool() as pool:
            args = [["_One_", 2], ["_Two_", 3]]
            for name, wait, startedAt, endedAt in pool.imap_unordered(my_wait, args):     
                print('Task %s started at %s wait %s ended at %s' % (name,
                    startedAt, wait, endedAt))
    except ValueError as e:
        print(e)
</code></pre>

<p>This method is not suitable for long-timed, low-workload tasks, as it will only run as many of the tasks in parallel as the number of child processes it is managing (but this is something you can set). It's also not great if you need to run different functions.</p>

python multiprocessing python-3.5