Exit a multiprocessing script
<p>I am trying to exit a multiprocessing script when an error is thrown by the target function, but instead of quitting, the parent process just hangs.</p>

<p>This is the test script I use to replicate the problem:</p>

<pre><code>#!/usr/bin/python3.5

import time, multiprocessing as mp

def myWait(wait, resultQueue):
    startedAt = time.strftime("%H:%M:%S", time.localtime())
    time.sleep(wait)
    endedAt = time.strftime("%H:%M:%S", time.localtime())
    name = mp.current_process().name
    resultQueue.put((name, wait, startedAt, endedAt))

# queue initialisation
resultQueue = mp.Queue()

# process creation arg: (process number, sleep time, queue)
proc =  [
    mp.Process(target=myWait, name = ' _One_', args=(2, resultQueue,)),
    mp.Process(target=myWait, name = ' _Two_', args=(2, resultQueue,))
    ]

# starting processes
for p in proc:
    p.start()

for p in proc:
    p.join()

# print results
results = {}
for p in proc:
    name, wait, startedAt, endedAt = resultQueue.get()
    print('Process %s started at %s wait %s ended at %s' % (name, startedAt, wait, endedAt))
</code></pre>

<p>This works perfectly, I can see the parent script spawning two child processes in <code>htop</code> but when I want to force the parent script to exit if an error is thrown in the <code>myWait</code> target function the parent process just hangs and doesn't even spawn any child process. I have to <code>ctrl-c</code> to kill it.</p>

<pre><code>def myWait(wait, resultQueue):
    try:
        # do something wrong
    except:
        raise SystemExit
</code></pre>

<p>I have tried every way to exit the function (e.g. <code>exit()</code>, <code>sys.exit()</code>, <code>os._exit()</code>...) to no avail.</p>

<p>Firstly, your code has a major issue: you're trying to join the processes before flushing the content of the queues, if any, which can result in a deadlock. See the section titled 'Joining processes that use queues' here: <a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming" rel="nofollow">https://docs.python.org/3/library/multiprocessing.html#multiprocessing-programming</a></p>

<p>Secondly, the call to <code>resultQueue.get()</code> will block until it receives some data, which never happens
if an exception is raised from the <code>myWait</code> function and that no data has been pushed into the queue before then. So make it non-blocking and make it check for any data in a loop until it finally receives something or that something's wrong.</p>

<p>Here's a quick'n'dirty fix to give you the idea:</p>

<pre><code>#!/usr/bin/python3.5

import multiprocessing as mp
import queue
import time

def myWait(wait, resultQueue):
    raise Exception("error!")

# queue initialisation
resultQueue = mp.Queue()

# process creation arg: (process number, sleep time, queue)
proc =  [
    mp.Process(target=myWait, name = ' _One_', args=(2, resultQueue,)),
    mp.Process(target=myWait, name = ' _Two_', args=(2, resultQueue,))
    ]

# starting processes
for p in proc:
    p.start()

# print results
results = {}
for p in proc:
    while True:
        if not p.is_alive():
            break

        try:
            name, wait, startedAt, endedAt = resultQueue.get(block=False)
            print('Process %s started at %s wait %s ended at %s'
                  % (name, startedAt, wait, endedAt))
            break
        except queue.Empty:
            pass

for p in proc:
    p.join()
</code></pre>

<p>The function <code>myWait</code> will throw an exception but both processes will still join and the program will exit nicely.</p>

python multiprocessing python-3.5