Why do processes take long time to join even when the map call to pool is completed?
<p>This is yet another question regarding <code>multiprocessing</code> module in <code>Python 3.</code>5. My problem is that I know all the forked processed have done their job (I can see their result in <code>Queue</code>), the AsyncResult.result() returns True which means that jobs are completed but when I proceed with PoolObj.join(), it takes forever. I know I can PoolObj.terminate() and carry on with my life, but I want to know why the heck does this happen?</p>

<p>I'm using following code:</p>

<pre><code>def worker(d):
    queue.put(d)

def gen_data():
    for i in range(int(1e6)):
        yield i

if __name__ == "__main__":
    queue = Queue(maxsize=-1)
    pool = Pool(processes=12)
    pool_obj_worker = pool.map_async(worker, gen_data(), chunksize=1)
    pool.close()

    print ('Lets run the workers...\n')
    while True:
        if pool_obj_worker.ready():
            if pool_obj_worker.successful():
                print ('\nAll processed successfully!') # I can see this quickly, so my jobs are done
            else:
                print ('\nAll processed. Errors encountered!')
            sys.stdout.flush()
            print (q.qsize()) # The size is right that means all workers have done their job
            pool.join() # will get stuck here for long long time
            queue.put('*')           
            break
    print ('%d still to be processed' %
           pool_obj_worker._number_left)
    sys.stdout.flush()
    time.sleep(0.5)
</code></pre>

<p>Am I doing it wrong? Please enlighten me. Or are the processes holding <code>join()</code> have gone zombie?</p>

<p>The issue here is that you are using an extra <code>Queue</code> in your worker, other than the one fournished by <code>Pool</code>. 
When the processes finish their work, they will all join the <code>FeederThread</code> used in the <code>multiprocessing.Queue</code> and these calls will hang (probably because all the threads call <code>join</code> simultaneously and there can be some weird race conditions, it is not easy to investigate).</p>

<p>Adding <code>multiprocessing.util.log_to_stderr(10)</code> permits to reveal that your processes hang while joining the queue feeder thread.</p>

<p>To solve your issue, you can either use <code>multiprocessing.SimpleQueue</code> instead of <code>multiprocessing.Queue</code> (no hang in join as there is no feeder thread) or try using the method <code>pool.unordered_imap</code> which provides the same kind of behavior as what you seem to implement (give back an unordered generator containing the results returned by worker).</p>

python parallel-processing multiprocessing zombie-process