Multiprocessing process does not join when putting complex dictionary in return queue
<p>Given a pretty standard read/write multithreaded process with a read Queue and a write Queue:</p>

<p>8 times <code>worker done</code> is printed, but the join() statement is never passed. But if I replace <code>queue_out.put(r)</code> by `queue_out.put(1) it works. </p>

<p>This is melting my brain, probably something really stupid. Should I make a copy of my dictionary and put that in the return Queue? Did I make a stupid mistake somewhere? </p>

<p><strong>Process function</strong></p>

<pre><code>def reader(queue_in, queue_out, funktion):
    # Read from the queue
    while True:
        r = queue_in.get()
        if r == 'DONE':
            return
        funktion(r) # funktion adds additional keys to the dictionary
        queue_out.put(r) # &lt;---- replacing r by 1 does let me join()
    print "worker done" # &lt;----- this happens
</code></pre>

<p><strong>Populate the input queue</strong></p>

<pre><code>def writer(generator, queue):
    # Write to the queue
    for r in enumerate(generator):
        # r is a complex dictionary
        queue.put(r)    
    print "writer done"
    for _ in range(0, WORKERS):
        queue.put((-1, "DONE"))
</code></pre>

<p><strong>The rest</strong></p>

<pre><code>WORKERS = 8

# init Queues
queue_in = Queue()
queue_out = Queue()

# Start processes, with input and output quests
readers = []
for _ in range(0, WORKERS):
    p = Process(target=reader, args=(queue_in, queue_out, funktion))
    p.daemon = True
    p.start()
    readers.append(p)

writer(generator, queue_in)

for p in readers:
    p.join()

print "joined"  # &lt;---- this never happens

queue_in.close()

while not queue_out.empty():
    print queue_out.get()
queue_out.close()
</code></pre>

<p>I <em>think</em> I have pieced this together from two sources as I always have the same problem. I think the important thing is that this is in Windows. </p>

<p>Note from <a href="https://docs.python.org/2/library/multiprocessing.html" rel="nofollow">the documentation</a></p>

<blockquote>
  <p>Since Windows lacks os.fork() it has a few extra restrictions:</p>
</blockquote>

<p>Then read the answers <a href="http://stackoverflow.com/questions/25391025/what-exactly-is-python-multiprocessing-modules-join-method-doing">here</a> that <code>join()</code> is for forked processed. </p>

<p>I have always managed to run <code>multiprocessing</code> in a similar fashion to you without using <code>join()</code> and not seen any errors - I'm more than happy for a counterexample to explain why it's needed. Indeed, removing it has corrected your issue.</p>

<p>And <a href="http://rhodesmill.org/brandon/2010/python-multiprocessing-linux-windows/" rel="nofollow">this article</a> goes into more depth about the differences with child processes in multiprocessing between operating systems. I do think that the issue with <code>join()</code>, specifically, should be more explicit in the documentation.</p>

python multithreading dictionary queue multiprocessing