Why am I unable to join this thread in python?
<p>I am writing a multithreading class. The class has a <code>parallel_process()</code> function that is overridden with the parallel task. The data to be processed is put in the <code>queue</code>. The <code>worker()</code> function in each thread keeps calling <code>parallel_process()</code> until the <code>queue</code> is empty. Results are put in the <code>results</code> Queue object. The class definition is:  </p>

<pre><code>import threading
try:
    from Queue import Queue
except ImportError:
    from queue import Queue


class Parallel:

    def __init__(self, pkgs, common=None, nthreads=1):
        self.nthreads = nthreads
        self.threads = []
        self.queue = Queue()
        self.results = Queue()
        self.common = common
        for pkg in pkgs:
            self.queue.put(pkg)

    def parallel_process(self, pkg, common):
        pass

    def worker(self):
        while not self.queue.empty():
            pkg = self.queue.get()
            self.results.put(self.parallel_process(pkg, self.common))
            self.queue.task_done()
        return

    def start(self):
        for i in range(self.nthreads):
            t = threading.Thread(target=self.worker)
            t.daemon = False
            t.start()
            self.threads.append(t)

    def wait_for_threads(self):
        print('Waiting on queue to empty...')
        self.queue.join()
        print('Queue processed. Joining threads...')
        for t in self.threads:
            t.join()
            print('...Thread joined.')

    def get_results(self):
        results = []
        print('Obtaining results...')
        while not self.results.empty():
            results.append(self.results.get())
        return results
</code></pre>

<p>I use it to create a parallel task:  </p>

<pre><code>    class myParallel(Parallel):    # return square of numbers in a list
            def parallel_process(self, pkg, common):
                return pkg**2

    p = myParallel(range(50),nthreads=4)
    p.start()
    p.wait_for_threads()
    r = p.get_results()
    print('FINISHED')
</code></pre>

<p>However all threads do not join every time the code is run. Sometimes only 2 join, sometimes no thread joins. I do not think I am blocking the threads from finishing. What reason could there be for <code>join()</code> to not work here?</p>

<p>@Brendan Abel identified the cause.  I'd like to suggest a different solution:  <code>queue.join()</code> is usually a Bad Idea too.  Instead, create a unique value to use as a sentinel:</p>

<pre><code>class Parallel:
    _sentinel = object()
</code></pre>

<p>At the end of <code>__init__()</code>, add one sentinel to the queue for each thread:</p>

<pre><code>    for i in range(nthreads):
        self.queue.put(self._sentinel)
</code></pre>

<p>Change the start of <code>worker()</code> like so:</p>

<pre><code>    while True:
        pkg = self.queue.get()
        if pkg is self._sentinel:
            break
</code></pre>

<p>By the construction of the queue, it won't be empty until each thread has seen its sentinel value, so there's no need to mess with the unpredictable <code>queue.size()</code>.</p>

<p>Also remove the <code>queue.join()</code> and <code>queue.task_done()</code> cruft.</p>

<p>This will give you reliable code that's easy to modify for fancier scenarios.  For example, if you want to add more work items <em>while</em> the threads are running, fine - just write another method to say "I'm done adding work items now", and move the loop adding sentinels into that.</p>

python multithreading