Full blocking generator in twisted thread that runs forever
<p>Let's assume we have a <em>blackbox</em> generator that is fully blocking and runs forever. We're running Twisted, so standard way to handle blocking things is use <code>defers</code> or <code>defers + threads</code>.</p>

<pre><code># some very naive example
from twisted.internet import reactor

def aSillyBlockingMethod():
    for results in fullblocking_blockbox_generator_that_runs_forever():
        print results

reactor.callInThread(aSillyBlockingMethod)
reactor.run()
</code></pre>

<p>Yay! Job's done! ... but not exactly.</p>

<p>Everything is great until we want to stop whole Twisted app. One of the shutting down steps is call something like ~<code>threadpool.join()</code> which is waiting for all threads to finish themselves. But not our fullbocking ethernal <em>blackbox</em> generator. We can't do much in loop because we need yielded results from generator that could never happen.</p>

<p>I've browse whole internet for ideas. Nothing. Maybe I'm missing something important or there is a clever way to handle this?</p>

<p>I took the answer provided from <a href="http://stackoverflow.com/a/2244899/2172464">this post</a> and added example code. Also keep in mind it's always best to have well defined start and end points for threads. As you can see, it can cause some pretty sticky situations if a thread runs forever in an uncontrolled fashion. With that being said, sometimes some situations are unavoidable (but that doesn't mean we shouldn't try to avoid it whenever possible).</p>

<p>If the <em>blackbox</em> doesn't require anything from Twisted, you could start it as a daemon thread outside of Twisted's control which will stop when your application stops. In general, this is <strong>very dangerous</strong> because this kills threads immediately which could leave things in an corrupted state. In any case, here is an example:</p>

<pre><code>from threading import Thread, Event
from time import sleep
from twisted.internet import reactor

class BlackBox(Thread):

    def __init__(self, stop_event=None):
        super(BlackBox, self).__init__()
        if not stop_event:
            stop_event = Event()
        self.stop_event = stop_event

    def run(self):
        while True:
            print('running in thread')
            if self.stop_event.is_set():
                print('STOPPING!')
                break
            sleep(1)

t = BlackBox()
t.daemon = True
t.start()
# reactor.callLater(5, t.stop_event.set)      # if you want to stop thread after 5 seconds
reactor.run()
</code></pre>

<p>The better option would have a control flow within your thread code, such as checking for an <code>Event</code>, and initiate a trigger when the app shuts down. To capture such a trigger in Twisted, use <code>reactor.addSystemEventTrigger</code>.</p>

<pre><code>from threading import Event
from time import sleep
from twisted.internet import reactor

def aSillyBlockingMethod(event):
    while True:
        print('running in thread')
        if event.is_set():
            print('STOPPING')
            break
        sleep(1)

stop_event = Event()
reactor.callInThread(aSillyBlockingMethod, stop_event)
reactor.addSystemEventTrigger('before', 'shutdown', stop_event.set)
reactor.run()
</code></pre>

<p>Option #2 is a much better solution than daemon threads because the threaded code doesn't arbitrarily stop, you could let Twisted manage the thread, and the thread can stop at an appropriate time (deemed by the developer). I hope this helps</p>

python multithreading twisted