Gevent: Using two queues with two consumers without blocking each other at the same time
<p>I have the problem that I need to write values generated by a consumer to disk. I do not want to open a new instance of a file to write every time, so I thought to use a second queue and a other consumer to write to disk from a singe Greenlet. The problem with my code is that the second queue does not get consumed async from the first queue. The first queue finishes first and then the second queue gets consumed.
I want to write values to disk at the same time then other values get generated.
Thanks for help!</p>

<pre><code>#!/usr/bin/python
#- * -coding: utf-8 - * -
import gevent #pip install gevent
from gevent.queue import *
import gevent.monkey
from timeit import default_timer as timer
from time import sleep
import cPickle as pickle

def save_lineCount(count):
    with open("count.p", "wb") as f:
        pickle.dump(count, f)

def loader():
    for i in range(0,3):
        q.put(i)

def writer():
    while True:
        task = q_w.get()
        print "writing",task
        save_lineCount(task)

def worker():
    while not q.empty():
        task = q.get()
        if task%2:
            q_w.put(task)
            print "put",task
            sleep(10)

def asynchronous():
    threads = []
    threads.append(gevent.spawn(writer))
    for i in range(0, 1):
        threads.append(gevent.spawn(worker))
    start = timer()
    gevent.joinall(threads,raise_error=True)
    end = timer()
    #pbar.close()
    print "\n\nTime passed: " + str(end - start)[:6]


q = gevent.queue.Queue()
q_w = gevent.queue.Queue()
gevent.spawn(loader).join()
asynchronous()
</code></pre>

<p>In general, that approach should work fine. There are some problems with this specific code, though:</p>

<ul>
<li><p>Calling <code>time.sleep</code> will cause all greenlets to block. You either need to call <code>gevent.sleep</code> or monkey-patch the process in order to have just one greenlet block (I see <code>gevent.monkey</code> imported, but <code>patch_all</code> is not called). I suspect that's the major problem here.</p></li>
<li><p>Writing to a file is also synchronous and causes all greenlets to block. You can use <code>FileObjectThread</code> if that's a major bottleneck.</p></li>
</ul>

python multithreading queue gevent greenlets