python: socket.sendall() hogs GIL?
<p>i have a multithreaded program (about 20 threads; a mixture of producer/consumers with many queues)</p>

<p>in one of the threads, it pops strings from a queue and send it to a remote program</p>

<pre><code># it starts the thread like this
workQ = Queue.Queue()
stop_thr_event = threading.Event()
t = threading.Thread( target=worker, args=(stop_thr_event,) )


# in the thread's worker function
def worker(stop_event):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_address = (myhost, int(myport))
    sock.connect(server_address)
    while True:
        try:
            item = workQ.get(timeout=1)

            if print_only:
                print item
            else:
                if item.startswith("key:"):
                    item = "{%s}" % item
                    sock.sendall(item)

            workQ.task_done()
        except Queue.Empty, msg:
            if stop_event.isSet():
                break
</code></pre>

<p>intermittently, my program will just hang, none of the threads are doing any work</p>

<p>after trial and error, i found that my program only hangs with this thread running</p>

<p>my only guess is that sendall() is hogging the GIL and my whole program hangs</p>

<p>1) is this even a plausible theory?<br>
2) if my theory is correct, what can i do so that sendall() doesnt hog the GIL? make it a nonblock send?</p>

<p>You're wrong. No network activities hold GIL and sendall() is not an exception!</p>

<pre><code>item=workQ.get()
socket.sendall() **# may take long time here.**
workQ.task_done() 
</code></pre>

<p>Because sendall() may take long time and other threads which use workQ <strong>can not take turn to run before you call task_done()</strong> ==> this is why it seems that your whole program is in hang.</p>

python multithreading sockets deadlock hang