python: socket.sendall() hogs GIL?
<p>i have a multithreaded program (about 20 threads; a mixture of producer/consumers with many queues)</p>

<p>in one of the threads, it pops strings from a queue and send it to a remote program</p>

<pre><code># it starts the thread like this
workQ = Queue.Queue()
stop_thr_event = threading.Event()
t = threading.Thread( target=worker, args=(stop_thr_event,) )


# in the thread's worker function
def worker(stop_event):
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_address = (myhost, int(myport))
    sock.connect(server_address)
    while True:
        try:
            item = workQ.get(timeout=1)

            if print_only:
                print item
            else:
                if item.startswith("key:"):
                    item = "{%s}" % item
                    sock.sendall(item)

            workQ.task_done()
        except Queue.Empty, msg:
            if stop_event.isSet():
                break
</code></pre>

<p>intermittently, my program will just hang, none of the threads are doing any work</p>

<p>after trial and error, i found that my program only hangs with this thread running</p>

<p>my only guess is that sendall() is hogging the GIL and my whole program hangs</p>

<p>1) is this even a plausible theory?<br>
2) if my theory is correct, what can i do so that sendall() doesnt hog the GIL? make it a nonblock send?</p>

<p>GIL-hogging will not cause a program to hang.  It may harm the performance of the program, but this is a far cry from hanging.  It is much more likely that you are experiencing some form of <a href="http://en.wikipedia.org/wiki/Deadlock" rel="nofollow">deadlock</a>.  The GIL cannot participate in a deadlock because the interpreter is constantly releasing and re-acquiring it, acquiring or releasing the GIL is generally not dependent on acquiring or releasing any other resources, and other locks do not depend on the GIL either.</p>

<p>Your use of the <code>stop_thr_event</code> lock is rather peculiar.  It would be more common for the master to simply put a series of "we're done, go home" objects into the queue, and for the workers to detect these objects and return when they are recognized.  This also ties into the rule of thumb that the only correct values for <code>timeout</code> are zero and infinity (i.e. no timeout).  In the current case, your worker is waiting for one second, checking the event, waiting for one second, etc., and <a href="https://blogs.msdn.microsoft.com/oldnewthing/20060124-17/?p=32553" rel="nofollow">polling is a Bad Thing</a>.</p>

<p>Now, if by "hang" you mean the program occasionally freezes up for short periods of time before resuming, that <em>is</em> poor performance, so perhaps the GIL could be to blame.  But the socket is not the problem.  The problem is that you may have a large number of threads contending for the GIL (because they're all trying to poll once per second), and if you're still on 2.x, you don't have <a href="https://docs.python.org/3/whatsnew/3.2.html#multi-threading" rel="nofollow">the new GIL</a>.  Eliminating the polling will help with this.</p>

python multithreading sockets deadlock hang