readable socket times out on recv
<p>I have a 'jobs' server which accepts requests from a client (there are 8 clients sending requests from another machine). The server then submits a 'job' (a 'job' is just an executable which writes a results file to disk), and on a 'jobs manager' thread waits until the job is done. When a job is done it sends a message to the client that a results files is ready to be copied back to the client.</p>

<p>On the main thread I use <code>select</code> to read incoming connections from clients, as well as jobs requests:</p>

<pre><code>readable, writable, exceptional = select.select(inputs, [], [])
</code></pre>

<p>where <code>inputs</code> is a list of accepted connections (sockets), and this list also includes the <code>server</code> socket. All sockets are set to non-blocking. To my best understanding, if this call to <code>select</code> returns a non-empty <code>readable</code>, it means some elements of <code>inputs</code> has incoming data waiting to be read.
I am reading data using the following logic (<code>SIZE</code> is a constant):</p>

<pre><code>for s in readable:
    if s is not server:
        try:
            socket_ok = True
            data = s.recv(SIZE)
        except socket.error as e:        
            print ('ERROR socket error: ' + str(e) )
            socket_ok = False
        except Exception as e:
            print ('ERROR error reading from socket: ' + str(e))
            socket_ok = False
        if not socket_ok:
            # do something
</code></pre>

<p>I have 2 problems:</p>

<ul>
<li>Sometimes I get a <code>[Errno 110] Connection timed out</code> exception, and I don't understand why - if I have a readable socket, doesn't it mean it has some data to be read? </li>
<li>How to deal with this exception - the <code>#do something</code> part. I can do a 'cleanup' - delete the running jobs which were requested by the timed-out socket, and remove the dead socket from the list. But I have no way of letting the client know that it should stop waiting for these jobs' results. Ideally I would like to reconnect somehow, because the jobs themselves keep running and produce results which I don't want to throw away.</li>
</ul>

<p><strong>EDIT</strong> I realized now that the jobs manager thread also have access to the sockets via a <code>Queue</code> instance - if a job is finished, the thread sends a 'job done' message through the relevant socket - so maybe the <code>send</code> and <code>recv</code> methods of the same socket cause some kind of race condition? But anyway, I don't see how this can cause a 'connection timed out' error.</p>

<p>A solution that was just a guess and seems to work: On the client side, I am using a blocking <code>recv</code> method to get message from the server that the job is done. Since a job can take a long time (e.g if the cluster running the jobs is low on resources), I guessed that maybe the socket waiting was the cause of the time-out. So instead of using <code>recv</code> in blocking mode, I use it with time-out of 5 seconds, so I can send a dummy message to the server every 5 seconds to keep the connection alive until a message is received. Now I don't get the exception (on the server side) any more.</p>

python sockets client-server