Running time in a while loop (really slow)
<p>Following is my code:</p>

<pre><code>import socket
import time
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send(b'GET /code/romeo.txt HTTP/1.1\n')
mysock.send(b'Host: www.py4inf.com\n\n')
all = b""

while True:
    data = mysock.recv(512)
    all = all + data
    if len(data) &lt; 1:
        break

mysock.close()

stuff = all.decode()
position = stuff.find('\r\n\r\n')
print(stuff[position+4:])
</code></pre>

<p>There must be something wrong because it takes almost 30 seconds to invoke break in while loop.
However, if I change the code <code>if len(data) &lt; 1:</code> to <code>if len(data) &lt; 100:</code> it took just 0.5 second.</p>

<p>Please help. It haunted me for a while.
The sample website: <a href="http://www.py4inf.com/code/romeo.txt" rel="nofollow">http://www.py4inf.com/code/romeo.txt</a></p>

<p>Web servers don't have to close connections immediately.In fact, they may be looking for another http request.  Just add <code>print(data)</code> after the recv and you'll see you get the data, then a pause, then <code>b''</code>, meaning the server finally closed the socket.</p>

<p>You'll also notice that the server sends a header that includes "Content-Length: 167\r\n". Once the header has finished, the server will send exactly 167 bytes of data. You could parse out the header yourself but that's why we have client libraries like <code>urllib</code> and <code>requests</code>.</p>

<p>I was curious about how much would need to be added to the request header to get the connection to close immediately, and <code>Connection: close</code> seemed to do it. This returns right away:</p>

<pre><code>import socket
import time
mysock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect(('www.py4inf.com', 80))
mysock.send(b'GET /code/romeo.txt HTTP/1.1\n')
mysock.send(b'Connection: close\n')
mysock.send(b'Host: www.py4inf.com\n\n')
all = b""

while True:
    data = mysock.recv(512)
    all = all + data
    if len(data) &lt; 1:
        break

mysock.close()

stuff = all.decode()
position = stuff.find('\r\n\r\n')
print(stuff[position+4:])
</code></pre>

python sockets