Why is Python 3 http.client so much faster than python-requests?
<p>I was testing different Python HTTP libraries today and I realized that <a href="https://docs.python.org/3/library/http.client.html" rel="nofollow"><code>http.client</code></a> library seems to perform much much faster than <a href="http://docs.python-requests.org/en/master/" rel="nofollow"><code>requests</code></a>. </p>

<p>To test it you can run following two code samples.</p>

<pre><code>import http.client

conn = http.client.HTTPConnection("localhost", port=8000)
for i in range(1000):
    conn.request("GET", "/")
    r1 = conn.getresponse()
    body = r1.read()
    print(r1.status)

conn.close()
</code></pre>

<p>and here is code doing same thing with python-requests:</p>

<pre><code>import requests

with requests.Session() as session:
    for i in range(1000):
        r = session.get("http://localhost:8000")
        print(r.status_code)
</code></pre>

<p>If I start SimpleHTTPServer:</p>

<pre><code>&gt; python -m http.server
</code></pre>

<p>and run above code samples (I'm using Python 3.5.2). I get following results:</p>

<p>http.client:</p>

<pre><code>0.35user 0.10system 0:00.71elapsed 64%CPU 
</code></pre>

<p>python-requests:</p>

<pre><code>1.76user 0.10system 0:02.17elapsed 85%CPU 
</code></pre>

<p>Are my measurements and tests correct? Can you reproduce them too? If yes does anyone know what's going on inside <code>http.client</code> that make it so much faster? Why is there such big difference in processing time?</p>

<p>Based on profiling both, the main difference appears to be that the <code>requests</code> version is doing a DNS lookup for every request, while the <code>http.client</code> version is doing so once.</p>

<pre><code># http.client
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1974    0.541    0.000    0.541    0.000 {method 'recv_into' of '_socket.socket' objects}
     1000    0.020    0.000    0.045    0.000 feedparser.py:470(_parse_headers)
    13000    0.015    0.000    0.563    0.000 {method 'readline' of '_io.BufferedReader' objects}
...

# requests
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1481    0.827    0.001    0.827    0.001 {method 'recv_into' of '_socket.socket' objects}
     1000    0.377    0.000    0.382    0.000 {built-in method _socket.gethostbyname}
     1000    0.123    0.000    0.123    0.000 {built-in method _scproxy._get_proxy_settings}
     1000    0.111    0.000    0.111    0.000 {built-in method _scproxy._get_proxies}
    92000    0.068    0.000    0.284    0.000 _collections_abc.py:675(__iter__)
...
</code></pre>

<p>You're providing the hostname to <code>http.client.HTTPConnection()</code> once, so it makes sense it would call <code>gethostbyname</code> once. <code>requests.Session</code> probably could cache hostname lookups, but it apparently does not. </p>

<p>EDIT: After some further research, it's not just a simple matter of caching. There's a function for determining whether to bypass proxies which ends up invoking <code>gethostbyname</code> regardless of the actual request itself.</p>

python http python-requests