Python 3 Requests or urllib - how to always add a header?
<p>Title says it all: is there a 'best' way to always add a header to every request? I've got an internal tool that wants to send request ids to other internal tools; I'm looking for a blessed solution. I've skimmed the docs of both and it seems that this isn't a popular thing to ask for as I can't find a cookbook example.</p>

<p>I'm thinking of a couple solutions:</p>

<ol>
<li>Wrap requests in my own thin wrapper and use that. Need to teach codevelopers to remember to not <code>import requests</code> but <code>import myrequestswrapper as requests</code>.</li>
<li>Monkey-patch requests. I don't like monkey patching, but maybe just this once...? I dread the time when there comes a need to <em>not</em> send a header to this one particular system.</li>
</ol>

<p>edit: Why I'm not considering a requests.Session: it stores cookies and needs to be disposed of as it keeps its connection open.</p>

<p>Create a <a href="http://docs.python-requests.org/en/master/user/advanced/#session-objects" rel="nofollow">session object</a>, which is the 1st thing shown under <a href="http://docs.python-requests.org/en/master/user/advanced/" rel="nofollow">advanced usage</a>:</p>

<pre><code>s = requests.Session()
s.headers.update({'x-some-header': 'the value'})
s.get('http://httpbin.org/headers')
</code></pre>

<p>and use the session to perform requests. As you've stated that you do not wish to persist cookies between requests, you could subclass the <code>Session</code>:</p>

<pre><code>In [64]: from requests.adapters import HTTPAdapter

In [65]: from requests.cookies import cookiejar_from_dict

In [66]: class CookieMonsterSession(Session):
    ...:     
    ...:     def __init__(self, *args, **kwgs):
    ...:         super(CookieMonsterSession, self).__init__(*args, **kwgs)
    ...:         # Override default adapters with 0-pooling adapters
    ...:         self.mount('https://', HTTPAdapter(pool_connections=1,
    ...:                                            pool_maxsize=0))
    ...:         self.mount('http://', HTTPAdapter(pool_connections=1,
    ...:                                           pool_maxsize=0))
    ...:     @property
    ...:     def cookies(self):
    ...:         """ Freshly baked cookies, always!"""
    ...:         return cookiejar_from_dict({})
    ...:     @cookies.setter
    ...:     def cookies(self, newcookies):
    ...:         """ OM NOM NOM NOM..."""
    ...:         pass
    ...:     

In [67]: s = CookieMonsterSession()

In [69]: real_s = Session()

In [70]: s.get('http://www.google.fi')
Out[70]: &lt;Response [200]&gt;

In [71]: s.cookies
Out[71]: &lt;RequestsCookieJar[]&gt;

In [72]: real_s.get('http://www.google.fi')
Out[72]: &lt;Response [200]&gt;

In [73]: real_s.cookies
Out[73]: &lt;RequestsCookieJar[Cookie(version=0, name='NID', value='86=14qy...Rurx', port=None, port_specified=False, domain='.google.fi', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1489744358, discard=False, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)]&gt;
</code></pre>

<p>It is unfortunate that the <code>Session</code> is by design difficult to extend and configure, so "disabling" cookies, and modifying pooling like this is a hack and prone to break, if and when <code>Session</code> is updated the slightest. Also we've disabled the 2 main features of <code>Session</code> just for persistent headers.</p>

<p>Wrapping the basic API methods is probably the cleaner and safer approach:</p>

<pre><code># customrequests.py
from functools import wraps
from requests import api as requests_api

custom_headers = {}


def _header_wrapper(f):
    @wraps(f)
    def wrapper(*args, **kwgs):
        headers = kwgs.pop('headers', None) or {}
        headers.update(custom_headers)
        return f(*args, headers=headers, **kwgs)

    return wrapper

request = _header_wrapper(requests_api.request)
get = _header_wrapper(requests_api.get)
options = _header_wrapper(requests_api.options)
head = _header_wrapper(requests_api.head)
post = _header_wrapper(requests_api.post)
put = _header_wrapper(requests_api.put)
patch = _header_wrapper(requests_api.patch)
delete = _header_wrapper(requests_api.delete)
</code></pre>

<p>In action:</p>

<pre><code>In [1]: import customrequests as requests

In [2]: print(requests.get('http://httpbin.org/headers').text)
{
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.11.1"
  }
}


In [3]: requests.custom_headers['X-Test'] = "I'm always here"

In [4]: print(requests.get('http://httpbin.org/headers').text)
{
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.11.1", 
    "X-Test": "I'm always here"
  }
}
</code></pre>

python python-requests urllib