Python requests caching authentication headers
<p>I have used python's requests module to do a POST call(within a loop) to a single URL with varying sets of data in each iteration. I have already used session reuse to use the underlying TCP connection for each call to the URL during the loop's iterations. </p>

<p>However, I want to further speed up my processing by 1. caching the URL and the authentication values(user id and password) as they would remain the same in each call 2. Spawning multiple sub-processes  which could take a certain number of calls and pass them as a group thus allowing for parallel processing of these smaller sub-processes</p>

<p>Please note that I pass my authentication as headers in base64 format and a pseudo code of my post call would typically look like this:</p>

<pre><code>S=requests.Session()
url='https://example.net/'
Loop through data records:
    headers={'authorization':authbase64string,'other headers'}
    data="data for this loop"
    #Post call
    r=s.post(url,data=data,headers=headers)
response=r.json()
#end of loop and program
</code></pre>

<p>Please review the scenario and suggest any techniques/tips which might be of help-</p>

<p>Thanks in advance,
Abhishek</p>

<p>You can:</p>

<ol>
<li>do it as you described (if you want to make it faster then you can run it using multiprocessing) and e.g. add headers to session, not request.</li>
<li>modify target server and allow to send one post request with multiple data (so you're going to limit time spent on connecting, etc)</li>
<li>do some optimalizations on server side, so it's going to reply faster (or just store requests and send you response using some callback)</li>
</ol>

<p>It would be much easier if you described the use case :)</p>

python python-requests cache-control