Downloading a list files with python requests on a cert authenticated resource
<p>I've been on here all day long looking for a usable answer to this question, but haven't found something that works for my use case.</p>

<p>I am trying to download a bunch of files from a server that checks a client cert for authentication.  I also have a list array of specific files I want to download in an automated way.  I am using python 2.7.  What I'd like to do is to wait for FileOne.zip to download before looping back to start downloading FileTwo.zip, and so on.  Here's the code:</p>

<pre><code>import requests
import shutil

dlList = ["FileOne.zip", "FileTwo.zip", "FileThree.zip"]
cCert = r'C:\Temp\client_cert.pem'
cKey = r'C:\Temp\client_key.pem'

for i in dlList:
    url = ("https://my.server.com/files/" + i)
    r = requests.get(url, cert=(cCert, cKey), stream=True)
    with open(i, "wb") as f:
    r.raw.decode_content = True
    shutil.copyfileobj(r.raw, f)
</code></pre>

<p>The cert is working fine; I'm getting 200 response.</p>

<p>But when run, the script creates 3 files in the directory called FileOne.zip, FileTwo.zip, etc, but they are each only 2K and the files themselves are a couple hundred MB each.</p>

<p>What I'd like to do is get one file completed and then move on to the next.  Once that's working I can figure out how to multi-thread it.  But right now I just want to get the files down correctly...</p>

<p>Ok, here is what I did to fix it:</p>

<p>instead of</p>

<pre><code>r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)
</code></pre>

<p>I chunked it using:</p>

<pre><code>with open(i, "wb") as f:
    for chunk in r.iter_content(chunk_size=1024):
        if chunk:
            f.write(chunk)
</code></pre>

<p>I honestly don't know how efficient this is, but it seems to work.  Still would be happy on any advice on how I could tighten this up or move to concurrent downloads.</p>

python ssl python-multithreading downloading