Strange behavior of urllib.request in two different cases while webscrapping
<p>I am trying to download bhavcopy of NSE EoD data for last 15 years using web scrapping through urllib.request.</p>

<p>I am seeing that urllib.request is behaving strangely, it is working in one case but in another case it is throwing me error 403 Access Denies ..</p>

<p>I used the HTTP Headers for masking but in one case it is failing ..</p>

<p>Here is the code</p>

<pre><code>import urllib.request
def downloadCMCSV(year="2001",mon="JAN",dd="01"):
    #baseurl = "https://www.nseindia.com"
    headers = {'Host':'www.nseindia.com:443',
               'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
               'Accept-Encoding':'gzip, deflate, sdch, br',
               'Accept-Language':'en-US,en;q=0.8',
               #'Cookie':'NSE-TEST-1=1809850378.20480.0000; pointer=1; sym1=ONGC; pointerfo=1; underlying1=ONGC; instrument1=FUTSTK; optiontype1=-; expiry1=27OCT2016; strikeprice1=-',
               'Cookie':'NSE-TEST-1=1809850378.20480.0000; pointer=1; sym1=ONGC; pointerfo=1; underlying1=ONGC; instrument1=FUTSTK; optiontype1=-; expiry1=27OCT2016; strikeprice1=-; JSESSIONID=B4CA0543FF4C33FD9EA9D18B95238DB4',
               'Referer':'Referer: https://www.nseindia.com/products/content/equities/equities/archieve_eq.htm',
               'Upgrade-Insecure-Requests':'1',
               'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'}
    filename = "cm%s%s%sbhav.csv" % (dd,mon,year)
    urlcm = "https://www.nseindia.com/content/historical/EQUITIES/%s/%s/%s.zip" % (year, mon, filename)
    print(urlcm)
    request = urllib.request.Request(urlcm, headers = headers)
    #print(dir(request))
    #print(request.headers)
    try:
        response = urllib.request.urlopen(request)
    except urllib.error.HTTPError as e:
        if e.code == 404:
            print("Bhavcopy not available for", year, mon, dd)
            return
        print(e.code)
        print(e.read())  
        return
    if response.code == 200:
        print("The response is good", response.length)

if __name__ == "__main__":
    #getAll()
    downloadCMCSV('2001','JAN', '01')
    downloadCMCSV('2016','JAN', '01')
</code></pre>

<p>Got Output as below </p>

<pre><code>https://www.nseindia.com/content/historical/EQUITIES/2001/JAN/cm01JAN2001bhav.csv.zip
403
b'&lt;HTML&gt;&lt;HEAD&gt;\n&lt;TITLE&gt;Access Denied&lt;/TITLE&gt;\n&lt;/HEAD&gt;&lt;BODY&gt;\n&lt;H1&gt;Access Denied&lt;/H1&gt;\n \nYou don\'t have permission to access "http&amp;#58;&amp;#47;&amp;#47;www&amp;#46;nseindia&amp;#46;com&amp;#47;content&amp;#47;historical&amp;#47;EQUITIES&amp;#47;2001&amp;#47;JAN&amp;#47;cm01JAN2001bhav&amp;#46;csv&amp;#46;zip" on this server.&lt;P&gt;\nReference&amp;#32;&amp;#35;18&amp;#46;33210f17&amp;#46;1476700779&amp;#46;13b4f615\n&lt;/BODY&gt;\n&lt;/HTML&gt;\n'
https://www.nseindia.com/content/historical/EQUITIES/2016/JAN/cm01JAN2016bhav.csv.zip
The response is good 58943
</code></pre>

<p>Could you guys please help me what mistake I am doing ?</p>

<p>Pass a <em>user-agent</em> , <code>'Accept': '*/*'</code> and the <em>referer</em> headers:</p>

<pre><code>url = "https://www.nseindia.com/content/historical/EQUITIES/2001/JAN/cm01JAN2001bhav.csv.zip"
r = request.Request(url, headers={'User-Agent': 'mybot', 'Accept': '*/*',
                                  "Referer": "https://www.nseindia.com/products/content/equities/equities/archieve_eq.htm"})

print(request.urlopen(r))
</code></pre>

<p>You don't need cookies or anything else set.</p>

python url web-scraping urllib http-status-code-403