Python scraping: 403 and 503 Errors
<p>I am trying to crawl some public information(apple app's info) on a website. </p>

<p>This website requires log-in in order to perform actions such as "search app/developer". Although there are many website provide similar information, but I consider this particular website provides the most complete and detailed info for each app. </p>

<p>I, as a valid user, am able to perform the task.</p>

<p>However, when I try to access the info via python code, I have the encountered 403 Error when sending POST request, and 504 Error when sending Get request. </p>

<p>I have tried using </p>

<ol>
<li><p>real userAgent header </p></li>
<li><p>fake-useragent" package</p></li>
<li><p>FancyOpener[/sth like that, shown depreciated for python 3.4]</p></li>
<li><p>HttpAuthM..[/sth like that, for authentication, still doesn't work]</p></li>
</ol>

<p>I guess the website is highly against automate access, but the detailed info there is highly useful. Is there any way I could workaround this issue? </p>

<p>Thanks!!</p>

<p>I tried this header:</p>

<pre><code>ua = {#'User-Agent':'Mozilla/5.0 (compatible; Googlebot/2.1; +Googlebot - Webmaster Tools Help)',  
      'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36',  
      'Connection':'Keep-Alive',  
      'Accept-Language':'zh-CN,zh;q=0.8',  
      'Accept-Encoding':'gzip,deflate,sdch',  
      'Accept':'*/*',  
      'Accept-Charset':'GBK,utf-8;q=0.7,*;q=0.3',  
      'Cache-Control':'max-age=0'  
      }  
</code></pre>

<p><a href="http://i.stack.imgur.com/LUcfv.jpg" rel="nofollow">503 Error</a></p>

<p><a href="http://i.stack.imgur.com/4zTDM.jpg" rel="nofollow">403 Error</a></p>

<pre><code>------------------------------------------------ HTTPError   
Traceback (most recent call last) &lt;ipython-input-43-421b27c5194e&gt; in &lt;module&gt;()
     68 data= data.encode('utf-8')
     69 request = urq.Request(url, data, headers = ua)
---&gt; 70 response = urq.urlopen(request)
     71 the_page = response.read()
     72 print(the_page)

c:\python34\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    159     else:
    160         opener = _opener
--&gt; 161     return opener.open(url, data, timeout)
    162 
    163 def install_opener(opener):

c:\python34\lib\urllib\request.py in open(self, fullurl, data, timeout)
    468         for processor in self.process_response.get(protocol, []):
    469             meth = getattr(processor, meth_name)
--&gt; 470             response = meth(req, response)
    471 
    472         return response

c:\python34\lib\urllib\request.py in http_response(self, request, response)
    578         if not (200 &lt;= code &lt; 300):
    579             response = self.parent.error(
--&gt; 580                 'http', request, response, code, msg, hdrs)
    581 
    582         return response

c:\python34\lib\urllib\request.py in error(self, proto, *args)
    506         if http_err:
    507             args = (dict, 'default', 'http_error_default') + orig_args
--&gt; 508             return self._call_chain(*args)
    509 
    510 # XXX probably also want an abstract factory that knows when it makes

c:\python34\lib\urllib\request.py in _call_chain(self, chain, kind, meth_name, *args)
    440         for handler in handlers:
    441             func = getattr(handler, meth_name)
--&gt; 442             result = func(*args)
    443             if result is not None:
    444                 return result

c:\python34\lib\urllib\request.py in http_error_default(self, req, fp, code, msg, hdrs)
    586 class HTTPDefaultErrorHandler(BaseHandler):
    587     def http_error_default(self, req, fp, code, msg, hdrs):
--&gt; 588         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    589 
    590 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: FORBIDDEN
----------------------------------------------
</code></pre>

<p>The below result I obtained by using "Advanced REST client", which is a chrome extension to send request. Notice how on a page that doesn't require log in, the code is 200; the other one was 403 at login page.See link in comment below</p>

<p>[access success][3]</p>

<p>[access fail][4]</p>

<p>plain python requests package is enough, you shouldn't need other packages.</p>

<p>I'm sure your problem is just you're not perfectly emulating a browser request.
On Google Chrome and Mozilla Firefox you should be able to see requests headers from a developer panel.</p>

<p>Be sure you always use the <strong>same</strong> session object.</p>

<p>Be sure not to forget to set proper headers:</p>

<ul>
<li>User-Agent</li>
<li>Accept</li>
<li>Accept-Language</li>
<li>Accept-Encoding</li>
<li><strong>Referer</strong>  (url of previous GET request)</li>
<li>Connection (keep-alive)</li>
<li><strong>Host</strong> ( abc.website.com )</li>
</ul>

<p>.</p>

<pre><code>session.headers = {
    'User-Agent' : 'real one',
    ...
}
</code></pre>

<p>Be sure to respect redirects:</p>

<pre><code>session.get(url, allow_redirects=True, timeout=x_secs)
</code></pre>

<p>In post requests be sure you send all the required fields, there may be some hidden ones (like the csfr token).</p>

python web-crawler http-error