python urllib2.urlopen returns error 500 while Chrome loads the page
<p>I have a web-page (<a href="http://rating.chgk.info/api/tournaments/3506/" rel="nofollow">http://rating.chgk.info/api/tournaments/3506/</a>) I want to open in Python 2 via urllib2. It opens perfectly well in my browser but when I do this:</p>

<pre><code>import urllib2
url = 'http://rating.chgk.info/api/tournaments/3506/'
urllib2.urlopen(url)
</code></pre>

<p>i get HTTP Error 500.</p>

<p>I tried tweaking User-Agent and Accept headers but nothing worked. What else can be the problem?</p>

<p>You need to first visit the page on the site to get the session cookie set:</p>

<pre><code>In [7]: import requests

In [8]: requests.get("http://rating.chgk.info/api/tournaments/3506")
Out[8]: &lt;Response [500]&gt;

In [9]: with requests.Session() as session:
    ...:     session.get("http://rating.chgk.info/index.php/api")
    ...:     response = session.get("http://rating.chgk.info/api/tournaments/3506")
    ...:     print(response.status_code)
    ...:     
200
</code></pre>

python web-scraping urllib2