Bypassing intrusive cookie statement with requests library
<p>I'm trying to crawl a website using the <code>requests</code> library. However, the particular website I am trying to access (<a href="http://www.vi.nl/matchcenter/vandaag.shtml" rel="nofollow">http://www.vi.nl/matchcenter/vandaag.shtml</a>) has a very intrusive cookie statement.</p>

<p>I am trying to access the website as follows:</p>

<pre><code>from bs4 import BeautifulSoup as soup
import requests
website = r"http://www.vi.nl/matchcenter/vandaag.shtml"
html = requests.get(website, headers={"User-Agent": "Mozilla/5.0"})
htmlsoup = soup(html.text, "html.parser")
</code></pre>

<p>This returns a web page that consists of just the cookie statement with a big button to accept. If you try accessing this page in a browser, you find that pressing the button redirects you to the requested page. How can I do this using <code>requests</code>?</p>

<p>I considered using <code>mechanize.Browser</code> but that seems a pretty roundabout way of doing it.</p>

<p>Try setting:</p>

<pre><code>cookies = dict(BCPermissionLevel='PERSONAL')
html = requests.get(website, headers={"User-Agent": "Mozilla/5.0"}, cookies=cookies)
</code></pre>

<p>This will bypass the cookie consent page and will land you staight to the page.</p>

<p><strong>Note</strong>: You could find the above by analyzing the javascript code that is run on the cookie concent page, it is a bit obfuscated but it should not be difficult. If you run into the same type of problem again, take a look at what kind of cookies does the javascript code that is executed upon a event's handling sets.</p>

python cookies beautifulsoup python-requests