Bypassing intrusive cookie statement with requests library
<p>I'm trying to crawl a website using the <code>requests</code> library. However, the particular website I am trying to access (<a href="http://www.vi.nl/matchcenter/vandaag.shtml" rel="nofollow">http://www.vi.nl/matchcenter/vandaag.shtml</a>) has a very intrusive cookie statement.</p>

<p>I am trying to access the website as follows:</p>

<pre><code>from bs4 import BeautifulSoup as soup
import requests
website = r"http://www.vi.nl/matchcenter/vandaag.shtml"
html = requests.get(website, headers={"User-Agent": "Mozilla/5.0"})
htmlsoup = soup(html.text, "html.parser")
</code></pre>

<p>This returns a web page that consists of just the cookie statement with a big button to accept. If you try accessing this page in a browser, you find that pressing the button redirects you to the requested page. How can I do this using <code>requests</code>?</p>

<p>I considered using <code>mechanize.Browser</code> but that seems a pretty roundabout way of doing it.</p>

<p>I have found <a href="http://stackoverflow.com/questions/7164679/how-to-send-cookies-in-a-post-request-with-the-python-requests-library">this</a> SO question which asks how to send cookies in a post using requests. The accepted answer states that the latest build of Requests will build CookieJars for you from simple dictionaries. Below is the POC code included in the original answer.</p>

<pre><code>import requests

cookie = {'enwiki_session': '17ab96bd8ffbe8ca58a78657a918558'}

r = requests.post('http://wikipedia.org', cookies=cookie)
</code></pre>

python cookies beautifulsoup python-requests