Authentication while scraping via BeautifulSoup in python
<p>I have created a piece of code to scrape an article off the ft.com website.  </p>

<pre><code>url = ""
r = requests.get(url)
soup = bs4.BeautifulSoup(r.content, "html.parser")
for a in soup.find_all('div', {"id":"storyContent"}):
    print a
</code></pre>

<p><strong>1) On the website, there is a div tag with id:storyContent but I get no output as a result of this code which means that it didn' enter the loop at all! What might the reason be?</strong><br>
Now ft.com does not give access to articles without entering username and password.<br>
I have logged into ft.com using chrome.<br>
Suppose my username, password details are the following:<br>
Username : bs@sb.com<br>
Pass: 12345<br>
I need to know either of the following:<br>
<strong>2) How can I provide this authentication in my code?<br>
3) How can I use the session on chrome (on which I'm already logged in) to acces the webpage/article details.<br>
4) If authentication is the resson behind no output!<br>
5) I am trying to get the article's body out of the webpage.</strong><br>
Thanks!</p>

<p>Rather start with this.</p>

<pre><code>url = "http://www.ft.com"
r = requests.get(url)
soup = BeautifulSoup(r.content, "html.parser")
for a in soup:
    print a
</code></pre>

<p>Then add a requests when you find the key:value pair required</p>

<pre><code>r = requests.post('http://www.ft.com/xxx', data = {'key':'value'})
</code></pre>

python python-2.7