I want to crawl data from 1 to 10 pages automatically from website.How can i do it?
<pre><code>import requests
from bs4 import BeautifulSoup
My_Url = "http://questions.consumercomplaints.in/page/2"
Data = requests.get(My_Url)
Soup = BeautifulSoup(Data.content)
head_id = Soup.find_all({"div":"href"})
len(head_id)
for i in head_id:
    print i.text 
</code></pre>

<p>From above code i scrapped (reviews/complaints) from web page 2.
How do i craw data automatically from Page 3 to 10 (<a href="http://questions.consumercomplaints.in/page/3" rel="nofollow">http://questions.consumercomplaints.in/page/3</a>) </p>

<p>Why not surround your function in a ranged for loop?</p>

<pre><code>import requests
from bs4 import BeautifulSoup
for i in range(3,11):
    My_Url = "http://questions.consumercomplaints.in/page/" + str(i)
    Data = requests.get(My_Url)
    Soup = BeautifulSoup(Data.content)
    head_id = Soup.find_all({"div":"href"})
    len(head_id)
    for i in head_id:
        print i.text 
</code></pre>

<p>Have look at how the range function works <a href="http://pythoncentral.io/pythons-range-function-explained/" rel="nofollow">here</a>.</p>

python python-2.7 web-scraping web-crawler ipython