Scraping several pages with BeautifulSoup
<p>I'd like to scrape through several pages of a website using Python and BeautifulSoup4. The pages differ by only a single number in their URL, so I could actually make a declaration like this:</p>

<pre><code>theurl = "beginningofurl/" + str(counter) + "/endofurl.html"
</code></pre>

<p>The link I've been testing with is this:</p>

<p>And my python script is this: <a href="http://www.worldofquotes.com/topic/Art/1/index.html" rel="nofollow">http://www.worldofquotes.com/topic/Art/1/index.html</a></p>

<pre><code>import urllib
import urllib.request
from bs4 import BeautifulSoup


def category_crawler():
    ''' This function will crawl through an entire category, regardless how many pages it consists of. '''

    pager = 1

    while pager &lt; 11:
        theurl = "http://www.worldofquotes.com/topic/Nature/"+str(pager)+"/index.html"
        thepage = urllib.request.urlopen(theurl)
        soup = BeautifulSoup(thepage, "html.parser")

        for link in soup.findAll('blockquote'):
            sanitized = link.find('p').text.strip()
            spantext = link.find('a')
            writer = spantext.find('span').text
            print(sanitized)
            print(writer)
            print('---------------------------------------------------------')


        pager += 1

category_crawler()
</code></pre>

<p>So the question is: how to change the hardcoded number in the while loop into a solution that makes the script automatically recognize that it passed the last page, and then it quits automatically?</p>

<p>Try with <code>requests</code> (avoiding redirections) and check if you get new quotes.</p>

<pre><code>import requests
from bs4 import BeautifulSoup


def category_crawler():
    ''' This function will crawl through an entire category, regardless how many pages it consists of. '''

    pager = 1

    while pager &lt; 11:
        theurl = "http://www.worldofquotes.com/topic/Art/"+str(pager)+"/index.html"
        thepage = requests.get(theurl, allow_redirects=False).text
        soup = BeautifulSoup(thepage, "html.parser")

        for link in soup.find_all('blockquote'):
            sanitized = link.find('p').text.strip()
            spantext = link.find('a')
            writer = spantext.find('span').text
            if not sanitized:
                break
            print(sanitized)
            print(writer)
            print('---------------------------------------------------------')


        pager += 1

category_crawler()
</code></pre>

python beautifulsoup