Scraping several pages with BeautifulSoup
<p>I'd like to scrape through several pages of a website using Python and BeautifulSoup4. The pages differ by only a single number in their URL, so I could actually make a declaration like this:</p>

<pre><code>theurl = "beginningofurl/" + str(counter) + "/endofurl.html"
</code></pre>

<p>The link I've been testing with is this:</p>

<p>And my python script is this: <a href="http://www.worldofquotes.com/topic/Art/1/index.html" rel="nofollow">http://www.worldofquotes.com/topic/Art/1/index.html</a></p>

<pre><code>import urllib
import urllib.request
from bs4 import BeautifulSoup


def category_crawler():
    ''' This function will crawl through an entire category, regardless how many pages it consists of. '''

    pager = 1

    while pager &lt; 11:
        theurl = "http://www.worldofquotes.com/topic/Nature/"+str(pager)+"/index.html"
        thepage = urllib.request.urlopen(theurl)
        soup = BeautifulSoup(thepage, "html.parser")

        for link in soup.findAll('blockquote'):
            sanitized = link.find('p').text.strip()
            spantext = link.find('a')
            writer = spantext.find('span').text
            print(sanitized)
            print(writer)
            print('---------------------------------------------------------')


        pager += 1

category_crawler()
</code></pre>

<p>So the question is: how to change the hardcoded number in the while loop into a solution that makes the script automatically recognize that it passed the last page, and then it quits automatically?</p>

<p>Here is my attempt. </p>

<p>Minor issue: put a <code>try-except</code> block in the code in case the redirection leads you somewhere that doesn't exist. </p>

<p>Now, the main issue: how to avoid parsing stuff you already parsed. Keep a record of urls that you have parsed. Then detect if the <em>actual</em> url from the page <code>urllib</code> is reading from (using the <code>geturl()</code> method from from <code>thepage</code>) has already been read. Worked on my Mac OSX machine. </p>

<p>Note: there are 10 pages in total according to what I see from the website and this method does not require prior knowledge about the page's HTML- it works in general.</p>

<pre><code>import urllib
import urllib.request
from bs4 import BeautifulSoup


def category_crawler():
    ''' This function will crawl through an entire category, regardless how many pages it consists of. '''
    urlarchive = [];
    pager = 1
    while True:
        theurl = "http://www.worldofquotes.com/topic/Nature/"+str(pager)+"/index.html"
        thepage = None;
        try:
            thepage = urllib.request.urlopen(theurl)
            if thepage.geturl() in urlarchive:
                break;
            else:
                urlarchive.append(thepage.geturl());
                print(pager);
        except:
            break;
        soup = BeautifulSoup(thepage, "html.parser")

        for link in soup.findAll('blockquote'):
            sanitized = link.find('p').text.strip()
            spantext = link.find('a')
            writer = spantext.find('span').text
            print(sanitized)
            print(writer)
            print('---------------------------------------------------------')


        pager += 1

category_crawler()
</code></pre>

python beautifulsoup