How to Scrape Page HTML and follow Next Link in Selenium
<p>I'm trying to scrape a website for research, and I'm stuck. I want the scraper to read the page source, and append that to a local HTML file so I can analyze the data off-campus. I have experimented with <code>BeautifulSoup</code> and <code>Scrapy</code>, but I have found that I need to use <code>Selenium</code> to interact with the page to navigate through my university's authentication system. (I'm not including that code, because it's tangential to my question.)</p>

<p>When I run the script it navigates to the page and clicks the link, but it only saves the first page's HTML. It then duplicates and appends that page's HTML every time it click on the link. </p>

<p>How do I use <code>Selenium</code> to click on the next page link, scrape the HTML, and save to file until I reach the last page?</p>

<pre><code>source = driver.page_source

while True:
    with open("test.html", "a") as TestFile:
        TestFile.write(source)
        try:
            driver.implicitly_wait(200)
            driver.find_element_by_css_selector('li.next').click()
        except AttributeError:
            break
</code></pre>

<p>Edit: I added the AttributeError to the except and received the following error.</p>

<blockquote>
  <p>selenium.common.exceptions.StaleElementReferenceException: Message:
  stale element reference: element is not attached to the page document</p>
</blockquote>

<p>My assumption is that I need to slow down the <code>.click()</code>, which is why I originally had the implicit wait, but that doesn't seem to be doing the trick.</p>

<p>You need to assign the <code>page source</code> to <code>source</code> variable inside the while loop.</p>

<pre><code>source = driver.page_source

while True:
with open("test.html", "a") as TestFile:
    TestFile.write(source)
    try:
        driver.implicitly_wait(200)
        driver.find_element_by_css_selector('li.next').click()
        source = driver.page_source
    except AttributeError:
        break
</code></pre>

python selenium