Making continuous clicks on a pagination 'view-more' button in selenium (python) to load JS populated data
<p>I am trying to crawl the contents of a website by dynamically clicking a 'load-more' button. I saw some other <a href="http://stackoverflow.com/questions/30490549/how-to-click-on-load-button-dynamically-using-selenium-python/40071131#40071131">similar questions</a> but they seem to be getting other types of errors or are simply facing version issues. I am trying to parse the website <a href="https://angel.co/companies" rel="nofollow">https://angel.co/companies</a>. Below is my code. </p>

<pre><code>with closing(Chrome()) as browser:
    browser.get(url)
    # wait for the page to load
    while True:
        try:
            WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.more")))
            WebDriverWait(browser, timeout=10).until(EC.text_to_be_present_in_element((By.CSS_SELECTOR, 'div.more'), 'More'))
        except:
            break
        element = browser.find_element_by_css_selector('div.more').click()
</code></pre>

<p>This code does not even hit the click option once. However, if I bring the second-wait-condition after the click() call (like below), it clicks the 'More' button once, the data for that pagination-step loads and then there are no more clicks done.</p>

<pre><code>with closing(Chrome()) as browser:
    browser.get(url)
    # wait for the page to load
    while True:
        try:
            WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.more")))
        except:
            break
        element = browser.find_element_by_css_selector('div.more').click()
        WebDriverWait(browser, timeout=10).until(EC.text_to_be_present_in_element((By.CSS_SELECTOR, 'div.more'), 'More'))
</code></pre>

<p>Can someone help me find out what am I doing wrong?</p>

<p>When websites like that dynamically load in content that way, it tends to wreak havoc on the page DOM, constantly invalidating element as new ones are being loaded in. I have found the best approach is to organize your code in way that puts the selenium calls into their own functions, which you then decorate with a retry decorator. If/when exceptions get thrown (StaleElementReferenceException, UnknownElementException, etc), you can immediately retry the call.</p>

<pre><code>from retry import retry
from explicit import waiter
from selenium import webdriver
from selenium.common.exceptions import StaleElementReferenceException


@retry(StaleElementReferenceException, tries=3, delay=0.5)
def click_more(driver):
    waiter.find_element(driver, 'div.more').click()

driver = webdriver.Chrome()

try:
    driver.get("https://angel.co/companies")

    while True:
        click_more(driver)

finally:
    driver.quit()
</code></pre>

<p>Full disclosure: Explicit is a python package I maintain and available from PyPI. It is basically the same thing as the WebDriverWait call you have.</p>

python selenium pagination web-crawler selenium-chromedriver