Python selenium click element while displayed
<p>I'm scraping a dynamic page that requires the user to click a "Load More Results" button several times in order to get all data. Is there a better way to approach the task of clicking on an element while displayed? </p>

<pre><code>def clickon(xpath):
    try:
        element = driver.find_element_by_xpath(xpath)
    except:
        print "RETRYING CLICKON() for %s" % (xpath)
        time.sleep(1)
        clickon(xpath)
    else:
        element.click()
        time.sleep(3)

def click_element_while_displayed(xpath):
    element = driver.find_element_by_xpath(xpath)
    try:
        while element.is_displayed():
            clickon(xpath)
    except:
        pass
</code></pre>

<p>I suspect you are asking this question because the current solution is slow. This is mostly because you have these hardcoded <code>time.sleep()</code> delays which wait for more than they usually should. To tackle this problem I'd start using <a href="http://selenium-python.readthedocs.io/waits.html#explicit-waits" rel="nofollow">Explicit Waits</a> - initialize an endless loop and break it once selenium stopped waiting for the button to be clickable:</p>

<pre><code>from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

wait = WebDriverWait(driver, 10)

while True:
   try:
       element = wait.until(EC.element_to_be_clickable((By.XPATH, xpath)))
       element.click()
   except TimeoutException:
       break  # cannot click the button anymore

   # TODO: wait for the results of the click action
</code></pre>

<p>Now the last <code>TODO</code> part is also important - here I suggest you wait for a specific condition that would indicate that click resulted into something on a page - for instance, more results were loaded. For example, you can use a custom Expected Condition similar to <a href="http://stackoverflow.com/a/30177865/771848">this one</a>.</p>

python selenium selenium-webdriver