How to set selenium to process the next url when the current one loads more than 30 seconds?
<p>I want to know what to put into exception condition. I'm currently using the pass statement, but I'm not sure whether it does exactly what I want it to do. The reason why I want to implement this is because some webpages takes more than 30 seconds to load completely, like: taobao.com. My code is as followed:</p>

<pre><code>from selenium import webdriver 
from time import sleep 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.common.exceptions import WebDriverException

profile = webdriver.FirefoxProfile() 
profile.add_extension(extension = '/Users/wayne/Desktop/fourthparty/extension/fourthparty.xpi') 

driver = webdriver.Firefox(profile)
def scan(cutoff): 
    with open('top-1m.csv', 'r') as f: 
        for num, url in enumerate(f): 
            if (num == 500): 
                return

            url = url.split(',')[1] 
            driver.get('http://www.' + url) 
            sleep(30) 

            try: 
                driver.set_page_load_timeout(30) 
            except TimeoutException: 
                pass

if __name__ == "__main__": 
    scan(500)
</code></pre>

<p>You do the <code>set_page_load_timeout</code> once, usually shortly after instantiating the driver. You should then wrap the <code>driver.get</code> in the try except, like so:</p>

<pre><code>from __future__ import print_function

from selenium import webdriver
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import TimeoutException

profile = webdriver.FirefoxProfile()
profile.add_extension(extension = '/Users/wayne/Desktop/fourthparty/extension/fourthparty.xpi')
driver = webdriver.Firefox(profile)
driver.set_page_load_timeout(30)

def scan(cutoff):
    with open('top-1m.csv', 'r') as f:
        for num, url in enumerate(f):
            if (num == 500):
                return

            url = url.split(',')[1]
            try:
                driver.get('http://www.' + url)
            except TimeoutException:
                print("Caught and handled slow page timeout exception")

            # Do something here I guess?


if __name__ == "__main__":

scan(500)
</code></pre>

python selenium