Create a new instance of a generator in python
<p>I am trying to scrape a page which has many links to pages which contain ads.  What I am currently doing to navigate it is going to the first page with the list of ads and getting the link for the individual ads.  After that, I check to make sure that I haven't scraped any of the links by pulling data from my database.  The code below basically gets all the href attributes and joins them as a list.  After, I crosscheck it against the the list of links I have stored in my database of pages I have already scraped.  So basically it will return a list of the links I haven't scraped yet.  </p>

<pre><code>@staticmethod
def _scrape_home_urls(driver):
    home_url_list = list(home_tab.find_element_by_tag_name('a').get_attribute('href') for home_tab in driver.find_elements_by_css_selector('div[class^="nhs_HomeResItem  clearfix"]'))
    return (home_url for home_url in home_url_list if home_url not in(url[0] for url in NewHomeSource.outputDB()))
</code></pre>

<p>Once it scrapes all the links of that page, it goes to the next one.  I tried to reuse it by calling _scrape_home_urls() again </p>

<pre><code>    NewHomeSource.unique_home_list = NewHomeSource._scrape_home_urls(driver)
    for x in xrange(0,limit):

        try:
            home_url = NewHomeSource.unique_home_list.next()

        except StopIteration:
            page_num = int(NewHomeSource.current_url[NewHomeSource.current_url.rfind('-')+1:]) + 1 #extract page number from url and gets next page by adding 1. example: /.../.../page-3
            page_url = NewHomeSource.current_url[:NewHomeSource.current_url.rfind('-')+1] + str(page_num)
            print page_url
            driver.get(page_url)
            NewHomeSource.current_url = driver.current_url
            NewHomeSource.unique_home_list = NewHomeSource._scrape_home_urls(driver)
            home_url = NewHomeSource.unique_home_list.next()

            #and then I use the home_url to do some processing within the loop
</code></pre>

<p>Thanks in advance.</p>

<p>It looks to me like your code would be a lot simpler if you put the logic that scrapes successive pages into a generator function. This would let you use <code>for</code> loops rather than messing around and calling <code>next</code> on the generator objects directly:</p>

<pre><code>def urls_gen(driver):
    while True:
        for url in NewHomeSource._scrape_home_urls(driver):
            yield url
        page_num = int(NewHomeSource.current_url[NewHomeSource.current_url.rfind('-')+1:]) + 1 #extract page number from url and gets next page by adding 1. example: /.../.../page-3
        page_url = NewHomeSource.current_url[:NewHomeSource.current_url.rfind('-')+1] + str(page_num)
        print page_url
        driver.get(page_url)
        NewHomeSource.current_url = driver.current_url
</code></pre>

<p>This will transparently skip over pages that don't have any unprocessed links. The generator function yields the url values indefinitely. To iterate on it with a limit like your old code did, use <code>enumerate</code> and <code>break</code> when the limit is reached:</p>

<pre><code>for i, home_url in urls_gen(driver):
    if  i == limit:
        break

    # do stuff with home_url here
</code></pre>

<p>I've not changed your code other than what was necessary to change the iteration. There are quite a few other things that could be improved however. For instance, using a shorter variable than <code>NewHomeSource.current_url</code> would make the lines of the that figure out the page number and then the next page's URL much more compact and readable. It's also not clear to me where that variable is initially set. If it's not used anywhere outside of this loop, it could easily be changed to a local variable in <code>urls_gen</code>.</p>

<p>Your <code>_scrape_home_urls</code> function is probably also very inefficient. It looks like it does a database query for every url it returns (not one lookup before checking all of the urls). Maybe that's what you want it to do, but I suspect it would be much faster done another way.</p>

python generator yield-keyword