Yield both items and callback request in scrapy
<p>Disclaimer: I'm pretty new to both Python and Scrapy.</p>

<p>I'm trying to get my spider to gather urls from the start url, follow those gathered urls and both:</p>

<ol>
<li>scrape the next page for specific items (and eventually return them)</li>
<li>gather more specific urls from the next page and follow these urls.</li>
</ol>

<p>I want to be able to continue this process of yielding both items and callback requests, but I am not quite sure how to do it.
Currently my code only returns urls, and no items. I'm obviously doing something wrong. Any feedback would be greatly appreciated.</p>

<pre><code>class VSSpider(scrapy.Spider):
    name = "vs5"
    allowed_domains = ["votesmart.org"]
    start_urls = [
                  "https://votesmart.org/officials/WA/L/washington-state-legislative#.V8M4p5MrKRv",
                  ]

    def parse(self, response):
        sel = Selector(response)
        #this gathers links to the individual legislator pages, it works
        for href in response.xpath('//h5/a/@href'): 
            url = response.urljoin(href.extract())
            yield scrapy.Request(url, callback=self.parse1)

    def parse1(self, response):
        sel = Selector(response)
        items = []
        #these xpaths are on the next page that the spider should follow, when it first visits an individual legislator page
        for sel in response.xpath('//*[@id="main"]/section/div/div/div'):
            item = LegislatorsItems()
            item['current_office'] = sel.xpath('//tr[1]/td/text()').extract()
            item['running_for'] = sel.xpath('//tr[2]/td/text()').extract()
            items.append(item)
        #this is the xpath to the biography of the legislator, which it should follow and scrape next
        for href in response.xpath('//*[@id="folder-bio"]/@href'):
            url = response.urljoin(href.extract())
            yield scrapy.Request(url, callback=self.parse2, meta={'items': items})

    def parse2(self, response):
        sel = Selector(response)
        items = response.meta['items']
        #this is an xpath on the biography page
        for sel in response.xpath('//*[@id="main"]/section/div[2]/div/div[3]/div/'):
            item = LegislatorsItems()
            item['tester'] = sel.xpath('//div[2]/div[2]/ul/li[3]').extract()
            items.append(item)
            return items
</code></pre>

<p>Thanks!</p>

<p>There are 2 levels of your problem.<br></p>

<p><strong>1.</strong> Bio url is not available with JS disabled. Turn off JS in your browser and check this page:
<a href="https://votesmart.org/candidate/126288/derek-stanford" rel="nofollow">https://votesmart.org/candidate/126288/derek-stanford</a></p>

<p>You should see the tag with empty href and correct url hidden under comment. </p>

<pre><code>&lt;a href="#" class="folder" id="folder-bio"&gt;
&lt;!--&lt;a href='/candidate/biography/126288/derek-stanford' itemprop="url" class='more'&gt;
           See Full Biographical and Contact Information&lt;/a&gt;--&gt;
</code></pre>

<p>For extracting bio url, you can get this comment with xpath selector "/comment()", and then extract url with regexp. </p>

<p>Or, if url structure is common for all pages, just form url yourself: replace "/candidate/" in link with "/candidate/biography/".</p>

<blockquote>
  <p><strong>NB!</strong> If you face unexpected issues, one of the first actions - disable JS and look at the page as Scrapy see it. Test all selectors.</p>
</blockquote>

<hr>

<p><strong>2.</strong> Your usage of items is very complicated. If "one item = one person", you should just define one item in "parse_person" and pass it to "parse_bio". </p>

<p>Take a look at the updated code. I rewrote some parts while finding the issue. Notes:</p>

<ul>
<li>You don't need (in most cases) to create "items" list and append items to it. Scrapy manage with items itself.</li>
<li>"sel = Selector(response)" has no sense in your code, you can throw it.</li>
</ul>

<p>This code is tested with Scrapy 1.0 and Python 3.5, though with earlier versions should work too.</p>

<pre><code>from scrapy import Spider, Request

class VSSpider(Spider):
    name = "vs5"
    allowed_domains = ["votesmart.org"]
    start_urls = ["https://votesmart.org/officials/WA/L/washington-state-legislative"]

    def parse(self, response):
        for href in response.css('h5 a::attr(href)').extract():
            person_url = response.urljoin(href)
            yield Request(person_url, callback=self.parse_person)

    def parse_person(self, response):  # former "parse1"
        # define item, one for both parse_person and bio function
        item = LegislatorsItems()

        # extract text from left menu table and populate to item
        desc_rows = response.css('.span-abbreviated td::text').extract()
        if desc_rows:
            item['current_office'] = desc_rows[0]
            item['running_for'] = desc_rows[1] if len(desc_rows) &gt; 1 else None

        # create right bio url and pass item to it
        bio_url = response.url.replace('votesmart.org/candidate/', 
                                       'votesmart.org/candidate/biography/')
        return Request(bio_url, callback=self.parse_bio, meta={'item': item})

    def parse_bio(self, response):  # former "parse2"
        # get item from meta, add "tester" data and return
        item = response.meta['item']
        item['tester'] = response.css('.item.first').xpath('//li[3]').extract()
        print(item)   # for python 2: print item 
        return item
</code></pre>

python callback scrapy