Troubleshooting item loaders between callbacks
<p>In order to understand the "Naive approach" example in <a href="http://oliverguenther.de/2014/08/almost-asynchronous-requests-for-single-item-processing-in-scrapy/" rel="nofollow">http://oliverguenther.de/2014/08/almost-asynchronous-requests-for-single-item-processing-in-scrapy/</a> </p>

<p>I am trying to replicate that code. The idea is to have a single item populated where each of the fields are sourced from a different website. </p>

<p>I am trying to understand why I get the following behavior from the below code when I run it and I export the result  into a csv file using <code>scrapy crawl compSpider -o prices.csv</code>.</p>

<p>The code actually populates the nic_price with the relevant price but it doesn't do the same with the tester_price. </p>

<p>I believe it should do so as the item loader object is passed in the request meta field from the first callback [firstRequest], where the item loader object is created, to the second call back [parseDescription1], where the item loader object is finally loaded into an item.</p>

<p>I have tested that the css selectors work. Can someone please help me understand why am I getting that behaviour?</p>

<h2>ITEM DECLARATION</h2>

<pre><code>import scrapy
class ProductItem(scrapy.Item):       
    nic_price = scrapy.Field()
    tester_price = scrapy.Field()
</code></pre>

<h2>SPIDER CODE</h2>

<pre><code>import scrapy
from scrapy.http import Request
from scrapy.loader import ItemLoader
from comparator.items import ProductItem


class Compspider(scrapy.Spider):
    name = "compSpider"


    #start_urls = (  'https://www.shop.niceic.com/',   )

    def start_requests(self):  

        yield Request(
            'https://www.shop.niceic.com/6559-megger-mft1711-multifunction-tester-1008-121',  callback=self.firstRequest)


    def firstRequest(self, response):
        l = ItemLoader(item=ProductItem(), response=response)
        l.add_css('nic_price', 'div.product-info p.product-price span[itemprop="price"]::text')

        yield Request('https://www.tester.co.uk/test-safe-pen-co-meter', meta={'loader' : l}, callback= self.parseDescription1)

    def parseDescription1(self, response):

        # Recover item(loader)
        l = response.meta['loader']

        # Use just as before
        l.add_css('tester_price', 'div.price-breakdown div.price-excluding-tax span.price::text')

        yield l.load_item()
</code></pre>

<p>The css selector is do definitely work. The problem resides in the itemloader variable <strong>l</strong> in which you are saving in the meta dict, it points to the response from <strong>firstRequest</strong> callback, not to the <strong>parseDescription1</strong> callback in which this css selector can work</p>

<p>'div.price-breakdown div.price-excluding-tax span.price::text'</p>

<p>to solve this, just create a new itemloader <strong>parseDescription1</strong> or better load the item itself rather than its loader in the meta dict as follows...</p>

<pre><code>    def firstRequest(self, response):

        l = ItemLoader(item=ProductItem(), response=response)

        l.add_css('nic_price', 'div.product-info p.product-price span[itemprop="price"]::text')

        yield Request('https://www.tester.co.uk/test-safe-pen-co-meter',
                  meta={'item': l.load_item()},
                  callback=self.parseDescription1)

    def parseDescription1(self, response):

        # Recover item
        item = response.meta['item']

        l = ItemLoader(item=ProductItem(), response=response)

        # Use just as before
        l.add_css('tester_price', 'div.price-breakdown div.price-excluding-tax span.price::text')

        l.add_value('nic_price', item['nic_price'])

        yield l.load_item()
</code></pre>

python csv scrapy web-crawler