Scrapy: TypeError: 'Request' object is not iterable
<p>I'm making a spider using Scrapy (1.1.2) to scrap products. I managed to get it to work and scrape enough data, but now, I want for each element to make new request to the <code>product page</code> and scrap, for example the product description.</p>

<p>First, here's my last working code</p>

<p><strong>spider.py</strong> (except)</p>

<pre><code>class ProductScrapSpider(Spider):
    name = "dmoz"
    allowed_domains = ["example.com"]
    start_urls = [
        "http://www.example.com/index.php?id_category=24"
        # ...
    ]

    def parse(self, response):
        for sel in response.xpath("a long string"):
            mainloader = ProductLoader(selector=sel)
            mainloader.add_value('category', 'Category Name')
            mainloader.add_value('meta', self.get_meta(sel))
            # more data

            yield mainloader.load_item()

            # Follows the pagination
            next_page = response.css("li#pagination_next a::attr('href')")
            if next_page:
                url = response.urljoin(next_page[0].extract())
                yield scrapy.Request(url, self.parse)

    def get_meta(self, response):
        metaloader = ProductMetaLoader(selector=response)
        metaloader.add_value('store', "Store name")
        # more data

        yield metaloader.load_item()
</code></pre>

<p><strong>Output</strong></p>

<pre><code>[
 {
     "category": "Category Name",
     "price": 220000,
     "meta": {
         "baseURL": "",
         "name": "",
         "store": "Store Name"
     },
     "reference": "100XXX100"
 },
 ...
]
</code></pre>

<p>After reading the documentation and some answers here, I've altered the <code>get_meta</code> method and added a callback for the request <code>get_product_page</code>:</p>

<p><strong>new_spider.py</strong> (except)</p>

<pre><code>def get_meta(self, response):
    metaloader = ProductMetaLoader(selector=response)
    metaloader.add_value('store', "Store name")
    # more data

    items = metaloader.load_item()
    new_request = scrapy.Request(items['url'], callback=self.get_product_page)

    # Passing the metadata
    new_request.meta['item'] = items

    # The source of the problem
    yield new_request

def get_product_page(self, response):
    sel = response.selector.css('.product_description')
    items = response.meta['item']

    new_meta = items
    new_meta.update({'product_page': sel[0].extract()})

    return new_meta
</code></pre>

<p><strong>Expected output</strong></p>

<pre><code>[
 {
     "category": "Category Name",
     "price": 220000,
     "meta": {
         "baseURL": "",
         "name": "",
         "store": "Store Name",
         "product_page": "&lt;div&gt; [...] &lt;/div&gt;" 
     },
     "reference": "100XXX100"
 },
 ...
]
</code></pre>

<p><strong>Error</strong></p>

<pre><code>TypeError: 'Request' object is not iterable
</code></pre>

<p><a href="http://pastebin.com/napscirN" rel="nofollow">full output</a></p>

<p>I couldn't find anything about this error, so please help me fix it.</p>

<p>Thanks a lot.</p>

<p>The error you experience (<code>TypeError: 'Request' object is not iterable</code>) happened because a <code>Request</code> instance is being put into a field of the item (in the updated <code>get_meta</code> method function), while the feed exporter cannot serialize it.</p>

<p>You would need to return the get meta request to Scrapy, together with a meta argument to pass the half-parsed item. Here's an example of the updated <code>parse</code> method and a new <code>parse_get_meta</code> method:</p>

<pre><code>def parse(self, response):
    for sel in response.xpath("a long string"):
        mainloader = ProductLoader(selector=sel)
        mainloader.add_value('category', 'Category Name')
        #mainloader.add_value('meta', self.get_meta(sel))
        # more data
        item = mainloader.load_item()
        get_meta_req = self.get_meta(sel)
        get_meta_req['meta']['item'] = item
        yield get_meta_req.replace(callback=self.parse_get_meta)

def parse_get_meta(self, response):
    """Parses a get meta response"""
    item = response.meta['item']
    # Parse the response and load the data here, e.g. item['foo'] = bar
    pass
    # Finally return the item
    return item
</code></pre>

<p>See also: <a href="http://doc.scrapy.org/en/latest/topics/request-response.html#topics-request-response-ref-request-callback-arguments" rel="nofollow">http://doc.scrapy.org/en/latest/topics/request-response.html#topics-request-response-ref-request-callback-arguments</a></p>

python python-2.7 scrapy scrapy-spider