Scrapy: Looping through search results only returns first item
<p>I'm scraping a site by going through the search page, then looping through all results within. However it only seems to be returning the first result for each page. I also don't think it's hitting the start page's results either.</p>

<p>Secondly, the price is returning as some sort of Unicode (Â£ symbol) - how can I remove it altogether just leaving the price?</p>

<pre><code> 'regular_price': [u'\xa38.59'],
</code></pre>

<p>Here is the HTML:
<a href="http://pastebin.com/F8Lud0hu" rel="nofollow">http://pastebin.com/F8Lud0hu</a></p>

<p>Here's the spider:</p>

<pre><code>import scrapy
import random
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from scrapy.selector import Selector
from cdl.items import candleItem

class cdlSpider(CrawlSpider):
        name = "cdl"
        allowed_domains = ["www.xxxx.co.uk"]
        start_urls = ['https://www.xxxx.co.uk/advanced_search_result.php']

        rules = [
                Rule(LinkExtractor(
                        allow=['advanced_search_result\.php\?sort=2a&amp;page=\d*']),
                        callback='parse_listings',
                        follow=True)
        ]

        def parse_listings(self, response):
                sel = Selector(response)
                urls = sel.css('a.product_img')

                for url in urls:
                        url = url.xpath('@href').extract()[0]
                        return scrapy.Request(url,callback=self.parse_item)

        def parse_item(self, response):

                candle = candleItem()

                n = response.css('.prod_info_name h1')

                candle['name'] = n.xpath('.//text()').extract()[0]


                if response.css('.regular_price'):
                        candle['regular_price'] = response.css('.regular_price').xpath('.//text()').extract()
                else:
                        candle['was_price'] = response.css('.was_price strong').xpath('.//text()').extract()
                        candle['now_price'] = response.css('.now_price strong').xpath('.//text()').extract()

                candle['referrer'] = response.request.headers.get('Referer', None)
                candle['url'] = response.request.url

                yield candle
</code></pre>

<p>Yes it's returning only the first result because of your <code>parse_listing</code> method (you're returning the first url and you should be yielding it). I would do something like:</p>

<pre><code>def parse_listings(self, response):
    for url in response.css('a.product_img::attr(href)').extract():
        yield Request(url, callback=self.parse_item)
</code></pre>

<p>In that case I would even do something like:</p>

<pre><code>class CdlspiderSpider(CrawlSpider):
    name = 'cdlSpider'
    allowed_domains = ['www.xxxx.co.uk']
    start_urls = ['https://www.xxxx.co.uk/advanced_search_result.php']

    rules = [
        Rule(LinkExtractor(allow='advanced_search_result\.php\?sort=2a&amp;page=\d*')),
        Rule(LinkExtractor(restrict_css='a.product_img'), callback='parse_item')
        ]

    def parse_item(self, response):
        ...
        if response.css('.regular_price'):
            candle['regular_price'] = response.css('.regular_price::text').re_first(r'\d+\.?\d*')
        else:
            candle['was_price'] = response.css('.was_price strong::text').re_first(r'\d+\.?\d*')
            candle['now_price'] = response.css('.now_price strong::text').re_first(r'\d+\.?\d*')
        ...
        return candle
</code></pre>

python scrapy