Extract data by pages one by one using single scrapy spider
<p>I am trying to extract data from <a href="https://www.goodreads.com/list/show/19793.I_Marked_My_Calendar_For_This_Book_s_Release?page=1" rel="nofollow">goodreads.</a></p>

<p>I want to crawl pages one by one using some time delay.</p>

<p>My spider looks like:</p>

<pre><code>import scrapy
import unidecode
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from lxml import html


class ElementSpider(scrapy.Spider):
    name = 'books'
    download_delay = 3
    allowed_domains = ["https://www.goodreads.com"]
    start_urls = ["https://www.goodreads.com/list/show/19793.I_Marked_My_Calendar_For_This_Book_s_Release?page=1",
                   ]
    rules = (Rule(LinkExtractor(allow=(), restrict_xpaths=('//a[@class="next_page"]',)), callback="parse", follow= True),)

    def parse(self, response):
        for href in response.xpath('//div[@id="all_votes"]/table[@class="tableList js-dataTooltip"]/tr/td[2]/div[@class="js-tooltipTrigger tooltipTrigger"]/a/@href'):
            full_url = response.urljoin(href.extract())
            print full_url
            yield scrapy.Request(full_url, callback=self.parse_books)

        next_page = response.xpath('.//a[@class="button next"]/@href').extract()
        if next_page:
            next_href = next_page[0]
            print next_href
            next_page_url = 'https://www.goodreads.com' + next_href
            request = scrapy.Request(url=next_page_url)
            yield request

    def parse_books(self, response):
        yield{
            'url': response.url,
            'title':response.xpath('//div[@id="metacol"]/h1[@class="bookTitle"]/text()').exract(),

        } 
</code></pre>

<p>Please suggest what I do so can extract all pages data by running spider once.</p>

<p>I make changes in my code and it works. change is -</p>

<pre><code>request = scrapy.Request(url=next_page_url)
</code></pre>

<p>should be</p>

<pre><code>request = scrapy.Request(next_page_url, self.parse)
</code></pre>

<p>When I comment <code>allowed_domains = ["https://www.goodreads.com"]</code> It works well otherwise there is no data saved in json file.
Can anyone one explain why??  </p>

python web scrapy scrapy-spider