Stop Scrapy spider when it meets a specified URL
<p>This question is very similar to <a href="http://stackoverflow.com/questions/4448724/force-my-scrapy-spider-to-stop-crawling">Force my scrapy spider to stop crawling</a> and some others asked several years ago. However, the suggested solutions there are either dated for Scrapy 1.1.1 or not precisely relevant. <strong>The task is to close the spider when it reaches a certain URL.</strong> You definitely need this when crawling a news website for your media project, for instance.</p>

<p>Among the settings <code>CLOSESPIDER_TIMEOUT</code> <code>CLOSESPIDER_ITEMCOUNT</code> <code>CLOSESPIDER_PAGECOUNT</code> <code>CLOSESPIDER_ERRORCOUNT</code>, item count and page count options are close but not enough since you never know the number of pages or items.</p>

<p>The <code>raise CloseSpider(reason='some reason')</code> exception seems to do the job but so far it does it in a bit weird way. I follow the <a href="https://www.packtpub.com/big-data-and-business-intelligence/learning-scrapy" rel="nofollow">âLearning Scrapyâ</a> textbook and the structure of my code looks like the one in the book.</p>

<p>In <code>items.py</code> I make a list of items:</p>

<pre><code>class MyProjectItem(scrapy.Item):

    Headline = scrapy.Field()
    URL = scrapy.Field()
    PublishDate = scrapy.Field()
    Author = scrapy.Field()

    pass
</code></pre>

<p>In <code>myspider.py</code>  I use the <code>def start_requests()</code> method where the spider takes the pages to process, parse each index page in <code>def parse()</code>, and  specify the XPath for each item in <code>def parse_item()</code>:</p>

<pre><code>class MyProjectSpider(scrapy.Spider):
    name = 'spidername'
    allowed_domains = ['domain.name.com']


    def start_requests(self):

        for i in range(1,3000): 
            yield scrapy.Request('http://domain.name.com/news/index.page'+str(i)+'.html', self.parse)


    def parse(self, response):

        urls = response.xpath('XPath for the URLs on index page').extract()           
        for url in urls:
            # The urls are absolute in this case. Thereâs no need to use urllib.parse.urljoin()
            yield scrapy.Request(url, callback=self.parse_item)


    def parse_item(self, response):

        l = ItemLoader(item=MyProjectItem(), response=response)

        l.add_xpath('Headline', 'XPath for Headline')
        l.add_value('URL', response.url)
        l.add_xpath ('PublishDate', 'XPath for PublishDate')
        l.add_xpath('Author', 'XPath for Author')

        return l.load_item()
</code></pre>

<p>If <code>raise CloseSpider(reason='some reason')</code> exception is placed in <code>def parse_item()</code>, it still scrapes a number of items before it finally stops:</p>

<pre><code>if l.get_output_value('URL') == 'http://domain.name.com/news/1234567.html':
    raise CloseSpider('No more news items.')
</code></pre>

<p>If itâs placed in <code>def parse()</code> method to stop when the specific URL is reached, it stops after grabbing only the first item from the index page which contains that specific URL: </p>

<pre><code>def parse(self, response):       

    most_recent_url_in_db = 'http://domain.name.com/news/1234567.html '
    urls = response.xpath('XPath for the URLs on index page').extract()

    if most_recent_url_in_db not in urls:
        for url in urls:
            yield scrapy.Request(url, callback=self.parse_item)
    else:
        for url in urls[:urls.index(most_recent_url_in_db)]:
            yield scrapy.Request(url, callback=self.parse_item)
        raise CloseSpider('No more news items.')
</code></pre>

<p>For example, if you have 5 index pages (each of them has 25 item URLs) and <code>most_recent_url_in_db</code> is on page 4, it means that youâll have all items from pages 1-3 and only the first item from page 4. Then the spider stops. If <code>most_recent_url_in_db</code> is number 10 in the list, items 2-9 from index page 4 wonât appear in your database.</p>

<p>The âhackyâ tricks with <code>crawler.engine.close_spider()</code> suggested in many cases or the ones shared in <a href="http://stackoverflow.com/questions/9699049/how-do-i-stop-all-spiders-and-the-engine-immediately-after-a-condition-in-a-pipe">How do I stop all spiders and the engine immediately after a condition in a pipeline is met?</a> donât seem to work.</p>

<p>What should be the method to properly complete this task?</p>

<p>When you raise <code>close_spider() exception</code>, ideal assumption is that scrapy should stop immediately, abandoning all other activity (any future page requests, any processing in pipeline ..etc)</p>

<p>but this is not the case, When you raise <code>close_spider() exception</code>, scrapy will try to close it's current operation's <strong>gracefully</strong> , meaning it will stop the current request but it will wait for any other request pending in any of the queues( there are multiple queues!)</p>

<p>(i.e. if you are not overriding default settings and have more than 16 start urls, scrapy make 16 requests at a time)</p>

<p>Now if you want to stop spider as soon as you Raise <code>close_spider() exception</code>, you will want to clear three Queues</p>

<p>-- At spider middleware level ---</p>

<ul>
<li>spider.crawler.engine.slot.scheduler.mqs  -> Memory Queue future request</li>
<li>spider.crawler.engine.slot.inprogress  -> Any In-progress Request</li>
</ul>

<p>-- download middleware level ---</p>

<ul>
<li>spider.requests_queue -> pending Request in Request queue</li>
</ul>

<p>flush all this queues by overriding proper middle-ware to prevent scrapy from visiting any further pages</p>

python scrapy