How to persist state when the crawler dies abruptly?
<p>this quetion is in reference to <a href="http://stackoverflow.com/questions/39211490/scrapy-spider-does-not-store-state-persistent-state">Scrapy spider does not store state (persistent state)</a></p>

<p>I have followed the following link to persist state of the crawler <a href="http://doc.scrapy.org/en/latest/topics/jobs.html" rel="nofollow">http://doc.scrapy.org/en/latest/topics/jobs.html</a></p>

<p>Now this works perfectly fine when the crawler ends properly, with a interrupt or a Ctrl+C.</p>

<p>I have noticed that the spider does not shut down properly when</p>

<ol>
<li>You hit Ctrl +C multiple times.</li>
<li>The server capacity is hit.</li>
<li>Any other reason due to which it ends abruptly</li>
</ol>

<p>The spider when runs again , shuts itself down on the first url crawled.</p>

<p>How to achieve a persistent state of the crawler when something like above happens?
Cause or else it ends up crawling the whole bunch of urls again.</p>

<p>Logs when the spider runs again: </p>

<pre><code>2016-08-30 08:14:11 [scrapy] INFO: Scrapy 1.1.2 started (bot: maxverstappen)
2016-08-30 08:14:11 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'maxverstappen.spiders', 'SPIDER_MODULES': ['maxverstappen.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'maxverstappen'}
2016-08-30 08:14:11 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.spiderstate.SpiderState']
2016-08-30 08:14:11 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-08-30 08:14:11 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-08-30 08:14:12 [scrapy] INFO: Enabled item pipelines:
['maxverstappen.pipelines.MaxverstappenPipeline']
2016-08-30 08:14:12 [scrapy] INFO: Spider opened
2016-08-30 08:14:12 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-08-30 08:14:12 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024
2016-08-30 08:14:12 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.inautonews.com/robots.txt&gt; (referer: None)
2016-08-30 08:14:12 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.thecheckeredflag.com/robots.txt&gt; (referer: None)
2016-08-30 08:14:12 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.inautonews.com/&gt; (referer: None)
2016-08-30 08:14:12 [scrapy] DEBUG: Crawled (200) &lt;GET http://www.thecheckeredflag.com/&gt; (referer: None)
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered duplicate request: &lt;GET http://www.inautonews.com/&gt; - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.newsnow.co.uk': &lt;GET http://www.newsnow.co.uk/h/Life+&amp;+Style/Motoring&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.americanmuscle.com': &lt;GET http://www.americanmuscle.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.extremeterrain.com': &lt;GET http://www.extremeterrain.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.autoanything.com': &lt;GET http://www.autoanything.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.bmwcoop.com': &lt;GET http://www.bmwcoop.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.automotorblog.com': &lt;GET http://www.automotorblog.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'twitter.com': &lt;GET https://twitter.com/inautonews&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.facebook.com': &lt;GET https://www.facebook.com/inautonews&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'plus.google.com': &lt;GET https://plus.google.com/+Inautonewsplus&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.histats.com': &lt;GET http://www.histats.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.hamiltonf1site.com': &lt;GET http://www.hamiltonf1site.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.joshwellsracing.com': &lt;GET http://www.joshwellsracing.com/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.jensonbuttonfan.net': &lt;GET http://www.jensonbuttonfan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.fernandoalonsofan.net': &lt;GET http://www.fernandoalonsofan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.markwebberfan.net': &lt;GET http://www.markwebberfan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.felipemassafan.net': &lt;GET http://www.felipemassafan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.nicorosbergfan.net': &lt;GET http://www.nicorosbergfan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.nickheidfeldfan.net': &lt;GET http://www.nickheidfeldfan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.lewishamiltonblog.net': &lt;GET http://www.lewishamiltonblog.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.timoglockfan.net': &lt;GET http://www.timoglockfan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.jarnotrullifan.net': &lt;GET http://www.jarnotrullifan.net/&gt;
2016-08-30 08:14:12 [scrapy] DEBUG: Filtered offsite request to 'www.brunosennafan.net': &lt;GET http://www.brunosennafan.net/&gt;
2016-08-30 08:14:12 [scrapy] INFO: Closing spider (finished)
2016-08-30 08:14:12 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 896,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 4,
 'downloader/response_bytes': 35353,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 4,
 'dupefilter/filtered': 149,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 8, 30, 8, 14, 12, 724932),
 'log_count/DEBUG': 28,
 'log_count/INFO': 7,
 'offsite/domains': 22,
 'offsite/filtered': 23,
 'request_depth_max': 1,
 'response_received_count': 4,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/disk': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/disk': 2,
 'start_time': datetime.datetime(2016, 8, 30, 8, 14, 12, 13456)}
2016-08-30 08:14:12 [scrapy] INFO: Spider closed (finished)
</code></pre>

<p>One way of doing this is seperating discovery and consumer logic by having two spiders. One to discovery product urls another to consume those urls and return results for every one of them. If for some reason the consumer dies mid run it can easily resume the crawl because discovery queue was not affected by this crash.</p>

<p>There's already a great tool for scrapy by the scrapy guys which does exactly that. It's called <a href="https://github.com/scrapinghub/frontera" rel="nofollow">Frontera</a></p>

<blockquote>
  <p>Frontera is a web crawling framework consisting of crawl frontier, and
  distribution/scaling primitives, allowing to build a large scale
  online web crawler.</p>
  
  <p>Frontera takes care of the logic and policies to follow during the
  crawl. It stores and prioritises links extracted by the crawler to
  decide which pages to visit next, and capable of doing it in
  distributed manner.</p>
</blockquote>

<p>It sounds complicated but it's pretty straigh-forward. However if you run something small scale and one off you might just want to approach this manually. You can run discovery spider and output results in json then parse that json in a persistant manner(i.e. popping values out of it) in your consumer spider.</p>

python scrapy web-crawler