CrawlerProcess vs CrawlerRunner
<p><a href="http://scrapy.readthedocs.io/en/latest/topics/practices.html#run-scrapy-from-a-script" rel="nofollow">Scrapy 1.x documentation</a> explains that there are two ways to <em>run a Scrapy spider from a script</em>:</p>

<ul>
<li>using <a href="http://scrapy.readthedocs.io/en/latest/topics/api.html#scrapy.crawler.CrawlerProcess" rel="nofollow"><code>CrawlerProcess</code></a></li>
<li>using <a href="http://scrapy.readthedocs.io/en/latest/topics/api.html#scrapy.crawler.CrawlerRunner" rel="nofollow"><code>CrawlerRunner</code></a></li>
</ul>

<p>What is the difference between the two? When should I use "process" and when "runner"?</p>

<p>CrawlerRunner:</p>

<blockquote>
  <p>This class shouldnât be needed (since Scrapy is responsible of using it accordingly) unless writing scripts that manually handle the crawling process. See Run Scrapy from a script for an example.</p>
</blockquote>

<p>CrawlerProcess:</p>

<blockquote>
  <p>This utility should be a better fit than CrawlerRunner if you arenât running another Twisted reactor within your application.</p>
</blockquote>

<p>It sounds like the CrawlerProcess is what you want unless you're adding your crawlers to an existing Twisted application.</p>

python web-scraping scrapy