Scrapy - running spider from a python script
<p>I am trying to run scrapy from a <code>python</code> script according to documentation <a href="http://scrapy.readthedocs.io/en/0.16/topics/practices.html" rel="nofollow">http://scrapy.readthedocs.io/en/0.16/topics/practices.html</a></p>

<pre><code>def CrawlTest():

    spider = PitchforkSpider(domain='"pitchfork.com"')
    crawler = Crawler(Settings())
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()
    log.start()
    reactor.run() # the script will block here
</code></pre>

<p>but when I run it, I get the following error:</p>

<pre><code>AttributeError: 'Settings' object has no attribute 'update_settings'
</code></pre>

<p>has something been deprecated? what is wrong here?</p>

<p>my version is <code>Scrapy 1.1.2</code></p>

<p>You are looking at Scrapy 0.16 docs but using Scrapy 1.1.2. </p>

<p>Here is the <a href="http://scrapy.readthedocs.io/en/latest/topics/practices.html#run-scrapy-from-a-script" rel="nofollow">correct documentation page</a>.</p>

<p>FYI, you should now be using <code>CrawlerProcess</code> or <code>CrawlerRunner</code>.</p>

python web-scraping scrapy