ReactorNotRestartable error in while loop with scrapy
<p>I get <code>twisted.internet.error.ReactorNotRestartable</code> error when I execute following code:</p>

<pre><code>from time import sleep
from scrapy import signals
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from scrapy.xlib.pydispatch import dispatcher

result = None

def set_result(item):
    result = item

while True:
    process = CrawlerProcess(get_project_settings())
    dispatcher.connect(set_result, signals.item_scraped)

    process.crawl('my_spider')
    process.start()

    if result:
        break
    sleep(3)
</code></pre>

<p>For the first time it works, then I get error. I create <code>process</code> variable each time, so what's the problem?</p>

<p>By default, <a href="https://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.CrawlerProcess" rel="nofollow"><code>CrawlerProcess</code></a>'s <a href="https://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.CrawlerProcess.start" rel="nofollow"><code>.start()</code></a> will stop the Twisted reactor it creates when all crawlers have finished.</p>

<p>You should call <code>process.start(stop_after_crawl=False)</code> if you create <code>process</code> in each iteration.</p>

<p>Another option is to handle the Twisted reactor yourself and use <a href="https://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.CrawlerRunner" rel="nofollow"><code>CrawlerRunner</code></a>. <a href="https://doc.scrapy.org/en/latest/topics/practices.html#running-multiple-spiders-in-the-same-process" rel="nofollow">The docs have an example</a> on doing that.</p>

python python-2.7 scrapy twisted