Python scrapy ReactorNotRestartable substitute
<p>I have been trying to make an app in Python using <code>Scrapy</code> that has the following functionality:</p>

<ul>
<li>A <strong><em>rest api</em></strong> (I had made that using flask) listens to all requests to crawl/scrap and return the response after crawling.(the crawling part is short enough, so the connection can be keep-alive till crawling gets completed.)</li>
</ul>

<p>I am able to do this using the following code:</p>

<pre><code>items = []
def add_item(item):
    items.append(item)

# set up crawler
crawler = Crawler(SpiderClass,settings=get_project_settings())
crawler.signals.connect(add_item, signal=signals.item_passed)

# This is added to make the reactor stop, if I don't use this, the code stucks at reactor.run() line.
crawler.signals.connect(reactor.stop, signal=signals.spider_closed) #@UndefinedVariable 
crawler.crawl(requestParams=requestParams)
# start crawling 
reactor.run() #@UndefinedVariable
return str(items)
</code></pre>

<p>Now the problem I am facing is after making the reactor stop (which seems necessary to me since I don't want to stuck to the <code>reactor.run()</code>). I couldn't accept the further request after first request. After first request gets completed, I got the following error:</p>

<pre><code>Traceback (most recent call last):
  File "c:\python27\lib\site-packages\flask\app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "c:\python27\lib\site-packages\flask\app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "c:\python27\lib\site-packages\flask\app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "F:\my_workspace\jobvite\jobvite\com\jobvite\web\RequestListener.py", line 38, in submitForm
    reactor.run() #@UndefinedVariable
  File "c:\python27\lib\site-packages\twisted\internet\base.py", line 1193, in run
    self.startRunning(installSignalHandlers=installSignalHandlers)
  File "c:\python27\lib\site-packages\twisted\internet\base.py", line 1173, in startRunning
    ReactorBase.startRunning(self)
  File "c:\python27\lib\site-packages\twisted\internet\base.py", line 684, in startRunning
    raise error.ReactorNotRestartable()
ReactorNotRestartable
</code></pre>

<p>Which is obvious, since we can not restart the reactor.</p>

<p>So my questions are:</p>

<p>1) How could I provide support for the next requests to crawl? </p>

<p>2) Is there any way to move to next line after reactor.run() without stopping it?</p>

<p>I recommend you using a queue system like <a href="http://python-rq.org/" rel="nofollow">Rq</a> (for simplicity, but there are few others).<br>
You could have a craw function:</p>

<pre><code>from twisted.internet import reactor
import scrapy
from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings

from spiders import MySpider

def runCrawler(url, keys, mode, outside, uniqueid): 

    runner = CrawlerRunner( get_project_settings() )

    d = runner.crawl( MySpider, url=url, param1=value1, ... )

    d.addBoth(lambda _: reactor.stop())
    reactor.run()
</code></pre>

<p>Then in your main code, use the Rq queue in order to collect crawler executions:</p>

<pre><code># other imports
pool = redis.ConnectionPool( host=REDIS_HOST, port=REDIS_PORT, db=your_redis_db_number)
redis_conn =redis.Redis(connection_pool=pool)  

q = Queue('parse', connection=redis_conn)

# urlSet is a list of http:// or https:// like url's
for url in urlSet:
    job = q.enqueue(runCrawler, url, param1, ... , timeout=600 )
</code></pre>

<p>Do not forget to start a rq worker process, working for the same queue name (here <strong>parse</strong>).  For example, execute in a terminal session:</p>

<pre><code>rq worker parse
</code></pre>

python flask scrapy reactor twisted.internet