Running Multiple spiders in scrapy for 1 website in parallel?
<p>I want to crawl a website with 2 parts and my script is not as fast as i need.</p>

<p>is it possible to luanch 2 spiders, one for scraping the first part and the second one for the second part? </p>

<p>i tried to have 2 diffrent classes, and run them</p>

<pre><code>scrapy crwal firstSpider
scrapy crawl secondSpider
</code></pre>

<p>but i think that it is not smart.</p>

<p>i read the <a href="http://scrapyd.readthedocs.io/en/latest/api.html" rel="nofollow">documentation of scrapyd</a> but i don't know if it's good for my case.</p>

<p>I think what you are looking for is something like this:</p>

<pre><code>import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider1(scrapy.Spider):
    # Your first spider definition
    ...

class MySpider2(scrapy.Spider):
    # Your second spider definition
    ...

process = CrawlerProcess()
process.crawl(MySpider1)
process.crawl(MySpider2)
process.start() # the script will block here until all crawling jobs are finished
</code></pre>

<p>You can read more at: <a href="http://doc.scrapy.org/en/1.1/topics/practices.html#running-multiple-spiders-in-the-same-process" rel="nofollow">running-multiple-spiders-in-the-same-process</a>.</p>

python web-scraping scrapy web-crawler scrapy-spider