scrapy run spider from path
<p>A number of suggestions on running scrapy suggest doing this in order to start scrapy via script, or to debug in an IDE, etc:</p>

<pre><code>from scrapy import cmdline

cmdline.execute(("scrapy runspider spider-file-name.py").split())
</code></pre>

<p>This works, so long as the script is placed in the project directory, but if not try to give it an absolute or relative path. For example:</p>

<pre><code>import os

from scrapy import cmdline

this_file_path = os.path.dirname(os.path.realpath(__file__))
base_path = this_file_path.replace('bootstrap', '')
full_path = base_path + "path/to/spiders/some-spider.py"
print full_path

cmdline.execute(("scrapy runspider " + full_path).split())
</code></pre>

<p>With this, I get:</p>

<pre><code>2016-09-28 10:49:29 [scrapy] INFO: Scrapy 1.1.2 started (bot: scrapybot)
2016-09-28 10:49:29 [scrapy] INFO: Overridden settings: {}
Usage
=====
  scrapy runspider [options] &lt;spider_file&gt;

spider-main.py: error: Unable to load '/Users/name/intellij-workspace/crawling/scrape/scrape/spiders/some-spider.py': No module named items
</code></pre>

<p>Is there a way to run and debug scrapy spiders from an absolute path? Ideally, I need to have this to debug in an IDE.</p>

<p>It's highly advised to use a distributed crawling software but if you really want to do it like this just for some dirty testing here it is</p>

<pre><code>import subprocess

project_path="/Users/name/intellij-workspace/crawling/scrape"
subprocess.Popen(["scrapy","runspider","scrape/spiders/some-spider.py"],cwd=project_path)
</code></pre>

python scrapy