ImportError: No module named scrapyproject.settings
<p>I have a scrapy project, the idea is to execute crawler and get the result back. I am using Flask as api end application and also using virtualenvironment.</p>

<pre><code>from scrapy.spiderloader import SpiderLoader
from scrapy.utils.project import get_project_settings

@app.route("/")
def hello():
    settings = get_project_settings()
    loader = SpiderLoader(settings)
    spiders = loader.list()
    // validate spider
    cmd = "scrapy crawl test --nolog --output-format=json -o -"
    proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
    output = proc.stdout.read()
    return Response(output, mimetype='application/json')
</code></pre>

<p>The problem is when the lines below are used inside executing the subprocess</p>

<pre><code>settings = get_project_settings()
loader = SpiderLoader(settings)
spiders = loader.list()
</code></pre>

<p>the result is empty and it throws exceptions</p>

<pre><code>Traceback (most recent call last):
  File "/var/www/projects/ENV/bin/scrapy", line 11, in &lt;module&gt;
    sys.exit(execute())
  File "/var/www/projects/ENV/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 108, in execute
    settings = get_project_settings()
  File "/var/www/projects/ENV/local/lib/python2.7/site-packages/scrapy/utils/project.py", line 60, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "/var/www/projects/ENV/local/lib/python2.7/site-packages/scrapy/settings/__init__.py", line 282, in setmodule
    module = import_module(module)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named scrapyproject.settings
</code></pre>

<p>When the code to get the spiders list is commented it executes the test spider without any issues.</p>

<p>It is worth to note that the code to get all the spiders works fine.</p>

<pre><code>@app.route("/spiders")
def spiders():
    settings = get_project_settings()
    loader = SpiderLoader(settings)
    jsonify(loader.list())
</code></pre>

<p>I have also looked into a few questions / answers mentioning it may be problem with virtualenvironment. But wondering why it only behaves when certain code is executed.</p>

<p>If you are wondering why I have not used / executed <a href="http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script" rel="nofollow">scrapy via script</a> . What I need is to get the output without saving to any file system or db or anything via pipeline.</p>

<p><a href="https://twitter.com/chrisguitarguy/status/775155136452624384" rel="nofollow">Christopher Davis</a> mentioned to pass the environment copy to subprocess.</p>

<pre><code>proc = subprocess.Popen(..., env=os.environ.copy())
</code></pre>

<p>Which actually opened my eye to notice the introduction of <code>SCRAPY_SETTINGS_MODULE</code> once the <code>get_project_settings()</code> is called, which in turn when running the command to execute <code>scrapy</code> feels it is loaded. But not truly loaded. </p>

<p>So the solution is remove the environment variable as </p>

<pre><code>del os.environ['SCRAPY_SETTINGS_MODULE']
</code></pre>

<p>before you call the subprocess. </p>

<p>Hope that will help someone and save their time.</p>

<p>Thank you</p>

python scrapy scrapy-spider