Scrapy API - Spider class init argument turned to None
<p>After a fresh install of Miniconda 64-bit exe installer for Windows and Python 2.7 on Windows 7, through which I get Scrapy, here is what is installed:</p>

<ul>
<li>Python 2.7.12</li>
<li>Scrapy 1.1.1</li>
<li>Twisted 16.4.1</li>
</ul>

<p>This minimal code, run from "python scrapy_test.py" (using Scrapy API):</p>

<pre><code>#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-

import scrapy.spiders.crawl
import scrapy.crawler
import scrapy.utils.project

class MySpider(scrapy.spiders.crawl.CrawlSpider) :
    name = "stackoverflow.com"
    allowed_domains = ["stackoverflow.com"]
    start_urls = ["http://stackoverflow.com/"]
    download_delay = 1.5

    def __init__(self, my_arg = None) :
        print "def __init__"

        self.my_arg = my_arg
        print "self.my_arg"
        print self.my_arg

    def parse(self, response) :
        pass

def main() :
    my_arg = "Value"

    process = scrapy.crawler.CrawlerProcess(scrapy.utils.project.get_project_settings())
    process.crawl(MySpider(my_arg))
    process.start()

if __name__ == "__main__" :
    main()
</code></pre>

<p>gives this ouput:</p>

<pre><code>[scrapy] INFO: Scrapy 1.1.1 started (bot: scrapy_project)
[scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'scrapy_project.spiders', 'SPIDER_MODULES': ['scrapy_project.spiders'], 'ROBOTSTXT_OBEY': True, 'BOT_NAME': 'scrapy_project'}
def __init__
self.my_arg
Value
[scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
def __init__
self.my_arg
None
[...]
</code></pre>

<p>Notice how the init method was run twice and how the stored argument got turned to None after the second run, which is not what I want. Is this supposed to happen??</p>

<p>If I change:</p>

<pre><code>def __init__(self, my_arg = None) :
</code></pre>

<p>to:</p>

<pre><code>def __init__(self, my_arg) :
</code></pre>

<p>the output is:</p>

<pre><code>[...]
Unhandled error in Deferred:
[twisted] CRITICAL: Unhandled error in Deferred:


Traceback (most recent call last):
  File "scrapy_test.py", line 28, in main
    process.crawl(MySpider(my_arg))
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 163, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 167, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\twisted\internet\defer.py", line 1331, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- &lt;exception caught here&gt; ---
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\twisted\internet\defer.py", line 1185, in _inlineCallbacks
    result = g.send(result)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 90, in crawl
    six.reraise(*exc_info)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 71, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 94, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\crawl.py", line 96, in from_crawler
    spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\__init__.py", line 50, in from_crawler
    spider = cls(*args, **kwargs)
exceptions.TypeError: __init__() takes exactly 2 arguments (1 given)
[twisted] CRITICAL:
Traceback (most recent call last):
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\twisted\internet\defer.py", line 1185, in _inlineCallbacks
    result = g.send(result)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 90, in crawl
    six.reraise(*exc_info)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 71, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\crawler.py", line 94, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\crawl.py", line 96, in from_crawler
    spider = super(CrawlSpider, cls).from_crawler(crawler, *args, **kwargs)
  File "C:\Users\XYZ\Miniconda2\lib\site-packages\scrapy\spiders\__init__.py", line 50, in from_crawler
    spider = cls(*args, **kwargs)
TypeError: __init__() takes exactly 2 arguments (1 given)
</code></pre>

<p>No clue how to get around this problem. Any idea?</p>

<p>Here is the method definition for <a href="http://doc.scrapy.org/en/latest/topics/api.html#scrapy.crawler.CrawlerProcess.crawl" rel="nofollow"><code>scrapy.crawler.CrawlerProcess.crawl()</code></a>:</p>

<blockquote>
  <p><code>crawl(crawler_or_spidercls, *args, **kwargs)</code></p>
  
  <ul>
  <li><strong>crawler_or_spidercls</strong> (<code>Crawler</code> instance, <code>Spider</code> subclass or string) â already created crawler, or a spider class or spiderâs name
  inside the project to create it</li>
  <li><strong>args</strong> (<em>list</em>) â arguments to initialize the spider</li>
  <li><strong>kwargs</strong> (<em>dict</em>) â keyword arguments to initialize the spider</li>
  </ul>
</blockquote>

<hr>

<p>This means you should be passing the name of your <code>Spider</code> separately from the <code>kwargs</code> needed to initialize said <code>Spider</code>, like so:</p>

<pre><code>process.crawl(MySpider, my_arg = 'Value')
</code></pre>

python scrapy