How to add instance variable to Scrapy CrawlSpider?
<p>I am running a CrawlSpider and I want to implement some logic to stop following some of the links in mid-run, by passing a function to <code>process_request</code>.</p>

<p>This function uses the spider's <strong>class</strong> variables in order to keep track of the current state, and depending on it (and on the referrer URL), links get dropped or continue to be processed:</p>

<pre><code>class BroadCrawlSpider(CrawlSpider):
    name = 'bitsy'
    start_urls = ['http://scrapy.org']
    foo = 5

    rules = (
        Rule(LinkExtractor(), callback='parse_item', process_request='filter_requests', follow=True),
    )

    def parse_item(self, response):
        &lt;some code&gt;

    def filter_requests(self, request):
        if self.foo == 6 and request.headers.get('Referer', None) == someval:
             raise IgnoreRequest("Ignored request: bla %s" % request)
        return request
</code></pre>

<p>I think that if I were to run several spiders on the same machine, they would all use the same <strong>class</strong> variables which is not my intention.</p>

<p>Is there a way to add <strong>instance</strong> variables to CrawlSpiders? Is only a single instance of the spider created when I run Scrapy?</p>

<p>I could probably work around it with a dictionary with values per process ID, but that will be ugly...</p>

<p>I think <a href="http://doc.scrapy.org/en/latest/topics/spiders.html#spider-arguments" rel="nofollow">spider arguments</a> would be the solution in your case.</p>

<p>When invoking scrapy like <code>scrapy crawl some_spider</code>, you could add arguments like <code>scrapy crawl some_spider -a foo=bar</code>, and the spider would receive the values via its constructor, e.g.:</p>

<pre><code>class SomeSpider(scrapy.Spider):
    def __init__(self, foo=None, *args, **kwargs):
        super(SomeSpider, self).__init__(*args, **kwargs)
        # Do something with foo
</code></pre>

<p>What's more, as <a href="https://github.com/scrapy/scrapy/blob/master/scrapy/spiders/__init__.py#L30" rel="nofollow"><code>scrapy.Spider</code> actually sets all additional arguments as instance attributes</a>, you don't even need to explicitly override the <code>__init__</code> method but just access the <code>.foo</code> attribute. :)</p>

python scrapy scrapy-spider