Python Scrapy - Execute code after spider exits
<p>I am not able to find an answer for that question. How can I execute a python code after a scrapy spider exits:</p>

<p>I did the following inside the function which parses the response (def parse_item(self, response):) :
 self.my_function()
Than I defined my_function(), but the problem is that it is still inside the loop of the spider. My main idea is to execute a given code in a function outside the spider's loop with the gathered data.
Thanks.</p>

<p>Use the function <a href="http://doc.scrapy.org/en/latest/topics/signals.html#spider-closed" rel="nofollow">closed</a> of the Scrapy class as follows:</p>

<pre><code>class MySpider(scrapy.Spider):
    # some attributes
    spider_attr=[]

    def parse(self, response):
        # do your logic here
        # page_text = response.xpath('//text()').extract()
        self.spider_attr.append(whatever)

    def closed( self, reason ):
        # will be called when the crawler process ends
        # any code 
        # do something with collected data 
        for i in self.spider_attr: 
            print i
</code></pre>

python scrapy