Use scrapy as an item generator
<p>I have an existing script (main.py) that requires data to be scraped.</p>

<p>I started a scrapy project for retrieving this data. Now, is there any way main.py can retrieve the data from scrapy as an Item generator, rather than persisting data using the Item pipeline?</p>

<p>Something like this would be really convenient, but I couldn't find out how to do it, if it's feasible at all.</p>

<pre><code>for item in scrapy.process():
</code></pre>

<p>I found a potential solution there: <a href="https://tryolabs.com/blog/2011/09/27/calling-scrapy-python-script/" rel="nofollow">https://tryolabs.com/blog/2011/09/27/calling-scrapy-python-script/</a>, using multithreading's queues. </p>

<p>Even though I understand this behaviour is not compatible with distributed crawling, which is what Scrapy is intended for, I'm still a little surprised that you wouldn't have this feature available for smaller projects.</p>

<p>You could send json data out from the crawler and grab the results. It can be done as follows:</p>

<p>Having the spider:</p>

<pre><code>class MySpider(scrapy.Spider):
    # some attributes
    accomulated=[]

    def parse(self, response):
        # do your logic here
        page_text = response.xpath('//text()').extract()
        for text in page_text:
            if conditionsAreOk( text ):
                self.accomulated.append(text)

    def closed( self, reason ):
        # call when the crawler process ends
        print JSON.dumps(self.accomulated)
</code></pre>

<p>Write a runner.py script like:</p>

<pre><code>import sys
from twisted.internet import reactor

import scrapy

from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging    
from scrapy.utils.project import get_project_settings

from spiders import MySpider 

def main(argv): 

    url = argv[0]

    configure_logging({'LOG_FORMAT': '%(levelname)s: %(message)s', 'LOG_ENABLED':False })
    runner = CrawlerRunner( get_project_settings() )

    d = runner.crawl( MySpider, url=url)

    # For Multiple in the same process
    #
    # runner.crawl('craw')
    # runner.crawl('craw2')
    # d = runner.join()

    d.addBoth(lambda _: reactor.stop())
    reactor.run() # the script will block here until the crawling is finished


if __name__ == "__main__":
   main(sys.argv[1:])
</code></pre>

<p>And then call it from your main.py as:</p>

<pre><code>import json, subprocess, sys, time

def main(argv): 

    # urlArray has http:// or https:// like urls
    for url in urlArray:    
        p = subprocess.Popen(['python', 'runner.py', url ], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = p.communicate()

        # do something with your data
        print out
        print json.loads(out)

        # This just helps to watch logs
        time.sleep(0.5)


if __name__ == "__main__":
   main(sys.argv[1:])
</code></pre>

<p><strong>Note</strong>  </p>

<p>This is not the best way of using Scrapy as you know, but for fast results which do not require a complex post processing, this solution can provide what you need.</p>

<p>I hope it helps.</p>

python scrapy scrapy-pipeline