Is it possible to run just through the item pipeline without crawling with Scrapy?
<p>I have a <code>.jl</code> file with items I've scraped. I have another pipeline now which was not present when I was doing the scraping. Is it possible to run just the pipeline, and have it apply the new pipeline without doing the crawl/scrape again?</p>

<p>Quick answer: Yes.</p>

<p>To bypass the downloader while having other components of scrapy working, you could use a customized downloader middleware which returns <code>Response</code> objects in its <code>process_request</code> method. Check the details: <a href="http://doc.scrapy.org/en/latest/topics/downloader-middleware.html" rel="nofollow">http://doc.scrapy.org/en/latest/topics/downloader-middleware.html</a></p>

<p>But in your case I personally think you could use some simple code to download the <code>.jl</code> file from your local file system. A quick (and full) example:</p>

<pre><code># coding: utf8

import json
import scrapy


class SampleSpider(scrapy.Spider):

    name = 'sample_spider'
    start_urls = [
        'file:///tmp/some_file.jl',
    ]
    custom_settings = {
        'ITEM_PIPELINES': {
            'your_pipeline_here': 100,
        },
    }

    def parse(self, response):
        for line in response.body.splitlines():
            jdata = json.loads(line)
            yield jdata
</code></pre>

<p>Just replace <code>'/tmp/some_file.jl'</code> with your actual path to the file.</p>

python scrapy