Only cache 200 with scrapy
<p>I am using <code>scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware</code> to cache scrapy requests. I'd like it to only cache if status is 200. Is that the default behavior? Or do I need to specify <code>HTTPCACHE_IGNORE_HTTP_CODES</code> to be everything <em>except</em> 200?</p>

<p>Yes, by default <code>HttpCacheMiddleware</code> run a <code>DummyPolicy</code> for the requests. It pretty much does nothing special on it's own so you need to set <code>HTTPCACHE_IGNORE_HTTP_CODES</code> to everything except 200.</p>

<p><a href="https://github.com/scrapy/scrapy/blob/master/scrapy/extensions/httpcache.py#L18" rel="nofollow">Here's the source for the DummyPolicy</a>
And these are the lines that actually matter:</p>

<pre><code>class DummyPolicy(object):

    def __init__(self, settings):
        self.ignore_http_codes = [int(x) for x in settings.getlist('HTTPCACHE_IGNORE_HTTP_CODES')]

    def should_cache_response(self, response, request):
        return response.status not in self.ignore_http_codes
</code></pre>

<p>So in reality you can also extend this and override <code>should_cache_response()</code> to something that would check for <code>200</code> explicitly, i.e. <code>return response.status == 200</code> and then set it as your cache policy via <a href="http://doc.scrapy.org/en/latest/topics/downloader-middleware.html?highlight=ignore_http_codes#httpcache-policy" rel="nofollow"><code>HTTPCACHE_POLICY</code> setting</a>.</p>

python scrapy