How to detect HTTP response status code and set a proxy accordingly in scrapy?
<p>Is there a way to set a new proxy ip (e.g.: from a pool) according to the HTTP response status code?
For example, start up with an IP form an IP list till it gets a 503 response (or another http error code), then use the next one till it gets blockedï¼and so on, something like:</p>

<pre><code>if http_status_code in [403, 503, ..., n]:
    proxy_ip = 'new ip'
    # Then keep using it till it's gets another error code
</code></pre>

<p>Any ideas?</p>

<p>Scrapy has a downloader middleware which is enabled by default to handle proxies. It's called <a href="http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware" rel="nofollow">HTTP Proxy Middleware</a> and what it does is allows you to supply meta key <code>proxy</code> to your <code>Request</code> and use that proxy for this request.</p>

<p>There are few ways of doing this.<br>
First one, straight-forward just use it in your spider code:</p>

<pre><code>def parse(self, response):
    if response.status in range(400, 600):
        return Request(response.url, 
                       meta={'proxy': 'http://myproxy:8010'}
                       dont_filter=True)  # you need to ignore filtering because you already did one request to this url
</code></pre>

<p>Another more elegant way would be to use custom downloader middleware which would handle this for multiple callbacks and keep your spider code cleaner:</p>

<pre><code>from project.settings import PROXY_URL
class MyDM(object):
    def process_response(self, request, response, spider):
        if response.status in range(400, 600):
            logging.debug('retrying [{}]{} with proxy: {}'.format(response.status, response.url, PROXY_URL)
            return Request(response.url, 
                           meta={'proxy': PROXY_URL}
                           dont_filter=True)
        return response
</code></pre>

<p>Note that by default scrapy doesn't let through any response codes other than <code>200</code> ones. Scrapy automatically handles redirect codes <code>300</code> with <code>Redirect middleware</code> and raises request errors on <code>400</code> and <code>500</code> with HttpError middleware. To handle requests other than 200 you need to either:</p>

<p>Specify that in Request Meta:  </p>

<pre><code>Request(url, meta={'handle_httpstatus_list': [404,505]})
# or for all 
Request(url, meta={'handle_httpstatus_all': True})
</code></pre>

<p>Set a project/spider wide parameters:</p>

<pre><code>HTTPERROR_ALLOW_ALL = True  # for all
HTTPERROR_ALLOWED_CODES = [404, 505]  # for specific
</code></pre>

<p>as per <a href="http://doc.scrapy.org/en/latest/topics/spider-middleware.html#httperror-allowed-codes" rel="nofollow">http://doc.scrapy.org/en/latest/topics/spider-middleware.html#httperror-allowed-codes</a></p>

python scrapy