Scrapy - Filtered duplicate request
<p>I'm working with scrapy. I want to loop through a db table and grab the starting page for each scrape (random_form_page), then yield a request for each start page. Please note that I am hitting an api to get a proxy with the initial request. I want to set up each request to have its own proxy, so using the callback model I have:   </p>

<pre><code>def start_requests(self):
    for x in xrange(8): 
        random_form_page = session.query(....

        PR = Request(
            'htp://my-api',
            headers=self.headers,
            meta={'newrequest': Request(random_form_page,  headers=self.headers)},
            callback=self.parse_PR
        )
        yield PR
</code></pre>

<p>I notice:</p>

<pre><code>[scrapy] DEBUG: Filtered duplicate request: &lt;GET 'htp://my-api'&gt; - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
</code></pre>

<p>In my code I can see that although it loops through 8 times it only yields a request for the first page. The others I assume are being filtered out. I've looked at <a href="http://doc.scrapy.org/en/latest/topics/settings.html#dupefilter-class" rel="nofollow">http://doc.scrapy.org/en/latest/topics/settings.html#dupefilter-class</a> but still unsure how to turn off this filtering action. How can I turn off the filtering?</p>

<p>use </p>

<blockquote>
  <p>dont_filter = True in Request object</p>
</blockquote>

<pre><code>def start_requests(self):
    for x in xrange(8): 
        random_form_page = session.query(....

        PR = Request(
            'htp://my-api',
            headers=self.headers,
            meta={'newrequest': Request(random_form_page,  headers=self.headers)},
            callback=self.parse_PR,
            dont_filter = True
        )
        yield PR
</code></pre>

python scrapy