scrapy list of requests delayed
<p>I need to scrape a list of web pages 10 times at 5 minute intervals.
This is to collect URLs for later scraping.  Another way to look at it is</p>

<pre><code>url_list = []
for i in 1:10 {
   url_list += scrape request
   url_list += scrape request
   url_list += scrape request
   sleep 5 min
}
for site in url_list
    scrape site
</code></pre>

<p>How can I add a delay between the sets, but no delay between the scraping requests?</p>

<p>How can I achieve this?</p>

<p>Thanks</p>

<p>You can use <code>DOWNLOAD_DELAY</code> project settings or <code>download_delay</code> spider class attribute with a number of desired delay in seconds. </p>

<p><a href="http://doc.scrapy.org/en/latest/topics/settings.html#download-delay" rel="nofollow">Official docs on download_delay</a></p>

python scrapy