Scrapy not calling parse function with start_requests
<p>I am fairly new to Python and Scrapy, but something just seems not right. According to documentation and example, re-implementing <em>start_requests</em> function will cause Scrapy to use return of <em>start_requests</em> instead of <em>start_urls</em> array variable.</p>

<p>Everything works fine with <em>start_urls</em>, but when I add <em>start_requests</em>, it does not go into <em>parse</em> function. Documentation states that <em>parse</em> method is</p>

<blockquote>
  <p>the default callback used by Scrapy to process downloaded responses,
  when their requests donât specify a callback</p>
</blockquote>

<p>but the <em>parse</em> is never executed, tracing my logger prints.</p>

<p>Here is my code, it is very short as I am just toying around with it.</p>

<pre><code>class Crawler(scrapy.Spider):

    name = 'Hearthpwn'
    allowed_domains = ['hearthpwn.com']
    storage_dir = 'C:/Users/Michal/PycharmProjects/HearthpwnCrawler/'
    start_urls = ['http://www.hearthpwn.com/decks/645987-nzoth-warrior']

    def start_requests(self):

        logging.log(logging.INFO, "Loading requests")
        yield Request(url='http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter')

    def parse(self, response):

        logging.log(logging.INFO, "parsing response")

        filename = response.url.split("/")[-1] + '.html'
        with open('html/' + filename, 'wb') as f:
            f.write(response.body)

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(Crawler)
process.start()
</code></pre>

<p>And print of the console:</p>

<pre><code>2016-10-12 15:33:39 [scrapy] INFO: Scrapy 1.2.0 started (bot: scrapybot)
2016-10-12 15:33:39 [scrapy] INFO: Overridden settings: {'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'}
2016-10-12 15:33:39 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.logstats.LogStats']
2016-10-12 15:33:39 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-10-12 15:33:39 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-10-12 15:33:39 [scrapy] INFO: Enabled item pipelines:
[]
2016-10-12 15:33:39 [scrapy] INFO: Spider opened
2016-10-12 15:33:39 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-10-12 15:33:39 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-10-12 15:33:39 [root] INFO: Loading requests
2016-10-12 15:33:41 [scrapy] DEBUG: Redirecting (302) to &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter?cookieTest=1&gt; from &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter&gt;
2016-10-12 15:33:41 [scrapy] DEBUG: Redirecting (302) to &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter&gt; from &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter?cookieTest=1&gt;
2016-10-12 15:33:41 [scrapy] DEBUG: Filtered duplicate request: &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter&gt; - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2016-10-12 15:33:41 [scrapy] INFO: Closing spider (finished)
2016-10-12 15:33:41 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 655,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 1248,
 'downloader/response_count': 2,
 'downloader/response_status_count/302': 2,
 'dupefilter/filtered': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2016, 10, 12, 13, 33, 41, 740724),
 'log_count/DEBUG': 4,
 'log_count/INFO': 8,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2016, 10, 12, 13, 33, 39, 441736)}
2016-10-12 15:33:41 [scrapy] INFO: Spider closed (finished)
</code></pre>

<p>Thanks for any leads.</p>

<pre><code>2016-10-12 15:33:41 [scrapy] DEBUG: Redirecting (302) to &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter?cookieTest=1&gt; from &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter&gt;
2016-10-12 15:33:41 [scrapy] DEBUG: Redirecting (302) to &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter&gt; from &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter?cookieTest=1&gt;
2016-10-12 15:33:41 [scrapy] DEBUG: Filtered duplicate request: &lt;GET http://www.hearthpwn.com/decks/646673-s31-legend-2eu-3asia-smorc-hunter&gt; - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
</code></pre>

<p>What happens here is that the website redirects you several times and you end up crawling the same url twice because of it. Scrapy spider by default has middleware which filters out duplicate requests, so you need to set parameter <code>dont_filter</code> to <code>True</code> when creating the Request object to ignore this middleware.</p>

<p>e.g.:</p>

<pre><code>def start_requests(self):
    yield ('http://scrapy.org', dont_filter=True) 
</code></pre>

python request scrapy