Yielding multiple starting point requests with scrapy
<p>I am working with the following code ( simplified):</p>

<pre><code>def parse(self, response):

    print('hello')
    for x in xrange(8): 
        print x

        random_form_page = session.query(....


        PR = Request(
            'htp://my-api',
            headers=self.headers,
            meta={'newrequest': Request(random_form_page,  headers=self.headers)},
            callback=self.parse_PR
        )
        yield PR
</code></pre>

<p>I want to loop through a db table and grab the starting page for each scrape (random_form_page), then yield a request for each start page. In my code I can see that although it loops through 8 times it only yields a request for the first start page. What am I doing wrong?</p>

<p>You should be using <a href="http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.start_requests" rel="nofollow"><code>start_requests()</code> method</a> instead of <code>parse()</code>:</p>

<pre><code>def start_requests(self):
    for x in xrange(8): 
        random_form_page = session.query(....

        PR = Request(
            'htp://my-api',
            headers=self.headers,
            meta={'newrequest': Request(random_form_page,  headers=self.headers)},
            callback=self.parse_PR
        )
        yield PR
</code></pre>

<p>You should also omit <code>start_urls</code> if it is set.</p>

python scrapy