How to constructing url in start_urls list in scrapy framework python
<p>I am new to scrapy and python.<br>
In my case:</p>

<h2>Page A:</h2>

<pre><code>http://www.example.com/search?keyword=city&amp;style=1&amp;page=1  
http://www.example.com/search?keyword=city&amp;style=1&amp;page=2  
http://www.example.com/search?keyword=city&amp;style=1&amp;page=3 
</code></pre>

<p>Rules is:   </p>

<pre><code>    `for i in range(50):
        "http://www.example.com/search?keyword=city&amp;style=1&amp;page=%s" % i`
</code></pre>

<h2>Page B:</h2>

<pre><code>http://www.example.com/city_detail_0001.html  
http://www.example.com/city_detail_0100.html  
http://www.example.com/city_detail_0053.html  
</code></pre>

<p>No rules, Because Page B is match the keyword for search.    </p>

<p>So, This meansï¼If I want grab some information from Page B,<br>
First, I must use the Page A to sifting link of the Page B.<br>
In the past, I usually two step:<br>
1. I create scrapy A, and grab the Page B's link in a txt file<br>
2. And in scrapy B, I read the txt file to the "start_urls"   </p>

<p>Now, can u please guide me, that how can i construct the "start_urls" in one spider? </p>

<p>the <a href="http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.start_requests" rel="nofollow"><code>start_requests</code></a> method is what you need. After that, keep passing the requests and parse the response bodies on callback methods.</p>

<pre><code>class MySpider(Spider):
    name = 'example'

    def start_requests(self):
        for i in range(50):
            yield Request('myurl%s' % i, callback=self.parse)

    def parse(self, response):
        # get my information for page B
        yield Request('pageB', callback=self.parse_my_item)

    def parse_my_item(self, response):
        item = {}
        # real parsing method for my items    
        yield item
</code></pre>

python regex scrapy