how to get the next page link in scrapy
<p>how to get the link "/zufang/dongcheng/pg2/" in "ä¸ä¸é¡µ"ï¼</p>

<pre><code>&lt;a href="/zufang/dongcheng/pg2/" data-page="2"&gt;ä¸ä¸é¡µ&lt;/a&gt;
</code></pre>

<p>I have tried this, but got nothing.</p>

<p>the url is "<a href="http://bj.lianjia.com/zufang/dongcheng/" rel="nofollow">http://bj.lianjia.com/zufang/dongcheng/</a>"</p>

<pre><code>start_urls = (
            'http://bj.lianjia.com/zufang/dongcheng/',
    )

rules = (Rule(LinkExtractor(allow=(), restrict_xpaths=('/html/body/div[4]/div[2]/div/div[2]/div[2]/a[5]',)), callback="parse", follow= True),)


def parse(self, response):
            l = ItemLoader(item = ItjuziItem(),response=response)
            for i in range(0,len(response.xpath("//div[@class='info-panel']/h2/a/text()").extract())):
                    info = response.xpath("//div[@class='info-panel']/h2/a/text()").extract()[i].encode('utf-8')
                    local = response.xpath("//div[@class='info-panel']").xpath(".//span[@class='region']/text()").extract()[i].encode('utf-8') 
                    house_layout = response.xpath("//div[@class='info-panel']").xpath(".//span[@class='zone']//text()").extract()[i].encode('utf-8')
                    house_square = response.xpath("//div[@class='info-panel']").xpath(".//span[@class='meters']/text()").extract()[i].encode('utf-8')
                    house_orientation = response.xpath("//div[@class='info-panel']").xpath(".//div[@class='where']//span/text()").extract()[(i + 1) * 4 - 1].encode('utf-8')
                    district = response.xpath("//div[@class='info-panel']").xpath(".//div[@class='con']/a/text()").extract()[i].encode('utf-8')[:-6]
                    floor = response.xpath("//div[@class='info-panel']").xpath(".//div[@class='con']//text()").extract()[(i + 1) * 5 - 3].encode('utf-8')
                    building_year = response.xpath("//div[@class='info-panel']").xpath(".//div[@class='con']//text()").extract()[(i + 1) * 5 - 1].encode('utf-8')
                    price_month = response.xpath("//div[@class='info-panel']").xpath(".//span[@class='num']//text()").extract()[(i + 1) * 2 - 2].encode('utf-8')
                    person_views = response.xpath("//div[@class='info-panel']").xpath(".//span[@class='num']//text()").extract()[(i + 1) * 2 - 1].encode('utf-8')
                    tags = []
                    for j in range(0,len(response.xpath("//div[@class='view-label left']")[i].xpath(".//span//text()").extract())):
                            tags.append(response.xpath("//div[@class='view-label left']")[i].xpath(".//span//text()").extract()[j].encode("utf-8"))
                    l.add_value('info',info)
                    l.add_value('local',local)
                    l.add_value('house_layout',house_layout)
                    l.add_value('house_square',house_square)
                    l.add_value('house_orientation',house_orientation)
                    l.add_value('district',district)
                    l.add_value('floor',floor)
                    l.add_value('building_year',building_year)
                    l.add_value('price_month',price_month)
                    l.add_value('person_views',person_views)
                    l.add_value('tags',tags)
                    print l
                    return l.load_item()
</code></pre>

<p>Extract the link and attach it to the current url</p>

<pre><code>from urlparse import urljoin
yield scrapy.Request(urljoin(response.url,response.xpath("//a[@data-page]/@href").extract_first()), callback=self.parse)
</code></pre>

<p>or increment the page</p>

<pre><code>for i in range(2,20):
    yield scrapy.Request("http://bj.lianjia.com/zufang/dongcheng/pg"+ str(i), callback=self.parse)
</code></pre>

python regex scrapy web-crawler