scrapy LxmlLinkExtractor to crawl relative urls
<p>I want to crawl all the relative urls under tag in <a href="http://news.qq.com/" rel="nofollow">http://news.qq.com/</a></p>

<p>the code of mine is that:</p>

<pre><code>import scrapy
from scrapy.selector import Selector  
from homework.items import HomeworkItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.lxmlhtml import LxmlLinkExtractor

class News1Spider(scrapy.Spider):
    name = "News1"
    allowed_domains = ["http://news.qq.com/"]
    start_urls = (
        'http://news.qq.com/',
    )
    rules = (
        Rule(LxmlLinkExtractor(restrict_xpaths='//div[@class="Q-   tpList"]/div/a/@href'),callback='parse'),
    )

    def parse(self, response):
        sel = Selector(response)
    #lis = sel.xpath('//div[@class="Q-tpList"]')
    #item = TutorialItem()
    #for li in lis:
        title = sel.xpath('//div[@id=C-Main-Article-QQ]/div[1]/text()').extract()
        content =sel.xpath('//div[@id=Cnt-Main-Article-QQ]/p/text()').extract()
        print title
</code></pre>

<p>when running cmd scrapy crawl News1</p>

<p>I cant get the title in command window ,Could you please tell me how to revise it and why?  thanks</p>

<p>You're subclassing <code>Spider</code>, but since you have <code>start_urls</code>, I think you meant to use <code>CrawlSpider</code>. In this case, you need to revise your structure, since <code>parse</code> is actually used internally by <code>CrawlSpider</code> to find new links to crawl:</p>

<pre><code>rules = (
    Rule(LxmlLinkExtractor(restrict_xpaths='//div[@class="Q-   tpList"]/div/a/@href'), callback='parse_page'),
)

def parse_page(self, response):
    ...
</code></pre>

<p>You should fix this class name and remove the spaces as well:</p>

<pre><code>//div[@class="Q-   tpList"]/div/a/@href
                ^^^
</code></pre>

<p>Finally, I think you're using an old version of Scrapy. I suggest you upgrade now before writing more code using the old API, since it'll be harder to switch later.</p>

python scrapy web-crawler