scrapy LxmlLinkExtractor to crawl relative urls
<p>I want to crawl all the relative urls under tag in <a href="http://news.qq.com/" rel="nofollow">http://news.qq.com/</a></p>

<p>the code of mine is that:</p>

<pre><code>import scrapy
from scrapy.selector import Selector  
from homework.items import HomeworkItem
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.lxmlhtml import LxmlLinkExtractor

class News1Spider(scrapy.Spider):
    name = "News1"
    allowed_domains = ["http://news.qq.com/"]
    start_urls = (
        'http://news.qq.com/',
    )
    rules = (
        Rule(LxmlLinkExtractor(restrict_xpaths='//div[@class="Q-   tpList"]/div/a/@href'),callback='parse'),
    )

    def parse(self, response):
        sel = Selector(response)
    #lis = sel.xpath('//div[@class="Q-tpList"]')
    #item = TutorialItem()
    #for li in lis:
        title = sel.xpath('//div[@id=C-Main-Article-QQ]/div[1]/text()').extract()
        content =sel.xpath('//div[@id=Cnt-Main-Article-QQ]/p/text()').extract()
        print title
</code></pre>

<p>when running cmd scrapy crawl News1</p>

<p>I cant get the title in command window ,Could you please tell me how to revise it and why?  thanks</p>

<p>First thing you are either using an outdate scrapy version or having bad imports since right now scrapy has only 1 type of link extractors - LinkExtractor (which is renamed LxmlExtractor)</p>

<p>I've tested this and it works perfectly fine:</p>

<pre><code>$ scrapy shell 'http://news.qq.com/'
from scrapy.linkextractors import LinkExtractor
LinkExtractor(restrict_xpaths=['//div[@class="Q-tpList"]/div/a']).extract_links(response)
# got 43 results
</code></pre>

<p>Note no spaces in xpath <code>@class</code> check and it points to <code>a</code> node rather than <code>@href</code> attribute because LinkExtractor extract nodes not parameters.</p>

python scrapy web-crawler