In scrapy, I use XPATH to pick HTML and got many unnecessary "" and ,?
<p>I faced with a problem with parsing <a href="http://so.gushiwen.org/view_20788.aspx" rel="nofollow">http://so.gushiwen.org/view_20788.aspx</a></p>

<p><img src="http://i.stack.imgur.com/W3jfH.png" alt="Inspector"></p>

<p>This is what I want:</p>

<pre><code>"detail_text": ["
    å¯¥è½å¤è¡å®«ï¼å®«è±å¯å¯çº¢ãç½å¤´å®«å¥³å¨ï¼é²åè¯´çå®ã 
"],
</code></pre>

<p>but I got this :</p>

<pre><code>"detail_text": ["
    ", "
    ", "
    ", "
    ", "
    å¯¥è½å¤è¡å®«ï¼å®«è±å¯å¯çº¢ã", "ç½å¤´å®«å¥³å¨ï¼é²åè¯´çå®ã 
"],
</code></pre>

<p>and this is my code :</p>

<pre><code>#spider
class Tangshi3Spide(scrapy.Spider):
    name = "tangshi3"
    allowed_domains = ["gushiwen.org"]
    start_urls = [
        "http://so.gushiwen.org/view_20788.aspx"
    ]
    def __init__(self):
        self.items = []

    def parse(self, response):
        sel = Selector(response)
        sites = sel.xpath('//div[@class="main3"]/div[@class="shileft"]')
        domain = 'http://so.gushiwen.org'
        for site in sites:
            item = Tangshi3Item()
            item['detail_title'] = site.xpath('div[@class="son1"]/h1/text()').extract()
            item['detail_dynasty'] = site.xpath(
                u'div[@class="son2"]/p/span[contains(text(),"æä»£ï¼")]/parent::p/text()').extract()
            item['detail_translate_note_url'] = site.xpath('div[@id="fanyiShort676"]/p/a/u/parent::a/@href').extract()
            item['detail_appreciation_url'] = site.xpath('div[@id="shangxiShort787"]/p/a/u/parent::a/@href').extract()
            item['detail_background_url'] = site.xpath('div[@id="shangxiShort24492"]/p/a/u/parent::a/@href').extract()
            #question line
            item['detail_text'] = site.xpath('div[@class="son2"]/text()').extract()
            self.items.append(item)
        return self.items



#pipeline
class Tangshi3Pipeline(object):
    def __init__(self):
        self.file = codecs.open('tangshi_detail.json', 'w',     encoding='utf-8')

    def process_item(self, item, spider):
        line = json.dumps(dict(item))
        self.file.write(line.decode("unicode_escape"))
        return item
</code></pre>

<p>how can I get the right text?</p>

<p>You can add predicate <code>[normalize-space()]</code> to avoid picking up empty text nodes i.e those containing whitespaces only :</p>

<pre><code>item['detail_text'] = site.xpath('div[@class="son2"]/text()[normalize-space()]').extract()
</code></pre>

python xpath scrapy