Using XPath with Scrapy
<p>I am new to using Scrapy and is trying get all the URLs of the listings on the page using Xpath.</p>

<p>The first xpath works</p>

<pre><code>sel.xpath('//[contains(@class, "attraction_element")]')
</code></pre>

<p>but the second xpath is giving an error</p>

<pre><code>get_parsed_string(snode_attraction, '//[@class="property_title"]/a/@href')
</code></pre>

<p>What is wrong and how can we fix it?</p>

<p><strong>Scrapy Code</strong></p>

<pre><code>def clean_parsed_string(string):
    if len(string) &gt; 0:
        ascii_string = string
        if is_ascii(ascii_string) == False:
            ascii_string = unicodedata.normalize('NFKD', ascii_string).encode('ascii', 'ignore')
        return str(ascii_string)
    else:
        return None


def get_parsed_string(selector, xpath):
    return_string = ''
    extracted_list = selector.xpath(xpath).extract()
    if len(extracted_list) &gt; 0:
        raw_string = extracted_list[0].strip()
        if raw_string is not None:
            return_string = htmlparser.unescape(raw_string)
    return return_string


class TripAdvisorSpider(Spider):
    name = 'tripadvisor'

    allowed_domains = ["tripadvisor.com"]
    base_uri = "http://www.tripadvisor.com"
    start_urls = [
        base_uri + '/Attractions-g155032-Activities-c47-t163-Montreal_Quebec.html'
    ]


    # Entry point for BaseSpider
    def parse(self, response):

        tripadvisor_items = []

        sel = Selector(response)
        snode_attractions = sel.xpath('//[contains(@class, "attraction_element")]')

        # Build item index
        for snode_attraction in snode_attractions:
            print clean_parsed_string(get_parsed_string(snode_attraction, '//[@class="property_title"]/a/@href'))
</code></pre>

<p>Both are not valid XPath expressions, you need to add the tag names after the <code>//</code>. You can also use a wildcard <code>*</code>:</p>

<pre><code>snode_attractions = sel.xpath('//*[contains(@class, "attraction_element")]')
</code></pre>

<p>Note that aside from that you second XPath expression that is used in a loop has to be context specific and start with a dot:</p>

<pre><code># Build item index
for snode_attraction in snode_attractions:
    print clean_parsed_string(get_parsed_string(snode_attraction, './/*[@class="property_title"]/a/@href'))
</code></pre>

<p>Also note that you don't need to instantiate a <code>Selector</code> object and ca use <code>response.xpath()</code> shortcut directly.</p>

<hr>

<p>Note that a more concise and, arguably, more readable version of the same logic implementation would be to use <em>CSS selectors</em>:</p>

<pre><code>snode_attractions = response.css('.attraction_element')
for snode_attraction in snode_attractions:
    print snode_attraction.css('.property_title &gt; a::attr("href")').extract_first()
</code></pre>

python python-2.7 xpath scrapy