Scrapy SgmlLinkExtractor how to define XPath
<p>I want to retreive the cityname and citycode and store it in one string variable. The image shows the precise location:</p>

<p><a href="http://i.stack.imgur.com/PmMcq.png" rel="nofollow"><img src="http://i.stack.imgur.com/PmMcq.png" alt="enter image description here"></a></p>

<p>Google Chrome gave me the following XPath:</p>

<pre><code>//*[@id="page"]/main/div[4]/div[2]/div[1]/div/div/div[1]/div[2]/div/div[1]/div/a[1]/span
</code></pre>

<p>So I defined the following statement in scrapy to get the desired information: </p>

<pre><code>plz = response.xpath('//*[@id="page"]/main/div[4]/div[2]/div[1]/div/div/div[1]/div[2]/div/div[1]/div/a[1]/span/text()').extract()
</code></pre>

<p>However I was not successful, the string remains empty. What XPath definition should I use instead?</p>

<p>The DOM is manipulated by javascript, so what chrome shows is the xpath after 
the all the stuff has happened.</p>

<p>If all you want is to get the cities, you can get it this way (using scrapy):</p>

<pre><code>city_text = response.css('.detail-address span::text').extract_first()
city_code, city_name = city_text.split(maxsplit=1)
</code></pre>

<p>Or you can manipulate the JSON in CDATA to get all the data you need:</p>

<pre><code>cdata_text = response.xpath('//*[@id="tdakv"]/text()').extract_first()
json_str = cdata_text.splitlines()[2]
json_str = json_str[json_str.find('{'):]
data = json.loads(json_str)                # import json
city_code = data['kvzip']
city_name = data['kvplace']
</code></pre>

python regex xpath scrapy