Union of node and function on node in XPath
<p>I am using Scrapy to crawl some webpages. I want to write an XPath query that will, within a parent <code>&lt;div&gt;</code>, append a couple of characters of text to any child <code>&lt;a&gt;</code> nodes, while extracting the text of the  div's <code>self</code> node normally. Essentially it is like a normal <code>descendant-or-self</code> or <code>//</code> query, just written with <code>|</code> and calling the <code>concat</code> function on the descendants (which, if they exist, will be <code>&lt;a&gt;</code> tags).</p>

<p>These all return a value:</p>

<ol>
<li><code>my_div.xpath('div[@class="my_class"]/text()).extract()</code></li>
<li><code>my_div.xpath('concat(\'@\', div[@class="my_class"]/a/text())').extract()</code></li>
<li><code>my_div.xpath('div[@class="my_class"]/text() | div[@class="my_class"]/a/text()').extract()</code></li>
</ol>

<p>However attempting to combine (1) and (2) above in the format of (3):</p>

<p><code>my_div.xpath('div[@class="my_class"]/text() | 
concat(\'@\', div[@class="my_class"]/a/text())').extract()</code></p>

<p>results in the following error:</p>

<p><code>ValueError: XPath error: Invalid type in div[@class="my_class"]/text() | concat('@', div[@class="my_class"]/a/text())
</code></p>

<p>How do I get XPath to recognize the union of a node with a function called on a node?</p>

<p>I think it doesn't work because concat is doesn't actually return a path, and <code>|</code> is used to select multiple <em>paths</em></p>

<blockquote>
  <p>By using the | operator in an XPath expression you can select several paths.  </p>
</blockquote>

<p>as per <a href="http://www.w3schools.com/xsl/xpath_syntax.asp" rel="nofollow">http://www.w3schools.com/xsl/xpath_syntax.asp</a></p>

<p>Why not just split it into two? Generally you use <a href="https://doc.scrapy.org/en/latest/topics/loaders.html" rel="nofollow">ItemLoaders</a> with your spider. So you can simply add as many paths and/or values as you like.</p>

<pre><code>mil = MyItemLoader(response=response)
mil.add_xpath('name', 'xpath1')
mil.add_xpath('name', 'xpath2')
mil.load_item()
# {'name': ['values_of_xpath1','values_of_xpath2']
</code></pre>

<p>If you want to preserve tree order you can try:</p>

<pre><code>nodes = my_div.xpath('div[@class="my_class"]')
text = []
for node in nodes:
    text.append(node.xpath("text()").extract_first())
    text.append(node.xpath("a/text()").extract_first())
text = '@'.join(text)
</code></pre>

<p>You can probably simplify it with list comprehension but you get the idea: extract the nodes and iterate through nodes for both values.</p>

python xpath scrapy