Union of node and function on node in XPath
<p>I am using Scrapy to crawl some webpages. I want to write an XPath query that will, within a parent <code>&lt;div&gt;</code>, append a couple of characters of text to any child <code>&lt;a&gt;</code> nodes, while extracting the text of the  div's <code>self</code> node normally. Essentially it is like a normal <code>descendant-or-self</code> or <code>//</code> query, just written with <code>|</code> and calling the <code>concat</code> function on the descendants (which, if they exist, will be <code>&lt;a&gt;</code> tags).</p>

<p>These all return a value:</p>

<ol>
<li><code>my_div.xpath('div[@class="my_class"]/text()).extract()</code></li>
<li><code>my_div.xpath('concat(\'@\', div[@class="my_class"]/a/text())').extract()</code></li>
<li><code>my_div.xpath('div[@class="my_class"]/text() | div[@class="my_class"]/a/text()').extract()</code></li>
</ol>

<p>However attempting to combine (1) and (2) above in the format of (3):</p>

<p><code>my_div.xpath('div[@class="my_class"]/text() | 
concat(\'@\', div[@class="my_class"]/a/text())').extract()</code></p>

<p>results in the following error:</p>

<p><code>ValueError: XPath error: Invalid type in div[@class="my_class"]/text() | concat('@', div[@class="my_class"]/a/text())
</code></p>

<p>How do I get XPath to recognize the union of a node with a function called on a node?</p>

<p>In XPath 1.0, a <a href="https://www.w3.org/TR/xpath/#location-paths" rel="nofollow"><em>location path</em></a>  returns a <a href="https://www.w3.org/TR/xpath/#node-sets" rel="nofollow"><em>node-set</em></a>. The <a href="https://www.w3.org/TR/xpath/#function-concat" rel="nofollow"><code>concat</code></a> function returns a <a href="https://www.w3.org/TR/xpath/#strings" rel="nofollow"><em>string</em></a>. The <a href="https://www.w3.org/TR/xpath/#node-sets" rel="nofollow"><em>union operator</em> <code>|</code></a> computes the union of its operands, which must be <em>node-sets</em>.</p>

python xpath scrapy