Union of node and function on node in XPath
<p>I am using Scrapy to crawl some webpages. I want to write an XPath query that will, within a parent <code>&lt;div&gt;</code>, append a couple of characters of text to any child <code>&lt;a&gt;</code> nodes, while extracting the text of the  div's <code>self</code> node normally. Essentially it is like a normal <code>descendant-or-self</code> or <code>//</code> query, just written with <code>|</code> and calling the <code>concat</code> function on the descendants (which, if they exist, will be <code>&lt;a&gt;</code> tags).</p>

<p>These all return a value:</p>

<ol>
<li><code>my_div.xpath('div[@class="my_class"]/text()).extract()</code></li>
<li><code>my_div.xpath('concat(\'@\', div[@class="my_class"]/a/text())').extract()</code></li>
<li><code>my_div.xpath('div[@class="my_class"]/text() | div[@class="my_class"]/a/text()').extract()</code></li>
</ol>

<p>However attempting to combine (1) and (2) above in the format of (3):</p>

<p><code>my_div.xpath('div[@class="my_class"]/text() | 
concat(\'@\', div[@class="my_class"]/a/text())').extract()</code></p>

<p>results in the following error:</p>

<p><code>ValueError: XPath error: Invalid type in div[@class="my_class"]/text() | concat('@', div[@class="my_class"]/a/text())
</code></p>

<p>How do I get XPath to recognize the union of a node with a function called on a node?</p>

<p>Update: this is what I did:</p>

<pre class="lang-py prettyprint-override"><code>item['div_text'] = []
div_nodes = definition.xpath('div[@class="my_class"]/a | div[@class="my_class"]/text()')
for n in div_nodes:
    if n.xpath('self::a'):
        item['div_text'].append("@%s" % n.xpath('text()').extract_first())
    else:
        item['div_text'].append(n.extract())
</code></pre>

python xpath scrapy