Processing html text nodes with scrapy and XPath
<p>I'm using scrapy to process documents like this one:</p>

<pre><code>...
&lt;div class="contents"&gt;
    some text
    &lt;ol&gt;
        &lt;li&gt;
            more text
        &lt;/li&gt;
        ...
    &lt;/ol&gt;
&lt;/div&gt;
...
</code></pre>

<p>I want to collect all the text inside the contents area into a string.
I also need the '1., 2., 3....' from the <code>&lt;li&gt;</code> elements, so my result should be <code>'some text 1. more text...'</code> <br></p>

<p>So, I'm looping over <code>&lt;div class="contents"&gt;</code>'s children </p>

<pre><code>for n in response.xpath('//div[@class="contents"]/node()'):
    if n.xpath('self::ol'):
        result += process_list(n)
    else:
        result += n.extract()
</code></pre>

<p>If <code>n</code>is an ordered list, I loop over its elements and add a number to <code>li/text()</code> (in <code>process_list()</code>). If <code>n</code>is a text node itself, I just read its value.
However, <code>'some text'</code> doesn't seem to be part of the node set, since the loop doesn't get inside the <code>else</code> part. My result is <code>'1. more text'</code></p>

<p>Finding text nodes relative to their parent node works:</p>

<pre><code>response.xpath('//div[@class="contents"]//text()')
</code></pre>

<p>finds all the text, but this way I can't add the list item numbers.</p>

<p>What am I doing wrong and is there a better way to achieve my task?</p>

<p>If <code>n</code> is not an <code>ol</code> element, <code>self::ol</code> yields an empty node set. What is <code>n.xpath(...)</code> supposed to return when the result of the expression is an empty node set?</p>

<p>An empty node set is "falsy" in XPath, but you're not evaluating it as a boolean in XPath, only in Python. Is an empty node set falsy in Python?</p>

<p>If that's the problem, you could fix it by changing the <code>if</code> statement to</p>

<pre><code>if n.xpath('boolean(self::ol)'):
</code></pre>

<p>or</p>

<pre><code>if n.xpath('count(self::ol) &gt; 1'):
</code></pre>

python xpath scrapy