Processing html text nodes with scrapy and XPath
<p>I'm using scrapy to process documents like this one:</p>

<pre><code>...
&lt;div class="contents"&gt;
    some text
    &lt;ol&gt;
        &lt;li&gt;
            more text
        &lt;/li&gt;
        ...
    &lt;/ol&gt;
&lt;/div&gt;
...
</code></pre>

<p>I want to collect all the text inside the contents area into a string.
I also need the '1., 2., 3....' from the <code>&lt;li&gt;</code> elements, so my result should be <code>'some text 1. more text...'</code> <br></p>

<p>So, I'm looping over <code>&lt;div class="contents"&gt;</code>'s children </p>

<pre><code>for n in response.xpath('//div[@class="contents"]/node()'):
    if n.xpath('self::ol'):
        result += process_list(n)
    else:
        result += n.extract()
</code></pre>

<p>If <code>n</code>is an ordered list, I loop over its elements and add a number to <code>li/text()</code> (in <code>process_list()</code>). If <code>n</code>is a text node itself, I just read its value.
However, <code>'some text'</code> doesn't seem to be part of the node set, since the loop doesn't get inside the <code>else</code> part. My result is <code>'1. more text'</code></p>

<p>Finding text nodes relative to their parent node works:</p>

<pre><code>response.xpath('//div[@class="contents"]//text()')
</code></pre>

<p>finds all the text, but this way I can't add the list item numbers.</p>

<p>What am I doing wrong and is there a better way to achieve my task?</p>

<p>Scrapy's Selectors use <code>lxml</code> under the hood, but <a href="https://bugs.launchpad.net/lxml/+bug/996134" rel="nofollow"><code>lxml</code> doesn't work with XPath calls on text nodes</a>. </p>

<pre><code>&gt;&gt;&gt; import scrapy
&gt;&gt;&gt; s = scrapy.Selector(text='''&lt;div class="contents"&gt;
...     some text
...     &lt;ol&gt;
...         &lt;li&gt;
...             more text
...         &lt;/li&gt;
...         ...
...     &lt;/ol&gt;
... &lt;/div&gt;''')
&gt;&gt;&gt; s.xpath('.//div[@class="contents"]/node()')
[&lt;Selector xpath='.//div[@class="contents"]/node()' data='\n    some text\n    '&gt;, &lt;Selector xpath='.//div[@class="contents"]/node()' data='&lt;ol&gt;\n        &lt;li&gt;\n            more text\n'&gt;, &lt;Selector xpath='.//div[@class="contents"]/node()' data='\n'&gt;]
&gt;&gt;&gt; for n in s.xpath('.//div[@class="contents"]/node()'):
...     print(n.xpath('self::ol'))
... 
[]
[&lt;Selector xpath='self::ol' data='&lt;ol&gt;\n        &lt;li&gt;\n            more text\n'&gt;]
[]
</code></pre>

<p>But you could hack into the underlying lxml object to test it's type for a text node (it's "hidden" in a <code>.root</code> attribute of each scrapy Selector):</p>

<pre><code>&gt;&gt;&gt; for n in s.xpath('.//div[@class="contents"]/node()'):
...     print([type(n.root), n.root])
... 
[&lt;class 'str'&gt;, '\n    some text\n    ']
[&lt;class 'lxml.etree._Element'&gt;, &lt;Element ol at 0x7fa020f2f9c8&gt;]
[&lt;class 'str'&gt;, '\n']
</code></pre>

<p>An alternative is to use some HTML-to-text conversion library like <a href="https://pypi.python.org/pypi/html2text" rel="nofollow">html2text</a></p>

<pre><code>&gt;&gt;&gt; import html2text
&gt;&gt;&gt; html2text.html2text("""&lt;div class="contents"&gt;
...     some text
...     &lt;ol&gt;
...         &lt;li&gt;
...             more text
...         &lt;/li&gt;
...         ...
...     &lt;/ol&gt;
... &lt;/div&gt;""")
'some text\n\n  1. more text \n...\n\n'
</code></pre>

python xpath scrapy