XPath returns empty list. Why is it ignoring targeted div element?
<p>I'm a newbee to XPath and Scrapy. I'm trying to target a node which does not have a unique class (i.e. <code>class="pubBody"</code>). </p>

<p>Already tried: 
<a href="http://stackoverflow.com/questions/28163626/xpath-not-contains-a-and-b">xpath not contains A and B</a></p>

<p>This should be a simple task but XPath just misses the second item. I am doing this from the scrapy shell. On the command prompt:</p>

<p>scrapy shell "<a href="http://www.sciencedirect.com/science/journal/00221694/" rel="nofollow">http://www.sciencedirect.com/science/journal/00221694/</a>"</p>

<p>I am looking for the second div:</p>

<pre><code>&lt;div id="issueListHeader" class="pubBody"&gt;...&lt; /div&gt;

&lt;div class="pubBody"&gt;... &lt; /div&gt; 
</code></pre>

<p>I can only get the first but not the second. The best answers to similar questions suggested trying something like:</p>

<pre><code>hxs.xpath('//div[contains(@class,"pubBody") and not(contains(@id,"issueListHeader"))]') 
</code></pre>

<p>but this returns an empty list for some reason. Any help please? Must be missing something silly, I've tried this for days!</p>

<p>Other details:</p>

<p>Once in the scrapy shell:</p>

<pre><code>import scrapy

xs = scrapy.Selector(response)

hxs.xpath('//div[@class="pubBody"]')
</code></pre>

<p>Which works only for the first div element:</p>

<pre><code>[&lt;Selector xpath='//div[@class="pubBody"]' data='&lt;div id="issueListHeader" class="pubBody'&gt;]
</code></pre>

<p>For the failed second div element I've also tried:</p>

<pre><code>hxs.xpath('//div[@class="pubBody" and not(@id="issueListHeader")]').extract_first()

hxs.xpath('//div[starts-with(@class, "pubBody") and not(re:test(@id, "issueListHeader"))]')
</code></pre>

<p>Also directly copied the XPath from Chrome, but also returns '[]':</p>

<pre><code>hxs.xpath('//*[@id="issueList"]/div/form/div[2]')
</code></pre>

<p>I suspect the problem is that the source for the page you'r trying to parse (<a href="http://www.sciencedirect.com/science/journal/00221694/" rel="nofollow">http://www.sciencedirect.com/science/journal/00221694/</a>) is not valid XML due to the <code>&lt;link ...&gt;</code> nodes/elements/tags not having closing tags.  There may be other problems, but those are the first ones I found.</p>

<p>I'm rusty on Javascript, but you may try navigating down the DOM down to a lower level in the page (ie. body or some other node closer to the elements you're trying to target) and then perform the XPath from that level.</p>

<p><strong>UPDATE:</strong> I just tried removing the <code>&lt;head&gt;</code> of the document and passing it through an XML parser and it still breaks on sever <code>&lt;input&gt;</code> nodes that are not closed.  Unless I'm forgetting some special JavaScript XML/XPath rules methods that dismiss closing tags I suspect you might be better suited to use something like JQuery to find the elements you're looking for.</p>

python html xpath scrapy html-parsing