XPath returns empty list. Why is it ignoring targeted div element?
<p>I'm a newbee to XPath and Scrapy. I'm trying to target a node which does not have a unique class (i.e. <code>class="pubBody"</code>). </p>

<p>Already tried: 
<a href="http://stackoverflow.com/questions/28163626/xpath-not-contains-a-and-b">xpath not contains A and B</a></p>

<p>This should be a simple task but XPath just misses the second item. I am doing this from the scrapy shell. On the command prompt:</p>

<p>scrapy shell "<a href="http://www.sciencedirect.com/science/journal/00221694/" rel="nofollow">http://www.sciencedirect.com/science/journal/00221694/</a>"</p>

<p>I am looking for the second div:</p>

<pre><code>&lt;div id="issueListHeader" class="pubBody"&gt;...&lt; /div&gt;

&lt;div class="pubBody"&gt;... &lt; /div&gt; 
</code></pre>

<p>I can only get the first but not the second. The best answers to similar questions suggested trying something like:</p>

<pre><code>hxs.xpath('//div[contains(@class,"pubBody") and not(contains(@id,"issueListHeader"))]') 
</code></pre>

<p>but this returns an empty list for some reason. Any help please? Must be missing something silly, I've tried this for days!</p>

<p>Other details:</p>

<p>Once in the scrapy shell:</p>

<pre><code>import scrapy

xs = scrapy.Selector(response)

hxs.xpath('//div[@class="pubBody"]')
</code></pre>

<p>Which works only for the first div element:</p>

<pre><code>[&lt;Selector xpath='//div[@class="pubBody"]' data='&lt;div id="issueListHeader" class="pubBody'&gt;]
</code></pre>

<p>For the failed second div element I've also tried:</p>

<pre><code>hxs.xpath('//div[@class="pubBody" and not(@id="issueListHeader")]').extract_first()

hxs.xpath('//div[starts-with(@class, "pubBody") and not(re:test(@id, "issueListHeader"))]')
</code></pre>

<p>Also directly copied the XPath from Chrome, but also returns '[]':</p>

<pre><code>hxs.xpath('//*[@id="issueList"]/div/form/div[2]')
</code></pre>

<p>The problem is that the <em>HTML is very far from being well-formed on this page</em>. To demonstrate, see how the same exact CSS selector is producing 0 results with Scrapy and producing 94 in <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" rel="nofollow"><code>BeautifulSoup</code></a>:</p>

<pre><code>In [1]: from bs4 import BeautifulSoup

In [2]: soup = BeautifulSoup(response.body, 'html5lib')  # note: "html5lib" has to be installed

In [3]: len(soup.select(".article h4 a"))
Out[3]: 94

In [4]: len(response.css(".article h4 a"))
Out[4]: 0
</code></pre>

<p>Same goes for the <code>pubBody</code> element you are trying to locate:</p>

<pre><code>In [6]: len(response.css(".pubBody"))
Out[6]: 1

In [7]: len(soup.select(".pubBody"))
Out[7]: 2
</code></pre>

<p>So, try hooking up <code>BeautifulSoup</code> to fix/clean up the HTML - ideally through a <a href="http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#writing-your-own-downloader-middleware" rel="nofollow">middleware</a>.</p>

<hr>

<p>I've created a simple <a href="https://github.com/alecxe/scrapy-beautifulsoup" rel="nofollow"><code>scrapy_beautifulsoup</code> middleware</a> to easily hook up into the project:</p>

<ul>
<li><p>install it via pip:</p>

<pre><code>pip install scrapy-beautifulsoup
</code></pre></li>
<li><p>configure the middleware in <code>settings.py</code>:</p>

<pre><code>DOWNLOADER_MIDDLEWARES = {
    'scrapy_beautifulsoup.middleware.BeautifulSoupMiddleware': 543
}
BEAUTIFULSOUP_PARSER = "html5lib" 
</code></pre></li>
</ul>

<p>Profit.</p>

python html xpath scrapy html-parsing