Scrapy XPath Incorrectly when there is Unicode in the page
<p>I want to get all divs that have category class.</p>

<p>Take a look at this page: www.postkhmer.com/áááááá¶ááá¶áá· </p>

<p><a href="https://i.stack.imgur.com/WvFAN.jpg" rel="nofollow"><img src="https://i.stack.imgur.com/WvFAN.jpg" alt="enter image description here"></a></p>

<p>in scrapy shell: <code>scrapy shell 'www.postkhmer.com/áááááá¶ááá¶áá·'</code></p>

<p><a href="https://i.stack.imgur.com/KpZUq.png" rel="nofollow"><img src="https://i.stack.imgur.com/KpZUq.png" alt="enter image description here"></a></p>

<p>As you see I got only 2 elements back. </p>

<p><code>
scrapy fetch --nolog http://www.postkhmer.com/áááááá¶ááá¶áá· &gt; page.html
scrapy shell ./page.html
response.xpath('//div[@class="category"]')
</code>
Still got only 2 elements back. But when I open page.html in Sublime. </p>

<p>I got 15 matches:
<a href="https://i.stack.imgur.com/S9J8u.png" rel="nofollow"><img src="https://i.stack.imgur.com/S9J8u.png" alt="enter image description here"></a></p>

<p>The most interesting part is: when I remove the anchor link from the 2nd category: </p>

<p><a href="https://i.stack.imgur.com/y1rfG.png" rel="nofollow"><img src="https://i.stack.imgur.com/y1rfG.png" alt="enter image description here"></a></p>

<p>and i run <code>response.xpath('//div[@class="category"]')</code> in the scrapy shell again, I got 3 elements:</p>

<p><a href="https://i.stack.imgur.com/kGRrS.png" rel="nofollow"><img src="https://i.stack.imgur.com/kGRrS.png" alt="enter image description here"></a></p>

<p>I'm like what the hell!? Can someone help me to solve this problem please?</p>

<p>I've uploaded the file in <a href="http://www.filedropper.com/scrapypage" rel="nofollow">here</a> incase you want to test locally. </p>

<p>when you save the page into a local file <code>page.html</code>, you skip the http header that contains encoding information.
Later on, when you open this file , either with scrapy or sublime, they have no clue what was the original encoding of the document.</p>

<p><strong>Recommendation</strong>: never used documents saved to a file for parsing.</p>

python xpath scrapy