XPath in scrapy returns elements which don't exist
<p>I am creating a new scrapy spider and everything is going pretty good, although I have a problem with one of the websites, where response.xpath is returning objects in the list which doesn't exist in html code:</p>

<pre><code>{"pdf_name": ["\n\t\t\t\t\t\t\t\t\t", "ZZZZZZ", "\n\t\t\t\t\t\t\t\t\t", "PDF", "\n\t\t\t\t\t\t\t\t"],
{"pdf_name": ["\n\t\t\t\t\t\t\t\t\t\t", "YYYYYY", "\n\t\t\t\t\t\t\t\t\t\t", "XXXXXX"]}
</code></pre>

<p>As you can see below,  these "empty" objects (\t and \n) are not included in HTML tags. If I understand correctly, xpath is including whitespaces before tags:</p>

<pre><code>&lt;div class="inner d-i-b va-t" role="group"&gt;
                        &lt;a class="link-to" href="A.pdf" target="_blank"&gt;
                                    &lt;i class="offscreen"&gt;ZZZZZZ&lt;/i&gt;
                                    &lt;span&gt;PDF&lt;/span&gt;
                                &lt;/a&gt;

                                &lt;div class="text-box"&gt;
                                    &lt;a href="A.pdf"&gt;
                                        &lt;i class="offscreen"&gt;YYYYYY&lt;/i&gt;
                                        &lt;p&gt;XXXXXX&lt;/p&gt;&lt;/a&gt;
                                &lt;/div&gt;
                            &lt;/div&gt;
</code></pre>

<p>I know that I can strip() strings and remove white spaces, although it would only mitigate the issue, not remove the main problem, which is including white spaces in results.</p>

<p>Why is it happening? How to limit XPath results only to tags (I thought previously that it is done by default)?</p>

<p><strong>Spider code - parse function (pdf_name is causing problems)</strong></p>

<pre><code>def parse(self, response):

    # Select all links to pdfs
    for pdf in response.xpath('//a[contains(@href, ".pdf")]'):
        item = PdfItem()

        # Create a list of text fields for links to PDFs and their descendants
        item['pdf_name'] = pdf.xpath('descendant::text()').extract()

        yield item
</code></pre>

<p>Whitespace is part of the document. Just because <em>you</em> think it is unimportant does not make it go away.</p>

<p>A text node is a text node, whether it consists of <code>' '</code> (the space character) or any other character makes no difference at all.</p>

<p>You can normalize the whitespace with the <code>normalize-space()</code> XPath function:</p>

<pre><code>def parse(self, response):
    for pdf_link in response.xpath('//a[contains(@href, ".pdf")]'):
        item = PdfItem()
        item['pdf_name'] = pdf_link.xpath('normalize-space(.)').extract()
        yield item
</code></pre>

<p>First, <code>normalize-space()</code> converts its argument to string, which is done by concatenating all descendant text nodes. Then it trims leading and trailing spaces and collapses any consecutive whitespace (including line breaks) into a single space. Something like this <code>'\n   bla \n\n bla  '</code> would become <code>'bla bla'</code>.</p>

python xpath scrapy