Scrapy: Exclude content inside script tags in the HTML body
<p>I am currently extracting the entire text inside the body tag (excluding spacing like \r\n) using the following code:</p>

<pre><code>full_text = response.xpath('normalize-space(/html/body)').extract()
</code></pre>

<p>The problem is this is picking up javascript inside script tags within body.</p>

<p>Do you know how I can exclude the content within any script tags?</p>

<p>I've tried doing this but it isn't working:</p>

<pre><code>full_text = response.xpath('normalize-space(/html/body/*[not(self::script)])').extract()
</code></pre>

<p>Any help appreciated.</p>

<p>you can follow the answer on this question <a href="http://stackoverflow.com/a/19780110/2204978">Scraping text without javascript code using scrapy</a></p>

<pre><code>from w3lib.html import remove_tags, remove_tags_with_content

input = hxs.select('//div[@id="content"]').extract()
output = remove_tags(remove_tags_with_content(input, ('script', )))
</code></pre>

python xpath scrapy