How to test standalone code outside of a scrapy spider?
<p>I can test my xpaths on a HTML body string by running the two lines under (1) below.</p>

<p>But what if I have a local file <code>myfile.html</code> whose content is exactly the <code>body</code> string. How would I run some standalone code, <strong>outside of a spider</strong>? I am seeking something that is roughly similar to the lines under (2) below.</p>

<p>I am aware that <code>scrapy shell myfile.html</code> tests xpaths. My intention is to run some Python code on the response, and hence <code>scrapy shell</code> is lacking (or else tedious).</p>

<pre><code>from scrapy.selector import Selector
from scrapy.http import HtmlResponse

# (1)

body = '&lt;html&gt;&lt;body&gt;&lt;span&gt;Hello&lt;/span&gt;&lt;/body&gt;&lt;/html&gt;'
print Selector(text=body).xpath('//span/text()').extract()

# (2)

response = HtmlResponse(url='file:///tmp/myfile.html')
print Selector(response=response).xpath('//span/text()').extract()
</code></pre>

<p>You can have a look at <a href="https://github.com/scrapy/scrapy/blob/master/tests/test_linkextractors.py" rel="nofollow">how scrapy tests for link extractors</a> are implemented.</p>

<p>They use a <code>get_test_data()</code> helper to <a href="https://github.com/scrapy/scrapy/blob/129421c7e31b89b9b0f9c5f7d8ae59e47df36091/tests/__init__.py#L31" rel="nofollow">read a file</a> as bytes,
and then instanciate a <code>Response</code> with a fake URL and body set to the bytes read with <code>get_test_data()</code>:</p>

<pre><code>...
            body = get_testdata('link_extractor', 'sgml_linkextractor.html')
            self.response = HtmlResponse(url='http://example.com/index', body=body)
...
</code></pre>

<p>Depending on the encoding used for the local HTML files, you may need to pass a non-UTF-8 <code>encoding</code> argument to <code>Response</code></p>

python xpath scrapy