TypeError: 'HtmlResponse' object is not iterable
<p>I'm new to python, but trying to get my head around it in order to use Scrapy for work.</p>

<p>I'm currently following this tutorial:
<a href="http://scrapy2.readthedocs.io/en/latest/intro/tutorial.html" rel="nofollow">http://scrapy2.readthedocs.io/en/latest/intro/tutorial.html</a></p>

<p>I've having trouble with this part (from the tutorial): </p>

<pre><code>def parse(self, response):
    for sel in response.xpath('//ul/li'):
        title = sel.xpath('a/text()').extract()
        link = sel.xpath('a/@href').extract()
        desc = sel.xpath('text()').extract()
        print title, link, desc
</code></pre>

<p>The problem I'm having is with the <code>for sel in response.xpath('//ul/li'):</code> part. I understand that line to essentially narrow down what is crawled to anything that matches the xpath <code>//ul/li</code>.</p>

<p>However, in my implementation, I can't narrow down the page to one singular section. I tried to get around this by selecting the entire HTML, see my attempt below:</p>

<pre><code>   def parse(self, response):
           for sel in response.xpath('//html'):
            title = sel.xpath('//h1/text()').extract()
            author = sel.xpath('//head/meta[@name="author"]/@content').extract()
            mediumlink = sel.xpath('//head/link[@rel="author"]/@href').extract()
            print title, author, mediumlink
</code></pre>

<p>The xpaths work both in a Chrome plugin I use, and using <code>response.xpath('//title').extract()</code> in <code>scrapy shell</code></p>

<p>I've tried changing the line to this:</p>

<p><code>for sel in response.xpath('//html'):</code> and <code>for sel in response.xpath('html'):</code></p>

<p>But each time I get this:</p>

<pre><code>2016-10-16 14:33:43 [scrapy] ERROR: Spider error processing &lt;GET https://medium.com/swlh/how-our-app-went-from-20-000-day-to-2-day-in-revenue-d6892a2801bf#.smmwwqxlf&gt; (referer: None)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 587, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Users/Matthew/Sites/crawl/tutorial/tutorial/spiders/medium_Spider.py", line 11, in parse
    for sel in response:
TypeError: 'HtmlResponse' object is not iterable
</code></pre>

<p>Could somebody give me some pointers on how best to resolve this? Go easy on me, my skills are not so hot. Thanks!</p>

<p>As the error message states</p>

<pre><code> for sel in response:
</code></pre>

<p>You try to iterate through the <code>response</code> object in your <code>medium_Spider.py</code> file at <strong>line 11</strong>.</p>

<p>However <code>response</code> is an <code>HtmlResponse</code> not an <em>iterable</em> which you could use in a <code>for</code> loop --> you are missing some method call on <code>response</code>. Try to do the loops as you have written in your question:</p>

<pre><code>for sel in response.xpath('//html'):
</code></pre>

<p>Where <code>.xpath('//html')</code> returns an iterable which you can use in the <code>for</code> loop.</p>

python xpath scrapy scrapy-spider