Scrapy - TypeError: Cannot convert unicode body - HtmlResponse has no encoding
<p>When I try to construct a <code>HtmlResponse</code> object in Scrapy like this:</p>

<pre><code>scrapy.http.HtmlResponse(url=self.base_url + dealer_url[0], body=dealer_html)
</code></pre>

<p>I got this error:</p>

<pre><code>Traceback (most recent call last):

  File "d:\kerja\hit\python~1\oliver~1\machin~2\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks

    current.result = callback(current.result, *args, **kw)

  File "D:\Kerja\HIT\Python Projects\Oliver Morris\machinery\machinery\machinery\spiders\fwi.py", line 69, in parse_items

    dealer_page = scrapy.http.HtmlResponse(url=self.base_url + dealer_url[0], body=dealer_html)

  File "d:\kerja\hit\python~1\oliver~1\machin~2\lib\site-packages\scrapy\http\response\text.py", line 27, in __init__

    super(TextResponse, self).__init__(*args, **kwargs)

  File "d:\kerja\hit\python~1\oliver~1\machin~2\lib\site-packages\scrapy\http\response\__init__.py", line 18, in __init__

    self._set_body(body)

  File "d:\kerja\hit\python~1\oliver~1\machin~2\lib\site-packages\scrapy\http\response\text.py", line 43, in _set_body

    type(self).__name__)

TypeError: Cannot convert unicode body - HtmlResponse has no encoding
</code></pre>

<p>Does anyone know how to solve this error? </p>

<p><a href="http://doc.scrapy.org/en/latest/topics/request-response.html#htmlresponse-objects" rel="nofollow">HtmlResponse</a> is trying to detect encoding:</p>

<blockquote>
  <p>The HtmlResponse class is a subclass of TextResponse which adds
  encoding auto-discovering support by looking into the HTML meta
  http-equiv attribute. See TextResponse.encoding.</p>
</blockquote>

<p>So basically the html string you provide to <code>body</code> parameter(<code>dealer_html</code> in your case) doesn't have encoding specified.
As per <a href="http://www.w3schools.com/TAGS/att_meta_http_equiv.asp" rel="nofollow">w3 docs of <code>http-equiv</code></a> it should have:</p>

<blockquote>
<pre><code>HTML 4.01: &lt;meta http-equiv="content-type" content="text/html; charset=UTF-8"&gt;
HTML5: &lt;meta charset="UTF-8"&gt;
</code></pre>
</blockquote>

<p>In this case you can either fix your html or specify encoding when creating the <code>HtmlResponse</code> object via <code>encoding</code> parameter:</p>

<pre><code>HtmlResponse(url='http://scrapy.org', body=u'some body', encoding='utf-8')
</code></pre>

python python-2.7 encoding scrapy scrapy-spider