Python how to decode unicode with hex characters
<p>I have extracted a string from web crawl script as following:</p>

<pre><code>u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
</code></pre>

<p>I want to decode <code>u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'</code> with utf-8.
With <a href="http://ddecode.com/hexdecoder/" rel="nofollow">http://ddecode.com/hexdecoder/</a>, I can see the result is <code>'ãä¸­å­ã'</code></p>

<p>I tried using the following syntax but failed.</p>

<pre><code>msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
result = msg.decode('utf8')
</code></pre>

<p>Error:</p>

<pre><code>Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-11: ordi
nal not in range(128)
</code></pre>

<p>May I ask how to decode the string correctly?</p>

<p>Thanks for help.</p>

<p>The problem with</p>

<pre><code>msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
result = msg.decode('utf8')
</code></pre>

<p>is that you are trying to decode Unicode. That doesn't really make sense. You can encode <em>from</em> Unicode to some type of encoding, or you can decode a byte string <em>to</em> Unicode.</p>

<p>When you do  </p>

<pre><code>msg.decode('utf8')
</code></pre>

<p>Python 2 sees that <code>msg</code> is Unicode. It knows that it can't decode Unicode so it "helpfully" assumes that you want to encode <code>msg</code> with the default ASCII codec so the result of that transformation can be decoded to Unicode using the UTF-8 codec. Python 3 behaves much more sensibly: that code would simply fail with </p>

<pre><code>AttributeError: 'str' object has no attribute 'decode'
</code></pre>

<p>The technique given in kennytm's answer: </p>

<pre><code>msg.encode('latin1').decode('utf-8')
</code></pre>

<p>works because the Unicode codepoints less than 256 correspond directly to the characters in the <a href="https://en.wikipedia.org/wiki/Latin-1_Supplement_%28Unicode_block%29" rel="nofollow">Latin1</a> encoding (aka ISO 8859-1). </p>

<p>Here's some Python 2 code that illustrates this:</p>

<pre><code>for i in xrange(256):
    lat = chr(i)
    uni = unichr(i)
    assert lat == uni.encode('latin1')
    assert lat.decode('latin1') == uni
</code></pre>

<p>And here is the equivalent Python 3 code:</p>

<pre><code>for i in range(256):
    lat = bytes([i])
    uni = chr(i)
    assert lat == uni.encode('latin1')
    assert lat.decode('latin1') == uni
</code></pre>

<p>You may find this article helpful: <a href="http://nedbatchelder.com/text/unipain.html" rel="nofollow">Pragmatic Unicode</a>, which was written by SO veteran Ned Batchelder.</p>

<p>Unless you are forced to use Python 2 I strongly advise you to switch to Python 3. It will make handling Unicode far less painful.</p>

python utf-8 python-2.x