Converting double slash utf-8 encoding
<p>I cannot get this to work!  I have a text file from a save game file parser with a bunch of UTF-8 Chinese names in it in byte form, like this in the source.txt:</p>

<p>\xe6\x89\x8e\xe5\x8a\xa0\xe6\x8b\x89</p>

<p>But, no matter how I import it into Python (3 or 2), I get this string, at best:</p>

<p>\\xe6\\x89\\x8e\\xe5\\x8a\\xa0\\xe6\\x8b\\x89</p>

<p>I have tried, like other threads have suggested, to re-encode the string as UTF-8 and then decode it with unicode escape, like so:</p>

<pre><code>stringName.encode("utf-8").decode("unicode_escape")
</code></pre>

<p>But then it messes up the original encoding, and gives this as the string:</p>

<p>'Ã¦\x89\x8eÃ¥\x8a\xa0Ã¦\x8b\x89' (printing this string results in: Ã¦Ã¥ Ã¦ )</p>

<p>Now, if I manually copy and paste b + the original string in the filename and encode this, I get the correct encoding.  For example:</p>

<pre><code>b'\xe6\x89\x8e\xe5\x8a\xa0\xe6\x8b\x89'.encode("utf-8")
</code></pre>

<p>Results in:  'æå æ'</p>

<p>But, I can't do this programmatically.  I can't even get rid of the double slashes.  </p>

<p>To be clear, source.txt contains single backslashes.  I have tried importing it in many ways, but this is the most common: </p>

<pre><code>with open('source.txt','r',encoding='utf-8') as f_open:
    source = f_open.read()
</code></pre>

<p>Okay, so I clicked the answer below (I think), but here is what works:</p>

<pre><code>from ast import literal_eval
decodedString = literal_eval("b'{}'".format(stringVariable)).decode('utf-8')
</code></pre>

<p>I can't use it on the whole file because of other encoding issues, but extracting each name as a string (stringVariable) and then doing that works!  Thank you!</p>

<p>To be more clear, the original file is not just these messed up utf encodings.  It only uses them for certain fields.  For example, here is the beginning of the file:</p>

<pre><code>{'m_cacheHandles': ['s2ma\x00\x00CN\x1f\x1b"\x8d\xdb\x1fr \\\xbf\xd4D\x05R\x87\x10\x0b\x0f9\x95\x9b\xe8\x16T\x81b\xe4\x08\x1e\xa8U\x11',
                's2ma\x00\x00CN\x1a\xd9L\x12n\xb9\x8aL\x1d\xe7\xb8\xe6\xf8\xaa\xa1S\xdb\xa5+\t\xd3\x82^\x0c\x89\xdb\xc5\x82\x8d\xb7\x0fv',
                's2ma\x00\x00CN\x92\xd8\x17D\xc1D\x1b\xf6(\xedj\xb7\xe9\xd1\x94\x85\xc8`\x91M\x8btZ\x91\xf65\x1f\xf9\xdc\xd4\xe6\xbb',
                's2ma\x00\x00CN\xa1\xe9\xab\xcd?\xd2PS\xc9\x03\xab\x13R\xa6\x85u7(K2\x9d\x08\xb8k+\xe2\xdeI\xc3\xab\x7fC',
                's2ma\x00\x00CNN\xa5\xe7\xaf\xa0\x84\xe5\xbc\xe9HX\xb93S*sj\xe3\xf8\xe7\x84`\xf1Ye\x15~\xb93\x1f\xc90',
                's2ma\x00\x00CN8\xc6\x13F\x19\x1f\x97AH\xfa\x81m\xac\xc9\xa6\xa8\x90s\xfdd\x06\rL]z\xbb\x15\xdcI\x93\xd3V'],
'm_campaignIndex': 0,
'm_defaultDifficulty': 7,
'm_description': '',
'm_difficulty': '',
'm_gameSpeed': 4,
'm_imageFilePath': '',
'm_isBlizzardMap': True,
'm_mapFileName': '',
'm_miniSave': False,
'm_modPaths': None,
'm_playerList': [{'m_color': {'m_a': 255, 'm_b': 255, 'm_g': 92,   'm_r': 36},
               'm_control': 2,
               'm_handicap': 0,
               'm_hero': '\xe6\x89\x8e\xe5\x8a\xa0\xe6\x8b\x89',
</code></pre>

<p>All of the information before the 'm_hero': field is not utf-8.  So using ShadowRanger's solution works if the file is only made up of these fake utf-encodings, but it doesn't work when I have already parsed m_hero as a string and try to convert that.  Karin's solution does work for that.</p>

<p>So there are several different ways to interpret having the data "in byte form." Let's assume you really do:</p>

<pre class="lang-py prettyprint-override"><code>s = b'\xe6\x89\x8e\xe5\x8a\xa0\xe6\x8b\x89'
</code></pre>

<p>The <code>b</code> prefix indicates those are bytes. Without getting into the 
whole mess that is bytes vs codepoints/characters and the long differences
between Python 2 and 3, the <code>b</code>-prefixed string indicates those are intended 
to be bytes (e.g. raw UTF-8 bytes).</p>

<p>Then just decode it, which converts UTF-8 encoding (which you already
have in the bytes, into true Unicode characters. In Python 2.7, e.g.:</p>

<pre><code>print s.decode('utf-8')
</code></pre>

<p>yields:</p>

<pre><code>æå æ
</code></pre>

<p>One of your examples did an encode followed by a decode, which can only lead to sorrow and pain. If your variable holds true UTF-8 bytes, you only need the decode.</p>

<p><strong>Update</strong> Based on discussion, it appears the data isn't really in UTF-8 bytes, but a string-serialized version of same. There are a lot of ways to get from string serial to bytes. Here's mine:</p>

<pre><code>from struct import pack

def byteize(s):
    """
    Given a backslash-escaped string serialization of bytes,
    decode it into a genuine byte string.
    """
    bvals = [int(s[i:i+2], 16) for i in range(2, len(s), 4)]
    return pack(str(len(bvals)) + 'B', *bvals)
</code></pre>

<p>Then:</p>

<pre><code>print byteize(s).decode('utf-8')
</code></pre>

<p>as before yields:</p>

<pre><code>æå æ
</code></pre>

<p>This <code>byteize()</code> isn't as general as the <code>literal_eval()</code>-based <a href="http://stackoverflow.com/a/39283896/240490">accepted answer</a>, but <code>%timeit</code> benchmarking shows it to be about 33% faster on short strings. It could be further accelerated by swapping out <code>range</code> for <code>xrange</code> under Python 2. The <code>literal_eval</code> approach wins handily on long strings, however, given its lower-level nature.</p>

<pre><code>100000 loops, best of 3: 6.19 Âµs per loop
100000 loops, best of 3: 8.3 Âµs per loop
</code></pre>

python unicode encoding utf-8