Converting double slash utf-8 encoding
<p>I cannot get this to work!  I have a text file from a save game file parser with a bunch of UTF-8 Chinese names in it in byte form, like this in the source.txt:</p>

<p>\xe6\x89\x8e\xe5\x8a\xa0\xe6\x8b\x89</p>

<p>But, no matter how I import it into Python (3 or 2), I get this string, at best:</p>

<p>\\xe6\\x89\\x8e\\xe5\\x8a\\xa0\\xe6\\x8b\\x89</p>

<p>I have tried, like other threads have suggested, to re-encode the string as UTF-8 and then decode it with unicode escape, like so:</p>

<pre><code>stringName.encode("utf-8").decode("unicode_escape")
</code></pre>

<p>But then it messes up the original encoding, and gives this as the string:</p>

<p>'Ã¦\x89\x8eÃ¥\x8a\xa0Ã¦\x8b\x89' (printing this string results in: Ã¦Ã¥ Ã¦ )</p>

<p>Now, if I manually copy and paste b + the original string in the filename and encode this, I get the correct encoding.  For example:</p>

<pre><code>b'\xe6\x89\x8e\xe5\x8a\xa0\xe6\x8b\x89'.encode("utf-8")
</code></pre>

<p>Results in:  'æå æ'</p>

<p>But, I can't do this programmatically.  I can't even get rid of the double slashes.  </p>

<p>To be clear, source.txt contains single backslashes.  I have tried importing it in many ways, but this is the most common: </p>

<pre><code>with open('source.txt','r',encoding='utf-8') as f_open:
    source = f_open.read()
</code></pre>

<p>Okay, so I clicked the answer below (I think), but here is what works:</p>

<pre><code>from ast import literal_eval
decodedString = literal_eval("b'{}'".format(stringVariable)).decode('utf-8')
</code></pre>

<p>I can't use it on the whole file because of other encoding issues, but extracting each name as a string (stringVariable) and then doing that works!  Thank you!</p>

<p>To be more clear, the original file is not just these messed up utf encodings.  It only uses them for certain fields.  For example, here is the beginning of the file:</p>

<pre><code>{'m_cacheHandles': ['s2ma\x00\x00CN\x1f\x1b"\x8d\xdb\x1fr \\\xbf\xd4D\x05R\x87\x10\x0b\x0f9\x95\x9b\xe8\x16T\x81b\xe4\x08\x1e\xa8U\x11',
                's2ma\x00\x00CN\x1a\xd9L\x12n\xb9\x8aL\x1d\xe7\xb8\xe6\xf8\xaa\xa1S\xdb\xa5+\t\xd3\x82^\x0c\x89\xdb\xc5\x82\x8d\xb7\x0fv',
                's2ma\x00\x00CN\x92\xd8\x17D\xc1D\x1b\xf6(\xedj\xb7\xe9\xd1\x94\x85\xc8`\x91M\x8btZ\x91\xf65\x1f\xf9\xdc\xd4\xe6\xbb',
                's2ma\x00\x00CN\xa1\xe9\xab\xcd?\xd2PS\xc9\x03\xab\x13R\xa6\x85u7(K2\x9d\x08\xb8k+\xe2\xdeI\xc3\xab\x7fC',
                's2ma\x00\x00CNN\xa5\xe7\xaf\xa0\x84\xe5\xbc\xe9HX\xb93S*sj\xe3\xf8\xe7\x84`\xf1Ye\x15~\xb93\x1f\xc90',
                's2ma\x00\x00CN8\xc6\x13F\x19\x1f\x97AH\xfa\x81m\xac\xc9\xa6\xa8\x90s\xfdd\x06\rL]z\xbb\x15\xdcI\x93\xd3V'],
'm_campaignIndex': 0,
'm_defaultDifficulty': 7,
'm_description': '',
'm_difficulty': '',
'm_gameSpeed': 4,
'm_imageFilePath': '',
'm_isBlizzardMap': True,
'm_mapFileName': '',
'm_miniSave': False,
'm_modPaths': None,
'm_playerList': [{'m_color': {'m_a': 255, 'm_b': 255, 'm_g': 92,   'm_r': 36},
               'm_control': 2,
               'm_handicap': 0,
               'm_hero': '\xe6\x89\x8e\xe5\x8a\xa0\xe6\x8b\x89',
</code></pre>

<p>All of the information before the 'm_hero': field is not utf-8.  So using ShadowRanger's solution works if the file is only made up of these fake utf-encodings, but it doesn't work when I have already parsed m_hero as a string and try to convert that.  Karin's solution does work for that.</p>

<p>I'm assuming you're using Python 3. In Python 2, strings are bytes by default, so it would just work for you. But in Python 3, strings are unicode and interpretted as unicode, which is what makes this problem harder if you have a byte string being read as unicode.</p>

<p>This solution was inspired by mgilson's answer. We can literally evaluate your unicode string as a byte string by using <a href="https://docs.python.org/3/library/ast.html#ast.literal_eval" rel="nofollow"><code>literal_eval</code></a>:</p>

<pre><code>from ast import literal_eval

with open('source.txt', 'r', encoding='utf-8') as f_open:
    source = f_open.read()
    string = literal_eval("b'{}'".format(source)).decode('utf-8')
    print(string)  # æå æ
</code></pre>

python unicode encoding utf-8