Python2: Using .decode with errors='replace' still returns errors
<p>So I have a <code>message</code> which is read from a file of unknown encoding. I want to send to a webpage for display. I've grappled a lot with UnicodeErrors and have gone through many Q&amp;As on StackOverflow and think I have decent understand of how Unicode and encoding works. My current code looks like this</p>

<pre><code>try :
            return message.decode(encoding='utf-8')
        except:
            try:
                return message.decode(encoding='latin-1')
            except:
                try:
                    print("Unable to entirely decode in latin or utf-8, will replace error characters with '?'")
                    return message.decode(encoding='utf-8', errors="replace")
</code></pre>

<p>The returned message is then dumped into a JSON and send to the front end.</p>

<p>I assumed that because I'm using <code>errors="replace"</code>on the last <code>try except</code> that I was going to avoid exceptions at the expense of having a few '?' characters in my display. An acceptable cost.</p>

<p>However, it seems that I was too hopeful, and for some files I still get a <code>UnicodeDecodeException</code> saying "ascii codecs cannot decode" for some character. Why doesn't <code>errors="replace"</code> just take care of this? </p>

<p>(also as a bonus question, what does ascii have to do with any of this?.. I'm specifying UTF-8)</p>

<p>You should not get a <code>UnicodeDecodeError</code> with <code>errors='replace'</code>. Also <code>str.decode('latin-1')</code> should never fail, because ISO-8859-1 has a valid character mapping for every possible byte sequence.</p>

<p>My suspicion is that <code>message</code> is already a <code>unicode</code> string, not bytes. Unicode text has already been âdecodedâ from bytes and can't be decoded any more.</p>

<p>When you call <code>.decode()</code> an a <code>unicode</code> string, Python 2 tries to be helpful and decides to <em>encode</em> the Unicode string back to bytes (using the default encoding), so that you have something that you can really decode. This implicit encoding step <em>doesn't</em> use <code>errors='replace'</code>, so if there are any characters in the Unicode string that aren't in the default encoding (probably ASCII) you'll get a <code>Unicode<strong>En</strong>codeError</code>.</p>

<p>(Python 3 no longer does this as it is terribly confusing.)</p>

<p>Check the type of <code>message</code> and assuming it is indeed <code>Unicode</code>, work back from there to find where it was decoded (possibly implicitly) to replace that with the correct decoding.</p>

python python-2.7 unicode character-encoding