Python2: Using .decode with errors='replace' still returns errors
<p>So I have a <code>message</code> which is read from a file of unknown encoding. I want to send to a webpage for display. I've grappled a lot with UnicodeErrors and have gone through many Q&amp;As on StackOverflow and think I have decent understand of how Unicode and encoding works. My current code looks like this</p>

<pre><code>try :
            return message.decode(encoding='utf-8')
        except:
            try:
                return message.decode(encoding='latin-1')
            except:
                try:
                    print("Unable to entirely decode in latin or utf-8, will replace error characters with '?'")
                    return message.decode(encoding='utf-8', errors="replace")
</code></pre>

<p>The returned message is then dumped into a JSON and send to the front end.</p>

<p>I assumed that because I'm using <code>errors="replace"</code>on the last <code>try except</code> that I was going to avoid exceptions at the expense of having a few '?' characters in my display. An acceptable cost.</p>

<p>However, it seems that I was too hopeful, and for some files I still get a <code>UnicodeDecodeException</code> saying "ascii codecs cannot decode" for some character. Why doesn't <code>errors="replace"</code> just take care of this? </p>

<p>(also as a bonus question, what does ascii have to do with any of this?.. I'm specifying UTF-8)</p>

<p>decode with error replace implements the 'replace' error handling (for <strong>text encodings</strong> only): substitutes '?' for encoding errors (to be encoded by the codec), and '\ufffd' (the Unicode replacement character) for decoding errors</p>

<p>text encodings means A "codec which encodes Unicode strings to bytes."</p>

<p>maybe your data is malformed - u should try 'ignore' error handling where malformed data is ignored and encoding or decoding is continued without further notice.</p>

<pre><code>message.decode(encoding='utf-8', errors="ignore")
</code></pre>

python python-2.7 unicode character-encoding