How to detect if a string is already utf8-encoded?
<p>I have some strings like this:</p>

<pre><code>u'ThaÃÃÂ¯lande'
</code></pre>

<p>This was "ThaÃ¯lande" and I dont know how it's been encoded, but I need to bring it back to "ThaÃ¯lande", then URL-encode it.</p>

<p>Is there a way to guess if a string has already been encoded with Python 2?</p>

<p>You have what is called a <a href="https://en.wikipedia.org/wiki/Mojibake" rel="nofollow">Mojibake</a>. You could use statistical analysis to see if there is a unusual number of Latin-1 characters in there in a combination typical of UTF-8 bytes, or if there are any CP1252-specific characters in there.</p>

<p>There already is a package that does this for you <em>and</em> repairs the damage if a Mojibake is detected: <a href="https://ftfy.readthedocs.io/en/latest/" rel="nofollow"><code>ftfy</code></a>:</p>

<blockquote>
  <p>The goal of ftfy is to take in bad Unicode and output good Unicode, for use in your Unicode-aware code.</p>
</blockquote>

<p>and</p>

<blockquote>
  <p>The ftfy.fix_encoding() function will look for evidence of mojibake and, when possible, it will undo the process that produced it to get back the text that was supposed to be there.</p>
  
  <p>Does this sound impossible? Itâs really not. UTF-8 is a well-designed encoding that makes it obvious when itâs being misused, and a string of mojibake usually contains all the information we need to recover the original string.</p>
</blockquote>

python python-2.7 character-encoding