How to capture all letters from different languages in python?
<p>I have a corpus of different texts from different languages.<br>
I want to capture all characters. I use <strong>python 2.7</strong> and <strong>defaultencodingsetting</strong> is <strong>utf-8</strong>.<br>
I do not know why when I use this code for German umlaut it prints out German umlaut correctly :    </p>

<pre><code>'Erd\xC3\xA4pfel'.decode('unicode-escape').encode('latin1').decode('utf-8')    
</code></pre>

<p>Result is: 
<strong>ErdÃ¤pfel</strong>   </p>

<p>but when I use this code :  </p>

<pre><code>'Erd\xC3\xA4pfel'.decode('unicode-escape').encode('utf-8').decode('utf-8')
</code></pre>

<p>result is :
<strong>ErdÃÂ¤pfel</strong> which is different.  </p>

<p>I am not familiar with text mining.I know that for example latin1 encoding does not contain French letter which is not desired in my project.
How can I convert all unicode escape strings in my corpus regardless of their language to respective character?    </p>

<p><strong>Utf-8</strong> according to documentations contains all languages but why it does not print out German umlaut correctly while latin1 encoding prints out correctly?  </p>

<p><strong>PS: Lowercase in unicode escaping characters sequences is not the case. I have tried both and results were the same.</strong></p>

<p>You <em>already</em> have UTF-8 encoded data. There are no string literal characters to escape in your bytestring. You are looking at the <code>repr()</code> output of a string where non-printable ASCII characters are shown as escape sequences because that makes the value easily copy-pastable in an ASCII-safe way. The <code>\xc3</code> you see is <em>one byte</em>, not separate characters:</p>

<pre><code>&gt;&gt;&gt; 'Erd\xC3\xA4pfel'
'Erd\xc3\xa4pfel'
&gt;&gt;&gt; 'Erd\xC3\xA4pfel'[3]
'\xc3'
&gt;&gt;&gt; 'Erd\xC3\xA4pfel'[4]
'\xa4'
&gt;&gt;&gt; print 'Erd\xC3\xA4pfel'
ErdÃ¤pfel
</code></pre>

<p>You'd have to use a raw string literal or doubled backslashes to actually getting escape sequences that <code>unicode-escape</code> would handle:</p>

<pre><code>&gt;&gt;&gt; '\\xc3\\xa4'
'\\xc3\\xa4'
&gt;&gt;&gt; '\\xc3\\xa4'[0]
'\\'
&gt;&gt;&gt; '\\xc3\\xa4'[1]
'x'
&gt;&gt;&gt; '\\xc3\\xa4'[2]
'c'
&gt;&gt;&gt; '\\xc3\\xa4'[3]
'3'
&gt;&gt;&gt; print '\\xc3\\xa4'
\xc3\xa4
</code></pre>

<p>Note how there is a separate <code>\</code> backslash character in that string (echoed as <code>\\</code>, <em>escaped</em> again).</p>

<p>Next to interpreting actual escape sequences he <code>unicode-escape</code> decodes your data as Latin-1, so you end up with a Unicode string with the character <a href="https://codepoints.net/U+00C3" rel="nofollow">U+00C3 LATIN CAPITAL LETTER A WITH TILDE</a> in it. Encoding that back to Latin-1 bytes gives you the <code>\xC3</code> byte again, and you are back to UTF-8 bytes. Decoding then as UTF-8 works correctly.</p>

<p>But your second attempt encoded the U+00C3 LATIN CAPITAL LETTER A WITH TILDE codepoint to UTF-8, and <em>that</em> encoding gives you the byte sequence <code>\xc3\x83</code>. Printing those bytes to your UTF-8 terminal will show the <code>Ã</code> character. The other byte, <code>\xA4</code> became <a href="https://codepoints.net/U+00A4" rel="nofollow">U+00A4 CURRENCY SIGN</a>, and the UTF-8 byte sequence for that is <code>\xc2\xa4</code>, which prints as <code>Â¤</code>.</p>

<p>There is <em>absolutely no need</em> to decode as <code>unicode-escape</code> here. Just leave the data <em>as is</em>. Or, perhaps, decode as UTF-8 to get a <code>unicode</code> object:</p>

<pre><code>&gt;&gt;&gt; 'Erd\xC3\xA4pfel'.decode('utf8')
u'Erd\xe4pfel'
&gt;&gt;&gt; print 'Erd\xC3\xA4pfel'.decode('utf8')
ErdÃ¤pfel
</code></pre>

<p>If your <strong>actual</strong> data (and not the test you did) contains <code>\xhh</code> escape sequences that encode UTTF-8, then don't use <code>unicode-escape</code> to decode those sequences <em>either</em>. Use <code>string-escape</code> so you get a byte string containing UTF-8 data (which you can then decode to Unicode as needed):</p>

<pre><code>&gt;&gt;&gt; 'Erd\\xc3\\xa4pfel'
'Erd\\xc3\\xa4pfel'
&gt;&gt;&gt; 'Erd\\xc3\\xa4pfel'.decode('string-escape')
'Erd\xc3\xa4pfel'
&gt;&gt;&gt; 'Erd\\xc3\\xa4pfel'.decode('string-escape').decode('utf8')
u'Erd\xe4pfel'
&gt;&gt;&gt; print 'Erd\\xc3\\xa4pfel'.decode('string-escape').decode('utf8')
ErdÃ¤pfel
</code></pre>

python unicode