python replace unicode characters
<p>I wrote a program to read in Windows DNS debugging log, but inside always got some funny characters in the domain field. </p>

<p>Below is one of the example:</p>

<pre>(13)\xc2\xb5\xc2\xb1\xc2\xbe\xc3\xa2p\xc3\xb4\xc2\x8d(5)example(3)com(0)'</pre>

<p>I want to replace all the <code>\x..</code> with a <code>?</code></p>

<p>I explicitly type \xc2 as follows works </p>

<pre><code>line = '(13)\xc2\xb5\xc2\xb1\xc2\xbe\xc3\xa2p\xc3\xb4\xc2\x8d(5)example(3)com(0)'
re.sub('\\\xc2', '?', line)
result: '(13)?\xb5?\xb1?\xbe\xc3\xa2p\xc3\xb4?\x8d(5)example(3)com(0)'
</code></pre>

<p>But its not working if I write as follow:</p>

<p><code>re.sub('\\\x..', '?', line)</code></p>

<p>How I can write a regular expression to replace them all?</p>

<p>There are better tools for this job than regex, you could try for example:</p>

<pre><code>&gt;&gt;&gt; line
'(13)\xc2\xb5\xc2\xb1\xc2\xbe\xc3\xa2p\xc3\xb4\xc2\x8d(5)example(3)com(0)'
&gt;&gt;&gt; line.decode('ascii', 'ignore')
u'(13)p(5)example(3)com(0)'
</code></pre>

<p>That skips non-ascii characters.  Or with replace, you can swap them for a '?' placeholder:</p>

<pre><code>&gt;&gt;&gt; print line.decode('ascii', 'replace')
(13)ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½ï¿½pï¿½ï¿½ï¿½ï¿½(5)example(3)com(0)
</code></pre>

<p>But the best solution is to find out what erroneous encoding/decoding caused the <a href="https://en.wikipedia.org/wiki/Mojibake" rel="nofollow">mojibake</a> to happen in the first place, so you can recover data by using the correct code pages.  </p>

<p>There is an excellent answer about unbaking emojibake <a href="http://stackoverflow.com/a/24141326/674039">here</a>.  Note that it's an inexact science, and a lot of the crucial information is actually in the comment thread under that answer. </p>

python mojibake