Using .replace() method on a string read from HTML file containing Unicode
<p>I want to read a .html file as raw text and replace instances of a substring that contains unicode characters with another substring. Assume that the file <code>mm03.html</code> contains only one line of text:</p>

<pre><code>&lt;span style='font-size:14.0pt'&gt;Â«testÂ»&lt;/span&gt;
</code></pre>

<p>I would like to read <code>mm03.html</code>, parse its raw text as a string, and call replace so that the output will look like this:</p>

<pre><code>&lt;span style='font-size:14.0pt'&gt;TEST&lt;/span&gt;
</code></pre>

<p>The first time I tried to do this, I wrote the following code...</p>

<pre><code># -*- coding: utf-8 -*-
import codecs
htmlBase = codecs.open("mm03.html",'r')
htmlFill = htmlBase.read()
print htmlFill
htmlFill = htmlFill.replace("Â«testÂ»","TEST")
print htmlFill
htmlBase.close()
</code></pre>

<p>...with the expectation that it would first print the original line listed above, and then the the second. Instead, it listed the first line twice.</p>

<p>Okay. So it's probably a Unicode decoding problem, right? Maybe, but when I modify the code according to Unicode-related advice found all over this site, problems of varying shades persist. Moreover, the desired functionality can be achieved by defining htmlBase explicitly as...</p>

<pre><code>htmlBase = """&lt;span style='font-size:14.0pt'&gt;Â«testÂ»&lt;/span&gt;"""
</code></pre>

<p>...which leads me to believe there's something I don't know about reading html files in python. I've tried opening mmo3.html in 'w' mode, but that doesn't seem to work and tends to nuke the original file. It doesn't make much sense that a string read from a read-only file should itself be read-only, but I might be wrong.</p>

<p>Following are several script/output pairs I've chewed through.</p>

<ol>
<li><p>Adding the unquoted character 'u' before the string I wish to replace</p>

<pre><code># -*- coding: utf-8 -*-
import codecs
htmlBase = codecs.open("mm03.html",'r')
htmlFill = htmlBase.read()
print htmlFill
htmlFill = htmlFill.replace(u"Â«testÂ»","TEST")
print htmlFill
htmlBase.close()
</code></pre>

<p>Output:</p>

<pre class="lang-none prettyprint-override"><code>&lt;span style='font-size:14.0pt'&gt;Â½testâ&lt;/span&gt;
Traceback (most recent call last):
  File "test2.py", line 6, in &lt;module&gt;
    htmlFill = htmlFill.replace(u"â¬Â½testâ¬â","TEST")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xab in position 31: ordinal not in range(128)
</code></pre></li>
<li><p>Applying .decode('utf-8') to the string passed from .read()</p>

<pre><code># -*- coding: utf-8 -*-
import codecs
htmlBase = codecs.open("mm03.html",'r')
htmlFill = htmlBase.read().decode('utf-8')
print htmlFill
htmlFill = htmlFill.replace(u"Â«testÂ»","TEST")
print htmlFill
htmlBase.close()
</code></pre>

<p>Output:</p>

<pre class="lang-none prettyprint-override"><code>Traceback (most recent call last):
  File "test2.py", line 4, in &lt;module&gt;
    htmlFill = htmlBase.read().decode('utf-8')
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xab in position 31: invalid start byte
</code></pre></li>
<li><p>Applying .encode('utf-8') to the string passed from .read()</p>

<pre><code># -*- coding: utf-8 -*-
import codecs
htmlBase = codecs.open("mm03.html",'r')
htmlFill = htmlBase.read().encode('utf-8')
print htmlFill
htmlFill = htmlFill.replace(u"Â«testÂ»","TEST")
print htmlFill
htmlBase.close()
</code></pre>

<p>Output:</p>

<pre class="lang-none prettyprint-override"><code>Traceback (most recent call last):
  File "test2.py", line 4, in &lt;module&gt;
    htmlFill = htmlBase.read().encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xab in position 31: ordinal not in range(128)
</code></pre></li>
<li><p>Applying .decode('utf-8') to the string passed from .read(), without the 'u' suffix on the target substring</p>

<pre><code># -*- coding: utf-8 -*-
import codecs
htmlBase = codecs.open("mm03.html",'r')
htmlFill = htmlBase.read().decode('utf-8')
print htmlFill
htmlFill = htmlFill.replace("Â«testÂ»","TEST")
print htmlFill
htmlBase.close()
</code></pre>

<p>Output:</p>

<pre class="lang-none prettyprint-override"><code>Traceback (most recent call last):
  File "test2.py", line 4, in &lt;module&gt;
    htmlFill = htmlBase.read().decode('utf-8')
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xab in position 31: invalid start byte
</code></pre></li>
<li><p>Applying .encode('utf-8') to the string passed from .read(), without the 'u' suffix on the target substring</p>

<pre><code># -*- coding: utf-8 -*-
import codecs
htmlBase = codecs.open("mm03.html",'r')
htmlFill = htmlBase.read().encode('utf-8')
print htmlFill
htmlFill = htmlFill.replace("Â«testÂ»","TEST")
print htmlFill
htmlBase.close()
</code></pre>

<p>Output:</p>

<pre class="lang-none prettyprint-override"><code>Traceback (most recent call last):
  File "test2.py", line 4, in &lt;module&gt;
    htmlFill = htmlBase.read().encode('utf-8')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xab in position 31: ordinal not in range(128)
</code></pre></li>
</ol>

<p>You need to decode the string that you want to replace before passing it to <code>str.replace()</code>. This works for me:</p>

<pre><code># -*- coding: utf-8 -*-
import codecs
htmlBase = codecs.open("mm03.html",'r')
htmlFill = htmlBase.read()
htmlFill = codecs.decode(htmlFill,'utf-8')
substr = codecs.decode("Â«testÂ»",'utf-8')
htmlFill = htmlFill.replace(substr,"TEST")
print htmlFill
htmlBase.close()
</code></pre>

python html string unicode io