Python codecs encoding not working
<p>I have this code</p>

<pre><code>import collections
import csv
import sys
import codecs
from xml.dom.minidom import parse
import xml.dom.minidom

String = collections.namedtuple("String", ["tag", "text"])

def read_translations(filename): #Reads a csv file with rows made up of 2 columns: the string tag, and the translated tag
    with codecs.open(filename, "r", encoding='utf-8') as csvfile:
        csv_reader = csv.reader(csvfile, delimiter=",")
        result = [String(tag=row[0], text=row[1]) for row in csv_reader]
    return result
</code></pre>

<p>The CSV file I'm reading contains Brazilian portuguese characters. When I try to run this, I get an error: </p>

<pre><code>'utf8' codec can't decode byte 0x88 in position 21: invalid start byte
</code></pre>

<p>I'm using Python 2.7. As you can see, I'm encoding with codecs, but it doesn't work. </p>

<p>Any ideas?</p>

<p>The idea of this line:</p>

<pre><code>with codecs.open(filename, "r", encoding='utf-8') as csvfile:
</code></pre>

<p>is to say "This file was saved as utf-8. Please make appropriate conversions when reading from it."</p>

<p>That works fine if the file was actually saved as utf-8. If some other encoding was used, then it is bad.</p>

<p><strong>What then?</strong></p>

<p>Determine which encoding was used. Assuming the information cannot be obtained from the software which created the file - guess.</p>

<p>Open the file normally and print each line:</p>

<pre><code>with open(filename, 'rt') as f:
    for line in f:
        print repr(line)
</code></pre>

<p>Then look for a character which is not ASCII, e.g. Ã± - this letter will be printed as some code, e.g.:</p>

<pre><code>'espa\xc3\xb1ol'
</code></pre>

<p>Above, Ã± is represented as <code>\xc3\xb1</code>, because that is the utf-8 sequence for it.</p>

<p>Now, you can check what various encodings would give and see which is right:</p>

<pre><code>&gt;&gt;&gt; ntilde = u'\N{LATIN SMALL LETTER N WITH TILDE}'
&gt;&gt;&gt; 
&gt;&gt;&gt; print repr(ntilde.encode('utf-8'))
'\xc3\xb1'
&gt;&gt;&gt; print repr(ntilde.encode('windows-1252'))
'\xf1'
&gt;&gt;&gt; print repr(ntilde.encode('iso-8859-1'))
'\xf1'
&gt;&gt;&gt; print repr(ntilde.encode('macroman'))
'\x96'
</code></pre>

<p>Or print all of them:</p>

<pre><code>for c in encodings.aliases.aliases:
    try:
        encoded = ntilde.encode(c)
        print c, repr(encoded)
    except:
        pass
</code></pre>

<p>Then, when you have guessed which encoding it is, use that, e.g.:</p>

<pre><code>with codecs.open(filename, "r", encoding='iso-8859-1') as csvfile:
</code></pre>

python python-2.7