python - different encoding results by setting the "encoding" parameter in diff. functions
<p>I have a function like </p>

<pre><code>f = open('workfile', 'r', encoding='utf-8')
df = pandas.read_csv(...)
</code></pre>

<p>, which opens a csv file. When I tried to set the <code>encoding</code> parameter to <code>read_csv</code> function, I got other encoding results, than by setting the parameter to <code>open()</code> function: for example the <strong>â¬</strong> and <strong>Ã¶</strong> characters were misinterpreted by setting the parameter to <code>read_csv</code> function. Can anyone explain me, why is it so?</p>

<p>This is what happens in your program</p>

<pre><code>f = open('workfile', 'r', encoding='utf-8') # 1
df = pandas.read_csv(f, encoding=e) # 2
</code></pre>

<p>(1) The file is told to decode bytes using the encoding 'utf-8'. If you print the representation of the file handle f it will show something like </p>

<pre><code> &lt;_io.TextIOWrapper name='workfile' mode='r' encoding='utf-8'&gt;
</code></pre>

<p>When you pull text from this wrapper you will get a unicode string.</p>

<p>(2) read_csv() is told to use some encoding e. Therefore it will convert the unicode string to bytes (doing an implicit encode() -- which will use 'utf-8' on my system) which then are decoded with the decoding e.</p>

<p>Here is a little test program for illustration</p>

<pre><code>import pandas

for file in ['workfile-utf-8.csv', 'workfile-cp1252.csv']:
    for file_encoding in ['utf-8', 'cp1252']:
        for pandas_encoding in [None, 'utf-8', 'cp1252']:
            with open(file, 'r', encoding=file_encoding) as fp:
                try:
                    print('***', file, fp, pandas_encoding)
                    df = pandas.read_csv(fp, encoding=pandas_encoding)
                    print(df)
                except Exception as ex:
                    print(ex)
</code></pre>

<p>The files mentioned are in an encoding that is reflected in their name.</p>

<p>The output should be like this (may depend on your environment)</p>

<pre><code>(1) workfile-utf-8.csv &lt;_io.TextIOWrapper name='workfile-utf-8.csv' mode='r'
encoding='utf-8'&gt; None
       a          b        c
0      HÃ¤llo       â¬uro       Ãl
(2) workfile-utf-8.csv &lt;_io.TextIOWrapper name='workfile-utf-8.csv' mode='r'
encoding='utf-8'&gt; utf-8
       a          b        c
0      HÃ¤llo       â¬uro       Ãl
(3) workfile-utf-8.csv &lt;_io.TextIOWrapper name='workfile-utf-8.csv' mode='r'
encoding='utf-8'&gt; cp1252
        a            b         c
0      HÃÂ¤llo       Ã¢âÂ¬uro       Ãâl
(4) workfile-utf-8.csv &lt;_io.TextIOWrapper name='workfile-utf-8.csv' mode='r'
encoding='cp1252'&gt; None
        a            b         c
0      HÃÂ¤llo       Ã¢âÂ¬uro       Ãâl
(5) workfile-utf-8.csv &lt;_io.TextIOWrapper name='workfile-utf-8.csv' mode='r'
encoding='cp1252'&gt; utf-8
        a            b         c
0      HÃÂ¤llo       Ã¢âÂ¬uro       Ãâl
(6) workfile-utf-8.csv &lt;_io.TextIOWrapper name='workfile-utf-8.csv' mode='r'
encoding='cp1252'&gt; cp1252
          a                b            c
0      HÃÆÃÂ¤llo       ÃÂ¢Ã¢â¬Å¡ÃÂ¬uro       ÃÆÃ¢â¬âl
(7) workfile-cp1252.csv &lt;_io.TextIOWrapper name='workfile-cp1252.csv'
mode='r' encoding='utf-8'&gt; None
Error tokenizing data. C error: Calling read(nbytes) on source failed. Try engine='python'.
(8) workfile-cp1252.csv &lt;_io.TextIOWrapper name='workfile-cp1252.csv'
mode='r' encoding='utf-8'&gt; utf-8
Error tokenizing data. C error: Calling read(nbytes) on source failed. Try engine='python'.
(9) workfile-cp1252.csv &lt;_io.TextIOWrapper name='workfile-cp1252.csv'
mode='r' encoding='utf-8'&gt; cp1252
Error tokenizing data. C error: Calling read(nbytes) on source failed. Try engine='python'.
(10) workfile-cp1252.csv &lt;_io.TextIOWrapper name='workfile-cp1252.csv'
mode='r' encoding='cp1252'&gt; None
       a          b        c
0      HÃ¤llo       â¬uro       Ãl
(11) workfile-cp1252.csv &lt;_io.TextIOWrapper name='workfile-cp1252.csv'
mode='r' encoding='cp1252'&gt; utf-8
       a          b        c
0      HÃ¤llo       â¬uro       Ãl
(12) workfile-cp1252.csv &lt;_io.TextIOWrapper name='workfile-cp1252.csv'
mode='r' encoding='cp1252'&gt; cp1252
        a            b         c
0      HÃÂ¤llo       Ã¢âÂ¬uro       Ãâl
</code></pre>

<p>(1) File is utf-8 -> decode utf-8 -> use as is -> ok</p>

<p>(2) File is utf-8 -> decode utf-8 -> (encode w. default) -> (decode utf-8) -> ok here, but not in other environments</p>

<p>(3) File is utf-8 -> decode utf-8 -> (encode w. default) -> (decode cp1252) -> turns HÃ¤llo into HÃÂ¤llo, etc.</p>

<p>...</p>

<p>(7) File is cp1252 -> decode utf-8 -> raises UnicodeDecodeError, and leads to an error</p>

<p>...</p>

<p>(11) File is cp1252 -> decode cp1252 -> (encode w. default) -> (decode utf-8) -> ok here, but not in other environments</p>

<p>...</p>

<p>Interesting (and funny to look at) is in particular case (6) turning HÃ¤llo,â¬uro, Ãl into HÃÆÃÂ¤llo,ÃÂ¢Ã¢â¬Å¡ÃÂ¬uro,ÃÆÃ¢â¬âl</p>

<p>It corresponds to a sequence:</p>

<pre><code>&gt;&gt;&gt; x1 = 'HÃ¤llo,â¬uro, Ãl'
&gt;&gt;&gt; x1
'HÃ¤llo,â¬uro, Ãl'
&gt;&gt;&gt; x2 = x1.encode()
&gt;&gt;&gt; x2
b'H\xc3\xa4llo,\xe2\x82\xacuro, \xc3\x96l'
&gt;&gt;&gt; x3 = x2.decode('cp1252')
&gt;&gt;&gt; x3
'HÃÂ¤llo,Ã¢âÂ¬uro, Ãâl'
&gt;&gt;&gt; x4 = x3.encode()
&gt;&gt;&gt; x4
b'H\xc3\x83\xc2\xa4llo,\xc3\xa2\xe2\x80\x9a\xc2\xacuro, \xc3\x83\xe2\x80\x93l'
&gt;&gt;&gt; x5 = x4.decode('cp1252')
&gt;&gt;&gt; x5
'HÃÆÃÂ¤llo,ÃÂ¢Ã¢â¬Å¡ÃÂ¬uro, ÃÆÃ¢â¬âl'
</code></pre>

python pandas character-encoding