python write file dealing with encode
<p>I'm confused. I need HELP!!!
I'm dealing with a file contains Chinese characters,for instance, let's call it <code>a.TEST</code>, and here is what's inside.</p>

<pre><code>ä½ å¥½ ä¸­å½ Hello China 1 2 3
</code></pre>

<p>You don't need to understand what the chinese means.(Actually it's 'hello China')</p>

<pre><code>&gt;&gt;&gt; f=open('wr.TRAIN')
&gt;&gt;&gt; print f.read()
ä½ å¥½ ä¸­å½ Hello China 1 2 3

&gt;&gt;&gt; f.seek(0)
&gt;&gt;&gt; content = f.readline()
&gt;&gt;&gt; content
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
&gt;&gt;&gt; print content
ä½ å¥½ ä¸­å½ Hello China 1 2 3
&gt;&gt;&gt; type(content)
&lt;type 'str'&gt;
&gt;&gt;&gt; isinstance(content,unicode)
False
</code></pre>

<p>Here comes the <strong>first Question</strong>: Why python shell give me the <code>utf-8</code>of <code>content</code> when i just type <code>content</code>,meanwhile <code>print content</code> cmd can output the form that I want to see?</p>

<p>The <strong>Second Question</strong>: what's the difference between <code>unicode</code> and <code>str</code>?
Someone told me that <code>encode</code> is convert <code>unicode</code> to <code>str</code>, but what i learned from <a href="https://docs.python.org/2/howto/unicode.html" rel="nofollow">Unicode HowTo</a> tells me <code>encode</code> is convert <code>unicode</code> to <code>utf-8</code></p>

<p>Not over yet! :)</p>

<p>here is <code>test.py</code></p>

<pre><code>#!/usr/bin/python
#-*- coding: utf-8 -*-

fr = open('a.TEST')
fw = open('out.TEST','w')

content = fr.readline()
content_list = content.split()
print content
fw.write('{0}'.format(content_list))

fr.close()
fw.close()
</code></pre>

<p><a href="http://i.stack.imgur.com/4XevK.png" rel="nofollow"><img src="http://i.stack.imgur.com/4XevK.png" alt="enter image description here"></a></p>

<p><a href="http://i.stack.imgur.com/Oc52y.png" rel="nofollow"><img src="http://i.stack.imgur.com/Oc52y.png" alt="enter image description here"></a></p>

<p><strong>Third Question</strong>:Why the chinese character turn into utf-8 code when I do <code>.split()</code>?</p>

<p>and I thought <code>fw.write('{0}'.format(content_list).decode('utf-8'))</code> will work, but it doesn't. 
I don't want what's written into <code>out.TEST</code> is character encoding form, I want it to be exactly the character that look like originallyï¼ä½ å¥½ï¼. How to do it?</p>

<h2>What is Encoding</h2>

<p>A file consists of bytes. You can represent each byte with a number between 0 and 255 (or 0x00 and 0xFF in hexadecimal).</p>

<p>Text is also written as bytes. There is an agreement on the way text is written. That is an encoding. The most basic encoding is ASCII and other encodings are usually based on it. For example, ASCII defines that number 65 (0x41) represents 'A', 66 (0x42) represents 'B' etc.</p>

<h2>How are Strings Represented</h2>

<p>In python, you can define a string using numeric values:</p>

<pre><code>&gt;&gt;&gt; '\x41\x42\x43'
'ABC'
</code></pre>

<p><code>'\x41\x42\x43'</code> is exactly the same thing as <code>'ABC'</code>. Python will always represent the string using the more readable textual representation (<code>'ABC'</code>).</p>

<p>However, some numeric values are not printable characters, so they will be represented in numeric form:</p>

<pre><code>&gt;&gt;&gt; '\x00\x01\x02\x03\x04'
'\x00\x01\x02\x03\x04'
</code></pre>

<p>Others characters have aliases to make your job easier:</p>

<pre><code>&gt;&gt;&gt; '\x0a\x0d\x09'
'\n\r\t'
</code></pre>

<h2>Different Encodings</h2>

<p><a href="http://www.asciitable.com/" rel="nofollow">ASCII table</a> defines meaning of numbers 0-127 and includes only the english alphabet. Numbers 128-255 are undefined. So, other encodings define a meaning for 128-255. Yet others change the meaning of the whole range 0-255.</p>

<p>There are many encodings and they define 128-255 differently.</p>

<p>For example, character 185 (0xB9) is <code>Ä</code> in windows-1250 encoding, but it is <code>Å¡</code> in iso-8859-2 encoding.</p>

<p>So, what happens if you print <code>\xb9</code>? It depends on the encoding used in the console. In my case (my console uses cp852 encoding) it is:</p>

<pre><code>&gt;&gt;&gt; print '\xb9'
â£
</code></pre>

<p>Because of that ambiguity, string <code>'\xb9'</code> will never be <em>represented</em> as <code>'â£'</code> (nor <code>'Ä'</code>...). That would hide the true value. It will be represented as the numeric value:</p>

<pre><code>&gt;&gt;&gt; '\xb9'
'\xb9'
</code></pre>

<p>Also:</p>

<pre><code>&gt;&gt;&gt; 'â£'
'\xb9'
</code></pre>

<p>See also the string from the question in my console:</p>

<pre><code>&gt;&gt;&gt; content = '\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
&gt;&gt;&gt;
&gt;&gt;&gt; content
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
&gt;&gt;&gt;
&gt;&gt;&gt; print content
ÅÅ»Ã¡ÅÄÅ» ÅÅÅÅÅ¤Å» Hello China 1 2 3
</code></pre>

<p><strong>But what happens if variable is just entered in the console?</strong></p>

<p>When a variable is enteren in cosole without <code>print</code>, its representation is printed. It is the same as the following:</p>

<pre><code>&gt;&gt;&gt; print repr(content)
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
</code></pre>

<h2>What is Unicode?</h2>

<p><a href="http://unicode-table.com/en/#control-character" rel="nofollow">Unicode table</a> aims to define a numeric representation of all characters in the world and more. It can actually do that, because it is not limited to 256 values (or to any other limit actually). This is not an encoding, but a universal mapping of numbers to characters.</p>

<p>For example, unicode defines that number 353 (0x0161) is character <code>Å¡</code>. That is allways true regardless of your locale and encodings you use. That character can be stored in files (or memory) in any encoding which supports <code>Å¡</code>.</p>

<h2>What is UTF-8?</h2>

<p>When encoding a unicode character, one can use any encoding, but not all of them will support all characters.</p>

<p>For example, <code>Å¡</code> (unicode 0x0161) can be encoded in iso-8869-2 as 0xB9, but it cannot be encoded in iso-8869-1 at all.</p>

<p>So, to be able to encode anything, you need an encoding which supports <em>every</em> unicode character. UTF-8 is one of those encodings, but there are others:</p>

<pre><code>&gt;&gt;&gt; u'\u0161'.encode('utf-7')
'+AWE-'
&gt;&gt;&gt; u'\u0161'.encode('utf-8')
'\xc5\xa1'
&gt;&gt;&gt; u'\u0161'.encode('utf-16le')
'a\x01'
&gt;&gt;&gt; u'\u0161'.encode('utf-16be')
'\x01a'
&gt;&gt;&gt; u'\u0161'.encode('utf-32le')
'a\x01\x00\x00'
&gt;&gt;&gt; u'\u0161'.encode('utf-32be')
'\x00\x00\x01a'
</code></pre>

<p>The good thing about utf-8 is that the whole ASCII range is unchanged and as long as only ASCII is used, only one byte is used per character:</p>

<pre><code>&gt;&gt;&gt; u'abcdefg'.encode('utf-8')
'abcdefg'
</code></pre>

<h2>Unicode in Python 2</h2>

<p><strong>Important:</strong> This is really specific to Python 2. Python 3 is different.</p>

<p>Unlike <code>str</code> objects, which are strings of bytes, <code>unicode</code> objects are strings of unicode characters.</p>

<p>They can be encoded into a <code>str</code> in chosen encoding, or decoded from <code>str</code> in chosen encoding.</p>

<p>A unicode string is specified using <code>u</code> before the opening quote. The characters inside are interpreted using current encoding, or they can be specified in numeric format <code>\uHEX</code>:</p>

<pre><code>&gt;&gt;&gt; u'ABCD'
u'ABCD'
&gt;&gt;&gt;
&gt;&gt;&gt; u'\u0041\u0042\u0043'
u'ABC'
&gt;&gt;&gt; u'Å¡Ã¢Å¯'
u'\u0161\xe2\u016f'
</code></pre>

<h1>And Now the Answers</h1>

<h2>First Question</h2>

<ul>
<li><code>contents</code> prints <code>repr(contents)</code></li>
<li><code>print contents</code> prints <code>contents</code></li>
</ul>

<h2>Second Question</h2>

<p><strong>UTF-8</strong> strings are byte strings (<code>str</code>). You get them by encoding the <code>unicode</code>:</p>

<pre><code>&gt;&gt;&gt; u'\u0161'.encode('utf-8')
'\xc5\xa1'
&gt;&gt;&gt; '\xc5\xa1'.decode('utf-8')
u'\u0161'
</code></pre>

<p>So yes, <code>encode</code> converts <code>unicode</code> to <code>str</code>. The <code>str</code> can be utf-8, but it does not have to be.</p>

<h2>Third Question</h2>

<p><strong>A)</strong> <em>"Why the chinese character turn into utf-8 code when I do .split()?"</em></p>

<p>They were utf-8 all the time.</p>

<p><strong>B)</strong> <em>"I thought fw.write('{0}'.format(content_list).decode('utf-8')) will work"</em></p>

<p><code>content_list</code> is not a string. It is a list. When a list is converted to a string, it is done using its <code>repr</code>, which also does <code>repr</code> of all of the contents.</p>

<p>For example:</p>

<pre><code>&gt;&gt;&gt; 'a \n a \n a'
'a \n a \n a'
&gt;&gt;&gt; print 'a \n a \n a'
a
 a
 a
&gt;&gt;&gt; print ['a \n a \n a']
['a \n a \n a']
</code></pre>

<p>The last print printed repr(list) which contains repr(str).</p>

python unicode encoding utf-8 character-encoding