python write file dealing with encode
<p>I'm confused. I need HELP!!!
I'm dealing with a file contains Chinese characters,for instance, let's call it <code>a.TEST</code>, and here is what's inside.</p>

<pre><code>ä½ å¥½ ä¸­å½ Hello China 1 2 3
</code></pre>

<p>You don't need to understand what the chinese means.(Actually it's 'hello China')</p>

<pre><code>&gt;&gt;&gt; f=open('wr.TRAIN')
&gt;&gt;&gt; print f.read()
ä½ å¥½ ä¸­å½ Hello China 1 2 3

&gt;&gt;&gt; f.seek(0)
&gt;&gt;&gt; content = f.readline()
&gt;&gt;&gt; content
'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd Hello China 1 2 3\n'
&gt;&gt;&gt; print content
ä½ å¥½ ä¸­å½ Hello China 1 2 3
&gt;&gt;&gt; type(content)
&lt;type 'str'&gt;
&gt;&gt;&gt; isinstance(content,unicode)
False
</code></pre>

<p>Here comes the <strong>first Question</strong>: Why python shell give me the <code>utf-8</code>of <code>content</code> when i just type <code>content</code>,meanwhile <code>print content</code> cmd can output the form that I want to see?</p>

<p>The <strong>Second Question</strong>: what's the difference between <code>unicode</code> and <code>str</code>?
Someone told me that <code>encode</code> is convert <code>unicode</code> to <code>str</code>, but what i learned from <a href="https://docs.python.org/2/howto/unicode.html" rel="nofollow">Unicode HowTo</a> tells me <code>encode</code> is convert <code>unicode</code> to <code>utf-8</code></p>

<p>Not over yet! :)</p>

<p>here is <code>test.py</code></p>

<pre><code>#!/usr/bin/python
#-*- coding: utf-8 -*-

fr = open('a.TEST')
fw = open('out.TEST','w')

content = fr.readline()
content_list = content.split()
print content
fw.write('{0}'.format(content_list))

fr.close()
fw.close()
</code></pre>

<p><a href="http://i.stack.imgur.com/4XevK.png" rel="nofollow"><img src="http://i.stack.imgur.com/4XevK.png" alt="enter image description here"></a></p>

<p><a href="http://i.stack.imgur.com/Oc52y.png" rel="nofollow"><img src="http://i.stack.imgur.com/Oc52y.png" alt="enter image description here"></a></p>

<p><strong>Third Question</strong>:Why the chinese character turn into utf-8 code when I do <code>.split()</code>?</p>

<p>and I thought <code>fw.write('{0}'.format(content_list).decode('utf-8'))</code> will work, but it doesn't. 
I don't want what's written into <code>out.TEST</code> is character encoding form, I want it to be exactly the character that look like originallyï¼ä½ å¥½ï¼. How to do it?</p>

<p>In the beginning, there was just english characters, and people was not satisfied.</p>

<p>Then they want to display every character in the world.But there is problem. One byte can only represent 255 characters. There just simply not enough place to hold them. </p>

<p>Then people decide to use two byte to represent one character.And call it 'utf8'.</p>

<p>No matter what characters you write in, it's all store in byte form.</p>

<p>In Python, there is no such datatype called 'unicode', just 'str'. And 'unicode' is an encoding system of 'str'. </p>

<p>'\xe4\xbd\xa0\xe5\xa5\xbd \xe4\xb8\xad\xe5\x9b\xbd' is byte form of "ä½ å¥½ ä¸­å½".
It can not display without an encoding system specified.</p>

<p>I suppose you could blame linux/unix. Python has no problem to display 'utf-8' characters, while 'cat' cannot.</p>

python unicode encoding utf-8 character-encoding