How to handle unknow encoding
<p>I'm having some issues with a Python script that needs to open files with different encoding.</p>

<p><strong>I'm usually using this:</strong></p>

<pre><code>with open(path_to_file, 'r') as f:
    first_line = f.readline()
</code></pre>

<p>And that works great when the file is properly encode.</p>

<p><strong>But sometimes, it doesn't work, for example <a href="http://www.cjoint.com/doc/16_09/FInqYhqOOYi_test.Txt" rel="nofollow">with this file</a>, I've got this:</strong></p>

<pre><code>In [22]: with codecs.open(filename, 'r') as f:
    ...:    a = f.readline()
    ...:    print(a)
    ...:    print(repr(a))
    ...:     
ï¿½ï¿½Test for StackOverlow

'\xff\xfeT\x00e\x00s\x00t\x00 \x00f\x00o\x00r\x00 \x00S\x00t\x00a\x00c\x00k\x00O\x00v\x00e\x00r\x00l\x00o\x00w\x00\r\x00\n'
</code></pre>

<p><strong>And I would like to search some stuff on those lines. Sadly with that method, I can't:</strong></p>

<pre><code>In [24]: "Test" in a
Out[24]: False
</code></pre>

<p><strong>I've found a lot of questions here referring to the same type of issues:</strong></p>

<ol>
<li><a href="http://stackoverflow.com/questions/491921/unicode-utf8-reading-and-writing-to-files-in-python">Unicode (utf8) reading and writing to files in python</a></li>
<li><a href="http://stackoverflow.com/questions/22216076/unicodedecodeerror-utf8-codec-cant-decode-byte-0xa5-in-position-0-invalid-s">UnicodeDecodeError: &#39;utf8&#39; codec can&#39;t decode byte 0xa5 in position 0: invalid start byte</a></li>
<li><a href="http://programmers.stackexchange.com/questions/187169/how-to-detect-the-encoding-of-a-file">http://programmers.stackexchange.com/questions/187169/how-to-detect-the-encoding-of-a-file</a></li>
<li><a href="http://stackoverflow.com/questions/1979171/how-can-i-escape-xff-xfe-to-a-readable-string">how can i escape &#39;\xff\xfe&#39; to a readable string</a></li>
</ol>

<p>But can't manage to decode the file properly with them... </p>

<p><strong>With codecs.open():</strong></p>

<pre><code>In [17]: with codecs.open(filename, 'r', "utf-8") as f:
    a = f.readline()
    print(a)
   ....:     
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
&lt;ipython-input-17-0e72208eaac2&gt; in &lt;module&gt;()
      1 with codecs.open(filename, 'r', "utf-8") as f:
----&gt; 2     a = f.readline()
      3     print(a)
      4 

/usr/lib/python2.7/codecs.pyc in readline(self, size)
    688     def readline(self, size=None):
    689 
--&gt; 690         return self.reader.readline(size)
    691 
    692     def readlines(self, sizehint=None):

/usr/lib/python2.7/codecs.pyc in readline(self, size, keepends)
    543         # If size is given, we call read() only once
    544         while True:
--&gt; 545             data = self.read(readsize, firstline=True)
    546             if data:
    547                 # If we're at a "\r" read one extra character (which might

/usr/lib/python2.7/codecs.pyc in read(self, size, chars, firstline)
    490             data = self.bytebuffer + newdata
    491             try:
--&gt; 492                 newchars, decodedbytes = self.decode(data, self.errors)
    493             except UnicodeDecodeError, exc:
    494                 if firstline:

UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte
</code></pre>

<p><strong>with encode('utf-8):</strong></p>

<pre><code>In [18]: with codecs.open(filename, 'r') as f:
    a = f.readline()
    print(a)
   ....:     a.encode('utf-8')
   ....:     print(a)
   ....:     
ï¿½ï¿½Test for StackOverlow

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
&lt;ipython-input-18-7facc05b9cb1&gt; in &lt;module&gt;()
      2     a = f.readline()
      3     print(a)
----&gt; 4     a.encode('utf-8')
      5     print(a)
      6 

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
</code></pre>

<p><strong>I've found a way to change file encoding automatically with Vim:</strong></p>

<pre><code>system("vim '+set fileencoding=utf-8' '+wq' %s" % path_to_file)
</code></pre>

<p>But I would like to do this without using Vim...</p>

<p>Any help will be appreciate.</p>

<p>It looks like you need to detect the encoding in the input file. The <code>chardet</code> library mentioned in the answer to <a href="http://stackoverflow.com/questions/436220/python-is-there-a-way-to-determine-the-encoding-of-text-file">this question</a> might help (though note the proviso that complete encoding detection is not possible).</p>

<p>Then you can write the file out in a known encoding, perhaps. When dealing with Unicode remember that it MUST be encoded into a suitable bytestream before being communicated outside the process. Decode on input, then encode on output.</p>

python python-2.7 encoding