Apparently, Python Strings are not 'born equal'
<p>I'm trying to wrap my brain around the 'text encoding standards'. When interpreting a bunch of bytes as 'text', one has to know which 'encoding sheme' applies. Possible candidates that I know of:</p>

<ul>
<li>ASCII: Very basic encoding scheme, supports 128 characters.</li>
<li>CP-1252: Windows encoding scheme for the Latin alphabet. Also known as 'ANSI'.</li>
<li>UTF-8: A coding scheme for the Unicode table (1.114.112 characters). Represents each character with one byte if possible, more bytes if needed (max. 4 bytes).</li>
<li>UTF-16: Another coding scheme for the Unicode table (1.114.112 characters). Represents each character with min 2 bytes, max 4 bytes.</li>
<li>UTF-32: Yet another coding scheme for the Unicode table. Represents each character with 4 bytes.</li>
<li>. . .</li>
</ul>

<p>Now I would expect that Python consistently uses one encoding scheme for its built-in String type. I did the following test, and the result makes me shiver. I start to believe that Python is not consistently sticking to one encoding scheme to store its Strings internally. In other words: Python Strings seem to be 'not born equal'..</p>

<p><strong>EDIT :</strong></p>

<p>I forgot to mention that I'm using Python 3.x . Sorry :-)</p>

<p><strong>1. The test</strong></p>

<p>I have two simple text files in a folder: <code>myAnsi.txt</code> and <code>myUtf.txt</code>. As you can guess, the first is encoded in the <code>CP-1252</code> encoding scheme, also known as <code>ANSI</code>. The latter is encoded in <code>utf-8</code>. In my test, I open each file and read out its content. I assign the content to a native Python String variable. Then I close the file. After that, I create a new file and write the content of the String variable to that file. Here is the code to do all that:</p>

<pre><code>    ##############################
    #    TEST ON THE ANSI-coded  #
    #    FILE                    #
    ##############################
    import os
    file = open(os.getcwd() + '\\myAnsi.txt', 'r')
    fileText = file.read()
    file.close()

    file = open(os.getcwd() + '\\outputAnsi.txt', 'w')
    file.write(fileText)
    file.close()

    # A print statement here like:
    #    &gt;&gt; print(fileText)
    # will raise an exception.
    # But if you're typing this code in a python terminal,
    # you can just write:
    #    &gt;&gt; fileText
    # and get the content printed. In my case, it is the exact
    # content of the file.
    # PS: I use the native windows cmd.exe as my Python terminal ;-)

    ##############################
    #    TEST ON THE Utf-coded   #
    #    FILE                    #
    ##############################
    import os
    file = open(os.getcwd() + '\\myUtf.txt', 'r')
    fileText = file.read()
    file.close()

    file = open(os.getcwd() + '\\outputUtf.txt', 'w')
    file.write(fileText)
    file.close()

    # A print statement here like:
    #    &gt;&gt; print(fileText)
    # will just work fine (at least for me).

    ############# END OF TEST #############
</code></pre>

<p><strong>2. The result I would expect</strong></p>

<p>Let us suppose that Python consistently sticks to one internal coding scheme - for example <code>utf-8</code> - for all its Strings. Assigning other content to a String would lead to some sort of implicit conversion. Under these assumptions, I would expect both output files to be of the <code>utf-8</code> type:</p>

<pre><code>    outputAnsi.txt   -&gt;   utf-8 encoded
    outputUtf.txt    -&gt;   utf-8 encoded
</code></pre>

<p><strong>3. The result I get</strong></p>

<p>The result I get is this:</p>

<pre><code>    outputAnsi.txt   -&gt;   CP-1252 encoded (ANSI)
    outputUtf.txt    -&gt;   utf-8 encoded
</code></pre>

<p>From these results, I have to conclude that the String variable <code>fileText</code> somehow stores the encoding scheme it adheres to.</p>

<p>Many people tell me in their answers:</p>

<blockquote>
  <p>When no encoding is passed explicitly, <code>open()</code> uses the preferred
  system encoding both for reading and for writing.</p>
</blockquote>

<p>I just cannot wrap my brain around that statement. If open() uses the 'preferred system encoding' - say <code>cp1252</code> as example - then both <code>*.txt</code> outputs should be encoded in that way, wouldn't they?</p>

<p><strong>4. Questions..</strong></p>

<p>My test raises several questions to me:</p>

<p>(1) When I open a file to read its content, how does Python know the encoding scheme of that file? I did not specify it when opening the file.</p>

<p>(2) Apparently a Python String can adhere to any encoding scheme supported by Python. So not all Python Strings are born equal. How do you find out the encoding scheme of a particular String, and how do you convert it? Or how do you make sure your freshly created Python String is of the expected type?</p>

<p>(3) When I create a file, how does Python decide in what encoding scheme the file will be created? I did not specify the encoding scheme when creating those files in my test. Nevertheless, Python made a different (!) decision in each case.</p>

<p><strong>5. Extra information (based on the comments to this question):</strong></p>

<ul>
<li>Python version: Python 3.x (installed from Anaconda)</li>
<li>Operating system: Windows 10</li>
<li>Terminal: Standard Windows command prompt <code>cmd.exe</code></li>
<li>Some questions raised about the temporary variable <code>fileText</code>. Apparently the instruction <code>print(fileText)</code> does not work for the ANSI case. An exception is thrown. But in the python terminal window, I can simply type the variable name <code>fileText</code> and get the file content printed out.</li>
<li>Encoding detection of files: Bottom right corner of Notepad++ for first check, online tool for double check: <a href="https://nlp.fi.muni.cz/projects/chared/" rel="nofollow">https://nlp.fi.muni.cz/projects/chared/</a></li>
<li>The output files <code>outputAnsi.txt</code> and <code>outputUtf.txt</code> do not exist at the start of the test. They are created at the very moment that I issue the <code>open(..)</code> command with the <code>'w'</code> option.</li>
</ul>

<p><strong>6. The actual files (for completeness):</strong></p>

<p>I got several comments encouraging me to share the actual files on which I'm doing this test. Those files were quite large, so I've trimmed them down and re-did the tests. Results are similar. Here are the files (PS: of course, my files contain source code, what else?):</p>

<p><strong>myAnsi.txt</strong></p>

<pre><code>/*
******************************************************************************
**
**  File        : LinkerScript.ld
**
**  Author      : Auto-generated by Ac6 System Workbench
**
**  Abstract    : Linker script for STM32F746NGHx Device from STM32F7 series
**
**  Target      : STMicroelectronics STM32
**
**  Distribution: The file is distributed âas is,â without any warranty
**                of any kind.
**
*****************************************************************************
** @attention
**
** &lt;h2&gt;&lt;center&gt;&amp;copy; COPYRIGHT(c) 2014 Ac6&lt;/center&gt;&lt;/h2&gt;
**
*****************************************************************************
*/

/* Entry Point */
/*ENTRY(Reset_Handler)*/
ENTRY(Default_Handler)

/* Highest address of the user mode stack */
_estack = 0x20050000;    /* end of RAM */

_Min_Heap_Size = 0;      /* required amount of heap  */
_Min_Stack_Size = 0x400; /* required amount of stack */

/* Memories definition */
MEMORY
{
  RAM (xrw)     : ORIGIN = 0x20000000, LENGTH = 320K
  ROM (rx)      : ORIGIN = 0x8000000, LENGTH = 1024K
}
</code></pre>

<p>The print statement of the <code>fileText</code> variable leads to the following exception:</p>

<pre><code>&gt;&gt;&gt; print(fileText)
    Traceback (most recent call last):
      File "&lt;stdin&gt;", line 1, in &lt;module&gt;
      File "C:\Anaconda3\lib\encodings\cp850.py", line 19, in encode
        return codecs.charmap_encode(input,self.errors,encoding_map)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\u201c' in position 357: character maps to &lt;undefined&gt;
</code></pre>

<p>But just typing the name of the variable prints out the contents without problems:</p>

<pre><code>&gt;&gt;&gt; fileText
    ### contents of the file are printed out :-) ###
</code></pre>

<p><strong>myUtf.txt</strong></p>

<pre><code>/*--------------------------------------------------------------------------------------------------------------------*/
/*           _ _ _                                                                                                    */
/*          / -,- \                   __  _            _                                                              */
/*         //  |  \\                 / __\ | ___   ___| | __                   _            _                         */
/*         |   0--,|                / /  | |/ _ \ / __| |/ /    __ ___ _ _  __| |_ __ _ _ _| |_ ___                   */
/*         \\     //               / /___| | (_) | (__|   &lt;    / _/ _ \ ' \(_-&lt;  _/ _` | ' \  _(_-&lt;                   */
/*          \_-_-_/                \____/|_|\___/ \___|_|\_\   \__\___/_||_/__/\__\__,_|_||_\__/__/                   */
/*--------------------------------------------------------------------------------------------------------------------*/


#include "clock_constants.h"
#include "../CMSIS/stm32f7xx.h"
#include "stm32f7xx_hal_rcc.h"


/*--------------------------------------------------------------------------------------------------*/
/*          S y s t e m C o r e C l o c k       i n i t i a l        v a l u e                      */
/*--------------------------------------------------------------------------------------------------*/
/*                                                                                                  */
/* This variable is updated in three ways:                                                          */
/*      1) by calling CMSIS function SystemCoreClockUpdate()                                        */
/*      2) by calling HAL API function HAL_RCC_GetHCLKFreq()                                        */
/*      3) each time HAL_RCC_ClockConfig() is called to configure the system clock frequency        */
/*          Note: If you use this function to configure the system clock; then there                */
/*                is no need to call the 2 first functions listed above, since SystemCoreClock      */
/*                variable is updated automatically.                                                */
/*                                                                                                  */
uint32_t SystemCoreClock = 16000000;
const uint8_t AHBPrescTable[16] = {0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 6, 7, 8, 9};


/*--------------------------------------------------------------------------------------------------*/
/*          S y s t e m C o r e C l o c k       v a l u e      u p d a t e                          */
/*--------------------------------------------------------------------------------------------------*/
/*                                                                                                  */
/* @brief  Update SystemCoreClock variable according to Clock Register Values.                      */
/*         The SystemCoreClock variable contains the core clock (HCLK), it can                      */
/*         be used by the user application to setup the SysTick timer or configure                  */
/*         other parameters.                                                                        */
/*--------------------------------------------------------------------------------------------------*/
</code></pre>

<p>When no encoding is passed explicitly, <a href="https://docs.python.org/3.5/library/functions.html#open" rel="nofollow"><code>open()</code> uses the preferred system encoding</a> both for reading and for writing (not sure exactly how the preferred encoding is detected on Windows).</p>

<p>So, when you write:</p>

<pre><code>file = open(os.getcwd() + '\\myAnsi.txt', 'r')
file = open(os.getcwd() + '\\outputAnsi.txt', 'w')
file = open(os.getcwd() + '\\myUtf.txt', 'r')
file = open(os.getcwd() + '\\outputUtf.txt', 'w')
</code></pre>

<p>All four files are opened using the same encoding, both for reading and for writing.</p>

<p>You have to pass <code>encoding='cp1252'</code> or <code>encoding='utf-8'</code> if you want to be sure that files are opened using these encodings:</p>

<pre><code>file = open(os.getcwd() + '\\myAnsi.txt', 'r', encoding='cp1252')
file = open(os.getcwd() + '\\outputAnsi.txt', 'w', encoding='cp1252')
file = open(os.getcwd() + '\\myUtf.txt', 'r', encoding='utf-8')
file = open(os.getcwd() + '\\outputUtf.txt', 'w', encoding='utf-8')
</code></pre>

<p>(As a side note, I'm not a Windows expert, but I think you can write <code>'myAnsi.txt'</code> instead of <code>os.getcwd() + '\\myAnsi.txt'</code>.)</p>

<hr>

<p>Apart from that, you have to consider that some characters are represented in the same way with different encodings. For example, the string <code>hello</code> has the same representation in ASCII, CP-1252 or UTF-8. In general, you have to use some non-ASCII characters to see some differences:</p>

<pre><code>&gt;&gt;&gt; 'hello'.encode('cp1252')
b'hello'
&gt;&gt;&gt; 'hello'.encode('utf-8')
b'hello'  # different encoding, same byte representation
</code></pre>

<p>Not only that, but some byte strings can be perfectly valid in two distinct encodings, even though they can have different meanings, so that when you try to decode a file with the wrong encoding you don't get an error, but a weird string:</p>

<pre><code>&gt;&gt;&gt; b'\xe2\x82\xac'.decode('utf-8')
'â¬'
&gt;&gt;&gt; b'\xe2\x82\xac'.decode('cp1252')
'Ã¢âÂ¬'  # same byte representation, different string
</code></pre>

<hr>

<p>For the record, <a href="https://docs.python.org/3.5/c-api/unicode.html" rel="nofollow">Python uses UTF-8, UTF-16 or UTF-32</a> to represent strings internally. Python tries to use the "shortest" representation, even though UTF-8 and UTF-16 are used without continuation bytes, so that lookups are always O(1).</p>

<hr>

<p>In short, you have read two files using the system encoding and written two files using the same encoding (therefore without any transformation). The content of the files you have read are compatible with both CP-1252 and UTF-8.</p>

python python-3.x unicode encoding utf-8