Apparently, Python Strings are not 'born equal'
<p>I'm trying to wrap my brain around the 'text encoding standards'. When interpreting a bunch of bytes as 'text', one has to know which 'encoding sheme' applies. Possible candidates that I know of:</p>

<ul>
<li>ASCII: Very basic encoding scheme, supports 128 characters.</li>
<li>CP-1252: Windows encoding scheme for the Latin alphabet. Also known as 'ANSI'.</li>
<li>UTF-8: A coding scheme for the Unicode table (1.114.112 characters). Represents each character with one byte if possible, more bytes if needed (max. 4 bytes).</li>
<li>UTF-16: Another coding scheme for the Unicode table (1.114.112 characters). Represents each character with min 2 bytes, max 4 bytes.</li>
<li>UTF-32: Yet another coding scheme for the Unicode table. Represents each character with 4 bytes.</li>
<li>. . .</li>
</ul>

<p>Now I would expect that Python consistently uses one encoding scheme for its built-in String type. I did the following test, and the result makes me shiver. I start to believe that Python is not consistently sticking to one encoding scheme to store its Strings internally. In other words: Python Strings seem to be 'not born equal'..</p>

<p><strong>EDIT :</strong></p>

<p>I forgot to mention that I'm using Python 3.x . Sorry :-)</p>

<p><strong>1. The test</strong></p>

<p>I have two simple text files in a folder: <code>myAnsi.txt</code> and <code>myUtf.txt</code>. As you can guess, the first is encoded in the <code>CP-1252</code> encoding scheme, also known as <code>ANSI</code>. The latter is encoded in <code>utf-8</code>. In my test, I open each file and read out its content. I assign the content to a native Python String variable. Then I close the file. After that, I create a new file and write the content of the String variable to that file. Here is the code to do all that:</p>

<pre><code>    ##############################
    #    TEST ON THE ANSI-coded  #
    #    FILE                    #
    ##############################
    import os
    file = open(os.getcwd() + '\\myAnsi.txt', 'r')
    fileText = file.read()
    file.close()

    file = open(os.getcwd() + '\\outputAnsi.txt', 'w')
    file.write(fileText)
    file.close()

    # A print statement here like:
    #    &gt;&gt; print(fileText)
    # will raise an exception.
    # But if you're typing this code in a python terminal,
    # you can just write:
    #    &gt;&gt; fileText
    # and get the content printed. In my case, it is the exact
    # content of the file.
    # PS: I use the native windows cmd.exe as my Python terminal ;-)

    ##############################
    #    TEST ON THE Utf-coded   #
    #    FILE                    #
    ##############################
    import os
    file = open(os.getcwd() + '\\myUtf.txt', 'r')
    fileText = file.read()
    file.close()

    file = open(os.getcwd() + '\\outputUtf.txt', 'w')
    file.write(fileText)
    file.close()

    # A print statement here like:
    #    &gt;&gt; print(fileText)
    # will just work fine (at least for me).

    ############# END OF TEST #############
</code></pre>

<p><strong>2. The result I would expect</strong></p>

<p>Let us suppose that Python consistently sticks to one internal coding scheme - for example <code>utf-8</code> - for all its Strings. Assigning other content to a String would lead to some sort of implicit conversion. Under these assumptions, I would expect both output files to be of the <code>utf-8</code> type:</p>

<pre><code>    outputAnsi.txt   -&gt;   utf-8 encoded
    outputUtf.txt    -&gt;   utf-8 encoded
</code></pre>

<p><strong>3. The result I get</strong></p>

<p>The result I get is this:</p>

<pre><code>    outputAnsi.txt   -&gt;   CP-1252 encoded (ANSI)
    outputUtf.txt    -&gt;   utf-8 encoded
</code></pre>

<p>From these results, I have to conclude that the String variable <code>fileText</code> somehow stores the encoding scheme it adheres to.</p>

<p>Many people tell me in their answers:</p>

<blockquote>
  <p>When no encoding is passed explicitly, <code>open()</code> uses the preferred
  system encoding both for reading and for writing.</p>
</blockquote>

<p>I just cannot wrap my brain around that statement. If open() uses the 'preferred system encoding' - say <code>cp1252</code> as example - then both <code>*.txt</code> outputs should be encoded in that way, wouldn't they?</p>

<p><strong>4. Questions..</strong></p>

<p>My test raises several questions to me:</p>

<p>(1) When I open a file to read its content, how does Python know the encoding scheme of that file? I did not specify it when opening the file.</p>

<p>(2) Apparently a Python String can adhere to any encoding scheme supported by Python. So not all Python Strings are born equal. How do you find out the encoding scheme of a particular String, and how do you convert it? Or how do you make sure your freshly created Python String is of the expected type?</p>

<p>(3) When I create a file, how does Python decide in what encoding scheme the file will be created? I did not specify the encoding scheme when creating those files in my test. Nevertheless, Python made a different (!) decision in each case.</p>

<p><strong>5. Extra information (based on the comments to this question):</strong></p>

<ul>
<li>Python version: Python 3.x (installed from Anaconda)</li>
<li>Operating system: Windows 10</li>
<li>Terminal: Standard Windows command prompt <code>cmd.exe</code></li>
<li>Some questions raised about the temporary variable <code>fileText</code>. Apparently the instruction <code>print(fileText)</code> does not work for the ANSI case. An exception is thrown. But in the python terminal window, I can simply type the variable name <code>fileText</code> and get the file content printed out.</li>
<li>Encoding detection of files: Bottom right corner of Notepad++ for first check, online tool for double check: <a href="https://nlp.fi.muni.cz/projects/chared/" rel="nofollow">https://nlp.fi.muni.cz/projects/chared/</a></li>
<li>The output files <code>outputAnsi.txt</code> and <code>outputUtf.txt</code> do not exist at the start of the test. They are created at the very moment that I issue the <code>open(..)</code> command with the <code>'w'</code> option.</li>
</ul>

<p><strong>6. The actual files (for completeness):</strong></p>

<p>I got several comments encouraging me to share the actual files on which I'm doing this test. Those files were quite large, so I've trimmed them down and re-did the tests. Results are similar. Here are the files (PS: of course, my files contain source code, what else?):</p>

<p><strong>myAnsi.txt</strong></p>

<pre><code>/*
******************************************************************************
**
**  File        : LinkerScript.ld
**
**  Author      : Auto-generated by Ac6 System Workbench
**
**  Abstract    : Linker script for STM32F746NGHx Device from STM32F7 series
**
**  Target      : STMicroelectronics STM32
**
**  Distribution: The file is distributed âas is,â without any warranty
**                of any kind.
**
*****************************************************************************
** @attention
**
** &lt;h2&gt;&lt;center&gt;&amp;copy; COPYRIGHT(c) 2014 Ac6&lt;/center&gt;&lt;/h2&gt;
**
*****************************************************************************
*/

/* Entry Point */
/*ENTRY(Reset_Handler)*/
ENTRY(Default_Handler)

/* Highest address of the user mode stack */
_estack = 0x20050000;    /* end of RAM */

_Min_Heap_Size = 0;      /* required amount of heap  */
_Min_Stack_Size = 0x400; /* required amount of stack */

/* Memories definition */
MEMORY
{
  RAM (xrw)     : ORIGIN = 0x20000000, LENGTH = 320K
  ROM (rx)      : ORIGIN = 0x8000000, LENGTH = 1024K
}
</code></pre>

<p>The print statement of the <code>fileText</code> variable leads to the following exception:</p>

<pre><code>&gt;&gt;&gt; print(fileText)
    Traceback (most recent call last):
      File "&lt;stdin&gt;", line 1, in &lt;module&gt;
      File "C:\Anaconda3\lib\encodings\cp850.py", line 19, in encode
        return codecs.charmap_encode(input,self.errors,encoding_map)[0]
    UnicodeEncodeError: 'charmap' codec can't encode character '\u201c' in position 357: character maps to &lt;undefined&gt;
</code></pre>

<p>But just typing the name of the variable prints out the contents without problems:</p>

<pre><code>&gt;&gt;&gt; fileText
    ### contents of the file are printed out :-) ###
</code></pre>

<p><strong>myUtf.txt</strong></p>

<pre><code>/*--------------------------------------------------------------------------------------------------------------------*/
/*           _ _ _                                                                                                    */
/*          / -,- \                   __  _            _                                                              */
/*         //  |  \\                 / __\ | ___   ___| | __                   _            _                         */
/*         |   0--,|                / /  | |/ _ \ / __| |/ /    __ ___ _ _  __| |_ __ _ _ _| |_ ___                   */
/*         \\     //               / /___| | (_) | (__|   &lt;    / _/ _ \ ' \(_-&lt;  _/ _` | ' \  _(_-&lt;                   */
/*          \_-_-_/                \____/|_|\___/ \___|_|\_\   \__\___/_||_/__/\__\__,_|_||_\__/__/                   */
/*--------------------------------------------------------------------------------------------------------------------*/


#include "clock_constants.h"
#include "../CMSIS/stm32f7xx.h"
#include "stm32f7xx_hal_rcc.h"


/*--------------------------------------------------------------------------------------------------*/
/*          S y s t e m C o r e C l o c k       i n i t i a l        v a l u e                      */
/*--------------------------------------------------------------------------------------------------*/
/*                                                                                                  */
/* This variable is updated in three ways:                                                          */
/*      1) by calling CMSIS function SystemCoreClockUpdate()                                        */
/*      2) by calling HAL API function HAL_RCC_GetHCLKFreq()                                        */
/*      3) each time HAL_RCC_ClockConfig() is called to configure the system clock frequency        */
/*          Note: If you use this function to configure the system clock; then there                */
/*                is no need to call the 2 first functions listed above, since SystemCoreClock      */
/*                variable is updated automatically.                                                */
/*                                                                                                  */
uint32_t SystemCoreClock = 16000000;
const uint8_t AHBPrescTable[16] = {0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 6, 7, 8, 9};


/*--------------------------------------------------------------------------------------------------*/
/*          S y s t e m C o r e C l o c k       v a l u e      u p d a t e                          */
/*--------------------------------------------------------------------------------------------------*/
/*                                                                                                  */
/* @brief  Update SystemCoreClock variable according to Clock Register Values.                      */
/*         The SystemCoreClock variable contains the core clock (HCLK), it can                      */
/*         be used by the user application to setup the SysTick timer or configure                  */
/*         other parameters.                                                                        */
/*--------------------------------------------------------------------------------------------------*/
</code></pre>

<p>CP-1252 is basically a byte for byte codec; it can decode arbitrary bytes, including the bytes from UTF-8 encoding. So effectively, assuming you're on Windows using a Western locale, where the default encoding provided to <code>open</code> is <code>cp-1252</code>, if you never work with the string in Python, just read and write it, you may as well have just read and written in binary mode. You'd only see a problem if you tried to use the string in ways that exposed the problem.</p>

<p>For example, consider the following test file with a single UTF-8 encoded character in it:</p>

<pre><code>with open('utf8file.txt', 'w', encoding='utf-8') as f:
    f.write('Ã©')
</code></pre>

<p>The actual bytes in that file are <code>C3 A9</code>.</p>

<p>If you read that file in <code>cp-1252</code>, it will happily do so, because every byte is a legal <code>cp-1252</code> byte:</p>

<pre><code>with open('utf8file.txt') as f:
    data = f.read()
</code></pre>

<p>But it's not the string <code>'Ã©'</code>, it's what those two bytes happen to represent in <code>cp-1252</code>: <code>"ÃÂ©"</code> (you can print them or check the length and you'll see that, assuming your console encoding handles non-ASCII)</p>

<p>If you're just writing it back though, without using it, you'd never see this; the output step is (default) encoding <code>"ÃÂ©"</code> as <code>C9 A9</code>, which restores the original bytes that you expect.</p>

<p>Your problem is that most files are legal <code>cp-1252</code> text files (and it's possible Python will silently read unassigned bytes as equivalent Unicode ordinals; I know it does so for <code>latin-1</code> for unassigned bytes like <code>\x8d</code>), and when they're legal, reading as such and writing back in the same encoding is non-mutating.</p>

python python-3.x unicode encoding utf-8