chcp 65001 codepage results in program termination without any error
<p><strong>Problem</strong><br>
The problem arises when I want to <strong>input</strong> Unicode character in Python interpreter (for simplicity I have used a-umlaut in the example, but I have first encountered this for Farsi characters). Whenever I use python with <code>chcp 65001</code> code page and then try to input even one Unicode character, Python exits without any error.</p>

<p>I have spent days trying to solve this problem to no avail. But today, I found a thread on <a href="https://bugs.python.org/issue1602" rel="nofollow">python website</a>, another on <a href="https://bugs.mysql.com/bug.php?id=66682" rel="nofollow">MySQL</a> and another on Lua-users which issues were raised regarding this sudden exit, although without any solution and some saying that <code>chcp 65001</code> is inherently broken.</p>

<p>It would be good to know once and for all whether this problem is chcp-design-related or there is a possible workaround.</p>

<p><strong>Reproduce Error</strong></p>

<p><code>chcp 65001</code></p>

<blockquote>
  <p>Python 3.X:</p>
</blockquote>

<p>Python shell</p>

<p><code>print('Ã¤')</code></p>

<p>result: it just exits the shell</p>

<p><strong>however</strong>, this works <code>python.exe -c "print('Ã¤')"</code>
and also this : <code>print('\u00e4')</code></p>

<p>result: Ã¤</p>

<blockquote>
  <p>in Luajit2.0.4</p>
</blockquote>

<p><code>print('Ã¤')</code></p>

<p>result: it just exits the shell</p>

<p>however this works: <code>print('\xc3\xa4')</code></p>

<p><strong>I have come up with this observation so far:</strong></p>

<ol>
<li>direct output with the command prompt works.</li>
<li>Unicode-based , hex-based equivalent of the character works.</li>
</ol>

<p><strong>So</strong>
  This is not a Python bug <strong>and</strong> that we can't use a Unicode character directly in CLI programs in Windows command prompt or any of its Wrapper like Conemu, Cmder (I am using Cmder to be able to see and use Unicode character in Windows shell and I have done so without any problem). Is this correct?</p>

<p>To use Unicode in the Windows console for Python 2.7 and 3.x (prior to 3.6), install and enable <a href="https://pypi.python.org/pypi/win_unicode_console">win_unicode_console</a>. This uses the wide-character functions <a href="https://msdn.microsoft.com/en-us/library/ms684958"><code>ReadConsoleW</code></a> and <a href="https://msdn.microsoft.com/en-us/library/ms687401"><code>WriteConsoleW</code></a>, just like other Unicode-aware console programs such as cmd.exe and powershell.exe. For Python 3.6, a new <code>io._WindowsConsoleIO</code> raw I/O class has been added. It reads and writes UTF-8 encoded text (for cross-platform compatibility with Unix -- "get a byte" -- programs), but internally it uses the wide-character API by transcoding to and from UTF-16LE. </p>

<p>The problem you're experiencing with non-ASCII input is reproducible in the console for all Windows versions up to and including Windows 10. The console host process, i.e. conhost.exe, wasn't designed for UTF-8 (codepage 65001) and hasn't been updated to support it consistently. In particular, non-ASCII input causes an empty read. This in turn causes Python's REPL to exit and built-in <code>input</code> to raise <code>EOFError</code>. </p>

<p>The problem is that conhost encodes its UTF-16 input buffer assuming a single-byte codepage, such as the OEM and ANSI codepages in Western locales (e.g. 437, 850, 1252). UTF-8 is a multibyte encoding in which non-ASCII characters are encoded as 2 to 4 bytes. To handle UTF-8 it would need to encode in multiple iterations of <code>M / 4</code> characters, where M is the remaining bytes available from the N-byte buffer. Instead it assumes a request to read N bytes is a request to read N characters. Then if the input has one or more non-ASCII characters, the internal <code>WideCharToMultiByte</code> call fails due to an undersized buffer, and the console returns a 'successful' read of 0 bytes.</p>

<p>You may not observe exactly this problem in Python 3.5 if the pyreadline module is installed. Python 3.5 automatically tries to import <code>readline</code>. In the case of pyreadline, input is read via the wide-character function <a href="https://msdn.microsoft.com/en-us/library/ms684961"><code>ReadConsoleInputW</code></a>. This is a low-level function to read console input records. In principle it should work, but in practice entering <code>print('Ã¤')</code> gets read by the REPL as <code>print('')</code>. For a non-ASCII character, <code>ReadConsoleInputW</code> returns a sequence of Alt+Numpad <code>KEY_EVENT</code> records. The sequence is a lossy OEM encoding, which can be ignored except for the last record, which has the input character in the <code>UnicodeChar</code> field. Apparently pyreadline ignores the entire sequence.</p>

<p>Prior to Windows 8, output using codepage 65001 is also broken. It prints a trail of garbage text in proportion to the number of non-ASCII characters. In this case the problem is that <code>WriteFile</code> and <code>WriteConsoleA</code> incorrectly return the number of UTF-16 codes written to the screen buffer instead of the number of UTF-8 bytes. This confuses Python's buffered writer, leading to repeated writes of what it thinks are the remaining unwritten bytes. This problem was fixed in Windows 8 as part of rewriting the internal console API to use the ConDrv device instead of an LPC port. Older versions of Windows can use ConEmu or ANSICON to work around this bug.</p>

python windows unicode cmd codepages