python os.walk and unicode error
<p>two questions:
1. why does </p>

<pre><code>In [21]:                                                                                   
   ....:     for root, dir, file in os.walk(spath):
   ....:         print(root)
</code></pre>

<p>print the whole tree but</p>

<pre><code>In [6]: for dirs in os.walk(spath):                             
...:     print(dirs)    
</code></pre>

<p>chokes on this unicode error?</p>

<pre><code>UnicodeEncodeError: 'charmap' codec can't encode character '\u2122' in position 1477: character maps to &lt;undefined&gt;
</code></pre>

<p>[NOTE: this is the TM symbol]</p>

<ol start="2">
<li>I looked at these answers</li>
</ol>

<p><a href="http://stackoverflow.com/questions/22184178/scraping-works-well-until-i-get-this-error-ascii-codec-cant-encode-character">Scraping works well until I get this error: &#39;ascii&#39; codec can&#39;t encode character u&#39;\u2122&#39; in position</a></p>

<p><a href="http://stackoverflow.com/questions/30539882/whats-the-deal-with-python-3-4-unicode-different-languages-and-windows/30551552#30551552">What&#39;s the deal with Python 3.4, Unicode, different languages and Windows?</a></p>

<p><a href="http://stackoverflow.com/questions/16346914/python-3-2-unicodeencodeerror-charmap-codec-cant-encode-character-u2013-i?noredirect=1&amp;lq=1">python 3.2 UnicodeEncodeError: &#39;charmap&#39; codec can&#39;t encode character &#39;\u2013&#39; in position 9629: character maps to &lt;undefined&gt;</a></p>

<p><a href="https://github.com/Drekin/win-unicode-console" rel="nofollow">https://github.com/Drekin/win-unicode-console</a></p>

<p><a href="https://docs.python.org/3/search.html?q=IncrementalDecoder&amp;check_keywords=yes&amp;area=default" rel="nofollow">https://docs.python.org/3/search.html?q=IncrementalDecoder&amp;check_keywords=yes&amp;area=default</a></p>

<p>and tried these variations</p>

<pre><code>----&gt; 1 print(dirs, encoding='utf-8')                                                           
TypeError: 'encoding' is an invalid keyword argument for this function       
In [11]: &gt;&gt;&gt; u'\u2122'.encode('ascii', 'ignore')                                                
Out[11]: b''                       

print(dirs).encode(âutf=8â)
</code></pre>

<p>all to no effect.</p>

<p>This was done with python 3.4.3 and visual studio code 1.6.1 on Windows 10.  The default settings in Visual Studio Code include:</p>

<blockquote>
  <p>// The default character set encoding to use when reading and writing
  files.
      "files.encoding": "utf8",</p>
</blockquote>

<p>python 3.4.3
visual studio code 1.6.1
ipython 3.0.0</p>

<p><strong>UPDATE EDIT</strong>
I tried this again in the Sublime Text REPL, running a script. Here's what I got:</p>

<pre><code># -*- coding: utf-8 -*-
import os

spath = 'C:/Users/Semantic/Documents/Align' 

with open('os_walk4_align.txt', 'w') as f:
    for path, dirs, filenames in os.walk(spath):
        print(path, dirs, filenames, file=f)

Traceback (most recent call last):
File "listdir_test1.py", line 8, in &lt;module&gt;
print(path, dirs, filenames, file=f)
File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2605' in position 300: character maps to &lt;undefined&gt;
</code></pre>

<p>This code is only 217 characters long, so where does âposition 300â come from?</p>

<p>Here's a test case:</p>

<pre><code>C:\TEST
ââââdir1
â       file1â¢
â
ââââdir2
        file2
</code></pre>

<p>Here's a script (Python 3.x):</p>

<pre><code>import os

spath = r'c:\test'

for root,dirs,files in os.walk(spath):
    print(root)

for dirs in os.walk(spath):                             
    print(dirs)
</code></pre>

<p>Here's the output, on an IDE that supports UTF-8 (PythonWin, in this case):</p>

<pre><code>c:\test
c:\test\dir1
c:\test\dir2
('c:\\test', ['dir1', 'dir2'], [])
('c:\\test\\dir1', [], ['file1â¢'])
('c:\\test\\dir2', [], ['file2'])
</code></pre>

<p>Here's the output, on my Windows console, which defaults to <code>cp437</code>:</p>

<pre><code>c:\test
c:\test\dir1
c:\test\dir2
('c:\\test', ['dir1', 'dir2'], [])
Traceback (most recent call last):
  File "C:\test.py", line 9, in &lt;module&gt;
    print(dirs)
  File "C:\Python33\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2122' in position 47: character maps to &lt;undefined&gt;
</code></pre>

<p>For Question 1, the reason <code>print(root)</code> works is that no directory had a character that wasn't supported by the output encoding, but <code>print(dirs)</code> is now printing a tuple containing <code>(root,dirs,files)</code> and one of the files has an unsupported character in the Windows console.</p>

<p>For Question 2, the first example misspelled <code>utf-8</code> as <code>utf=8</code>, and the second example didn't declare an encoding for the file the output was written to, so it used a default that didn't support the character.</p>

<p>Try this:</p>

<pre><code>import os

spath = r'c:\test'

with open('os_walk4_align.txt', 'w', encoding='utf8') as f:
    for path, dirs, filenames in os.walk(spath):
        print(path, dirs, filenames, file=f)
</code></pre>

<p>Content of <code>os_walk4_align.txt</code>, encoded in UTF-8:</p>

<pre><code>c:\test ['dir1', 'dir2'] []
c:\test\dir1 [] ['file1â¢']
c:\test\dir2 [] ['file2']
</code></pre>

python unicode encoding utf-8 vscode