Why is 'Ã©' and 'Ã©' encoding to different bytes?
<h1>Question</h1>

<p>Why is the same character encoding to different bytes in different parts of my code base?</p>

<h1>Context</h1>

<p>I have a unit test that generates a temporary file tree and then checks to make sure my scan actually finds the file in question.</p>

<pre><code>def test_unicode_file_name():
    test_regex = "Ã©"
    file_tree = {"files": ["Ã©"]} # File created with python.open()
    with TempTree(file_tree) as tmp_tree:
        import pdb; pdb.set_trace()
        result = tasks.find_files(test_regex, root_path=tmp_tree.root_path)
        expected = [os.path.join(tmp_tree.root_path, "Ã©")]
        assert result == expected
</code></pre>

<h1>Function that's failing</h1>

<pre><code>for dir_entry in scandir(current_path):
    if dir_entry.is_dir():
        dirs_to_search.append(dir_entry.path)

    if dir_entry.is_file():
        testing = dir_entry.name
        if filename_regex.match(testing):
            results.append(dir_entry.path)
</code></pre>

<h1>PDB Session</h1>

<p>When I started digging into things I found that the test character (copied from my unit test) and the character in <code>dir_entry.name</code> encoded to different bytes.</p>

<pre><code>(Pdb) testing
'Ã©'
(Pdb) 'Ã©'
'Ã©'
(Pdb) testing == 'Ã©'
False
(Pdb) testing in 'Ã©'
False
(Pdb) type(testing)
&lt;class 'str'&gt;
(Pdb) type('Ã©')
&lt;class 'str'&gt;
(Pdb) repr(testing)
"'Ã©'"
(Pdb) repr('Ã©')
"'Ã©'"
(Pdb) 'Ã©'.encode("utf-8")
b'\xc3\xa9'
(Pdb) testing.encode("utf-8")
b'e\xcc\x81'
</code></pre>

<p>Your operating system (MacOS, at a guess) has converted the filename <code>'Ã©'</code> to <a href="https://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms" rel="nofollow">Unicode Normal Form D</a>, decomposing it into an unaccented <code>'e'</code> and a combining acute accent. You can see this clearly with a quick session in the Python interpreter:</p>

<pre><code>&gt;&gt;&gt; import unicodedata
&gt;&gt;&gt; e1 = b'\xc3\xa9'.decode()
&gt;&gt;&gt; e2 = b'e\xcc\x81'.decode()
&gt;&gt;&gt; [unicodedata.name(c) for c in e1]
['LATIN SMALL LETTER E WITH ACUTE']
&gt;&gt;&gt; [unicodedata.name(c) for c in e2]
['LATIN SMALL LETTER E', 'COMBINING ACUTE ACCENT']
</code></pre>

<p>To ensure that you're comparing like with like, you can convert the filename given by <code>dir_entry.name</code> back to Normal Form C before testing it against your regex:</p>

<pre><code>import unicodedata

for dir_entry in scandir(current_path):
    if dir_entry.is_dir():
        dirs_to_search.append(dir_entry.path)

    if dir_entry.is_file():
        testing = unicodedata.normalize('NFC', dir_entry.name)
        if filename_regex.match(testing):
            results.append(dir_entry.path)
</code></pre>

python python-3.x unicode normalization