Tracking down implicit unicode conversions in Python 2
<p>I have a large project where at various places problematic implicit Unicode conversions (coersions) were used in the form of e.g.:
</p>

<pre class="lang-python prettyprint-override"><code>someDynamicStr = "bar" # could come from various sources

# works
u"foo" + someDynamicStr
u"foo{}".format(someDynamicStr)

someDynamicStr = "\xff" # uh-oh

# raises UnicodeDecodeError
u"foo" + someDynamicStr
u"foo{}".format(someDynamicStr)
</code></pre>

<p>(Possibly other forms as well.)</p>

<p><strong>Now I would like to track down those usages, especially those in actively used code.</strong></p>

<p>It would be great if I could easily replace the <code>unicode</code> constructor with a wrapper which checks whether the input is of type <code>str</code> and the <code>encoding</code>/<code>errors</code> parameters are set to the default values and then notifies me (prints traceback or such).</p>

<p><em>/edit:</em></p>

<p>While not directly related to what I am looking for I came across this gloriously horrible hack for how to make the decode exception go away altogether (the decode one only, i.e. <code>str</code> to <code>unicode</code>, but not the other way around, see <a href="https://mail.python.org/pipermail/python-list/2012-July/627506.html" rel="nofollow">https://mail.python.org/pipermail/python-list/2012-July/627506.html</a>).</p>

<p>I don't plan on using it but it might be interesting for those battling problems with invalid Unicode input and looking for a quick fix (but please think about the side effects):</p>

<pre class="lang-python prettyprint-override"><code>import codecs
codecs.register_error("strict", codecs.ignore_errors)
codecs.register_error("strict", lambda x: (u"", x.end)) # alternatively
</code></pre>

<p>(An internet search for <code>codecs.register_error("strict"</code> revealed that apparently it's used in some real projects.)</p>

<p><em>/edit #2:</em></p>

<p>For explicit conversions I made a snippet with the help of <a href="http://stackoverflow.com/a/4025310">a SO post on monkeypatching</a>:</p>

<pre class="lang-python prettyprint-override"><code>class PatchedUnicode(unicode):
  def __init__(self, obj=None, encoding=None, *args, **kwargs):
    if encoding in (None, "ascii", "646", "us-ascii"):
        print("Problematic unicode() usage detected!")
    super(PatchedUnicode, self).__init__(obj, encoding, *args, **kwargs)

import __builtin__
__builtin__.unicode = PatchedUnicode
</code></pre>

<p>This only affects explicit conversions using the <code>unicode()</code> constructor directly so it's not something I need.</p>

<p><em>/edit #3:</em></p>

<p>The thread "<a href="https://stackoverflow.com/questions/6738987/extension-method-for-python-built-in-types">Extension method for python built-in types!</a>" makes me think that it might actually not be easily possible (in CPython at least).</p>

<p><em>/edit #4:</em></p>

<p>It's nice to see many good answers here, too bad I can only give out the bounty once.</p>

<p>In the meantime I came across a somewhat similar question, at least in the sense of what the person tried to achieve: <a href="http://stackoverflow.com/q/2851481">Can I turn off implicit Python unicode conversions to find my mixed-strings bugs?</a>
Please note though that throwing an exception would <strong>not</strong> have been OK in my case. Here I was looking for something which might point me to the different locations of problematic code (e.g. by printing smth.) but not something which might exit the program or change its behavior (because this way I can prioritize what to fix).</p>

<p>On another note, the people working on the Mypy project (which include Guido van Rossum) might also come up with something similar helpful in the future, see the discussions at <a href="https://github.com/python/mypy/issues/1141" rel="nofollow">https://github.com/python/mypy/issues/1141</a> and more recently <a href="https://github.com/python/typing/issues/208" rel="nofollow">https://github.com/python/typing/issues/208</a>.</p>

<p><em>/edit #5</em></p>

<p>I also came across the following but didn't have yet the time to test it: <a href="https://pypi.python.org/pypi/unicode-nazi" rel="nofollow">https://pypi.python.org/pypi/unicode-nazi</a></p>

<p>You can register a custom encoding which prints a message whenever it's used:</p>

<p>Code in <code>ourencoding.py</code>:</p>

<pre><code>import sys
import codecs
import traceback

# Define a function to print out a stack frame and a message:

def printWarning(s):
    sys.stderr.write(s)
    sys.stderr.write("\n")
    l = traceback.extract_stack()
    # cut off the frames pointing to printWarning and our_encode
    l = traceback.format_list(l[:-2])
    sys.stderr.write("".join(l))

# Define our encoding:

originalencoding = sys.getdefaultencoding()

def our_encode(s, errors='strict'):
    printWarning("Default encoding used");
    return (codecs.encode(s, originalencoding, errors), len(s))

def our_decode(s, errors='strict'):
    printWarning("Default encoding used");
    return (codecs.decode(s, originalencoding, errors), len(s))

def our_search(name):
    if name == 'our_encoding':
        return codecs.CodecInfo(
            name='our_encoding',
            encode=our_encode,
            decode=our_decode);
    return None

# register our search and set the default encoding:
codecs.register(our_search)
reload(sys)
sys.setdefaultencoding('our_encoding')
</code></pre>

<p>If you import this file at the start of our script, then you'll see warnings for implicit conversions:</p>

<pre><code>#!python2
# coding: utf-8

import ourencoding

print("test 1")
a = "hello " + u"world"

print("test 2")
a = "hello âº " + u"world"

print("test 3")
b = u" ".join(["hello", u"âº"])

print("test 4")
c = unicode("hello âº")
</code></pre>

<p>output:</p>

<pre><code>test 1
test 2
Default encoding used
 File "test.py", line 10, in &lt;module&gt;
   a = "hello âº " + u"world"
test 3
Default encoding used
 File "test.py", line 13, in &lt;module&gt;
   b = u" ".join(["hello", u"âº"])
test 4
Default encoding used
 File "test.py", line 16, in &lt;module&gt;
   c = unicode("hello âº")
</code></pre>

<p>It's not perfect as test 1 shows, if the converted string only contain ASCII characters, sometimes you won't see a warning.</p>

python python-2.7 debugging unicode monkeypatching