Tracking down implicit unicode conversions in Python 2
<p>I have a large project where at various places problematic implicit Unicode conversions (coersions) were used in the form of e.g.:
</p>

<pre class="lang-python prettyprint-override"><code>someDynamicStr = "bar" # could come from various sources

# works
u"foo" + someDynamicStr
u"foo{}".format(someDynamicStr)

someDynamicStr = "\xff" # uh-oh

# raises UnicodeDecodeError
u"foo" + someDynamicStr
u"foo{}".format(someDynamicStr)
</code></pre>

<p>(Possibly other forms as well.)</p>

<p><strong>Now I would like to track down those usages, especially those in actively used code.</strong></p>

<p>It would be great if I could easily replace the <code>unicode</code> constructor with a wrapper which checks whether the input is of type <code>str</code> and the <code>encoding</code>/<code>errors</code> parameters are set to the default values and then notifies me (prints traceback or such).</p>

<p><em>/edit:</em></p>

<p>While not directly related to what I am looking for I came across this gloriously horrible hack for how to make the decode exception go away altogether (the decode one only, i.e. <code>str</code> to <code>unicode</code>, but not the other way around, see <a href="https://mail.python.org/pipermail/python-list/2012-July/627506.html" rel="nofollow">https://mail.python.org/pipermail/python-list/2012-July/627506.html</a>).</p>

<p>I don't plan on using it but it might be interesting for those battling problems with invalid Unicode input and looking for a quick fix (but please think about the side effects):</p>

<pre class="lang-python prettyprint-override"><code>import codecs
codecs.register_error("strict", codecs.ignore_errors)
codecs.register_error("strict", lambda x: (u"", x.end)) # alternatively
</code></pre>

<p>(An internet search for <code>codecs.register_error("strict"</code> revealed that apparently it's used in some real projects.)</p>

<p><em>/edit #2:</em></p>

<p>For explicit conversions I made a snippet with the help of <a href="http://stackoverflow.com/a/4025310">a SO post on monkeypatching</a>:</p>

<pre class="lang-python prettyprint-override"><code>class PatchedUnicode(unicode):
  def __init__(self, obj=None, encoding=None, *args, **kwargs):
    if encoding in (None, "ascii", "646", "us-ascii"):
        print("Problematic unicode() usage detected!")
    super(PatchedUnicode, self).__init__(obj, encoding, *args, **kwargs)

import __builtin__
__builtin__.unicode = PatchedUnicode
</code></pre>

<p>This only affects explicit conversions using the <code>unicode()</code> constructor directly so it's not something I need.</p>

<p><em>/edit #3:</em></p>

<p>The thread "<a href="https://stackoverflow.com/questions/6738987/extension-method-for-python-built-in-types">Extension method for python built-in types!</a>" makes me think that it might actually not be easily possible (in CPython at least).</p>

<p><em>/edit #4:</em></p>

<p>It's nice to see many good answers here, too bad I can only give out the bounty once.</p>

<p>In the meantime I came across a somewhat similar question, at least in the sense of what the person tried to achieve: <a href="http://stackoverflow.com/q/2851481">Can I turn off implicit Python unicode conversions to find my mixed-strings bugs?</a>
Please note though that throwing an exception would <strong>not</strong> have been OK in my case. Here I was looking for something which might point me to the different locations of problematic code (e.g. by printing smth.) but not something which might exit the program or change its behavior (because this way I can prioritize what to fix).</p>

<p>On another note, the people working on the Mypy project (which include Guido van Rossum) might also come up with something similar helpful in the future, see the discussions at <a href="https://github.com/python/mypy/issues/1141" rel="nofollow">https://github.com/python/mypy/issues/1141</a> and more recently <a href="https://github.com/python/typing/issues/208" rel="nofollow">https://github.com/python/typing/issues/208</a>.</p>

<p><em>/edit #5</em></p>

<p>I also came across the following but didn't have yet the time to test it: <a href="https://pypi.python.org/pypi/unicode-nazi" rel="nofollow">https://pypi.python.org/pypi/unicode-nazi</a></p>

<p>What you can do is the following:</p>

<p>First create a custom encoding. I will call it "lascii" for "logging ASCII":</p>

<pre><code>import codecs
import traceback

def lascii_encode(input,errors='strict'):
    print("ENCODED:")
    traceback.print_stack()
    return codecs.ascii_encode(input)


def lascii_decode(input,errors='strict'):
    print("DECODED:")
    traceback.print_stack()
    return codecs.ascii_decode(input)

class Codec(codecs.Codec):
    def encode(self, input,errors='strict'):
        return lascii_encode(input,errors)
    def decode(self, input,errors='strict'):
        return lascii_decode(input,errors)

class IncrementalEncoder(codecs.IncrementalEncoder):
    def encode(self, input, final=False):
        print("Incremental ENCODED:")
        traceback.print_stack()
        return codecs.ascii_encode(input)

class IncrementalDecoder(codecs.IncrementalDecoder):
    def decode(self, input, final=False):
        print("Incremental DECODED:")
        traceback.print_stack()
        return codecs.ascii_decode(input)

class StreamWriter(Codec,codecs.StreamWriter):
    pass

class StreamReader(Codec,codecs.StreamReader):
    pass

def getregentry():
    return codecs.CodecInfo(
        name='lascii',
        encode=lascii_encode,
        decode=lascii_decode,
        incrementalencoder=IncrementalEncoder,
        incrementaldecoder=IncrementalDecoder,
        streamwriter=StreamWriter,
        streamreader=StreamReader,
    )
</code></pre>

<p>What this does is basically the same as the ASCII-codec, just that it prints a message and the current stack trace every time it encodes or decodes from unicode to lascii.</p>

<p>Now you need to make it available to the codecs module so that it can be found by the name "lascii". For this you need to create a search function that returns the lascii-codec when it's fed with the string "lascii". This is then registered to the codecs module:</p>

<pre><code>def searchFunc(name):
    if name=="lascii":
        return getregentry()
    else:
        return None

codecs.register(searchFunc)
</code></pre>

<p>The last thing that is now left to do is to tell the sys module to use 'lascii' as default encoding:</p>

<pre><code>import sys
reload(sys) # necessary, because sys.setdefaultencoding is deleted on start of Python
sys.setdefaultencoding('lascii')
</code></pre>

<p><strong>Warning:</strong>
This uses some deprecated or otherwise unrecommended features. It might not be efficient or bug-free. Do not use in production, only for testing and/or debugging.</p>

python python-2.7 debugging unicode monkeypatching