Trying to get a grip on CachedPropery in clang\cindex.py
<p>This is related to other <a href="http://stackoverflow.com/questions/39194326/libclang-with-python-binding-asserterror">question</a> I had, which left with no answer...
I trying to understand what's going on under the hood of the <a href="https://github.com/llvm-mirror/clang/tree/master/bindings/python" rel="nofollow">Python binding</a> to <a href="http://clang.llvm.org/doxygen/group__CINDEX.html" rel="nofollow">libclang</a>, and having really hard-time doing so.</p>

<p>I've read TONs of articles about both <code>decorators</code> and <code>descriptors</code> in Python, in order to understand how the <a href="https://github.com/llvm-mirror/clang/blob/master/bindings/python/clang/cindex.py#L122" rel="nofollow">CachedProperty class in clang/cindex.py</a> works, but still can't get all the pieces together.</p>

<p>The most related texts I've seen is <a href="http://stackoverflow.com/questions/6598534/explanation-of-a-decorator-class-in-python">one SO answer</a>,  and this <a href="http://code.activestate.com/recipes/577452-a-memoize-decorator-for-instance-methods/" rel="nofollow">code recipe</a> in ActiveState. This helps me a bit, but - as I mentioned - I'm still not there.</p>

<p>So, let's cut to the chase:
I want to understand why am I getting <code>AssertionError</code> on creating CIndex. I will post here only the relevant code (cindex.py is 3646 lines long..), and I hope I don't miss anything that is relevant to me.
My code has only one relevant line, which is:</p>

<pre><code>index = clang.cindex.Index.create()
</code></pre>

<p>This reffers to <a href="https://github.com/llvm-mirror/clang/blob/master/bindings/python/clang/cindex.py#L2291" rel="nofollow">line 2291 in cindex.py</a>, which yields:</p>

<pre><code>return Index(conf.lib.clang_createIndex(excludeDecls, 0))
</code></pre>

<p>From now on, there's a series of function calls, which I can't explain why and WTH did they come from. I'll list the code and <code>pdb</code> output along the questions that relevant to each part:</p>

<p>(Important thing to notice ahead: conf.lib defined like this:)</p>

<pre><code>class Config:
    ...snip..

    @CachedProperty
    def lib(self):
        lib = self.get_cindex_library()
        ...
        return lib
</code></pre>

<p><strong>CachedProperty code:</strong></p>

<pre><code>class CachedProperty(object):
    """Decorator that lazy-loads the value of a property.

    The first time the property is accessed, the original property function is
    executed. The value it returns is set as the new value of that instance's
    property, replacing the original method.
    """

    def __init__(self, wrapped):
        self.wrapped = wrapped
        try:
            self.__doc__ = wrapped.__doc__
        except:
            pass

    def __get__(self, instance, instance_type=None):
        if instance is None:
            return self

        value = self.wrapped(instance)
        setattr(instance, self.wrapped.__name__, value)

        return value
</code></pre>

<p><strong><code>Pdb</code> output:</strong></p>

<pre><code>-&gt; return Index(conf.lib.clang_createIndex(excludeDecls, 0))
(Pdb) s
--Call--
&gt; d:\project\clang\cindex.py(137)__get__()
-&gt; def __get__(self, instance, instance_type=None):
(Pdb) p self
&lt;clang.cindex.CachedProperty object at 0x00000000027982E8&gt;
(Pdb) p self.wrapped
&lt;function Config.lib at 0x0000000002793598&gt;
</code></pre>

<ol>
<li>Why the next call after
<code>Index(conf.lib.clang_createIndex(excludeDecls, 0))</code> is to
<code>CachedProperty.__get__</code> method? What about the <code>__init__</code>?</li>
<li>If the <code>__init__</code> method isn't get called, how comes that self.wrapped has
value?</li>
</ol>

<p><strong><code>Pdb</code> output:</strong></p>

<pre><code>(Pdb) r
--Return--
&gt; d:\project\clang\cindex.py(144)__get__()-&gt;&lt;CDLL 'libcla... at 0x27a1cc0&gt;
-&gt; return value
(Pdb) n
--Call--
&gt; c:\program files\python35\lib\ctypes\__init__.py(357)__getattr__()
-&gt; def __getattr__(self, name):
(Pdb) r
--Return--
&gt; c:\program files\python35\lib\ctypes\__init__.py(362)__getattr__()-&gt;&lt;_FuncPtr obj...000000296B458&gt;
-&gt; return func
(Pdb)
</code></pre>

<ol start="3">
<li>Where <code>CachedProperty.__get__</code> should return value to? Where the call for <code>CDLL.__getattr__</code> method come from?</li>
</ol>

<p><strong>MOST CRITICAL PART, for me</strong></p>

<pre><code>(Pdb) n
--Call--
&gt; d:\project\clang\cindex.py(1970)__init__()
-&gt; def __init__(self, obj):
(Pdb) p obj
40998256
</code></pre>

<p>This is the <a href="https://github.com/llvm-mirror/clang/blob/master/bindings/python/clang/cindex.py#L2044" rel="nofollow">creation of <code>ClangObject</code></a>, which class Index inherits from.</p>

<ol start="4">
<li>But - where there's any call to <code>__init__</code> with one parameter? Is this is the one that <code>conf.lib.clang_createIndex(excludeDecls, 0)</code> returning?</li>
<li>Where is this number (40998256) coming from? I'm getting the same number over and over again. As far as I understand, it should be just a number, but a <code>clang.cindex.LP_c_void_p object</code> and that's why the assertion failed.</li>
</ol>

<p>To sum it up, the best for me will be step-by-step guidance of the functions invocation over here, cause I'm felling a little lost in all this...</p>

<p><strong>SOLUTION to the last 2 questions:</strong>
The problem lays in the difference between Python 2 and 3 in the <code>map()</code> function. While Python 2 actually <strong>does</strong> the mapping, Python 3 only return iterator, which you can use later on. This results the <code>register_function</code> method on <code>config.lib</code> to run without actually register any function - hence the wrong translation of the returning value.</p>

<p><strong>Fix:</strong> Change <a href="https://github.com/llvm-mirror/clang/blob/master/bindings/python/clang/cindex.py#L3657" rel="nofollow">map(register, functionList)</a> to list(map(register, functionList))</p>

<p>Still, thanks <a href="http://stackoverflow.com/users/100297/martijn-pieters">@Martijn</a>, because of you I was able to move on from this <code>CachedProperty</code>...:)</p>

<p>The <code>CachedProperty</code> object is a <a href="https://docs.python.org/3/howto/descriptor.html" rel="nofollow">descriptor object</a>; the <code>__get__</code> method is called automatically whenever Python tries to access an attribute on an instance that is only available on the class <em>and</em> has a <code>__get__</code> method.</p>

<p>Using <code>CachedProperty</code> as a decorator means it is called and an instance of <code>CachedProperty</code> is created that replaces the original function object on the <code>Config</code> class. It is the <code>@CachedProperty</code> line that causes <code>CachedProperty.__init__</code> to be called, and the instance ends up on the <code>Config</code> class as <code>Config.lib</code>. Remember, the syntax</p>

<pre><code>@CachedProperty
def lib(self):
    # ...
</code></pre>

<p>is essentially executed as</p>

<pre><code>def lib(self):
    # ...
lib = CachedProperty(lib)
</code></pre>

<p>so this creates an instance of <code>CachedProperty()</code> with <code>lib</code> passed in as the <code>wrapped</code> argument, and then <code>Config.lib</code> is set to that object.</p>

<p>You can see this in the debugger; one step up you could inspect <code>type(config).lib</code>:</p>

<pre><code>(Pdb) type(config)
&lt;class Config at 0x00000000027936E&gt;
(Pdb) type(config).lib
&lt;clang.cindex.CachedProperty object at 0x00000000027982E8&gt;
</code></pre>

<p>In the rest of the codebase <code>config</code> is an instance of the <code>Config</code> class. At first, that instance has no <code>lib</code> name in the <code>__dict__</code> object, so the instance has no such attribute:</p>

<pre><code>(Pdb) 'lib' in config.__dict__
False
</code></pre>

<p>So trying to get <code>config.lib</code> has to fall back to the class, where Python finds the <code>Config.lib</code> attribute, and this is a descriptor object. Instead of using <code>Config.lib</code> directly, Python returns the result of calling <code>Config.lib.__get__(config, Config)</code> in that case.</p>

<p>The <code>__get__</code> method then executes the original function (referenced by <code>wrapped</code>) and stores that in <code>config.__dict__</code>. So <em>future</em> access to <code>config.lib</code> will find that result, and the descriptor on the class is not going to be used after that.</p>

<p>The <code>__getattr__</code> method is called to satisfy the <em>next</em> attribute in the <code>conf.lib.clang_createIndex(excludeDecls, 0)</code> expression; <code>config.lib</code> returns a dynamically loaded library from <code>cdll.LoadLibrary()</code> (via <code>CachedProperty.__get__()</code>), and that <a href="https://docs.python.org/3/library/ctypes.html#ctypes.CDLL" rel="nofollow">specific object type</a> is handled by the Python <a href="https://docs.python.org/3/library/ctypes.html" rel="nofollow">ctypes libary</a>. It translates attributes to specific C calls for you; here that's the <code>clang_createIndex</code> method; see <a href="https://docs.python.org/3/library/ctypes.html#accessing-functions-from-loaded-dlls" rel="nofollow"><em>Accessing functions from loaded dlls</em></a>.</p>

<p>Once the call to <code>conf.lib.clang_createIndex(excludeDecls, 0)</code> completes, that resulting object is indeed passed to <code>Index()</code>; the <a href="https://github.com/llvm-mirror/clang/blob/master/bindings/python/clang/cindex.py#L2283" rel="nofollow"><code>Index()</code> class</a> itself has no <code>__init__</code> method, but the base class <a href="https://github.com/llvm-mirror/clang/blob/master/bindings/python/clang/cindex.py#L2044" rel="nofollow"><code>ClangObject</code></a> does.</p>

<p>Whatever that return value is, it has a <em>representation</em> that looks like an integer number. However, it almost certainly is not an <code>int</code>.  You can see what type of object that is by using <code>type()</code>, see what attributes it has with <code>dir()</code>, etc. I'm pretty certain it is a <a href="https://docs.python.org/3/library/ctypes.html#ctypes.c_void_p" rel="nofollow"><code>ctypes.c_void_p</code> data type</a> <em>representing</em> a <code>clang.cindex.LP_c_void_p</code> value (it is a Python object that proxies for the real C value in memory); it'll represent as an integer:</p>

<blockquote>
  <p>Represents the C <code>void *</code> type. The value is represented as integer. The constructor accepts an optional integer initializer.</p>
</blockquote>

<p>The rest of the <code>clang</code> Python code will just pass this value back to more C calls proxied by <code>config.lib</code>.</p>

python clang python-decorators libclang python-descriptors