why python's pickle is not serializing a method as default argument?
<p>I am trying to use pickle to transfer python objects over the wire between 2 servers. I created a simple class, that subclasses <code>dict</code> and I am trying to use pickle for the marshalling:</p>

<pre><code>def value_is_not_none(value):
    return value is not None

class CustomDict(dict):
    def __init__(self, cond=lambda x: x is not None):
        super().__init__()
        self.cond = cond

    def __setitem__(self, key, value):
        if self.cond(value):
            dict.__setitem__(self, key, value)
</code></pre>

<p>I first tried to use <code>pickle</code> for the marshalling, but when I un-marshalled I received an error related to the <code>lambda</code> expression.</p>

<p>Then I tried to do the marshalling with <code>dill</code> but it seemed the <code>__init__</code> was not called.</p>

<p>Then I tried again with <code>pickle</code>, but I passed the <code>value_is_not_none()</code> function as the <code>cond</code> parameter - again the <code>__init__()</code> does not seemed to be invoked and the un-marshalling failed on the <code>__setitem__()</code> (<code>cond</code> is <code>None</code>).</p>

<p>Why is that? what am I missing here?</p>

<p>If I try to run the following code:</p>

<pre><code>obj = CustomDict(cond=value_is_not_none)
obj['hello'] = ['world']

payload = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
obj2 = pickle.loads(payload)
</code></pre>

<p>it fails with </p>

<pre><code>AttributeError: 'CustomDict' object has no attribute 'cond'
</code></pre>

<p>This is a different question than: <a href="http://stackoverflow.com/questions/16626429/python-cpickle-pickling-lambda-functions">Python, cPickle, pickling lambda functions</a>
as I tried using <code>dill</code> with <code>lambda</code> and it failed to work, and I also tried passing a function and it also failed.</p>

<p><code>pickle</code> is loading your dictionary data <em>before</em> it has restored the attributes on your instance. As such the <code>self.cond</code> attribute is not yet set when <code>__setitem__</code> is called for the dictionary key-value pairs.</p>

<p>Note that <code>pickle</code> will never call <code>__init__</code>; instead it'll create an entirely <em>blank</em> instance and restore the <code>__dict__</code> attribute namespace on that directly.</p>

<p>You have two options:</p>

<ul>
<li><p>default to <code>cond=None</code> and ignore the condition if it is still set to <code>None</code>:</p>

<pre><code>class CustomDict(dict):
    def __init__(self, cond=None):
        super().__init__()
        self.cond = cond

    def __setitem__(self, key, value):
        if getattr(self, 'cond', None) is None or self.cond(value):
            dict.__setitem__(self, key, value)
</code></pre>

<p>The <code>getattr()</code> there is needed because a blank instance has no <code>cond</code> attribute at all (it is not set to <code>None</code>, the attribute is entirely missing). You could add <code>cond = None</code> to the class:</p>

<pre><code>class CustomDict(dict):
    cond = None
</code></pre>

<p>and then just test for <code>if self.cond is None or self.cond(value):</code>.</p></li>
<li><p>Define a custom <a href="https://docs.python.org/3/library/pickle.html#object.__reduce__" rel="nofollow"><code>__reduce__</code> method</a> to control how the initial object is created when restored:</p>

<pre><code>def _default_cond(v): return v is not None

class CustomDict(dict):
    def __init__(self, cond=_default_cond):
        super().__init__()
        self.cond = cond

    def __setitem__(self, key, value):
        if self.cond(value):
            dict.__setitem__(self, key, value)

    def __reduce__(self):
        return (CustomDict, (self.cond,), None, None, iter(self.items()))
</code></pre>

<p><code>__reduce__</code> is expected to return a tuple with:</p>

<ul>
<li>A callable that can be pickled directly (here the class does fine)</li>
<li>A tuple of positional arguments for that callable; on unpickling the first element is called passing in the second as arguments, so by setting this to <code>(self.cond,)</code> we ensure that the new instance is created with <code>cond</code> passed in as an argument and <em>now</em> <code>CustomDict.__init__()</code> <em>will</em> be called.</li>
<li>The next 2 positions are for a <code>__setstate__</code> method (ignored here) and for list-like types, so we set these to <code>None</code>.</li>
<li>The last element is an iterator for the key-value pairs that pickle then will restore for us.</li>
</ul>

<p></p>

<p>Note that I replaced the default value for <code>cond</code> with a function here too so you don't have to rely on <code>dill</code> for the pickling.</p></li>
</ul>

python dictionary lambda pickle