sentry-youtrack plugin: PicklingError: Can't pickle <type 'generator'>: attribute lookup __builtin__.generator failed
<ul>
<li>sentry: 8.4.0</li>
<li>sentry-youtrack plugin: 0.3.1</li>
<li>youtrack-6.5.17105</li>
<li>python-memcached: 1.53</li>
</ul>

<p>I'm trying to integrate <a href="https://www.jetbrains.com/youtrack/" rel="nofollow">youtrack</a> into <a href="https://github.com/getsentry/sentry" rel="nofollow">sentry</a> using <a href="https://github.com/bogdal/sentry-youtrack" rel="nofollow">this</a> plugin.</p>

<p>The problem is the page seems to hang when we click More --> Create YouTrack Issue. Looking at the syslog, I saw this:</p>

<pre><code>Traceback (most recent call last):
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/utils/decorators.py", line 29, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/utils/decorators.py", line 99, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/utils/decorators.py", line 25, in bound_func
    return func(self, *args2, **kwargs2)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/web/frontend/base.py", line 172, in dispatch
    return self.handle(request, *args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/web/frontend/group_plugin_action.py", line 25, in handle
    response = plugin.get_view_response(request, group)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 113, in get_view_response
    return super(YouTrackPlugin, self).get_view_response(request, group)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/plugins/base/v1.py", line 296, in get_view_response
    response = self.view(request, group)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 131, in view
    return view(request, group, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/plugins/bases/issue.py", line 169, in view
    form = self.get_new_issue_form(request, group, event)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 77, in get_new_issue_form
    project_fields=self.get_project_fields(group.project),
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 57, in get_project_fields
    return cached_fields(self.get_option('ignore_fields', project))
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/utils.py", line 16, in wrapper
    cache.set(key, result, timeout)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/cache/backends/memcached.py", line 82, in set
    self._cache.set(key, value, self._get_memcache_timeout(timeout))
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 740, in set
    return self._set("set", key, val, time, min_compress_len, noreply)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1060, in _set
    return _unsafe_set()
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set
    store_info = self._val_to_store_info(val, min_compress_len)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
    pickler.dump(val)
PicklingError: Can't pickle &lt;type 'generator'&gt;: attribute lookup __builtin__.generator failed
</code></pre>

<p>Follow the traceback (<a href="https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/utils.py#L6" rel="nofollow">https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/utils.py#L6</a>):</p>

<pre><code>def cache_this(timeout=60):
    def decorator(func):
        def wrapper(*args, **kwargs):
            def get_cache_key(*args, **kwargs):
                params = list(args) + kwargs.values()
                return md5("".join(map(str, params))).hexdigest()
            key = get_cache_key(func.__name__, *args, **kwargs)
            result = cache.get(key)
            if not result:
                result = func(*args, **kwargs)
                cache.set(key, result, timeout)
            return result
        return wrapper
    return decorator
</code></pre>

<p>I understand that we got this error because <code>result</code> is a generator.</p>

<p><a href="https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/plugin.py#L51" rel="nofollow">https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/plugin.py#L51</a>:</p>

<pre><code>def get_project_fields(self, project):
    @cache_this(600)
    def cached_fields(ignore_fields):
        yt_client = self.get_youtrack_client(project)
        return yt_client.get_project_fields(
            self.get_option('project', project), ignore_fields)
    return cached_fields(self.get_option('ignore_fields', project))
</code></pre>

<p><a href="https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/youtrack.py#L198" rel="nofollow">https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/youtrack.py#L198</a>:</p>

<pre><code>def get_project_fields(self, project_id, ignore_fields=None):
    ignore_fields = ignore_fields or []
    for field in self.get_project_fields_list(project_id):
        if not field['name'] in ignore_fields:
            yield self._get_custom_project_field_details(field)
</code></pre>

<p>So, I'm trying to convert it into a list:</p>

<pre><code>        if not result:
            result = func(*args, **kwargs)
            cache.set(key, [f for f in result], timeout)
        return result
</code></pre>

<p>but still got the same error:</p>

<pre><code>      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/utils.py", line 16, in wrapper
        cache.set(key, [f for f in result], timeout)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/cache/backends/memcached.py", line 82, in set
        self._cache.set(key, value, self._get_memcache_timeout(timeout))
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 740, in set
        return self._set("set", key, val, time, min_compress_len, noreply)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1060, in _set
        return _unsafe_set()
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set
        store_info = self._val_to_store_info(val, min_compress_len)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
        pickler.dump(val)
    PicklingError: Can't pickle &lt;type 'generator'&gt;: attribute lookup __builtin__.generator failed
</code></pre>

<p>Then I tried to log the value of <code>val</code> into a file:</p>

<pre><code>        try:
            pickler.dump(val)
        except Exception:
            with open('/tmp/quanta.log', 'a+') as f:
                f.write(str(val))
</code></pre>

<p>but that file is not created. And the strange thing is the traceback said that the error happened at the same line as before:</p>

<pre><code>      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/utils.py", line 16, in wrapper
        cache.set(key, [f for f in result], timeout)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/cache/backends/memcached.py", line 82, in set
        self._cache.set(key, value, self._get_memcache_timeout(timeout))
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 740, in set
        return self._set("set", key, val, time, min_compress_len, noreply)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1060, in _set
        msg = msg[1]
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set

      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
        try:
    PicklingError: Can't pickle &lt;type 'generator'&gt;: attribute lookup __builtin__.generator failed
</code></pre>

<p>So, I have two questions:</p>

<ol>
<li>Why the error still said that <code>Can't pickle &lt;type 'generator'&gt;...</code> while it had been converted to a list?</li>
<li>How can I debug this situation to know the value of <code>val</code> before calling <code>pickler.dump(val)</code>?</li>
</ol>

<p>This is a bug for sentry-youtrack, it should not cache a generator object. python-memcached might have a pyc file, that's why it did not dump the value like you modified. And you added (i for i in list) which also is a generator.</p>

<p>You should use getsentry/sentry-youtrack since it has the <a href="https://github.com/getsentry/sentry-youtrack/commit/7989f3db695e2a7b3bf0eff18b6e4140d27e8d0f" rel="nofollow">correct fix</a> for caching generator. </p>

python django generator sentry youtrack