Django AdminEmailHandler hangs when DB is not accesible
<p>I noticed that when my database goes down, the queries to my Django app gives timeout instead of returning 500 to the client instantaneously. </p>

<p>Tracing the problem I set database connect_timeout in config to 5s (<a href="http://stackoverflow.com/questions/1084488/how-to-set-timeout-for-database-connection-in-django">explained here</a>) and the exception appears faster on the logs but after printing the exception the client does not receive the result until 30s more.</p>

<pre><code>OperationalError: (2003, "Can't connect to MySQL server on 'db.example.com' (4)")
</code></pre>

<p>Debugging what is happening with <a href="http://stackoverflow.com/questions/1118183/how-to-debug-in-django-the-good-way">PDB</a> I found that the problem is in the <strong>django.utils.log AdminEmailHandler</strong>, when it tries to generate the mail of the exception. </p>

<p>To generate the mail it calls <strong>django.views.debug ExceptionReporter.get_traceback_text()</strong> that looks for all frames on the traceback and then for all variables on each frame, and one of these variables is the <strong>queryset</strong> that triggered the exception.</p>

<p>Generating the error email makes multiple accesses to the <strong>queryset</strong> that generated the DB connection timeout, generating more DB timeouts and thus the error response to the client takes so long.</p>

<p>What is the best approach to avoid this problem?</p>

<p>While no better alternative is given, this is the solution I have implemented.</p>

<p>I have setup a custom AdminEmailHandler that inspect's the exception and in case of database OperationalError skips the exception and sends and error instead.</p>

<pre><code>import logging

from django.utils.log import AdminEmailHandler
from django.db.utils import OperationalError

logger = logging.getLogger('mylogger')

class CustomAdminEmailHandler(AdminEmailHandler):

        # When mail is because of exception conencting to DB, avoid rendering the email,
        # rendering email makes connections to DB making the query hang in multiple DB 
        # connection timeouts.
        if record.exc_info:
            exc_type, exc_value, exc_traceback = record.exc_info
            if exc_type == OperationalError:
                logger.error(exc_value, exc_info=False)
                return

        super(CustomAdminEmailHandler, self).emit(record)
</code></pre>

<p>And in settings.py</p>

<pre><code>LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '%(asctime)s %(levelname)s %(module)s:%(funcName)s() %(message)s'
        },
    },
    'handlers': {
        'send_mail': {
            'level': 'ERROR',
            'class': 'myapp.handlers.CustomdAdminEmailHandler',
            'include_html': False,
        },
    },
    'loggers': {
        'django': {
            'handlers': ['send_mail'],
            'level': 'INFO'
        },
        'mylogger': {
            'handlers': ['send_mail'],
            'level': 'INFO'
        },
    }
}
</code></pre>

<p>I have also set database connection timeout to 1 second, I don't want the requests to be queued in case that the database is not accessible, this may lead to a very high load when the connection returns that can get the servers down.</p>

python mysql django