Why is Python logging framework losing messages?
<p>I have a Python 3.4 application that uses logging extensively. I have two FileHandlers and a StreamHandler registered. Everything works as expected except that sometimes, and it seems to happen after the <code>requests</code> library throws an exception, the log files lose all the accumulated messages and start with new messages. I'm assuming that for some reason the FileHandlers reopened the files with <code>mode='w'</code>, but I don't understand why. Any ideas?</p>

<p>The main program sets up the loggers as follows:</p>

<pre><code># Set up root logger - two handlers logging to files
fh_debug = logging.FileHandler('Syncer.debug', mode='w', encoding='utf-8')
fh_debug.setLevel(logging.DEBUG)
fh_log = logging.FileHandler('Syncer.log', mode='w', encoding='utf-8')
fh_log.setLevel(logging.INFO)
fh_formatter = logging.Formatter(
    '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
fh_debug.setFormatter(fh_formatter)
fh_log.setFormatter(fh_formatter)
logging.getLogger().addHandler(fh_debug)
logging.getLogger().addHandler(fh_log)

# Add console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch_formatter = logging.Formatter('%(message)s')
ch.setFormatter(ch_formatter)
logging.getLogger().addHandler(ch)

# Need to set the logging level of the logger as well as the handlers
logging.getLogger().setLevel(logging.DEBUG)

# Set up the logger for this module
logger = logging.getLogger("Syncer")

logger.debug('Logger started.')
</code></pre>

<p>The modules simply contain</p>

<pre><code>logger = logging.getLogger(__name__)
</code></pre>

<p>Your issue is that you chose the wrong <code>mode</code> for your <code>FileHandler</code>.</p>

<p>The default mode of <code>FileHandler</code> is <code>a</code> which means appends new lines to the log file.</p>

<blockquote>
  <p>class logging.FileHandler(filename, mode='a', encoding=None, delay=False)</p>
</blockquote>

<p>You modified the default mode to <code>w</code> which truncate file to zero length or create a new file for writing. That's why you lost all accumulated messages.</p>

<p><strong>Change it to <code>mode='a'</code> or just delete <code>mode='w'</code> and then your logger will work.</strong></p>

<p>Read the <a href="https://webdocs.cs.ualberta.ca/~jdg001/local-archive/python-3.4-reference/library/logging.handlers.html" rel="nofollow">official python docs here</a></p>

python python-3.x logging python-requests error-logging