Python Logging for a module shared by different scripts
<p>I like the python logging infrastructure and I want to use it for a number of different overnight jobs that I run.  A lot of these jobs use module X let's say.  I want the logging for module X to write to a log file not dependent on module X, but based on the job that ultimately led to calling functionality in module X.  </p>

<p>So if overnight_script_1.py calls foo() in module X, I want the log of foo() to go to overnight_script_1.log.  I also want overnight_script_2.py's call of foo() to log to overnight_script_2.log.</p>

<p>A potential solution to this problem would be to set up the log file based on looking at the 0th argument to sys.argv which can be mapped to my preferred file.  Something seems hacky about doing it this way.  Is there a preferred design pattern for doing this? I don't want to rummage through different log files based on the module where the function was called to find my diagnostic information for one of my scripts.  Here is some code because I am not sure I am making myself clear.</p>

<p>Here is script1.py</p>

<pre><code>import X
import logging_utils as lu
import sys

logname=sys.argv[0][:-3] # logname==script1 with the .py cut off
logger=lu.setup_logger(log_name) # assume this does the formatting and sets the filehandlers
# furthermore assume the file handler is set so that the output goes to script1.log.

# do a bunch of thing
logger.info('I am doing a bunch of things in script1 and I will now call X.foo()')
X.foo() # see module X below
logger.info('I finished X.foo()')
</code></pre>

<p>Similary, here is script2.py</p>

<pre><code>import X
import logging_utils as lu
import sys

logname=sys.argv[0][:-3] # logname==script2 with the .py cut off
logger=lu.setup_logger(log_name) # assume this does the formatting and sets the filehandlers
# furthermore assume the file handler is set so that the output goes to script2.log.

# do a bunch of thing
logger.info('I am doing a bunch of things in script2 and I will now call X.foo()')
X.foo() # see module X below
logger.info('I finished X.foo()')
</code></pre>

<p>Here is X.py</p>

<pre><code>import logging
import sys
logname=sys.argv[0][:-3] # could be script1 or script2
logger=logging.getLogger(logname)

def foo():
    try:
        i=1/0
    except:
        logger.error('oops - division by zero')
</code></pre>

<p>Then I want to run:</p>

<p>python script1.py</p>

<p>python script2.py</p>

<p>and get two log files script1.log and script2.log where the division by zero error that occurred in module X is logged in each.</p>

<p>I believe you should just follow the standard set up for library code:</p>

<p>Say you have a package <code>mypkg</code> and you want this package to log information while letting the user of the package decide which level of logging to use and where the output should go.</p>

<p>The <code>mypkg</code> should <strong>not</strong> set the level nor any handler for the logs.
Module <code>x</code> of <code>mypkg</code> should be something like this:</p>

<pre><code>import logging

logger = logging.getLogger(__name__)


def my_function():
    logger.info('something')
</code></pre>

<p>No configuration of the logging whatsoever just <code>getLogger(__name__)</code>.</p>

<p>Then your <code>script.py</code> that uses <code>mypkg</code> and wants to log at level <code>INFO</code> on the console will do:</p>

<pre><code>import logging
import mypkg

root_logger = logging.getLogger()
console_handler = logging.StreamHandler()

root_logger.setLevel(logging.INFO)
root_logger.addHandler(console_handler)

mypkg.my_function()  # produces output to stderr
</code></pre>

<p>While <code>script2.py</code> will log at level <code>DEBUG</code> to a file:</p>

<pre><code>import logging
import mypkg


root_logger = logging.getLogger()
file_handler = logging.FileHandler('some_file.log')

root_logger.setLevel(logging.DEBUG)
root_logger.addHandler(file_handler)

mypkg.my_function()
</code></pre>

<p>Note that by setting the level and handler on the root logger we are setting the level "globally". If the user has some own loggers that he wants to use with level <code>DEBUG</code> but he wants to use level <code>INFO</code> for <code>mypkg</code> he can do:</p>

<pre><code>root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)

mypkg_logger = logging.getLogger('mypkg')
mypkg_logger.setLevel(logging.INFO)

handler = #... whatever you want
root_logger.addHandler(handler)
</code></pre>

<p>If your <code>mypkg</code> contains the modules <code>x.py</code>,  <code>y.py</code>, <code>a.py</code>, <code>b.py</code> and <code>c.py</code> and you want to log the functions inside <code>x</code> at <code>DEBUG</code> and those inside <code>y</code> at <code>WARNING</code> and those for <code>a/b/c.py</code> at <code>INFO</code> you can do so by setting the levels on the corresponding loggers:</p>

<pre><code>mypkg_logger = logging.getLogger('mypkg')
my_pkg_logger.setLevel(logging.INFO)

x_logger = logging.getLogger('mypkg.x')
x_logger.setLevel(logging.DEBUG)
y_logger = logging.getLogger('mypkg.y')
y_logger.setLevel(logging.WARNING)
</code></pre>

<p>Put the handlers only attached to the <code>root_logger</code> and everything should work fine.</p>

<hr>

<p>Note: on some versions of python the <code>logging</code> module might warn you that you defined a logger with no handlers for the library modules. To fix ths you can use a <code>NullHandler</code> that just drops the logging messages:</p>

<pre><code># x.py

logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())
</code></pre>

<p>So the <code>NullHandler</code> literally does nothing, except from preventing the <code>logging</code> module from complaining. AFAIK newer versions do not need this at all.</p>

python logging