Python Logging for a module shared by different scripts
<p>I like the python logging infrastructure and I want to use it for a number of different overnight jobs that I run.  A lot of these jobs use module X let's say.  I want the logging for module X to write to a log file not dependent on module X, but based on the job that ultimately led to calling functionality in module X.  </p>

<p>So if overnight_script_1.py calls foo() in module X, I want the log of foo() to go to overnight_script_1.log.  I also want overnight_script_2.py's call of foo() to log to overnight_script_2.log.</p>

<p>A potential solution to this problem would be to set up the log file based on looking at the 0th argument to sys.argv which can be mapped to my preferred file.  Something seems hacky about doing it this way.  Is there a preferred design pattern for doing this? I don't want to rummage through different log files based on the module where the function was called to find my diagnostic information for one of my scripts.  Here is some code because I am not sure I am making myself clear.</p>

<p>Here is script1.py</p>

<pre><code>import X
import logging_utils as lu
import sys

logname=sys.argv[0][:-3] # logname==script1 with the .py cut off
logger=lu.setup_logger(log_name) # assume this does the formatting and sets the filehandlers
# furthermore assume the file handler is set so that the output goes to script1.log.

# do a bunch of thing
logger.info('I am doing a bunch of things in script1 and I will now call X.foo()')
X.foo() # see module X below
logger.info('I finished X.foo()')
</code></pre>

<p>Similary, here is script2.py</p>

<pre><code>import X
import logging_utils as lu
import sys

logname=sys.argv[0][:-3] # logname==script2 with the .py cut off
logger=lu.setup_logger(log_name) # assume this does the formatting and sets the filehandlers
# furthermore assume the file handler is set so that the output goes to script2.log.

# do a bunch of thing
logger.info('I am doing a bunch of things in script2 and I will now call X.foo()')
X.foo() # see module X below
logger.info('I finished X.foo()')
</code></pre>

<p>Here is X.py</p>

<pre><code>import logging
import sys
logname=sys.argv[0][:-3] # could be script1 or script2
logger=logging.getLogger(logname)

def foo():
    try:
        i=1/0
    except:
        logger.error('oops - division by zero')
</code></pre>

<p>Then I want to run:</p>

<p>python script1.py</p>

<p>python script2.py</p>

<p>and get two log files script1.log and script2.log where the division by zero error that occurred in module X is logged in each.</p>

<p>You could probably use as many RotatingFileHandler(s) as different modules you have and setup their filenames accordingly </p>

<p><a href="https://docs.python.org/2/library/logging.handlers.html" rel="nofollow">https://docs.python.org/2/library/logging.handlers.html</a></p>

<p>-Edit- </p>

<p>Based on the comment then you may want to use a <code>Formatter</code> that takes into account where was the call made (to know if the call to the shared library was made from <code>module_1</code> or <code>module_2</code> and so on) and then and then simply log to the same file.</p>

<p>There are some caveats to think about: </p>

<ul>
<li>I'm not sure if there is order preservation in the writing of the file because there may be two or more processes that write to that file</li>
<li>The log file may increase in size arbitrarily ending up with 0 disk space</li>
</ul>

<p><a href="https://docs.python.org/2/library/logging.html#formatter-objects" rel="nofollow">https://docs.python.org/2/library/logging.html#formatter-objects</a>
<a href="https://docs.python.org/2/library/logging.html#logrecord-attributes" rel="nofollow">https://docs.python.org/2/library/logging.html#logrecord-attributes</a>
<a href="https://docs.python.org/2.7/library/logging.config.html#configuration-file-format" rel="nofollow">https://docs.python.org/2.7/library/logging.config.html#configuration-file-format</a></p>

python logging