Python, using threading with multiprocessing
<p>Can someone explain why threading don't work in multiprocessing.Process.</p>

<p>I've attached some example to explain my problem.</p>

<p>I have a process that executed every second and write to file. When I run it from shell, it works as expected.</p>

<p><strong>stat_collect.py</strong></p>

<pre><code>#!/usr/bin/env python
# -*- coding: utf-8 -*-

from threading import Timer
from os import path
from datetime import datetime

STAT_DATETIME_FMT = '%Y-%m-%d %H:%M:%S'


def collect_statistics():
    my_file = 'test.file'
    if not path.exists(my_file):
        with open(my_file, 'w') as fp:
            fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')
    else:
        with open(my_file, 'a') as fp:
            fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')

    Timer(1, collect_statistics).start()


if __name__ == '__main__':
    collect_statistics()
</code></pre>

<p>When I try to run it from other script (to work in background):</p>

<pre><code>#!/usr/bin/env python

from multiprocessing import Process
from stat_collect import collect_statistics  # logger sc

if __name__ == '__main__':
    # This don't work
    p = Process(target=collect_statistics)
    p.start()

    while True:
        pass
</code></pre>

<p>Method collect_statistics executed only once, but if I use Thread(target=collect_statistics).start() it works as if I run it from shell. Why this is happen?</p>

<p>Here is what is going on:</p>

<ol>
<li>You start your process </li>
<li><code>collect_statistics</code> runs</li>
<li>Timer is started</li>
<li>now the function called in the process(<code>collect_statistics</code>) is finished, so the process
quit, killing the timer in the same time.</li>
</ol>

<p>Here is how to fix it :</p>

<p><strong>stat_collect.py</strong></p>

<pre><code>#!/usr/bin/env python
# -*- coding: utf-8 -*-

from threading import Timer
from os import path
from datetime import datetime
import time

STAT_DATETIME_FMT = '%Y-%m-%d %H:%M:%S'


def collect_statistics():
    while True:
        my_file = 'test.file'
        if not path.exists(my_file):
            with open(my_file, 'w') as fp:
                fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')
        else:
            with open(my_file, 'a') as fp:
                fp.write(datetime.now().strftime(STAT_DATETIME_FMT) + '\n')

        time.sleep(1)


if __name__ == '__main__':
    collect_statistics()
</code></pre>

<p>And for the calling script :</p>

<pre><code>#!/usr/bin/env python

from multiprocessing import Process
from stat_collect import collect_statistics  # logger sc

if __name__ == '__main__':
    # This don't work
    p = Process(target=collect_statistics)
    p.start()
    p.join() # wait until process is over, e.g forever
</code></pre>

<p><code>p.join()</code> is just here to replace you infinite while loop, which is taking a lot of ressource for nothing.</p>

python multithreading python-multiprocessing