Figure out if called from function without main guard
<p>If a module is imported from a script without a main guard (<code>if __name__ == '__main__':</code>), doing any kind of parallelism in some function in the module will result in an infinite loop on Windows. Each new process loads all of the sources, now with <code>__name__</code> not equal to <code>'__main__'</code>, and then continues execution in parallel. If there's no main guard, we're going to do another call to the same function in each of our new processes, spawning even more processes, until we crash. It's only a problem on Windows, but the scripts are also executed on osx and linux.</p>

<p>I could check this by writing to a special file on disk, and read from it to see if we've already started, but that limits us to a single python script running at once. The simple solution of modifying all the calling code to add main guards is not feasible because they are spread out in many repositories, which I do not have access to. Thus, I would like to parallelize, when main guards are used, but fallback to single threaded execution when they're not.</p>

<p>How do I figure out if I'm being called in an import loop due to a missing main guard, so that I can fallback to single threaded execution?</p>

<p><strong>Here's some demo code:</strong></p>

<p>lib with parallel code:</p>

<pre><code>from multiprocessing import Pool


def _noop(x):
    return x


def foo():
    p = Pool(2)
    print(p.map(_noop, [1, 2, 3]))
</code></pre>

<p>Good importer (with guard):</p>

<pre><code>from lib import foo

if __name__ == "__main__":
    foo()
</code></pre>

<p>Bad importer (without guard):</p>

<pre><code>from lib import foo

foo()
</code></pre>

<p>where the bad importer fails with this RuntimeError, over and over again:</p>

<pre><code>    p = Pool(2)
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\context.py", line 118, in Pool
    context=self.get_context())
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\pool.py", line 168, in __init__
    self._repopulate_pool()
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\pool.py", line 233, in _repopulate_pool
    w.start()
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\context.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\popen_spawn_win32.py", line 34, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 144, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\filip.haglund\AppData\Local\Programs\Python\Python35\lib\multiprocessing\spawn.py", line 137, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
</code></pre>

<p>Since you're using <code>multiprocessing</code>, you can also use it to detect if you're the main process or a child process. However, these features are not documented and are therefore just implementation details that could change without warning between python versions. </p>

<p>Each process has a <code>name</code>, <code>_identity</code> and <code>_parent_pid</code>. You can check any of them to see if you're in the main process or not. In the main process <code>name</code> will be <code>'MainProcess'</code>, <code>_identity</code> will be <code>()</code>, and <code>_parent_pid</code> will be <code>None</code>).</p>

<p>My solution allows you to continue using <code>multiprocessing</code>, but just modifies child processes so they can't keep creating child processes forever. It uses a decorator to change <code>foo</code> to a no-op in child processes, but returns <code>foo</code> unchanged in the main process. This means when the spawned child process tries to execute <code>foo</code> nothing will happen (as if it had been executed inside a <code>__main__</code> guard.</p>

<pre><code>from multiprocessing import Pool
from multiprocessing.process import current_process

def run_in_main_only(func):
    if current_process().name == "MainProcess":
        return func
    else:
        def noop(*args, **kwargs):
            pass
        return noop

def _noop(_ignored):
    p = current_process()
    return p.name, p._identity, p._parent_pid

@run_in_main_only
def foo():
    with Pool(2) as p:
        for result in p.map(_noop, [1, 2, 3]):
            print(result) # prints something like ('SpawnPoolWorker-2', (2,), 10720)

if __name__ == "__main__":
    print(_noop(1)) # prints ('MainProcess', (), None)
</code></pre>

python windows multithreading