A print function makes a multiprocessing program fail
<p>In the following code, I'm trying to create a sandboxed master-worker system, in which changes to global variables in a worker don't reflect to other workers.</p>

<p>To achieve this, a new process is created each time a task is created, and to make the execution parallel, the creation of processes itself is managed by <code>ThreadPoolExecutor</code>.</p>

<pre><code>import time
from concurrent.futures import ThreadPoolExecutor
from multiprocessing import Pipe, Process


def task(conn, arg):
  conn.send(arg * 2)


def isolate_fn(fn, arg):

  def wrapped():
    parent_conn, child_conn = Pipe()
    p = Process(target=fn, args=(child_conn, arg), daemon=True)
    try:
      p.start()
      r = parent_conn.recv()
    finally:
      p.join()
    return r

  return wrapped


def main():
  with ThreadPoolExecutor(max_workers=4) as executor:
    pair = []

    for i in range(0, 10):
      pair.append((i, executor.submit(isolate_fn(task, i))))

      # This function makes the program broken.
      # 
      print('foo')

    time.sleep(2)

    for arg, future in pair:
      if future.done():
        print('arg: {}, res: {}'.format(arg, future.result()))
      else:
        print('not finished: {}'.format(arg))

  print('finished')

main()
</code></pre>

<p>This program works fine, until I put the <code>print('foo')</code> function inside the loop.  If the function exists, some tasks remain unfinished, and what is worse, this program itself doesn't finish.</p>

<p>Results are not always the same, but the following is the typical output:</p>

<pre><code>foo
foo
foo
foo
foo
foo
foo
foo
foo
foo
arg: 0, res: 0
arg: 1, res: 2
arg: 2, res: 4
not finished: 3
not finished: 4
not finished: 5
not finished: 6
not finished: 7
not finished: 8
not finished: 9
</code></pre>

<p>Why is this program so fragile?</p>

<p>I use Python 3.4.5.</p>

<p>Try using</p>

<pre><code>from multiprocessing import set_start_method

... rest of your code here ....

if __name__ == '__main__':
    set_start_method('spawn')
    main()
</code></pre>

<p>If you search Stackoverflow for python multiprocessing and multithreading you will find a a fair few questions mentioning similar hanging issues. (esp. for python version 2.7 and 3.2)</p>

<p>Mixing multithreading and multiprocessing ist still a bit of an issue and even the python docs for <a href="https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods" rel="nofollow">multiprocessing.set_start_method</a> mention that. In your case <em>'spawn'</em> and <em>'forkserver'</em> should work without any issues.</p>

<p>Another option might be to use MultiProcessingPool directly, but this may not be possible for you in a more complex use case.</p>

<p>Btw. <em>'Not Finished'</em> may still appear in your output, as you are not waiting for your sub processes to finish, but the whole code should not hang anymore and always finish cleanly.</p>

python python-3.x python-multithreading python-multiprocessing