Why can I not create / access futures in callables submitted to ProcessPoolExecutor?
<p>Why does this code work with threads but not processes?</p>

<pre><code>import concurrent.futures as f
import time
def wait_on_b():
    time.sleep(2)
    print(b.result())
    return 5

def wait_5():
    time.sleep(2)
    return 6

THREADS = False
if THREADS:
    executor = f.ThreadPoolExecutor()
else:
    executor = f.ProcessPoolExecutor()
a = executor.submit(wait_on_b)
b = executor.submit(wait_5)
print(a.result()) # works fine if THREADS, BrokenProcessPool otherwise
</code></pre>

<p>The <a href="https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor" rel="nofollow">docs</a> do warn:</p>

<blockquote>
  <p>Calling Executor or Future methods from a callable submitted to a ProcessPoolExecutor will result in deadlock.</p>
</blockquote>

<p>The docs don't seem to mention raising an exception, so does it mean <code>ProcessPoolExecutor</code> somehow discovered the deadlock and resolved it by killing both processes?</p>

<p>More importantly, why is this deadlock unavoidable with processes (and avoidable with threads), and what is the workaround if I want to use multiple processes with futures, without being so restricted?</p>

<p>When using threads the <strong>memory is shared</strong> between all threads and that's why <em>wait_on_b</em> can access <em>b</em>.</p>

<p>Using processes, A <strong>new memory space</strong> is created for each process (copy of the old one <em>in fork mode</em>) so You will get a copy of <em>b</em> with a broken PIPE since it is not the <strong>real</strong> <em>b</em> (just a copy)</p>

<p>BTW: on windows there is <strong>no fork</strong>, so <em>b</em> (the memory is totally new) does not exists and you'll get a </p>

<pre><code>concurrent.futures.process._RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "C:\Anaconda3\lib\concurrent\futures\process.py", line 175, in _process_worker
        r = call_item.fn(*call_item.args, **call_item.kwargs)
      File "C:\Users\yglazner\My Documents\LiClipse Workspace\anaconda_stuff\mproc.py", line 5, in wait_on_b
        print(b.result())
    NameError: name 'b' is not defined
    """
</code></pre>

python python-3.x python-multithreading python-multiprocessing concurrent.futures