Using Python to kick off Python subprocesses without delay
<p>I want to launch multiple instances of a python script using <code>subprocess.call</code>, but the kick off script waits for each to complete. How do I prevent it from waiting from going one by one, without waiting for the previous job to complete?</p>

<pre><code>step = 5
for n in range(5, 11, step):
    subprocess.call(["python", cwd + "/" + "subprocess.py", str(n - step), str(n)])
</code></pre>

<p>That's the documented behaviour of <a href="https://docs.python.org/3/library/subprocess.html#subprocess.call" rel="nofollow"><code>subprocess.call()</code></a> so you can't use it that way. Instead you can use <a href="https://docs.python.org/3/library/subprocess.html#subprocess.Popen" rel="nofollow"><code>subprocess.Popen()</code></a>.</p>

<pre><code>import subprocess
import os.path

processes = []
step = 5
for n in range(5, 11, step):
    processes.append(subprocess.Popen(['python', os.path.join(cwd, 'child.py'), str(n - step), str(n)]))

for p in processes:    # wait for the child processes to terminate, avoid zombies
    p.wait()
</code></pre>

<p>Note that it is a bad idea to name a file <code>subprocess.py</code>, especially if it is in the same directory as your main script - an <code>import subprocess</code> will import the local version, not the system version. I've renamed it to <code>child.py</code> in the above code.</p>

<p>It is also important that the parent process waits for the child processes. Omitting this can lead to <a href="https://en.wikipedia.org/wiki/Zombie_process" rel="nofollow">"zombie" processes</a> in Linux.</p>

<p>If you are using Python 3 you could investigate use of the <a href="https://docs.python.org/3/library/asyncio.html#module-asyncio" rel="nofollow"><code>asyncio</code></a> module.</p>

python concurrency subprocess