From subprocess.Popen to multiprocessing
<p>I got a function that invokes a process using <code>subprocess.Popen</code> in the following way:</p>

<pre><code>    def func():
        ...
        process = subprocess.Popen(substr, shell=True, stdout=subprocess.PIPE)

        timeout = {"value": False}
        timer = Timer(timeout_sec, kill_proc, [process, timeout])
        timer.start()

        for line in process.stdout:
            lines.append(line)

        timer.cancel()
        if timeout["value"] == True:
            return 0
        ...
</code></pre>

<p>I call this function from other function using a loop (e.g from range(1,100) ) , how can I make multiple calls to the function with multiprocessing? that each time several processes will run in parallel</p>

<p>The processes doesn't depend on each other, the only constraint is that each process would be 'working' on only one index (e.g no two processes will work on index 1)</p>

<p>Thanks for your help</p>

<p>Just add the index to your <code>Popen</code> call and create a <a href="https://docs.python.org/3.6/library/multiprocessing.html" rel="nofollow">worker pool</a> with as many CPU cores you have available.</p>

<pre><code>import multiprocessing

def func(index):
    ....
    process = subprocess.Popen(substr + " --index {}".format(index), shell=True, stdout=subprocess.PIPE)
    ....

if __name__ == '__main__':
    p = multiprocessing.Pool(multiprocessing.cpu_count())
    p.map(func, range(1, 100))
</code></pre>

python multiprocessing