Using xargs for parallel Python scripts
<p>I currently have a bash script, script.sh, with two nested loops. The first enumerates possible values for a, and the second enumerates possible values for b, like</p>

<pre><code>#!/bin/sh
for a in {1..10}
do
    for b in {1..10}
    do
        nohup python script.py $a $b &amp;
    done
done
</code></pre>

<p>So this spawns off 100 Python processes running script.py, one for each (a,b) pair. However, my machine only has 5 cores, so I want to cap the number of processes at 5 to avoid thrashing/wasteful switching. The goal is that I am always running 5 processes until all 100 processes are done.</p>

<p>xargs seems to be one way to do this, but I don't know how to pass these arguments to xargs. I've checked other similar questions but don't understand the surrounding bash jargon well enough to know what's happening. For example, I tried</p>

<pre><code>seq 1 | xargs -i --max-procs=5 bash script.sh
</code></pre>

<p>but this doesn't seem to do anything - script.sh runs as before and still spawns off 100 processes.</p>

<p>I assume I'm misunderstanding how xargs works.</p>

<p>Thanks!</p>

<p>If you use bash, then the following should work:</p>

<pre><code>#!/bin/bash

for a in {1..10}
do
    for b in {1..10}
    do
        if [ `jobs | wc -l` -lt 6 ]; then # less than 6 background jobs
            nohup python script.py $a $b &amp;
        else
            wait -n   # wait for any background job to terminate
        fi
    done
done
</code></pre>

python bash parallel-processing xargs