Using xargs for parallel Python scripts
<p>I currently have a bash script, script.sh, with two nested loops. The first enumerates possible values for a, and the second enumerates possible values for b, like</p>

<pre><code>#!/bin/sh
for a in {1..10}
do
    for b in {1..10}
    do
        nohup python script.py $a $b &amp;
    done
done
</code></pre>

<p>So this spawns off 100 Python processes running script.py, one for each (a,b) pair. However, my machine only has 5 cores, so I want to cap the number of processes at 5 to avoid thrashing/wasteful switching. The goal is that I am always running 5 processes until all 100 processes are done.</p>

<p>xargs seems to be one way to do this, but I don't know how to pass these arguments to xargs. I've checked other similar questions but don't understand the surrounding bash jargon well enough to know what's happening. For example, I tried</p>

<pre><code>seq 1 | xargs -i --max-procs=5 bash script.sh
</code></pre>

<p>but this doesn't seem to do anything - script.sh runs as before and still spawns off 100 processes.</p>

<p>I assume I'm misunderstanding how xargs works.</p>

<p>Thanks!</p>

<p>GNU Parallel is made for exactly these kinds of jobs:</p>

<pre><code>parallel python script.py ::: {1..10} ::: {1..10}
</code></pre>

<p>If you need $a and $b placed differently you can use {1} and {2} to refer to the two input sources:</p>

<pre><code>parallel python script.py --option-a {1} --option-b {2} ::: {1..10} ::: {1..10}
</code></pre>

<p>GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a <code>for</code> loop.</p>

<p>If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:</p>

<p><img src="http://i.stack.imgur.com/uH0Dh.png" alt="Simple scheduling"></p>

<p>GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:</p>

<p><img src="http://i.stack.imgur.com/17FsG.png" alt="GNU Parallel scheduling"></p>

<p><strong>Installation</strong></p>

<p>If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:</p>

<pre><code>(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
</code></pre>

<p>For other installation options see <a href="http://git.savannah.gnu.org/cgit/parallel.git/tree/README" rel="nofollow">http://git.savannah.gnu.org/cgit/parallel.git/tree/README</a></p>

<p><strong>Learn more</strong></p>

<p>See more examples: <a href="http://www.gnu.org/software/parallel/man.html" rel="nofollow">http://www.gnu.org/software/parallel/man.html</a></p>

<p>Watch the intro videos: <a href="https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1" rel="nofollow">https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1</a></p>

<p>Walk through the tutorial: <a href="http://www.gnu.org/software/parallel/parallel_tutorial.html" rel="nofollow">http://www.gnu.org/software/parallel/parallel_tutorial.html</a></p>

<p>Sign up for the email list to get support: <a href="https://lists.gnu.org/mailman/listinfo/parallel" rel="nofollow">https://lists.gnu.org/mailman/listinfo/parallel</a></p>

python bash parallel-processing xargs