Using xargs for parallel Python scripts
<p>I currently have a bash script, script.sh, with two nested loops. The first enumerates possible values for a, and the second enumerates possible values for b, like</p>

<pre><code>#!/bin/sh
for a in {1..10}
do
    for b in {1..10}
    do
        nohup python script.py $a $b &amp;
    done
done
</code></pre>

<p>So this spawns off 100 Python processes running script.py, one for each (a,b) pair. However, my machine only has 5 cores, so I want to cap the number of processes at 5 to avoid thrashing/wasteful switching. The goal is that I am always running 5 processes until all 100 processes are done.</p>

<p>xargs seems to be one way to do this, but I don't know how to pass these arguments to xargs. I've checked other similar questions but don't understand the surrounding bash jargon well enough to know what's happening. For example, I tried</p>

<pre><code>seq 1 | xargs -i --max-procs=5 bash script.sh
</code></pre>

<p>but this doesn't seem to do anything - script.sh runs as before and still spawns off 100 processes.</p>

<p>I assume I'm misunderstanding how xargs works.</p>

<p>Thanks!</p>

<p>This would actually look more like:</p>

<pre><code>#!/bin/bash
for a in {1..10}; do
  for b in {1..10}; do
    printf '%s\0' "$a" "$b"
  done
done | xargs -0 -x -n 2 -P 5 python script.py
</code></pre>

<p>Note that there's no <code>nohup</code>, nor any <code>&amp;</code> -- to track the number of concurrent invocations, <code>xargs</code> needs to be directly executing the Python script, and that process can't exit until it's complete.</p>

<p>The non-standard (but widely available) <code>-0</code> extension requires input to be in NUL-delimited form (as created with <code>printf '%s\0'</code>); this ensures correct behavior with arguments having spaces, quotes, backslashes, etc.</p>

<p>The likewise non-standard <code>-P 5</code> sets the maximum number of processes (in a way slightly more portable than <code>--max-procs=5</code>, which is supported on GNU but not modern BSD xargs).</p>

<p>The <code>-n 2</code> indicates that each instance of the Python script receives only two arguments, thus starting one per pair of inputs.</p>

<p>The <code>-x</code> (used in conjunction with <code>-n 2</code>) indicates that if a single Python instance can't be given two arguments (for instance, if the arguments are so long that both can't fit on a single command line), this should be treated as a failure, rather than invoking a Python instance with only one argument.</p>

python bash parallel-processing xargs