Running python in parallel
<p>I have this bash script:</p>

<pre><code>#!/bin/bash
for i in `seq 1 32`;
do
python run.py file$i.txt &amp;
if (( $i % 10 == 0 )); then wait; fi
done
wait
</code></pre>

<p>I want to run it on a cluster with 64 cores. How will the 10 files be distributed among the cores? IF I have 10 free cores at the same time, each core will take one file, or there is a more complex way? Also, if all of the ore are free, should I run 64 files at a time, or it might encounters problems if I use all of them (like the program might be slowed down due to memory issues?</p>

<p>When you run the scripts from the shell in paralel, you are delegating responsibility of process managment to the OS scheduler, which is not a bad idea, because the OS knows well how to handle process through the processor cores, but sometimes you need more control about the tasks execution (maybe because you will run a really large number of tasks) if so, in my opinion the best approach will be run your tasks using the python multiprocessing module and program the tasks execution logic.</p>

python bash parallel-processing