Supercomputer: Dead simple example of a program to run in supercomputer
<p>I am learning how to use supercomputers to make the good use of resources.
Let's say I have a python script, that will create a text file with given random number.</p>

<h1>myfile.py</h1>

<pre><code># Imports
import random,os

outdir = 'outputs'
if not os.path.exists(outdir):
    os.makedirs(outdir)

with open (outdir+'/temp.txt','w') as f :
    a = random.randint(0,9)
    f.write(str(a))
</code></pre>

<p>This will create only one text file in the local machine.<br>
Is there any way I can use the multiple instances of this program, use multiple nodes and get multiple outputs?  </p>

<p>I got a template for mpiexec in C program which looks like this, but I could not find any template for python program.</p>

<pre><code>#PBS -N my_job
#PBS -l walltime=0:10:00
#PBS -l nodes=4:ppn=12
#PBS -j oe

cd $PBS_O_WORKDIR

mpicc -O2 mpi-hello.c -o mpi-hello

cp $PBS_O_WORKDIR/* $PFSDIR
cd $PFSDIR

mpiexec ./mpi-hello

cp $PFSDIR/* $PBS_O_WORKDIR
</code></pre>

<p><strong>Note:</strong>  On a single node using multiple cores I can write a bash script like this:  </p>

<pre><code>for i in `seq 1 10`;
    do
        python myfile.py &amp;&amp; cp temp.txt outputs/out$i.txt &amp;
    done
</code></pre>

<p>But I want to utilize different <strong>nodes</strong>.<br>
<strong>Required output:</strong>
outputs/out1.txt,out2.txt,out3.txt  etc</p>

<p>Some related links are following:<br>
<a href="https://www.osc.edu/sites/osc.edu/files/documentation/Batch%20Training%20-%2020150312%20-%20OSC.pdf" rel="nofollow">https://www.osc.edu/sites/osc.edu/files/documentation/Batch%20Training%20-%2020150312%20-%20OSC.pdf</a><br>
<a href="https://www.osc.edu/~kmanalo/multithreadedsubmission" rel="nofollow">https://www.osc.edu/~kmanalo/multithreadedsubmission</a>  </p>

<p>Take a look to this link it may solve your problem</p>

<p><a href="http://materials.jeremybejarano.com/MPIwithPython/introMPI.html" rel="nofollow">http://materials.jeremybejarano.com/MPIwithPython/introMPI.html</a></p>

<p>so your code may be something like:</p>

<pre><code>from mpi4py import MPI
import random,os

outdir = 'outputs'
comm = MPI.COMM_WORLD
rank = comm.Get_rank()

if not os.path.exists(outdir):
    os.makedirs(outdir)

with open (outdir+'/temp%s.txt' % rank,'w') as f :
    a = random.randint(0,9)
    f.write(str(a))
</code></pre>

<p>and the pbs file:</p>

<pre><code>#!/bin/bash
################################################################################
#PBS -N myfile.py
#PBS -l nodes=7:ppn=4
#PBS -l walltime=30:30:00:00
#PBS -m bea
##PBS -M mail@mail.mail
###############################################################################

cores=$(awk 'END {print NR}' $PBS_NODEFILE)
mpirun -np $cores python myfile.py
</code></pre>

python supercomputers mpiexec