Using python's multiprocessing on slurm
<p>I am trying to run some parallel code on slurm, where the different processes do not need to communicate. Naively I used python's slurm package. However, it seems that I am only using the cpu's on one node.</p>

<p>For example, if I have 4 nodes with 5 cpu's each, I will only run 5 processes at the same time. How can I tell multiprocessing to run on different nodes?</p>

<p>The python code looks like the following</p>

<pre><code>import multiprocessing

def hello():
    print("Hello World")

pool = multiprocessing.Pool() 
jobs = [] 
for j in range(len(10)):
    p = multiprocessing.Process(target = run_rel)
    jobs.append(p)
    p.start() 
</code></pre>

<p>The problem is similar to <a href="http://stackoverflow.com/questions/32594734/slurm-multiprocessing-python-job">this one</a>, but there it has not been solved in detail.</p>

<p>Your current code will run 10 times on 5 processor, on a SINGLE node where you start it. It has nothing to do with SLURM now. </p>

<p>You will have to <code>SBATCH</code> the script to SLURM.</p>

<p>If you want to run this script on 5 cores with SLURM modify the script like this:</p>

<pre><code>#!/usr/bin/python3

#SBATCH --output=wherever_you_want_to_store_the_output.log
#SBATCH --partition=whatever_the_name_of_your_SLURM_partition_is
#SBATCH -n 5 # 5 cores

import sys
import os
import multiprocessing

# Necessary to add cwd to path when script run
# by SLURM (since it executes a copy)
sys.path.append(os.getcwd())

def hello():
    print("Hello World")

pool = multiprocessing.Pool() 
jobs = [] 
for j in range(len(10)):
    p = multiprocessing.Process(target = run_rel)
    jobs.append(p)
    p.start() 
</code></pre>

<p>And then execute the script with </p>

<pre><code>sbatch my_python_script.py
</code></pre>

<p>On one of the nodes where SLURM is installed</p>

<p>However this will allocate your job to a SINGLE node as well, so the speed will be the very same as you would just run it on a single node. </p>

<p>I dont know why would you want to run it on different nodes when you have just 5 processes. It will be faster just to run on one node. If you allocate more then 5 cores, in the beginning of the python script, then SLURM will allocate more nodes for you. </p>

python multiprocessing slurm