SLURM with python multiprocessing give inconsistent results
<p>We have a small HPC with 4*64 cores, and it has SLURM installed in it.</p>

<p>The nodes are:</p>

<pre><code>sinfo -N -l
Mon Oct  3 08:58:12 2016
NODELIST    NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT FEATURES REASON              
dlab-node1      1     dlab*        idle   64   2:16:2 257847        0      1   (null) none                
dlab-node2      1     dlab*        idle   64   2:16:2 257847        0      1   (null) none                
dlab-node3      1     dlab*        idle   64   2:16:2 257847        0      1   (null) none                
dlab-node4      1     dlab*        idle   64   2:16:2 257847        0      1   (null) none  
</code></pre>

<p>To test the SLURM i wrote a little script in python with multiprocessing:</p>

<pre><code>import multiprocessing
import os
def func(i):
    print(n_procs)

n_procs = int(os.environ['SLURM_JOB_CPUS_PER_NODE'].split('(')[0]) * int(os.environ['SLURM_JOB_NUM_NODES'])
p = multiprocessing.Pool(n_procs)
list(p.imap_unordered(func, [i for i in range(n_procs*2)]))
</code></pre>

<p>I use the following batch <code>sh</code> script to run it with SLURM</p>

<pre><code>#!/bin/bash
#
#SBATCH -p dlab                # partition (queue)
#SBATCH -N 2                      # number of nodes
#SBATCH -n 64                     # number of cores
#SBATCH --mem 250                 # memory pool for all cores
#SBATCH -t 0-2:00                 # time (D-HH:MM)
#SBATCH -o slurm.%N.%j.out        # STDOUT
#SBATCH -e slurm.%N.%j.err        # STDERR

python3 asd.py
</code></pre>

<p>As i would expect this would print <code>128</code> <code>256</code> times to the STDOUT file.</p>

<p>However if i run this multiple times i get very different amount of lines (they all contain <code>128</code> which is correct)</p>

<p>For the first run i got 144 lines, the second time i got 256 (which is correct) and the third time i get 184.</p>

<p>What is the problem? Should i investigate something inside the configuration of SLURM, or there is something wrong within python <code>multiprocessing</code>? </p>

<p>From sbatch man page:</p>

<blockquote>
  <p>SLURM_JOB_CPUS_PER_NODE</p>
  
  <p>Count of processors available to the job <strong>on this node</strong>.  Note the select/linear plugin allocates entire nodes to jobs, so the value indicates the total count of CPUs on the node.  The select/cons_res plugin allocates individual processors to jobs, so this number indicates the number of processors on this node allocated to the job</p>
</blockquote>

<p>As highlighted, the variable will only return the number on cpus allocated in the node where the script is running. If you want to have an homogeneous allocation you should specify <code>--ntasks-per-node=32</code></p>

<p>Also, bear in mind that multiprocessing will not spawn processes in more than one node. If you want to span multiple nodes you have a nice documentation <a href="https://rcc.uchicago.edu/docs/tutorials/kicp-tutorials/running-jobs.html#multiprocessing" rel="nofollow">here</a></p>

python python-3.x multiprocessing slurm