How to import a local python module when using the sbatch command in SLURM
<p>I was using the cluster manager <a href="http://slurm.schedmd.com/" rel="nofollow">slurm</a> and I was running a submission script with sbatch (with a python interpeter). The sbatch submission imported one of my modules called <code>main_nn.py</code>. The module is located in the same place as my submission directory, however, python fails to find it even though the file exists. I am having a hard time figuring it out why this is happening. My python file looks as follow:</p>

<pre><code>#!/usr/bin/env python
#SBATCH --job-name=Python

print('hi')

import main_nn
</code></pre>

<p>however the output of my slurm dump file is:</p>

<pre><code>hi
Traceback (most recent call last):
    File "/home/slurm/slurmd/job3223398/slurm_script", line6, in &lt;module&gt;
        import main_nn
ImportError: No module named main_nn
</code></pre>

<p>I tried checking if the module <code>main_nn</code> was in the current directory and it was there indeed. Thus, the first thing that seemed suspicious to me was that the error in the slurm file said the location of my script was at <code>"/home/slurm/slurmd/job3223398/slurm_script"</code> rather than at <code>path_to_project</code>. Thus I went ahead an added the line </p>

<p>os.system('pwd')</p>

<p>to see where my script was executing from and to my surprise it was executing at <code>path_to_project</code> rather than at <code>"/home/slurm/slurmd/job3223398/slurm_script"</code> which must mean that sbatch is doing something funky to executed a script at one location but make it think its at another. If this is the case how am I suppose to do an import in python where the module is in the same location as in my submission script? Am I forced to put it in a package and trick python to think its in a package/library?</p>

<p>As Slurm copies the submission script to a specific location on the compute node to run it, your Python script will not find the modules that are in the submission directory.</p>

<p>But Slurm correctly sets the current working directory so you can explicitly add it to the python path with something like:</p>

<pre><code>sys.path.append(os.getcwd()) 
</code></pre>

<p>near the beginning of your script.</p>

python slurm