How can I get Spark to see code in a different module?
<p>I have complicated function that I run over a dataset in spark using the map function. It is in a different python module. When map is called, the executor nodes do not have that code and then the map function fails.</p>

<pre><code>s_cobDates = getCobDates() #returns a list of dates
sb_dataset = sc.broadcast(dataset) #fyi - it is not trivial to slice this into chunks per date

def sparkInnerLoop(n_cobDate):
   n_dataset = sb_dataset.value
   import someOtherModule
   return someOtherModule.myComplicatedCalc(n_dataset)

results = s_cobDates.map(sparkInnerLoop).collect()
</code></pre>

<p>Spark then fails as it can't import myOtherModule.</p>

<p>So far I have got round it by creating a python package that contains someOtherModule and deploying that to the cluster in advance of my spark jobs, but that doesn't make for rapid prototyping.</p>

<p>How can I get spark to send the complete code to the executor nodes, without inlining all the code into "sparkInnerLoop"? That code is used elsewhere in my solution and I don't want code duplication.</p>

<p>I'm using an eight node cluster in stand alone mode, v 1.6.2, and the driver is running on my workstation in pycharm.</p>

<p>Well the above answer works, it falls down if your modules are part of a package. Instead, its possible to zip your modules and then add the zip file to your spark context and then they have the correct package name.</p>

<pre><code>def ziplib():
    libpath = os.path.dirname(__file__)  # this should point to your packages directory
    zippath = r'c:\Temp\mylib-' + randstr.randstr(6) + '.zip'
    zippath = os.path.abspath(zippath)
    zf = zipfile.PyZipFile(zippath, mode='w')
    try:
        zf.debug = 3  # making it verbose, good for debugging
        zf.writepy(libpath)
        return zippath  # return path to generated zip archive
    finally:
        zf.close()

sc = SparkContext(conf=conf)

zip_path = ziplib()  # generate zip archive containing your lib
zip_path = pathlib.Path(zip_path).as_uri()
sc.addPyFile(zip_path)  # add the entire archive to SparkContext
</code></pre>

python apache-spark pyspark