How can I get Spark to see code in a different module?
<p>I have complicated function that I run over a dataset in spark using the map function. It is in a different python module. When map is called, the executor nodes do not have that code and then the map function fails.</p>

<pre><code>s_cobDates = getCobDates() #returns a list of dates
sb_dataset = sc.broadcast(dataset) #fyi - it is not trivial to slice this into chunks per date

def sparkInnerLoop(n_cobDate):
   n_dataset = sb_dataset.value
   import someOtherModule
   return someOtherModule.myComplicatedCalc(n_dataset)

results = s_cobDates.map(sparkInnerLoop).collect()
</code></pre>

<p>Spark then fails as it can't import myOtherModule.</p>

<p>So far I have got round it by creating a python package that contains someOtherModule and deploying that to the cluster in advance of my spark jobs, but that doesn't make for rapid prototyping.</p>

<p>How can I get spark to send the complete code to the executor nodes, without inlining all the code into "sparkInnerLoop"? That code is used elsewhere in my solution and I don't want code duplication.</p>

<p>I'm using an eight node cluster in stand alone mode, v 1.6.2, and the driver is running on my workstation in pycharm.</p>

<p>It is possible to dynamically distribute Python modules using <code>SparkContext.addPyFile</code></p>

<pre><code>modules_to_distribute = ["foo.py", "bar.py"]

for module in modules_to_distribute:
    sc.addPyFile(module)
</code></pre>

<p>All files distributed this way are placed on the Python path and accessible to the worker interpreters. It is also possible to distribute complete packages using egg files.</p>

<p>In general this method works only with pure Python libraries and cannot be used with C extensions.</p>

python apache-spark pyspark