Under what circumstances must I use py-files option of spark-submit?
<p>Just poking around spark-submit, I was under the impression that if my application has got dependencies on other .py files then I have to distribute them using the py-files option (see <a href="http://spark.apache.org/docs/latest/submitting-applications.html#bundling-your-applications-dependencies" rel="nofollow">bundling your applications dependencies</a>). I took that to mean any file had to be declared using py-files yet the following works fine... two <code>.py</code> files:  </p>

<p><code>spark_submit_test_lib.py</code>:</p>

<pre><code>def do_sum(sc) :
  data = [1, 2, 3, 4, 5]
  distData = sc.parallelize(data)
  return distData.sum()
</code></pre>

<p>and <code>spark_submit_test.py</code>:  </p>

<pre><code>from pyspark import SparkContext, SparkConf
from spark_submit_test_lib import do_sum
conf = SparkConf().setAppName('JT_test')
sc = SparkContext(conf=conf)
print do_sum(sc)
</code></pre>

<p>submitted using: </p>

<pre><code>spark-submit --queue 'myqueue' spark_submit_test.py
</code></pre>

<p>All worked fine. Code ran, yields the correct result, spark-submit terminates gracefully.<br>
However, I would have thought having read the documentation that I would have had to do this: </p>

<pre><code>spark-submit --queue 'myqueue' --py-files spark_submit_test_lib.py spark_submit_test.py
</code></pre>

<p>That still worked of course. I'm just wondering why the former worked as well. Any suggestions?</p>

<p>You must be submitting this in local environment where your driver and executors runs on the same machine , that is the the reason it worked ,but if you deploy in cluster and try to run from there  you have to use --pf-files option.</p>

<p>Please check the <a href="http://spark.apache.org/docs/latest/#running-the-examples-and-shell" rel="nofollow">link</a> for more details</p>

python apache-spark