Submitting Python Application with Apache Spark Submit
<p>I am trying to follow the examples on the Apache Spark documentation site: <a href="https://spark.apache.org/docs/2.0.0-preview/submitting-applications.html" rel="nofollow">https://spark.apache.org/docs/2.0.0-preview/submitting-applications.html</a></p>

<p>I started a Spark standalone cluster and want to run the example Python application. I am in my spark-2.0.0-bin-hadoop2.7 directory and ran the following command</p>

<pre><code>./bin/spark-submit \
--master spark://207.184.161.138:7077 \
examples/src/main/python/pi.py \
1000
</code></pre>

<p>However, I get the error</p>

<pre><code>jupyter: '/Users/MyName/spark-2.0.0-bin- \
hadoop2.7/examples/src/main/python/pi.py' is not a Jupyter command
</code></pre>

<p>This is what my bash_profile looks like</p>

<pre><code>#setting path for Spark
export SPARK_PATH=~/spark-2.0.0-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'
</code></pre>

<p>What am I doing wrong?</p>

<p>The <code>PYSPARK_DRIVER_PYTHON</code> and <code>PYSPARK_DRIVER_PYTHON_OPTS</code> are meant for running the ipython/jupyter shell when opening the pyspark shell ( More info at <a href="http://stackoverflow.com/questions/31862293/how-to-load-ipython-shell-with-pyspark">How to load IPython shell with PySpark</a> ).</p>

<p>You can set this up like:</p>

<pre><code>alias snotebook='PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook $SPARK_PATH/bin/pyspark --master local[2]'
</code></pre>

<p>So that it doesn't interfere with pyspark when submitting</p>

python apache-spark pyspark