Why does spark-submit in YARN cluster mode not find python packages on executors?
<p>I am running a <code>boo.py</code> script on AWS EMR using <code>spark-submit</code> (Spark 2.0).</p>

<p>The file finished successfully when I use</p>

<pre><code>python boo.py
</code></pre>

<p>However, it failed when I run</p>

<pre><code>spark-submit --verbose --deploy-mode cluster --master yarn  boo.py
</code></pre>

<p>The log on <code>yarn logs -applicationId ID_number</code> shows:</p>

<pre><code>Traceback (most recent call last):
File "boo.py", line 17, in &lt;module&gt;
import boto3
ImportError: No module named boto3
</code></pre>

<p>The <code>python</code> and <code>boto3</code> module I am using is </p>

<pre><code>$ which python
/usr/bin/python
$ pip install boto3
Requirement already satisfied (use --upgrade to upgrade): boto3 in /usr/local/lib/python2.7/site-packages
</code></pre>

<p>How do I append this library path so that <code>spark-submit</code> could read the <code>boto3</code> module?</p>

<p>When you are running spark, part of the code is running on the driver, and part is running on the executors.</p>

<p>Did you install boto3 on the driver only, or on driver + all executors (nodes) which might run your code?</p>

<p>One solution might be - to install boto3 on all executors (nodes)</p>

<p><strong>how to install python modules on Amazon EMR nodes</strong>:</p>

<p><a href="http://stackoverflow.com/questions/31525012/how-to-bootstrap-installation-of-python-modules-on-amazon-emr">How to bootstrap installation of Python modules on Amazon EMR?</a></p>

python apache-spark pyspark