ImportError: No module named caffe while running spark-submit
<p>While running a <code>spark-submit</code> on a spark standalone cluster comprised of one master and 1 worker, the <code>caffe</code> python module does not get imported due to error <code>ImportError: No module named caffe</code></p>

<p>This doesn't seem to be an issue whenever I run a job locally by 
<code>spark-submit --master local script.py</code> the <code>caffe</code> module gets imported just fine.</p>

<p>The environmental variables are currently set under <code>~/.profile</code> for spark and caffe and they are pointing to the <code>PYTHONPATH</code>.</p>

<p>Is <code>~/.profile</code> the correct location to set these variables or perhaps a system wide configuration is needed such as adding the variables under <code>/etc/profile.d/</code></p>

<p>Please note that the CaffeOnSpark team ported Caffe to a distributed environment backed by Hadoop and Spark. You cannot, I am 99.99% sure, use Caffe alone <em>(without any modifications)</em> in a Spark cluster or any distributed environment per se. (Caffe team is known to be working on this).</p>

<p>If you need distributed deep-learning using Caffe, please follow the building method mentioned here in <a href="https://github.com/yahoo/CaffeOnSpark/wiki/build" rel="nofollow">https://github.com/yahoo/CaffeOnSpark/wiki/build</a> to build CaffeOnSpark for that and use CaffeOnSpark instead of Caffe. </p>

<p>But, best bet will be to follow either this <a href="https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_standalone" rel="nofollow">GetStarted_standalone wiki</a> or <a href="https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_yarn" rel="nofollow">GetStarted_yarn wiki</a> to create you a distributed environment to carry out deep-learning.</p>

<p>Further, to add python, please go through <a href="https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_python" rel="nofollow">GetStarted_python wiki</a>.</p>

<p>Also, since you mentioned that you are using Ubuntu <a href="https://github.com/yahoo/CaffeOnSpark/issues/164" rel="nofollow">here</a>, please use <code>~/.bashrc</code> to update environment your variables. You will have to source the file after the changes: <code>source ~/.bashrc</code></p>

python ubuntu apache-spark caffe pycaffe