Spark problems with imports in Python
<p>We are running a spark-submit command on a python script that uses Spark to parallelize object detection in Python using Caffe. The script itself runs perfectly fine if run in a Python-only script, but it returns an import error when using it with Spark code. I know the spark code is not the problem because it works perfectly fine on my home machine, but it is not functioning well on AWS. I am not sure if this somehow has to do with the environment variables, it is as if it doesn't detect them.</p>

<p>These environment variables are set:  </p>

<pre><code>SPARK_HOME=/opt/spark/spark-2.0.0-bin-hadoop2.7
PATH=$SPARK_HOME/bin:$PATH
PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
PYTHONPATH=/opt/caffe/python:${PYTHONPATH}
</code></pre>

<p>Error:</p>

<pre><code>16/10/03 01:36:21 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 172.31.50.167): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
 File "/opt/spark/spark-2.0.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 161, in main
   func, profiler, deserializer, serializer = read_command(pickleSer, infile)
 File "/opt/spark/spark-2.0.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 54, in read_command
   command = serializer._read_with_length(file)
 File "/opt/spark/spark-2.0.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 164, in _read_with_length
   return self.loads(obj)
 File "/opt/spark/spark-2.0.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 422, in loads
   return pickle.loads(obj)
 File "/opt/spark/spark-2.0.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/cloudpickle.py", line 664, in subimport
   __import__(name)
ImportError: ('No module named caffe', &lt;function subimport at 0x7efc34a68b90&gt;, ('caffe',))
</code></pre>

<p>Does anyone know why this would be an issue?</p>

<p>This package from Yahoo manages what we're trying to do by shipping Caffe as a jar dependency and then uses it again in Python. But I haven't found any resources on how to build it and import it ourselves.</p>

<p><a href="https://github.com/yahoo/CaffeOnSpark">https://github.com/yahoo/CaffeOnSpark</a></p>

<p>You probably havenât compiled the caffe python wrappers in your AWS environment.  For reasons that completely escape me (and several others, <a href="https://github.com/BVLC/caffe/issues/2440" rel="nofollow">https://github.com/BVLC/caffe/issues/2440</a>) pycaffe is not available as a pypi package, and you have to compile it yourself.  You should follow the compilation/make instructions here or automate it using ebextensions if you are in an AWS EB environment: <a href="http://caffe.berkeleyvision.org/installation.html#python" rel="nofollow">http://caffe.berkeleyvision.org/installation.html#python</a></p>

python apache-spark pyspark caffe pycaffe