Running PySpark using Cronjob (crontab)
<p>First, I assume that we have <code>SPARK_HOME</code> set up, in my case it's at <code>~/Desktop/spark-2.0.0</code>. Basically, I want to run my PySpark script using Cronjob (e.g. <code>crontab -e</code>). My question is how to add environment path to make Spark script works with Cronjob. Here is my sample script, <code>example.py</code></p>

<pre><code>import os
from pyspark import SparkConf, SparkContext

# Configure the environment
if 'SPARK_HOME' not in os.environ:
    os.environ['SPARK_HOME'] = '~/Desktop/spark-2.0.0'

conf = SparkConf().setAppName('example').setMaster('local[8]')
sc = SparkContext(conf=conf)

if __name__ == '__main__':
    ls = range(100)
    ls_rdd = sc.parallelize(ls, numSlices=10)
    ls_out = ls_rdd.map(lambda x: x+1).collect()

    f = open('test.txt', 'w')
    for item in ls_out:
        f.write("%s\n" % item) # save list to test.txt
</code></pre>

<p>My bash script in <code>run_example.sh</code> is as follows</p>

<pre><code>rm test.txt

~/Desktop/spark-2.0.0/bin/spark-submit \
  --master local[8] \
  --driver-memory 4g \
  --executor-memory 4g \
  example.py
</code></pre>

<p>Here, I want to run <code>run_example.sh</code> every minutes using <code>crontab</code>. However, I don't know how to custom path when I run <code>crontab -e</code>. So far, I only see this <a href="https://zedar.gitbooks.io/spark-hadoop-notes/content/spark_job_running_in_cron.html" rel="nofollow">Gitbook link</a>. I have something like this in my Cronjob editor that doesn't run my code yet.</p>

<pre><code>#!/bin/bash

# add path to cron (this line is the one I don't know)
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$HOME/anaconda/bin

# run script every minutes
* * * * * source run_example.sh
</code></pre>

<p>Thanks in advance!</p>

<p>What you can do is , add following line in your .bashrc file in home location.</p>

<pre><code>export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$HOME/anaconda/bin
</code></pre>

<p>then you can have following entry in crontab</p>

<pre><code>* * * * * source ~/.bashrc;sh run_example.sh
</code></pre>

<p>This line will execute your .bashrc file first, which will set the PATH value, then it will execute run_example.sh</p>

<p>Alternatively, you can set the PATH in run_example.sh only,</p>

<pre><code>export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$HOME/anaconda/bin
rm test.txt

~/Desktop/spark-2.0.0/bin/spark-submit \
  --master local[8] \
  --driver-memory 4g \
  --executor-memory 4g \
  example.py
</code></pre>

python cron pyspark crontab