Execute Python scripts with Selenium via Crontab
<p>I have several python scripts that use selenium webdriver on a Debian server. If I run them manually from the terminal (usually as root) everything is ok but every time I tried to run them via crontab I have an exception like this:</p>

<pre><code>WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.
</code></pre>

<p>Try for example this script:</p>

<pre><code>from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from pyvirtualdisplay import Display
from selenium import webdriver
import datetime
import logging

FIREFOX_PATH = '/usr/bin/firefox'

if __name__ == '__main__':
    cur_date = datetime.datetime.now().strftime('%Y-%m-%d')
    logging.basicConfig(filename="./logs/download_{0}.log".format(cur_date),
                        filemode='w',
                        level=logging.DEBUG,
                        format='%(asctime)s - %(levelname)s - %(message)s')
    try:
        display = Display(visible=0, size=(800, 600))
        display.start()
        print 'start'
        logging.info('start')
        binary = FirefoxBinary(FIREFOX_PATH,
                               log_file='/home/egor/dev/test/logs/firefox_binary_log.log')
        driver = webdriver.Firefox()
        driver.get("http://google.com")
        logging.info('title: ' + driver.title)
        driver.quit()
        display.stop()
    except:
        logging.exception('')
    logging.info('finish')
    print 'finish'
</code></pre>

<p>The crontab command for it:</p>

<pre><code>0 13 * * * cd "/home/egor/dev/test" &amp;&amp; python test.py
</code></pre>

<p>The log file for this script looks like this:</p>

<pre><code>2016-09-27 16:30:01,742 - DEBUG - param: "['Xvfb', '-help']" 
2016-09-27 16:30:01,743 - DEBUG - command: ['Xvfb', '-help']
2016-09-27 16:30:01,743 - DEBUG - joined command: Xvfb -help
2016-09-27 16:30:01,745 - DEBUG - process was started (pid=23042)
2016-09-27 16:30:01,747 - DEBUG - process has ended
2016-09-27 16:30:01,748 - DEBUG - return code=0
2016-09-27 16:30:01,748 - DEBUG - stdout=
2016-09-27 16:30:01,751 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']" 
2016-09-27 16:30:01,751 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']
2016-09-27 16:30:01,751 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1724
2016-09-27 16:30:01,753 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']" 
2016-09-27 16:30:01,753 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']
2016-09-27 16:30:01,753 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1725
2016-09-27 16:30:01,755 - DEBUG - process was started (pid=23043)
2016-09-27 16:30:01,755 - DEBUG - DISPLAY=:1725
2016-09-27 16:30:01,855 - INFO - start
2016-09-27 16:30:31,965 - ERROR - 
Traceback (most recent call last):
  File "test.py", line 31, in &lt;module&gt;
    driver = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 103, in __init__
    self.binary, timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable
    % (self.profile.path))
WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.

2016-09-27 16:30:31,966 - INFO - finish
</code></pre>

<p>What I have tried:</p>

<ol>
<li>Ensure that the script file is owned by root</li>
<li>Use export DISPLAY=:0; or export DISPLAY=:99; in crontab command</li>
<li>set the HOME variable in the crontab to the path of the home directory of the user that the cronjob was being run as</li>
</ol>

<p>I'm really stuck with this problem.</p>

<p>I have python 2.7.10, selenium 2.53.6 with Xvbf and Firefox 47.0.1 on Debian 7.7</p>

<p>The problem concerns environment variables: cron is started by the system and knows nothing about user environments.</p>

<p>So, the solution to a problem is to make cron run a shell script that sets the needed environment variables first and then runs the script. In my case I needed to set the <code>PATH</code> varibable like this: <code>PATH=/root/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin</code>
Also, it may be useful to set <code>HOME</code> or <code>DISPLAY</code> varibles in some cases.</p>

python selenium debian crontab