Pig: is it possible to use pytz or dateutils for Python udfs?
<p>I am using <code>datetime</code> in some Python udfs that I use in my <code>pig</code> script. So far so good. I use pig 12.0 on Cloudera 5.5</p>

<p>However, I also need to use the <code>pytz</code> or <code>dateutil</code> packages as well and they dont seem to be part of a vanilla python install. </p>

<p>Can I use them in my <code>Pig</code> udfs in some ways? If so, how? I think <code>dateutil</code> is installed on my nodes (I am not admin, so how can I actually check that is the case?), but when I type:</p>

<pre><code>import sys
#I append the path to dateutil on my local windows machine. Is that correct?
sys.path.append('C:/Users/me/AppData/Local/Continuum/Anaconda2/lib/site-packages')

from dateutil import tz
</code></pre>

<p>in my <code>udfs.py</code> script, I get:</p>

<pre><code>2016-08-30 09:56:06,572 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1121: Python Error. Traceback (most recent call last):
  File "udfs.py", line 23, in &lt;module&gt;
    from dateutil import tz
ImportError: No module named dateutil
</code></pre>

<p>when I run my pig script.</p>

<p>All my other python udfs (using <code>datetime</code> for instance) work just fine. Any idea how to fix that?</p>

<p>Many thanks!</p>

<p><strong>UPDATE</strong></p>

<p>after playing a bit with the python path, I am now able to </p>

<pre><code>import dateutil 
</code></pre>

<p>(at least Pig does not crash). But if I try:</p>

<pre><code>from dateutil import tz
</code></pre>

<p>I get an error. </p>

<pre><code>  from dateutil import tz 
  File "/opt/python/lib/python2.7/site-packages/dateutil/tz.py", line 16, in &lt;module&gt;
    from six import string_types, PY3
  File "/opt/python/lib/python2.7/site-packages/six.py", line 604, in &lt;module&gt;
    viewkeys = operator.methodcaller("viewkeys")
AttributeError: type object 'org.python.modules.operator' has no attribute 'methodcaller'
</code></pre>

<p>How to overcome that? I use tz in the following manner</p>

<pre><code>to_zone = dateutil.tz.gettz('US/Eastern')
from_zone = dateutil.tz.gettz('UTC')
</code></pre>

<p>and then I change the timezone of my timestamps. Can I just import dateutil to do that? what is the proper syntax?</p>

<p><strong>UPDATE 2</strong></p>

<p>Following yakuza's suggestion, I am able to </p>

<pre><code>import sys
sys.path.append('/opt/python/lib/python2.7/site-packages')
sys.path.append('/opt/python/lib/python2.7/site-packages/pytz/zoneinfo')

import pytz
</code></pre>

<p>but now I get and error again </p>

<pre><code>Caused by: Traceback (most recent call last): File "udfs.py", line 158, in to_date_local File "__pyclasspath__/pytz/__init__.py", line 180, in timezone pytz.exceptions.UnknownTimeZoneError: 'America/New_York'
</code></pre>

<p>when I define</p>

<pre><code>to_zone = pytz.timezone('America/New_York')
from_zone = pytz.timezone('UTC')
</code></pre>

<p>Found some hints here <a href="http://stackoverflow.com/questions/9158846/unknowntimezoneerror-exception-raised-with-python-application-compiled-with-py2e">UnknownTimezoneError Exception Raised with Python Application Compiled with Py2Exe</a></p>

<p>What to do? Awww, I just want to convert timezones in Pig :(</p>

<p>Well, as you probably know all Python UDF functions are not executed by Python interpreter, but Jython that is distributed with Pig. By default in 0.12.0 it should be <a href="http://stackoverflow.com/questions/17711451/python-udf-version-with-jython-pig">Jython 2.5.3</a>. Unfortunately <code>six</code> package supports Python starting from <a href="https://pypi.python.org/pypi/six" rel="nofollow">Python 2.6</a> and it's package required by <a href="https://github.com/dateutil/dateutil/blob/master/setup.py" rel="nofollow"><code>dateutil</code></a>. However <code>pytz</code> seems not to have such dependency, and should support Python versions starting from <a href="https://pypi.python.org/pypi/pytz" rel="nofollow">Python 2.4</a>.</p>

<p>So to achieve your goal you should distribute <code>pytz</code> package to all your nodes for version 2.5 and in your Pig UDF add it's path to <code>sys.path</code>. If you complete same steps you did for <code>dateutil</code> everything should work as you expect. We are using very same approach with <code>pygeoip</code> and it works like a charm.</p>

<h1>How does it work</h1>

<p>When you run Pig script that references some Python UDF (more precisely Jython UDF), you script gets compiled to map/reduce job, all <code>REGISTER</code>ed files are included in JAR file, and are distributed on nodes where code is actually executed. Now when your code is executed, Jython interpreter is started and executed from Java code. So now when Python code is executed on each node taking part in computation, all Python imports are resolved locally on node. Imports from standard libraries are taken from Jython implementation, but all "packages" have to be install otherwise, as there is no <code>pip</code> for it. So to make external packages available to Python UDF you have to install required packages manually using other <code>pip</code> or install from sources, but remember to download package <strong>compatible with Python 2.5</strong>! Then in every single UDF file, you have to append <code>site-packages</code> on each node, where you installed packages (it's important to use same directory on each node). For example:</p>

<pre><code>import sys
sys.path.append('/path/to/site-packages')
# Imports of non-stdlib packages
</code></pre>

<h1>Proof of concept</h1>

<p>Let's assume some we have following files:</p>

<p><code>/opt/pytz_test/test_pytz.pig</code>:</p>

<pre><code>REGISTER '/opt/pytz_test/test_pytz_udf.py' using jython as test;

A = LOAD '/opt/pytz_test/test_pytz_data.csv' AS (timestamp:int);
B = FOREACH A GENERATE
    test.to_date_local(timestamp);

STORE B INTO '/tmp/test_pytz_output.csv' using PigStorage(',');
</code></pre>

<p><code>/opt/pytz_test/test_pytz_udf.py</code>:</p>

<pre><code>from datetime import datetime
import sys

sys.path.append('/usr/lib/python2.6/site-packages/')

import pytz

@outputSchema('date:chararray')
def to_date_local(unix_timestamp):
    """
    converts unix timestamp to a rounded date
    """
    to_zone = pytz.timezone('America/New_York')
    from_zone = pytz.timezone('UTC')

    try :
        as_datetime = datetime.utcfromtimestamp(unix_timestamp)
            .replace(tzinfo=from_zone).astimezone(to_zone)
            .date().strftime('%Y-%m-%d')
    except:
        as_datetime = unix_timestamp
    return as_datetime
</code></pre>

<p><code>/opt/pytz_test/test_pytz_data.csv</code>:</p>

<pre><code>1294778181
1294778182
1294778183
1294778184
</code></pre>

<p>Now let's install <code>pytz</code> on our node (it has to be installed using Python version on which <code>pytz</code> is compatible with Python 2.5 (2.5-2.7), in my case I'll use Python 2.6):</p>

<p><code>sudo pip2.6 install pytz</code></p>

<p><strong>Please make sure, that file</strong> <code>/opt/pytz_test/test_pytz_udf.py</code> <strong>adds to <code>sys.path</code> reference to <code>site-packages</code> where <code>pytz</code> is installed.</strong></p>

<p>Now once we run Pig with our test script:</p>

<p><code>pig -x local /opt/pytz_test/test_pytz.pig</code></p>

<p>We should be able to read output from our job, which should list:</p>

<pre><code>2011-01-11
2011-01-11
2011-01-11
2011-01-11
</code></pre>

python apache-pig jython cloudera pytz