Pig: is it possible to use pytz or dateutils for Python udfs?
<p>I am using <code>datetime</code> in some Python udfs that I use in my <code>pig</code> script. So far so good. I use pig 12.0 on Cloudera 5.5</p>

<p>However, I also need to use the <code>pytz</code> or <code>dateutil</code> packages as well and they dont seem to be part of a vanilla python install. </p>

<p>Can I use them in my <code>Pig</code> udfs in some ways? If so, how? I think <code>dateutil</code> is installed on my nodes (I am not admin, so how can I actually check that is the case?), but when I type:</p>

<pre><code>import sys
#I append the path to dateutil on my local windows machine. Is that correct?
sys.path.append('C:/Users/me/AppData/Local/Continuum/Anaconda2/lib/site-packages')

from dateutil import tz
</code></pre>

<p>in my <code>udfs.py</code> script, I get:</p>

<pre><code>2016-08-30 09:56:06,572 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1121: Python Error. Traceback (most recent call last):
  File "udfs.py", line 23, in &lt;module&gt;
    from dateutil import tz
ImportError: No module named dateutil
</code></pre>

<p>when I run my pig script.</p>

<p>All my other python udfs (using <code>datetime</code> for instance) work just fine. Any idea how to fix that?</p>

<p>Many thanks!</p>

<p><strong>UPDATE</strong></p>

<p>after playing a bit with the python path, I am now able to </p>

<pre><code>import dateutil 
</code></pre>

<p>(at least Pig does not crash). But if I try:</p>

<pre><code>from dateutil import tz
</code></pre>

<p>I get an error. </p>

<pre><code>  from dateutil import tz 
  File "/opt/python/lib/python2.7/site-packages/dateutil/tz.py", line 16, in &lt;module&gt;
    from six import string_types, PY3
  File "/opt/python/lib/python2.7/site-packages/six.py", line 604, in &lt;module&gt;
    viewkeys = operator.methodcaller("viewkeys")
AttributeError: type object 'org.python.modules.operator' has no attribute 'methodcaller'
</code></pre>

<p>How to overcome that? I use tz in the following manner</p>

<pre><code>to_zone = dateutil.tz.gettz('US/Eastern')
from_zone = dateutil.tz.gettz('UTC')
</code></pre>

<p>and then I change the timezone of my timestamps. Can I just import dateutil to do that? what is the proper syntax?</p>

<p><strong>UPDATE 2</strong></p>

<p>Following yakuza's suggestion, I am able to </p>

<pre><code>import sys
sys.path.append('/opt/python/lib/python2.7/site-packages')
sys.path.append('/opt/python/lib/python2.7/site-packages/pytz/zoneinfo')

import pytz
</code></pre>

<p>but now I get and error again </p>

<pre><code>Caused by: Traceback (most recent call last): File "udfs.py", line 158, in to_date_local File "__pyclasspath__/pytz/__init__.py", line 180, in timezone pytz.exceptions.UnknownTimeZoneError: 'America/New_York'
</code></pre>

<p>when I define</p>

<pre><code>to_zone = pytz.timezone('America/New_York')
from_zone = pytz.timezone('UTC')
</code></pre>

<p>Found some hints here <a href="http://stackoverflow.com/questions/9158846/unknowntimezoneerror-exception-raised-with-python-application-compiled-with-py2e">UnknownTimezoneError Exception Raised with Python Application Compiled with Py2Exe</a></p>

<p>What to do? Awww, I just want to convert timezones in Pig :(</p>

<p>From the answer to <a href="http://stackoverflow.com/questions/7831649/how-do-i-make-hadoop-find-imported-python-modules-when-using-python-udfs-in-pig">a different but related question</a>, it seems that you should be able to use resources as long as they are available on each of the nodes.</p>

<p>I think you can then add the path as described in <a href="http://stackoverflow.com/a/23807481/983722">this answer regarding jython</a>, and load the modules as usual.</p>

<blockquote>
  <p>Append the location to the sys.path in the Python script: </p>

<pre><code>import sys
sys.path.append('/usr/local/lib/python2.7/dist-packages')
import happybase
</code></pre>
</blockquote>

python apache-pig jython cloudera pytz