Python/Numpy: problems with type conversion in vectorize and item
<p>I am writing a function to extract values from datetimes over arrays.  I want the function to operate on a Pandas DataFrame or a numpy ndarray.  </p>

<p>The values should be returned in the same way as the Python datetime properties, e.g. </p>

<pre><code>from datetime import datetime
dt = datetime(2016, 10, 12, 13)
dt.year
  =&gt; 2016
dt.second
  =&gt; 0
</code></pre>

<p>For a DataFrame this is reasonably easy to handle using <code>applymap()</code> (although there may well be a better way).  I tried the same approach for numpy ndarrays using <code>vectorize()</code>, and I'm running into problems.  Instead of the values I was expecting, I end up with very large integers, sometimes negative.</p>

<p>This was pretty baffling at first, but I figured out what is happening: the vectorized function is using <code>item</code> instead of <code>__get__</code> to get the values out of the ndarray.  This seems to automatically convert each <code>datetime64</code> object to a <code>long</code>:</p>

<pre><code>nd[1][0]
  =&gt; numpy.datetime64('1986-01-15T12:00:00.000000000')
nd[1].item()
  =&gt; 506174400000000000L
</code></pre>

<p>The long seems to be the number of nanoseconds since epoch (1970-01-01T00:00:00).  Somewhere along the line the values are converted to integers and they overflow, hence the negative numbers.</p>

<p>So that's the problem.  Please can someone help me fix it?  The only thing I can think of is doing the conversion manually, but this would effectively mean reimplementing a chunk of the <code>datetime</code> module.</p>

<p>Is there some alternative to <code>vectorize</code> that doesn't use <code>item()</code>?</p>

<p>Thanks!</p>

<p>Minimal code example:</p>

<pre><code>## DataFrame works fine
import pandas as pd
from datetime import datetime

df = pd.DataFrame({'dts': [datetime(1970, 1, 1, 1), datetime(1986, 1, 15, 12),
                         datetime(2016, 7, 15, 23)]})
exp = pd.DataFrame({'dts': [1, 15, 15]})

df_func = lambda x: x.day    
out = df.applymap(df_func)

assert out.equals(exp)

## numpy ndarray is more difficult
from numpy import datetime64 as dt64, timedelta64 as td64, vectorize  # for brevity

# The unary function is a little more complex, especially for days and months where the minimum value is 1
nd_func = lambda x: int((dt64(x, 'D') - dt64(x, 'M') + td64(1, 'D')) / td64(1, 'D'))

nd = df.as_matrix()
exp = exp.as_matrix()
  =&gt; array([[ 1],
            [15],
            [15]])

# The function works as expected on a single element...
assert nd_func(nd[1][0]) == 15

# ...but not on an ndarray
nd_vect = vectorize(nd_func)
out = nd_vect(nd)
  =&gt; array([[    -105972749999999],
            [ 3546551532709551616],
            [-6338201187830896640]])
</code></pre>

<p>In Py3 the error is <code>OverflowError: Python int too large to convert to C long</code>.</p>

<pre><code>In [215]: f=np.vectorize(nd_func,otypes=[int])
In [216]: f(dts)
... 
OverflowError: Python int too large to convert to C long
</code></pre>

<p>but if I change the datetime units, it runs ok</p>

<pre><code>In [217]: f(dts.astype('datetime64[ms]'))
Out[217]: array([ 1, 15, 15])
</code></pre>

<p>We could dig into this in more depth, but this seems to be simplest solution.</p>

<p>Keep in mind that <code>vectorize</code> is a convenience function; it makes iterating over multidimensions easier. But for a 1d array it is basically</p>

<pre><code>np.array([nd_func(i) for i in dts])
</code></pre>

<p>But note that we don't have to use iteration:</p>

<pre><code>In [227]: (dts.astype('datetime64[D]') - dts.astype('datetime64[M]') + td64(1,'D')) / td64(1,'D').astype(int)
Out[227]: array([ 1, 15, 15], dtype='timedelta64[D]')
</code></pre>

python datetime numpy