dask dataframe how to convert column to to_datetime
<p>I am trying to convert one column of my dataframe to datetime. Following the discussion here <a href="https://github.com/dask/dask/issues/863" rel="nofollow">https://github.com/dask/dask/issues/863</a> I tried the following code:</p>

<pre><code>import dask.dataframe as dd
df['time'].map_partitions(pd.to_datetime, columns='time').compute()
</code></pre>

<p>But I am getting the following error message</p>

<pre><code>ValueError: Metadata inference failed, please provide `meta` keyword
</code></pre>

<p>What exactly should I put under meta? should I put a dictionary of ALL the columns in df or only of the 'time' column? and what type should I put? I have tried dtype and datetime64 but none of them work so far.</p>

<p>Thank you and I appreciate your guidance,</p>

<p><strong>Update</strong></p>

<p>I will include here the new error messages:</p>

<p>1) Using Timestamp</p>

<pre><code>df['trd_exctn_dt'].map_partitions(pd.Timestamp).compute()

TypeError: Cannot convert input to Timestamp
</code></pre>

<p>2) Using datetime and meta</p>

<pre><code>meta = ('time', pd.Timestamp)
df['time'].map_partitions(pd.to_datetime,meta=meta).compute()
TypeError: to_datetime() got an unexpected keyword argument 'meta'
</code></pre>

<p>3) Just using date time: gets stuck at 2%</p>

<pre><code>    In [14]: df['trd_exctn_dt'].map_partitions(pd.to_datetime).compute()
[                                        ] | 2% Completed |  2min 20.3s
</code></pre>

<p>Also, I would like to be able to specify the format in the date, as i would do in pandas:</p>

<pre><code>pd.to_datetime(df['time'], format = '%m%d%Y'
</code></pre>

<p><strong>Update 2</strong></p>

<p>After updating to Dask 0.11, I no longer have problems with the meta keyword. Still, I can't get it past 2% on a 2GB dataframe.</p>

<pre><code>df['trd_exctn_dt'].map_partitions(pd.to_datetime, meta=meta).compute()
    [                                        ] | 2% Completed |  30min 45.7s
</code></pre>

<p><strong>Update 3</strong></p>

<p>worked better this way:</p>

<pre><code>def parse_dates(df):
  return pd.to_datetime(df['time'], format = '%m/%d/%Y')

df.map_partitions(parse_dates, meta=meta)
</code></pre>

<p>I'm not sure whether it's the right approach or not</p>

<h3>Use <code>astype</code></h3>

<p>You can use the <code>astype</code> method to convert the dtype of a series to a NumPy dtype</p>

<pre><code>df.time.astype('M8[us]')
</code></pre>

<p>There is probably a way to specify a Pandas style dtype as well (edits welcome)</p>

<h3>Use map_partitions and meta</h3>

<p>When using black-box methods like <code>map_partitions</code>, dask.dataframe needs to know the type and names of the output.  There are a few ways to do this listed in the docstring for <code>map_partitions</code>.</p>

<p>You can supply an empty Pandas object with the right dtype and name</p>

<pre><code>meta = pd.Series([], name='time', dtype=pd.Timestamp)
</code></pre>

<p>Or you can provide a tuple of <code>(name, dtype)</code> for a Series or a dict for a DataFrame</p>

<pre><code>meta = ('time', pd.Timestamp)
</code></pre>

<p>Then everything should be fine</p>

<pre><code>df.time.map_partitions(pd.Timestamp, meta=meta)
</code></pre>

<p>If you were calling <code>map_partitions</code> on <code>df</code> instead then you would need to provide the dtypes for everything.  That isn't the case in your example though.</p>

python pandas dask