Storing pure python datetime.datetime in pandas DataFrame
<p>Since <code>matplotlib</code> doesn't support <a href="https://github.com/pydata/pandas/issues/8113" rel="nofollow">either</a><code>pandas.TimeStamp</code> <a href="http://stackoverflow.com/questions/22048792/how-do-i-display-dates-when-plotting-in-matplotlib-pyplot">or</a><code>numpy.datetime64</code>, and  there are <a href="http://stackoverflow.com/questions/27472548/pandas-scatter-plotting-datetime">no simple workarounds</a>, I decided to convert a native pandas date column into a pure python <code>datetime.datetime</code> so that scatter plots are easier to make.</p>

<p>However:</p>

<pre><code>t = pd.DataFrame({'date': [pd.to_datetime('2012-12-31')]})
t.dtypes # date    datetime64[ns], as expected
pure_python_datetime_array = t.date.dt.to_pydatetime() # works fine
t['date'] = pure_python_datetime_array # doesn't do what I hoped
t.dtypes # date    datetime64[ns] as before, no luck changing it
</code></pre>

<p>I'm guessing pandas auto-converts the pure python <code>datetime</code> produced by <code>to_pydatetime</code> into its native format. I guess it's convenient behavior in general, but is there a way to override it?</p>

<p>For me, the steps look like this:</p>

<ol>
<li>convert timezone with pytz </li>
<li>convert to_datetime with pandas and make that the index</li>
<li>plot and autoformat</li>
</ol>

<p>Starting df looks like this:</p>

<p><a href="http://i.stack.imgur.com/L2WPm.png" rel="nofollow"><img src="http://i.stack.imgur.com/L2WPm.png" alt="before converting timestamps"></a></p>

<ol>
<li><code>import pytz</code>
   <code>ts['posTime']=[x.astimezone(
                   pytz.timezone('US/Pacific')) for x in ts['posTime']]</code></li>
</ol>

<p>I can see that it worked because the timestamps changed format:</p>

<p><a href="http://i.stack.imgur.com/KHk1U.png" rel="nofollow"><img src="http://i.stack.imgur.com/KHk1U.png" alt="after timezone conversion"></a></p>

<ol start="2">
<li><p><code>sample['posTime'] = pandas.to_datetime(sample['posTime'])</code></p>

<p><code>sample.index = sample['posTime']</code></p></li>
</ol>

<p>At this point, just plotting with pandas (which uses matplotlib under the hood) gives me a nice rotation and totally the wrong format:</p>

<p><a href="http://i.stack.imgur.com/0bPCU.png" rel="nofollow"><img src="http://i.stack.imgur.com/0bPCU.png" alt="after pandas datetime conversion"></a></p>

<ol start="3">
<li><p>However, there's nothing wrong with the format of the objects. I can now make a scatterplot with matplotlib and it autoformats the datetimes as you'd expect. </p>

<p><code>plt.scatter(sample['posTime'].values, sample['Altitude'].values)</code></p>

<p><code>fig = plt.gcf()</code></p>

<p><code>fig.set_size_inches(9.5, 3.5)</code></p></li>
</ol>

<p><a href="http://i.stack.imgur.com/Tqy0w.png" rel="nofollow"><img src="http://i.stack.imgur.com/Tqy0w.png" alt="formatted"></a> </p>

<ol start="4">
<li>If you use the auto format method, you can zoom in and it will continue to automatically choose the appropriate format (but you still have to set the scale manually). </li>
</ol>

<p><a href="http://i.stack.imgur.com/n5vP7.png" rel="nofollow"><img src="http://i.stack.imgur.com/n5vP7.png" alt="autoformatted"></a></p>

python python-3.x datetime pandas