Python Pandas Debugging on to_datetime
<p>Millions of records of data is in my dataframe.  I have to convert of the string columns to datetime.  I'm doing it as follows:</p>

<pre><code>allData['Col1'] = pd.to_datetime(allData['Col1'])
</code></pre>

<p>However some of the strings are not valid datetime strings, and thus I get a value error.  I'm not very good at debugging in Python, so I'm struggling to find the reason why some of the data items are not convertible.  </p>

<p>I need Python to show me the row number, as well as the value that is not convertible, instead of throwing out a useless error that tells me nothing.  How can I achieve this?</p>

<p>You can use <a href="http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing" rel="nofollow"><code>boolean indexing</code></a> with condition where check <code>NaT</code> values by <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.isnull.html" rel="nofollow"><code>isnull</code></a> created <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.to_datetime.html" rel="nofollow"><code>to_datetime</code></a> with parameter <code>errors='coerce'</code> - it create <code>NaT</code> where are invalid datetime:</p>

<pre><code>allData1 = allData[pd.to_datetime(allData['Col1'], errors='coerce').isnull()]
</code></pre>

<p>Sample:</p>

<pre><code>allData = pd.DataFrame({'Col1':['2015-01-03','a','2016-05-08'],
                        'B':[4,5,6],
                        'C':[7,8,9],
                        'D':[1,3,5],
                        'E':[5,3,6],
                        'F':[7,4,3]})

print (allData)
   B  C        Col1  D  E  F
0  4  7  2015-01-03  1  5  7
1  5  8           a  3  3  4
2  6  9  2016-05-08  5  6  3

print (pd.to_datetime(allData['Col1'], errors='coerce'))
0   2015-01-03
1          NaT
2   2016-05-08
Name: Col1, dtype: datetime64[ns]

print (pd.to_datetime(allData['Col1'], errors='coerce').isnull())
0    False
1     True
2    False
Name: Col1, dtype: bool


allData1 = allData[pd.to_datetime(allData['Col1'], errors='coerce').isnull()]
print (allData1)
   B  C Col1  D  E  F
1  5  8    a  3  3  4
</code></pre>

python string datetime debugging pandas