Debug the "NaN" error of sklearn using pandas dataframe input
<p>I read my input as pandas dataframe and filled NaN by:</p>

<pre><code>df = df.fillna(0)
</code></pre>

<p>After that I split into train and test set and do a classification using sklearn.</p>

<pre><code>features = df.drop('class',axis=1)
labels = df['class']
features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.3, random_state=42)
clf.fit(features_train, labels_train)   
</code></pre>

<p>But I still got an "NaN error": ValueError: Input contains NaN, infinity or a value too large for dtype('float32'). It seems that fillna() didn't find the missing data. How can I find where the "NaN" is?</p>

<p>You ask </p>

<blockquote>
  <p>How can I find where the "NaN" is</p>
</blockquote>

<p>Would it be helpful to visualize where the problematic data lie in the frame?</p>

<p>You could try <code>matplotlib.pyplot.spy</code></p>

<pre><code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# lets make some initial clean data
df = pd.DataFrame(
    data={
        'alpha': [0, 1, 2],
        'beta': [3, 4, 5],
        'gamma': [6, 7, 8]
    },
    index=['one', 'two', 'three']
)
# add some problematic points
# `NaN`s, infinities and stuff that is 
#  just not numeric
df.loc['one', 'beta'] = 'not a number but not NaN'
df.loc['two', 'alpha'] = np.NaN
df.loc['three', 'gamma'] = np.infty

fig, axes = plt.subplots(1, 3)
axes[0].spy(df.isnull())
axes[0].set_title('NaN elements')
axes[1].spy(df == np.infty)
axes[1].set_title('infinite elements')
axes[2].spy(~df.applymap(np.isreal))
axes[2].set_title('Non numeric elements')
</code></pre>

python pandas scikit-learn