pandas dataframe fillna() not working?
<p>I have a data set in which I am performing a principal components analysis (PCA). I get a <code>ValueError</code> message when I try to transform the data. Below is some of the code:</p>

<pre><code>import pandas as pd
import numpy as np
import matplotlib as mpl
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA as sklearnPCA

data = pd.read_csv('test.csv',header=0)
X = data.ix[:,0:1000].values   # values of 1000 predictor variables
Y = data.ix[:,1000].values     # values of binary outcome variable
sklearn_pca = sklearnPCA(n_components=2)
X_std = StandardScaler().fit_transform(X)
</code></pre>

<p>It is here that I get the following error message:</p>

<pre class="lang-none prettyprint-override"><code>ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
</code></pre>

<p>So I then checked whether the original data set had any NaN values:</p>

<pre><code>print(data.isnull().values.any())   # prints True
data.fillna(0)                      # replace NaN values with 0
print(data.isnull().values.any())   # prints True
</code></pre>

<p>I don't understand why <code>data.isnull().values.any()</code> is still printing <code>True</code> even after I replaced the NaN values with 0.</p>

<p>You have to replace data by the returned object from <code>fillna</code></p>

<p>Small reproducer:</p>

<pre><code>import pandas as pd

data = pd.DataFrame(data=[0,float('nan'),2,3])

print(data.isnull().values.any())   # prints True
data = data.fillna(0)                      # replace NaN values with 0
print(data.isnull().values.any())   # prints False now :)
</code></pre>

python pandas dataframe missing-data