Python Pandas: Reuse stored means correctly to replace nan
<p>Over some data, I computed means columnwise.</p>

<p>Let's say the data looks like this</p>

<pre><code> A   B   C   ...  Z
0.1 0.2 0.15 ... 0.17
 .   .   .        .
 .   .   .        .
 .   .   .        .
</code></pre>

<p>I used the mean() function of DataFrame and as result I got</p>

<pre><code>A  some_mean_A
B  some_mean_B
...
Z  some_mean_Z
</code></pre>

<p>For replacing NaN, I use fillna(). It works for the case of computing the mean and using it during the same execution.</p>

<p>But as soon as I save the means in a file and read it to use it in a different .py file, I get rubbish. The reason is the file with the means are not interpreted correctly. In the new dataset, each NaN of the column A should be replaced by some_mean_A. Same for B and the rest till Z. But this is not happening, because by reading the means with read_csv(), I get the following</p>

<pre><code>0   1
A  some_mean_A
B  some_mean_B
...
Z  some_mean_Z
</code></pre>

<p>When I use this with fillna(), I do not get the expected result.</p>

<p>So, I hope you are understanding my problem. Do you know how to solve this problem?</p>

<p>EDIT 1.0:</p>

<p>This is how I compute and store the means:</p>

<pre><code>df_mean = df.mean()
df.fillna(df_mean, inplace=True)  // df is the dataframe for dataset where it works
df_mean.to_csv('mean.csv')
</code></pre>

<p>This is how I read the means:</p>

<pre><code>df_mean = pd.read_csv('mean.csv', header=None)
</code></pre>

<p><code>df.mean()</code> returns a Series. In that Series, values are the means of columns and the indices are the column names. It is a one-dimensional structure. However, if you read that file using <code>pd.read_csv</code>'s default parameters, it will read it as a DataFrame: one column for the column names, and another column for the means. To get the same data structure, you need to specify the index and pass <code>squeeze=True</code>. This way, pandas will read it into a Series:</p>

<pre><code>df_mean = pd.read_csv('mean.csv', header=None, index_col=0, squeeze=True)
</code></pre>

<p>would give you the same Series for the mean vector. You can add <code>rename_axis(None)</code> at the end to get rid of the index name (I think this requires pandas 0.18.0):</p>

<pre><code>df_mean = pd.read_csv('mean.csv', header=None, index_col=0).squeeze().rename_axis(None)
</code></pre>

python python-3.x pandas