python dataframe std deviation of a column/row containing all Nan values except one element
<p>I have a python dataframe like this  </p>

<pre><code>In [212]: d3  
Out[212]: 
     a    b    c    d    e  
a  0.0  0.0  0.0  0.0  0.0  
b  1.0  1.0  1.0  1.0  1.0  
c  2.0  2.0  2.0  2.0  2.0  
d  3.0  3.0  3.0  3.0  3.0  
e  3.0  4.0  4.0  4.0  4.0  
f  5.0  5.0  5.0  5.0  0.0  
p  NaN  3.0  NaN  3.0  NaN  
q  NaN  4.0  4.0  4.0  NaN  
r  NaN  5.0  NaN  NaN  NaN  
</code></pre>

<p>And when I find standard deviation using <code>d3.std(axis=1)</code>, I get SD of row 'r' is <strong>NaN</strong> and I was expecting <strong>0</strong> for the same.</p>

<pre><code>In [213]: d3.std(axis=1)
Out[213]:
a    0.000000
b    0.000000
c    0.000000
d    0.000000
e    0.447214
f    2.236068
p    0.000000
q    0.000000
r         NaN
dtype: float64
</code></pre>

<p>When I try the same using numpy's <code>nanstd()</code> function this gives me desired result.</p>

<pre><code>In [225]: np.nanstd(d3.loc['r',:])
Out[225]: 0.0
</code></pre>

<p>But I have a dataframe and I want to use dataframe's std function like this <code>d3['new_col']=np.where(d3.std(axis=1)==0, d3.min(axis=1), np.nan)</code><br>
this results into  </p>

<pre><code>In [226]: d3['new_col'] = np.where(d3.std(axis=1)==0, d3.min(axis=1), np.nan)

In [227]: d3
Out[227]:
     a    b    c    d    e  new_col
a  0.0  0.0  0.0  0.0  0.0      0.0
b  1.0  1.0  1.0  1.0  1.0      1.0
c  2.0  2.0  2.0  2.0  2.0      2.0
d  3.0  3.0  3.0  3.0  3.0      3.0
e  3.0  4.0  4.0  4.0  4.0      NaN
f  5.0  5.0  5.0  5.0  0.0      NaN
p  NaN  3.0  NaN  3.0  NaN      3.0
q  NaN  4.0  4.0  4.0  NaN      4.0
r  NaN  5.0  NaN  NaN  NaN      NaN
</code></pre>

<p>I am expecting rightmost bottom most value to be <strong>5.0</strong> not <strong>NaN</strong>.</p>

<p>Use arg <code>ddof=0</code> to make <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.std.html" rel="nofollow"><code>df.std</code></a> behave like <code>np.std</code> as it always leaves the last value while calculating standard deviation by default (<em>ddof=1</em>):</p>

<pre><code>df['new_col'] = np.where(df.std(axis=1, ddof=0)==0, df.min(axis=1), np.nan)
</code></pre>

<p><a href="http://i.stack.imgur.com/tY8jr.png" rel="nofollow"><img src="http://i.stack.imgur.com/tY8jr.png" alt="enter image description here"></a></p>

python pandas dataframe standard-deviation