Can I get a trimmed mean of all columns in a dataframe with nan values?
<p>I seem to be going around in circles trying to solve this problem. I'd be very grateful for any help from the stackoverflow brains trust.</p>

<p>The problem is that I want to get the trimmed mean of all the columns in a pandas dataframe (i.e. the mean of the values in a given column, excluding the max and the min values). It's likely that some columns will have nan values. Basically, I want to get the exact same functionality as the pandas.DataFrame.mean function, except that it's the trimmed mean.</p>

<p>The obvious solution is to use the scipy tmean function, and iterate over the df columns. So I did: </p>

<pre><code>import scipy as sp

trim_mean = []

for i in data_clean3.columns:
    trim_mean.append(sp.tmean(data_clean3[i]))
</code></pre>

<p>This worked great, until I encountered nan values, which caused tmean to choke. Worse, when I dropped the nan values in the dataframe, there were some datasets that were wiped out completely as they had an nan value in every column. This means that when I amalgamate all my datasets into a master set, there'll be holes on the master set where the trimmed mean should be.</p>

<p>Does anyone know of a way around this? As in, is there a way to get tmean to behave like the standard scipy stats functions and ignore nan values? </p>

<p>(Note that my code is calculating a big number of descriptive statistics on large datasets with limited hardware; highly involved or inefficient workarounds might not be optimal. Hopefully, though, I'm just missing something simple.)</p>

<p>(<strong>EDIT</strong>: Someone suggested in a comment (that has since vanished?) that I should used the trim_mean scipy function, which allows you to top and tail a specific proportion of the data. This is just to say that this solution won't work for me, as my datasets are of unequal sizes, so I cannot specify a fixed proportion of data that will be OK to remove in every case; it must always just be the max and the min values.) </p>

<p>you colud use df.mean(skipna =True) <a href="http://pandas.pydata.org/pandas-docs/version/0.17.1/generated/pandas.DataFrame.mean.html" rel="nofollow">DataFrame.mean</a></p>

<pre><code>df1 = pd.DataFrame([[5, 1, 'a'], [6, 2, 'b'],[7, 3, 'd'],[np.nan, 4, 'e'],[9, 5, 'f'],[5, 1, 'g']],  columns =  ["A", "B", "C"])
print df1
df1 = df1[df1.A != df1.A.max()] # Remove max values
df1 = df1[df1.A != df1.A.min()] # Remove min values

print "\nDatafrmae after removing max and min\n"
print df1
print "\nMean of A\n"

print df1["A"].mean(skipna =True)
</code></pre>

<p>output </p>

<pre><code>         A  B  C
0  5.0  1  a
1  6.0  2  b
2  7.0  3  d
3  NaN  4  e
4  9.0  5  f
5  5.0  1  g

Datafrmae after removing max and min

     A  B  C
1  6.0  2  b
2  7.0  3  d
3  NaN  4  e

Mean of A

6.5
</code></pre>

python pandas scipy code-statistics