Can I get a trimmed mean of all columns in a dataframe with nan values?
<p>I seem to be going around in circles trying to solve this problem. I'd be very grateful for any help from the stackoverflow brains trust.</p>

<p>The problem is that I want to get the trimmed mean of all the columns in a pandas dataframe (i.e. the mean of the values in a given column, excluding the max and the min values). It's likely that some columns will have nan values. Basically, I want to get the exact same functionality as the pandas.DataFrame.mean function, except that it's the trimmed mean.</p>

<p>The obvious solution is to use the scipy tmean function, and iterate over the df columns. So I did: </p>

<pre><code>import scipy as sp

trim_mean = []

for i in data_clean3.columns:
    trim_mean.append(sp.tmean(data_clean3[i]))
</code></pre>

<p>This worked great, until I encountered nan values, which caused tmean to choke. Worse, when I dropped the nan values in the dataframe, there were some datasets that were wiped out completely as they had an nan value in every column. This means that when I amalgamate all my datasets into a master set, there'll be holes on the master set where the trimmed mean should be.</p>

<p>Does anyone know of a way around this? As in, is there a way to get tmean to behave like the standard scipy stats functions and ignore nan values? </p>

<p>(Note that my code is calculating a big number of descriptive statistics on large datasets with limited hardware; highly involved or inefficient workarounds might not be optimal. Hopefully, though, I'm just missing something simple.)</p>

<p>(<strong>EDIT</strong>: Someone suggested in a comment (that has since vanished?) that I should used the trim_mean scipy function, which allows you to top and tail a specific proportion of the data. This is just to say that this solution won't work for me, as my datasets are of unequal sizes, so I cannot specify a fixed proportion of data that will be OK to remove in every case; it must always just be the max and the min values.) </p>

<p>consider <code>df</code></p>

<pre><code>np.random.seed()
data = np.random.choice((0, 25, 35, 100, np.nan),
                        (1000, 2),
                        p=(.01, .39, .39, .01, .2))
df = pd.DataFrame(data, columns=list('AB'))
</code></pre>

<p>Construct your mean using sums and divide by relevant normalizer.</p>

<pre><code>(df.sum() - df.min() - df.max()) / (df.notnull().sum() - 2)

A    29.707674
B    30.402228
dtype: float64
</code></pre>

<hr>

<pre><code>df.mean()

A    29.756987
B    30.450617
dtype: float64
</code></pre>

<hr>

python pandas scipy code-statistics