Applying an operation on multiple columns with a fixed column in pandas
<p>I have a dataframe as shown below. The last column shows the sum of values from all the columns i.e. <code>A</code>,<code>B</code>,<code>D</code>,<code>K</code> and <code>T</code>. Please note some of the columns have <code>NaN</code> as well.</p>

<pre><code>word1,A,B,D,K,T,sum
na,,63.0,,,870.0,933.0
sva,,1.0,,3.0,695.0,699.0
a,,102.0,,1.0,493.0,596.0
sa,2.0,487.0,,2.0,15.0,506.0
su,1.0,44.0,,136.0,214.0,395.0
waw,1.0,9.0,,34.0,296.0,340.0
</code></pre>

<p>How can I calculate the entropy for each row? i.e. I should find something like following</p>

<pre><code>df['A']/df['sum']*log(df['A']/df['sum']) + df['B']/df['sum']*log(df['B']/df['sum']) + ...... + df['T']/df['sum']*log(df['T']/df['sum'])
</code></pre>

<p>The condition is that whenever the value inside the <code>log</code> becomes <code>zero</code> or <code>NaN</code>, the whole value should be treated as zero (by definition, the log will return an error as log 0 is undefined).</p>

<p>I am aware of using lambda operation to apply on individual columns. Here I am not able to think for a pure pandas solution where a fixed column <code>sum</code> is applied on different columns <code>A</code>,<code>B</code>,<code>D</code> etc.. Though I can think of a simple loopwise iteration over CSV file with hard-coded column values.</p>

<pre><code>df1 = df.iloc[:, :-1]
df2 = df1.div(df1.sum(1), axis=0)
df2.mul(np.log(df2)).sum(1)

word1
na    -0.247187
sva   -0.038471
a     -0.469774
sa    -0.184881
su    -0.958774
waw   -0.464188
dtype: float64
</code></pre>

<h3>Setup</h3>

<pre><code>from StringIO import StringIO
import pandas as pd

text = """word1,A,B,D,K,T,sum
na,,63.0,,,870.0,933.0
sva,,1.0,,3.0,695.0,699.0
a,,102.0,,1.0,493.0,596.0
sa,2.0,487.0,,2.0,15.0,506.0
su,1.0,44.0,,136.0,214.0,395.0
waw,1.0,9.0,,34.0,296.0,340.0"""

df = pd.read_csv(StringIO(text), index_col=0)

df
</code></pre>

<p><a href="http://i.stack.imgur.com/I01af.png" rel="nofollow"><img src="http://i.stack.imgur.com/I01af.png" alt="enter image description here"></a></p>

python pandas dataframe sum multiple-columns