Applying a custom function on a Pandas series using groupby and pd.isnull
<p>I have a sample dataframe which generically looks like this:</p>

<pre><code>df = pd.Dataframe({'Class': [1, 2, 3, 2, 1, 2, 3, 2],
                   'Sex': [1, 0, 0, 0, 1, 1, 0, 1],
                   'Age': [15, 24, 13, 28, 29, NaN, 34, 27]})
</code></pre>

<p>Which displays as:</p>

<pre><code>    Age  Class  Sex
0  15.0      1    1
1  24.0      2    0
2  13.0      2    0
3  28.0      2    0
4  29.0      1    1
5   NaN      2    1
6  34.0      1    0
7  27.0      2    1
</code></pre>

<p>What I'd like  to do is fill in each of the NaN values in the 'Age' series with the median value for all entries that have their 'Class' and 'Sex' grouping.  </p>

<p>So for example, when I access these values like so:</p>

<pre><code>df.groupby(['Class', 'Sex'])['Age'].median()
</code></pre>

<p>and get:</p>

<pre><code>  Class  Sex
   1      0      34.0
          1      22.0
   2      0      24.0
          1      27.0
</code></pre>

<p>I'd like to write a function that automatically fills the extant NaN value with 27 since that is the median of the entries that have a Class value of 2 and a Sex value of 1.</p>

<p>Right now I have:</p>

<pre><code>df['Age'] = df.groupby(['Class', 'Sex'])['Age'].apply(lambda x: x.median() if pd.isnull(x) else x)
</code></pre>

<p>and am getting the following error:</p>

<pre><code>ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
</code></pre>

<p>However, a very similar syntax was used in the answers for <a href="http://stackoverflow.com/questions/14714181/conditional-logic-on-pandas-dataframe">this question</a> and <a href="http://stackoverflow.com/questions/34861086/replacing-null-values-in-a-pandas-dataframe-using-applymap">this one</a>, so I'm not quite sure why mine doesn't work, particularly, the latter also uses the isnull method in its lambda function so it's not clear to me why mine doesn't work but that one does.</p>

<p>I've also tried using the fillna method like so:</p>

<pre><code>df['Age'] = df['Age'].fillna(df.groupby(['Class', 'Sex'])['Age'].median())
</code></pre>

<p>But got the following error message:</p>

<pre><code>ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long long'
</code></pre>

<p>I'm open to other methods which get the same value, but prefer something that relies entirely on Pandas methods without having to use a separate for-loop and passing it into the 'Apply' method to make it as concise as possible.</p>

<p>Thank you.</p>

<p>One option would be to use <code>transform</code> to replace null values with median for the <code>Age</code> column:</p>

<pre><code>df['Age'] = df.groupby(['Class', 'Sex']).Age.transform(lambda col: col.where(col.notnull(), col.median()))

df

#   Age Class   Sex
#0  15.0    1   1
#1  24.0    2   0
#2  13.0    3   0
#3  28.0    2   0
#4  29.0    1   1
#5  27.0    2   1
#6  34.0    3   0
#7  27.0    2   1
</code></pre>

<p>Alternatively use <code>replace</code> method instead of <code>where</code> also works:</p>

<pre><code>df['Age'] = df.groupby(['Class', 'Sex']).Age.transform(lambda col: col.replace(np.nan, col.median()))
</code></pre>

python pandas