Pandas DataFrame, smart apply of a complex function to groupby result
<p>I have a <code>pandas.DataFrame</code> with 3 columns of type <code>str</code> and <code>n</code> other columns of type <code>float64</code>.</p>

<p>I need to group rows by one of the three <code>str</code> columns and apply a function <code>myComplexFunc()</code> which will reduce `ÌN  rows to one row.</p>

<p><code>myComplexFunc()</code> take only rows of type <code>float64</code>.</p>

<p>This can be done with some for loops but it will not be efficient, So I tried to use the <a href="http://pandas.pydata.org/pandas-docs/stable/groupby.html#flexible-apply" rel="nofollow">flexible apply</a> of <code>pandas</code> but it seems that it runs the heavy code of <code>myComplexFunc()</code> twice!</p>

<p>To be more clear, here is a minimal example</p>

<p><strong>Let "df" be a dataFrame like this</strong> :
</p>

<pre><code>df
&gt;&gt;
     A      B         C         D
0  foo    one  0.406157  0.735223
1  bar    one  1.020493 -1.167256
2  foo    two -0.314192 -0.883087
3  bar  three  0.271705 -0.215049
4  foo    two  0.535290  0.185872
5  bar    two  0.178926 -0.459890
6  foo    one -1.939673 -0.523396
7  foo  three -2.125591 -0.689809
</code></pre>

<p><strong>myComplexFunc()</strong></p>

<pre class="lang-py prettyprint-override"><code>def myComplexFunc(rows):
  # Some transformations that will return 1 row
  result = some_transformations(rows)
  return result
</code></pre>

<p><strong>What I want :</strong></p>

<pre class="lang-py prettyprint-override"><code># wanted apply is the name of the wanted method
df.groupby("A").wanted_apply(myComplexFunc)

&gt;&gt; 
    A    C            D
0  foo   new_c0_foo   new_d0_foo
1  bar   new_c0_bar   new_d0_bar
</code></pre>

<p>The column <code>B</code> have been removed because it's not of type <code>float64</code>. </p>

<p>Thanks in advance</p>

<p>You can filter DataFrame by <code>dtype</code> by <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.select_dtypes.html" rel="nofollow"><code>select_dtypes</code></a>, but then need aggreagate by <code>Series</code> <code>df.A</code>:</p>

<pre><code>def myComplexFunc(rows):
    return rows  + 10

df = df.select_dtypes(include=[np.float64]).groupby([df.A]).apply(myComplexFunc)
print (df)
           C          D
0  10.406157  10.735223
1  11.020493   8.832744
2   9.685808   9.116913
3  10.271705   9.784951
4  10.535290  10.185872
5  10.178926   9.540110
6   8.060327   9.476604
7   7.874409   9.310191
</code></pre>

<p>because if use only <code>A</code>:</p>

<pre><code>df = df.select_dtypes(include=[np.float64]).groupby('A').apply(myComplexFunc)
</code></pre>

<p>get</p>

<blockquote>
  <p>KeyError: 'A'</p>
</blockquote>

<p>and it is right - all string columns are excluded (<code>A</code> and <code>B</code>).</p>

<pre><code>print (df.select_dtypes(include=[np.float64]))
          C         D
0  0.406157  0.735223
1  1.020493 -1.167256
2 -0.314192 -0.883087
3  0.271705 -0.215049
4  0.535290  0.185872
5  0.178926 -0.459890
6 -1.939673 -0.523396
7 -2.125591 -0.689809
</code></pre>

python string pandas dataframe group-by