Python - In-line boolean evaluation without IF statements
<p>I am trying to assess the value of a column of a dataframe to determine the value of another column.  I did this by using an <code>if</code> statement and <code>.apply()</code> function successfully. I.e. </p>

<pre><code>if Col x &lt; 0.3:
    return y
elif Col x &gt; 0.6:
    return z
</code></pre>

<p>Etc.  The problem is this takes quite a while to run with a lot of data.  Instead I am trying to use the following logic to determine the new column value:</p>

<p>(x&lt;0.3)*y + (x>0.6)*z</p>

<p>So Python evaluates TRUE/FALSE and applies the correct value.  This seems to work much faster, the only thing is Python says: 
"UserWarning: evaluating in Python space because the '*' operator is not supported by numexpr for the bool dtype, use '&amp;' instead
  unsupported[op_str]))"</p>

<p>Is this a problem?  Should I be using "&amp;"?  I feel using "&amp;" would be incorrect when multiplying.</p>

<p>Thank you!</p>

<p>From what I have read so far, the performance gap is issued by the <em>parser</em> backend chosen by <code>pandas</code>. There's the regular python parser as a backand and, additionally, a pandas parsing backend.<br>
The docs say, that there is no performance gain if using plain old python over pandas here: <a href="http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html#pandas-eval-backends" rel="nofollow">Pandas eval Backends</a></p>

<p>However, you obviously hit a white spot in the pandas backend; i.e. you formed an expression that cannot be evaluated using pandas. The result is that pandas falls back to the original python parsing backend, as stated in the resulting UserWarning:</p>

<blockquote>
  <p>UserWarning: evaluating in Python space because the '*' operator is not supported by numexpr for the bool dtype, use '&amp;' instead
    unsupported[op_str]))</p>
</blockquote>

<p><a href="http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html#technical-minutia-regarding-expression-evaluation" rel="nofollow">(More on this topic)</a></p>

<h2>Timing evaluations</h2>

<p>So, as we now know about different parsing backends, it's time to check a few options provided by <code>pandas</code> that are suitable for your desired dataframe operation (complete script below):</p>

<pre><code>expr_a = '''(a &lt; 0.3) * 1 + (a &gt; 0.6) * 3 + (a &gt;= 0.3) * (a &lt;= 0.6) * 2'''
</code></pre>

<ol>
<li>Evaluate the expression as a string using the <code>pandas</code> backend</li>
<li>Evaluate the <strong>same</strong> string using the <code>python</code> backend</li>
<li>Evaluate the expression string with external variable reference using <code>pandas</code></li>
<li>Solve the problem using <code>df.apply()</code></li>
<li>Solve the problem using <code>df.applymap()</code></li>
<li>Direct submission of the expression (no string evaluation)</li>
</ol>

<p>The results on my machine for a dataframe with 10,000,000 random float values in one column are:</p>

<pre><code>(1) Eval (pd)    0.240498406269
(2) Eval (py)    0.197919774926
(3) Eval @ (pd)  0.200814546686
(4) Apply        3.242620778595
(5) ApplyMap     6.542354086152
(6) Direct       0.140075372736
</code></pre>

<p>The major points explaining the performance differences are most likely the following:</p>

<ul>
<li>Using a python function (as in <code>apply()</code> and <code>applymap()</code>) is (<strong>of course!</strong>) much slower than using functionality completely implemented in C</li>
<li>String evaluation is expensive (see (6) vs (2))</li>
<li>The overhead (1) has over (2) is probably the backend choice and fallback to also using the <code>python</code> backend, because <code>pandas</code> does not evaluate <code>bool * int</code>.</li>
</ul>

<p>Nothing new, eh?</p>

<h2>How to proceed</h2>

<p>We basically just proved what our gut feeling was telling us before (namely: pandas chooses the right backend for a task).</p>

<p>As a consequence, I think it is <strong>totally okay</strong> to ignore the UserWarning, as long as you know the underlying <em>hows and whys</em>.</p>

<p>Thus: Keep going and have <code>pandas</code> use the fastest of all implementations, which is, as usual, the C functions.</p>

<h2>The Test Script</h2>

<pre><code>from __future__ import print_function

import sys
import random
import pandas as pd
import numpy as np
from timeit import default_timer as timer

def conditional_column(val):
    if val &lt; 0.3:
        return 1
    elif val &gt; 0.6:
        return 3
    return 2

if __name__ == '__main__':
    nr = 10000000
    df = pd.DataFrame({
        'a': [random.random() for _ in range(nr)]
    })

    print(nr, 'rows')

    expr_a = '''(a &lt; 0.3) * 1 + (a &gt; 0.6) * 3 + (a &gt;= 0.3) * (a &lt;= 0.6) * 2'''
    expr_b = '''(@df.a &lt; 0.3) * 1 + (@df.a &gt; 0.6) * 3 + (@df.a &gt;= 0.3) * (@df.a &lt;= 0.6) * 2'''
    fmt = '{:16s} {:.12f}'

    # Evaluate the string expression using pandas parser
    t0 = timer()
    b = df.eval(expr_a, parser='pandas')
    print(fmt.format('(1) Eval (pd)', timer() - t0))

    # Evaluate the string expression using python parser
    t0 = timer()
    c = df.eval(expr_a, parser='python')
    print(fmt.format('(2) Eval (py)', timer() - t0))

    # Evaluate the string expression using pandas parser with external variable access (@)
    t0 = timer()
    d = df.eval(expr_b, parser='pandas')
    print(fmt.format('(3) Eval @ (pd)', timer() - t0))

    # Use apply to map the if/else function to each row of the df
    t0 = timer()
    d = df['a'].apply(conditional_column)
    print(fmt.format('(4) Apply', timer() - t0))

    # Use element-wise apply (WARNING: requires a dataframe and walks ALL cols AND rows)
    t0 = timer()
    e = df.applymap(conditional_column)
    print(fmt.format('(5) ApplyMap', timer() - t0))

    # Directly access the pandas series objects returned by boolean expressions on columns
    t0 = timer()
    f = (df['a'] &lt; 0.3) * 1 + (df['a'] &gt; 0.6) * 3 + (df['a'] &gt;= 0.3) * (df['a'] &lt;= 0.6) * 2
    print(fmt.format('(6) Direct', timer() - t0))
</code></pre>

python performance if-statement boolean apply