Is this approach "vectorized" - used against medium dataset it is relatively slow
<p>I have this data frame:</p>

<pre><code>df = pd.DataFrame({'a' : np.random.randn(9),
             'b' : ['foo', 'bar', 'blah'] * 3,
             'c' : np.random.randn(9)})
</code></pre>

<p>This function:</p>

<pre><code>def my_test2(row, x):
    if x == 'foo':
        blah = 10
    if x == 'bar':
        blah = 20
    if x == 'blah':
        blah = 30
    return (row['a'] % row['c']) + blah
</code></pre>

<p>I am then creating 3 new columns like this:</p>

<pre><code>df['Value_foo'] = df.apply(my_test2, axis=1, x='foo')
df['Value_bar'] = df.apply(my_test2, axis=1, x='bar')
df['Value_blah'] = df.apply(my_test2, axis=1, x='blah')
</code></pre>

<p>It runs ok but when I make my_test2 more complex and expand df to several thousand rows it is slow - is the above what I hear described as "vectorized"? Can I easily speed things up?</p>

<p>As Andrew, Ami Tavory and Sohier Dane have already mentioned in comments there are two "slow" things in your solution:</p>

<ol>
<li><code>.apply()</code> is generally slow as it loops under the hood.</li>
<li><code>.apply(..., axis=1)</code> is <strong>extremely</strong> slow (even compared to <code>.apply(..., axis=0)</code>) as it does looping on the row basis</li>
</ol>

<p>Here is a vectorized approach:</p>

<pre><code>In [74]: d = {
   ....:   'foo': 10,
   ....:   'bar': 20,
   ....:   'blah': 30
   ....: }

In [75]: d
Out[75]: {'bar': 20, 'blah': 30, 'foo': 10}

In [76]: for k,v in d.items():
   ....:         df['Value_{}'.format(k)] = df.a % df.c + v
   ....:

In [77]: df
Out[77]:
          a     b         c  Value_bar  Value_blah  Value_foo
0 -0.747164   foo  0.438713  20.130262   30.130262  10.130262
1 -0.185182   bar  0.047253  20.003828   30.003828  10.003828
2  1.622818  blah -0.730215  19.432174   29.432174   9.432174
3  0.117658   foo  1.530249  20.117658   30.117658  10.117658
4  2.536363   bar -0.100726  19.917499   29.917499   9.917499
5  1.128002  blah  0.350663  20.076014   30.076014  10.076014
6  0.059516   foo  0.638910  20.059516   30.059516  10.059516
7 -1.184688   bar  0.073781  20.069590   30.069590  10.069590
8  1.440576  blah -2.231575  19.209001   29.209001   9.209001
</code></pre>

<p>Timing against 90K rows DF:</p>

<pre><code>In [80]: big = pd.concat([df] * 10**4, ignore_index=True)

In [81]: big.shape
Out[81]: (90000, 3)

In [82]: %%timeit
   ....: big['Value_foo'] = big.apply(my_test2, axis=1, x='foo')
   ....: big['Value_bar'] = big.apply(my_test2, axis=1, x='bar')
   ....: big['Value_blah'] = big.apply(my_test2, axis=1, x='blah')
   ....:
1 loop, best of 3: 10.5 s per loop

In [83]: big = pd.concat([df] * 10**4, ignore_index=True)

In [84]: big.shape
Out[84]: (90000, 3)

In [85]: %%timeit
   ....: for k,v in d.items():
   ....:     big['Value_{}'.format(k)] = big.a % big.c + v
   ....:
100 loops, best of 3: 7.24 ms per loop
</code></pre>

<p><strong>Conclusion:</strong> vectorized approach is 1450 times faster...</p>

python pandas dataframe