optimization & numpy vectorization: repeat set of arrays
<p>I am getting used to the awesomness of <code>numpy.apply_along_axis</code> and I was wondering whether I could take the vectorization to the next level - mainly for speed purposes, that is using the potential of the function by trying to eliminate the for loop I have in the code below. </p>

<pre><code>from pandas import DataFrame
import numpy as np
from time import time

list_away = []

new_data_morphed = DataFrame(np.random.rand(1000, 5)).transpose().as_matrix()
group_by_strat_full = DataFrame([
    [0, 1, 4],
    [1, 2, 3],
    [2, 3, 4],
    [0, 1, 4]], columns=['add_mask_1', 'add_mask_2', 'add_mask_3', ])

all_the_core_strats = [lambda x: x.prod(), lambda x: sum(x), ]

def run_strat_mod(df_vals, core_strat, dict_mask_1, dict_mask_2, dict_mask_3, list_away):
    slice_df = df_vals[[dict_mask_1, dict_mask_2, dict_mask_3]]
    # TODO: this comprehension list should be vectorized
    to_append = [np.apply_along_axis(lambda x: u(x), 0, slice_df) for u in core_strat]
    list_away.append(to_append)

t1 = time()
results_3 = group_by_strat_full.apply(lambda x: run_strat_mod(
    new_data_morphed,
    all_the_core_strats,
    x['add_mask_1'],
    x['add_mask_2'],
    x['add_mask_3'],
    list_away), axis=1)

t2 = time()
print(abs(t1 - t2))
</code></pre>

<p>in order to do that, I was thinking of repeating the initial set of arrays, that is <code>slice_df</code> so that I could apply <code>numpy.apply_along_axis</code> to a new <code>all_the_core_strats_mod</code>.</p>

<p>with output, from this:</p>

<pre><code>    print(slice_df)
    [[[ 0.91302268  0.6172959   0.05478723 ...,  0.37028638  0.52116891
    0.14158221]
  [ 0.72579223  0.78732047  0.61335979 ...,  0.46359203  0.27593171
    0.73700975]
  [ 0.21706977  0.87639447  0.44936619 ...,  0.44319643  0.53712003
    0.8071096 ]]
</code></pre>

<p>to this:</p>

<pre><code>slice_df = np.array([df_vals[[dict_mask_1, dict_mask_2, dict_mask_3]]] * len(core_strat))
print(slice_df)

[[[ 0.91302268  0.6172959   0.05478723 ...,  0.37028638  0.52116891
    0.14158221]
  [ 0.72579223  0.78732047  0.61335979 ...,  0.46359203  0.27593171
    0.73700975]
  [ 0.21706977  0.87639447  0.44936619 ...,  0.44319643  0.53712003
    0.8071096 ]]

 [[ 0.91302268  0.6172959   0.05478723 ...,  0.37028638  0.52116891
    0.14158221]
  [ 0.72579223  0.78732047  0.61335979 ...,  0.46359203  0.27593171
    0.73700975]
  [ 0.21706977  0.87639447  0.44936619 ...,  0.44319643  0.53712003
    0.8071096 ]]]
</code></pre>

<p>and then </p>

<pre><code>def all_the_core_strats_mod(x):
    return [x[0].prod(), sum(x[1])]

to_append = np.apply_along_axis(all_the_core_strats_mod, 0, slice_df)
</code></pre>

<p>but it's not working as I imagined (applying functions separately to each duplicated block). </p>

<p>any ideas are welcome (the faster the better!)</p>

<pre><code>def foo(x):
   print(x)  # add for diagnosis
   return [x[0].prod(), x[1].sum()]
</code></pre>

<p>You apply it to a 3d array, which for simplicity I'll use:</p>

<pre><code>In [64]: x=np.arange(2*3*2).reshape(2,3,2)

In [66]: np.apply_along_axis(foo,0,x)
[0 6]
[1 7]
[2 8]
[3 9]
[ 4 10]
[ 5 11]
Out[66]: 
array([[[ 0,  1],
        [ 2,  3],
        [ 4,  5]],

       [[ 6,  7],
        [ 8,  9],
        [10, 11]]])
</code></pre>

<p>So <code>apply_along_axis</code> is passing to <code>foo</code>, <code>x[:,0,0]</code>, <code>x[:,0,1]</code>, <code>x[:,1,0]</code>, etc.  Taking <code>prod</code> and <code>sum</code> on those 2 numbers isn't very exciting.</p>

<p><code>apply_along_axis</code> is just a convenient way of doing:</p>

<pre><code>for i in range(x.shape[1]):
   for j in range(x.shape[2]):
      ret[:,i,j] = foo(x[:,i,j])
</code></pre>

python arrays numpy vectorization