Parallelize pandas apply
<p>New to pandas, I already want to parallelize a row-wise apply operation. So far I found <a href="http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby">Parallelize apply after pandas groupby</a> However, that only seems to work for grouped data frames.</p>

<p>My use case is different: I have a list of holidays and for my current row/date want to find the no-of-days before and after this day to the next holiday.</p>

<p>This is the function I call via apply:</p>

<pre><code>def get_nearest_holiday(x, pivot):
    nearestHoliday = min(x, key=lambda x: abs(x- pivot))
    difference = abs(nearesHoliday - pivot)
    return difference / np.timedelta64(1, 'D')
</code></pre>

<p>How can I speed it up?</p>

<h1>edit</h1>

<p>I experimented a bit with pythons pools - but it was neither nice code, nor did I get my computed results.</p>

<p>For the parallel approach this is the answer based on <a href="http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby">Parallelize apply after pandas groupby</a>:</p>

<pre><code>from joblib import Parallel, delayed
import multiprocessing

def get_nearest_dateParallel(df):
    df['daysBeforeHoliday'] = df.myDates.apply(lambda x: get_nearest_date(holidays.day[holidays.day &lt; x], x))
    df['daysAfterHoliday']  =  df.myDates.apply(lambda x: get_nearest_date(holidays.day[holidays.day &gt; x], x))
    return df

def applyParallel(dfGrouped, func):
    retLst = Parallel(n_jobs=multiprocessing.cpu_count())(delayed(func)(group) for name, group in dfGrouped)
    return pd.concat(retLst)

print ('parallel version: ')
# 4 min 30 seconds
%time result = applyParallel(datesFrame.groupby(datesFrame.index), get_nearest_dateParallel)
</code></pre>

<p>but I prefer @NinjaPuppy's approach because it does not require O(n * number_of_holidays) </p>

python pandas parallel-processing apply embarrassingly-parallel