Parallelize pandas apply
<p>New to pandas, I already want to parallelize a row-wise apply operation. So far I found <a href="http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby">Parallelize apply after pandas groupby</a> However, that only seems to work for grouped data frames.</p>

<p>My use case is different: I have a list of holidays and for my current row/date want to find the no-of-days before and after this day to the next holiday.</p>

<p>This is the function I call via apply:</p>

<pre><code>def get_nearest_holiday(x, pivot):
    nearestHoliday = min(x, key=lambda x: abs(x- pivot))
    difference = abs(nearesHoliday - pivot)
    return difference / np.timedelta64(1, 'D')
</code></pre>

<p>How can I speed it up?</p>

<h1>edit</h1>

<p>I experimented a bit with pythons pools - but it was neither nice code, nor did I get my computed results.</p>

<p>I think going down the route of trying stuff in parallel is probably over complicating this. I haven't tried this approach on a large sample so your mileage may vary, but it should give you an idea...</p>

<p>Let's just start with some dates...</p>

<pre><code>import pandas as pd

dates = pd.to_datetime(['2016-01-03', '2016-09-09', '2016-12-12', '2016-03-03'])
</code></pre>

<p>We'll use some holiday data from <code>pandas.tseries.holiday</code> - note that in effect we want a <code>DatetimeIndex</code>...</p>

<pre><code>from pandas.tseries.holiday import USFederalHolidayCalendar

holiday_calendar = USFederalHolidayCalendar()
holidays = holiday_calendar.holidays('2016-01-01')
</code></pre>

<p>This gives us:</p>

<pre><code>DatetimeIndex(['2016-01-01', '2016-01-18', '2016-02-15', '2016-05-30',
               '2016-07-04', '2016-09-05', '2016-10-10', '2016-11-11',
               '2016-11-24', '2016-12-26',
               ...
               '2030-01-01', '2030-01-21', '2030-02-18', '2030-05-27',
               '2030-07-04', '2030-09-02', '2030-10-14', '2030-11-11',
               '2030-11-28', '2030-12-25'],
              dtype='datetime64[ns]', length=150, freq=None)
</code></pre>

<p>Now we find the indices of the nearest nearest holiday for the original dates using <code>searchsorted</code>:</p>

<pre><code>indices = holidays.searchsorted(dates)
# array([1, 6, 9, 3])
next_nearest = holidays[indices]
# DatetimeIndex(['2016-01-18', '2016-10-10', '2016-12-26', '2016-05-30'], dtype='datetime64[ns]', freq=None)
</code></pre>

<p>Then take the difference between the two:</p>

<pre><code>next_nearest_diff = pd.to_timedelta(next_nearest.values - dates.values).days
# array([15, 31, 14, 88])
</code></pre>

<p>You'll need to be careful about the indices so you don't wrap around, and for the previous date, do the calculation with the <code>indices - 1</code> but it should act as (I hope) a relatively good base.</p>

python pandas parallel-processing apply embarrassingly-parallel