Optimization of for loop in python
<p>I am executing the following code for different time stamps and each will have close to one million records. It took more than one hour for one date and I have the data for a total of 35 dates.</p>

<p>Is there a way to optimize this code?</p>

<pre><code>def median(a, b, c,d,e):
    I=[a,b,c,d,e]
    I.sort()
    return I[2]

for i in range(2, len(df['num'])-2):
    num_smooth= median(df['num'][i-1], df['num'][i-2], df['num'][i],
                       df['num'][i+1], df['num'][i+2])
    df.set_value(i,'num_smooth',num_smooth)
df['num_smooth'].fillna(df['num'], inplace=True)

...........................................
Remaining code
</code></pre>

<p>I'm guessing that your <code>df</code> is a Pandas <code>DataFrame</code> object. Pandas has built-in functionality to compute rolling statistics, including a rolling median. This functionality is available via the <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.rolling.html" rel="nofollow"><code>rolling</code></a> method on Pandas <code>Series</code> and <code>DataFrame</code> objects.</p>

<pre><code>&gt;&gt;&gt; s = pd.Series(np.random.rand(10))
&gt;&gt;&gt; s
0    0.500538
1    0.598179
2    0.747391
3    0.371498
4    0.244869
5    0.930303
6    0.327856
7    0.317395
8    0.190386
9    0.976148
dtype: float64
&gt;&gt;&gt; s.rolling(window=5, center=True).median()
0         NaN
1         NaN
2    0.500538
3    0.598179
4    0.371498
5    0.327856
6    0.317395
7    0.327856
8         NaN
9         NaN
dtype: float64
</code></pre>

<p>See the Pandas documentation on <a href="http://pandas.pydata.org/pandas-docs/stable/computation.html#window-functions" rel="nofollow">Window Functions</a> for more general information on using <code>rolling</code> and related functionality. As a general rule, when performance matters you should prefer using built-in Pandas and NumPy functions and methods over explicit Python-level <code>for</code> loops, though as always, you should profile your solutions to be sure. On my machine, working with a <code>df['num']</code> series containing one million random floats, the <code>rolling</code>-based solution takes approximately 129 seconds, while the <code>for</code>-loop based solution takes around 0.61 seconds, so using <code>rolling</code> speeds the code up by a factor of over 200.</p>

<p>So in your case,</p>

<pre><code>df['num_smooth'] = df['num'].rolling(window=5, center=True).median()
</code></pre>

<p>along with the <code>fillna</code> step that you already have should give you something close to what you need.</p>

<p>Note that the syntax for computing rolling statistics changed in Pandas 0.18, so you'll need at least version 0.18 to use the above code. For earlier versions of Pandas, look into the <a href="http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.rolling_median.html#pandas-rolling-median" rel="nofollow"><code>rolling_median</code></a> function.</p>

python loops pandas for-loop optimization