Optimizing for loop in python
<p>I have a dataframe (df) with distance traveled and I have assigned a label based on certain conditions.</p>

<pre><code>distance=[0,0.0001,0.20,1.23,4.0]
df = pd.DataFrame(distance,columns=["distance"])
df['label']=0
for i in range(0, len(df['distance'])):   
      if (df['distance'].values[i])&lt;=0.10:
          df['label'][i]=1
      elif (df['distance'].values[i])&lt;=0.50:
          df['label'][i]=2
      elif (df['distance'].values[i])&gt;0.50:
          df['label'][i]=3
</code></pre>

<p>This is working fine. However, I have more than 1 million records with distance and this for loop is taking longer time than expected. Can we optimize this code to reduce the execution time?</p>

<p>In general, you shouldn't loop over DataFrames unless it's absolutely necessary. You'll usually get much better performance using a built-in Pandas function that's already been optimized, or by using a vectorized approach.  </p>

<p>In this case, you can use <code>loc</code> and <a href="http://pandas.pydata.org/pandas-docs/stable/indexing.html#boolean-indexing" rel="nofollow">Boolean indexing</a> to do the assignments:</p>

<pre><code># Initialize as 1 (eliminate need to check the first condition).
df['label'] = 1

# Case 1: Between 0.1 and 0.5
df.loc[(df['distance'] &gt; 0.1) &amp; (df['distance'] &lt;= 0.5), 'label'] = 2

# Case 2: Greater than 0.5
df.loc[df['distance'] &gt; 0.5, 'label'] = 3
</code></pre>

<p>Another option is to use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.cut.html" rel="nofollow"><code>pd.cut</code></a>.  This is a method is a little more specialized to the example problem in the question.  Boolean indexing is a more general method.</p>

<pre><code># Get the low and high bins.
low, high = df['distance'].min()-1, df['distance'].max()+1

# Perform the cut.  Add one since the labels start at zero by default.
df['label'] = pd.cut(df['distance'], bins=[low, 0.1, 0.5, high], labels=False) + 1
</code></pre>

<p>You could also use <code>labels=[1,2,3]</code> in the code above, and not add 1 to the result.  This would give <code>df['labels']</code> categorical dtype instead of integer dtype though.  Depending on your use case this may or may not be important.</p>

<p>The resulting output for either method:</p>

<pre><code>   distance  label
0    0.0000      1
1    0.0001      1
2    0.2000      2
3    1.2300      3
4    4.0000      3
</code></pre>

python pandas for-loop optimization