How to replace the for loop by faster option in Python
<p>I have two data frames, which have approximately 10 000 rows. They are similar as a and b below but with more rows.</p>

<pre><code> a
 Out[9]: 
     end  start
 0   4.0      3
 1   5.5      5
 2   7.5      7
 3   9.5      9
 4  11.5     11
 5  15.0     14
 6  18.0     17
 7  21.0     20
 8  26.0     25
 9  31.0     30

 b
 Out[10]: 
        status
 moment       
 8.0         o
 10.0        o
 14.5        o
 16.0        o
 19.0        o
 27.0        o
 28.0        o
 30.5        o
 35.0        o
 40.0        o
 50.0        o
</code></pre>

<p>I have to find all moments in dataframe b which belong to between end and start in dataframe a.</p>

<p>I developed for loop for that and it works well with small dataframes.</p>

<pre><code> for r in a.index:    
     for k in b.index:
         if a.ix[r,'start'] &lt;k and k &lt;a.ix[r,'end']:
             b.ix[k,'status']='m' # replaces m to o if moment is in between start and end
</code></pre>

<p>Below you can see how for loop have replaced o -> m when moment is in between start and end.</p>

<pre><code> n [12]: b
 Out[12]: 
        status
 moment       
 8.0         o
 10.0        o
 14.5        m
 16.0        o
 19.0        o
 27.0        o
 28.0        o
 30.5        m
 35.0        o
 40.0        o
 50.0        o
</code></pre>

<p>When I try to use it with huge dataframes (more than 10 000 rows in dataframe) it cannot get results any more within reasonable time.</p>

<p>Do you have any ideas how to elaborate my for loop faster and suitable for longer dataframes?</p>

<p>Your solution is <code>O(n^2)</code> in running time. As far as i see all of the dataframes are sorted, if this is the case for the whole <code>DF</code> you can do a divide and conquer method to make it <code>O(nlogn)</code>. However coding it is not easy, you will have to look it up, understand the D&amp;C methods and write it as a recursive function. I think you can create a so called <code>BinarySearch</code> algorithm for this problem which is O(logn) for each element, so O(nlogn). </p>

<p>If i am wrong and the <code>DF</code> is not sorted, but you have to do this kind of search multiple times i would advise to sort it first. Sorting can also be done in D&amp;C, its usually called <code>MergeSort</code> and its O(nlogn) as well.</p>

python pandas