Removing elements from the list when looping over it
<p>This part of my code does not scale if <strong>dimension</strong> gets bigger. </p>

<p>I loop over my data and accumulate them every <strong>dt</strong> time window. To do this I compare lower and upper time value. When I reach upper bound, I break the <strong>for loop</strong> for efficiency. The next time I run <strong>for loop</strong> I want to start not from its beginning but from the element I stopped previously, for efficiency. <strong>How can I do that?</strong> </p>

<p>I tried to remove/pop elements of the list but indexes get messed up. I read that I cannot modify the list I loop over, but my goal seems to be not uncommon so there has to be solution. I don't care about original <strong>data</strong> list later in my code, I only want optimization of my accumulation.</p>

<pre><code># Here I generate data for you to show my problem
from random import randint
import numpy as np

dimension = 200
times = [randint(0, 1000) for p in range(0, dimension)]
times.sort()
values = [randint(0, dimension) for p in range(0, dimension)]
data = [(values[k], times[k]) for k in range(dimension)]
dt = 50.0
t = min(times)
pixels = []
timestamps = []

# this is my problem
while (t &lt;= max(times)):
    accumulator = np.zeros(dimension)
    for idx, content in enumerate(data):
        # comparing lower bound of the 'time' window
        if content[1] &gt;= t:
            # comparing upper bound of the 'time' window
            if (content[1] &lt; t + dt):
                accumulator[content[0]] += 1
                # if I pop the first element from the list after accumulating, indexes are screwed when looping further
                # data.pop(0)
            else:
                # all further entries are bigger because they are sorted
                break

    pixels.append(accumulator)
    timestamps.append(t)
    t += dt
</code></pre>

<p>In a simpler form, I think you are trying to do:</p>

<pre><code>In [158]: times=[0, 4, 6, 10]
In [159]: data=np.arange(12)
In [160]: cnt=[0 for _ in times]
In [161]: for i in range(len(times)-1):
     ...:     for d in data:
     ...:         if d&gt;=times[i] and d&lt;times[i+1]:
     ...:             cnt[i]+=1
     ...:             
In [162]: cnt
Out[162]: [4, 2, 4, 0]
</code></pre>

<p>And you are trying to make this <code>data</code> loop more efficient by breaking form the loop when <code>d</code> gets too large, and by starting the next loop after items which have already been counted.</p>

<p>Adding the break is easy as you've done:</p>

<pre><code>In [163]: cnt=[0 for _ in times]
In [164]: for i in range(len(times)-1):
     ...:     for d in data:
     ...:         if d&gt;=times[i]:
     ...:             if d&lt;times[i+1]:
     ...:                 cnt[i]+=1
     ...:             else:
     ...:                 break

In [165]: cnt
Out[165]: [4, 2, 4, 0]
</code></pre>

<p>One way to skip the counted stuff is to replace the <code>for d in data</code> with a index loop; and keep track of where we stopped last time around:</p>

<pre><code>In [166]: cnt=[0 for _ in times]
In [167]: start=0
     ...: for i in range(len(times)-1):
     ...:     for j in range(start,len(data)):
     ...:         d = data[j]
     ...:         if d&gt;=times[i]:
     ...:             if d&lt;times[i+1]:
     ...:                 cnt[i]+=1
     ...:             else:
     ...:                 start = j
     ...:                 break
     ...:                 
In [168]: cnt
Out[168]: [4, 2, 4, 0]
</code></pre>

<p>A <code>pop</code> based version requires that I work with a list (my <code>data</code> is an array), a requires inserting the value back at the break</p>

<pre><code>In [186]: datal=data.tolist()
In [187]: cnt=[0 for _ in times]
In [188]: for i in range(len(times)-1):
     ...:     while True:
     ...:         d = datal.pop(0)
     ...:         if d&gt;=times[i]:
     ...:             if d&lt;times[i+1]:
     ...:                 cnt[i]+=1
     ...:             else:
     ...:                 datal.insert(0,d)
     ...:                 break
     ...:             
In [189]: cnt
Out[189]: [4, 2, 4, 0]
In [190]: datal
Out[190]: [10, 11]
</code></pre>

<p>This isn't perfect, since I still have items on the list at the end (my <code>times</code> don't cover the whole <code>data</code> range).  But it tests the idea.</p>

<p>Here's something closer to your attempt:</p>

<pre><code>In [203]: for i in range(len(times)-1):
     ...:     for d in datal[:]:
     ...:         if d&gt;=times[i]:
     ...:             if d&lt;times[i+1]:
     ...:                 cnt[i]+=1
     ...:                 datal.pop(0)
     ...:             else:
     ...:                 break
     ...:       
</code></pre>

<p>The key difference is that I iterate on a copy of <code>datal</code>.  That way the <code>pop</code> affects <code>datal</code>, but doesn't affect the current iteration.  Admittedly there's a cost to the copy, so the speed up might be significant.</p>

<p>A different approach would be to loop on <code>data</code>, and step <code>time</code> as the <code>t</code> and <code>t+dt</code> boundaries are crossed.</p>

<pre><code>In [222]: times=[0, 4, 6, 10,100]
In [223]: cnt=[0 for _ in times]; i=0
In [224]: for d in data:
     ...:     if d&gt;=times[i]:
     ...:         if d&lt;times[i+1]:
     ...:             cnt[i]+=1
     ...:         else:
     ...:             i += 1
     ...:             cnt[i]+=1
     ...:             
In [225]: cnt
Out[225]: [4, 2, 4, 2, 0]
</code></pre>

python list numpy iteration pop