Set a current row in a dataframe based off of future values
<p>Given:</p>

<pre class="lang-py prettyprint-override"><code>    d = {
    'datetime': ['2010-01-08 09:45:00', '2010-01-08 10:00:00',
               '2010-01-08 10:15:00', '2010-01-08 10:30:00',
               '2010-01-08 10:45:00', '2010-01-08 11:00:00',
               '2010-01-08 11:15:00', '2010-01-08 11:30:00',
               '2010-01-08 11:45:00', '2010-01-08 12:00:00',
               '2010-01-08 12:15:00', '2010-01-08 12:30:00',
               '2010-01-08 12:45:00', '2010-01-08 13:00:00',
               '2010-01-08 13:15:00', '2010-01-08 13:30:00',
               '2010-01-08 13:45:00', '2010-01-08 14:00:00',
               '2010-01-08 14:15:00', '2010-01-08 14:30:00',
               '2010-01-08 14:45:00', '2010-01-08 15:00:00',
               '2010-01-08 15:15:00', '2010-01-08 15:30:00',
               '2010-01-08 15:45:00', '2010-01-08 16:00:00',
               '2010-01-08 16:15:00'],
    'Total-tops': [0,-1,-1,2,3,0,0,4,0,0,0,0,5,6,7,8,-1,0,0,0,0,0,0,0,-1,-1,2]
}

df = pandas.DataFrame(d)
df = df.set_index('datetime')
</code></pre>

<p>I want to add another column which is a boolean for whether that row will break or not. A break means the tops are at a number greater than 1 and then a -1 occurs somewhere in the future. For example the first 2 will break at the next -1 it encounters. Here is the desired dataframe:
<a href="http://i.stack.imgur.com/3TlcG.png" rel="nofollow"><img src="http://i.stack.imgur.com/3TlcG.png" alt="desired_dataframe"></a></p>

<p>Here is the function I am currently using, but it runs very slow, since I iterate over all rows.</p>

<pre class="lang-py prettyprint-override"><code>def does_break(data):
    cur_breaks = []

    for index, row in data.iterrows():
        if row['Total-tops'] &gt; 1:
            # Get all rows after this time that are new tops
            breaks = data[(data['Total-tops'] == -1) &amp; (data.index.time &gt; index.time())]
            if len(breaks) &gt; 0:
                cur_breaks.append(True)
            else:
                cur_breaks.append(False)
        else:
            cur_breaks.append(False)
    return cur_breaks
</code></pre>

<p>You can use the ungainly expression</p>

<pre><code>In [56]: import numpy as np

In [57]: ((np.cumsum((df['Total-tops'] == -1)[:: -1])[:: -1] &gt; 0) &amp; (df['Total-tops'] &gt; 0)).astype(int)
Out[57]: 
datetime
2010-01-08 09:45:00    0
2010-01-08 10:00:00    0
2010-01-08 10:15:00    0
2010-01-08 10:30:00    1
2010-01-08 10:45:00    1
2010-01-08 11:00:00    0
2010-01-08 11:15:00    0
2010-01-08 11:30:00    1
2010-01-08 11:45:00    0
2010-01-08 12:00:00    0
2010-01-08 12:15:00    0
2010-01-08 12:30:00    0
2010-01-08 12:45:00    1
2010-01-08 13:00:00    1
2010-01-08 13:15:00    1
2010-01-08 13:30:00    1
2010-01-08 13:45:00    0
2010-01-08 14:00:00    0
2010-01-08 14:15:00    0
2010-01-08 14:30:00    0
2010-01-08 14:45:00    0
2010-01-08 15:00:00    0
2010-01-08 15:15:00    0
2010-01-08 15:30:00    0
2010-01-08 15:45:00    0
2010-01-08 16:00:00    0
2010-01-08 16:15:00    0
Name: Total-tops, dtype: int64
</code></pre>

<p>(Of course, for your new column, you can use <code>df['breaks'] = ...</code>.)</p>

<p>What this does is as follows:</p>

<ol>
<li>We find where the values are -1, and reverse. Now any operations we do on the past (in particular <code>cumsum</code>), is really performed on the future.</li>
<li>We find the cumulative sum, and reverse again. At this point, the meaning is how many times in the future will we see a -1.</li>
<li>We find where the result is larger than 0, since we don't care <em>how many times</em> will we see a -1, only <em>whether</em> we will see it.</li>
<li>Finally, we also require that the current entry is positive. This is just the definition from your question.</li>
</ol>

python pandas