Speed up code that doesn't use groupby()?
<p>I have two pieces of code (doing the same job) which takes in array of <code>datetime</code> and produces clusters of <code>datetime</code> which have difference of 1 hour.</p>

<p>First piece is:</p>

<pre><code>def findClustersOfRuns(data):
    runClusters = []
    for k, g in groupby(itertools.izip(data[0:-1], data[1:]),
                        lambda (i, x): (i - x).total_seconds() / 3600):
        runClusters.append(map(itemgetter(1), g))
</code></pre>

<p>Second piece is:</p>

<pre><code>def findClustersOfRuns(data):
    if len(data) &lt;= 1:
        return []
        current_group = [data[0]]
        delta = 3600
        results = []

        for current, next in itertools.izip(data, data[1:]):
            if abs((next - current).total_seconds()) &gt; delta:
                # Here, `current` is the last item of the previous subsequence
                # and `next` is the first item of the next subsequence.
                if len(current_group) &gt;= 2:
                    results.append(current_group)

                current_group = [next]
                continue

            current_group.append(next)

        return results
</code></pre>

<p>The first code takes 5 minutes to execute while second piece takes few seconds. I am trying to understand why.</p>

<p>The data over which I am running the code has size:</p>

<pre><code>data.shape
(13989L,)
</code></pre>

<p>The data contents is as:</p>

<pre><code>data
array([datetime.datetime(2016, 10, 1, 8, 0),
       datetime.datetime(2016, 10, 1, 9, 0),
       datetime.datetime(2016, 10, 1, 10, 0), ...,
       datetime.datetime(2019, 1, 3, 9, 0),
       datetime.datetime(2019, 1, 3, 10, 0),
       datetime.datetime(2019, 1, 3, 11, 0)], dtype=object)
</code></pre>

<p>How do I improve the first piece of code to make it run as fast?</p>

<p>Based on the size, it looks you are having a huge <code>list</code> of elements i.e. huge <code>len</code>. Your second code is having just one <code>for</code> loop where as your first approach has many. You see just one right? They are in the form of <code>map()</code>, <code>groupby()</code>. Multiple iteration on the huge list is adding huge cost to the time complexity. <em>These are not just additional iterations, but also these are slower than the normal <code>for</code> loop.</em> </p>

<p>I made a comparison for another post which you might find useful <a href="http://stackoverflow.com/a/39518977/2063361">Comparing list comprehensions and explicit loops</a>.</p>

<p>Also, the usage of <code>lambda</code> function is adding up extra time. </p>

<p>However, you may further improve the execution time of the code by storing <code>results.append</code> to a separate variable say <code>my_func</code> and make a call as <code>my_func(current_group)</code>.</p>

<p>Few more comparisons are:</p>

<ul>
<li><a href="http://stackoverflow.com/q/36261621/2063361">Python 3: Loops, list comprehension and map slower compared to Python 2</a></li>
<li><a href="http://stackoverflow.com/a/28584157/2063361">Speed/efficiency comparison for loop vs list comprehension vs other methods</a></li>
</ul>

python lambda