itertools.groupby function seems inconsistent
<p>I'm having trouble understanding exactly what it is that this function does because of I guess, the programming magic around its use?</p>

<p>It seems to me like it returns a list of keys (unique letters in a string) paired with iterators, that reference a list of the number of each of those letters in the original string, but sometimes it seems like this is not the case.</p>

<p>For example:</p>

<pre><code>import itertools

x = list(itertools.groupby("AAABBB"))
print x
</code></pre>

<p>which prints:</p>

<pre><code>[('A', &lt;itertools._grouper object at 0x101a0b050), 
 ('B', &lt;itertools._grouper object at 0x101a0b090)]
</code></pre>

<p>This seems correct, we have our unique keys paired with iterators. But when I run:</p>

<pre><code>print list(x[0][1])
</code></pre>

<p>I get:</p>

<pre><code>[]
</code></pre>

<p>and when I run</p>

<pre><code>for k, g in x:
    print k + ' - ' + g
</code></pre>

<p>I get:</p>

<pre><code>B - &lt;itertools._grouper object at 0x1007eedd5&gt;
</code></pre>

<p>It ignores the first element. This seems counter-intuitive, because if I just change the syntax a little bit:</p>

<pre><code>[list(g) for k, g in itertools.groupby("AAABBB")]
</code></pre>

<p>I get:</p>

<pre><code>[["A", "A", "A"], ["B", "B", "B"]]
</code></pre>

<p>which is right, and aligns with what I think this function should be doing.</p>

<p>However, if I once again change the syntax just a bit:</p>

<pre><code>[list(thing) for thing in [g for k, g in itertools.groupby(string)]]
</code></pre>

<p>I get back:</p>

<pre><code>[[], ['B']]
</code></pre>

<p>These two list comprehensions should be directly equivalent, but they return different results.</p>

<p>What is going on? Insight would be extremely appreciated.</p>

<p>To get the answers you expect, convert the returned iterators to a list.</p>

<p><a href="https://docs.python.org/3/library/itertools.html#itertools.groupby" rel="nofollow"><em>Groupby</em></a> consumes an input iterator lazily (that means that it reads data only as needed).  To find a new group, it needs to read up to next non-equal element (the first member of the next group).  If you <em>list</em> the subgroup iterator, it will advance the input to the end of the current group.</p>

<p>In general, if you advance to the next group, then the previously returned subgroup iterator won't have an data and will appear empty.  So, if you need the data in the subgroup iterator, you need to <em>list</em> it <strong>before</strong> advancing to the next group.</p>

<p>The reason for this behavior is that iterators are all about looking a one piece of data at a time and not keeping anything unnecessary in memory.</p>

<p>Here's some code that make all the operations visible:</p>

<pre><code>from itertools import groupby

def supply():
    'Make the lazy input visible'
    for c in 'aaaaabbbcdddddddeeee':
        print('supplying %r' % c)
        yield c

print("\nCase where we don't consume the sub-iterator")
for k, g in groupby(supply()):
    print('Got group for %r' % k)

print("\nCase where we do consume the sub-iterator before advancing")
for k, g in groupby(supply()):
    print('Got group for %r' % k)
    print(list(g))
</code></pre>

<p>In the example "that is driving you crazy", the <em>list</em> operation is being applied too late (in the outer list comprehension).  The solution is to move the <em>list</em> step to the inner comprehension:</p>

<pre><code>&gt;&gt;&gt; import itertools
&gt;&gt;&gt; [list(g) for k, g in itertools.groupby('aaaaabbbb')]
&gt;&gt;&gt; [['a', 'a', 'a', 'a', 'a'], ['b', 'b', 'b', 'b']]
</code></pre>

<p>If you don't really care about conserving memory, then running <code>grouped = [list(g) for k, g in itertools.groupby(data)]</code> is a perfectly reasonable way to go.  Then you can lookup data in any of the sublists whenever you want and not be subject to rules about when the iterator is consumed.  In general, list of lists are easier to work with than iterators.  Hope this helps :-)</p>

python group-by list-comprehension itertools