Generator expression evaluation with several ... for ... in ... parts
<p><strong>Question</strong>: What does Python do under the hood when it sees this kind of expression?  </p>

<pre><code>sum(sum(i) for j in arr for i in j)
</code></pre>

<hr>

<p><strong>My thoughts</strong>: <em>The above expression works.</em> But as it is written in <a href="https://docs.python.org/3.5/reference/executionmodel.html#resolution-of-names" rel="nofollow">Python's docs</a>:</p>

<blockquote>
  <p>generator expressions are implemented using a function scope</p>
</blockquote>

<p>Not to be verbose :) I have an array with the following layout (<em>as an example</em>):  </p>

<pre><code>&gt;&gt;&gt; arr = [
           [[1,2,3], [4,5,6]],
           [[7,8,9],[10,11,12]]
          ]
</code></pre>

<p>At first, I try to sum all elements of <code>arr</code> with the following expression:</p>

<pre><code>&gt;&gt;&gt; sum(sum(i) for i in j for j in arr)
NameError: name 'j' is not defined
</code></pre>

<p>It raises <code>NameError</code>, but why not <code>UnboundLocalError: local variable 'j' referenced before assignment</code> if it is implemented using a function scope, what is evaluation rules for <code>for ... in ...</code> from left-to-right or from right-to-left? And what is an equivalent generator function for this generator expression?</p>

<hr>

<p><strong>EDIT</strong>:</p>

<p>I catch the idea. Thanks @vaultah for some insight. In this case <code>j</code> is the argument that is send to generator expression:</p>

<pre><code>&gt;&gt;&gt; sum(sum(i) for i in j for j in arr)  # NameError
</code></pre>

<p>that's why I get this weird <code>NameError</code>.  </p>

<hr>

<p><a href="http://stackoverflow.com/a/39552690/6793085">@Eric answer</a> shows that generator expression:</p>

<pre><code>&gt;&gt;&gt; sum(sum(i) for j in arr for i in j)
</code></pre>

<p>is equivalent to: </p>

<pre><code>&gt;&gt;&gt; def __gen(arr):
        for j in arr:
            for i in j:
                yield sum(i)

&gt;&gt;&gt; sum(__gen(arr))
</code></pre>

<hr>

<p>Whether it is a generator or a list comprehension, the comprehension nesting is the same. It is easier to see what is going on with a list comprehension and that is what I will use in the examples below.</p>

<p>Given:</p>

<pre><code>&gt;&gt;&gt; arr
[[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]
</code></pre>

<p>You can flatten the List of Lists of Ints by 1 level using a nested list comprehension (or generator):</p>

<pre><code>&gt;&gt;&gt; [e for sl in arr for e in sl]
[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]]
</code></pre>

<p>You can flatten completely, given that structure, by nesting again (example only; there are better ways to flatten a deeply nested list):</p>

<pre><code>&gt;&gt;&gt; [e2 for sl2 in [e for sl in arr for e in sl] for e2 in sl2]
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
</code></pre>

<p>Since <code>sum</code> takes an iterable, the second flattening is not necessary in your example:</p>

<pre><code>&gt;&gt;&gt; [sum(e) for sl in arr for e in sl]
[6, 15, 24, 33]   # sum of those is 78...
</code></pre>

<hr>

<p>The general form of a comprehension is:</p>

<pre><code>[ expression      for a_variable in a_DEFINED_sequence     optional_predicate ]
</code></pre>

<p>You can get the same <code>NameError</code> you are seeing on your nested comprehension by using a non defined name:</p>

<pre><code>&gt;&gt;&gt; [c for c in not_defined]
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
NameError: name 'not_defined' is not defined
</code></pre>

<p>So the error you see on <code>sum(sum(i) for i in j for j in arr)</code> is because <code>j</code> has not been defined yet. Comprehensions are evaluated left to right, inner to outer. The definition of <code>j</code> as a sequence is to the right of its attempted use. </p>

<p>To unroll the list comprehension into nested loops, the <em>inner</em> (or left hand) section becomes the <em>outer</em> loop:</p>

<pre><code>for sl in arr:
    for sl2 in sl:
        for e in sl2:
           # now you have each int in the LoLoInts...
           # you could use yield e for a generator here
</code></pre>

<hr>

<p>Your final question: Why do you get a <code>TypeError</code> with <code>gen = (j for j in arr)</code>?</p>

<p>That generator expression does nothing. Example:</p>

<pre><code>&gt;&gt;&gt; [j for j in arr]
[[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]
&gt;&gt;&gt; [j for j in arr] == arr
True
</code></pre>

<p>So the expression <code>(j for j in arr)</code> just returns a generator over <code>arr</code>. </p>

<p>And <code>sum</code> does not know how to add that or arr either:</p>

<pre><code>&gt;&gt;&gt; sum(arr)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
TypeError: unsupported operand type(s) for +: 'int' and 'list'
</code></pre>

<p>Since <code>gen</code> in your example is returning the same data structure, that is your error. </p>

<p>To fix it:</p>

<pre><code>&gt;&gt;&gt; gen=(e for sl in arr for e in sl)
&gt;&gt;&gt; sum(sum(li) for li in gen)
78
</code></pre>

python python-3.x scope nested generator