Cleanest way to combine reduce and map in Python
<p>I'm doing a little deep learning, and I want to grab the values of all hidden layers.  So I end up writing functions like this:</p>

<pre><code>def forward_pass(x, ws, bs):
    activations = []
    u = x
    for w, b in zip(ws, bs):
        u = np.maximum(0, u.dot(w)+b)
        activations.append(u)
    return activations
</code></pre>

<p>If I didn't have to get the intermediate values, I'd use the much less verbose form:</p>

<pre><code>out = reduce(lambda u, (w, b): np.maximum(0, u.dot(w)+b), zip(ws, bs), x)
</code></pre>

<p>Bam.  All one line, nice and compact.  But I can't keep any of the intermediate values.</p>

<p>So, what is there any way to have my cake (nice compact one-liner) and eat it too (return intermediate values)?</p>

<p><strong>Edit:  My conclusions so far:</strong><br>
In Python 2.x, there is no clean one-liner for this.<br>
In Python 3, there is <code>itertools.accumulate</code>, but it is still not really clean because it doesn't accept an "initial" input, as <code>reduce</code> does.<br>
In Python 4, I hereby request a "map-reduce comprehension":  </p>

<pre><code>activations = [u=np.maximum(0, u.dot(w)+b) for w, b in zip(ws, bs) from u=x]  
</code></pre>

<p>Which would also give a useful job to that <code>from</code> keyword, which normally just sits around doing nothing after all the imports are done.</p>

<p>The <code>dot</code> tells me you are using one or more numpy arrays.  So I'll try:</p>

<pre><code>In [28]: b=np.array([1,2,3])
In [29]: x=np.arange(9).reshape(3,3)
In [30]: ws=[x,x,x]

In [31]: forward_pass(x,ws,bs)
Out[31]: 
[array([[ 16,  19,  22],
        [ 43,  55,  67],
        [ 70,  91, 112]]), 
 array([[ 191,  248,  305],
        [ 569,  734,  899],
        [ 947, 1220, 1493]]), 
 array([[ 2577,  3321,  4065],
        [ 7599,  9801, 12003],
        [12621, 16281, 19941]])]
</code></pre>

<p>In py3 I have to write the <code>reduce</code> solution as:</p>

<pre><code>In [32]: functools.reduce(lambda u, wb: np.maximum(0,
                   u.dot(wb[0])+wb[1]), zip(ws, bs), x)
Out[32]: 
array([[ 2577,  3321,  4065],
       [ 7599,  9801, 12003],
       [12621, 16281, 19941]])
</code></pre>

<p>That intermediate value <code>u</code> that is passed from one evaluation to the next makes a list comprehension tricky.</p>

<p><code>accumulate</code> uses the first item as the start.  I can work around that with a function like</p>

<pre><code>def foo(u, wb):
    if u[0] is None: u=x   # x from global
    return np.maximum(0, u.dot(wb[0])+wb[1])
</code></pre>

<p>Then I need to add extra start values to <code>ws</code> and <code>bs</code>:</p>

<pre><code>In [56]: list(itertools.accumulate(zip([None,x,x,x], np.array([0,1,2,3])), foo))
Out[56]: 
[(None, 0), 
 array([[ 16,  19,  22],
        [ 43,  55,  67],
        [ 70,  91, 112]]), 
 array([[ 191,  248,  305],
        [ 569,  734,  899],
        [ 947, 1220, 1493]]), 
 array([[ 2577,  3321,  4065],
        [ 7599,  9801, 12003],
        [12621, 16281, 19941]])]
</code></pre>

<p>Here's a list comprehension version, using an external <code>u</code>:</p>

<pre><code>In [66]: u=x.copy()
In [67]: def foo1(wb):
    ...:     v = np.maximum(0, u.dot(wb[0])+wb[1])
    ...:     u[:]=v
    ...:     return v
    ...: 
In [68]: [foo1(wb) for wb in zip(ws,bs)]
Out[68]: 
[array([[ 16,  19,  22],
        [ 43,  55,  67],
        [ 70,  91, 112]]), 
 array([[ 191,  248,  305],
        [ 569,  734,  899],
        [ 947, 1220, 1493]]), 
 array([[ 2577,  3321,  4065],
        [ 7599,  9801, 12003],
        [12621, 16281, 19941]])]
</code></pre>

<p>No real advantage over the original loop with <code>append</code>.</p>

<p><code>numpy.ufunc</code> have an <code>accumulate</code> method, but that isn't easy to use with custom Python functions.  So there is a  <code>np.maximum.accumulate</code>, but I'm not sure how that could be used in this case.  (also <code>np.cumsum</code> which is <code>np.add.accumulate</code>).</p>

python numpy deep-learning reduce