How to append elements to a list by using reduceByKey in pyspark
<p>I'm kind of stuck trying to solve a problem in pyspark. After do same calculations by using map function, I have a RDD that contains a list of dicts in this way:</p>

<pre><code>[{key1: tuple1}, {key1: tuple2}....{key2: tuple1}, {keyN: tupleN}] 
</code></pre>

<p>I pretend to append for each key a list with all the tuples with the same key, obtaining something like:</p>

<pre><code>[{key1: [tuple1, tuple2, tuple3...]}, {key2: [tuple1, tuple2....]}] 
</code></pre>

<p>I think an example it's more illustrative:</p>

<pre><code>[{0: (0, 1.0)}, {0: (1, 0.0)}, {1: (0, 0.0)}, {1: (1, 1.0)}, {2:(0,0.0)}... ]
</code></pre>

<p>And I would like to obtain list of dicts like this:</p>

<pre><code>[{0: [(0, 1.0), (1, 0.0)}, {1: [(0, 0.0), (1, 1.0)]}, {2:[(0,0.0),...]},...]
</code></pre>

<p>I'm trying to avoid using "combineByKey" function because it lasts too much time, there is any possibility to do that with "reduceByKey"?? </p>

<p>Thanks you all very much.</p>

<p>Here's a possible solution without using reduceByKey but just python builtin functions:</p>

<pre><code>from collections import defaultdict


inp = [{0: (0, 1.0)}, {0: (1, 0.0)}, {1: (0, 0.0)},
       {1: (1, 1.0)}, {2: (0, 0.0)}]

out = defaultdict(list)

for v in inp:
    for k, v1 in v.iteritems():
        out[k].append(v1)

out = [{k: v} for k, v in out.iteritems()]
print out
</code></pre>

python append pyspark rdd reduce