Sum up the index value in multiple array with pyspark
<p>I have 50 array with float values (<code>50*7</code>). How am I suppose to sum up the 50 arrays on same index to one with PySpark map-reducer function.</p>

<p>Example:</p>

<pre><code>array1 = {1,2,3,4,5,6,7}
array2 = {3,4,2,3,5,6,7}
....
</code></pre>

<p>the result should be <code>array3 = {4,6,5,7,10,12,14}</code>.</p>

<p>This is a project requirement to use PySpark on Map-Reducer platform.</p>

<p>Now I can figure out the map part:</p>

<pre><code>NUM_SAMPLES = 50
result = sc.parallelize(xrange(0, NUM_SAMPLES)).map(random_generation)
</code></pre>

<p>The result here contains 50 arrays. Function <code>random_generation</code> gives one array with 7 random numbers.</p>

<p>Please anyone can provide me the suggestion on the reduce part.</p>

<p>Edit: I think it's easier to use DataFrame.</p>

<pre><code>from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .getOrCreate()

arrays = [
    [1,2,3,4,5,6,7],
    [3,4,2,3,5,6,7],
    [1,2,3,4,5,6,7],
]

df = spark.createDataFrame(arrays)
s = df.groupBy().sum().collect()

print s
print list(s[0])
</code></pre>

<p><strong>Result</strong></p>

<pre><code>[Row(sum(_1)=5, sum(_2)=8, sum(_3)=8, sum(_4)=11, sum(_5)=15, sum(_6)=18, sum(_7)=21)]
[5, 8, 8, 11, 15, 18, 21]
</code></pre>

python mapreduce pyspark