pyspark: difference between using (,) and [,] for pair representation for reducedByKey
<p>I am applying a map and then reduceByKey transformations on an RDD using pyspark. I tried both of the following syntax, and both of them seem to work:</p>

<p>case 1:</p>

<pre><code>my_rdd_out = my_rdd.map(lambda r: [r['my_id'], [[r['my_value']]]])\
                                .reduceByKey(lambda a, b: a+b)\
                                .map(lambda r: r[1])
</code></pre>

<p>case 2:</p>

<pre><code>my_rdd_out = my_rdd.map(lambda r: (r['my_id'], [[r['my_value']]]))\
                                .reduceByKey(lambda a, b: a+b)\
                                .map(lambda r: r[1])
</code></pre>

<p>The r here is of the class <code>from pyspark.sql import Row</code>.
In case 1, the map output pair is in bracket; in case 2, the map output pair is in parenthesis. Though both works, I am wondering is there any difference between using [] and () for representing a pair that will be the input for reduceByKey later? Thanks!</p>

<p>The difference between a <code>tuple</code> and a <code>list</code> in python is that <code>tuple</code> object are immutable so they are hashable. <code>list</code> objects are not hashable since they can be modified using their reference.</p>

<p>In your case you can use any of them (or the <code>reduceByKey</code> method wouldn't apply to both tuples and lists), it's just a convenience to avoid casting one into another when you get the object from some caller (The method only needs to iterate through the collection, does not care what kind of collection it is).</p>

<p>here is an implementation of <code>reduceByKey</code> lifted from <a href="https://gist.github.com/Juanlu001/562d1ec55be970403442" rel="nofollow">here</a></p>

<pre><code>def reduceByKey(func, iterable):
    """Reduce by key.
    Equivalent to the Spark counterpart
    Inspired by http://stackoverflow.com/q/33648581/554319
    1. Sort by key
    2. Group by key yielding (key, grouper)
    3. For each pair yield (key, reduce(func, last element of each grouper))
    """
    get_first = lambda p: p[0]
    get_second = lambda p: p[1]
    # iterable.groupBy(_._1).map(l =&gt; (l._1, l._2.map(_._2).reduce(func)))
    return map(
        lambda l: (l[0], reduce(func, map(get_second, l[1]))),
        groupby(sorted(iterable, key=get_first), get_first)
    )
</code></pre>

<p>In your example you have <code>tuple(&lt;something&gt;).reduceByKey(lambda &lt;something&gt;)</code>. Obviously, the iterable is the <code>tuple</code> and the <code>func</code> is the lambda expression.</p>

<p>As you can see, the input just needs to be an iterable. index access is not even required.</p>

<p>You could have passed a <code>set</code>, a <code>deque</code>, a generator comprehension, whatever. It contains no conversion to list or tuple whatsoever.</p>

<p>It doesn't even need to get all data at the same time, just one at a time (generator functions/comprehensions would work too): elegant way of avoiding useless temporary object creation.</p>

<p>This requires that <code>iterable</code> is only iterated through once in the function, which is the case here with the <code>sorted</code> function that generates a <code>list</code>.</p>

python lambda row pyspark