Spark: Extract zipped keys without computing values
<p>Spark has a <a href="http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.zip" rel="nofollow"><code>zip()</code></a> function to combine two RDDs.  It also has functions to split them apart again: <a href="http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.keys" rel="nofollow"><code>keys()</code></a> and <a href="http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.values" rel="nofollow"><code>values()</code></a>.  But to my surprise, if you ask for just the <code>keys()</code>, both RDDs are fully computed, even if the values weren't necessary for the computation.</p>

<p>In this example, I create an RDD of <code>(key, value)</code> pairs, but then I only ask for the keys.  Why are the values computed anyway?  Does Spark make no attempt to simplify it's internal DAG in such cases?</p>

<pre class="lang-py prettyprint-override"><code>In [1]: def process_value(val):
   ...:     print "Processing {}".format(val)
   ...:     return 2*val
   ...:

In [2]: k = sc.parallelize(['a','b','c'])

In [3]: v = sc.parallelize([1,2,3]).map(process_value)

In [4]: zipped = k.zip(v)

In [5]: zipped.keys().collect()
Processing 1
Processing 2
Processing 3

Out[5]: ['a', 'b', 'c']
</code></pre>

<p>If you look at the <a href="https://github.com/apache/spark/blob/v2.0.0/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala#L1249" rel="nofollow">source</a> (at least at 2.0) keys() is simply implemented as </p>

<p><code>rdd.map(_._1)</code></p>

<p>I.e. returning the first attribute of the tuple, so the tuple must be fully instantiated.</p>

<p>This might have worked if zip returned <code>RDD[Seq[K, V]]</code> or some other lazy data structure, but a tuple is not a lazy data structure.</p>

<p>In short: no.</p>

python apache-spark pyspark