pyspark: expand a DenseVector to tuple in a RDD
<p>I have the following RDD, each record is a tuple of (bigint, vector):</p>

<pre><code>myRDD.take(5)

[(1, DenseVector([9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432])),
 (1, DenseVector([9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432])),
 (0, DenseVector([5.0, 20.0, 0.3444, 0.3295, 54.3122, 4.0])),
 (1, DenseVector([9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432])),
 (1, DenseVector([9.2463, 2.0, 0.392, 0.3381, 162.6437, 7.9432]))]
</code></pre>

<p>How do I expand the Dense vector and make it part of a tuple? i.e. I want the above to become:</p>

<pre><code>[(1, 9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432),
 (1, 9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432),
 (0, 5.0, 20.0, 0.3444, 0.3295, 54.3122, 4.0),
 (1, 9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432),
 (1, 9.2463, 2.0, 0.392, 0.3381, 162.6437, 7.9432)]
</code></pre>

<p>Thanks!</p>

<p>Well, since <code>pyspark.ml.linalg.DenseVector</code> (or <code>mllib</code>) is iterbale (provide <code>__len__</code> and <code>__getitem__</code> methods) you can treat it like any other python collections, for example:</p>

<pre><code>def as_tuple(kv):
    """
    &gt;&gt;&gt; as_tuple((1, DenseVector([9.25, 1.0, 0.31, 0.31, 162.37])))
    (1, 9.25, 1.0, 0.31, 0.31, 162.37)
    """
    k, v = kv
    # Use *v.toArray() if you want to support Sparse one as well.
    return (k, *v)
</code></pre>

<p>For Python 2 replace:</p>

<pre><code>(k, *v)
</code></pre>

<p>with:</p>

<pre><code>from itertools import chain

tuple(chain([k], v))
</code></pre>

<p>or:</p>

<pre><code>(k, ) + tuple(v)
</code></pre>

<p>If you want to convert values to Python (not NumPy) scalars use:</p>

<pre><code>v.toArray().tolist()
</code></pre>

<p>in place of <code>v</code>.</p>

python apache-spark pyspark rdd