pyspark: append/merge PythonRDD to a pyspark dataframe
<p>I am using the following code to create a clustering model, then classify each record to certain cluster:</p>

<pre><code>from pyspark.mllib.clustering import KMeans
from pyspark.mllib.linalg import Vectors

spark_df = sqlContext.createDataFrame(pandas_df)
rdd = spark_df.rdd.map(lambda data: Vectors.dense([float(c) for c in data]))
model = KMeans.train(rdd, 2, maxIterations=10,  initializationMode="random")

result = model.predict(red)
</code></pre>

<p>How do I append the predicted result back to the spark_df as an additional column? Thanks!</p>

<p><code>pyspark.mllib.clustering.KMeansModel</code> is one of rare models that can be used directly inside PySpark transformation so you can simply <code>map</code> with <code>predict</code>:</p>

<pre><code>rdd.map(lambda point: (model.predict(point), point))
</code></pre>

<p>In general case when it is not possible <code>zip</code> is the right tool for the job:</p>

<pre><code>rdd.zip(model.predict(rdd))
</code></pre>

python apache-spark pyspark spark-dataframe apache-spark-mllib