pyspark: convert RDD[DenseVector] to dataframe
<p>I have the following rdd:</p>

<p>rdd.take(5) gives me:</p>

<pre><code>[DenseVector([9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699]),
 DenseVector([9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699]),
 DenseVector([5.0, 20.0, 0.3444, 0.3295, 54.3122, 4.0, 4.0, 9.0]),
 DenseVector([9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699]),
 DenseVector([9.2463, 2.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699])]
</code></pre>

<p>I would like to make it a data frame which should look like:</p>

<pre><code>-------------------------------------------------------------------
| features                                                        |
-------------------------------------------------------------------
| [9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699] |
|-----------------------------------------------------------------|
| [9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699] |
|-----------------------------------------------------------------| 
| [5.0, 20.0, 0.3444, 0.3295, 54.3122, 4.0, 4.0, 9.0]             |
|-----------------------------------------------------------------|
| [9.2463, 1.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699] |
|-----------------------------------------------------------------|
| [9.2463, 2.0, 0.392, 0.3381, 162.6437, 7.9432, 8.3397, 11.7699] |
|-----------------------------------------------------------------|
</code></pre>

<p>Is this possible? I tried to use <code>df_new = sqlContext.createDataFrame(rdd,['features'])</code> , but it didn't work. Does anyone have any suggestion? Thanks!</p>

<p>Map to <code>tuples</code> first:</p>

<pre><code>rdd.map(lambda x: (x, )).toDF(["features"])
</code></pre>

<p>Just keep in mind that as of Spark 2.0 there are two different <code>Vector</code> implementation an <code>ml</code> algorithms require <code>pyspark.ml.Vector</code>.</p>

python pyspark spark-dataframe apache-spark-ml