AttributeError: 'DataFrame' object has no attribute 'map'
<p>I wanted to convert the spark data frame to add using the code below:</p>

<pre><code>from pyspark.mllib.clustering import KMeans
spark_df = sqlContext.createDataFrame(pandas_df)
rdd = spark_df.map(lambda data: Vectors.dense([float(c) for c in data]))
model = KMeans.train(rdd, 2, maxIterations=10, runs=30, initializationMode="random")
</code></pre>

<p>The detailed error message is:</p>

<pre><code>---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
&lt;ipython-input-11-a19a1763d3ac&gt; in &lt;module&gt;()
      1 from pyspark.mllib.clustering import KMeans
      2 spark_df = sqlContext.createDataFrame(pandas_df)
----&gt; 3 rdd = spark_df.map(lambda data: Vectors.dense([float(c) for c in data]))
      4 model = KMeans.train(rdd, 2, maxIterations=10, runs=30, initializationMode="random")

/home/edamame/spark/spark-2.0.0-bin-hadoop2.6/python/pyspark/sql/dataframe.pyc in __getattr__(self, name)
    842         if name not in self.columns:
    843             raise AttributeError(
--&gt; 844                 "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
    845         jc = self._jdf.apply(name)
    846         return Column(jc)

AttributeError: 'DataFrame' object has no attribute 'map'
</code></pre>

<p>Does anyone know what I did wrong here? Thanks!</p>

<p>You can't <code>map</code> a dataframe, but you can convert the dataframe to an RDD and map that by doing <code>spark_df.rdd.map()</code>. Prior to Spark 2.0, <code>spark_df.map</code> would alias to <code>spark_df.rdd.map()</code>. With Spark 2.0, you must explicitly call <code>.rdd</code> first. </p>

python apache-spark pyspark spark-dataframe apache-spark-mllib