Pyspark ml can't fit the model and always "AttributeError: 'PipelinedRDD' object has no attribute '_jdf'
<pre><code>data = sqlContext.sql("select a.churn,b.pay_amount,c.all_balance from db_bi.t_cust_churn a left join db_bi.t_cust_pay b on a.cust_id=b.cust_id left join db_bi.t_cust_balance c on a.cust_id=c.cust_id limit 5000").cache()

def labelData(df):
    return df.map(lambda row: LabeledPoint(row[0], row[1:]))
traindata = labelData(data) --this step works well.
from pyspark.ml.classification import LogisticRegression   
lr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)
lrModel = lr.fit(lrdata)
lrModel = lr.fit(lrdata)
</code></pre>

<hr>

<pre><code>AttributeError                            Traceback (most recent call last)
&lt;ipython-input-40-b84a106121e6&gt; in &lt;module&gt;()
----&gt; 1 lrModel = lr.fit(lrdata)

/home/hadoop/spark/python/pyspark/ml/pipeline.pyc in fit(self, dataset, params)
     67                 return self.copy(params)._fit(dataset)
     68             else:
---&gt; 69                 return self._fit(dataset)
     70         else:
     71             raise ValueError("Params must be either a param map or a list/tuple of param maps, "

/home/hadoop/spark/python/pyspark/ml/wrapper.pyc in _fit(self, dataset)
    131 
    132     def _fit(self, dataset):
--&gt; 133         java_model = self._fit_java(dataset)
    134         return self._create_model(java_model)
    135 

/home/hadoop/spark/python/pyspark/ml/wrapper.pyc in _fit_java(self, dataset)
    128         """
    129         self._transfer_params_to_java()
--&gt; 130         return self._java_obj.fit(dataset._jdf)
    131 
    132     def _fit(self, dataset):

AttributeError: 'PipelinedRDD' object has no attribute '_jdf'   
</code></pre>

<p>I guess you are using the tutorial for the latest spark version <a href="https://spark.apache.org/docs/latest/ml-classification-regression.html" rel="nofollow">(2.0.1)</a> with 
<code>pyspark.ml.classification import LogisticRegression</code> whereas you need some other version, e.g. <a href="https://spark.apache.org/docs/1.6.2/mllib-linear-methods.html" rel="nofollow">1.6.2</a> with <code>pyspark.mllib.classification import LogisticRegressionWithLBFGS, LogisticRegressionModel</code>. Note the different libraries.</p>

python apache-spark pyspark apache-spark-mllib