pyspark: TypeError: condition should be string or Column
<p>I am trying to filter an RDD based like below:</p>

<pre><code>spark_df = sc.createDataFrame(pandas_df)
spark_df.filter(lambda r: str(r['target']).startswith('good'))
spark_df.take(5)
</code></pre>

<p>But got the following errors:</p>

<pre><code>TypeErrorTraceback (most recent call last)
&lt;ipython-input-8-86cfb363dd8b&gt; in &lt;module&gt;()
      1 spark_df = sc.createDataFrame(pandas_df)
----&gt; 2 spark_df.filter(lambda r: str(r['target']).startswith('good'))
      3 spark_df.take(5)

/usr/local/spark-latest/python/pyspark/sql/dataframe.py in filter(self, condition)
    904             jdf = self._jdf.filter(condition._jc)
    905         else:
--&gt; 906             raise TypeError("condition should be string or Column")
    907         return DataFrame(jdf, self.sql_ctx)
    908 

TypeError: condition should be string or Column
</code></pre>

<p>Any idea what I missed? Thank you!</p>

<p>I have been through this and have settled to using a UDF:</p>

<pre><code>from pyspark.sql.functions import udf
from pyspark.sql.types import BooleanType

filtered_df = spark_df.filter(udf(lambda target: target.startswith('good'), 
                                  BooleanType())(spark_df.target))
</code></pre>

<p>More readable would be to use a normal function definition instead of the lambda</p>

python pandas pyspark spark-dataframe rdd