pyspark: TypeError: condition should be string or Column
<p>I am trying to filter an RDD based like below:</p>

<pre><code>spark_df = sc.createDataFrame(pandas_df)
spark_df.filter(lambda r: str(r['target']).startswith('good'))
spark_df.take(5)
</code></pre>

<p>But got the following errors:</p>

<pre><code>TypeErrorTraceback (most recent call last)
&lt;ipython-input-8-86cfb363dd8b&gt; in &lt;module&gt;()
      1 spark_df = sc.createDataFrame(pandas_df)
----&gt; 2 spark_df.filter(lambda r: str(r['target']).startswith('good'))
      3 spark_df.take(5)

/usr/local/spark-latest/python/pyspark/sql/dataframe.py in filter(self, condition)
    904             jdf = self._jdf.filter(condition._jc)
    905         else:
--&gt; 906             raise TypeError("condition should be string or Column")
    907         return DataFrame(jdf, self.sql_ctx)
    908 

TypeError: condition should be string or Column
</code></pre>

<p>Any idea what I missed? Thank you!</p>

<p><code>DataFrame.filter</code>, which is an alias for <code>DataFrame.where</code>, expects a SQL expression expressed either as a <code>Column</code>:</p>

<pre><code>spark_df.filter(col("target").like("good%"))
</code></pre>

<p>or equivalent SQL string:</p>

<pre><code>spark_df.filter("target LIKE 'good%'")
</code></pre>

<p>I believe you're trying here to use <code>RDD.filter</code> which is completely different method:</p>

<pre><code>spark_df.rdd.filter(lambda r: r['target'].startswith('good'))
</code></pre>

<p>and does not benefit from SQL optimizations.</p>

python pandas pyspark spark-dataframe rdd