Call a function for a filtered dataframe in pyspark
<p>I am having data such as ,</p>

<p>data</p>

<pre><code>ID          filter
1             A
2             A
3             A
4             A
5             B
6             B
7             B
8             B
</code></pre>

<p>I want to apply a function for the dataframe,</p>

<pre><code>def add(x):
    y = x+1
    return(y)

from pyspark.sql.functions import *                 
from pyspark.sql.functions import udf

ol_val = udf(add, StringType())

data = data.withColumn("sum",ol_val(data.ID))
</code></pre>

<p>this gives an output,</p>

<p>data</p>

<pre><code>ID          filter        sum
1             A            2
2             A            3
3             A            4
4             A            5
5             B            6
6             B            7
7             B            8
8             B            9
</code></pre>

<p><strong>I want to apply this function only when filter = A</strong> and <strong>for the remaining part I want it be NULL</strong>. The output I want here is,</p>

<p>data</p>

<pre><code>ID          filter        sum
1             A            2
2             A            3
3             A            4
4             A            5
5             B            NULL
6             B            NULL
7             B            NULL
8             B            NULL
</code></pre>

<p>Here the value is NULL because it didnt satisfy the condition filter= A. I want the function to be applied only when filter = A.</p>

<p>Can anybody help me in changing the code inorder to get this output in pyspark?</p>

<p>What you need is to use <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=functions#pyspark.sql.functions.when" rel="nofollow">when</a> and <a href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=functions#pyspark.sql.Column.otherwise" rel="nofollow">otherwise</a>. By the way you don't have to create that <code>UDF</code>.</p>

<pre><code>df = sc.parallelize([
    (1, "a"),
    (1, "b"),
    (3, "c")
  ]).toDF(["id", "filter"])

df.select("*", when(col("filter") == lit("a"), col("id") + 1).otherwise(None).alias("result")).show()
</code></pre>

<p>If you really need to call that function, you can simply replace <code>col("id") + 1</code> by     <code>yourUDF(col("id"))</code></p>

python python-2.7 python-3.x apache-spark pyspark