Spark Dataframe-Python
<p>In pandas, I can successfully run the following:</p>

<pre><code>def car(t)
    if t in df_a:
       return df_a[t]/df_b[t]
    else:
       return 0
</code></pre>

<p>But how can I do the exact same thing with spark dataframe?Many thanks!<br>
The data is like this</p>

<pre><code>df_a
a 20
b 40
c 60

df_b
a 80
b 50
e 100
</code></pre>

<p>The result should be 0.25 when input car(a)</p>

<p>First you have to <code>join</code> both dataframes, then you have to <code>filter</code> by the letter you want and <code>select</code> the operation you need.</p>

<pre class="lang-py prettyprint-override"><code>df_a = sc.parallelize([("a", 20), ("b", 40), ("c", 60)]).toDF(["key", "value"])
df_b = sc.parallelize([("a", 80), ("b", 50), ("e", 100)]).toDF(["key", "value"])

def car(c):
  return df_a.join(df_b, on=["key"]).where(df_a["key"] == c).select((df_a["value"] / df_b["value"]).alias("ratio")).head()

car("a")

# Row(ratio=0.25)
</code></pre>

python apache-spark