PYSPARK : How to work with dataframes?
<p>I have the following dataframes</p>

<pre><code>from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.functions import *

sc = SparkContext()
sql = SQLContext(sc)

df1 = sql.createDataFrame([("Mark", 68), ("John", 59), ("Mary", 49)], ['Name', \
'Weight'])

df2 = sql.createDataFrame([("White", 68), ("Smith", 59), ("Gary", 49)], ['Name', \
    'Weight'])
</code></pre>

<p>Now I want to randomly choose n = 2 (can be any number) pairs from the weight columns and create the following pairs, each pair consist of two unequal weights:</p>

<pre><code>(68, 59)
(49, 68) 
</code></pre>

<p>then I want to choose from df1 only those with weight 68 and 49, and from df2 only those with weight 59 and 68 and create another dataframe:</p>

<pre><code>df3 = sql.createDataFrame([("Mark", 68, "Smith", 59), ("Mary", 49, "White", 68)], ['Name1', \
    'Weight1', 'Name2', 'Weight2'])
</code></pre>

<p>I'm working with big data. Given n, I first need to generate n pairs and then create the final dataframe.</p>

<p>Try:</p>

<pre><code>&gt;&gt;&gt; df1.where(df1['Weight'].between(68, 59)).union(df2.where(df2['Weight'].between(49, 68)))
</code></pre>

python apache-spark pyspark