Spark RDD to DataFrame python
<p>I am trying to convert the Spark RDD to a DataFrame. I have seen the documentation and example where the scheme is passed to 
<code>sqlContext.CreateDataFrame(rdd,schema)</code> function. </p>

<p>But I have 38 columns or fields and this will increase further. If I manually give the schema specifying each field information, that it going to be so tedious job.</p>

<p>Is there any other way to specify the schema without knowing the information of the columns prior.</p>

<p>See,</p>

<p>There is two ways of convert an RDD to DF in spark.</p>

<p><code>toDF()</code> and <code>createDataFrame(rdd, schema)</code></p>

<p>I will show you how you can do that dynamically.</p>

<h2>toDF()</h2>

<p>The <code>toDF()</code> command gives you the way to convert an <code>RDD[Row]</code> to a Dataframe. The point is, the object <code>Row()</code> can receive a <code>**kwargs</code> argument. So, there is an easy way to do that.</p>

<pre><code>from pyspark.sql.types import Row

#here you are going to create a function
def f(x):
    d = {}
    for i in range(len(x)):
        d[str(i)] = x[i]
    return d

#Now populate that
df = rdd.map(lambda x: Row(**f(x))).toDF()
</code></pre>

<p>This way you are going to be able to create a dataframe dynamically.</p>

<h1>createDataFrame(rdd, schema)</h1>

<p>Other way to do that is creating a dynamic schema. How?</p>

<p>This way:</p>

<pre><code>from pyspark.sql.types import StructType
from pyspark.sql.types import StructField
from pyspark.sql.types import StringType

schema = StructType([StructField(str(i), StringType(), True) for i in range(32)])

df = sqlContext.createDataFrame(rdd, schema)
</code></pre>

<p>This seccond way is cleanner to do that...</p>

<p>So this is how you can create dataframes dynamically.</p>

python apache-spark pyspark spark-dataframe