How can i save RDD to a single parquet file?
<p>I work with pyspark 2.0, hadoop 2.7.2.
And here is my code:</p>

<pre><code>def func(df):
    new_df = pd.DataFrame(df['id'])
    new_df['num'] = new_df['num'] * 12
    return new_df

set = sqlContext.read.parquet("data_set.parquet")
columns = set.columns
map_res = set.rdd.mapPartitions(lambda iter_: func(pd.DataFrame(list(iter_), 
                                                   columns=columns)))
</code></pre>

<p>Now, I need to save <strong>map_res</strong> RDD as a parquet file <em>new.parquet</em>. 
Is there any way i can do it without creating a large dataframe before the saving? Or may be there is a possibility of saving each partition of RDD separately and then merge all saved files?</p>

<p>P.s. I want to manage without creating a dataframe due to its realy large size.</p>

<p>I suggest this:</p>

<pre><code>dataframes = []
#creating index
map_res = map_res.zipWithIndex()
# setting index as key
map_res = map_res.map(lambda x: (x[1],x[0]))
# creating one spark df per element
for i in range(0, map_res.count()):
    partial_dataframe_pd  = map_res.lookup(i)
    partial_dataframe = sqlContext.createDataFrame(partial_dataframe_pd)
    dataframes.append(partial_dataframe)
# concatination
result_df = dataframes.pop()
for df in dataframes:
    result_df.union(df)   
#saving
result_df.write.parquet("...")
</code></pre>

<p>If you have small number of partitions (2-100) then it should work rather fast. </p>

python hadoop apache-spark pyspark rdd