How can i save RDD to a single parquet file?
<p>I work with pyspark 2.0, hadoop 2.7.2.
And here is my code:</p>

<pre><code>def func(df):
    new_df = pd.DataFrame(df['id'])
    new_df['num'] = new_df['num'] * 12
    return new_df

set = sqlContext.read.parquet("data_set.parquet")
columns = set.columns
map_res = set.rdd.mapPartitions(lambda iter_: func(pd.DataFrame(list(iter_), 
                                                   columns=columns)))
</code></pre>

<p>Now, I need to save <strong>map_res</strong> RDD as a parquet file <em>new.parquet</em>. 
Is there any way i can do it without creating a large dataframe before the saving? Or may be there is a possibility of saving each partition of RDD separately and then merge all saved files?</p>

<p>P.s. I want to manage without creating a dataframe due to its realy large size.</p>

<p>There are only 2 ways to do this: </p>

<p>One is use <code>"coalesce(1)"</code>
This will make sure that all the data is saved into 1 file rather than multiple files (200 is the spark default no of partitions) use <code>dataframe.write.save("/this/is/path")</code>. </p>

<p>The other option is write the output to a hive table and then use <code>hive -e "select * from table" &gt; data.tsv</code> which will be tab separated.</p>

python hadoop apache-spark pyspark rdd