Spark coalesce vs collect, which one is faster?
<p>I am using <code>pyspark</code> to process 50Gb data using AWS EMR with ~15 m4.large cores. </p>

<p>Each row of the data contains some information at a specific time on a day. I am using the following <code>for</code> loop to extract and aggregate information for every hour. Finally I <code>union</code> the data, as I want my result to save in <strong>one</strong> csv file.</p>

<pre><code># daily_df is a empty pyspark DataFrame
for hour in range(24):
    hourly_df = df.filter(hourFilter("Time")).groupby("Animal").agg(mean("weights"), sum("is_male"))
    daily_df = daily_df.union(hourly_df)
</code></pre>

<p>As of my knowledge, I have to perform the following to force the <code>pyspark.sql.Dataframe</code> object to save to 1 csv files (approx 1Mb) instead of 100+ files:</p>

<pre><code>daily_df.coalesce(1).write.csv("some_local.csv")
</code></pre>

<p>It seems it took about 70min to finish this progress, and I am wondering if I can make it faster by using <code>collect()</code> method like?</p>

<pre><code>daily_df_pandas = daily_df.collect()
daily_df_pandas.to_csv("some_local.csv")
</code></pre>

<p>To save as single file these are options</p>

<p>Option 1 : 
<code>coalesce</code>(1) (no shuffle data over network) or <code>repartition</code>(1) or <code>collect</code> may work for small data-sets, but large data-sets it may not perform, as expected.since all data will be moved to one partition on one node </p>

<p>option 1 would be fine if a single executor has more RAM for use than the driver.</p>

<p>Option 2 : 
Other option would be <code>FileUtil.copyMerge()</code>  - to merge the outputs into a single file. </p>

<p>Option 3 : 
after getting part files you can use hdfs <code>getMerge</code> command </p>

<p>Now you have to decide based on your requirements... which one is safer/faster</p>

<p>also, can have look at <a href="http://stackoverflow.com/questions/36723908/dataframe-save-after-join-is-creating-numerous-part-files">Dataframe save after join is creating numerous part files</a></p>

python apache-spark pyspark