Spark coalesce vs collect, which one is faster?
<p>I am using <code>pyspark</code> to process 50Gb data using AWS EMR with ~15 m4.large cores. </p>

<p>Each row of the data contains some information at a specific time on a day. I am using the following <code>for</code> loop to extract and aggregate information for every hour. Finally I <code>union</code> the data, as I want my result to save in <strong>one</strong> csv file.</p>

<pre><code># daily_df is a empty pyspark DataFrame
for hour in range(24):
    hourly_df = df.filter(hourFilter("Time")).groupby("Animal").agg(mean("weights"), sum("is_male"))
    daily_df = daily_df.union(hourly_df)
</code></pre>

<p>As of my knowledge, I have to perform the following to force the <code>pyspark.sql.Dataframe</code> object to save to 1 csv files (approx 1Mb) instead of 100+ files:</p>

<pre><code>daily_df.coalesce(1).write.csv("some_local.csv")
</code></pre>

<p>It seems it took about 70min to finish this progress, and I am wondering if I can make it faster by using <code>collect()</code> method like?</p>

<pre><code>daily_df_pandas = daily_df.collect()
daily_df_pandas.to_csv("some_local.csv")
</code></pre>

<p>Both <code>coalesce(1)</code> and <code>collect</code> are pretty bad in general but with expected output size around 1MB it doesn't really matter. It simply shouldn't be a bottleneck here.</p>

<p>One simple improvement is to drop <code>loop</code> -> <code>filter</code> -> <code>union</code> and perform a single aggregation:</p>

<pre><code>df.groupby(hour("Time"), col("Animal")).agg(mean("weights"), sum("is_male"))
</code></pre>

<p>If that's not enough then most likely the issue here is configuration (the good place to start could be adjusting <code>spark.sql.shuffle.partitions</code> if you don't do that already).</p>

python apache-spark pyspark