find mean and corr of 10,000 columns in pyspark Dataframe
<p>I have DF with 10K columns and 70Million rows. I want to calculate the mean and corr of 10K columns. I did below code but it wont work due to code size 64K issue (<a href="https://issues.apache.org/jira/browse/SPARK-16845" rel="nofollow">https://issues.apache.org/jira/browse/SPARK-16845</a>)</p>

<p>Data:</p>

<pre><code>region dept week sal val1  val2  val3 ... val10000   
 US    CS   1     1    2    1     1   ...  2 
 US    CS   2     1.5  2    3     1   ...  2
 US    CS   3     1    2    2     2.1      2
 US    ELE  1     1.1  2    2     2.1      2
 US    ELE  2     2.1  2    2     2.1      2
 US    ELE  3     1    2    1     2   .... 2
 UE    CS   1     2    2    1     2   .... 2
</code></pre>

<p>Code:</p>

<pre><code>aggList =  [func.mean(col) for col in df.columns]  #exclude keys
df2= df.groupBy('region', 'dept').agg(*aggList)
</code></pre>

<p>code 2</p>

<pre><code>aggList =  [func.corr('sal', col).alias(col) for col in df.columns]  #exclude keys
df2  = df.groupBy('region', 'dept', 'week').agg(*aggList)
</code></pre>

<p>this fails. Is there any alternative way to overcome this bug? and any one tried DF with 10K columns?. Is there any suggestion on performance improvement?</p>

<p>We also ran into the 64KB issue, but in a where clause, which is filed under another bug report. What we used as a workaround, is simply, to do the operations/transformations in several steps.</p>

<p>In your case, this would mean, that you don't do all the aggregatens in one step. Instead loop over the relevant columns in an outer operation:</p>

<ul>
<li>Use <a href="https://spark.apache.org/docs/1.5.1/api/java/org/apache/spark/sql/DataFrame.html#select(org.apache.spark.sql.Column...)" rel="nofollow"><code>select</code></a> to create a temporary dataframe, which just contains columns you need for the operation.</li>
<li>Use the <code>groupBy</code> and <code>agg</code> like you did, except not for a list of aggregations, but just for on (or two, you could combine the <code>mean</code> and <code>corr</code>.</li>
<li>After you received references to all temporary dataframes, use <a href="https://spark.apache.org/docs/1.5.1/api/java/org/apache/spark/sql/DataFrame.html#withColumn(java.lang.String,%20org.apache.spark.sql.Column)" rel="nofollow"><code>withColumn</code></a> to append the aggregated columns from the temporary dataframes to a result df. </li>
</ul>

<p>Due to the lazy evaluation of a Spark DAG, this is of course slower as doing it in one operation. But it should evaluate the whole analysis in one run.</p>

python apache-spark pyspark spark-dataframe