User defined aggregation function for spark dataframe (python)
<p>I have the below spark dataframe, where id is int and attributes is a list of string</p>

<pre><code>id | attributes
1  | ['a','c', 'd']
2  | ['a', 'e']
1  | ['e', 'f']
1  | ['g']
3  | ['a', 'b']
2  | ['e', 'g']
</code></pre>

<p>I need to perform an aggregation, where the attributes lists for each id are concatenated. The results of the aggregation are:</p>

<pre><code>id | concat(attributes)
1  | ['a', 'c', 'd', 'e', 'f', 'g']
2  | ['a', 'e', 'e', 'g']
3  | ['a', 'b']
</code></pre>

<p>Is there a way to achieve this using python?</p>

<p>Thanks.</p>

<p>One way is to create a new frame, using reduceByKey:</p>

<pre><code>&gt;&gt;&gt; df.show()
+---+----------+
| id|attributes|
+---+----------+
|  1| [a, c, d]|
|  2|    [a, e]|
|  1|    [e, f]|
|  1|       [g]|
|  3|    [a, b]|
|  2|    [e, g]|
+---+----------+

&gt;&gt;&gt; custom_list = df.rdd.reduceByKey(lambda x,y:x+y).collect()
&gt;&gt;&gt; new_df = sqlCtx.createDataFrame(custom_list, ["id", "attributes"])
&gt;&gt;&gt; new_df.show()
+---+------------------+
| id|        attributes|
+---+------------------+
|  1|[a, c, d, e, f, g]|
|  2|      [a, e, e, g]|
|  3|            [a, b]|
+---+------------------+
</code></pre>

<blockquote>
  <p>reduceByKey(func, [numTasks]):</p>
  
  <p>When called on a dataset of (K, V)
  pairs, returns a dataset of (K, V) pairs where the values for each key
  are aggregated using the given reduce function func, which must be of
  type (V,V) => V. Like in groupByKey, the number of reduce tasks is
  configurable through an optional second argument.</p>
</blockquote>

python apache-spark spark-dataframe udf