python, pyspark : get sum of a pyspark dataframe column values
<p>say I have a dataframe like this</p>

<pre><code>name age city
abc   20  A
def   30  B
</code></pre>

<p>i want to add a summary row at the end of the dataframe, so result will be like</p>

<pre><code>name age city
abc   20  A
def   30  B
All   50  All
</code></pre>

<p>So String 'All', I can easily put, but how to get sum(df['age']) ###column object is not iterable</p>

<pre><code>data = spark.createDataFrame([("abc", 20, "A"), ("def", 30, "B")],["name", "age", "city"])
data.printSchema()
#root
 #|-- name: string (nullable = true)
 #|-- age: long (nullable = true)
 #|-- city: string (nullable = true)
res = data.union(spark.createDataFrame([('All',sum(data['age']),'All')], data.columns))  ## TypeError: Column is not iterable
#Even tried with data['age'].sum() and got error.   If i am using [('All',50,'All')], it is doing fine. 
</code></pre>

<p>I  usually work on Pandas dataframe and new to Spark. Might be my undestanding about spark dataframe is not that matured.</p>

<p>Please suggest, how to get the sum over a dataframe-column in pyspark. And if there is any better way to add/append a row to end of a dataframe.
Thanks.</p>

<p>Spark SQL has a dedicated module for column functions <a href="http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#module-pyspark.sql.functions" rel="nofollow"><code>pyspark.sql.functions</code></a>.<br>
So the way it works is:</p>

<pre><code>from pyspark.sql import functions as F
data = spark.createDataFrame([("abc", 20, "A"), ("def", 30, "B")],["name", "age", "city"])

res = data.unionAll(
    data.select([
        F.lit('All').alias('name'), # create a cloumn named 'name' and filled with 'All'
        F.sum(data.age).alias('age'), # get the sum of 'age'
        F.lit('All').alias('city') # create a column named 'city' and filled with 'All'
    ]))
res.show()
</code></pre>

<p>Prints:</p>

<pre><code>+----+---+----+
|name|age|city|
+----+---+----+
| abc| 20|   A|
| def| 30|   B|
| All| 50| All|
+----+---+----+
</code></pre>

python pyspark pyspark-sql