Pyspark: show histogram of a data frame column
<p>In pandas data frame, I am using the following code to plot histogram of a column:</p>

<pre><code>my_df.hist(column = 'field_1')
</code></pre>

<p>Is there something that can achieve the same goal in pyspark data frame? (I am in Jupyter Notebook) Thanks!</p>

<p>Unfortunately I don't think that there's a clean <code>plot()</code> or <code>hist()</code> function in the PySpark Dataframes API, but I'm hoping that things will eventually go in that direction.</p>

<p>For the time being, you could compute the histogram in Spark, and plot the computed histogram as a bar chart. Example:</p>

<pre><code>import pandas as pd
import pyspark.sql as sparksql

# Let's use UCLA's college admission dataset
file_name = "http://www.ats.ucla.edu/stat/data/binary.csv"

# Creating a pandas dataframe from Sample Data
pandas_df = pd.read_csv(file_name)

sql_context = sparksql.SQLcontext(sc)

# Creating a Spark DataFrame from a pandas dataframe
spark_df = sql_context.createDataFrame(df)

spark_df.show(5)
</code></pre>

<p>This is what the data looks like:</p>

<pre><code>Out[]:    +-----+---+----+----+
          |admit|gre| gpa|rank|
          +-----+---+----+----+
          |    0|380|3.61|   3|
          |    1|660|3.67|   3|
          |    1|800| 4.0|   1|
          |    1|640|3.19|   4|
          |    0|520|2.93|   4|
          +-----+---+----+----+
          only showing top 5 rows


# This is what we want
df.hist('gre');
</code></pre>

<p><a href="http://i.stack.imgur.com/vdVfB.png" rel="nofollow">Histogram when plotted in using df.hist()</a></p>

<pre><code># Doing the heavy lifting in Spark. We could leverage the `histogram` function from the RDD api

gre_histogram = spark_df.select('gre').rdd.flatMap(lambda x: x).histogram(11)

# Loading the Computed Histogram into a Pandas Dataframe for plotting

    pd.DataFrame(zip(list(gre_histogram)[0],
                     list(gre_histogram)[1]),columns=['bin','frequency']).set_index('bin').plot(kind='bar');
</code></pre>

<p><a href="http://i.stack.imgur.com/AMYkM.png" rel="nofollow">Histogram computed by using RDD.histogram()</a></p>

python pyspark spark-dataframe jupyter-notebook