Word count: 'Column' object is not callable
<pre><code>from pyspark.sql.functions import split, explode

sheshakespeareDF = sqlContext.read.text(fileName).select(removePunctuation(col('value')))

shakespeareDF.show(15, truncate=False)
</code></pre>

<p>The dataframe looks like this:</p>

<p><a href="http://i.stack.imgur.com/3pAy9.png" rel="nofollow"><img src="http://i.stack.imgur.com/3pAy9.png" alt="enter image description here"></a></p>

<pre><code>ss = split(shakespeareDF.sentence," ")
shakeWordsDFa =explode(ss)

shakeWordsDF_S=sqlContext.createDataFrame(shakeWordsDFa,'word')
</code></pre>

<p>Any idea what am I doing wrong? Tip says <code>Column is not iterable</code>.</p>

<p>What should I do? I just want to change <code>shakeWordsDFa</code> to dataframe and rename.  </p>

<p>Just use select:</p>

<pre><code>shakespeareDF = sc.parallelize([
    ("from fairest creatures we desire increase", ),
    ("that thereby beautys rose might never die", ),
]).toDF(["sentence"])

(shakespeareDF
    .select(explode(split("sentence", " ")).alias("word"))
    .show(4))

## +---------+
## |     word|
## +---------+
## |     from|
## |  fairest|
## |creatures|
## |       we|
## +---------+
## only showing top 4 rows
</code></pre>

<p>Spark SQL columns are not data structures. There are not bound to a data and are meaningful only when evaluated in a context of a specific <code>DataFrame</code>. This way <code>Columns</code> behave more like functions. </p>

python apache-spark pyspark