Word count: 'Column' object is not callable
<pre><code>from pyspark.sql.functions import split, explode

sheshakespeareDF = sqlContext.read.text(fileName).select(removePunctuation(col('value')))

shakespeareDF.show(15, truncate=False)
</code></pre>

<p>The dataframe looks like this:</p>

<p><a href="http://i.stack.imgur.com/3pAy9.png" rel="nofollow"><img src="http://i.stack.imgur.com/3pAy9.png" alt="enter image description here"></a></p>

<pre><code>ss = split(shakespeareDF.sentence," ")
shakeWordsDFa =explode(ss)

shakeWordsDF_S=sqlContext.createDataFrame(shakeWordsDFa,'word')
</code></pre>

<p>Any idea what am I doing wrong? Tip says <code>Column is not iterable</code>.</p>

<p>What should I do? I just want to change <code>shakeWordsDFa</code> to dataframe and rename.  </p>

<p>Note that split() returns an array of Strings, u can't use them directly to create Data Frame (it is 'ss' in your case)</p>

<p>So you explode() them into a column &amp; give an alias to it. 
Select would take your processed column &amp; creates new dataframe.</p>

<pre><code>newDF = (shakespeareDF
          .select(explode(split(shakespeareDF['sentence'], ' ')).alias('word')
                 )
        )
</code></pre>

python apache-spark pyspark