pyspark: convert a bytearray field to string in data frame
<p>I am reading a parquet file to a dataframe:</p>

<pre><code>my_df = sqlContext.read.parquet('hdfs://my_server/user/hive/warehouse/my_db.db/my_table')
</code></pre>

<p>if I do:</p>

<pre><code>my_df.head()
</code></pre>

<p>I got:</p>

<pre><code>Row(id=bytearray(b'00000000000000000000000000000000'), numcores=8, ...)
</code></pre>

<p>and when I do </p>

<pre><code>my_df.show()
</code></pre>

<p>the first field id looks like:</p>

<pre><code>[30 30 30 30 30 3...]
</code></pre>

<p>How do I change the bytearray field and make it show as a string output? Thanks!</p>

<p>If you mean to change what <code>head()</code> returns to you, that's not gonna happen, since the <a href="https://spark.apache.org/docs/1.6.2/api/python/pyspark.sql.html#pyspark.sql.DataFrame.head" rel="nofollow">prototype</a> doesn't provide any such functionality:</p>

<blockquote>
  <p>head(n=None)</p>
</blockquote>

python apache-spark pyspark spark-dataframe