explode array of array- (Dataframe) pySpark
<p>I have a dataframe like this:</p>

<pre><code>  +-----+--------------------+
|index|              merged|
+-----+--------------------+
|    0|[[2.5, 2.4], [3.5...|
|    1|[[-1.0, -1.0], [-...|
|    2|[[-1.0, -1.0], [-...|
|    3|[[0.0, 0.0], [0.5...|
|    4|[[0.5, 0.5], [1.0...|
|    5|[[0.5, 0.5], [1.0...|
|    6|[[-1.0, -1.0], [0...|
|    7|[[0.0, 0.0], [0.5...|
|    8|[[0.5, 0.5], [1.0...|
+-----+--------------------+
</code></pre>

<p>And I want to explode the merged column into </p>

<pre><code>+-----+-------+-------+
|index|Column1|Column2|
+-----+-------+-------+
|    0|    2.5|   2.4 |
|    1|    3.5|    0.5|
|    2|   -1.0|   -1.0|
|    3|   -1.0|   -1.0|
|    4|   0.0 |   0.0 |
|    5|    0.5|   0.74|
+-----+-------+-------+
</code></pre>

<p>Each tuple [[2.5, 2.4], [3.5,0,5]] repensente two columns, knowing that 2,5 and 3,5  will be stored in column 1 and (2.4,0,5) will be stored in second Column </p>

<p>So I tried this</p>

<pre><code>df= df.withColumn("merged", df["merged"].cast("array&lt;array&lt;float&gt;&gt;"))
df= df.withColumn("merged",explode('merged'))
</code></pre>

<p>then i will apply a udf to create another DF</p>

<p>but i can't cast the data or apply explode, and I received the error</p>

<pre><code>pyspark.sql.utils.AnalysisException: u"cannot resolve 'cast(merged as array&lt;array&lt;float&gt;)' due to data type mismatch: cannot cast StringType to ArrayType(StringType,true)
</code></pre>

<p>I tried also</p>

<pre><code>df= df.withColumn("merged", df["merged"].cast("array&lt;string&gt;"))
</code></pre>

<p>but nothing works
and if I apply explode without cast, I receive </p>

<pre><code>pyspark.sql.utils.AnalysisException: u"cannot resolve 'explode(merged)' due to data type mismatch: input to function explode should be array or map type, not StringType;
</code></pre>

<p>You could try the following code:</p>

<pre><code>from pyspark import SparkConf, SparkContext                        
from pyspark.sql import SparkSession                               

from pyspark.sql.types import FloatType, StringType, IntegerType   
from pyspark.sql.functions import udf, col                         


def col1_calc(merged):                                             
    return merged[0][0]                                            

def col2_calc(merged):                                             
    return merged[0][1]                                            

if __name__ == '__main__':                                         
    spark = SparkSession \                                         
        .builder \                                                 
        .appName("Python Spark SQL Hive integration example") \    
        .getOrCreate()                                             

    df = spark.createDataFrame([                                   
        (0, [[2.5,2.4],[3.5]]),                                    
        (1, [[-1.0,-1.0],[3.5]]),                                  
        (2, [[-1.0,-1.0],[3.5]]),                                  
    ], ["index", "merged"])                                        

    df.show()                                                      

    column1_calc = udf(col1_calc, FloatType())                     
    df = df.withColumn('Column1', column1_calc(df['merged']))      
    column2_calc = udf(col2_calc, FloatType())                     
    df = df.withColumn('Column2', column2_calc(df['merged']))      

    df = df.select(['Column1', 'Column2', 'index'])                
    df.show()         
</code></pre>

<p>Output:</p>

<pre><code>+-------+-------+-----+
|Column1|Column2|index|
+-------+-------+-----+
|    2.5|    2.4|    0|
|   -1.0|   -1.0|    1|
|   -1.0|   -1.0|    2|
+-------+-------+-----+
</code></pre>

python apache-spark pyspark spark-dataframe