I am trying to de-serialize the the spark data frame into another data frame as expected below.

Existing Dataframe Data:

Existing Dataframe schema:

Expected Dataframe:

Can anyone help me on this?

You can use the explode function for that.

from pyspark.sql.functions import explode 
df.withColumn("ns2:fileName", explode(df.ns2:fileName))

EDIT

df.withColumn("result", explode(zip($"ns2:fileName", $"ns2:alias"))).select($"result._1".alias("ns2:fileName"), $"result._2".alias("ns2:alias"))

Possible duplicate.