PySpark - How to output JSON with specific fields?
<p>The JSON format looks like:</p>

<pre><code>{
  "name": "aaa",
  "address": {
    "street": "blv abc",
    "street_num": "122"
  }
}
</code></pre>

<p>I would read the data from parquet files and execute a sql query on them, like finding all those living at street <code>blv abc</code>. But I just want to output the <code>name</code> and <code>address.street</code> as:</p>

<pre><code>{
  "name": "aaa",
  "address": {
    "street": "blv abc"
  }
}
</code></pre>

<p>How can I output only <code>name</code> and <code>address.street</code>?</p>

<p>The DataFrameReader schema might not work for me since I need to execute some SQL query before output which might need to filter on <code>street_num</code>.</p>

<p>My last resort for this kind of unusual data transformation is</p>

<pre><code>from pyspark.sql.types import Row

def transform(row):
    d = row.asDict()  # now in python data types

    del d['address']['street_num']

    return Row(**d)

new = dataframe.rdd.map(transform)
</code></pre>

<p>I suppose you want to remove a lot of fields to keep the records slim, otherwise it doesn't really worth it.</p>

python json apache-spark pyspark