Selectively flattening a nested JSON structure
<p>So this is a problem that I have no idea where to even start so even just a pointer in the right direction would be great.</p>

<p>So I have data that looks like so:</p>

<pre><code>data = {
   "agg": {
      "agg1": [
         {
            "keyWeWant": "*-20.0",
            "asdf": 0,
            "asdf": 20,
            "asdf": 14,
            "some_nested_agg": [
               {
                  "keyWeWant2": 20,
                  "to": 25,
                  "doc_count": 4,
                  "some_nested_agg2": {
                     "count": 7,
                     "min": 2,
                     "max": 5,
                     "keyWeWant3": 2.857142857142857,
                     "sum": 20
                  }
               },
               {
                  "keyWeWant2": 25,
                  "to": 30,
                  "doc_count": 10,
                  "some_nested_agg2": {
                     "count": 16,
                     "min": 2,
                     "max": 10,
                     "keyWeWant3": 6.375,
                     "sum": 102
                  }
               }
            ]
         },
         {
         ...
         },
         {
         ...
         },
         ...
      ]
   }
}
</code></pre>

<p>Now from the example, within 'agg' there are N 'agg1' results, within each 'agg1' result there is a 'keyWeWant'. Each 'agg1' result also has a list of 'some_nested_agg' results which each contain a 'keyWeWant2'. Each 'keyWeWant2' value is associated a single 'keyWeWant' value somewhere up in the hierarchy. Similarly each 'keyWeWant2' also contains a set of results for 'some_nested_agg2' (not a list but rather a map this time). Each of the set of results contains a 'keyWeWant3'.</p>

<p>Now I want to flatten this structure while still preserving the association between 'keyWeWant', 'keyWeWant2', and 'keyWeWant3' (I'm essentially de-normalizing) to get something like so:</p>

<p>What I want the function to look like:</p>

<pre><code>[
   {
      "keyWeWant" : "*-20",
      "keyWeWant2" : 20,
      "keyWeWant3" : 2.857142857142857
   },
   {
      "keyWeWant" : "*-20",
      "keyWeWant2" : 25,
      "keyWeWant3" : 6.375
   },
   {
   ...
   },
   {
   ...
   }
]
</code></pre>

<p>This is an example where there is only depth 3 but there could be arbitrary depth with some nested values being lists and some being arrays/list.</p>

<p>What I would like to do is write a function to take in the keys I want and where to find them, and then go get the keys and denormalize.</p>

<p>Something that looks like:</p>

<pre><code>function_name(data_map, {
   "keyWeWant" : ['agg', 'agg1'],
   "keyWeWant2" : ['agg', 'agg1', 'some_nested_agg'],
   "keyWeWant" : ['agg', 'agg1', 'some_nested_agg', 'some_nested_agg2']
})
</code></pre>

<p>Any ideas? I'm familiar with Java, Clojure, Java-script, and Python and am just looking for a way to solve this that's relatively simple.</p>

<p>Here is a JavaScript (ES6) function you could use:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="true">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>function flatten(data, keys) {
    var key = keys[0];
    if (key in data)
        keys = keys.slice(1);
    var res = keys.length &amp;&amp; Object.keys(data)
        .map( key =&gt; data[key] )
        .filter( val =&gt; Object(val) === val )
        .reduce( (res, val) =&gt; res.concat(flatten(val, keys)), []);
    return !(key in data) ? res
        : (res || [{}]).map ( obj =&gt; Object.assign(obj, { [key]: data[key] }) );
}

// Sample data
var data = {
   "agg": {
      "agg1": [
         {
            "keyWeWant": "*-20.0",
            "asdf": 0,
            "asdf": 20,
            "asdf": 14,
            "some_nested_agg": [
               {
                  "keyWeWant2": 20,
                  "to": 25,
                  "doc_count": 4,
                  "some_nested_agg2": {
                     "count": 7,
                     "min": 2,
                     "max": 5,
                     "keyWeWant3": 2.857142857142857,
                     "sum": 20
                  }
               },
               {
                  "keyWeWant2": 25,
                  "to": 30,
                  "doc_count": 10,
                  "some_nested_agg2": {
                     "count": 16,
                     "min": 2,
                     "max": 10,
                     "keyWeWant3": 6.375,
                     "sum": 102
                  }
               }
            ]
         },
      ]
   }
};

// Flatten it by array of keys
var res = flatten(data, ['keyWeWant', 'keyWeWant2', 'keyWeWant3']);

// Output result
console.log(res);</code></pre>
</div>
</div>
</p>

<h3>Alternative using paths</h3>

<p>As noted in comments, the above code does not use path information; it just looks in all arrays. This could be an issue if the keys being looked for also occur in paths that should be ignored.</p>

<p>The following alternative will use path information, which should be passed as an array of sub-arrays, where each sub-array first lists the path keys, and as last element the value key to be retained:</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>function flatten(data, [path, ...paths]) {
    return path &amp;&amp; (
        Array.isArray(data)
            ? data.reduce( (res, item) =&gt; res.concat(flatten(item, arguments[1])), [] )
            : path[0] in data &amp;&amp; (
                path.length &gt; 1 
                    ? flatten(data[path[0]], [path.slice(1), ...paths])
                    : (flatten(data, paths) || [{}]).map ( 
                        item =&gt; Object.assign(item, { [path[0]]: data[path[0]] }) 
                    )
            )
    );
}

// Sample data
var data = {
   "agg": {
      "agg1": [
         {
            "keyWeWant": "*-20.0",
            "asdf": 0,
            "asdf": 20,
            "asdf": 14,
            "some_nested_agg": [
               {
                  "keyWeWant2": 20,
                  "to": 25,
                  "doc_count": 4,
                  "some_nested_agg2": {
                     "count": 7,
                     "min": 2,
                     "max": 5,
                     "keyWeWant3": 2.857142857142857,
                     "sum": 20
                  }
               },
               {
                  "keyWeWant2": 25,
                  "to": 30,
                  "doc_count": 10,
                  "some_nested_agg2": {
                     "count": 16,
                     "min": 2,
                     "max": 10,
                     "keyWeWant3": 6.375,
                     "sum": 102
                  }
               }
            ]
         },
      ]
   }
};

// Flatten it by array of keys
var res = flatten(data, [
    ['agg', 'agg1', 'keyWeWant'], 
    ['some_nested_agg', 'keyWeWant2'], 
    ['some_nested_agg2', 'keyWeWant3']]);

// Output result
console.log(res);</code></pre>
</div>
</div>
</p>

javascript python json elasticsearch clojure