Selectively flattening a nested JSON structure
<p>So this is a problem that I have no idea where to even start so even just a pointer in the right direction would be great.</p>

<p>So I have data that looks like so:</p>

<pre><code>data = {
   "agg": {
      "agg1": [
         {
            "keyWeWant": "*-20.0",
            "asdf": 0,
            "asdf": 20,
            "asdf": 14,
            "some_nested_agg": [
               {
                  "keyWeWant2": 20,
                  "to": 25,
                  "doc_count": 4,
                  "some_nested_agg2": {
                     "count": 7,
                     "min": 2,
                     "max": 5,
                     "keyWeWant3": 2.857142857142857,
                     "sum": 20
                  }
               },
               {
                  "keyWeWant2": 25,
                  "to": 30,
                  "doc_count": 10,
                  "some_nested_agg2": {
                     "count": 16,
                     "min": 2,
                     "max": 10,
                     "keyWeWant3": 6.375,
                     "sum": 102
                  }
               }
            ]
         },
         {
         ...
         },
         {
         ...
         },
         ...
      ]
   }
}
</code></pre>

<p>Now from the example, within 'agg' there are N 'agg1' results, within each 'agg1' result there is a 'keyWeWant'. Each 'agg1' result also has a list of 'some_nested_agg' results which each contain a 'keyWeWant2'. Each 'keyWeWant2' value is associated a single 'keyWeWant' value somewhere up in the hierarchy. Similarly each 'keyWeWant2' also contains a set of results for 'some_nested_agg2' (not a list but rather a map this time). Each of the set of results contains a 'keyWeWant3'.</p>

<p>Now I want to flatten this structure while still preserving the association between 'keyWeWant', 'keyWeWant2', and 'keyWeWant3' (I'm essentially de-normalizing) to get something like so:</p>

<p>What I want the function to look like:</p>

<pre><code>[
   {
      "keyWeWant" : "*-20",
      "keyWeWant2" : 20,
      "keyWeWant3" : 2.857142857142857
   },
   {
      "keyWeWant" : "*-20",
      "keyWeWant2" : 25,
      "keyWeWant3" : 6.375
   },
   {
   ...
   },
   {
   ...
   }
]
</code></pre>

<p>This is an example where there is only depth 3 but there could be arbitrary depth with some nested values being lists and some being arrays/list.</p>

<p>What I would like to do is write a function to take in the keys I want and where to find them, and then go get the keys and denormalize.</p>

<p>Something that looks like:</p>

<pre><code>function_name(data_map, {
   "keyWeWant" : ['agg', 'agg1'],
   "keyWeWant2" : ['agg', 'agg1', 'some_nested_agg'],
   "keyWeWant" : ['agg', 'agg1', 'some_nested_agg', 'some_nested_agg2']
})
</code></pre>

<p>Any ideas? I'm familiar with Java, Clojure, Java-script, and Python and am just looking for a way to solve this that's relatively simple.</p>

<p>There is probably a better way to solve this particular problem (using some ElasticSearch library or something), but here's a solution in Clojure using your requested input and output data formats. </p>

<p>I placed this test data in a file called <code>data.json</code>:</p>

<pre><code>{
    "agg": {
        "agg1": [
            {
                "keyWeWant": "*-20.0",
                "asdf": 0,
                "asdf": 20,
                "asdf": 14,
                "some_nested_agg": [
                    {
                        "keyWeWant2": 20,
                        "to": 25,
                        "doc_count": 4,
                        "some_nested_agg2": {
                            "count": 7,
                            "min": 2,
                            "max": 5,
                            "keyWeWant3": 2.857142857142857,
                            "sum": 20
                        }
                    },
                    {
                        "keyWeWant2": 25,
                        "to": 30,
                        "doc_count": 10,
                        "some_nested_agg2": {
                            "count": 16,
                            "min": 2,
                            "max": 10,
                            "keyWeWant3": 6.375,
                            "sum": 102
                        }
                    }]
            }]}
}
</code></pre>

<p>Then <a href="https://github.com/dakrone/cheshire" rel="nofollow">Cheshire JSON library</a> parses the data to a Clojure data structure:</p>

<pre><code>(use '[cheshire.core :as cheshire])

(def my-data (-&gt; "data.json" slurp cheshire/parse-string))
</code></pre>

<p>Next the paths to get are defined as follows:</p>

<pre><code>(def my-data-map
  {"keyWeWant"  ["agg", "agg1"],
   "keyWeWant2" ["agg", "agg1", "some_nested_agg"],
   "keyWeWant3" ["agg", "agg1", "some_nested_agg", "some_nested_agg2"]})
</code></pre>

<p>It is your <code>data_map</code> above without ":", single quotes changed to double quotes and the last "keyWeWant" changed to "keyWeWant3".</p>

<p><code>find-nested</code> below has the semantics of Clojure's <a href="https://clojuredocs.org/clojure.core/get-in" rel="nofollow"><code>get-in</code></a>, only then it works on maps with vectors, and returns all values instead of one. 
When <code>find-nested</code> is given a search vector it finds all values in a nested map where some values can consist of a vector with a list of maps. Every map in the vector is checked.</p>

<pre><code>(defn find-nested
  "Finds all values in a coll consisting of maps and vectors.

  All values are returned in a tree structure:
  i.e, in your problem it returns (20 25) if you call it with
  (find-nested ['agg', 'agg1', 'some_nested_agg', 'keyWeWant2'] 
  my-data).

  Returns nil if not found."
  [ks c]
  (let [k (first ks)]
    (cond (nil? k)    c
          (map? c)    (find-nested (rest ks) (get c k))
          (vector? c) (if-let [e (-&gt; c first (get k))]
                        (if (string? e) e ; do not map over chars in str
                            (map (partial find-nested (rest ks)) e))
                        (find-nested ks (into [] (rest c)))) ; create vec again
          :else       nil)))
</code></pre>

<p><code>find-nested</code> finds the values for a search path:</p>

<pre><code>(find-nested ["agg", "agg1", "some_nested_agg", "keyWeWant2"] my-data) 
; =&gt; (20 25)
</code></pre>

<p>If all the paths towards the "keyWeWant's are mapped over <code>my-data</code> these are the slices of a <code>tree</code>:</p>

<blockquote>
  <p>(*-20.0<br>
   (20                25)<br>
   (2.857142857142857 6.375))</p>
</blockquote>

<p>The structure you ask for (all end results with paths getting there) can be obtained from this <code>tree</code> in <code>function-name</code> like this:</p>

<pre><code>(defn function-name
  "Transforms data d by finding (nested keys) via data-map m in d and 
  flattening the structure."
  [d m]
  (let [tree               (map #(find-nested (conj (second %) (first %)) d) m)
        leaves             (last tree)
        leaf-indices       (range (count leaves))
        results            (for [index leaf-indices]
                             (map (fn [slice]
                                    (if (string? slice)
                                      slice
                                      (loop [node (nth slice index)]
                                        (if node
                                          node
                                          (recur (nth slice (dec index)))))))
                                  tree))
        results-with-paths (mapv #(zipmap (keys m) %) results)
        json               (cheshire/encode results-with-paths)]
    json))
</code></pre>

<p><code>results</code> uses a <a href="https://clojuredocs.org/clojure.core/loop" rel="nofollow"><code>loop</code></a> to step back if a <code>leaf-index</code> is larger than that particular slice. I think it will work out for deeper nested structures as well -if a next slice is always double the size of a previous slice or the same size it should work out -, but I have not tested it.</p>

<p>Calling (<code>function-name</code> <code>my-data</code> <code>my-data-map</code>) leads to a JSON string in your requested format:</p>

<blockquote>
  <p>[{<br>
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"keyWeWant": "<em>-20.0",<br>
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"keyWeWant2": 20,<br>
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"keyWeWant3": 2.857142857142857 }<br>
  &nbsp;{<br>
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"keyWeWant": "</em>-20.0",<br>
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"keyWeWant2" 25,<br>
  &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;"keyWeWant3" 6.375 }]</p>
</blockquote>

<p>/edit
I see you were looking for a relatively simple solution, that this is not. :-) maybe there is one without having it available in a library. I would be glad to find out how it can be simplified.</p>

javascript python json elasticsearch clojure