Deep Kalman Filters Can Filter

Blanka Hovarth<EMAIL_ADDRESS>
Department of Mathematics, University of Oxford

Oxford-Man Institute for Quantitative Finance
Anastasis<EMAIL_ADDRESS>
Department of Mathematics, McMaster University

Vector Institute
Yannick Limmer<EMAIL_ADDRESS>
Department of Mathematics, University of Oxford

Oxford-Man Institute for Quantitative Finance
Xuwei Yang<EMAIL_ADDRESS>
Department of Mathematics, McMaster University

Deep Kalman filters (DKFs) are a class of neural network models that generate Gaussian probability measures from sequential data.  Though DKFs are inspired by the Kalman filter, they lack concrete theoretical ties to the stochastic filtering problem, thus limiting their applicability to areas where traditional model-based filters have been used, e.g. model calibration for bond and option prices in mathematical finance.  We address this issue in the mathematical foundations of deep learning by exhibiting a class of continuous-time DKFs which can approximately implement the conditional law of a broad class of non-Markovian and conditionally Gaussian signal processes given noisy continuous-times measurements.  Our approximation results hold uniformly over sufficiently regular compact subsets of paths, where the approximation error is quantified by the worst-case 2-Wasserstein distance computed uniformly over the given compact set of paths.

§ INTRODUCTION

In several scientific problems, e.g. in mathematical finance, one is interested in predicting an unobservable signal process $X_{\cdot}$ given the information available in noisy observations $Y_{\cdot}$. Here both of the processes are often assumed to have continuous paths and are defined on a filtered probability space $(\Omega,\mathcal{F},(\mathcal{F}_t)_{t\in [0:T]},\mathbb{P})$.  For example, market sentiment(signal) impacts market prices(observations), however, only the latter of these two can be directly measured.  The objective of the robust stochastic filtering problem, studied by <cit.>, is to identify a continuous function $f_t:C([0:t],\mathbb{R}^{d_Y})\rightarrow \mathcal{P}(\mathbb{R}^{d_X})$, with $d_X, d_Y \in \N$ for $t>0$, satisfying

\begin{align}
\label{goal:Robust_Filtering}
\mathbb{P}(X_t\in \cdot|\mathcal{F}_t^Y)
\end{align}

where $\mathbb{P}(X_t\in \cdot|\mathcal{F}_t^Y)$ is the conditional law of the $d_X$-dimensional signal process $X_t$ given the $\sigma$-algebra generated by $(Y_s)_{s\in [0:t]}$. The key innovation in (<ref>) is the continuity, and the uniqueness, of $f_t$.  In contrast, a Borel $f_t$ satisfying (<ref>) exists by elementary measure-theoretic <cit.>.
The continuity of $f_t$ is qualified by equipping the set of Borel probability measures on $\mathbb{R}^{d_X}$, denoted by $\mathcal{P}(\mathbb{R}^{d_X})$, with the weak topology and by equipping the set of continuous paths from $[0:t]$ to $\mathbb{R}^{d_Y}$, denoted by $C([0:t],\mathbb{R}^{d_Y})$, with the uniform norm.

When one has access to a robust representation (<ref>), then they can reliably predict the conditional law of $X_t$ even subject to imperfections on the observed historical data in $y_{[0:t]}\in C([0:t],\mathbb{R}^{d_Y})$.  These robust representations are particularly invaluable in mathematical finance, where continuous streams of data are often noisy.  More broadly, stochastic filters are indispensable in situations where latent parameters influence or obscure market factors.  Applications include the computation of optimal investment under partial information <cit.>, the estimation of volatility from observed intra-day stock prices <cit.>, estimation of interest rates <cit.>, estimation of spot price estimation for commodity futures <cit.>, hedging of credit derivatives <cit.>, estimation of equilibria under asymmetric information <cit.>, and calibration of option pricing models <cit.>.  Pathwise, or so-called robust, formulations were later derived in <cit.>.  Indeed, the abilities of stochastic filters to estimate unobservable variables, track market dynamics, and improve the precision of financial models have made them a staple tool in the quantitative finance toolbox, with several books treating the role of stochastic filtering in finance <cit.>.

In its general form, the stochastic filtering problem is a well-understood mathematical problem.   Many of the fundamental questions in stochastic filtering, such as existence and dynamics for the evolution of the conditional law of signal process $X_t$ given the realizations of the measurement process $Y_{\cdot}$, were solved in a series of papers <cit.>.  Nevertheless, the infinite-dimensionality of general stochastic filters, being measure-valued path-dependent processes, make the problem computationally intractable.

The exception to this rule is so-called finite-dimensional filters. These encapsulate rare situations in which the dynamics of $X_{\cdot}$ and $Y_{\cdot}$ give way to optimal filters that are finitely parameterized.  Examples include the Kalman-Bucy filters <cit.> where closed-forms are derived under the assumption of Markovian Gaussian noise and affine OU-type dynamics of $X_{\cdot}$ and $Y_{\cdot}$, the <cit.> filter where all involved quantitative are finite-state Markov processes, and the Beňes filter <cit.> which relies a particular set of one-dimensional dynamics.

The computational intractability of the general filtering problem leads to the use of approximately optimal filters.  These approaches include particle filters which aim at dynamically approximating optimal filters using an evolving interacting particle system <cit.>, linear relaxations of the optimal filtering functional for affine processes which can be numerically computed by solving specific stochastic Riccati equations <cit.>, or by projection of the optimal filter onto finite-dimensional manifolds of exponential families <cit.> where it can be tracked using finitely parameterized representations.  The latter two approaches work well if the coupled system $(X_{\cdot},Y_{\cdot})$ follows the postulated affine dynamics, and the former particle-filtering approach is well-suited to low-dimensional settings. Nevertheless, one would like to have access to (approximate) optimal finite dimensional filters for a broader range of situations.

One possibility could be via deep learning, since deep neural networks have demonstrated that they can efficiently solve a variety of previously intractable numerical problems. The expressivity of neural networks on classical learning problems has inspired the use of deep learning approaches to the numerical stochastic filtering problems; e.g. <cit.>, and the development of measure-valued deep learning models; e.g. <cit.>.  At the forefront of these methods, are the class of deep Kalman filters (DKFs) <cit.>, these are the primary object of study in this paper.  Broadly speaking, a DKF is a deep neural network model which maps historical data to Gaussian measures.  Though these deep learning-powered models are inspired by Kalman filters, there is still little connection between them and the stochastic filtering problem (<ref>).

This paper constructs a DKF model which can uniformly approximate the map $f_t$ in the robust representation (<ref>), to arbitrary precision, for any pair of non-Markovian $\mathbb{F}\eqdef (\mathcal{F}_s)_{s\in [0:T]}$-adapted conditionally Gaussian stochastic processes studied by <cit.>.

[The DKF is a Universal Conditionally-Gaussian Filter]

Let $(X_{\cdot},Y_{\cdot})$ be the partially-observed system (<ref>)-(<ref>) below, subject to mild regularity conditions.  For any suitable compact subset $K\subseteq C([0:T],\mathbb{R}^{d_Y})$, every $T>0$, and any approximation error $\varepsilon>0$ there exists a DKF model $\hat{F}:[0:T]\times C([0:T]\times\mathbb{R}^{d_Y})\rightarrow\mathcal{N}_{d_X}$ satisfying the uniform estimate

\[
\max_{0\le t\le T,\,y_{\cdot}\in K}\,
\mathcal{W}_p\big(
% f^{\star}(y_{\cdot})
\mathbb{P}(X_t \in \cdot \vert y_{[0:t]})
\hat{F}(t,y_{\cdot})
\big)
\varepsilon
\]

for every $1\le p\le 2$.

The DKF model in Informal Theorem <ref> has two inputs: a time $t$ and a continuous path $y_{\cdot}$ defined up to some time $T\ge t$.  The first parameter $t$ acts as a mask hiding the future evolution of the observation (input) path $y_{\cdot}$ beyond the current time $t$.  We note that there is no loss of generality in assuming that the path $y_{\cdot}$ is defined up to time $T$, since any path can be trivially extended beyond the current time $t$ by setting $y_{s}=y_t$ for all $s\in [t,T]$; e.g. as in the Functional Itô Calculus <cit.>.

The mask, i.e. the parameter $t$, fills the same analogous role to recursions in classical stochastic filters.  That is, by varying the time parameter $t$, the prediction of the DKFs evolves into the future without having to retrain the DKF model.

Our uniform estimates hold for coupled stochastic differential equations governed by

\begin{align}
\label{eq:filtering_dynamics__signal}
% \tag{Sig}
dX_t & = [a_0(t,Y_{[0:t]}) + a_1(t,Y_{[0:t]}) X_t]dt + \sum_{i=1}^2\, b_i(t,Y_{[0:t]})dW^{(i)}_t \\
\label{eq:filtering_dynamics__observation}
% \tag{Obs}
dY_t & = [A_0(t,Y_{[0:t]}) + A_1(t,Y_{[0:t]}) X_t]dt + \sum_{i=1}^2\, B_i(t,Y_{[0:t]})dW^{(i)}_t
\end{align}

where $a_0$ and $A_0$ respectively take values in $\mathbb{R}^{d_X}$ and $\mathbb{R}^{d_Y}$, and where $a_1,A_1,b_1,b_2,B_1,B_2$ are matrix-valued of respective dimensions $d_X\times d_X$, $d_Y\times d_X$, $d_X\times d_X$, $d_X\times d_Y$, $d_Y\times d_X$, and $d_Y\times d_Y$, and the entries of $a_0, a_1, b_0, b_1, A_0, A_1, B_0, B_1$ are measurable nonanticipative functionals on the measurable space $\smash{([0:T] \times C([0:T], \R^{d_Y}), \cB_{[0:T]} \times \cB_T^{d_Y})}$, and where $\smash{\cB_t^d  \eqdef  \sigma(C([0: t], \R^d))}$ denotes the $\sigma$-algebra generated by continuous paths on $[0: t]$ to $\R^d$.
We will always assume that the filtration $\mathbb{F}$ is right-continuous and that the probability space $(\Omega,\mathcal{F},\mathbb{F},\mathbb{P})$ supports independent $\mathbb{F}$-adapted Brownian motions, $\smash{W^{(1)}_{\cdot}\eqdef (W_t^{(1)})_{0\le t\le T}}$ and $\smash{W^{(2)}_{\cdot}\eqdef (W^{(2)}_t)_{0\le t\le T}}$, of respective dimensions $d_X$ and $d_Y$, for positive integers $d_X$ and $d_Y$. The requirements for the dynamics eq:filtering_dynamics__signal,eq:filtering_dynamics__observation are detailed in <Ref> below.

§.§ The Deep Kalman Filter Model

The proposed deep Kalman filter (DKF), illustrated in <Ref>, generates predictions on the finite-dimensional metric space $(\mathcal{N}_{d_X},\mathcal{W}_2)$ whose points are non-singular Gaussian measures, and whose distance between points is quantified by the $2$-Wasserstein distance.  These predictions are generated from the continuous input paths via the following three-phase process.

Architecture of a Deep Kalman Filter which can approximate the conditional law of the signal process $X_{\cdot}$ given source paths, possibly (but not necessarily) taken by the observation process.

* The observed infinite-dimensional continuous path is encoded into a finite-length real vector.  Surprisingly, our proposed encoding layer, coined the pathwise attention mechanism, adaptively implements a stable and lossless encoding for a broad class of compact subsets of $K$ paths in $C([0:T],\mathbb{R}^{d_Y})$.  By losslessness, we mean that it is injective for a broad class of compact subsets of $K$ include classes of piecewise linear paths, any finite (training) set, any $K$ which is isometric to a closed Riemannian manifold.  By stability, we mean it in the sense of constructive approximation theory, e.g. <cit.>, namely that the map encoding layer is Lipschitz and so is its inverse.  Stability is desirable since it implies that minor numerical errors, e.g. rounding, do not lead to drastically different downstream predictions.  Finally, by adaptivity we mean that the encoding layer is is devoid of any projection onto a finite-dimensional (Schauder) basis, and its parameters can be chosen to suit the specific geometry of the compact set of paths $K$ on which the approximation of the optimal filter is to be performed.

* In the next phase, the DKF processes the vectors encoding the observed paths and generates “deep features” which are then passed along to the output layer for prediction generation.  This phase leverages the efficient approximation capacity of MLPs, e.g. <cit.>, to flexibly process the encoded features in a task-specific manner, by adaptive to specific dynamics of the coupled system (<ref>)-(<ref>).

* In its final processing phase, the DKF decodes the “deep features” generated by the MLP to $\mathcal{N}_{d_X}$-valued predictions, which are then used to approximately implement the predictions of the optimal filter (<ref>).  This decoder is a modified instance of the geometric attention mechanism of <cit.>.  Intuitively, this layer translates Euclidean data to a point in a generalized geodesic convex hull of a finite number containing the image of $K$ under the optimal filter.

§.§ Organization of Paper

<Ref> formalizes the basic required regularity conditions on the coupled system (<ref>)-(<ref>) and it overviews any background material.  <Ref> rigorously introduces the relevant DKF model.  <Ref> contains the paper's main theoretical results, as well as an overview of the proof methodology.  Several results of independent interest, both concerning the approximation capacity of the DKF model and the local Lipschitz regularity of the robust filtering map (<ref>) are discussed.  Detailed proofs of our results are relegated to <Ref>.

§ NOTATION

This section serves as a reference, which records the notation used throughout our manuscript.  In what follows, $N,I,J\in \mathbb{N}_+$, $A$ is an arbitrary $I\times J$ matrix, $x,x_1,\dots,x_J\in \mathbb{R}^N$, $f$ is an arbitrary real-valued function on $\mathbb{R}$, and $t\in \mathbb{R}$.

* Componentwise Composition: $\smash{f\bullet x\eqdef (f(x_n))_{n=1}^N}$ for any $N\in \mathbb{N}_+$.

* Rectified Linear Unit (ReLU): $\operatorname{ReLU}:\mathbb{R}\rightarrow\mathbb{R}$ given by $\smash{\operatorname{ReLU}(t)\eqdef \max\{0,t\}}$.

* Rowwise Product: $\smash{v\odot X\eqdef (v_i\,X_{i,j})_{i=1,\dots,N,\,j=1,\dots,d}}$.

* Softmax Function: $\smash{\operatorname{Softmax}(x)\eqdef (e^{x_n}/\sum_{i=1}^N\,e^{x_i})_{n=1}^N}$.

* Sparsity: $\smash{\|(A_{i,j})_{i,j=1}^{I,J}\|_0\eqdef \#\{A_{i,j}\neq 0:\,i,j=1,\dots,I\}}$.

* Vector Concatenation: $\smash{\oplus_{j=1}^N x_j\eqdef (x_1,\dots,x_J)^{\top}}$ is the $J\times N$-matrix whose $j^{th}$ row is $x_j$.

* Vectorization: $\smash{\operatorname{vec}((A_{i,j})_{i,j=1}^{I,J})\eqdef (A_{1,1},\dots,,A_{I,1},\dots,A_{1, J}, \dots, A_{I,J})}$.

* Euclidean Norm:
$\smash{\|A\| = (\sum_{i,j=1}^{I,J} |A_{i,j}|^2 )^{1/2}}$.

* Supremum Norm:
$\smash{\|A\|_\infty=\sup \{ A_{i,j} : \, i=1,..., I, j=1, ..., J \}}$.

§ THE SETTING

This section formalizes the setting in which our analysis takes place.  We begin by specifying the dynamics of the stochastic processes in the filtering problem.  Next, the source space of paths considered is introduced.  Subsequently, we formulate the target space of probability measures wherein the optimal filter lies.

§.§ Regularity Conditions on the Partially-Observed System

We henceforth maintain the following basic regularity assumptions on the dynamics of the coupled system (<ref>)-(<ref>).

[Regularity Conditions: Dynamics]

We will assume the following uniform bounds:

\begin{align*}
\vert \process{a_1}{t, y}{i, j}\vert \le L
\mbox{ and }
\vert \process{A_1}{t, y}{k, j}\vert \le L
\end{align*}

for $y \in C([0:t], \R^{d_Y})$, $t \in [0:T]$, $i,j=1,\dots,d_X$, and $k=1,\dots,d_Y$.
Further, we require the following integrability conditions:[In particular, the integrability conditions imply that $\bE\big[\sum_{j = 1}^{d_X}(X_0)_j^4\big] < \infty$.]

* $\int_0^T \bE[\process{a_0}{t, Y_{[0:t]}}{i}^4 + \process{b_1}{t, Y_{[0:t]}}{ij}^4 + \process{b_2}{t, Y_{[0:t]}}{ij}^4] dt < \infty$,

* $\int_0^T \bE[\process{b_2}{t, Y_{[0:t]}}{ij}^4] dt < \infty$,

hold for $i,j=1,\dots,d_X$, and $k,l=1,\dots,d_Y$ and all paths $y_{[0:T]} \in C([0:T], \R^{d_Y})$. We define

\begin{align*}
B \circ B \eqdef B_1B_1^\top + B_2B_2^\top, \quad
b \circ B \eqdef b_1B_1^\top + b_2B_2^\top, \quad
b \circ b \eqdef b_1b_1^\top + b_2b_2^\top.
\end{align*}

and require that the matrix $B \circ B$ is uniformly non-singular, that is, its inverse is uniformly bounded; there are constants $L_1, L_2 \in \R$ as well as a non-decreasing right-continuous function $K: [0:T] \to [0:1]$ such that for every $x, y \in C([0:T], \R^{d_Y})$ and every $k=1,2$, and $i,j=1,\dots,d$ it holds for all $0 \leq t\leq T$ that

* $\vert (B_k)_{i,j}(t, x) - (B_k)_{i,j}(t, y) \vert^2
\le
L_1 \int_0^{t} \vert x_s - y_s \vert^2 dK(s) + L_2 \vert x_t - y_t \vert^2$,

* $(B_k)_{i,j}(t, x)^2
\le
L_1 \int_0^t \vert (1 + \vert x_s \vert^2 ) dK(s) + L_2 (1 +\vert x_t \vert^2)$,

* $\int_0^T \bE[\vert \process{A_1}{(t, Y_{[0:t]})}{i, j}(X_t)_j \vert] dt < \infty$,

* $\bE[\vert (X_t)_j \vert] < \infty, \quad \forall t \in [0:T]$,

* $\bP\big( \int_0^T (\process{A_1}{t, Y_{[0:t]}}{i, j} \bE[(X_t)_j \vert \cY_t])^2dt  < \infty \big) = 1$,

for indices $i=1,\dots,d_Y$ and $j=1,\dots,d_X$, where $\vert \cdot \vert$ is the Euclidean distance on $\R^{d_Y}$.

We further impose the following assumptions on the dynamics (<ref>), (<ref>):

* Local Lipschitz continuity in the path component uniformly in time[
By local Lipschitz continuity uniformly in time we mean that there exists a constant that is a local Lipschitz-constant for the path at all times $t \in [0,T]$.
], as well as global Lipschitz continuity in the time component of $a_0$, $a_1$, $b \circ b$, $b \circ B$, $ A_0$, $A_1$, $(b \circ B)$, $(B \circ B)^{-1}$ with respect to the $l^2$-norm, or Frobenius norm if matrix-valued.

* Positive semi-definiteness of $(b \circ b)(t, y_{[0:t]})- (b \circ B)(B \circ B)(b \circ B)^\top(t, y_{[0:t]})$ for all times $t \in [0:T]$ and paths $y_{[0:T]} \in C^1([0:T], \R^{d_X})$.

*  Let $G_t(y_{[0:t]})$ be a solution of $\partial_tG_t(y_{[0:t]}) = \Tilde{a}_1(t, y_{[0:t]})G_t(y_{[0:t]})$ with $G_0(y_{[0:t]}) = I_{d_X}$ for any path $y_{[0:T]} \in C^1([0:T], \R^{d_X})$, where

\begin{align*}
\Tilde{a}_1^\top(t, y_{[0:t]}) \eqdef a_1(t, y_{[0:t]}) - (b \circ B)(B \circ B)A_1(t, y_{[0:t]}).
\end{align*}

There exist constants $K_1, K_2 > 0$ s.t. uniformly $K_1 \leq \tr(G_t(y_{[0:t]})) \leq K_2$ as well as $K_3 > 0$ s.t. $\tr(A_1^\top(B \circ B)^{-1}A_1(t, y_{[0:t]})) \leq K_3$.

Eventually we require for the initial conditions that

* the conditional distribution of $X_0$ given $Y_0$ is normal,

* the covariance matrix $\Sigma_0 := \mathbb{E}[(X_0 - \mu_0)(X_0 - \mu_0)^\intercal \vert Y_0]$ is positive definite, where $\mu_0 := \mathbb{E}[X_0 \vert Y_0]$, and that

*  Lipschitz continuity of $\mu_0, \Sigma_0$ in $Y_0$ w.r.t. the $\Vert \cdot \Vert_2$-norm holds.

§.§ Source Spaces: Regular Compact Subsets of Path Space

Fix a time-horizon $T>0$.  For every $0<t\le T$, let $C([0:t],\mathbb{R}^{d_Y})$ denote the set of continuous “paths”, i.e. functions, from $[0:t]$ to the $d$-dimensional Euclidean space $\mathbb{R}^{d_Y}$.  We equip this set with the uniform norm, making it a Banach space, where the norm of a path $y_{\cdot}\in C([0:t],\mathbb{R}^{d_Y})$ is defined by

\[
\|y\|_t
\eqdef
\max_{0\le s\le t}\, \|y_s\|_2,
\]

where $\|\cdot\|_2$ is the Euclidean norm on $\mathbb{R}^{d_Y}$.

For every $0<t\le T$, similarly to the Horizontal Extension of a path in the Functional Itô calculus <cit.>, we may canonically embed any path $y\in C([0:t],\mathbb{R}^{d_Y})$ into a path $\bar{y} \in C([0:T],\mathbb{R}^{d_Y})$ via by extending the “frozen version after time $t$” defined by

\[
\bar{y}(s)\eqdef \begin{cases}
y(s) &: \mbox{ if } 0\le s \le t\\
y(t) & : \mbox{ if } t<s\le T.
\end{cases}
\]

Moreover, note that the map $\bar{\cdot}: C([0:t],\mathbb{R}^{d_Y}) \to C([0:T],\mathbb{R}^{d_Y})$ is a linear isometric embedding of the Banach spaces, since

\[
\|\bar{y}\|_T = \max_{0\le s\le T}\, \|y(s)\|_2
= \max_{0\le s\le t}\, \|y(s)\|_2 = \|y\|_t,
\]

since $y(s)=y(t)$ for all $t<s\le T$.  Conversely, the restriction $\bar{y}_{[0:t]}$ of a path $\bar{y}\in C([0:T],\mathbb{R}^{d_Y})$ to any shorter time-interval $[0:t]$ for $0\le t\le T$ defines a non-expansive linear operator from $C([0:T],\mathbb{R}^{d_Y})$ to $C([0:t],\mathbb{R}^{d_Y})$ since

\[
\|\bar{y}_{[0:t]}\|_t
\max_{0\le s\le t}\, \|\bar{y}_{[0:t]}(s)\|_2
\le
\max_{0\le s\le T}\, \|\bar{y}_{[0:t]}(s)\|_2
\max_{0\le s\le T}\, \|\bar{y}(s)\|_2
\|\bar{y}\|_T.
\]

Therefore, in what follows, we will always consider the domain of our path space to be the Banach space $C([0:T],\mathbb{R}^{d_Y})$.
As show in <cit.>, even in finite dimensions, there are rather regular functions which cannot be approximated by standard deep neural networks parameterized determined by a number of trainable parameters which is a polynomial in the reciprocal approximation error.  In other words, the curse of dimensionality is generally unavoidable when approximating rather regular functions between finite-dimensional Banach spaces, and thus the problem can only be exacerbated in finite-dimensions; see <cit.>.

The curse of dimensionality cannot be broken, but can typically be avoided either by restricting the class of functions being approximated, e.g. in <cit.>, or the regularity of the compact subsets of the input space on which the uniform approximation is to hold; e.g. in <cit.>.  Recent advances in infinite-dimensional approximation of functions not taking values in a linear space – as is the case in our results – show that such restrictions are a sufficient requirement for obtaining universality, see <cit.>. However, the necessity of such restrictions on compact subsets is still unknown.  The explicit effects of restricting the (fractal) dimension and diameter of compact sets of which a deep neural network approximation is to hold is explicitly studied in <cit.>.

In this paper, we adopt the second approach, since we cannot restrict the function class which is determined by the stochastic filtering problem.
We therefore restrict to compact subsets $K$ of the input space which are regular, in that they are either isometric to some compact Riemannian manifold or they are comprised of piecewise linear paths with finitely many pieces.  In particular, we exclude fractal-like subsets of the path space $C([0:T],\mathbb{R}^{d_Y})$ which need not be compressible into finitely many dimensions without “loosing information” (i.e. for which there is no bi-Lipschitz embedding into a finite-dimensional normed space).

In what follows, we denote the Riemannian volume of a connected $(C^2)$ Riemannian manifold $(\mathcal{M},g)$ by $\operatorname{Vol}(\mathcal{M},g)$.  We denote its geodesic distance function by $d_g$.
[Domain Regularity]

Fix a compact $K\subseteq C([0:T],\mathbb{R}^{d_Y})$ and a “latent dimension $\ldim\in \mathbb{N}_+$”.
Suppose either that:

* Finite Domains: $K$ is finite.

*  Piecewise Linear Domains: There are $\pldim \in \mathbb{N}_+$, $0=t_0<\dots<t_\pldim=T$, and $C_K>0$ such that $y_{\cdot}\in K$ if and only if:

* For $i=0,\dots,\pldim-1$ there is a $d\times d$ matrix $A^{(i)}$ and a $b^{(i)}\in \mathbb{R}^{d_Y}$ satisfying

\[
\begin{aligned}
& y(t)=A^{(i)}\,t+b^{(i)}
\,
\,
\mbox{ for all } t\in (t_i,t_{i+1})
\end{aligned}
\]

* $y_0=0$,

* $\max_{i=1,\dots,\pldim}\,|y(t_i)|\le C_K$.

* Smooth Domains: There exists a compact and connected $\ldim$-dimensional Riemannian manifold $(\mathcal{M},g)$ for which $(\mathcal{M},d_g)$ is isometric to $(K,\|\cdot\|_T)$.

Evidently, their domains $K$ satisfying <Ref> <ref> are easily exhibited, and they correspond to any training set.  One can interpret results in this case as paralleling interpolation results for deep learning, e.g. <cit.>, but allowing for some arbitrarily small error as in <cit.>.

Compact subsets of the path-space $C([0:T],\mathbb{R}^{d_Y})$ satisfying <Ref> <ref> are also plentiful.  The following consequence of the Banach-Mazur theorem <cit.>, and related results such as <cit.>, illustrate this by guaranteeing that $C([0:T],\mathbb{R}^{d_Y})$ contains isometric copies of any compact metric space.  Consequentially, $C([0:T],\mathbb{R}^{d_Y})$ contains isometric copies of any compact and connected Riemannian manifold.

Let $\mathcal{X}$ be a compact metric space.  For every $T>0$ and each $d\in \mathbb{N}_+$, there exists a compact subset $K\subseteq C([0:T],\mathbb{R}^{d_Y})$ and an isometry from $\mathcal{X}$ onto $K$.

We obtain explicit estimates for the case where $(K,d_{\cdot})$ satisfies <Ref> <ref>, if the “model” smooth manifold $\mathcal{M}$ is topologically regular, in that sense that it is aspherical.
This means that their homotopy groups $\pi_i(K)$ all vanish for all indices $i\ge 2$, what is for instance the case if $\mathcal{M}$ is a Torus. See e.g. <cit.> for definitions and details on homotopy groups.

Domains $K$ satisfying <Ref> <ref> include any piecewise linear interpolation of real-world financial time-series data, or simulated data.  This is because only a finite number of inflection points can be observed, and prices are not unbounded.

In this way, <Ref> summarizes a rich family of compact subsets of the path-space $C([0:T],\mathbb{R}^{d_Y})$ which possess a sufficient degree of structure to be losslessly encoded into finitely many dimensions by our model's encoding layer, called its pathwise attention mechanism.

§.§ Target Space - The 2-Wasserstein Space of Probability Measures

Fix $1 \leq p \leq 2$. The $p$-Wasserstein space $\mathcal{P}_p(\mathbb{R}^{d_X})$ consists of all probability measures on $\mathbb{R}^{d_X}$ with finite second moment; i.e. $\mathbb{P}\in \mathcal{P}_p(\mathbb{R}^{d_X})$ if

\[
\mathbb{E}_{X\sim \mathbb{P}}[\|X\|^p]<\infty
\]

The $2$-Wasserstein metric $\mathcal{W}_p$ on $\mathbb{P}_p(\mathbb{R}^{d_X})$ is given by minimizing the optimal cost of transporting mass between any two measures $\mathbb{P}$ and $\mathbb{Q}$ via randomized transport plans.  This can be formalized by the Kanotovich problem

\[
\mathcal{W}_p(\mathbb{P}, \mathbb{Q})^p
\eqdef
\inf_{(X_1,X_2)\sim \pi;\,X_1\sim \mathbb{P},\,X_2\sim \mathbb{Q}}\,
\mathbb{E}_{\pi}[\|X_1-X_2\|^p]
\]

Generally, the $p$-Wasserstein distance between measures can be computationally taxing, requiring a super-quadratic complexity to compute <cit.>.  However, the $2$-Wasserstein distance is not always computationally intractable as for instance in the case where $\mathbb{P}=N(\mu^{(1)},\Sigma^{(1)})$ and $\mathbb{Q}=N(\mu^{(2)},{\Sigma^{(2)}})$ are Gaussian measures on $\mathbb{R}^{d_X}$, <cit.> showed that it admits the following closed-form expression

\begin{align}
\mathcal{W}_2(\mathbb{P},\mathbb{Q})^2
\| \mu^{(1)} - \mu^{(2)} \|_2^2
\operatorname{tr}(\Sigma^{(1)}) + \operatorname{tr}(\Sigma^{(2)})
\,
\operatorname{tr}
\big(
\Sigma^{(1)}
\big)^{1/2}
, \label{id:WassersteinExplicit}
\end{align}

provided that $\Sigma^{(1)}$ and $\Sigma^{(2)}$ are invertible; in particular, they are positive-definite.

§ THE MODEL: AN ATTENTION-BASED DEEP KALMAN FILTER

§.§ Phase 1 - Encoding via Pathwise Attention

We consider an attention-type feature encoding which modifies the attention mechanism of <cit.> and the graph attention mechanism of <cit.> to the context of continuous-time path sources/inputs.  Our attention mechanism, called pathwise attention, encodes key information about a novel path $y_{\cdot}$ and, like those attention mechanisms, allows us to adaptively encode any newly observed path in terms of its similarity scores to “known/previously generated paths”.

The similarity score component $\operatorname{sim}_T^{\theta_0}$ encodes a path into finite-dimensional vectorial data, by relating its similarity to a dictionary of previously observed “contextual paths”.

The positional encoding component $\operatorname{post}_T^{\theta_0}$ of the attention mechanism allows the similarity scores to further adapt to any novel path by extracting sample-positions on the novel paths.

[1]Our pathwise attention mechanism $\operatorname{attn}_t^{\theta}$, in <Ref>, relies on two methods for adaptive feature extraction.  The first component of the mechanism, in <ref>, in is a similarity score which ranks the similarity of any novel path $x_{\cdot}$ to a dictionary of “recognized/saved” reference paths.  The second component of the mechanism, in <ref>, samples points on the novel path $x_{\cdot}$ and perturbs the similarity scores by them.  The adeptness of the pathwise attention mechanism, unlike basis-based methods, is rooted in the fact that nearly all of its parameters are trainable, which allows it to be tasks-specific (similarly to the advantage of deep MLPs have over regression via basis-functions).

Fix positive integers $\rdim$, $\sdim$, $d_Y$, and a “time horizon” $T> 0$.
Given a set of $\rdim$ distinct paths $\{y^{(n)}\}_{n=1}^\rdim$ in $C([0,1],\mathbb{R}^{d_Y})$, an $\sdim \times \rdim$-matrix  $B$, $b\in \mathbb{R}^{\sdim}$, an $\sdim \times \sdim$-matrix $A$, and $a\in \mathbb{R}^{\sdim}$.  Denote $\theta_0\eqdef (A,B,a,b,\{y^{(n)}\}_{n=1}^\rdim)$.

The similarity score, is a map $\operatorname{sim}_T^{\theta_0}:C([0,1],\mathbb{R}^{d_Y})\rightarrow\mathbb{R}^\sdim$, mapping any $y_{\cdot}\in C([0,1],\mathbb{R}^{d_Y})$ to

\begin{equation}
\label{eq:sim_score}
\operatorname{sim}_T^{\theta_0}:
\mapsto
\operatorname{Softmax}
\biggl(
\Big(
\big(\|y_{\cdot}-y^{(n)}_{\cdot}\|_t\big)_{n=1}^\rdim
\Big)
\biggr)
\end{equation}

Illustrated in <Ref>, our pathwise attention mechanism extracts features by reinterpreting two of the key features of attention mechanisms from natural language processing (NLP).  First, it utilizes similarity scores which rank the likeness of any novel path against a dictionary of references paths.  These reference paths can either arise from real/stress historical (market) scenarios, they can be generated synthetically, or a combination of either.  These similarity scores are illustrated in <Ref>, and they serve as pathwise analogues of contextual keys and queries in attention mechanisms in NLP.

In the notation of <Ref>, fix a positive integer $\pdim$, and set of query times $0\le t_1<\dots< t_\tdim\le T$, an $\pdim\times {d_Y}$ matrix $V$, and an $\pdim\times \tdim$ matrix $U$.  Set $\theta_1\eqdef (V,U,\{t_n\}_{n=1}^\tdim)$.
The positional encoding, is a map $\operatorname{post}_T^{\theta_0}:C([0,1],\mathbb{R}^{d_Y})\rightarrow\mathbb{R}^{\pdim \times d_Y}$, defined by

\begin{equation}
\label{eq:pos_encoding}
\operatorname{post}_T^{\theta_1}
(y_{\cdot} )
\eqdef
U \,
(\oplus_{j=1}^\tdim \, y_{t_j})
+ V
\end{equation}

for any $y_{\cdot}\in C([0,1],\mathbb{R}^{d_Y})$.

Next, the positional encoding of any novel path, illustrated in <Ref>, encodes snapshots of this path at various times.  These indicate changes in any sample path, and they fully determined piecewise linear paths with finitely many pieces (as are generated in simulation studies).  This data then combines with the weights of the similarity scores to produce the features encodings of our pathwise attention mechanism.

In the notation of <Ref>, set $\sdim=\pdim$.  A pathwise attention, with parameter $\theta\eqdef (\theta_0,\theta_1,C)$, is the map $\operatorname{attn}_T^{\theta}:[0,T]\times C([0,T],\mathbb{R}^{d_Y})\rightarrow \mathbb{R}^{\edim + 1}$ with representation

\begin{align}
\label{eq:featuremap}
\operatorname{attn}_T^{\theta}(t,y_{\cdot})
\eqdef
\Big(
\operatorname{vec}\big(
\operatorname{sim}_T^{\theta_0}(y_{\cdot})
\odot
\operatorname{post}_T^{\theta_1}(y_{\cdot})
\big)
\Big)
\end{align}

for any $y_{\cdot}\in C([0,1],\mathbb{R}^{d_Y})$ and $0\le t\le T$, where $C$ is an $\edim \times (\sdim d_Y)$-dimensional matrix.

Let $K\subseteq C([0:T],\mathbb{R}^{d_Y})$ satisfy <Ref>.  Then, there exists an $\edim \in \mathbb{N}_+$ and a parameter $\theta$ as in <Ref> such that, $\operatorname{attn}_T^{\theta}$ restricts to a bi-Lipschitz embedding of $(K,\|\cdot\|_T)$ into $(\mathbb{R}^{\edim+1},\|\cdot\|_2)$.

Estimates for the encoding dimension $\edim$ are recorded in <Ref> on a case-by-base basis.

Complexity Estimates for Lossless Pathwise Attention Encoding of Regular Compact Domains.

Type of Compact Domain
Encoding Dimension ($\edim$)     Ass. <ref>

$\mathcal{O}(\log(\# K))$

P.W. Lin. ($\pldim$ Pieces)
$\mathcal{O}(\pldim d)$

Iso. Comp. Riemann.

Iso. Comp. Riemann $\&$ Aspherical
$\smash{\mathcal{O}\big(\mathfrak{N}_{d_g}(\mathcal{M},\operatorname{Vol}(\mathcal{M},g)^{1/\ldim} )\big)}$

*Fix an $A\subset \mathcal{M}$ and $\delta>0$.  The quantity $\mathfrak{N}_{d_g}(A,\delta)$ denotes the minimum number, $I$, of points $\{x_i\}_{i=1}^I \subseteq A$ for which every $x\in A$ is contained in a geodesic ball of radius $\delta$ about some $x_i$, for $i\in \{1,\dots,I\}$.

§.§ Phase 2 - Multi-Layered Perceptrons (MLPs) Transformation

Let us briefly recall the structure of a standard MLP, before formalizing our DKF.
In what follows, we fix a real-valued activation function $\sigma$ defined on $\mathbb{R}$ satisfying:
[<cit.> Condition]

The activation function $\sigma$ is continuous, non-affine, and there exists an $t\in \mathbb{R}$ such that $\sigma$ is differentiable at $t$ and $\sigma^{\prime}(t)\neq 0$.

Examples of activation functions satisfying this condition are the $\operatorname{PReLU}$ function $t\mapsto \max\{0,t\}+a\min\{0,t\}$, for a hyperparameter $a\in \mathbb{R}$, the $\tanh$ function used in the numerical PDE literature <cit.>, the sine function used in SIRENs <cit.>, and the Swish map $t\mapsto \frac{t}{1+e^{-\beta t}}$ of <cit.>, where $0\le \beta$ is a hyperparameter.

Fix an encoding dimension $\edim$ and a target dimension $\ddim$; both of which are positive integers.  A multi-layer perceptron (MLP), also called feedforward neural network, from $\mathbb{R}^\edim$ to $\mathbb{R}^\ddim$ with activation function $\sigma$ is a map $\hat{f}:\mathbb{R}^\edim\rightarrow\mathbb{R}^\ddim$ with iterative representation: for each $x\in \mathbb{R}^\edim$

\begin{equation}
\label{eq:MLP}
\begin{aligned}
\hat{f}(x) & \eqdef A^{(J)}\,x^{(J)}+b^{(J)},
\\
x^{(j+1)} & \eqdef
\sigma%_{\bar{\alpha}^{(j)}}
\bullet
\big(
\,
\qquad
\mbox{for }
\\
x^{(0)} & \eqdef  x.
\end{aligned}
\end{equation}

where for $j=0,\dots,J-1$, each $A^{(j)}$ is a $d_{j+1}\times d_j$ matrix, $b^{(j)}\in \mathbb{R}^{d_{j+1}}$, $d_0=\edim$ and $d_{J+1}=\ddim$.

§.§ Phase 3 - Decoding via Geometric Attention Mechanism

Fix $\ddim \in \mathbb{N}_+$.  In what follows, we denote the orthogonal projection of $\mathbb{R}^\ddim$ onto the $\ddim$-simplex

\[
\Delta_\ddim
\eqdef
\Big\{
w \in [0,1]^\ddim:\,
\sum_{n=1}^\ddim\,w_n = 1
\Big\}
\]

by $P_{\Delta_\ddim}$.  We note that the closedness and convexity of $\Delta_\ddim$ implies that $P_{\Delta_\ddim}$ is a well-defined $1$-Lipschitz map, see <cit.>.

Let $\ddim,d_X \in \mathbb{N}_+$.  A geometric attention mechanism is a map $\operatorname{g-attn}_\ddim: \mathbb{R}^\ddim \rightarrow \mathcal{N}_{d_X}$ with representation

\[
\operatorname{g-attn}_\ddim^{\vartheta}(v)
\mathcal{N}_{d_X}\Biggl(
\sum_{n=1}^\ddim\,
\cdot
\sum_{n=1}^\ddim\,
\cdot
\Biggr)
\]

where $m^{(1)},\dots,m^{(\ddim)}\in \mathbb{R}^{d_X}$, $A^{(1)},\dots,A^{(\ddim)} \in \R^{d_X\times d_X}$; and $\vartheta\eqdef (m^{(n)},A^{(n)})_{n=1}^\ddim$.

§.§ The DKF Model

We may now formalize the discrete Kalman filter model of <Ref>.

Let $d_X, d_Y \in\mathbb{N}_+$, $T>0$, and an activation function $\sigma:\mathbb{R}\rightarrow \mathbb{R}$.
A function $\hat{F}:C([0:T],\mathbb{R}^{d_Y})\rightarrow \mathcal{N}_{d_X}$ is called a discrete Kalman Filter if it admits the representation

\begin{equation}
\label{eq:DKF_formalization}
\hat{F}
\operatorname{g-attn}_\ddim^{\vartheta}
\circ
\hat{f}
\circ
\operatorname{attn}_T^{\theta}
\end{equation}

where $\operatorname{g-attn}_\ddim^{\vartheta}$ is as in <Ref>, $\operatorname{attn}_T^{\theta}$ is as in <Ref>, and $\hat{f}$ is an MLP as in (<ref>); such that the composition (<ref>) is well-defined.

§ MAIN RESULT

We are now ready to state our main theorem, which shows that the DKF (<ref>) is indeed capable of asymptotically optimally filtering the coupled system (<ref>)-(<ref>).  Our guarantees are of a non-asymptotic form; in that they depend on the complexity of the network.

Let $d_X, d_Y \in \mathbb{N}_+$ and $K\subset C([0:T],\mathbb{R}^{d_Y})$ satisfy <Ref>.  Suppose that the coupled system (<ref>)-(<ref>) satisfies <Ref>.
For every $T>0$ there exists a DKF $\hat{F}:[0:T]\times C([0:T]\times\mathbb{R}^{d_Y})\rightarrow\mathcal{N}_{d_X}$ satisfying the uniform estimate

\[
\max_{0\le t\le T,\,y_{\cdot}\in K}\,
\mathcal{W}_p\big(
% f^{\star}(y_{\cdot})
\mathbb{P}(X_t \in \cdot \vert y_{[0:T]})
\hat{F}(t,y_{\cdot})
\big)
\varepsilon
\]

for every $1\le p\le 2$.  Furthermore, <Ref> records the complexity estimates of $\hat{F}$.

Complexity Estimates for the DKF model $\hat{F}$ in <Ref>.

$\sigma$ Regularity     Depth     Width     Encode ($\edim$)     Decode ($\ddim$)

\smash{
\mathcal{O}\big(
\varepsilon^{-\ddim}
\big)

\smash{
\mathcal{O}\big(
\varepsilon^{-\ddim}
\big)
$\mathcal{O}(1)$     $\mathcal{O}\big(\varepsilon^{-1}\big)$

Smooth $\&$ Non-poly.

\smash{
\mathcal{O}\big(
\varepsilon^{-4\ddim-1}
\big)

$ \ddim + \edim +3$
$\mathcal{O}(1)$     $\mathcal{O}\big(\varepsilon^{-1}\big)$

Poly. $\&$ Non-affine

\smash{
\mathcal{O}\big(
\varepsilon^{-8\ddim-6}
\big)

$\mathcal{O}(1)$     $\mathcal{O}\big(\varepsilon^{-1}\big)$

Non-Smooth $\&$ Non-poly.

$\mathcal{O}(1)$     $\smash{\mathcal{O}\big(\varepsilon^{-1}\big)}$

§.§ Proof Overview

This section overviews the derivation of our main result, namely, <Ref>.

Results in <cit.> to show that  $\mathbb{P}(X_t\in \cdot|\mathcal{Y}_t)$ is conditionally Gaussian, they denote an explicit formulation of the resulting distribution which can then analyze directly.
In particular, <Ref> allows the application of <cit.> and we obtain that for each $0\le t\le T$ and each path $y_{[0:t]}\in C([0:t], \R^{d_Y})$ the probability measure $f_t(y_{[0:t]})$ is a $d_X$-dimensional Gaussian measure.  We denote the mean and covariance of this measure by $\mu(y_{[0:t]})$ and $\Sigma(y_{[0:t]})$.  Thus,

\begin{align}
\mathcal{N}(\mu(y_{[0:t]}),\Sigma(y_{[0:t]}))
\sim
\mathbb{P}(X_t\in \cdot|y_{[0:t]})
\eqdef
, \label{id:LipsterShiryaevRepresentation}
\end{align}

for each $0\le t\le T$ and each path $y_{[0:t]}\in C([0:t], \R^{d_Y})$.

A function is uniformly approximable by a deep learning model if it is continuous.  If there is no clear favourable structure in the function being approximated, e.g. smoothness <cit.> or neural network-like structure <cit.>, then the best available approximation rates are those corresponding to those where the target function is Lipschitz; see <cit.>.
Thus, the first step to obtaining approximability guarantees for the optimal filter by deep neural networks, which depend on relatively few trainable parameters, is to show that the optimal filter is a locally-Lipschitz function of observed paths.

Under <Ref>, $f_t$ from (<ref>) is locally Lipschitz-continuous. In particular, for every time $t \in [0:T]$, path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ and $\epsilon > 0$ there exists constant $C \ge 0$ such that for all times $s \in [0:T]$ and paths $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\vert t -s \vert < \epsilon$, $\| y^{(1)}_{[0:T]} - y^{(2)}_{[0:T]} \|_T < \epsilon$ holds

\begin{align*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y^{(1)}_{[0:t]}), \mathbb{P}(X_s \in \cdot \vert y^{(2)}_{[0:s]}))
\leq C
\big(
\Vert y^{(1)}_{[0:T]} - y^{(2)}_{[0:T]} \Vert_T
\vert t - s \vert
\big).
\end{align*}

Having established the regularity of the optimal filter for the coupled system (<ref>)-(<ref>), we need to verify that maps sharing the same domain (input space), codomain (output space), and regularity as the optimal filter are approximable by the proposed DKF model.  This is the content of the next proposition, which acts as its standalone approximation theorem for the DKF.

Complexity Estimates for transformer-type model $\hat{F}$ in <Ref>.

$\sigma$ Regularity     Depth     Width     Encode ($\edim$)     Decode ($\ddim$)

\smash{
\mathcal{O}\big(
(LV( L))^{-\ddim}\,\varepsilon^{-\ddim}
\big)

\smash{
\mathcal{O}\big(
\big)
$\mathcal{O}(1)$     $\mathcal{O}\big(L\varepsilon^{-1}\big)$

Smooth $\&$ Non-poly.

\smash{
\mathcal{O}\big(
L^{4\ddim+1}\, \varepsilon^{-4\ddim-1}
\big)

$ \ddim + \edim +2$
$\mathcal{O}(1)$     $\mathcal{O}\big(L\varepsilon^{-1}\big)$

Poly. $\&$ Non-affine

\smash{
\mathcal{O}\big(
L^{8\ddim+6}\, \varepsilon^{-8\ddim-6}
\big)

$\mathcal{O}(1)$     $\mathcal{O}\big(L\varepsilon^{-1}\big)$

Non-Sm. $\&$ Non-poly.

$\mathcal{O}(1)$     $\smash{\mathcal{O}\big(L\varepsilon^{-1}\big)}$

*Where $V(t)$ is the inverse of $s\mapsto s^4\,\log_3(t+2)$ on $[0,\infty)$ evaluated at $131 t$.

Let $d_X, d_Y \in \mathbb{N}_+$, $L>0$, and $K\subset C([0:T],\mathbb{R}^{d_Y})$ satisfying <Ref>.
For every $0<\varepsilon< 1/2$ and every $L$-Lipschitz function $f:([0,T]\times K,|\cdot| \times \|\cdot\|_{\infty})\rightarrow (\mathcal{N}_{d_X},\mathcal{W}_2)$ there exists a DKF $\smash{\hat{F}}$

satisfying the uniform estimate

\[
\sup_{t, x\in [0:T]\times K}\,
\mathcal{W}_p\big(
f(t, x_\cdot) ,
\hat{F}(t,x_\cdot)
\big)
\varepsilon
\]

for every $1\le p\le 2$.  Furthermore, the depth, width, encoding dimension $(\edim)$ and Decoding dimension $(\ddim)$ are all recorded in <Ref> depending on the activation function $\sigma$.

We now have at our disposal, the tools to prove our main result.
of <Ref>

Since the coupled system (<ref>)-(<ref>) satisfies <Ref>, <Ref> implies that the optimal filter $f_T$ in (<ref>)
is a locally-Lipschitz map from $([0:T],  C([0:T],\mathbb{R}^{d_Y}),\vert \cdot \vert \times \|\cdot\|_T)$ to $(\mathcal{N}_{d_X},\mathcal{W}_2)$.  Since $K\subseteq C([0:T],\mathbb{R}^{d_Y})$ is compact then, $f|_K$ is Lipschitz.

Since $K\subset C([0:T],\mathbb{R}^{d_Y})$ satisfies <Ref> and $0<\varepsilon<1/2$ then, <Ref> implies that there is a DKF $\hat{F}:C([0:T],\mathbb{R}^{d_Y})\rightarrow\mathcal{N}_{d_X}$ with representation (<ref>) and complexity recorded in <Ref> satisfying

\[
\sup_{t, y\in [0:T]\times K}\,
\mathcal{W}_p\big(
\hat{F}(t,y)
\big)
\varepsilon
\]

This completes the proof of <Ref>.

§ DISCUSSION AND FUTURE WORK

Our main result, namely Theorem <ref>, showed that there are deep Kalman filters which can solve the approximately stochastic filtering problem in continuous time for the coupled system (<ref>)-(<ref>), to arbitrary accuracy.  This complements the experimental work of <cit.>, which shows that deep Kalman filters can be trained to offer state-of-the-art empirical performance.

From the theory perspective, it would be interesting to analyze the statistical learning theoretic properties of DKFs.  One would expect that elementary generalization bounds could be derived by using the bounds on VC-dimension derived in <cit.> and a version of the results of <cit.> or by directly modifying the transport theoretic arguments of <cit.>, to incorporate the effects of the pathwise and geometric attention layers in (<ref>).  Nevertheless, for problems in mathematical finance where one only observes a single training path (one-shot learning), the most interesting question in that direction is: How to train the DKFs in a way such that they have robust statistical guarantees when only a single training path is observed.

§ PROOFS

This section contains the derivations of our paper's results.

§.§ Isometric Copies of Every Compact Riemannian Manifold in the Path Space – Proof of Proposition 1

of <Ref>

Since $(\mathcal{X},d_{\mathcal{X}})$ is compact then the Kuratowksi embedding $\phi_1:\,x\mapsto d_{\mathcal{X}}(\cdot,x)$ is an isometric embedding of $\mathcal{X}$ into the Banach space $C(\mathcal{X})$ with its uniform norm (since $\mathcal{X}$ is compact); where $d_{\mathcal{X}}$ denotes the metric on $\mathcal{X}$.  By the Banach-Mazur theorem, there exists an isometric embedding of $\phi_2:C(\mathcal{X})\rightarrow C([0,1])$.
Since the map $\psi_3:C([0:1])\rightarrow C([0:T],\mathbb{R}^{d_Y})$, given by $f\mapsto (f(\cdot /T),0,\dots,0)$, is an isometric embedding; then $\smash{\phi\eqdef \phi_3\circ\phi_2\circ \phi_1}$ is an isometric embedding of $\mathcal{X}$ into $C([0:T],\mathbb{R}^{d_Y})$.

§.§ Stability of the Optimal Filter - Proof of Lipschitz Continuity

We now prove the local Lipschitz stability of the optimal filter (<ref>), for the coupled system (<ref>)-(<ref>), as a function of its observed/input path.  The proof builds on a series of Lemmata, which demonstrate the local Lipschitz continuity of the parameters defining the optimal filter.

We first observe that, <Ref> grants us access to <cit.>, which implies that $\mu_t\eqdef \mu(y_{[0:t]})$ and $\Sigma_t\eqdef \Sigma(y_{[0:t]})$ are

\begin{align}
&=[a_0(t,y_{[0:t]}) + a_1(t,y_{[0:t]}) \mu_t]dt +[(b\circ B)(t,y_{[0:t]})+\Sigma_t A_1^{\top}(t,y_{[0:t]})]\label{id:ClosedFormMu}
\\
& \pushright{
\times (B\circ B)^{-1}(t,y_{[0:t]}) [dy_t - (A_0(t,y_{[0:t]}) + A_1(t,y_{[0:t]}))\mu_t dt]  \notag
\\
\partial_t\Sigma_t
& = a_1(t,y_{[0:t]}) \Sigma_t + \Sigma_t a_1^{\top}(t,y_{[0:t]}) + (b\circ b)(t,y_{[0:t]})
\label{id:ClosedFormSigma} \\
& \pushright{
- [(b\circ B)(t,y_{[0:t]}) + \Sigma_t A_1^{\top}(t,y_{[0:t]}) ]
(B\circ B)^{-1} (t,y_{[0:t]})} \notag \\
& \pushright{\times [(b\circ B)(t,y_{[0:t]}) + \Sigma_t A_1^{\top}(t,y_{[0:t]})]^{\top} }
\notag
\end{align}

with initial conditions $\mu_0\eqdef\mathbb{E}[X_0|Y_0]$ and $\Sigma_0\eqdef \mathbb{E}[(X_0-\mu_0)(X_0-\mu_0)^{\top}|Y_0]$; where

\begin{align*}
B \circ B \eqdef B_1B_1^\top + B_2B_2^\top, \quad
b \circ B \eqdef b_1B_1^\top + b_2B_2^\top, \quad
b \circ b \eqdef b_1b_1^\top + b_2b_2^\top.
\end{align*}

Furthermore, <cit.> implies that $\Sigma_t$ is positive definite for all $t \in [0:T]$ since we have assume that $\Sigma_0$ in <Ref>.

To clarify the argument's direction, we first record the proof of <Ref>.  We subsequently derive the lemmata on which it relies.

of <Ref>

Fix $t \in [0:T]$, a path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ and $\epsilon > 0$.
By <Ref> we can fix a $K_1 > 0$ such that for every time $0 < s < T$ holds

\begin{align*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y^{(1)}_{[0:t]}), \mathbb{P}(X_s \in \cdot \vert y^{(1)}_{[0:s]})) \leq K_1 \vert t-s \vert_t.
\end{align*}

By <Ref> we can fix a $K_2 > 0$ such that for every path $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\| y^{(1)}_{[0:T]} - y^{(2)}_{[0:T]} \|_T < \epsilon$ holds

\begin{align*}
\mathcal{W}_2(
\mathbb{P}(X_u \in \cdot \vert y^{(1)}_{[0:u]})
\mathbb{P}(X_u \in \cdot \vert y^{(2)}_{[0:u]})
\leq
K_2 \Vert y^{(1)}_{[0:T]} - y^{(2)}_{[0:T]} \Vert_T.
\end{align*}

We see by the triangular inequality of the $\mathcal{W}_2$-distance (that holds as we handle normal distributions, see (<ref>)),

\begin{align*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y^{(1)}_{[0:t]}), \mathbb{P}(X_s \in \cdot \vert y^{(2)}_{[0:s]})) \\
& \quad \leq
\mathcal{W}_2(
\mathbb{P}(X_t \in \cdot \vert y^{(1)}_{[0:t]})
\mathbb{P}(X_s \in \cdot \vert y^{(1)}_{[0:s]})
\mathcal{W}_2(
\mathbb{P}(X_s \in \cdot \vert y^{(1)}_{[0:s]})
\mathbb{P}(X_s \in \cdot \vert y^{(2)}_{[0:s]})
\\
& \quad \leq \max\{ K_1, K_2 \}
\big(
\Vert y^{(1)}_{[0:T]} - y^{(2)}_{[0:T]} \Vert_T
\vert t - s \vert
\big)
\end{align*}

what concludes the proof.

§.§.§ Lipschitz-Continuity in the Path-Component of the Optimal Filter

Under <Ref>, $f$ from (<ref>) is locally Lipschitz-continuous uniformly in time. In particular, for every path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ and $\epsilon > 0$ there exists constant $C \ge 0$ such that for all paths $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \|_t < \epsilon$ holds

\begin{align*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y^{(1)}_{[0:t]}), \mathbb{P}(X_t \in \cdot \vert y^{(2)}_{[0:t]})) \leq C \Vert y^{(1)}_{[0:T]} - y^{(2)}_{[0:T]} \Vert_T, \quad \forall t \in [0:T].
\end{align*}

of Lemma <ref>

By (<ref>), <Ref>, and <Ref> there exists a non-negative constant $\bar{K} < \infty$ satisfying the following for every pair of paths $y^{(1)}_{\cdot},y^{(2)}_{\cdot} \in C([0:T],\|\cdot\|_2)$ and all time $t \in [0:T]$

\begin{multline*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y^{(1)}_{[0:t]}), \mathbb{P}(X_t \in \cdot \vert y^{(2)}_{[0:t]}))
\\
\leq
\bar{K}
\,
\sqrt{\| \mu(y_{[0:t]}^{(1)}) - \mu(y_{[0:t]}^{(2)}) \|_2^2 + \|\Sigma(y_{[0:t]}^{(1)})-\Sigma(y_{[0:t]}^{(2)})\|_2^2}.
\end{multline*}

We see from <Ref> and <Ref>, there exists a non-negative constant $K < \infty$, satisfying

\begin{align*}
\begin{split}
\|\Sigma(y_{[0:t]}^{(1)})-\Sigma(y_{[0:t]}^{(2)})\|_2^2
& \leq
K^2 \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t^2,
\\
\|\mu(y_{[0:t]}^{(1)})-\mu(y_{[0:t]}^{(2)})\|_2^2
& \leq
K^2 \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t^2,
\end{split}
\qquad \forall t \in [0:T]
\end{align*}

Therefore, for each $y_{\cdot}^{(1)},y^{(2)}_{\cdot}\in C([0:T],\mathbb{R}^{d_Y})$ we have

\begin{align*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y^{(1)}_{[0:t]}), \mathbb{P}(X_t \in \cdot \vert y^{(2)}_{[0:t]}))
\leq
\sqrt{2} K \bar{K}\Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t,
\qquad \forall t \in [0:T]
\end{align*}

and the statement follows.

Under <Ref> <ref> $\mu_\cdot$ from (<ref>) is locally Lipschitz-continuous uniformly in time. This means that for every path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ there exists constant $K \in \R$, $\epsilon > 0$ such that for all paths $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \|_t < \epsilon$ holds

\begin{align*}
\big\| \mu^{(1)}_t - \mu^{(2)}_t \big\|_2
\leq
K \big\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \big\|_t,
\qquad \forall t \in [0:T].
\end{align*}

of Lemma <ref>

\begin{align}
\Phi_1(s, y_{[0:s]}) & \eqdef a_1(s, y_{[0:s]})
- [(b\circ B)(s, y_{[0:s]}) + \Sigma_t A_1^{{\top}}(s, y_{[0:s]}) ] \label{id:Phi1} \\
& \hspace{3cm} \times  (B\circ B)^{-1}(s, y_{[0:s]}) [A_0(s, y_{[0:s]}) + A_1(s, y_{[0:s]})], \notag \\
\Phi_2(s, y_{[0:s]}) & \eqdef a_0(s, y_{[0:s]}) + [(b\circ B)(s, y_{[0:s]}) + \Sigma_t A_1^{{\top}}(s, y_{[0:s]}) ] (B\circ B)^{-1}(s, y_{[0:s]}) \label{id:Phi2}.
\end{align}

Note that conditions of <Ref> and <Ref> are satisfied, therefore, $\Phi_1$ and $\Phi_2$ are globally Lipschitz w.r.t. time and locally Lipschitz w.r.t. the path.

By (<ref>) and for $t \in [0,T]$, $\mu_t$ can be written as

\begin{align}
\mu_t = & \mu_0 + \int_0^t a_0(s, y_{[0:s]}) ds  + \int_0^t \Phi_2(s, y_{[0:s]}) d y_{[0:s]}
+ \int_0^t \Phi_1(s, y_{[0:s]}) \mu_s ds .   \notag
\end{align}

We further have that

\begin{align*}
\mu^{(1)}_t - \mu^{(2)}_t
& =
\mu_0^{(1)} - \mu_0^{(2)}
\int_0^t a_0(s, y_{[0:s]}^{(1)}) -  a_0(s, y_{[0:s]}^{(2)} )  ds \\
& \quad +
\int_0^t \Phi_2(s, y^{(1)}_{[0: s]}) d y^{(1)}_{[0:s]}
\int_0^t \Phi_2(s, y^{(2)}_{[0 : s]})  d  y^{(2)}_{[0 : s]} \\
& \quad +
\int_0^t \Phi_1(s, y^{(1)}_{[0 : s]}) -   \Phi_1(s, y^{(2)}_{[0 : s]}) \mu^{(2)}_s ds
\int_0^t  \Phi_1(s, y^{(1)}_{[0 : s]})\, (\mu^{(1)}_s - \mu^{(2)}_s ) ds .
\end{align*}

By the triangle inequality, Jensen's inequality, and Cauchy-Schwarz inequality, we have

\begin{align}
\big\| \mu^{(1)}_t - \mu^{(2)}_t \big\|_2 \leq & \gamma(t) + \int_0^t \big\| \Phi_1(s, y^{(1)}_{[0 : s]}) \big\|_2 \cdot
\big\| \mu^{(1)}_s - \mu^{(2)}_s \big\|_2 ds ,\notag
\end{align}

\begin{align*}
\gamma(t) \eqdef & \big\| \mu_0^{(1)} - \mu_0^{(2)} \big\|_2
+ \underbrace{\Big\| \int_0^t a_0(s, y_{[0:s]}^{(1)}) -  a_0(s, y_{[0:s]}^{(2)} )  ds \Big\|_2}_{\term{t:gammaA}} \\
& + \underbrace{ \Big\| \int_0^t \Phi_2(s, y^{(1)}_{[0: s]}) d y^{(1)}_{[0:s]}
- \int_0^t \Phi_2(s, y^{(2)}_{[0 : s]})  d  y^{(2)}_{[0 : s]} \Big\|_2 }_{\term{t:gammaB}} \\
& + \underbrace{\Big\| \int_0^t \Phi_1(s, y^{(1)}_{[0 : s]}) -   \Phi_1(s, y^{(2)}_{[0 : s]}) \mu^{(2)}_s ds \Big\|_2}_{\term{t:gammaC}} .
\end{align*}

For an application of Grönwall's inequality, it is sufficient to show $\Vert \gamma(t) \Vert_2 < \infty$. We have due to $a_0$ being Lipschitz with constant $K_1 < \infty$,

\begin{align*}
\eqref{t:gammaA}
\leq \int_0^t \Vert a_0(s, y_{[0:s]}^{(1)}) - a_0(s, y_{[0:s]}^{(2)}) \Vert_2 ds
\leq \int_0^t K_1 \Vert y_{[0:s]}^{(1)} - y_{[0:s]}^{(2)} \Vert_s ds
\leq t K_1 \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t.
\end{align*}

Further, we see that

\begin{align}
\eqref{t:gammaB}
\leq &
\underbrace{\Big\| \int_0^t \Phi_2(s, y^{(2)}_{[0:s]}) - \Phi_2(s, y^{(1)}_{[0:s]}) d y^{(2)}_{[0:s]} \Big\|_2}_{\term{t:gammaE}}
\underbrace{ \Big\| \int_0^t \Phi_2(s, y^{(1)}_{[0:s]})
d ( y^{(2)}_{[0:s]} - y^{(1)}_{[0:s]} ) \Big\|_2 }_{\term{t:gammaD}} . \notag
\end{align}

Using Itô's isometry, we have

\begin{align*}
\eqref{t:gammaE}^2
& =  \int_0^t
\big\| \Phi_2(s, y^{(2)}_{[0:s]}) - \Phi_2(s, y^{(1)}_{[0:s]})\big\|_2^2
d \, (B\circ B)(s, y^{(2)}_{[0:s]}) \\
& \leq
K_2^2 \Vert y^{(2)}_{[0:t]} - y^{(1)}_{[0:t]} \Vert^2_t
\int_0^t d \, (B\circ B)(s, y^{(2)}_{[0:s]})
\end{align*}

where the last inequality holds due to (path-)Lipschitz continuity of $\Phi_2$ uniformly in time with constant $K_2 < \infty$, given by <Ref>. Next, let $\lbrace [s_{i-1}:s_{i}] \vert i \in \myset{1}{I}\rbrace$, $I \in \N$ be a fine enough partition of $[0:t]$, we see due to the (time-) Lipschitz continuity of $B \circ B$ together with the definition of the Riemann-Stieltjes integral that

\begin{align*}
\eqref{t:gammaE}^2
& \leq \epsilon_I + K_2^2 \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert^2_t \cdot \sum_{i = 1}^I \Vert (B\circ B)(y^{(2)}_{[0:s_i]}) - (B\circ B)(y^{(2)}_{[0:s_{i-1}]}) \Vert_2  \\
& \leq \epsilon_I + K_2^2 T \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t^2.
\end{align*}

As $\epsilon_I \to 0$ for $I \to \infty$, we obtain $\eqref{t:gammaE} \leq K_2\sqrt{T} \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t$.

Integrating by parts, we find that

\begin{align}
\eqref{t:gammaD}
& = \Big\| \Phi_2(t, y^{(1)}_{[0:t]}) (y^{(2)}_{[0:t]} - y^{(1)}_{[0:t]})
- \Phi_2(0, y^{(1)}_0) (y^{(2)}_0 - y^{(1)}_0)
\notag \\
& \pushright{- \int_0^t ( y^{(2)}_{[0:s]} - y^{(1)}_{[0:s]} ) \, d \Phi_2(s, y^{(1)}_{[0:s]})  \Big\|_2} \notag \\
& \leq \Big\| \Phi_2(t, y^{(1)}_{[0:t]}) (y^{(2)}_{[0:t]} - y^{(1)}_{[0:t]})
- \Phi_2(0, y^{(1)}_0) (y^{(2)}_0 - y^{(1)}_0) \Big\|_2
\notag \\
& \pushright{+ \Big\|  \int_0^t ( y^{(2)}_{[0:s]} - y^{(1)}_{[0:s]} ) \, d \Phi_2(s, y^{(1)}_{[0:s]})  \Big\|_2} \notag \\
& \leq 2 \max_{0 \leq s \leq T} \Big\| \Phi_2(s, y^{(1)}_{[0:s]}) \Big\|_2
\cdot \Big\|  y^{(2)}_{[0:s]} - y^{(1)}_{[0:s]} \Big\|_t
+ \underbrace{\Big\|  \int_0^t ( y^{(2)}_{[0:s]} - y^{(1)}_{[0:s]} ) \, d \Phi_2(s, y^{(1)}_{[0:s]})  \Big\|_2}_{\term{t:gammaF}}. \notag
\end{align}

<Ref> yields (time-)Lipschitz continuity of $\Phi_2$, implying $\max_{s \in [0:T]} \| \Phi_2(s, y^{(1)}_{[0:s]}) \|_2 =: K_3 < \infty$, together with (path-) Lipschitz continuity of $\Phi_2$ uniformly in time with constant $K_4 < \infty$ yields

\begin{align*}
\eqref{t:gammaF}
& \leq \epsilon_I + \sum_{i = 1}^I \big\| y^{(2)}_{[0:s_i]} - y^{(1)}_{[0:s_i]} \big\|_2  \big\Vert \Phi_2(s_i, y^{(1)}_{[0:s_i]}) - \Phi_2(s_{i-1}, y^{(1)}_{[0:s_{i-1}]}) \big\Vert_2 \\
& \leq \epsilon_I + K_4 t \Vert y^{(2)}_{[0:t]} - y^{(1)}_{[0:t]} \Vert_t ,
\end{align*}

which holds for $\epsilon_I \to 0$ for $I \to \infty$; we conclude that

\begin{align*}
\eqref{t:gammaB} \leq \left(K_2 \sqrt{T} + K_3 + K_4 T\right) \Vert y^{(2)}_{[0:t]} - y^{(1)}_{[0:t]} \Vert_t.
\end{align*}

Next, we consider

\begin{align*}
\eqref{t:gammaC}
\leq \int_0^t \big\| \Phi_1(s, y^{(1)}_{[0 : s]}) - \Phi_1(s, y^{(2)}_{[0 : s]}) \big\|_2 \big\|\mu^{(2)}_s \big\|_2 ds
\leq  K_5 t \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t \max_{s \in [0:t]} \big\|\mu^{(2)}_s \big\|_2
\end{align*}

what holds due to (path-)Lipschitz continuity of $\Phi_1$ uniformly in time with constant $K_5$ which is obtained by <Ref>. We conclude, since $\max_{s \in [0:T]} \|\mu^{(2)}_s \|_2 := K_6 < \infty$ due to <Ref>, that $\gamma(t) \leq K \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t$ with $K := K_1 + K_2 \sqrt{T} + K_3 + K_4 T + K_5 K_6 T$.

With that, we are able to apply Grönwall's inequality and obtain

\begin{align}
\big\| \mu^{(1)}_t - \mu^{(2)}_t \big\|_2
& \leq \gamma(t) + \int_0^t \gamma(s) \big\| \Phi_1(s, y^{(1)}_{[0 : s]}) \big\|_2
\exp\Big( \int_s^t \big\| \Phi_1(r, y^{(1)}_{[0 : r]}) \big\|_2 d r \Big) ds
\notag   \\
& \leq K \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t \notag \\
& \pushright{\left(
+ \int_0^t \big\| \Phi_1(s, y^{(1)}_{[0 : s]}) \big\|_2
\exp\Big( \int_s^t \big\| \Phi_1(r, y^{(1)}_{[0 : r]}) \big\|_2 d r \Big) ds
\right)}
\notag
\end{align}

Note that $\Phi_1$ is (time-)Lipschitz continuous, as shown in <Ref>, therefore implying that $\max_{s \in [0:T]} \| \Phi_1(s, y^{(1)}_{[0:s]}) \|_2 < \infty$. As constants $K_1$, $K_2$, $K_3$, $K_3$, $K_4$, $K_5$, and $K_6$ were chosen independently of $t \in [0:T]$ and <Ref> <ref> holds, we conclude the proof.

Let <Ref> <ref> hold. Then,

* for any path $y \in C([0:T]), \R^d)$ there is a constant $K \in \R$ s.t. for all $t > s \in [0:T]$ holds

\begin{align*}
\| \Phi_1(t, y_{[0:t]}) - \Phi_1(s, y_{[0:s]}) \|_2 \leq K \vert t-s \vert.
\end{align*}

where $\Phi_1$ is defined as in (<ref>).

* Also, for every path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ there exists a constant $C\ge 0$ such that for all $\epsilon > 0$, paths $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\| y^{(1)}_{[0:T]} - y^{(2)}_{[0:T]} \|_T < \epsilon$, and times $t \in [0:T]$ holds

\begin{align*}
\big\| \Phi_1(t, y^{(1)}_{[0:t]}) - \Phi_1(t, y^{(2)}_{[0:t]}) \big\|_2
\leq C \big\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \big\|_t.
\end{align*}

of Lemma <ref>

* The statement holds if $a_1$, $ (b\circ B)(B\circ B)^{-1}A_0$, $ (b\circ B)(B\circ B)^{-1}A_1$, $ \Sigma_\cdot A_1(B\circ B)^{-1}A_0$, $ \Sigma_\cdot A_1(B\circ B)^{-1}A_1$ are globally Lipschitz continuous w.r.t. the time component. As $a_1$, $ (b \circ B)$, $ \Sigma_\cdot$, $ (B\circ B)^{-1}$, $ A_0$, $ A_1$ satisfy this already (see <Ref>) and with this are also bounded on the interval $[0:T]$, their product is globally Lipschitz continuous w.r.t. time as well.

* The assertion holds if $a_1$, $ (b\circ B)(B\circ B)^{-1}A_0$, $ (b\circ B)(B\circ B)^{-1}A_1$, $ \Sigma_\cdot A_1(B\circ B)^{-1}A_0$, $ \Sigma_\cdot A_1(B\circ B)^{-1}A_1$ are locally Lipschitz continuous w.r.t. the path component uniformly in time. As $a_1$, $ (b \circ B)$, $ \Sigma_\cdot$, $ (B\circ B)^{-1}$, $ A_0$, $ A_1$ are locally Lipschitz continuous w.r.t. the path component (see <Ref>) uniformly in time, this also follows for their product.

Let <Ref> <ref> be fulfilled. Then,

* for any path $y \in C([0:T]), \R^d)$ there is a constant $C\ge 0$ s.t. for all $t > s \in [0:T]$ holds

\begin{align*}
\| \Phi_2(t, y_{[0:t]}) - \Phi_2(s, y_{[0:s]}) \|_2 \leq C \vert t-s \vert.
\end{align*}

where $\Phi_2$ is defined as in (<ref>).

* Additionally, for every path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ there exists constant $C\ge 0$, such that for all $\epsilon > 0$,  paths $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \|_t < \epsilon$, and times $t \in [0:T]$ holds

\begin{align*}
\big\| \Phi_2(t, y^{(1)}_{[0:t]}) - \Phi_2(t, y^{(2)}_{[0:t]}) \big\|_2
\leq C \big\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \big\|_t.
\end{align*}

of Lemma <ref>

* The assertion holds if $a_0$, $(b \circ B)(B\circ B)^{-1}$, $\Sigma_\cdot A_1(B\circ B)^{-1}$ are globally Lipschitz continuous w.r.t. the time component. As $a_0$, $(b \circ B)$, $\Sigma_\cdot$, $ (B\circ B)^{-1}$, $ A_1$ satisfy this already (see <Ref>) and with this are also bounded on the interval $[0:T]$, their product is globally Lipschitz continuous w.r.t. time as well.

* The statement holds if $a_0$, $(b \circ B)(B\circ B)^{-1}$, $\Sigma_\cdot A_1(B\circ B)^{-1}$ are locally Lipschitz continuous w.r.t. the path component. As $a_0$, $(b \circ B)$, $\Sigma_\cdot$, $ (B\circ B)^{-1}$, $ A_1$ are locally Lipschitz continuous w.r.t. the path component (see <Ref>) this also follows for their product.

Under <Ref> <ref>, $\Sigma_\cdot$ from (<ref>) is locally Lipschitz-continuous uniformly in time. This means, for every path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ there exists constant $C \in \R$ such that for all $\epsilon > 0$, paths $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \|_t < \epsilon$, and times $t \in [0:T]$ holds

\begin{align*}  \big\| \Sigma^{(1)}_t - \Sigma^{(2)}_t \big\|_2
\leq C \big\| y^{1}_{[0:t]} - y^{2}_{[0:t]} \big\|_t ,
\end{align*}

where $\Vert \cdot \Vert_2$ on the left-hand side refers to the Frobenius norm.

of Lemma <ref>

From (<ref>) and for $t \in [0:T]$, we obtain the integral representation of $\Sigma_t$

\begin{align}
\Sigma_t
& \Sigma_0 + \int_0^t
a_1(t,y_{[0:s]}) \Sigma_s
\Sigma_s a_1^{\top}(s,y_{[0:s]})
(b\circ b)(s,y_{[0:s]}) \hspace{3cm}
\notag\\
&\pushright{- [
(b\circ B)(s,y_{[0:s]})
\Sigma_s A_1^{\top}(s,y_{[0:s]})
(B\circ B)^{-1} (s,y_{[0:s]}) } \hspace{1cm} \notag \\
(b\circ B)(s,y_{[0:s]})
\Sigma_s A_1^{\top}(s,y_{[0:s]})
ds.} \notag
\end{align}

Let $\Sigma^{(1)}_t$, $\Sigma^{(2)}_t$ be the covariance matrices corresponding to paths $y^{(1)}_\cdot, y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ where $y^{(1)}_\cdot$ is arbitrary and $y^{(2)}_\cdot$ s.t. $\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \|_t < \epsilon$. The difference $\Sigma^{(1)}_t - \Sigma^{(2)}_t$ satisfies the integral representation

\begin{multline*}
\Sigma^{(1)}_t - \Sigma^{(2)}_t
= \Sigma^{(1)}_0 - \Sigma^{(2)}_0
+ \int_0^t
\Xi_1(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s)  d s
\\
+ \int_0^t (\Sigma^{(1)}_s - \Sigma^{(2)}_s)
\Xi_2(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s) \\
\Xi_2^\top (s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s)  (\Sigma^{(1)}_s - \Sigma^{(2)}_s) ds ,
\end{multline*}

where we denote

\begin{align}
& \Xi_1(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s)\notag \\
& \quad\quad\quad \eqdef
(a_1(s, y^{(1)}_{[0:s]}) - a_1(s, y^{(2)}_{[0:s]}) ) \Sigma^{(1)}_s
\Sigma^{(1)}_s (a_1^\top(s, y^{(1)}_{[0:s]}) - a_1^\top(s, y^{(2)}_{[0:s]})) \notag \\
& \quad\quad\quad\quad +
(b\circ b)(s, y^{(1)}_{[0:s]}) - (b\circ b) (s, y^{(2)}_{[0:s]}) \notag\\
& \quad\quad\quad\quad -
\Sigma^{(2)}_s [
A_1^\top (B\circ B)^{-1} (b\circ B)(s, y^{(1)}_{[0:s]})
A_1^\top (B\circ B)^{-1} (b\circ B)(s, y^{(2)}_{[0:s]})
] \notag\\
& \quad\quad\quad\quad -
(b\circ B) (B\circ B)^{-1} A_1(s, y^{(1)}_{[0:s]})
(b\circ B) (B\circ B)^{-1} A_1(s, y^{(2)}_{[0:s]})
] \Sigma^{(2)}_s \notag\\
& \quad\quad\quad\quad -
\Sigma^{(2)}_s [
A_1^\top (B\circ B)^{-1} A_1(s,y^{(1)}_{[0:s]})
A_1^\top (B\circ B)^{-1} A_1(s,y^{(2)}_{[0:s]})
] \Sigma^{(1)}_s \notag\\
& \quad\quad\quad\quad -
(b\circ B) (B\circ B)^{-1} (b\circ B)^\top(s, y^{(1)}_{[0:s]})
(b\circ B) (B\circ B)^{-1} (b\circ B)^\top(s, y^{(2)}_{[0:s]})
] \label{Xi1}
\end{align}

\begin{multline}
\Xi_2(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s) \eqdef  a_1^\top(s, y^{(2)}_s)
+ A_1^\top(s, y^{(1)}_{[0:s]})
(B\circ B)^{-1}(s, y^{(1)}_{[0:s]})  \\
\times(b\circ B) (s, y^{(1)}_{[0:s]})
+ A_1^\top(s, y^{(1)}_{[0:s]})
(B\circ B)^{-1}(s, y^{(1)}_{[0:s]}) A_1(s, y^{(1)}_{[0:s]}) \Sigma^{(1)}_s
.   \label{Xi2}
\end{multline}

By the triangular inequality, Jensen's inequality, and Cauchy-Schwarz inequality, we have

\begin{align}
\big\| \Sigma^{(1)}_t - \Sigma^{(2)}_t \big\|_2
\leq & \big\| \Sigma^{(1)}_0 - \Sigma^{(2)}_0 \big\|_2
+ \int_0^t \big\| \Xi_1(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s) \big\|_2 ds \notag \\
& + 2 \int_0^t \big\| \Sigma^{(1)}_s - \Sigma^{(2)}_s \big\|_2 \cdot \big\| \Xi_2(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s)  \big\|_2
ds . \notag
\end{align}

By Grönwall's inequality, it holds that

\begin{multline}
\big\| \Sigma^{(1)}_t - \Sigma^{(2)}_t \big\|_2
\leq
\big\| \Sigma^{(1)}_0 - \Sigma^{(2)}_0 \big\|_2 + \int_0^t \big\| \Xi_1(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s) \big\|_2 ds
\\
+ \int_0^t
\Big( \big\| \Sigma^{(1)}_0 - \Sigma^{(2)}_0 \big\|_2 + \int_0^s \big\| \Xi_1(r, y^{(1)}_{[0:r]}, y^{(2)}_{[0:r]}, \Sigma^{(1)}_r, \Sigma^{(2)}_r) \big\|_2 d r
\Big)  \cdot  \\
\cdot 2 \big\|  \Xi_2(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s) \big\|_2 \cdot \\ \cdot \exp \Big( 2 \int_s^t \big\|  \Xi_2(r, y^{(1)}_{[0:r]}, y^{(2)}_{[0:r]}, \Sigma^{(1)}_r, \Sigma^{(2)}_r) \big\|_2
d r  \Big) d s . \label{Gronwall:Sigma1-Sigma2}
\end{multline}

From <Ref>, we know that there exists a constant $K \in \R$ such that

\begin{align}
\big\|  \Xi_1(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s)  \big\|_2
\leq K \big\| y^{(1)}_{[0:s]} - y^{(2)}_{[0:s]}  \big\|_s .
\label{Xi1Lip}
\end{align}

Combining (<ref>) and (<ref>), we obtain

\begin{multline}
\big\| \Sigma^{(1)}_t - \Sigma^{(2)}_t \big\|_2
\leq
\big( \big\| \Sigma^{(1)}_0 - \Sigma^{(2)}_0 \big\|_2  + KT \big\| y^{(1)} - y^{(2)} \big\|_T \big)  \cdot
\\
\cdot \Big[ 1+  \int_0^t 2 \big\| \Xi_2(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s)  \big\|_2 \cdot
\\
\cdot \exp \Big( 2 \int_s^t \big\|  \Xi_2(r, y^{(1)}_{[0:r]}, y^{(2)}_{[0:r]}, \Sigma^{(1)}_r, \Sigma^{(2)}_r) \big\|_2
d r  \Big) d s \Big] .
\label{Sigma1-2bddXi2}
\end{multline}

Since $a_1$, $A_1$, $(B\circ B)^{-1}$ and $b\circ B$ are globally (time-) Lipschitz continuous, $\Xi_2$ is uniformly bounded, according to (<ref>). Therefore, from (<ref>) we obtain a constant $C \in \R$ such that

\begin{align}
\big\| \Sigma^{(1)}_t - \Sigma^{(2)}_t \big\|_2
\leq
C ( \big\| \Sigma^{(1)}_0 - \Sigma^{(2)}_0 \big\|_2 + \big\| y^{(1)} - y^{(2)} \big\|_T ) .  \notag
\end{align}

We conclude the proof due to <Ref> <ref>.

Let $\Xi_1$ be defined as in (<ref>), and let <Ref> <ref> hold.
Then, for every path $y^{(1)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ there exists constant $K \in \R$, such that for all  $\epsilon > 0$, paths $y^{(2)}_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ with $\| y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \|_t < \epsilon$, and times $s \in [0:T]$ holds

\begin{align*}
\big\|  \Xi_1(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s)  \big\|_2
\leq
K \big\| y^{(1)}_{[0:s]} - y^{(2)}_{[0:s]}  \big\|_s .
\end{align*}

of Lemma <ref>

Since $a_1$, $b \circ b$, $b \circ B$, $A_1$, $(B \circ B)^{-1}$ are locally Lipschitz continuous in the path component uniformly over all times, we conclude that $a_1$, $b \circ b$, $A_1^\top(B \circ B)^{-1}(b \circ B)$, $(b \circ B)(B \circ B)^{-1}A_1$, $A_1^\top (B \circ B)^{-1}A_1$, and $(b \circ B)(B \circ B)^{-1}(b \circ B)^\top$ have the same property. By this, together with the triangular inequality, there exists a constant $K_1 \in \R$

\begin{align*}
\big\Vert \Xi_1(s, y^{(1)}_{[0:s]}, y^{(2)}_{[0:s]}, \Sigma^{(1)}_s, \Sigma^{(2)}_s) \big \Vert
\\& \quad
\leq
2K_1 \big\Vert y^{(1)}_{[0:s]} - y^{(2)}_{[0:s]}  \big\Vert_s \Big(\big\Vert\Sigma^{(1)}_s\big\Vert_2 + \big\Vert\Sigma^{(2)}_s \big\Vert_2\Big)
K_1 \big\Vert y^{(1)}_{[0:s]} - y^{(2)}_{[0:s]}  \big\Vert_s \big\Vert\Sigma^{(1)}_s\big\Vert_2\big\Vert\Sigma^{(2)}_s\big\Vert_2
\\ &
\pushright{
K_1 \big\Vert y^{(1)}_{[0:s]} - y^{(2)}_{[0:s]}  \big\Vert_s
\\& \quad
K_1 \big\Vert y^{(1)}_{[0:s]} - y^{(2)}_{[0:s]}  \big\Vert_s
\Big(
2 \Vert\Sigma^{(1)}_s \big\Vert_2
2 \Vert\Sigma^{(2)}_s \big\Vert_2
\big\Vert\Sigma^{(1)}_s\big\Vert_2 \big\Vert\Sigma^{(2)}_s \big\Vert_2
\Big).
\end{align*}

With that, it is left to show that $\Vert\Sigma^{(1)}_s \Vert_2 < \infty$ and $\Vert\Sigma^{(2)}_s \Vert_2 < {K}_2$ for some $K_2 \in \R$. The former is trivially fulfilled due to the continuity of $a_1, b \circ b, b \circ B, A_1, (B \circ B)^{-1}$.
Since $\Sigma_t$ is continuous in the path-component and the closure of the set

\begin{align*}
\{y \in C^{1}([0:T]) \,:\, \Vert y_{[0:t]} - y^{(1)}_{[0:t]} \Vert_2 < \epsilon \}
\end{align*}

is compact, we know $\Vert\Sigma^{(2)}_s \Vert_2$ is bounded.

§.§.§ Lipschitz-Continuity in the Time-Component of the Optimal Filter

Under <Ref>, $f_t$ from (<ref>) is locally Lipschitz-continuous in the time-component. In particular, for every path $y_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$, $t \in [0:T]$, and  $\epsilon > 0$ there exists constant $C \ge 0$ such that for all times $s \in [0:T]$ with $\| t - s \|_t < \epsilon$ holds

\begin{align*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y_{[0:t]}), \mathbb{P}(X_s \in \cdot \vert y_{[0:s]})) \leq C \vert t - s \vert.
\end{align*}

of Lemma <ref>

As in the proof of <Ref>, we argue that by (<ref>), <Ref>, and <Ref> there exists a non-negative constant $\bar{K} < \infty$ satisfying the following for every path $y_{\cdot} \in C([0:T],\|\cdot\|_2)$ and all times $t, s \in [0:T]$

\begin{multline*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y_{[0:t]}), \mathbb{P}(X_s \in \cdot \vert y_{[0:s]}))
\\
\leq
\bar{K}
\,
\sqrt{\| \mu(y_{[0:t]}) - \mu(y_{[0:s]}) \|_2^2 + \|\Sigma(y_{[0:t]})-\Sigma(y_{[0:s]} \|_2^2}.
\end{multline*}

We see from <Ref> and <Ref> that there exists a non-negative Lipschitz-constant $K < \infty$ for both $\mu$ and $\Sigma$ such that

\begin{align*}
\mathcal{W}_2(\mathbb{P}(X_t \in \cdot \vert y_{[0:t]}), \mathbb{P}(X_s \in \cdot \vert y_{[0:s]})) \leq \sqrt{2}\bar{K}K \vert t - s \vert,
\end{align*}

and the statement follows.

Let the <Ref> <ref> be fulfilled. Then, for any path $y \in C([0:T]), \R^d)$ there is a constant $K \in \R$ s.t. for all $t > s \in [0:T]$ holds

\begin{align*}
\| \mu_t - \mu_s \|_2 \leq K \vert t-s \vert.
\end{align*}

where $\mu$ is defined as in (<ref>).

of Lemma <ref>

With the notation of <Ref>, we have

\begin{multline*}
\| \mu_t - \mu_s \|_2
\\ \leq
\underbrace{\Big\Vert \int_s^t a_0(u, y_{[0:u]})du \Big\Vert_2}_{\term{t:muA}}
+ \underbrace{\Big\Vert\int_s^t \Phi_2(u, y_{[0:u]})dy_{[0:u]} \Big\Vert_2}_{\term{t:muB}}
+ \underbrace{\Big\Vert \int_s^t \Phi_1(u, y_{[0:u]})\mu_u du \Big \Vert_2}_{\term{t:muC}}.
\end{multline*}

We see that $\eqref{t:muA} \leq K \vert t - s \vert$ as $\max_{u \in [0:T]}\Vert a_0(u, y_{[0:u]}) \Vert_2 < \infty$ due to (time-)Lipschitz-continuity of $a_0$.

By the use of integration by parts,

\begin{align*}
\eqref{t:muB}
& =
\Big\|
\Phi_2(t, y_{[0:t]}) y_{[0:t]}
- \Phi_2(s, y_{[0:s]}) y_{[0:s]}
- \int_0^t y_{[0:u]} \, d \Phi_2(u, y_{[0:u]})
\Big\|_2 \\
& \leq
\underbrace{
\Big\|
\Phi_2(t, y_{[0:t]}) y_{[0:t]}
- \Phi_2(0, y_0) y_0
\Big\|_2
\underbrace{
\Big\|
\int_0^t y_{[0:u]} \, d \Phi_2(u, y_{[0:u]})
\Big\|_2
\end{align*}

we see that $\eqref{t:muB1} \leq K \Vert y^{(1)}_{[0:t]} - y^{(2)}_{[0:t]} \Vert_t$ due to (path-)Lipschitz continuity of $\Phi_2$.
Moreover, let $\mySet{[u_{i-1}:u_{i}]}{i \in \myset{1}{I}}, I \in \N$ be a fine enough partition of $[s:t]$, then we see due to global (time-)Lipschitz continuity of $\Phi_2$ and the definition of the Riemann-Stieltjes integral that

\begin{align*}
\eqref{t:muB2}
& \leq \epsilon_I + \sum_{i = 1}^I   \Vert y_{[0:u_i]} \Vert_2  \big\| \Phi_2(u_i, y_{[0:u_i]}) - \Phi_2(u_{i-1}, y_{[0:u_{i-1}]})  \big\|_2 \\
& \leq \epsilon_I + \Vert y_{[0:t]} \Vert_t \sum_{i = 1}^I K \vert u_i - u_{i-1} \vert
\end{align*}

and, since $\epsilon_I \to 0$ for $I \to \infty$, it follows $\eqref{t:muB} \leq K \vert t - s \vert$.

Due to $\eqref{t:muA} < \infty$ and $\eqref{t:muB} < \infty$, we can apply Grönwall's inequality to

\begin{align*}
\| \mu_t \|_2
\le &
\overbrace{\Big\Vert \int_0^t a_0(u, y_{[0:u]})du \Big\Vert_2
+ \Big\Vert\int_0^t \Phi_2(u, y_{[0:u]})dy_{[0:u]} \Big\Vert_2}^{ \eqdef  \alpha(t)} \\
& + \int_0^t \big\Vert \Phi_1(u, y_{[0:u]})\big\Vert_2 \big\Vert \mu_u \big\Vert_2 du,
\end{align*}

which yields

\begin{align*}
\| \mu_t \|_2
& \leq
\alpha(t) + \int_0^t \alpha(s)\big\Vert \Phi_1(s, y_{[0:s]}) \big \Vert_2
\exp\left(
\int_s^t \big\Vert \Phi_1(u, y_{[0:u]}) \big \Vert_2 du
\right) ds
\leq
K t
\end{align*}

where the last equality holds due to $\alpha(s) \leq K s$ for all $s \in [0:T]$ and (time-)Lipschitz continuity of $\Phi_1$, implying $\max_{u \in [0:t]}\Vert \Phi_1(u, y_{[0:u]}) \Vert_2 < \infty$.

With that, we have

\begin{align*}
\eqref{t:muC} \leq  \int_s^t \big\Vert \Phi_1(u, y_{[0:u]})\big\Vert_2 \big\Vert \mu_u \big \Vert_2du   \leq \max_{u \in [0:t]}\Vert \Phi_1(u, y_{[0:u]}) \Vert_2 K T \vert t - s \vert
\end{align*}

what proves the claim.

Let <Ref> <ref> hold. Then, for any path $y \in C([0:T]), \R^d)$ there is a constant $K \in \R$ s.t. for all $t > s \in [0:T]$ holds

\begin{align*}
\| \Sigma_t - \Sigma_s \|_2 \leq K \vert t-s \vert.
\end{align*}

where $\Sigma$ is defined as in (<ref>).

of Lemma <ref>

Note that $\Sigma_\cdot$ is continuous in time. From (<ref>), we obtain

\begin{align*}
\Vert \Sigma_t  - \Sigma_s \Vert_2
= & \int_s^t
\Vert a_1(t,y_{[0:u]}) \Sigma_u \Vert_2
+ \Vert \Sigma_u a_1^{\top}(u,y_{[0:u]}) \Vert_2
+ \Vert (b\circ b)(u,y_{[0:u]}) \Vert_2 \\
& \quad \quad
+ \Vert  (b\circ B)(u,y_{[0:u]})
(B\circ B)^{-1} (u,y_{[0:u]})
(b\circ B)(u,y_{[0:u]})^{\top} \Vert_2 \\
& \quad \quad
+ \Vert  (b\circ B)(u,y_{[0:u]})
(B\circ B)^{-1} (u,y_{[0:u]})
\Sigma_u A_1^{\top}(u,y_{[0:u]})^{\top} \Vert_2 \\
& \quad \quad
+ \Vert  \Sigma_u A_1^{\top}(u,y_{[0:u]})
(B\circ B)^{-1} (u,y_{[0:u]})
(b\circ B)(u,y_{[0:u]})^{\top} \Vert_2 \\
& \quad \quad
+ \Vert  \Sigma_u A_1^{\top}(u,y_{[0:u]})
(B\circ B)^{-1} (u,y_{[0:u]})
\Sigma_u A_1^{\top}(u,y_{[0:u]})^{\top} \Vert_2
du .
\end{align*}

As all components are continuous, we can bound them by their maximal value on $[0:T]$. With this, we conclude $\| \Sigma_t - \Sigma_s \|_2 \leq K \vert t-s \vert$.

§.§.§ Lower Loewner order bound

Let $\Sigma$ be defined as in <Ref> and <Ref> <ref>, <ref>, hold. Then there exists an $r > 0$, s.t. for all times $t \in [0:T]$ and paths  $y_\cdot \in C^{\color{blue} 1}([0:T], \R^{d_Y})$ holds that
$\Sigma_t \preccurlyeq r I_d$.

of Lemma <ref>

By the Courant-Fisher Theorem, see <cit.>, the statement is equivalent to finding a lower bound on the eigenvalues of $\Sigma$, which we will show by finding an upper bound on the Eigenvalues of $\Sigma^{-1}$, which exists as it is positive definite.

Following the proof of <cit.>, we note that

\begin{align*}
\partial_t\Sigma_t^{-1}
= &
- \Tilde{a}_1^\top(t, y_{[0:t]})\Sigma_t^{-1}
- \Sigma_t^{-1}\Tilde{a}_1(t, y_{[0:t]})
+ A_1^\top(B \circ B)^{-1}A_1(t, y_{[0:t]}) \\
&   - \Sigma_t^{-1}\left[
(b \circ b)(t, y_{[0:t]})
- (b \circ B)(B \circ B)(b \circ B)^\top(t, y_{[0:t]})
\right]\Sigma_t^{-1}
\end{align*}

\begin{align*}
\Tilde{a}_1^\top(t, y_{[0:t]}) \eqdef a_1(t, y_{[0:t]}) - (b \circ B)(B \circ B)A_1(t, y_{[0:t]}).
\end{align*}

Now, let $G_t(y_{[0:t]})$ be a solution of $\partial_tG_t(y_{[0:t]}) = \Tilde{a}_1(t, y_{[0:t]})G_t(y_{[0:t]})$ with $G_0(y_{[0:t]}) = I_{d_X}$.
Then, as pointed out in <cit.>, we arrive due to $(b \circ b)(t, y_{[0:t]})- (b \circ B)(B \circ B)(b \circ B)^\top(t, y_{[0:t]})$ being positive semi-definite at

\begin{multline*}
\tr( G_t(y_{[0:t]}) \Sigma_t^{-1} G_t(y_{[0:t]})^\top ) \\
\leq
\tr(\Sigma_0^{-1})
\int_0^T
\tr\left(
A_1^\top(B \circ B)^{-1}A_1(s, y_{[0:s]})
\right)
\end{multline*}

As $G_t(y_{[0:t]})$ is positive definite, we conclude by submultiplicativity of positive semi-definite matrices that

\begin{align*}
\tr(\Sigma_t^{-1})
\tr(
G_t(y_{[0:t]}) \Sigma_t^{-1} G_t(y_{[0:t]})^\top
) \\
\tr(G_t(y_{[0:t]})^{-1})^2
\tr(
G_t(y_{[0:t]}) \Sigma_t^{-1} G_t(y_{[0:t]})^\top
\end{align*}

\begin{multline*}
\tr\left(
A_1^\top(B \circ B)^{-1}A_1(s, y_{[0:s]})
\right)
\\ \leq
\tr\left(
\right)^2
\tr\left(
A_1^\top(B \circ B)^{-1}A_1(s, y_{[0:s]})
\right).
\end{multline*}

As we assumed that there exist constants $K_1, K_2 > 0$ s.t. uniformly $K_1 \leq \tr(G_t(y_{[0:t]})) \leq K_2$ as well as $K_3 > 0$ s.t. $\tr(A_1^\top(B \circ B)^{-1}A_1(t, y_{[0:t]})) \leq K_3$ we conclude $\tr(\Sigma_t^{-1}) < K$ uniformly for some constant $K > 0$. As the trace is the sum of all Eigenvalues, this proves the claim.

§.§ Stable and Lossless Encoding by Pathwise Attention - Proof of Proposition 2

In the case where the “latent geometry” of $K$ has additional structure, we may guarantee that the attention mechanism $\operatorname{attn}_T$ is linearly-stable with linearly-stable inverse, how to generate the reference paths $y^{(1)}$, $\dots$, $y^{(\rdim)}$ in $C([0:T],\mathbb{R}^{d_Y})$, and quantitative estimates on how many paths must be generated.

Under <Ref> <ref>, there exists a constant $C_{\ldim}>0$, depending only on $\ldim$, and a $\delta>0$ depending only on $(M,g)$ such that any $\delta$-packing $\{y^{(n)}\}_{n=1}^\ldim$ of $K$, there are matrices $A,B,b,V,U,C$ and vectors $a,b$ for which the parameter $\theta$, as in <Ref>, defining a pathwise attention mechanism $\operatorname{attn}_T^{\theta}:C([0:T],\mathbb{R}^{d_Y})\rightarrow \mathbb{R}^{\edim}$ which restricts to a bi-Lipschitz embedding of $(K,\|\cdot\|_T)$ into $(\mathbb{R}^{\edim+1},\|\cdot\|_2)$.
Furthermore, $\|\theta\|_0 \in \mathcal{O}(\edim^2)$.

Moreover, if $(\mathcal{M},g)$ is aspherical then:

(i) $\delta \in \mathcal{O}( \operatorname{Vol}(\mathcal{M},g)^{1/\ldim} )$

(ii) $\|\theta\|_0 \in \mathcal{O}(\edim^2)$,

(iii) $C_{\ldim}\in \mathcal{O}(\ldim^{3(\ldim+1)/2})$

In particular, the attention mechanism $\operatorname{attn}_T$ satisfies <cit.> for any Borel probability measure $\mathbb{P}_{\mathrm{in}}$ on $K$.

of Lemma <ref>

* Estimating $\rdim$ on the Riemannian Manifold $(\mathcal{M},g)$.
Since $(K,\|\cdot\|_T)$ be isometric to $(\mathcal{M},d_g)$ where $d_g$ is the geodesic distance on an aspherical compact $\ldim$-dimensional Riemannian manifold $(\mathcal{M},g)$, then Gromov's systolic inequality <cit.> implies that the systole of $\mathcal{M}$ denoted by $\operatorname{sys}(\mathcal{M})$ satisfies

\begin{equation}
\label{eq:Systolic}
\operatorname{sys}(\mathcal{M})
\le
\,\tilde{C}_{\ldim}
\,
\operatorname{Vol}(\mathcal{M},g)^{1/\operatorname{dim}(\mathcal{M})}
\end{equation}

where $\operatorname{Vol}(\mathcal{M},g)$ denotes the Riemannian volume of $(\mathcal{M},g)$ and $\tilde{C}_{\ldim}>0$ is a universal constant only depending on $\ldim$ satisfying

\begin{align*}
\Tilde{C}_{\ldim}
6 (\ldim+1)\ldim^{\ldim}\,\sqrt{(\ldim+1)!}
\,
\end{align*}

By Stirling's approximation, $\sqrt{(\ldim+1)!} \in \mathcal{O}(\ldim^{(\ldim+1)/2})$; whence it follows that $\tilde{C}_{\ldim}\in
% \mathcal{O}(\ldim^{\ldim+1}\, \ldim^{(\ldim+1)/2})$
\mathcal{O}(\ldim^{3(\ldim+1)/2})$.

In the proof of <cit.> (circa <cit.>) we see that if $\delta = \operatorname{Sys}(\mathcal{M})/10$ then given any $\delta$-net $\widetilde{\mathbb{X}}$ in $(\mathcal{M},g)$, the map $\varphi_2:(\mathcal{M},d_g)\mapsto (\mathbb{R}^{\#\widetilde{\mathbb{X}}},\|\cdot\|_2)$ given for any $p\in M$ by

\[
\varphi_2:p\mapsto \big(d_g(p,u)\big)_{u\in \widetilde{\mathbb{X}}}
\]

is a bi-Lipschitz embedding.  Set $C_{\ldim}\eqdef \tilde{C}_{\ldim}/10$ and $\rdim\eqdef \#\widetilde{\mathbb{X}}$.  Therefore,

\[
\delta
\le
\,
\operatorname{Vol}(\mathcal{M},g)^{1/\ldim}
\]

Enumerate $\widetilde{\mathbb{X}}\eqdef \{u_n\}_{n=1}^\rdim$.
We note that if $K$ is not aspherical but if it is only isometric to a closed Riemannian manifold, then the conclusion still holds, however, without this explicit upper bound on $\delta$.

*  Building the Feature Map with Well-posed Inverse.
Let $\varphi_1:(\mathcal{M},d_g)\rightarrow (K,\|\cdot\|_T)$ by an any isometry, which we have postulated to exist.  Set $\smash{\mathbb{X}\eqdef \{\varphi(u):\,u\in \widetilde{\mathbb{X}}\}=\{y^{(n)}\}_{n=1}^\rdim}$ where, for $n=1,\dots,\rdim$ we define $y^{(n)}\eqdef \phi_1(u_n)$.
Define $\smash{\varphi: C([0:T],\mathbb{R}^{d_Y})\rightarrow \mathbb{R}^\rdim}$ by $\smash{y_{\cdot}\mapsto (\|y_{\cdot}-y^{(n)}_{\cdot}\|_{T})_{n=1}^\rdim}$.  Observe that, for every path $y\in K$, the following holds

\begin{align}
\nonumber
\varphi(y_{\cdot})
\eqdef &
\Big(
\|y_{\cdot}-y^{(n)}_{\cdot}\|_{T}
\Big)_{n=1}^\rdim
\\
\nonumber
= &
\Big(
\big\|
\varphi_1\circ \varphi_1|_{\varphi_1(K)}^{-1}(y_{\cdot})
\varphi_1\circ \varphi_1|_{\varphi_1(K)}^{-1}(y^{(n)}_{\cdot})
\big\|_T
\Big)_{n=1}^\rdim
\\
\label{proofFeatureMap_isometry_assumption}
= &
\Big(
\varphi_1|_{\varphi_1(K)}^{-1}(y_{\cdot})
\varphi_1|_{\varphi_1(K)}^{-1}(y^{(n)}_{\cdot})
\big)
\Big)_{n=1}^\rdim
\\
= &
\label{proofFeatureMap_injectivity}
\Big(
\varphi_1|_{\varphi_1(K)}^{-1}(y_{\cdot})
\big)
\Big)_{n=1}^\rdim
\\
\nonumber
= &
\varphi_2\circ \varphi_1|_{\varphi_1(K)}^{-1}(y_{\cdot})
\end{align}

where (<ref>) holds since $\varphi_2$ is an isometry and (<ref>) holds unambiguously since $\varphi_1$ is a bijection from $\mathcal{M}$ to $K$.  Since every isometry is a bi-Lipschitz map, the compositions of bi-Lipschitz maps is again bi-Lipschitz, and since we have just shown that $\varphi|_{K}=\varphi_2\circ \varphi_1|_{\phi(K)}^{-1}$ then, $\phi|_{K}$ is a bi-Lipschitz embedding of $K$ into the $\rdim$-dimensional Euclidean space.

*  Aligning to a Hyperplane in $\mathbb{R}^{\sdim}$ with a Shallow ReLU Neural Network.
Set $\sdim \eqdef \rdim+1$.  Let $b$ be the zero vector in $\mathbb{R}^{2\rdim}$.
We now consider a variation of the example on <cit.>, the respective $\rdim \times 2\rdim$ and $2\rdim\times \rdim$ block-matrices $A_1$ and $B$

\[
\begin{pmatrix}
I_\rdim & ~ -I_\rdim
\end{pmatrix}
\mbox{ and }
\begin{pmatrix}
I_\rdim \\
\end{pmatrix}
\]

are such that the ReLU neural network $\tilde{\psi}:\mathbb{R}^\rdim\rightarrow \mathbb{R}^\rdim$, with $\tilde{\psi}(u)\eqdef A_1\operatorname{ReLU}\bullet(Bu+b)$ satisfies $\tilde{\psi}(u)=u$, for each $u\in \mathbb{R}^\rdim$.  Consider the $\sdim\times \rdim$ block-matrix $A_2$ and the vector $a\in \mathbb{R}^{\sdim}$ given by

\[
\begin{pmatrix}
I_\rdim \\
0 \\
\end{pmatrix}
\mbox{ and }
\begin{cases}
0 & \mbox{ if } i=1,\dots,\rdim\\
1 & \mbox{ if } i=\sdim
\end{cases}
\]

Set $A\eqdef A_2A_1$, $\psi\eqdef A\operatorname{ReLU}\bullet(Bu+b)+a$, and note that $\psi$ is a ReLU neural network.  A direct computation shows that, $\|A\|_0=2\rdim$, $\|B\|_0=2\rdim$, $\|a\|_0=1$, and $\|b\|_0=0$.

*  Injectivity of Softmax Function on Hyperplane.
Note that, for each $u\in \mathbb{R}^\rdim$

\[
\psi(u) = (u_1,\dots,u_N,1)^{\top}
\]

therefore bijectively $\psi$ maps $\mathbb{R}^\rdim$ onto the $\rdim$-dimensional hyperplane $H\eqdef \{(u,1):\,u\in \mathbb{R}^\rdim\}$ in $\mathbb{R}^{\sdim}$.  Observe that the Softmax function, given for any $u\in \mathbb{R}^{\sdim}$ by $\smash{\operatorname{Softmax}:u\mapsto (e^{u_j}/\sum_{k=1}^\rdim\,e^{u_k})_{j=1}^\sdim}$, maps $H$ surjectively and continuously onto the image set

\begin{align*}
\operatorname{Softmax}(\mathbb{R}^{\sdim})\subseteq \operatorname{int}(\Delta_\sdim)\eqdef \Big\{v\in [0,1]^{\sdim}:\,\sum_{i=1}^{\sdim}\,v_i=1\Big\};
\end{align*}

i.e. of the interior of the $\sdim$-simplex.  Since,

\begin{equation}
\label{eq:inverse_softmax_interior}
\mapsto
\big(\ln(v_i)-\ln(v_{\sdim}) + 1\big)_{i=1}^{\sdim}
\end{equation}

is a continuous right-inverse of $\operatorname{Softmax}$ on the interior of the image set $\operatorname{Softmax}(\mathbb{R}^{\sdim})$, the softmax function $\operatorname{Softmax}$ defines a continuous bijection from $H$ onto $\operatorname{Softmax}(\mathbb{R}^{\sdim})$.

Therefore, the map $\Psi\eqdef \operatorname{Softmax}\circ \psi\circ \phi$ is a continuous and injective when restricted to $K$.

Since $K$ is compact and $\Psi$ is continuous then $\Psi(K)$ is a compact subset of the interior $\operatorname{int}(\Delta_\sdim)$ of the $\sdim$-simplex, then,

\begin{align*}
C^{\star}\eqdef \max_{u\in \Phi(K)}~\min_{v\in \Delta_{\sdim}\setminus \operatorname{int}(\Delta_{\sdim})}\,\|u-v\|>0.
\end{align*}

Therefore, $\Psi(K)$ is contained in the set

\[
\Delta_{\sdim}^{\star}
\eqdef
\{
\bar{\Delta}_{\sdim}
\}
\]

where $\bar{\Delta}_{\sdim}\eqdef (1/\sdim,\dots,1/\sdim)$ is the barycenter of the $\sdim$-simplex $\Delta_{\sdim}$ for some $t^{\star}\in [0,1)$ such that $\Psi(K)\subseteq \Delta_{\sdim}^{\star}$ (which is possible since $C^{\star}>0$).  Since the map (<ref>) is locally Lipschitz on $\operatorname{int}(\Delta_\sdim)$, it is Lipschitz on the compact set $\Delta_{\sdim}^{\star}$ then $\Psi$ has a Lipschitz inverse on its image since it is the composition of Lipschitz functions with Lipschitz inverses.

*  Representation as an Attention Mechanism.
Set $\theta_0 = (A,B,a,b, \{y^{(n)}_{\cdot}\}_{n=1}^\rdim)$ and observe that the map $\Psi$ is of the form (<ref>).
We continue by choosing a map of the form (<ref>). For this, let $\tdim \in \N$, set $\pdim \eqdef \sdim$ and let $U$ be the $\pdim \times \tdim$-dimensional zero matrix, and let $V$ be the $\pdim \times d_Y$-dimensional matrix whose entries are all equal to $1$.  Set $\theta_1\eqdef (U,V,\{n/{\pdim}\}_{n=1}^{\tdim})$.  Define the encoding dimension $\edim \eqdef \sdim$ and let $C$ be the $\edim \times \edim\cdot  d_Y$-dimensional matrix given by $Cx = (x_{n \cdot d_Y})_{n=1}^{\edim}$ for every $x\in \mathbb{R}^{\edim d_Y}$.  Set $\theta\eqdef (\theta_0,\theta_1,C)$ and observe that

\[
\operatorname{attn}_T^{\theta}(t,y_{\cdot})
\eqdef
\Big(
\operatorname{vec}\big(
\operatorname{sim}_T^{\theta_0}(y_{\cdot})
\odot
\operatorname{post}_T^{\theta_1}(y_{\cdot})
\big)
\Big)
\big(1_{[0:T]}\times \Psi\big)
\]

for each $y_{\cdot}\in K.$ and every $y\in [0:T]$.
Consequentially, $\operatorname{attn}_T^{\theta_{\cdot}}$ is a bi-Lipschitz embedding of $[0:T]\times K$ to $\mathbb{R}^{1 + \edim}$.  Note that $\|C\|_0= \pdim$, $\|U\|_0=0$, and $\|V\|_0= \pdim d_Y$.  Thus, $\|\theta\|_0\in \mathcal{O}(\pdim \cdot d_Y + \rdim)$.

This concludes the proof.

Under <Ref> <ref> and let $\rdim \eqdef \#K$.
Set $\edim=192\lceil\log(\rdim)\rceil$ and $\{y_i\}_{i=1}^k$ enumerate $K$.
There is a parameter $\theta$ as in <Ref> defining a pathwise attention mechanism $\operatorname{attn}_T^{\theta}:C([0:T],\mathbb{R}^{d_Y})\rightarrow \mathbb{R}^{\edim}$ which restricts to a $(2^{-1/2},(3\rdim/6)^{1/2})$-bi-Lipschitz embedding of $(K,\|\cdot\|_T)$ into $(\mathbb{R}^{1+\edim},\|\cdot\|_2)$.

In particular, $\operatorname{attn}_T$ satisfies <cit.>.

of Lemma <ref>

We argue similarly to the proof of <Ref>, with only Steps <ref> and <ref> of its proof being replaced with the following argument.

Steps 1-2 (Modified): Building the Feature Map.

Enumerate $\smash{K=\{x_n\}_{n=1}^\rdim}$.  The map $\smash{\phi_1:(K,\|\cdot\|_T)\rightarrow (\mathbb{R}^\rdim,\|\cdot\|_{\infty})}$ given for every $x\in K$ by $\smash{x\mapsto \big(\|x-x_n\|_t\big)_{n=1}^\rdim}$ is an isometric embedding, called the Kuratowski embedding (see <cit.>).  The optimal constants for the equivalence of the Euclidean $\|\cdot\|_2$ and max $\|\cdot\|_{\infty}$ norms are given by

\begin{equation}
\label{eq:equivalence_2infty_norms}
\|u\|_{\infty} \le \|u\|_2
\quad\mbox{ and }\quad
\|u\|_2\le \rdim^{1/2}\,\|u\|_{\infty}
\end{equation}

for each $u\in\mathbb{R}^\rdim$.  Therefore, the “set theoretic identity map” $\phi_2:(\mathbb{R}^\rdim,\|\cdot\|_{\infty})\rightarrow (\mathbb{R}^\rdim,\|\cdot\|_2)$ is bi-Lipschitz with optimal (shrinking and expansion) constants given by (<ref>).
Define $\sdim \eqdef \lceil48\,\ln(\rdim)\rceil$.
By the Johnson-Lindenstrauss Lemma, with “small” constant given in the derivation of <cit.>
[We use this formulation since the constant in $4*48$ as opposed to $4*200$ in the standard derivation; found for example in <cit.>.]
, exists a linear map $\phi_3:(\mathbb{R}^\rdim,\|\cdot\|_2)\rightarrow (\mathbb{R}^{\sdim},\|\cdot\|_2)$, i.e. $\varphi_3(u)=A_0\,u$ for some $\sdim \times \rdim$-matrix $A_0$, satisfying: For every $u,v\in \mathbb{R}^\rdim$

\[
\,
\|u-v\|_2
\le
\|A_0\,u-A_0\,v\|_2
\le
\,
\|u-v\|_2
\]

Consequentially, the map $\varphi:(K,\|\cdot\|_T)\rightarrow (\mathbb{R}^{\sdim},\|\cdot\|_2)$ given by $\varphi\eqdef \varphi_3\circ \varphi_2\circ \varphi_1$ is $(2^{-1/2},(3\rdim/2)^{1/2})$-bi-Lipschitz, since the Kuratowksi embedding $\varphi_1$ is an isometry and $\varphi_2$ satisfies (<ref>); this is because for each $x,\tilde{x}\in K$ we have

\begin{align*}
\,
\|x-\tilde{x}\|_T
= &
\,
\|\varphi_1(x)-\varphi_1(\tilde{x})\|_{\infty}
\\
\le &
\,
\|\varphi_2\circ \varphi_1(x)-\varphi_2\circ \varphi_1(\tilde{x})\|_2
\\
\le &
\|\varphi_3\circ \varphi_2\circ \varphi_1(x)-\varphi_3\circ \varphi_2\circ \varphi_1(\tilde{x})\|_2
\\
\eqdef &
\|\varphi(x)-\varphi(\tilde{x})\|_2
\\
\eqdef &
\|\varphi_3\circ \varphi_2\circ \varphi_1(x)-\varphi_3\circ \varphi_2\circ \varphi_1(\tilde{x})\|_2
\\
\le &
\,
\|\varphi_2\circ \varphi_1(x)-\varphi_2\circ \varphi_1(\tilde{x})\|_2
\\
\le &
\,
\rdim^{1/2}\,
\| \varphi_1(x)-\varphi_1(\tilde{x}) \|_{\infty}
\\
= &
\,
\| x-\tilde{x} \|_T
\end{align*}

Remaining Steps.

The rest of the proof is identical to Steps step:GFM_3,step:GFM_4,step:GFM_5[Here $\sdim$ corresponds to $\sdim - 1$ in Step <ref>.] of the proof of <Ref> but with $B$ defined instead as $B\eqdef B_1A_0$, where $B_1=  (I_\sdim\, \,I_\sdim)^{\top}$.

This concludes the proof.

Fix $\pldim \in \mathbb{N}_+$, $0=t_0<\dots<t_\pldim=T$, and a constant $C_K>0$.
Let $K\subset C([0:T],\mathbb{R}^{d_Y})$ satisfy <Ref> <ref>.
Then, there is a parameter $\theta$, as in <Ref>, defining a pathwise attention mechanism $\operatorname{attn}_T^{\theta}:C([0:T],\mathbb{R}^{d_Y})\rightarrow \mathbb{R}^{\edim + 1}$, with $\edim \eqdef \pldim \cdot d_Y$ which restricts to a bi-Lipschitz embedding of $(K,\|\cdot\|_T)$ into $(\mathbb{R}^{\edim + 1},\|\cdot\|_2)$.
Furthermore, $\|\theta\|_0\in \mathcal{O}(\pldim \cdot d_Y)$.

In particular, $\operatorname{attn}_T$ satisfies <cit.>.

of Lemma <ref>

Let $\rdim \in \N_+$, set $\sdim \eqdef \pldim$, and define $A, B$ as zero-matrices as well as $a, b$ as zero-vectors according to the dimensions in <Ref>.
By fixing $\theta_0=(A,B,a,b)$, we observe that

\begin{align}
\label{eq:bijection_samplingA}
\operatorname{sim}_T^{\theta_0}(y_{\cdot}) = 1 \in \R^{\pldim}
\end{align}

for all $y\in C([0:T],\mathbb{R}^{d_Y})$.

Set $\tdim \eqdef \pdim \eqdef \pldim$. Then, let $V$ be the zero matrix and $U$ the identity matrix according to the dimensions in <Ref>[The point $t_0$ is not sampled since every path $y_{\cdot}$ in $K$ satisfies $y_0=0$ and, thus, there is no need to sample it at time $0$.]. Set $\theta_1=(U,V,\{t_n\}_{n=1}^\pldim)$ and with that,

\begin{equation}
\label{eq:bijection_samplingB}
\operatorname{pos}_T^{\theta_1}(y_{\cdot})
\end{equation}

In particular, the map $\operatorname{pos}_T^{\theta_1}$ in (<ref>) defines a linear bijection between $K$ and the set $B_{C_K}(0)^\pldim$, where $B_{C_K}(0) := \lbrace z \in \R^{d_Y} : \vert z \vert \leq C_K\rbrace$.  Since $B_{C_K}(0)^\pldim$ is a subset of the finite-dimensional Banach space $(\mathbb{R}^{\pldim \times d_Y},\|\cdot\|_2)$ and all norms on a finite-dimensional normed are equivalent, then, $\operatorname{pos}_T^{\theta_1}$ is a bi-Lipschitz embedding of $K$ into $(\mathbb{R}^{\pldim \times d_Y},\|\cdot\|_2)$.

Fix $\edim \eqdef \pldim \cdot d_Y$ and let $C$ be the $\edim \times \edim$-dimensional matrix with $C_{i,j}=1$ for $i,j=1,\dots,\edim$.  Set $\theta=(\theta_0,\theta_1,C)$.  By construction $\|\theta\|_0\in \mathcal{O}(\pldim \cdot d_Y)$.

\begin{equation}
\label{eq:bijection_sampling__RelabelingStuff}
\operatorname{attn}_T^{\theta}(t,y_{\cdot})
\eqdef
\Big(
\operatorname{vec}\big(
\operatorname{sim}_T^{\theta_0}(y_{\cdot})
\odot
\operatorname{post}_T^{\theta_1}(y_{\cdot})
\big)
\Big)
\big(t,(y_1,\dots,y_\pldim)\big)
\end{equation}

Together, eq:bijection_samplingA,eq:bijection_samplingB imply that $\operatorname{attn}_T^{\theta}$ defines a bi-Lipschitz embedding of $K$ into the $\edim+1$-dimensional Euclidean space.

of <Ref>

Follows directly from <Ref>.

§.§ Approximability of Locally Lipschitz Maps By DKFs - Proof of Proposition 4

We now prove our main universal approximation theorem, <Ref>.  We show that the target space/codomain of any considered DKF is geometrically regular, in the sense of <cit.>.  Using this fact, we then combine and apply the results of <cit.> to deduce the result.

§.§.§ Polish QAS Space Structure on the space of Positive Semi-Definite Matrices

Let $d_X \in \mathbb{N}_+$, and $\operatorname{Sym}_{0,d_X}$ denote the set of $d_X\times d_X$ symmetric positive semi-definite matrices and let $\|\cdot\|_F$ denote the Frobenius norm on the set of $d_X\times d_X$ matrices.[The $0$ emphasizes that there is no rank-restriction on the matrices in $\operatorname{Sym}_{0,d_X}$ unlike, for example, in <cit.>.]  Consider the $2$-product metric on $\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ given for any $(m^{(1)},A),(m^{(2)},B)$ by

\begin{equation}
\label{eq:Quantization}
\operatorname{d}_{2,F}\big(
\big)^2
\eqdef
\|m^{(1)}-m^{(2)}\|^2+\|A-B\|_F^2
\end{equation}

We show that $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},\operatorname{d}_{2,F})$ satisfies the conditions of <cit.>; namely <cit.>.

This requires defining a few maps first.
For $q\in \mathbb{N}$ define the map $\mathcal{Q}_q:\mathbb{R}^{d_X+d_X^2}\rightarrow \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ by sending any $(m,A)\in \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ to

\[
\mathcal{Q}_q((m,A))
\eqdef
\]

The family $\mathcal{Q}\eqdef (\mathcal{Q}_q)_{q\in \mathbb{N}_+}$ quantizes $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},\operatorname{d}_{2,F})$, in the sense of <cit.>.

Next, we consider the so-called mixing function $\eta:\cup_{N\in \mathbb{N}_+}\,\Delta_N\times (\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X})^{N} \rightarrow
\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$, where $N\in\mathbb{N}_+$ and $\Delta_N$ is the $N$-simplex, defined for any $N\in\mathbb{N}_+$, $w\in \Delta_N$, and $(m^{(1)},A^{(1)}),\dots,(m^{(N)},A^{(N)})\in \mathbb{R}^{d_X}\times \mathbb{R}^{d_X^2}$ by

\begin{equation}
\label{eq:MixingFunction}
\eta\big(
\{(m^{(n)},A^{(n)}\}_{n=1}^N
\big)
\eqdef
\sum_{n=1}^N\,
\cdot
\end{equation}

Note that by the convexity of $\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ and the fact that each $w\in \Delta_N$, for some $N\in \mathbb{N}_+$, then $\eta$ does indeed take values in $\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$.  The mixing function $\eta$ will be used to inscribe “abstract geodesic simplices” in $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},\operatorname{d}_{2,F})$ thereby endowing it with the structure of an approximately simplicial space, in the sense of <cit.>.

$(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},\mathcal{Q},\eta)$ is a barycentric QAS space, in the sense of <cit.>.  In particular, it satisfies <cit.>.

of Lemma <ref>

We begin by showing that $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},\mathcal{Q},\eta)$ is a QAS space, as defined in <cit.>.  We also show that it is barycentric, meaning that it admits a $1$-barycenter map as defined, for example, in <cit.>.

QAS Space Structure

Since $\operatorname{Sym}_{0,d_X}$ is a closed convex subset of space $(\mathbb{R}^{d\times d},\|\cdot\|_F)$.  Since the Cartesian product of closed subsets is a closed subset of $(\mathbb{R}^{d^2+d},d_{2,F})$ and since the product of convex sets is again convex by <cit.> then, $\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ is a closed and convex subset of the normed linear space $(\mathbb{R}^{d+d^2},d_{2,F})$.

For $q\in \mathbb{N}$ define the map $\mathcal{Q}_q:\mathbb{R}^{d+d^2}\rightarrow \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ as sending any $(m,A)\in \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ to

\[
\mathcal{Q}_q((m,A))
\eqdef
\]

The map $\mathcal{Q}_q$ is a surjection since every symmetric matrix is the square of some $d\times d$ matrix <cit.>.
Therefore, $\mathcal{Q}_{\cdot}=(\mathcal{Q}_q)_{q\in\mathbb{N}_+}$ trivially satisfies <cit.> with modulus of quantizability, defined on <cit.>, given by $\mathcal{Q}_{K}(\varepsilon)=D_1\eqdef d(d+1)$ for every compact subset $K$ of $\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ and for every $\varepsilon>0$.

Observe that, for any $N\in \mathbb{N}_+$, $w\in \Delta_N$, and $(m^{(1)},A^{(1)}),\dots,(m^{(N)},A^{(N)})\in \mathbb{R}^{d_X}\times \mathbb{R}^{d\times d}$ we have

\begin{align}
\nonumber
& d_{2,F}\Big(
\eta\big(
\{(m^{(n)},A^{(n)}\}_{n=1}^N
\big)
\Big) \\
\notag
& \quad =
\Big(
\big\|
\big(\sum_{n=1}^N\,w_n\,m^{(n)}\big) - m^{(i)}
\big\|^2
\big\|
\big(\sum_{n=1}^N\,w_n\,A^{(n)}\big) - A^{(i)}
\big\|_F^2
\big)^{1/2}
\\
\label{eq:Because_Simplex}
& \quad =
\Big(
\big\|
\sum_{n=1}^N\,w_n\,m^{(n)}
\sum_{n=1}^N\,w_n\,m^{(i)}
\big\|^2
\big\|
\sum_{n=1}^N\,w_n\,A^{(n)}
\sum_{n=1}^N\,w_n\,A^{(i)}
\big\|_F^2
\big)^{1/2}
\\
\nonumber
& \quad \le
\Big(\sum_{n=1}^N\,w_n\Big)^2\,
\big\|
\big\|^2
\Big(\sum_{n=1}^N\,w_n\Big)^2\,
\big\|
\big\|_F^2
\big)^{1/2}
\\
\nonumber
& \quad \le
\sum_{n=1}^N\,w_n
\,
\Big(
\big\|
\big\|^2
\big\|
\big\|_F^2
\big)^{1/2}
\\
\nonumber
& \quad =
\sum_{n=1}^N\,w_n
\,
\big)^1
\end{align}

for $i=1,\dots,N$,
where (<ref>) holds since $\sum_{n=1}^N\,w_n=1$ since $w\in \Delta_N$.  Thus, $\eta$ is a mixing function and therefore $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F})$ is approximately simplicial, as defined in <cit.>.
Consequentially, $\smash{(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F},\mathcal{Q},\eta)}$ is a QAS space; as defined in <cit.>.

Since $(\mathbb{R}^{d_X}\times \operatorname{Sym}_d,d_{2,F})$ is a normed linear space then <cit.> shows that the only contracting barycenter map is given by $\mathcal{P}_1(\mathbb{R}^{d_X}\times \operatorname{Sym}_{d},d_{2,F})\ni \mathbb{P}\rightarrow \mathbb{E}_{X\sim \mathbb{P}}[X]\in \mathbb{R}^{d_X}\times \operatorname{Sym}_{d}$.
Since $\operatorname{Sym}_{0,d_X}$ is a closed convex subset of the normed linear space $\operatorname{Sym}_d$ then Jensen's inequality, as formulated in <cit.>, implies that for each $\mathbb{P}\in \mathcal{P}_1(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F})$ we have $\mathbb{E}_{X\sim \mathbb{P}}[X]\in \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$; where $\mathbb{E}_{X\sim \mathbb{P}}[X]$ denote the Bochner integral of a random variable with law $\mathbb{P}$.  Consequentially,
$\mathcal{P}_1(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F})\ni \mathbb{P}\rightarrow \mathbb{E}_{X\sim \mathbb{P}}[X]\in \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ is a contracting barycenter map.  Thus, $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F})$ is barycentric metric space.

Since is a barycentric QAS space then, <cit.> is satisfied.

We are now ready to prove the following result, as we have established the barycentricity and the QAS space structure of $(\mathbb{R}^{d_X}\times\operatorname{Sym}_{0,d_X}, d_{2,F},\eta,\mathcal{Q}_{\cdot})$,

Fix an activation function $\sigma$ satisfying <Ref>.
For every $K\subset C([0:T],\mathbb{R}^{d_X})$ satisfying <Ref>, each $0<\delta,\alpha\le 1$, $0 \le L$, and every $(L,\alpha)$-Hölder
[That is, $f$ is $\alpha$ Hölder with optimal Hölder coefficient $L$.]
function $f:[0:T]\times K \rightarrow \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ there is a map $\hat{g}: [0:T]\times C([0:T],\mathbb{R}^{d_Y}) \rightarrow \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ satisfying the uniform estimate

\begin{equation}
\label{lem:approximation_Sym__approximation}
\max_{(t,y_{\cdot}) \in [0:T]\times K}\,
d_{2,F}(\hat{g}(t,y_{\cdot}),f(t, y_{\cdot})) < \delta
\end{equation}

with representation

\begin{equation}
\label{lem:approximation_Sym__approximation___representation}
\hat{g}(t, y_{\cdot})
% =
%     \hat{\eta}(\hat{f}\circ \phi(x_{\cdot}),Z_n)
\sum_{i=1}^\ddim\,
P_{\Delta_\ddim}(\hat{f}\circ \operatorname{attn}^\theta(t, y_{\cdot}))_i
\cdot
\end{equation}

where $\ddim \in \mathbb{N}_+$, $m^{(1)},\dots,m^{(\ddim)}\in \mathbb{R}^{d_X}$, $A^{(1)},\dots,A^{(\ddim)}\in \mathbb{R}^{d\times d}$, $P_{\Delta_\ddim}:\mathbb{R}^\ddim\rightarrow\Delta_\ddim$ is the Euclidean (orthogonal) projection onto the $\ddim$-simplex, and an MLP $\hat{f}:\mathbb{R}^\edim\rightarrow \mathbb{R}^\ddim$ with activation function $\sigma$.  The depth, width, encoding dimension ($\edim$), and decoding dimension ($\ddim$) are recorded in <Ref>.

Complexity Estimates for transformer-type model $\hat{g}$ in <Ref>.

$\sigma$ Regularity     Depth     Width     Encode ($\edim$)     Decode ($\ddim$)

\mathcal{O}\big(
(LV( L))^{-\edim-1}\,\varepsilon^{-\edim-1}
\big)

\mathcal{O}\big(
\big)
$\mathcal{O}(1)$     $\mathcal{O}\big(L\varepsilon^{-1}\big)$

Smooth $\&$ Non-poly.

%%% GENERAL FORM
%  \mathcal{O}\left(
% \frac{
%     m (2\operatorname{diam}(K))^{2p}
%     }{
%     \left(C_0\omega^{-1} \left(
%         f
% , \frac{C \, \epsilon}{(1+\frac{p}{4})m} \right)\right)^{2p} }
% \right)
%%% Simplified FORM for Lipschitz Maps
\mathcal{O}\left(
L^{4\edim+5}\, \varepsilon^{-4\edim-5}
\right)

$ \mathcal{O}(L\epsilon^{-1} + \edim +3)$
$\mathcal{O}(1)$     $\mathcal{O}\big(L\varepsilon^{-1}\big)$

Poly. $\&$ Non-affine

%%% GENERAL FORM
%      \mathcal{O}\left(
% \frac{
%     m(m+p)(2\operatorname{diam}(K))^{4p+2}
%     }{
%     \left(C_0\,\omega^{-1} \left(
%     f
%     , \frac{C\,\epsilon}{(1+\frac{p}{4})m} \right)\right)^{4p+2}
%     }
%     \right)
%%% Simplified FORM for Lipschitz Maps
\mathcal{O}\big(
L^{8\edim+14}\, \varepsilon^{-8m-14} %m=(\edim-1)
\big)

$\mathcal{O}(L\epsilon^{-1} + \edim +4)$
$\mathcal{O}(1)$     $\mathcal{O}\big(L\varepsilon^{-1}\big)$

$C(\rr)$ $\&$ Non-poly.

$\mathcal{O}(L\epsilon^{-1} + \edim +3)$
$\mathcal{O}(1)$     $\mathcal{O}\big(L\varepsilon^{-1}\big)$

*Where $V(t)$ is the inverse of $s\mapsto s^4\,\log_3(t+2)$ on $[0,\infty)$ evaluated at $131 t$.

of Lemma <ref>

We work in the notation of <cit.>, or rather, its quantitative version <cit.>.
Our objective is to apply <cit.> by verifying each of the conditions of <cit.>.

* Implementing a Bi-Lipschitz Feature Map with the Attention Layer.

We first show that the conditions of <cit.> are met, by verifying that the parameters of the attention layer (<ref>)  can be chosen such that $\operatorname{attn}^\theta$ is a suitable feature map.

When convenient, let $\operatorname{attn}^{\theta}$ be as in either of <Ref> depending on which assumption of Assumptions <ref> (i), (ii), or (iii) holds.  For convenience, we denote the map $\operatorname{attn}$ by $\varphi$.  These lemmata show that the map $\varphi$ is a bi-Lipschitz embedding of $([0:T]\times K,\|\cdot\|\times \|\cdot\|_T)$ into a Euclidean space $(\mathbb{R}^N,\|\cdot\|_2)$.

We also observe that every bi-Lipschitz map is a quasi-symmetric map[See <cit.>.], as defined on <cit.>.  Thus, <cit.> implies that $(\mathcal{M},d_g)$ is a doubling metric space, as defined on <cit.>.  Since $(\mathcal{M},d_g)$ and $(K,\|\cdot\|_T)$ since both are isometric then $(K,\|\cdot\|_T)$ is also a doubling metric space (see <cit.>).  Thus, $([0:T]\times K,\|\cdot\|\times \|\cdot\|_T)$ is a doubling metric space.  We have thus verified <cit.>.

* Feature Space Geometry.
Since the codomain of $\phi$ is simply a Euclidean space, then, the constant sequence of identity maps $\{T^i\eqdef 1_{\mathbb{R}^{\edim+1}}\}_{i=1}^{\infty}$ are trivially finite-rank linear operators realizing the bounded approximation property on any compact subset of $\mathbb{R}^{\edim+1}$.  That is, for each non-empty compact $A\subseteq\mathbb{R}^{\edim+1}$,

\[
\lim\limits_{i\mapsto \infty}\,\max_{u\in A}\,\|u-T^i(u)\|_2 = \lim\limits_{i\mapsto \infty}\,\max_{u\in A}\,\|u-1_{\mathbb{R}^N}(u)\|_2 = 0
\]

and the operator norm $\|1_{\mathbb{R}^{\edim+1}}\|_{\mathrm{op}}=1$. Thus, $(T^i)_{i\in \mathbb{N}}$ implements the $1$-BAP ($1$-bounded approximation property) on $\mathbb{R}^{\edim+1}$ for every $i$.  Therefore, <cit.> holds.

* Geometry of $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},\mathcal{Q},\eta)$.
<Ref> shows that $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},\mathcal{Q},\eta)$ is barycentric and it is a QAS space with quantized mixing function, see <cit.>, given for any $\ddim \in\mathbb{N}_+$, $u\in \mathbb{R}^\ddim$, and $(m^{(1)},A^{(1)}),\dots,(m^{(N)},A^{(\ddim)})\in \mathbb{R}^{d_X}\times \mathbb{R}^{d_X^2}$ by

\begin{equation}
\label{eq:QMF}
\hat{\eta}\big(
\{(m^{(n)},A^{(n)}\}_{n=1}^\ddim
\big)
\eqdef
\sum_{n=1}^\ddim\,
\cdot
\end{equation}

This verifies <cit.>.

* Determining The Euclidean Universal Approximator.

We now verify that the class of all MLPs with activation function $\sigma$ satisfying <Ref>, thus assumption <cit.> holds.  The case where $\sigma=\operatorname{ReLU}$ and $\sigma\neq \operatorname{ReLU}$ are treated separately.

First, consider the case where $\sigma\neq \operatorname{ReLU}$.
Then, for each $\edim,\ddim,c\in \mathbb{N}_+$ let $\mathcal{F}_{\edim, \ddim, c}$ denote the family of maps $f:\mathbb{R}^\edim\rightarrow \mathbb{R}^\ddim$ with representation (<ref>) and satisfying

\begin{equation}
\label{eq:GeneralSigma_Version}
J \le c
\mbox{ and }
\max_{j=0,\dots,J}\,
\le \edim+\ddim+3
\end{equation}

Since $\sigma$ was assumed to satisfy <Ref> then by <cit.>, as formulated in <cit.>, implies that $\mathcal{F}_{\cdot}$ is a universal approximator in the sense of <cit.>.  Moreover, its rate function is recorded in <cit.>.  Therefore, $\mathcal{F}_{\cdot}$, as defined in (<ref>), verifies <cit.>.

Next, suppose that $\sigma=\operatorname{ReLU}$.  Then, for each $\edim,\ddim,c\in \mathbb{N}_+$ let $\mathcal{F}_{\edim,\ddim,c}$ denote the family of maps $f:\mathbb{R}^\edim\rightarrow \mathbb{R}^\ddim$ with representation (<ref>) and satisfying

\begin{equation}
\label{eq:ReLU_Version}
J \le c
\mbox{ and }
\max_{j=0,\dots,J}\,
\le c
\end{equation}

By <cit.>, $\mathcal{F}_{\cdot}$ is a universal approximator in the sense of <cit.>.  Moreover, its rate function is given in <cit.>, as recorded in <cit.>.  In either case, $\mathcal{F}_{\cdot}$, as defined in (<ref>), verifies <cit.>.

* Applying <cit.>.
Steps $1$ though $5$ verify that the conditions of <cit.> are indeed met.  Furthermore, we have just shown that we are in the special case where the feature decomposition $\{([0,T]\times K,\varphi)\}$ of $([0,T]\times K,\|\cdot\|_T)$, as defined in <cit.>, is a singleton.
Therefore, by <cit.> for every $\varepsilon>0$ there is a map $\hat{F}:[0,T]\times K\rightarrow \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ with representation
[Since the barycentric decomposition $\{(K,\varphi)\}$ of $(K,\|\cdot\|_T,\mu)$ has exactly one part then the partition of unity <cit.> is trivial and $\psi_1(x_{\cdot})=1$ for each $x_{\cdot}\in K$.]

\begin{align}
\label{eq:transformer_pre_mapped_BEGIN}
\hat{F}(y_{\cdot})
= &
\beta_{\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}}
\big(
\delta_{\hat{\eta}(P_{\Delta_\ddim}\circ \hat{f}_n\circ \phi(y_{\cdot}),(Z_n)_{n=1}^\edim)}
\big)
\\
\label{eq:transformer_barycentercancellation}
= &
\hat{\eta}(P_{\Delta_\ddim}\circ \hat{f}_n\circ \phi(y_{\cdot}),(Z_n)_{n=1}^\edim)
\\
\label{eq:transformer_pre_mapped_END}
= &
\sum_{n=1}^\ddim
\,
\big(
P_{\Delta_\ddim}\circ \hat{f}_n\circ \phi(y_{\cdot})
\big)
\cdot
\end{align}

for each $y_{\cdot}\in K$ satisfying

\[
\sup_{y_{\cdot}\in K}\,
\hat{F}(y_{\cdot})
\big)
\varepsilon_A + \varepsilon_Q + \varepsilon_E = \varepsilon
\]

where $\varepsilon_E=0$ and $\varepsilon_A\eqdef \varepsilon_Q\eqdef \varepsilon/2$ and where $\hat{f}\in \mathcal{F}_{d_n,N_n,c_n}$, $Z_n\in \mathbb{R}^{N_n\times D_n}$, for some positive integers $d_n,c_n,D_n$, and $N_1,\dots,N_N\in \mathbb{N}_+$ recorded in <cit.>, and $\smash{\beta_{\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}}}$ is a $1$-Lipschitz barycenter map on $(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F})$ (which exists by <Ref>).  We observe that that (<ref>) follows from the fact that the barycenter map $\beta_{\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}}$ is a right-inverse of the map $\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}\ni (m,B)\mapsto \delta_{(m,B)}\in \mathcal{P}_1(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X})$, where $\mathcal{P}_1(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X})$ denotes the $1$-Wasserstein space on $(\mathcal{P}_1(\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}),d_{2,F})$ and (<ref>) follows from the expression for $\hat{\eta}$ given in (<ref>).  Consequentially, eq:transformer_pre_mapped_BEGIN,eq:transformer_barycentercancellation,eq:transformer_pre_mapped_END reduces to (<ref>).

* Tallying Parameters.
Since the quantitative version of <cit.> held, namely <cit.>, then we obtain the following parameter estimates

* $\edim=\mathcal{N}_{\operatorname{pack}}\big(K,C_{\ldim}\,\operatorname{Vol}(\mathcal{M},g)^{1/\ldim}\big) \in \mathcal{O}(1)$,

* $c$ is recorded in <Ref> as the depth of the network $\hat{f}$,

* The expression of $\ddim$ is recorded, in detail, atop <cit.> and is

\[
\ddim
\le
\big(
C_{K,1}^{\lceil \frac1{4\alpha}\rceil}
\big)^{
\log_2(\operatorname{diam}(K)) - \frac1{\alpha}\log_2(\epsilon_A/(2L C_{K:2}))
\in
\mathcal{O}_{K}\big(
\big)
\]

for constants $C_{K,1},C_{K,2},C_K>0$ depending only on the compact set $K$ and on the mixing function $\eta$, $\alpha=1$ as $f$ is $1$-Hölder, and where $\mathcal{O}$ suppresses a constant depending only on $K$ and on the mixing function $\eta$ in (<ref>).

This concludes the proof.

§.§.§ Proof of the Main Approximation Lemma

In what follows, for any $d\in \mathbb{N}_+$, let $\overline{\mathcal{N}_d}$ denotes the set of Gaussian measure on $\mathbb{R}^d$ equipped with the $2$-Wasserstein metric $\mathcal{W}_2$.

The Lemmata in the previous sections together with the main results of <cit.> and <cit.> to deduce our main approximation theoretic tool, namely, <Ref>.

of <Ref>

* Bounded The Local Lipschitz Stability of $\rho$ on $f(K)$.
By <Ref>, the map $\varrho:(\mathcal{N}_{d_X},\mathcal{W}_2)\rightarrow (\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F})$ is continuous.  By definition of the product topology, see <cit.>, the projection map $\pi:(\mathbb{R}^{d_X},\operatorname{Sym}_{0,d_X},d_{2,F})\ni (\mu,\Sigma)\rightarrow \Sigma \in (\operatorname{Sym}_{0,d_X},\|\cdot\|_F)$ is continuous.
Since the composition of continuous functions is again continuous, then the map

\[
g: ([0,T]\times K,
| \cdot | \times \|\cdot\|_T)\ni (t,y_{\cdot}) \mapsto
\pi\circ \varrho\circ f(t,y_{\cdot})
\in (\mathbb{R},|\cdot|)
\]

is continuous.  By <cit.>, $\tilde{K}\eqdef g(K)$ is a compact subset of $(\mathbb{R},|\cdot|)$.

By <cit.> the map $\lambda_{\text{min}}:(\operatorname{Sym}_{0,d_X},\|\cdot\|_F)\rightarrow (\mathbb{R},|\cdot|)$ which sends any $d_X \times d_X$ square matrix $\Sigma$ to its minimal eigenvalue $\lambda_{\text{min}}(\Sigma)$ is continuous.  Since every continuous function with compact domain achieves its minimum on its domain then, there exists some $\Sigma_0\in \tilde{K}$ minimizing $\lambda_{\text{min}}$; by which we mean that

\begin{equation}
\label{eq:prop:Universal_Approximation_Theorem__PathToNd___minimaleigenvalueachieved}
\lambda_{\text{min}}(\Sigma_0)
\min_{\Sigma\in g(K)}\,\lambda_{\text{min}}(\Sigma)
\infty
\end{equation}

Since $f$ takes values in $\mathcal{N}_{d_X}$ then $\pi\circ \varrho(f(x))$ is positive definite, for each $x\in K$.  In particular, $\lambda_{\text{min}}(\Sigma_0)>0$.  Consequentially, (<ref>) implies that

\begin{equation}
\label{eq:prop:Universal_Approximation_Theorem__PathToNd___minimaleigenvalue}
\eqdef
\lambda_{\text{min}}(\Sigma_0)
\min_{\Sigma\in g(K)}\,\lambda_{\text{min}}(\Sigma)
\infty
\end{equation}

By <Ref>, we have that there is an $r>0$ such that $r=\lambda_{\operatorname{min}}(\Sigma_t)$ for all $0\le t\le T$.
Consequentially, <Ref> implies that the map $\varrho:(\mathcal{N}_{d_X},\mathcal{W}_2)\rightarrow (\mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X},d_{2,F})$ is Lipschitz on $\tilde{K}$ with Lipschitz constant bounded-above by $\operatorname{Lip}\big(\varrho|f(K)\big)$.  Consequentially, we have that

\begin{equation}
\label{eq:prop:Universal_Approximation_Theorem__PathToNd___Lipschitzvarrho}
\operatorname{Lip}\big(\varrho\circ f | K\big)
\le
\operatorname{Lip}\big(\varrho|f(K)\big)
\,
\operatorname{Lip}\big(f|K\big)
\le
\max\big\{1,\frac{\sqrt{d}}{2\sqrt{r}}\big\}
\,
\end{equation}

* Approximating $\varrho \circ f$ on $K$.
Fix $\varepsilon>0$ and fix the “perturbed approximation error”

\begin{equation}
\label{eq:prop:Universal_Approximation_Theorem__VALUEOFDelta}
\delta
\eqdef
\min\biggl\{
\frac{\varepsilon
\sqrt{d}2\sqrt{
\,\sqrt{T^2+\operatorname{diam}(K)^2} + 1
\biggr\}
\end{equation}

We apply <Ref> to deduce that there exists a map $\hat{F}: [0:T]\times K \rightarrow \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}$ with representation (<ref>) satisfying the uniform estimate
\begin{equation}
\label{eq:prop:Universal_Approximation_Theorem__PathToNd___uniformestimate}
\max_{(t,y_{\cdot}) \in [0:T]\times K }\,
d_{2,F}(\hat{F}(t, y_\cdot),\varrho\circ f(t,y_\cdot))
\delta
\end{equation}

Since <Ref> showed that $\varrho$ has a locally bi-Lipschitz homeomorphism then, in particular, $\varrho^{-1}$ exists and it is Lipschitz continuous.
Consider the $1$-thickening of $\varrho\circ f(K)$ defined by

\[
\bar{B}_1
\eqdef
\{(m,\Sigma)\in \mathbb{R}^{d_X}\times \operatorname{Sym}_{0,d_X}:\,(\exists (t,y_{\cdot} )\in [0:T]\times K )\, d_{2,F}(f(t, y_\cdot),(m,\Sigma))\le 1\}
\]

Since $\delta$ was defined, in (<ref>), to be at-most $1$, then (<ref>) implies that

\begin{equation}
\label{eq:prop:Universal_Approximation_Theorem__PathToNd___CONTAINEMENTAPPROXIMATION}
\hat{F}(K)\subseteq \bar{B}_1
\end{equation}

From (<ref>) and <Ref> we deduce the uniform estimate

\begin{align}
\label{eq:Approxim_LEMMA__BEGIN}
& \max_{(t,y_{\cdot})\in [0:T]\times K}\,
\mathcal{W}_2\big(
\rho^{-1}\circ F(t,y_{\cdot})
\big)
\\
\nonumber
= &
\max_{(t,y_{\cdot})\in [0:T]\times K}\,
\mathcal{W}_2\big(
\varrho^{-1}\circ \hat{F}(t,y_{\cdot})
\varrho^{-1}\circ \varrho\circ f(t,y_{\cdot})
\big)
\\
\nonumber
\le &
\operatorname{Lip}\big(
\varrho^{-1}
\bar{B}_1
\big)
\,
\max_{(t,y_{\cdot})\in [0:T]\times K}\,
\hat{F}(t,y_{\cdot})
\varrho\circ f(t,y_{\cdot})
\big)
\\
\nonumber
\le &
\operatorname{Lip}\big(
\varrho^{-1}
\bar{B}_1
\big)
\,
\delta
\\
\label{eq:prop:Universal_Approximation_Theorem__PathToNd___BOUND_A}
\le &
\operatorname{Lip}\big(
\varrho^{-1}
\bar{B}_1
\big)
\,
\frac{\varepsilon
\sqrt{d}2\sqrt{
\,\sqrt{T^2+\operatorname{diam}(K)^2} + 1
\\
% \label{eq:prop:Universal_Approximation_Theorem__PathToNd___BOUND_B}
% \le &
%     \sqrt{d}2\sqrt{
%             L
%     \,\sqrt{T^2+\operatorname{diam}(K)^2} + 1
%         }
%     \,
%     \varepsilon
%     \,
%     \frac1{
%     \sqrt{d}2\sqrt{
%         L
% \,\sqrt{T^2+\operatorname{diam}(K)^2} + 1
%     }
%     }
% \\
\label{eq:Approxim_LEMMA__END}
\le &
\varepsilon
\end{align}

where (<ref>) followed from <Ref> and the definition of $\bar{B}_1$
and bound (<ref>) followed from the estimate (<ref>) for $\operatorname{Lip}( \varrho^{-1}  | \bar{B}_1 )$.  Since $\mathcal{W}_p\le \mathcal{W}_2$ for all $1\le p\le 2$, see <cit.> then, the estimates in (<ref>)-(<ref>) imply that

\begin{align}
\max_{(t,y_{\cdot}) \in [0:T]\times K}\,
\mathcal{W}_p\big(
\hat{F}(t,y), f(y)
\big)
\le &
\varepsilon
\end{align}

for all $1\le p\le 2$; as claimed.

* Counting Parameters.
Using $\delta$, as defined in (<ref>), in the place of $\varepsilon$ in <Ref> and noting that $\delta^{-1} \in \mathcal{O}(\varepsilon^{-1})$ yields the conclusion.

§.§.§ Polish QAS Space Structure on $d$-dimensional Gaussian Measures - OLD Construction

package this nicely...

Let $\overline{\mathcal{N}_{d_X}}$ denote the set of Gaussian measure on $\mathbb{R}^{d_X}$ equipped with the $2$-Wasserstein metric $\mathcal{W}_2$.
We now show that $\overline{\mathcal{N}_{d_X}}$ is a QAS space and we exhibit one such QAS structure; in the sense of <cit.>.
Show that $\overline{\mathcal{N}_{d_X}}$ can be identified with all $d$-dimensional Guassian measures (using geodesic completeness)...

$(\overline{\mathcal{N}_{d_X}},\mathcal{W}_2,\hat{\eta})$ is a barycenteric, Polish QAS space with quantized mixing function $\hat{\eta}:\cup_{N=1}^{\infty}\, [\Delta_N\times \mathbb{R}^{d(d+3)/2}]\rightarrow \overline{\mathcal{N}_{d_X}}$ defined for every $N\in \mathbb{N}_+$, each $w\in \Delta_N$, and every $(m_1,A_1)$, $\dots$, $(m_N,A_N)$ we have

\[
\hat{\eta}(w,(m_n,A_n)_{n=1}^N)
\eqdef
\eta\big(w,
\mathcal{N}(m_n,\operatorname{sym}(A_n)^{\top}\operatorname{sym}(A_n))
\big)
\]

Moreover, details...

polish details

For every $q\in \mathbb{N}_+$, define the map $\mathcal{Q}_q:\mathbb{R}^{d(d+3)/2}\rightarrow \overline{\mathcal{N}_{d_X}}$ sending any $(m,A)\in \mathbb{R}^{d_X}\times \mathbb{R}^{d(d+1)/2}\cong \mathbb{R}^{d(d+3)/2}$ to the measure

\begin{equation}
\label{eq:quantization}
\mathcal{Q}(m,A)
\eqdef
\mathcal{N}(m,\operatorname{sym}(A)^{\top}\operatorname{sym}(A))
\end{equation}

Note that, $\mathcal{Q}_1=\mathcal{Q}_q$ for every $q\in \mathbb{N}_+$.  Furthermore, if $\mathcal{N}(m,\Sigma)\in \overline{\mathcal{N}_{d_X}}$ then $\Sigma$ is symmetric and positive semi-definite.  Thus, citation there is some symmetric matrix $B$ such that $\Sigma=B^2$; where $B^2\eqdef B^{\top}B$.  Since $\operatorname{sym}$ is a linear isometry between the Euclidean spaces $(\mathbb{R}^{d(d-1)/2},\|\cdot\|_2)$ and the Hilbert space of $d\times d$ symmetric matrices with Frobenius inner product citation then, for every $\mathcal{N}(m,\Sigma)\in \overline{\mathcal{N}_{d_X}}$ there exists some $(m,A)\in \mathbb{R}^d\times \mathbb{R}^{d(d+1)/2}$ such that

\[
\mathcal{Q}(m,A)
\mathcal{N}(m,\Sigma)
\]

Therefore, for any compact subset $K\subset \overline{\mathcal{N}_{d_X}}$ and any $\varepsilon>0$ the modulus of quantizability, denoted by $q_{K,\varepsilon}$ and defined in <cit.>, is identically $d(d+3)/2$.  In particular, $\overline{\mathcal{N}_{d_X}}$ is quantizable by the “constant quantization” $\mathcal{Q}_{\cdot}\eqdef \{\mathcal{Q}_q\}_{q=1}^{\infty}$.

Let us now show that $\overline{\mathcal{N}_{d_X}}$ is approximately simplicial; in the sense of <cit.>.
For any $N\in \mathbb{N}_+$, denote the $N$-simplex by $\Delta_N\eqdef \{w\in [0,1]^N:\,\sum_{n=1}^N\,w_n=1\}$.
Define the map $\eta:\cup_{N=1}^{\infty}\,[\Delta_N\times \mathcal{P}_2(\mathbb{R}^d)]\rightarrow\mathcal{P}_2(\mathbb{R}^d)$ as follows: for any $N\in \mathbb{N}_+$ and $\mathcal{N}(m_1,\Sigma_1)$, $\dots$, $\mathcal{N}(m_N,\Sigma_N)$ we define $\Delta_N\ni w\mapsto
\eta \big(
\big)
$ as a measurable selection of the set of optimizers of

\[
V(w) \eqdef \inf_{\nu \in \mathcal{P}_2(\mathbb{R}^d)}\,
\sum_{n=1}^N\, w_n \mathcal{W}_2\big(\nu,\mathcal{N}(m_n,\Sigma_n)\big)^2
\]

As shown <cit.>, such an $\eta$ is well-defined as satisfies <cit.>.  Therefore, $\big(\overline{\mathcal{N}_{d_X}},\mathcal{W}_2,\eta,\mathcal{Q}_{\cdot}\big)$ is a QAS space.  Moreover, since $\overline{\mathcal{N}_{d_X}}$ is, by definition, a closed subset of $(\mathcal{P}_2(\mathbb{R}^d),\mathcal{W}_2)$ and $(\mathcal{P}_2(\mathbb{R}^d),\mathcal{W}_2)$ is a Polish space, by <cit.>, since $\mathbb{R}^d$ is Polish; then, $\overline{\mathcal{N}_{d_X}}$ is a Polish space.

Furthermore, by <cit.> if $N\in \mathbb{N}_+$, $w\in \Delta_N$, and $\mathcal{N}(m_1,\Sigma_1)$, $\dots$, $\mathcal{N}(m_N,\Sigma_N)$ are such that $\Sigma_1$, $\dots$, $\Sigma_N$ are all non-singular then $\eta \big(
\big)
$ is uniquely determined by

\begin{equation}
\label{eq:simplified_quantization}
\eta \big(
\big)
\mathcal{N}\Biggl(
\sum_{n=1}^N\,w_n\,m_n
\Biggr)
\end{equation}

where $S_w$ is the unique solution to the matrix equation $S =\sum_{n=1}^N\, w_n\, (S^{1/2}\Sigma_nS^{1/2})^{1/2}$.

Thus, $\hat{\eta}$ is a quantized mixing map $\hat{\eta}$.  Moreover, the expression for $\eta$ can be simplified using (<ref>), in the obvious way, when $\operatorname{sym}(A_n)^{\top}\operatorname{sym}(A_n)$ is positive-definite for $n=1,\dots,N$.details

I wonder if I can leverage quantization to ensure this nice expression always holds by adding $1/q\cdot I_d$ at quantization level $\mathcal{Q}_q$...then the uniform quantization rate (modulus oQ) must be verified and is not fully trivial.

§ AN AUXILIARY LEMMA - I. PINELIS

This section records an auxiliary lemma due to Iosif Pinelis, <cit.>.  We include the result and its proof here to keep our manuscript self-contained.

Fix $d\in \mathbb{N}_+$ and $R,r>0$.  For every $m^{(1)},m^{(2)}\in \mathbb{R}^d$ and each $d\times d$ symmetric positive semi-definite matrix $A,B$ satisfying: $\|A\|_F,\|B\|_F\le R$ and $A-r\cdot I_d$ and $B-r\cdot I_d$ are positive semi-definite
I.e. $A,B\ge r\cdot I_d$ where $\ge $ is the partial ordering on the set of $d\times d$-dimensional symmetric positive-definite matrices given by $A\ge B$ if and only if $A-B$ is positive semi-definite.
then the following lower-bound holds

\begin{align*}
% Lipschitz Inverse
\frac1{
\min\{1,\sqrt{d}(2\sqrt{R})\}
\,
\sqrt{\|m^{(1)}-m^{(2)}\|^2+\|A-B\|_F^2}
\le &
\mathcal{W}_2\big(
\mathcal{N}(m^{(1)},A)
\mathcal{N}(m^{(2)},B)
\big)
\end{align*}

Moreover, the following upper-bound also holds

\begin{align*}
\mathcal{W}_2\big(
\mathcal{N}(m^{(1)},A)
\mathcal{N}(m^{(2)},B)
\big)
\le &
\max\Big\{1,\frac{\sqrt{d}}{2\sqrt{r}}\Big\}
\,
\sqrt{\|m^{(1)}-m^{(2)}\|^2+\|A-B\|_F^2}
\end{align*}

In particular, the map $\varrho:(\overline{\mathcal{N}}_d,\mathcal{W}_2)
\rightarrow
(\mathbb{R}^d\times \operatorname{Sym}_{0,d},\|\cdot\|\times \|\cdot\|_F)
$ is locally-Lipschitz; where $\|\cdot\|\times \|\cdot\|_F$ denotes the product of the Euclidean norm on $\mathbb{R}^d$ and the Fröbenius norm on the space of $d\times d$-dimensional symmetric positive semi-definite matrices $\operatorname{Sym}_{0,d}$.

of Lemma <ref>

By <cit.> the $2$-Wasserstein distance between $\mathcal{N}(m^{(1)},A)$ and $\mathcal{N}(m^{(2)},B)$ satisfies

\begin{equation*}
\mathcal{W}_2(\mathcal{N}(m^{(1)},A),\mathcal{N}(m^{(2)},B))=\sqrt{\|m^{(1)}-m^{(2)}\|^2+W_2(\mathcal{N}(0,A),\mathcal{N}(0,B))^2}.
\label{10}
\end{equation*}

Thus, $\|m^{(1)}-m^{(2)}\|\le \mathcal{W}_2(\mathcal{N}(m^{(1)},A),\mathcal{N}(m^{(2)},B))$.  Moreover, for any unit vector $u\in\R^n=\R^{n\times1}$ we have

\begin{align*}
\mathbb{E}_{X\sim \mathcal{N}(0,A),\,Y\sim \mathcal{N}(0,B)}\|X-Y\|^2
\ge &
\mathbb{E}_{X\sim \mathcal{N}(0,A),\,Y\sim \mathcal{N}(0,B)}(u^\top X-u^\top Y)^2
\\
\ge&
(\sqrt{u^\top A u}-\sqrt{u^\top B u})^2,
\end{align*}

where the last inequality holds since $u^\top X$ and $u^\top Y$ have Gaussian law with zero-mean random with respective variances $u^\top A u$ and $u^\top B u$.
Again using the inequality $\|m^{(1)}-m^{(2)}\|\le \mathcal{W}_2(\mathcal{N}(m^{(1)},A),\mathcal{N}(m^{(2)},B))$, for any unit vector $u\in\R^n=\R^{n\times1}$, we have

\begin{equation*}
\begin{aligned}
\mathcal{W}_2(\mathcal{N}(a,A),\mathcal{N}(b,B))&\ge \mathcal{W}_2(\mathcal{N}(0,A),\mathcal{N}(0,B)) \\
&\ge|\sqrt{u^\top A u}-\sqrt{u^\top B u}| \\
&=\frac{|u^\top A u-u^\top B u|}{\sqrt{u^\top A u}+\sqrt{u^\top B u}} \\
&\ge\frac{|u^\top(A-B)u|}{\sqrt{\|A\|}+\sqrt{\|B\|}} \\
\end{aligned}
\end{equation*}

for some unit vector $u\in\R^n=\R^{n\times1}$, where $\|M\|$ is the spectral norm of a matrix $M$.

\begin{equation*}
\|A-B\|\le(\sqrt{\|A\|}+\sqrt{\|B\|\,})\, \mathcal{W}_2(\mathcal{N}(m^{(1)},A),\mathcal{N}(m^{(2)},B)). \label{30}
\end{equation*}

The conclusion follows upon combining $\|m^{(1)}-m^{(2)}\|\le \mathcal{W}_2(N(m^{(1)},A),N(m^{(2)},B))$ and
$\|A-B\|\le (\sqrt{\|A\|}+\sqrt{\|B\|}) \mathcal{W}_2(\mathcal{N}(m^{(1)},A),\mathcal{N}(m^{(2)},B))$ together with the observation that $\|\cdot\|_{\lambda:2}\le \|\cdot\|_F\le \sqrt{d}\|\cdot\|_{\lambda:2}$; where

\[
\|A\|_{\lambda:2}\eqdef \sqrt{\max_{i=1,\dots,d}\,\lambda_{\text{max}}(C^{\top}C})
\]

denotes the spectral norm on the set of $d\times d$ matrices and where $\lambda_{\text{max}}(C^{\top}C)$ denotes the largest eigenvalue $C^{\top}C$ for a given $d\times d$ matrix $C$.