# Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows

Sibylle Marcotte  Rémi Gribonval  Gabriel Peyré

###### Abstract

Conservation laws are well-established in the context of Euclidean gradient
flow dynamics, notably for linear or ReLU neural network training. Yet, their
existence and principles for non-Euclidean geometries and momentum-based
dynamics remain largely unknown. In this paper, we characterize “all”
conservation laws in this general setting. In stark contrast to the case of
gradient flows, we prove that the conservation laws for momentum-based
dynamics exhibit temporal dependence. Additionally, we often observe a
“conservation loss” when transitioning from gradient flow to momentum
dynamics. Specifically, for linear networks, our framework allows us to
identify all momentum conservation laws, which are less numerous than in the
gradient flow case except in sufficiently over-parameterized regimes. With
ReLU networks, no conservation law remains. This phenomenon also manifests in
non-Euclidean metrics, used e.g. for Nonnegative Matrix Factorization (NMF):
all conservation laws can be determined in the gradient flow context, yet none
persists in the momentum case.

Machine Learning, ICML

##  1 Introduction

Discovering functions that remain unchanged during the optimization of neural
networks is important to gain insight into the properties of trained models.
While these laws are understood in Euclidean gradient flows, they are much
less studied in non-Euclidean metrics or momentum dynamics.

##### Conservation laws of Euclidean gradient flows

are known and extensively used to study the training of linear and ReLU
networks without momentum. These laws correspond to “balancedness properties”
between neurons across layers (Saxe et al., 2013; Du et al., 2018; Arora et
al., 2019). They can be leveraged to understand which specific attributes
(e.g. sparsity, low-rank, etc.) the optimization process tends to select from
the potentially infinite pool of solutions (Saxe et al., 2013; Bah et al.,
2022; Arora et al., 2018; Tarmoun et al., 2021; Min et al., 2021). They can
also be used to prove under restrictive conditions the global convergence of
gradient descent (Du et al., 2018; Arora et al., 2019; Bah et al., 2022;
Chizat & Bach, 2020; Ji & Telgarsky, 2019; Min et al., 2021). For linear and
ReLU neural networks, the number of conservation laws for a gradient flow
dynamic is determined by the dimension of a Lie algebra and there are no more
conservation laws than these “balancedness” laws (Marcotte et al., 2024).

##### Momentum and non-Euclidean metrics.

Momentum and non-Euclidean metrics are two key ideas to accelerate the
convergence of optimization schemes, enable the use of larger step sizes, and
take into account constraints on the weights. The initial idea dates back to
Polyak’s heavy ball (Polyak, 1964), which introduces a momentum in the
gradient descent algorithm to perform an extrapolation step. In the
continuous-time limit of using small step sizes, this corresponds to using a
second-order differential equation. Nesterov’s acceleration (Nesterov, 1983)
goes one step further by progressively increasing the momentum strength during
the dynamics, reaching a faster convergence rate on the class of smooth
functions. A complementary idea to better capture the curvature of the loss is
to use spatially varying metrics. The simplest cases are data-independent
metrics, which are Hessian of some potential function (Raskutti & Mukherjee,
2015). This corresponds to the continuous-time counterpart to the mirror
descent algorithm (Nemirovskij & Yudin, 1983), which is closely related to
optimization schemes using Bregman’s divergences (Bregman, 1967). Another
advantage of these non-Euclidean metrics is that they can naturally enforce
constraints, such as positivity when using the mirror descent metric
associated with the Shannon entropy potential (Bubeck et al., 2015). Data-
dependent metrics estimate the local curvature using variations around the
idea of natural gradient (Amari, 1998). They are popular for training neural
networks, using efficient low-rank approximations of the metric (Martens,
2010; Martens & Sutskever, 2012; Martens & Grosse, 2015).

##### Conservation laws and momentum.

In sharp contrast with first-order flows, conservation laws of momentum flows
and non-Euclidean metrics remain mostly unexplored. The simplest approach to
derive conservation laws is to apply Noether’s theorem (Noether, 1918) to
transformations leaving the loss invariant. Leveraging Lagrangian’s
formulations of the flows (Wibisono et al., 2016), this leads to preserving
some form of inertial quantities (Tanaka & Kunin, 2021). While gradient flows
can be seen as a small-momentum limit of second-order flow, this limit is
singular. In particular, invariances of the loss do not immediately lead to
conservation laws for gradient flows (Zhao et al., 2023; Tanaka & Kunin, 2021;
Kunin et al., 2021). One of the goals of this paper is to expose the
fundamental differences in the structure and number of the conservation laws
for gradient and momentum flows.

##### Contributions.

We define the concept of conservation laws for Momentum flows ( Section 2.3.2)
and show how to extend the framework from paper (Marcotte et al., 2024) for
non-Euclidean gradient flows and momentum flow settings (Proposition 2.9). We
prove the time independence of conservation laws in the gradient flow case,
and a non-trivial time dependence in the momentum case  (Theorem 2.1). We
uncover new conservation laws for linear networks in the Euclidean momentum
case (Theorem 4.1). These new laws are complete, as proved theoretically for
depth-two cases (Theorem 4.3, Proposition 4.2), and algorithmically through
formal computations for deeper cases (Section 4.1); We show that, in contrast,
there is no conservation law for ReLU networks in the Euclidean momentum case,
as proved theoretically for two-layer cases, and by formal computations for
deeper cases (Section 4.2). We shed light on a quasi systematic loss of
conservation when transitioning from Euclidean gradient flows to the Euclidean
momentum setting (Section 4); this loss also occurs in a non-Euclidean
context, such as in Non-negative Matrix Factorization (NMF) or for Input
Convex Neural Networks (ICNN) implemented with two-layer ReLU networks where
we find out new conservation laws for gradient flows (Theorem 4.5 and Theorem
4.6, and find none in the momentum case (See Section 4.3, Section 4.4) ; We
obtain new conservation laws in the Natural Gradient Flow case (Section 4.5).

##  2 Conservation Laws for Momentum Flows

We formalize the concept of conservation laws for momentum dynamics and
establish their generic time-dependence properties. We characterize their most
important properties in relation to certain linear spaces of vector fields.

###  2.1 Momentum dynamics

We explore learning problems with features $x_{i}\in\mathbb{R}^{m}$ and
targets $y_{i}\in\mathcal{Y}$ (typically for regression, with
$\mathcal{Y}=\mathbb{R}^{n}$) or labels (for classification) within the scope
of supervised learning. In the context of unsupervised or self-supervised
learning, the $y_{i}$ can be treated as constant. We denote
$z_{i}\coloneqq(x_{i},y_{i})$ and $Z=(x_{i},y_{i})_{i}$. Prediction is
accomplished through a parametric function
$g(\theta,\cdot):\mathbb{R}^{m}\to\mathbb{R}^{n}$ (such as a neural network),
which is trained by empirically minimizing a cost over the parameters
$\theta\in\Theta\subseteq\mathbb{R}^{D}$

$\mathcal{E}_{Z}(\theta)\coloneqq\textstyle\sum_{i}\ell(g({\theta},x_{i}),y_{i}),$
(1)

with $\ell$ a loss function. The parameter set $\Theta\subseteq\mathbb{R}^{D}$
is typically either the whole parameter space $\mathbb{R}^{D}$ or an open set
of “non-degenerate” parameters for linear or ReLU networks.

This paper studies quantities that are preserved during the minimization of
$\mathcal{E}_{Z}$ defined in (1) using dynamical flows. First-order dynamics
corresponds to gradient flows

$\dot{\theta}(t)=-M_{Z}(\theta(t))\nabla\mathcal{E}_{Z}(\theta(t))$ (2)

where $M_{Z}(\theta)\in\mathbb{R}^{D\times D}$ is typically a positive semi-
definite matrix. Conservation laws $h(\theta)$ for these flows have been
studied in-depth in the Euclidean case ($M$ is the identity matrix
$\mathrm{Id}_{D}$) (Marcotte et al., 2024), and we study what happens in a
non-Euclidean setting. Another goal is to go beyond first-order dynamics and
to analyze which functions $h(t,\theta,\dot{\theta})$ are preserved during the
momentum flow of $\mathcal{E}_{Z}$:

$\ddot{\theta}(t)+\tau(t)\dot{\theta}(t)=-M_{Z}(t,\theta(t),\dot{\theta}(t))\nabla\mathcal{E}_{Z}(\theta(t)).$
(3)

We will always assume that $t\mapsto\tau(t)$ is infinitely smooth. To
anticipate our findings, let us immediately mention that introducing momentum
leads to conserved quantities that depend on time and velocity, and results in
many cases of interest in a reduction of conservation properties. The latter
phenomenon consistently emerges across all our examples, whether in Euclidean
($M=\mathrm{Id}_{D}$) or non-Euclidean settings, such as in non-negative
matrix factorization. Consequently, we will draw comparisons with the gradient
flow scenario.

###  2.2 Main Examples

We consider several settings of practical interest.

##### Examples of models.

Prime examples are two-layer linear or ReLU networks, where $\theta=(U,V)$
with matrices $U,V$ of appropriate sizes and $g(\theta,x)\coloneqq
U\sigma(V^{\top}x)$. Here $\sigma=\mathtt{id}$ for the linear case, while
$\sigma$ is the entrywise ReLU activation function for ReLU networks. In the
latter case, deeper examples as well as biases can also be considered.

##### Example of flows.

In terms of flows, we consider: Gradient flows (GF), corresponding to (2).
This can informally be thought of as using $\tau=\infty$ in (3) (noticing that
the matrix $M_{Z}$ in (3) can depend on $\tau$ via $t$). Heavy ball, i.e. (3)
with a fixed $\tau(t)=\tau<\infty$. This corresponds to a continuous-time
limit of Poliak’s heavy ball acceleration method (Polyak, 1964). Nesterov
acceleration, i.e. (3) with $\tau(t)=3/t$. This corresponds to the flow
introduced by (Su et al., 2016) as a continuous-time limit of Nesterov
accelerated gradient descent scheme (Nesterov, 1983).

##### Example of metrics.

We illustrate our findings on: Euclidean geometry, i.e.
$M_{Z}(t,\theta,\dot{\theta})=\mathrm{Id}_{D}$. Mirror geometry, associated to
the Shannon entropy potential, uses $M_{Z}(\theta)=\mathrm{diag}(\theta)$ for
gradient flows, and in the Heavy ball case, it uses
$M_{Z}(t,\theta,\dot{\theta})=\mathrm{diag}(\dot{\theta}+\tau\theta)$
(Wibisono et al., 2016). The associated flow is a continuous time limit of
mirror descent (Nemirovskij & Yudin, 1983). While such approaches were
initially developed for first-order schemes, their extension to second-order
flows that we use is derived in (Wibisono et al., 2016) as a flow for a
Bregman-type Lagrangian. If this paper particularly focuses on the case of the
Shannon entropy potential as an example, note that our theory applies to any
mirror potential. Natural gradient (Amari, 1998) avoids the issues of Newton’s
method using a first-order estimation of the curvature (hence the naming
“Hessian free”). Assuming again $\tau(t)=\tau$ is constant, it uses a data-
dependent metric
$M_{Z}(t,\theta,\dot{\theta})=H_{Z}(\dot{\theta}+\tau\theta)^{\dagger}$, where
$A^{{\dagger}}$ denotes the pseudo-inverse of $A$ and where (for the mean
square loss function)
$H_{Z}(\xi)\coloneqq\frac{1}{n}\sum_{i}\partial_{1}g(\xi,x_{i})^{\top}\partial_{1}g(\xi,x_{i})$
is a proxy for the Hessian that captures the curvature of the loss. Here
$\partial_{1}g$ is the differential of $g$ with respect to its first variable.

##### Running examples.

We consider several examples. Principal component analysis (PCA) corresponds
to linear neural networks ($\sigma=\mathtt{id}$) with Euclidean geometry to
perform dimensionality reduction via matrix factorization. Multilayer
Perceptrons (MLP) use $\sigma=$ReLU and Euclidean geometry. Non-negative
matrix factorization (NMF) uses a linear network (i.e. $\sigma=$Id) with
mirror geometry (for the Shannon entropy) to impose positivity on the factors
(Lee & Seung, 1999). Input Convex Neural Networks (ICNN) (Amos et al., 2017)
use a hybrid Euclidean/Mirror (for the Shannon entropy) geometry with
$\sigma=$ReLU to impose positivity on some weights, to represent convex
functions. It finds applications in implicit deep learning (Amos et al., 2017)
and to compute optimal transport in high dimension (Makkuva et al., 2020).

This table indicates the number of conservation laws that we characterize (for
two layers and $r$ hidden neurons) for  gradient flow ($\tau(t)=\infty$) and
for  momentum flows ($\tau(t)<+\infty$).

For the example of PCA, the information presented in the table corresponds to
the case where $r$ is small enough, and the general case is fully addressed in
Proposition 4.4.

###  2.3 Conserved functions

In the GF (resp. MF – momentum flow) scenario, a function $h(t,\theta)$ (resp.
$h(t,\theta,\dot{\theta})$) is conserved if for each solution111The existence
of such a solution is discussed in Definition 2.2. $\theta(\cdot)$ to the ODE
(2) (resp. (3)) with arbitrary initialization, the quantity $h(t,\theta(t))$
(resp. $h(t,\theta(t),\dot{\theta}(t))$) remains constant in time.

####  2.3.1 Time-dependence: GF vs MF

We postpone the formal definition of conserved functions and their
characterization to first highlight an important fact (See Appendix A for a
proof): in the momentum setting conservation laws can depend non-trivially on
time. This is in sharp contrast with the gradient flow case.

######  Theorem 2.1 (Structure theorem).

Let $h(t,\theta)$ (resp. $h(t,\theta,\dot{\theta})$) be a conserved function
for the ODE (2) (resp. (3) with $\tau(t)=\tau$) when its right-hand side is
zero. For all $t$ and $\theta$, one has $h(t,\theta)=h(0,\theta)$ (resp.
$h(t,\theta,\dot{\theta})={H}(\theta+\frac{\dot{\theta}}{\tau},\dot{\theta}\exp(\tau
t))$, where ${H}(a,b)\coloneqq h(0,a-\frac{b}{\tau},b)$). In particular,
conserved functions in the MF scenario can be expressed with $2D$ variables
(instead of $2D+1$).

Conservation laws (to be soon formally defined) are notably conserved
functions of the ODE when the right-hand side is zero, hence the above theorem
directly applies.

####  2.3.2 Formal definition via phase-space lifting

To formally define conserved functions in the momentum case, we generalize the
related notions from (Marcotte et al., 2024) to encompass dependencies on time
$t$ and velocity $\dot{\theta}$.

Notations Given an open subset $\Omega\subseteq\Theta\subseteq\mathbb{R}^{D}$,
we denote
$\tilde{\Omega}\coloneqq\mathbb{R}\times\Omega\times\mathbb{R}^{D}\subseteq\mathbb{R}^{2D+1}$.
In particular
$\tilde{\Theta}\coloneqq\mathbb{R}\times\Theta\times\mathbb{R}^{D}$.

######  Definition 2.2 (Conservation through a flow).

Consider an open subset $\Omega\subseteq\mathbb{R}^{D}$ and a function
$F\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R}^{2D})$. By the Cauchy-Lipschitz
theorem, for each initial condition
$\texttt{init}\coloneqq(t_{\text{init}},\omega_{\text{init}})\in\tilde{\Omega}$,
there exists a unique maximal solution
$t\in\left(t_{\text{init}}-\eta_{\text{init}},t_{\text{init}}+\eta_{\text{init}}\right)\mapsto\omega(t,\texttt{init})$
of the ODE $\dot{\omega}(t)=F(t,\omega(t))$ with
$\omega(t_{\text{init}})=\omega_{\text{init}}$. A function
$h:\tilde{\Omega}\subseteq\mathbb{R}^{2D+1}\to\mathbb{R}$ is conserved on
$\Omega$ through the flow $F$ if
$h(t,\omega(t,\texttt{init}))=h(t_{\text{init}},\omega_{\text{init}})$ for
each choice of $\texttt{init}\in\tilde{\Omega}$ and every
$t\in\left(t_{\text{init}}-\eta_{\text{init}},t_{\text{init}}+\eta_{\text{init}}\right)$.

Rewrite (3) as $\dot{\omega}(t)=F(t,\omega(t))$ with
$\omega\coloneqq(\theta,\dot{\theta})$ and

$F_{Z}\left(t,\begin{pmatrix}{\theta}\\\
\dot{\theta}\end{pmatrix}\right)\coloneqq\begin{pmatrix}\dot{\theta}\\\
-\tau(t)\dot{\theta}-M_{Z}(t,\theta,\dot{\theta})\nabla\mathcal{E}_{Z}(\theta)\end{pmatrix}.$
(4)

With this expression, Definition 2.2 can be specialized.

######  Definition 2.3 (Conservation during the flow (3) with a given
dataset).

Given an open subset $\Omega\subseteq\Theta\subseteq\mathbb{R}^{D}$ and a
dataset $Z$ such that $(t,\theta,\dot{\theta})\mapsto
M_{Z}(t,\theta,\dot{\theta})\nabla\mathcal{E}_{Z}(\theta)\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R}^{D})$,
a function $h:\tilde{\Omega}\subseteq\mathbb{R}^{2D+1}\to\mathbb{R}$ is
conserved on $\Omega$ during the flow (3) if it is conserved through $F_{Z}$.

The next definition allows us to study which functions are conserved during
“all” flows defined by the ODE (3). The smoothness assumptions will enable
simpler characterizations of such functions in due time.

######  Definition 2.4 (Conservation during the flow (3) with “any” dataset).

Consider an open subset $\Omega\subset\Theta$ and a loss $\ell(z,y)$ such that
$\ell(\cdot,y)$ is $\mathcal{C}^{2}$-differentiable for all $y\in\mathcal{Y}$.
A function $h:\tilde{\Omega}\subseteq\mathbb{R}^{2D+1}\to\mathbb{R}$ is
conserved on $\Omega$ during the flow (3) for any data set if, for each data
set $Z$ such that $g(\cdot,x_{i})\in\mathcal{C}^{2}(\Omega,\mathbb{R})$ for
each $i$ and $(t,\theta,\dot{\theta})\mapsto
M_{Z}(t,\theta,\dot{\theta})\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R}^{D})$,
the function $h$ is conserved on $\Omega$ during the flow (3).

While the above definitions are local, we are rather interested in functions
defined for the whole parameter space $\Theta$, hence the following definition
which mimics its equivalent in the gradient flow case (Marcotte et al., 2024).

######  Definition 2.5.

A function $h:\tilde{\Theta}\mapsto\mathbb{R}$ is locally conserved during the
flow (3) on $\Theta$ for any data set if for each open subset
$\Omega\subseteq\Theta$, $h$ is conserved on $\Omega$ for any data set.

######  Example 2.6.

As a first simple example, consider a two-layer linear neural network in
dimension 1 (both for the input and output), with two hidden neurons. In that
case, the parameter is
$\theta=(u_{1},u_{2},v_{1},v_{2})\subseteq\mathbb{R}^{4}=\Theta$ and the model
writes $g(\theta,x)=(u_{1}v_{1}+u_{2}v_{2})x$. Computing the derivative of
$t\mapsto h(t,\theta(t),\dot{\theta}(t))$, where
$h(t,\theta,\dot{\theta})\coloneqq
u_{1}\dot{u}_{2}-\dot{u}_{1}u_{2}+v_{1}\dot{v}_{2}-\dot{v}_{1}v_{2}$, one can
directly check that it vanishes, hence $h$ is locally conserved on
$\tilde{\Theta}=\mathbb{R}^{9}$ for any data set during the flow (3) for
$M=\mathrm{Id}_{4}$ and $\tau=0$.

A characterization of $\mathcal{C}^{1}$ conserved functions (see proof in
Appendix B) is the “orthogonality” of their gradients to an associated vector
field. This is the momentum analog to a similar property for gradient flows
(Marcotte et al., 2024).

######  Proposition 2.7 (Smooth functions conserved through a given flow).

Given $F\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R}^{2D})$, a function
$h\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R})$ is conserved through the flow
induced by $F$ if and only if $\langle\nabla
h(\alpha),(1,F(\alpha)^{\top})^{\top}\rangle=0$ for all
$\alpha\in\tilde{\Omega}$.

The following characterization is proved in Appendix C.

######  Proposition 2.8.

A function
$h:\tilde{\Theta}\mapsto\mathbb{R}\in\mathcal{C}^{1}(\tilde{\Theta},\mathbb{R})$
is locally conserved on $\Theta$ for any data set if and only if $\nabla
h(\alpha)\perp W_{\alpha}^{\mathtt{mom}}$ for all $\alpha\in\tilde{\Theta}$,
where for all $\alpha\coloneqq(t,\theta,\dot{\theta})\in\tilde{\Theta}$:

$W_{\alpha}^{\mathtt{mom}}\coloneqq\underset{Z\in\mathcal{Z}_{\theta}}{\mathrm{span}}\left\\{(1,F_{Z}(\alpha)^{\top})^{\top}\right\\},$

with $\mathcal{Z}_{\theta}$ the set of all data sets $Z=(x_{i},y_{i})_{i}$
such that there exists a neighborhood ${\Omega}$ of $\theta$ such that for all
$i$, $g(\cdot,x_{i})\in\mathcal{C}^{2}(\Omega,\mathbb{R}^{n})$ and such that
$M_{Z}(\cdot)\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R}^{D\times D})$.

The subspace $W_{\alpha}^{\mathtt{mom}}\subseteq\mathbb{R}^{2D+1}$ (that
characterizes conserved functions for a momentum flow) is linked with its
counterpart in a Euclidean GF dynamic (Marcotte et al., 2024), a subspace
denoted $W_{\theta}\subseteq\mathbb{R}^{D}$ where
$\alpha=(t,\theta,\dot{\theta})$. The space $W_{\theta}$ characterizes
conserved functions for a gradient flow in an Euclidean setting, and its
definition is recalled in Appendix D where the following proposition is
proved.

######  Proposition 2.9.

Assume that $M_{Z}(t,\theta,\dot{\theta})=M(t,\theta,\dot{\theta})$ does not
depend on the data set $Z$ and that
$M(\cdot)\in\mathcal{C}^{1}(\tilde{\Theta},\mathbb{R}^{D\times D})$. Assume
that for each $y\in\mathcal{Y}$ the loss $\ell(z,y)$ is
$\mathcal{C}^{2}$-differentiable with respect to $z\in\mathbb{R}^{n}$ and that
for each $\theta\in{\Theta}$, there exists a training dataset
$Z=(x_{i},y_{i})_{i}$ such that $\nabla\mathcal{E}_{Z}(\theta)=0$ and such
that for all $i$, $\theta\mapsto g(\theta,x_{i})$ is $\mathcal{C}^{2}$ in a
neighborhood of $\theta$. Then, for each
$\alpha\coloneqq(t,\theta,\dot{\theta})\in\tilde{\Theta}$:

$W_{\alpha}^{\mathtt{mom}}=\mathbb{R}\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\
-\tau(t)\dot{\theta}\end{smallmatrix}\right)+\left(\begin{smallmatrix}0\\\
0\\\ M(t,\theta,\dot{\theta})W_{\theta}\end{smallmatrix}\right).$ (5)

When the subspace $W_{\theta}$ for Euclidean gradient flows is known (Marcotte
et al., 2024), the above link allows us to leverage this knowledge in non-
Euclidean, momentum flow scenarios. Extensions to matrices $M_{Z}$ that depend
on the dataset $Z$, e.g. with natural gradient metrics, are briefly discussed
(in the gradient case) in Section 4.5.

######  Remark 2.10.

The assumption on the loss $\ell$ in Proposition 2.9 holds for classical
losses, see Appendix D.

######  Example 2.11.

Revisiting Example 2.6, we know that
$h(t,\theta,\dot{\theta}):=u_{1}\dot{u}_{2}-\dot{u}_{1}u_{2}+v_{1}\dot{v}_{2}-\dot{v}_{1}v_{2}$
is conserved for any data set (with $M=\mathrm{Id}_{4})$ during (3), hence by
Proposition 2.8 one has for each $\alpha=(t,\theta,\dot{\theta})$: $\nabla
h(\alpha)\perp W^{\mathtt{mom}}_{\alpha}$. By (5), one has in particular
$\nabla
h(\alpha)=(0,\dot{u}_{2},-\dot{u}_{1},\dot{v}_{2},-\dot{v}_{1},-u_{2},u_{1},-v_{2},v_{1})^{\top}\perp(1,\dot{u}_{1},\dot{u}_{2},\dot{v}_{1},\dot{v}_{2},0,\ldots,0)=\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\ 0\end{smallmatrix}\right)$ (remember that $\tau=0$). This will
be further elaborated on in Example 2.18.

The analog of Proposition 2.9 for gradient flows was established in (Marcotte
et al., 2024) in the Euclidean case, and is naturally extended to the non-
Euclidean case.

######  Proposition 2.12 (Locally conserved function for any data set for
(2)).

Assume $M_{Z}(\cdot)\in\mathcal{C}^{1}(\Theta,\mathbb{R}^{D})$. A function
$h:\Theta\mapsto\mathbb{R}\in\mathcal{C}^{1}(\Theta,\mathbb{R})$ is locally
conserved on $\Theta$ for any data set during the flow (2) if and only if
$\nabla h(\theta)\perp
W_{\theta}^{\texttt{grad}}\coloneqq\mathrm{span}\\{M_{Z}(\theta)\nabla\mathcal{E}_{Z}(\theta):Z\in\mathcal{Z}_{\theta}^{\prime}\\}$,
with $\mathcal{Z}_{\theta}^{\prime}$ the set of all data sets
$Z=(x_{i},y_{i})_{i}$ such that there exists a neighborhood ${\Omega}$ of
$\theta$ such that for all $i$,
$g(\cdot,x_{i})\in\mathcal{C}^{2}(\Omega,\mathbb{R}^{n})$. When $M_{Z}$ does
not depend on $Z$, we simple have
$W_{\theta}^{\texttt{grad}}=M(\theta)W_{\theta}$.

###  2.4 From conserved functions to conservation laws

To provide an algorithmic procedure to determine these functions, (Marcotte et
al., 2024) makes the fundamental hypothesis that the model $g(\theta,x)$ can
be (locally) factored via a reparametrization $\phi$ as $f(\phi(\theta),x)$.
They require that the model $g(\theta,x)$ satisfies the following central
assumption.

######  Assumption 2.13 (Local reparameterization).

There exists a dimension $d$ and a function
$\phi\in\mathcal{C}^{2}(\Theta,\mathbb{R}^{d})$ such that: for each parameter
$\theta_{0}$ in the open set $\Theta\subseteq\mathbb{R}^{D}$, for each
$x\in\mathcal{X}$ such that $\theta\mapsto g(\theta,x)$ is $\mathcal{C}^{2}$
in a neighborhood of $\theta_{0}$, there is a neighborhood $\Omega$ of
$\theta_{0}$ and a function
$f(\cdot,x)\in\mathcal{C}^{2}(\phi(\Omega),\mathbb{R}^{n})$ such that

$\vspace{-0.2cm}\forall\theta\in\Omega,\quad g(\theta,x)=f(\phi(\theta),x).$
(6)

Moreover, (Marcotte et al., 2024) shows that such reparametrizations exist for
linear and layered ReLU neural networks of any depth. They are respectively
denoted $\phi_{\mathtt{Lin}}(\cdot)$ and $\phi_{\mathtt{ReLU}}(\cdot)$, and
detailed in (Marcotte et al., 2024, Examples 2.10, 2.11 and Appendix C). We
recall the expression of such reparametrizations in the two-layer case.

######  Example 2.14.

(Factorization for two-layer linear neural networks) In the two-layer case,
with $r$ neurons, denoting $\theta=(U,V)\in\mathbb{R}^{n\times
r}\times\mathbb{R}^{m\times r}$ (so that $D=(n+m)r$), we can factorize
$g(\theta,x)\coloneqq UV^{\top}x$ by the reparametrization
$\phi_{\mathtt{Lin}}(\theta)\coloneqq UV^{\top}\in\mathbb{R}^{n\times m}$
(identified with $\mathbb{R}^{d}$, $d=mn$).

######  Example 2.15 (Factorization for two-layer ReLU networks).

Consider $g(\theta,x)=\big{(}\sum_{j=1}^{r}u_{k,j}\sigma(\langle
v_{j},x\rangle+b_{j})+c_{k}\big{)}_{k=1}^{n}$, with
$\sigma(t)\coloneqq\max(t,0)$ the ReLU activation function and
$v_{j}\in\mathbb{R}^{m}$, $u_{k,j}\in\mathbb{R}$, $b_{j},c_{k}\in\mathbb{R}$.
Then, denoting $\theta=(U,V,b,c)$ with
$U=(u_{k,j})_{k,j}=:(u_{1},\cdots,u_{r})\in\mathbb{R}^{n\times r}$,
$V=(v_{1},\cdots,v_{r})\in\mathbb{R}^{m\times r}$,
$b=(b_{1},\cdots,b_{r})^{\top}\in\mathbb{R}^{r}$ and
$c=(c_{1},\cdots,c_{n})\in\mathbb{R}^{n}$ (so that $D=(n+m+1)r+n$), we can
locally factorize $g(\theta,x)$ by the reparametrization:
$\phi_{\mathtt{ReLU}}(\theta)=((u_{j}v_{j}^{\top},u_{j}b_{j})_{j=1}^{r},c)$.
In particular, in the case without bias ($(b,c)=(0,0)$), the reparametrization
is defined by $\phi_{\mathtt{ReLU}}(\theta)=(\phi_{j})_{j=1}^{r}$ where
$\phi_{j}=\phi_{j}(\theta)\coloneqq u_{j}v_{j}^{\top}\in\mathbb{R}^{n\times
m}$ (here $d=rmn$): the reparametrization $\phi_{\mathtt{ReLU}}(\theta)$
contains $r$ matrices of size $m\times n$ (each of rank at most one).

Thanks to this reparametrization and under a mild assumption on the loss
$\ell$, Marcotte et al. (2024, Theorem 2.14) show that for linear neural
networks of any depth (resp. for two-layer ReLU neural networks), the
functions that are locally conserved for any data set in a Euclidean gradient
flow dynamic are entirely characterized by the trace
$\mathcal{W}(\theta)\coloneqq\mathrm{span}\\{w(\theta):w(\cdot)\in\mathcal{W}\\}\subseteq\mathbb{R}^{D}$
of a finite-dimensional linear space of functions, $\mathcal{W}$, determined
by $\phi_{\mathtt{Lin}}$ (resp. $\phi_{\mathtt{ReLU}}$). For these cases, they
show that for all $\theta\in\Theta$, the space $W_{\theta}$ defined in
Appendix D satisfies $W_{\theta}=\mathcal{W}_{\phi}(\theta)$ with

$\mathcal{W}_{\phi}\coloneqq\mathrm{span}\\{\nabla\phi_{1}(\cdot),\cdots,\nabla\phi_{d}(\cdot)\\},$
(7)

where the components $\phi_{i}:\mathbb{R}^{D}\to\mathbb{R}$, $1\leq i\leq d$
of $\phi$ are assumed to satisfy
$\nabla\phi_{i}\in\mathcal{C}^{1}(\Theta,\mathbb{R}^{D})$.

Assuming that $M$ does not depend on $Z$, in the following theorem, we combine
(Marcotte et al., 2024, Theorem 2.14) and Proposition 2.9 to show that, under
an assumption on the loss $\ell$,
$W_{\alpha}^{\mathtt{mom}}\subseteq\mathbb{R}^{D}$ is the trace of
$\mathcal{W}_{\phi}^{\mathtt{mom}}\coloneqq\mathrm{span}\\{\chi_{i}(\cdot):0\leq
i\leq d\\}\subseteq\mathcal{C}^{1}(\widetilde{\Theta},\mathbb{R}^{D})$ where

$\chi_{0}(\alpha)\coloneqq\left(\begin{smallmatrix}1\\\ \dot{\theta}\\\
-\tau(t)\dot{\theta}\end{smallmatrix}\right);\chi_{i}(\alpha)\coloneqq\left(\begin{smallmatrix}0\\\
0\\\
M(t,\theta,\dot{\theta})\nabla\phi_{i}(\theta)\end{smallmatrix}\right),i\geq
1,$ (8)

with $\alpha=(t,\theta,\dot{\theta})$. Similarly the subspace
$W_{\theta}^{\texttt{grad}}$ of Proposition 2.12 is the trace of

$\vspace{-0.22cm}\mathcal{W}_{\phi}^{\texttt{grad}}\coloneqq{\mathrm{span}}_{i}\\{M(\cdot)\nabla\phi_{i}(\cdot)\\}\subseteq\mathcal{C}^{1}({\Theta},\mathbb{R}^{D}).$
(9)

######  Theorem 2.16.

Assume that the loss $(z,y)\mapsto\ell(z,y)$ is
$\mathcal{C}^{2}$-differentiable with respect to $z\in\mathbb{R}^{n}$ for each
$y\in\mathcal{Y}$, and that it satisfies the condition:

$\underset{y\in\mathcal{Y}}{\mathrm{span}}\\{\nabla_{z}\ell(z,y)\\}=\mathbb{R}^{n},\forall
z\in\mathbb{R}^{n}.$ (10)

Assume that $M(\cdot)\in\mathcal{C}^{1}(\tilde{\Theta},\mathbb{R}^{D\times
D})$ (resp. $M(\cdot)\in\mathcal{C}^{1}({\Theta},\mathbb{R}^{D\times D})$).
Then, for linear neural networks, one has for all $\alpha\in\tilde{\Theta}$
with $\Theta\coloneqq\mathbb{R}^{D}$ (resp. for all $\theta\in\Theta$:

$W_{\alpha}^{\mathtt{mom}}=\mathcal{W}_{\phi_{\mathtt{Lin}}}^{\mathtt{mom}}(\alpha)\quad\text{(resp.
}W_{\alpha}^{\mathtt{mom}}=\mathcal{W}_{\phi_{\mathtt{Lin}}}^{\texttt{grad}}(\theta)).$
(11)

The same holds for two-layer ReLU networks with $\phi_{\mathtt{ReLU}}$ from
Example 2.15 and $\Theta$ the (open) set of all parameters $\theta$ such that
hidden neurons are associated to pairwise distinct “hyperplanes” (see
(Marcotte et al., 2024, Theorem 2.8)).

Condition (10) is the same as in (Marcotte et al., 2024) and is satisfied for
standard losses such as the quadratic loss. Theorem 2.16 motivates the
following definition.

######  Definition 2.17 (Conservation law for (3)).

A real-valued function $h\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R})$ is a
_conservation law of $\phi$_ for (3) if for all
$\alpha\coloneqq(t,\theta,\dot{\theta})\in\tilde{\Omega}$, $\nabla
h(\alpha)\perp\mathcal{W}_{\phi}^{\mathtt{mom}}(\alpha)$.

Combining Proposition 2.8 and Theorem 2.16, the functions
$h\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R})$ that are locally conserved on
$\Theta$ for any data set are exactly the conservation laws of
$\phi_{\mathtt{Lin}}$ (resp. of $\phi_{\mathtt{ReLU}}$ for linear (resp. ReLU)
two-layer networks).

######  Example 2.18.

Revisiting again Example 2.6, the model $g$ is factorized by
$\phi_{\mathtt{Lin}}(\theta)\coloneqq(u_{1},u_{2})\times(v_{1},v_{2})^{\top}=u_{1}v_{1}+u_{2}v_{2}\in\mathbb{R}$
with $\theta\coloneqq(u,v)\in\Theta=\mathbb{R}^{4}$. We know that
$h(t,\theta,\dot{\theta})\coloneqq
u_{1}\dot{u}_{2}-\dot{u}_{1}u_{2}+v_{1}\dot{v}_{2}-\dot{v}_{1}v_{2}$ is
locally conserved during (3) (with $M=\mathrm{Id}_{4}$) on $\tilde{\Theta}$
for any data set and indeed: by Example 2.11, we already have $\nabla
h(\alpha)\perp\chi_{0}(\alpha)$ for all
$\alpha\coloneqq(t,\theta,\dot{\theta})$ (remember that $\tau=0$) and we also
have $\nabla
h(\alpha)=(0,\dot{u}_{2},-\dot{u}_{1},\dot{v}_{2},-\dot{v}_{1},-u_{2},u_{1},-v_{2},v_{1})^{\top}\perp(0,\ldots,0,v_{1},v_{2},u_{1},u_{2})^{\top}=(0,\ldots,0,\nabla\phi(\theta)^{\top})^{\top}={\chi_{1}(\alpha)}$.
Thus $\nabla h(\alpha)\perp\mathcal{W}_{\phi}^{\mathtt{mom}}(\alpha)$: $h$ is
a conservation law of $\phi_{\mathtt{Lin}}$.

Again, we generalize from (Marcotte et al., 2024) the notion of conservation
laws for $\phi$ for a non-Euclidean GF, when $M$ does not depend on the data
set $Z$.

######  Definition 2.19 (Conservation law for (2)).

A function $h:\Theta\mapsto\mathbb{R}\in\mathcal{C}^{1}(\Theta,\mathbb{R})$ is
a conservation law of $\phi$ for (2) if $\nabla
h(\theta)\perp\mathcal{W}_{\phi}^{\texttt{grad}}(\theta)$.

Conservation laws are known for the linear case $\phi_{\mathtt{Lin}}$ (Arora
et al., 2019, 2018) and the ReLu case $\phi_{\mathtt{ReLu}}$ (Du et al.,
2018).

######  Example 2.20 (Conservation laws for linear and ReLu neural networks in
Euclidean GF scenario).

If $\theta\coloneqq(U_{1},\cdots,U_{q})$ satisfies (2) with
$M_{Z}(\theta)=\mathrm{Id}_{D}$, then for each $i=1,\cdots,q-1$ the function
$\theta\mapsto U_{i}^{\top}U_{i}-U_{i+1}U_{i+1}^{\top}$ (resp. the function
$\theta\mapsto\text{diag}\left(U_{i}^{\top}U_{i}-U_{i+1}U_{i+1}^{\top}\right)$)
defines a set of conservation laws for $\phi_{\mathtt{Lin}}$ (resp. for
$\phi_{\mathtt{ReLu}}$).

##  3 Finding Conservation Laws

In this section, we propose a constructive way to build some conservation
laws. Then we explain how we can certify if there are conservation laws that
are missing or not.

###  3.1 Constructing Conservation Laws

Conservation laws can be built using formal calculus, via the orthogonal
relation that defines conservation laws (Definition 2.19 and Definition 2.17),
as done for Euclidean-GF in (Marcotte et al., 2024). One can also exploit
invariances in the spirit of Noether’s theorem (Noether, 1918; Wibisono et
al., 2016; Tanaka & Kunin, 2021).

##### Using formal calculus.

By Definition 2.17 (resp. Definition 2.19), $h$ is a conservation law for (3)
(resp. (2)) if

$\langle\nabla h(\alpha),\chi_{i}(\alpha)\rangle=0,\
\forall\alpha=(t,\theta,\dot{\theta})\in\tilde{\Theta},\ 0\leq i\leq d,$ (12)

where the $\chi_{i}$ are the $d+1$ vector fields defined in (8) that span the
linear function space $W_{\alpha}^{\mathtt{mom}}$ (resp. if

$\langle\nabla h(\theta),M(\theta)\nabla\phi_{i}(\theta)\rangle=0,\
\forall\theta\in\Theta,\ 1\leq i\leq d.$ (13)

One could seek conservation laws that belong to the finite-dimensional linear
space of polynomials of a given degree, as in (Marcotte et al., 2024) for the
Euclidean GF scenario. However, for MF scenarios, Theorem 2.1 suggests that
the conserved functions include a non-polynomial term in $t$, specifically
$\exp(\tau t)$ when $\tau(t)=\tau\neq 0$. Thus, in the MF case when
$\tau(t)=\tau$, instead of considering conserved functions under the form
$h(t,\theta,\dot{\theta})$, we use a change a variable $s\coloneqq\exp(\tau
t)$ to consider “modified” conserved functions $\tilde{h}$ that are under the
form $\tilde{h}(s,\theta,\dot{\theta})\coloneqq h((\ln
s)/\tau,\theta,\dot{\theta})$. Simple calculus yields $\nabla
h(t,\theta,\dot{\theta})=\mathrm{diag}(\tau s,1,\ldots
1)\nabla\tilde{h}(s,\theta,\dot{\theta}),$ therefore, given any constant $c$
and any vector field
$f:\Omega\times\mathbb{R}^{D}\to\mathbb{R}^{D}\times\mathbb{R}^{D}$, we have:
for every
$(s,\theta,\dot{\theta})\in\mathbb{R}_{+}^{*}\times\Omega\times\mathbb{R}^{D}$:
$\nabla h(t,\theta,\dot{\theta})\perp\left(\begin{smallmatrix}c\\\
f(\theta,\dot{\theta})\end{smallmatrix}\right)\Leftrightarrow\nabla\tilde{h}(s,\theta,\dot{\theta})\perp\left(\begin{smallmatrix}\tau
sc\\\ f(\theta,\dot{\theta})\end{smallmatrix}\right).$ Recalling that we
consider the case $\tau(t)=\tau$, all the vector fields
$\chi_{i}(t,\theta,\dot{\theta})$ from (8) involved in the definition of
$W_{\alpha}^{\mathtt{mom}}$ do not depend on time, and their first coordinates
are constant. Consequently, defining
$\tilde{\chi}_{i}(s,\theta,\dot{\theta})\coloneqq\mathrm{diag}(\tau s,1,\ldots
1)\chi_{i}(0,\theta,\dot{\theta}),$ we can solve222Concretely this is
expressed as a linear system. (12) with the new vector fields
$\tilde{\chi}_{i}$ and determine all polynomial $\tilde{h}$ of a given degree
such that
$\nabla\tilde{h}(s,\theta,\dot{\theta})\perp\tilde{\chi}_{i}(s,\theta,\dot{\theta})$
for all $i$. Finally, since $s=\exp(\tau t)$, we can determine all
conservation laws that are polynomial in $\theta,\dot{\theta}$ and in
$\exp(\tau t)$.

In the case where $\tau(t)$ is not constant, it remains possible to try
directly to solve (12) with the initial vector fields $\chi_{i}$ in
$\mathcal{W}_{\phi}^{\mathtt{mom}}$ and see if there are polynomial
conservation laws $h$ at a given degree. In particular, for a Nesterov flow
with Euclidean metric, $\tau(t)=3/t$, it turns out that there are polynomial
conservation laws (see Section 4.1). Our code to compute them is available at
https://github.com/sibyllema/Conservation_laws_ICML.

##### Using invariances – gradient flows.

For gradient flows, invariances of the cost (1) directly lead to conservation
laws.

######  Definition 3.1 (Invariant transformation on the cost (1)).

A (one-parameter) transformation on an open set
$\Omega\subseteq\Theta\subseteq\mathbb{R}^{D}$ is a map
$T:\mathbb{R}\times\Omega\to\mathbb{R}^{D}$ such that $T(\cdot,\theta)$ is
differentiable for each $\theta\in\Omega$ and $T(0,\cdot)=\mathtt{id}$. This
transformation leaves invariant the cost (1) if for all $\theta\in\Omega$ and
for all $\epsilon\geq 0$,
$\mathcal{E}_{Z}(T(\epsilon,\theta))=\mathcal{E}_{Z}(T(0,\theta))=\mathcal{E}_{Z}(\theta)$.
When this holds, simple calculus yields for every $\theta\in\Omega$:

$\Big{\langle}\nabla\mathcal{E}_{Z}(\theta),\frac{\partial}{\partial\epsilon}T(\epsilon,\theta)\Big{|}_{\epsilon=0}\Big{\rangle}=0.$
(14)

We denote
$\Delta_{T}(\cdot)\coloneqq\frac{\partial}{\partial\epsilon}T(\epsilon,\cdot)\Big{|}_{\epsilon=0}$.

######  Example 3.2 (Fondamental example of linear transformation).

Let us consider $\theta\coloneqq(U,V)\in\mathbb{R}^{n\times
r}\times\mathbb{R}^{m\times r}$. We define $T^{A}$ the linear transformation
where for all $\epsilon\geq 0$:

$T^{A}(\epsilon,U,V)=\big{(}U\exp(\epsilon A),V\exp(-\epsilon
A^{\top})\big{)}.$ (15)

Simple calculus yields $\Delta_{T^{A}}(U,V)=(UA,-VA^{\top})$. Considering
$g(\theta,x)\coloneqq UV^{\top}x$ and any $A\in\mathbb{R}^{r\times r}$,
$T^{A}$ is an invariant transformation on (1): for all $\epsilon$ and
$\theta$, $g(T^{A}(\epsilon,\theta),\cdot)=g(\theta,\cdot)$ hence
$\mathcal{E}_{Z}(T^{A}(\epsilon,\theta))=\mathcal{E}_{Z}(\theta)$. Considering
now $g(\theta,x)\coloneqq U\sigma(V^{\top}x)$ with $\sigma$ the ReLU
activation function, a similar reasoning shows that $T^{A}$ is an invariant
transformation on (1) if $A\in\mathbb{R}^{r\times r}$ is diagonal.

From loss invariance to conservation laws for GF. In the context of a gradient
flow (2), the consequence (14) of invariance rewrites as

$\langle
M_{Z}\left(\theta(t)\right)^{-1}\dot{\theta}(t),\Delta_{T}(\theta(t))\rangle=0$
(16)

as soon as $M_{Z}(\theta)$ is invertible for every $\theta$.

As a particular consequence, for gradient flows with linear networks in a
Euclidean setting $(M\equiv\mathrm{Id}_{D})$, since (16) holds for $T=T^{A}$
with any matrix $A\in\mathbb{R}^{r\times r}$, denoting $\langle
M,N\rangle\coloneqq\mathtt{Tr}(M^{\top}N)$ we obtain
$0=\langle\dot{U}(t),U(t)A\rangle-\langle\dot{V}(t),V(t)A\rangle$ at each time
and for any $A$. Specializing this to any symmetric matrix, we obtain that
$0=\langle\dot{U},UA\rangle-\langle\dot{V},VA\rangle=1/2\frac{\mathrm{d}}{\mathrm{d}t}(\langle
U,UA\rangle-\langle V,VA\rangle)$. Thus for every symmetric matrix $A$,
$\langle U,UA\rangle-\langle V,VA\rangle$ is conserved, which coincides with
all conservation laws in that case (Arora et al., 2019, 2018; Marcotte et al.,
2024).

Similarly, for ReLU neural networks in a Euclidean setting (without bias for
simplicity, this holds with bias too as detailed in the proof of Theorem 4.6),
by restricting ourselves to elementary diagonal matrices $A=E_{i,i}$ where
$E_{i,i}$ is the one-hot matrix in $\mathbb{R}^{r\times r}$ with the
$(i,i)$-th entry being $1$, $i=1,\cdots,r$, we obtain that for all $i$,
$0=\langle\dot{U}_{i}(t),U_{i}(t)\rangle-\langle\dot{V}_{i}(t),V_{i}(t)\rangle=\tfrac{1}{2}\frac{\mathrm{d}}{\mathrm{d}t}(\langle
U_{i},U_{i}\rangle-\langle V_{i},V_{i}\rangle)$. Thus for all $i$,
$\|U_{i}\|^{2}-\|V_{i}\|^{2}$ is conserved, recovering all conservation laws
(Du et al., 2018; Marcotte et al., 2024).

##### Using invariances – momentum flows.

Invariances of the cost (1) are replaced in Noether’s theorem by invariances
of a Lagrangian compatible with the flow (3).

######  Definition 3.3 (Lagrangian).

A Lagrangian (Noether, 1918; Wibisono et al., 2016) is a function of three
variables $\mathcal{L}\left(t,\theta,v\right)$. The second order dynamical
system (3) is compatible with this Lagrangian if for every solution
$\theta(t)$ of (3) the Euler-Lagrange equation of $\mathcal{L}$ holds:

$\frac{\mathrm{d}}{\mathrm{d}t}\Big{(}\partial_{v}\mathcal{L}\big{(}t,\theta(t),\dot{\theta}(t)\big{)}\Big{)}=\partial_{{\theta}}\mathcal{L}\big{(}t,\theta(t),\dot{\theta}(t)\big{)}.$

######  Example 3.4 (Euclidean momentum (Wibisono et al., 2016)).

In the case $M=\mathrm{Id}_{D}$ and $\tau(t)=\tau$ the map

$\mathcal{L}\left(t,\theta,v\right)\coloneqq\exp(\tau
t)\big{(}\tfrac{\|v\|^{2}}{2}-\mathcal{E}_{Z}(\theta)\big{)}$ (17)

is a Lagrangian of the second-order dynamical system (3). In the Euclidean
case ($M=\mathrm{Id}_{d}$) with a Nesterov flow ($\tau(t)=3/t$),

$\mathcal{L}\left(t,\theta,v\right)\coloneqq\tfrac{t^{3}}{2}\big{(}\tfrac{\|v\|^{2}}{2}-\mathcal{E}_{Z}(\theta)\big{)}$
(18)

is a Lagrangian of the second-order dynamical system (3).

More generally, (Wibisono et al., 2016) gives a Lagrangian associated to (3),
subject to: a) an hypothesis of an “ideal scaling condition” (Wibisono et al.,
2016, Equation 2.2b)); and b) the assumption that the matrix $M$ is the
inverse of the Hessian of an explicitly known metric $\psi$.

In general, a transformation leaving the cost invariant does not necessarily
leave invariant the Lagrangian of the associated dynamic (e.g., revisiting
Example 3.2, $T^{A}$ leaves the cost invariant for any
$A\in\mathbb{R}^{r\times r}$, yet only skew matrices $A$ leave the Lagrangian
(17) invariant). However, to obtain a conserved function with Noether
theorem333The full version of Noether theorem with a non-zero right-hand side
is recalled in (Tanaka & Kunin, 2021)., one needs to consider transformations
leaving invariant the Lagrangian: in that case the function
$\partial_{v}\mathcal{L}(t,\theta(t),\dot{\theta}(t))\cdot\Delta_{T}(\theta(t))$
is conserved.

######  Theorem 3.5 (Noether theorem).

Let $(\theta(t),\dot{\theta}(t))$ be a solution of (3). Then for each
transformation $T$ that leaves $\mathcal{L}$ invariant,
$\frac{\mathrm{d}}{\mathrm{d}t}\left(\partial_{v}\mathcal{L}(t,\theta(t),\dot{\theta}(t))\cdot\Delta_{T}(\theta(t))\right)=0.$

###  3.2 Finding the number of conservation laws

While being able to build conservation laws is beneficial, the question
remains: how to ensure that we have derived “all” possible laws? This first
requires to recall the definition of independent conserved functions (Marcotte
et al., 2024, Definition 2.18), to avoid functional redundancies.

######  Definition 3.6.

A family of $N$ functions $(h_{1},\cdots,h_{N})$ in
$\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R})$ is independent if for all
$\alpha\in\tilde{\Omega}$ the vectors $(\nabla h_{1}(\alpha),\cdots,\nabla
h_{N}(\alpha))$ are linearly independent.

(Marcotte et al., 2024, Theorem 3.3) link the number of independent
conservation laws to the dimension of a space involving Lie algebras.
Knowledge about Lie algebras is not mandatory in the main body of this paper,
basics are recalled in Section H.1 (See (Marcotte et al., 2024)) to support
the proofs. The space $\mathrm{Lie}(\mathcal{W})$ is the generated Lie algebra
of $\mathcal{W}$ (See Section H.1), it is entirely characterized by
$\mathcal{W}$.

######  Theorem 3.7.

If $\mathrm{dim}(\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha))$ is
locally constant then each $\alpha\in\tilde{\Omega}\subseteq\mathbb{R}^{2D+1}$
admits a neighborhood $\tilde{\Omega^{\prime}}$ such that there are
$2D+1-\mathrm{dim}(\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha))$
independent conservation laws of $\phi$ for (3) on $\tilde{\Omega^{\prime}}$.

The same theorem holds for $\mathcal{W}_{\phi}^{\texttt{grad}}$ by replacing
$2D+1$ by $D$. Besides, given a finite set of vector fields, Marcotte et al.
(2024) provide a code (detailed in Section 3.3) that computes the dimension of
the trace of their generated Lie algebra: in their case, they apply it to the
vector fields $(\nabla\phi_{i}(\cdot))_{i}$ that span $\mathcal{W}_{\phi}$ (as
defined in (7)). We can directly use this code with the finite set of vector
fields that span $\mathcal{W}_{\phi}^{\mathtt{mom}}$ as defined in (8) (resp.
on the fields $(M(\cdot)\nabla\phi_{i}(\cdot))_{i}$ that span
$\mathcal{W}_{\phi}^{\texttt{grad}}$ for the non-Euclidean GF case, see (9)).

By computing the number of independent conservation laws with this code in the
Euclidean GF scenario, Marcotte et al. (2024) (in their Section 3.3) establish
that the set of known conservation laws for linear and ReLu neural networks
(recalled in Example 2.20) is complete. In particular, they fully work out
theoretically the $2$-layer case and show (Marcotte et al., 2024, Proposition
4.2, Corollary 4.4):

######  Proposition 3.8.

If $(U;V)$ has full rank noted $\mathtt{rk}$, then in a neighborhood of
$(U,V)$, the set of $\mathtt{rk}/2\cdot(2r+1-\mathtt{rk})$ independent
conservation laws given by Example 2.20 is complete: there exists no other
conservation law.

##  4 Conservation for Neural Networks

We now exemplify our results in different settings. We study the conservation
laws for neural networks with $q$ layers, and either a linear or ReLU
activation. We write $\theta\coloneqq(U_{1},\cdots,U_{q})$ with
$U_{i}\in\mathbb{R}^{n_{i-1}\times n_{i}}$ the weight matrices. We also
consider the impact of the choice of metric. The striking outcome of this
analysis is that momentum flows radically change the structure of the
conservation laws. There are fewer (or even none) conserved quantities when
using momentum flows, and this phenomenon appears both for Euclidean and non-
Euclidean geometries.

###  4.1 PCA: linear networks with Euclidean geometry

The following theorem (proved in Appendix E) gives a set of conserved
functions when $g(\theta,x)\coloneqq U_{1}\ldots U_{q}x$. Here
$\mathcal{A}_{n}$ denotes the set of skew-symmetric matrices in
$\mathbb{R}^{n\times n}$.

######  Theorem 4.1.

Consider the model $g(\theta,x)\coloneqq U_{1}\ldots U_{q}x$. For all
$i=1,\cdots q-1$ and for all $A\in\mathcal{A}_{n_{i}}$, the function

$\exp\big{(}\textstyle\int^{t}\tau(s)\mathrm{d}s\big{)}\big{(}\langle\dot{U}_{i},U_{i}A\rangle+\langle\dot{U}_{i+1},A^{\top}U_{i+1}\rangle\big{)}$
(19)

is a conservation law for (3) with $M=\mathrm{Id}_{D}$, where
$\int^{t}\tau(s)\mathrm{d}s$ is a primitive of $\tau(\cdot)$. Moreover, for
each $i$ such that $n_{i-1}=n_{i+1}=1$ and all $A\in\mathcal{A}_{n_{i}}$ the
function
$\exp\big{(}\textstyle\int^{t}\tau(s)\mathrm{d}s\big{)}\big{(}\langle\dot{U}_{i},U_{i+1}^{\top}A\rangle+\langle\dot{U}_{i+1}^{\top},U_{i}A\rangle\big{)}$
is an additional conservation law.

In particular, for the heavy ball case $\tau(t)=\tau$, these functions exactly
correspond to the ones obtained by solving (12) with the change of variable
explained in Section 3.1. For the Euclidean Nesterov case $\tau(t)=3/t$, these
functions are also directly obtained by solving (12), with a polynomial term
$t^{3}$ in $t$. As discussed in Appendix F, in the case
$(n_{i-1},n_{i+1})\neq(1,1)$, the associated conserved functions can also be
found using Noether theorem with $T^{A}$ (defined in Example 3.2) for every
skew matrix $A$, and for the case $(n_{i-1},n_{i+1})=(1,1)$, the new
associated conserved functions can also be found using Noether theorem with a
new linear transformation $T^{\prime A}$ (defined in (29)).
The above theorem gives a set of conserved functions. A priori, they are not
all independent. The following proposition (proved in Appendix G) gives the
number of independent conserved functions given by Theorem 4.1 in the case
$q=2$. We rewrite $\theta=(U,V)$ as a vertical matrix concatenation denoted
$(U;V)\in\mathbb{R}^{(n+m)\times r}$.

######  Proposition 4.2.

Consider $\widetilde{\Theta}$ the set of $\alpha=(t,\theta,\dot{\theta})$ such
that $(\theta,\dot{\theta})=(U;V;\dot{U};\dot{V})\in\mathbb{R}^{2(n+m)\times
r}$ has full rank, and assume $(n,m)\neq(1,1)$. Then if $2(n+m)\leq r$,
Theorem 4.1 gives exactly $(n+m)[(r-2(n+m))+r-1]$ independent conserved
functions. If $2(n+m)>r$, Theorem 4.1 gives exactly $r(r-1)/2$ independent
conserved functions.

Now we want to establish if there are other conservation laws independent from
the ones obtained in Theorem 4.1. By using Theorem 3.7, we only need to
determine the dimension of the trace of a Lie algebra characterized by
$\mathcal{W}_{\phi_{\mathtt{Lin}}}^{\mathtt{mom}}$. We fully work out
theoretically the case $q=2$ as detailed in the following theorem and show
that there no other conservation laws. See Appendix H for a proof.

######  Theorem 4.3.

Consider $g(\theta,x)\coloneqq UV^{\top}x$ with $(U,V)\in\mathbb{R}^{n\times
r}\times\mathbb{R}^{m\times r}$. If
$(U;V;\dot{U};\dot{V})\in\mathbb{R}^{2(n+m)\times r}$ has full rank noted
$\mathtt{rk}$ then, in a neighborhood of $(t,U,V,\dot{U},\dot{V})$:

  * •

If $(n,m)\neq(1,1)$, there are exactly
$\mathtt{rk}/2\cdot\left(2r-1-\mathtt{rk}\right)$ independent conservation
laws of $\phi_{\mathtt{Lin}}$.

  * •

If $n=m=1$ and if $r\geq 4$, there are exactly $4r-6$ independent conservation
laws of $\phi_{\mathtt{Lin}}$.

For deeper cases, we computed
$\dim[\mathrm{Lie}(\mathcal{W}_{\phi_{\mathtt{Lin}}}^{\mathtt{mom}}){(\alpha)}]$
using the method explained in Section 3.2 with the vector fields that generate
$W_{\phi_{\mathtt{Lin}}}^{\mathtt{mom}}$ on a sample of depths/widths of small
size. This empirically confirmed (see Section M.1) that there are no other
conservation laws for deeper cases too.

Comparing the number of independent conservation laws in the GF scenario given
in Proposition 3.8 with the one in the MF scenario given in the last Theorem
4.3, we obtain as highlighted next a loss of conservation when transitioning
from Euclidean gradient flows to the Euclidean momentum setting and when
$r\leq n+m$. Notice that this case includes the factorization by matrices up
to full rank and even in mildly over-parameterized regimes. By contrast, in
the more over-parameterized regime $r>n+m$, we obtain a gain in conservation.
A proof can be found in Appendix I.

######  Proposition 4.4.

Consider $g(\theta,x)\coloneqq UV^{\top}x$ with $(U,V)\in\mathbb{R}^{n\times
r}\times\mathbb{R}^{m\times r}$. Assume both $(U;V)\in\mathbb{R}^{(n+m)\times
r}$ and $(U;V;\dot{U};\dot{V})\in\mathbb{R}^{2(n+m)\times r}$ have full rank
and $(n,m)\neq(1,1)$. Denote $N_{GF}$ (resp. $N_{MF}$) the number of
independent conservation laws of $\phi_{\mathtt{Lin}}$ for (2) (resp. for (3))
in a neighborhood of $(U,V)$ (resp. of $(t,U,V,\dot{U},\dot{V})$, then if
$r\leq n+m$: $N_{GF}-N_{MF}=r>0,$ else: $N_{GF}-N_{MF}\leq 0$.

###  4.2 MLP: ReLU Networks with Euclidean geometry

For the case $q=2$ without bias, since
$\phi_{\mathtt{ReLU}}(\theta)=(\phi_{j}(\theta))_{j=1}^{r}$ is decoupled into
$r$ functions $\phi_{j}(\theta)\coloneqq
u_{j}v_{j}^{\top}\in\mathbb{R}^{n\times m}$ each depending on a separate block
of coordinates, Jacobian matrices and Hessian matrices are block-diagonal.
Therefore Lie brackets computations can be done separately for each block,
using Theorem 4.3 with $r=1$ . We obtain that there is no conserved function
for Euclidean momentum flow with a two-layer ReLU network without bias. For
deeper ReLU networks including with bias, by computing on a sample of
depths/widths of small size the number of conservation laws as explained in
Section 3.2 using the finite set of vector fields that generates
$\mathcal{W}_{\phi_{\mathtt{ReLU}}}^{\mathtt{mom}}$, we obtain that there is
no conservation law (see details in Section M.1).

###  4.3 NMF: Linear Networks with Mirror geometry

Non-negative matrix factorization (NMF) is an example of a two-layer linear
neural network, and we use the mirror geometry associated to the Shannon
entropy potential both on $U$ and $V$ to enforce non-negativity. The following
theorem (proved in Appendix J) gives a set of conserved functions in the
gradient flow case (2). We denote $1_{\ell}\in\mathbb{R}^{\ell}$ the vector
with all coordinates equal to 1.

######  Theorem 4.5.

Consider $g(\theta,x)\coloneqq UV^{\top}x$ with
$\theta=(U,V)\in\mathbb{R}^{n\times r}\times\mathbb{R}^{m\times r}$. Then

$1_{n}^{\top}U-1_{m}^{\top}V$ (20)

defines $r$ independent conservation laws for (2) with
$M(\theta)\coloneqq\mathrm{diag}(\theta)$.

These functions can be found by solving (13) with formal calculus with the
vector fields that generate
$\mathcal{W}_{\phi_{\mathtt{Lin}}}^{\texttt{grad}}$ in this non-Euclidean
gradient flow case. As for the Euclidean GF case, these functions can also be
linked to invariance on the cost as they coincide with the time integration of
(16):
$\langle\mathrm{diag}(\theta)^{-1}\dot{\theta},\Delta_{T^{A}}(\theta)\rangle=0$
when $A=E_{i,i}$ for each $i=1,\cdots,r$. By computing the number of
independent conservation laws with the method explained in Section 3.2, we
obtain that there are no other conservation laws (see Section M.2). In
contrast, by computing the number of independent conservation laws for the
vector fields that generate $\mathcal{W}_{\phi}^{\mathtt{mom}}$ for the non-
Euclidean momentum flow case, we found that there is no conservation law at
all in that case. Once again there are fewer conservation laws for MF than GF.

###  4.4 ICNN: mixing Mirror/Euclidean geometries

In this section, we only treat the two-layer ReLU case of ICNNs (Amos et al.,
2017), corresponding to $g(\theta,x)=U\sigma(V^{\top}x+b^{\top})$, with
$\theta=(U;V)\in\mathbb{R}^{(m+n)\times r}$ and $\sigma(t)\coloneqq\max(t,0)$.
We employ the mirror geometry associated to the Shannon entropy potential on
$U$ to enforce non-negativity ($U\geq 0$) and the Euclidean metric on $V$ and
$b$. The following theorem (proved in Appendix K) gives a set of conservation
laws for GF.

######  Theorem 4.6.

Consider $g(\theta,x)=U\sigma(V^{\top}x+b^{\top})$. Denote $U_{j}$ (resp.
$V_{j}$ / $b_{j}$) the $j$-th column of $U$ (resp column of $V$ / entry of
$b$). For all $j=1,\cdots,r$, the function

$1_{n}^{\top}U_{j}-\frac{1}{2}\left(\|V_{j}\|^{2}+b_{j}^{2}\right)$ (21)

is a conservation law for (2) with
$M(U,V,b)\coloneqq\mathrm{diag}[(U,1_{m\times r},1_{1\times r})].$

By computing the number of independent conservation laws with the method
explained in Section 3.2, we obtain that there are no other conservation laws.
In contrast, by computing the number of independent conservation laws for the
vector fields that generate
$\mathcal{W}_{\phi_{\mathtt{ReLU}}}^{\mathtt{mom}}$ for the non-Euclidean
momentum flow case, we found that there is no conservation law at all in that
case (see Section M.3). Thus, once again we observe a loss of conservation
from GF to MF regimes.

###  4.5 Natural gradient

Our main result on conservation laws for momentum, Proposition 2.9, only holds
when $M=M_{Z}$ is independent of the dataset $Z$. While we cannot apply it to
the natural gradient setting, we can still conduct (a part of) our study for
the non-Euclidean GF. We obtain that all conservation laws known for the
Euclidean gradient flow case recalled in Example 2.20 are conservation laws
for the Natural gradient flow. See Appendix L for a proof.

## Conclusion

In this paper, we examined conservation laws for gradient or momentum flows
within Euclidean as well as non-Euclidean geometries. One notable constraint
of this theory is its limitation to continuous-time flow. Appendix N studies
numerically the impact of the time discretization on MLP and NMF problems. It
also studies the impact of momentum on quantities preserved by GF.
Understanding these approximate conservations is an important open problem. We
also anticipate that our theory will be adaptable to the study of the SGD
case.

##### Acknowledgement

The work of G. Peyré was supported by the European Research Council (ERC
project NORIA) and the French government under management of Agence Nationale
de la Recherche as part of the “Investissements d’avenir” program, reference
ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). The work of R. Gribonval was
partially supported by the AllegroAssai ANR project ANR-19-CHIA-0009 and the
SHARP ANR Project ANR-23-PEIA-0008 of the PEPR IA, funded in the framework of
the France 2030 program.

##### Impact Statement.

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none of
which we feel must be specifically highlighted here.

## References

  * Amari (1998) Amari, S.-I.  Natural gradient works efficiently in learning.  _Neural computation_ , 10(2):251–276, 1998.
  * Amos et al. (2017) Amos, B., Xu, L., and Kolter, J. Z.  Input convex neural networks.  In _International Conference on Machine Learning_ , pp. 146–155. PMLR, 2017.
  * Arora et al. (2018) Arora, S., Cohen, N., and Hazan, E.  On the optimization of deep networks: Implicit acceleration by overparameterization.  In _International Conference on Machine Learning_ , pp. 244–253. PMLR, 2018.
  * Arora et al. (2019) Arora, S., Cohen, N., Golowich, N., and Hu, W.  A convergence analysis of gradient descent for deep linear neural networks.  In _International Conference on Learning Representations_ , 2019.
  * Bah et al. (2022) Bah, B., Rauhut, H., Terstiege, U., and Westdickenberg, M.  Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers.  _Information and Inference: A Journal of the IMA_ , 11(1):307–353, 2022.
  * Bonnard et al. (2018) Bonnard, B., Chyba, M., and Rouot, J.  _Geometric and Numerical Optimal Control - Application to Swimming at Low Reynolds Number and Magnetic Resonance Imaging_.  SpringerBriefs in Mathematics. Springer Int. Publishing, 2018.
  * Bregman (1967) Bregman, L. M.  The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming.  _USSR computational mathematics and mathematical physics_ , 7(3):200–217, 1967.
  * Bubeck et al. (2015) Bubeck, S. et al.  Convex optimization: Algorithms and complexity.  _Foundations and Trends® in Machine Learning_ , 8(3-4):231–357, 2015.
  * Chizat & Bach (2020) Chizat, L. and Bach, F.  Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss.  In _Conf. on Learning Theory_ , pp. 1305–1338. PMLR, 2020.
  * Du et al. (2018) Du, S. S., Hu, W., and Lee, J. D.  Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Głuch & Urbanke (2021) Głuch, G. and Urbanke, R.  Noether: The more things change, the more stay the same.  _arXiv preprint arXiv:2104.05508_ , 2021.
  * Ji & Telgarsky (2019) Ji, Z. and Telgarsky, M.  Gradient descent aligns the layers of deep linear networks.  In _International Conference on Learning Representations_ , 2019.
  * Kunin et al. (2021) Kunin, D., Sagastuy-Brena, J., Ganguli, S., Yamins, D. L., and Tanaka, H.  Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics.  In _International Conference on Learning Representations_ , 2021.
  * LeCun et al. (2010) LeCun, Y., Cortes, C., and Burges, C.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Lee & Seung (1999) Lee, D. D. and Seung, H. S.  Learning the parts of objects by non-negative matrix factorization.  _Nature_ , 401(6755):788–791, 1999.
  * Makkuva et al. (2020) Makkuva, A., Taghvaei, A., Oh, S., and Lee, J.  Optimal transport mapping via input convex neural networks.  In _International Conference on Machine Learning_ , pp. 6672–6681. PMLR, 2020.
  * Marcotte et al. (2024) Marcotte, S., Gribonval, R., and Peyré, G.  Abide by the law and follow the flow: Conservation laws for gradient flows.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Martens (2010) Martens, J.  Deep learning via hessian-free optimization.  In _Proceedings of the 27th International Conference on International Conference on Machine Learning_ , pp. 735–742, 2010.
  * Martens & Grosse (2015) Martens, J. and Grosse, R.  Optimizing neural networks with kronecker-factored approximate curvature.  In _International conference on machine learning_ , pp. 2408–2417. PMLR, 2015.
  * Martens & Sutskever (2012) Martens, J. and Sutskever, I.  _Training Deep and Recurrent Networks with Hessian-Free Optimization_.  Springer, 2012.
  * Min et al. (2021) Min, H., Tarmoun, S., Vidal, R., and Mallada, E.  On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks.  In _International Conference on Machine Learning_ , pp. 7760–7768. PMLR, 2021.
  * Nemirovskij & Yudin (1983) Nemirovskij, A. S. and Yudin, D. B.  _Problem complexity and method efficiency in optimization_.  Wiley-Interscience, 1983.
  * Nesterov (1983) Nesterov, Y. E.  A method of solving a convex programming problem with convergence rate o$\bigl{(}k^{2}\bigr{)}$.  In _Doklady Akademii Nauk_ , volume 269, pp. 543–547. Russian Academy of Sciences, 1983.
  * Noether (1918) Noether, E.  Invariante variationsprobleme.  _Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathematisch-Physikalische Klasse_ , 1918:235–257, 1918.
  * Polyak (1964) Polyak, B. T.  Some methods of speeding up the convergence of iteration methods.  _Ussr computational mathematics and mathematical physics_ , 4(5):1–17, 1964.
  * Raskutti & Mukherjee (2015) Raskutti, G. and Mukherjee, S.  The information geometry of mirror descent.  _IEEE Transactions on Information Theory_ , 61(3):1451–1457, 2015.
  * Saxe et al. (2013) Saxe, A. M., McClelland, J. L., and Ganguli, S.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  _arXiv preprint arXiv:1312.6120_ , 2013.
  * Su et al. (2016) Su, W., Boyd, S., and Candes, E. J.  A differential equation for modeling nesterov’s accelerated gradient method: Theory and insights.  _Journal of Machine Learning Research_ , 17(153):1–43, 2016.
  * Tanaka & Kunin (2021) Tanaka, H. and Kunin, D.  Noether’s learning dynamics: Role of symmetry breaking in neural networks.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Tarmoun et al. (2021) Tarmoun, S., Franca, G., Haeffele, B. D., and Vidal, R.  Understanding the dynamics of gradient flow in overparameterized linear models.  In _International Conference on Machine Learning_ , pp. 10153–10161. PMLR, 2021.
  * The Sage Developers (2022) The Sage Developers.  _SageMath, the Sage Mathematics Software System (Version 9.7)_ , 2022.  https://www.sagemath.org.
  * Wibisono et al. (2016) Wibisono, A., Wilson, A. C., and Jordan, M. I.  A variational perspective on accelerated methods in optimization.  _proceedings of the National Academy of Sciences_ , 113(47):E7351–E7358, 2016.
  * Zhao et al. (2023) Zhao, B., Ganev, I., Walters, R., Yu, R., and Dehmamy, N.  Symmetries, flat minima, and the conserved quantities of gradient flow.  In _The Eleventh International Conference on Learning Representations_ , 2023.

##  Appendix A Proof of Theorem 2.1

See 2.1

###### Proof.

GF case: Let $h(t,\theta)$ be a conserved function for the ODE (2) with a
right-hand side equal to zero with any initial condition $\theta_{0}$. In that
case, (2) rewrites $\dot{\theta}=0$. For each initialization $\theta_{0}$, the
solution of this ODE is $\theta(t)=\theta_{0}$ for $t\in\mathbb{R}$ and by
definition of a conserved function, one has
$h(t,\theta_{0})=h(t,\theta(t))=h(0,\theta(0))=h(0,\theta_{0})$.

MF case: Let $h(t,\theta,\dot{\theta})$ be a conserved function for the ODE
(3) with a right-hand side equal to zero, an arbitrary initial condition
$(t_{0},\theta_{0},\dot{\theta}_{0})$, and $\tau(t)\coloneqq\tau$. In that
case, (3) rewrites $\ddot{\theta}+\tau\dot{\theta}=0$. For each initialization
$(\theta_{0},\dot{\theta}_{0})$, the solution of this ODE is:
$\theta(t)=\theta_{0}+\dot{\theta}_{0}/\tau\left(1-\exp(-t\tau)\right)$, for
every $t\in\mathbb{R}$, and it satisfies
$(\theta(0),\dot{\theta}(0))=(\theta_{0},\dot{\theta}_{0})$.

Since $h$ is conserved,
$h(0,\theta_{0},\dot{\theta}_{0}){=h(t,\theta(t),\dot{\theta}(t))}=h\left(t,\theta_{0}+\dot{\theta}_{0}/\tau\left(1-\exp(-t\tau)\right),\dot{\theta}_{0}\exp(-t\tau)\right)$
for each $t\in\mathbb{R}$. This holds for any $\theta_{0},\dot{\theta}_{0}$,
and given any $\theta,\dot{\theta}$ one can find $\theta_{0},\dot{\theta}_{0}$
such that
$(\theta,\dot{\theta})=(\theta_{0}+\dot{\theta}_{0}/\tau\left(1-\exp(-t\tau)\right),\dot{\theta}_{0}\exp(-t\tau))$.
Expliciting the expression of such $\theta_{0},\dot{\theta}_{0}$ in terms of
$\theta,\dot{\theta}$ yields

$h(t,\theta,\dot{\theta})=h(0,\theta-\dot{\theta}/\tau(\exp(t\tau)-1),\dot{\theta}\exp(\tau
t))={H}\left(\theta+\frac{\dot{\theta}}{\tau},\dot{\theta}\exp(\tau
t)\right).\qed$

##  Appendix B Proof of Proposition 2.7

See 2.7

###### Proof.

We will use that $\langle\nabla
h(\alpha),(1,F(\alpha)^{\top})^{\top}\rangle=\partial
h(\alpha)(1,F(\alpha)^{\top})^{\top}$ with $\partial h$ the Jacobian of $h$.

Assume that $\partial h(\alpha)(1,F(\alpha)^{\top})^{\top}=0$ for all
$\alpha\in\tilde{\Omega}$. Then for all
$\texttt{init}\coloneqq(t_{\text{init}},\omega_{\text{init}})\in\tilde{\Omega}$
and for all
$t\in\left(t_{\text{init}}-\eta_{\text{init}},t_{\text{init}}+\eta_{\text{init}}\right),$
denoting
$\dot{\omega}(t,\texttt{init})\coloneqq\frac{\mathrm{d}}{\mathrm{d}t}\omega(t,\texttt{init})$
we have

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}h(t,\omega(t,\texttt{init}))=\partial
h(t,\omega(t,\texttt{init})){(1,\dot{\omega}(t,\texttt{init})^{\top})^{\top}}=\partial
h(t,\omega(t,\texttt{init}))(1,F(t,\omega(t,\texttt{init}))^{\top})^{\top}=0.$

Thus: $h(t,\omega(t,\texttt{init}))=h(\texttt{init})$, i.e., $h$ is conserved
through $\chi$.

Conversely, assume that there exists $(t_{0},\omega_{0})\in\tilde{\Omega}$
such that $\partial
h(t_{0},\omega_{0})(1,F(t_{0},\omega_{0})^{\top})^{\top}\neq 0$. Then by
continuity of $z\in\tilde{\Omega}\mapsto\partial h(z)(1,F(z)^{\top})^{\top}$,
there exists $r>0$ such that $\partial h(z)(1,F(z)^{\top})^{\top}\neq 0$ on
$B(\left(t_{0},\omega_{0}\right),r)$. With
${\texttt{init}}=(t_{0},\omega_{0})$ by continuity of
$t\mapsto\omega(t,\texttt{init})$, there exists $\epsilon>0$, such that for
all $t\in(t_{0},t_{0}+\epsilon)$, $\omega(t,\texttt{init})\in
B(\left(t_{0},\omega_{0}\right),r)$. Then for all
$t\in(t_{0},t_{0}+\epsilon)$:$\frac{\mathrm{d}}{\mathrm{d}t}h(t,\omega(t,\texttt{init}))=\partial
h(t,\omega(t,\texttt{init}))(1,F(t,\omega(t,\texttt{init}))^{\top})^{\top}\neq
0,$ hence $h$ is not conserved through the flow induced by $F$. ∎

##  Appendix C Proof of Proposition 2.8

Recall that $F_{Z}$ is defined in (4).

See 2.8

###### Proof.

Let us consider
$h:\tilde{\Theta}\mapsto\mathbb{R}\in\mathcal{C}^{1}(\tilde{\Theta},\mathbb{R})$.
We first show the direct implication. We assume that $h$ is locally conserved
on $\Theta$ for any data set. Let $\alpha\in\tilde{\Theta}$ and let
$Z=(x_{i},y_{i})_{i}\in\mathcal{Z}_{\theta}$. By definition of
$\mathcal{Z}_{\theta}$, there exists a neighborhood of $\theta$
$\Omega\subseteq\Theta$ such that for all $i$,
$g(\cdot,x_{i})\in\mathcal{C}^{2}(\Omega,\mathbb{R}^{n})$ and such that
$M_{Z}(\cdot)\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R}^{D\times D})$. Then
by definition of being locally conserved on $\Theta$ for any data set, $h$ is
in particular conserved on $\Omega$ for any data set. Thus in particular $h$
is conserved on $\Omega$ during the flow $F_{Z}$. By Proposition 2.7, $\nabla
h(\alpha)\perp(1,F_{Z}(\alpha)^{\top})^{\top}$. This holds for any
$Z\in\mathcal{Z}_{\theta}$, thus $\nabla h(\alpha)\perp
W_{\alpha}^{\mathtt{mom}}$. As it is true for any $\theta\in\tilde{\Theta}$,
we have the direct implication.

We now show the converse implication. We assume that $\nabla h(\alpha)\perp
W_{\alpha}^{\mathtt{mom}}$ for all $\alpha\in\tilde{\Theta}$. Let us consider
an open subset $\Omega\subseteq\Theta$, and let us consider
$Z=(x_{i},y_{i})_{i}$ a data set such that
$g(\cdot,x_{i})\in\mathcal{C}^{2}(\Omega,\mathbb{R})$ for each $i$ and
$(t,\theta,\dot{\theta})\mapsto
M_{Z}(t,\theta,\dot{\theta})\in\mathcal{C}^{1}(\tilde{\Omega},\mathbb{R}^{D\times
D})$. In particular, $Z\in\mathcal{Z}_{\theta}$. Thus, for any
$\alpha\in\tilde{\Omega}$, as $\nabla h(\alpha)\perp
W_{\alpha}^{\mathtt{mom}}$, one has $\nabla
h(\alpha)\perp(1,F_{Z}(\alpha)^{\top})^{\top}$, and by Proposition 2.7, $h$ is
conserved on $\Omega$ during the flow $F_{Z}$. As this holds for any
$\Omega\subseteq\Theta$, $h$ is locally conserved on $\Theta$ for any data
set. ∎

##  Appendix D Proof of Proposition 2.9

See 2.9

###### Proof.

Let $z\coloneqq(t,\theta,\dot{\theta})\in\tilde{\Theta}$. Let us denote
$\mathcal{Z}_{\theta}^{\prime}$ the collection of all data set
$Z=(x_{i},y_{i})_{i}$ such that for all $i$, $g(\cdot,x_{i})$ is
$\mathcal{C}^{2}$-differentiable in the neighborhood of $\theta$. Let
$z\coloneqq(t,\theta,\dot{\theta})\in\tilde{\Theta}$. First, let us recall
that (See Proposition 2.7 of (Marcotte et al., 2024)) we have:

$W_{\theta}=\underset{Z\in\mathcal{Z}_{\theta}^{\prime}}{\mathrm{span}}\\{\nabla\mathcal{E}_{Z}(\theta)\\}.$
(22)

Then as by assumption
${M(\cdot)}\in\mathcal{C}^{1}(\tilde{\Theta},\mathbb{R}^{D\times D})$, we have
that $\mathcal{Z}^{\prime}_{\theta}=\mathcal{Z}_{\theta}$ with
$\mathcal{Z}_{\theta}$ defined as in Proposition 2.8. Thus, we can rewrite:

$\displaystyle
W_{\alpha}^{\mathtt{mom}}=\underset{Z\in\mathcal{Z}_{\theta}^{\prime}}{\mathrm{span}}\left\\{(1,F_{Z}(\alpha)^{\top})^{\top}\right\\}\stackrel{{\scriptstyle\eqref{eq:DefFZ}}}{{=}}\underset{Z\in\mathcal{Z}_{\theta}^{\prime}}{\mathrm{span}}\left\\{\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\
-\tau(t)\dot{\theta}\end{smallmatrix}\right)+\left(\begin{smallmatrix}0\\\
0\\\
-M(t,\theta,\dot{\theta})\nabla\mathcal{E}_{Z}(\theta)\end{smallmatrix}\right)\right\\}.$

Thus:
$W_{\alpha}^{\mathtt{mom}}\subseteq\mathbb{R}\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\
-\tau(t)\dot{\theta}\end{smallmatrix}\right)+\underset{Z\in\mathcal{Z}_{\theta}^{\prime}}{\mathrm{span}}\left(\begin{smallmatrix}0\\\
0\\\
-M(t,\theta,\dot{\theta})\nabla\mathcal{E}_{Z}(\theta)\end{smallmatrix}\right)=\mathbb{R}\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\
-\tau(t)\dot{\theta}\end{smallmatrix}\right)+\left(\begin{smallmatrix}0\\\
0\\\ -M(t,\theta,\dot{\theta})W_{\theta}\end{smallmatrix}\right),$ which gives
the direct inclusion.

Let us show the converse inclusion. By assumption, there exists
$Z_{0}\in\mathcal{Z}_{\theta}^{\prime}$ such that
$\nabla\mathcal{E}_{Z_{0}}(\theta)=0$, so that
$W_{\alpha}^{\mathtt{mom}}\ni(1,F_{Z_{0}}(\alpha)^{\top})^{\top}=\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\ -\tau(t)\dot{\theta}\end{smallmatrix}\right)$, and thus
$\mathbb{R}\left(\begin{smallmatrix}1\\\ \dot{\theta}\\\
-\tau(t)\dot{\theta}\end{smallmatrix}\right)\subseteq
W_{\alpha}^{\mathtt{mom}}$. Then for all $Z\in\mathcal{Z}_{\theta}^{\prime}$,

$\left(\begin{smallmatrix}0\\\ 0\\\
{-M(t,}\theta,\dot{\theta})\nabla\mathcal{E}_{Z}(\theta)\end{smallmatrix}\right)=\underbrace{\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\
-\tau(t)\dot{\theta}\end{smallmatrix}\right)+\left(\begin{smallmatrix}0\\\
0\\\
{-M(t,}\theta,\dot{\theta})\nabla\mathcal{E}_{Z}(\theta)\end{smallmatrix}\right)}_{{=(1,F_{Z}(\alpha)^{\top})^{\top}}\in
W_{\alpha}^{\mathtt{mom}}}-\underbrace{\left(\begin{smallmatrix}1\\\
\dot{\theta}\\\ -\tau(t)\dot{\theta}\end{smallmatrix}\right)}_{\in
W_{\alpha}^{\mathtt{mom}}}\in W_{\alpha}^{\mathtt{mom}},$

and thus $\left(\begin{smallmatrix}0\\\ 0\\\
-M(t,\theta,\dot{\theta})W_{\theta}\end{smallmatrix}\right)=\underset{Z\in\mathcal{Z}_{\theta}^{\prime}}{\mathrm{span}}\left(\begin{smallmatrix}0\\\
0\\\
-M(t,\theta,\dot{\theta})\nabla\mathcal{E}_{Z}(\theta)\end{smallmatrix}\right)\subseteq
W_{\alpha}^{\mathtt{mom}}$, which gives the converse inclusion. ∎

We now show that the following assumption holds for classical losses in
machine learning.

######  Assumption D.1.

For all $\theta\in{\Theta}$, there exists $Z\in\mathcal{Z}_{\theta}^{\prime}$
such that $\nabla\mathcal{E}_{Z}(\theta)=0$.

######  Lemma D.2.

D.1 holds for the mean-square error loss and the cross-entropy loss.

###### Proof.

Let $\theta\in\Theta$.

Mean-square error loss. The mean-squared error loss is defined by
$(z,y)\mapsto\ell_{2}(z,y)\coloneqq\|y-z\|^{2}$. Let consider $x$ such that
$g(\cdot,x)$ is $\mathcal{C}^{2}$ on a neighborhood of $\theta$. Then consider
$y\coloneqq g(\theta,x)$. By definition
$Z=(x,y)\in\mathcal{Z}_{\theta}^{\prime}$. Moreover one has in a neighborhood
of $\theta$: $\nabla\mathcal{E}_{Z}(\theta^{\prime})=\partial
g(\theta^{\prime},x)^{\top}\nabla_{z}\ell_{2}(g(\theta^{\prime},x),y))=2\partial
g(\theta^{\prime},x)^{\top}(g(\theta^{\prime},x)-y))$. Thus
$\nabla\mathcal{E}_{Z}(\theta)=0$.

Cross-entropy loss. The cross-entropy loss is defined by
$(z\in\mathbb{R}^{n},y\in\Sigma_{n})\mapsto\ell_{\mathtt{cross}}(z,y)\coloneqq\text{KL}(\text{softmax}(z),y)$,
where
$\Sigma_{n}\coloneqq\\{(y_{1},\cdots,y_{n}){\in\mathbb{R}_{+}^{n}}:\sum_{i=1}^{n}y_{i}=1\\}$
is the simplex and KL is the Kullback–Leibler divergence defined on
$\Sigma_{n}\times\Sigma_{n}$ by
$\text{KL}(r,q)\coloneqq\sum_{i=1}^{n}r_{i}\log(r_{i}/q_{i})$. In particular
by taking $x$ such that $g(\cdot,x)$ is $\mathcal{C}^{2}$ on a neighborhood of
$\theta$ and by taking $y=\text{softmax}(g(\theta,x))$ and $Z=(x,y)$ (in
$\mathcal{Z}_{\theta}^{\prime}$), we obtain $\nabla\mathcal{E}_{Z}(\theta)=0$.
∎

##  Appendix E Proof of Theorem 4.1

We consider linear networks with $\theta\coloneqq(U_{1},\ldots,U_{q})$,
$g(\theta,x)\coloneqq U_{1}\ldots U_{q}x$. See 4.1

###### Proof.

We first treat the general case. The specific case $(n_{i-1},n_{i+1})=(1,1)$
will come next. Let us denote for $A\in\mathcal{A}_{n_{i}}$:

$h_{A}(t,\theta,\dot{\theta})\coloneqq\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\left\langle\dot{U}_{i},U_{i}A\right\rangle+\left\langle\dot{U}_{i+1},A^{\top}U_{i+1}\right\rangle\right).$
(23)

Observe that (3) implies

$\ddot{U}_{i}(t)+\tau(t)\dot{U}_{i}(t)=-\nabla_{U_{i}}\mathcal{E}(\theta(t))$
(24)

for every $1\leq i\leq q$, where to ease further computation each
$\nabla_{U_{i}}$ is reshaped as the $n_{i-1}\times n_{i}$ matrix $U_{i}$. Then

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}h_{A}(t,\theta(t),\dot{\theta}(t))$
$\displaystyle=\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\left\langle\ddot{U}_{i},U_{i}A\right\rangle+\underbrace{\left\langle\dot{U}_{i},\dot{U}_{i}A\right\rangle}_{=0\text{
as }A\text{ is a skew matrix
}}+\left\langle\ddot{U}_{i+1},A^{\top}U_{i+1}\right\rangle+\underbrace{\left\langle\dot{U}_{i+1},A^{\top}\dot{U}_{i+1}\right\rangle}_{=0\text{
as }A\text{ is a skew matrix }}\right)$
$\displaystyle\quad\quad+\tau(t)h_{A}(t,\theta(t),\dot{\theta}(t))$
$\displaystyle\stackrel{{\scriptstyle\eqref{eq:DefMFBlockAnnex}}}{{=}}-\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\langle\nabla_{U_{i}}\mathcal{E}_{Z}(\theta),U_{i}A\rangle+\langle\nabla_{U_{i+1}}\mathcal{E}_{Z}(\theta),\underbrace{A^{\top}}_{=-A}U_{i+1}\rangle\right)$
$\displaystyle=0\text{ as }T^{A}\text{ leaves }\mathcal{E}_{Z}\text{invariant
(See \eqref{eq:invarianceloss} and
\lx@cref{creftypecap~refnum}{example:linear_transf})},$

where here $T^{A}(\epsilon,U_{i},U_{i+1})=\big{(}U_{i}\exp(\epsilon
A),\exp(-\epsilon A)U_{i+1}\big{)},$ and
$\Delta_{T^{A}}(U_{i},U_{i+1})=(U_{i}A,-AU_{i+1})$.

Special case where $(n_{i-1}=n_{i+1}=1)$.  For any $A\in\mathcal{A}_{n_{i}}$,
we denote:

$g_{A}(t,\theta,\dot{\theta})\coloneqq\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\left\langle\dot{U}_{i},U_{i+1}^{\top}A\right\rangle+\left\langle\dot{U}_{i+1}^{\top},U_{i}A\right\rangle\right)$

By our assumption on the dimensions, $U_{i+1}^{\top}A$ is an $n_{i+1}\times
n_{i}=1\times n_{i}$ matrix, just as $\dot{U}_{i}$, and similarly for $U_{i}A$
and $\dot{U}_{i+1}^{\top}$, so $g_{A}$ is indeed well-defined. We will prove
below that (again with gradients properly reshaped as matrices)

$\nabla_{U_{i}}\mathcal{E}_{Z}(\theta)\propto U_{i+1}^{\top}\ \text{and}\
\nabla_{U_{i+1}}\mathcal{E}_{Z}(\theta)\propto U_{i}^{\top}.$ (25)

As a result, using again (24) we compute

$\displaystyle\frac{\mathrm{d}}{\mathrm{d}t}g_{A}(t,\theta(t),\dot{\theta}(t))$
$\displaystyle=\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\left\langle\ddot{U}_{i},U_{i+1}^{\top}A\right\rangle+\left\langle\ddot{U}_{i+1}^{\top},U_{i}A\right\rangle+\underbrace{\left\langle\dot{U}_{i},\dot{U}_{i+1}^{\top}A\right\rangle+\left\langle\dot{U}_{i+1}^{{\top}},\dot{U}_{i}A\right\rangle}_{{=\langle\dot{U}_{i+1}\dot{U}_{i}+\dot{U}_{i}^{\top}\dot{U}_{i+1}^{\top},A\rangle}=0\text{
as }A\text{ is a skew matrix }}\right)$
$\displaystyle\quad\quad+\tau(t)g_{A}(t,\theta(t),\dot{\theta}(t))$
$\displaystyle=-\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\underbrace{\langle\underbrace{\nabla_{U_{i}}\mathcal{E}_{Z}(\theta)}_{\propto
U_{i+1}^{\top}\text{ by }\eqref{eq:prop1}},U_{i+1}^{\top}A\rangle}_{=0\text{
as }A\text{ is a skew matrix
}}+\underbrace{\langle\underbrace{[\nabla_{U_{i+1}}\mathcal{E}_{Z}(\theta)]^{\top}}_{\propto
U_{i}\text{ by }\eqref{eq:prop1}},U_{i}A\rangle}_{=0\text{ as }A\text{ is a
skew matrix }}\right)$ $\displaystyle=0.$

Let us now prove (25) as claimed. First, to simplify the notations, let us
define:

$\widehat{V_{j}}=\left\\{\begin{array}[]{ll}U_{1}...U_{j-1}\mbox{ for all
}j=2,...,q,\\\ \mathrm{Id}_{n_{0}}\mbox{ for }j=1.\end{array}\right.$

and

$\widehat{W_{j}}=\left\\{\begin{array}[]{ll}U_{j+1}...U_{q}\mbox{ for all
}j=1,...,q-1,\\\ \mathrm{Id}_{n_{q}}\mbox{ for j=q.}\end{array}\right.$

Let us now derive an expression $\nabla_{U_{i}}\mathcal{E}_{Z}$. Given that
$g(\theta,x)=U_{1}\ldots U_{q}x$,we can factorize
$\mathcal{E}_{Z}(\theta)=F\circ\phi(\theta)$ with $\phi(\theta)\coloneqq
U_{1}\ldots U_{q}\in\mathbb{R}^{n_{0}\times n_{q}}$ (identified with
$\mathbb{R}^{d}$, $d=n_{0}n_{q}$), and $F:\mathbb{R}^{d}\to\mathbb{R}$. The
chain rule for Jacobians yields

$\partial\mathcal{E}_{Z}\left(\theta\right)=\partial
F\left(\phi(\theta)\right)\cdot\partial\phi\left(\theta\right)$ (26)

hence for all $(H_{1},\cdots,H_{q})\in\mathbb{R}^{D}$ (with components
$H_{i}\in\mathbb{R}^{n_{i-1}\times n_{i}}$ seen as vectors in
$\mathbb{R}^{n_{i-1}n_{i}}$):

$\displaystyle\left\langle\nabla\mathcal{E}_{Z}\left(\theta\right),(H_{1},\cdots,H_{q})\right\rangle$
$\displaystyle=\partial\mathcal{E}_{Z}\left(\theta\right)\cdot(H_{1},\cdots,H_{q})\stackrel{{\scriptstyle\eqref{comp}}}{{=}}\left\langle\nabla
F\left(\phi(\theta)\right),\partial\phi\left(\theta\right)\cdot(H_{1},\cdots,H_{q})\right\rangle$
$\displaystyle=\left\langle\nabla
F\left(\phi(\theta)\right),\sum_{j=1}^{q}U_{1}...U_{j-1}H_{j}U_{j+1}...U_{q}\right\rangle=\sum_{j=1}^{q}\left\langle\nabla
F\left(\phi(\theta)\right),\widehat{V_{j}}H_{j}\widehat{W_{j}}\right\rangle$
$\displaystyle=\sum_{j=1}^{q}\mathtt{Tr}\left([{\nabla
F\left(\phi(\theta)\right)}]^{\top}\widehat{V_{j}}H_{j}\widehat{W_{j}}\right)=\sum_{j=1}^{q}\mathtt{Tr}\left(H_{j}\widehat{W_{j}}[{\nabla
F\left(\phi(\theta)\right)}]^{\top}\widehat{V_{j}}\right)$
$\displaystyle=\sum_{j=1}^{q}\left\langle H_{j},\widehat{V_{j}}^{\top}{\nabla
F\left(\phi(\theta)\right)}\widehat{W_{j}}^{\top}\right\rangle.$

Then by Riez theorem, viewing again for convenience $\nabla F$ as its matrix
reshaped version, we get that for each $1\leq j\leq q$:

$\nabla_{U_{j}}\mathcal{E}_{Z}\left(\theta\right)=\underbrace{\widehat{V_{j}}^{\top}}_{n_{j-1}\times
n_{0}}\underbrace{{\nabla F\left(\phi(\theta)\right)}}_{n_{0}\times
n_{q}}\underbrace{\widehat{W_{j}}^{\top}}_{n_{q}\times n_{j}}.$ (27)

Thus by specializing to $j\in\\{i,i+1\\}$ and by developing (27), one has:

$\nabla_{U_{i}}\mathcal{E}_{Z}(\theta)=\underbrace{a}_{\text{ size
}n_{i-1}\times n_{i+1}}U_{i+1}^{\top},\quad\text{ and
}\quad\nabla_{U_{i+1}}\mathcal{E}_{Z}(\theta)=U_{i}^{\top}\underbrace{b}_{\text{
size }n_{i-1}\times n_{i+1}},$

thus establishing (25). ∎

##  Appendix F Application of Noether Theorem

Invariances of $\mathcal{E}_{Z}$ with respect to certain transformations have
been used in the proof of Theorem 4.1. Here we establish more direct
connections with Noether’s theorem (see Theorem 3.5). For the sake of brevity
we describe the case $q=2$ but the same reasoning can easily be adapted to any
$q\geq 2$ by considering the invariances associated to each pair $U_{i}$,
$U_{i+1}$.

Invariances valid for any dimension. We consider the flow (3) in the Euclidean
case ($M=\mathrm{Id}_{d}$), with $\theta\coloneqq(U,V)\in\mathbb{R}^{n\times
r}\times\mathbb{R}^{m\times r}$ and $g(\theta,x)\coloneqq UV^{\top}x$. We
recall that for all $A\in\mathbb{R}^{r\times r}$, the linear transformation
$T^{A}$ from (15) leaves the cost (1) invariant, and that (see Example 3.2)
that: $\Delta_{T^{A}}(U,V)=(UA,-VA^{\top}).$ Moreover for any $A$ in
$\mathcal{A}_{r}$ the space of skew-symmetric matrices in $\mathbb{R}^{r\times
r}$, $T^{A}$ leaves invariant the following Lagrangian (See for example (Głuch
& Urbanke, 2021)), with which the Euclidian MF is compatible:

$\mathcal{L}\left(t,\theta,v\right)\coloneqq\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(1/2\|v\|^{2}-\mathcal{E}_{Z}(\theta)\right).$
(28)

Thus by Noether theorem (Theorem 3.5), for any $A\in\mathcal{A}_{r}$, a
conserved function is

$\displaystyle{h(t,\theta,\dot{x})\coloneqq}\partial_{v}\mathcal{L}(t,\theta,\dot{\theta})\cdot\Delta_{T}^{A}(\theta(t))$
$\displaystyle=\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\langle\dot{U},UA\rangle-\langle\dot{V},VA^{\top}\rangle\right)$
$\displaystyle=\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\langle\dot{U},UA\rangle+\langle\dot{V},VA\rangle\right).$

A supplementary invariance when $(n,m)=(1,1)$. Assume $(n,m)=(1,1)$, so that
$U,V\in\mathbb{R}^{1\times r}$ and $\theta\in\mathbb{R}^{1\times 2r}$, and
consider $T^{\prime A}$ the linear transformation where for all $\epsilon\geq
0$:

$T^{\prime A}(\epsilon,U,V)=\Big{(}V\exp(-\epsilon A^{\top}),U\exp(\epsilon
A)\Big{)}.$ (29)

One has: $\Delta_{T^{\prime A}}(U,V)=(-VA^{\top},UA)$. It is easy to check
that since $g(\theta,x)\coloneqq UV^{\top}x$, we have $g(T^{\prime
A}(\epsilon,\cdot))=g(\theta,\cdot)$ hence $T^{\prime A}$ leaves invariant the
cost (1). Moreover, one can also easily check that for any skew matrix
$A\in\mathcal{A}_{r}$, the transformation $T^{\prime A}$ also leaves the
Lagrangian (28) invariant. Thus, by Noether theorem, for any
$A\in\mathcal{A}_{r}$, the quantity

$\displaystyle{h(t,\theta,\dot{x})\coloneqq}\partial_{v}\mathcal{L}(t,\theta,\dot{\theta})\cdot\Delta_{T^{\prime
A}}(\theta)$
$\displaystyle=\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\langle\dot{U},-VA^{\top}\rangle+\langle\dot{V},UA\rangle\right)$
$\displaystyle=\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}\left(\langle\dot{U},VA\rangle+\langle\dot{V},UA\rangle\right)$

is conserved.

##  Appendix G Proof of Proposition 4.2

See 4.2

###### Proof.

First, observe that if $r=1$, then Theorem 4.1 gives zero conserved functions
as $\mathcal{A}_{1}=\\{0\\}$, and indeed here $2(n+m)>r=1$ and $r(r-1)=0$.

We now focus on the case where $r>1$ and denote $U=(u_{1},\cdots,u_{r})$,
$V=(v_{1},\cdots,v_{r})$, $\dot{U}=(\dot{u}_{1},\cdots,\dot{u}_{r})$,
$\dot{V}=(\dot{v}_{1},\cdots,\dot{v}_{r})$, with
$u_{i},\dot{u_{i}}\in\mathbb{R}^{n}$ and $v_{i},\dot{v}_{i}\in\mathbb{R}^{m}$.
For every $1\leq i,j\leq r$, by Theorem 4.1 the elementary skew matrix
$A_{i,j}\coloneqq E_{i,j}-E_{j,i}\in\mathcal{A}_{r}$ leads to a conserved
function $h_{i,j}\coloneqq h_{A_{i,j}}$ (see (23)). Since $(n,m)\neq(1,1)$,
each conserved function predicted by Theorem 4.1 is of the form $h_{A}$,
$A\in\mathcal{A}_{r}$, hence is a linear combination of $h_{i,j}$, $1\leq
i,j\leq r$. As a result, conserved function predicted by Theorem 4.1 satisfy
$\nabla h_{A}(\alpha)\in\mathrm{span}\\{\nabla h_{i,j}(\alpha):1\leq i,j\leq
r\\}$ for every $\alpha=(t,\theta,\dot{\theta})$. We will show below that for
any $\alpha\in\widetilde{\Theta}$, there is a set of indices
$S\subseteq\\{1,\ldots r\\}$ of cardinality $R=\min(r,2(n+m))$ and a
neighborhood $\widetilde{\Omega}$ of $\alpha$ such that, with
$T\coloneqq\\{(i,j)\in S\times S,i<j\\}\cup(S\times S^{c})$, we have for every
$\alpha^{\prime}\in\widetilde{\Omega}$:

  * •

$\nabla h_{k,\ell}(\alpha^{\prime})\in\mathrm{span}\\{\nabla
h_{i,j}(\alpha^{\prime}):(i,j)\in T\\}$ for every $(k,\ell)\notin T$;

  * •

the vectors $\nabla h_{i,j}(\alpha^{\prime})$, $(i,j)\in T$ are linearly
independent.

This will conclude since

$\sharp T=R(R-1)/2+R(r-R)=\begin{cases}r(r-1)/2&\text{if}\ R=r<2(n+m)\\\
(n+m)[(r-2(n+m))+r-1]&\text{if}\ R=2(n+m)\leq r.\end{cases}$

To proceed, specialize the definition (23) of $h_{A}$ to our context: $q=2$,
$U_{1}=U$ and $U_{2}=V^{\top}$ yields
$h_{A}(\alpha)=h_{A}(t,\theta,\dot{\theta})=\exp\Big{(}\int^{t}\tau(s)\mathrm{d}s\Big{)}H_{A}(\theta,\dot{\theta})$
with
$H_{A}(\theta,\dot{\theta})\coloneqq\langle\dot{U},UA\rangle+\langle\dot{V}^{\top},V^{\top}A^{\top}\rangle=\langle\dot{U},UA\rangle+\langle\dot{V},VA\rangle=\langle\dot{U}A^{\top},U\rangle+\langle\dot{V}A^{\top},V\rangle$.
As a result

$\nabla
h_{i,j}(t,\theta,\dot{\theta})=(\tau(t)h_{i,j}(t,\theta,\dot{\theta}),\exp(\int^{t}\tau(s)\mathrm{d}s)\times[\nabla
H_{A_{i,j}}(\theta,\dot{\theta})]^{\top})^{\top}$ (30)

where, up to proper reshaping

$\nabla H_{A}(\theta,\dot{\theta})=\left(\begin{matrix}\nabla_{U}H_{A}\\\
\nabla_{V}H_{A}\\\ \nabla_{\dot{U}}H_{A}\\\
\nabla_{\dot{V}}H_{A}\end{matrix}\right)=\left(\begin{matrix}\dot{U}A^{\top}\\\
\dot{V}A^{\top}\\\ UA\\\ VA\end{matrix}\right)\stackrel{{\scriptstyle
A^{\top}=-A}}{{=}}\left(\begin{matrix}-\dot{U}A\\\ -\dot{V}A\\\ UA\\\
VA\end{matrix}\right)=\underbrace{(-\dot{U};-\dot{V};U;V)}_{=:W}A.$

hence the $2(n+m)\times r$ matrix $W=W(\alpha)$ will play a special role.
Denote $w_{j}(\alpha)$, $1\leq j\leq r$ its columns.

Observe that given $\alpha=(t,\theta,\dot{\theta})\in\widetilde{\Theta}$,
$R\coloneqq\min(r,2(n+m))$ is both the rank of
$(\theta,\dot{\theta})=(U;V;\dot{U};\dot{V})$ and the rank of $W$. Hence,
given $\alpha\in\widetilde{\Theta}$, there is a subset of indices
$S\subseteq\\{1,\ldots r\\}$ of cardinality $R$ such that the vectors
$w_{j}(\alpha)$, $j\in S$ are linearly independent, while for $k\notin S$ we
have $w_{k}(\alpha)\in\mathrm{span}(w_{j}(\theta,\dot{\theta}),j\in S)$. By
standard calculus, there is a neighborhood
$\widetilde{\Omega}\subseteq\widetilde{\Theta}$ of $\alpha$ such that these
properties remain valid (with the same $S$) for every
$\alpha^{\prime}\in\widetilde{\Omega}$. We show below that this implies the
claimed linear (in)dependence properties of the vectors $\nabla
h_{i,j}(\alpha^{\prime})$. From now on we omit the dependence in
$\alpha^{\prime}$ for the sake of brevity.

By (30), to show that the vectors $\nabla h_{i,j}$, $(i,j)\in T$ are linearly
independent, it is sufficient to show the linear independence of $\nabla
H_{A_{i,j}}\coloneqq WA_{i,j}$, $(i,j)\in T$. Assume that $0_{2(n+m)\times
r}=\sum_{(i,j)\in T}\lambda_{i,j}WA_{i,j}$. Our goal is now to show that
$\lambda_{k,\ell}=0$ for every $(k,\ell)\in T=(T\cap(S\times S))\cup(S\times
S^{c})$. We first prove it for every $(k,\ell)\in S\times S^{c}$, then on
$T\cap(S\times S)$.

First consider $(k,\ell)\in S\times S^{c}$. For any $(i,j)\in T$, by the
definition of $T$ we have $i\in S$, hence $\ell\neq i$ since $\ell\notin S$.
Using the standard notation for canonical vectors and Kronecker deltas, we
thus have $e_{i}^{\top}e_{\ell}=\delta_{i,\ell}=0$, and since
$A_{i,j}=e_{i}e_{j}^{\top}-e_{j}e_{i}^{\top}$ we obtain
$A_{i,j}e_{\ell}=e_{i}\delta_{j,\ell}$, so that
$WA_{i,j}e_{\ell}=\delta_{j,\ell}w_{i}$ and

$0=\sum_{(i,j)\in T}\lambda_{i,j}WA_{i,j}e_{\ell}=\sum_{(i,j)\in
T}\lambda_{i,j}\delta_{j,\ell}w_{i}=\sum_{i\in S}\Big{(}\sum_{j:(i,j)\in
T}\lambda_{i,j}\delta_{j,\ell}\Big{)}w_{i}.$

where by convention an empty sum is zero. By the linear independence of
$w_{i}$, $i\in S$ we get for each $i\in S$ that $0=\sum_{j:(i,j)\in
T}\lambda_{i,j}\delta_{j,\ell}$. Since $(k,\ell)\in T$, specializing to
$i\coloneqq k\in S$, we obtain $0=\sum_{j:(k,j)\in
T}\lambda_{k,j}\delta_{j,\ell}=\lambda_{k,\ell}$ as claimed.

Since the above holds for any $(k,\ell)\in S\times S^{c}$, and given the
definition of $T$, we have established that in fact

$0_{2(n+m)\times r}=\sum_{(i,j)\in T\cap(S\times
S)}\lambda_{i,j}WA_{i,j}=W\Big{(}\sum_{(i,j)\in T\cap(S\times
S)}\lambda_{i,j}A_{i,j}\Big{)}=WB,$

where $B\coloneqq\sum_{(i,j)\in T\cap(S\times S)}\lambda_{i,j}A_{i,j}$.
Observe that, by definition of $A_{i,j}$, we have $B(S^{c},:)=0$ (the rows of
$B$ indexed by $S^{c}$ are zero), hence $0=WB=W(:,S)B(S,:)$. By the linear
independence of the columns $w_{j}$, $j\in S$ of $W(:,S)$, we conclude that
$B(S,:)=0$, hence $B=0$, that is to say $\sum_{(i,j)\in T\cap(S\times
S)}\lambda_{i,j}A_{i,j}=0$. Since $(i,j)\in T$ implies $i\neq j$ and as the
matrices $A_{i,j}$, $i\neq j$, are linearly independent, we conclude that
$\lambda_{i,j}=0$ for every $(i,j)\in T\cap(S\times S)$.

This establishes that $\nabla h_{i,j}(\alpha^{\prime})$, $(i,j)\in T$, are
linearly independent.

To conclude the proof, there remains to show that $\nabla
h_{k,\ell}(\alpha^{\prime})\in\mathrm{span}\\{\nabla
h_{i,j}(\alpha^{\prime}):(i,j)\in T\\}$ for every $(k,\ell)\notin T$.

Consider $(k,\ell)\notin T$. As $w_{k}$ and $w_{\ell}$ are linear combinations
of $\\{w_{j}\\}_{j\in S}$ there exists $(\alpha_{j})_{j}\neq(0)$ and
$(\beta_{j})_{j}\neq(0)$ such that $w_{k}=\sum_{j\in S}\alpha_{j}w_{j}$ and
$w_{\ell}=\sum_{j\in S}\beta_{j}w_{j}$. As a result

$\nabla h_{k,\ell}=\sum_{j\in S}(\alpha_{j}\nabla h_{j,\ell}-\beta_{j}\nabla
h_{j,k})-\sum_{i,j\in S}\alpha_{i}\beta_{j}\nabla h_{i,j}.$

∎

##  Appendix H Proof of Theorem 4.3

First, we recall some definitions/results about Lie algebra, directly taken
from (Marcotte et al., 2024) and we add a supplementary lemma (Lemma H.2) that
states useful results.

###  H.1 Background on Lie algebra

A Lie algebra $\mathcal{A}$ is a vector space endowed with a bilinear map
$[\cdot,\cdot]$, called a Lie bracket, that verifies for all
$X,Y,Z\in{\mathcal{A}}$: $[X,X]=0$ and the Jacobi identity:
$[X,[Y,Z]]+[Y,[Z,X]]+[Z,[X,Y]]=0.$

Typically, the Lie algebra of interest is the set of infinitely smooth vector
fields
${\mathcal{A}\coloneqq}\mathcal{C}^{\infty}(\tilde{\Theta},\mathbb{R}^{2D+1})$,
endowed with the Lie bracket $[\cdot,\cdot]$ defined by

$[\chi_{1},\chi_{2}]:\quad\alpha\in\tilde{\Theta}\mapsto[\chi_{1},\chi_{2}](\alpha)\coloneqq\partial\chi_{1}(\alpha)\chi_{2}(\alpha)-\partial\chi_{2}(\alpha)\chi_{1}(\alpha),$
(31)

with $\partial\chi(\alpha)\in\mathbb{R}^{(2D+1)\times(2D+1)}$ the jacobian of
$\chi$ at $\alpha$. The space $\mathbb{R}^{n\times n}$ of matrices is also a
Lie algebra endowed with the Lie bracket $[A,B]\coloneqq AB-BA.$ This can be
seen as a special case of (31) in the case of _linear_ vector fields, i.e.
$\chi(\alpha)=A\alpha$.

##### Generated Lie algebra

Let $\mathcal{A}$ be a Lie algebra and let $\mathcal{W}\subset\mathcal{A}$ be
a vector subspace of $\mathcal{A}$. There exists a smallest Lie algebra that
contains $\mathcal{W}$. It is denoted $\mathrm{Lie}(\mathcal{W})$ and called
the generated Lie algebra of $\mathcal{W}$. The following proposition (See
Definition 20 of (Bonnard et al., 2018)) constructively characterizes
$\mathrm{Lie}(\mathcal{W})$, where for vector subspaces
$[\mathcal{W},\mathcal{W}^{\prime}]\coloneqq\\{[\chi_{1},\chi_{2}]:\chi_{1}\in\mathcal{W},\chi_{2}\in\mathcal{W^{\prime}}\\}$,
and
$\mathcal{W}+\mathcal{W}^{\prime}=\\{\chi_{1}+\chi_{2}:\chi_{1}\in\mathcal{W},\chi_{2}\in\mathcal{W}^{\prime}\\}$.

######  Proposition H.1.

Given any vector subspace $\mathcal{W}\subseteq\mathcal{A}$ we have
$\mathrm{Lie}(\mathcal{W})=\bigcup_{k}\mathcal{W}_{k}$ where:

$\left\\{\begin{array}[]{ll}\mathcal{W}_{0}&\coloneqq\mathcal{W}\\\
\mathcal{W}_{k}&\coloneqq\mathcal{W}_{k-1}+[\mathcal{W}_{0},\mathcal{W}_{k-1}]\
\text{ for }\ k\geq 1.\end{array}\right.$

######  Lemma H.2.

let $\mathcal{W}\subseteq\mathcal{X}\left(\tilde{\Omega}\right)$ be a vector
space. Then by considering

$\mathcal{D}(\mathcal{W})\coloneqq\mathrm{span}\\{a(\cdot)\chi(\cdot):a\in\mathcal{C}^{\infty}(\tilde{\Omega},\mathbb{R}),\chi\in\mathcal{W}\\},$
(32)

one has
$\mathcal{D}(\mathrm{Lie}(\mathcal{D}(\mathcal{W})))=\mathrm{Lie}(\mathcal{D}(\mathcal{W}))$
and for all $\alpha\in\tilde{\Omega}$,
$[\mathrm{Lie}(\mathcal{W})](\alpha)=[\mathrm{Lie}(\mathcal{D}(\mathcal{W}))](\alpha)$.

###### Proof.

We first show by recursion that for all $k\in\mathbb{N}$:

$\mathcal{D}(\mathcal{W})_{k}=\mathcal{D}\left(\mathcal{W}_{k}\right).$ (33)

where the iterates are defined in Proposition H.1.

Initialisation. One has:
$\mathcal{D}(\mathcal{W})_{0}=\mathcal{D}(\mathcal{W})=\mathcal{D}(\mathcal{W}_{0})$
by definition of the first iterate.

Recursion. Let $k\in\mathbb{N}$ and assume that $k$ satifies (33). Let us show
that
$\mathcal{D}(\mathcal{W})_{k+1}=\mathcal{D}\left(\mathcal{W}_{k+1}\right)$. By
definition (Proposition H.1), one has

$\mathcal{D}\left(\mathcal{W}\right)_{k+1}=\mathcal{D}\left(\mathcal{W}\right)_{k}+[\mathcal{D}\left(\mathcal{W}\right),\mathcal{D}\left(\mathcal{W}\right)_{k}].$
(34)

Thus
$\mathcal{D}\left(\mathcal{W}\right)_{k+1}\stackrel{{\scriptstyle\eqref{eq:recurrence}}}{{=}}\mathcal{D}\left(\mathcal{W}_{k}\right)+[\mathcal{D}\left(\mathcal{W}\right),\mathcal{D}\left(\mathcal{W}_{k}\right)].$
We first show the direct inclusion
$\mathcal{D}(\mathcal{W})_{k+1}\subseteq\mathcal{D}(\mathcal{W}_{k+1})$. Since
$\mathcal{W}_{k}\subseteq\mathcal{W}_{k+1}$ by construction, we have
$\mathcal{D}(\mathcal{W}_{k})\subseteq\mathcal{D}(\mathcal{W}_{k+1})$, hence
it is enough to show that
$[\mathcal{D}\left(\mathcal{W}\right),\mathcal{D}\left(\mathcal{W}_{k}\right)]\subseteq\mathcal{D}\left(\mathcal{W}_{k+1}\right)$.
Let $X\in\mathcal{D}\left(\mathcal{W}\right)$ and
$Y\in\mathcal{D}\left(\mathcal{W}_{k}\right)$. By definition of the operator
$\mathcal{D}$, there are smooth real-valued functions $a_{i},b_{j}$ and
$\chi_{i}\in\mathcal{W},\mu_{j}\in\mathcal{W}_{k}$ such that
$X(\cdot)=\sum_{1}^{m_{1}}a_{i}(\cdot)\chi_{i}(\cdot)$ and
$Y(\cdot)=\sum_{1}^{m_{2}}b_{j}(\cdot)\mu_{j}(\cdot)$ on $\tilde{\Omega}$ and
we deduce by bilinearity of the Lie brackets that
$[X,Y](\cdot)=\sum_{i,j}[a_{i}\chi_{i},b_{j}\mu_{j}](\cdot)$ on
$\tilde{\Omega}$. Moreover, one has:

$[a_{i}\chi_{i},b_{j}\mu_{j}]=\underbrace{a_{i}b_{j}[\chi_{i},\mu_{j}]}_{\in\mathcal{D}([\mathcal{W},\mathcal{W}_{k}])}+\underbrace{b_{j}[(\partial
a_{i})\mu_{j}]\chi_{i}}_{\in\mathcal{D}(\mathcal{W})}-\underbrace{a_{i}[(\partial
b_{j})\chi_{i}]\mu_{j}}_{\in\mathcal{D}(\mathcal{W}_{k}))},$ (35)

where, due to dimensions, both $(\partial a_{i})\mu_{j}$ and $(\partial
b_{j})\chi_{i}$ are smooth scalar-valued functions. Thus
$[X,Y]\in\mathcal{D}\left([\mathcal{W},\mathcal{W}_{k}]+\mathcal{W}_{k}\right)=\mathcal{D}(\mathcal{W}_{k+1})$.
Finally,
$[\mathcal{D}\left(\mathcal{W}\right),\mathcal{D}\left(\mathcal{W}_{k}\right)]\subseteq\mathcal{D}\left(\mathcal{W}_{k+1}\right)$
and thus
$\mathcal{D}(\mathcal{W})_{k+1}\subseteq\mathcal{D}\left(\mathcal{W}_{k+1}\right)$.
We now show the converse inclusion. Let $X\in\mathcal{D}(\mathcal{W}_{k+1})$.
There are smooth real-valued functions $a_{i}$ and
$Y_{i}\in\mathcal{W}_{k+1}$, such that $X=\sum_{i}a_{i}Y_{i}$. As by
definition: $\mathcal{W}_{k+1}=\mathcal{W}_{k}+[\mathcal{W}_{k},\mathcal{W}]$,
there exists $\chi_{i},v_{i}\in\mathcal{W}_{k},\mu_{i}\in\mathcal{W}$ such
that: $Y_{i}=v_{i}+[\chi_{i},\mu_{i}]$. Thus
$X=\sum_{i}a_{i}\left(v_{i}+[\chi_{i},\mu_{i}]\right)$. As
$\mathcal{D}(\mathcal{W}_{k+1})$ is a linear space, it is enough to show that
both $a_{i}v_{i}$ and $a_{i}[\chi_{i},\mu_{i}]$ are in
$\mathcal{D}(\mathcal{W}_{k+1})$. As
$v_{i}\in\mathcal{W}_{k}\subseteq\mathcal{W}_{k+1}$, one has directly that
$a_{i}v_{i}\in\mathcal{D}(\mathcal{W}_{k+1})$. Finally by using again equality
(35) (with $b_{i}\equiv 1$ and $j=i$), one has:

$a_{i}[\chi_{i},\mu_{i}]=[a_{i}\chi_{i},\mu_{i}]-(\partial
a_{i}\mu_{i})\chi_{i}.$

Both $a_{i}\chi_{i}$ and $(\partial a_{i}\mu_{i})\chi_{i}$ are in
$\mathcal{D}(\mathcal{W}_{k})\stackrel{{\scriptstyle\eqref{eq:recurrence}}}{{=}}\mathcal{D}(\mathcal{W})_{k}$
since $a_{i}$ and $\partial a_{i}\mu_{i}$ are smooth real-valued functions. We
also have $\mu_{i}\in\mathcal{W}\subseteq\mathcal{D}(\mathcal{W})$ hence,
using the characterization (34), one has
$a_{i}[\chi_{i},\mu_{i}]\in\mathcal{D}(\mathcal{W})_{k+1}$ and thus
$X\in\mathcal{D}(\mathcal{W})_{k+1}$, which concludes the recursion.

We now prove that
$\mathcal{D}(\mathrm{Lie}(\mathcal{D}(\mathcal{W})))=\mathrm{Lie}(\mathcal{D}(\mathcal{W}))$.
One inclusion is trivial so we only need to prove the other one. Let
$X\in\mathrm{Lie}(\mathcal{D}(\mathcal{W}))$ and let us consider $a$ a smooth
real-valued function. To conclude, we need to show that
$aX\in\mathrm{Lie}(\mathcal{D}(\mathcal{W}))$. By definition of the generated
Lie algebra Proposition H.1, there exists $k$ such that
$X\in\mathcal{D}(\mathcal{W})_{k}$. Then, by using (33), one has
$X\in\mathcal{D}(\mathcal{W}_{k})$ and thus
$aX\in\mathcal{D}(\mathcal{W}_{k})\stackrel{{\scriptstyle\eqref{eq:recurrence}}}{{=}}\mathcal{D}(\mathcal{W})_{k}\subseteq\mathrm{Lie}(\mathcal{D}(\mathcal{W}))$.

Finally, we now prove that for all $\alpha\in\tilde{\Omega}$,
$[\mathrm{Lie}(\mathcal{W})](\alpha)=[\mathrm{Lie}(\mathcal{D}(\mathcal{W}))](\alpha)$.
Let $\alpha\in\tilde{\Omega}$. By using (33), one has for all
$k\in\mathbb{N}$,
$[\mathcal{D}(\mathcal{W})_{k}](\alpha)=[\mathcal{D}\left(\mathcal{W}_{k}\right)](\alpha)=[\mathcal{W}_{k}](\alpha).$
Then
$[\mathrm{Lie}(\mathcal{D}(\mathcal{W}))](\alpha)=[\mathrm{Lie}(\mathcal{W})](\alpha)$,
which concludes the proof. ∎

###  H.2 Proof of Theorem 4.3

First, we recall the statement of the theorem for the reader’s convenience.

See 4.3

The proof of this theorem relies on Theorem 3.7: we compute
$\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha)$ with
$\phi=\phi_{\mathtt{Lin}}$ and show that its dimension is locally constant
around $\alpha=(t,\theta,\dot{\theta})$ where, $\theta=\mathtt{vec}((U;V))$
and $\dot{\theta}=\mathtt{vec}((\dot{U};\dot{V}))$ are vectorized versions of
two $(n+m)\times r$ matrices such that
$(U;V;\dot{U};\dot{V})\in\mathbb{R}^{2(n+m)\times r}$ has full rank.

First we make $\mathcal{W}_{\phi}^{\mathtt{mom}}$ more explicit. Recall that
$\mathcal{W}_{\phi}^{\mathtt{mom}}$ is the functional linear space spanned by
the functions $\alpha\mapsto\chi_{i}(\alpha)$ defined in (8). Since we
consider the Euclidean geometry, we have $M=\mathrm{Id}_{D}$ so that any
function $\chi\in\mathcal{W}_{\phi}^{\mathtt{mom}}$ is a linear combination of
$\chi_{i}(\cdot)$, $0\leq i\leq d$, hence it satisfies

$\chi(\alpha)=\beta\chi_{0}(\alpha)+\sum_{i=1}^{d}\gamma_{i}\chi_{i}(\alpha)=\begin{pmatrix}\beta\\\
\beta\dot{\theta}\\\
[\partial\phi(\theta)]^{\top}\gamma-\tau(t)\beta\dot{\theta}\end{pmatrix}$

for some $\beta\in\mathbb{R}$ and
$\gamma\coloneqq(\gamma_{i})_{i=1}^{d}\in\mathbb{R}^{d}$. Since $d=nm$, we can
write $\gamma=\mathtt{vec}(\Delta)$ with $\Delta\in\mathbb{R}^{n\times m}$,
and leveraging (Marcotte et al., 2024, Proposition H.2), we obtain
$\partial\phi(\theta)^{\top}\gamma=\partial\phi(\theta)^{\top}\mathtt{vec}(\Delta)=\mathtt{vec}\left(S_{\Delta}(U;V)\right)$
where $S_{\Delta}\coloneqq\begin{pmatrix}0&\Delta\\\
\Delta^{\top}&0\end{pmatrix}\in\mathbb{R}^{(n+m)\times(n+m)}$. Using a basic
property444this property can indeed serve as an operational definition of the
Kronecker product between matrices. of Kronecker products
($(\mathbf{A}\otimes\mathbf{B})\mathtt{vec}(\mathbf{X})=\mathtt{vec}(\mathbf{B}\mathbf{X}\mathbf{A}^{\top})$),
this is further rewritten as
$\partial\phi(\theta)^{\top}\gamma=\mathtt{vec}\left(S_{\Delta}(U;V)\mathrm{Id}_{r}\right)=(\mathrm{Id}_{r}\otimes
S_{\Delta})\theta$. Overall we obtain that $\mathcal{W}_{\phi}^{\mathtt{mom}}$
is the collection of all vector fields

$\chi_{\beta,\Delta}(\alpha)\coloneqq\begin{pmatrix}\beta\\\
\beta\dot{\theta}\\\ (\mathrm{Id}_{r}\otimes
S_{\Delta})\theta-\tau(t)\beta\dot{\theta}\end{pmatrix}.$

Second, we express $\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha)$.
We highlight in purple the results and reasoning steps specific to $n=m=1$.

######  Proposition H.3.

Denote $\mathcal{S}_{\ell}\subset\mathbb{R}^{\ell\times\ell}$ the space of
symmetric matrices, $H\coloneqq\begin{pmatrix}0&\mathrm{Id}_{n+m}\\\
-\mathrm{Id}_{n+m}&0\end{pmatrix}$, and for any square matrix $M$ of size
$2(n+m)$ and $\beta\in\mathbb{R}$ denote

$\eta_{\beta,M}(\alpha)\coloneqq\begin{pmatrix}0\\\
[\mathrm{Id}_{r}\otimes(HM)]\cdot\begin{pmatrix}\theta\\\
\dot{\theta}\end{pmatrix}\end{pmatrix}+\beta\begin{pmatrix}1\\\ 0_{(n+m)r}\\\
-\tau(t)\dot{\theta}\end{pmatrix}.$

We have
$\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha)=\\{\eta_{\beta,M}(\alpha):\beta\in\mathbb{R},M\in\mathcal{S}\\}$
where $\mathcal{S}\coloneqq\mathcal{S}_{2(n+m)}$ when $(n,m)\neq(1,1)$, while
for $n=m=1$,
$\mathcal{S}\coloneqq\tilde{\mathcal{S}}_{4}\coloneqq\Big{\\{}\begin{pmatrix}S_{1}&S\\\
S&S_{2}\end{pmatrix}:S,S_{1},S_{2}\in\mathcal{S}_{2}^{\prime}\Big{\\}}\subsetneq\mathcal{S}_{4}$,
where $\mathcal{S}_{2}^{\prime}\subsetneq\mathcal{S}_{2}$ is the set of
symmetric matrices on the form $\begin{pmatrix}a&b\\\ b&a\end{pmatrix}$.

###### Proof.

First we show that
$\mathcal{D}(\mathcal{W}_{\phi}^{\mathtt{mom}})\subseteq\mathcal{D}(\mathcal{W})$,
where
$\mathcal{W}\coloneqq\\{\eta_{\beta,M}:\beta\in\mathbb{R},M\in\mathcal{S}\\}$
and with the operator $\mathcal{D}$ defined by (32). One has (see Section H.3
for more details):

$\chi_{\beta,\Delta}=\eta_{\beta,M},\text{ where
}M\coloneqq\begin{pmatrix}-S_{\Delta}&0\\\
0&\beta\mathrm{Id}_{(n+m)}\end{pmatrix}.$ (36)

The matrix $M$ is symmetric and belongs to $\mathcal{S}$ even when $n=m=1$.
Thus $\mathcal{W}_{\phi}^{\mathtt{mom}}\subseteq\mathcal{W}$ and
$\mathcal{D}(\mathcal{W}_{\phi}^{\mathtt{mom}})\subseteq\mathcal{D}(\mathcal{W})$.

We now prove that $\mathcal{D}(\mathcal{W})$ is a Lie algebra. Since for all
$a,b$ smooth real-valued functions and for all $X,Y\in\mathcal{W}$ one has:

$[aX,bY]=ab[X,Y]+\underbrace{b(\partial
aY)X}_{\in\mathcal{D}(\mathcal{W})}-\underbrace{a(\partial
bX)Y}_{\in\mathcal{D}(\mathcal{W})},$

where, due to dimensions, both $(\partial a)Y$ and $(\partial b)X$ are smooth
scalar-valued functions, it is enough to show that
$[\mathcal{W},\mathcal{W}]\subseteq\mathcal{D}(\mathcal{W})$. Moreover, since
$\eta_{\beta,M}=\eta_{0,M}+\beta\eta_{1,0}$, it is enough to check that
$[\eta_{0,M},\eta_{0,M^{\prime}}]$ and $[\eta_{1,0},\eta_{0,M}]$ are elements
of $\mathcal{D}(\mathcal{W})$ whenever $M,M^{\prime}\in\mathcal{S}$.

  * •

We first obtain (see Section H.3 for more details) that

$[\eta_{0,M},\eta_{0,M^{\prime}}]=\eta_{0,M^{\prime\prime}},\text{ where
}M^{\prime\prime}\coloneqq MHM^{\prime}-M^{\prime}HM.$ (37)

Since $H^{\top}=-H$ it is straightforward to check that
$M^{\prime\prime}\in\mathcal{S}_{2(n+m)}$, hence
$M^{\prime\prime}\in\mathcal{S}$ when $(n,m)\neq(1,1)$, and we let the reader
check that when $(n,m)=(1,1)$ we also have $M^{\prime\prime}\in\mathcal{S}$ as
soon as $M,M^{\prime}\in\mathcal{S}$ (note that $\mathcal{S}_{2}^{\prime}$ is
stable by matrix multiplication and is commutative). Therefore
$[\eta_{0,M},\eta_{0,M^{\prime}}]=\eta_{0,M^{\prime\prime}}\in\mathcal{W}\subseteq\mathcal{D}(\mathcal{W})$.

  * •

We now show that $[\eta_{0,M},\eta_{1,0}]\in\mathcal{D}(\mathcal{W})$ for
every $M\in\mathcal{S}$. Since $M\in\mathcal{S}$ we can write it as
$M=\begin{pmatrix}S_{1}&S\\\ S^{\top}&S_{2}\end{pmatrix}$ with
$S\in\mathbb{R}^{(n+m)\times(n+m)},S_{1},S_{2}\in\mathcal{S}_{n+m}$ (when
$n=m=1$ we further have $S,S_{1},S_{2}\in\mathcal{S}^{\prime}_{2}$). We then
obtain (see Section H.3 for more details):

$[\eta_{0,M},\eta_{1,0}](\alpha)=\tau(t)\eta_{0,M^{\prime}}(\alpha)\text{ with
}M^{\prime}\coloneqq\begin{pmatrix}S_{1}&0\\\ 0&-S_{2}\end{pmatrix}.$ (38)

Since $t\mapsto\tau(t)$ is $\mathcal{C}^{\infty}$ and
$M^{\prime}\in\mathcal{S}$ (even when $n=m=1$) this implies
$[\eta_{0,M},\eta_{1,0}]\in\mathcal{D}(\mathcal{W})$.

This establishes as claimed that $\mathcal{D}(\mathcal{W})$ is indeed a Lie
algebra.

Finally we prove that
$\mathcal{W}\subseteq\mathrm{Lie}(\mathcal{D}(\mathcal{W}_{\phi}^{\mathtt{mom}}))$.
This is where the necessity to impose the more restricted definition of
$\mathcal{S}$ for $n=m=1$ will become evident. Before proving this inclusion,
observe that by Lemma H.2 it will imply
$\mathcal{D}(\mathcal{W})\subseteq\mathcal{D}\left(\mathrm{Lie}(\mathcal{D}(\mathcal{W}_{\phi}^{\mathtt{mom}}))\right)=\mathrm{Lie}(\mathcal{D}(\mathcal{W}_{\phi}^{\mathtt{mom}}))$,
hence combined with what we already proved it implies
$\mathcal{D}(\mathcal{W})=\mathrm{Lie}(\mathcal{D}(\mathcal{W}_{\phi}^{\mathtt{mom}}))$.

As a shorthand denote
$\mathcal{V}\coloneqq\mathcal{D}(\mathcal{W}_{\phi}^{\mathtt{mom}})$. First,
we will building matrix sets $\mathcal{M}_{k},\mathcal{M}^{\prime}_{k}$ such
that $\eta_{0,M}\in\mathcal{V}_{k}$ for every
$M\in\mathcal{M}_{k}\cup\mathcal{M}^{\prime}_{k}$. Then we will show that
$\mathcal{W}\subseteq\\{\eta_{0,M},M\in\mathcal{M}\\}+{\mathcal{V}}$ where
$\mathcal{M}$ is the linear span of all built matrix sets. Since
$\mathcal{V}_{k}\subseteq\mathrm{Lie}(\mathcal{V})$ for every $k$, this will
yield the desired conclusion.

$\bullet$ We first prove that $\eta_{0,M}\in\mathcal{V}_{k}$ for every
$M\in\mathcal{M}_{k}$.

The set $\mathcal{M}_{0}$ is defined as the collection of all matrices that
write as

$M=\begin{pmatrix}-S_{\Delta}&0\\\ 0&0\end{pmatrix}$

for some $\Delta$. By (36), for any $\Delta$ we have
$\eta_{0,M}=\chi_{0,\Delta}\in\mathcal{V}_{0}$.

The set $\mathcal{M}_{1}$ is defined as the linear span of $\mathcal{M}_{0}$
and of the set of all matrices that write as

$M=\begin{pmatrix}0&S_{\Delta}\\\ S_{\Delta}&0\end{pmatrix}.$

Consider $M_{1}\coloneqq\begin{pmatrix}-S_{\Delta}&0\\\
0&0\end{pmatrix}\in\mathcal{M}_{0}$ and $M_{2}\coloneqq\begin{pmatrix}0&0\\\
0&\mathrm{Id}_{n+m}\end{pmatrix}$. As
$\eta_{1,M_{2}}=\eta_{0,M_{2}}+\eta_{1,0}$, by bilinearity of Lie brackets:

$[\chi_{0,\Delta},\chi_{1,0}](\alpha)\stackrel{{\scriptstyle\eqref{eq:lien_chi_et_mu}}}{{=}}[\eta_{0,M_{1}},\eta_{1,M_{2}}](\alpha)=[\eta_{0,M_{1}},\eta_{0,M_{2}}](\alpha)+[\eta_{0,M_{1}},\eta_{1,0}](\alpha)\stackrel{{\scriptstyle\eqref{eq:liebracketdeuxM},\eqref{eq:liebracketmixé}}}{{=}}\eta_{0,M_{3}}(\alpha)+\tau(t)\eta_{0,M_{4}}(\alpha),$

where
$M_{3}\stackrel{{\scriptstyle\eqref{eq:liebracketdeuxM}}}{{\coloneqq}}M_{1}HM_{2}-M_{2}HM_{1}=\begin{pmatrix}0&-S_{\Delta}\\\
-S_{\Delta}&0\end{pmatrix}$ and
$M_{4}\stackrel{{\scriptstyle\eqref{eq:liebracketmixé}}}{{=}}M_{1}\in\mathcal{M}_{0}$.
Thus:
$\eta_{0,M_{3}}(\alpha)=[\chi_{0,\Delta},\chi_{1,0}](\alpha)-\tau(t)\eta_{0,M_{4}}(\alpha)\in\mathcal{V}_{1}$
since $t\mapsto\tau(t)$ is $\mathcal{C}^{\infty}$.

The set $\mathcal{M}_{2}$ is defined as the linear span of $\mathcal{M}_{1}$
and of the set of matrices

$M=\begin{pmatrix}0&0\\\ 0&S_{\Delta}\end{pmatrix}.$

Consider $M_{1}\coloneqq\begin{pmatrix}0&S_{\Delta}\\\
S_{\Delta}&0\end{pmatrix}\in\mathcal{M}_{1}$ and
$M_{2}\coloneqq\begin{pmatrix}0&0\\\ 0&\mathrm{Id}_{n+m}\end{pmatrix}$. Since
$\eta_{0,M_{1}}\in\mathcal{V}_{1}$ and $\chi_{1,0}\in\mathcal{V}$, we have
$[\eta_{0,M_{1}},\eta_{1,M_{2}}]\stackrel{{\scriptstyle\eqref{eq:lien_chi_et_mu}}}{{=}}[\eta_{0,M_{1}},\chi_{1,0}]\in\mathcal{V}_{2}$
and as:
$[\eta_{0,M_{1}},\eta_{1,M_{2}}](\alpha)=[\eta_{0,M_{1}},\eta_{0,M_{2}}](\alpha)+[\eta_{0,M_{1}},\eta_{1,0}](\alpha)\stackrel{{\scriptstyle\eqref{eq:liebracketdeuxM},\eqref{eq:liebracketmixé}}}{{=}}\eta_{0,M_{3}}(\alpha)+\tau(t)\eta_{0,M_{4}}(\alpha),$
where
$M_{3}\stackrel{{\scriptstyle\eqref{eq:liebracketdeuxM}}}{{\coloneqq}}M_{1}HM_{2}-M_{2}HM_{1}=-2\begin{pmatrix}0&0\\\
0&S_{\Delta}\end{pmatrix}$ and
$M_{4}\stackrel{{\scriptstyle\eqref{eq:liebracketmixé}}}{{=}}0$. Thus
$\eta_{0,M_{4}}=0$ and we obtain
$\eta_{0,M_{3}}=[\eta_{0,M_{1}},\chi_{1,0}]\in\mathcal{V}_{2}$.

The set $\mathcal{M}_{3}$ is defined as the linear span of $\mathcal{M}_{2}$
and of the set of matrices

$M=\begin{pmatrix}0&S_{\Delta}S_{\Delta^{\prime}}\\\
S_{\Delta^{\prime}}S_{\Delta}&0\end{pmatrix},$

for any $\Delta,\Delta^{\prime}$. Such a matrix satisfies
$M=M_{2}HM_{1}-M_{1}HM_{2}$ where $M_{1},M_{2}$ are specified as
$M_{1}\coloneqq\begin{pmatrix}0&0\\\
0&S_{\Delta}\end{pmatrix}\in\mathcal{M}_{2}$ and
$M_{2}\coloneqq\begin{pmatrix}S_{\Delta^{\prime}}&0\\\
0&0\end{pmatrix}\in\mathcal{M}_{0}.$ By (37) it follows that
$\eta_{0,M}=[\eta_{0,M_{2}},\eta_{0,M_{1}}]\in\mathcal{V}_{3}$.

Before defining $\mathcal{M}_{5}$ we further explicit matrices contained in
$\mathcal{M}_{3}$. Given any $\Delta,\Delta^{\prime}\in\mathbb{R}^{n\times
m}$, denote $B_{1}\coloneqq\Delta{\Delta^{\prime}}^{\top}$ and
$B_{2}\coloneqq\Delta^{\top}{\Delta^{\prime}}$. Since
$S_{\Delta}S_{\Delta^{\prime}}=\begin{pmatrix}\Delta{\Delta^{\prime}}^{\top}&0\\\
0&\Delta^{\top}{\Delta^{\prime}}\end{pmatrix}$ we have

$\begin{pmatrix}0&0&B_{1}&0\\\ 0&0&0&B_{2}\\\ B_{1}^{\top}&0&0&0\\\
0&B_{2}^{\top}&0&0\end{pmatrix}\in\mathcal{M}_{3},$ (39)

for any pair of matrices $B_{1},B_{2}$ that can be written as above. We
explicit a few such matrices.

In the case $n=m=1$, $B_{1}=B_{2}=b\in\mathbb{R}$, and as
$\begin{pmatrix}0&S_{\Delta}\\\ S_{\Delta}&0\end{pmatrix}\in\mathcal{M}_{1}$
and $S_{\Delta}=\begin{pmatrix}0&a\\\ a&0\end{pmatrix}$ with $a\in\mathbb{R}$,
one has

$\begin{pmatrix}0&B\\\ B&0\end{pmatrix}\in\mathcal{M}_{3},\text{ for any
}B\in\mathcal{S}^{\prime}_{2}.$ (40)

When $(n,m)\neq(1,1)$, consider any $1\leq i,k\leq n$, $1\leq j,l\leq m$, and
$\Delta\coloneqq E_{i,j}\in\mathbb{R}^{n\times m}$ and
$\Delta^{\prime}\coloneqq E_{k,l}\in\mathbb{R}^{n\times m}$. Since
$B_{1}\coloneqq\Delta{\Delta^{\prime}}^{\top}=E_{i,j}E_{l,k}=\delta_{j,l}E_{i,k}\in\mathbb{R}^{n\times
n}$ and
$B_{2}\coloneqq\Delta^{\top}{\Delta^{\prime}}=E_{j,i}E_{k,l}=\delta_{i,k}E_{j,l}\in\mathbb{R}^{m\times
m}$, we can reach:

  * •

$(B_{1},B_{2})=(E_{i,i},E_{j,j})$, for any $1\leq i\leq n$ and $1\leq j\leq
m$, by choosing $k:=i$ and $l:=j$;

  * •

(in the case $m>1$) $(B_{1},B_{2})=(0,E_{j,l})$ for any $1\leq j\neq l\leq m$,
by choosing e.g. $i=k=1$;

  * •

(in the case $n>1$): $(B_{1},B_{2})=(E_{i,k},0)$ for any $1\leq i\neq k\leq
n$, by choosing $j=l=1$.

The two following steps are specific to the case $(n,m)\neq(1,1)$ only.

The set $\mathcal{M}_{5}$ (defined only when $(n,m)\neq(1,1)$ – and we skip
the definition of $\mathcal{M}_{4}$) is defined as the linear span of
$\mathcal{M}_{3}$ and of the set of matrices

$M=\begin{pmatrix}0&0&0&E_{i,j}\\\ 0&0&-E_{i,j}^{\top}&0\\\ 0&-E_{i,j}&0&0\\\
E_{i,j}^{\top}&0&0&0\end{pmatrix}$ (41)

with $1\leq i\leq n$ and $1\leq j\leq m$. Since $(n,m)\neq(1,1)$, without loss
of generality, assume that $n>1$ (a similar construction can be done in the
case $m>1$). Observe that $M=M_{2}HM_{1}-M_{1}HM_{2}$ where $M_{1},M_{2}$ are
specified as $M_{1}\coloneqq\begin{pmatrix}0&S_{\Delta}\\\
S_{\Delta}&0\end{pmatrix}\in\mathcal{M}_{1}$ (NB: not $\mathcal{M}_{0}$) with
$\Delta:=E_{i,j}$, and $M_{2}\coloneqq\begin{pmatrix}0&0&E_{k,k}&0\\\
0&0&0&E_{j,j}\\\ E_{k,k}&0&0&0\\\
0&E_{j,j}&0&0\end{pmatrix}\in\mathcal{M}_{3}$ (by (39)) for some $1\leq k\neq
i\leq n$ (such a choice of $k$ is possible since $n>1$). By (37) it follows
(using Jacobi identity and the fact that $M_{1}\in\mathcal{M}_{1}$) that
$\eta_{0,M}=[\eta_{0,M_{2}},\eta_{0,M_{1}}]\in\mathcal{V}_{5}$ (but not
$\mathcal{V}_{4}$).

Again, before defining the set $\mathcal{M}_{7}$ we show that for any
$\Delta_{1}\in\mathbb{R}^{n\times m},\Delta_{2}\in\mathbb{R}^{m\times n}$ we
have

${M^{\prime}\coloneqq}\begin{pmatrix}0&0&0&\Delta_{1}\\\ 0&0&\Delta_{2}&0\\\
0&\Delta_{2}^{\top}&0&0\\\
\Delta_{1}^{\top}&0&0&0\end{pmatrix}\in\mathcal{M}_{5}.$ (42)

Indeed $2M^{\prime}=\begin{pmatrix}0&0&0&\Delta_{1}\\\
0&0&-\Delta_{1}^{\top}&0\\\ 0&-\Delta_{1}&0&0\\\
\Delta_{1}^{\top}&0&0&0\end{pmatrix}+\begin{pmatrix}0&S_{\Delta_{1}}\\\
S_{\Delta_{1}}&0\end{pmatrix}+\begin{pmatrix}0&0&0&-\Delta_{2}^{\top}\\\
0&0&\Delta_{2}&0\\\ 0&\Delta_{2}^{\top}&0&0\\\
-\Delta_{2}&0&0&0\end{pmatrix}+\begin{pmatrix}0&S_{\Delta_{2}^{\top}}\\\
S_{\Delta_{2}^{\top}}&0\end{pmatrix}$ where the first and third terms are
combinations of matrices shaped as (41), while the second and last belong to
$\mathcal{M}_{1}$.

The set $\mathcal{M}_{7}$ (defined only when $(n,m)\neq(1,1)$ – and again we
skip the definition of $\mathcal{M}_{6}$) is defined as the linear span of
$\mathcal{M}_{5}$ and of the set of matrices

$M=\begin{pmatrix}0&0&E_{i,i}&0\\\ 0&0&0&-E_{j,j}\\\ E_{i,i}&0&0&0\\\
0&-E_{j,j}&0&0\end{pmatrix}$ (43)

with $1\leq i\leq n$ and $1\leq j\leq m$. Observe that
$M=M_{2}HM_{1}-M_{1}HM_{2}$ where
$M_{1}\coloneqq\begin{pmatrix}0&S_{\Delta}\\\
S_{\Delta}&0\end{pmatrix}\in\mathcal{M}_{1}$ (NB: not $\mathcal{M}_{0}$) with
$\Delta=E_{i,j}$, and ${M_{2}\coloneqq}\begin{pmatrix}0&0&0&0\\\
0&0&E_{j,i}&0\\\ 0&E_{j,i}^{\top}&0&0\\\
0&0&0&0\end{pmatrix}\in\mathcal{M}_{5}$ (by (42)). By (37) it follows (using
again Jacobi identity and the fact that $M_{1}\in\mathcal{M}_{1}$) that
$\eta_{0,M}=[\eta_{0,M_{2}},\eta_{0,M_{1}}]\in\mathcal{V}_{7}$ (and not
$\mathcal{V}_{6}$).

As $\mathcal{M}_{7}$ is a vector space and since we have already (39) with
$(B_{1},B_{2})=(E_{ii},E_{jj})$, by linear combination with matrices shaped as
in (43) we obtain that any matrix shaped as in (39) with arbitrary diagonal
$B_{1},B_{2}$ also belongs to $\mathcal{M}_{7}$. Arbitrary off-diagonal terms
can be obtained by combining matrices shaped as in (39) (if $m>1$, $B_{1}=0$
and $B_{2}=E_{j,l}$ and if $n>1$, $B_{1}=E_{i,k}$ and $B_{2}=0$), and we
obtain

$M=\begin{pmatrix}0&0&B_{1}&0\\\ 0&0&0&B_{2}\\\ B_{1}^{\top}&0&0&0\\\
0&B_{2}^{\top}&0&0\end{pmatrix}\in\mathcal{M}_{7},\text{ for each
}B_{1}\in\mathbb{R}^{n\times n},B_{2}\in\mathbb{R}^{m\times m}.$ (44)

Finally, combining (44) and (42) one has

$\begin{pmatrix}0&B\\\ B^{\top}&0\end{pmatrix}\in\mathcal{M}_{7}\text{ for any
}B\in\mathbb{R}^{(n+m)\times(n+m)}.$ (45)

The set $\mathcal{M}_{8}$ (resp. $\mathcal{M}^{\prime}_{4}$) is defined as the
linear span of $\mathcal{M}_{7}$ (resp. of $\mathcal{M}_{3}$ ) and of the set
of matrices

$M=\begin{pmatrix}\Delta_{1}\Delta^{\top}+\Delta\Delta_{1}^{\top}&0&0&0\\\
0&\Delta_{2}\Delta+\Delta^{\top}\Delta_{2}^{\top}&0&0\\\ 0&0&0&0\\\
0&0&0&0\end{pmatrix},\text{{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. with $\Delta_{2}=\Delta_{1}^{\top}$)}} },$

and

$M^{\prime}=\begin{pmatrix}0&0&0&0\\\ 0&0&0&0\\\
0&0&\Delta\Delta_{2}+\Delta_{2}^{\top}\Delta^{\top}&0\\\
0&0&0&\Delta^{\top}\Delta_{1}+\Delta_{1}^{\top}\Delta\end{pmatrix},\text{{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. with $\Delta_{2}=\Delta_{1}^{\top}$)}} },$

which satisfy $M=M_{2}HM_{1}-M_{1}HM_{2}$ and
$M^{\prime}=M_{2}^{\prime}HM_{1}-M_{1}HM_{2}^{\prime}$ where
$M_{1}\coloneqq\begin{pmatrix}0&0&0&\Delta_{1}\\\ 0&0&\Delta_{2}&0\\\
0&\Delta_{2}^{\top}&0&0\\\
\Delta_{1}^{\top}&0&0&0\end{pmatrix}\in\mathcal{M}_{5}$ (in the case
$(n,m)=(1,1)$, $M_{1}=\begin{pmatrix}0&S_{\Delta_{1}}\\\
S_{\Delta_{1}}&0\end{pmatrix}\in\mathcal{M}_{1}$),
$M_{2}\coloneqq\begin{pmatrix}S_{\Delta}&0\\\
0&0\end{pmatrix}\in\mathcal{M}_{0}$ and
$M_{2}^{\prime}\coloneqq-\begin{pmatrix}0&0\\\
0&S_{\Delta}\end{pmatrix}\in\mathcal{M}_{2}.$

By (37) it directly follows that
$\eta_{0,M}=[\eta_{0,M_{2}},\eta_{0,M_{1}}]\in\mathcal{V}_{6}\subseteq\mathcal{V}_{8}$
(resp. $\eta_{0,M}\in\mathcal{V}_{2}\subseteq\mathcal{V}_{4}$). Similarly,
using Jacobi identity and the fact that $M^{\prime}_{2}\in\mathcal{M}_{2}$
(resp. that $M_{1}\in\mathcal{M}_{1}$) we obtain that
$\eta_{0,M^{\prime}}=[\eta_{0,M_{2}^{\prime}},\eta_{0,M_{1}}]\in\mathcal{V}_{8}$
(resp. $\eta_{0,M^{\prime}}\in\mathcal{V}_{4}$).

Again, we now explicit matrices belonging to $\mathcal{M}_{8}$ (resp. to
$\mathcal{M}^{\prime}_{4}$).

By considering $M_{3}\coloneqq\begin{pmatrix}S_{\Delta^{\prime\prime}}&0\\\
0&0\end{pmatrix}\in\mathcal{M}_{0}$, we obtain that
$M+M_{3}=\begin{pmatrix}S_{1}&\Delta^{\prime\prime}&0&0\\\
{\Delta^{\prime\prime}}^{\top}&S_{2}&0&0\\\ 0&0&0&0\\\
0&0&0&0\end{pmatrix}\in\mathcal{M}_{8}^{\prime}\text{{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. $\mathcal{M}_{4}^{\prime}$)}} },$ with
$S_{1}\coloneqq\Delta_{1}\Delta^{\top}+\Delta\Delta_{1}^{\top}\in\mathcal{S}_{n}$
and
$S_{2}\coloneqq\Delta_{2}\Delta+\Delta^{\top}\Delta_{2}^{\top}\in\mathcal{S}_{m}$
(resp. in the case $n=m=1$, one has $S_{1}=S_{2}$ as
$\Delta_{2}=\Delta_{1}^{\top}$), and since this holds for any choice of
$\Delta,\Delta_{1},\Delta_{2},\Delta^{\prime\prime}$, one has

$\begin{pmatrix}S&0\\\
0&0\end{pmatrix}\in\mathcal{M}_{8}^{\prime}\text{{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. $\mathcal{M}_{4}^{\prime}$)}} }\text{ for any
}S\in\mathcal{S}_{n+m}\text{{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. $\mathcal{S}_{n+m}^{\prime}$)}} }.$ (46)

Similarly, by considering $M_{3}^{\prime}\coloneqq\begin{pmatrix}0&0\\\
0&S_{\Delta^{\prime\prime}}\end{pmatrix}\in\mathcal{M}_{2}$, we obtain that
$M^{\prime}+M_{3}^{\prime}=\begin{pmatrix}0&0&0&0\\\ 0&0&0&0\\\
0&0&S_{1}&\Delta^{\prime\prime}\\\
0&0&{\Delta^{\prime\prime}}^{\top}&S_{2}\end{pmatrix}\in\mathcal{M}_{8}^{\prime}\text{{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. $\mathcal{M}_{4}^{\prime}$)}} },$ with
$S_{1}\coloneqq\Delta_{1}\Delta^{\top}+\Delta\Delta_{1}^{\top}\in\mathcal{S}_{n}$
and
$S_{2}\coloneqq\Delta_{2}\Delta+\Delta^{\top}\Delta_{2}^{\top}\in\mathcal{S}_{m}$
(resp. in the case $n=m=1$, one has $S_{1}=S_{2}$ as
$\Delta_{2}=\Delta_{1}^{\top}$), and thus one has

$\begin{pmatrix}0&0\\\
0&S\end{pmatrix}\in\mathcal{M}^{\prime}_{8}\text{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. $\mathcal{M}^{\prime}_{4}$)}}\text{ for any
}S\in\mathcal{S}_{n+m}\text{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. $\mathcal{S}^{\prime}_{2}$)}}.$ (47)

Thus by combining (46) and (47), one has:

$\begin{pmatrix}S_{1}&0\\\
0&S_{2}\end{pmatrix}\in\mathcal{M}^{\prime}_{8}\text{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{
(resp. $\mathcal{M}^{\prime}_{4}$)}}\text{ for each
}S_{1},S_{2}\in\mathcal{S}_{n+m}{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}{\text{
(resp. }\mathcal{S}_{2}^{\prime})}}.$ (48)

$\bullet$ We now show that
$\mathcal{W}\subseteq\\{\eta_{0,M},M\in\mathcal{M}\\}+{\mathcal{V}}$ where
$\mathcal{M}$ is the linear span of all built matrix sets.

First by combining Equation 45 and Equation 48, one has
$\\{\eta_{0,M},M\in\mathcal{S}\\}\subseteq\\{\eta_{0,M},M\in\mathcal{M}\\}$

For any $\beta\in\mathbb{R}$ one has:
$\beta\eta_{1,0}(\alpha)=\beta\begin{pmatrix}1\\\ 0\\\
-\tau(t)\dot{\theta}\end{pmatrix}=\chi_{\beta,0}(\alpha)-\beta\begin{pmatrix}0\\\
\dot{\theta}\\\ 0\end{pmatrix}=\chi_{\beta,0}(\alpha)-\eta_{0,\beta
M}(\alpha)$, where $M\coloneqq\begin{pmatrix}0&0\\\
0&\mathrm{Id}_{n+m}\end{pmatrix}\in\mathcal{M}$ (by (47)), and
$\chi_{\beta,0}\in{\mathcal{V}}$. Therefore
$\beta\eta_{1,0}\in\\{\eta_{0,M},M\in\mathcal{M}\\}+{\mathcal{V}}$. Then, for
any $\beta$, $M\in\mathcal{S}$,
$\eta_{\beta,M}=\beta\eta_{1,0}+\eta_{0,M}\in\\{\eta_{0,M},M\in\mathcal{M}\\}+{\mathcal{V}}$
so that: $\mathcal{W}\subseteq\\{\eta_{0,M},M\in\mathcal{M}\\}+{\mathcal{V}}$.

$\bullet$ Conclusion.

Since $\mathcal{V}_{k}\subseteq\mathrm{Lie}(\mathcal{V})$ for every $k$ and
since $\mathcal{V}\subseteq\mathrm{Lie}(\mathcal{V})$, we get
$\mathcal{W}\subseteq\\{\eta_{0,M},M\in\mathcal{M}\\}+{\mathcal{V}}{\subseteq\cup_{k}\mathcal{V}_{k}}\subseteq\mathrm{Lie}(\mathcal{V}).$

Conclusion. By using Lemma H.2, one has for all $\alpha$,
$\mathrm{Lie}\left(\mathcal{D}\left(\mathcal{W}_{\phi}^{\mathtt{mom}}\right)\right)(\alpha)=\mathrm{Lie}\left(\mathcal{W}_{\phi}^{\mathtt{mom}}\right)(\alpha)$,
and thus
$\mathcal{W}(\alpha)=\mathcal{D}(\mathcal{W})(\alpha)=\mathrm{Lie}\left(\mathcal{D}\left(\mathcal{W}_{\phi}^{\mathtt{mom}}\right)\right)(\alpha)=\mathrm{Lie}\left(\mathcal{W}_{\phi}^{\mathtt{mom}}\right)(\alpha)$,
which concludes the proof of Proposition H.3. ∎

Eventually, what we need to compute is the dimension of the trace
$\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha)$ for any
$\alpha=(t,U,V,\dot{U},\dot{V})$.

######  Proposition H.4.

Consider $\alpha=(t,{\theta,\dot{\theta}})$ such that
$\left(U;V;\dot{U};\dot{V}\right)\in\mathbb{R}^{2(n+m)\times r}$ has full rank
where $\theta=\mathtt{vec}(U;V)$ and
$\dot{\theta}=\mathtt{vec}(\dot{U};\dot{V})$. Then:

  1. 1.

if $2(n+m)\leq r$ and if $(n,m)\neq(1,1)$, then
$\mathrm{dim}\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})\left(\alpha\right)=(n+m)(2(n+m)+1)+1$;

  2. 2.

if $2(n+m)>r$ and if $(n,m)\neq(1,1)$, then
$\mathrm{dim}\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})\left(\alpha\right)=2(n+m)r+1-r(r-1)/2$;

  3. 3.

if $(n,m)=(1,1)$ and if $r\geq 4$, then
$\mathrm{dim}\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})\left(\alpha\right)=6+1$.

###### Proof.

Let us consider the linear applications:

$\Gamma:M\in\mathcal{S}\mapsto(\mathrm{Id}_{r}\otimes(HM))\begin{pmatrix}\theta\\\
\dot{\theta}\end{pmatrix}=\mathtt{vec}\left(HM\begin{pmatrix}U\\\ V\\\
\dot{U}\\\
\dot{V}\end{pmatrix}\right)\quad\text{and}\quad\bar{\Gamma}:M\in\mathcal{S}\mapsto\begin{pmatrix}0\\\
\Gamma(M)\end{pmatrix}.$

By Proposition H.3, we have
$\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha)=\mathrm{span}\\{\eta_{\beta,M}(\alpha):\beta\in\mathbb{R},M\in\mathcal{S}\\}$.
By linearity of $(\beta,M)\mapsto\eta_{\beta,M}(\alpha)$ it follows that
$\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha)=\mathbb{R}\eta_{1,0}(\alpha)+\bar{\Gamma}(\mathcal{S})$.
Since the first coordinate of $\eta_{1,0}(\alpha)$ is nonzero, it does not
belong to $\bar{\Gamma}(\mathcal{S})$, hence $\mathrm{dim}\
\mathrm{Lie}(\mathcal{W}_{\phi}^{\mathtt{mom}})(\alpha)=\mathrm{dim}(\bar{\Gamma}(\mathcal{S}))+1=\mathrm{dim}(\Gamma(\mathcal{S}))+1=\operatorname{rank}(\Gamma)+1.$
By the rank–nullity theorem, we have: $\mathrm{dim}\ \operatorname{ker}\
(\Gamma)+\operatorname{rank}\ (\Gamma)=\mathrm{dim}\ \mathcal{S}.$ We now
distinguish two cases.

1st case: $2(n+m)\leq r$. Then as $H$ is invertible ($H^{-1}=-H$) and
$\left(U;V;\dot{U};\dot{V}\right)$ has full rank $2(n+m)$, $\Gamma$ is
injective and we obtain $\operatorname{rank}(\Gamma)=\mathrm{dim}\mathcal{S}$.
When $(n,m)\neq(1,1)$ we have $\mathcal{S}=\mathcal{S}_{2(n+m)}$ hence this
yields
$\operatorname{rank}(\Gamma)=\frac{2(n+m)[2(n+m)-1]}{2}=(n+m)[2(n+m)+1]$. In
the case $n=m=1$, the assumption $2(n+m)\leq r$ reads $r\geq 4$, and the
associated rank is equal to $6$.

2d case: $2(n+m)>r$. Since $H$ is invertible, $\operatorname{ker}(\Gamma)$ is
the set of matrices $M\in\mathcal{S}$ such that $M(U;V;\dot{U};\dot{V})=0$.
Denote $M_{i}$, $1\leq i\leq 2(n+m)$ the rows of such a matrix, so that
$M^{\top}=(M_{1};\cdots;M_{2(n+m)})$. Denoting $C_{j}$, $1\leq j\leq r$ the
columns of $\left(U;V;\dot{U};\dot{V}\right)$. and
$\mathcal{C}\coloneqq\underset{j=1,\cdots,r}{\mathrm{span}}C_{j}$, we observe
that since $\left(U;V;\dot{U};\dot{V}\right)$ has full rank $r=\min(2(m+n),r)$
the columns $C_{j}$ are linearly independent and $\mathrm{dim}\
\mathcal{C}=r$. Since $M\times\left(U;V;\dot{U};\dot{V}\right)=0$, we have
$\langle M_{i},C_{j}\rangle=0$ for all $1\leq i\leq 2(n+m)$ and $1\leq j\leq
r$, i.e., each $M_{i}\in\mathbb{R}^{2(n+m)}$ belongs to $\mathcal{C}^{\perp}$,
of dimension $\mathrm{dim}\ \mathcal{C}^{\perp}=2(n+m)-r$.

To determine $\mathrm{dim}\ \operatorname{ker}(\Gamma)$ we now count the
number of degrees of freedom to choose $M\in\mathcal{S}$ such that
$M_{i}\in\mathcal{C}^{\perp}$ for every $i$. We only treat the case
$(n,m)\neq(1,1)$, where $\mathcal{S}$ is simply the set of symmetric matrices
characterized by $M^{\top}=M$.

We first show the following lemma.

######  Lemma H.5.

The matrix $C\in\mathbb{R}^{2(n+m)\times r}$ has full rank $r$ if and only if
there exists a subset $T$ of $2(n+m)-r$ indices such that the horizontal
concatenation $\left(C,\mathrm{Id}_{T}\right)$ is invertible, where
$\mathrm{Id}_{T}\in\mathbb{R}^{2(n+m)\times(2(n+m)-r)}$ is the restriction of
the identity matrix to its columns indexed by $T$.

###### Proof.

The converse implication is clear. Let us show the direct one. By denoting
$e_{1},\cdots,e_{2(n+m)}$ the canonical basis in $\mathbb{R}^{2(n+m)}$, there
is $i_{1}$ such that $e_{i_{1}}$ is linearly independent from all $C_{j}$:
otherwise all $e_{i}$ would be spanned by $C_{1},\cdots,C_{r}$, i.e. we would
have ${\mathrm{span}}\\{e_{i}:1\leq i\leq 2(n+m)\\}\subseteq\mathcal{C}$ hence
$2(n+m)\leq r$, which contradicts our assumption. Similarly, by recursion,
after finding $i_{1},\cdots,i_{k}$ for some $k<2(n+m)-r$ such that
$i_{1},\cdots,i_{k}$ are linearly independent from $C_{1},\dots,C_{r}$ (so
that
$\tilde{\mathcal{C}}\coloneqq{\mathrm{span}}\\{\mathcal{C},e_{i_{l}}:1\leq
l\leq k\\}$ has dimension $r+k<2(n+m)$), there exists $i_{k+1}$ such that
$e_{i_{k+1}}$ is linearly independent from all $C_{j}$ and all
$e_{i_{1}},\cdots,e_{i_{k}}$. Stopping this construction when $k=2(n+m)-r$
yields $T:=\\{i_{1},\ldots,i_{k}\\}$. ∎

Consider the index set $T=\\{i_{1},\cdots,i_{2(n+m)-r}\\}$ given by Lemma H.5,
so that $\left(C,\mathrm{Id}_{T}\right)\in\mathbb{R}^{2(n+m)\times 2(n+m)}$ is
invertible.

We first build the column $M_{i_{1}}$, which can be chosen arbitrarily in
$\mathcal{C}^{\perp}$, a space of dimension $2(n+m)-r$. Then, the $i_{1}$-th
coordinate of $M_{i_{2}}$ is determined by $M_{i_{1}}$ (and equal to its
$i_{2}$-th one) as $M$ is a symmetric matrix, and its remaining $2(n+m)-1$
coordinates can be freely chosen provided that $M_{i_{2}}$ belongs to
$\mathcal{C}^{\perp}$. Thus, $M_{i_{2}}$ can be arbitrarily chosen in the
affine space of dimension $2(n+m)-r-1$ defined by

$\left(C,e_{i_{1}}\right)^{\top}M_{i_{2}}=\begin{pmatrix}0\\\ \cdots\\\ 0\\\
M_{i_{1}}[i_{2}]\end{pmatrix},$

where the matrix $\left(C,e_{i_{1}}\right)^{\top}\in\mathbb{R}^{(r+1)\times
2(n+m)}$ has full rank $r+1$ by construction. By recursion, after building $k$
columns $M_{i_{1}},\cdots,M_{i_{k}}$ with $k<2(n+m)-r$, the coordinates
indexed by $i_{1},\cdots,i_{k}$ of the column $M_{i_{k+1}}$ are determined by
$M_{i_{1}},\cdots,M_{i_{k}}$ to ensure that $M$ is a symmetric matrix, and the
remaining $2(n+m)-k$ coordinates must ensure that
$M_{i_{k+1}}\in\mathcal{C}^{\perp}$. Thus $M_{i_{k+1}}$ can be arbitrarily
chosen in the affine space of dimension $2(n+m)-r-k$ defined by

$\left(C,e_{i_{1}},\cdots,e_{i_{k}}\right)^{\top}M_{i_{k+1}}=\begin{pmatrix}0\\\
\cdots\\\ 0\\\ M_{i_{1}}[i_{k+1}]\\\ \cdots\\\
M_{i_{k}}[i_{k+1}]\end{pmatrix},$

where the matrix
$\left(C,e_{i_{1}},\cdots,e_{i_{k}}\right)^{\top}\in\mathbb{R}^{(r+k)\times
2(n+m)}$ has full rank $r+k$ by construction. Finally the dimension of
$\mathrm{ker}(\Gamma)$ is equal to:

$\sum_{i=0}^{2(n+m)-r}(2(n+m)-r-i)=(2(n+m)-r)(2(n+m)-r+1)/2.$

Eventually we obtain $\operatorname{rank}(\Gamma)=2(n+m)r-r(r-1)/2$. ∎

###  H.3 Some derivations.

##### Details on how we obtained (36).

Given $\beta,\Delta$ the matrix $M\coloneqq\begin{pmatrix}-S_{\Delta}&0\\\
0&\beta\mathrm{Id}_{(n+m)}\end{pmatrix}$ is symmetric and belongs to
$\mathcal{S}$ even when $n=m=1$, and