# Taming the Interacting Particle Langevin Algorithm
\- the superlinear case

Tim Johnston  School of Mathematics, University of Edinburgh, UK  Nikolaos
Makras  School of Mathematics, University of Edinburgh, UK  Sotirios Sabanis
School of Mathematics, University of Edinburgh, UK  The Alan Turing Institute,
UK  National Technical University of Athens, Greece

###### Abstract

Recent advances in stochastic optimization have yielded the interacting
particle Langevin algorithm (IPLA), which leverages the notion of interacting
particle systems (IPS) to efficiently sample from approximate posterior
densities. This becomes particularly crucial within the framework of
Expectation-Maximization (EM), where the E-step is computationally challenging
or even intractable. Although prior research has focused on scenarios
involving convex cases with gradients of log densities that grow at most
linearly, our work extends this framework to include polynomial growth. Taming
techniques are employed to produce an explicit discretization scheme that
yields a new class of stable, under such non-linearities, algorithms which are
called tamed interacting particle Langevin algorithms (tIPLA). We obtain non-
asymptotic convergence error estimates in Wasserstein-2 distance for the new
class under an optimal rate.

## 1 Introduction

The Expectation-Maximization (EM) algorithm is widely used for locating
maximizers of posterior distributions. Applications span, but are not confined
to, hyperparameter estimation, mixture models, hidden variable models, and
variational inference [5]. At its essence, each iteration of the EM algorithm
consists of two fundamental steps: the Expectation (E) and the Maximization
(M) step. The algorithm is defined by alternating iteratively between these
two steps. Given a data specification $p_{\theta}(x,y)$ parameterized by
$\theta$, where x represents the latent variable (interpreted as incomplete
data) and y the observed data, our aim is to find $\theta^{*}$ that maximizes
the marginal likelihood
$q_{\theta}(x)=\int_{\mathbb{R}^{d^{x}}}p_{\theta}(x,y)dy$.
When the integral in the E-step is computationally challenging or intractable,
Markov Chain Monte Carlo (MCMC) methods are often employed, traditionally
using Metropolis-Hastings-type algorithms [4]. However, these approaches
introduce scalability concerns and are susceptible to local mode entrapment.
In this landscape, a new method was introduced by [2], such that samples of
the latent space variable were generated via an Unadjusted Langevin Algorithm
(ULA) chain. Nevertheless, the general applicability of these results is
curtailed by particular choices of step sizes. An alternative avenue was
pioneered in [12], in which the study of the limiting behaviour of various
gradient flows associated with appropriate free energy functionals led to an
interacting particle system (IPS) that provides efficient estimates for
maximum likelihood estimations. Further, [1] thoughtfully expanded on this
framework by injecting noise into the dynamics of the parameter $\theta$
itself, thereby transitioning from deterministic to stochastic dynamics. This
key modification provides a stochastic system with an invariant measure,
allowing the establishment of non-asymptotic convergence to $\theta^{*}$, the
maximizer of the marginal likelihood.
In our work, while the convexity assumption is maintained, we address the
challenge posed by superlinear growth exhibited by gradients of log densities,
which makes other known algorithms, such as vanilla Langevin based algorithms,
unstable. To counteract this, we implement taming techniques, initially
researched for non-globally Lipschitz drifts for SDEs in [10] and subsequently
in [18] and [19]. The latter approach has found applications in optimization
and machine learning and led to the design of new MCMC algorithms as one
typically deals with high nonlinear objective function, see e.g.
[3],[16],[14],[20]. The underlying principle of all these algorithms is the
rescaling of ULA’s drift coefficient in such a way that maintains stability
without significantly increasing computational complexity as in the case of
implicit schemes, or by introducing additional constrains via adaptive
stepsizes.
In this paper, we study two new algorithms from the tIPLA class, namely the
coordinate wise version, known as tIPLAc, and the uniformly tamed version
tIPLAu. Those algorithms are tamed versions of IPLA (developed in [1]) as
explained in Subsection 2.2. The optimal rate of convergence is recovered for
both algorithms and our estimates are explicit regarding their dependencies on
dimension and the number of particles employed $N$.

### 1.1 Notation

We conclude this subsection with some basic notation. For
$u,v\in\mathbb{R}^{d}$, define the scalar product $\langle
u,v\rangle=\sum_{i=1}^{d}u_{i}v_{i}$ and the Euclidian norm $|u|={\langle
u,u\rangle}^{1/2}$. For all continuously differentiable functions $f$ and we
denote by $\nabla f$ it’s gradient. The integer part of a real number x is
denoted by $\lfloor x\rfloor$. For any $p\in\mathbb{N}$, we denote by
$\mathcal{P}(\mathbb{R}^{d})$ the set of probability measures on
$\mathcal{B}(\mathbb{R}^{d})$ and by
$\mathcal{P}_{p}(\mathbb{R}^{d})=\\{\pi\in\mathcal{P}:\int_{\mathbb{R}^{d}}|x|^{p}_{p}d\pi(x)<\infty\\}$
the set of all probability measures over $\mathcal{B}(\mathbb{R}^{d})$ with
finite $p$-th moment. For any two Borel probability measures $\mu$ and $\nu$,
we define the Wasserstein distance of order $p\geq 1$ as

$W_{p}(\mu,\nu)=\left(\inf_{\zeta\in\prod(\mu,\nu)}\int_{\mathbb{R}^{d}\times\mathbb{R}^{d}}|x-y|^{p}d\zeta(x,y)\right)^{1/p},$

where by $\prod(\mu,\nu)$ we denote the set of transference plans of $\mu$ and
$\nu$. Moreover, for all $\mu,\ \nu\in\mathcal{P}_{p}(\mathbb{R}^{d})$, there
exists a transference plan $\zeta^{*}\in\prod(\mu,\nu)$ such that for any
coupling $(X,Y)$ distributed according to $\zeta^{*}$,
$W_{p}(\mu,\nu)=\mathbb{E}^{1/p}\left[\left|X-Y\right|^{p}\right]$.

## 2 Setting and Definitions

### 2.1 Initial setup

Let $p_{\theta}(x,\cdot)$ be the aforementioned joint probability density
function of the latent variable $x$ and for fixed (observed) data $y$. The
goal of maximum marginal likelihood estimation (MMLE) is to find the parameter
$\theta^{*}$ that maximises the marginal likelihood (Dempster et al.)[5]. To
deal with the aforementioned log-density we define the negative log-likelihood
for fixed $y\in\mathbb{R}^{d^{y}}$ as follows

$U_{y}(\theta,x):=-\log p_{\theta}(x,y),$

thus gaining the following notation for the quantity we are interested in
maximizing $k_{y}(\theta)=q_{\theta}(y)=\int p_{\theta}(x,y)dx=\int
e^{-U_{y}(\theta,x)}dx$. Lastly, for matters of clarity we also denote
$h_{y}(v)=\nabla U_{y}(v)$ so that $h^{x}_{y}(v)=\nabla_{x}U_{y}(v)$ and
$h^{\theta}_{y}(v)=\nabla_{\theta}U_{y}(v)$, where $v:=(\theta,x)$.
Henceforth, we drop the reference to the (fixed) data, i.e. to $y$, for
reasons of brevity. Following the convention of [12], $N$ particles
$\mathcal{X}_{t}^{i,N}$ for $i\in\\{1,\ldots,N\\}$ are used to estimate the
gradient of $q_{\theta}$, which are governed by following continuous-time
dynamics:

$\displaystyle
d\vartheta_{t}^{N}=-\dfrac{1}{N}\sum_{j=1}^{N}\nabla_{\theta}U(\vartheta_{t}^{N},\mathcal{X}^{j,N}_{t})+\sqrt{\dfrac{2}{N}}dB_{t}^{0,N},$
(1) $\displaystyle
d\mathcal{X}_{t}^{i,N}=-\nabla_{x}U(\vartheta_{t}^{N},\mathcal{X}^{i,N}_{t})+\sqrt{2}dB_{t}^{i,N},$
(2)

$\text{for }i=1,..,N$, where $\\{(B_{t})^{i,N}_{t\geq 0}\\}_{0\leq i\leq N}$
is a family of independent Brownian motions. The discrete time Markov chain
associated with the above IPS (1)-(2) is obtained by the corresponding Euler-
Maruyama discretization scheme of the given Langevin SDEs:

$\displaystyle\theta^{\lambda}_{0}=$ $\displaystyle\theta_{0},\
\theta_{n+1}^{\lambda}=\theta_{n}^{\lambda}-\dfrac{\lambda}{N}\sum_{i=1}^{N}h^{\theta}(\theta_{n}^{\lambda},X_{n}^{i,\lambda})+\sqrt{\dfrac{2\lambda}{N}}\xi^{(0)}_{n+1},$
$\displaystyle X^{i,\lambda}_{0}=$ $\displaystyle x^{i}_{0},\
X_{n+1}^{i,\lambda}=X_{n}^{i,\lambda}-\lambda
h^{x}(\theta_{n}^{\lambda},X_{n}^{i,\lambda})+\sqrt{2\lambda}\xi^{(i)}_{n+1},\
\forall i\in\\{1,...,N\\},$

where ${\theta}_{0}\in\mathbb{R}^{d^{\theta}}$,
$x^{i}_{0}\in\mathbb{R}^{d^{x}}$, $\lambda>0$ the step-size parameter and for
$i=0$ and $\forall i\in\\{1,\ldots,N\\}$$,(\xi^{(i)}_{n})_{n\in\mathbb{N}}$
are i.i.d. standard $d^{\theta}$ and $d^{x}$ respectively dimensional Gaussian
variables. For reasons that become apparent after the introduction of the
taming functions in Subsection 2.2.2 we also consider the following dynamics
which are a different time-scaled version of the original equations (1)-(2):

$\displaystyle d\vartheta_{t}^{N}=$
$\displaystyle-\dfrac{1}{N^{p+1}}\sum_{j=1}^{N}\nabla_{\theta}U(\vartheta_{t}^{N},\mathcal{X}^{j,N}_{t})+\sqrt{\dfrac{2}{N^{p+1}}}dB_{t}^{0,N},$
(3) $\displaystyle d\mathcal{X}_{t}^{i,N}=$
$\displaystyle-\dfrac{1}{N^{p}}\nabla_{x}U(\vartheta_{t}^{N},\mathcal{X}^{i,N}_{t})+\sqrt{\dfrac{2}{N^{p}}}dB_{t}^{i,N},$
(4)

where $p=2\ell+1$ is controlled by the order of polynomial growth $\ell$ in
$\nabla U(\theta,x)$, see Remark 1. Such an adjustment allows the
generalization of the results in [1] to the case of superlinear drift
coefficients while keeping the underlying stationary distribution of the
dynamics identical.

### 2.2 Taming approach

####  2.2.1 Introduction of the underlying taming technique

For a fixed $T>0$, consider an SDE given by

$\displaystyle dY(t)=b(t,Y(t))dt+\sigma(t,Y(t))dB_{t},\ \forall t\in[0,T],$
(5)

where $b(t,y)$ and $\sigma(t,y)$ are assumed to be
$\mathcal{B}(\mathbb{R}_{+})\times\mathcal{B}(\mathbb{R}^{d})-$measurable
functions. Then, the discrete time Markov chain associated with the ULA
algorithm is obtained by the Euler-Maruyama discretization scheme of SDE(5)
and is defined for all $n\in\mathbb{N}$ by:

$Y(t_{n+1})=Y(t_{n})-\lambda
b(t_{n},Y(t_{n}))+\sqrt{\lambda}\sigma(t_{n},Y(t_{n}))\xi_{n+1},$

where $\lambda>0$ is the stepsize and $\left(\xi_{n}\right)_{n\geq 1}$ are
i.i.d. standard Gaussian random variables. In the case where the drift
coefficient $b$ is superlinear, it is shown in [9] that ULA is unstable in the
sense that any $p-$absolute moment of the algorithm $(p\geq 1)$ diverges to
infinity. In the SDE approximation literature, a new class of explicit
numerical schemes has been introduced to study the case of non-globally
Lipschitz conditions by modifying both the drift and diffusion coefficients in
such a way that they grow at most linearly, for example see, [19], [21] and
[11]. The efficiency of such schemes and their respective properties of
$\mathcal{L}^{p}$ convergence create a strong incentive to extend those
techniques to sampling and optimization. This adjustment is key to the
development of algorithms that approximate non-linear systems while remaining
computationally tractable. Typically tamed schemes are given by,

$Y(t_{n+1})=Y(t_{n})-\lambda
b_{\lambda}(t_{n},Y(t_{n}))+\sqrt{\lambda}\sigma_{\lambda}(t_{n},Y(t_{n}))\xi_{n+1},$

for an appropriate choice of taming functions
$b_{\lambda}:\mathbb{R}^{d}\mapsto\mathbb{R}^{d}$ and
${\sigma}_{\lambda}:\mathbb{R}^{d}\mapsto\mathbb{R}^{d\times d}$.

####  2.2.2 Application in the IPLA framework

We extend this notion of taming to our setting when dealing with the
superlinear nature of $\nabla U$ and a constant diffusion coefficient. To this
end, we introduce a family of a taming functions $(h_{\lambda})_{\lambda\geq
0}$ with
$h_{\lambda}:\mathbb{R}^{d^{\theta}}\times\mathbb{R}^{d^{x}}\to\mathbb{R}^{d^{\theta}}\times\mathbb{R}^{d^{x}}$
which are close approximations of $\nabla U$ in a sense made precise below.
We suggest two such taming functions $h_{\lambda}(v)$, one of which is
uniformly tamed and another by using a coordinate-wise approach:

$\displaystyle h_{\lambda,u}(v)=\dfrac{h(v)-\mu
v}{1+{\lambda}^{1/2}N^{-p/2}|h(v)-\mu v|}+\mu v,$ (6) $\displaystyle
h_{\lambda,c}(v)=\left(\dfrac{h^{(i)}(v)-\mu
v^{(i)}}{1+{\lambda}^{1/2}|h^{(i)}(v)-\mu v^{(i)}|}+\mu
v^{(i)}\right)_{i\in\\{1,\ldots,d^{\theta}+d^{x}\\}},$ (7)

where $\mu$ is the strong convexity constant given in A2. In [3] it’s
experimentally established that the coordinate-wise version outperforms the
uniform taming approach. This is in agreement with the observation that
uniform taming cannot distinguish between the different levels of contribution
each coordinate offers to the gradient. However, there is a trafeoff for using
the coordinate-wise approach, as in order to obtain an appropriate
dissipativity condition for the tamed function $h_{\lambda}$, one needs to
require additional smoothness on the potential $U$.
We propose two different algorithms for the EM implementation within the tIPLA
class, which are determined by the choice of the taming function and the
stochastic dynamics. We find that when the uniform taming is used, then the
time-scaled dynamics (3)-(4) are more suitable as the addition of the exponent
in the number of particles $N$ is essential to guarantee convergence. Hence
(3)-(4) paired with (6) leads to the following scheme:

$\displaystyle\theta^{\lambda,u}_{0}=\theta_{0},\
\theta_{n+1}^{\lambda,u}=\theta_{n}^{\lambda,u}-\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h_{\lambda,u}^{\theta}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})+\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi^{(0)}_{n+1},$
(8) $\displaystyle X^{i,\lambda,u}_{0}=x^{i}_{0},\
X_{n+1}^{i,\lambda,u}=X_{n}^{i,\lambda,u}-\dfrac{\lambda}{N^{p}}h_{\lambda,u}^{x}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})+\sqrt{\dfrac{2\lambda}{N^{p}}}\xi^{(i)}_{n+1},\
\forall i\in\\{1,...,N\\}.$ (9)

In the case of the coordinate-wise taming, the extra smoothness required
obviates any modification to the original dynamics. Therefore one uses (1)-(2)
with (7) to obtain the second algorithm:

$\displaystyle\theta^{\lambda,c}_{0}=\theta_{0},\
\theta_{n+1}^{\lambda,c}=\theta_{n}^{\lambda,c}-\dfrac{\lambda}{N}\sum_{i=1}^{N}h_{\lambda,c}^{\theta}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})+\sqrt{\dfrac{2\lambda}{N}}\xi^{(0)}_{n+1},$
(10) $\displaystyle
X^{i,\lambda,c}_{0}=x^{i}_{0},X_{n+1}^{i,\lambda,c}=X_{n}^{i,\lambda,c}-\lambda
h_{\lambda,c}^{x}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})+\sqrt{2\lambda}\xi^{(i)}_{n+1},\
\forall i\in\\{1,...,N\\}.$ (11)

It is important to note that despite the differences in their setup, both
algorithms yield the same non-asymptotic convergence behavior and dependence
on the dimension of the E optimization challenge. Both schemes are summarised
in the algorithms (tIPLAu) and (tIPLAc) given below.

Algorithm 1 Tamed Interacting Particle Langevin Algorithm (tIPLAu)

Require:
N,$\lambda$,$\pi_{\text{init}}\in\mathcal{P}(\mathbb{R}^{d^{\theta}})\times\mathcal{P}((\mathbb{R}^{d^{x}})^{N})$

Draw $(\theta_{0},\\{X_{0}^{i,N}\\}_{1\leq i\leq N})$ from $\pi_{\text{init}}$

for n = $0$ to $n_{T}=\lfloor T/\lambda\rfloor$ do

$\theta_{n+1}^{\lambda,u}=\theta_{n}^{\lambda,u}-\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})+\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi^{(0)}_{n+1}$

$X_{n+1}^{i,\lambda,u}=X_{n}^{i,\lambda,u}-\dfrac{\lambda}{N^{p}}h^{x}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})+\sqrt{\dfrac{2\lambda}{N^{p}}}\xi^{(i)}_{n+1},\
\forall i\in{1,...,N}$

end for

return $\theta_{n_{T}+1}$

It is apparent that the newly proposed algorithms, designated as tIPLAu and
tIPLAc, incorporate foundational elements that exist in [12], which employed
deterministic dynamics in the $\theta$ component. Their design profits also
from the approach in [1] which introduced stochastic noise in the
$\theta-$direction, allowing for the explicit calculation of the invariant
measure.

Algorithm 2 Tamed Interacting Particle Langevin Algorithm - Coordinatewise
(tIPLAc)

Require:
N,$\lambda$,$\pi_{\text{init}}\in\mathcal{P}(\mathbb{R}^{d^{\theta}})\times\mathcal{P}((\mathbb{R}^{d^{x}})^{N})$

Draw $(\theta_{0},\\{X_{0}^{i,N}\\}_{1\leq i\leq N})$ from $\pi_{\text{init}}$

for n = $0$ to $n_{T}=\lfloor T/\lambda\rfloor$ do

$\theta_{n+1}^{\lambda,c}=\theta_{n}^{\lambda,c}-\dfrac{\lambda}{N}\sum_{i=1}^{N}h^{\theta}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})+\sqrt{\dfrac{2\lambda}{N}}\xi^{(0)}_{n+1}$

$X_{n+1}^{i,\lambda,c}=X_{n}^{i,\lambda,c}-\lambda
h^{x}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})+\sqrt{2\lambda}\xi^{(i)}_{n+1},\
\forall i\in{1,...,N}$

end for

return $\theta_{n_{T}+1}$

A pivotal enhancement that comes with this approach involves the application
of a taming technique via the recalibration of the original drift terms,
$h_{x}$ and $h_{\theta}$, into tamed counterparts, as described in (6) and
(7). This adjustment is instrumental in deriving nonasymptotic results,
particularly in cases characterized by superlinear gradients in the log-
likelihood function.

### 2.3 Essential quantities and proof strategy of main result

In this subsection, we lay out the analytical framework and convergence
properties of the sequences $(\theta_{n}^{\lambda,\cdot})_{n\geq 0}$ of either
tIPLAu $(\theta_{n}^{\lambda,u})_{n\geq 0}$ or tIPLAc
$(\theta_{n}^{\lambda,c})_{n\geq 0}$, resulting in their convergence to the
minimiser $\theta^{*}$. A central concept of the technical analysis is the
rescaling of the original dynamics of the following form

$\displaystyle\mathcal{Z}_{t}^{N}=\left(\vartheta_{t}^{N},N^{-1/q}\mathcal{X}_{t}^{1,N},\ldots,N^{-1/q}\mathcal{X}_{t}^{N,N}\right),$
(12)

along with the corresponding rescaling that is required for the algorithms and
their continuous time interpolations, see more details in Appendix A. This
rescaled version of the particle system is pivotal in bridging the dynamics
represented by either (1)-(2) or (3)-(4) with the pairs
$\mathcal{V}_{t}^{i,N}:=(\vartheta_{t}^{N},\mathcal{X}_{t}^{i,N})$ generated
by the algorithms. The essence of this rescaling lies in its ability to equate
the moments of $\mathcal{Z}_{t}^{N}$ with the averaged moments of the pairs
$\mathcal{V}_{t}^{i,N}$ across any $\mathbb{L}^{q}$ norm. However, given our
focus on Wasserstein-2 distance, we opt for a rescaling factor of $q=2$. This
choice yields the critical property:

$|\mathcal{Z}_{t}|^{2}=\dfrac{1}{N}\sum_{i=1}^{N}|\mathcal{V}_{t}^{i}|^{2}.$

It is imperative to note that this convenient rescaling does not detract from
the analysis objectives, as our primary interest is in the convergence of the
$\theta-$component. The introduction of particles serves primarily to
facilitate sampling, hence their specific scaling does not impact the core
convergence analysis.
In the present analysis, the interacting particle systems defined in equations
(1)-(2) and (3)-(4) are deliberately constructed to target the same invariant
measure $\pi_{*}^{N}$ characterized by the density
$Z^{-1}\exp(-\sum_{i=1}^{N}U(\theta,x^{i}))$. This aspect is critical as it
assigns the number of particles $N$ a role akin to the inverse temperature
parameter encountered in simulated annealing algorithms, thereby regulating
the concentration of the $\theta-$marginal of the invariant measure
$\pi_{\Theta}^{N}$, towards the minimizer $\theta^{*}$. This behavior, wherein
$\pi_{\Theta}^{N}\to\delta_{\theta^{*}}$ as $N\to\infty$, is established in
Proposition 2.
Moreover, the convergence rate of the Langevin diffusion to its invariant
measure is a standard result, one could consult [7] and the references within.
This is further generalized to particle systems in the research conducted by
[1]. By integrating their findings into our framework, we arrive at
Proposition 3 and 4. Equipped with these guarantees, we can now theoretically
approximate the minimizer $\theta^{*}$, and subsequently, our analysis focuses
to the discretization errors inherent in the proposed schemes.
One expects a dimensional scaling of at least $\mathcal{O}(d^{1/2})$ for the
Wasserstein-2 numerical error of Langevin based algorithms, for example [6].
In the context of the E optimization where $N$ particles are generated, the
true dimentionality of the problem is $d^{\theta}+N\cdot d^{x}$, which can be
naively interpreted as the need to increase the number of iterations to
achieve convergence as $N$ increases. However, this dependency on $N$ is
effectively mitigated by making use of the implied symmetry within the
dynamics of the interacting particle system by considering only the
$\theta$-marginal in the analysis. Such an approach enables us to attain the
classical Euler-Maruyama convergence rate for the numerical solutions of
stochastic differential equations, as demonstrated in Proposition 5 and 6.
In closing, one decomposes the global $\mathcal{L}^{2}$-error, denoted as
$\mathbb{E}^{1/2}\left[|{\theta}^{*}-{\theta}^{\lambda,\cdot}_{n}|^{2}\right]$,
into three distinct components:

$\mathbb{E}^{1/2}\left[|{\theta}^{*}-{\theta}^{\lambda,\cdot}_{n}|^{2}\right]\leq
W_{2}(\delta_{\theta^{*}},\pi_{\Theta}^{N})+W_{2}(\pi_{\Theta}^{N},\mathcal{L}(\vartheta^{N}_{n\lambda}))+W_{2}(\mathcal{L}(\vartheta^{N}_{n\lambda}),\mathcal{L}(\theta_{n}^{\lambda,\cdot})),$

where ${\theta}^{\lambda,\cdot}_{n}$ stands for the iterates of either of the
algorithms, tIPLAu and tIPLAc alike. Moreover,
$W_{2}(\delta_{\theta^{*}},\pi_{\Theta}^{N})$ quantifies the deviation of the
invariant measure from the minimizer,
$W_{2}(\pi_{\Theta}^{N},\mathcal{L}(\vartheta^{N}_{n\lambda}))$ captures the
discrepancy between the law of the dynamics and their invariant measure, and
$W_{2}(\mathcal{L}(\vartheta^{N}_{n\lambda}),\mathcal{L}(\theta_{n}^{\lambda,\cdot}))$
encompasses the error induced by the discretization scheme, namely, the
divergence between the law of the iterations of our algorithms and their
continuous counterparts.

## 3 Main Assumptions and Results

In this section we provide the assumptions on the potential $U$ and it’s
gradient $\nabla U$ that define the framework, under which the main results
for the non-asymptotic behaviour of the newly proposed algorithms, tIPLAu and
tIPLAc, are derived.

Let $d:=d^{\theta}+d^{x}$, $U$ be a $C^{1}\left(\mathbb{R}^{d}\right)$
function and recall that $h(v):=h(\theta,x):=\nabla U(\theta,x)$ is a locally
Lipschitz function in $\theta$ and $x$.

A1. There exist $L>0$ and $\ell>0$ such that

$|h(v)-h(v^{\prime})|\leq
L\left(1+|v|^{\ell}+|v^{\prime}|^{\ell}\right)|v-v^{\prime}|,\ \forall
v,v^{\prime}\in\mathbb{R}^{d}.$

Additionally we require $U$ to be $\mu-$strongly convex.

A2. There exists $\mu>0$ such that

$\left\langle
v-v^{\prime},h(v)-h(v^{\prime})\right\rangle\geq\mu|v-v^{\prime}|^{2},\
\forall v,v^{\prime}\in\mathbb{R}^{d}.$

Assumption A1 is a significant relaxation of the global Lipschitz condition
which is widely used in the literature. It allows the gradient $h$ to grow
polynomially fast at infinity. Assumption A2 guarantees that $U$ has a unique
minimiser and also can be seen as a monotonicity type condition which is
satisfied by the drift coefficients of the corresponding SDEs.
We also present a growth condition for the gradient $\nabla U$, which is
essential in the case where the coordinate-wise taming function (7) is used.

A3. For each $i$ in $\\{1,\ldots,d\\}$ there exists $\mu>0$ such that

$h^{(i)}(v)v^{(i)}\geq\dfrac{\mu}{2}|v^{(i)}|^{2}-\dfrac{1}{2\mu}|h^{(i)}_{0}|^{2}.$

Assumption A3 is a coordinate-wise dissipativity type condition which plays a
crucial role in establishing moment bounds for tIPLAc. Its need stems from the
fact that Remark 2 does not guarantee that the taming function (7) preservers
the dissipativity of the drift $h(v)=\nabla U(v)$.
Here, we emphasize that Assumption A3 is primarily influenced by the
structural properties of the taming function. This assumption could
potentially be relaxed to necessitate only argument-wise dissipativity, as
described by the following mathematical formulation:

$h^{\theta}(v)\theta\geq\dfrac{\mu}{2}|\theta|^{2}-\dfrac{1}{2\mu}|h^{\theta}_{0}|^{2}\text{
and }h^{x}(v)x\geq\dfrac{\mu}{2}|x|^{2}-\dfrac{1}{2\mu}|h^{x}_{0}|^{2}.$

This relaxation is applicable if one considers the taming function

$h_{\lambda,a}^{w}(v)=\dfrac{h^{w}(v)-\mu w}{1+{\lambda}^{1/2}|h^{w}(v)-\mu
w|}+\mu w\ \text{for }w=\theta,\ x,$

which represents an intermediate scenario bridging the uniform and coordinate-
wise approaches. Such consideration leads to an alternative formulation of the
tIPLA algorithm. However, this variant of the tIPLA algorithm does not capture
our interest, primarily due to its inherent limitations. Specifically, it
fails to adequately discriminate between the components of the gradient that
exhibit explosive behavior and those that do not. While simultaneously, this
alternative formulation imposes more stringent requirements than those
necessitated by the uniform case.

A4. The initial condition
$z_{0}^{N}=\left(\theta_{0},N^{-1/2}x_{0}^{1},\ldots,N^{-1/2}x_{0}^{N}\right)$
is such that

$\mathbb{E}\left[|z_{0}^{N}|^{2p_{0}}\right]<\infty,$

where $p_{0}=2(\ell+1)$ and $\ell$ being the order of polynomial growth of
$\nabla U(\theta,x)$ as given in Assumption A1.

Remark 1. One notices that Assumption A1 allows for control of the growth of
$h(v)=\nabla U(v)$, i.e., for every $v\in\mathbb{R}^{d}$,

$h(v)\leq K(1+|v|^{\ell+1}),$

where $K=2L+|h_{0}|$.

Remark 2.In view of Assumption A2, strong convexity implies dissipativity,
i.e, for every $v\in\mathbb{R}^{d}$,

$\langle v,h(v)\rangle\geq\dfrac{\mu}{2}|v|^{2}-b,$

where $b=\dfrac{1}{2\mu}|h_{0}|^{2}$.

### 3.1 Taming functions and inherited properties

Important properties of the suggested taming functions, which act as drift
coefficients in the corresponding numerical schemes, are established in this
subsection.

Property 1. For all $\lambda>0$ and $v\in\mathbb{R}^{d}$ one has

$|h_{\lambda,u}(v)|\leq\mu|v|+{\lambda}^{-1/2}N^{p/2}.$

The original and tamed functions are sufficiently close for $\lambda>0$ small
enough.

Property 2.For all $\lambda>0$ and $v\in\mathbb{R}^{d}$ one has

$|h_{\lambda,u}(v)-h(v)|\leq C_{1}{\lambda}^{1/2}N^{-p/2}(1+|v|^{2(\ell+1)}),$

where $C_{1}=2^{2(\ell+3/2)}\max\\{K^{2},\mu^{2}\\}$.

The tamed function $h_{\lambda,u}(v)$ inherits the dissipativity condition
established in Remark 2.

Property 3. For all $\lambda>0$ and $v\in\mathbb{R}^{d}$ one has

$\langle v,h_{\lambda,u}(v)\rangle\geq\dfrac{\mu}{2}|v|^{2}-b,$

where $b$ is given in Remark 2. To see this consider the cases in which
$\langle h(v)-\mu v,v\rangle$ is greater or less than $0$ separately.
In the context of unified framework established in [15], the properties 1-3
therein are recovered for $\delta=2$ and $\gamma=1/2$ which is on par with
TH$\epsilon$O POULA [13] and TUSLA [16] aglorithms.

Comment. Regarding tIPLAc, where the coordinate-wise taming function (7) is
used, Properties 1-2 are also guaranteed without the $N$-factor and also a
slightly worse coefficient in Property 1, that is:
$|h_{\lambda,c}(v)|\leq\mu|v|+(d^{\theta}+d^{x}){\lambda}^{-1/2}$. However for
Property 3 to be recovered, we require the additional smoothness provided by
Assumption A3.

### 3.2 Preliminary Theoretical Statements

Both interacting SDEs defined by (1)-(2) and (3)-(4) respectively, admit a
strong solution under A1-A2. This follows since each of the drift coefficients
of the SDEs are locally Lipschitz functions and dissipative in view of Remark
2 whereas the diffusion coefficients are constant, one could consult
[17](Theorem 2.3.5) for more details. Moreover, they exhibit the same
invariant measure, as one would expect from a standard Langevin diffusion.

Proposition 1. Let A1 and A2 hold. Then, the measure $\pi_{*}^{N}$
characterized by the density $Z^{-1}\exp^{-\sum_{i=1}^{N}U(\theta,x^{i})}$,
with $Z$ being the normalizing constant, is the invariant measure for both the
interacting particles systems (1)-(2) and (3)-(4).

###### Proof.

The existence of an invariant measure is established by invoking the Krylov-
Bogoliubov Theorem, as presented in Theorem 7.1 [8]. This theorem guarantees
the existence of an invariant measure for a Markov process generated by it’s
corresponding semigroup, given a tightness condition on the sequence of
probability measured associated with that process. According to Da Prato’s
Proposition 7.10 [8], the establishment of uniform moment bounds serves as a
sufficient condition for tightness, which is provided in Lemma 1. The
uniqueness of the invariant measure follows from Theorem 7.16(ii) [8], as
Hypothesis 7.13 is satisfied due to A1 and A2. Furthermore, one can verify
that $\pi_{*}^{N}$ is indeed the invariant measure in discussion by checking
that for any $\phi\in C^{\infty}_{c}(\mathbb{R}^{d})$ one has
$\int_{\mathbb{R}^{d}}\mathcal{L}\phi\ d\pi_{*}^{N}=0$, where $\mathcal{L}$ is
the infinitesimal generator of the process. See the proof of Proposition 2 in
[1] for more details. ∎

Henceforward we consider exclusively the $\theta-$marginal of the invariant
measure and we show that indeed the number of particles $N$ plays the role of
the inverse temperature parameter in temperature annealing algorithms in the
sense that it grants us control on the marginal’s concentration around the
minimiser.

Proposition 2. Let $\pi_{\Theta}^{N}$ denote the $\theta-$marginal of the
invariant measure and $\theta^{*}$ as the maximiser of $k(\theta)$. Then,
under A2, for any $N\in\mathbb{N}$, one has the bound

$W_{2}(\pi_{\Theta}^{N},\delta_{\theta^{*}})\leq\sqrt{\dfrac{2d^{\theta}}{\mu
N}}.$

###### Proof.

Follows from Proposition 3 in [1]. ∎

Lastly we need the following ergodicity result to obtain the convergence of
the law of the IPS to the invariant measure.

Proposition 3. Let A1, A2 and A4 hold and consider the first component, namely
$\vartheta^{N}_{t}$, of the continuous dynamics (3)-(4). Then, for any
$N\in\mathbb{N}$,

$W_{2}\left(\mathcal{L}(\vartheta^{N}_{t}),\pi_{\Theta}^{N}\right)\leq e^{-\mu
t}\left(\mathbb{E}^{1/2}\left[|z_{0}^{N}-z^{*}|^{2}\right]+\left(\dfrac{d^{x}N+d^{\theta}}{\mu
N^{p}}\right)^{1/2}\right).$

###### Proof.

Follows closely from the proof of Proposition 4 in [1]. One can easily verify
that although the corresponding A1 in [1] invokes global Lipschitz continuity
rather than local, its need there is solely to ensure the existence of a
unique global solution, a fact that is already established at the start of
Subsection 3.2. A further difference is the additional factor $N^{-p}$ instead
of $N^{-1}$ which appears in the last term of the upper bound. ∎

Proposition 4. Let A1, A3 and A4 hold and consider the first component, namely
$\vartheta^{N}_{t}$, of the continuous dynamics (1)-(2). Then, for any
$N\in\mathbb{N}$,

$W_{2}\left(\mathcal{L}(\vartheta^{N}_{t}),\pi_{\Theta}^{N}\right)\leq e^{-\mu
t}\left(\mathbb{E}^{1/2}\left[|z_{0}^{N}-z^{*}|^{2}\right]+\left(\dfrac{d^{x}N+d^{\theta}}{\mu
N}\right)^{1/2}\right).$

###### Proof.

Follows also closely from the proof of Proposition 4 in [1] with the note that
A3 implies A2. ∎

### 3.3 Discretisation Error Estimates

We consider the continuous interpolation of the algorithm tIPLAu which
approximates the time-scaled version of the continuous dynamics (3)-(4)
denoted as $\overline{{Z}}_{t}^{\lambda,u}$. We note that the law of the
discretized process and it’s interpolation coincide at grid-points, i.e.
$\mathcal{L}({Z}_{n}^{\lambda,u})=\mathcal{L}(\overline{{Z}}_{n}^{\lambda,u})$.
The continuous-time interpolation of the tIPLAu can be defined as:

$\displaystyle\overline{\theta}_{0}^{\lambda,u}=$ $\displaystyle\theta_{0},\
d\overline{\theta}_{t}^{\lambda,u}=-\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{\lfloor{t}\rfloor}^{\lambda,u},X^{i,\lambda,u}_{\lfloor{t}\rfloor})dt+\sqrt{\dfrac{{2\lambda}}{N^{p+1}}}dB_{t}^{0,\lambda},$
(13) $\displaystyle\overline{X}_{0}^{i,\lambda,u}=$ $\displaystyle x^{i}_{0},\
d\overline{X}_{t}^{i,\lambda,u}=-\dfrac{\lambda}{N^{p}}h^{x}_{\lambda,u}(\theta_{\lfloor{t}\rfloor}^{\lambda,u},X^{i,\lambda,u}_{\lfloor{t}\rfloor})dt+\sqrt{\dfrac{{2\lambda}}{N^{p}}}dB_{t}^{i,\lambda},$
(14)

for all $i\in\\{1,\ldots,N\\}$, while the time-changed SDEs (3)-(4)are given
by:

$\displaystyle d{\vartheta}_{\lambda
t}^{N}=-\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}({\vartheta}^{N}_{\lambda
t},\mathcal{X}_{\lambda
t}^{i,N})dt+\sqrt{\dfrac{2\lambda}{N^{p+1}}}dB_{t}^{0,\lambda},$ (15)
$\displaystyle d\mathcal{X}_{\lambda
t}^{i,N}=-\dfrac{\lambda}{N^{p}}h^{x}({\vartheta}^{N}_{\lambda
t},\mathcal{X}_{\lambda
t}^{i,N})dt+\sqrt{\dfrac{2\lambda}{N^{p}}}dB_{t}^{i,\lambda},$ (16)

where in both cases $B_{t}^{\lambda}:=B_{\lambda t}/\sqrt{\lambda},t\geq 0,$
is a Brownian motion under its completed natural filtration
$\mathcal{F}^{\lambda}_{t}:=\mathcal{F}_{\lambda t}.$

Proposition 5. Let A1, A2 and A4 hold and consider the process
${\vartheta}_{\lambda t}^{N}$ as given by the dynamics (15)-(16). Then, for
every $\lambda_{0}<N^{2\ell+1}/4\mu$, there exists a constant $C>0$
independent of $N,n,\lambda$ such that for any $\lambda\in(0,\lambda_{0})$,

$\mathbb{E}^{1/2}\left[|\overline{\theta}^{\lambda,u}_{n}-\vartheta^{N}_{n\lambda}|^{2}\right]\leq\lambda^{1/2}C_{|z_{0}|,\mu,b,\ell,L}{(1+d^{\theta}/N+d^{x})}^{\ell+1},$

for all $n\in\mathbb{N}$.

###### Proof.

The proof is postponed to Appendix A. ∎

Following the same lines we define the corresponding auxiliary processes for
the iterates of tIPLAc. The continuous-time interpolation is given by:

$\displaystyle\overline{\theta}_{0}^{\lambda,c}$ $\displaystyle=\theta_{0},\
d\overline{\theta}_{t}^{\lambda,c}=-\dfrac{\lambda}{N}\sum_{i=1}^{N}h^{\theta}_{\lambda,c}(\theta_{\lfloor{t}\rfloor}^{\lambda,c},X^{i,\lambda,c}_{\lfloor{t}\rfloor})dt+\sqrt{\dfrac{{2\lambda}}{N}}dB_{t}^{0,\lambda},$
(17) $\displaystyle\overline{X}_{0}^{i,\lambda,c}$ $\displaystyle=x^{i}_{0},\
d\overline{X}_{t}^{i,\lambda,c}=-\lambda
h^{x}_{\lambda,c}(\theta_{\lfloor{t}\rfloor}^{\lambda,c},X^{i,\lambda,c}_{\lfloor{t}\rfloor})dt+\sqrt{2\lambda}dB_{t}^{i,\lambda},$
(18)

and the time changed SDEs of (1)-(2) by

$\displaystyle d{\vartheta}_{\lambda
t}^{N}=-\dfrac{\lambda}{N}\sum_{i=1}^{N}h^{\theta}({\vartheta}^{N}_{\lambda
t},\mathcal{X}_{\lambda
t}^{i,N})dt+\sqrt{\dfrac{2\lambda}{N}}dB_{t}^{0,\lambda},$ (19) $\displaystyle
d\mathcal{X}_{\lambda t}^{i,N}=-{\lambda}h^{x}({\vartheta}^{N}_{\lambda
t},\mathcal{X}_{\lambda t}^{i,N})dt+\sqrt{2\lambda}dB_{t}^{i,\lambda},$ (20)

where the Brownian motions are defined as in (15)-(16).

Proposition 6. Let A1, A3 and A4 hold and consider the process
${\vartheta}_{\lambda t}^{N}$ as given by the dynamics (19)-(20). Then, for
every $\lambda_{0}<1/4\mu$, there exists a constant $C>0$ independent of
$N,n,\lambda$ such that for any $\lambda\in(0,\lambda_{0})$,

$\mathbb{E}^{1/2}\left[|\overline{\theta}^{\lambda,c}_{n}-\vartheta^{N}_{n\lambda}|^{2}\right]\leq\lambda^{1/2}C_{|z_{0}|,\mu,b,\ell,L}{(1+d^{\theta}+d^{x})}^{\ell+1},$

for all $n\in\mathbb{N}$.

###### Proof.

The proof is postponed to Appendix A. ∎

### 3.4 Main results and global error

Theorem 1. Consider the iterates $({\theta}^{\lambda,u}_{n})_{n\geq 0}$ as
given in tIPLAu, and let A1, A2 and A4 hold. Then, for every
$\lambda_{0}<N^{2\ell+1}/4\mu$, there exists a constant $C>0$ independent of
$N,n,\lambda$ such that, for any $\lambda\in(0,\lambda_{0})$,

$\displaystyle\mathbb{E}^{1/2}\left[|{\theta}^{*}-{\theta}^{\lambda,u}_{n}|^{2}\right]$
$\displaystyle\leq\sqrt{\dfrac{2d^{\theta}}{\mu N}}+e^{-\mu
n\lambda/N^{2\ell+1}}\left(\mathbb{E}^{1/2}\left[|z_{0}^{N}-z^{*}|^{2}\right]+\left(\dfrac{d^{x}N+d^{\theta}}{\mu
N^{2\ell+1}}\right)^{1/2}\right)$
$\displaystyle+\lambda^{1/2}C_{|z_{0}|,\mu,b,\ell,L}(1+d^{\theta}/N+d^{x})^{\ell+1},$

for all $n\in\mathbb{N}$.

###### Proof.

Combining the results from Propositions 2, 3 and 5, we are able to decompose
the expectation into a term describing the concentration of the
$\pi_{\Theta}^{N}$ around $\theta^{*}$, a term describing the convergence of
the IPS to its invariant measure, and a term describing the error induced by
the time discretisation:

$\displaystyle\mathbb{E}^{1/2}\left[|{\theta}^{*}-{\theta}^{\lambda,\cdot}_{n}|^{2}\right]$
$\displaystyle=W_{2}(\delta_{\theta^{*}},\mathcal{L}(\theta^{\lambda,\cdot}_{n}))$
$\displaystyle\leq
W_{2}(\delta_{\theta^{*}},\pi_{\Theta}^{N})+W_{2}(\pi_{\Theta}^{N},\mathcal{L}(\vartheta^{N}_{n\lambda}))+W_{2}(\mathcal{L}(\vartheta^{N}_{n\lambda}),\mathcal{L}(\theta_{n}^{\lambda,\cdot})).$
(21)

Substituting the bounds from the aforementioned Propositions yields the final
inequality. ∎

Theorem 2. Consider the iterates $({\theta}^{\lambda,c}_{n})_{n\geq 0}$ as
given in tIPLAc, and let A1, A3 and A4 hold. Then, for every
$\lambda_{0}<1/4\mu$, there exists a constant $C>0$ independent of
$N,n,\lambda$ such that, for any $\lambda\in(0,\lambda_{0})$,

$\displaystyle\mathbb{E}^{1/2}\left[|{\theta}^{*}-{\theta}^{\lambda,c}_{n}|^{2}\right]$
$\displaystyle\leq\sqrt{\dfrac{2d^{\theta}}{\mu N}}+e^{-\mu
n\lambda}\left(\mathbb{E}^{1/2}\left[|z_{0}^{N}-z^{*}|^{2}\right]+\left(\dfrac{d^{x}N+d^{\theta}}{\mu
N}\right)^{1/2}\right)$
$\displaystyle+\lambda^{1/2}C_{|z_{0}|,\mu,b,\ell,L}(1+d^{\theta}+d^{x})^{\ell+1},$

for all $n\in\mathbb{N}$.

###### Proof.

To prove the above inequality, one replaces the bound for
$W_{2}(\pi_{\Theta}^{N},\mathcal{L}(\vartheta^{N}_{n\lambda}))$ in (21) given
by Proposition 3 by the bound given by Proposition 4 and respectively the
bound for
$W_{2}(\mathcal{L}(\vartheta^{N}_{n\lambda}),\mathcal{L}(\theta_{n}^{\lambda,\cdot}))$
given by Proposition 5 by the bound given by Proposition 6. ∎

It is observed that the dependence on the dimension $d^{\theta}$ of the
parameter space in Theorem 2 exhibits a slight deterioration compared to that
presented in Theorem 1 and in Theorem 1 of [1]. Notably, the factor $1/N$ is
absent in the third term of Theorem 2. This is interpreted as a compromise
entailed by the use of the coordinate-wise taming function (7), which requires
the direct derivation of moment bounds for the pairs $\mathcal{V}_{t}^{i,N}$,
as discussed in Subsection 2.3. Consequently, this approach results in the
loss of the symmetrical structure that is attained by considering the
$N$-particle system $\mathcal{X}_{t}^{i,N}$ in its entirety.

## References

  * [1] Ö. Akyildiz et al.  “Interacting Particle Langevin Algorithm for Maximum Marginal Likelihood Estimation”  In _ArXiv_ abs/2303.13429, 2023
  * [2] Valentin De Bortoli, Alain Durmus, Marcelo Pereyra and Ana Fernandez Vidal  “Efficient stochastic optimisation by unadjusted Langevin Monte Carlo”  In _Statistics and Computing_ 31, 2019
  * [3] Nicolas Brosse, Alain Durmus, Éric Moulines and Sotirios Sabanis  “The tamed unadjusted Langevin algorithm”  In _Stochastic Processes and their Applications_ 129.10, 2019, pp. 3638–3663
  * [4] Ngoc Huy Chau et al.  “On Stochastic Gradient Langevin Dynamics with Dependent Data Streams: The Fully Nonconvex Case”  In _SIAM Journal on Mathematics of Data Science_ 3.3, 2021, pp. 959–986
  * [5] A.. Dempster, N.. Laird and D.. Rubin  “Maximum Likelihood from Incomplete Data via the EM Algorithm”  In _Journal of the Royal Statistical Society. Series B (Methodological)_ 39.1, 1977, pp. 1–38
  * [6] Alain Durmus and Éric Moulines  “High-dimensional Bayesian inference via the unadjusted Langevin algorithm”  In _Bernoulli_ 25.4A Bernoulli Society for Mathematical StatisticsProbability, 2019, pp. 2854–2882
  * [7] Alain Durmus and Éric Moulines  “Nonasymptotic convergence analysis for the unadjusted Langevin algorithm”  In _The Annals of Applied Probability_ 27.3 Institute of Mathematical Statistics, 2017, pp. 1551–1587
  * [8] Da Prato G  “An Introduction to Infinite-Dimensional Analysis”  Springer, 2006
  * [9] Martin Hutzenthaler, Arnulf Jentzen and Peter E. Kloeden  “Strong and weak divergence in finite time of Euler’s method for stochastic differential equations with non-globally Lipschitz continuous coefficients”  In _Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 467.2130 The Royal Society, 2010, pp. 1563–1576
  * [10] Martin Hutzenthaler, Arnulf Jentzen and Peter E. Kloeden  “Strong convergence of an explicit numerical method for SDEs with nonglobally Lipschitz continuous coefficients”  In _The Annals of Applied Probability_ 22.4 Institute of Mathematical Statistics, 2012
  * [11] Chaman Kumar and Sotirios Sabanis  “On explicit approximations for Lévy driven SDEs with super-linear diffusion coefficients”  In _Electronic Journal of Probability_ 22.none Institute of Mathematical StatisticsBernoulli Society, 2017, pp. 1–19
  * [12] Juan Kuntz, Jen Ning Lim and Adam M. Johansen  “Particle algorithms for maximum likelihood training of latent variable models”  In _Proceedings of The 26th International Conference on Artificial Intelligence and Statistics_ 206, Proceedings of Machine Learning Research  PMLR, 2023, pp. 5134–5180
  * [13] Dong-Young Lim, Ariel Neufeld, Sotirios Sabanis and Ying Zhang  “Langevin dynamics based algorithm e-TH$\epsilon$O POULA for stochastic optimization problems with discontinuous stochastic gradient”  In _ArXiv_ abs/2210.13193, 2022
  * [14] Dong-Young Lim, Ariel Neufeld, Sotirios Sabanis and Ying Zhang  “Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function”  In _IMA Journal of Numerical Analysis_ , 2023
  * [15] Dong-Young Lim and Sotirios Sabanis  “Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks”  In _To appear, Journal of Machine Learning Research_ , 2024
  * [16] Attila Lovas, Iosif Lytras, Miklós Rásonyi and Sotirios Sabanis  “Taming Neural Networks with TUSLA: Nonconvex Learning via Adaptive Stochastic Gradient Langevin Algorithms”  In _SIAM Journal on Mathematics of Data Science_ 5.2, 2023, pp. 323–345
  * [17] Xuerong Mao  “Stochastic Differential Equations and Applications”  Elsevier, 2007
  * [18] Sotirios Sabanis  “A note on tamed Euler approximations”  In _Electronic Communications in Probability_ 18.none Institute of Mathematical StatisticsBernoulli Society, 2013, pp. 1–10
  * [19] Sotirios Sabanis  “Euler approximations with varying coefficients: The case of superlinearly growing diffusion coefficients”  In _The Annals of Applied Probability_ 26.4 Institute of Mathematical Statistics, 2016
  * [20] Sotirios Sabanis and Ying Zhang  “Higher order Langevin Monte Carlo algorithm”  In _Electronic Journal of Statistics_ 13.2 Institute of Mathematical StatisticsBernoulli Society, 2019, pp. 3805–3850
  * [21] Sotirios Sabanis and Ying Zhang  “On explicit order 1.5 approximations with varying coefficients: The case of super-linear diffusion coefficients”  In _Journal of Complexity_ 50 Elsevier BV, 2019, pp. 84–115

##  Appendix A Proofs of subsection 3

### A.1 Uniform moments bounds

Lemma 1. Consider either the dynamics given by (1)-(2) or (3)-(4) and let
Assumptions A1-A2 hold, then there exists a constant $C>0$ such that

$\sup_{t\geq 0}\mathbb{E}\left[|\mathcal{Z}_{t}^{N}|^{2}\right]\leq C.$

###### Proof.

Consider the rescaling dynamics as given in (12) for the original system of
equations (1)-(2):

$\displaystyle|\mathcal{Z}_{t}^{N}|^{2}=|\vartheta_{t}^{N}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|\mathcal{X}_{t}^{i,N}|^{2}.$

By applying standard arguments using stopping times, Grönwall’s lemma and
Fatou’s lemma, one obtains that there is a constant $c$, which depends on
time, such that $\sup_{0\leq t\leq
T}\mathbb{E}\left[|\mathcal{Z}_{t}^{N}|^{2}\right]\leq c$ for any $T>0$.
Moreover, through the use of Itô’s formula one derives

$\displaystyle|\mathcal{Z}_{t}^{N}|^{2}$
$\displaystyle\leq|\vartheta_{0}^{N}|^{2}-2\int_{0}^{t}\langle\vartheta_{s}^{N},\dfrac{1}{N}\sum_{i=1}^{N}h^{\theta}(\mathcal{V}_{s}^{i,N})\rangle
ds+\dfrac{2d^{\theta}t}{N}+2\sqrt{\dfrac{2}{N}}\int_{0}^{t}\vartheta_{s}^{N}dB_{s}^{0,N}$
$\displaystyle+\dfrac{1}{N}\sum_{i=1}^{N}|\mathcal{X}_{0}^{i,N}|^{2}-\dfrac{2}{N}\sum_{i=1}^{N}\int_{0}^{t}\langle\mathcal{X}_{s}^{i,N},h^{x}(\mathcal{V}_{s}^{i,N})\rangle
ds+2d^{x}t+\dfrac{2\sqrt{2}}{N}\sum_{i=1}^{N}\int_{0}^{t}\mathcal{X}_{s}^{i,N}dB_{s}^{i,N}ds$
$\displaystyle\leq|\mathcal{Z}_{0}^{N}|^{2}-2\int_{0}^{t}\dfrac{1}{N}\sum_{i=1}^{N}\langle\mathcal{V}_{s}^{i,N},h(\mathcal{V}_{s}^{i,N})\rangle
ds+2(d^{\theta}/N+d^{x})t$
$\displaystyle+2\sqrt{\dfrac{2}{N}}\int_{0}^{t}\vartheta_{s}^{N}dB_{s}^{0,N}+\dfrac{2\sqrt{2}}{N}\sum_{i=1}^{N}\int_{0}^{t}\mathcal{X}_{s}^{i,N}dB_{s}^{i,N}ds$
$\displaystyle\leq|\mathcal{Z}_{0}^{N}|^{2}-\mu\int_{0}^{t}|\mathcal{Z}_{s}^{N}|^{2}ds+2bt+2(d^{\theta}/N+d^{x})t$
$\displaystyle+2\sqrt{\dfrac{2}{N}}\int_{0}^{t}\vartheta_{s}^{N}dB_{s}^{0,N}+\dfrac{2\sqrt{2}}{N}\sum_{i=1}^{N}\int_{0}^{t}\mathcal{X}_{s}^{i,N}dB_{s}^{i,N}ds.$

Taking the expectation on both sides results in

$\displaystyle\mathbb{E}\left[|\mathcal{Z}_{t}^{N}|^{2}\right]$
$\displaystyle\leq\mathbb{E}\left[|\mathcal{Z}_{0}^{N}|^{2}\right]-\mu\int_{0}^{t}\mathbb{E}\left[|\mathcal{Z}_{s}^{N}|^{2}\right]ds+2(b+d^{\theta}/N+d^{x})t.$

One notices that

$\displaystyle\dfrac{d}{dt}e^{\mu
t}\mathbb{E}\left[|\mathcal{Z}_{t}^{N}|^{2}\right]\leq e^{\mu t}C,$

which via integrating implies

$\displaystyle\mathbb{E}\left[|\mathcal{Z}_{t}^{N}|^{2}\right]\leq
C/\mu\Rightarrow\sup_{t\geq
0}\mathbb{E}\left[|\mathcal{Z}_{t}^{N}|^{2}\right]\leq C,$

as C is a constant independent of time. The corresponding result regarding
equations (3)-(4) follows by going through the same steps, where the
calculations differ only up to a constant in the SDEs coefficients. ∎

### A.2 Uniformly tamed scheme - tIPLAu

####  A.2.1 Key quantities for the proof of the main Lemmas.

The following definitions that refer to the rescaled dynamics (13)-(14) of the
algorithm tIPLAu and its continuous time interpolations (15)-(16) are given
as:

$\displaystyle
Z^{\lambda,u}_{n}=\left(\theta_{n+1}^{\lambda,u},N^{-1/2}X_{n+1}^{1,\lambda,u},\ldots,N^{-1/2}X_{n+1}^{N,\lambda,u}\right),$
(22)
$\displaystyle\overline{Z}^{\lambda,u}_{t}=\left(\overline{\theta}^{\lambda,u}_{t},N^{-1/2}\overline{X}^{1,\lambda,u}_{t},\ldots,N^{-1/2}\overline{X}^{N,\lambda,u}_{t}\right),$
(23) $\displaystyle\mathcal{Z}_{\lambda t}^{N}=\left(\vartheta_{\lambda
t}^{N},N^{-1/2}\mathcal{X}_{\lambda
t}^{1,N},\ldots,N^{-1/2}\mathcal{X}_{\lambda t}^{N,N}\right).$ (24)

####  A.2.2 Moment and increment bounds

Lemma 2. Let A1, A2 and A4 hold. Then, for any $0\leq\lambda<N^{p}/4\mu$, it
holds that

$\mathbb{E}\left[|Z^{\lambda,u}_{n}|^{2}\right]\leq
C_{|z_{0}|,\mu,b}(1+d^{\theta}/N+d^{x}),$

for a constant $C>0$ independent of $N,n,\lambda,d^{x}\text{ and }d^{\theta}$,
which is given in the proof.

###### Proof.

Consider the rescaled iterates as described in (22):

$\displaystyle\left|Z_{n+1}^{\lambda,u}\right|^{2}=\left|\theta_{n+1}^{\lambda,u}\right|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}\left|X_{n+1}^{i,\lambda,u}\right|^{2}$
$\displaystyle=\left|\theta_{n}^{\lambda,u}-\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\right|^{2}+\dfrac{2\lambda}{N^{p+1}}|\xi^{(0)}_{n+1}|^{2}$
$\displaystyle+2\sqrt{\dfrac{2\lambda}{N^{p+1}}}\left\langle\theta_{n}^{\lambda,u}-\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\xi_{n+1}^{(0)}\right\rangle$
$\displaystyle+\dfrac{1}{N}\sum_{i=1}^{N}\left(\left|X_{n}^{i,\lambda,u}-\dfrac{\lambda}{N^{p}}h^{x}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\right|^{2}+\dfrac{2\lambda}{N^{p}}|\xi^{(i)}_{n+1}|^{2}\right.$
$\displaystyle+\left.2\sqrt{\dfrac{2\lambda}{N^{p}}}\left\langle
X_{n}^{i,\lambda,u}-\dfrac{\lambda}{N^{p}}h^{x}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\xi_{n+1}^{(i)}\right\rangle\right).$

Taking the conditional expectation on both sides with respect to the
filtration generated by $Z_{n}^{\lambda,u}$ the cross terms are vanishing to
$0$ due to the independence between the $\xi^{(i)}_{n+1}$’s and
$Z_{n}^{\lambda,u}$, yielding

$\displaystyle\mathbb{E}\left[\left|Z_{n+1}^{\lambda,u}\right|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle=\mathbb{E}\left[\left|\theta_{n}^{\lambda,u}\right|^{2}|Z_{n}^{\lambda,u}\right]-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\mathbb{E}\left[\langle\theta_{n}^{\lambda,u},h_{\lambda,u}^{\theta}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\rangle|Z_{n}^{\lambda,u}\right]$
$\displaystyle+\dfrac{\lambda^{2}}{N^{2(p+1)}}\mathbb{E}\left[\left|\sum_{i=1}^{N}h_{\lambda,u}^{\theta}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\right|^{2}|Z_{n}^{\lambda,u}\right]+\dfrac{2\lambda
d^{\theta}}{N^{p+1}}$
$\displaystyle+\dfrac{1}{N}\sum_{i=1}^{N}\mathbb{E}\left[\left|X_{n}^{i,\lambda,u}\right|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\mathbb{E}\left[\langle
X_{n}^{i,\lambda},h_{\lambda,u}^{x}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\rangle|Z_{n}^{\lambda,u}\right]$
$\displaystyle+\dfrac{\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}\mathbb{E}\left[|h_{\lambda,u}^{x}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}|Z_{n}^{\lambda,u}\right]+\dfrac{2\lambda
d^{x}}{N^{p}}.$

Furthermore by using the elementary inequality $(t_{1}+\ldots+t_{m})^{p}\leq
m^{p-1}(t_{1}^{p}+\ldots,t_{m}^{p})$ and the fact that all of the expressions
within the conditional expectations are measurable we obtain

$\displaystyle\mathbb{E}\left[\left|Z_{n+1}^{\lambda,u}\right|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle\leq|\theta_{n}^{\lambda,u}|+\dfrac{1}{N}\sum_{i=1}^{N}|X_{n}^{i,\lambda,u}|^{2}+\dfrac{2\lambda
d^{\theta}}{N^{p+1}}++\dfrac{2\lambda d^{x}}{N^{p}}$
$\displaystyle-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle\theta_{n}^{\lambda},h_{\lambda,u}^{\theta}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\rangle$
$\displaystyle-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
X_{n}^{i,\lambda,u},h_{\lambda,u}^{x}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\rangle$
$\displaystyle+\dfrac{\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}|h_{\lambda}^{\theta}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}+\dfrac{\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}|h_{\lambda,u}^{x}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}.$

Recalling that $(h^{\theta}_{\lambda,u},h^{x}_{\lambda,u})=h_{\lambda,u}$,
further leads to

$\displaystyle\mathbb{E}\left[\left|Z_{n+1}^{\lambda,u}\right|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle\leq\left|Z_{n}^{\lambda,u}\right|^{2}-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
V_{n}^{i,\lambda,u},h_{\lambda,u}(V_{n}^{i,\lambda,u})\rangle+\dfrac{\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}|h_{\lambda,u}(V_{n}^{i,\lambda,u})|^{2}$
$\displaystyle+\dfrac{2\lambda}{N^{p}}(d^{\theta}/N+d^{x}).$

Now using Properties 1 and 3 of the taming function, we get

$\displaystyle\mathbb{E}\left[\left|Z_{n+1}^{\lambda,u}\right|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle\leq\left|Z_{n}^{\lambda,u}\right|^{2}-\dfrac{\mu\lambda}{N^{p+1}}\sum_{i=1}^{N}\left|V_{n}^{i,\lambda,u}\right|^{2}+\dfrac{2\lambda
b}{N^{p}}$
$\displaystyle+\dfrac{2\lambda^{2}\mu^{2}}{N^{2p+1}}\sum_{i=1}^{N}\left|V_{n}^{i,\lambda,u}\right|^{2}+\dfrac{2\lambda^{2}\lambda^{-1}N^{p}}{N^{2p}}+\dfrac{2\lambda}{N^{p}}(d^{\theta}/N+d^{x})$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{N^{p}}+\dfrac{2\lambda^{2}\mu^{2}}{N^{2p}}\right)\left|Z_{n}^{\lambda,u}\right|^{2}+\dfrac{2\lambda}{N^{p}}\left(b+1+d^{\theta}/N+d^{x}\right).$

By considering the restriction $\lambda<\dfrac{N^{p}}{4\mu}$ and iterating the
above bound, we conclude with

$\displaystyle\mathbb{E}\left[\left|Z_{n+1}^{\lambda,u}\right|^{2}\right]$
$\displaystyle\leq\left(1-\dfrac{\mu\lambda}{2N^{p}}\right)^{n}\mathbb{E}\left[\left|Z_{0}^{\lambda,u}\right|^{2}\right]+\dfrac{1-(1-\mu\lambda/2)^{n}}{\mu\lambda/2N^{p}}\dfrac{2\lambda}{N^{p}}(b+1+d^{\theta}/N+d^{x})$
$\displaystyle\leq\mathbb{E}\left[\left|Z_{0}^{\lambda,u}\right|^{2}\right]+\dfrac{4}{\mu}(b+1+d^{\theta}/N+d^{x})$
$\displaystyle\leq\left(\mathbb{E}\left[\left|Z_{0}^{\lambda,u}\right|^{2}\right]+\dfrac{4}{\mu}(b+1)\right)(1+d^{\theta}/N+d^{x})$
$\displaystyle\leq C_{|z_{0}|,\mu,b}(1+d^{\theta}/N+d^{x}).$

∎

Lemma 3. Let A1, A2 and A4 hold. Then, for any $0\leq\lambda<N^{p}/4\mu$ and
$q\in[2,2\ell+1)\cap\mathbb{N}$, it holds that,

$\mathbb{E}\left[|Z^{\lambda,u}_{n}|^{2q}\right]\leq
C_{|z_{0}|,b,q,\mu}(1+d^{\theta}/N+d^{x})^{q},$

for a constant $C>0$ independent of $N,n,\lambda,d^{x}\text{ and }d^{\theta}$.

###### Proof.

To make the forthcoming calculations more readable we define the following
auxiliary processes:

$\displaystyle\Delta_{n}^{\lambda,\theta}=\theta_{n}^{\lambda,u}-\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}\left(\theta_{n}^{\lambda},X^{i,\lambda}_{n}\right),\
G_{n}^{\lambda,\theta}=\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi^{(0)}_{n+1},$
$\displaystyle\Delta_{n}^{\lambda,x,i}=X_{n}^{i,\lambda,u}-\dfrac{\lambda}{N^{p}}h^{x}_{\lambda,u}\left(\theta_{n}^{\lambda},X^{i,\lambda}_{n}\right),\
G_{n}^{\lambda,x,i}=\sqrt{\dfrac{2\lambda}{N^{p}}}\xi^{(i)}_{n+1},$

but from this point onwards till the completion of the proof we denote
$\lambda/N^{p}$ by just $\lambda$ to make the computations less hectic. Recall
that have already established via Lemma 2 the bound

$\displaystyle
A_{n}^{\lambda}:=|\Delta_{n}^{\lambda,\theta}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|\Delta_{n}^{\lambda,i,x}|^{2}\leq\left(1-\dfrac{\lambda\mu}{2}\right)|Z_{n}^{\lambda,u}|^{2}+2\lambda
C,$

and lastly let us define the quantity

$\displaystyle
B^{\lambda}_{n}=2\langle\Delta_{n}^{\theta},G_{n}^{\lambda,\theta}\rangle+\dfrac{2}{N}\sum_{i=1}^{N}\langle\Delta_{n}^{\lambda,x,i},G_{n}^{\lambda,x,i}\rangle+|G_{n}^{\lambda,\theta}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|G_{n}^{\lambda,x,i}|^{2}.$

Regarding the $2q$-th moment one writes

$\displaystyle|Z_{n+1}^{\lambda,u}|^{2q}$
$\displaystyle=\left(A_{n}^{\lambda}+B^{\lambda}_{n}\right)^{q}$
$\displaystyle\leq(A_{n}^{\lambda})^{q}+2q(A_{n}^{\lambda})^{q-1}B^{\lambda}_{n}+\sum_{k=2}^{q}\binom{q}{k}|A_{n}^{\lambda}|^{q-k}|B^{\lambda}_{n}|^{k}.$
(25)

We shall deal with each term separately

$\displaystyle\mathbb{E}\left[(A_{n}^{\lambda})^{q}|Z_{n}^{\lambda,u}\right]$
$\displaystyle=(A_{n}^{\lambda})^{q}\leq\left(\left(1-\dfrac{\lambda\mu}{2}\right)|Z_{n}^{\lambda,u}|^{2}+2\lambda
C\right)^{q}$
$\displaystyle\leq\left(1+\dfrac{\lambda\mu}{4}\right)^{q-1}\left(1-\dfrac{\lambda\mu}{2}\right)^{q}|Z_{n}^{\lambda,u}|^{2q}+\left(1+\dfrac{4}{\lambda\mu}\right)^{q-1}2^{q}{\lambda}^{q}C^{q}$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{4}\right)^{q-1}\left(1-\dfrac{\lambda\mu}{2}\right)|Z_{n}^{\lambda,u}|^{2q}+\left(\lambda+\dfrac{4}{\mu}\right)^{q-1}\lambda(2C)^{q}$
$\displaystyle\leq r_{q}^{\lambda}|Z_{n}^{\lambda,u}|^{2q}+w_{q}^{\lambda}.$
(26)

where $r_{q}^{\lambda}=(1-\lambda\mu/4)^{q-1}(1-\lambda\mu/2)$ and
$w_{q}^{\lambda}=(\lambda+4/\mu)^{q-1}\lambda(2C)^{q}$. Notice that we made
use of the elementary equation
$(r+s)^{q}\leq(1+\epsilon)^{q-1}r^{q}+(1+1/\epsilon)^{q-1}s^{q}$ for the
choice $\epsilon=\lambda\mu/4$. Additionally for the shake of simplicity, let
us denote $d=d^{\theta}/N+d^{x}$. Now on a similar note with the previous term

$\displaystyle\mathbb{E}\left[2q(A_{n}^{\lambda})^{q-1}B^{\lambda}_{n}|Z_{n}^{\lambda,u}\right]$
$\displaystyle=2q(A_{n}^{\lambda})^{q-1}\mathbb{E}\left[B^{\lambda}_{n}|Z_{n}^{\lambda,u}\right]=4q\lambda\left(d^{\theta}/N+d^{x}\right)(A_{n}^{\lambda})^{q-1}$
$\displaystyle\leq
4q\lambda\left(d^{\theta}/N+d^{x}\right)\left(r_{q-1}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle\leq 4q\lambda
d\left(r_{q-1}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-1)}+w_{q-1}^{\lambda}\right).$
(27)

The 3rd term on (25) can be further expanded to

$\displaystyle\sum_{k=2}^{q}\binom{q}{k}|A_{n}^{\lambda}|^{q-k}|B^{\lambda}_{n}|^{k}$
$\displaystyle=\sum_{m=0}^{q-2}\binom{q}{m+2}|A_{n}^{\lambda}|^{q-2-m}|B^{\lambda}_{n}|^{m+2}$
$\displaystyle=\dfrac{q}{m+2}\dfrac{q-1}{m+1}\sum_{m=0}^{q-2}\binom{q-2}{m}|A_{n}^{\lambda}|^{q-2-m}|B^{\lambda}_{n}|^{m}|B^{\lambda}_{n}|^{2}$
$\displaystyle\leq
q(q-1)\left(|A_{n}^{\lambda}|+|B_{n}^{\lambda}|\right)^{q-2}|B^{\lambda}_{n}|^{2}$
$\displaystyle\leq
q(q-1)2^{q-3}|A_{n}^{\lambda}|^{q-2}|B_{n}^{\lambda}|^{2}+q(q-1)2^{q-3}|B_{n}^{\lambda}|^{q}$
$\displaystyle:=D+F.$ (28)

Taking the expectation of the above terms yields

$\displaystyle\mathbb{E}[D|Z_{n}^{\lambda,u}]$
$\displaystyle=q(q-1)2^{q-3}|A_{n}^{\lambda}|^{q-2}\mathbb{E}\left[|B_{n}^{\lambda}|^{2}|Z_{n}^{\lambda,u}\right],$

where

$\displaystyle\mathbb{E}\left[|B_{n}^{\lambda}|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle=\mathbb{E}\left[\left|2\langle\Delta_{n}^{\theta},G_{n}^{\lambda,\theta}\rangle+\dfrac{2}{N}\sum_{i=1}^{N}\langle\Delta_{n}^{\lambda,x,i},G_{n}^{\lambda,x,i}\rangle\\+|G_{n}^{\lambda,\theta}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|G_{n}^{\lambda,x,i}|^{2}\right|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle\leq
4\left(\mathbb{E}\left[4|\Delta_{n}^{\lambda,\theta}|^{2}|G_{n}^{\lambda,\theta}|^{2}|Z_{n}^{\lambda,u}\right]+\mathbb{E}\left[\dfrac{4}{N}\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{2}|G_{n}^{\lambda,x,i}|^{2}|Z_{n}^{\lambda,u}\right]\right.$
$\displaystyle+\left.\mathbb{E}\left[|G_{n}^{\lambda,x,i}|^{4}|Z_{n}^{\lambda,u}\right]+\mathbb{E}\left[\dfrac{1}{N}\sum_{i=1}^{N}|G_{n}^{\lambda,x,i}|^{4}|Z_{n}^{\lambda,u}\right]\right)$
$\displaystyle\leq
4\left(4|\Delta_{n}^{\lambda,\theta}|^{2}\mathbb{E}\left[|G_{n}^{\lambda,\theta}|^{2}|Z_{n}^{\lambda,u}\right]+\dfrac{4}{N}\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{2}\mathbb{E}\left[|G_{n}^{\lambda,x,i}|^{2}|Z_{n}^{\lambda,u}\right]\right.$
$\displaystyle+\left.\mathbb{E}\left[|G_{n}^{\lambda,x,i}|^{4}|Z_{n}^{\lambda,u}\right]+\mathbb{E}\left[\dfrac{1}{N}\sum_{i=1}^{N}|G_{n}^{\lambda,x,i}|^{4}|Z_{n}^{\lambda,u}\right]\right).$

Recalling that each $G_{n}^{\lambda,\cdot,i}$ is distributed according to a
Gaussian distribution, we subsequently derive

$\displaystyle\mathbb{E}\left[|B_{n}^{\lambda}|^{2}|Z_{n}^{\lambda,u}\right]$
$\displaystyle\leq 4\left(4(2\lambda
d^{\theta}/N)|\Delta_{n}^{\lambda,\theta}|^{2}+\dfrac{4}{N}\sum_{i=1}^{N}(2\lambda
d^{x})|\Delta_{n}^{\lambda,x,i}|^{2}+3(2\lambda d^{\theta}/N)^{2}+3(2\lambda
d^{x})^{2}\right)$ $\displaystyle\leq 16(2\lambda
d)\left(|\Delta_{n}^{\lambda,\theta}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{2}\right)+24(2\lambda
d)^{2}$ $\displaystyle\leq 16(2\lambda d)A_{n}^{\lambda}+24(2\lambda d)^{2}.$

By substituting this result, the term $D$ can be expressed as follows

$\displaystyle\mathbb{E}[D|Z_{n}^{\lambda,u}]$
$\displaystyle=q(q-1)2^{q-3}|A_{n}^{\lambda}|^{q-2}\left(16(2\lambda
d)A_{n}^{\lambda}+24(2\lambda d)^{2}\right)$ $\displaystyle=16(2\lambda
d)q(q-1)2^{q-3}\left(r_{q-1}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle+24(2\lambda
d)^{2}q(q-1)2^{q-3}\left(r_{q-2}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-2)}+w_{q-2}^{\lambda}\right).$
(29)

Additionally,

$\displaystyle\mathbb{E}[F|Z_{n}^{\lambda,u}]$
$\displaystyle=q(q-1)2^{q-3}|B_{n}^{\lambda}|^{q}$
$\displaystyle=q(q-1)2^{q-3}\mathbb{E}\left[\left|2\langle\Delta_{n}^{\theta},G_{n}^{\lambda,\theta}\rangle+\dfrac{2}{N}\sum_{i=1}^{N}\langle\Delta_{n}^{\lambda,x,i},G_{n}^{\lambda,x,i}\rangle\right.\right.$
$\displaystyle\left.\left.+|G_{n}^{\lambda,\theta}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|G_{n}^{\lambda,x,i}|^{2}\right|^{q}|Z_{n}^{\lambda,u}\right]$
$\displaystyle\leq
q(q-1)2^{q-3}4^{q-1}\left(\mathbb{E}\left[2^{q}|\Delta_{n}^{\lambda,\theta}|^{q}|G_{n}^{\lambda,\theta}|^{q}|Z_{n}^{\lambda,u}\right]\right.$
$\displaystyle+\mathbb{E}\left[\dfrac{2^{q}}{N^{q}}\left|\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}||G_{n}^{\lambda,x,i}|\right|^{q}|Z_{n}^{\lambda,u}\right]+\mathbb{E}\left[|G_{n}^{\lambda,\theta}|^{2q}|Z_{n}^{\lambda,u}\right]$
$\displaystyle\left.+\mathbb{E}\left[\dfrac{1}{N^{q}}N^{q-1}\sum_{i=1}^{N}|G_{n}^{\lambda,x,i}|^{2q}|Z_{n}^{\lambda,u}\right]\right).$
(30)

We handle the second term in (30) by applying the multinomial expansion:

$\left|\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}||G_{n}^{\lambda,x,i}|\right|^{q}=\sum_{k_{1}+\ldots+k_{N}=q}\binom{q}{k_{1},\ldots,k_{N}}\prod_{i=1}^{N}\left(|\Delta_{n}^{\lambda,x,i}||G_{n}^{\lambda,x,i}|\right)^{k_{i}},$

and using the fact that $\Delta_{n}^{\lambda,x,i}$ are
$Z_{n}^{\lambda,u}-$measurable, hence

$\displaystyle\dfrac{2^{q}}{N^{q}}\mathbb{E}\left[\left|\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|G_{n}^{\lambda,x,i}|\right|^{q}|Z_{n}^{\lambda,u}\right]$
$\displaystyle=\dfrac{2^{q}}{N^{q}}\mathbb{E}\left[\sum_{k_{1}+\ldots+k_{N}=q}\binom{q}{k_{1},\ldots,k_{N}}\prod_{i=1}^{N}\left(|\Delta_{n}^{\lambda,x,i}||G_{n}^{\lambda,x,i}|\right)^{k_{i}}|Z_{n}^{\lambda,u}\right]$
$\displaystyle=\dfrac{2^{q}}{N^{q}}\sum_{k_{1}+\ldots+k_{N}=q}\binom{q}{k_{1},\ldots,k_{N}}\prod_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{k_{i}}\prod_{i=1}^{N}\mathbb{E}\left[|G_{n}^{\lambda,x,i}|^{k_{i}}|Z_{n}^{\lambda,u}\right]$
$\displaystyle=\dfrac{2^{q}}{N^{q}}\sum_{k_{1}+\ldots+k_{N}=q}\binom{q}{k_{1},\ldots,k_{N}}\prod_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{k_{i}}\prod_{i=1}^{N}\left((2\lambda
d^{x})^{k_{i}/2}k_{i}!!\right)$
$\displaystyle=\dfrac{2^{q}}{N^{q}}\sum_{k_{1}+\ldots+k_{N}=q}\binom{q}{k_{1},\ldots,k_{N}}\prod_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{k_{i}}(2\lambda
d^{x})^{\sum_{i=1}^{N}k_{i}/2}k_{i}!!$
$\displaystyle\leq\dfrac{2^{q}q!!(2\lambda
d^{x})^{q/2}}{N^{q}}\sum_{k_{1}+\ldots+k_{N}=q}\binom{q}{k_{1},\ldots,k_{N}}\prod_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{k_{i}}$
$\displaystyle=\dfrac{2^{q}q!!(2\lambda
d^{x})^{q/2}}{N^{q}}\left(\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|\right)^{q}\leq\dfrac{2^{q}q!!(2\lambda
d^{x})^{q/2}}{N^{q-1}}\left(\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{2}\right)^{q/2}$
$\displaystyle\leq 2^{q}q!!(2\lambda
d^{x})^{q/2}\left(\dfrac{1}{N}\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{2}\right)^{q/2}.$

Plugging these results into (30) we get

$\displaystyle\mathbb{E}[F|Z_{n}^{\lambda,u}]$ $\displaystyle\leq
q(q-1)2^{3q-5}\left(2^{q}q!!(2\lambda
d^{\theta}/N)^{q/2}\left(|\Delta_{n}^{\lambda,\theta}|^{2}\right)^{q/2}\right.$
$\displaystyle\left.+2^{q}q!!(2\lambda
d^{x})^{q/2}\left(\dfrac{1}{N}\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{2}\right)^{q/2}+(2q)!!(2\lambda
d^{\theta}/N)^{q}+(2q)!!(2\lambda d^{x})^{q}\right)$ $\displaystyle\leq
q(q-1)2^{3q-5}\left(2^{q}q!!(2\lambda
d)^{q/2}\left(|\Delta_{n}^{\lambda,\theta}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|\Delta_{n}^{\lambda,x,i}|^{2}\right)^{q/2}+2(2q)!!(2\lambda
d)^{q}\right)$ $\displaystyle\leq q(q-1)2^{4q-5}(2q)!!\left((2\lambda
d)^{q/2}\left|A_{n}^{\lambda}\right|^{q/2}+(2\lambda d)^{q}\right)$
$\displaystyle\leq q(q-1)2^{4q-5}(2q)!!(2\lambda
d)^{q/2}\left(r^{\lambda}_{q/2}|Z_{n}^{\lambda,u}|^{q}+w^{\lambda}_{q/2}\right)$
$\displaystyle+q(q-1)2^{4q-5}(2q)!!(2\lambda d)^{q}.$ (31)

Combining (29),(31) yields the following bound for (28),

$\displaystyle\mathbb{E}\left[\sum_{k=2}^{q}\binom{q}{k}|A_{n}^{\lambda}|^{q-k}|B^{\lambda}_{n}|^{k}\right]$
$\displaystyle\leq 16(2\lambda
d)q(q-1)2^{q-3}\left(r_{q-1}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle+24(2\lambda
d)^{2}q(q-1)2^{q-3}\left(r_{q-2}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-2)}+w_{q-2}^{\lambda}\right)$
$\displaystyle+(2q)!!(2\lambda
d)^{q/2}q(q-1)2^{4q-5}\left(r^{\lambda}_{q/2}|Z_{n}^{\lambda,u}|^{q}+w^{\lambda}_{q/2}\right)$
$\displaystyle+(2q)!!(2\lambda d)^{q}q(q-1)2^{4q-5}.$ (32)

Hence one obtains for (25) via (26),(27) and (32),

$\displaystyle\mathbb{E}\left[|Z_{n+1}|^{2q}|Z_{n}\right]$ $\displaystyle\leq
r_{q}^{\lambda}|Z_{n}^{\lambda,u}|^{2q}+w_{q}^{\lambda}$
$\displaystyle+4q\left(2\lambda
d\right)(1+4(q-1)2^{q-3})\left(r_{q-1}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle+24(2\lambda
d)^{2}q(q-1)2^{q-3}\left(r_{q-2}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-2)}+w_{q-2}^{\lambda}\right)$
$\displaystyle+(2q)!!(2\lambda
d)^{q/2}q(q-1)2^{4q-5}\left(r^{\lambda}_{q/2}|Z_{n}^{\lambda,u}|^{q}+w^{\lambda}_{q/2}\right)$
$\displaystyle+(2q)!!(2\lambda d)^{q}q(q-1)2^{4q-5}.$ (33)

Consider
$|Z_{n}^{\lambda,u}|\geq\sqrt{\dfrac{2d}{\mu/4}}\left\\{(2q)!!q(q-1)2^{4q-5}\right\\}^{1/2}\geq\sqrt{\dfrac{2d}{\mu/4}}\left\\{(2q)!!q(q-1)2^{4q-5}\right\\}^{1/q}$.
Then one observes

$\displaystyle\mathbb{E}\left[|Z_{n+1}^{\lambda,u}|^{2q}|Z_{n}\right]$
$\displaystyle\leq r_{q}^{\lambda}|Z_{n}^{\lambda,u}|^{2q}+w_{q}^{\lambda}$
$\displaystyle+\dfrac{\lambda\mu}{2\cdot
4}|Z_{n}^{\lambda,u}|^{2}\left(r_{q-1}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-1)}\right)+\dfrac{(\lambda\mu)^{2}}{2\cdot
4^{2}}|Z_{n}^{\lambda,u}|^{4}\left(r_{q-2}^{\lambda}|Z_{n}^{\lambda,u}|^{2(q-2)}\right)$
$\displaystyle+\dfrac{(\lambda\mu)^{q/2}}{2\cdot
4^{q/2}}|Z_{n}^{\lambda,u}|^{q}\left(r^{\lambda}_{q/2}|Z_{n}^{\lambda,u}|^{q}\right)+(2q)!!q(q-1)2^{4q-5}(2\lambda
d)^{q}$ $\displaystyle+(2q)!!q(q-1)2^{4q-5}\left((2\lambda
d)w_{q-1}^{\lambda}+(2\lambda d)^{2}w_{q-2}^{\lambda}+(2\lambda
d)^{q/2}w_{q/2}^{\lambda}\right)$ $\displaystyle\leq
r^{\lambda}_{q/2}\left(\left(1-\dfrac{\lambda\mu}{4}\right)^{q/2}+\dfrac{1}{2}\left(\dfrac{\lambda\mu}{4}\right)\left(1-\dfrac{\lambda\mu}{4}\right)^{q/2-1}\right.$
$\displaystyle\left.+\dfrac{1}{2}\left(\dfrac{\lambda\mu}{4}\right)^{2}\left(1-\dfrac{\lambda\mu}{4}\right)^{q/2-2}+\dfrac{1}{2}\left(\dfrac{\lambda\mu}{4}\right)^{q/2}\right)|Z_{n}^{\lambda,u}|^{2q}$
$\displaystyle+w_{q}^{\lambda}+(2q)!!q(q-1)2^{4q-5}(2\lambda d)^{q}$
$\displaystyle+(2q)!!q(q-1)2^{4q-5}\left((2\lambda
d)w_{q-1}^{\lambda}+(2\lambda d)^{2}w_{q-2}^{\lambda}+(2\lambda
d)^{q/2}w_{q/2}^{\lambda}\right).$

Using the fact that $\lambda\mu\leq 1$ we get

$\displaystyle\mathbb{E}\left[|Z_{n+1}^{\lambda,u}|^{2q}|Z_{n}\right]$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)|Z_{n}^{\lambda,u}|^{2q}$
$\displaystyle+w_{q}^{\lambda}+(2q)!!q(q-1)2^{4q-5}(2\lambda d)^{q}$
$\displaystyle+(2q)!!q(q-1)2^{4q-5}\left((2\lambda
d)w_{q-1}^{\lambda}+(2\lambda d)^{2}w_{q-2}^{\lambda}+(2\lambda
d)^{q/2}w_{q/2}^{\lambda}\right)$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)|Z_{n}^{\lambda,u}|^{2q}+(2q)!!q(q-1)2^{4q-2}(8/\mu)^{q-1}\lambda(2C)^{q}.$

Consequently, on
$\\{|Z_{n}^{\lambda,u}|\leq\sqrt{\dfrac{2d}{\mu/4}}\left\\{(2q)!!q(q-1)2^{4q-5}\right\\}^{1/2}\\}$,
we have

$\displaystyle\mathbb{E}\left[|Z_{n+1}^{\lambda,u}|^{2q}|Z_{n}\right]$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)|Z_{n}^{\lambda,u}|^{2q}$
$\displaystyle+(2q)!!q(q-1)2^{4q-2}(8/\mu)^{q-1}\lambda(2C)^{q}$
$\displaystyle+\left((2q)!!q(q-1)2^{4q-2}\right)^{q/2}(8/\mu)^{q-1}\lambda(2C)^{q}.$

So all in all (33) results to

$\displaystyle\mathbb{E}\left[|Z_{n+1}^{\lambda,u}|^{2q}\right]$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)\mathbb{E}\left[|Z_{n}^{\lambda,u}|^{2q}\right]+M_{q}(8/\mu)^{q}C^{q}$
$\displaystyle\leq C_{|z_{0}|,b,q,\mu}(1+d^{\theta}/N+d^{x})^{q},$

where
$C_{|z_{0}|,b,q,\mu}=\mathbb{E}\left[|Z_{0}^{\lambda}|^{2q}\right]M_{q}(8/\mu)^{q}(b+1)^{q}$
and $M_{q}=(2q)!!q(q-1)2^{6q-5}$. ∎

Lemma 4. Let A1, A2 and A4 hold. Then, for every $\lambda_{0}<N^{p}/4\mu$,
there exists a constant $C>0$ independent of $N,n,\lambda$ such that for any
$\lambda\in(0,\lambda_{0})$ one has

$\mathbb{E}\left[|Z^{\lambda,u}_{n+1}-Z^{\lambda,u}_{n}|^{4}\right]\leq\lambda^{2}N^{-2p}C_{|z_{0}|,\mu,b}(1+d^{\theta}/N+d^{x})^{2}.$

###### Proof.

$\displaystyle|Z^{\lambda,u}_{n+1}-Z^{\lambda,u}_{n}|^{4}=\left(|\theta^{\lambda,u}_{n+1}-\theta^{\lambda,u}_{n}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|X_{n+1}^{i,\lambda,u}-X_{n}^{i,\lambda,u}|^{2}\right)^{2}$
$\displaystyle=\left(\left|\dfrac{-\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})+\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi_{n+1}^{(0)}\right|^{2}\right.$
$\displaystyle\left.+\dfrac{1}{N}\sum_{i=1}^{N}\left|-\dfrac{\lambda}{N^{p}}h^{x}_{\lambda}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})+\sqrt{\dfrac{2\lambda}{N^{p}}}\xi_{n+1}^{(i)}\right|^{2}\right)^{2}$

Moreover

$\displaystyle|Z^{\lambda,u}_{n+1}-Z^{\lambda,u}_{n}|^{4}\leq\left(\dfrac{\lambda^{2}}{N^{2(p+1)}}|\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}+\dfrac{\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}|h^{x}_{\lambda}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}\right.$
$\displaystyle-2\langle\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi_{n+1}^{(0)}\rangle$
$\displaystyle\left.-2\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h^{x}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\sqrt{\dfrac{2\lambda}{N^{p}}}\xi_{n+1}^{(i)}\rangle+\dfrac{2\lambda}{N^{p+1}}|\xi_{n+1}^{(0)}|^{2}+\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}|\xi_{n+1}^{(i)}|^{2}\right)^{2}$
$\displaystyle\leq\dfrac{\lambda^{4}}{N^{4p+2}}\left(\sum_{i=1}^{N}|h_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}\right)^{2}+2\left(\dfrac{\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}|h_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}\right)$
$\displaystyle\times\left(-2\langle\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi_{n+1}^{(0)}\rangle\right.$
$\displaystyle\left.-2\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h^{x}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\sqrt{\dfrac{2\lambda}{N^{p}}}\xi_{n+1}^{(i)}\rangle+\dfrac{2\lambda}{N^{p+1}}|\xi_{n+1}^{(0)}|^{2}+\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}|\xi_{n+1}^{(i)}|^{2}\right)$
$\displaystyle+\left|-2\langle\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi_{n+1}^{(0)}\rangle\right.$
$\displaystyle\left.-2\dfrac{\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h^{x}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u}),\sqrt{\dfrac{2\lambda}{N^{p}}}\xi_{n+1}^{(i)}\rangle+\dfrac{2\lambda}{N^{p+1}}|\xi_{n+1}^{(0)}|^{2}+\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}|\xi_{n+1}^{(i)}|^{2}\right|^{2}.$
(34)

The first term in (34) is immediately bounded by the moments of
$Z^{\lambda,u}_{n}$,

$\displaystyle\mathbb{E}\left[\dfrac{\lambda^{4}}{N^{4p+2}}\left(\sum_{i=1}^{N}|h_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}\right)^{2}|Z^{\lambda,u}_{n}\right]$
$\displaystyle\leq\dfrac{\lambda^{4}}{N^{4p+2}}\left(2\mu^{2}\sum_{i=1}^{N}|V_{n}^{i,\lambda,u}|^{2}+2\lambda^{-1}N^{p}\right)^{2}$
$\displaystyle\leq
4\mu^{4}\dfrac{\lambda^{4}}{N^{4p}}\dfrac{1}{N^{2}}\left(\sum_{i=1}^{N}|V_{n}^{i,\lambda,u}|^{2}\right)^{2}+4\dfrac{\lambda^{2}}{N^{2p}}$
$\displaystyle\leq\mu^{2}\dfrac{\lambda^{2}}{N^{2p}}|Z^{\lambda,u}_{n}|^{4}+4\dfrac{\lambda^{2}}{N^{2p}}.$
(35)

where the last inequality follows due to the restriction $\lambda<N^{p}/4\mu$.
Notice that in the second term the inner products $\langle\cdot,\cdot\rangle$
vanish under expectation

$\displaystyle
2\left(\dfrac{\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}|h_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}\right)\left(\dfrac{2\lambda}{N^{p+1}}\mathbb{E}\left[|\xi_{n+1}^{(0)}|^{2}|Z^{\lambda,u}_{n}\right]+\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\mathbb{E}\left[|\xi_{n+1}^{(i)}|^{2}|Z^{\lambda,u}_{n}\right]\right)$
$\displaystyle\leq
2\dfrac{\lambda^{2}}{N^{2p}}\left(2\mu^{2}|Z^{\lambda,u}_{n}|^{2}+2\lambda^{-1}N^{p}\right)\left(2\dfrac{\lambda}{N^{p}}d^{\theta}/N+2\dfrac{\lambda}{N^{p}}d^{x}\right)\leq
8\dfrac{\lambda^{3}}{N^{3p}}d(\mu^{2}|Z^{\lambda,u}_{n}|^{2}+\lambda^{-1}N^{p})$
$\displaystyle\leq
2d\mu\dfrac{\lambda^{2}}{N^{2p}}|Z^{\lambda,u}_{n}|^{2}+8d\dfrac{\lambda^{2}}{N^{2p}}.$
(36)

Using the usual elementary inequality and the Cauchy-Schwartz on the last term
of (34) we get

$\displaystyle\dfrac{16\lambda^{2}}{N^{2p+2}}\left|\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\right|^{2}\mathbb{E}\left[|\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi_{n+1}^{(0)}|^{2}\right]$
$\displaystyle+\dfrac{16\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}|h^{x}_{\lambda}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}\mathbb{E}\left[|\sqrt{\dfrac{2\lambda}{N^{p}}}\xi_{n+1}^{(i)}|^{2}\right]$
$\displaystyle+\dfrac{16\lambda^{2}}{N^{2p+2}}\mathbb{E}\left[|\xi_{n+1}^{(0)}|^{4}\right]+\dfrac{16\lambda^{2}}{N^{2p+1}}\sum_{i=1}^{N}\mathbb{E}\left[|\xi_{n+1}^{(i)}|^{4}\right]$
$\displaystyle\leq
16\dfrac{\lambda^{2}}{N^{2p}}\left(2\dfrac{\lambda}{N^{p}}d^{\theta}/N\right)\dfrac{1}{N}\sum_{i=1}^{N}|h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}$
$\displaystyle+16\dfrac{\lambda^{2}}{N^{2p}}\left(2\dfrac{\lambda}{N^{p}}d^{x}\right)\dfrac{1}{N}\sum_{i=1}^{N}|h^{x}_{\lambda}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})|^{2}$
$\displaystyle+16\dfrac{\lambda^{2}}{N^{2p}}(d^{\theta}/N)^{2}+16\dfrac{\lambda^{2}}{N^{2p}}(d^{x})^{2}$

which by using Property 1 of the taming function, can be further written as

$\displaystyle\dfrac{16\lambda^{2}}{N^{2p+2}}\left|\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{n}^{\lambda,u},X_{n}^{i,\lambda,u})\right|^{2}\mathbb{E}\left[|\sqrt{\dfrac{2\lambda}{N^{p+1}}}\xi_{n+1}^{(0)}|^{2}\right]$
$\displaystyle\leq
16\dfrac{\lambda^{3}}{N^{3p}}(2d)\dfrac{1}{N}\sum_{i=1}^{N}\left(2\mu^{2}|V_{n}^{i,\lambda,u}|^{2}+2\lambda^{-1}N^{p}\right)+16\dfrac{\lambda^{2}}{N^{2p}}d^{2}$
$\displaystyle\leq 16\cdot
4\dfrac{\lambda^{3}}{N^{3p}}\mu^{2}d|Z^{\lambda,u}_{n}|^{2}+16\cdot
4\dfrac{\lambda^{2}}{N^{2p}}d+16\dfrac{\lambda^{2}}{N^{2p}}d^{2}$
$\displaystyle\leq
16d\mu\dfrac{\lambda^{2}}{N^{2p}}|Z^{\lambda,u}_{n}|^{2}+64\dfrac{\lambda^{2}}{N^{2p}}d+16\dfrac{\lambda^{2}}{N^{2p}}d^{2}.$
(37)

Hence (34) in view of (35),(36) and (37) can be further bounded as

$\displaystyle\mathbb{E}\left[|Z^{\lambda,u}_{n+1}-Z^{\lambda,u}_{n}|^{4}|Z^{\lambda,u}_{n}\right]$
$\displaystyle\leq\mu^{2}\dfrac{\lambda^{2}}{N^{2p}}|Z^{\lambda,u}_{n}|^{4}+18d\mu\dfrac{\lambda^{2}}{N^{2p}}|Z^{\lambda,u}_{n}|^{2}+4\dfrac{\lambda^{2}}{N^{2p}}(4+18d+d^{2})$
$\displaystyle\leq\dfrac{\lambda^{2}}{N^{2p}}\left(\mu^{2}|Z^{\lambda,u}_{n}|^{4}+18d\mu|Z^{\lambda,u}_{n}|^{2}+4(4+18d+d^{2})\right).$

The final result follows by taking the expectation and using the Lemmas 2-3

$\displaystyle\mathbb{E}\left[|Z^{\lambda,u}_{n+1}-Z^{\lambda,u}_{n}|^{4}\right]$
$\displaystyle\leq\dfrac{\lambda^{2}}{N^{2p}}81\left(\mu^{2}C^{2}_{|z_{0}|,\mu,b}+18\mu
C_{|z_{0}|,b,2,\mu}+4\right)(1+d^{\theta}/N+d^{x})^{2}.$

∎

####  A.2.3 Proof of Proposition 5

###### Proof.

Consider the usual split on the difference between the interpolation and the
scaled dynamics and apply the Itô’s formula for $x\to|x|^{2}$

$\displaystyle|\overline{Z}^{\lambda,u}_{t}-{\mathcal{Z}}_{\lambda
t}|^{2}=|\overline{\theta}^{\lambda,u}_{t}-{\vartheta}_{\lambda
t}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|\overline{X}^{i,\lambda,u}_{t}-\mathcal{X}_{\lambda
t}^{i}|^{2}$
$\displaystyle=-2\lambda\int_{0}^{t}\left\langle\dfrac{1}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}_{\lambda,u}(\theta_{\lfloor{t}\rfloor}^{\lambda,u},X^{i,\lambda,u}_{\lfloor{t}\rfloor})-\dfrac{1}{N^{p+1}}\sum_{i=1}^{N}h^{\theta}({\vartheta}_{\lambda
t}^{\lambda},\mathcal{X}_{\lambda
t}^{i}),\overline{{\theta}}_{t}^{\lambda,u}-{\vartheta}_{\lambda
t}\right\rangle ds$
$\displaystyle-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\int_{0}^{t}\left\langle
h^{x}_{\lambda,u}(\theta_{\lfloor{t}\rfloor}^{\lambda,u},X^{i,\lambda,u}_{\lfloor{t}\rfloor})-h^{x}({\vartheta}_{\lambda
t}^{\lambda},\mathcal{X}_{\lambda
t}^{i}),\overline{X}_{t}^{i,\lambda,u}-\mathcal{X}_{\lambda
t}^{i}\right\rangle ds$
$\displaystyle=-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\int_{0}^{t}\langle
h_{\lambda,u}(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(\mathcal{V}_{\lambda
t}^{i}),\overline{V}_{t}^{i,\lambda,u}-\mathcal{V}_{\lambda t}^{i}\rangle ds.$

In order to make the forthcoming calculations more readable we define the
following quantities:

$e^{i}_{t}=\overline{V}_{t}^{i,\lambda,u}-\mathcal{V}_{\lambda t}^{i}\
\text{and}\ e_{t}=\overline{Z}^{\lambda,u}_{t}-\mathcal{Z}_{\lambda t},$

so that it follows: $1/N\sum_{i=1}^{N}|e^{i}_{t}|^{2}=|e_{t}|^{2}.$ Then by
taking the derivative of the above expansion we obtain

$\displaystyle\dfrac{d}{dt}|e_{t}|^{2}$
$\displaystyle=-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h_{\lambda,u}(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(\mathcal{V}_{\lambda
t}^{i}),e_{t}^{i}\rangle$
$\displaystyle=-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h(\overline{V}_{t}^{i,\lambda,u})-h(\mathcal{V}_{\lambda
t}^{i}),e_{t}^{i}\rangle-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h_{\lambda,u}(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(V_{\lfloor{t}\rfloor}^{i,\lambda,u}),e_{t}^{i}\rangle$
$\displaystyle-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(\overline{V}_{t}^{i,\lambda,u}),e_{t}^{i}\rangle$
$\displaystyle:=k_{1}(t)+k_{2}(t)+k_{3}(t).$ (38)

The first term $k_{1}(t)$ is controlled through the convexity of the initial
potential, the taming error $k_{2}(t)$ can be controlled via the properties of
the taming function and the additional moment bounds that we have established,
which are also used to control the discretisation error $k_{3}(t)$. In
particular ,

$\displaystyle k_{1}(t)$
$\displaystyle=-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h(\overline{V}_{t}^{i,\lambda,u})-h(\mathcal{V}_{\lambda
t}^{i}),e_{t}^{i}\rangle$
$\displaystyle\leq-\dfrac{2\lambda\mu}{N^{p+1}}\sum_{i=1}^{N}|e_{t}^{i}|^{2}=-\dfrac{2\lambda\mu}{N^{p}}|e_{t}|^{2}.$
(39)

Using the Cauchy-Schwarz inequality and the $\epsilon$-Young inequality for
$\epsilon=\mu/2$ yields

$\displaystyle k_{2}(t)$
$\displaystyle=-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h_{\lambda,u}(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(V_{\lfloor{t}\rfloor}^{i,\lambda,u}),e_{t}^{i}\rangle$
$\displaystyle\leq\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\left(\dfrac{\mu}{4}|e_{t}^{i}|^{2}+\dfrac{1}{\mu}|h_{\lambda,u}(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(V_{\lfloor{t}\rfloor}^{i,\lambda,u})|^{2}\right)$
$\displaystyle\leq\dfrac{2\lambda\mu}{4N^{p}}|e_{t}|^{2}+\dfrac{2\lambda}{\mu
N^{p+1}}\sum_{i=1}^{N}\dfrac{\lambda
C_{1}}{N^{p}}(1+|V_{\lfloor{t}\rfloor}^{i,\lambda,u}|^{4(\ell+1)})$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}|e_{t}|^{2}+\dfrac{2\lambda^{2}C_{1}}{\mu
N^{2p}}+\dfrac{2\lambda^{2}C_{1}}{\mu
N^{p}}\left(\dfrac{1}{N}\sum_{i=1}^{N}|V_{\lfloor{t}\rfloor}^{i,\lambda,u}|^{2}\right)^{2(\ell+1)}$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}|e_{t}|^{2}+\dfrac{2\lambda^{2}C_{1}}{\mu
N^{p}}+\dfrac{2\lambda^{2}C_{1}}{\mu
N^{p}}|Z_{\lfloor{t}\rfloor}^{\lambda,u}|^{4(\ell+1)}.$ (40)

Note that the constant $C_{1}$ was introduced in Property 2 and that we
choosed $p=2\ell+1$ in order to get an expression with a power of
$|Z_{\lfloor{t}\rfloor}^{\lambda,u}|$. Similarly for the last term we get

$\displaystyle k_{3}(t)$
$\displaystyle=-\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\langle
h(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(\overline{V}_{t}^{i,\lambda,u}),e_{t}^{i}\rangle$
$\displaystyle\leq\dfrac{2\lambda}{N^{p+1}}\sum_{i=1}^{N}\left(\dfrac{\mu}{4}|e_{t}^{i}|^{2}+\dfrac{1}{\mu}|h(V_{\lfloor{t}\rfloor}^{i,\lambda,u})-h(\overline{V}_{t}^{i,\lambda,u})|^{2}\right)$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}|e_{t}|^{2}+\dfrac{2\lambda
C_{2}}{\mu
N^{p+1}}\sum_{i=1}^{N}\left((1+|V_{\lfloor{t}\rfloor}^{i,\lambda,u}|^{2\ell}+|\overline{V}_{t}^{i,\lambda,u}|^{2\ell})|V_{\lfloor{t}\rfloor}^{i,\lambda,u}-\overline{V}_{t}^{i,\lambda,u}|^{2}\right)$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}|e_{t}|^{2}+\dfrac{2\lambda
C_{2}}{\mu
N^{p+1}}\sum_{i=1}^{N}\left(1+|V_{\lfloor{t}\rfloor}^{i,\lambda,u}|^{2\ell}+|\overline{V}_{t}^{i,\lambda,u}|^{2\ell}\right)\sum_{i=1}^{N}|V_{\lfloor{t}\rfloor}^{i,\lambda,u}-\overline{V}_{t}^{i,\lambda,u}|^{2}.$

By reordering, taking the expectation and using the Holder’s inequality we
further get:

$\displaystyle\mathbb{E}\left[k_{3}(t)\right]$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]$
$\displaystyle+\dfrac{2\lambda C_{2}}{\mu
N^{(p+1)/2}}\mathbb{E}\left[\dfrac{1}{N^{\ell}}\sum_{i=1}^{N}\left(1+|V_{\lfloor{t}\rfloor}^{i,\lambda,u}|^{2\ell}+|\overline{V}_{t}^{i,\lambda,u}|^{2\ell}\right)\dfrac{1}{N}\sum_{i=1}^{N}|V_{\lfloor{t}\rfloor}^{i,\lambda,u}-\overline{V}_{t}^{i,\lambda,u}|^{2}\right]$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{2\lambda
C_{2}}{\mu}\mathbb{E}\left[\left(1+|Z_{\lfloor{t}\rfloor}^{\lambda,u}|^{2\ell}+|\overline{Z}_{t}^{\lambda,u}|^{2\ell}\right)|Z_{\lfloor{t}\rfloor}^{\lambda,u}-\overline{Z}_{t}^{\lambda,u}|^{2}\right]$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{6\lambda
C_{2}}{\mu}\mathbb{E}^{1/2}\left[\left(1+|Z_{\lfloor{t}\rfloor}^{\lambda,u}|^{4\ell}+|\overline{Z}_{t}^{\lambda,u}|^{4\ell}\right)\right]\mathbb{E}^{1/2}\left[|Z_{\lfloor{t}\rfloor}^{\lambda,u}-\overline{Z}_{t}^{\lambda,u}|^{4}\right]$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{6\lambda^{2}C_{2}C_{3}}{\mu
N^{p}}\mathbb{E}^{1/2}\left[\left(1+|Z_{\lfloor{t}\rfloor}^{\lambda,u}|^{4\ell}+|\overline{Z}_{t}^{\lambda,u}|^{4\ell}\right)\right](1+d^{\theta}/N+d^{x})$
$\displaystyle\leq\dfrac{\lambda\mu}{2N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{6\lambda^{2}C_{2}C_{3}C_{4}}{\mu
N^{p}}(1+d^{\theta}/N+d^{x})^{\ell+1}.$ (41)

Taking the expectation for the rest of the terms in (39) and (40) respectively
and combining them with (41), one obtains in view of (38),

$\displaystyle\dfrac{d}{dt}\mathbb{E}\left[|e_{t}|^{2}\right]$
$\displaystyle\leq-\dfrac{\mu\lambda}{N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{2\lambda^{2}C_{1}}{\mu
N^{p}}+\dfrac{2\lambda^{2}C_{1}C_{5}}{\mu
N^{p}}(1+d^{\theta}/N+d^{x})^{2(\ell+1)}$
$\displaystyle+\dfrac{6\lambda^{2}C_{2}C_{3}C_{4}}{\mu
N^{p}}(1+d^{\theta}/N+d^{x})^{\ell+1}$
$\displaystyle\leq-\dfrac{\mu\lambda}{N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{\lambda^{2}C_{6}}{\mu
N^{p}}(1+d^{\theta}/N+d^{x})^{2(\ell+1)},$

by multiplying both sides by the integrating factor $e^{\lambda\mu t/N^{p}}$
and rearranging the term one gets

$\displaystyle\dfrac{d}{dt}\left(e^{\lambda\mu
t/N^{p}}\mathbb{E}\left[|e_{t}|^{2}\right]\right)\leq\dfrac{\lambda^{2}C_{6}}{\mu
N^{p}}(1+d^{\theta}/N+d^{x})^{2(\ell+1)}e^{\lambda\mu t/N^{p}}$
$\displaystyle\mathbb{E}\left[|e_{t}|^{2}\right]\leq\lambda\dfrac{C_{6}}{\mu^{2}}(1+d^{\theta}/N+d^{x})^{2(\ell+1)}.$

∎

### A.3 Coordinate-wise tamed scheme - tIPLAc

####  A.3.1 Key quantities for the proof of the main Lemmas.

The following definitions that refer to the rescaled dynamics (17)-(18) of the
algorithm tIPLAc and its continuous time interpolations (19)-(20) are given
as:

$\displaystyle
Z^{\lambda,c}_{n}=\left(\theta_{n+1}^{\lambda,c},N^{-1/2}X_{n+1}^{1,\lambda,c},\ldots,N^{-1/2}X_{n+1}^{N,\lambda,c}\right),$
(42)
$\displaystyle\overline{Z}^{\lambda,c}_{t}=\left(\overline{\theta}^{\lambda,c}_{t},N^{-1/2}\overline{X}^{1,\lambda,c}_{t},\ldots,N^{-1/2}\overline{X}^{N,\lambda,c}_{t}\right),$
(43) $\displaystyle\mathcal{Z}_{\lambda t}^{N}=\left(\vartheta_{\lambda
t}^{N},N^{-1/2}\mathcal{X}_{\lambda
t}^{1,N},\ldots,N^{-1/2}\mathcal{X}_{\lambda t}^{N,N}\right).$ (44)

####  A.3.2 Moment and increment bounds

Lemma 5. Let A1, A3 and A4 hold. Then, for any $0\leq\lambda<1/4\mu$, it holds
that,

$\mathbb{E}\left[|V_{n}^{i,\lambda,c}|^{2}\right]\leq
C_{|z_{0}|,\mu,b}(1+d^{\theta}+d^{x}),$

for a constant $C>0$ independent of $N,n,\lambda,d^{x}\text{ and }d^{\theta}$,
$\forall i\in\\{1,\ldots,N\\}$.

###### Proof.

$\displaystyle\left|V_{n+1}^{i,\lambda,c}\right|^{2}$
$\displaystyle=\left|\theta_{n+1}^{\lambda,c}\right|^{2}+\left|X_{n+1}^{i,\lambda,c}\right|^{2}$
$\displaystyle=\left|\theta_{n}^{\lambda,c}-\dfrac{\lambda}{N}\sum_{i=1}^{N}h^{\theta}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})\right|^{2}+\dfrac{2\lambda}{N}|\xi^{(0)}_{n+1}|^{2}$
$\displaystyle+2\sqrt{\dfrac{2\lambda}{N}}\left\langle\theta_{n}^{\lambda,c}-\dfrac{\lambda}{N}\sum_{i=1}^{N}h^{\theta}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c}),\xi_{n+1}^{(0)}\right\rangle$
$\displaystyle+\left|X_{n}^{i,\lambda,c}-\lambda
h^{x}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})\right|^{2}+2\lambda|\xi^{(i)}_{n+1}|^{2}$
$\displaystyle+2\sqrt{2\lambda}\left\langle X_{n}^{i,\lambda,c}-\lambda
h^{x}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c}),\xi_{n+1}^{(i)}\right\rangle.$

Taking the conditional expectation on both sides with respect to the
filtration generated by $V_{n}^{i,\lambda,c}$ the cross terms are vanishing to
$0$, yielding

$\displaystyle\mathbb{E}\left[\left|V_{n+1}^{i,\lambda,c}\right|^{2}|V_{n}^{i,\lambda,c}\right]$
$\displaystyle=\mathbb{E}\left[\left|\theta_{n}^{\lambda,c}\right|^{2}|V_{n}^{i,\lambda,c}\right]-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\mathbb{E}\left[\langle\theta_{n}^{\lambda,c},h_{\lambda,c}^{\theta}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})\rangle|V_{n}^{i,\lambda,c}\right]$
$\displaystyle+\dfrac{\lambda^{2}}{N^{2}}\mathbb{E}\left[\left|\sum_{i=1}^{N}h_{\lambda,c}^{\theta}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})\right|^{2}|V_{n}^{i,\lambda,c}\right]+\dfrac{2\lambda
d^{\theta}}{N}+\mathbb{E}\left[\left|X_{n}^{i,\lambda,c}\right|^{2}|V_{n}^{i,\lambda,c}\right]$
$\displaystyle-2\lambda\mathbb{E}\left[\langle
X_{n}^{i,\lambda,c},h_{\lambda,c}^{x}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})\rangle|V_{n}^{i,\lambda,c}\right]$
$\displaystyle+{\lambda}^{2}\mathbb{E}\left[|h_{\lambda,c}^{x}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})|^{2}|V_{n}^{i,\lambda,c}\right]+2\lambda
d^{x}.$

Furthermore by using the elementary inequality $(t_{1}+\ldots+t_{m})^{p}\leq
m^{p-1}(t_{1}^{p}+\ldots,t_{m}^{p})$ and the fact that all of the expressions
within the conditional expectations are measurable we obtain

$\displaystyle\mathbb{E}\left[\left|V_{n+1}^{i,\lambda,c}\right|^{2}|V_{n}^{i,\lambda,c}\right]$
$\displaystyle\leq|\theta_{n}^{\lambda,c}|-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle\theta_{n}^{\lambda,c},h_{\lambda,c}^{\theta}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})\rangle+\dfrac{\lambda^{2}}{N}\sum_{i=1}^{N}|h_{\lambda}^{\theta}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})|^{2}$
$\displaystyle+\dfrac{2\lambda
d^{\theta}}{N}+|X_{n}^{i,\lambda,c}|^{2}-2\lambda\langle
X_{n}^{i,\lambda,c},h_{\lambda,c}^{x}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})\rangle$
$\displaystyle+{\lambda}^{2}|h_{\lambda,c}^{x}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})|^{2}+2\lambda
d^{x}$
$\displaystyle\leq\left|V_{n}^{i,\lambda,c}\right|^{2}-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\sum_{j=1}^{d^{\theta}}\theta_{n,j}^{\lambda}\cdot
h^{\theta_{j}}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})$
$\displaystyle+\dfrac{\lambda^{2}}{N}\sum_{i=1}^{N}\sum_{j=1}^{d^{\theta}}|h_{\lambda,c}^{\theta_{j}}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})|^{2}-2\lambda\sum_{j=1}^{d^{x}}X_{n,j}^{i,\lambda}\cdot
h^{x_{j}}_{\lambda,c}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})$
$\displaystyle+\lambda^{2}\sum_{j=1}^{d^{x}}|h_{\lambda,c}^{x_{j}}(\theta_{n}^{\lambda,c},X_{n}^{i,\lambda,c})|^{2}+2\lambda(d^{\theta}/N+d^{x})$
$\displaystyle\leq\left|V_{n}^{i,\lambda,c}\right|^{2}-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\sum_{j=1}^{d^{\theta}}\left(\dfrac{\mu}{2}|\theta^{\lambda,c}_{n,j}|^{2}-b\right)$
$\displaystyle+\dfrac{\lambda^{2}}{N}\sum_{i=1}^{N}\sum_{j=1}^{d^{\theta}}\left(2\mu^{2}|\theta^{\lambda,c}_{n,j}|^{2}+2\lambda^{-1}\right)-2\lambda\sum_{j=1}^{d^{x}}\left(\dfrac{\mu}{2}|X^{i,\lambda,c}_{n,j}|^{2}-b\right)$
$\displaystyle+\lambda^{2}\sum_{j=1}^{d^{x}}\left(2\mu^{2}|X^{i,\lambda,c}_{n,j}|^{2}+2\lambda^{-1}\right)+2\lambda(d^{\theta}/N+d^{x}).$

Executing the summations yields

$\displaystyle\mathbb{E}\left[\left|V_{n+1}^{i,\lambda,c}\right|^{2}|V_{n}^{i,\lambda,c}\right]$
$\displaystyle\leq\left|V_{n}^{i,\lambda,c}\right|^{2}-\lambda\mu|\theta^{\lambda,c}_{n}|^{2}-\lambda\mu|X^{i,\lambda,c}_{n}|^{2}+2\lambda^{2}\mu^{2}|\theta^{\lambda,c}_{n}|^{2}+2\lambda^{2}\mu^{2}|X^{i,\lambda,c}_{n}|^{2}$
$\displaystyle+2\lambda
b(d^{\theta}+d^{x})+2\lambda(d^{\theta}+d^{x})+2\lambda(d^{\theta}/N+d^{x})$
$\displaystyle\leq(1-\lambda\mu+2\lambda^{2}\mu^{2})\left|V_{n}^{i,\lambda,c}\right|^{2}+2\lambda(b+1)(d^{\theta}+d^{x})+2\lambda(d^{\theta}/N+d^{x}).$

Consider the restriction $\lambda\mu<1/4$, then this implies that

$\displaystyle\mathbb{E}\left[\left|V_{n+1}^{i,\lambda,c}\right|^{2}\right]$
$\displaystyle\leq(1-\lambda\mu/2)\mathbb{E}\left[\left|V_{n}^{i,\lambda,c}\right|^{2}\right]+2\lambda(b+1)(d^{\theta}+d^{x})+2\lambda(d^{\theta}/N+d^{x}).$

Iterating the above bound finally yields

$\displaystyle\mathbb{E}\left[\left|V_{n+1}^{i,\lambda,c}\right|^{2}\right]$
$\displaystyle\leq(1-\lambda\mu/2)^{n}\left|V_{0}^{i,\lambda,c}\right|^{2}+\dfrac{1-(1-\lambda\mu/2)^{n}}{\lambda\mu/2}4\lambda(b+1)(d^{\theta}+d^{x})$
$\displaystyle\leq\left(\left|V_{0}\right|^{2}+(8/\mu)(b+1)\right)(1+d^{\theta}+d^{x}).$

∎

Lemma 6. Let A1, A3 and A4 hold. Then, for any $0\leq\lambda<1/4\mu$ and
$q\in[2,2\ell+1)\cap\mathbb{N}$, it holds that,

$\mathbb{E}\left[|V_{n}^{i,\lambda,c}|^{2q}\right]\leq
C_{|z_{0}|,\mu,b,p}(1+d^{\theta}+d^{x})^{q},$

for a constant $C>0$ independent of $N,n,\lambda,d^{x}\text{ and }d^{\theta}$,
$\forall i\in\\{1,\ldots,d^{\theta}+Nd^{x}\\}.$

###### Proof.

Similarly as in the proof of Lemma 3 we define the following auxiliary
quantities:

$\displaystyle
A_{n}^{\lambda}:=|\Delta_{n}^{\lambda,\theta}|^{2}+|\Delta_{n}^{\lambda,i,x}|^{2},$
$\displaystyle
B^{\lambda}_{n}=2\langle\Delta_{n}^{\theta},G_{n}^{\lambda,\theta}\rangle+2\langle\Delta_{n}^{\lambda,x,i},G_{n}^{\lambda,x,i}\rangle+|G_{n}^{\lambda,\theta}|^{2}+|G_{n}^{\lambda,x,i}|^{2}.$

Now for the 2q-th Moment one writes:

$\displaystyle|V^{i,\lambda,c}_{n+1}|^{2q}$
$\displaystyle=\left(A_{n}^{\lambda}+B^{\lambda}_{n}\right)^{q}$
$\displaystyle\leq(A_{n}^{\lambda})^{q}+2q(A_{n}^{\lambda})^{q-1}B^{\lambda}_{n}+\sum_{k=2}^{q}\binom{q}{k}|A_{n}^{\lambda}|^{q-k}|B^{\lambda}_{n}|^{k}$
(45)

We shall deal with each term separately,

$\displaystyle\mathbb{E}\left[|A_{n}^{\lambda}|^{q}|Z_{n}^{\lambda}\right]$
$\displaystyle=(A_{n}^{\lambda})^{q}\leq\left((1-\lambda\mu/2)\left|V_{n}^{i,\lambda,c}\right|^{2}+2\lambda(b+1)(d^{\theta}+d^{x})\right)^{q}$
$\displaystyle\leq\left(1+\dfrac{\lambda\mu}{4}\right)^{q-1}\left(1-\dfrac{\lambda\mu}{2}\right)^{q}|V^{i,\lambda,c}_{n}|^{2q}+\left(1+\dfrac{4}{\lambda\mu}\right)^{q-1}2^{q}{\lambda}^{q}C^{q}$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{4}\right)^{q-1}\left(1-\dfrac{\lambda\mu}{2}\right)|V^{i,\lambda,c}_{n}|^{2q}+\left(\lambda+\dfrac{4}{\mu}\right)^{q-1}\lambda(2C)^{q}$
$\displaystyle\leq r_{q}^{\lambda}|V^{i,\lambda,c}_{n}|^{2q}+w_{q}^{\lambda}.$
(46)

where $r_{q}^{\lambda}=(1-\lambda\mu/4)^{q-1}(1-\lambda\mu/2)$ and
$w_{q}^{\lambda}=(\lambda+4/\mu)^{q-1}\lambda(2C)^{q}$.
On a similar note,

$\displaystyle\mathbb{E}\left[2q(A_{n}^{\lambda})^{q-1}B^{\lambda}_{n}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle=2q(A_{n}^{\lambda})^{q-1}\mathbb{E}\left[B^{\lambda}_{n}|Z_{n}^{\lambda}\right]=4q\lambda\left(d^{\theta}/N+d^{x}\right)(A_{n}^{\lambda})^{q-1}$
$\displaystyle\leq
4q\lambda\left(d^{\theta}/N+d^{x}\right)\left(r_{q-1}^{\lambda}|V^{i,\lambda,c}_{n}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle\leq 4q\left(2\lambda
d\right)\left(r_{q-1}^{\lambda}|V^{i,\lambda,c}_{n}|^{2(q-1)}+w_{q-1}^{\lambda}\right).$
(47)

The 3rd term on (45) can be further expanded to

$\displaystyle\sum_{k=2}^{q}\binom{q}{k}|A_{n}^{\lambda}|^{q-k}|B^{\lambda}_{n}|^{k}$
$\displaystyle=\sum_{\ell=0}^{q-2}\binom{q}{\ell+2}|A_{n}^{\lambda}|^{q-2-k}|B^{\lambda}_{n}|^{k+2}$
$\displaystyle=\dfrac{q}{\ell+2}\dfrac{q-1}{\ell+1}\sum_{\ell=0}^{q-2}\binom{q}{\ell}|A_{n}^{\lambda}|^{q-2-k}|B^{\lambda}_{n}|^{k}|B^{\lambda}_{n}|^{2}$
$\displaystyle\leq
q(q-1)\left(|A_{n}^{\lambda}|+|B_{n}^{\lambda}|\right)^{q-2}|B^{\lambda}_{n}|^{2}$
$\displaystyle\leq
q(q-1)2^{q-3}|A_{n}^{\lambda}|^{q-2}|B_{n}^{\lambda}|^{2}+q(q-1)2^{q-3}|B_{n}^{\lambda}|^{q}.$
$\displaystyle=D+F$ (48)

Taking the expectation of the above terms yields

$\displaystyle\mathbb{E}[D|V^{i,\lambda,c}_{n}]$
$\displaystyle=q(q-1)2^{q-3}|A_{n}^{\lambda}|^{q-2}\mathbb{E}\left[|B_{n}^{\lambda}|^{2}|V^{i,\lambda,c}_{n}\right],$

where

$\displaystyle\mathbb{E}\left[|B_{n}^{\lambda}|^{2}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle=\mathbb{E}\left[\left|2\langle\Delta_{n}^{\theta},G_{n}^{\lambda,\theta}\rangle+2\langle\Delta_{n}^{\lambda,x,i},G_{n}^{\lambda,x,i}\rangle+|G_{n}^{\lambda,\theta}|^{2}+|G_{n}^{\lambda,x,i}|^{2}\right|^{2}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle\leq
4\left(\mathbb{E}\left[4|\Delta_{n}^{\lambda,\theta}|^{2}|G_{n}^{\lambda,\theta}|^{2}|V^{i,\lambda,c}_{n}\right]+\mathbb{E}\left[4|\Delta_{n}^{\lambda,x,i}|^{2}|G_{n}^{\lambda,x,i}|^{2}|V^{i,\lambda,c}_{n}\right]\right.$
$\displaystyle+\left.\mathbb{E}\left[|G_{n}^{\lambda,x,i}|^{4}|V^{i,\lambda,c}_{n}\right]+\mathbb{E}\left[|G_{n}^{\lambda,x,i}|^{4}|V^{i,\lambda,c}_{n}\right]\right)$
$\displaystyle\leq 4\left(4(2\lambda
d^{\theta}/N)|\Delta_{n}^{\lambda,\theta}|^{2}+4(2\lambda
d^{x})|\Delta_{n}^{\lambda,x,i}|^{2}+12(2\lambda d^{\theta}/N)^{2}+12(2\lambda
d^{x})^{2}\right)$ $\displaystyle\leq 16(2\lambda
d)\left(|\Delta_{n}^{\lambda,\theta}|^{2}+|\Delta_{n}^{\lambda,x,i}|^{2}\right)+96(2\lambda
d)^{2}$ $\displaystyle\leq 16(2\lambda d)A_{n}^{\lambda}+96(2\lambda d)^{2}.$

with $d=\max\\{d^{\theta}/N,d^{x}\\}$. plugging in this result for the term D
we get

$\displaystyle\mathbb{E}[D|V^{i,\lambda,c}_{n}]$
$\displaystyle=q(q-1)2^{q-3}|A_{n}^{\lambda}|^{q-2}\left(16(2\lambda
d)A_{n}^{\lambda}+96(2\lambda d)^{2}\right)$ $\displaystyle=16(2\lambda
d)q(q-1)2^{q-3}\left(r_{q-1}^{\lambda}|V_{n}^{\lambda}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle+96(2\lambda
d)^{2}q(q-1)2^{q-3}\left(r_{q-2}^{\lambda}|V_{n}^{\lambda}|^{2(q-2)}+w_{q-2}^{\lambda}\right).$

Finally,

$\displaystyle\mathbb{E}[F|V^{i,\lambda,c}_{n}]$
$\displaystyle=q(q-1)2^{q-3}|B_{n}^{\lambda}|^{q}$
$\displaystyle=q(q-1)2^{q-3}\mathbb{E}\left[\left|2\langle\Delta_{n}^{\theta},G_{n}^{\lambda,\theta}\rangle+2\langle\Delta_{n}^{\lambda,x,i},G_{n}^{\lambda,x,i}\rangle+|G_{n}^{\lambda,\theta}|^{2}+|G_{n}^{\lambda,x,i}|^{2}\right|^{q}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle\leq
q(q-1)2^{q-3}4^{q-1}\left(\mathbb{E}\left[2^{q}|\Delta_{n}^{\lambda,\theta}|^{q}|G_{n}^{\lambda,\theta}|^{q}|V^{i,\lambda,c}_{n}\right]+\mathbb{E}\left[2^{q}|\Delta_{n}^{\lambda,x,i}|^{q}|G_{n}^{\lambda,x,i}|^{q}|V^{i,\lambda,c}_{n}\right]\right.$
$\displaystyle\left.+\mathbb{E}\left[|G_{n}^{\lambda,\theta}|^{2q}|V^{i,\lambda,c}_{n}\right]+\mathbb{E}\left[|G_{n}^{\lambda,x,i}|^{2q}|V^{i,\lambda,c}_{n}\right]\right)$
$\displaystyle\leq q(q-1)2^{3q-5}\left(2^{q}(2\lambda
d^{\theta}/N)^{q/2}q!!\left(|\Delta_{n}^{\lambda,\theta}|^{2}\right)^{q/2}+2^{q}q!!(2\lambda
d^{x})^{q/2}\left(|\Delta_{n}^{\lambda,x,i}|^{2}\right)^{q/2}\right.$
$\displaystyle\left.+(2q)!!(2\lambda d^{\theta}/N)^{q}+(2q)!!(2\lambda
d^{x})^{q}\right)$ $\displaystyle\leq(q(q-1))^{q+1}2^{4q-5}\left((2\lambda
d)^{q/2}\left(|\Delta_{n}^{\lambda,\theta}|^{2}+|\Delta_{n}^{\lambda,x,i}|^{2}\right)^{q/2}+2(2\lambda
d)^{q}\right)$ $\displaystyle\leq(q(q-1))^{q+1}2^{4q-5}\left((2\lambda
d)^{q/2}\left|A_{n}^{\lambda}\right|^{q/2}+2(2\lambda d)^{q}\right)$
$\displaystyle\leq(q(q-1))^{q+1}2^{4q-5}(2\lambda
d)^{q/2}\left(r^{\lambda}_{q/2}|V_{n}^{\lambda}|^{q}+w^{\lambda}_{q/2}\right)$
$\displaystyle+(q(q-1))^{q+1}2^{4q-5}2(2\lambda d)^{q}.$

Combining the above results in view of (48) yields

$\displaystyle\mathbb{E}\left[\sum_{k=2}^{q}\binom{q}{k}|A_{n}^{\lambda}|^{q-k}|B^{\lambda}_{n}|^{k}\right]$
$\displaystyle\leq 16(2\lambda
d)q(q-1)2^{q-3}\left(r_{q-1}^{\lambda}|V_{n}^{\lambda}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle+96(2\lambda
d)^{2}q(q-1)2^{q-3}\left(r_{q-2}^{\lambda}|V_{n}^{\lambda}|^{2(q-2)}+w_{q-2}^{\lambda}\right)$
$\displaystyle+(q(q-1))^{q+1}2^{4q-5}(2\lambda
d)^{q/2}\left(r^{\lambda}_{q/2}|V_{n}^{\lambda}|^{q}+w^{\lambda}_{q/2}\right)$
$\displaystyle+(q(q-1))^{q+1}2^{4q-4}(2\lambda d)^{q}.$ (49)

Substituting (46),(47) and (49) into (45) also yields

$\displaystyle\mathbb{E}\left[|V^{i,\lambda,c}_{n+1}|^{2q}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle\leq r_{q}^{\lambda}|V^{i,\lambda,c}_{n}|^{2q}+w_{q}^{\lambda}$
$\displaystyle+4q\left(2\lambda
d\right)(1+4(q-1)2^{q-3})\left(r_{q-1}^{\lambda}|V^{i,\lambda,c}_{n}|^{2(q-1)}+w_{q-1}^{\lambda}\right)$
$\displaystyle+96(2\lambda
d)^{2}q(q-1)2^{q-3}\left(r_{q-2}^{\lambda}|V^{i,\lambda,c}_{n}|^{2(q-2)}+w_{q-2}^{\lambda}\right)$
$\displaystyle+(q(q-1))^{q+1}2^{4q-5}(2\lambda
d)^{q/2}\left(r^{\lambda}_{q/2}|V^{i,\lambda,c}_{n}|^{q}+w^{\lambda}_{q/2}\right)$
$\displaystyle+(q(q-1))^{q+1}2^{4q-4}(2\lambda d)^{q}.$ (50)

Consider
$|V^{i,\lambda,c}_{n}|\geq\sqrt{\dfrac{2d}{\mu/4}}\left\\{2(q(q-1))^{q+1}2^{4(q-1)}\right\\}^{1/2}\geq\sqrt{\dfrac{2d}{\mu/4}}\left\\{2(q(q-1))^{q+1}2^{4(q-1)}\right\\}^{1/q}$.
Then one observes

$\displaystyle\mathbb{E}\left[|V^{i,\lambda,c}_{n+1}|^{2q}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle\leq
r_{q}^{\lambda}|V_{n}^{i,\lambda,c}|^{2q}+w_{q}^{\lambda}+\dfrac{\lambda\mu}{2\cdot
4}|V^{i,\lambda,c}_{n}|^{2}\left(r_{q-1}^{\lambda}|V^{i,\lambda,c}_{n}|^{2(q-1)}\right)$
$\displaystyle+\dfrac{(\lambda\mu)^{2}}{2\cdot
4^{2}}|V^{i,\lambda,c}_{n}|^{4}\left(r_{q-2}^{\lambda}|V^{i,\lambda,c}_{n}|^{2(q-2)}\right)$
$\displaystyle+\dfrac{(\lambda\mu)^{q/2}}{2\cdot
4^{q/2}}|V_{n}^{i,\lambda,c}|^{q}\left(r^{\lambda}_{q/2}|V_{n}^{i,\lambda,c}|^{q}\right)+(q(q-1))^{q+1}2^{4(q-1)}(2\lambda
d)^{q}$ $\displaystyle+(q(q-1))^{q+1}2^{4(q-1)}\left((2\lambda
d)w_{q-1}^{\lambda}+(2\lambda d)^{2}w_{q-2}^{\lambda}+(2\lambda
d)^{q/2}w_{q/2}^{\lambda}\right)$ $\displaystyle\leq
r^{\lambda}_{q/2}\left(\left(1-\dfrac{\lambda\mu}{4}\right)^{q/2}+\dfrac{1}{2}\left(\dfrac{\lambda\mu}{4}\right)\left(1-\dfrac{\lambda\mu}{4}\right)^{q/2-1}\right.$
$\displaystyle\left.+\dfrac{1}{2}\left(\dfrac{\lambda\mu}{4}\right)^{2}\left(1-\dfrac{\lambda\mu}{4}\right)^{q/2-2}+\dfrac{1}{2}\left(\dfrac{\lambda\mu}{4}\right)^{q/2}\right)|V_{n}^{i,\lambda,c}|^{2q}$
$\displaystyle+w_{q}^{\lambda}+(q(q-1))^{q+1}2^{4(q-1)}(2\lambda d)^{q}$
$\displaystyle+(q(q-1))^{q+1}2^{4(q-1)}\left((2\lambda
d)w_{q-1}^{\lambda}+(2\lambda d)^{2}w_{q-2}^{\lambda}+(2\lambda
d)^{q/2}w_{q/2}^{\lambda}\right).$

Using the fact that $\lambda\mu\leq 1$ we get

$\displaystyle\mathbb{E}\left[|V^{i,\lambda,c}_{n+1}|^{2q}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)|V^{i,\lambda,c}_{n}|^{2q}$
$\displaystyle+w_{q}^{\lambda}+(q(q-1))^{q+1}2^{4(q-1)}(2\lambda d)^{q}$
$\displaystyle+(q(q-1))^{q+1}2^{4(q-1)}\left((2\lambda
d)w_{q-1}^{\lambda}+(2\lambda d)^{2}w_{q-2}^{\lambda}+(2\lambda
d)^{q/2}w_{q/2}^{\lambda}\right)$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)|V^{i,\lambda,c}_{n}|^{2q}+(q(q-1))^{q+1}2^{4(q-1)}10^{q}C^{q}(d\lambda)^{q}.$

Consequently, on
$\\{|V^{i,\lambda,c}_{n}|\leq\sqrt{\dfrac{2d}{\mu/4}}\left\\{2(q(q-1))^{q+1}2^{4(q-1)}\right\\}^{1/2}\\}$,
we get

$\displaystyle\mathbb{E}\left[|V^{i,\lambda,c}_{n+1}|^{2q}|V^{i,\lambda,c}_{n}\right]$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)|V^{i,\lambda,c}_{n}|^{2q}$
$\displaystyle+(q(q-1))^{q+1}2^{4(q-1)}10^{q}C^{q}(d\lambda)^{q}$
$\displaystyle+3\left(q(q-1)^{q+1}2^{4(q-1)}\right)^{q}(2d)^{q}\left(\dfrac{4}{\mu}\right)^{q}2^{q-1}.$

Hence the bound in (50) is further improved to

$\displaystyle\mathbb{E}\left[|V^{i,\lambda,c}_{n+1}|^{2q}\right]$
$\displaystyle\leq\left(1-\dfrac{\lambda\mu}{2}\right)\mathbb{E}\left[|V^{i,\lambda,c}_{n}|^{2q}\right]+C(d,q,\lambda,\mu),$

and the usual bound for recursive sequences follows. ∎

####  A.3.3 Proof of Proposition 6

###### Proof.

Consider the usual split on the difference between the interpolation and the
scaled dynamics and apply the Itô’s formula for $x\to|x|^{2}$

$\displaystyle|\overline{Z}^{\lambda,c}_{t}-\mathcal{Z}_{\lambda
t}|^{2}=|\overline{\theta}^{\lambda,c}_{t}-{\vartheta}_{\lambda
t}|^{2}+\dfrac{1}{N}\sum_{i=1}^{N}|\overline{X}^{i,\lambda,c}_{t}-\mathcal{X}_{\lambda
t}^{i}|^{2}$
$\displaystyle=-2\lambda\int_{0}^{t}\left\langle\dfrac{1}{N}\sum_{i=1}^{N}h^{\theta}_{\lambda,c}(\theta_{\lfloor{t}\rfloor}^{\lambda,c},X^{i,\lambda,c}_{\lfloor{t}\rfloor})-\dfrac{1}{N}\sum_{i=1}^{N}h^{\theta}({\vartheta}_{\lambda
t}^{\lambda},\mathcal{X}_{\lambda
t}^{i}),\overline{{\theta}}_{t}^{i,\lambda,c}-{\vartheta}_{\lambda
t}^{i}\right\rangle ds$
$\displaystyle-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\int_{0}^{t}\left\langle
h^{x}_{\lambda,c}(\theta_{\lfloor{t}\rfloor}^{\lambda,c},X^{i,\lambda,c}_{\lfloor{t}\rfloor})-h^{x}({\vartheta}_{\lambda
t}^{\lambda},\mathcal{X}_{\lambda
t}^{i}),\overline{X}_{t}^{i,\lambda,c}-\mathcal{X}_{\lambda
t}^{i}\right\rangle ds$
$\displaystyle=-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\int_{0}^{t}\langle
h_{\lambda,c}(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(\mathcal{V}_{\lambda
t}^{i}),\overline{V}_{t}^{i,\lambda,c}-\mathcal{V}_{\lambda t}^{i}\rangle ds.$

In order to make the forthcoming calculations more readable we define the
following quantities:

$e^{i}_{t}=\overline{V}_{t}^{i,\lambda,c}-\mathcal{V}_{\lambda t}^{i}\
\text{and}\ e_{t}=\overline{Z}^{\lambda,c}_{t}-\mathcal{Z}_{\lambda t},$

so that it follows: $1/N\sum_{i=1}^{N}|e^{i}_{t}|^{2}=|e_{t}|^{2}.$
Then by taking the derivative of the above expansion we obtain

$\displaystyle\dfrac{d}{dt}|e_{t}|^{2}$
$\displaystyle=-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle
h_{\lambda,c}(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(\mathcal{V}_{\lambda
t}^{i}),e_{t}^{i}\rangle$
$\displaystyle=-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle
h(\overline{V}_{t}^{i,\lambda,c})-h(\mathcal{V}_{\lambda
t}^{i}),e_{t}^{i}\rangle$
$\displaystyle-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle
h_{\lambda,c}(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(V_{\lfloor{t}\rfloor}^{i,\lambda,c}),e_{t}^{i}\rangle$
$\displaystyle-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle
h(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(\overline{V}_{t}^{i,\lambda,c}),e_{t}^{i}\rangle$
$\displaystyle=k_{1}(t)+k_{2}(t)+k_{3}(t).$ (51)

The first term $k_{1}(t)$ is controlled through the convexity of the initial
potential, the taming error $k_{2}(t)$ can be controlled via the properties of
the taming function and the additional moment bounds that we have established,
which are also used to control the discretisation error $k_{3}(t)$. In
particular,

$\displaystyle k_{1}(t)$
$\displaystyle=-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle
h(\overline{V}_{t}^{i,\lambda,c})-h(\mathcal{V}_{\lambda
t}^{i,\lambda}),e_{t}^{i}\rangle$
$\displaystyle\leq-\dfrac{2\lambda\mu}{N}\sum_{i=1}^{N}|e_{t}^{i}|^{2}=-2\lambda\mu|e_{t}|^{2}.$
(52)

Using the Cauchy-Schwarz inequality and the $\epsilon$-Young inequality for
$\epsilon=\mu/2$ yields

$\displaystyle k_{2}(t)$
$\displaystyle=-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle
h_{\lambda,c}(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(V_{\lfloor{t}\rfloor}^{i,\lambda,c}),e_{t}^{i}\rangle$
$\displaystyle\leq\dfrac{2\lambda}{N}\sum_{i=1}^{N}\left(\dfrac{1}{\mu}|h_{\lambda,c}(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(V_{\lfloor{t}\rfloor}^{i,\lambda,c})|^{2}+\dfrac{\mu}{4}|e_{t}^{i}|^{2}\right)$
$\displaystyle\leq 2\dfrac{\lambda\mu}{4}|e_{t}|^{2}+\dfrac{2\lambda}{\mu
N}\sum_{i=1}^{N}2\lambda
C_{2}^{2}(1+|V_{\lfloor{t}\rfloor}^{i,\lambda,c}|^{4(\ell+1)})$
$\displaystyle\leq
2\dfrac{\lambda\mu}{4}|e_{t}|^{2}+\dfrac{4\lambda^{2}}{\mu}C_{2}^{2}+\dfrac{4\lambda^{2}}{\mu
N}\sum_{i=1}^{N}|V_{\lfloor{t}\rfloor}^{i,\lambda,c}|^{4(\ell+1)}.$ (53)

Similarly for the last term we get

$\displaystyle k_{3}(t)$
$\displaystyle=-\dfrac{2\lambda}{N}\sum_{i=1}^{N}\langle
h(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(\overline{V}_{t}^{i,\lambda,c}),e_{t}^{i}\rangle$
$\displaystyle\leq\dfrac{2\lambda}{N}\sum_{i=1}^{N}\left(\dfrac{1}{\mu}|h(V_{\lfloor{t}\rfloor}^{i,\lambda,c})-h(\overline{V}_{t}^{i,\lambda,c})|^{2}+\dfrac{\mu}{4}|e_{t}^{i}|^{2}\right)$
$\displaystyle\leq 2\dfrac{\lambda\mu}{4}|e_{t}|^{2}+\dfrac{2\lambda
L^{2}}{\mu
N}\sum_{i=1}^{N}\left((1+|V_{\lfloor{t}\rfloor}^{i,\lambda,c}|^{\ell}+|\overline{V}_{t}^{i,\lambda,c}|^{\ell})^{2}|V_{\lfloor{t}\rfloor}^{i,\lambda,c}-\overline{V}_{t}^{i,\lambda,c}|^{2}\right).$
(54)

By combining the above bounds (52),(53) and (54) in view of (51) and taking
the expectation, one obtains

$\displaystyle\dfrac{d}{dt}\mathbb{E}\left[|e_{t}|^{2}\right]$
$\displaystyle\leq-\mu\lambda\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{4\lambda^{2}C_{2}^{2}}{\mu}+\dfrac{4\lambda^{2}}{\mu}\dfrac{1}{N}\sum_{i=1}^{N}\mathbb{E}\left[|V_{\lfloor{t}\rfloor}^{i,\lambda,c}|^{4(\ell+1)}\right]$
$\displaystyle+\dfrac{2\lambda
L^{2}}{\mu}\dfrac{1}{N}\sum_{i=1}^{N}\mathbb{E}^{1/2}\left[(1+|V_{\lfloor{t}\rfloor}^{i,\lambda,c}|^{\ell}+|\overline{V}_{t}^{i,\lambda,c}|^{\ell})^{4}\right]\mathbb{E}^{1/2}\left[|V_{\lfloor{t}\rfloor}^{i,\lambda,c}-\overline{V}_{t}^{i,\lambda,c}|^{4}\right]$
$\displaystyle\leq-\mu\lambda\mathbb{E}\left[|e_{t}|^{2}\right]+\dfrac{4\lambda^{2}(C_{2}^{2}+C_{3})}{\mu}$
$\displaystyle+\dfrac{2\lambda
L^{2}}{\mu}\left(6\max\\{1,2C_{4}\\}\right)\lambda\left(\mu
C_{5}+6\sqrt{d\mu}C_{6}+10d+2\right)$
$\displaystyle\leq-\mu\lambda\mathbb{E}\left[|e_{t}|^{2}\right]+\lambda^{2}M(L,\mu,d,\ell)\leq\lambda\left(-\dfrac{\mu}{2}\mathbb{E}\left[|e_{t}|^{2}\right]+\lambda
M\right).$

by multiplying both sides by the integrating factor $e^{\lambda\mu t/2}$ and
rearranging the term one gets

$\displaystyle\dfrac{d}{dt}\left(e^{\lambda\mu
t/2}\mathbb{E}\left[|e_{t}|^{2}\right]\right)\leq\lambda^{2}Me^{\lambda\mu
t/2}$
$\displaystyle\mathbb{E}\left[|e_{t}|^{2}\right]\leq\lambda\dfrac{2M}{\mu}.$

∎