# Motion of Poly-vectors in Fractal Clifford Spaces

###### Abstract

Fractal Clifford Spaces (FCS) may be considered as a challenging approach to
the unification of micro-physics and macro-physics. Trajectories of these
manifolds are described by different poly-vectors describing paths and their
deviation equations for not only to test particles but also for charged,
rotating and rotating charged objects. Due to relaxing the condition of
differentiability for all extended objects in the manifold. In this approach,
a specialized Bazanski Lagrangian will be discussed to describe the fractal
version of extended objects in a fractal space-time.

Magd E. Kahil111 Faculty of Engineering, Modern Sciences and Arts University,
Giza, Egypt
e.mail<EMAIL_ADDRESS>
##  1 Introduction

The problem of unification has to be revisited every now and then, providing a
new description of the structure of space-time such as replacing point like by
p-branes as described within Clifford space, the building blocks of this
manifolds is based on poly-vectors rather than vectors [1] and types of
differentiability involving fractals [2]. This may lead many authors [3] to
propose some aspects that play a vital role to present it.

Following this track, it is worth mentioning to examine motion of poly-vectors
as defined in a fractal Clifford Space as an approach towards unification of
micro and macro physics. This farfetched dream may require thinking outside
the box in the realm of relaxing the usual common assumptions that keep each
field to be treated independently. In our study we are going to describe a
fractal space time which is continuous but not differentiable. This step may
give rise to widen the scope of work to expect not only one geodesic between
two points but infinite geodesics [4]. This would lead to consider two
different types of differentiation, forward and backward. Such an ensemble
makes to describe velocity in a complex space [5].

The paper is of concerned with revisiting the notation of path and path
deviation equations fractal and their counterparts of spinning and spinning
deviation objects in a specific type of Clifford space describing gravity
using fractal derivatives in order to observe their associated manifold such
is continuous and not differentiable.

###  1.1 Fractal Space-times: A brief Introduction

Trajectories of nature are fractal, the necessity, we study curves that are
continuous but not differentiable, and this may give rise to define two types
of differentiations forward and backward. This makes the number of geodesics
infinite between two arbitrary points in that prescribed space. A reliable
approach to understand nature is based on systems that are continuous but not
differentiable. One can demonstrate that the length of continuous and nowhere
differentiable curve is dependent on resolution $\epsilon\rightarrow 0$ and
further that ${\it{L}}(\epsilon)\rightarrow\infty$ when $\epsilon\rightarrow
0$, the curve is fractal (Nottale 1996). Scale relativity theory (SRT) is an
approach to revisit quantum mechanics from different domains of scale laws
ranging from microscopic to macroscopic. If the particle is subject to the
Brownian motion of unknown origin which is described by two Winer Processes
(forward - backward), when they combine together they yield the complex nature
of the wave function and they perform Newton’s equation of motion into the
Schroedinger equation [5,6].

Nature needs to be revisited from different perspectives, due to sudden
discrepancies to understand the cause of striking problems in cosmology like
dark energy and in dark . This may led many scientists to become skeptic in
all methods of computation and detection. Yet, one of good approaches to start
revisiting the laws of nature from their building blocks. In other words,
relaxing conditions that are fundamentally crucial in their constructions.
This may led us to derive equations of motion for objects described not only
by points but also as p-branes, in a space not differentiable but fractal. So,
examining the behavior of particles spinning, charged or even both in fractal
space-time, whose trajectories are continuous but not differentiable.

Such a confinement may be in need to introduce its underlying mechanism which
enable us to describe them. We also express the corresponding Bazanski method
which may be represented in this realm.

###  1.2  Fractal Geometry: An overview

Agop et al [7]that point out in Notalle’s model of scale relativity, it is
supposed that micro particles are taking place on fractal curves with fractal
dimensions $D_{F}$ defining a manifold of fractal space time. This one has the
following characteristic behavior: The number of geodesics is infinite between
two arbitrary points; a statistical (fluid-like) description was accepted.
Each geodesic is considered a fractal curve [7].
The breaking of differential time reflection invariance. The case on non-
differentiability gives rise to define the following functions:

$\frac{df_{+}}{dt}=\lim_{\Delta t\rightarrow 0^{+}}\frac{f(t+\Delta t,\Delta
t)-f(t,\Delta t)}{\Delta t}$ (1)

and

$\frac{df_{-}}{dt}=\lim_{\Delta t\rightarrow 0^{-}}\frac{f(t,\Delta
t)-f(t-\Delta t,\Delta t)}{\Delta t}$ (2)

According to scale relativity it implies to the following relations:

$dX_{+}=v_{+}dt+d\xi_{+},$ (3)

and

$dX_{-}=v_{-}dt+d\xi_{-}$ (4)

where $v_{+}$ and $v_{-}$ are forward and backward mean velocities such that

$v_{+}=\frac{dX_{+}}{dt}=\lim_{\Delta t\rightarrow 0^{+}}<\frac{X(t+\Delta
t)-X(t)}{\Delta t}>,$ (5)

and

$v_{-}=\frac{dX_{-}}{dt}=\lim_{\Delta t\rightarrow
0^{-}}<\frac{X(t)-\Delta(t-\Delta t)}{\Delta t}>,$ (6)

such as $d\xi_{\pm}$ is the measure of non-differentiability with properties

$<d\xi_{\pm}>=0.$ (7)

If $\frac{(v_{+}+v_{-})}{2}$ is consider a differentiable speed , then
$\frac{(v_{+}-v_{-})}{2}$ is counted to be a non-differentiable speed, such
that $d_{\pm}V=dV_{\pm}$.
Agop et al (2006) described the associated fractal covariant derivative in the
following way []:

$\bar{\Gamma}^{\lambda}_{\mu\nu}=\frac{1}{2}\tilde{g}^{\lambda\gamma}(g_{\nu\gamma,\mu}+\tilde{g}_{\gamma\mu,\nu}-g_{\mu\nu,\gamma})+\frac{1}{2}\tilde{g}^{\lambda\gamma}(C^{\rho}_{\mu\nu}\tilde{g}_{\rho\gamma}+C^{\rho}_{\gamma\nu}\tilde{g}_{\rho\nu}-C^{\rho}_{\nu\gamma}\tilde{g}_{\rho\mu})$
(8)

where $\bar{\Gamma}^{\lambda}_{\mu\nu}$ is the affine connection, $g_{\mu\nu}$
the metric tensor, and $C^{\lambda}_{\mu\nu}$ the structure constants defined
by the non-commutativity of the partial derivative $\bar{\partial}_{\rho}$
i.e.

$[\bar{\partial}_{\mu},\bar{\partial}_{\nu}]=C^{\lambda}_{\mu\nu}\partial_{\lambda}$
(9)

Thus, the fractal covariant derivative of a vector $X^{\mu}$ is defined as
follows:

$\bar{\nabla}_{\rho}X^{\mu}=\bar{\partial}_{\rho}X^{\mu}+\bar{\Gamma}^{\mu}_{\delta\rho}X^{\delta}.$
(10)

Thus, the commutation of two covariant derivatives becomes

$[\bar{\nabla}_{\mu},\bar{\nabla}_{\nu}]=C^{\lambda}_{\mu\nu}\bar{\nabla}_{\lambda},$
(11)

Also, the Riemannian curvature tensor $\bar{R}^{\sigma}_{\gamma\mu\nu}$ is
defined as follows:

$\bar{R}^{\rho}_{\gamma\mu\nu}=\bar{\Gamma}^{\rho}_{\gamma\nu,\mu}-\bar{\Gamma}^{\rho}_{\gamma\mu,\nu}+\bar{\Gamma}^{\delta}_{\gamma\nu,}\bar{\Gamma}^{\rho}_{\delta\mu}-\bar{\Gamma}^{\delta}_{\gamma\mu,}\bar{\Gamma}^{\rho}_{\delta\nu}-C^{\delta}_{\mu\nu}\bar{\Gamma}^{\rho}_{\delta\gamma}$
(12)

The extra term $C^{\alpha}_{\mu\nu}\bar{\Gamma}^{\rho}_{\alpha\delta}$ is due
to the effect of non-commutation of the curvature tensor.

The torsion of fractal space time as anti-symmetric part of affine connection
can be amended in a fractal space-time as defined [5]

$\bar{\Lambda}^{A}_{BC}=\bar{\Gamma}^{A}_{BC}-\bar{\Gamma}^{A}_{CB}-\bar{C}^{A}_{BC}.$
(13)

Thus, it can be found that for an arbitrary poly-vector $A^{M}$,

$\frac{\bar{\nabla}A^{M}}{\bar{\nabla}x^{\mu\nu}}=[\nabla_{\mu},\nabla_{\nu}]A^{M}+C^{N}_{\mu\nu}D_{N}A^{M}$
(14)

where , $\frac{\bar{\nabla}}{\bar{\nabla}x^{\mu\nu}}$ is the covariant
derivative with respect to a plane $x^{\mu\nu}$, such that

$\frac{\bar{\nabla}s}{\bar{\nabla}x^{\mu\nu}}=[\bar{\nabla}_{\mu},\bar{\nabla}_{\nu}]s=K^{\rho}_{\mu\nu}\partial_{\rho}s+C^{\rho}_{\mu\nu}D_{\rho}s,$
(15)

where, $K^{\rho}_{\mu\nu}$ is the torsion tensor.

$\frac{\bar{\nabla}a^{\alpha}}{\bar{\nabla}x^{\mu\nu}}=[\bar{\nabla}_{\mu},\bar{\nabla}_{\nu}]a^{\alpha}+C^{\delta}_{\mu\nu}\bar{\nabla}_{\delta}a^{\alpha}=\bar{R}^{\alpha}_{\rho\mu\nu}a^{\rho}+K^{\rho}_{\mu\nu}\bar{\nabla}_{\rho}a^{\alpha}+C^{\rho}_{\mu\nu}\bar{\nabla}_{\rho}a^{\alpha}+C^{\delta}_{\mu\nu}\bar{\nabla}_{\delta}a^{\alpha}.$
(16)

Yet, this type of torsion (15) can related to the notation of torsion as
mentioned by Hammond [8] as commutation of the potential associated by a
prescribed scalar field $\phi$ [8]

$K^{\alpha}_{\mu\nu}=\frac{1}{2}(\delta^{\alpha}_{\mu}\phi_{,\nu}-\delta^{\alpha}_{\nu}\phi_{,\mu})+C^{\delta}_{\mu\nu}{\phi_{,\delta}}.$
(17)

The fractal metric tensor is defined as [9]

$\tilde{g}_{\mu\nu}=g_{\mu\nu}+\gamma_{\mu\nu}$ (18)

such that $\gamma_{\mu\nu}$ is the fluctuation metric tensor (defined within
fractal dimensions).
The fractal covariant derivative is defined as follows [3]:

$\frac{\bar{\nabla}}{\bar{\nabla}S}=\frac{{D}}{{D}S}+V^{\alpha}.{D}_{\alpha}-i\mu({D}^{\mu}{D}_{\mu}+\xi
R)$ (19)

where, $\frac{D}{DS}$ is the well-known covariant derivative as described in
the context of Riemannian geometry. Nowadays, in which its corresponding
geodesic may be described as follows:

$\frac{\bar{\nabla}U^{\alpha}}{\bar{\nabla}S}=0.$ (20)

Also, the same technique can be applied to get the variation with respect to
the tangent vector $U^{\rho}$ to get the geodesic deviation equations:

$\frac{\bar{\nabla}^{2}\Psi^{\alpha}}{\bar{\nabla}S^{2}}=\bar{R}^{\alpha}_{.\beta\gamma\delta}\Psi^{\gamma}U^{\beta}U^{\delta}$
(21)

The above method has been applied in different geometries than the Riemannian
one e.g. the above Lagrangian has been modified to describe the path equation
of charged object to take the following form :

$L=\tilde{g}_{\alpha\beta}\tilde{V}^{\alpha}{\frac{\tilde{D}\tilde{\Psi}^{\beta}}{\tilde{D}S}}+\tilde{f}_{\beta}\tilde{\Psi}^{\beta}$
(22)

where

${\tilde{f}_{\beta}=a_{1}F_{\alpha\beta}\tilde{V}^{\beta}+a_{2}\bar{R}_{\alpha\beta\gamma\delta}\tilde{S}^{\gamma\delta}\tilde{V}^{\alpha}},$

$a_{1}$ and $a_{2}$ are parameters may take the following values
${\frac{e}{m}}$ and ${\frac{1}{2m}}$ to be adjusted with the original Lorentz
force equation and Papapetrou equation [10] as well as Dixon equation [11].

##  2  Fractal Clifford Space: Underlying Geometry

In this section, we are going to introduce an extended object as described as
a poly-vector existed in a fractal space. A point in a Fractal Clifford Space
(FCS) is defined as a set of holographic coordinates
$(s,x^{\mu},x^{\mu\nu},...)$ forming the coordinates of a poly-vector . Each
one is expressed within bases
$\\{\gamma_{A}\\}=\\{1,\gamma_{a_{1}},\gamma_{a_{1}a_{2}},\gamma_{a_{1}a_{2}a_{3}}....\\}$
, $a_{1}<a_{2}<a_{3}<a_{4}<a_{5}<...$, $r=1,2,3....,$ where
$\gamma_{a_{1}a_{2}a_{3}...}=a_{1}\wedge a_{2}\wedge a_{3}....$ .
It is well known that the local basis $\gamma_{\mu}$, is related to the tetrad
field $e^{a}_{\mu}$ such that [12]

$\gamma_{\mu}=e^{a}_{\mu}\gamma_{a}$

in which

$A=a+\frac{1}{1!}a^{\mu}\gamma_{\mu}+\frac{1}{2!}a^{\mu\nu}\gamma_{\mu}\wedge\gamma_{\nu}+..........\frac{1}{n!}a^{\mu_{1}.....\mu_{n}}\gamma_{\mu_{1}}\wedge......\gamma_{\mu_{n}}$
(23)

The fractal derivative of C-space is defined as follows

$\bar{d}A=\frac{\partial A}{\partial X^{B}}\bar{d}X^{B}$ (24)

i.e

$\bar{d}A=\frac{\partial A}{\partial s}+\frac{\partial A}{\partial
x^{\mu}}\bar{d}x^{\mu}+\frac{\partial A}{\partial
x^{\mu\nu}}\bar{d}x^{\mu\nu}+......$ (25)

in other words, if one takes $A=\gamma_{\mu}$ , then

$\bar{d}\gamma_{\mu}=\frac{\partial\gamma}{\partial
s}+\frac{\partial\gamma_{\mu}}{\partial
x^{\mu}}\bar{d}x^{\mu}+\frac{\partial\gamma_{\mu}}{\partial
x^{\mu\nu}}\bar{d}x^{\mu\nu}+......$ (26)

which becomes

$\bar{d}\gamma_{\mu}=\frac{\partial\gamma}{\partial
s}+\Gamma^{\alpha}_{\mu\nu}\bar{d}x^{\nu}+\Gamma^{\alpha}_{\mu[\nu,\rho]}\bar{d}x^{\nu\rho}+......$
(27)

which can be reduced to

$\bar{d}\gamma_{\mu}=\frac{\partial\gamma}{\partial
s}+\bar{\Gamma}^{\alpha}_{\mu\nu}\bar{d}x^{\nu}+\frac{1}{2}\bar{R}^{\alpha}_{\beta\nu\rho}\bar{d}x^{\nu\rho}+......$
(28)

where $\bar{R}^{\alpha}_{\beta\nu\rho}$ is the curvature of fractal space-
time.
Also, it can be found that for an arbitrary poly-vector $A^{M}$

$\frac{\bar{D}A^{M}}{\bar{D}x^{\mu\nu}}=[\bar{D}_{\mu},\bar{D}_{\nu}]A^{M}$
(29)

where $\frac{\bar{D}}{\bar{D}x^{\mu\nu}}$ is the covariant derivative with
respect to a plane $x^{\mu\nu}$, such that

$\frac{\bar{D}s}{\bar{D}x^{\mu\nu}}=[\bar{D}_{\mu},\bar{D}_{\nu}]s=\bar{K}^{\rho}_{\mu\nu}\partial_{\rho}s,$
(30)

where $K^{\rho}_{\mu\nu}$ is the torsion tensor.

$\frac{\bar{D}a^{\alpha}}{\bar{D}x^{\mu\nu}}=[\bar{D}_{\mu},\bar{D}_{\nu}]a^{\alpha}=\bar{R}^{\alpha}_{\rho\mu\nu}a^{\rho}+\bar{K}^{\rho}_{\mu\nu}\bar{D}_{\rho}a^{\alpha},$
(31)

Yet, this type of torsion can related to the notation of torsion as mentioned
by Hammond [9] as commutator the potential associated by a prescribed scalar
field $\phi$

$\bar{K}^{\alpha}_{\mu\nu}=\frac{1}{2}(\delta^{\alpha}_{\mu}\phi_{\bar{,}\nu}-\delta^{\alpha}_{\nu}\phi_{\bar{,}\mu})$
(32)

Thus, we can figure out that the torsion as defined in C-space (Riemannian
Type) by means of the parameters $s$ may act like an independent scalar field
defined in the usual context of Riemannian-Cartan geometry.

From examining equations (30) and (31), one can find the existence of torsion
tensor in the even presence of symmetric affine connection apart from its
conventional notation definition of being the anti-symmetric part of an affine
connection as in the context of non-Riemannian geometries. Accordingly, owing
to C-space one may realize the dispute between the reliability of torsion
propagating or non-propagating this can be resolved by means of of we describe
torsion as a result of covariant differentiation of areas of holographic
coordinates and non-propagating as being defined as anti-symmetric parts of an
affine connection of poly-vectors or vectors, if one utilizes in the internal
or external coordinate its corresponding non-symmetric affine connection.

Consequently, we can regard that the geometry described within the coordinates
of poly-vector described not only Riemannian but also a non-Riemannian, i.e.
the composition of a Riemannian affine connection for the poly-vector is not
necessarily Riemannian as well, This may through some light to find out that a
Riemannian Poly-vector affine connection and curvature (external coordinate
capital Latin letters) may be described by non- Riemannian quantities as
describing its holographic coordinates (internal coordinate (Greek letters).

The arising notation of continuous and not spaces evolves the existence of a
vital parameter called resolution [4]. This means that scale relativity is
going to extending the original relativity. It is worth mentioning that
Nottale 1993 initiated this concept paving the way to a huge number of papers
examining different types of physics using scale relativity (SR). One of very
fundamental aspects due to implementing notation of SR which gives rise that
physical quantities are becoming scale-dependent [ref.]. In other words, the
length $\it{L(\epsilon)}$ of the fractal curve is becoming function of
resolution such that

$\it{L(\epsilon)}=\it{L_{0}(1+\frac{\lambda}{\epsilon})^{\delta}}$

where $L_{0}$ the classical length $\lambda$ is a constant to be fixed like
the Planck length microscopic scale or the Compton length for the macroscopic
scale, $epsilon$ is scale of resolution acts as a fractal dimension: for
better adjustment $\epsilon$ becomes $\ln{\epsilon}$ and $\delta$. ” Scale
relativity is an approach to understand quantum mechanics, and physical
domains to define chaotic system as well as large scale cosmological”
Accordingly the poly-vector may be expressed in fractal space as follows

$X^{A}=X^{A}_{0}(1+\frac{\lambda_{0}}{\Lambda})^{\delta}$ (33)

Thus, in order to implement the concept of scale relativity on poly-vectors as
defined in Clifford space, it has to allocate the quantities of $\lambda$ and
$\epsilon$ of scale relativity to be specified as $\lambda_{0}$ the electron
wave length and $\Lambda$ the Planck length.

Accordingly the poly-vector may be expressed in fractal space as follows

$X^{A}=X^{A}_{0}(1+\frac{\lambda_{0}}{\Lambda})^{\delta}$ (34)

##  3 The Bazanki Approach for Extended Objects of Fractal Clifford Space

The Bazanski approach [13] to obtain an equation of an object may be found as
follows:

$L=G_{AB}W^{A}\frac{\bar{D}\Phi^{B}}{\bar{D}\Xi}$ (35)

Taking the variation with respect to $\Phi^{C}$ and $W^{C}$ we obtain after
some manipulations one finds

$\frac{\bar{D}W^{C}}{\bar{D}S}=0,$ (36)

and

$\frac{\bar{D}^{2}\Phi^{C}}{\bar{D}\bar{S}^{2}}=\bar{R}^{C}_{BDE}W^{E}W^{D}\Phi^{E},$
(37)

where $\bar{;}$ is the fractal covariant derivative.

###  3.1 Spinning and Spinning Deviation Equations of a Fractal Clifford Space

We suggest the equivalent Bazanski Lagrangian [14-19] for deriving the
equations for spinning and spinning poly-vectors of a Fractal Clifford Space
(FCS) to be

$L=G_{AB}P^{A}\frac{\bar{D}\Psi^{B}}{\bar{D}\bar{S}}+S_{AB}\frac{\bar{D}\Psi^{AB}}{\bar{D}\bar{S}}+\bar{F}_{A}\Psi^{A}+\bar{M}_{AB}\Psi^{AB}.$
(38)

such that

$P^{A}=mU^{A}+U_{\beta}\frac{\bar{D}S^{AB}}{\bar{D}S}$

where $P^{\mu}$ is the momentum poly-vector
$\bar{F}^{\mu}=\frac{1}{2}\bar{R}^{\mu}_{\nu\rho\delta}S^{\rho\delta}U^{\nu}$,
$\bar{R}^{\alpha}_{\beta\rho\sigma}$ is its corresponding Riemann curvature,
$\frac{\bar{D}}{\bar{D}\bar{S}}$ is the fractal covariant derivative with
respect to a parameter $S$,$S^{\alpha\beta}$ is the spin poly-tensor, and
$M^{\mu\nu}=P^{\mu}U^{\nu}-P^{\nu}U^{\mu}$ such that
$U^{\alpha}=\frac{\bar{d}x^{\alpha}}{\bar{d}s}$ is the unit tangent poly-
vector to the geodesic one. In a similar way as performed in (40), by taking
the variation with respect to $\Psi^{\mu}$ and$\Psi^{\mu\nu}$ simultaneously
one obtains

$\frac{\bar{D}P^{M}}{\bar{D}\bar{S}}=\bar{F}^{M},$ (39)
$\frac{\bar{D}S^{MN}}{\bar{D}\bar{S}}=\bar{M}^{MN},$ (40)

Using the following identity on both equations (41) and (42)

$A^{D}_{\bar{;}NH}-A^{D}_{\bar{;}HN}=\bar{R}^{D}_{BNH}A^{B},$ (41)

such that $A^{D}$ is an arbitrary poly-vector. Multiplying both sides with
arbitrary poly-vectors, $U^{H}\Psi^{B}$ as well as using the following
condition

$U^{A}_{\bar{;}H}\Psi^{H}=\Psi^{A}_{\bar{;}H}U^{H},$ (42)

and $\Psi^{A}$ is its deviation poly-vector associated to the unit poly-vector
tangent $U^{A}$. Also in a similar way:

$S^{AB}_{\bar{;}H}\Psi^{H}=\Psi^{AB}_{\bar{;}H}U^{H},$ (43)

We obtain the corresponding deviation equations:

$\frac{\bar{D}^{2}\Psi^{A}}{\bar{D}S^{2}}=\bar{R}^{A}_{BHC}P^{B}U^{H}\Phi^{C}+\bar{F}^{A}_{\bar{;}H}\Psi^{H},$
(44)

and

$\frac{\bar{D}^{2}\Psi^{AB}}{\bar{D}\bar{S}^{2}}=S^{[BD}{\bar{R}}^{A]}_{DHC}U^{H}\Psi^{C}+\bar{M}^{AB}_{\bar{;}H}\Psi^{H}.$
(45)

###  3.2 The Generalized Dixon Equation in C-space

As a byproduct, we suggest the following Lagrangian which enables us to obtain
the spinning charged poly-vector in FCS .

$L=G_{AB}P^{A}\frac{\bar{D}\Psi^{B}}{\bar{D}S}+S_{AB}\frac{\bar{D}\Psi^{AB}}{\bar{D}S}+\tilde{F_{A}}\Psi^{A}U^{B}+\tilde{M}_{AB}\Psi^{AB},$
(46)

such that

$\tilde{F_{A}}=\frac{1}{m}(e\bar{F}_{AB}+\frac{1}{2}{\bar{R}}_{ABCD}S^{CD})$

and

$\tilde{M}_{AB}=(P_{A}U^{B}-P_{B}U^{U}+\bar{F}_{AC}S^{C}_{.~{}B}-\bar{F}_{AC}S_{A~{}.}^{.~{}C})$

. Taking the variation with respect to $\Psi^{\mu}$ and$\Psi^{\mu\nu}$
simultaneously we obtain

$\frac{\bar{D}P^{M}}{\bar{D}S}=\frac{\epsilon}{M}\bar{F}^{M}_{B}U^{B}+\frac{1}{2M}\bar{R}^{M}_{NEQ}S^{EQ}U^{N},$
(47)
$\frac{\bar{D}S^{MN}}{\bar{D}S}=P^{M}U^{N}-P^{N}U^{M}+\bar{F}^{MC}S_{C}^{.~{}N}-\bar{F}^{CN}S^{M}_{.~{}C}.$
(48)

Thus, applying the same laws of commutation as illustrated in (43)and(44), we
obtain their corresponding deviation equation

$\frac{\bar{D}^{2}\Psi^{A}}{\bar{D}S^{2}}=\bar{R}^{A}_{BHC}P^{B}U^{H}\Phi^{C}+(\frac{\epsilon}{M}\bar{F}^{M}_{B}U^{B}+\frac{1}{2m}{\bar{R}}^{M}_{NEQ}S^{EQ}U^{N})_{\bar{;}H}\Psi^{H},$
(49)

and

$\frac{\bar{D}^{2}\Psi^{AB}}{\bar{D}S^{2}}=S^{[BD}{R}^{A]}_{DHC}U^{H}\Psi^{C}+(P^{M}U^{N}-P^{N}U^{M}+\bar{F}^{MC}S_{C}^{.~{}N}-\bar{F}^{CN}S^{M}_{.~{}C})_{\bar{;}H}\Psi^{H}.$
(50)

##  4 Conclusions

In this paper we have defined a new type of path equations for extended
objects as defined in a fractal space as an attempt to combine micro-physics
with macro-physics. Owing to this idea one may express extended objects in
Clifford space as points described in Fractal Space. Accordingly, their path
and path deviation equations are obtained as in equation (30 and 31) not only
those ones but their spinning and spinning deviation equations in as shown in
(46), (47), (49)and (50).
However, such an approach is regarded as a prototype to comprehend the
unification problem. Yet, the work has to be extended using strong theories of
gravity such as bi-metric theory in order to examine the problem of
singularity as well as fluctuations of space time, which will be expressed in
our future work.

## References

[1]C. Castro and M. Pavsic, Progress in Physics vol 1, 32 (2005)
[2] L. Nottale, Chaos, Solitons and Fractals vol 7 , No.6 ,877 (1996)
[3] M. E. Kahil and T. Harko , 85 Proceedings MEARIM- IAU Regional Meetings
April 5-10,2008 , eds. A.A Hady and M.I. Wanas, Cairo, Egypt
[4] L. Nottale Astronomy and Astrophysics, 327, 867(1997)
[5] M. Agop and I. Gottieb, J. Math Phys. 47, 05353 (2006)
[6] D. da Rocha and L. Nottale Chaos, Solitons and Fractals 16, 565, (2003).
[7] M. Agop and I. Gottieb, Eur. Phys. Journal D 49, 239 (2008)
[8] R. Hammond Reports on Progress in physics 65, 599.
[9] L. Nottale, Marie- Noelle Celerier and T. Lerner arXiv: hep-th/0605280
(2006).
[10]A Papapetrou Proceedings of Royal Society London A 209 , 248(1951)
[11] W.G. Dixon Proceedings of Royal Society London A 314 , 499(190)
[12] Magd E. Kahil Journal of Modern Physics (2020) 11,1865 DOI :
10.4236/jmp.2020.1111116 .
[13] S.L. Bazanski J. Math. Phys. 30, 1018 (1989)
[14]M.I. Wanas, M. Melek, and M.E. Kahil, Astrophys. Space Sci.,228, 273. ;
gr-qc/0207113 (1995).
[15]M.I. Wanas, M. Melek, M.E. Kahil Gravit. Cosmol., 6 , 319 (2000).
[16] Magd E. Kahil J. Math. Phys. 47,052501 (2006)
[17] Magd E. Kahil Gravit. Cosmol. 23, 70 (2017)
[18] Magd E. Kahil Gravit. Cosmol. 24, 83 (2018)
[19] Magd E. Kahil ADAP 3, 136 (2018)


# Bounds all around: training energy-based models with bidirectional bounds

Cong Geng, Jia Wang, Zhiyong Gao
Shanghai Jiao Tong University
{gengcong, jiawang<EMAIL_ADDRESS>
&Jes Frellsen, Søren Hauberg11footnotemark: 1
Technical University of Denmark
{jefr<EMAIL_ADDRESS>equal contribution

###### Abstract

Energy-based models (EBMs) provide an elegant framework for density
estimation, but they are notoriously difficult to train. Recent work has
established links to generative adversarial networks, where the EBM is trained
through a minimax game with a variational value function. We propose a
bidirectional bound on the EBM log-likelihood, such that we maximize a lower
bound and minimize an upper bound when solving the minimax game. We link one
bound to a gradient penalty that stabilize training, thereby provide grounding
for best engineering practice. To evaluate the bounds we develop a new and
efficient estimator of the Jacobi-determinant of the EBM generator. We
demonstrate that these developments significantly stabilize training and yield
high-quality density estimation and sample generation.

##  1 Energy-based models

Energy-based models (EMBs) are probabilistic models that draw inspiration from
physics and have a long history in machine learning [Hopfield, 1982, Hinton
and Sejnowski, 1983, Smolensky, 1986]. An EBM is specified in terms of an
energy function $E_{\theta}:\mathcal{X}\rightarrow\mathbb{R}$ that is
parameterized by $\theta$ and defines a probability distribution over
$\mathcal{X}$ from the Gibbs density:

$p_{\theta}(\mathbf{x})=\frac{\exp(-E_{\theta}(\mathbf{x}))}{Z_{\theta}},\qquad
Z_{\theta}=\int\exp(-E_{\theta}(\mathbf{x}))\mathrm{d}\mathbf{x},$ (1)

where $Z_{\theta}$ is the normalization constant or partition function. In
principle, any density can be described this way for a suitable choice of
$E_{\theta}$. EBMs are typically learned using maximum likelihood estimation
(MLE), where we wish to find a value of $\theta$ that minimized the _negative_
data log-likelihood:

$L(\theta):=-\mathbb{E}_{\mathbf{x}\sim p_{\text{data}}(\mathbf{x})}\left[\log
p_{\theta}(\mathbf{x})\right]=\mathbb{E}_{\mathbf{x}\sim
p_{\text{data}}}[E_{\theta}(\mathbf{x})]+\log Z_{\theta},$ (2)

where $p_{\text{data}}$ is the data generating distribution.

The fundamental challenge with EBMs for non-standard energy functions is the
lack of a closed-form expression for the normalization constant, $Z_{\theta}$,
which hinders exact pointwise density evaluation, sampling and learning.
Therefore, these tasks are traditionally performed using approximate methods
such as Markov chain Monte Carlo (MCMC, Metropolis et al., 1953, Neal et al.,
2011). Similar to the Boltzmann learning rule [Hinton and Sejnowski, 1983,
Osogami, 2017], gradient-based learning for MLE involves evaluating the
gradient $\nabla_{\theta}L(\theta)=\mathbb{E}_{\mathbf{x}\sim
p_{\text{data}}}[\nabla_{\theta}E_{\theta}(\mathbf{x})]-\mathbb{E}_{\mathbf{x}\sim
p_{\theta}}[\nabla_{\theta}E_{\theta}(\mathbf{x})]$, where the expectation in
the last term necessitates MCMC approximation. Recently, a series of papers
[Kim and Bengio, 2016, Kumar et al., 2019, Abbasnejad et al., 2020, Che et
al., 2020] have established links between EBMs and _Wasserstein GANs_ (WGANs,
Arjovsky et al., 2017), such that EBMs can be approximately learned through a
minimax game. Xie et al. [2017, 2018c] and Wu et al. [2018] also employed the
adversarial game of EBM. This allows for significant improvements in EBM
learning over MCMC based methods [Grathwohl et al., 2021].

In this paper, we remark that variational bounds on the value function of the
minimax game can be problematic as this is both maximized and minimized. We
propose to both upper and lower bound the negative log-likelihood, and
alternate between their respective minimization and maximization to alleviate
this concern. Evaluation of the bounds requires evaluating the entropy of the
generator, and we provide a new efficient estimator. We link the upper bound
to the use of gradient penalties which are known to stabilize training.
Experimentally, we demonstrate that our approach matches or surpasses state-
of-the-art on diverse tasks at negligible performance increase.

###  1.1 Background: Variational bounds and minimax games

Following Grathwohl et al. [2021], we can lower bound the log normalization
constant using a proposal (variational) distribution $p_{g}(\mathbf{x})$ and
Jensen’s inequality:

$\log Z_{\theta}=\log\mathbb{E}_{\mathbf{x}\sim
p_{g}}\\!\left[\frac{\exp(-E_{\theta}(\mathbf{x}))}{p_{g}(\mathbf{x})}\right]\geq\mathbb{E}_{\mathbf{x}\sim
p_{g}}\\!\left[\log\frac{\exp(-E_{\theta}(\mathbf{x}))}{p_{g}(\mathbf{x})}\right]=-\mathbb{E}_{\mathbf{x}\sim
p_{g}}[E_{\theta}(\mathbf{x})]+H[p_{g}]$ (3)

where $H[p_{g}]=-\mathbb{E}_{p_{g}}[\log p_{g}]$ is the (differential) entropy
of the proposal distribution. This means that we also obtain a lower bound on
the negative log likelihood function given by

$\mathcal{L}(\theta):=\mathbb{E}_{\mathbf{x}\sim
p_{\text{data}}(\mathbf{x})}[E_{\theta}(\mathbf{x})]-\mathbb{E}_{\mathbf{x}\sim
p_{g}(\mathbf{x})}[E_{\theta}(\mathbf{x})]+H[p_{g}]\leq L(\theta).$ (4)

We note that this bound is tight when $p_{g}=p_{\theta}$, since the bound on
the log normalization constant (3) can be equivalently expressed as $\log
Z_{\theta}\geq\log Z_{\theta}-\mathrm{KL}\left(p_{g}\|p_{\theta}\right)$. To
tighten the bound, we seek a proposal distribution $p_{g}$ that maximises
$\mathcal{L}(\theta)$, while for MLE we want to find a $\theta$ that minimizes
$\mathcal{L}(\theta)$. Accordingly, MLE can be formulated as a minimax game:

$\min_{E_{\theta}}\max_{p_{g}}\left\\{\mathcal{L}(\theta)\right\\}=\min_{E_{\theta}}\max_{p_{g}}\left\\{\mathbb{E}_{\mathbf{x}\sim
p_{\text{data}}(\mathbf{x})}[E_{\theta}(\mathbf{x})]-\mathbb{E}_{\mathbf{x}\sim
p_{g}(\mathbf{x})}[E_{\theta}(\mathbf{x})]+H[p_{g}]\right\\}.$ (5)

This gives a tractable and MCMC-free approach to EBM learning, assuming we can
evaluate the entropy [Grathwohl et al., 2021].

Our key issue is that both minimizing and maximizing a lower bound is
potentially unstable. By minimizing a lower bound, we run the risk of finding
an optimum, where the bound is loosest, rather than where it is informative
about the optima of the true objective. In particular, a minima of the lower
bound may be $-\infty$, which is rather unhelpful. Good results have been
reported from using a lower bound in the minimax game [Dai et al., 2017, Kumar
et al., 2019, Grathwohl et al., 2021], but the conceptual issue remains.

The WGAN loss function is, as noted by Arjovsky et al. [2017], very similar to
the lower bound $\mathcal{L}(\theta)$,

$L_{\text{WGAN}}=\mathbb{E}_{\mathbf{x}\sim p_{\text{data
}}}[E_{\theta}(\mathbf{x})]-\mathbb{E}_{\mathbf{x}\sim
p_{g}}\left[E_{\theta}(\mathbf{x})\right],$ (6)

with the missing entropy term being the only difference. Lessons from the WGAN
can, thus, be expected to apply to the EBM setting as well. For example, the
success of _gradient clipping_ [Arjovsky et al., 2017] and _gradient
penalties_ [Gulrajani et al., 2017] in WGAN training, inspired Grathwohl et
al. [2021] to heuristically use gradient penalties in variational inference to
stabilize training. Our work will provide justification to such heuristics.

##  2 Approximate minimax games through bidirectional bounds

Figure 1: The bidirectional bounds ‘sandwich’ the negative data log-
likelihood.

Instead of solving a minimax game with a lower bounded value function (5), we
propose to bound $L(\theta)$ from both above and below:

$\lfloor L(\theta)\rfloor\leq L(\theta)\leq\lceil L(\theta)\rceil.$ (7)

With this, we can now follow an optimization strategy of alternating

  1. 1.

minimize $\lceil L(\theta)\rceil$ with respect to $E_{\theta}$.

  2. 2.

maximize $\lfloor L(\theta)\rfloor$ with respect to $p_{g}$.

This avoids the potential pitfalls of minimizing a lower bound.

###  2.1 A lower bound for maximizing $p_{g}$

We already have a lower bound $\mathcal{L}(\theta)$ on $L(\theta)$ in Equation
4. The first two terms of the bound can readily be evaluated by sampling from
$p_{\text{data}}$ and $p_{g}$, respectively. The entropy of $p_{g}$ is less
straight-forward. Following standard practice in GANs, we define $p_{g}$
through a base variable $\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$,
which is passed through a ‘generator’ network
$G:\mathbb{R}^{d}\rightarrow\mathbb{R}^{D}$, i.e.
$\mathbf{x}=G(\mathbf{z})\sim p_{g}$. This constructs a density over a
$d$-dimensional manifold111Note that Eq. 8 is only the density _on_ the
spanned manifold, and that the density is zero _off_ the manifold. in
$\mathbb{R}^{D}$, which by standard change-of-variables is

$\log p_{g}(G(\mathbf{z}))=\log
p_{0}(\mathbf{z})-\frac{1}{2}\log\det(\mathbf{J}_{\mathbf{z}}^{\intercal}\mathbf{J}_{\mathbf{z}}),$
(8)

where $\mathbf{J}_{\mathbf{z}}\in\mathbb{R}^{D\times d}$ is the Jacobian of
$G$ at $\mathbf{z}$. This assume that the Jacobian exist, i.e. that the
generator $G$ has at least one derivative. The entropy of $p_{g}$ is then

$H[p_{g}]=-\mathbb{E}_{\mathbf{x}\sim p_{g}}\left[\log
p_{g}(\mathbf{x})\right]=H[p_{0}]+\mathbb{E}_{\mathbf{z}\sim
p_{0}}\left[\frac{1}{2}\log\det(\mathbf{J}_{\mathbf{z}}^{\intercal}\mathbf{J}_{\mathbf{z}})\right].$
(9)

The entropy of $p_{0}$ trivially evaluates to
$H[p_{0}]=\nicefrac{{d}}{{2}}(1+\log(2\pi))$. To avoid an expensive evaluation
of the log-determinant-term, we note that it is easily bounded from below:

$\frac{1}{2}\log\det(\mathbf{J}_{\mathbf{z}}^{\intercal}\mathbf{J}_{\mathbf{z}})=\frac{1}{2}\sum_{i=1}^{d}\log
s_{i}^{2}\geq d\log s_{1},$ (10)

where $s_{d}\geq\ldots\geq s_{1}$ are the singular values of the Jacobian
$\mathbf{J}_{\mathbf{z}}$. Note that in general, this bound is not tight. With
this, we have our final lower bound:

$L(\theta)\geq\lfloor L(\theta)\rfloor=\mathbb{E}_{\mathbf{x}\sim
p_{\text{data}}(\mathbf{x})}[E_{\theta}(\mathbf{x})]-\mathbb{E}_{\mathbf{x}\sim
p_{g}(\mathbf{x})}[E_{\theta}(\mathbf{x})]+H[p_{0}]+\mathbb{E}_{\mathbf{z}\sim
p_{0}}\left[d\log s_{1}(\mathbf{z})\right].$ (11)

###  2.2 An upper bound for minimizing $E_{\theta}$

There are many ways to arrive at an upper bound for the log-likelihood. Due to
the strong ties between the lower bound (4) and the WGAN objective, we take
inspiration from how WGANs are optimized. The original WGAN paper [Arjovsky et
al., 2017] noted optimization instabilities unless _gradient clipping_ was
applied. Gulrajani et al. [2017] noted that this clipping could be avoided by
adding a _gradient penalty_ , such that the loss become

$\displaystyle L_{\text{WGAN-GP}}$
$\displaystyle=L_{\text{WGAN}}+\lambda\mathbb{E}_{\hat{\mathbf{x}}\sim
g(\mathbf{x})}\left[(\|\nabla_{\hat{\mathbf{x}}}E_{\theta}(\hat{\mathbf{x}})\|_{2}-1)^{2}\right],$
(12)

where $\hat{x}$ is a uniform interpolation between real data and generated
samples. This is generally acknowledged as a good way to train a WGAN
[Gulrajani et al., 2017]. We observe that this loss is similar to our lower
bound, but the gradient penalty adds a positive term, such that one could
speculate that we might have an upper bound. With this intuition in hand, we
prove the following statement in the supplementary material.

######  Theorem 1.

Suppose $f:\mathcal{X}\rightarrow\mathbb{R}$ is L-Lipschitz continuous, $g(x)$
is a probability density function with finite support, then there exists
constants $M,m\geq 0$ and $p\geq 1$ such that:

$\log\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[f(\mathbf{x})\right]-\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[\log f(\mathbf{x})\right]\leq M(\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[|\nabla_{\mathbf{x}}\log
f(\mathbf{x})|^{p}\right]+m)^{\nicefrac{{1}}{{p}}}.$ (13)

Note that if $\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[|\nabla_{\mathbf{x}}\log f(\mathbf{x})|^{p}\right]+m\geq
1$, the bound in Equation 13 can be simplified by dropping the power
$\nicefrac{{1}}{{p}}$,

$\log\left[\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[f(\mathbf{x})\right]\right]-\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[\log f(\mathbf{x})\right]\leq M\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[|\nabla_{\mathbf{x}}\log f(\mathbf{x})|^{p}\right]+Mm.$
(14)

For our model, we set
$f(\mathbf{x})=\frac{\exp(-E_{\theta}(\mathbf{x}))}{p_{g}(\mathbf{x})}$ and
$g(\mathbf{x})=p_{g}(\mathbf{x})$. Empirically, we observe the bound
simplification in Equation 14 holds through-out most of the training, and use
the following upper bound:

$\lceil L(\theta)\rceil=\mathcal{L}(\theta)+M\mathbb{E}_{\mathbf{x}\sim
p_{g}(\mathbf{x})}\left[|\nabla_{\mathbf{x}}E_{\theta}(\mathbf{x})+\nabla_{\mathbf{x}}\log
p_{g}(\mathbf{x})|^{p}\right]+Mm.$ (15)

For a given choice of $f(\mathbf{x})$, we can view $m$ as a constant and the
term $Mm$ can be ignored during optimization. This upper bound can directly be
interpreted as the lower bound (4) plus a gradient penalty, albeit one of a
different form than the traditional WGAN penalty, which is derived purely from
a regularization perspective. Our upper bound can, thus, be seen as a
justification of the regularization from a maximum likelihood perspective.

#### Bound tightness

When $p_{\theta}=p_{g}$, we have that
$\left|\nabla_{\mathbf{x}}E_{\theta}(\mathbf{x})+\nabla_{\mathbf{x}}\log
p_{g}(\mathbf{x})\right|^{p}=0$. In Theorem 1, $m$ is a constant related to
the Lipschitz constant of $\log f(\mathbf{x})$ satisfying
$\left|\nabla_{\mathbf{x}}\log
f(\mathbf{x})\right|^{p}\leq\left|\nabla_{\mathbf{y}}\log
f(\mathbf{y})\right|^{p}+m$ for all $\mathbf{x},\mathbf{y}$ (see proof for
details). When $p_{\theta}=p_{g}$ we also have $\left|\nabla_{\mathbf{x}}\log
f(\mathbf{x})\right|^{p}=0$, such that $m=0$. Our upper bound is then $\lceil
L(\theta)\rceil=\mathcal{L}(\theta)=L(\theta)$, and hence it is tight.

##  3 Numerical evaluation of the bounds

#### Evaluating the lower bound

To evaluate the lower bound in Equation 11, we need the smallest singular
value of the Jacobian $\mathbf{J}=\partial_{\mathbf{z}}G(\mathbf{z})$. Recall
that this singular value satisfy
$s_{1}=\|\mathbf{J}\mathbf{v}_{\text{min}}\|_{2}=\min_{\mathbf{v}\neq\mathbf{0}}\frac{\|\mathbf{J}\mathbf{v}\|_{2}}{\|\mathbf{v}\|_{2}}$.
We can then evaluate the singular value by finding $\mathbf{v}_{\text{min}}$
with an iterative optimization algorithm, where we opt to use the celebrated
single-vector _LOBPCG algorithm_ [Knyazev, 1998]. This method performs an
iterative minimization of the generalized Rayleigh quotient,

$\displaystyle\rho(\mathbf{v}):=\frac{\mathbf{v}^{\intercal}\mathbf{J}^{\intercal}\mathbf{J}\mathbf{v}}{\mathbf{v}^{\intercal}\mathbf{v}},$
(16)

which converges to $\mathbf{v}_{\text{min}}$. The gradient of
$\rho(\mathbf{v})$ is proportional to
$r=\mathbf{J}^{\intercal}\mathbf{J}\mathbf{v}-\rho(\mathbf{v})\mathbf{v}$. To
avoid computing the Jacobian $\mathbf{J}$, we use Jacobian-vector products,
which can be efficiently evaluated using automatic differentiation. To compute
$\mathbf{J}^{\intercal}\mathbf{J}\mathbf{v}$, we use the following trick (in
pytorch-notation):

$\displaystyle\mathbf{J}^{\intercal}\mathbf{J}\mathbf{v}=\left((\mathbf{J}\mathbf{v})^{\intercal}\mathbf{J}\right)^{\intercal}=\nabla_{\mathbf{z}}\left((\mathbf{J}\mathbf{v})^{\intercal}\\!\\!\\!\texttt{.detach()}\cdot
G(\mathbf{z})\right)^{\intercal}.$ (17)

The optimal learning rate for this iterative scheme can be found by maximizing
the Rayleigh quotient (16). Finally, we follow the suggestions of Knyazev
[2001] to improve numerical stability and accelerate convergence, which we
omit here for brevity.

#### Evaluating the upper bound

There are two challenges when evaluating Equation 15. The first is to compute
$\nabla_{\mathbf{x}}\log p_{g}(\mathbf{x})$, where we empirically found that
existing methods [Shi et al., 2018, Li and Turner, 2018] were too inefficient
for our needs. To evaluate the term $\mathbb{E}_{\mathbf{x}\sim
p_{g}(\mathbf{x})}\left[|\nabla_{\mathbf{x}}E_{\theta}(\mathbf{x})+\nabla_{\mathbf{x}}\log
p_{g}(\mathbf{x})|^{p}\right]$, we further loosen the bound

$\displaystyle|\nabla_{G(\mathbf{z})}E_{\theta}(G(\mathbf{z}))+\nabla_{G(\mathbf{z})}\log
p_{g}(G(\mathbf{z}))|^{p}$
$\displaystyle\leq\frac{|\nabla_{G(\mathbf{z})}E_{\theta}(G(\mathbf{z}))\mathbf{J}_{\mathbf{z}}+\nabla_{G(\mathbf{z})}\log
p_{g}(G(\mathbf{z}))\mathbf{J}_{\mathbf{z}}|^{p}}{s_{1}^{p}}$ (18)
$\displaystyle\leq\frac{|\nabla_{G(\mathbf{z})}E_{\theta}(G(\mathbf{z}))\mathbf{J}_{\mathbf{z}}+\nabla_{\mathbf{z}}\log
p_{g}(G(\mathbf{z}))|^{p}}{s_{1}^{p}}.$

where $s_{1}$ is the smallest singular value of $\mathbf{J}_{\mathbf{z}}$.
Detailed derivations are in the supplementary material. If we choose $p=2$,
then we can use Hutchinson’s estimator [1989]:

$|\nabla_{\mathbf{x}}E_{\theta}(\mathbf{x})\mathbf{J}_{\mathbf{z}}+\nabla_{\mathbf{z}}\log
p_{g}(G(\mathbf{z}))|^{2}=\mathbb{E}_{\mathbf{v}}\left[\left(\nabla_{\mathbf{x}}E_{\theta}(\mathbf{x})\mathbf{J}_{\mathbf{z}}\mathbf{v}+\nabla_{\mathbf{z}}\log
p_{g}(G(\mathbf{z}))\mathbf{v}\right)^{2}\right],$ (19)

where $\mathbf{v}\sim\mathcal{N}\left(\mathbf{0},\mathbf{I}_{d}\right)$. This
is easily evaluated using automatic differentiation.

The second challenge is to evaluate $\log p_{g}(\mathbf{x})$ which needs the
Jacobian of the generator $G(\mathbf{z})$ as dictated by Equation 8. Here, we
opt to use our entropy estimator as described above. We could alternatively
use Hutchinson’s estimator as proposed by Kumar et al. [2020]. Experimentally
we do not observe much difference between these two estimators.

##  4 Related work

In machine learning, there has been a long-standing interest in EBMs dating
back to Hopfield networks [Hopfield, 1982], Boltzmann machines [Hinton and
Sejnowski, 1983, Ackley et al., 1985] and restricted Boltzmann machines
[Smolensky, 1986, Hinton, 2002], see e.g. reviews in the works by LeCun et al.
[2006] and Scellier [2020]. Learning and evaluation of these models are
difficult since the normalization constant cannot be efficiently evaluated.
MLE-based learning, such as the Boltzmann learning rule, relies on expensive
MCMC sampling to estimate the gradient, and more advanced MCMC methods are
used to reliably estimate the normalization constant (see e.g. Salakhutdinov
and Murray, 2008, Grosse et al., 2013, Burda et al., 2015, Frellsen et al.,
2016). For images, MCMC-based learning has been used to learn non-deep EBMs of
both textures [Zhu et al., 1998, Zhu and Mumford, 1998] and natural images
[Xie et al., 2015, 2016]. Learning algorithms that avoid the costly MCMC
approximation have been heavily investigated. For instance, Hinton [2002]
proposed k-step Contrastive Divergence (CD-k) to approximate the negative
phase log-likelihood gradient, and Hyvärinen [2005] proposed an alternative
method to train non-normalized graphical models using score matching. Deep
versions of EBMs have subsequently been proposed, such as deep belief networks
[Hinton et al., 2006] and deep Boltzmann machines [Salakhutdinov and Hinton,
2009].

In recent years, there have been renewed interest in deep generative EBMs,
particularly for image generation. Grathwohl et al. [2021] gave an excellent
overview of the current developments and distinct drawbacks of three classes
of learning methods, which we summarize here: (1) MLE methods with MCMC
sampling are slow and unstable, and while (2) score matching based methods are
comparable faster, they are also unstable and do not work with discontinuous
nonlinearities. (3) Noise-contrastive estimation [Gutmann and Hyvärinen, 2010]
do not have these drawback, but it does not scale well with the data
dimensionality.

Our proposed method belongs to a fourth class of algorithms that sidestep the
costly MCMC sampling by using a simultaneously learned generator or
variational distribution. Kim and Bengio [2016] proposed using a generator
function and an adversarial-like training strategy similar to ours. They
update the generator using the same lower bound as us, but their entropy
approximation is quite different. Furthermore, their method does not have a
gradient penalty like our upper bound when optimizing the energy function.
Consequently, the energy function needs to be explicitly designed to prevent
it from growing to infinity, limiting its potential.

The method proposed by Zhai et al. [2016] plays a min-max game to jointly
optimize the energy function and generator using the same lower bound as us.
However, their method relies on a specific designed bounded multi-modal energy
function, which limits its potential. Furthermore, their approximation of the
generator entropy comes with no theoretical guarantees, and their
regularisation of the energy function does not constitute an upper bound.

Dai et al. [2017] proposed an adversarial learning framework for jointly
learning the energy function and a generator. They considered two approaches
to maximising the generator entropy: One, which maximizes the entropy by
minimizing the conditional entropy using a variational upper bound, and
another, which makes isotropic Gaussian assumptions for the data, which is not
suitable for high-dimensional data. Kumar et al. [2019] and Abbasnejad et al.
[2019] also consider adversarial learning but different approaches to
estimating the entropy of the generator. Kumar et al. [2019] estimated the
entropy through its connection to mutual information, but they need an
additional network to measure the entropy term. Abbasnejad et al. [2019]
maximized the entropy by approximating the generator function’s Jacobian log-
determinant. However, their method is impractical in high dimensions as the
Jacobian is computationally expensive. Han et al. [2019] use an adversarial
learning strategy in their divergence triangle loss, but their training
mechanism is radically different from ours, and they rely on an extra encoder
for learning the generator.

Xie et al. [2018a, b, 2021a, 2021b] proposed _cooperative learning_ of the
energy function and a generator. However, the cooperative learning approach
relies on MCMC or Langevin dynamics to draw samples from the EBM, which is
expensive and difficult to tune. Like our work, VERA by Grathwohl et al.
[2021] avoids the use of MCMC. VERA plays a min-max game and uses variational
inference to approximate the gradient of the entropy term, which is different
from ours. Furthermore, VERA uses a gradient penalty as a regularizer for the
energy function, which is a heuristic unlike our upper bound, and their method
is memory-consuming, and the hyperparameters are difficult to adjust.

##  5 Experiments

To demonstrate the efficiency of our energy-based model with bidirectional
bounds (EBM-BB) we compare against a range of methods that represent state-of-
the-art. As representatives of the GAN literature, we consider _deep
convolutional GANs (DCGANs)_ by Radford et al. [2016], _spectrally normalized
GANS (SNGANs)_ by Miyato et al. [2018] and the _WGAN with zero-centered
gradient penalty (WGAN-0GP)_ by Thanh-Tung et al. [2019]. As representatives
of EBMs, we consider the _maximum entropy generators (MEG)_ by Kumar et al.
[2019] and the _variational entropy regularized approximate maximum likelihood
(VERA)_ estimator [Grathwohl et al., 2021]. As representatives of CoopNets, we
consider two similar methods [Xie et al., 2018a, 2021a] and _EBM-VAE_ [Xie et
al., 2021b]. We also consider _NCSN_ by Song and Ermon [2019] and _DDPM_ by Ho
et al. [2020] as representatives of a score matching model and a diffusion
model respectively. To investigate the influence of our derived upper bound,
we introduce another baseline in which we replace our upper bound with a zero-
centered gradient penalty. We denote the resulting model _EBM-0GP_.

###  5.1 Training details

On toy data and MNIST, all models are realized with multi-layer perceptrons
(MLPs), while for natural images, we use the convolutional architecture from
StudioGAN [Kang and Park, 2020]. Details of the network architecture are given
in the supplementary material. All experiments are conducted on a single 12GB
NVIDIA Titan GPU using a pytorch [Paszke et al., 2017] implementation. To
improve training, we use a positive margin for our energy function to balance
the bounds. Specifically, we let

$\lceil L(\theta)\rceil=\lfloor
L(\theta)\rfloor+\left[M\mathbb{E}_{\mathbf{x}\sim
p_{g}(\mathbf{x})}\left[|\nabla_{\mathbf{x}}E_{\theta}(\mathbf{x})+\nabla_{\mathbf{x}}\log
p_{g}(\mathbf{x})|^{p}\right]-\zeta\right]_{+},$ (20)

where $[\cdot]_{+}=\max(0,\cdot)$ is the usual hinge. In all experiments,
$\zeta=1$ as this allows us to apply the simplified bound in Equation 15.

###  5.2 Toy data

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
(a) Data | (b) WGAN-0GP | (c) MEG | (d) VERA | (e) EBM-0GP (ours) | (f) EBM-BB (ours)
Figure 2: Density estimation on the 25-Gaussians and swiss-roll datasets.  Table 1: Number of captured modes and KL divergence between the real and generated distributions. Model | Modes$\uparrow$ | KL$\downarrow$
---|---|---
DCGAN | 392 $\pm$ 7.4 | 8.012 $\pm$ 0.056
SNGAN | 441 $\pm$ 39.0 | 2.755 $\pm$ 0.033
WGAN-0GP | 1000 $\pm$ 0.0 | 0.048 $\pm$ 0.003
MEG | 1000 $\pm$ 0.0 | 0.042 $\pm$ 0.004
VERA | 989 $\pm$ 9.0 | 0.152 $\pm$ 0.037
EBM-0GP(ours) | 1000 $\pm$ 0.0 | 0.039 $\pm$ 0.003
EBM-BB (ours) | 1000 $\pm$ 0.0 | 0.045 $\pm$ 0.003

Figure 2 shows estimated densities on the 25-Gaussians and swiss-roll datasets
using both our methods, WGAN-0GP and two baselines. We observe that WGAN’s
discriminator is not suitable as a density estimator. This is unsurprising as
WGAN is not supposed to provide a density estimate. MEG and EBM-0GP can have
inaccurate density information in some edge and peripheral areas. This may be
due to insufficient or excessive maximization of entropy since the zero-
centered gradient penalty is not a principled objective for maximum
likelihood. VERA and our EBM-BB can learn a sharp distribution, but our method
is more stable in some inner regions of the 25-Gaussians and edge regions of
the swiss-roll.

###  5.3 MNIST

#### Mode counting

Mode collapse is a frequently occurring phenomenon in which the generator
function maps all latent input to a small number of points in the observation
space. In particular, GANs are plagued by this problem. Since the generator is
trained to maximize the entropy of the generated distribution, several EBM-
based models have been shown to capture all the modes of the data
distribution. To empirically verify that our model also captures a variety of
modes in the data distribution, we follow the test procedure of Kumar et al.
[2019]. We train our generative model on the StackedMNIST dataset, a synthetic
dataset created by stacking MNIST on different channels. The true total number
of modes is $1{,}000$, and they are counted using a pretrained MNIST
classifier. The KL divergence is calculated empirically between the generated
mode distribution and the data distribution. The results appear in Table 1. As
expected, GAN-based methods suffer from mode collapse except for WGAN-0GP. All
the EBM-based methods capture all or nearly all the modes of the data
distribution. Our model EBM-0GP captures all modes and report the smallest KL
measures. It worth mentioned that WGAN-0GP obtains comparable results, which
was also observed by Grathwohl et al. [2021].

#### Entropy Estimation

Figure 3: Entropy estimators.

In Figure 3, we explore the quality of our entropy estimator measured on MNIST
[LeCun, 1998]. We use networks with fully connected layers as the Jacobian is
then easily derived in closed-form, giving us a ground truth for the entropy.
We compare our estimator (red) with the ground truth (black) and Hutchinson’s
estimator (blue), as was proposed by Kumar et al. [2020]. Finally, as we only
run our iterative estimator for a few steps, we also compare with an estimator
that uses the smallest singular value computed with high precision (green). We
observe that Hutchinson’s estimator is reasonable close to the ground truth
but provides an upper bound, making it inapplicable for our lower bound. We
further observe a noticeable gap between our estimator and the high precision
singular value estimator. Finally, we see that all estimators follow roughly
the same trend, which suggests that our estimator provides a suitable target
for optimization.

###  5.4 Natural Images

#### Image generation

The data studied thus far are simple and perhaps do not challenge our upper
bound and entropy estimator. We, therefore, train our model on the standard
benchmark $32\times 32$ CIFAR-10 [Krizhevsky et al., 2009] dataset and the
$64\times 64$ cropped
ANIMEFACE222https://www.kaggle.com/splcher/animefacedataset dataset, which
both represent a significant increase in complexity. Similar to recent work on
GANs [Miyato et al., 2018], we report Inception Score (IS), Fréchet Inception
Distance (FID) scores and two $F_{\beta}$ scores [Sajjadi et al., 2018]. We
compare with competitive GANs, EBM baselines, CoopNets, NCSN and DDPM (Table
3). All GANs and EBMs are reproduced using the same network architecture (see
supplements for details) except DCGAN and IGEBM, where we respectively borrow
the results from StudioGAN [Kang and Park, 2020] and the original
implementation [Du and Mordatch, 2019]. We use the DCGAN [Radford et al.,
2016] network architecture for CIFAR-10 and a Resnet architecture [Kang and
Park, 2020] for ANEMIFACE. Parameters are chosen as in the original papers.
For VERA [Grathwohl et al., 2021], we were unable to reproduce the reported
performance, so we choose hyper-parameters according to an extensive grid
search. Like the original paper, we choose the entropy weight to be $0.0001$.
For CoopNets and NCSN, we report the results from the original papers. For
DDPM, we used a public available
implementation.333https://github.com/rosinality/denoising-diffusion-pytorch

From Table 3, we see that our EBM-BB model is the best performing EBM on
CIFAR-10, though it is surpassed by NCSN and DDPM. This is not surprising as
NCSN and DDPM focus on sample quality rather than optimising a data
likelihood. Further note that sampling NCSN and DDPM is significantly more
expensive than our method. On ANIMEFACE our method is highly competitive.
Figures 4 and 5 show samples from different models. For ANIMEFACE, our method
generates more diverse samples in terms of face parts than the baselines. All
models predominantly generate female faces. This suggests that while our
approach captures more density modes than the baselines, all approaches still
misses several modes. We further draw attention to several corrupt samples
generated by VERA, despite extensive parameter search. For CIFAR-10, we
observe no immediate differences in the generative capabilities between
models.

|  |  |
---|---|---|---
(a) WGAN-0GP | (b) MEG | (c) VERA | (d) EBM-BB (ours)
Figure 4:  Generated samples on CIFAR-10 with our method and various methods.  |  |  |
---|---|---|---
(a) WGAN-0GP | (b) MEG | (c) VERA | (d) EBM-BB (ours)
Figure 5:  Generated samples on ANIMEFACE with our method and other generative
models

#### Capacity usage

For the proposal distribution $p_{g}$ to be adaptive, the associated generator
network $G(\mathbf{z})$ should be able to use as much of its available
capacity as possible. To compare different methods, we consider an implicit
measure of capacity usage $\mathcal{C}_{\mathbf{z}}$ that locally measures the
intrinsic dimensionality of $G(\mathbf{z})$. In particular, we use the
anisotropy index [Wang and Ponce, 2021] that, for a given $\mathbf{z}$,
measure the standard deviation of the norm of the directional derivative of
$G(\mathbf{z})$ along the input dimensions, i.e.

$\displaystyle\mathcal{C}_{\mathbf{z}}$
$\displaystyle=\mathrm{std}\left(\\{\|\mathbf{J}_{\mathbf{z}}\mathbf{e}_{i}\|\\}_{i=1\ldots
d}\right),$ (21)

Table 2: Anisotropy indices ($\downarrow$).

Model | CIFAR-10 | ANIMEFACE
---|---|---
WGAN-0GP | 1.0823 $\pm$ 0.005 | 2.4366 $\pm$ 0.129
MEG | 0.9467 $\pm$ 0.009 | 2.3600 $\pm$ 0.142
VERA | 3.9694 $\pm$ 0.055 | 2.2929 $\pm$ 0.2074
EBM-0GP (ours) | 1.0688 $\pm$ 0.0226 | 3.4560 $\pm$ 0.0521
EBM-BB (ours) | 0.9431 $\pm$ 0.0191 | 1.8016 $\pm$ 0.1345

where $\mathrm{std}(\cdot)$ computes the standard deviation of the input, and
$\mathbf{e}_{i}$ denotes the $i^{\mathrm{th}}$ standard basis vector. A small
$\mathcal{C}_{\mathbf{z}}$ indicates that the different input dimensions
contribute equally to the output of $G$, which imply good capacity usage. We
measure the mean value of $\mathcal{C}_{\mathbf{z}}$ for
$\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{d})$ and report mean and
standard deviation of the result running for several times in Table 2. We
observe that most models, with the exception of VERA, perform well on
CIFAR-10, while on ANIMEFACE there is more diversity. In both cases, EBM-BB
has the best capacity usage. The large difference between EBM-0GP and EBM-BB
on ANIMEFACE suggest that our proposed upper bound helps increase the entropy
of the proposal distribution $p_{g}$.

Table 3: Comparison in terms of FID, Inception Score, $F_{8}$ score (weights recall higher than precision) and $F_{\nicefrac{{1}}{{8}}}$ score (weights precision higher than recall). Model | Inception$\uparrow$ | FID$\downarrow$ | $F_{8}\\!\\!\uparrow$ | $F_{\nicefrac{{1}}{{8}}}\\!\uparrow$
---|---|---|---|---
CIFAR-10
DCGAN | 6.64 | 49.03 | 0.795 | 0.83
WGAN-0GP | 7.24 $\pm$ 0.035 | 29.31 $\pm$ 0.185 | 0.92 $\pm$ 0.010 | 0.95 $\pm$ 0.010
CoopNets [2018a] | 6.55 | 36.4 | - | -
CoopNets [2021a] | - | 33.61 | - | -
EBM-VAE | 6.65 | 36.2 | - | -
IGEBM | 6.78 | 38.2 | - | -
MEG | 6.62 $\pm$ 0.243 | 34.55 $\pm$ 1.145 | 0.88 $\pm$ 0.001 | 0.92 $\pm$ 0.010
VERA | 5.06 $\pm$ 0.555 | 66.38 $\pm$ 6.635 | 0.58 $\pm$ 0.080 | 0.79 $\pm$ 0.005
NCSN | 8.87 $\pm$ 0.12 | 25.32 | - | -
DDPM | 9.03 | 7.76 | 0.98 | 0.99
EBM-0GP (ours) | 6.90 $\pm$ 0.032 | 35.42 $\pm$ 0.582 | 0.90 $\pm$ 0.004 | 0.93 $\pm$ 0.002
EBM-BB (ours) | 7.45 $\pm$ 0.014 | 28.63 $\pm$ 0.290 | 0.93 $\pm$ 0.001 | 0.95 $\pm$ 0.008
ANIMEFACE
WGAN-0GP | 2.22 $\pm$ 0.030 | 9.76 $\pm$ 0.674 | 0.95 $\pm$ 0.005 | 0.98 $\pm$ 0.005
MEG | 2.20 $\pm$ 0.020 | 9.31 $\pm$ 0.007 | 0.95 $\pm$ 0.005 | 0.98 $\pm$ 0.001
VERA | 2.15 $\pm$ 0.001 | 41.00 $\pm$ 1.072 | 0.515 $\pm$ 0.078 | 0.78 $\pm$ 0.013
DDPM | 2.18 | 8.81 | 0.94 | 0.98
EBM-0GP (ours) | 2.26 $\pm$ 0.017 | 20.53 $\pm$ 0.524 | 0.889 $\pm$ 0.008 | 0.909 $\pm$ 0.019
EBM-BB (ours) | 2.26 $\pm$ 0.005 | 12.75 $\pm$ 0.045 | 0.94 $\pm$ 0.001 | 0.96 $\pm$ 0.005

#### Out-of-distribution detection

As EBMs are density estimators, they should, in principle, assign low
likelihood to out-of-distribution (OOD) observations. OOD detection
performance can then be seen as a proxy for the quality of the estimated
density. To test how well our model fares in this regard, we follow previous
work [Hendrycks and Gimpel, 2017, Hendrycks et al., 2019, Alemi et al., 2018,
Choi et al., 2018, Ren et al., 2019, Havtorn et al., 2021] and report the
threshold independent evaluation metrics of Area Under the Receiver Operator
Characteristic (AUROC$\uparrow$), Area Under the Precision Recall Curve
(AUPRC$\uparrow$) and False Positive Rate at 80% (FPR80$\downarrow$), where
the arrow indicates the direction of improvement of the metrics. The results
are reported in Table 4. Each column of the table takes the form ‘In-
distribution / Out-of-distribution’ in reference to the training and test set,
respectively. We observe that on CIFAR-10 / SVHN, EBM-BB is the top performer.
On CIFAR-10 / CIFAR-100 the overall performance degraded significantly,
EBM-0GP fared noticeable better than other models. The overall degradation may
be caused by the strong similarity between CIFAR-10 and CIFAR-100. For
ANIMEFACE / Bedroom a similar situation occurs, even if these datasets are
highly dissimilar, but we observe EBM-0GP is much better than other models.

No clear winner can be found from this study. We note that our method
consistently performs well, but surprisingly, so does WGAN-0GP even if it is
not a density estimator. As OOD detection is a task that comes with many
subtle pitfalls [Havtorn et al., 2021], we suggest that these results should
be taken with a grain of salt even if our model performs well.

Table 4: AUROC$\uparrow$, AUPRC$\uparrow$ and FPR80$\downarrow$ for OOD
detection for ‘train / test’ datasets.

Model | CIFAR-10 / SVHN | CIFAR-10 / CIFAR-100 | ANIMEFACE / Bedroom
---|---|---|---
AUROC$\uparrow$ | AUPRC$\uparrow$ | FPR80$\downarrow$ | AUROC$\uparrow$ | AUPRC$\uparrow$ | FPR80$\downarrow$ | AUROC$\uparrow$ | AUPRC$\uparrow$ | FPR80$\downarrow$
WGAN-0GP | 0.8 | 0.83 | 0.39 | 0.54 | 0.55 | 0.77 | 0.51 | 0.48 | 0.72
MEG | 0.79 | 0.81 | 0.42 | 0.52 | 0.53 | 0.8 | 0.56 | 0.53 | 0.77
VERA | 0.62 | 0.64 | 0.64 | 0.51 | 0.51 | 0.79 | 0.60 | 0.525 | 0.6
EBM-0GP (ours) | 0.66 | 0.69 | 0.67 | 0.64 | 0.63 | 0.64 | 0.67 | 0.63 | 0.53
EBM-BB (ours) | 0.88 | 0.86 | 0.1997 | 0.53 | 0.52 | 0.7765 | 0.53 | 0.505 | 0.83

#### Running time

Table 5: Total training time.

Model | Iterations | Runtime
---|---|---
CIFAR-10
WGAN-0GP | 100000 | 4h
MEG | 100000 | 4.5h
VERA | 200000 | 10h
EBM-0GP | 200000 | 15h
EBM-BB | 200000 | 20h
ANIMEFACE
WGAN-0GP | 100000 | 13h
MEG | 100000 | 20h
VERA | 200000 | 44h
EBM-0GP | 100000 | 38h
EBM-BB | 100000 | 40h

Table 5 gives an overview of the time required to train the proposed models
and the baselines. All models were trained on a single 12GB Titan GPU. We
observe that our model should be trained with a lower learning rate, and
therefore may increase the total number of epochs.

##  6 Discussion

The first observation in our work is that current methods for training energy-
based models (EBMs) interchangeably minimize and maximize a lower bound. As
this may be a potential source of training instability, we propose to bound
the negative log-likelihood from above and below and switch between bounds
when minimizing and maximizing. The lower bound, $\lfloor L(\theta)\rfloor$,
is similar to existing ones, but we provide a new algorithm for its
realization. Unlike past work, this algorithm does not need additional
networks and does not rely on hard-to-tune parameters; our only parameter
controls the number of iterations for the Jacobian approximation and
represents a trade-off between the accuracy of the bound and computational
budget. The upper bound, $\lceil L(\theta)\rceil$, is new to the literature
but similar to the common regularization practice of introducing gradient
penalties. To the best of our knowledge, this is the first time that gradient
penalties have been derived from the perspective of bounding the log-
likelihood. It is rewarding that current better engineering practice can be
justified from a probabilistic perspective. Empirically, we find our proposed
model generally performs as well or better than some of the current state-of-
the-art on a variety of tasks. The evidence suggests that our bidirectional
bounds allow the generator to increase its entropy and capacity usage. We see
this both directly and through improved sample quality on diverse datasets.

#### Limitations

The current drawbacks of the method are mainly three concerns. First, we find
that a smaller-than-usual learning rate helps our model. While we expect this
to be a matter of implementation rather than a more profound concern, the
current implication is that training is approximately twice that of a EBM
baseline. Second, our bounds rely on computing the smallest singular value of
the Jacobian of the generator. We provide an efficient implementation of a
method for this task, but it needs to run for several iterations to guarantee
convergence. In practice, we stop after a fixed, low number of iterations,
which we found to work well, but technically this violates the bound. This
seems to be an unavoidable aspect of our approach, but we find that the
benefits significantly compensate for this issue. The availability of a fast
(approximate) entropy estimator that does not require additional networks and
parameters is highly valuable when training EBMs. Finally, for the upper
bound, we need an estimate of the volume of support $M$ of the proposal
distribution $p_{g}$. We do not have a viable method for this, and, in
practice, we treat $M$ as a hyper-parameter. We have not found it difficult to
tune this parameter, but it nonetheless constitutes a limitation.

#### Negative societal impact

All high-capacity generative models carry the risk of being used for
misinformation, and our model is no exception. The value of EBMs over GANs is
that they come with a likelihood function, which is more valuable in data
analysis, than e.g. for creating deepfakes.

## Acknowledgments and Disclosure of Funding

This work was funded in part by the Novo Nordisk Foundation through the Center
for Basic Machine Learning Research in Life Science (NNF20OC0062606). It also
received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research, innovation programme (757360), National science
foundation of China under grant 61771305 and Shanghai Municipal Science and
Technology Major Project (2021SHZDZX0102). JF was supported in part by the
Novo Nordisk Foundation (NNF20OC0065611) and the Independent Research Fund
Denmark (9131-00082B). SH was supported in part by a research grant (15334)
from VILLUM FONDEN. The authors are grateful to anonymous reviewers and the
handling area chair for valuable discussions and feedback on an early version
of this manuscript.

## References

  * Abbasnejad et al. [2019] M. E. Abbasnejad, Q. Shi, A. v. d. Hengel, and L. Liu.  A generative adversarial density estimator.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10782–10791, 2019.
  * Abbasnejad et al. [2020] M. E. Abbasnejad, J. Shi, A. van den Hengel, and L. Liu.  Gade: A generative adversarial approach to density estimation and its applications.  _International Journal of Computer Vision_ , 128(10):2731–2743, 2020.
  * Ackley et al. [1985] D. H. Ackley, G. E. Hinton, and T. J. Sejnowski.  A learning algorithm for boltzmann machines.  _Cognitive Science_ , 9(1):147–169, 1985.
  * Alemi et al. [2018] A. A. Alemi, I. Fischer, and J. V. Dillon.  Uncertainty in the variational information bottleneck.  _arXiv preprint arXiv:1807.00906_ , 2018.
  * Arjovsky et al. [2017] M. Arjovsky, S. Chintala, and L. Bottou.  Wasserstein generative adversarial networks.  In _International conference on machine learning_ , pages 214–223. PMLR, 2017.
  * Burda et al. [2015] Y. Burda, R. Grosse, and R. Salakhutdinov.  Accurate and conservative estimates of MRF log-likelihood using reverse annealing.  In G. Lebanon and S. V. N. Vishwanathan, editors, _Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics_ , volume 38 of _Proceedings of Machine Learning Research_ , pages 102–110, San Diego, California, USA, 09–12 May 2015. PMLR.
  * Che et al. [2020] T. Che, R. Zhang, J. Sohl-Dickstein, H. Larochelle, L. Paull, Y. Cao, and Y. Bengio.  Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 12275–12287. Curran Associates, Inc., 2020.
  * Choi et al. [2018] H. Choi, E. Jang, and A. A. Alemi.  WAIC, but why? Generative ensembles for robust anomaly detection.  _arXiv preprint arXiv:1810.01392_ , 2018.
  * Dai et al. [2017] Z. Dai, A. Almahairi, P. Bachman, E. Hovy, and A. Courville.  Calibrating energy-based generative adversarial networks.  In _International Conference on Learning Representations_ , 2017.
  * Du and Mordatch [2019] Y. Du and I. Mordatch.  Implicit generation and modeling with energy based models.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Frellsen et al. [2016] J. Frellsen, O. Winther, Z. Ghahramani, and J. Ferkinghoff-Borg.  Bayesian generalised ensemble markov chain monte carlo.  In A. Gretton and C. C. Robert, editors, _Proceedings of the 19th International Conference on Artificial Intelligence and Statistics_ , volume 51 of _Proceedings of Machine Learning Research_ , pages 408–416, Cadiz, Spain, 09–11 May 2016. PMLR.
  * Grathwohl et al. [2021] W. S. Grathwohl, J. J. Kelly, M. Hashemi, M. Norouzi, K. Swersky, and D. Duvenaud.  No MCMC for me: Amortized sampling for fast and stable training of energy-based models.  In _International Conference on Learning Representations_ , 2021.
  * Grosse et al. [2013] R. B. Grosse, C. J. Maddison, and R. R. Salakhutdinov.  Annealing between distributions by averaging moments.  In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc., 2013.
  * Gulrajani et al. [2017] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville.  Improved training of wasserstein GANs.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Gutmann and Hyvärinen [2010] M. Gutmann and A. Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , pages 297–304. JMLR Workshop and Conference Proceedings, 2010.
  * Han et al. [2019] T. Han, E. Nijkamp, X. Fang, M. Hill, S.-C. Zhu, and Y. N. Wu.  Divergence triangle for joint training of generator model, energy-based model, and inferential model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8670–8679, 2019.
  * Havtorn et al. [2021] J. D. Havtorn, J. Frellsen, S. Hauberg, and L. Maaløe.  Hierarchical VAEs know what they don’t know.  In M. Meila and T. Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 4117–4128. PMLR, 18–24 Jul 2021.
  * Hendrycks and Gimpel [2017] D. Hendrycks and K. Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _International Conference on Learning Representations_ , 2017.
  * Hendrycks et al. [2019] D. Hendrycks, M. Mazeika, and T. Dietterich.  Deep anomaly detection with outlier exposure.  In _International Conference on Learning Representations_ , 2019.
  * Hinton [2002] G. E. Hinton.  Training products of experts by minimizing contrastive divergence.  _Neural computation_ , 14(8):1771–1800, 2002\.
  * Hinton and Sejnowski [1983] G. E. Hinton and T. J. Sejnowski.  Optimal perceptual inference.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , volume 448, 1983.
  * Hinton et al. [2006] G. E. Hinton, S. Osindero, and Y.-W. Teh.  A fast learning algorithm for deep belief nets.  _Neural computation_ , 18(7):1527–1554, 2006\.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 6840–6851. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf.
  * Hopfield [1982] J. J. Hopfield.  Neural networks and physical systems with emergent collective computational abilities.  _Proceedings of the National Academy of Sciences_ , 79(8):2554–2558, 1982.  ISSN 0027-8424.
  * Hutchinson [1989] M. F. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines.  _Communications in Statistics-Simulation and Computation_ , 18(3):1059–1076, 1989.
  * Hyvärinen [2005] A. Hyvärinen.  Estimation of non-normalized statistical models by score matching.  _Journal of Machine Learning Research_ , 6(4), 2005.
  * Kang and Park [2020] M. Kang and J. Park.  ContraGAN: Contrastive learning for conditional image generation.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 21357–21369. Curran Associates, Inc., 2020.
  * Kim and Bengio [2016] T. Kim and Y. Bengio.  Deep directed generative models with energy-based probability estimation.  _arXiv preprint arXiv:1606.03439_ , 2016.
  * Knyazev [1998] A. V. Knyazev.  Preconditioned eigensolvers—an oxymoron.  _Electron. Trans. Numer. Anal_ , 7:104–123, 1998.
  * Knyazev [2001] A. V. Knyazev.  Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method.  _SIAM journal on scientific computing_ , 23(2):517–541, 2001.
  * Krizhevsky et al. [2009] A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * Kumar et al. [2020] A. Kumar, B. Poole, and K. Murphy.  Regularized autoencoders via relaxed injective probability flow.  In _International Conference on Artificial Intelligence and Statistics_ , pages 4292–4301. PMLR, 2020.
  * Kumar et al. [2019] R. Kumar, S. Ozair, A. Goyal, A. Courville, and Y. Bengio.  Maximum entropy generators for energy-based models.  _arXiv preprint arXiv:1901.08508_ , 2019.
  * LeCun [1998] Y. LeCun.  The MNIST database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * LeCun et al. [2006] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang.  A tutorial on energy-based learning.  _Predicting structured data_ , 1(0), 2006.
  * Li and Turner [2018] Y. Li and R. E. Turner.  Gradient estimators for implicit models.  In _International Conference on Learning Representations_ , 2018.
  * Metropolis et al. [1953] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller.  Equation of state calculations by fast computing machines.  _The journal of chemical physics_ , 21(6):1087–1092, 1953.
  * Miyato et al. [2018] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida.  Spectral normalization for generative adversarial networks.  In _International Conference on Learning Representations_ , 2018.
  * Neal et al. [2011] R. M. Neal et al.  MCMC using hamiltonian dynamics.  _Handbook of markov chain monte carlo_ , 2(11):2, 2011.
  * Neudecker [1992] H. Neudecker.  A matrix trace inequality.  _Journal of mathematical analysis and applications_ , 166(1):302–303, 1992.
  * Osogami [2017] T. Osogami.  Boltzmann machines and energy-based models.  _arXiv preprint arXiv:1708.06008_ , 2017.
  * Paszke et al. [2017] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  _NIPS 2017 Workshop Autodiff_ , 2017.
  * Radford et al. [2016] A. Radford, L. Metz, and S. Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  In _International Conference on Learning Representations_ , 2016.
  * Ren et al. [2019] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan.  Likelihood ratios for out-of-distribution detection.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Sajjadi et al. [2018] M. S. M. Sajjadi, O. Bachem, M. Lučić, O. Bousquet, and S. Gelly.  Assessing Generative Models via Precision and Recall.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2018.
  * Salakhutdinov and Hinton [2009] R. Salakhutdinov and G. Hinton.  Deep boltzmann machines.  In D. van Dyk and M. Welling, editors, _Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics_ , volume 5 of _Proceedings of Machine Learning Research_ , pages 448–455, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16–18 Apr 2009\. PMLR.
  * Salakhutdinov and Murray [2008] R. Salakhutdinov and I. Murray.  On the quantitative analysis of deep belief networks.  In A. McCallum and S. Roweis, editors, _Proceedings of the 25th Annual International Conference on Machine Learning (ICML 2008)_ , pages 872–879. Omnipress, 2008.
  * Scellier [2020] B. Scellier.  _A deep learning theory for neural networks grounded in physics_.  PhD thesis, Université de Montréal, Quebec, Canada, 2020.
  * Shi et al. [2018] J. Shi, S. Sun, and J. Zhu.  A spectral approach to gradient estimation for implicit distributions.  In _International Conference on Machine Learning_ , pages 4644–4653. PMLR, 2018.
  * Smolensky [1986] P. Smolensky.  _Information Processing in Dynamical Systems: Foundations of Harmony Theory_ , page 194–281.  MIT Press, Cambridge, MA, USA, 1986.
  * Song and Ermon [2019] Y. Song and S. Ermon.  Generative modeling by estimating gradients of the data distribution.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Thanh-Tung et al. [2019] H. Thanh-Tung, T. Tran, and S. Venkatesh.  Improving generalization and stability of generative adversarial networks.  In _International Conference on Learning Representations_ , 2019.
  * Wang and Ponce [2021] B. Wang and C. R. Ponce.  The geometry of deep generative image models and its applications.  _arXiv preprint arXiv:2101.06006_ , 2021.
  * Wu et al. [2018] Y. N. Wu, J. Xie, Y. Lu, and S.-C. Zhu.  Sparse and deep generalizations of the frame model.  _Annals of Mathematical Sciences and Applications_ , 3(1):211–254, 2018.
  * Xie et al. [2015] J. Xie, W. Hu, S.-C. Zhu, and Y. N. Wu.  Learning sparse FRAME models for natural image patterns.  _International Journal of Computer Vision_ , 114(2):91–112, 2015.
  * Xie et al. [2016] J. Xie, Y. Lu, S.-C. Zhu, and Y. N. Wu.  Inducing wavelets into random fields via generative boosting.  _Applied and Computational Harmonic Analysis_ , 41(1):4–25, 2016.
  * Xie et al. [2017] J. Xie, S.-C. Zhu, and Y. Nian Wu.  Synthesizing dynamic patterns by spatial-temporal generative convnet.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ , pages 7093–7101, 2017.
  * Xie et al. [2018a] J. Xie, Y. Lu, R. Gao, and Y. N. Wu.  Cooperative learning of energy-based model and latent variable model via MCMC teaching.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018a.
  * Xie et al. [2018b] J. Xie, Y. Lu, R. Gao, S.-C. Zhu, and Y. N. Wu.  Cooperative training of descriptor and generator networks.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(1):27–45, 2018b.
  * Xie et al. [2018c] J. Xie, Z. Zheng, R. Gao, W. Wang, S.-C. Zhu, and Y. N. Wu.  Learning descriptor networks for 3d shape synthesis and analysis.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8629–8638, 2018c.
  * Xie et al. [2021a] J. Xie, Z. Zheng, X. Fang, S.-C. Zhu, and Y. N. Wu.  Cooperative training of fast thinking initializer and slow thinking solver for conditional learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021a.
  * Xie et al. [2021b] J. Xie, Z. Zheng, and P. Li.  Learning energybased model with variational auto-encoder as amortized sampler.  In _The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI)_ , volume 2, 2021b.
  * Zhai et al. [2016] S. Zhai, Y. Cheng, R. Feris, and Z. Zhang.  Generative adversarial networks as variational training of energy based models.  _arXiv preprint arXiv:1611.01799_ , 2016.
  * Zhu and Mumford [1998] S. C. Zhu and D. Mumford.  Grade: Gibbs reaction and diffusion equations.  In _Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271)_ , pages 847–854. IEEE, 1998.
  * Zhu et al. [1998] S. C. Zhu, Y. Wu, and D. Mumford.  Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling.  _International Journal of Computer Vision_ , 27(2):107–126, 1998.

Bounds all around: training energy-based models with bidirectional bounds
— Supplementary Material —

Cong Geng, Jia Wang, Zhiyong Gao
Shanghai Jiao Tong University
{gengcong, jiawang<EMAIL_ADDRESS>
&Jes Frellsen11footnotemark: 1, Søren Hauberg11footnotemark: 1
Technical University of Denmark
{jefr<EMAIL_ADDRESS>
##  Appendix A Theoretical part

###  A.1 Proof of Theorem 1

######  Proof.

$\displaystyle\log\left[\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[f(x)\right]\right]-\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[\log f(x)\right]=\log\int f(x)g(x)dx-\int(\log
f(x))g(x)dx$ (22) $\displaystyle=\log\left[f(\eta)\int g(x)dx\right]-\int(\log
f(x))g(x)dx=\log f(\eta)\int g(x)dx-\int(\log f(x))g(x)dx$
$\displaystyle=\int\left[\log f(\eta)-\log f(x)\right]g(x)dx=\int
g(x)\int_{0}^{1}\frac{d\left[\log f(t\eta+(1-t)x)\right]}{dt}dtdx$
$\displaystyle=\int g(x)\int_{0}^{1}\nabla_{\tilde{x}}\log
f(\tilde{x})(\eta-x)dtdx$ $\displaystyle\leq\int
g(x)\left(\int_{0}^{1}|\nabla_{\tilde{x}}\log
f(\tilde{x})|^{p}dt\right)^{\frac{1}{p}}\left(\int_{0}^{1}|\eta-x|^{q}dt\right)^{\frac{1}{q}}dx$
$\displaystyle\leq\int g(x)|\eta-x|\left(\int_{0}^{1}|\nabla_{\tilde{x}}\log
f(\tilde{x})|^{p}dt\right)^{\frac{1}{p}}dx$ $\displaystyle\leq\left(\int
g(x)\int_{0}^{1}|\nabla_{\tilde{x}}\log
f(\tilde{x})|^{p}dtdx\right)^{\frac{1}{p}}\left(\int
g(x)|\eta-x|^{q}dx\right)^{\frac{1}{q}},$

where $\tilde{x}=t\eta+(1-t)x$. The second equation is derived by mean value
theorem for definite integrals. The first inequality is derived by Holder’s
inequality, so $p,q\geq 1$ and $\frac{1}{p}+\frac{1}{q}=1$. Because $g(x)$ has
finite support, there exists an $M\geq 0$ that satisfying: $|\eta-x|\leq M$,
then we can get:

$\displaystyle\log\left[\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[f(x)\right]\right]-\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[\log f(x)\right]$ $\displaystyle\leq M\left(\int
g(x)\int_{0}^{1}|\nabla_{\tilde{x}}\log
f(\tilde{x})|^{p}dtdx\right)^{\frac{1}{p}}$ (23) $\displaystyle\leq
M\left(\int g(x)|\nabla_{\hat{x}}\log f(\hat{x})|^{p}dx\right)^{\frac{1}{p}},$

where $\hat{x}=t_{0}\eta+(1-t_{0})x$ for a $t_{0}$ ($0\leq t_{0}\leq 1$) using
the mean value theorem. Because $f(x)$ is L-Lipschitz continuous, then $\log
f(x)$ is also Lipschitz continuous, so there exists an $m\geq 0$ satisfying

$|\nabla_{\hat{x}}\log f(\hat{x})|^{p}\leq|\nabla_{x}\log
f(x)|^{p}+m,for\forall x,$ (24)

so we can get

$\displaystyle\log\left[\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[f(x)\right]\right]-\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[\log f(x)\right]$ $\displaystyle\leq M\left(\int
g(x)|\nabla_{x}\log f(x)|^{p}dx+m\right)^{\frac{1}{p}}$ (25)
$\displaystyle\leq M(\mathbb{E}_{\mathbf{x}\sim
g(\mathbf{x})}\left[|\nabla_{x}\log f(x)|^{p}\right]+m)^{\frac{1}{p}}.$

###  A.2 Proof of Eq. (18)

######  Proof.

For a vector $u\in R^{1\times n}$ and a matrix $J\in R^{n\times m}$, we have

$\displaystyle|uJ|_{2}^{2}$
$\displaystyle=uJJ^{T}u^{T}=\operatorname{tr}(JJ^{T}u^{T}u)$ (26)
$\displaystyle\geq\lambda(JJ^{T})\operatorname{tr}(u^{T}u),$

where $\lambda$ is the smallest eigenvalue of $JJ^{T}$. The inequality holds
because $JJ^{T}$ is a real symmetric matrix and $u^{T}u$ is positive
semidefinite [Neudecker, 1992]. Then we set
$J=\mathbf{J}_{\mathbf{z}}$,$u=\nabla_{\mathbf{x}}E_{\theta}(G(\mathbf{z}))+\nabla_{G(\mathbf{z})}\log
p_{g}(G(\mathbf{z}))$, we can obtain:

$\displaystyle|\nabla_{\mathbf{x}}E_{\theta}(G(\mathbf{z}))\mathbf{J}_{\mathbf{z}}+\nabla_{G(\mathbf{z})}\log
p_{g}(G(\mathbf{z}))\mathbf{J}_{\mathbf{z}}|_{2}^{2}\geq\lambda(\mathbf{J}_{\mathbf{z}}\mathbf{J}_{\mathbf{z}}^{T})\operatorname{tr}(u^{T}u)$
(27)
$\displaystyle=\lambda(\mathbf{J}_{\mathbf{z}}\mathbf{J}_{\mathbf{z}}^{T})|u|_{2}^{2}=\lambda(\mathbf{J}_{\mathbf{z}}\mathbf{J}_{\mathbf{z}}^{T})|\nabla_{\mathbf{x}}E_{\theta}(G(\mathbf{z}))+\nabla_{G(\mathbf{z})}\log
p_{g}(G(\mathbf{z}))|_{2}^{2}$

Because $\lambda(\mathbf{J}_{\mathbf{z}}\mathbf{J}_{\mathbf{z}}^{T})$ is the
square of the smallest singular value of $\mathbf{J}_{\mathbf{z}}$, which we
represent by $s_{1}$. So we can obtain:

$|\nabla_{\mathbf{x}}E_{\theta}(G(\mathbf{z}))+\nabla_{G(\mathbf{z})}\log
p_{g}(G(\mathbf{z}))|_{2}\leq\frac{|\nabla_{\mathbf{x}}E_{\theta}(G(\mathbf{z}))\mathbf{J}_{\mathbf{z}}+\nabla_{G(\mathbf{z})}\log
p_{g}(G(\mathbf{z}))\mathbf{J}_{\mathbf{z}}|_{2}}{s_{1}}.$ (28)

This proves Eq. 18.

##  Appendix B Model Architecture

In order apply the change-of-variables formula to get a density for the
generator, we assume that $G:\mathbb{R}^{d}\rightarrow\mathbb{R}^{D}$ spans an
immersed $d$-dimensional manifold in $\mathbb{R}^{D}$. This assumption place
some restrictions on the architecture of the generator neural network.

The governing assumption is that the Jacobian of $G$ exist and has full rank.
Existence is ensured as long as the chosen activation functions have at least
one derivative almost everywhere. Smooth activations naturally satisfy this
assumption, but it is worth noting that e.g. the ReLU activation function has
a single point where the derivative is not defined. As long as the linear map
preceding the activation is not degenerate, then the non-smooth region has
measure zero, and the change-of-variables technique still applies.

We cannot guarantee that the Jacobian has full rank through clever choices of
neural architectures. However, we note that one requirement is that no hidden
layer may have dimensionality below the $d$ dimensions of the latent space.
This is a natural requirement for the generator anyway. In our model, we aim
to maximize the entropy of the generator, which encourages the generator to
create as diverse samples as possible. In practice this ensures that the
Jacobian has full rank as a degenerate Jacobian implies a reduction of
entropy. Note that this is not a theoretical guarantee against degenerate
Jacobians during optimization, but in practice we have at no point experienced
problems in this regard.

###  B.1 Practical experimental settings

For the toy and MNIST datasets, we use multi-layer perceptrons (MLPs)
networks, while for CIFAR-10 and ANIMEFACE datasets, we use a DCGAN network
and Resnet architecture, respectively.

Table 6: The network architecture trained for toy datasets Operation | Input | Output
---|---|---
Energy
Linear, PReLU | 2 | 100
Linear, PReLU | 100 | 100
Linear | 100 | 1
Generator
Linear, PReLU, BN | 2 | 100
Linear, PReLU, BN | 100 | 100
Linear | 100 | 2
Table 7: The network architecture trained for MNIST dataset Operation | Input | Output
---|---|---
Energy
Linear, PReLU | 2352 | 2000
Linear, PReLU | 2000 | 1000
Linear, PReLU | 1000 | 500
Linear, PReLU | 500 | 250
Linear, PReLU | 250 | 250
Linear | 250 | 1
Generator
Linear, BN, PReLU | 128 | 500
Linear, BN, PReLU | 500 | 1000
Linear, BN, PReLU | 1000 | 2000
Linear, Tanh | 2000 | 2352
Table 8: The network architecture for CIFAR-10 dataset Operation | Kernel | Strides | Channels | Output size
---|---|---|---|---
Energy |
Conv2D, LReLU | 3 | 1 | 64 | 32
Conv2D, LReLU | 4 | 2 | 64 | 16
Conv2D, LReLU | 3 | 1 | 128 | 16
Conv2D, LReLU | 4 | 2 | 128 | 8
Conv2D, LReLU | 3 | 1 | 256 | 8
Conv2D, LReLU | 4 | 2 | 256 | 4
Conv2D, LReLU | 3 | 1 | 512 | 4
Flatten |  |  |  | 8192
Linear |  |  |  | 1
Generator |
Linear |  |  |  | 8192
Reshape |  |  | 512 | 4
ConvTranspose2D, BN, ReLU | 4 | 2 | 256 | 8
ConvTranspose2D, BN, ReLU | 4 | 2 | 128 | 16
ConvTranspose2D, BN, ReLU | 4 | 2 | 64 | 32
Conv2D, Tanh | 3 | 1 | 3 | 32
Table 9: The energy network architecture for ANIMEFACE dataset Operation | Kernel | Strides | Channels | Output size
---|---|---|---|---
(ResBlock0) |  |  |  |
Left: Conv2D, BN, LReLU | 3 | 1 | 64 | 64
Conv2D | 3 | 1 | 64 | 64
AvgPool2D | 2 | 2 | 64 | 32
Right: AvgPool2D, BN | 2 | 2 | 3 | 32
Conv2D | 1 | 1 | 64 | 32
Overall: Add |  |  | 64 | 32
(ResBlock1) |  |  |  |
Left: BN, LReLU,Conv2D | 3 | 1 | 128 | 32
BN, LReLU,Conv2D | 3 | 1 | 128 | 32
AvgPool2D | 2 | 2 | 128 | 16
Right: BN, Con2D | 1 | 1 | 128 | 32
AvgPool2D | 2 | 2 | 128 | 16
Overall: Add |  |  | 128 | 16
(ResBlock2) |  |  |  |
Left: BN, LReLU,Conv2D | 3 | 1 | 256 | 16
BN, LReLU,Conv2D | 3 | 1 | 256 | 16
AvgPool2D | 2 | 2 | 256 | 8
Right: BN, Con2D | 1 | 1 | 256 | 16
AvgPool2D | 2 | 2 | 256 | 8
Overall: Add |  |  | 256 | 8
(ResBlock3) |  |  |  |
Left: BN, LReLU,Conv2D | 3 | 1 | 512 | 8
BN, LReLU,Conv2D | 3 | 1 | 512 | 8
AvgPool2D | 2 | 2 | 512 | 4
Right: BN, Con2D | 1 | 1 | 512 | 8
AvgPool2D | 2 | 2 | 512 | 4
Overall: Add |  |  | 512 | 4
(ResBlock4) |  |  |  |
Left: BN, LReLU,Conv2D | 3 | 1 | 1024 | 4
BN, LReLU,Conv2D | 3 | 1 | 1024 | 4
Right: BN, Con2D | 1 | 1 | 1024 | 4
Overall: Add |  |  | 1024 | 4
LReLU |  |  | 1024 | 4
Sum |  |  |  | 1024
Linear |  |  |  | 1
Table 10: The generator network architecture for ANIMEFACE dataset Operation | Kernel | Strides | Channels | Output size
---|---|---|---|---
Linear |  |  |  | 16384
Reshape |  |  | 1024 | 4
(ResBlock0) |  |  |  |
Left: BN, ReLU, NN-Upsampling, Conv2D | 3 | 1 | 512 | 8
BN, ReLU, Conv2D | 3 | 1 | 512 | 8
Right: NN-Upsampling, Conv2D | 1 | 1 | 512 | 8
Overall: Add |  |  | 512 | 8
(ResBlock1) |  |  |  |
Left: BN, ReLU, NN-Upsampling, Conv2D | 3 | 1 | 256 | 16
BN, ReLU, Conv2D | 3 | 1 | 256 | 16
Right: NN-Upsampling, Conv2D | 1 | 1 | 256 | 16
Overall: Add |  |  | 256 | 16
(ResBlock2) |  |  |  |
Left: BN, ReLU, NN-Upsampling, Conv2D | 3 | 1 | 128 | 32
BN, ReLU, Conv2D | 3 | 1 | 128 | 32
Right: NN-Upsampling, Conv2D | 1 | 1 | 128 | 32
Overall: Add |  |  | 128 | 32
(ResBlock3) |  |  |  |
Left: BN, ReLU, NN-Upsampling, Conv2D | 3 | 1 | 64 | 64
BN, ReLU, Conv2D | 3 | 1 | 64 | 64
Right: NN-Upsampling, Conv2D | 1 | 1 | 64 | 64
Overall: Add |  |  | 64 | 64
BN, ReLU, Conv2D, Tanh | 3 | 1 | 3 | 64

##  Appendix C Training Details

In Table 11, we specify the hyperparameters used when training our models for
each dataset. We choose $p=2$ in Eq (20) for our implementation. We normalize
the data to be in [-1, 1] and do not use dequantization. During training we
augment only using random horizontal flips. For cifar10 we used the official
train-test split from PyTorch, and for animeface we used a 85:15 train-test
split. For our upper bound, we set
$\frac{M}{s_{1}^{2}}=\frac{0.001}{z_{dim}}$, where $s_{1}$ is the smallest
singular value of Jacobian and $z_{dim}$ is the latent dimension. We observe
this setting can get a satisfying generation for all datasets. If we replace
the $z_{dim}$ with $\|v\|_{2}^{2}$ in high-dimensional data, where $v$ is a
random vector sampled in Eq 19, it will further improve the generation. For
out-of-distribution detection and capacity usage, we set
$\frac{M}{s_{1}^{2}}=\frac{0.1}{z_{dim}}$, because if we increase this value,
it will help the density estimation of the energy function. We also observe
that the network’s design affect performance. For example, removing batch
normalization in the energy function can stabilize training on the Animeface
dataset. The relationship between such design decisions in the context of EBMs
should be explored in future work.

Table 11: Selection of most important hyper-parameters and their setting. Datasets | Optimization | Learning rate | Batch size | Iterations/Epochs | latent dim
---|---|---|---|---|---
Toy | Adam(0.0,0.9) | 2e-4 | 200 | 150000 | 2
MNIST | Adam(0.0,0.9) | 2e-4 | 64 | 60(epochs) | 128
CIFAR-10 | Adam(0.0,0.999) | 5e-5 | 64 | 200000 | 128
ANIMEFACE | Adam(0.0,0.999) | 5e-5 | 64 | 100000 | 128


The Jacobi sums over Galois rings of arbitrary characters and their
applications in constructing asymptotically optimal codebooks

Deng-Ming Xu1 Chen Meng2 Gang Wang${}^{3^{*}}$ Fang-Wei Fu4