# Poisson summation formulas involving the sum-of-squares function

Nir Lev  Department of Mathematics, Bar-Ilan University, Ramat-Gan 5290002,
Israel<EMAIL_ADDRESS>and  Gilad Reti  Department of Mathematics, Bar-
Ilan University, Ramat-Gan 5290002, Israel<EMAIL_ADDRESS>
(Date: December 12, 2020)

###### Abstract.

We obtain new Poisson type summation formulas with nodes $\pm\sqrt{n}$ and
with weights involving the function $r_{k}(n)$ that gives the number of
representations of a positive integer $n$ as the sum of $k$ squares. Our
results extend summation formulas due to Guinand and Meyer that involve the
sum-of-three-squares function $r_{3}(n)$.

###### Key words and phrases:

Quasicrystals, Poisson summation formulas, sum of squares

###### 2010 Mathematics Subject Classification:

11E25, 42B10, 52C23

Research supported by ISF Grant No. 227/17 and ERC Starting Grant No. 713927.

##  1\. Introduction

### 1.1.

The classical Poisson summation formula asserts that for any function $f$ on
$\mathbb{R}$ satisfying sufficient smoothness and decay assumptions, we have

$\sum_{n\in\mathbb{Z}}f(n)=\sum_{n\in\mathbb{Z}}\widehat{f}(n)$ (1.1)

where $\widehat{f}(t)=\int_{\mathbb{R}}f(x)\exp(-2\pi itx)dx$ is the Fourier
transform of $f$. The equality (1.1) holds in particular for any function $f$
from the Schwartz class. There is an equivalent way to state the Poisson
summation formula, by saying that the measure
$\mu=\sum_{n\in\mathbb{Z}}\delta_{n}$ (the sum of unit masses at the integer
points) satisfies $\widehat{\mu}=\mu$, where $\widehat{\mu}$ is the Fourier
transform of $\mu$ understood in the sense of temperate distributions.

The problem of which other Poisson type formulas may exist, was studied by
different authors. A recent result proved in [LO13, LO15] states that if
$\mu=\sum_{\lambda\in\Lambda}a_{\lambda}\delta_{\lambda}$ is a pure point
measure on $\mathbb{R}$ supported on a uniformly discrete set $\Lambda$, and
if the Fourier transform $\widehat{\mu}$ (in the sense of temperate
distributions) is also a pure point measure $\widehat{\mu}=\sum_{s\in
S}b_{s}\delta_{s}$ supported on a uniformly discrete set $S$, then the measure
$\mu$ can be obtained from Poisson’s formula using dilation and a finite
number of shifts, modulations, and taking linear combinations. (We recall that
a set is said to be _uniformly discrete_ if the distance between any two of
its points is bounded from below by some positive constant.)

On the other hand, in [LO16] the authors established the existence of pure
point measures $\mu$ whose distributional Fourier transform $\widehat{\mu}$ is
also a pure point measure, and such that the supports of both $\mu$ and
$\widehat{\mu}$ are _locally finite_ sets (i.e. they have no finite
accumulation points), but the support of $\mu$ contains only finitely many
elements of any arithmetic progression. In particular, such a measure $\mu$
cannot be obtained from Poisson’s summation formula using the procedures
mentioned above (see also [Kol16]).

We note that interest in the subject has been largely inspired by the
experimental discovery in the middle of 80’s of quasicrystalline materials,
i.e. nonperiodic atomic structures that have a discrete diffraction pattern,
see [Mey95], [Lag00]. For recent related work see e.g. [Mey16], [Mey17],
[LO17], [Fav18] and the references therein.

### 1.2.

There is a remarkable summation formula due to Guinand [Gui59, p. 265] which
states that if $\varphi$ is an odd Schwartz function on $\mathbb{R}$, and if
$\psi=\widehat{\varphi}$ is the Fourier transform of $\varphi$, then

$\varphi^{\prime}(0)+\sum_{n=1}^{\infty}\frac{r_{3}(n)}{\sqrt{n}}\varphi(\sqrt{n})=i\psi^{\prime}(0)+i\sum_{n=1}^{\infty}\frac{r_{3}(n)}{\sqrt{n}}\psi(\sqrt{n})$
(1.2)

where $r_{k}(n)$ is the number of representations of $n$ as the sum of $k$
squares, that is,

$r_{k}(n)=\\#\\{m\in\mathbb{Z}^{k}:|m|^{2}=n\\}.$ (1.3)

Notice that while (1.2) holds for odd functions only, one can obtain a
summation formula that is valid for an arbitrary Schwartz function $f$ (i.e.
not necessarily an odd function) by applying (1.2) to $\varphi(t):=f(t)-f(-t)$
and $\psi(t):=\widehat{f}(t)-\widehat{f}(-t)$. This yields a nonclassic
Poisson type formula with nodes at the points $\pm\sqrt{n}$ and with weights
involving the sum-of-three-squares function $r_{3}(n)$.

The summation formula (1.2) can be equivalently stated by saying that the
Fourier transform of the temperate distribution

$\sigma_{3}:=-2\delta^{\prime}_{0}+\sum_{n=1}^{\infty}\frac{r_{3}(n)}{\sqrt{n}}(\delta_{\sqrt{n}}-\delta_{-\sqrt{n}})$
(1.4)

is $-i\sigma_{3}$. In particular, the distribution $\sigma_{3}$ is supported
on the locally finite set of points $\pm\sqrt{n}$, and its Fourier transform
$\widehat{\sigma}_{3}$ is supported on the same set.

Guinand’s formula (1.2) remained little known until it was recently
rediscovered by Meyer, see [Mey16]. In that paper, a new proof of Guinand’s
formula was given, and moreover new examples of Poisson type summation
formulas generalizing Guinand’s one were obtained.

There is a recent result due to Radchenko and Viazovska [RV19] which states
that a Schwartz function $f$ is uniquely determined by the values of $f$ and
$\widehat{f}$ at the points $\pm\sqrt{n}$ and the values $f^{\prime}(0)$ and
${\widehat{f}}\,^{\prime}(0)$. Moreover, there exists a linear summation
formula that recovers the function $f$ from these values [RV19, Theorems 1 and
7]. As pointed out in [RV19, pp. 77–78], one can use this result to derive
Guinand’s formula (1.2).

Remarkably, the Poisson and Guinand formulas (and their linear combinations)
are _the only_ weighted summation formulas that involve only the values of $f$
and $\widehat{f}$ at the points $\pm\sqrt{n}$ and the values $f^{\prime}(0)$
and ${\widehat{f}}\,^{\prime}(0)$. In other words, if $\sigma$ is a temperate
distribution such that both $\sigma$ and its Fourier transform
$\widehat{\sigma}$ can be expressed as a weighted sum of terms of the form
$\delta_{0}$, $\delta^{\prime}_{0}$, $\delta_{\sqrt{n}}$ or
$\delta_{-\sqrt{n}}$ $(n=1,2,3,\dots)$, then $\sigma$ must be a linear
combination of the Poisson measure $\mu=\sum_{n\in\mathbb{Z}}\delta_{n}$ and
the Guinand distribution $\sigma_{3}$. This is a consequence of [Via18,
Theorem 5.2]. We note that the Poisson component of $\sigma$ corresponds to
its even part, while the Guinand component is the odd part of $\sigma$.

##  2\. Results

### 2.1.

In this paper we obtain new Poisson type summation formulas with nodes
$\pm\sqrt{n}$, and with weights involving the function $r_{k}(n)$ defined by
(1.3) that gives the number of representations of $n$ as the sum of $k$
squares.

For each positive integer $k$, consider a distribution $\sigma_{k}$ on
$\mathbb{R}$ defined by

$\sigma_{k}:=-2\delta^{\prime}_{0}+\sum_{n=1}^{\infty}\frac{r_{k}(n)}{\sqrt{n}}(\delta_{\sqrt{n}}-\delta_{-\sqrt{n}}).$
(2.1)

We observe that $\sum_{n=1}^{N}r_{k}(n)n^{-1/2}$ increases polynomially in
$N$, which implies that $\sigma_{k}$ is a temperate distribution. It is
obtained from Guinand’s distribution (1.4) by replacing the sum-of-three-
squares function $r_{3}(n)$ with the sum-of-$k$-squares function $r_{k}(n)$.

It is clear that $\sigma_{k}$ is an odd distribution supported on the set of
points $\pm\sqrt{n}$. Guinand’s result states that for $k=3$, the Fourier
transform $\widehat{\sigma}_{3}$ is supported on the same set of points, and
in fact, $\widehat{\sigma}_{3}=-i\sigma_{3}$. In the present paper we obtain
an extension of this result to all odd values of $k$ greater than $3$. Our
result provides an explicit expression for the Fourier transform
$\widehat{\sigma}_{k}$, which implies in particular that it is again supported
on the points $\pm\sqrt{n}$. However we will see that $\widehat{\sigma}_{k}$
is not $-i\sigma_{k}$ if $k>3$.

### 2.2.

For simplicity we first state the result for $k=5$.

######  Theorem 2.1.

Let $\varphi$ be an odd Schwartz function on $\mathbb{R}$, and let
$\psi=\widehat{\varphi}$ be the Fourier transform of $\varphi$. Then

$\varphi^{\prime}(0)+\sum_{n=1}^{\infty}\frac{r_{5}(n)}{\sqrt{n}}\varphi(\sqrt{n})=-\frac{i}{6\pi}\psi^{\prime\prime\prime}(0)+\frac{i}{2\pi}\sum_{n=1}^{\infty}\frac{r_{5}(n)}{n^{3/2}}\Big{[}\psi(\sqrt{n})-\sqrt{n}\,\psi^{\prime}(\sqrt{n})\Big{]}.$
(2.2)

The result establishes a new summation formula with nodes $\pm\sqrt{n}$ and
with weights involving the sum-of-five-squares function $r_{5}(n)$. We observe
that the right hand side of (2.2) involves not only the values of $\psi$, but
also those of its derivative $\psi^{\prime}$, at the points $\sqrt{n}$
$(n=1,2,3,\dots)$. This is not the case in Guinand’s formula (1.2).

Theorem 2.1 can be equivalently stated by saying that the Fourier transform of
the distribution $\sigma_{5}$ is given by

$\widehat{\sigma}_{5}=-\frac{i}{3\pi}\delta^{\prime\prime\prime}_{0}-\frac{i}{2\pi}\sum_{n=1}^{\infty}\frac{r_{5}(n)}{n^{3/2}}\Big{[}(\delta_{\sqrt{n}}-\delta_{-\sqrt{n}})+\sqrt{n}\,(\delta^{\prime}_{\sqrt{n}}+\delta^{\prime}_{-\sqrt{n}})\Big{]}.$
(2.3)

As a consequence we obtain that $\widehat{\sigma}_{5}$ is also supported on
the set of points $\pm\sqrt{n}$.

### 2.3.

Theorem 2.1 is a special case of the next result, which yields a different
summation formula for each $k=3,5,7,9,\dots$. To state the result, we define
the numbers

$\alpha_{k}:=\frac{(-1)^{(k-3)/2}}{(k-2)!!}\Big{(}\frac{1}{2\pi}\Big{)}^{(k-3)/2},\quad\beta_{j,k}:=\frac{(-1)^{j}\,(k-j-3)!}{j!(k-2j-3)!!}\Big{(}\frac{1}{2\pi}\Big{)}^{(k-3)/2}$
(2.4)

where $k\geqslant 3$ is an odd integer, and $0\leqslant j\leqslant(k-3)/2$.

######  Theorem 2.2.

Let $\varphi$ be an odd Schwartz function on $\mathbb{R}$, and let
$\psi=\widehat{\varphi}$ be the Fourier transform of $\varphi$. For each odd
integer $k\geqslant 3$ we have

$\varphi^{\prime}(0)+\sum_{n=1}^{\infty}\frac{r_{k}(n)}{\sqrt{n}}\varphi(\sqrt{n})=i\alpha_{k}\,\psi^{(k-2)}(0)+i\sum_{n=1}^{\infty}\frac{r_{k}(n)}{n^{(k-2)/2}}\Big{[}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,n^{j/2}\,\psi^{(j)}(\sqrt{n})\Big{]},$

where the coefficients $\alpha_{k}$ and $\beta_{j,k}$ are given by (2.4).

We thus obtain, for each $k=3,5,7,9,\dots$, a new Poisson type summation
formula with weights involving the sum-of-$k$-squares function $r_{k}(n)$. The
nodes remain at the points $\pm\sqrt{n}$, but the number of derivatives at
each node depends on $k$ and it increases with $k$. We observe that (1.2) and
(2.2) are the special cases obtained for $k=3$ and $5$.

The result is equivalent to saying that for each odd integer $k\geqslant 3$,
the Fourier transform of the distribution $\sigma_{k}$ defined by (2.1) is
given by

$\widehat{\sigma}_{k}=2i\alpha_{k}\delta^{(k-2)}_{0}-i\sum_{n=1}^{\infty}\frac{r_{k}(n)}{n^{(k-2)/2}}\Big{[}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,n^{j/2}\,\Big{(}(-1)^{j}\,\delta^{(j)}_{\sqrt{n}}-\delta^{(j)}_{-\sqrt{n}}\Big{)}\Big{]}.$
(2.5)

In particular, $\sigma_{k}$ is a temperate distribution supported on the
locally finite set of points $\pm\sqrt{n}$, and its Fourier transform
$\widehat{\sigma}_{k}$ is supported on the same set.

### 2.4.

Our proof of the results above is inspired by Meyer’s proof in [Mey16] of
Guinand’s formula (1.2). We will obtain Theorem 2.2 as a consequence of the
following result, which generalizes [Mey16, Theorem 7].

######  Theorem 2.3.

Let $\mu=\sum_{\lambda\in\Lambda}a(\lambda)\delta_{\lambda}$ be a measure on
$\mathbb{R}^{k}$, $k\in\\{3,5,7,9,\dots\\}$, whose support $\Lambda$ is a
locally finite set. Suppose that $\mu$ is a temperate distribution, and that
its Fourier transform $\widehat{\mu}=\sum_{s\in S}b(s)\delta_{s}$ is also a
measure, supported on a locally finite set $S$. Assume that $0\notin\Lambda$,
$0\notin S$, and let $\sigma$ be a measure on $\mathbb{R}$ defined by

$\sigma:=\sum_{\lambda\in\Lambda}\frac{a(\lambda)}{|\lambda|}(\delta_{|\lambda|}-\delta_{-|\lambda|}).$
(2.6)

Then $\sigma$ is a temperate distribution whose one-dimensional Fourier
transform is

$\widehat{\sigma}=-i\sum_{s\in
S}\frac{b(s)}{|s|^{k-2}}\Big{[}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,|s|^{j}\,\Big{(}(-1)^{j}\,\delta^{(j)}_{|s|}-\delta^{(j)}_{-|s|}\Big{)}\Big{]},$
(2.7)

where the coefficients $\beta_{j,k}$ are defined in (2.4).

The result in [Mey16, Theorem 7] corresponds to the case $k=3$ in Theorem 2.3.

We observe that the measure $\sigma$ in (2.6) is supported on the locally
finite subset $\\{\pm|\lambda|:\lambda\in\Lambda\\}$ of the real line. It
follows from Theorem 2.3 that the Fourier transform $\widehat{\sigma}$ is a
distribution supported on the (also locally finite) set $\\{\pm|s|:s\in S\\}$.

In Section 5 we will give a more general version of Theorem 2.3, in which the
origin is allowed to belong to the supports $\Lambda,S$ of the measures $\mu$
and $\widehat{\mu}$ (Theorem 5.1).

### 2.5.

As an example, Theorem 2.3 applies to the measure
$\mu_{(\eta,\xi)}=\sum_{m\in\mathbb{Z}^{k}}e^{2\pi i\langle
m,\xi\rangle}\delta_{m+\eta}$ where $\eta$ and $\xi$ are two vectors in
$\mathbb{R}^{k}\setminus\mathbb{Z}^{k}$. It follows from Poisson’s formula
that the Fourier transform of $\mu_{(\eta,\xi)}$ is given by
$\widehat{\mu}_{(\eta,\xi)}=e^{-2\pi
i\langle\eta,\xi\rangle}\mu_{(\xi,-\eta)}$. Hence applying Theorem 2.3 to this
measure yields the following generalization of [Mey16, Theorem 5].

######  Corollary 2.4.

Let $\eta$ and $\xi$ be two vectors in
$\mathbb{R}^{k}\setminus\mathbb{Z}^{k}$, $k\in\\{3,5,7,9,\dots\\}$. Then the
one-dimensional Fourier transform of the measure

$\sigma:=\sum_{m\in\mathbb{Z}^{k}}\frac{e^{2\pi i\langle
m,\xi\rangle}}{|m+\eta|}(\delta_{|m+\eta|}-\delta_{-|m+\eta|})$ (2.8)

is the distribution

$\widehat{\sigma}=-ie^{-2\pi
i\langle\eta,\xi\rangle}\sum_{m\in\mathbb{Z}^{k}}\frac{e^{-2\pi i\langle
m,\eta\rangle}}{|m+\xi|^{k-2}}\Big{[}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,|m+\xi|^{j}\,\Big{(}(-1)^{j}\,\delta^{(j)}_{|m+\xi|}-\delta^{(j)}_{-|m+\xi|}\Big{)}\Big{]}.$
(2.9)

### 2.6.

In order to prove Theorem 2.2, we will give in Section 5 a more general
version of Theorem 2.3 (Theorem 5.1), in which the supports $\Lambda,S$ of the
measures $\mu$ and $\widehat{\mu}$ are allowed to contain the origin. Theorem
2.2 will then be deduced by applying this result to the Poisson measure
$\mu=\sum_{m\in\mathbb{Z}^{k}}\delta_{m}$ which satisfies $\widehat{\mu}=\mu$.

We observe that (in the same spirit as Meyer’s proof of Guinand’s formula)
there exist distributional limits for (2.8) and (2.9) as $\eta$ and $\xi$ tend
to zero, and these limits coincide with (2.1) and (2.5) respectively.

The rest of the paper is devoted to the proofs of the results stated above.

##  3\. Preliminaries

In this section we briefly recall some preliminary background in the theory of
Schwartz distributions (see [Rud91] for more details) and obtain a few basic
results that will be used later on.

### 3.1.

We denote by $\mathcal{D}(\mathbb{R}^{k})$ the space of all infinitely smooth,
compactly supported functions on $\mathbb{R}^{k}$. A sequence of functions
$\varphi_{j}$ in $\mathcal{D}(\mathbb{R}^{k})$ is said to converge in the
space $\mathcal{D}(\mathbb{R}^{k})$ if (i) there exists a compact set
$K\subset\mathbb{R}^{k}$ such that $\operatorname{supp}(\varphi_{j})\subset K$
for all $j$; and (ii) for each multi-index $m=(m_{1},\dots,m_{k})$, the
sequence of partial derivatives $\partial^{m}\varphi_{j}$ converges uniformly
on $K$ as $j\to\infty$. A _distribution_ is a linear functional on
$\mathcal{D}(\mathbb{R}^{k})$ which is continuous with respect to the
convergence in the space $\mathcal{D}(\mathbb{R}^{k})$.

We denote by $\langle\sigma,\varphi\rangle$ the action of a distribution
$\sigma$ on a function $\varphi$ belonging to $\mathcal{D}(\mathbb{R}^{k})$.

The _Schwartz space_ $\mathcal{S}(\mathbb{R}^{k})$ consists of all infinitely
smooth functions $\varphi$ on $\mathbb{R}^{k}$ such that for each $n\geqslant
0$ and each multi-index $m=(m_{1},\dots,m_{k})$, the norm

$\|\varphi\|_{n,m}:=\sup_{x\in\mathbb{R}^{k}}|x|^{n}|\partial^{m}\varphi(x)|$

is finite. A sequence $\varphi_{j}$ is said to converge in the space
$\mathcal{S}(\mathbb{R}^{k})$ if it converges with respect to each one of the
norms $\|\cdot\|_{n,m}$. A _temperate distribution_ is a distribution that can
be extended to a linear functional on the space $\mathcal{S}(\mathbb{R}^{k})$
in such a way that the extension is continuous with respect to the convergence
in the space $\mathcal{S}(\mathbb{R}^{k})$.

If $\sigma$ is a temperate distribution then the notation
$\langle\sigma,\varphi\rangle$ can be extended to denote the action of
$\sigma$ on any function $\varphi$ from the Schwartz space
$\mathcal{S}(\mathbb{R}^{k})$.

If $\varphi$ is a Schwartz function on $\mathbb{R}^{k}$ then its Fourier
transform is defined by

$\widehat{\varphi}(\xi)=\int_{\mathbb{R}^{k}}\varphi(x)\,e^{-2\pi
i\langle\xi,x\rangle}dx.$

The Fourier transform $\widehat{\sigma}$ of a temperate distribution $\sigma$
is defined by
$\langle\widehat{\sigma},\varphi\rangle=\langle\sigma,\widehat{\varphi}\rangle$.

A distribution $\sigma$ on $\mathbb{R}$ is said to be _even_ (respectively
_odd_) if we have $\langle\sigma,\varphi\rangle=0$ for every odd (respectively
even) function $\varphi\in\mathcal{D}(\mathbb{R})$.

### 3.2.

The following result is a version of the well-known Hadamard Lemma, which
states that if $f$ is a function of the class $C^{r}(\mathbb{R})$ (that is,
$f$ is $r$-times continuously differentiable) such that $f(0)=0$, then
$f(t)=tg(t)$ where $g$ is a function of the class $C^{r-1}(\mathbb{R})$.

######  Lemma 3.1.

Let $f$ be a function in the Schwartz space $\mathcal{S}(\mathbb{R})$,
$f(0)=0$. Then

$f(t)=tg(t),\quad t\in\mathbb{R},$ (3.1)

where $g$ is a function in $\mathcal{S}(\mathbb{R})$. Moreover, the mapping
which takes $f$ to $g$ is a continuous linear mapping from the space of
Schwartz functions vanishing at the origin into $\mathcal{S}(\mathbb{R})$.

###### Proof.

Define $g(t):=\int_{0}^{1}f^{\prime}(tu)du$, then $g$ is an infinitely smooth
function and satisfies condition (3.1). The fact that $g(t)=f(t)/t$ $(t\neq
0)$ implies that $\|g\|_{n,m}$ is finite for every $n$ and $m$, and thus
$g\in\mathcal{S}(\mathbb{R})$. The mapping which takes $f$ to $g$ is thus a
linear mapping from the space of Schwartz functions vanishing at the origin
into $\mathcal{S}(\mathbb{R})$. Now suppose that $f_{j}\to f$ in
$\mathcal{S}(\mathbb{R})$. If we denote by $g_{j}$ and $g$ the functions
corresponding to $f_{j}$ and $f$ respectively by the mapping, then $g_{j}\to
g$ pointwise. The continuity of the mapping thus follows from the closed graph
theorem (see [Rud91, Theorem 2.15]). ∎

### 3.3.

######  Lemma 3.2.

Let $f$ be an even function in the Schwartz space $\mathcal{S}(\mathbb{R})$,
and let $F$ be a radial function on $\mathbb{R}^{k}$ defined by
$F(x):=f(|x|)$, $x\in\mathbb{R}^{k}$. Then $F$ belongs to the Schwartz space
$\mathcal{S}(\mathbb{R}^{k})$. Moreover, the mapping which takes $f$ to $F$ is
a continuous linear mapping from the space of even functions in
$\mathcal{S}(\mathbb{R})$ into $\mathcal{S}(\mathbb{R}^{k})$.

###### Proof.

Since $f$ is an even function we have $f^{\prime}(0)=0$. Hence by Lemma 3.1
there is a function $g\in\mathcal{S}(\mathbb{R})$ such that
$f^{\prime}(t)=tg(t)$, $t\in\mathbb{R}$. We claim that

$\frac{\partial F}{\partial x_{i}}(x)=g(|x|)x_{i},\quad x=(x_{1},\dots,x_{k})$
(3.2)

for each $1\leqslant i\leqslant k$. Indeed, this follows from the chain rule
for $x\neq 0$, while for $x=0$ we simply observe that both sides of (3.2)
vanish. Since the right-hand side of (3.2) is a continuous function for each
$i$, we obtain that $F$ is a function in the class $C^{1}(\mathbb{R}^{k})$.

Applying the same considerations to the (even) function $g$ in place of $f$,
we obtain that the function $G(x):=g(|x|)$ is also in the class
$C^{1}(\mathbb{R}^{k})$. This implies that the right-hand side of (3.2)
belongs to $C^{1}(\mathbb{R}^{k})$ for each $i$, and it follows that $F\in
C^{2}(\mathbb{R}^{k})$. Continuing in this way we obtain that $F\in
C^{r}(\mathbb{R}^{k})$ for every $r$, hence $F$ is infinitely smooth.

It is straightforward to check that $\|F\|_{n,m}$ is finite for every $n$ and
every multi-index $m=(m_{1},\dots,m_{k})$, which implies that
$F\in\mathcal{S}(\mathbb{R}^{k})$. The mapping which takes $f$ to $F$ is thus
a linear mapping from the space of even functions in $\mathcal{S}(\mathbb{R})$
into $\mathcal{S}(\mathbb{R}^{k})$.

Finally, suppose that $f_{j}\to f$ in $\mathcal{S}(\mathbb{R})$. If $F_{j}$
and $F$ are the functions corresponding to $f_{j}$ and $f$ respectively by the
mapping, then $F_{j}\to F$ pointwise. This establishes the continuity of the
mapping again as a consequence of the closed graph theorem. ∎

##  4\. Fourier transform of a radial function in odd dimensions

### 4.1.

Let $f$ be an even Schwartz function on $\mathbb{R}$, and suppose that we use
$f$ to construct a radial function $F_{k}$ on $\mathbb{R}^{k}$, defined by
$F_{k}(x):=f(|x|)$. The $k$-dimensional Fourier transform $\widehat{F}_{k}$ of
the function $F_{k}$ is again a radial function, so there is an even function
$f_{k}$ on $\mathbb{R}$ such that $\widehat{F}_{k}(\xi)=f_{k}(|\xi|)$. It
turns out that if the dimension $k$ is an odd integer, then there exists an
explicit expression for the new function $f_{k}$ in terms of the one-
dimensional Fourier transform $\widehat{f}$ and its derivatives. This
expression is given in the following result:

######  Theorem 4.1.

Let $f$ be an even Schwartz function on $\mathbb{R}$, and let $k\geqslant 3$
be an odd integer. Define a radial function $F_{k}$ on $\mathbb{R}^{k}$ by
$F_{k}(x):=f(|x|)$. Then the $k$-dimensional Fourier transform of $F_{k}$ is
given by

$\widehat{F}_{k}(\xi)=-\frac{1}{2\pi|\xi|^{k-1}}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,|\xi|^{j+1}\,\widehat{f}^{\>(j+1)}(|\xi|),\quad\xi\in\mathbb{R}^{k}\setminus\\{0\\}$
(4.1)

where the coefficients $\beta_{j,k}$ are defined as in (2.4), and where
$\widehat{f}$ is the one-dimensional Fourier transform of the function $f$.

This result was obtained in [GT13, Corollary 1.2]. In this paper the authors
also give an expression for $\widehat{F}_{k}$ for even dimensions $k$, but in
this case the expression is not “explicit”, i.e. it is not given in terms of
the one-dimensional Fourier transform $\widehat{f}$. Below we give a proof of
Theorem 4.1 that is different from the one in [GT13] (our method can also be
used to establish the corresponding result for even dimensions $k$).

The assumption in Theorem 4.1 that $f$ is a Schwartz function is not essential
for the conclusion. The result remains valid (with the same proof) e.g. for
any $f$ such that $(1+|t|)^{k-1}f(t)$ is in $L^{1}(\mathbb{R})$, a condition
which ensures that $F_{k}$ is a function in $L^{1}(\mathbb{R}^{k})$.

### 4.2.

We now turn to the proof of Theorem 4.1. We begin with a simple claim about a
recurrence relation satisfied by the Fourier transforms of the surface
measures on the unit spheres in $\mathbb{R}^{k}$.

######  Lemma 4.2.

Let $\mu_{k}$ denote the surface measure on the $(k-1)$-dimensional unit
sphere in $\mathbb{R}^{k}$. The Fourier transform $\widehat{\mu}_{k}$ is a
radial function, hence there is an even function $s_{k}$ satisfying
$\widehat{\mu}_{k}(\xi)=s_{k}(|\xi|)$, $\xi\in\mathbb{R}^{k}$. Then the
recurrence relation

$s_{k}(t)=(2\pi t^{2})^{-1}((k-4)s_{k-2}(t)-2\pi s_{k-4}(t)),\quad t\neq 0,$
(4.2)

is valid for each (even or odd) integer $k\geqslant 5$.

This follows from the recurrence relation satisfied by the Bessel functions:

###### Proof of Lemma 4.2.

It is well-known (see for instance [SW71, p. 154]) that the Fourier transform
of the surface measure $\mu_{k}$ satisfies
$\widehat{\mu}_{k}(\xi)=2\pi|\xi|^{-(k-2)/2}J_{(k-2)/2}(2\pi|\xi|)$, where
$J_{\nu}$ is the Bessel function of order $\nu$. It is also known that the
Bessel functions satisfy the recurrence relation $2\nu
z^{-1}J_{\nu}(z)=J_{\nu-1}(z)+J_{\nu+1}(z)$, see [Leb72, Section 5.3]. For
$\nu=(k-4)/2$ this relation yields (4.2), as one can verify in a
straightforward manner. ∎

### 4.3.

Equipped with Lemma 4.2, we can now give the proof of Theorem 4.1.

###### Proof of Theorem 4.1.

Recall that given an even Schwartz function $f$ on $\mathbb{R}$ we define a
radial function $F_{k}$ on $\mathbb{R}^{k}$ by $F_{k}(x):=f(|x|)$. We observe
that $F_{k}$ is in the Schwartz space, by Lemma 3.2. Since the Fourier
transform of a radial function is also radial, there is an even function
$f_{k}$ such that $\widehat{F}_{k}(\xi)=f_{k}(|\xi|)$, $\xi\in\mathbb{R}^{k}$.
This allows us to define a linear operator $A_{k}$ on the space of even
Schwartz functions on $\mathbb{R}$, given by $A_{k}f:=f_{k}$.

Using $k$-dimensional spherical integration we have

$\widehat{F}_{k}(\xi)=\int_{\mathbb{R}^{k}}f(|x|)\,e^{-2\pi
i\langle\xi,x\rangle}\,dx=\int_{0}^{\infty}f(r)\widehat{\mu}_{k}(r\xi)r^{k-1}\,dr,$
(4.3)

where $\mu_{k}$ is the surface measure on the $(k-1)$-dimensional unit sphere
in $\mathbb{R}^{k}$. If we let $s_{k}$ be the function defined in Lemma 4.2,
then this yields the formula

$A_{k}f(t)=\int_{0}^{\infty}f(r)s_{k}(rt)r^{k-1}\,dr.$ (4.4)

We note that this formula is well-known and can be found e.g. in [SW71, p.
155].

Next, we define another family of operators $B_{k}$ $(k=1,3,5,7,\dots)$ on the
space of even Schwartz functions on $\mathbb{R}$. We let
$B_{1}f(t):=\widehat{f}(t)$; while for $k=3,5,7,\dots$ we define

$B_{k}f(t):=-\frac{1}{2\pi
t^{k-1}}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,t^{j+1}\,\widehat{f}^{\>(j+1)}(t),\quad
t\neq 0$ (4.5)

(we do not specify the value of $B_{k}f$ at the point $t=0$). The assertion in
Theorem 4.1 may thus be reformulated by saying that $A_{k}f(t)=B_{k}f(t)$,
$t\neq 0$, for every even function $f$ in $\mathcal{S}(\mathbb{R})$. A
straightforward calculation whose details are omitted shows that the operators
$\\{B_{k}\\}$, $k=1,3,5,7,\dots$, satisfy the recurrence relation

$B_{k}f(t)=\frac{k-4}{2\pi
t^{2}}\,B_{k-2}f(t)-\frac{1}{t^{2}}B_{k-4}(t^{2}f(t)).$ (4.6)

In order to prove our claim, we will show that the sequence $\\{A_{k}\\}$,
$k=1,3,5,7,\dots$ satisfies the same recurrence relation as (4.6) with the
same initial conditions, and the conclusion will thus follow by induction on
$k$.

Indeed, using (4.4) together with the recurrence relation (4.2) we get

$A_{k}f(t)=\frac{k-4}{2\pi
t^{2}}\int_{0}^{\infty}f(r)r^{k-3}s_{k-2}(rt)\,dr-\frac{1}{t^{2}}\int_{0}^{\infty}(r^{2}f(r))r^{k-5}s_{k-4}(rt)\,dr,$

and thus we see that

$A_{k}f(t)=\frac{k-4}{2\pi
t^{2}}A_{k-2}f(t)-\frac{1}{t^{2}}A_{k-4}(t^{2}f(t)).$ (4.7)

It remains to check the base cases $k=1$ and $3$. If $k=1$ then
$F_{1}(x)=f(x)$ and so $A_{1}f(t)=\widehat{f}(t)=B_{1}f(t)$. For $k=3$ we have
$s_{3}(t)=2\sin(2\pi t)/t$, and (4.4) implies that

$A_{3}f(t)=\frac{2}{t}\int_{0}^{\infty}rf(r)\sin(2\pi rt)\,dr.$

But since $f$ is an even function, this yields

$A_{3}f(t)=\frac{1}{t}\int_{\mathbb{R}}rf(r)\sin(2\pi rt)\,dr=-\frac{1}{2\pi
t}\widehat{f}\;^{\prime}(t)=B_{3}f(t),$

as required. This concludes the proof of Theorem 4.1. ∎

######  Remark 4.3.

Theorem 4.1 provides an expression for $\widehat{F}_{k}(\xi)$ for all
$\xi\in\mathbb{R}^{k}\setminus\\{0\\}$, that is, except for the value at
$\xi=0$. This value may be not easy to find directly from (4.1) by continuity.
Fortunately, using a simple argument one can obtain also an explicit
expression for $\widehat{F}_{k}(0)$. Indeed, since $f$ is an even function, it
follows from (4.3) that

$\widehat{F}_{k}(0)=\widehat{\mu}_{k}(0)\int_{0}^{\infty}f(r)r^{k-1}\,dr=\tfrac{1}{2}\widehat{\mu}_{k}(0)\int_{\mathbb{R}}f(r)r^{k-1}\,dr.$

We observe that $\widehat{\mu}_{k}(0)$ is the total surface area of the unit
sphere in $\mathbb{R}^{k}$, which (since $k$ is odd) is known to be equal to
$2(2\pi)^{(k-1)/2}/(k-2)!!$. This yields the expression

$\widehat{F}_{k}(0)=-\frac{\alpha_{k}}{2\pi}\,\widehat{f}^{\>(k-1)}(0),$ (4.8)

where $\alpha_{k}$ is defined as in (2.4).

### 4.4.

As an application of Theorem 4.1 we consider the Fourier transform of the
surface measure on the $(k-1)$-dimensional unit sphere in $\mathbb{R}^{k}$. It
is well-known that for odd dimensions $k$, the Fourier transform of this
measure is expressible as a finite sum using powers, sines and cosines (this
can be inferred e.g. from the recurrence relation given in Lemma 4.2). Theorem
4.1 allows us to derive a closed-form expression for the Fourier transform of
the surface measure (such formulas are in fact known, see e.g. [KF49]).

######  Theorem 4.4.

Let $\mu_{k}$ be the surface measure on the $(k-1)$-dimensional unit sphere in
$\mathbb{R}^{k}$. If $k$ is an odd integer, $k\geqslant 3$, then the Fourier
transform of the measure $\mu_{k}$ is

$\widehat{\mu}_{k}(\xi)=\frac{2}{|\xi|^{k-2}}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,(2\pi|\xi|)^{j}\sin\big{(}2\pi|\xi|+\tfrac{\pi}{2}j\big{)},\quad\xi\in\mathbb{R}^{k}\setminus\\{0\\},$
(4.9)

where the coefficients $\beta_{j,k}$ are as in (2.4).

###### Proof.

Choose a non-negative, infinitely smooth, compactly supported, even function
$\varphi$ on $\mathbb{R}$, such that $\int_{\mathbb{R}}\varphi(t)dt=1$. For
each $\varepsilon>0$ define the function

$f_{\varepsilon}(t):=\varepsilon^{-1}\big{(}\varphi((t-1)/\varepsilon)+\varphi((t+1)/\varepsilon)\big{)},$

then $f_{\varepsilon}$ is an even function and
$f_{\varepsilon}\to\delta_{1}+\delta_{-1}$ as $\varepsilon\to 0$ in the sense
of distributions. In particular it follows from (4.4) that
$A_{k}f_{\varepsilon}(t)$ tends to $s_{k}(t)$ pointwise as $\varepsilon\to 0$,
where $s_{k}$ is the even function on $\mathbb{R}$ which satisfies
$\widehat{\mu}_{k}(\xi)=s_{k}(|\xi|)$, $\xi\in\mathbb{R}^{k}$.

Recall that in the proof of Theorem 4.1 we have shown that
$A_{k}f(t)=B_{k}f(t)$, $t\neq 0$, for every even function $f$ in the Schwartz
space, where $B_{k}f(t)$ is defined as in (4.5). We can therefore obtain the
value of $s_{k}(t)$ for $t\neq 0$ as the limit of $B_{k}f_{\varepsilon}(t)$ as
$\varepsilon\to 0$.

First we observe that
$\widehat{f}_{\varepsilon}(t)=2\,\widehat{\varphi}(\varepsilon t)\cos(2\pi
t)$. Then by the Leibniz rule we have

$\widehat{f}_{\varepsilon}^{\>(j+1)}(t)=2\,\widehat{\varphi}(\varepsilon
t)\big{\\{}\cos(2\pi t)\big{\\}}^{(j+1)}+\cdots,$

where the omitted terms tend to zero as $\varepsilon\to 0$. This implies that

$\lim_{\varepsilon\to
0}\widehat{f}_{\varepsilon}^{\>(j+1)}(t)=2\big{\\{}\cos(2\pi
t)\big{\\}}^{(j+1)}=-2(2\pi)^{j+1}\sin\big{(}2\pi t+\tfrac{\pi}{2}j\big{)}.$
(4.10)

Finally, combining (4.5) and (4.10) yields

$s_{k}(t)=\lim_{\varepsilon\to
0}B_{k}f_{\varepsilon}(t)=\frac{2}{t^{k-2}}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,(2\pi
t)^{j}\,\sin\big{(}2\pi t+\tfrac{\pi}{2}j\big{)},\quad t\neq 0,$ (4.11)

which is equivalent to the assertion in (4.9). ∎

######  Remark 4.5.

We could have proved Theorem 4.1 and Theorem 4.4 also the other way around,
that is, at the first stage we could have proved Theorem 4.4 by induction on
$k$, showing that (4.9) is valid using the recurrence relation (4.2) and
checking the base cases $k=3$ and $5$; while at the second stage we could have
derived Theorem 4.1 based on (4.3) and (4.9). We have opted for giving a
direct proof of Theorem 4.1, since it is the result that will be used in the
next section.

######  Remark 4.6.

The formula (4.9) for the Fourier transform of the surface measure $\mu_{k}$
is closely related to the _Bessel polynomials_ $\theta_{n}(z)$. This is a
sequence of polynomials satisfying the recurrence relation
$\theta_{n}(z)=(2n-1)\theta_{n-1}(z)+z^{2}\theta_{n-2}(z)$ with the initial
conditions $\theta_{0}(z)=1$ and $\theta_{1}(z)=z+1$. The $n$’th polynomial in
the sequence can be written explicitly as
$\theta_{n}(z)=(2\pi)^{n}\sum_{j=0}^{n}\beta_{j,k}(-z)^{j}$, where $k=2n+3$
and where the coefficients $\beta_{j,k}$ are the same as in (2.4). We can
therefore rewrite (4.9) in the form

$\widehat{\mu}_{k}(\xi)=\frac{2}{|\xi|^{k-2}}\,\operatorname{Im}\left\\{\frac{\theta_{n}(-2\pi
i|\xi|)}{(2\pi)^{n}}\,\exp(2\pi i|\xi|)\right\\},\quad n=\frac{k-3}{2},$
(4.12)

which clarifies the relation between the Bessel polynomials and the Fourier
transform of the surface measure $\mu_{k}$ for odd dimensions $k$. We refer to
[Gro78] for a comprehensive survey on the Bessel polynomials (see also
[KF49]).

##  5\. New Poisson type summation formulas

In this section we prove the results stated in Section 2. We also state and
prove Theorem 5.1 that generalizes Theorem 2.3 to the case where the origin is
allowed to belong to the supports $\Lambda,S$ of the measures $\mu$ and
$\widehat{\mu}$.

### 5.1.

First we will prove Theorem 2.3 which generalizes [Mey16, Theorem 7] to all
odd values of $k$ greater than $3$. Meyer’s proof in the case $k=3$ was based
on the following observation: Let a radial function $G$ on $\mathbb{R}^{3}$ be
defined by $G(x):=f(|x|)/|x|$, where $f$ is an odd Schwartz function on
$\mathbb{R}$. Then the three-dimensional Fourier transform of the function $G$
is given by $\widehat{G}(\xi)=i\widehat{f}(|\xi|)/|\xi|$.

The proof of Theorem 2.3 for an arbitrary odd integer $k\geqslant 3$ will
follow a similar line, where in this case we will use the connection given in
Theorem 4.1 between the one-dimensional Fourier transform of an even Schwartz
function on $\mathbb{R}$ and the $k$-dimensional Fourier transform of a radial
function on $\mathbb{R}^{k}$.

###### Proof of Theorem 2.3.

First we show that the distribution $\sigma$ defined by (2.6) is indeed a
temperate distribution. If $f$ is a function in the Schwartz space
$\mathcal{S}(\mathbb{R})$, then by Lemma 3.1 we have $f(t)-f(-t)=tg(t)$ where
$g$ is also in $\mathcal{S}(\mathbb{R})$. Then $g$ is an even function and so
by Lemma 3.2 the function $G(x):=g(|x|)$, $x\in\mathbb{R}^{k}$, is in the
Schwartz space $\mathcal{S}(\mathbb{R}^{k})$. Moreover, the mapping which
takes $f$ to $G$ is a continuous linear mapping from the space
$\mathcal{S}(\mathbb{R})$ into $\mathcal{S}(\mathbb{R}^{k})$. Observe that if
$f$ is compactly supported then we have

$\langle\sigma,f\rangle=\sum_{\lambda\in\Lambda}\frac{a(\lambda)}{|\lambda|}(f(|\lambda|)-f(-|\lambda|))=\sum_{\lambda\in\Lambda}a(\lambda)G(\lambda)=\langle\mu,G\rangle.$
(5.1)

Since the measure $\mu$ is assumed to be a temperate distribution, the mapping
$f\mapsto\langle\mu,G\rangle$ thus defines a continuous linear functional on
the Schwartz space $\mathcal{S}(\mathbb{R})$ which agrees with $\sigma$ on the
space $\mathcal{D}(\mathbb{R})$ of infinitely smooth, compactly supported
functions. This means that $\sigma$ is a temperate distribution.

Next we show that the Fourier transform of $\sigma$ is given by (2.7). Since
$\sigma$ is an odd distribution, the Fourier transform of $\widehat{\sigma}$
is $-\sigma$. Hence (2.7) is equivalent to saying that

$\langle\sigma,f\rangle=i\sum_{s\in
S}\frac{b(s)}{|s|^{k-2}}\Big{[}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,|s|^{j}\,\Big{(}\widehat{f}^{\>(j)}(|s|)-(-1)^{j}\widehat{f}^{\>(j)}(-|s|)\Big{)}\Big{]}$
(5.2)

for every Schwartz function $f$ such that $\widehat{f}$ is compactly
supported. If we keep using $g$ and $G$ to denote the two functions related to
$f$ as above, then by Theorem 4.1 we have

$\widehat{G}(\xi)=-\frac{1}{2\pi|\xi|^{k-1}}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,|\xi|^{j+1}\,\widehat{g}^{\>(j+1)}(|\xi|),\quad\xi\neq
0,$ (5.3)

while the property $tg(t)=f(t)-f(-t)$ implies that

$\widehat{g}^{\>(j+1)}(t)=-2\pi
i\big{(}\widehat{f}^{\>(j)}(t)-(-1)^{j}\widehat{f}^{\>(j)}(-t)\big{)}.$ (5.4)

It now follows using (5.3) and (5.4) that the right-hand side of (5.2) is
equal to

$\sum_{s\in
S}b(s)\widehat{G}(s)=\langle\widehat{\mu},\widehat{G}\rangle=\langle\mu,G\rangle=\langle\sigma,f\rangle$
(5.5)

as we had to show. (We note that in the second equality in (5.5) we used the
fact that the Fourier transform of $\widehat{G}$ is $G$, which follows from
the property $G(-x)=G(x)$.) ∎

### 5.2.

Corollary 2.4 now follows easily as a special case of Theorem 2.3.

###### Proof of Corollary 2.4.

We apply Theorem 2.3 to the measure

$\mu_{(\eta,\xi)}=\sum_{m\in\mathbb{Z}^{k}}e^{2\pi i\langle
m,\xi\rangle}\delta_{m+\eta}$

where the vectors $\eta$ and $\xi$ are in
$\mathbb{R}^{k}\setminus\mathbb{Z}^{k}$. By Poisson’s summation formula, the
Fourier transform of $\mu_{(\eta,\xi)}$ is the measure
$\widehat{\mu}_{(\eta,\xi)}=e^{-2\pi
i\langle\eta,\xi\rangle}\mu_{(\xi,-\eta)}$. Since neither $\eta$ nor $\xi$
belongs to $\mathbb{Z}^{k}$, the origin lies in neither the support of the
measure $\mu_{(\eta,\xi)}$ nor the support of its Fourier transform. Hence all
the assumptions in Theorem 2.3 are satisfied and the assertion in Corollary
2.4 follows. ∎

### 5.3.

Now we give a more general version of Theorem 2.3, in which the origin is
allowed to lie in the support $\Lambda$ of the measure $\mu$ or the support
$S$ of its Fourier transform $\widehat{\mu}$ (or both). The result can be
stated as follows:

######  Theorem 5.1.

Let $\mu$ be a measure satisfying the same assumptions as in Theorem 2.3
except that $\Lambda$ and $S$ may contain the origin. Then

$\sigma:=-2a(0)\delta^{\prime}_{0}+\sum_{\lambda\in\Lambda\setminus\\{0\\}}\frac{a(\lambda)}{|\lambda|}(\delta_{|\lambda|}-\delta_{-|\lambda|})$
(5.6)

is a temperate distribution on $\mathbb{R}$ whose one-dimensional Fourier
transform is

$\widehat{\sigma}=2ib(0)\alpha_{k}\delta^{(k-2)}_{0}-i\sum_{s\in
S\setminus\\{0\\}}\frac{b(s)}{|s|^{k-2}}\Big{[}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,|s|^{j}\,\Big{(}(-1)^{j}\,\delta^{(j)}_{|s|}-\delta^{(j)}_{-|s|}\Big{)}\Big{]},$
(5.7)

where the coefficients $\alpha_{k}$ and $\beta_{j,k}$ are defined as in (2.4).

If the support $\Lambda$ of the measure $\mu$ does not contain the origin,
then we understand the coefficient $a(0)$ in (5.6) to be zero; and similarly,
if the origin does not belong to the support $S$ of the measure
$\widehat{\mu}$, then the coefficient $b(0)$ in (5.7) is zero. If the origin
lies in neither $\Lambda$ nor $S$, the assertion in Theorem 5.1 reduces to
that of Theorem 2.3.

###### Proof of Theorem 5.1.

We keep using the same notation as in the proof of Theorem 2.3. We recall that
if $f$ is a function in the Schwartz space $\mathcal{S}(\mathbb{R})$ then
there is $g\in\mathcal{S}(\mathbb{R})$ such that $f(t)-f(-t)=tg(t)$. Then by
the continuity of $g$ we have $g(0)=2f^{\prime}(0)$, and the equality
$\langle\sigma,f\rangle=\langle\mu,G\rangle$ used in (5.1) remains valid also
in the case when $0\in\Lambda$.

To obtain the Fourier transform of $\sigma$ we argue as before. If $f$ is a
Schwartz function such that $\widehat{f}$ is compactly supported, then

$\langle\sigma,f\rangle=\langle\mu,G\rangle=\langle\widehat{\mu},\widehat{G}\rangle=b(0)\widehat{G}(0)+\sum_{s\neq
0}b(s)\widehat{G}(s).$ (5.8)

If we apply Remark 4.3 to the function $G$, and use (5.4) then we obtain

$\widehat{G}(0)=-\frac{\alpha_{k}}{2\pi}\,\widehat{g}^{\>(k-1)}(0)=2i\alpha_{k}\widehat{f}^{\>(k-2)}(0).$

It thus follows from (5.8) that

$\langle\sigma,f\rangle=2ib(0)\alpha_{k}\widehat{f}^{\>(k-2)}(0)+i\sum_{s\neq
0}\frac{b(s)}{|s|^{k-2}}\Big{[}\;\cdots\;\Big{]},$ (5.9)

where the omitted terms in the square brackets are as in (5.2). This confirms
(5.7). ∎

### 5.4.

Finally we prove Theorem 2.2 that extends Guinand’s formula (1.2) to all odd
values of $k$ greater than $3$. Guinand’s formula is equivalent to the
assertion that $\widehat{\sigma}_{3}=-i\sigma_{3}$, where $\sigma_{k}$ is the
distribution defined in (2.1) that involves the sum-of-$k$-squares function
$r_{k}(n)$. Theorem 2.2 is a consequence of the next result which provides an
expression for the Fourier transform $\widehat{\sigma}_{k}$ for all odd values
of $k$ greater than $3$.

######  Theorem 5.2.

Let $k$ be an odd integer, $k\geqslant 3$. Then the Fourier transform of the
temperate distribution

$\sigma_{k}:=-2\delta^{\prime}_{0}+\sum_{n=1}^{\infty}\frac{r_{k}(n)}{\sqrt{n}}\left(\delta_{\sqrt{n}}-\delta_{-\sqrt{n}}\right)$

is

$\widehat{\sigma}_{k}=2i\alpha_{k}\delta^{(k-2)}_{0}-i\sum_{n=1}^{\infty}\frac{r_{k}(n)}{n^{(k-2)/2}}\Big{[}\sum_{j=0}^{(k-3)/2}\beta_{j,k}\,n^{j/2}\,\Big{(}(-1)^{j}\,\delta^{(j)}_{\sqrt{n}}-\delta^{(j)}_{-\sqrt{n}}\Big{)}\Big{]}.$
(5.10)

###### Proof.

We apply Theorem 5.1 to the measure $\mu=\sum_{m\in\mathbb{Z}^{k}}\delta_{m}$
which satisfies $\widehat{\mu}=\mu$. ∎

##  6\. Remark

It is natural to ask what can be said about the Fourier transform of the
distribution $\sigma_{k}$ defined in (2.1) when $k$ is a positive _even_
integer (i.e. $k=2,4,6,8,\dots$). We expect that for these values of $k$ the
Fourier transform $\widehat{\sigma}_{k}$ does not have a discrete support. It
would be interesting to have a proof of this claim, but we do not address this
problem in the present work.

## References

  * [Fav18] S. Yu. Favorov, Tempered distributions with discrete support and spectrum. Bull. Hellenic Math. Soc. 62 (2018), 66–79.
  * [GT13] L. Grafakos, G. Teschl, On Fourier transforms of radial functions and distributions. J. Fourier Anal. Appl. 19 (2013), no. 1, 167–179.
  * [Gro78] E. Grosswald, Bessel polynomials. Lecture Notes in Mathematics, 698. Springer, Berlin, 1978.
  * [Gui59] A. P. Guinand, Concordance and the harmonic analysis of sequences. Acta Math. 101 (1959), 235–271.
  * [Kol16] M. N. Kolountzakis, Fourier pairs of discrete support with little structure. J. Fourier Anal. Appl. 22 (2016), no. 1, 1–5.
  * [KF49] H. L. Krall, O. Frink, A new class of orthogonal polynomials: The Bessel polynomials. Trans. Amer. Math. Soc. 65 (1949), 100–115.
  * [Lag00] J. C. Lagarias, Mathematical quasicrystals and the problem of diffraction. Directions in mathematical quasicrystals, 61–93, CRM Monogr. Ser., 13, Amer. Math. Soc., Providence, RI, 2000.
  * [Leb72] N. N. Lebedev, Special functions and their applications, Revised edition. Dover Publications, New York, 1972.
  * [LO13] N. Lev, A. Olevskii, Measures with uniformly discrete support and spectrum. C. R. Math. Acad. Sci. Paris 351 (2013), no. 15-16, 599–603.
  * [LO15] N. Lev, A. Olevskii, Quasicrystals and Poisson’s summation formula. Invent. Math. 200 (2015), no. 2, 585–606.
  * [LO16] N. Lev, A. Olevskii, Quasicrystals with discrete support and spectrum. Rev. Mat. Iberoam. 32 (2016), no. 4, 1341–1352.
  * [LO17] N. Lev, A. Olevskii, Fourier quasicrystals and discreteness of the diffraction spectrum. Adv. Math. 315 (2017), 1–26.
  * [Mey95] Y. Meyer, Quasicrystals, Diophantine approximation and algebraic numbers. Beyond quasicrystals (Les Houches, 1994), 3–16, Springer, Berlin, 1995.
  * [Mey16] Y. Meyer, Measures with locally finite support and spectrum. Proc. Natl. Acad. Sci. USA 113 (2016), no. 12, 3152–3158.
  * [Mey17] Y. Meyer, Measures with locally finite support and spectrum. Rev. Mat. Iberoam. 33 (2017), no. 3, 1025–1036.
  * [RV19] D. Radchenko, M. Viazovska, Fourier interpolation on the real line. Publ. Math. Inst. Hautes Études Sci. 129 (2019), 51–81.
  * [Rud91] W. Rudin, Functional analysis, Second edition. McGraw-Hill, New York, 1991.
  * [SW71] E. M. Stein, G. Weiss, Introduction to Fourier analysis on Euclidean spaces. Princeton University Press, 1971.
  * [Via18] M. Viazovska, Sharp sphere packings. Proc. Int. Cong. of Math. 2018, Vol. 2, 455–466.


# Feature Alignment for Approximated Reversibility in Neural Networks

Tiago de Souza Farias<EMAIL_ADDRESS>
Jonas Maziero<EMAIL_ADDRESS>
Departamento de Física, Centro de Ciências Naturais e Exatas
Universidade Federal de Santa Maria
Avenida Roraima 1000, Santa Maria, Rio Grande do Sul, 97105-900, Brazil

###### Abstract

We introduce feature alignment111Code available at
https://github.com/tiago939/feature_aligment., a technique for obtaining
approximate reversibility in artificial neural networks. By means of feature
extraction, we can train a neural network to learn an estimated map for its
reverse process from outputs to inputs. Combined with variational
autoencoders, we can generate new samples from the same statistics as the
training data. Improvements of the results are obtained by using concepts from
generative adversarial networks. Finally, we show that the technique can be
modified for training neural networks locally, saving computational memory
resources. Applying these techniques, we report results for three vision
generative tasks: MNIST, CIFAR-10, and celebA.

Keywords: Neural networks, features, reversibility, local training

##  1 Introduction

Feature extraction (Olah et al., 2017) is a set of techniques aiming to find
inputs, known as features, that maximize the activation of one or more
neurons. Usually, feature extraction is used as a method for model
interpretability, where one seeks to understand a neural network by analyzing
how much each neuron contributes for modelling the training data. The process
of obtaining these features is, in a sense, an attempt towards reversing a
neural network. Since a neural network is composed by functions which map
inputs to outputs, a feature is the input we would have given a target
activation for a group of posterior selected neurons.

Reversibility of neural networks relates to how well one can reverse the map
from the activation of target neurons to the input neurons (Aidan N. Gomez and
Grosse, 2017). Usually, neural networks are not reversible, due to a few
reasons: non-reversible activation functions, for example ReLU (Agarap, 2019),
non-orthogonal weights, and lack of one-to-one relationships. One advantage of
reversible neural networks, besides the reversing mapping, is its memory
efficiency: while non-invertible networks need to store all the activations
for the backward pass during training, reversible networks allows the storage
of fewer activations to update the trainable parameters.

Reversibility also constrains the number of possible models, as there are many
possible parameters configurations which model the data. For example, if one
considers an analytical function that one wishes to approximate by a
sufficiently parameterized neural network, with the pair of data
$\\{x,f(x)\\}$, there are a number of local minima which estimates the
function $x\rightarrow f(x)$, each one obtained by a different random
initialization of the neural network parameters (assuming optimal
convergence). By imposing the restriction of reversibility $f(x)\rightarrow
x$, we can reduce the number of optimal points toward which a neural network
can converge.

Memory is often a bottleneck for neural networks. The backpropagation
algorithm (Linnainmaa, 1976; Rumelhart et al., 1986), widely used in modern
deep learning for training, requires the storage of the activation of all
neurons of a network in order to update its parameters. Local training rules
(Baldi and Sadowski, 2016) allow a more efficient memory optimization of
neural networks. By constraining the trainable parameters, such as the
weights, to be updated only by local variables (the information contained in
the neurons that share the same parameter), we can reduce the memory
requirements to load a model in hardware such as CPUs and GPUs. This
constraint can save memory resources and has many potential applications, from
low-memory devices (Velichko, 2020; Sohoni et al., 2019) to training large
batch sizes (Gao and Zhong, 2020; You et al., 2017), and, even further, to
train very large neural networks (Jing and Xu, 2019).

This paper makes use of feature extraction for inverse mapping, a technique
for approximated reversibility of neural networks that can be implemented in
several architectures. We show that feature alignment makes neural networks
approximately reversible for a given training data. We also adapt the
technique for local training, showing results for the reconstruction of
images.

##  2 Related work

Several works have been done on the area of feature extraction, specially
applied for model interpretability and explainability (Shahroudnejad, 2021;
Fan et al., 2021; Gilpin et al., 2019). These techniques, used for extracting
features, usually consist in activation maximization (Mahendran and Vedaldi,
2016), where a group of neurons, which can involve from a single neuron up to
an entire layer (or channel for convolutions), is selected to extract the
feature by maximising its activation.

Many techniques of feature extraction consist in studying features in already
pretrained classifiers (Nguyen et al., 2016b). Other techniques consist in
searching for features in the latent space (Yujun Shen and Zhou, 2020).
Feature extraction can also be used for understanding which parts of an input
contribute the most for the target activations (Selvaraju et al., 2020;
Springenberg et al., 2015; Zintgraf et al., 2017).

In a generative process, we want to produce new examples with the same
statistical distribution as the training data. There are different techniques
to model the data for generation. Among these techniques, autoencoders,
generative adversarial networks, and normalizing flows are very popular.
Autoencoders (AE) have two networks: an encoder that projects the inputs into
a vector, usually with smaller dimension, and a decoder that reconstructs the
input from this vector. The compressed vector has a high level representation
of the model, in which each neuron contributes to properties beyond the data
level at the input layer (Lee et al., 2011). Autoencoders are commonly trained
in a unsupervised fashion, nevertheless there are variants that include
labeled information to further increase training for a specific objective.
Variational autoencoders (VAE) (Kingma and Welling, 2014, 2019; Doersch, 2021)
improve the sampling ability by projecting the data into a probabilistic
latent vector. Generative adversarial networks (GANs) (Goodfellow et al.,
2014; Salehi et al., 2020; Gui et al., 2020) are another example of a
generative method. By having two competing networks, a generative which takes
a random low-dimensional input and outputs an image, and a discriminator
network that compares the images from the training data set and the sampled
ones from the generator. The competition arises by training the generator to
fool the discriminator by generating images as closest to the training data
set as possible. Normalizing flows (Papamakarios, 2019; Kobyzev et al., 2020;
Kingma and Dhariwal, 2018) generate images by transforming a simple
distribution to a more complex one by a series of reversible transformations.

There have been works combining autoencoders with GANs (Larsen et al., 2016).
The work done in Refs. Nguyen et al. (2016a); Dosovitskiy and Brox (2016) is
related to ours. They synthesized new images with the same statistics as the
training data by imputing features to a generator network. The main difference
is that, in these previous articles, the features are obtained with a
pretrained network.

Most works on reversibility consist in architectural changes of neural
networks (Atapattu and Rekabdar, 2019; Schirrmeister et al., 2018; Grathwohl
et al., 2018; Behrmann et al., 2019; Leemon Baird and Ingkiriwang, 2005).
These changes guarantee a one-to-one relationship between inputs and outputs.
BiGAN (Donahue et al., 2017) constructs a generative network and a reverse
network that inputs images back to noise, which can be used to obtain a latent
representation of a data set directly.

Local learning rules have been explored since Donald Hebb proposed a simple
model for learning in the brain (Hebb, 1949). The main advantage of this kind
of learning algorithm is requiring lower memory resources. Some works are
biologically inspired (Krotov and Hopfield, 2019; Lindsey and Litwin-Kumar,
2020), while others focus solely on efficiency (Isomura and Toyoizumi, 2016,
2018). There is a growing body of work discussing whether the brain does
backpropagation (Whittington and Bogacz, 2019; Song et al., 2020), with some
approximations for training artificial neural networks (Millidge et al., 2020;
Salvatori et al., 2021; Lillicrap et al., 2020).

##  3 Methods

Let $E(\textbf{x};\theta)$ be a neural network with inputs x and trainable
parameters $\theta$. The output is a latent vector $\mathbf{z}_{x}$ with an
arbitrary number of neurons.

###  3.1 Feature alignment

The feature alignment encoder consists of a neural network with parameters
$\theta$ and an arbitrary number of latent variables as the output. From a
data set $\textbf{x}\in\textbf{X}$, $\textbf{z}_{x}=E(\textbf{x};\theta)$ is
the output from an input x. With the same network,
$z_{r}=E(\textbf{r};\theta)$ is the output from a random input $\mathbf{r}$
with the same dimension as the input data.

The feature $\hat{\textbf{r}}$ of $\textbf{z}(\textbf{x})$ is obtained by
minimizing a distance function $\mathcal{L}(\textbf{z}_{x},\textbf{z}_{r})$
with respect to the random inputs r. By minimizing this distance, the random
input will evolve continually according to a gradient flow:

$\frac{\partial r}{\partial t}=-\frac{\partial\mathcal{L}}{\partial r}.$ (1)

Since $\textbf{z}(\textbf{x})$ is fixed, r will evolve such as the function of
the random variables will approximate $\textbf{z}(\textbf{x})$ as much as
possible (See appendix A.1). We want to solve equation $(1)$ as efficiently as
possible in time and memory. By discretizing the gradient flow, we obtain an
approximation for the feature:

$r^{t}=r^{t-1}-\tau\frac{\partial\mathcal{L}}{\partial r},$ (2)

with $\tau$ being a hyperparameter that weights the contribution of the
gradient. Equation $(2)$ is similar to activation maximization, except that we
are minimising for the neurons to have a target activation. These updates are
done in $T$ time steps. Properly optimized, the solution to equation (2)
converges to a feature by approximating the inverse of the weights (see
Appendix 1). So then, optimizing the parameters of the network, the weight
matrix between layers will have the orthogonal property
$\textbf{w}^{T}\textbf{w}=\textbf{I}$, which implies in approximated
reversibility (see Appendix 2).

Figure 1: In feature alignment, we have two encoders that share the same
parameters. Left: at training time, the network is trained to approximate
$\hat{\textbf{r}}\approx\textbf{x}$ . Right: during inference, we sample
$\textbf{z}_{x}$ from either the data or a known distribution to obtain the
features.

After we extract the feature $\hat{\mathbf{r}}$, we measure how similar it is
to the inputs $\mathbf{x}$ by a new loss function
$\mathcal{L}(\mathbf{x},\hat{\mathbf{r}})$. This second loss function is used
for training the encoder by optimizing its parameters. As the neural network
is trained, the encoder learns, not only to map the inputs to the latent
variables, but also the reconstruction of the inputs from the latent vector.
Following training, we can reconstruct the inputs by knowing only the latent
vector. Figure 1 and Algorithm 1 summarize the feature alignment technique.
Notice that we initialize $\mathbf{r}$ with the same shape as $\mathbf{x}$.

Algorithm 1 Training with feature alignment

$\mathbf{z_{x}}=E(\mathbf{x};\theta)$

initialize $\mathbf{r}=\mathbf{x}.shape$

$t=0$

while $t<T$ do

$\mathbf{z_{r}}=E(\mathbf{r};\theta)$

$\mathcal{L}=||\mathbf{z_{x}}-\mathbf{z_{r}}||_{2}^{2}$

$r=r-\tau\frac{\partial\mathcal{C}}{\partial r}$

$t=t+1$

end while

$\hat{\textbf{r}}=\textbf{r}$

$\mathcal{C}=||\mathbf{x}-\hat{\textbf{r}}||_{2}^{2}$

update $\theta$ by optimizing $\mathcal{C}$

###  3.2 Variational autoencoders with feature alignment (VFA)

One issue with the encoder, and autoencoders in general, is its inability to
generate new samples with the same statistical distribution as the training
data. The latent variables from the data are associated to a distribution of
variables which might be too complicated for an effective sampling. To solve
this problem, we use a variational autoencoder (VAE) formulation, except by
removing the decoder network. In the VAE, the output of the encoder is coupled
with two layers that return the mean value $\mu_{x}$ and variance
$\sigma_{x}^{2}$ of the data. We constrain the latent vector to have a
distribution that is easy to sample (usually a Gaussian distribution), by
comparing two probability distributions via a metric such as the Kullback-
Leibler divergence. So, the cost function:

$\mathcal{L}=||\textbf{x}-\hat{\textbf{r}}||_{2}^{2}-\beta\mathcal{D}_{KL}(q_{\theta}(\textbf{z}|\textbf{x})||p(\textbf{z})),$
(3)

is used to train the feature encoder with a constraint to the output latent
variables from a known random probability distribution $p(\mathbf{z}),$ from
which we can easily sample. The constant $\beta$ is a hyper-parameter that
regularizes the latent vector for better disentanglement representation of the
data (Higgins et al., 2016; Burgess et al., 2018; Sikka et al., 2019).

We choose the distribution $p(\mathbf{z})$ according to the principle of
maximum entropy: since the latent variables are in the range
$(-\infty,+\infty)$, the Gaussian distribution is best suited in this case. We
then constrain each latent variable as to come from a Gaussian distribution
with zero mean value and variance equal to one. Just as the VAE, we can not
train the encoder by sampling directly from the mean and variance of the
latent vector. Instead, we use the renormalization trick: we sample a random
vector $\epsilon$ from a normal distribution, the latent vector is then:
$\mathbf{z_{x}}=\mu_{x}+\epsilon\odot\sigma^{2}_{x}$, with $\odot$ the
element-wise product. Note that we do not have an random normal vector for
$\mathbf{z_{r}}$, since its purpose is solely to reconstruct $\mathbf{x}$.
Figure 3 summarizes training a VAE with feature alignment.

Figure 2: Variational autoencoder with feature alignment. Left: training the
encoder to reconstruct the inputs $\mathbf{x}$. Right: we sample a random
normal vector $\epsilon$ to generate new data.

###  3.3 Improving the quality of the features

As will be shown in the results section, the features extracted using the
feature alignment trained with VAE are blurry, and this is due to the
variational autoencoder nature (Rezende and Viola, 2018). To improve the
quality of the generated images, we couple a generator network $G$ and a
discriminator network $D$ to the features. In this way, the feature acts as a
second latent vector, with a more complex distribution sampled from the
simpler one. This generator network is similar to the refiner network
presented in Ref. Atapattu and Rekabdar (2019), which take an image as input
and outputs an improved version of it.

The generator takes the feature as input and produces a new output which is
compared to the input x. The discriminator is trained as a generative
adversarial network, evaluating the probability that $G(\mathbf{\hat{r}})$ is
real or fake. The generator is updated by receiving gradients from the
discriminator. For more stable training, we use the least square loss for the
discriminator (Mao et al., 2017). Alternatively, we can use the Wasserstein
GAN formulation (Arjovsky et al., 2017), which replaces the discriminator by a
critic network that measures the score of the “ _realness_ ” of an image.

To reduce possible effects due to the posterior collapse problem in VAEs
(Takida et al., 2021; Havrylov and Titov, 2020; Lucas et al., 2019) and to
balance with the reconstruction loss, we propose a random schedule for the
$\beta$ variable. For each example during training, we sample $\beta$ from a
uniform distribution $\mathcal{U}(0,1)$.

While the reconstruction of images is optimized usually with pixel level loss,
we can also consider high level properties of the data. This kind of measure,
called perceptual loss (Johnson et al., 2016), compares the output of the
reconstruction with the original image at high level neurons (presented near
the end of the network). Here we consider the perceptual loss using the mean
and variance layers from the encoder network, enforcing the reconstruction to
have the same statistical properties of the original input.

Figure 3: Training in the VFA setting, with the addition of generator and
discriminator networks.

The final losses, for the encoder, generator, and discriminator are,
respectively:

$\displaystyle\mathcal{L}_{E}$
$\displaystyle=||\textbf{x}-\hat{\textbf{r}}||_{2}^{2}+\beta\mathcal{D}_{KL}(q_{\theta}(\textbf{z}|\textbf{x})||p(\textbf{z})),$
(4) $\displaystyle\mathcal{L}_{G}$
$\displaystyle=||1-D(G(\hat{\textbf{r}}))||_{2}^{2}+\lambda(||\mu(\textbf{x})-\mu(G(\hat{\textbf{r}}))||_{2}^{2}+||\sigma^{2}(\textbf{x})-\sigma^{2}(G(\hat{\textbf{r}}))||_{2}^{2}),$
(5) $\displaystyle\mathcal{L}_{D}$
$\displaystyle=||1-D(\textbf{x})||_{2}^{2}+||-1-D(G(\hat{\textbf{r}}))||_{2}^{2}.$
(6)

with $\lambda$ a hyperparameter that weights the perceptual loss contribution.

If a training data has additional information, such as labels, we can train a
neural network jointly with supervised training for specific related tasks
such as conditional generation, where we want samples that correspond to a
desired class. We can condition the latent variables to have different
distributions from different classes. The output of the network is trained
from the Gaussian distribution $\textbf{z}\sim Q(z_{i},c_{i}),$ with $c_{i}$
being a one-hot vector containing the class information. Similar to the work
done in Ref. Ardizzone et al. (2019), we couple a linear classification layer
on top of the network, parallel to the mean and variance layers. Since the
class layer is linear, at inference time we can choose a greater value for the
one-hot vector, which puts emphasis on the chosen class.

###  3.4 Local feature alignment

The rules for feature alignment were presented as a global rule: the auxiliary
loss is defined with the output layer and the loss is defined with the input
layer, so we have full communication between all layers. However, we can
reformulate this rule with local losses, similar to target propagation rules
(Farias and Maziero, 2018; Bengio, 2014; Ororbia et al., 2018): the auxiliary
loss and loss are defined as the interaction between two connected layers only
(or even individual neurons), as follows: for each layer from the first to the
last, we activate it from the its inputs and store a second activation from a
random input with the same dimension, we then extract the feature by
optimizing the random input with an auxiliary loss between activation of the
random output and the true input. Finally, the parameters of the chosen layer
is updated by optimizing the loss between the reconstruction and true input.
This technique of local training is summarized in the Algorithm 2 and
illustrated at FIG. 4.

Algorithm 2 Training with local feature alignment

1:for l=0, L do $\triangleright$ for each layer

2: $\textbf{z}_{x}^{l}=E(\textbf{x};\theta_{l})$

3: initialize $\textbf{r}=\textbf{x}.shape$

4: $t=0$

5: while $t<T$ do

6: $\textbf{z}_{r}^{l}=E(\textbf{r};\theta_{l})$

7: $\mathcal{L}_{l}=||\textbf{z}_{x}^{l}-\textbf{z}_{r}^{l}||_{2}^{2}$

8: $r=r-\tau\frac{\partial\mathcal{L}_{l}}{\partial r}$

9: $t=t+1$

10: end while

11: $\hat{\textbf{r}}=\textbf{r}$

12: $\mathcal{C}_{l}=||\textbf{x}-\hat{\textbf{r}}||^{2}$

13: update $\theta_{l}$ by optimizing $\mathcal{C}_{l}$

14: $\textbf{x}=\textbf{z}_{x}^{l}.detach$

15:end for

The neural network becomes a predictive machine, where each layer trains its
parameters by trying to predict the inputs by knowing the outputs. The
constraint of the local learning has a more pronounced effect on the non-
linearity of a neural network trained in this way. While backpropagation can
adjust all the parameters of a network so that the feature reconstructs the
input, local rules can only rely on very strict information content available.
Non reversible functions, such as the ReLU function, propagate loss of
information, which will lead to reconstructions with low fidelity. Thus, a
non-linear function must be carefully chosen in order to preserve as much
information as possible. The function inverse hyperbolic sine
($arsinh(x)=ln(x+\sqrt{1+x^{2}}))$), is similar to the hyperbolic tangent near
zero and logarithmic at large (absolute) values. This function has the
properties of being fully invertible, zero-centered mean, unbounded,
continuously differentiable and its gradient does not vanish as fast as
$tanh$. These properties make $arsinh$ a good candidate function for local
training.

Figure 4: The local training rule.

At inference time, we need to propagate the information backwards from the
output to the input layer by layer just as one would do normally with non-
local feature extraction. However, the non-linear function will play a major
role here, it being reversible will be required to approximate reversibility.
This is done by, after inputting the latent vector, applying the inverse of
the non-linear function after each layer that uses the function. This process
is summarized in the Algorithm 3.

Algorithm 3 Reconstruction with local feature alignment

1:sample $\mathbf{z}_{x}^{L}$

2:for l=L, 0 do

3: initialize $\mathbf{r}=\mathbf{x}.shape$ with $\mathbf{x}$ as
$\mathbf{z}_{x}^{l}=E(\mathbf{x};\theta_{l})$

4: $t=0$

5: while $t<T$ do

6: $\mathbf{z}_{r}^{l}=E(\mathbf{r};\theta_{l})$

7: $\mathcal{L}_{l}=||\mathbf{z}_{x}^{l}-\mathbf{z}_{r}^{l}||_{2}^{2}$

8: $r=r-\tau\frac{\partial\mathcal{L}_{l}}{\partial r}$

9: $t=t+1$

10: end while

11: $\hat{\mathbf{r}}=\mathbf{r}$

12: if layer = non-linear function $f$ then

13: $\hat{\mathbf{r}}=f^{-1}(\hat{\mathbf{r}})$

14: end if

15: $\mathbf{z}_{x}^{l}=\hat{\mathbf{r}}$

16:end for

##  4 Implementation details

The encoder network consists of a series of convolutional layers, similar to
the AlexNet architecture (Krizhevsky et al., 2012), but with stride one and
two for down-scaling, instead of maxpool, with LeakyReLU activation. The
generator network has three convolutional layers. The discriminator network
has the same architecture as the encoder, except for the last layer that
outputs a single value for each example. Only the generator uses batch
normalization after each convolution. All convolutions have kernel size $k=3$.

We use the Adam optimizer (Kingma and Ba, 2017) with learning rate
$\eta=0.00001$ and batch size $128$. The parameters of the encoder and
generator networks are initialized with orthogonal initialization (Saxe et
al., 2014; Hu et al., 2020). We set the hyperparameter $\lambda=0.01$ and
sample $\beta$ from a uniform distribution, with a different random value for
each training example.

From Appendix 2, the loss become unstable when the weights $w^{2}>2$, so we
restrict the weights to the range $-\sqrt{2}\leq w\leq\sqrt{2}$ by clamping
then, i.e.:

$w=\begin{cases}-\sqrt{2}&\text{if }w<-\sqrt{2},\\\ \sqrt{2}&\text{if
}w>\sqrt{2},\\\ w&\text{otherwise}.\end{cases}$ (7)

For the image data sets, the generator is composed by a few convolutional
layers, with same number of filters and kernel sizes.

We also report the results for the modified feature alignment for local
training as a proof of concept by training a encoder for reconstruction from a
latent vector.

For GAN, when used for reconstruction, we search the latent space that leads
to most similar images by optimizing
$\operatorname*{arg\,min}_{z}||x-G(z)||_{2}^{2}$.

##  5 Results

We compare the results against traditional variational autoencoders and
generative adversarial networks. The results show reconstruction of the inputs
with feature and generator. We also show random samples from the generator
network. We evaluate the Fréchet Inception Distance (FID) (Heusel et al.,
2018; Seitzer, 2020) for each data set as a measure of the quality of the
samples. The FID score is calculated by extracting the activation of the
global spatial pooling layer of a pre-trained Inception V3 model (Szegedy et
al., 2016), for equally numbered images from the data set (here we choose 10k
images) and sampled from a generator model:

$FID=||\mu_{1}-\mu_{2}||_{2}^{2}+tr(\Sigma_{1}+\Sigma_{2}-2\sqrt{\Sigma_{1}\Sigma_{2}}),$
(8)

with $\mu$ the mean of activations, $\Sigma$ the covariance matrix and $tr$
the trace function. The results are shown in Table 1.

Method |  MNIST |  CelebA |  CIFAR-10
---|---|---|---
VAE |  29.99 |  82.93 |  164.62
GAN |  18.72 |  31.40 |  63.36
$\hat{\textbf{r}}$ |  129.82 |  122.47 |  205.65
$G(\hat{\textbf{r}}$) |  38.53 |  128.35 |  69.89
Table 1: FID scores across three image data sets.

###  5.1 MNIST

The MNIST data set (Bengio and Haffner, 1998) is a set of 60000 grayscale
images with size $28\times 28$ pixels that contains hand drawings of digits
from zero to nine. Figure 5 shows the reconstruction of images by the features
and with the generator applied to them, compared with traditional AE, VAE and
GAN. While the features tend to be noisy, the generator can sharpen the images
to resemble better the original inputs. Figure 6 shows random samples from the
generator with a class layer coupled to the encoder. The same image also shows
the interpolation of the latent vector between four pairs of images to
demonstrate the continuity of transitions on the latent space.

Figure 5: Reconstruction of images of the MNIST data set from four different
models.  Figure 6: Left: random samples. Right: interpolation among four
images reconstructed from the data set.

###  5.2 CelebA

The CelebA data set (Liu et al., 2015) is a set of 202600 images of celebrity
faces. We resize the images to $64\times 64$ pixels. Figures 7 and 8 show the
reconstruction and sampling with interpolation between samples, respectively.
Without the perceptual loss (with $\lambda=1$), we noticed a failure on the
convergence of the generator network.

Figure 7: Reconstruction of images of the CelebA data set from four different
models.  Figure 8: Two sets of interpolation among four random sampled images.

###  5.3 CIFAR-10

The CIFAR-10 data set (Krizhevsky and Hinton, 2009) contains 70000 natural
images with size $32\times 32$ pixels across 10 different classes. Figure 9
shows the reconstruction of images from the data set. While the features do
approximate the original inputs, the transformation of the generator tends to
be more dissimilar due its loss being dependent only on the adversarial
contribution ($\lambda=0)$. Just as before, Figure 10 shows random samples and
interpolation, which show a diversity of images, albeit less perceptual
similar to the original data set.

Figure 9: Reconstruction of images on CIFAR-10. Figure 10: Left: random
samples. Right: interpolation among four images reconstructed from the data
set.

It is important to note that we do not expect feature alignment to exceed the
reconstruction and sample qualities of VAEs and GANs. The condition of
reversibility constrains the optimization of a neural network and thus has to
balance the reversibility cost with other losses. Nevertheless we compare the
reconstruction $L_{2}$ loss with other networks to analyse how different each
network is compared to same metric. These results are shown in FIG. 11.

As expected, autoenconders have the lowest loss, given that it is optimized
directly for reconstruction. The features should be compared to VAEs and the
generator with GAN, since they are optimized with similar losses. Here we can
see that absence of perceptual loss on the generator network decreases the
reconstruction (without optimizing the latent vector) ability.

Figure 11: Comparison of the reconstruction $L_{2}$ loss for AE, VAE, GAN,
$r$, and $G(r)$.

###  5.4 Local feature alignment

Here we show results from an encoder trained for reconstruction with the local
feature alignment training. Figures 12, 13, and 14 show reconstruction pairs
for the MNIST, CIFAR-10, and CelebA data sets respectively. We can see that
local training can reconstruct images despite the layers not receiving
information from the reconstruction loss of images, which can be explained
with the same reason of non-local feature alignment: the weights form a
orthogonal matrix which tries to reverse information between layers as much as
possible, which is limited only by the network capacity, directly related to
the number of neurons.

Figure 12: Local feature alignment on the MNIST data set. Each pair of images
contains the reconstruction and original, respectively. Figure 13: Local
feature alignment on the CIFAR-10 data set. Each pair of images contains the
reconstruction and original, respectively. Figure 14: Local feature alignment
on the CelebA data set. Each pair of images contains the reconstruction and
original, respectively.

##  6 Conclusions

We presented feature alignment, a technique that can approximately reverse
neural networks. We trained an encoder to predict its own input by optimizing
the features to match the inputs. By coupling a probabilistic layer with the
same formulation as the variational autoencoders, we have the ability to
generate new samples with the same statistical distribution of the training
data. To improve the quality of the generated samples, that suffer from noise
effects, we combined the generative adversarial network method by coupling a
generator and a discriminator network to the features. We also showed that the
technique can be modified to a local training rule, as opposed to
backpropagation, which offers the advantage of lowering memory resources for
training and extracting gradients of neural networks.

Mathematical analysis on the convergence of the proposed technique shows that
the weights converge to a pseudo-inverse matrix, which justifies the
convergence of a network trained in this way to map its outputs back to its
own inputs. The restriction is the network’s architecture itself, since the
bottlenecks do not allow for a one-to-one relationship.

The results show that the features can approximately match the inputs. Despite
not being able to improve on the sampling quality of other current generative
techniques, reversibility can offer advantages when a mapping of the outputs
back to its inputs is desired. The data set CIFAR-10 is notoriously difficult
because of the small image size and high variance, which led to samples with
high FID measure. By coupling a classification layer on the encoder network,
we can use label information for conditional generation of samples.
Additionally, the results of local training suggests that we can train neural
networks without feedback of global loss functions.

The proposed technique can be seen as a compromise between VAEs and GANs. The
full architecture has crisper samples than VAEs and a latent vector that can
be exploited. Feature extraction can be implemented across many different
neural network architectures. Thus, it can potentially be improved to further
increase the quality of the results.

##  A Mathematical analysis

This section studies the numerical analysis of the technique.

###  A.1 Convergence of the features

For two fully connected layers, we have
$a_{j}^{(x)}=\hat{a}_{j}=\sum_{i}w_{ij}x_{i}$ and
$a_{j}^{(r)}=\sum_{i}w_{ij}r_{i}$. The feature is obtained by optimizing the
squared $L_{2}$ loss between the two activations:

$\displaystyle\mathcal{L}=\frac{1}{2}(\hat{a}_{j}-a_{j}^{(r)})^{2},$ (9)

which follows that $r$ will evolve with the gradient flux:

$\displaystyle\frac{\partial r_{i}}{\partial
t}=-\frac{\partial\mathcal{L}}{\partial r_{i}}\ \therefore
r_{i}^{t}=r_{i}^{t-1}-\frac{\partial C}{\partial r^{t-1}}.$ (10)

Then we can evaluate $r^{t}$ at each time step $t$:

$\displaystyle
r_{i}^{1}=r_{i}^{0}-w_{ij}(\hat{a}_{j}-w_{ij}r^{0})=r_{i}^{0}(1-w_{ij}^{2})+w_{ij}\hat{a}_{j},$
(11) $\displaystyle
r_{i}^{2}=r_{i}^{1}(1-w_{ij}^{2})+w_{ij}\hat{a}_{j}=r_{i}^{0}(1-w_{ij}^{2})^{2}+[1+(1-w_{ij}^{2})]w_{ij}\hat{a}_{j},$
(12) $\displaystyle
r_{i}^{3}=r_{i}^{0}(1-w_{ij}^{2})^{3}+[1+(1-w_{ij}^{2})+(1-w_{ij}^{2})^{2}]w_{ij}\hat{a}_{j},$
(13) $\displaystyle
r_{i}^{4}=r_{i}^{0}(1-w_{ij}^{2})^{4}+[1+(1-w_{ij}^{2})+(1-w_{ij}^{2})^{2}+(1-w_{ij}^{2})^{3}]w_{ij}\hat{a}_{j}.$
(14)

With the pattern above, we generalize $r^{t}$ for any time step:

$\displaystyle
r_{i}^{p}=r_{i}^{0}(1-w_{ij}^{2})^{p}+\sum_{q=0}^{p-1}(1-w^{2})^{q}w_{ij}\hat{a}_{j}.$
(15)

Under the restriction of $w_{ij}^{2}\leq 2$, we have at limit as $t$ grows:

$\displaystyle\lim_{p\rightarrow\infty}r_{i}^{0}(1-w_{ij}^{2})^{p}=0\
\therefore\lim_{p\rightarrow\infty}\sum_{q=0}^{p-1}(1-w_{ij}^{2})^{q}w_{ij}\hat{a}_{j}=\frac{\hat{a}_{j}}{w_{ij}}.$
(16)

So the loss $\mathcal{L}\rightarrow 0$ as $p\rightarrow\infty$.

###  A.2 Convergence of the weights

The $L_{2}$ loss that update the parameters is:

$\displaystyle\mathcal{L}=\frac{1}{2}(x_{i}-r_{i}^{t})^{2}=\frac{1}{2}\left[x_{i}-r_{i}^{0}(1-w_{ij}^{2})^{t}-\sum_{q=0}^{t-1}(1-w_{ij}^{2})^{q}w_{ij}\hat{a}_{j}\right]^{2}.$
(17)

For one-shot, $T=1$, training, we have:

$\displaystyle\mathcal{L}=\frac{1}{2}\left[x_{i}-r_{i}^{0}(1-w_{ij}^{2})-w_{ij}\hat{a}_{j}\right]^{2}=\frac{1}{2}\left[x_{i}-r_{i}^{0}(1-w_{ij}^{2})-w_{ij}w_{ij}x_{i}\right]^{2}.$
(18)

We can rewrite the equation above in vector notation as follows:

$\displaystyle\mathcal{L}=\frac{1}{2}\left[\textbf{x}-\textbf{r}^{0}(\textbf{I}-\textbf{w}^{T}\textbf{w})-\textbf{w}^{T}\textbf{w}\textbf{x}\right]^{2}.$
(19)

For any $\textbf{r}^{0}$, the equation above has roots for $w_{ii}^{2}=0$. We
can see then that the loss is minimal when the weight matrix product is
orthogonal, i.e. $\textbf{w}^{T}\textbf{w}=\textbf{I}$, which has the
consequence that the transposed weight matrix is also its generalized
Moore–Penrose inverse or pseudo-inverse $\textbf{w}^{T}=\textbf{w}^{-1}$.

## References

  * Agarap (2019) Abien Fred Agarap.  Deep Learning using Rectified Linear Units (ReLU).  _arXiv:1803.08375_ , February 2019.
  * Aidan N. Gomez and Grosse (2017) Raquel Urtasun Aidan N. Gomez, Mengye Ren and Roger B. Grosse.  The reversible residual network: Backpropagation without storing activations.  _arXiv:1707.04585_ , 2017.
  * Ardizzone et al. (2019) Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W. Pellegrini, Ralf S. Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe.  Analyzing Inverse Problems with Invertible Neural Networks.  _arXiv:1808.04730_ , February 2019.
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein GAN.  _arXiv:1701.07875_ , December 2017.
  * Atapattu and Rekabdar (2019) Charith Atapattu and Banafsheh Rekabdar.  Improving the realism of synthetic images through a combination of adversarial and perceptual losses.  In _2019 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–7, July 2019.  doi: 10.1109/IJCNN.2019.8852449.  ISSN: 2161-4407.
  * Baldi and Sadowski (2016) Pierre Baldi and Peter Sadowski.  A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation.  _Neural Networks_ , 83:51–74, November 2016.  ISSN 08936080.  doi: 10.1016/j.neunet.2016.07.006.  arXiv: 1506.06472.
  * Behrmann et al. (2019) Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jörn-Henrik Jacobsen.  Invertible Residual Networks.  _arXiv:1811.00995_ , May 2019.
  * Bengio and Haffner (1998) Yann LeCun Leon Bottou Yoshua Bengio and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, November 1998.  ISSN 1558-2256.  doi: 10.1109/5.726791.
  * Bengio (2014) Yoshua Bengio.  How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation.  _arXiv:1407.7906_ , September 2014.
  * Burgess et al. (2018) Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner.  Understanding disentangling in beta-vae.  _arXiv:1804.03599_ , April 2018.
  * Doersch (2021) Carl Doersch.  Tutorial on Variational Autoencoders.  _arXiv:1606.05908_ , January 2021.
  * Donahue et al. (2017) Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell.  Adversarial Feature Learning.  _arXiv:1605.09782_ , April 2017.
  * Dosovitskiy and Brox (2016) Alexey Dosovitskiy and Thomas Brox.  Generating images with perceptual similarity metrics based on deep networks.  _arXiv:1602.02644_ , feb 2016.
  * Fan et al. (2021) Fenglei Fan, Jinjun Xiong, Mengzhou Li, and Ge Wang.  On Interpretability of Artificial Neural Networks: A Survey.  _arXiv:2001.02522_ , January 2021.
  * Farias and Maziero (2018) Tiago de Souza Farias and Jonas Maziero.  Gradient target propagation.  _arXiv:1810.09284_ , November 2018.
  * Gao and Zhong (2020) Fengli Gao and Huicai Zhong.  Study on the Large Batch Size Training of Neural Networks Based on the Second Order Gradient.  _arXiv:2012.08795_ , December 2020.
  * Gilpin et al. (2019) Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.  Explaining Explanations: An Overview of Interpretability of Machine Learning.  _arXiv:1806.00069_ , February 2019.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Networks.  _arXiv:1406.2661_ , June 2014.
  * Grathwohl et al. (2018) Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud.  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models.  _arXiv:1810.01367_ , October 2018.
  * Gui et al. (2020) Jie Gui, Zhenan Sun, Yonggang Wen, Dacheng Tao, and Jieping Ye.  A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications.  _arXiv:2001.06937_ , January 2020.
  * Havrylov and Titov (2020) Serhii Havrylov and Ivan Titov.  Preventing Posterior Collapse with Levenshtein Variational Autoencoder.  _arXiv:2004.14758_ , April 2020.
  * Hebb (1949) Donald Olding Hebb.  The organization of behavior; a neuropsychological theory.  _New York: Wiley_ , 1949.  doi: 10.1002/cne.900930310.
  * Heusel et al. (2018) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  _arXiv:1706.08500_ , January 2018.
  * Higgins et al. (2016) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.  November 2016.
  * Hu et al. (2020) Wei Hu, Lechao Xiao, and Jeffrey Pennington.  Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks.  _arXiv:2001.05992_ , January 2020.
  * Isomura and Toyoizumi (2016) Takuya Isomura and Taro Toyoizumi.  A Local Learning Rule for Independent Component Analysis.  _Scientific Reports_ , 6(1):28073, June 2016.  ISSN 2045-2322.  doi: 10.1038/srep28073.
  * Isomura and Toyoizumi (2018) Takuya Isomura and Taro Toyoizumi.  Error-Gated Hebbian Rule: A Local Learning Rule for Principal and Independent Component Analysis.  _Scientific Reports_ , 8(1):1835, January 2018\.  ISSN 2045-2322.  doi: 10.1038/s41598-018-20082-0.
  * Jing and Xu (2019) Kun Jing and Jungang Xu.  A Survey on Neural Network Language Models.  _arXiv:1906.03591_ , June 2019.
  * Johnson et al. (2016) Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution.  _arXiv:1603.08155_ , March 2016.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba.  Adam: A Method for Stochastic Optimization.  _arXiv:1412.6980_ , January 2017.
  * Kingma and Dhariwal (2018) Diederik P. Kingma and Prafulla Dhariwal.  Glow: Generative Flow with Invertible 1x1 Convolutions.  _arXiv:1807.03039_ , July 2018.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling.  Auto-Encoding Variational Bayes.  _arXiv:1312.6114_ , May 2014.
  * Kingma and Welling (2019) Diederik P. Kingma and Max Welling.  An Introduction to Variational Autoencoders.  _arXiv:1906.02691_ , December 2019.  doi: 10.1561/2200000056.
  * Kobyzev et al. (2020) Ivan Kobyzev, Simon J. D. Prince, and Marcus A. Brubaker.  Normalizing Flows: An Introduction and Review of Current Methods.  _arXiv:1908.09257_ , June 2020.  doi: 10.1109/TPAMI.2020.2992934.
  * Krizhevsky and Hinton (2009) Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  _Master’s thesis, Department of Computer Science, University of Toronto_ , 2009.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet Classification with Deep Convolutional Neural Networks.  _Advances in Neural Information Processing Systems_ , 25, 2012.
  * Krotov and Hopfield (2019) Dmitry Krotov and John J. Hopfield.  Unsupervised learning by competing hidden units.  _Proceedings of the National Academy of Sciences_ , 116(16):7723–7731, April 2019.  ISSN 0027-8424, 1091-6490.  doi: 10.1073/pnas.1820458116.
  * Larsen et al. (2016) Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther.  Autoencoding beyond pixels using a learned similarity metric.  _arXiv:1512.09300_ , February 2016.
  * Lee et al. (2011) Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng.  Unsupervised learning of hierarchical representations with convolutional deep belief networks.  _Communications of the ACM_ , 54(10):95–103, October 2011.  ISSN 0001-0782.  doi: 10.1145/2001269.2001295.
  * Leemon Baird and Ingkiriwang (2005) David Smalenberger Leemon Baird and Shawn Ingkiriwang.  One-step neural network inversion with PDF learning and emulation.  In _Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005._ , volume 2, pages 966–971 vol. 2, July 2005.  doi: 10.1109/IJCNN.2005.1555983.  ISSN: 2161-4407.
  * Lillicrap et al. (2020) Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton.  Backpropagation and the brain.  _Nature Reviews Neuroscience_ , 21(6):335–346, June 2020.  ISSN 1471-0048.  doi: 10.1038/s41583-020-0277-3.
  * Lindsey and Litwin-Kumar (2020) Jack Lindsey and Ashok Litwin-Kumar.  Learning to Learn with Feedback and Local Plasticity.  _arXiv:2006.09549_ , June 2020.
  * Linnainmaa (1976) Seppo Linnainmaa.  Taylor expansion of the accumulated rounding error.  _BIT Numerical Mathematics_ , 16(2):146–160, June 1976.  ISSN 1572-9125.  doi: 10.1007/BF01931367.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * Lucas et al. (2019) James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi.  Don’t Blame the ELBO! A Linear VAE Perspective on Posterior Collapse.  _arXiv:1911.02469_ , November 2019.
  * Mahendran and Vedaldi (2016) Aravindh Mahendran and Andrea Vedaldi.  Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images.  _International Journal of Computer Vision_ , 120(3):233–255, December 2016.  ISSN 0920-5691, 1573-1405.  doi: 10.1007/s11263-016-0911-8.  arXiv: 1512.02017.
  * Mao et al. (2017) Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley.  Least Squares Generative Adversarial Networks.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 2813–2821, October 2017.  doi: 10.1109/ICCV.2017.304.  ISSN: 2380-7504.
  * Millidge et al. (2020) Beren Millidge, Alexander Tschantz, Anil K. Seth, and Christopher L. Buckley.  Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain.  _arXiv:2009.05359_ , October 2020.
  * Nguyen et al. (2016a) Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  _arXiv:1605.09304_ , November 2016a.
  * Nguyen et al. (2016b) Anh Nguyen, Jason Yosinski, and Jeff Clune.  Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks.  _arXiv:1602.03616_ , May 2016b.
  * Olah et al. (2017) Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  _Distill_ , 2(11):e7, November 2017.  ISSN 2476-0757.  doi: 10.23915/distill.00007.
  * Ororbia et al. (2018) Alexander G. Ororbia, Ankur Mali, Daniel Kifer, and C. Lee Giles.  Conducting Credit Assignment by Aligning Local Representations.  _arXiv:1803.01834_ , July 2018.
  * Papamakarios (2019) George Papamakarios.  Neural Density Estimation and Likelihood-free Inference.  _arXiv:1910.13233_ , October 2019.
  * Rezende and Viola (2018) Danilo Jimenez Rezende and Fabio Viola.  Taming VAEs.  _arXiv:1810.00597_ , October 2018.
  * Rumelhart et al. (1986) David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams.  Learning representations by back-propagating errors.  _Nature_ , 323(6088):533–536, October 1986.  ISSN 1476-4687.  doi: 10.1038/323533a0.
  * Salehi et al. (2020) Pegah Salehi, Abdolah Chalechale, and Maryam Taghizadeh.  Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments.  _arXiv:2005.13178_ , May 2020.
  * Salvatori et al. (2021) Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, Rafal Bogacz, and Zhenghua Xu.  Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks.  _arXiv:2103.03725_ , March 2021.
  * Saxe et al. (2014) Andrew M. Saxe, James L. McClelland, and Surya Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  _arXiv:1312.6120_ , February 2014.
  * Schirrmeister et al. (2018) Robin Tibor Schirrmeister, Patryk Chrabaszcz, Frank Hutter, and Tonio Ball.  Training Generative Reversible Networks.  _arXiv:1806.01610_ , August 2018.
  * Seitzer (2020) Maximilian Seitzer.  pytorch-fid: FID Score for PyTorch.  https://github.com/mseitzer/pytorch-fid, August 2020.  Version 0.1.1.
  * Selvaraju et al. (2020) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization.  _International Journal of Computer Vision_ , 128(2):336–359, February 2020.  ISSN 0920-5691, 1573-1405.  doi: 10.1007/s11263-019-01228-7.  arXiv: 1610.02391.
  * Shahroudnejad (2021) Atefeh Shahroudnejad.  A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networks.  _arXiv:2102.01792_ , February 2021.
  * Sikka et al. (2019) Harshvardhan Sikka, Weishun Zhong, Jun Yin, and Cengiz Pehlevan.  A Closer Look at Disentangling in $\beta$-VAE.  _arXiv:1912.05127_ , December 2019.
  * Sohoni et al. (2019) Nimit Sharad Sohoni, Christopher Richard Aberger, Megan Leszczynski, Jian Zhang, and Christopher Ré.  Low-Memory Neural Network Training: A Technical Report.  _arXiv:1904.10631_ , April 2019.
  * Song et al. (2020) Yuhang Song, Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz.  Can the Brain Do Backpropagation? — Exact Implementation of Backpropagation in Predictive Coding Networks.  _Advances in Neural Information Processing Systems_ , 33:22566–22579, 2020.
  * Springenberg et al. (2015) Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for Simplicity: The All Convolutional Net.  _arXiv:1412.6806_ , April 2015.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the Inception Architecture for Computer Vision.  pages 2818–2826, 2016.
  * Takida et al. (2021) Yuhta Takida, Wei-Hsiang Liao, Toshimitsu Uesaka, Shusuke Takahashi, and Yuki Mitsufuji.  Preventing Posterior Collapse Induced by Oversmoothing in Gaussian VAE.  _arXiv:2102.08663_ , February 2021.
  * Velichko (2020) Andrei Velichko.  Neural Network for Low-Memory IoT Devices and MNIST Image Recognition Using Kernels Based on Logistic Map.  _Electronics_ , 9(9):1432, September 2020.  ISSN 2079-9292.  doi: 10.3390/electronics9091432.  arXiv: 2006.02824.
  * Whittington and Bogacz (2019) James C.R. Whittington and Rafal Bogacz.  Theories of Error Back-Propagation in the Brain.  _Trends in Cognitive Sciences_ , 23(3):235–250, March 2019.  ISSN 1364-6613.  doi: 10.1016/j.tics.2018.12.005.
  * You et al. (2017) Yang You, Igor Gitman, and Boris Ginsburg.  Large Batch Training of Convolutional Networks.  _arXiv:1708.03888_ , September 2017.
  * Yujun Shen and Zhou (2020) Xiaoou Tang Yujun Shen, Jinjin Gu and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA_ , page 9240–9249, 2020.  doi: 10.1109/CVPR42600.2020.00926.
  * Zintgraf et al. (2017) Luisa M. Zintgraf, Taco S. Cohen, and Max Welling.  A New Method to Visualize Deep Neural Networks.  _arXiv:1603.02518_ , June 2017.