# Experimental Realization of Entangled Coherent States in Two-dimensional
Harmonic Oscillators of a Trapped Ion

Honggi Jeon  Department of Physics and Astronomy, Seoul National University,
Seoul 08826, Republic of Korea  Automation and System Research Institute,
Seoul National University, Seoul 08826, Republic of Korea  Jiyong Kang
Automation and System Research Institute, Seoul National University, Seoul
08826, Republic of Korea  Department of Computer Science and Engineering,
Seoul National University, Seoul 08826, Republic of Korea  Jaeun Kim
Automation and System Research Institute, Seoul National University, Seoul
08826, Republic of Korea  Department of Computer Science and Engineering,
Seoul National University, Seoul 08826, Republic of Korea  Wonhyeong Choi
Automation and System Research Institute, Seoul National University, Seoul
08826, Republic of Korea  Department of Computer Science and Engineering,
Seoul National University, Seoul 08826, Republic of Korea  Inter-university
Semiconductor Research Center, Seoul National University, Seoul 08826,
Republic of Korea  Kyunghye Kim  Automation and System Research Institute,
Seoul National University, Seoul 08826, Republic of Korea  Department of
Computer Science and Engineering, Seoul National University, Seoul 08826,
Republic of Korea  Taehyun Kim<EMAIL_ADDRESS>Automation and System
Research Institute, Seoul National University, Seoul 08826, Republic of Korea
Department of Computer Science and Engineering, Seoul National University,
Seoul 08826, Republic of Korea  Inter-university Semiconductor Research
Center, Seoul National University, Seoul 08826, Republic of Korea  Institute
of Computer Technology, Seoul National University, Seoul 08826, Republic of
Korea  Institute of Applied Physics, Seoul National University, Seoul 08826,
Republic of Korea

###### Abstract

Entangled coherent states play pivotal roles in various fields such as quantum
computation, quantum communication, and quantum sensing. We experimentally
demonstrate the generation of entangled coherent states with the two-
dimensional motion of a trapped ion system. Using Raman transitions with
appropriate detunings, we simultaneously drive the red and blue sidebands of
the two transverse axes of a single trapped ion and observe multi-periodic
entanglement and disentanglement of its spin and two-dimensional motion. Then,
by measuring the spin state, we herald entangled coherent states of the
transverse motions of the trapped ion and observe the corresponding modulation
in the parity of the phonon distribution of one of the harmonic oscillators.
Lastly, we trap two ions in a linear chain and realize Mølmer–Sørensen gate
using two-dimensional motion.

##  I INTRODUCTION

For the last few decades, the coherent state has been the subject of intense
theoretical and experimental investigation [1]. It is considered to be a
quantum state with the most classical properties because its spread is the
minimal allowed by the uncertainty principle and its trajectory of time
evolution is identical to that of the classical harmonic oscillator [2]. Its
multipartite extension, the entangled coherent state, has been a useful
theoretical tool in various fields of quantum optics as it is the entangled
superposition of the most ”classical” quantum states. It has been used in
theoretical studies concerning quantum information processing [3, 4, 5, 6, 7,
8, 9], quantum metrology [10], and fundamental tests of physics such as Bell’s
inequality and Leggett’s inequality [11, 12]. Despite their sensitivity to
decoherence, entangled coherent states have been experimentally realized in a
few experiments involving photons [13] and superconducting circuits [14, 15].
The trapped ion has been an extremely valuable tool for studying the quantum
world because it is highly isolated from the environment yet can be precisely
controlled. The single-mode superposition of coherent states or cat states
have been realized in trapped ion systems in various experiments using the
motional state of the trapped ion [16, 17, 18, 19, 20]. There have been
several theoretical works on the implementation of entangled coherent states
in trapped ion systems [21, 22, 23, 24], but none have been experimentally
implemented so far.
In this work, we report on the realization of entangled coherent states with
the two dimensional motion of a trapped ion. We implement the simultaneous
spin-dependent force (SDF) on the ion in the two principal axes (X and Y) by
making the transverse trap potential nearly isotropic so that the secular
frequencies of the X and Y modes are very close. By choosing a laser detuning
between the X and Y mode frequencies and driving a bichromatic transition with
the blue and red sidebands, we excite the motional modes in the two radial
directions concurrently with varying ratios of coupling strengths to each
mode.

For a single ion, we generate Lissajous-curve-like motion in two dimensions
with various commensurate oscillation periods in each direction and observe
corresponding periodic variation in the spin state [25]. With mid-circuit
measurement, we decouple the spin from the motion and herald the entangled
coherent state of motion in two transverse axes. This is verified by observing
the modulation of phonon number parity, which results from the periodic
entanglement and disentanglement of the two motional modes. Also, in an ion
chain consisting of two ions, we demonstrate the successful generation of a
Bell state using Mølmer–Sørensen interaction where the geometric phase is
accumulated via motion in two spatial dimensions, which reduces the required
Rabi frequency compared to the one-dimension case.

##  II RESULTS

Figure 1: Experimental setup. (a) Schematic diagram of trap electrodes, ion,
and pulsed laser beams for Raman transition. $\Delta\vec{k}$ indicates the
direction of momentum transfer, which is the difference of the two pulsed
laser beams, $R_{V}$ and $R_{H}$. Upper right is the cross-section of the trap
in the transverse plane. The angles between $\Delta\vec{k}$ and the Y and X
principal axes are $24^{\circ}$ and $66^{\circ}$, respectively. The ion chain
is formed along the Z axis. (b) Representative spectra showing the blue
sidebands for the transverse modes of a single ion and (c) of a linear chain
of two ions where $X_{cm}$ and $Y_{cm}$ are in-phase modes and $X_{tilt}$ and
$Y_{tilt}$ are out-of-phase modes.

We trap ${}^{171}\mathrm{Yb^{+}}$ ions in the center of a blade-type trap with
dimensions specified in 1(a) [26, 27]. A radio-frequency (RF) voltage
oscillating at 15.3 MHz is applied to the RF blades for radial confinement,
and the other two blades are grounded. A DC offset voltage can be applied to
the RF blades to rotate the radial principal axes and change the separation
between the X and Y mode secular frequencies. The amplitude of the RF field is
actively stabilized by a PID controller [28]. As can be seen in 1(b) and (c),
the mean secular frequency for the radial modes is typically set to around
$\omega_{X,Y}\simeq 2\pi\times 1250$ kHz. The exact value changes by a few kHz
every day due to the thermal drift of the RF voltage sampling [29]. The
degeneracy of the transverse axes is lifted by $2\pi\times 27.8$ kHz, with the
Y axis frequency higher. A DC voltage of 1300 V is applied to the endcap
electrodes, and the resulting axial secular frequency is
$\omega_{Z}=2\pi\times 120\pm 0.12$ kHz. The qubit states are defined as
$\ket{\downarrow}=\ket{S_{1/2},F=0,m_{F}=0}$ and
$\ket{\uparrow}=\ket{S_{1/2},F=1,m_{F}=0}$. For experiments involving a single
ion, the qubit state is measured by the standard fluorescence detection method
[30]. For a two-ion chain, we use histogram fitting to infer the population of
the three possible classes of qubit states, {$\ket{\downdownarrows}$},
{$\ket{\downuparrows}$⟩, $\ket{\updownarrows}$} and {$\ket{\upuparrows}$}
[31].
The red sideband, blue sideband, and carrier transitions are implemented by
applying appropriate detunings to the stimulated Raman transition [32]. It is
realized by two perpendicular 355-nm pulsed laser beams which enter the trap
from the bottom $(R_{V})$ and the side $(R_{H})$. Their relative frequencies
and phases are controlled by acousto-optic modulators (AOMs). The Raman
transition momentum vector $\Delta\vec{k}$ is perpendicular to the Z axis and
has components in both radial axes with angles $\theta_{Y}=24^{\degree}$ and
$\theta_{X}=66^{\degree}$ as shown in 1(a). This results in asymmetric Lamb-
Dicke factors, $\eta_{X}=0.05$ and $\eta_{Y}=0.11$, for the modes. The beating
of the pulsed laser is stabilized by a feed-forward system which shifts the
driving RF frequency of the AOM that controls the $(R_{V})$ beam [33].

###  II.1 Entanglement of spin with multiple motional modes

We will use the notation $\ket{s}\ket{a}\ket{b}$ to specify the quantum state
of the system, where $s$ denotes the qubit state of the ion chain with
possible values of $\uparrow$ and $\downarrow$ for a single ion and their
tensor product for a chain of two ions. $a$ and $b$ indicate the quantum
states of the X and Y modes either in Fock state or coherent state basis. We
realize the SDF Hamiltonian by driving the red and blue sidebands of the
motional modes simultaneously with the same strength. When there is a
symmetric detuning from the sidebands, the position of the wave packet in
phase space modulated by a frequency proportional to the detuning [17],
resulting in a circular trajectory as shown in 2(e).

For a single trapped ion in a two-dimensional harmonic potential subject to a
symmetrically detuned bichromatic beam, we have the following interaction
Hamiltonian

$\begin{split}\hat{H}=\frac{\hbar\mathrm{\Omega}\eta_{X}}{2}\left({\hat{a}}_{X}e^{-i\left(\delta_{X}t+\phi_{M}\right)}+{\hat{a}}_{X}^{\dagger}e^{i\left(\delta_{X}t+\phi_{M}\right)}\right){\hat{\sigma}}_{\phi_{S}}+\\\
\frac{\hbar\mathrm{\Omega}\eta_{Y}}{2}\left({\hat{a}}_{Y}e^{-i(\delta_{Y}t+\phi_{M})}+{\hat{a}}_{Y}^{\dagger}e^{i(\delta_{Y}t+\phi_{M})}\right){\hat{\sigma}}_{\phi_{S}}\end{split}$
(1)

where $\eta_{j}$ and $\delta_{j}$ are the Lamb-Dicke factor for the $j$-th
axis and the detuning from the center-of-mass mode of the $j$-th axis,
respectively. $\hat{a_{j}}(\hat{a_{j}}^{\dagger})$ is the phonon annihilation
(creation) operator for the $j$-th axis and $\Omega$ is the Rabi frequency of
the Raman transition. The motion and spin phase of spin-dependent interaction
is proportional to the difference, $\phi_{M}=(\phi_{b}-\phi_{r})/2$, and sum,
$\phi_{S}=(\phi_{b}+\phi_{r})/2$, of the laser phases for the blue and red
sidebands, $\phi_{b}$ and $\phi_{r}$.

Figure 2: Entanglement of spin and the two motional modes. In (a-d) time
evolution of the spin state for various detuning ratios is observed. Partial
disentanglement of spin and motion takes place when wave packets return to the
origin in only one dimension. Complete disentanglement is observed when wave
packets return to the origin in both dimensions. Error bars indicate quantum
projection noise. Solid curves are fits to 3. $R$ is the ratio of detunings to
the radial modes, defined as $R=\delta_{X}/\delta_{Y}$. Values of $R$
estimated from fitting are (a) $-0.261\pm 0.001$, (b) $-0.653\pm 0.003$, (c)
$-1.547\pm 0.016$, and (d) $-3.892\pm 0.071$. Times at which each motional
mode is disentangled from the spin are indicated by vertical lines. Solid red
lines correspond to the Y mode and dashed blue lines to the X mode. (e) A
representative phase space diagram for the motional modes. In each phase
space, the wave function evolves into a coherent superposition of two wave
packets, corresponding to the $\phi_{S}$ basis spin eigenstates. The
trajectories are determined by Rabi frequency and detuning from each mode.

We will use $\ket{+}=1/\sqrt{2}(\ket{\downarrow}+e^{i\phi_{S}}\ket{\uparrow})$
and $\ket{-}=1/\sqrt{2}(\ket{\downarrow}-e^{-i\phi_{S}}\ket{\uparrow})$ to
indicate the eigenstates of the ${\hat{\sigma}}_{\phi_{S}}$ operator. We
assume $\phi_{S}=0$ for simplicity. The X axis terms and Y axis terms act on
their respective Hilbert spaces, yielding the following time evolution
operator with displacements in X and Y phase spaces defined as
$\alpha\left(t\right)=\eta_{X}\Omega/2\delta_{X}\left(1-e^{-i\delta_{X}t}\right)e^{-i\phi_{M}}$
and
$\beta\left(t\right)=\eta_{Y}\Omega/2\delta_{Y}\left(1-e^{-i\delta_{Y}t}\right)e^{-i\phi_{M}}$.
The time evolution operator is
$\hat{U}(t)=\ket{+}\bra{+}\hat{D}_{X}(\alpha(t))\hat{D}_{Y}(\beta(t))+\ket{-}\bra{-}\hat{D}_{X}(-\alpha(t))\hat{D}_{Y}(-\beta(t))$
where $\hat{D}_{X}(\alpha(t))$ and $\hat{D}_{Y}(\beta(t))$ are the
displacement operators defined as
$e^{\alpha(t)\hat{a}^{\dagger}_{X}-\alpha(t)^{*}\hat{a}_{X}}$ and
$e^{\beta(t)\hat{a}^{\dagger}_{Y}-\beta(t)^{*}\hat{a}_{Y}}$. Applying this to
the initial state of the ion,
$\ket{\psi(t=0)}=\ket{\downarrow}\ket{0}\ket{0}$, after sideband cooling and
qubit initialization, we get the following wave function which exhibits spin-
motion entanglement in both motional modes:

$\ket{\psi(t)}=\frac{1}{\sqrt{2}}(\ket{+}\ket{\alpha(t)}\ket{\beta(t)}+\ket{-}\ket{-\alpha(t)}\ket{-\beta(t)})$
(2)

The time evolution of the spin state for various ratios of detunings to the X
and Y modes, $R=\delta_{X}/\delta_{Y}$, is presented in 2(a)-(d). The dashed
lines are fits to the following equation [17]

$P_{\uparrow}\left(t\right)=\frac{1}{2}\left(1-e^{-\left({\bar{n}}_{X}+\frac{1}{2}\right)\left|2\alpha\left(t\right)\right|^{2}-\left({\bar{n}}_{Y}+\frac{1}{2}\right)\left|2\beta\left(t\right)\right|^{2}}e^{-t/\tau}\right)$
(3)

where $\tau$ is an empirical decoherence rate and ${\bar{n}}_{X}$ and
${\bar{n}}_{Y}$ are mean phonon numbers of the X and Y modes, which in our
system are $\simeq 0.2$ and $\simeq 0.1$, respectively. In each phase space,
the wave packets periodically move in a circular trajectory whose period is
defined by the inverse of the detuning of the bichromatic beam. When only one
of the motional modes return to the origin in the phase space, the spin states
only partially interfere and the measured spin state deviates from its
original state, $\ket{\downarrow}$. When the wave packets return to the origin
in both phase spaces at the same time, the spin state fully returns to the
initial state [17].

###  II.2 Generation of entangled coherent state and observation of phonon
number parity modulation

The tripartite entangled state of spin and the two motions can be transformed
into an entangled coherent state (ECS) of the two motional degrees of freedom
by projecting the spin state with mid-circuit measurement. Modifying this
sequence to displace only a single motional mode will produce a single-mode
cat state of motion, as experimentally shown by Kienzler et al (see the
Supplementary Material) [20]. We start the experimental sequence by cooling
the ion to the ground state with sideband cooling pulses. Then we apply the
two-mode SDF, which acts on both the X and Y motions simultaneously, for a
duration of $t_{SDF}$. In the following step, the ion is irradiated with a
near-resonant 369.5-nm laser beam that serves as the detection beam. It is
turned on for 500 $\mu$s, and the scattered photons are collected by a
photomultiplier tube. We then drive a blue sideband Rabi oscillation on the Y
mode for varying amounts of time and measure the spin state of the ion. This
sequence is shown in 3(a). We post-select the wave function with
$\ket{\downarrow}$ spin state which is heralded by the detection of less than
two photons during the mid-circuit detection phase.

Figure 3: Generation of entangled cat state and measurement of phonon state
distribution. (a) Experimental sequence used to generate entangled coherent
state and observe the modulation of its parity. (b) A representative plot for
phonon distribution of the Y mode with $R=-2/3$ when the X mode is
disentangled and (c) entangled. Orange bars are the theoretically expected
phonon population for a cat state and blue bars are population extracted by
fitting the blue sideband Rabi oscillation, which is presented in the insets.
Solid curves in the insets are fits to the blue sideband Rabi oscillation
model. (d), (e) Evolution of parity and mean phonon numbers as a functions of
$t_{SDF}$ for $R=-2$ and $R=-2/3$, respectively. In (d), the maximum magnitude
of the displacement in the Y phase space is
$|\beta|=\sqrt{\overline{n}_{Y}}\simeq 2$ and for the X phase space,
$|\alpha|=\sqrt{\overline{n}_{X}}\simeq 0.7$. In (e), $|\beta|\simeq 1.5$ and
$|\alpha|\simeq 1.0$ at maximum. Black line is a fit to phonon number parity
of the entangled coherent state. Solid red line is the mean phonon number in
the Y mode derived from the Rabi frequency and temperatures of each mode
obtained from the phonon number parity fitting. Dahsed blue line is the mean
phonon number of the X mode calculated the same way. All the error bars in
this figure represent standard errors of fitted parameters.

This results in the following wave function:

$\ket{\psi_{ECS}(t)}=\ket{\downarrow}\frac{\ket{\alpha(t)}\ket{\beta(t)}+\ket{-\alpha(t)}\ket{-\beta(t)}}{\sqrt{2+2e^{-2(|\alpha(t)|^{2}+|\beta(t)|^{2})}}}$
(4)

The complimentary data sets which have more than or equal to two photons
detected correspond to the entangled coherent state with the opposite phase
$\ket{\alpha(t)}\ket{\beta(t)}-\ket{-\alpha(t)}\ket{-\beta(t)}$, but we choose
not to use them because photon scattering affects coherence of motional states
via recoil [20]. The blue sideband Rabi oscillation is then fitted to the
following model to retrieve phonon number distribution of the Y mode [34, 35,
20]

$P_{\uparrow}\left(t_{BSB}\right)=\sum_{n=0}^{N}{\frac{p_{Y,n}(t)}{2}\left(1-\cos\left(\Omega_{n+1,n}t_{BSB}\right)e^{-t_{BSB}/\tau}\right)}$
(5)

where $p_{Y,n}(t)$ is the population for $n$-phonon state in Y-motion after
applying the SDF for $t$, and $N$ is the maximum phonon number we consider,
$\Omega_{n+1,n}$ is the first order blue sideband Rabi frequency for
$n$-phonon state, and $\tau$ is the coherence time. When $\alpha=0$, the wave
function in 4 is reduced to a single-mode cat state of the Y mode
$(\ket{\psi_{Y}(t)}=\ket{\downarrow}(\ket{\beta(t)}+\ket{-\beta(t)})/\sqrt{2+2e^{-2|\beta(t)|^{2}}})$
and the phonon population is expected to be only in the even number states.
However, for a non-zero $\alpha$, the interference between the two coherent
states with opposite phases in the Y phase space, $\ket{\beta}_{Y}$ and
$\ket{-\beta}_{Y}$, is suppressed by the motion in the X-axis,
$\ket{\alpha}_{X}$ and $\ket{-\alpha}_{X}$. Consequently, the parity of the Y
mode population is modulated as the size of the displacement in the X mode
changes. The resulting time evolution of the phonon population distribution is
as follows.

$\begin{split}p_{Y,n}(t)=Tr(\\{\ket{\downarrow}\bra{\downarrow}\otimes\hat{I}_{X}\otimes\ket{n}\bra{n}\\}\ket{\psi_{ECS}}\bra{\psi_{ECS}})\\\
=\frac{e^{-|\beta\left(t\right)|^{2}}(|\beta\left(t\right)|^{2n}/n!)}{1+e^{-2\left(\left|\alpha\left(t\right)\right|^{2}+\left|\beta\left(t\right)\right|^{2}\right)}}(1+\left(-1\right)^{n}e^{-2\left|\alpha\left(t\right)\right|^{2}})\end{split}$
(6)

where $\bra{n}$ is the $n$-th number state of the Y mode. 6 results in a
modulation of phonon number parity defined as
$\Pi\left(t\right)=\sum_{n}{\left(-1\right)^{n}p_{Y,n}(t)}$, because the
interference between the odd number states is suppressed by entanglement with
the X mode. We include the effect of imperfect sideband cooling in the model,
which leaves some population in the $1$-phonon state (see Methods). We
demonstrate the generation of entangled coherent states at various values of
$R=\delta_{X}/\delta_{Y}$. 3(d) corresponds to $R=-2$, where the ion is
periodically displaced in the X axis at a frequency which is twice of the
frequency of the periodic displacement in the Y axis. Therefore, according to
6, the parity of the phonon distribution of the Y motion is expected to be
modulated at half the period of its periodic displacement.

We repeat the same experiment with $R=-2/3$. In this case, the parity
modulation pattern is expected to span three periods of the Y displacement as
shown in 3(e). The observed variation in phonon number parity is in good
agreement with the theoretical model, and is a direct consequence of the
entanglement of the two motional modes. 3(b) and (c) are the two
representative phonon distributions. The Y mode displacement is maximum for
both, but the phonon number parity is $0.89\pm 0.09$ for 3(b) and $0.22\pm
0.06$ for 3(c). Also, 3(c) shows a clear deviation from the single-mode cat
state phonon distribution with a significant population in the $\ket{1}_{Y}$
and $\ket{3}_{Y}$ states. In 3(d) and (e), we also plotted the time evolution
of the mean phonon numbers of the Y mode, which approximately corresponds to
the square of the absolute value of the displacement in the Y mode phase
space. The theoretical curves for the mean phonon numbers of the X and Y modes
are calculated by using the Rabi frequency and $1$-phonon state population of
each mode inferred by fitting the phonon number parity data.

Figure 4: Characterization of Mølmer–Sørensen gate with two-dimensional
motion. (a) Normalized contributions of each mode for the geometric phase
needed for the generation of the Bell state,
$1/\sqrt{2}(\ket{\downdownarrows}+\ket{\upuparrows})$. $d_{2}$ is the detuning
from the $X_{cm}$ mode. At the detuning indicated by a vertical dashed line,
the detuning ratio is $R=-1/3$ and the X and Y mode contribute almost equally.
(b) Time evolution of the spin states of a two-qubit system under two-
dimensional Mølmer–Sørensen interaction when $R=-1/3$. The optimal gate time
$t_{g}=182\ \mu s$. Error bars represent quantum projection noise. (c) Qubit
state population oscillation as a function of the phase of the $\pi/2$-pulse.
Qubit state parity(not shown in the plot),
$\Pi(\phi)=P_{\uparrow\uparrow}(\phi)+P_{\downarrow\downarrow}(\phi)-(P_{\downarrow\uparrow}(\phi)+P_{\uparrow\downarrow}(\phi))$,
oscillates with an amplitude of $\Pi_{a}=0.852\pm 0.007$. Error bars are the
standard deviation calculated from five iterations of the same experiment. The
average population of the even states at $t_{g}$,
$P_{\uparrow\uparrow}+P_{\downarrow\downarrow}$, is $0.942\pm 0.009$ as shown
in the inset. The parity oscillation and even state population yield a gate
fidelity of $0.897\pm 0.006$.

###  II.3 Mølmer–Sørensen gate with two-dimensional motion

Next, we trap two ions in a linear chain and investigate how the two-
dimensional coherent motion can be utilized in an ion chain by realizing
Mølmer–Sørensen interaction [36, 37, 38] involving modes from multiple
principal axes. We first observe the time evolution of the two-qubit states
under two-dimensional Mølmer–Sørensen interaction, with the Rabi frequency and
detuning calibrated to generate a Bell state
($1/\sqrt{2}(\ket{\upuparrows}+\ket{\downdownarrows})$). Then we measure the
fidelity of the resulting state by observing qubit state parity oscillation,
which is implemented by applying a $\pi/2$-pulse that acts on both qubits and
scanning its phase, $\phi$ [37]. The gate fidelity is estimated at $R=-1/3$,
which in this context is defined as the ratio between the detunings to the
X-cm mode and the Y-tilt mode, $\delta_{X_{cm}}/{\delta_{Y}}_{tilt}$. The
measured gate fidelity is $89.7\pm 0.6$ % which is comparable to our single
axis Mølmer–Sørensen gate fidelity, $93.2\pm 0.6$ %. This indicates that
Mølmer–Sørensen interaction can be expanded into multiple dimensions
naturally. Also, the gate Rabi frequency is reduced compared to the single
axis case, because more phase spaces contribute to the geometric phase
($\Phi_{n}(t)={\eta_{n1}\eta_{n2}/(2d_{n})^{2}\left(d_{n}t-\sin\left(d_{n}t\right)\right)\Omega_{0}^{2}}$
where $\eta_{n1}$ and $\eta_{n2}$ are the Lamb-Dicke factors of each ion for
the $n$-th mode, $d_{n}$ is the detuning to $n$-th mode and $\Omega_{0}$ is
the Rabi frequency), as can be seen in 4(a). This effect is most pronounced at
$R=-1/3$ where the geometric phase contribution is similar for both axes, thus
the required Rabi frequency is reduced by a factor of $\simeq 1/\sqrt{2}$.
Here, the Rabi frequency needed to generate an equal superposition of
$\ket{\downdownarrows}$ and $\ket{\upuparrows}$ using both axes is $2\pi\times
86.1$ kHz, which is in agreement to the experimentally calibrated value of
$2\pi\times 81.3\pm 0.6$ kHz. This is $28.3\%$ lower compared to the Rabi
frequency required using only the X axis, and $30.1\%$ lower compared to using
only the Y axis, assuming the same gate time and gate detuning.

##  III Discussion

In this work, we have demonstrated the generation of entangled coherent states
with two-dimensional motion of a trapped ion. Our scheme uses the near-
degeneracy of the transverse modes of a linear Paul trap to excite the two
motional modes of a single ion simultaneously with detuned SDF, and does not
require second order interactions or two-phonon interactions as proposed in
[21, 22], which is advantageous in terms of the strength of the interaction.
We observed a periodic modulation in the single-mode phonon number parity
which is a direct consequence of the entanglement between the two phonon
modes. The loss of parity information is analogous to the decoherence of the
spin state when only the spin state is directly measured in a spin-motion
entangled system [17]. We have also shown that Mølmer–Sørensen interaction
with multiple ions can be easily realized with a lower laser power using a
two-dimensional spin-dependent force.

The size of the phase space displacements produced in our experiment is mainly
limited by the large Rabi frequency required to generate a strong SDF in both
X and Y modes. For a SDF with non-zero detuning, the maximum displacement is
limited because the phase space trajectory forms a circle with radius
proportional to the inverse of the detuning. Thus, the maximum displacement
can be increased by making the trap more isotropic in the transverse
directions. Alternatively, one can apply a SDF resonant to both the X and Y
modes, which can be realized by a tetrachromatic laser beam [23]. In this case
the size of the displacement will increase linearly with $t_{SDF}$ and the
coupling of the laser to each mode. Another limiting factor is the difficulty
of characterizing motional states with large displacements via blue sideband
Rabi oscillation. States with larger displacement magnitude are harder to
probe because the coherence time of a cat state scales inversely with the
square of magnitude of displacement [35].

The creation of an entangled coherent state with opposite phase,
$\ket{\downarrow}(\ket{\alpha}\ket{\beta}-\ket{-\alpha}\ket{-\beta})$, is
possible with a $\pi$-pulse phase-locked to the SDF preceding the mid-circuit
measurement [20].

Using our scheme, up to $3N$ modes can be entangled for an N-ion chain when
all the principal axes of the trapping potential are utilized. Especially, the
generation of a tripartite entangled coherent state of the X, Y and Z modes,
combined with a beam splitter interaction between the phonon modes [39, 40,
41], will enable the quantum teleportation protocol in Ref. [9] with a single
trapped ion.

There have been proposals and experiments of a Ramsey-type matter-wave
rotation sensor [42, 43], Rabi-type sensor [44] and Rashba and Dresselhaus-
type spin-orbit coupling for quantum simulation of topological insulators and
Majorana fermions in which a single ion is coherently manipulated in two or
more orthogonal spatial modes [25]. The coherent control of two-dimensional
motion of a trapped ion demonstrated in this work can be applied to realize
such experiments. Lastly, utilization of quantum motion in multiple axes for
the realization of entangling gates can reduce the experimental overhead
required to suppress interactions in multiple directions often employed in
trapped ion quantum computing setups, such as asymmetric trap geometry [45]
and trap RF voltage offset [46].

##  IV Methods

###  IV.1 Instruments

The pulsed laser beams are provided by a 355-nm mode-locked laser (Coherent,
Paladin Compact 355-4000). Its repetition rate fluctuates around 120.1 MHz due
to the thermal and acoustic perturbations in the laser cavity. The repetition
rate is monitored by an ultrafast photodetector (Alphalas, UPD-30-VSG-P). The
drift is compensated by an AOM in the path of the $R_{V}$ beam whose RF
frequency is updated at a rate of 50 kHz by a field-programmable-gate-array
(Digilent, Arty-S7) running a custom PID program [33]. The RF trap voltage is
sampled by a capacitive divider and rectified by a diode circuit [29]. The
rectified voltage is fed to a high-speed PID controller (New Focus, LB1005-S),
which controls the output power of the RF source to stabilize the trap RF
voltage [28].

###  IV.2 Experimental protocol

To drive the two-dimensional motion of a single trapped ion along a closed
trajectory, we first optimize the Raman detuning to a frequency where the
phase space trajectories for the X and Y axis can be closed simultaneously.
The protocol is as follows: (1) The ion is ground-state cooled by sequentially
applying sideband cooling pulses to the X and Y modes. (2) The spin state of
the ion is initialized to $\ket{\downarrow}$ via optical pumping. (3) With the
probe time set to $T=2\pi\times N/(\omega_{Y}-\omega_{X})$ where N is an
integer, we scan the Raman detuning between $\omega_{Y}$ and $\omega_{X}$ and
look for frequencies where the measured spin state is close to
$\ket{\downarrow}$, which indicates the simultaneous disentanglement of the
spin from the motional states in the X and Y axes. (4) To balance the red and
blue sideband transitions and calibrate out the differential Strak shift, the
RF power and frequency for the transitions are fine-tuned individually to
minimize the $\ket{\uparrow}$ state population.

In the entangled coherent state experiment, we limit the Rabi frequency of the
blue sideband transition used to probe the phonon distribution of the Y mode
to about $5$ kHz, so as not to excite the blue sideband transition of the X
mode. At this value, the expected maximum amplitude of the X mode blue
sideband Rabi oscillation is $0.7\%$, thus we did not include the excitation
of the X mode in the phonon distribution analysis.

The relatively low frequency of the blue sideband Rabi oscillation means that
even a small drift in the secular frequency of the trap can affect the phonon
state estimation results. The trap RF power is stabilized by a PID loop, but
it drifts slowly due to the temperature changes in the components of the PID
circuit at a rate of 2 kHz/hr in the worst case. Thus, we interleave a blue
sideband Ramsey spectroscopy experiment with the main experiment for every
data point in 3(d) and (e), and monitored the change of secular frequency
throughout data collection process. Data collection for each point in the
figures takes about 5 minutes and for all the points in each plot about 2
hours. We stop the experiment if the secular frequency of the Y mode changed
from the calibrated value by more than 300 Hz. We recalibrate the frequencies
for the blue sideband transition and the spin-dependent force, and then resume
the experiment. With a Rabi frequency of 5 kHz and detuning of 300 Hz, the
amplitude of the blue sideband Rabi oscillation decreases by $0.4\%$ and Rabi
frequency increases by $0.9\%$, which are negligible for the purposes of our
experiment. Also, we note that in the analysis of the blue sideband Rabi
oscillation, we use the exact form of
$\Omega_{n+1,n}=\Omega_{0,0}\bra{n+1}e^{i\eta_{Y}(\hat{a}^{{\dagger}}_{Y}+\hat{a}_{Y})}\ket{n}=\Omega_{0,0}exp{\left(-\eta_{Y}^{2}/2\right)}\eta_{Y}/\sqrt{n+1}{\
L}_{n}^{1}(\eta_{Y}^{2})$ where $\Omega_{0,0}$ is the carrier transition Rabi
frequency with zero phonons, $\eta$ is the Lamb-Dicke factor, and $L_{n}^{1}$
is the generalized Laguerre polynomial of n-th order [47], since in our data
the maximum value of $\eta_{Y}\sqrt{2{\bar{n}}_{Y}+1}$ is about 0.33 where the
Lamb-Dicke approximation becomes inaccurate.

Also, to eliminate the possibility of the slow drift during the experiment
affecting the observed pattern of parity modulation, we conducted the
experiment in randomized orders of $t_{SDF}$. The full randomized sequences of
$t_{SDF}$ used for the data sets in 3(d) and (e) are available in the
Supplementary Material.

###  IV.3 Effect of finite temperature on phonon number parity

The measured maximum value of parity shown in 3 does not reach unity because
of imperfect sideband cooling, which in our setup typically results in
$\overline{n}_{X}\simeq 0.2$ and $\overline{n}_{Y}\simeq 0.05$ . This finite
temperature effect is modelled by considering a mixed motional state with a
population of $p_{X,1}$ and $p_{Y,1}$ in the first excited state of each mode
and the rest in the motional ground states. We include the following three
initial states in the model. (i) $\ket{1}_{X}\ket{0}_{Y}$ with probability
$p_{X,1}\left(1-p_{Y,1}\right)$, (ii) $\ket{0}_{X}\ket{1}_{Y}$ with
probability $\left(1-p_{X,1}\right)p_{Y,1}$, and (iii)
$\ket{0}_{X}\ket{0}_{Y}$ with probability
$\left(1-p_{X,1}\right)\left(1-p_{Y,1}\right)$. The $\ket{1}_{X}\ket{1}_{Y}$
state is not considered since its probability is negligible. When the motional
state is the n-th excited state, the effect of the displacement operator and
the resulting phonon distribution can be calculated using number state
representations of the displacement operator,
$d_{mn}^{\alpha}=\bra{m}\hat{D}(\alpha)\ket{n}$. We employed the results of
[48] to calculate $d_{mn}^{\alpha}$.

For (i), the modified phonon distribution of the Y mode is as follows

$\begin{split}p_{Y,n}\left(t\right)=\frac{1}{\left(1+e^{-2\left(\left|\alpha\left(t\right)\right|^{2}+\left|\beta\left(t\right)\right|^{2}\right)}\right)}e^{-\left|\beta\left(t\right)\right|^{2}}\frac{\left|\beta\left(t\right)\right|^{2n}}{n!}\\\
\times\left(1+\left(-1\right)^{n}\frac{\left(d_{11}^{-2\alpha}+d_{11}^{2\alpha}\right)}{2}\right)\end{split}$

For (ii),

$\begin{split}p_{Y,n}\left(t\right)=\frac{1}{2\left(1+e^{-2\left(\left|\alpha\left(t\right)\right|^{2}+\left|\beta\left(t\right)\right|^{2}\right)}\right)}\\\
\times(\left|d_{n1}^{\beta}\right|^{2}+\left|d_{n1}^{-\beta}\right|^{2}+\left(\left(d_{n1}^{\beta}\right)^{\ast}d_{n1}^{-\beta}+d_{n1}^{\beta}{(d_{n1}^{-\beta})}^{\ast}\right)e^{-2\left|\alpha\right|^{2}})\end{split}$

.

The weighted sum of the phonon distributions corresponding to the above three
cases were used to fit the parity modulation data and extract the Rabi
frequency, $p_{X,1}$ and $p_{Y,1}$.

##  V Data Availability

The data supporting the findings in this study are available from the
corresponding author upon reasonable request.

##  VI Acknowledgments

This work was supported by Institute for Information & communications
Technology Planning & Evaluation (IITP) grant (No.2022-0-01040), the National
Research Foundation of Korea (NRF) grant (No. 2020R1A2C3005689, No.
2022M3E4A1083521), and the Samsung Research Funding & Incubation Center of
Samsung Electronics (No. SRFC-IT1901-09).

##  VII Author Contributions

H.J. designed, carried out and analyzed the experiments. J.Y.K. carried out
the experiments and implemented the experimental control system and the feed-
forward system. H.J., J.Y.K., J.E.K., W.H.C., and K.K. contributed to the
setup of the experimental apparatus. H.J. wrote and edited the manuscript.
T.K. supervised the experiment. All authors participated in the discussion of
the experimental results.

## References

  * Glauber [1963] R. J. Glauber, Phys. Rev. 131, 2766 (1963).
  * Sanders [2012] B. C. Sanders, J. Phys. A: Math. Theor. 45, 244002 (2012).
  * Jeong  _et al._ [2001] H. Jeong, M. S. Kim, and J. Lee, Phys. Rev. A 64, 052308 (2001).
  * Jeong and Kim [2002] H. Jeong and M. S. Kim, Phys. Rev. A 65, 042305 (2002).
  * Wang [2001] X. Wang, Phys. Rev. A 64, 022302 (2001).
  * Jeong and An [2006] H. Jeong and N. B. An, Phys. Rev. A 74, 022104 (2006).
  * Park and Jeong [2010] K. Park and H. Jeong, Phys. Rev. A 82, 062325 (2010).
  * Lee and Jeong [2013] S.-W. Lee and H. Jeong, Phys. Rev. A 87, 022326 (2013).
  * van Enk and Hirota [2001] S. J. van Enk and O. Hirota, Phys. Rev. A 64, 022313 (2001).
  * Joo  _et al._ [2012] J. Joo, K. Park, H. Jeong, W. J. Munro, K. Nemoto, and T. P. Spiller, Phys. Rev. A 86, 043828 (2012).
  * Stobińska  _et al._ [2007] M. Stobińska, H. Jeong, and T. C. Ralph, Phys. Rev. A 75, 052105 (2007).
  * Lee  _et al._ [2011] C.-W. Lee, M. Paternostro, and H. Jeong, Phys. Rev. A 83, 022102 (2011).
  * Ourjoumtsev  _et al._ [2009] A. Ourjoumtsev, F. Ferreyrol, R. Tualle-Brouri, and P. Grangier, Nature Phys 5, 189 (2009).
  * Wang  _et al._ [2022] Z. Wang, Z. Bao, Y. Wu, Y. Li, W. Cai, W. Wang, Y. Ma, T. Cai, X. Han, J. Wang, Y. Song, L. Sun, H. Zhang, and L. Duan, Science Advances 8, eabn1778 (2022).
  * Wang  _et al._ [2016] C. Wang, Y. Y. Gao, P. Reinhold, R. W. Heeres, N. Ofek, K. Chou, C. Axline, M. Reagor, J. Blumoff, K. M. Sliwa, L. Frunzio, S. M. Girvin, L. Jiang, M. Mirrahimi, M. H. Devoret, and R. J. Schoelkopf, Science 352, 1087 (2016).
  * Monroe  _et al._ [1996] C. Monroe, D. M. Meekhof, B. E. King, and D. J. Wineland, Science 272, 1131 (1996).
  * Haljan  _et al._ [2005] P. C. Haljan, K.-A. Brickman, L. Deslauriers, P. J. Lee, and C. Monroe, Phys. Rev. Lett. 94, 153602 (2005).
  * McDonnell  _et al._ [2007] M. J. McDonnell, J. P. Home, D. M. Lucas, G. Imreh, B. C. Keitch, D. J. Szwer, N. R. Thomas, S. C. Webster, D. N. Stacey, and A. M. Steane, Phys. Rev. Lett. 98, 063603 (2007).
  * Poschinger  _et al._ [2010] U. Poschinger, A. Walther, K. Singer, and F. Schmidt-Kaler, Phys. Rev. Lett. 105, 263602 (2010).
  * Kienzler  _et al._ [2016] D. Kienzler, C. Flühmann, V. Negnevitsky, H.-Y. Lo, M. Marinelli, D. Nadlinger, and J. Home, Phys. Rev. Lett. 116, 140402 (2016).
  * Gerry [1997] C. C. Gerry, Phys. Rev. A 55, 2478 (1997).
  * Zou  _et al._ [2001] X.-B. Zou, J. Kim, and H.-W. Lee, Phys. Rev. A 63, 065801 (2001).
  * Solano  _et al._ [2002] E. Solano, R. L. d. M. Filho, and N. Zagury, J. Opt. B: Quantum Semiclass. Opt. 4, S324 (2002).
  * Zhong  _et al._ [2018] Z.-R. Zhong, X.-J. Huang, Z.-B. Yang, L.-T. Shen, and S.-B. Zheng, Phys. Rev. A 98, 032311 (2018).
  * Rossetti  _et al._ [2016] R. F. Rossetti, G. D. d. M. Neto, J. C. Egues, and M. H. Y. Moussa, EPL 115, 53001 (2016).
  * Jeon  _et al._ [2021] H. Jeon, N. Park, J. Yu, Y.-D. Kwon, D. Yum, and W. Jhe, J. Korean Phys. Soc. 78, 251 (2021).
  * Schmidt-Kaler  _et al._ [2003] F. Schmidt-Kaler, H. Häffner, S. Gulde, M. Riebe, G. Lancaster, T. Deuschle, C. Becher, W. Hänsel, J. Eschner, C. Roos, and R. Blatt, Appl. Phys. B 77, 789 (2003).
  * Johnson  _et al._ [2016] K. G. Johnson, J. D. Wong-Campos, A. Restelli, K. A. Landsman, B. Neyenhuis, J. Mizrahi, and C. Monroe, Review of Scientific Instruments 87, 053110 (2016).
  * Park  _et al._ [2021] Y. Park, C. Jung, M. Seong, M. Lee, D. D. Cho, and T. Kim, Sensors 21, 1143 (2021).
  * Olmschenk  _et al._ [2007] S. Olmschenk, K. C. Younge, D. L. Moehring, D. N. Matsukevich, P. Maunz, and C. Monroe, Phys. Rev. A 76, 052314 (2007).
  * Turchette  _et al._ [1998] Q. A. Turchette, C. S. Wood, B. E. King, C. J. Myatt, D. Leibfried, W. M. Itano, C. Monroe, and D. J. Wineland, Phys. Rev. Lett. 81, 3631 (1998).
  * Hayes  _et al._ [2010] D. Hayes, D. N. Matsukevich, P. Maunz, D. Hucul, Q. Quraishi, S. Olmschenk, W. Campbell, J. Mizrahi, C. Senko, and C. Monroe, Phys. Rev. Lett. 104, 140501 (2010).
  * Mount  _et al._ [2016] E. Mount, D. Gaultney, G. Vrijsen, M. Adams, S.-Y. Baek, K. Hudek, L. Isabella, S. Crain, A. van Rynbach, P. Maunz, and J. Kim, Quantum Inf Process 15, 5281 (2016).
  * Meekhof  _et al._ [1996] D. M. Meekhof, C. Monroe, B. E. King, W. M. Itano, and D. J. Wineland, Phys. Rev. Lett. 76, 1796 (1996).
  * Turchette  _et al._ [2000] Q. A. Turchette, C. J. Myatt, B. E. King, C. A. Sackett, D. Kielpinski, W. M. Itano, C. Monroe, and D. J. Wineland, Phys. Rev. A 62, 053807 (2000).
  * Sørensen and Mølmer [2000] A. Sørensen and K. Mølmer, Phys. Rev. A 62, 022311 (2000).
  * Benhelm  _et al._ [2008] J. Benhelm, G. Kirchmair, C. F. Roos, and R. Blatt, Nature Phys 4, 463 (2008).
  * Lee  _et al._ [2005] P. J. Lee, K.-A. Brickman, L. Deslauriers, P. C. Haljan, L.-M. Duan, and C. Monroe, J. Opt. B: Quantum Semiclass. Opt. 7, S371 (2005).
  * Zhang  _et al._ [2018] J. Zhang, M. Um, D. Lv, J.-N. Zhang, L.-M. Duan, and K. Kim, Phys. Rev. Lett. 121, 160502 (2018).
  * Gan  _et al._ [2020] H. Gan, G. Maslennikov, K.-W. Tseng, C. Nguyen, and D. Matsukevich, Phys. Rev. Lett. 124, 170502 (2020).
  * Toyoda  _et al._ [2015] K. Toyoda, R. Hiji, A. Noguchi, and S. Urabe, Nature 527, 74 (2015).
  * Campbell and Hamilton [2017] W. C. Campbell and P. Hamilton, J. Phys. B: At. Mol. Opt. Phys. 50, 064002 (2017).
  * Shinjo  _et al._ [2021] A. Shinjo, M. Baba, K. Higashiyama, R. Saito, and T. Mukaiyama, Phys. Rev. Lett. 126, 153604 (2021).
  * Martínez-Garaot  _et al._ [2018] S. Martínez-Garaot, A. Rodriguez-Prieto, and J. G. Muga, Phys. Rev. A 98, 043622 (2018).
  * Madsen  _et al._ [2004] M. Madsen, W. Hensinger, D. Stick, J. Rabchuk, and C. Monroe, Appl Phys B 78, 639 (2004).
  * Yum  _et al._ [2020] D. Yum, T. Dutta, J. P. S. Cheng, and M. Mukherjee, J. Korean Phys. Soc. 77, 1143 (2020).
  * Leibfried  _et al._ [2003] D. Leibfried, R. Blatt, C. Monroe, and D. Wineland, Rev. Mod. Phys. 75, 281 (2003).
  * Cahill and Glauber [1969] K. E. Cahill and R. J. Glauber, Phys. Rev. 177, 1857 (1969).


*equal contribution

Knowledge distillation is the procedure of transferring "knowledge" from a large model (the teacher) to a more compact one (the student), often being used in the context of model compression.  When both models have the same architecture, this procedure is called self-distillation. Several works have anecdotally shown that a self-distilled student can outperform the teacher on held-out data. In this work, we systematically study self-distillation in a number of settings. We first show that even with a highly accurate teacher, self-distillation allows a student to surpass the teacher in all cases. Secondly, we revisit existing theoretical explanations of (self) distillation and identify contradicting examples, revealing possible drawbacks of these explanations. Finally, we provide an alternative explanation for the dynamics of self-distillation through the lens of loss landscape geometry. We conduct extensive experiments to show that self-distillation leads to flatter minima, thereby resulting in better generalization.

§ INTRODUCTION

In recent years, deep neural networks have found success in various tasks such as image classification [38, 54, 20], object detection [54], speech recognition [3], and language understanding [15].  But their success comes at the cost of incurring billions of model parameters.

As a consequence, it can be very challenging to deploy such cumbersome models on devices with constrained resources, and a plethora of model compression and acceleration methods have been developed to address this challenge.

One such method is knowledge distillation (KD), introduced by Bucilua et al. [7] and Hinton et al. [31] as a method of transferring knowledge from a large model (teacher) to another lightweight model (student) that is much easier to deploy without significant loss in performance. The intuition is that during training, the model needs to sift through a large set of possible from massive, highly redundant datasets, so a vast amount of representation capacity is needed. But during inference, the learned features might well be represented using smaller models. While Bucilua et al. [7] achieve this through matching predicted logits over a training dataset, Hinton et al. [31] introduce a tunable temperature the softmax outputs to better represent smaller probabilities in the model output.

In the original KD setting, the student model has fewer parameters than the teacher, thereby resulting in improved efficiency. However, even if model compression is not the goal, it is now folklore that distillation leads to improved model performance. A series of recent works have explored the setting when the teacher and student architectures are identical. Somewhat curiously, here too, KD leads to uniform boosts in student test accuracy [24, 59, 1, 46, 5, 64, 55]. This special case is often referred as self-distillation, and will be the central focus of our work.

Despite its promise, the reasons behind the success of self-distillation are not well-understood. At the face of it, both teacher and student have access to the same training dataset; the model capacities of the teacher and the student are identical; the training algorithm is identical (modulo possible choices of hyper-parameters). Where, then, are the benefits of self-distillation coming from?

Our contributions. In this paper, we systematically investigate the behavior of self-distillation, and uncover possible explanations for its success.

First, we perform a series of careful self-distillation experiments on standard image classification benchmarks. We confirm that even when the teacher has very high test accuracy, self-distillation can still enable the student to outperform its teacher.

Second, we revisit an existing theory of knowledge distillation called the multi-view hypothesis [2]. At a high-level, the hypothesis states that the teacher (for various reasons) typically only learns a strict subset of “views” (or facets) of the input data, and self-distillation enables the student to learn the rest of these views. We design a series of experiments that contradict this hypothesis, potentially unveiling its limitations as an explanation of KD.

Third, we investigate self-distillation through the lens of loss landscape geometry. We conduct a series of experiments to show that self-distillation encourages the student to find flatter minima (relative to the teacher). These findings are consistent with recent theoretical results on KD for shallow (kernel) models [46], and can be viewed as an alternative explanation for why self-distillation works: adding a "distillation" term flattens the loss landscape around minima, thereby improving generalization.

§ RELATED WORK

Knowledge distillation. Since its original introduction in [7, 31], many subsequent papers have introduced several refinements to KD. FitNets [53] focus on the intermediate representations by using regression to match the teacher and student feature activations. Similarly, attention transfer [62] deals with the feature maps instead of the output logits. Yonglong  [56] use a contrastive-based objective for transferring knowledge between networks. RKD [48] utilizes the distance-wise and angle-wise distillation losses that penalize structural differences in relations. Mishra and Marr [45] and Polino  [51] combines KD with network quantization to reduce bit precision of activations and weights. Xu  [58] use a conditional adversarial network to learn a loss function for knowledge distillation. Yin  [61]  generate class-conditional images for data-free KD. Additionally, KD has been explored beyond supervised learning. Lopez [43] extend KD to unsupervised, semi-supervised, and multi-task learning settings by combining frameworks from [31, 57].
Applications of KD have even made their way to recommender systems [36, 35], image retrieval [10], federated learning [41], and graph similarity computation [52].

Self-distillation. Several attempts to explain the behavior of self-distillation have already been made. Furlanello [24] shows that “dark knowledge” is a form of importance weighting. Dong [19] demonstrates that early-stopping is essential for self-distillation to harness dark-knowledge. Zhang and Sabuncu [64] provides empirical evidence that diversity in teacher predictions is correlated with the performance of the student in self-distillation. Based on this, they offer a new interpretation for teacher-student training as amortized a posteriori estimation of the softmax probability outputs, such that teacher predictions allow instance-specific regularization. They also propose a novel instance-specific label smoothing techniques that directly increase predictive diversity.

Mohabi  [46] provide a theoretical analysis of self-distillation in the classical regression setting where the student model is only trained on the soft labels provided by the teacher. In particular, they fit a nonlinear function to training data with models belonging to a Hilbert space under $L_2$ regularization. In this setting, multi-round self-distillation is progressively limiting the number of basic functions to represent the solution. Additionally, [5] build upon the previous analysis by also including the weighted-ground truth targets in the self-distillation procedure. They demonstrate that for fixed distillation weights, the ground-truth targets lessen the sparsification and regularization effect of the self-distilled solution. However, both [46] and [5] use the Mean Square Error (MSE) for the objective function, and therefore their results do not directly apply to image classification models trained using the cross-entropy loss.

Allen-Zhu and Li [2] study self-distillation under a more practical setting where the student is trained on a combination of soft-labels from the teacher and ground-truth targets. Specifically, the student objective function consists of a cross-entropy loss in the usual supervised task, and a Kullback-Leibler divergence term to encourage the student match the soft probabilities of the teacher model. They also introduce the "multi-view" hypothesis to explain how ensemble, knowledge distillation, and self-distillation work. We will discuss the hypothesis in more detail in Section <ref>. Finally, [55] systematically study the nature of (standard) knowledge distillation. They particularly study the problem through fidelity: how well the student can match its teacher's predictions, and generalization: the performance of a student on unseen held-out data. The work of [63] is perhaps closest to ours in spirit. However, their technical definition of self-distillation is different from what we consider, and therefore their observations do not directly port over to our setting.

Illustration of 2-round self-distillation. $f^{(0)}$ is the model trained from scratch using only ground-truth labels. $f^{(1)}$ is trained through self-distillation using both ground-truth labels $y$ and soft-labels $y^{(0)}$ from its teacher. The same procedure is used to train $f^{(2)}$.

§ PRELIMINARIES

Consider the supervised setting where $\mathcal{X}$  and $\mathcal{Y}$ denote the input and output (label) space respectively with $|\gY|=k$.

We wish to learn a classifier $f: \mathcal{X} \times \theta \rightarrow \mathbb{R}^k$ with parameters $\theta$ that maps input feature $x \in \mathcal{X}$ to a categorical predictive distribution over $\mathcal{Y}$.  Specifically, let $\mathbb{P}(y = i|\boldsymbol{x},\theta) = \sigma_i(f(\boldsymbol{x},\theta))$ where $\sigma(\cdot)$ is the standard softmax function.  We define $f(\boldsymbol{x},\theta)$ as the logits of the classifier $f$. We let $f_t$ and $f_s$ be respectively functions of the teacher and student, parameterized by $\theta_t$ and $\theta_s$. These functions are typically implemented as deep neural networks. When we refer to an ensemble of models, the logits $(\boldsymbol{z}_1, ..., \boldsymbol{z}_m)$ where $\boldsymbol{z}_i = f_i(x,\theta_i)$ are averaged to form the final logit vector, i.e. $\boldsymbol{z}_{ens} = \frac{1}{m} \sum_{i=1}^m \boldsymbol{z}_i$.

In conventional knowledge distillation, given a pre-trained teacher model, a student model is trained to emulate the teacher by minimizing the following objective:

\mathcal{L}_{KD} = \alpha \mathcal{L}_{CE}(\boldsymbol{z}_s,\boldsymbol{y}) + (1-\alpha) \mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t)

In the above equation, $\mathcal{L_{\textit{CE}}}(\boldsymbol{z}_s, \boldsymbol{y}) := - \sum_{j=1}^{k} y_j \text{log}\sigma_j(\boldsymbol{z}_s)$ is the usual cross-entropy loss between the student logits $z_s$ and labels $y$, and

$$\mathcal{L}_{KL}(\boldsymbol{z}_s, \boldsymbol{z}_t) := \tau^2 \sum_{j=1}^{k} \sigma_j(\boldsymbol{z}_t/\tau) \text{log}\sigma_j(\boldsymbol{z}_t/\tau) - \sigma_j(\boldsymbol{z}_t/\tau) \text{log}\sigma_j(\boldsymbol{z}_s/\tau)$$

is the Kullback-Leibler divergence between the scaled student and teacher logits. Here, $\tau > 0$ is a temperature hyperparameter, $\boldsymbol{z}_t = f(\boldsymbol{x},\theta_t)$, and $\boldsymbol{z}_s = f(\boldsymbol{x},\theta_s)$, while $\alpha \in [0,1)$ is a constant hyperparameter that controls the relative importance of the cross-entropy and Kullback-Leibler terms.

For self-distillation, the teacher and student will have the same model architecture. At round 0, the teacher model is trained from scratch. Subsequently, for every round of distillation, the teacher is the student model obtained in the previous step. We  denote the model at the $n^{th}$ step of distillation as $f^{(n)}$, parameterized with $\theta_n$. See Figure <ref> for an illustration.

§ DOES THE STUDENT ALWAYS SURPASS THE TEACHER?

First, we revisit (folklore) intuition in self-distillation, and check whether it is indeed correct. Self distillation is often used with the underlying assumption that the student must improve upon a teacher, and existing results on self-distillation have mostly supported this assumption. However, there arises a natural question: can self-distillation always improve upon a teacher trained on the same dataset from scratch (i.e., using only cross entropy)?

Specifically, is self-distillation a useful strategy that can improve upon even the highest-performing of teachers? We demonstrate that this is in fact true for single-round self-distillation through a series of experiments.

First, we start by noticing that teacher model accuracies that were previously reported in related literature on self-distillation [46, 5, 2] almost always lag behind the state-of-the-art. (See table: teacher comparison.). Therefore, it could be that any gains by a distilled student over the teacher might have been illusory, and could have been nullified if the teacher itself was trained better.

We know that model performance can, in practice, be further improved by (1) choosing the right set of hyperparameters and (2) adopting advanced data augmentation methodologies ([16], [12]). We leverage these to train better-performing teachers than ones that have been previously reported. We then train student models using self-distillation for a variety of architectures and datasets, and measure benefits (if any) of self-distillation in terms of test accuracy.

Experiment setup. For our experiments, we consider two architectures: ResNet18 and VGG16 trained on CIFAR-10 and CIFAR-100. We use several performance-improving heuristics, including a cosine learning rate schedule and early stopping. We also leverage modern data augmentation techniques, specifically AutoAugment [12] and Cutout [16], to train more accurate models. We choose the best models in every setting, and then use self-distillation to train the corresponding student models. We report all performance numbers in table: teacher comparison.

Table <ref> compares the reported teachers' test accuracy on the CIFAR-10/100 dataset to the student models we trained via self-distillation (details of training hyperparameters are provided in the Appendix). We infer the following observations based on Table <ref>.

(1) Teacher models used in [46, 5, 2] are relatively weak baselines; our ResNet18 teacher achieves $95.56\%$ test accuracy, which is even higher than larger architectures (e.g. ResNet34, ResNet50) used in previously published work.

(2) Self-distillation does indeed further boost generalization (e.g., $97.16\% \rightarrow 97.40\%$) even when the teacher is a strong classifier trained with heavy-duty data augmentation.

We therefore conclude that the aforementioned results are directionally correct: self-distillation really does improve upon teacher accuracy, even when the teachers themselves are strong classifiers. However, this still does not reveal any reasons behind this ubiquitous performance boost. Our next two sections address this matter.

Comparison of reported teacher and student performances from published self-distillation literature. A proper choice of training hyperparameters makes a baseline teacher outperform the self-distilled students reported in [46, 5, 2]. Moreover, our choice of architecture (e.g., ResNet18) has fewer parameters than the models of [46, 5, 2]. However, self-distillation does improve the generalization when teacher is trained with advanced data augmentation techniques such as Cutout [16] and AutoAugment [12].

Literature     Architecture     Dataset     Teacher     Student

1l|Mohabi et al.[46]     1c|ResNet50     1c|CIFAR-10     $80.5\%$     $81.3\%$

1l|Mohabi et al.[46]     1c|VGG16     1c|CIFAR-100     $55.0\%$     $56.5\%$

1l|Borup & Andersen [5]     1c|ResNet34     1c|CIFAR-10     $84\%$     $85\%$

1l|Allen-Zhu & Li [2]     1c|ResNet34     1c|CIFAR-10     $93.65\%$     $94.21\%$

1l|Allen-Zhu & Li [2]     1c|ResNet34     1c|CIFAR-100     $71.66\%$     $73.14\%$

1l|Ours     1c|ResNet18     1c|CIFAR-10     $\mathbf{95.56}\%$     $\mathbf{95.84}\%$

1c|Ours + Data Aug. ([16, 12])     1c|ResNet18     1c|CIFAR-10     $\mathbf{97.16}\%$     $\mathbf{97.40}\%$

1l|Ours     1c|VGG16     1c|CIFAR-10     $\mathbf{94.39}\%$     $\mathbf{94.50}\%$

1c|Ours + Data Aug. ([16, 12])     1c|VGG16     1c|CIFAR-10     $\mathbf{96.19}\%$     $\mathbf{96.49}\%$

1l|Ours     1c|ResNet18     1c|CIFAR-100     $\mathbf{76.30}\%$     $\mathbf{77.73}\%$

1l|Ours + Data Aug. ([16, 12])     1c|ResNet18     1c|CIFAR-100     $\mathbf{78.22}\%$     $\mathbf{80.71}\%$

§ CAN STUDENTS BECOME PROGRESSIVELY BETTER?

Self-distillation results on CIFAR-10. Data augmentation means leveraging Cutout and AutoAugment techniques. We report mean and standard deviations of test accuracy from three independent runs. $\uparrow$ (resp. $\downarrow$) stands for the increase (resp. decrease) in test accuracy relative to its teacher.

Architecture     Dataset     Data Aug.     $\alpha$     Teacher     Round 1     Round 2     Round 3     SAM

1l|ResNet18     1c|CIFAR-10     1c|No     0.2     $95.57 \pm 0.15$     $95.80\pm0.05 (\uparrow)$     $95.58 \pm 0.13 (\downarrow)$     $95.62 \pm 0.09 (\uparrow)$     $96.25 \pm 0.06$

1l|ResNet18     1c|CIFAR-10     1c|No     0.5     $95.57 \pm 0.15$     $95.84\pm0.10 (\uparrow)$     $95.60 \pm 0.17 (\downarrow)$     $95.59 \pm 0.01 (\downarrow)$     $96.25 \pm 0.06$

1l|ResNet18     1c|CIFAR-10     1c|No     0.8     $95.57 \pm 0.15$     $95.74\pm0.09 (\uparrow)$     $95.55 \pm 0.10 (\downarrow)$     $95.62 \pm 0.09 (\uparrow)$     $96.25 \pm 0.06$

1l|ResNet18     1c|CIFAR-10     1c|Yes     0.2     $97.15 \pm 0.07$     $97.24\pm0.05 (\uparrow)$     $97.39 \pm 0.01 (\uparrow)$     $97.44 \pm 0.04 (\uparrow)$     $97.42 \pm 0.04$

1l|ResNet18     1c|CIFAR-10     1c|Yes     0.5     $97.15 \pm 0.07$     $97.40\pm0.04 (\uparrow)$     $97.36 \pm 0.05 (\downarrow)$     $97.38 \pm 0.04 (\uparrow)$     $97.42 \pm 0.04$

1l|ResNet18     1c|CIFAR-10     1c|Yes     0.8     $97.15 \pm 0.07$     $97.28\pm0.07 (\uparrow)$     $97.38 \pm 0.11 (\uparrow)$     $97.43 \pm 0.05 (\uparrow)$     $97.42 \pm 0.04$

1l|VGG16     1c|CIFAR-10     1c|No     0.2     $94.39 \pm 0.11$     $94.45\pm0.12 (\uparrow)$     $94.25 \pm 0.09 (\downarrow)$     $94.25 \pm 0.04 (-)$     $95.02 \pm 0.17$

1l|VGG16     1c|CIFAR-10     1c|No     0.5     $94.39 \pm 0.11$     $94.50\pm0.12 (\uparrow)$     $94.26 \pm 0.14 (\downarrow)$     $94.16 \pm 0.17 (\downarrow)$     $95.02 \pm 0.17$

1l|VGG16     1c|CIFAR-10     1c|No     0.8     $94.39 \pm 0.11$     $94.38\pm0.07 (\downarrow)$     $94.35 \pm 0.06 (\downarrow)$     $94.30 \pm 0.11 (\downarrow)$     $95.02 \pm 0.17$

1l|VGG16     1c|CIFAR-10     1c|Yes     0.2     $96.19 \pm 0.05$     $96.36\pm0.15 (\uparrow)$     $96.33 \pm 0.05 (\downarrow)$     $96.29 \pm 0.05 (\downarrow)$     $96.61 \pm 0.12$

1l|VGG16     1c|CIFAR-10     1c|Yes     0.5     $96.19 \pm 0.05$     $96.49\pm0.08 (\uparrow)$     $96.36 \pm 0.08 (\downarrow)$     $96.39 \pm 0.04 (\uparrow)$     $96.61 \pm 0.12$

1l|VGG16     1c|CIFAR-10     1c|Yes     0.8     $96.19 \pm 0.05$     $96.36\pm0.03 (\uparrow)$     $96.42 \pm 0.06 (\uparrow)$     $96.37 \pm 0.05 (\downarrow)$     $96.61 \pm 0.12$

§.§ The multi-view hypothesis

Next, we revisit (seminal) prior work. In a thought-provoking paper, Allen-Zhu & Li [2] have proposed the “multi-view” hypothesis as a possible explanation as to why KD works so well. The multi-view hypothesis suggests that natural datasets (particularly for image classification) exhibit a special structure. Samples in such datasets consist of multiple “views” or concepts which when grouped together imply a class.  For example, a car image can be correctly classified when the model look at the headlights, the wheels, or the windows. Given a typical placement of a car in images, it is suffice to accurately predict a car using one of the above-mentioned features. The authors claim that several vision datasets (including CIFAR-10 and CIFAR-100) exhibit multi-view structure, and standard neural network models (such as ResNet-X) leverage this during training.

The authors support this hypothesis by analyzing ensembles of neural networks. They investigate how the improvement can be distilled into a single model using knowledge distillation. They then show that self-distillation is equivalent to implicitly combining ensembles and knowledge distillation to attain better test accuracy. They finally conclude that the performance boost can therefore be explained by the multi-view hypothesis. In particular, they argue that special structure in data is arguably necessary for ensemble to work.

Formally, a neural network trained using the cross-entropy loss from random initialization will:

* Learn one of the features $v \in \{v_1,v_2\}$ for the first label, and one of the features $v' \in \{v_3,v_4\}$ for the second label. As a result, $90\%$ of the training examples consisting of features $v$ and $v'$ are classified correctly. Once classified correctly, these samples contribute negligibly to the gradient.

* Afterwards, the network will memorize the remaining $10\%$ of the training data without learning any additional features, as there is not enough data remaining after the previous phase. This explains why models can achieve $100\%$ training accuracy but $90\%$ test accuracy.

To elaborate, under this hypothesis, an ensemble will learn more features than a single model. Further, during knowledge distillation, the student will be forced to learn additional features from the teacher. In both cases, the resulting model will have superior test accuracy compared to an individual model trained from scratch. In the case of self-distillation, the authors suggest that the procedure implicitly combines ensemble and knowledge distillation. Particularly, if the teacher learns $\gV_A$ features, the student is encouraged to also learn $\gV_A$. Subsequently, it purportedly learns additional features, $\gV_B$ on its own. Thus the self-distilled model performs better than the teacher by ensembling its independent features with those of the teacher, resulting in a larger learned set of features $\mathcal{V}_A \cup \mathcal{V}_B$

As empirical evidence, the authors show that one-round self-distillation allows students trained on CIFAR-10/100 surpass the teacher in test accuracy. They also show that when data that does not exhibit the multi-view structure (Gaussian like with target label generated by any fully-connected / residual / convolutional network), the ensemble does not improve upon any individual model in terms of test accuracy. Lastly, they demonstrate that if we first distill knowledge from an ensemble $ens_1$ to multiple student models, and create a second ensemble $ens_2$ from those student models, then the test accuracy of $ens_2$ does not exceed $ens_1$, and is in fact lower in many cases.

figureBAN vs Ensemble. The mean and standard deviation of accuracy is reported over 3 runs. The ensemble out-performs BAN at all stages, implying that training an ensemble is more effective than multiple rounds of self-distillation.

figureEnsemble as teachers. As more models are used as teachers, the student performance improves. However, the multi-view hypothesis is contradicted as the ensemble always outperforms the student.

Multi-round self-distillation
A natural extension to this line of thought would be to sequentially use self distillation to encourage student models to learn increasingly larger set of features – $\mathcal{V}_A \cup \mathcal{V}_B \cup \mathcal{V}_D$, where $\mathcal{V}_D$ are the features from model $D$ being implicitly introduced in the ensemble. We analyse if the multi-view hypothesis still holds in this case.

Experiment setup. For the first experiment, we train multiple individual models from scratch. An ensemble created from these models are then used as the teacher to perform knowledge distillation, where the student is a single model with the same architecture as the initial individual models. We increase the number of models in the ensemble from 2 to 9 and measure both the ensemble (teacher) and the student. In the next experiment, we perform self-distillation for 3 rounds. All the models have the same architecture. Each model model is trained for 600 epochs using Cutout and AutoAugment augmentations. We use $3$ different $\alpha$ values of $0.2$, $0.5$, and $0.8$. At each round, we save the model with the highest test accuracy and use it as the teacher for the next self-distillation round. An illustration of 2-round self-distillation can be seen in Figure <ref>. The architectures we consider for both of our experiments are ResNet18 and VGG16.

Results We report the our findings for the first experiment in figure <ref>. We observe that as we increase the number of models in an ensemble and use it as the teacher, then the student will also display better test accuracy. In other words, the more features we force the student to learn, the higher test accuracy it has. If the multi-view hypothesis correctly explains self-distillation, we expect that the student learns features from all the previous teachers in addition to its own independent features, thus achieving incrementally better test accuracy. However, our results for the second experiment show otherwise. From figure <ref>. While a single round of self-distillation consistently makes the student outperform the teacher, performing it for multiple rounds does not result in a stepwise better student. For example, when the model architecture is ResNet18, performing self-distillation using $\alpha = 0.2$ without self-distillation makes the test accuracy at every step evolve as follows: $95.17\% \rightarrow 95.8 \% \rightarrow  95.58 \% \rightarrow  96.25\%$. We can see that the accuracy fluctuates instead of progressively increasing, which holds fold for the majority of rows in table <ref>. This suggests that the multi-view hypothesis might not be sufficient to explain the success behind self-distillation.

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=100,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 94.81)
(1, 13.18)
(2, 11.44)
(3, 15.77)
(4, 45.94)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=7,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,

every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 5.01)
(1, 0.80)
(2, 0.65)
(3, 0.65)
(4, 2.62)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=100,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 94.81)
(1, 16.63)
(2, 9.77)
(3, 10.19)
(4, 45.94)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=7,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 5.01)
(1, 1.22)
(2, 0.89)
(3, 0.96)
(4, 2.62)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=100,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 94.81)
(1, 18.90)
(2, 12.10)
(3, 16.47)
(4, 45.94)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=7,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 5.01)
(1, 1.48)
(2, 0.93)
(3, 1.99)
(4, 2.62)

(a) $\alpha=0.2$ with augmentation     (b) $\alpha=0.5$ with augmentation     (c) $\alpha=0.8$ with augmentation

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=30,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 26.88)
(1, 9.84)
(2, 8.66)
(3, 10.05)
(4, 14.51)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=4,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 1.45)
(1, 0.72)
(2, 0.84)
(3, 0.88)
(4, 0.87)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=30,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 26.88)
(1, 11.45)
(2, 12.16)
(3, 12.20)
(4, 14.51)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=4,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 1.45)
(1, 0.76)
(2, 1.18)
(3, 1.23)
(4, 0.87)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=30,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 26.88)
(1, 12.41)
(2, 11.20)
(3, 10.07)
(4, 14.51)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=4,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 1.45)
(1, 1.09)
(2, 1.15)
(3, 0.98)
(4, 0.87)

(d) $\alpha=0.2$ w/o augmentation     (e) $\alpha=0.5$ w/o augmentation     (f) $\alpha=0.8$ w/o augmentation

Evolution of flatness measures in multi-step distillation  on ResNet18 for CIFAR-10. `T' and `S' stand for teacher and student models, respectively. Smaller trace (red) and $\lambda_{\max}$ (blue) values imply flatter minima. We observe the student with first round distillation enjoys getting a benefit finding flatter minima than the teacher. Surprisingly, self-distillation implicitly finds a flatter minima than SAM, which explicitly looks for the wider minima in its objective functions.

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=150,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 110.43)
(1, 12.95)
(2, 27.45)
(3, 17.55)
(4, 90.38)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=80,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 30.39)
(1, 1.08)
(2, 12.39)
(3, 4.52)
(4, 6.17)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=150,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 110.43)
(1, 11.31)
(2, 11.24)
(3, 10.61)
(4, 90.38)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=80,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 30.39)
(1, 1.06)
(2, 2.34)
(3, 1.67)
(4, 6.17)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=150,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 110.43)
(1, 17.57)
(2, 143.32)
(3, 16.01)
(4, 90.38)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=80,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 30.39)
(1, 2.13)
(2, 79.50)
(3, 1.76)
(4, 6.17)

(a) $\alpha=0.2$ with augmentation     (b) $\alpha=0.5$ with augmentation     (c) $\alpha=0.8$ with augmentation

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=20,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 9.70)
(1, 6.38)
(2, 5.30)
(3, 4.94)
(4, 13.75)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=4,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 2.85)
(1, 0.91)
(2, 0.98)
(3, 1.97)
(4, 2.96)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=20,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 9.70)
(1, 8.39)
(2, 4.61)
(3, 6.62)
(4, 13.75)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=4,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 2.85)
(1, 1.38)
(2, 1.54)
(3, 1.28)
(4, 2.96)

set layers

scale only axis,

axis y line*=left,
ymin=0, ymax=20,

ylabel style = align=center,
ylabel=Trace. <ref>,
grid style=dashed,
xticklabels=T, S1, S2, S3, SAM,
every axis plot/.append style=
bar width=.15,
bar shift=0pt,

[mark=none,red, ultra thick]
plot coordinates
(0, 9.70)
(1, 9.59)
(2, 10.35)
(3, 11.01)
(4, 13.75)

scale only axis,

axis y line*=right,
axis x line=none,
ymin=0, ymax=4,
legend style=at=(0.5,1.2),anchor=north,
ylabel style = align=center,
ylabel=Top eigenvalue <ref>,
every axis plot/.append style=
bar width=.15,
bar shift=7pt,
[mark=none,blue, ultra thick]
plot coordinates
(0, 2.85)
(1, 2.30)
(2, 2.42)
(3, 2.71)
(4, 2.96)

(d) $\alpha=0.2$ w/o augmentation     (e) $\alpha=0.5$ w/o augmentation     (f) $\alpha=0.8$ w/o augmentation

Evolution of flatness measures in multi-step distillation on VGG16 for CIFAR-10. `T' and `S' stand for teacher and student models, respectively. We observe the similar trends to Figure <ref>, which self-distillation implicitly finding wider minima than both teacher and SAM.

§.§ Do Born-Again Neural Networks Work?

Our proposal to perform multiple rounds of self-distillation is in fact not new, and dates back (at least) to Born-again Neural Networks (BAN) [24]. At a high level, this involves a re-training procedure that (essentially) performs multi-round self-distillation and then constructs an ensemble of the final models of every round to make predictions. Specifically, using the notation from Figure <ref>, the output of the corresponding Born-Again Neural Network is given by

$$f_{BAN} = (f^{(0)}(x) + f^{(1)}(x) + f^{(2)}(x))/3.$$

However, we discover that Born-Again Networks (BANs) actually perform worse than an ensemble over a collection of models trained independently from scratch.

Experimental setup. We use ResNet18 as the student models in BAN. We train the student models for 600 epochs, using SGD with momentum $0.9$, weight decay $3 \times 10^{-4}$, batch size 96, gradient clipping $5.0$, and an initial learning rate of $0.025$. The learning rate schedule is Cosine Annealing [44]. For data augmentation, we also use AutoAugment [13] and Cutout [17]. Additionally, we also train multiple models from scratch using the similar procedure to use for the ensemble. One can think of normal ensembling as a special case of BAN when $\alpha=1.0$. We report our BAN performance on CIFAR-10.

Results. Figure <ref> shows that BAN underperforms straightforward ensembling for all three choices of $\alpha$. Notice that as $\alpha$ increases, or as the student less depends on the teacher, BAN performance comes closer to that of an ensemble classifier. This suggests that it is more effective to just simply train an ensemble from scratch than performing several rounds of self-distillation as suggested by BAN.

(d) $\alpha = 0.2$     (e) $\alpha = 0.5$     (f) $\alpha = 0.8$

Tracking trace and top eigenvalue in distillation steps on ResNet18 (bottom row) for CIFAR-100. All models use augmentation.

Eigenspectrum of Hessian on ResNet18 from CIFAR-10 using [60]. The narrower eigenspectrum implies the flatter the loss surface. The explicit objective function from SAM (left) narrows down the eigenspectrum compared to the teacher model trained with regular cross-entropy loss. We further observe that the student model distillate from equivalent architecture (teacher) achieves an even flatter loss surface than SAM (left). The right plot compares the eigenspectrum of different students with various rounds.

§ SELF-DISTILLATION FINDS FLAT MINIMA

We have empirically demonstrated that contra the multi-view hypothesis, multiple rounds of self-distillation fail to yield progressively better students. In this section, we propose an alternative explanation for the success of self-distillation that is more consistent with this finding.

We specifically focus on the geometry of the (local) loss landscape around the learned model parameters. The connection between landscape flatness and generalization has been extensively studied from both the empirical and theoretical perspectives [37, 21, 34, 33], and flatter minima have been reported to give better generalization in various tasks [23, 50, 8]. We therefore hypothesize that self-distillation makes the student model attain flatter minima than the teacher.

To be clear, Dinh et al. [18] have shown that flatness on its own does not automatically imply better generalization in very deep models. Still, measuring and comparing flatness measures between the teacher and the student may provide insights on test accuracy. Similar to [9], we use the eigen-spectrum of the Hessian for the entire neural network to measure flatness of the loss landscape. Note that for ideal flat minima, all eigen-values of the Hessian should be positive and close to zero. This would necessarily result in also having a lower trace and lower top eigen-value $\lambda_{\max}$. We therefore also report the trace and the largest eigen-value as surrogate measures of flatness.

We use PyHessian [60] to estimate the trace, the top eigenvalue $\lambda_{max}$, and the eigen-spectral density of the models from <ref>. PyHessian leverages standard randomized linear algebra algorithms and automatic differentiation to estimate second-order properties of large neural network models. We report the results in Figure <ref> and <ref>.  We also trained a VGG16 and a ResNet18 with the recently proposed Sharpness-Aware Minimization (SAM) [23], an algorithm that explicitly encourages flat minima by modifying the training objective, as a suitable baseline for comparison.

figureContour visualization of the loss surface [40], original teacher model (left), student at self-distillation round 1 (middle), and student at self-distillation round 2 (right). All models used ResNet20 without skip connections. The training procedure of is similar to [40]

We notice that the teacher model trained with augmentation has a higher trace and $\lambda_{max}$ than without augmentation. Further, a single round of self-distillation will result in a student with lower trace and $\lambda_{max}$ than the teacher. Interestingly, performing multi-round self-distillation does not make successive students attain increasingly flatter minima, as the trace and $\lambda_{max}$ of models from subsequent rounds only fluctuate around those of the student from the first round.

Figure <ref> display the eigen-spectrum of the Hessian for ResNet18 on CIFAR-10. We can see that the overall distribution of eigenvalues of the student models is more concentrated around 0 compared to the teacher with or without SAM, therefore implying flatter minima.

Additionally, following [40], we trained a ResNet20 without skip connections for 2 self-distillation steps and visualize the loss surfaces similar to the authors. We demonstrate this in Figure <ref>. The borderlines of the teacher shows that it is much steeper than the students at both round 1 and 2. This suggests that the round-1 student achieves a flatter minima as compared to its teacher.

These observations, when combined, suggest that the self-distilled student exhibits relatively flatter minima when compared to a teacher trained from scratch. This is in line with the theoretically established results on induced regularization (in the context of shallow models [46]) and could be used to explain why (a single round of) self-distillation typically results provides better test accuracy.

§ DISCUSSION

In this work, we investigate several facets of self-distillation. We show that even with a strong teacher that is trained using modern techniques and augmentations, self-distillation still enables the student to surpass the teacher in terms of test accuracy. Secondly, we revisit previous literature on self-distillation and reveal potential limitations of these approaches. We then provide an alternative view on the success of self-distillation. In particular, we draw connections between self-distillation and loss geometry, and empirically show that the self-distilled student is encouraged to find flatter minima compared to the teacher; this may shed light on reasons behind its success.

As self-distillation (SD) is a special case of knowledge distillation (KD), we believe that understanding SD can help us develop better techniques for KD, which already has become a cornerstone of real-world state-of-the-art model building. An important open direction is the development of novel optimization procedures that implicitly perform (or emulate) self-distillation, resulting improved student performance while avoiding cumbersome (and resource-intensive) teacher-student knowledge transfer.

§ ACKNOWLEDGEMENTS

This work was supported in part by the National Science Foundation under grants CCF-2005804 and CCF-1815101, the US Department of Agriculture under grant USDA-NIFA:2021-67021-35329, and ARPA-E under grant DE:AR0001215.

§ APPENDIX

§.§ Experiment Details

For training the neural networks, we use SGD with momentum of $0.9$, learning rate $0.025$, weight decay $3 \times 10^{-4}$, batch-size 96, and gradient clipping value of $5.0$.

§.§ SVHN results

Accuracy     Trace     $\lambda_{\max}$

Teacher         95.23      197.53      9.24

Round 1 $(\alpha=0.5)$         95.94     205.79     11.200

Round 2 $(\alpha =0.5)$         95.67     98.62     8.30

Round 3 $(\alpha =0.5)$     95.17     271.71     12.873

SVHN Results for ResNet18

§.§ Additional BAN experiments

To investigate why BAN performs worse than normal ensembling, we calculate the difference (using Mean Square Error) between the logits of the student and the teacher at every self-distillation step and report it in Figure <ref>. We can see that the more self-distillation rounds that we perform, the more similar the predictive logits of the student model and those of its teacher model. Therefore, training BAN for multiple generations leads to initial improvements that gradually saturate, as observed by the authors, and this also indicates why increasing the number of rounds in BAN is less effective than taking an ensemble of models.

figureDifference between logits of student and its teacher on CIFAR-10 test set at every self-distillation step. We calculate the discrepency as follows $\frac{1}{n} \sum_{i=1}^n \lVert f^{(k)}(\boldsymbol{x}_i,\theta_k) - f^{(k-1)}(\boldsymbol{x}_i,\theta_{k-1})\lVert_2^2

§.§ Additional experiments on CIFAR-10/100

(a) $\alpha = 0.2, \tau = 20.0$     (b) $\alpha = 0.5, \tau = 20.0$     (c) $\alpha = 0.8, \tau = 20.0$

Tracking trace and top eigenvalue in distillation steps on VGG16 for CIFAR100. All models use augmentation.

(a) $\alpha = 0.2, \tau = 4.0$     (b) $\alpha = 0.5, \tau = 4.0$     (c) $\alpha = 0.8, \tau = 4.0$

Tracking trace and top eigenvalue in distillation steps on VGG16 for CIFAR10. All models use augmentation.

[1]
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai.
Variational information distillation for knowledge transfer.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), June 2019.

[2]
Zeyuan Allen-Zhu and Yuanzhi Li.
Towards understanding ensemble, knowledge distillation and
self-distillation in deep learning.
CoRR, abs/2012.09816, 2020.

[3]
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.
wav2vec 2.0: A framework for self-supervised learning of speech
CoRR, abs/2006.11477, 2020.

[4]
Yoshua Bengio and Yann LeCun.
Scaling learning algorithms towards AI.
In Large Scale Kernel Machines. MIT Press, 2007.

[5]
Kenneth Borup and Lars N. Andersen.
Even your teacher needs guidance: Ground-truth targets dampen
regularization imposed by self-distillation, 2021.

[6]
Léon Bottou.
Large-scale machine learning with stochastic gradient descent.
In Proceedings of COMPSTAT'2010, pages 177–186. Springer,

[7]
Cristian Buciluundefined, Rich Caruana, and Alexandru Niculescu-Mizil.
Model compression.
In Proceedings of the 12th ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining, KDD '06, page 535–541, New York,
NY, USA, 2006. Association for Computing Machinery.

[8]
Junbum Cha, Hancheol Cho, Kyungjae Lee, Seunghyun Park, Yunsung Lee, and
Sungrae Park.
Domain generalization needs stochastic weight averaging for
robustness on domain shifts.
CoRR, abs/2102.08604, 2021.

[9]
Pratik Chaudhari, Anna Choromańska, Stefano Soatto, Yann LeCun, Carlo
Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo
Entropy-sgd: Biasing gradient descent into wide valleys.
In ICLR, 2017.

[10]
Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang.
Darkrank: Accelerating deep metric learning via cross sample
similarities transfer.
In AAAI, 2018.

[11]
Jang Hyun Cho and Bharath Hariharan.
On the efficacy of knowledge distillation, 2019.

[12]
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.
Autoaugment: Learning augmentation strategies from data.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 113–123, 2019.

[13]
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and
Quoc V. Le.
Autoaugment: Learning augmentation policies from data.
CoRR, abs/1805.09501, 2018.

[14]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.
ImageNet: A Large-Scale Hierarchical Image Database.
In CVPR09, 2009.

[15]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
BERT: pre-training of deep bidirectional transformers for language
CoRR, abs/1810.04805, 2018.

[16]
Terrance DeVries and Graham W Taylor.
Improved regularization of convolutional neural networks with cutout.
arXiv preprint arXiv:1708.04552, 2017.

[17]
Terrance Devries and Graham W. Taylor.
Improved regularization of convolutional neural networks with cutout.
CoRR, abs/1708.04552, 2017.

[18]
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio.
Sharp minima can generalize for deep nets.
In ICML, 2017.

[19]
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang.
Distillation $\approx$ early stopping? harvesting dark knowledge
utilizing anisotropic information retrieval for overparameterized neural
network, 2019.

[20]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at
CoRR, abs/2010.11929, 2020.

[21]
Gintare Karolina Dziugaite and Daniel M. Roy.
Computing nonvacuous generalization bounds for deep (stochastic)
neural networks with many more parameters than training data.
CoRR, abs/1703.11008, 2017.

[22]
Gongfan Fang, Yifan Bao, Jie Song, Xinchao Wang, Donglin Xie, Chengchao Shen,
and Mingli Song.
Mosaicking to distill: Knowledge distillation from out-of-domain
data, 2021.

[23]
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.
Sharpness-aware minimization for efficiently improving
CoRR, abs/2010.01412, 2020.

[24]
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, and
Anima Anandkumar.
Born again neural networks, 2018.

[25]
Yaroslav Ganin and Victor Lempitsky.
Unsupervised domain adaptation by backpropagation, 2015.

[26]
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.
Deep learning, volume 1.
MIT Press, 2016.

[27]
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao.
Knowledge distillation: A survey.
International Journal of Computer Vision, 129(6):1789–1819,
Mar 2021.

[28]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition, 2015.

[29]
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan
Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob
Steinhardt, and Justin Gilmer.
The many faces of robustness: A critical analysis of
out-of-distribution generalization.
ICCV, 2021.

[30]
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.
Natural adversarial examples, 2021.

[31]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
Distilling the knowledge in a neural network.
arXiv preprint arXiv:1503.02531, 2015.

[32]
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh.
A fast learning algorithm for deep belief nets.
Neural Computation, 18:1527–1554, 2006.

[33]
Sepp Hochreiter and Jürgen Schmidhuber.
Flat minima.
Neural computation, 9(1):1–42, 1997.

[34]
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy
Fantastic generalization measures and where to find them.
CoRR, abs/1912.02178, 2019.

[35]
SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu.
DE-RRD: A knowledge distillation framework for recommender
CoRR, abs/2012.04357, 2020.

[36]
SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu.
Topology distillation for recommender system.
In Proceedings of the 27th ACM SIGKDD Conference on Knowledge
Discovery and Data Mining, KDD '21, page 829–839, New York, NY, USA, 2021.
Association for Computing Machinery.

[37]
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy,
and Ping Tak Peter Tang.
On large-batch training for deep learning: Generalization gap and
sharp minima.
CoRR, abs/1609.04836, 2016.

[38]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.
Imagenet classification with deep convolutional neural networks.
Commun. ACM, 60(6):84–90, may 2017.

[39]
Yann LeCun and Corinna Cortes.
MNIST handwritten digit database.
http://yann.lecun.com/exdb/mnist/, 2010.

[40]
Hao Li, Zheng Xu, Gavin Taylor, and Tom Goldstein.
Visualizing the loss landscape of neural nets.
CoRR, abs/1712.09913, 2017.

[41]
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi.
Ensemble distillation for robust model fusion in federated learning.
CoRR, abs/2006.07242, 2020.

[42]
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott E. Reed,
Cheng-Yang Fu, and Alexander C. Berg.
SSD: single shot multibox detector.
CoRR, abs/1512.02325, 2015.

[43]
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and
Vladimir Naumovich Vapnik.
Unifying distillation and privileged information.
CoRR, abs/1511.03643, 2016.

[44]
Ilya Loshchilov and Frank Hutter.
SGDR: stochastic gradient descent with restarts.
CoRR, abs/1608.03983, 2016.

[45]
Asit K. Mishra and Debbie Marr.
Apprentice: Using knowledge distillation techniques to improve
low-precision network accuracy.
CoRR, abs/1711.05852, 2017.

[46]
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett.
Self-distillation amplifies regularization in hilbert space, 2020.

[47]
Alexander Mordvintsev.
Inceptionism: Going deeper into neural networks, Jun 2015.

[48]
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.
Relational knowledge distillation.
CoRR, abs/1904.05068, 2019.

[49]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary
DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.
Automatic differentiation in pytorch, 2017.

[50]
Fabrizio Pittorino, Carlo Lucibello, Christoph Feinauer, Enrico M. Malatesta,
Gabriele Perugini, Carlo Baldassi, Matteo Negri, Elizaveta Demyanenko, and
Riccardo Zecchina.
Entropic gradient descent algorithms and wide flat minima.
CoRR, abs/2006.07897, 2020.

[51]
Antonio Polino, Razvan Pascanu, and Dan Alistarh.
Model compression via distillation and quantization.
CoRR, abs/1802.05668, 2018.

[52]
Can Qin, Handong Zhao, Lichen Wang, Huan Wang, Yulun Zhang, and Yun Fu.
Slow learning and fast inference: Efficient graph similarity
computation via knowledge distillation.
In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman
Vaughan, editors, Advances in Neural Information Processing Systems,
volume 34, pages 14110–14121. Curran Associates, Inc., 2021.

[53]
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo
Gatta, and Y. Bengio.
Fitnets: Hints for thin deep nets.
arXiv, 12 2014.

[54]
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition.
CoRR, abs/1409.1556, 2015.

[55]
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and
Andrew Gordon Wilson.
Does knowledge distillation really work?
CoRR, abs/2106.05945, 2021.

[56]
Yonglong Tian, Dilip Krishnan, and Phillip Isola.
Contrastive representation distillation, 2020.

[57]
Vladimir Vapnik and Rauf Izmailov.
Learning using privileged information: Similarity control and
knowledge transfer.
J. Mach. Learn. Res., 16(1):2023–2049, jan 2015.

[58]
Zheng Xu, Yen-Chang Hsu, and Jiawei Huang.
Learning loss for knowledge distillation with conditional adversarial
CoRR, abs/1709.00513, 2017.

[59]
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan Yuille.
Knowledge distillation in generations: More tolerant teachers educate
better students, 2018.

[60]
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W. Mahoney.
Pyhessian: Neural networks through the lens of the hessian.
CoRR, abs/1912.07145, 2019.

[61]
Hongxu Yin, Pavlo Molchanov, Zhizhong Li, Jose M. Alvarez, Arun Mallya, Derek
Hoiem, Niraj K. Jha, and Jan Kautz.
Dreaming to distill: Data-free knowledge transfer via deepinversion.
CoRR, abs/1912.08795, 2019.

[62]
Sergey Zagoruyko and Nikos Komodakis.
Paying more attention to attention: Improving the performance of
convolutional neural networks via attention transfer.
CoRR, abs/1612.03928, 2016.

[63]
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng
Be your own teacher: Improve the performance of convolutional neural
networks via self distillation.
CoRR, abs/1905.08094, 2019.

[64]
Zhilu Zhang and Mert R. Sabuncu.
Self-distillation as instance-specific label smoothing.
CoRR, abs/2006.05065, 2020.

[65]
Yanning Zhou, Hao Chen, Huangjing Lin, and Pheng-Ann Heng.
Deep semi-supervised knowledge distillation for overlapping cervical
cell instance segmentation.
CoRR, abs/2007.10787, 2020.