# Dual gauge concepts in electrodynamics, and new limitations on gauge
invariance

H. R. Reiss  Max Born Institute, Berlin, Germany  American University,
Washington, DC, USA<EMAIL_ADDRESS>
([)

###### Abstract

Gauge invariance, a core principle in electrodynamics, has two separate
meanings. One concept treats the photon as the gauge particle for
electrodynamics. It is based on symmetries of the Lagrangian, and requires no
mention of electric or magnetic fields. The second concept depends directly on
the electric and magnetic fields, and how they can be represented by potential
functions that are not unique. A general proof that potentials are more
fundamental than fields serves to resolve discrepancies. Physical symmetries,
however, are altered by gauge transformations and strongly limit gauge
freedom. A new constraint on the form of allowable gauge transformations must
be introduced that applies to both gauge concepts.

††preprint:

6 February 2022]

##  I Introduction

From the time of Maxwell it was noted that electric $\mathbf{E}$ and magnetic
$\mathbf{B}$ fields could be represented as derivatives of scalar and 3-vector
potentials $\phi$ and $\mathbf{A}$ (that can be combined into a 4-vector
potential $A^{\mu}$). The association is not, in general, unique, and each
such representation is called a gauge. Another gauge concept arose from the
work of Yang and Mills yangmills  on the properties of isospin in strong
nuclear interactions, a precursor to a unified theory of gauge fields that
encompasses strong and weak nuclear interactions as well as electrodynamics.
We will be concerned here only with the electrodynamics component of gauge
fields, where the photon is the gauge particle carrying the interaction. Thus
there are two distinct gauge concepts in electrodynamics, often improperly
conflated: the first will be referred to as Maxwellian, and the second as
Lagrangian, since it depends on symmetries of the Lagrangian density
$\mathcal{L}_{EM}$ of the electromagnetic field.

Both disciplines employ the same concept of a gauge transformation of the
4-vector potential $A^{\mu}$, given by the rotation

$A^{\mu}\longrightarrow\widetilde{A}^{\mu}=A^{\mu}-\partial^{\mu}\Lambda,$ (1)

where $\Lambda$ is any scalar function that satisfies the homogeneous wave
equation, and the Lorenz condition applies to both $A^{\mu}$ and
$\widetilde{A}^{\mu}$:

$\partial^{\mu}\partial_{\mu}\Lambda=0,\qquad\partial^{\mu}A_{\mu}=\partial^{\mu}\widetilde{A}_{\mu}=0.$
(2)

These conditions are such that electric and magnetic fields are unchanged by
the transformation. Gauge invariance refers to the requirement that a change
of gauge should not alter the outcome of calculations of electrodynamic
interactions.

The conventional point of view is that preservation of electric and magnetic
fields is sufficient to identify gauge invariance. However, with the knowledge
that potentials are the determining factor in gauge transformations, the new
governing principle is that physical symmetries must be preserved in a gauge
transformation. This is much more demanding than the traditional concept. An
important alteration in the original notion of gauge invariance is the
symmetry required to sustain the propagation property of plane waves. It is
shown that dependence on electric and magnetic fields is not sufficient to
sustain propagation, and the recasting to a potentials requirement strongly
constrains the possible class of equivalent gauges. The presence of a Coulomb
binding potential in addition to plane waves leaves no freedom at all in the
selection of gauge.

Gauge invariance is regarded as a bedrock principle in electrodynamics.
Maxwellian gauge concepts can be traced back to Maxwell’s equations of 1865
maxwell , and the Lagrangian form to the Yang-Mills paper of 1954 yangmills .
Despite these venerable origins, the fact that two inequivalent forms of gauge
invariance exist seems never to have been studied critically.

The following Section describes briefly the Lagrangian formulation, which has
remained free of controversy, unlike the Maxwellian formalism treated in much
more detail in Section III. However, a new limitation on gauge transformations
must be added to both gauge concepts to preserve the propagation property of
plane waves.

Difficulties with the meaning and application of the Maxwellian formalism
continue to the present day.

A proof, of much broader scope than that of Aharonov-Bohm, demonstrates that
potentials provide more information than electric and magnetic fields, and
that the fields can give erroneous predictions. When the Maxwellian field-
based discipline is altered to be in terms of potentials, then there is no
conflict with Lagrangian gauge transformations for plane wave fields. The
Lagrangian formalism examines only the photon 0field, equivalent to plane-wave
fields, whereas the Maxwellian formalism is not limited in this way, and thus
has a much broader scope.

Although gauge equivalence remains valid, it is shown that the concept of
gauge freedom is significantly restricted due to changes in physical
symmetries that follow from gauge transformations.

The status of potentials as the primary quantities in electrodynamics provides
insights as to why the tunneling model for atomic ionization is useful only
for some ranges of parameters, and why an analogous situation exists for the
concept of the local constant field approximation (LCFA).

Electromagnetic quantities are expressed in Gaussian units.

##  II Lagrangian Formalism

The contrast between the Maxwellian and Lagrangian approaches becomes evident
when the electromagnetic Lagrangian density is displayed:

$\mathcal{L}_{EM}=-F^{\mu\nu}F_{\mu\nu},\qquad
F^{\mu\nu}=\partial^{\mu}A^{\nu}-\partial^{\nu}A^{\mu}.$ (3)

$\mathcal{L}_{EM}$ is always exactly preserved in the gauge transformation
(1), which explains why the Lagrangian formalism is free of controversy.

Another essential point is that the Lagrangian formalism depends entirely on
the 4-vector potential $A^{\mu}$, so it can be explored without ever
mentioning $\mathbf{E}$ or $\mathbf{B}$ fields. This is in contrast to the
Maxwellian formalism, where the core consideration is how $\mathbf{E}$ and
$\mathbf{B}$ fields are related to potentials.

A unfortunate consequence of ignoring this disparity in gauge formulations is
exemplified by a recent pair of papers where an examination of Maxwellian
properties davis1  is deemed invalid on the grounds of the great success of
the Lagrangian formalism hees .

A previously unexplored limitation must be added to the standard requirements
for valid gauge transformations to preserve the propagation property of plane
waves. This is explained in Section IV.

##  III Maxwellian Formalism

Problems with the Maxwellian formalism can be divided into two categories. One
is simply an improper application of the formalism. The second category of
problems refers to fundamental failures to satisfy basic principles of physics
when fields are regarded as more fundamental than potentials.

We start with a quick review of fundamentals.

###  III.1 Basic Premises

From early stages of the study of electromagnetism in vacuum, the electric
field $\mathbf{E}$ and the magnetic field $\mathbf{B}$ were viewed as basic
measures. The connection of $\mathbf{E}$ and $\mathbf{B}$ to scalar and vector
potentials $\phi$ and $\mathbf{A}$ is not unique, so preservation of
$\mathbf{E}$ and $\mathbf{B}$ in gauge transformations was regarded as
fundamental. This was not called into question until the Aharonov-Bohm process
ehrsid ; ab ; berry  was noted (to be discussed below), but some authorities
jackokun  still maintain that the fields $\mathbf{E}$ and $\mathbf{B}$ remain
fundamental.

###  III.2 Improper Inferences

The often-controversial aspects of the Maxwellian formalism to be discussed
below do not occur at all in the Lagrangian formalism.

####  III.2.1 Length gauge

The improprieties to be reviewed are to be found principally in the Atomic,
Molecular, and Optical (AMO) physics community. The Appendix recounts a brief
history of the predilection of the AMO community to favor the length gauge,
with some saying that it is the only “correct” gauge.

####  III.2.2 Dipole approximation vs. long-wavelength approximation

The dipole approximation (DA) is taken in the AMO community to mean a complete
neglect of the magnetic component of a plane wave (PW). It is sometimes
confused with the long-wavelength approximation (LWA), which simply implies
that the wavelength of a PW is much larger than the extent of the atom. The
LWA does not mean that the magnetic field is neglected. The distinction is
important.

The DA is a much stronger constraint than the LWA because the DA sets
$\mathbf{B}=0$ from the outset. If $\mathbf{A}\left(t,\mathbf{r}\right)$ is
used in its full spacetime form to find
$\mathbf{B}\left(t,\mathbf{r}\right)=\mathbf{\nabla\times
A}\left(t,\mathbf{r}\right)$, and then the LWA is applied to $\mathbf{B}$, a
$\mathbf{B}\left(t\right)$ is found that becomes increasingly important as
intensity increases or as frequency decreases, with $\mathbf{B}$ becoming
significant with existing laboratory parameters hr101 ; hrtun . That is, the
two actions of evaluating $\mathbf{\nabla\times A}$ and applying the LWA are
not commutative.

If the DA is employed in its usual form with both $\mathbf{B}=0$ and the LWA,
then $\omega\rightarrow 0$ gives only a static electric field. On the other
hand, if the LWA is applied after
$\mathbf{B}\left(t,\mathbf{r}\right)=\mathbf{\nabla\times
A}\left(t,\mathbf{r}\right)$ is evaluated, the result is a low-frequency PW
that propagates at the speed of light. The outcome of the two procedures: a
static electric field vs. a field that propagates at the speed of light, could
hardly be more different.

####  III.2.3 Oscillatory electric vs. plane wave

An important error arises when the condition $\lambda\gg 1$ a.u. is viewed as
the only constraint on the DA, where $\lambda$ is the wavelength of the PW
field. This approach makes it natural to suppose that very long wavelengths
$\lambda$ (i.e. low frequencies $\omega$) are far removed from any possibility
of DA failures, and low frequency situations are then assumed to be governed
by what is called “adiabaticity”, approaching constant electric-field behavior
as $\omega\rightarrow 0$. An unfortunate effect was that analytical
approximations for strong-field processes were judged deficient unless they
approached constant-field results in the low frequency limit. See, for
example, Ref. joachain .

The actuality is that oscillatory electric-field effects lose all resemblance
to plane-wave field effects as $\omega\rightarrow 0$ hr101 ; hrtun ,
contradicting the adiabaticity argument. A plane-wave field propagates at the
speed of light for any frequency, no matter how low, with $\mathbf{E}$ and
$\mathbf{B}$ fields of equal magnitude. By contrast, the DA version infers an
oscillatory electric field that approaches a static limit with no magnetic
field at all.

###  III.3 Equations of motion

A paradox with the Maxwellian formalism was examined in the 1960s. All
quantum-mechanical equations of motion for particles in interaction with
electromagnetic fields have that interaction represented by potentials. The
prevailing view was that electric and magnetic fields are fundamental, but all
attempts mandelstam ; dewitt ; belinfante ; levy ; rohrlich ; priou  to
express the Schrödinger equation directly in terms of electric and magnetic
fields led to nonlocal theories. In retrospect this was an inevitable result.
Although fields are found from potentials by differentiation, which is a local
procedure, potentials are found from fields by integration, which is nonlocal.
Thus, any attempt to replace the potentials in quantum equations of motion by
fields will inevitably introduce nonlocality. It applies as well as for all
classical equations of motion except for the Newtonian form.

##  IV Potentials are fundamental; fields are secondary

The fact that equations of motion are expressed in terms of potentials, which
therefore govern the properties of the solutions of those equations, is a
strong indicator of the primacy of potentials. There are, however, even more
direct indicators: the Aharonov-Bohm effect and the properties of PWs
propagating in the vacuum.

###  IV.1 Aharonov-Bohm effect

A major development in the matter of potentials vs. fields is the 1959 paper
of Aharonov and Bohm (AB) ehrsid ; ab ; berry . AB considered the quantum-
mechanical experiment in which a beam of electrons is deflected as it passes
over a closed solenoid. There is a magnetic field within the solenoid, but no
field outside. There is, however, a potential outside, so it must be the
potential that gives rise to the deflection, meaning that the potential is
more fundamental than the field. Experimental verification of the deflection
was achieved in 1986 tonomura .

Many investigators accept that the AB effect is a universal proof that
potentials are more fundamental than fields, but others regard it as a
quantum-only effect, or that the effect is too particularized to be taken as a
general proof of the primacy of potentials over fields. Jackson and Okun
jackokun  maintain that it is possible to salvage the long-standing conviction
that fields are physical. Part of their argument is that $\mathbf{E}$ and
$\mathbf{B}$ can be measured in the laboratory, but $A^{\mu}$ cannot. However,
the ponderomotive potential $U_{p}$ is proportional to $A^{\mu}A_{\mu}$, and
$U_{p}$ is the fundamental measure of the nonperturbative interaction of a
charged particle with a plane-wave field hr62 ; hrmass ; hr100 . The
ponderomotive potential is observable as the cause of channel closings hr80 ;
muller ; hrepjd .

###  IV.2 Plane waves

The study of the properties of PWs (also known as transverse fields,
propagating fields, sourceless fields, laser fields, or simply “light”) is
fundamental in classical electrodynamics, quantum electrodynamics, and
astrophysics. To satisfy the Einstein basic principle of relativity that the
speed of light is the same in all inertial frames of reference einstein , the
$A^{\mu}$ that describes the PW can depend on the spacetime 4-vector $x^{\mu}$
only as the scalar product with the propagation 4-vector $k^{u}$:

$A^{\mu}\left(\varphi\right);\qquad\varphi\equiv k^{\nu}x_{\nu}=\omega
t-\mathbf{k\cdot r}$ (4)

It is possible to introduce a gauge transformation from a set of potentials
meeting the Einstein condition to a new set of potentials that violate it.
Starting with a 4-vector potential of the form given in Eq. (4), a gauge
transformation can be generated hr79 ; hr100  by the function

$\Lambda=A^{\mu}x_{\mu}.$ (5)

Equation (1) takes the form

$\widetilde{A}^{\mu}=-k^{\mu}x^{\nu}(d_{\varphi}A_{\nu})\text{ \ \ or \ \
}\widetilde{A}^{\mu}=-\frac{k^{\mu}}{\omega/c.}\mathbf{r\cdot
E}\left(\varphi\right),$ (6)

where the second expression in Eq. (6) is the result if the initial gauge
before transformation is the Coulomb gauge. Neither of the forms in Eq. (6)
can be descriptive of a propagating field. The occurrence of $x^{\nu}$ in
isolation from $k_{\nu}$ in the first expression and the appearance of
$\mathbf{r}$ in the second one, requires that an origin of coordinates must be
specified. This is a clear violation of the Einstein principle. Nevertheless,
the gauge transformation generated by Eq. (5) satisfies the Lorenz condition
and the homogeneous wave equation. It is a valid gauge transformation, meaning
that the $\mathbf{E}$ and $\mathbf{B}$ fields in the transformed gauge are
identical to those in the initial gauge.

The 4-potential $A^{\mu}\left(\varphi\right)$ is an appropriate form for the
description of a propagating electromagnetic field, but the potentials in Eq.
(6) cannot describe a propagating field. A knowledge of the potentials is
necessary to distinguish the physical from the unphysical situation; the
$\mathbf{E}$ and $\mathbf{B}$ fields are unable to make the distinction.

###  IV.3 Consequences of potentials over fields

Among the consequences that flow from the primacy of potentials, two will be
mentioned here because they are important for currently active research.

####  IV.3.1 Effects of plane waves vs. effects of oscillatory electric fields

When the effects of plane waves are treated with the DA, this is tantamount to
replacing a propagating field by one that is simply an oscillatory electric
field. That is, a wave that propagates with the speed of light is replaced by
a simple oscillatory electric field that has zero speed of propagation.

Both the DA and the LWA, as used in AMO physics, are subject to an upper limit
on frequency equivalent to the wavelength limit $\lambda\gg 1a.u.$ As the
frequency declines, effects predicted by the DA are close to those of the LWA,
but that correspondence vanishes when the coupling of the electric and
magnetic components of a PW field induces an oscillation of amplitude
$\beta_{0}$ ss  in the propagation direction approaching $1a.u.$, where

$\beta_{0}=\frac{U_{p}}{2mc\omega}\frac{1}{\left(1+2U_{p}/mc^{2}\right)}\approx\frac{U_{p}}{2mc\omega}.$
(7)

The quantity $U_{p}$ is the ponderomotive energy of a charged particle in the
field. As the lower limit on frequency approaches the value in Eq. (7), the
difference between DA and LWA predictions diverge. Since $U_{p}$ has the
0frequency dependence $1/\omega^{2}$, then

$\beta_{0}\backsim 1/\omega^{3}.$ (8)

As $\omega\rightarrow 0$, the DA approaches static electric field behavior,
whereas the LWA tends towards relativistic behavior. The onset of this extreme
divergence is very rapid, as Eq. (8) demonstrates.

The in-phase coupling between electric and magnetic fields in a PW produces a
force (radiation pressure) in the direction of propagation of the PW,
producing the classical figure-8 motion ss  in linearly polarized light. In
qualitative terms, calculations within the DA are in good agreement with
experiments until the point where effects of coupling between electric and
magnetic components become discernable hr101 ; hrtun . Several laboratories
now have the capability to explore non-DA behavior.

####  IV.3.2 Effects of plane waves vs. effects of constant crossed fields.

Constant crossed fields are an example showing the inability of electric and
magnetic fields to uniquely define an electromagnetic environment. One
approach to the significance of constant crossed field comes from an
examination of the Lorentz invariants of electromagnetic fields. Two Lorentz
invariants can be formed from the electromagnetic field tensor of Eq. (15):

$F^{\mu\nu}F_{\mu\nu}\thicksim\mathbf{E}^{2}-\mathbf{B}^{2},\qquad
F^{\mu\nu}\mathcal{F}_{\mu\nu}\thicksim\mathbf{E\cdot B,}$ (9)

where $\mathcal{F}_{\mu\nu}$ is the dual to the field tensor. When both of
these invariants have the value zero the field is called a null field. The
most important example is a PW field, where

$\left|\mathbf{E}\right|=\left|\mathbf{B}\right|,\qquad\mathbf{E\perp B}.$
(10)

The propagating field requires only the condition of Eq. (4), but it is also
possible to produce constant $\mathbf{E}$ and $\mathbf{B}$ fields that satisfy
the null field conditions of Eq. (10). Many authors have concluded that the
constant-crossed-field configuration provides a simple way to assess the
effects of a propagating field. This was initially proposed in Refs. nikrit ;
ritus , and has been explored by many authors as the LCFA.

Another line of reasoning is to regard a low-frequency laser field as having
so long a wavelength as compared to an object of atomic size that the
$\mathbf{E}$ and $\mathbf{B}$ fields approximate static conditions in the
interaction region. A good way to appraise this conjecture is to compare the
effects of a constant crossed field with those of a propagating field of
extremely long wavelength. This can be done in the context of a practical
example employed in Ref. hrnotun : an extremely-low-frequency (ELF) radio
transmitter employed by the US Navy sanguine  to communicate with deeply
submerged submarines at distances of thousands of kilometers from the
transmitting antenna. The system operated at 76 Hz, corresponding to a
wavelength of about 4,000 km. This is about 1/3 the diameter of the Earth. On
a laboratory scale, such a radio wave would appear to have $\mathbf{E}$ and
$\mathbf{B}$ values constant to very high accuracy. As has been shown above,
it is not possible to describe static $\mathbf{E}$ and $\mathbf{B}$ components
in terms of the propagating field potential of the form of Eq. (4).
Appropriate potentials for static $\mathbf{E}_{0}$ and $\mathbf{B}_{0}$ fields
are

$\phi=-\mathbf{r\cdot E}_{0},\qquad\mathbf{A}=-\frac{1}{2}\mathbf{r\times
B}_{0}.$ (11)

There is no gauge transformation that can relate the potentials obeying Eq.
(11) to those satisfying Eq. (4). The primacy of potentials over fields means
that the potentials of Eq. 11 represent an entirely different physical
environment from potentials in the form of Eq. (4).

When a solution of the Schrödinger equation is sought, the potentials
determine the outcome, so the potentials of Eq. (11) do not exhibit the
properties of PWs. This failure is not evident in terms of electric and
magnetic fields. It is only the potentials that determine a correct result.

As in the case of the near equivalence of the DA to true plane-wave effects in
a limited range of parameters, it has been found that at high intensities the
LCFA mimics PW behavior satisfactorily dipiazza1 ; dipiazza2 .

##  V Constraints on gauge freedom

###  V.1 Plane waves

The usual restrictions in a gauge transformation are that the generating
function should be a scalar function that satisfies the homogeneous wave
equation, and that the initial and transformed 4-vectors $A^{\mu}$ and
$\widetilde{A}^{\mu}$ should obey the Lorenz condition, as shown in Eq. (2).
These are not sufficient to preserve all necessary symmetries.

For plane waves, the property stated in Eq. (4) that both $A^{\mu}$ and
$\widetilde{A}^{\mu}$ are restricted to have spacetime dependence only in
combination with the propagation 4-vector $k^{\mu}$ as the product
$\varphi=k^{\mu}x_{\mu}$, means that $\partial^{\mu}\Lambda$ must also have
this property. This gives

$\partial^{\mu}\Lambda\left(\varphi\right)=\partial^{\mu}\left(\varphi\right)\frac{d}{d\varphi}\Lambda\left(\varphi\right)=k^{\mu}\Lambda^{\prime}\left(\varphi\right).$
(12)

That is, any change in the 4-vector potential resulting from a gauge
transformation has to lie on the lightcone.

This is a very strong constraint that applies both in classical and quantum
electrodynamics. It applies to plane waves, which are at the heart of
electrodynamics, yet it is not mentioned in standard texts on electrodynamics.

###  V.2 Plane waves + Coulomb potential

A common laboratory situation occurs when an electron subjected to a Coulomb
binding force is irradiated with a laser beam. The strong constraint (12)
imposed for the description of the laser beam becomes even stronger when the
binding potential must be included. The Dirac and Klein-Gordon equations must
now incorporate both interactions. This is done by adding the scalar potential
$V\backsim 1/r$ to the time component of the plane wave potential:

$A^{0}=A_{PW}^{0}+V.$ (13)

The Klein-Gordon equation, with time and space parts separated is

$\left[\left(\frac{i\hslash}{c}\partial_{t}-\frac{q}{c}A^{0}\right)^{2}-\left(ii\hslash\mathbf{\nabla}-\frac{q}{c}\mathbf{A}\right)^{2}-\left(mc\right)^{2}\right]\psi=0.$
(14)

When Eq. (13) is inserted into the first parenthesis in Eq. (14), and the
indicated square is performed, there will be a cross term $A_{PW}^{0}V$ that
causes a problem in the nonrelativistic reduction of the Klein-Gordon equation
to the Schrödinger equation schoene; schiff due to the singularity of the
Coulomb potential.

###  V.3 Beyond plane waves

The Lagrangian formalism is in terms of transverse fields, but the scope of
the Maxwellian is much broader. Transforming gauge means that there will be a
change in the functional dependence of the potentials. A change in functional
form of the potentials means that conservation principles will be altered. In
the case of plane waves it was shown that the condition in Eq. (12) must be
satisfied in order to maintain the propagation property of plane waves. The
general effect of changing the form of the potentials can be seen even in the
elementary problem of an electron in a constant electric field hr100 . That
case, from Ref. hr100  is replicated in the Appendix to illustrate the problem
in simple terms.

The overall conclusion is that gauge transformations do not automatically
preserve the outcome of a physical calculation. In general, additional
constraints must be satisfied, and these extra requirements may be very
restrictive.

##  VI Overview

The “traditional” concept of gauge invariance was based on the belief that
electric and magnetic fields are the physical determinants of an
electrodynamic environment. Gauge invariance was based on preservation of the
fields. A basic flaw in this argument is that the Schrödinger equation and
other equations of motion are based on potentials, so that a change of gauge
will alter the solution of an electrodynamic problem. An example is shown here
where a valid gauge transformation involving plane waves is such that the
propagation property of a set of potentials is lost as a result of the gauge
transformation. The solution of the transformed equation of motion is
unphysical even though the gauge transformation preserves the electric and
magnetic fields. The resolution of this apparent dilemma is that the
potentials determine the solution, not the fields.

With this alteration, there is then no conflict with gauge invariance based on
electromagnetism as the gauge field within the Standard Model, where only the
potentials are involved and electric and magnetic fields need not be
mentioned.

The example used to arrive at this conclusion is Eq. (5), and this gauge
transformation introduces a violation of the condition that plane waves
propagate with the speed of light in any Lorentz frame. An additional
condition must be added to the customary conditions governing gauge
transformations. This new constraint requires that all gauge transformations
involving plane waves must lie on the lightcone. That is, they must have the
same spacetime dependence as the propagation 4-vector $k^{\mu}$.

##  VII Appendix: Special status of the length gauge

Controversy about the Maxwellian formalism began with a 1952 paper by Lamb
lamb  and continues to the present davis1 ; hees ; heras ; davis2 , 70 years
later.

In Atomic, Molecular, and Optical (AMO) physics, two gauges are conventionally
employed: the “length gauge”, with interaction Hamiltonian

$H_{I}=-q\mathbf{r\cdot E}\left(t\right),$ (15)

and the “velocity gauge” with

$H_{I}=-\frac{q}{mc}\mathbf{A}\left(t\right)\mathbf{\cdot
p+}\frac{q^{2}}{2mc^{2}}\mathbf{A}^{2}\left(t\right).$ (16)

Both expressions employ the dipole approximation (DA), routinely adopted in
the AMO community, and defined to mean omitting spatial dependence in
$\mathbf{E}$ and $\mathbf{A}$. This leads to $\mathbf{B}=0,$ thus losing any
possibility of treating the propagation property of plane-wave fields that is
vital in the exploration of laser-induced phenomena. Plane-wave fields in the
vacuum have equal magnitudes of electric and magnetic fields:

$\left|\mathbf{E}^{PW}\right|=\left|\mathbf{B}^{PW}\right|,$ (17)

so setting $\mathbf{B}^{PW}=0$ is questionable in view of the Maxwellian
demand that $\mathbf{E}$ and $\mathbf{B}$ are both fundamental. Coupling of
the electric and magnetic components of a PW becomes vitally important when
field intensity is high and/or when field frequency is low hr101 ; hrtun .

Equation (15) can be descriptive of an oscillatory electric field, but it is a
scalar potential, which is fundamentally inadequate for the representation of
a vector field such as that of a laser hrepjd55 .

In 1952, Lamb lamb  did a second-order perturbative line-shape calculation and
found a plausible result in the length gauge, but an unacceptable asymmetry in
the velocity gauge. Lamb viewed this failure of gauge invariance as
perplexing, but Power and Zienau power  asserted that if length and velocity
gauges give different answers, the length gauge is correct. They leave
unresolved the dilemma about how gauge-equivalent expressions can lead to
different physical predictions. Fried demonstrated fried  in 1973 that Lamb
had erred in his velocity-gauge line shape calculation and, when properly
treated, the gauge conflict is removed; but this resolution of the difficulty
was not acknowledged in the AMO community. K. H. Yang yang  went further than
Power and Zienau by claiming to prove that the length gauge is a fundamental
gauge with which all calculations must begin, followed by a gauge
transformation if any other gauge is to be employed. This viewpoint gained
much support kobesmirl ; kobeyang ; schlicherbeck ; lambschlicher ; heidelberg
; bergues1 ; bergues2  even for problems with strong laser fields, despite the
fact that all plane-wave fields like those of a laser beam are vector fields.
Vector fields in interaction with matter cannot properly be represented by a
scalar potential.

A questionable belief widespread among defenders of the primacy of the
interaction Hamiltonian of Eq. (15) is the assumption that a gauge
transformation is a unitary transformation. If this was true then

$U\mathbf{r\cdot E}U^{-1}=\mathbf{r\cdot E},$ (18)

since there are no differential operators in $\mathbf{r\cdot E}$. This leads
to the concept of a “gauge-invariant gauge” or a “gauge-invariant formulation”
heidelberg . Apart from the transparent illogic (e.g. if one can transform
into a gauge-invariant gauge, then the inverse gauge transformation could not
exist), there is the fact that a gauge transformation is not, in general,
unitary hr100 . Starting from the well-known form-invariance of the
Schrödinger equation, it follows that the gauge-transformed Hamiltonian
$\widetilde{H}$ is hr100

$\widetilde{H}=UHU^{-1}-i\hslash\left(\partial_{t}U^{-1}\right),$ (19)

meaning that any time-dependence in the generating function for the gauge
transformation precludes unitarity. This holds true even if $\mathbf{E}$ is
time-independent.

For example, consider the elementary problem where a charged particle is
subjected to a constant electric field $\mathbf{E}_{0}$ hr100 . The field can
be specified by the potentials.

$\phi=-\mathbf{r\cdot E}_{0},\qquad\mathbf{A}=0.$ (20)

The problem is time-independent, meaning that energy is conserved. A gauge
transformation to the gauge where

$\widetilde{\phi}=0,\qquad\widetilde{\mathbf{A}}=-ct\mathbf{E}_{0}$ (21)

is accomplished with the generating function

$\Lambda=-ct\mathbf{r\cdot E}_{0}.$ (22)

The potentials are now independent of spatial coordinates, so that momentum is
conserved. Two things are manifest in this example: conservation principles
are altered, and time dependence of $U$ occurs even in an initially time-
independent problem.

This constant electric field problem is so elementary that if a particle
trajectory is sought, the same classical trajectory is found from either Eq.
(20) or Eq. (21), albeit with different procedures. That is because the
problem is such that both energy and momentum are conserved. In other
contexts, like the plane wave case, new constraints must be observed in a
change of gauge.

## References

  * (1) C. N. Yang and R. L. Mills, Conservation of isotopic spin and isospin gauge invariance, Phys. Rev. 96, 191 (1954).
  * (2) J. C. Maxwell, A dynamical theory of the electromagnetic field, Phil. Trans. Roy. Soc. London, 155, 459 (1965).
  * (3) A. Davis, Defining the electromagnetic potentials, Eur. J. Phys. 41, 045202 (2020).
  * (4) H. van Hees, Comment on ‘Defining the electromagnetic potentials’, Eur. J. Phys. 42, 028003 (2021).
  * (5) W. Ehrenberg and R. E. Siday, The refractive index in electron optics and the principles of dynamics, Proc. R. Soc. B 62, 8 (1949).
  * (6) Y. Aharonov and D. Bohm, Significance of potentials in the quantum theory, Phys. Rev. 115, 485 (1959).
  * (7) M. Berry, Aptly named Aharonov-Bohm effect has classical analog, long history, Physics Today, August 2010, p.8.
  * (8) J. D. Jackson and L. B. Okun, Historical roots of gauge invariance, Rev. Mod. Phys. 73, 663 (2001).
  * (9) H. R. Reiss, Limits on tunneling theories of strong-field ionization, Phys. Rev. Lett. 101, 043002 (2008); Erratum, Phys. Rev. Lett. 101, 155901(E) (2008).
  * (10) H. R. Reiss, The tunnelling model of laser-induced ionization and its failure at low frequencies, J. Phys. B 47 (2014) 204006.
  * (11) C. J. Joachain, N. J. Kylstra, and R. M. Potvliege, Atoms in Intense Laser Fields (Cambridge University Press, Cambridge, 2012).
  * (12) S. Mandelstam, Quantum mechanics without potentials, Ann. Phys. (N.Y.) 19, 1 (1962).
  * (13) B. S. DeWitt, Quantum theory without electrodynamic potentials, Phys. Rev. 125, 2189 (1962).
  * (14) F. J. Belinfante, Consequences of postulate of a complete commuting set of observables in quantum mechanics, Phys. Rev. 128, 2832 (1962).
  * (15) M. Lévy, Nonlocal quantum electrodynamics, Nucl. Phys. 57, 152 (1964).
  * (16) F. Rohrlich and F. Strocchi, Gauge independence and path independence, Phys. Rev. 139, B476 (1965).
  * (17) M. Priou, Quasi local electrodynamics, Nucl. Phys. 81, 641 (1966).
  * (18) A. Tonomura, et al., Evidence for Aharonov-Bohm effect with magnetic field completely shielded from electron wave, Phys. Rev. Lett. 56, 792 (1986).
  * (19) H. R. Reiss, Absorption of light by light, J. Math. Phys. 3, 59 (1962).
  * (20) H. R. Reiss, Mass shell of strong-field quantum electrodynamics, Phys. Rev. A 89, 022116 (2014).
  * (21) H. R. Reiss, Fundamental formulation of light-matter interactions revisited, Phys. Rev. A 100, 052105 (2019).
  * (22) ,H. R. Reiss, Effect of an intense electromagnetic field on a weakly bound system, Phys. Rev. A 22, 1786-1813 (1980).
  * (23) H. G. Muller, A. Tip, and van der Wiel, Ponderomotive force and AC Stark shift in multiphoton ionization, J. Phys. B 16, L679 (1983).
  * (24) H. R. Reiss, Special analytical properties of ultrastrong coherent fields, Eur. Phys. J. D 55, 365 (2009).
  * (25) A. Einstein, Zur Elektrodynamik bewegter Körper, Ann. Phys. 322, 891 (1905).
  * (26) H. R. Reiss, Field intensity and relativistic considerations in the choice of gauge in electrodynamics, Phys. Rev. A 19, 1140 (1979).
  * (27) E. S. Sarachik and G. T. Schappert, Classical theory of the scattering of intense laser radiation by free electrons, Phys. Rev. D 1, 2738-2753 (1970).
  * (28) A. I. Nikishov and V. I. Ritus, Ionization of systems bound by short-range forces by the field of an electromagnetic wave, Sov. Phys. JETP 23, 168 (1966) [Zh. Eksp. Teor. Fiz. 50, 255 (1966)].
  * (29) V. I. Ritus, Quantum effects of the interaction of elementary particles with an intense electromagnetic field, J. Sov. Laser Res. 6, 497-617 (1985) [Lebedeva Akad. Nauk SSSR 111, 5 (1979)].
  * (30) H. R. Reiss, No light at the end of the tunnel, Eur. Phys. J. D 75, 158 (2021).
  * (31) en.wikipedia.org/wiki/Project_Sanguine.
  * (32) A. Di Piazza, M. Tamburini, S. Meuren, and C. H. Keitel, Implementing nonlinear Compton scattering beyond the local-constant-field approximation, Phys. Rev. A 98, 012134 (2018).
  * (33) A. Di Piazza, M. Tamburini, S. Meuren, and C. H. Keitel, Improved local-constant-field approximation for strong-field QED codes, Phys. Rev. A 99, 022125 (2019).
  * (34) W. E. Lamb, Jr., Fine structure of the hydrogen atom, III, Phys. Rev. 85, 259 (1952).
  * (35) J. A. Heras, Comment on ‘Defining the electromagnetic potentials’, Eur. J. Phys. 42, 028001 (2021).
  * (36) A. M. Davis, Reply to Comments on ‘Defining the electromagnetic potentials’, Eur. J. Phys. 42, 028002 (2021).
  * (37) H. R. Reiss, Special analytical properties of ultrastrong coherent fields, Eur. Phys. J. D 55, 363 (2009).
  * (38) E. A. Power and S. Zienau, Coulomb gauge in non-relativistic quantum electrodynamics and the shape of spectral lines, Philos. Trans. R. Soc. Lond. A 251, 427 (1959).
  * (39) Z. Fried, Vector potential versus field intensity, Phys. Rev. A 8, 2835 (1973).
  * (40) K. H. Yang, Gauge transformations and quantum mechanics. 1. Gauge invariant interpretation of quantum mechanics, Ann. Phys. (N.Y.) 101, 62 (1976).
  * (41) D. H. Kobe and A. L. Smirl, Gauge invariant formulation of interaction of electromagnetic radiation and matter, Am. J. Phys. 46, 624 (1978).
  * (42) D. H. Kobe and K. H. Yang, Gauge-invariant non-relativistic limit of an electron in a time-dependent electromagnetic field, J. Phys. A 13, 3171 (1980).
  * (43) R. R. Schlicher, W. Becker, J. Bergou, and M. O. Scully, Quantum Electrodynamics and Quantum Optics, edited by A. O. Barut (Plenum, New York, 1984), p. 405.
  * (44) W. E. Lamb, R. R. Schlicher, and M. O. Scully, Matter-field interaction in atomic physics and quantum optics, Phys. Rev. A 36, 2763 (1987).
  * (45) M. Klaiber, K. Z. Hatsagortsyan, and C. H. Keitel, Gauge-invariant relativistic strong-field approximation, Phys. Rev. A 73, 053411 (2006).
  * (46) B. Bergues, Y. Ni, H. Helm, and I. Yu. Kiyan, Experimental Study of Photodetachment in a Strong Laser Field of Circular Polarization, Phys. Rev. Lett. 95, 263002 (2005).
  * (47) B. Bergues, Z. Ansari, D. Hanstorp, and I. Yu. Kiyan, Photodetachment in a strong laser field: An experimental test of Keldysh-like theories, Phys. Rev. A 75, 063415 (2007).


# Towards Redundancy-Free Sub-networks in Continual Learning

Cheng Chen  Jingkuan Song Jingkuan Song is the corresponding author. LianLi
Gao  Heng Tao Shen
University of Electronic Science and Technology of China
<EMAIL_ADDRESS>
###### Abstract

Catastrophic Forgetting (CF) is a prominent issue in continual learning.
Parameter isolation addresses this challenge by masking a sub-network for each
task to mitigate interference with old tasks. However, these sub-networks are
constructed relying on weight magnitude, which does not necessarily correspond
to the importance of weights, resulting in maintaining unimportant weights and
constructing redundant sub-networks. To overcome this limitation, inspired by
information bottleneck, which removes redundancy between adjacent network
layers, we propose Information Bottleneck Masked sub-network (IBM) to
eliminate redundancy within sub-networks. Specifically, IBM accumulates
valuable information into essential weights to construct redundancy-free sub-
networks, not only effectively mitigating CF by freezing the sub-networks but
also facilitating new tasks training through the transfer of valuable
knowledge. Additionally, IBM decomposes hidden representations to automate the
construction process and make it flexible. Extensive experiments demonstrate
that IBM consistently outperforms state-of-the-art methods. Notably, IBM
surpasses the state-of-the-art parameter isolation method with a 70% reduction
in the number of parameters within sub-networks and an 80% decrease in
training time. Codes are publicly available at
https://anonymous.4open.science/r/IBM-Net-2514.

##  1 Introduction

Currently, deep neural networks have demonstrated exceptional performance
across various tasks. Nevertheless, when exposed to new tasks, they tend to
forget previously acquired task-specific information, a phenomenon known as
Catastrophic Forgetting [26]. To address this issue, Continual Learning [31]
is proposed to retain previously acquired knowledge while acquiring new
information, evolving the networks toward human learning patterns. In recent
years, numerous effective continual learning methods have emerged. These
methods can be conceptually categorized as follows: regularization-based
methods [33, 40, 2, 20, 7], memory-based methods [5, 30, 4, 22, 6],
architecture-based methods [39, 15, 17, 25, 34].

((a))

((b))

Figure 1: The total number of parameters within sub-networks and average
training time for each task of WSN [15] and our IBM on CIFAR-100 with
Resnet-18. Significantly, our method surpasses WSN by 1.68% with a 70%
reduction in the number of parameters and an 80% decrease in training time.

While existing methods have achieved remarkable success, catastrophic
forgetting remains far from being solved. In the realm of architecture-based
methods, parameter-isolation [34, 15, 39, 17] methods, which involve
compressing or pruning network to construct a specialized sub-network for each
task to mitigate interference with old tasks, are arguably the most effective.
The fundamental criterion of these methods for selecting weights or neurons to
construct sub-networks remains roughly the same, i.e., their magnitude.
However, it has been demonstrated by MMC [24] that the magnitude of weights or
neurons does not necessarily correlate with their importance. Thus,
maintaining these unimportant weights or neurons will construct redundancy
sub-networks, resulting in an over-consumption of network capacity and a
decrease in performance.

Inspired by the success of information bottleneck (IB) theory [35, 8] to
penalize an information-theoretic measure of redundancy between adjacent
network layers, we conjecture that this ability can be utilized to reduce
redundancy within sub-networks, constructing redundancy-free sub-networks.
However, it is not trivial to effectively utilize the IB in the continual
learning context because of two critical issues. Firstly, the original
information bottleneck primarily compresses the network with a single task,
leaving the question of generalization across multiple tasks open. Thus,
finding an appropriate way to apply IB in the continual learning context is
imperative. Secondly, the IB compression process necessitates a ratio to
regulate each layer. Since lower layers of the network primarily handle visual
features while higher layers emphasize semantic information [38, 14], it is
evident that different layers should possess distinct compression ratios.
Previous methods [8, 23, 27] set these ratios in a heuristic way manually,
which is unsuitable for all tasks or datasets and impractical for deeper
networks.

In this work, we bridge the gap by proposing a novel parameter isolation
method called Information Bottleneck based sub-network Mask (IBM), a first
framework to effectively integrate information bottleneck into continual
learning and address these challenges. In a nutshell, this method formulates
an optimization object by computing inter-layer mutual information, such that
valuable information is accumulated into essential parameters while irrelevant
information is aggregated into expendable parameters. Once this optimization
process is completed, redundancy-free sub-networks are constructed by
selecting these essential parameters. Further, the key issue in continual
learning, _i.e_. catastrophic forgetting, is solved by freezing the essential
parameters within sub-networks. Besides, IBM maintains essential parameters
and re-initializes expendable parameters to facilitate knowledge transfer for
new tasks learning. Furthermore, drawing inspiration from recent studies [12,
24] that investigate feature distribution and its contribution to final
results, we investigate the impact of hidden representation on regulating the
compression process. To this end, a novel feature decomposing module is
proposed for IBM to decompose hidden representations to enable automatic and
flexible ratio settings.

We conduct extensive experiments on multiple CL benchmarks, which validate our
conjecture and demonstrate the effectiveness of IBM. On both datasets, our
method achieves state-of-the-art performance. Significantly, IBM surpasses WSN
(SOTA of parameter-isolation) by 1.68% with a 70% reduction in the number of
sub-networks parameters and an 80% decrease in training time, with ResNet-18
on CIFAR-100, as shown in Fig. 1. In summary, the main contributions can be
summarized as follows:

  * •

To the best of our knowledge, our work is the first to bring IB into continual
learning (CL) context to reduce redundancy within sub-networks.

  * •

We propose a method IBM to mitigate catastrophic forgetting and facilitate
knowledge transfer. Additionally, a novel feature detection module is proposed
to automate ratio settings for sub-networks construction, allowing for varying
ratios across different layers.

  * •

Extensive experiments confirm the innovative utilization of information
bottleneck to reduce redundancy and demonstrate that IBM consistently
outperforms SOTA methods while utilizing fewer parameters and training time.

Figure 2: An overview of the proposed IBM. After training on Task T, a binary
mask is constructed based on choosing a subset of variational parameters. The
mask and these parameters are copied and saved in a memory pool for inference.
Then, before training on T+1, we maintain the chosen parameters and re-
initialize the rest parameters to facilitate knowledge transfer. Finally, when
training on Task T+1, indicated by the masks learned before, the corresponding
weights of the network are frozen to solve catastrophic forgetting.

##  2 Related Work

Recently, numerous methods have been proposed to mitigate catastrophic
forgetting in the continual learning paradigm. These methods can be broadly
categorized into three groups: regularization-based, memory-based and
architecture-based methods.

Regularization-based methods [33, 40, 2, 20, 7] focus on curing a continual
learning network of its catastrophic forgetting by introducing an extra
regularization term in the loss function. _e.g_., EWC[33] penalizes the
changes of importance parameters when learning new tasks.

Memory-based methods [5, 30, 4, 22, 6] store previous samples or generate
samples for replaying while learning a new task. Some methods [5, 30, 4, 4]
use replayed samples from previous tasks to constrain the update of parameters
when learning the new task. During training on a new task of EEC [3],
reconstructed images from encoded episodes were replayed to avoid catastrophic
forgetting.

Architecture-based methods [39, 15, 17, 25, 34] design new architecture
modules to each task to prevent any possible forgetting. PNN [32] adds a
network to each task and lateral connections to the network of the previous
task while freezing previous task parameters. MNTDP [36] provides a learning
algorithm to search the modules to combine with, where these modules represent
atomic skills that can be composed to perform a certain task.

Parameter-isolation methods [34, 15, 39, 17], as a branch of architecture-
based, try to learn a sub-network for each task by pruning-based or
compression-based constraints. PackNet [25] literately prunes and re-trains
the network to construct masks, alleviating catastrophic forgetting by
freezing the masked parameter. However, its performance highly depends on the
quality of the pre-trained backbone. WSN [15]decouples the learning parameter
and the network structure into two separate learnable parameters, weights and
weight scores to selecte the weights with top-$k$ percent weight ranking
scores. SPG [17] computes an importance score for each weight relying on
gradients. The importance score of each parameter is used as a soft-mask to
constrain the gradient flow in the backward pass to prevent catastrophic
forgetting of previous knowledge. Although these methods have achieved
remarkable performance, they select weights or neurons based on magnitude,
resulting in redundant sub-networks. Our method is supported by information
bottleneck theory to penalize redundancy within sub-networks, achieving
redundancy-free sub-networks.

##  3 Methodology

To construct redundancy-free sub-networks, we propose Information Bottleneck
Masked sub-network (IBM), a first framework reducing the redundancy of sub-
networks in continual learning from an information theory. An overview of the
IBM is shown in Fig. 2.

###  3.1 Prelimilary

In the setup of supervised continual learning, a series of $T$ tasks are
learned sequentially. The task is denoted by a task descriptor,
$\tau\in\\{1,2,...,T\\}$ and its corresponding dataset
$\textit{D}_{\tau}=\\{{(x_{\tau,i},\mathit{y}_{\tau,i})}_{i=1}^{N_{\tau}}\\}$
which has $N_{\tau}$ example pairs. The $x_{\tau}$ ($\in\textit{X}$ ) is the
input vector and $\mathit{y}_{\tau}$ ($\in\textit{Y}$ ) is the target vector.
Let’s consider a $L$ hidden layer neural network, where at each layer network
computes the following function to get hidden representation:
$h_{l}=\mathit{f}_{l}(\mathit{W}_{l},h_{l-1})$. Here, $l\in\\{1,2...,L\\}$,
$\mathit{f}$ is the operation of the network layer and $\mathit{W}$ denotes
the layer-wise weight. At the first layer, $h_{0}$ = $x_{i}$ represents the
raw input data. This network aims to learn the sequence of tasks by solving
the following optimization procedure at each task $\tau$:

$\Theta^{*}_{\tau}=\underset{\Theta}{\textit{minimize}}\frac{1}{N_{t}}\sum_{i=1}^{N_{t}}\mathcal{L}_{cl}(\sum_{l=1}^{L}\mathit{f}_{l}(x_{i};\mathit{W}_{l}),\mathit{y_{i}}),$

where $\Theta^{*}_{\tau}$ denotes the optimal model for task $\tau$ and
$\mathcal{L}_{cl}(\cdot,\cdot)$ is a classification objective loss, _e.g_.
cross-entropy loss. It’s worth noting that the data $D_{\tau}$ is only
accessible when learning for task $\tau$. In this paper, we mainly focus on
the task incremental learning setting, by which task identifier $\tau$ for
each data is available during training and testing time.

###  3.2 Information Bottleneck Masked sub-network

As illustrated in [35], the traditional neural network layers can be
interpreted as a Markov chain:

$x\to h_{1}\to h_{2}\to\dots\to h_{L}\to\hat{y}\to y.$ (1)

The hidden layers extract information from previous layers, while the final
layer attempts to proximate the true distribution $p(y)$ via output
distribution $p(\hat{y}|h_{L})$. Through training the network, sufficient
information for accurately predicting $p(y)$ percolates through the network
for output. However, the internal representations will contain superfluous
content [35, 8]. Therefore, the Information bottleneck [35] principle
penalizes an information-theoretic measure of redundancy between adjacent
network layers to identify a mapping $\sum_{l=1}^{L}\mathit{f}_{l}$ of the
input variable that achieves maximum compression while retaining as much
information as possible about the output variable. More specifically, it would
like to minimize the mutual information $I(h_{l};h_{l-1})$, while
simultaneously maximizing the mutual information $I(h_{l};y)$. Namely, finding
an optimal representation $h_{l}$ is formulated as the minimization of the
following Lagrangian:

$\mathcal{L}_{l}=\gamma_{l}I(h_{l};h_{l-1})-I(h_{l};y),$ (2)

where Lagrangian multiplier $\gamma\geq 0$ serves as a trade-off parameter
between the compression of the representation and the amount of preserved
relevant information.

Next, we use variational inference to facilitate tractable computation of Eq.
2. Specifically, we apply variational distribution $q(h_{l})$ and $q(y|h_{L})$
to approximate distribution $p(h_{l})$ and $p(y|h_{L})$, respectively. Through
performing variational inference, we get the upper bound of $\mathcal{L}_{l}$:

$\begin{split}\mathcal{L}_{l}\leq\hat{\mathcal{L}_{l}}=\gamma_{l}&\mathbbm{E}_{{h_{l-1}\sim
p(h_{l-1}|x)}}[\mathbbm{KL}\left[p(h_{l}|h_{l-1})||q(h_{l})\right]]\\\
-&\mathbbm{E}_{h\sim p(h|x)}[\log q(y|h_{L})].\end{split}$ (3)

Combining all the upper bound $\mathcal{L}_{l}$ in the network,
$\mathcal{L}_{all}=\sum_{l}\mathcal{L}_{l},$ minimize $\mathcal{L}_{all}$ is
the final variational information bottleneck loss to assimilate information
management across all layers.

For dealing with the $p(h_{l}|h_{l-1})$, $q(h_{l})$ and $q(y|h_{L})$ in Eq. 3,
a parametric form of these distributions is needed. VIBNet [8] directly
applies the re-parameterization on the hidden representation $h_{l}$. However,
this operation impedes our effort to integrate IB into the continual learning
context through some experimental validation. Therefore, we opt to use the re-
parameterization on the weight of the network:

$h_{l}=f_{l}(h_{l-1},(\mu_{l}+\epsilon_{l}\odot\sigma_{l})\odot W_{l}),$ (4)

where $\mu_{l}$ and $\sigma_{l}$ are learnable variational parameters (we
abbreviate it as Va-Para below) and $\epsilon_{l}$ is a random parameter
sampled from $\mathcal{N}(0,I)$. The bias in all layers is omitted to avoid
clutter. In this way, we can select the specific weights for each task,
facilitating the integration of IBM into the continual learning context. With
the above definitions in mind, the parametric form of $p(h_{l}|h_{l-1})$
follows that:

$\begin{split}p(h_{l}|h_{l-1})=\mathcal{N}(&h_{l};f_{l}(W_{l}h_{l-1}\odot\mu_{l}),\\\
&diag[f_{l}(W_{l}^{2}h_{l-1}^{2}\odot\sigma_{l}^{2})].\end{split}$ (5)

It is worth noting that Gaussian noise has previously been multiplied with
layer-wise activations as a path towards network regularization [1, 16].
Proceeding further, we specify the parametric form of $q(h_{l})$ following
Gaussian distribution as:

$q(h_{l})=\mathcal{N}(h_{l};0,diag[\xi_{l}]),$ (6)

where $\xi_{l}$ is a variance that can be learned. These Gaussian assumptions
facilitate the creation of an interpretable and closed-form approximation for
the KL term in Eq. 3, enabling us to directly optimize $\xi_{l}$ Finally,
following several algebraic manipulations, we obtain the revised final loss
function:

$\mathcal{L}_{all}={\textstyle\sum_{l=1}^{L}}[\gamma_{l}\log(1+\frac{\mu^{2}}{\sigma^{2}})]-L\mathbbm{E}_{h\sim
p(h|x)}[\log q(y|h_{L})].$ (7)

The trade-off $\gamma$ provides us the flexibility to control the ratio of
compression across each layer.

The later term of Eq. 7 is a data fit term involving an expectation over
latent hidden states, _i.e_. a traditional objective loss. With respect to the
former, it serves as a regularizer based on the KL divergence. It is
noteworthy that the function $\log(1+x)$ is concave and non-decreasing on the
domain $[0,\infty]$, representing canonical characteristics of a sparsity-
promotion regularizer. Therefore, rather than favoring a solution with many
smaller, partially shrunken versions of the ratios, this type of sparsity
archetype:

$\alpha_{l,j}=\mu_{l,j}^{2}\sigma_{l,j}^{-2},\forall l,j,$ (8)

instead prefers pushing some percentages to exactly zero while leaving others
mostly unchanged [29, 8]. Here, $j$ is the $j$-th element in $\mu_{l}$ or
$\sigma_{l}$. Therefore, along with optimizing the object Eq. 7, the useless
information will be aggregated into certain expendable weights, as opposed to
distributing the information equally across all neurons in a layer [8]. Once
the optimization object converges, a redundancy-free sub-network can be
constructed by pruning these expendable weights and selecting the rest
weights. Specifically, we select valuable weights to construct a binary mask
matrix for each layer:$M_{l}=\alpha_{l,j}>1$ following [8], and save the Va-
Para in the memory for inference. Such that the hidden representation $h_{l}$
could be recreated:

$\begin{split}T^{\tau}_{l}&=(\mu_{l}+\epsilon_{l}\odot\sigma_{l})\odot
M_{l},\\\ h_{l}&=f(h_{l-1},T^{\tau}_{l}\odot W_{l}).\end{split}$ (9)

It is worth noting that since the activated weights of each layer are far less
than the total parameter, the pressure for saving these Va-Para is negligible.
Until now, we have constructed a redundancy-free sub-network for the current
task. The next challenge is how to apply this method within the context of
continual learning.

How to overcome Catastrophic Forgetting. In the continual learning paradigm,
before we train on new tasks, we combine all masks of the previous task:

$\begin{split}M=\mathbb{0},M_{all}={\textstyle\sum_{i=1}^{\tau}}M_{i}||M,\end{split}$
(10)

where $||$ represents the ‘or’ operation. The final $M_{all}$ indicates
weights that maintain valuable information of previous tasks. To solve the
catastrophic forgetting, we opt to freeze these weights, allowing updates only
on the weights that have not been selected. Specifically, when training on a
new task, before performing back-propagation, we modify the gradients
$\nabla_{W}\mathcal{L}$ of weights based on the binary masks learned before:

$\nabla_{W}\mathcal{L}=\nabla_{W}\mathcal{L}\odot(1-M_{all}),$ (11)

effectively freezing the weights selected for the previous tasks. When
performing inference on previous tasks, we can reproduce the same results as
before through the frozen weights and the saved Va-Para.

###  3.3 Re-initialization of Va-Para

However, since the sub-networks for old tasks are frozen, it will limit
knowledge transfer for learning new tasks [17]. To solve this problem, before
training on new tasks, we reuse the Va-Para that is selected by previous masks
and re-initialize the rest:

$\begin{split}\mu_{l}&=\mu_{l}\odot
M_{all,l}+\mu_{random}\odot(1-M_{all,l}),\\\ \sigma_{l}&=\sigma_{l}\odot
M_{all,l}+\sigma_{random}\odot(1-M_{all,l}).\end{split}$ (12)

Since the learned Va-Para and the frozen weights accumulate valuable
knowledge, the current optimization procedure can select positive knowledge
from these parameters and abandon negative one to facilitate current training.
Additionally, re-initialization enables the optimization to escape local
optima and discover a more favorable optimization space for future learning.

###  3.4 Feature Decomposing

As illustrated in Sec. 1, since lower layers predominantly handle visual
features while higher layers emphasize semantic information, thus different
layers should have different compression ratios. Thus, we investigate the
distribution of hidden representation to set the ratios automatically and
flexibly. In the continual learning community, SVD is a common tool to analyze
the hidden representations. It can be used to factorize hidden representations
$H=U\Sigma V^{T}\in\mathbbm{R}^{m\times n}$ into the product of three
matrices, where $U\in\mathbbm{R}^{m\times m}$ and $U\in\mathbbm{R}^{n\times
n}$ are orthogonal, and $\Sigma$ contains sorted singular values along its
main diagonal [9].

In this paper, to investigate the distribution of the hidden representation,
we also perform SVD on $h_{l}=U_{l}\Sigma_{l}V_{l}^{T}$ followed by its
$k$-rank approximation $(h_{l})_{k}$ according to the following criteria for
the given threshold $\delta$:

$\left\|(h_{l})_{k}\right\|^{2}_{F}\geq\delta\left\|h_{l}\right\|^{2}_{F},$
(13)

where $F$ represents Frobenius norm and $\delta$ ($0<\delta<1$) is the
threshold hyperparameter. By applying the $k$-rank approximation, the first
$k$ vectors in $U$ represent the significant representation since they contain
all the directions with the highest singular values in the representation.
Therefore, the ratio of $k$ over all channels is chosen as the final ratio of
compression of layer $l$, _i.e_. $\gamma_{i}$. Further, since the feature
distribution shifts along the training process, we perform this decomposing
process every $E$ epoch to dynamically adjust the ratio, achieving a more
precise compression.

##  4 Experiment

###  4.1 Experimental setting

#### Datasets

We conduct experiments on various continual learning benchmarks, including
CIFAR-100 [18], TinyImageNet, MiniImageNet [37]. The CIFAR-100 is constructed
by randomly splitting 100 classes of CIFAR-100 into 10 tasks, where each class
includes 500 training samples and 100 testing samples. The TinyImageNet is a
variant of the ImageNet dataset [10] comprising 10 randomized classes out of
the 200 classes for each task. Finally, MiniImageNet is a dataset formed by
partitioning the 100 classes of ImageNet into 10 sequential tasks, with each
task comprising 10 classes. Each class encompasses 500 training samples and
100 testing samples.

#### Baselines

We compare our method with state-of-the-art continual learning methods,
including memory-based: ER_ace [5], ER [30], Der [4], Der++ [4], Gdumb [28],
Gem [22], A-Gem [6], and regularization-based: SI [40], EWC_on [33].
Additionally, three parameter isolation methods are selected, _i.e_. Supsup
[39], WSN [15], SPG [17]. Moreover, we consider two simple methods: Multi-task
Learning which trains a new network for each task, and Finetune Learning which
learns without any mechanism to deal with catastrophic forgetting.

#### Training setup

For a fair comparison among CL methods, we conduct experiments based on the
public library [4]. Adam optimizer with a fixed learning rate of 0.001 and
batch size 256 is applied to all experiments. We adopt AlexNet [19] and
ResNet-18 [13] as the backbones and train them 300 epochs per task on all
datasets. Besides, we opt to retain 2000 samples for the memory-based methods
mentioned above. For our IBM, the threshold $\delta$ and epoch interval $E$
for feature decomposing is set to 0.97 and 50, respectively. We conduct each
experiment three times with different seeds to obtain averaged metrics. The
details can be found in the Appendix.

#### Evaluation metrics

We evaluate the performance on the following metrics: Average Accuracy,
Backward Transfer and Forward Transfer. (1) Average Accuracy (ACC): A average
accuracy of all tasks after training on the final task.
$ACC=\frac{1}{T}\sum_{i=1}^{T}A_{T,i},$ where $A_{T,i}$ is the performance on
$\tau=i$ task after training the final task ($\tau=T$).

(2) Backward Transfer (BWT): This metric measures the catastrophic forgetting
after learning on all tasks.
$BWT=\frac{1}{T-1}\sum_{i=1}^{T-1}(A_{T,i}-A_{i,i}),$ where $A_{i,i}$ is the
performance on $\tau=i$ task after training on the $\tau=i$ task.

(3) Forward Transfer (FWT): This metric indicates the positive influence of
previous tasks on the current task.
$FWT=\frac{1}{T-1}\sum_{i=1}^{T-1}(A_{i,i}-MT_{i}),$ where $MT_{i}$ is the
performance on task $\tau=i$ in Multi-task Learning.

Figure 3: The visualization of masked weight numbers within each layer of the
first sub-network constructed by WSN and IBM on CIFAR-100 with Resnet-18. The
significant reduction of weight numbers substantiates the conjecture of
employing the information bottleneck for reducing redundancy within sub-
networks.

###  4.2 Conjecture Validation

Here, we first validate our conjecture that information bottleneck can be
utilized for reducing redundancy within sub-networks. Figure 3 shows the
masked weights count for each layer of the first sub-network (for the first
task), where the results are obtained from WSN and our IBM with ResNet-18 on
CIFAR-100. Here, we choose WSN because it is the parameter-isolation SOTA
method now. In the case of WSN, weight selection involves an equal
distribution based on the top-$k$ percent weight ranking scores (set at 50% in
this experiment) across all layers. In contrast, our approach allows for
dynamic pruning of irrelevant weights and adapts the selection percentage
accordingly. Analysis of the figure reveals that WSN retains a limited number
of weights in lower layers but exhibits an abundance of weights in higher
layers, due to a significant rise in parameter quantity. In comparison to WSN,
our IBM employs slightly more weights in lower layers but demonstrates a
substantial reduction in the number of weights required in higher layers. This
phenomenon can be attributed to the necessity for our IBM to capture ample
visual information in the lower layers, allowing it to propagate through the
network and provide sufficient semantic information for the higher layers.
Consequently, our IBM achieves a 90% reduction in the total number of weights
and a better performance, surpassing WSN by 0.6%. The obtained results provide
empirical support for the conjecture that implementing an information
bottleneck is effective in reducing redundancy within sub-networks.

###  4.3 Comparisons with the State-of-the-arts

To evaluate the effectiveness of IBM in the continual learning context, we
first compare it with state-of-the-art methods with Alexnet on CIFAR-100.
Quantitative results are shown in Tab. 1. The results show that our IBM
achieves the best average accuracy of 82.69%. It is worth noting that, since
weights learned by previous tasks have been frozen, our method is proved to be
a forget-free model, resulting in 0% on backward transfer. In addition, with
the help of the re-initialization of Va-Para, IBM exhibits the capability to
transfer acquired knowledge to facilitate new task learning, achieving 3.52%
on forward transfer and outperforming previous parameter isolation methods.

Method | CIFAR-100
---|---
ACC (%) | BWT (%) | FWT (%)
Multi-task | 79.08±0.25 | 0 | 0
Finetune | 48.97±1.59 | -38.91±1.79 | 4.82±0.46
ER_ace [5] | 72.05±0.39 | -12.37±1.59 | 4.14±0.96
ER [30] | 71.79±0.05 | -13.55±0.11 | 4.82±0.35
Der [4] | 73.27±1.08 | -10.19±0.92 | 3.27±0.64
Der++ [4] | 74.12±0.72 | -10.04±1.23 | 3.98±0.13
SI [40] | 71.67±1.14 | -10.43±0.98 | 1.88±0.49
EWC_on [33] | 75.33±0.54 | -0.73±0.16 | -3.19±0.65
AGem [6] | 65.05±1.95 | -20.65±2.68 | 4.47±0.33
Gem [22] | 74.48±0.48 | -10.16±0.50 | 4.45±0.33
Gdumb [28] | 51.03±0.78 | -9.71±0.83 | -19.40±0.05
Supsup [39] | 67.30±0.66 | 0 | -11.83±0.67
SPG [17] | 69.07±0.94 | -5.52±0.97 | -5.17±0.44
WSN [15] | 82.28±0.18 | 0 | 3.11±0.23
IBM | 82.69±0.12 | 0 | 3.52±0.24

Table 1: Experimental results of Alexnet backbone on CIFAR-100 dataset. The
mean and standard deviation of the average accuracy (ACC), backward transfer
(BWT) and forward transfer (FWT) across three independent runs are reported.
The best results are highlighted in bold.

Method | CIFAR-100 | TinyImageNet | MiniImageNet
---|---|---|---
ACC (%) | BWT (%) | FWT (%) | ACC (%) | BWT (%) | FWT (%) | ACC (%) | BWT (%) | FWT (%)
Multi-task | 85.16±0.54 | 0 | 0 | 49.98±0.35 | 0 | 0 | 52.69±0.25 | 0 | 0
Finetune | 34.64±2.71 | -59.09±3.53 | 2.66±0.84 | 14.33±0.60 | -42.11±0.45 | 2.24±0.64 | 19.63±0.51 | -38.15±0.71 | 1.28±0.42
ER_ace [5] | 77.80±0.87 | -11.72±1.16 | 3.18±0.72 | 33.99±0.41 | -19.21±0.58 | 1.29±0.20 | 48.01±0.64 | -8.23±0.53 | 2.74±0.49
ER [30] | 77.60±0.46 | -12.01±0.26 | 3.25±0.14 | 33.43±0.55 | -24.30±9.24 | 0.70±0.62 | 46.91±0.66 | -8.51±0.13 | 1.88±0.82
Der [4] | 83.99±0.07 | -4.36±0.37 | 2.75±0.30 | 40.98±0.25 | -10.97±0.73 | 0.87±1.82 | 52.45±0.37 | -3.79±0.43 | 3.17±0.36
Der++ [4] | 84.23±0.31 | -4.58±0.59 | 3.19±0.58 | 40.70±0.53 | -11.73±0.45 | 1.28±1.20 | 53.12±0.72 | -3.81±0.37 | 3.86±0.37
SI [40] | 36.87±2.47 | -54.42±2.29 | 0.68±0.48 | 14.53±0.39 | -41.20±0.43 | 1.63±0.64 | 19.30±1.19 | -36.97±1.71 | -0.11±1.20
EWC_on [33] | 74.05±0.29 | -5.20±0.30 | -6.44±0.96 | 39.35±0.54 | -3.96±0.22 | -7.07±0.91 | 44.14±1.32 | -7.11±0.97 | -2.15±0.51
AGem [6] | 63.16±0.57 | -28.47±0.93 | 3.62±0.87 | 23.00±1.00 | -30.73±0.61 | 0.67±0.65 | 39.26±1.38 | -16.45±1.78 | 1.37±0.03
Gem [22] | 78.99±0.60 | -10.31±1.21 | 3.10±0.92 | 34.79±0.29 | -19.49±0.38 | 2.35±0.37 | 47.55±0.65 | -8.03±0.49 | 2.09±0.47
Gdumb [28] | 42.42±0.97 | -13.47±1.42 | -30.62±1.35 | 13.64±0.53 | -4.46±0.58 | -32.33±0.98 | 24.21±0.63 | -7.35±0.46 | -21.87±0.07
SPG [17] | 48.00±1.47 | -17.67±0.78 | -21.20±2.09 | 21.14±0.72 | -9.15±0.28 | -20.65±1.07 | 40.48±0.48 | -17.78±0.30 | 3.85±0.80
Supsup [39] | 79.94±0.41 | 0 | -4.97±0.26 | 40.27±0.40 | 0 | -9.72±0.61 | 50.68±0.14 | 0 | -2.01±0.22
WSN [15] | 86.47±0.22 | 0 | 1.30±0.47 | 49.36±0.34 | 0 | -0.62±0.30 | 52.63±0.53 | 0 | -0.06±0.72
IBM | 88.15±0.09 | 0 | 2.98±0.57 | 52.38±0.33 | 0 | 2.40±0.63 | 53.90±0.26 | 0 | 1.22±0.51

Table 2: The quantitative results of each baseline and our IBM based on
Resnet-18 backbone. We report the mean and standard deviation of the average
accuracy (ACC), backward transfer (BWT) and Forward transfer (FWT) across
three independent runs with different seeds under the same experimental setup.
Our method yields superior results among all datasets.

Then, to further verify the behavior of our IBM on a deeper network, we
conduct experiments on the ResNet-18 backbone. The comparative results on
three datasets are presented in Tab. 2. We have the following observations
from this table: Firstly, our method consistently achieves the best average
accuracy, gaining improvements of 1.68%, 3.02%, and 0.78% compared to SOTA
methods on CIFAR-100 (WSN), TinyImageNet (WSN) and MiniImageNet (Der++),
respectively. Secondly, our method even behaves better than Multi-task which
is seen as a strong baseline of continual learning, on all datasets. The
possible reasons are manifolds: (1). The compact sub-networks work better than
over-parameterized dense neural networks [11, 15, 21]. After further analysis
of the result of each task obtained by IBM and Multi-task, we observe that IBM
is always better than Multi-task. (Details of the results can be found in
Appendix.) (2). The Multi-task lacks the ability to transfer knowledge. Since
it trains a new network for each task, there is no interaction between
individual network, which ultimately limits the transferability of knowledge.
In contrast, our method leverages the re-initialization of Va-Para, allowing
IBM to selectively retain valuable knowledge and achieve effective knowledge
transfer. This is evident in the results for FWT, where our approach achieves
significant improvements, specifically 2.98%, 2.40%, and 1.22% on three
different datasets, respectively.

Variant | FD | Re-init | CIFAR-100 | TinyImageNet | MiniImageNet
---|---|---|---|---|---
Selected | Unselected | ACC (%) | FWT (%) | ACC (%) | FWT (%) | ACC (%) | FWT (%)
➀ | ✗ | ✗ | ✗ | 85.92±0.40 | 0.75±0.27 | 48.86±0.34 | -1.12±0.41 | 50.32±0.17 | -2.37±0.09
➁ | ✓ | ✗ | ✗ | 86.53±0.54 | 1.37±0.37 | 49.49±0.33 | -0.49±0.22 | 51.49±0.46 | -1.20±0.48
➂ | ✓ | ✗ | ✓ | 88.15±0.09 | 2.98±0.57 | 52.38±0.33 | 2.40±0.63 | 53.90±0.26 | 1.22±0.51
➃ | ✓ | ✓ | ✓ | 85.22±0.30 | 0.05±0.27 | 47.34±0.71 | -2.65±0.95 | 49.93±0.92 | -2.76±1.17

Table 3: The quantitative results for the ablation studies on feature
decomposing and re-initialization of IBM, employing Resnet-18 as the backbone.
It’s important to note that since our method does not suffer catastrophic
forgetting, we do not record the backward transfer. The terms “Selected” and
“Unselected” denote the subsets of Va-Para that are chosen by sub-network
masks and those that are not, respectively.

###  4.4 Ablation studies

Impact of proposed modules. Our IBM contains two key components, including a
re-initialization of the Va-Para (Re-init) process for transferring valuable
knowledge, as well as a feature decomposing (FD) for adjusting compression
ratios. We conduct component-wise analysis by progressively incorporating them
into a baseline model that omits feature decomposing and re-initialization.
Resnet-18 network is chosen as the backbone. Since the results in Tab. 1 and
Tab. 2 have confirmed our ability to address catastrophic forgetting, we do
not record the backward transfer in these experiments. The experimental
results are shown in Tab. 3. From ➀ and ➁, when equipped with feature
decomposing, we observe a significant increase on average accuracy (_e.g_.,
CIFAR-100: 85.92% _vs_. 86.53% , TinyImageNet: 48.86% _vs_. 49.49%,
MiniImageNet: 50.32% _vs_. 51.49%). This can be construed as the feature
decomposing exploring the most suitable ratio for each layer and adjusting
these ratios along the training process. What’s more, these experiment results
prove that the feature decomposing not only sets the ratio automatically and
flexibly as we designed but also has the ability to promote knowledge
transfer, _e.g_., resulting in a 1.17% increase on MiniImageNet in terms of
FWT. Additionally, compared ➁ and ➂, it can be observed that when training
with re-initialization, there is a more substantial improvement on average
accuracy (_e.g_., CIFAR-100: 86.53% _vs_. 88.15%, TinyImageNet: 49.49% _vs_.
52.38%, MiniImageNet: 51.49% _vs_. 53.90%) and forward transfer (_e.g_.,
CIFAR-100: 1.37% _vs_. 2.98%, TinyImageNet: -0.49% _vs_. 2.40%, MiniImageNet:
-1.20% _vs_. 1.22%), demonstrating the effectiveness of re-initialization to
extract valuable knowledge while eliminating the negative impact of redundant
information. These results also validate that the re-initialization enables
the optimization to escape local optima of previous task and discover a more
favorable optimization space for future learning. Finally, when re-
initializing the essential parameters (_i.e_., ➂ and ➃), it yields the worst
performances on all datasets. This occurs because the re-initialization of
valuable parameters results in a decline in the acquired knowledge.

Figure 4: The mean and standard deviation results of ablation studies about
feature decomposing interval on CIFAR-100 dataset with ResNet-18 as the
backbone. We choose 50 epoch interval as the main experiment setting to
balance the performance and efficiency.

Impact of feature decomposing frequency. To investigate the influence of
decomposing interval $E$ in feature decomposing, we conduct experiments on
CIFAR-100 with ResNet-18. In Fig. 4, we report the obtained results of the IBM
w/o or w/ the feature decomposing module at different training epochs,
including 10, 25, 50, 100 and 150. (The results of TinyImageNet and
MiniImageNet can be found in the Appendix). In this figure, dashed lines
represent the results of w/o variant in terms of average accuracy and forward
transfer. As can be observed, all variants with the feature module
consistently outperform w/o variant, affirming the effectiveness of feature
decomposing. In addition, with the decomposing interval increases, the
performances decrease. This trend can be attributed to the fact that, with a
smaller interval, the module can more precisely analyze the distribution over
time and adjust the compression ratio, leading to enhanced performance.
Nonetheless, a smaller interval demands more training time for analyzing the
hidden representation, which limits efficiency. Hence, to balance performance
and efficiency, we opt for a 50-epoch interval as the default setting for the
main experiments.

Method | TinyImageNet
---|---
ACC (%) | BWT (%) | FWT (%)
Multi-task | 59.14 | 0 | 0
Finetune | 16.32 | -49.53 | 4.23
ER_ace [5] | 43.83 | -21.03 | 4.67
ER [30] | 44.20 | -20.31 | 4.35
Der [4] | 50.86 | -11.78 | 2.91
Der++ [4] | 52.88 | -11.62 | 4.78
SI [40] | 14.35 | -50.85 | 3.52
EWC_on [33] | 49.45 | -2.62 | -7.20
AGem [6] | 25.16 | -38.16 | 2.27
Gem [22] | 48.86 | -16.34 | 5.24
Gdumb [28] | 23.48 | -5.08 | -30.83
SPG [17] | 16.98 | -5.53 | -36.89
Supsup [39] | 53.83 | 0 | -5.31
WSN [15] | 61.02 | 0 | 1.88
IBM | 63.34 | 0 | 4.20

Table 4: Experimental results of 20-tasks evaluation on TinyImageNet dataset.
We select Resnet-18 as the backbone.

###  4.5 Capacity for a longer sequence

A common issue in the parameter isolation is that since the network capacity
is limited, “free” parameters tend to saturate as more tasks are introduced,
leading to a diminished capacity for learning new tasks. To assess the
capability of our method in overcoming this challenge, we conducted
experiments using extended sequences on TinyImageNet, with the quantitative
results presented in Tab. 4 (Additional results for CIFAR-100 and MiniImageNet
are included in the Appendix). As can be seen, our IBM consistently
outperforms other methods, without suffering the limited capacity challenge.
This is due to the fact that IBM reduces redundancy in sub-networks, leaving
more capacity for future tasks. This advantage allows IBM to have more
capacity to learn fresh knowledge, which leads to better performance.

##  5 Conclusion

This paper proposes a novel parameter-isolation method based on the
information bottleneck for pruning the backbone network to construct
redundancy-free sub-networks for continual learning. Moreover, this method
freezes sub-networks to mitigate catastrophic forgetting and re-use the
essential parameters masked by these sub-networks to transfer valuable
knowledge. In addition, since the different layers of the network process
different information, the different layers should hold distinct prune ratios
to construct sub-networks. We propose a feature decomposing module to
decompose hidden representations to set the ratio automatically and flexibly.
The results of extensive experiments demonstrate the effectiveness of our
method.

## References

  * Achille and Soatto [2018] Alessandro Achille and Stefano Soatto.  Information dropout: Learning optimal representations through noisy computation.  _IEEE TPAMI_ , 2018.
  * Aljundi et al. [2018] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In _ECCV_ , 2018.
  * Ayub and Wagner [2021] Ali Ayub and Alan R. Wagner.  EEC: learning to encode and regenerate images for continual learning.  In _ICLR_ , 2021.
  * Buzzega et al. [2020] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  In _NeurIPS_ , 2020.
  * Caccia et al. [2022] Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky.  New insights on reducing abrupt representation change in online continual learning.  In _ICLR_ , 2022.
  * Chaudhry et al. [2019] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with A-GEM.  In _ICLR_ , 2019.
  * Chen et al. [2022] Cheng Chen, Ji Zhang, Jingkuan Song, and Lianli Gao.  Class gradient projection for continual learning.  In _ACM MM_ , 2022.
  * Dai et al. [2018] Bin Dai, Chen Zhu, Baining Guo, and David P. Wipf.  Compressing neural networks using the variational information bottleneck.  In _ICML_ , 2018.
  * Deisenroth et al. [2020] Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong.  _Mathematics for machine learning_.  Cambridge University Press, 2020.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Frankle and Carbin [2019] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _ICLR_ , 2019.
  * Gu et al. [2023] Qiao Gu, Dongsub Shim, and Florian Shkurti.  Preserving linear separability in continual learning by backward feature projection.  In _CVPR_ , 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Jing and Tian [2021] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  _IEEE TPAMI_ , 2021.
  * Kang et al. [2022] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D. Yoo.  Forget-free continual learning with winning subnetworks.  In _ICML_ , 2022.
  * Kingma et al. [2015] Diederik P. Kingma, Tim Salimans, and Max Welling.  Variational dropout and the local reparameterization trick.  In _NeurIPS_ , 2015.
  * Konishi et al. [2023] Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu.  Parameter-level soft-masking for continual learning.  In _ICML_ , 2023.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In _NeurIPS_ , 2012.
  * Li and Hoiem [2016] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  In _ECCV_ , 2016.
  * Lin et al. [2017] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou.  Runtime neural pruning.  In _NeurIPS_ , 2017.
  * Lopez-Paz and Ranzato [2017] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In _NeurIPS_ , 2017.
  * Louizos et al. [2017] Christos Louizos, Max Welling, and Diederik P. Kingma.  Learning sparse neural networks through l${}_{\mbox{0}}$ regularization.  _CoRR_ , 2017.
  * Luo et al. [2022] Xu Luo, Jing Xu, and Zenglin Xu.  Channel importance matters in few-shot image classification.  In _ICML_ , 2022.
  * Mallya and Lazebnik [2018] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In _CVPR_ , 2018.
  * McCloskey and Cohen [1989] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_. 1989.
  * Neklyudov et al. [2017] Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov.  Structured bayesian pruning via log-normal multiplicative noise.  In _NeurIPS_ , 2017.
  * Prabhu et al. [2020] Ameya Prabhu, Philip H. S. Torr, and Puneet K. Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In _ECCV_ , 2020.
  * Rao et al. [2003] Bhaskar D. Rao, Kjersti Engan, Shane F. Cotter, Jason A. Palmer, and Kenneth Kreutz-Delgado.  Subset selection in noise based on diversity measure minimization.  _IEEE TSP_ , 2003.
  * Riemer et al. [2019] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  In _ICLR_ , 2019.
  * Ring [1998] Mark B. Ring.  Child: A first step towards continual learning.  In _Learning to Learn_. Springer, 1998.
  * Rusu et al. [2016] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _CoRR_ , 2016.
  * Schwarz et al. [2018] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In _ICML_ , 2018.
  * Serrà et al. [2018] Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In _ICML_ , 2018.
  * Tishby and Zaslavsky [2015] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  _CoRR_ , 2015.
  * Veniat et al. [2021] Tom Veniat, Ludovic Denoyer, and Marc’Aurelio Ranzato.  Efficient continual learning with modular networks and task-driven priors.  In _ICLR_ , 2021.
  * Vinyals et al. [2016] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  In _NeurIPS_ , 2016.
  * Wang et al. [2015] Lijun Wang, Wanli Ouyang, Xiaogang Wang, and Huchuan Lu.  Visual tracking with fully convolutional networks.  In _ICCV_ , 2015.
  * Wortsman et al. [2020] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi.  Supermasks in superposition.  In _NeurIPS_ , 2020.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _ICML_ , 2017.

Supplementary Material

##  6 Training Detail

For hyper-parameters for other methods, we follow the public library [4]. The
detail hyper-parameters for baselines are shown in Tab. 5.

##  7 Performance on each task of IBM and Multi-task

The results for each task of IBM and Multi-task on three datasets are
presented in Tab. 6. Examination of the table reveals that, on the CIFAR-100
dataset, the performance of each task in IBM surpasses that of Multi-task.
This validate the hypothesis that the compact sub-networks work better than
over-parameterized dense neural networks [11, 15, 21]. Furthermore, these
results confirm that our IBM possesses the capability to selectively retain
valuable knowledge, facilitating effective knowledge transfer. On more
challenging datasets, namely TinyImageNet and MiniImageNet, initially, the
performance of IBM is slightly inferior to that of Multi-task. This can be
attributed to the fact that, as task difficulty increases, the sub-networks
are not as over-parameterized. However, when IBM transfers valuable knowledge,
this knowledge compensates for the parameter deficiency, leading to improved
performance.

Methods | Hyper-parameter
---|---
ER_ace | B:2000
ER | B:2000
DER | B:2000 Mini_batch_size:256 Alpha:0.3
DER++ | B:2000 Mini_batch_size:256 Alpha:0.3 Beta:0.5
SI | c:10.0 xi:0.1
EWC_on | e_lambda:1000 gamma:10
AGem | B:2000 Mini_batch_size:256
Gem | B:2000 Mini_batch_size:256 gamma 0.5
Gdumb | B:2000 Mini_batch_size:256 Maxlr:5e-4
| Minlr:1e-4 cutmix_alpha:1 fitting_epochs:200
Supsup | sparsity:8
WSN | sparsity:0.5

Table 5: List of hyperparameters for the baselines. “B” and “Mini_batch_size”
denote the buffer size and the buffer batch size for rehearsal, respectively.

##  8 Decomposing Results on TinyImageNet and MiniImageNet

We present the performance of feature decomposing variants with different
intervals in Fig. 4. Specifically, Fig. 5 displays the performance of these
variants on TinyImageNet and MiniImageNet, utilizing ResNet-18 as the
backbone. Notably, all variants employing feature decomposing exhibit superior
performance compared to the variant without this module, thereby confirming
the effectiveness of the feature decomposing module. Furthermore, the results
obtained with different intervals show a consistent trend on both datasets,
TinyImageNet and MiniImageNet, indicating the robustness of our feature
decomposing module with respect to varying intervals.

Method | Dataset | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---|---|---|---
IBM | CIFAR-100 | 87.8 | 88.2 | 86.6 | 88.2 | 88.6 | 86.7 | 89.3 | 86.0 | 90.6 | 90.4
TinyImageNet | 45.1 | 42.4 | 51.7 | 53.4 | 56.5 | 60.2 | 49.1 | 57.3 | 56.3 | 49.6
MiniImageNet | 53.3 | 56.6 | 50.1 | 52.0 | 52.6 | 58.4 | 58.4 | 51.8 | 51.4 | 55.4
2-12 Muiti-task | CIFAR-100 | 86.9 | 83.8 | 85.3 | 87.9 | 83.5 | 83.1 | 84.1 | 80.1 | 84.3 | 88.9
TinyImageNet | 46.2 | 44.1 | 49.9 | 51.2 | 53.4 | 57.7 | 47.0 | 53.0 | 54.9 | 46.5
MiniImageNet | 54.6 | 53.9 | 49.5 | 49.4 | 53.7 | 56.4 | 55.7 | 47.7 | 47.7 | 56.6

Table 6: Quantitative results for each task after training on every task on
CIFAR100, TinyImageNet and MiniImageNet. These experiments are conducted on
Resnet-18 backbone.

Figure 5: The mean and standard deviation results of ablation studies about
feature decomposing interval on TinyImageNet and MiniImageNet dataset with
ResNet-18 as the backbone.

##  9 Capacity for a longer sequence

This section provide the results for validating the the capability of our
method in overcoming limited capacity, shown in Tab. 7 and Tab. 8.

Method | CIFAR-100
---|---
ACC (%) | BWT (%) | FWT (%)
Multi-task | 89.06 | 0 | 0
Finetune | 27.47 | -67.82 | 2.84
ER_ace [5] | 81.53 | -10.98 | 2.90
ER [30] | 80.92 | -11.07 | 2.38
Der [4] | 84.23 | -6.41 | 1.26
Der++ [4] | 87.23 | -4.25 | 2.21
SI [40] | 29.77 | -63.73 | 1.25
EWC_on [33] | 74.56 | -4.09 | -10.61
AGem [6] | 67.04 | -26.13 | 2.80
Gem [22] | 86.26 | -6.35 | 3.23
Gdumb [28] | 55.48 | -10.55 | -23.56
SPG [17] | 51.75 | -20.79 | -17.56
Supsup [39] | 84.88 | 0 | -4.17
WSN [15] | 90.11 | 0 | 1.05
IBM | 90.44 | 0 | 1.38

Table 7: Experimental results of 20-tasks evaluation on CIFAR-100 dataset with
ResNet-18.

Method | MiniImageNet
---|---
ACC (%) | BWT (%) | FWT (%)
Multi-task | 66.5 | 0 | 0
Finetune | 26.9 | -43.52 | 1.74
ER_ace [5] | 62.81 | -7.68 | 3.61
ER [30] | 61.68 | -8.49 | 3.25
Der [4] | 63.07 | -5.56 | 1.85
Der++ [4] | 66.26 | -4.16 | 3.71
SI [40] | 26.45 | -43.73 | 1.49
EWC_on [33] | 58.68 | -3.99 | -4.03
AGem [6] | 51.36 | -18.14 | 2.09
Gem [22] | 63.02 | -7.71 | 3.84
Gdumb [28] | 40.97 | -6.66 | -19.2
SPG [17] | 39.88 | -27.53 | -0.35
Supsup [39] | 67.34 | 0 | 0.84
WSN [15] | 66.9 | 0 | 0.4
IBM | 68.45 | 0 | 1.95

Table 8: Experimental results of 20-tasks evaluation on MiniImageNet dataset
with ResNet-18.