# The non-condensed fraction of a homogeneous dilute Bose gas within the
improved Hartree-Fock approximation at finite temperature

Nguyen Van Thu  Department of Physics, Hanoi Pedagogical University 2, Hanoi
100000, Vietnam<EMAIL_ADDRESS>
###### Abstract

By means of Cornwall–Jackiw–Tomboulis effective action we investigate a dilute
weakly interacting Bose gas at finite temperature. The shift of critical
temperature is obtained in the universal form $\Delta
T_{C}/T_{C}^{(0)}=c.n_{0}^{1/3}a_{s}$ with constant $c$. The non-condensate
fraction is expressed in sum od three terms, which correspond to the quantum
fluctuation, thermal fluctuation and both.

Quantum and thermal fluctuations, critical temperature, Bose gas, improved
Hartree-Fock approximation

##  I Introduction

It is well-known that a system of Bose gas undergoes a condensate phase
transition when its temperature is lowed to the critical temperature [1, 2].
The result is a number of atoms in the Bose gas will be condensed and formed a
Bose-Einstein condensate (BEC). The criterion for the BEC is formed in
macroscopic scale is [3],

$\displaystyle n_{0}\lambda_{B}^{3}\leq\zeta(3/2),$ (1)

in which $n_{0}$ is particle density, which is defined as the number of
particle in a unit volume, $\zeta(x)$ is Zeta function. The de Broglie wave
length associated with a particle with mass of $m$ at temperature $T$ reads

$\displaystyle\lambda_{B}=\left(\frac{2\pi\hbar^{2}}{mk_{B}T}\right)^{1/2},$
(2)

with $\hbar$ and $k_{B}$ being the reduced Planck and Boltzmann constants,
respectively. In case of the ideal Bose gas, Eq. (1) gives the critical
temperature [3],

$\displaystyle
T_{C}^{(0)}=\frac{2\pi\hbar^{2}}{mk_{B}}\left[\frac{n_{0}}{\zeta(3/2)}\right]^{2/3}.$
(3)

For a weakly interacting Bose gas, the atomic interaction is assumed to be
pairwise and described by repulsive contact pseudo-potential (Lieb–Liniger
model) [4, 5]. The strength of this interaction is determined by a coupling
constant $g=4\pi\hbar^{2}a_{s}/m$ with $a_{s}$ the scattering length of the
$s$-wave. Effect from the repulsive interaction, the critical temperature is
shifted in compared with that of the ideal Bose gas

$\displaystyle\frac{\Delta T_{C}}{T_{C}^{(0)}}=c\alpha_{s}^{1/3},$ (4)

where $\alpha_{s}=n_{0}a_{s}^{3}$ is called the gas parameter and $\Delta
T_{C}\equiv T_{C}-T_{C}^{(0)}$. The condition of diluteness of the Bose gas is
$n_{s}\ll 1$ [6]. The constant $c$ is to be computed.

As the temperature is lower than the critical temperature, the lower the
temperature is, the more condensate particles there are. Theoretically, at
zero temperature all of the atoms will stay in the ground state [7]. In this
situation, the ground state is described by a wave function, which can be
found by solving the Gross-Pitaevskii (GP) equation [8, 9]. However, due to
interaction-induced quantum fluctuations in the BEC, some particles with
nonzero momentum reside in excited states instead of the ground state, even at
zero absolute temperature [3]. These particles get pushed out of the
condensate— the phenomenon of quantum depletion. The number of atoms in the
remaining condensate fraction due to quantum depletion was first studied in
1947 by N. N. Bogoliubov [10] up to order $1/2$ in the gas parameter by using
the second quantization formalism. The main idea is based on a quantum
description, where the particle operators are transformed into quasi-particle
operators, yielding an explicit diagonalization of the quantum Hamiltonian. In
1997, using the Bogoliubov theory and the semiclassical approach, the authors
of Ref. [11] investigated the quantum depletion in a Bose gas confined by a
harmonic trap. In the case of a homogeneous Bose gas their result corresponded
exactly with Bogoliubov’s result. Within the GP theory, the condensed fraction
was reproduced by S. Stringari [12]. Using an effective theory called
simplified approach, Carlen et. al. [13] verified the Bogoliubov’s result for
the condensed fraction of the BEC at low density. Recently, within the
framework of Cornwall–Jackiw–Tomboulis (CJT) effective action we studied
effect from the quantum fluctuation to non-condensate fraction of the
homogeneous Bose gas [14], in which not only Bogoliubov result was confirmed
but the higher-order terms of the gas parameter were also obtained. Moreover,
the non-condensate fraction of a confined Bose gas was investigated [15]. A
correction term due to the finite-size effect was found.

The main purpose of the present paper is to investigate the critical
temperature and non-condensate fraction of a dilute weakly interacting Bose
gas.

We set the stage for our calculations by starting with a dilute Bose gas
described by the following Lagrangian density [3],

${\cal L}=\psi^{*}\left(-i\hbar\frac{\partial}{\partial
t}-\frac{\hbar^{2}}{2m}\nabla^{2}\right)\psi-\mu\left|\psi\right|^{2}+\frac{g}{2}\left|\psi\right|^{4},$
(5)

wherein $\mu$ is the chemical potential. The field operator $\psi(\vec{r},t)$
depends on both the coordinate $\vec{r}$ and time $t$. According to Ref. [3],
the interaction potential between the atoms can be chosen as the hard-sphere
model. In this case, the strength of the interaction between the atoms is
determined by the coupling constant $g=4\pi\hbar^{2}a_{s}/m$, which is
expressed in terms of the $s$-wave scattering length $a_{s}$ by making use of
the Born approximation. Now, thermodynamic stability requires that $g>0$,
i.e., the boson interactions are repulsive.

This paper is organised as follows. In Section II we calculate the gap and
Schwinger-Dyson (SD) equations for a single Bose gas in the improved Hartree-
Fock (IHF) approximation by first recapitulating the regular HF method and
then calculating these expressions for the Cornwall-Jackiw-Tomboulis effective
potential with symmetry-restoring terms, hence the name improved Hartree-Fock
method. The non-condensate fraction and critical temperature are investigated
in Section III and the results are compared with both related approaches and
experimental data. Explicit expressions for the chemical potential, the
pressure and the free-energy density are calculated. Finally, we present the
conclusions and a future outlook in Section IV.

##  II The equations of state in the IHF approximation

In this Section, we will establish the equations of state for a Bose gas,
which are the gap and Schwinger-Dyson (SD) equations. To do so, we first
derive the Cornwall-Jackiw-Tomboulis (CJT) effective potential [16] in the IHF
approximation. As was previously mentioned, the condensed fraction is
investigated at zero temperature in spite of the fact that the CJT effective
action approach is constructed for a finite temperature $T>0$. The temperature
is set to be zero at the end of the calculations.

Let $\psi_{0}$ be the expectation value of the field operator in the tree-
approximation, the GP potential is then taken from (5)

$V_{GP}=-\mu\psi_{0}^{2}+\frac{g}{2}\psi_{0}^{4}.$ (6)

Note that henceforth the system is considered without any external fields.
Furthermore, the non-macroscopic part of the condensate moves as a whole so
that the lowest energy solution $\psi_{0}$ is real and plays the role of the
order parameter. The square of the order parameter $\psi_{0}^{2}$ is the
density of the condensate. Minimizing the potential (6) with respect to the
order parameter, one arrives at the gap equation

$\psi_{0}(-\mu+g\psi_{0}^{2})=0,$ (7)

and hence, for the broken phase

$\psi_{0}^{2}=\frac{\mu}{g}.$ (8)

In order to proceed within the framework of the HF approximation, the complex
field operator $\psi$ should first be decomposed in terms of the order
parameter $\psi_{0}$ and two real fields $\psi_{1}$ and $\psi_{2}$, which are
associated with quantum fluctuations of the field [6], i.e.,

$\psi\rightarrow\psi_{0}+\frac{1}{\sqrt{2}}(\psi_{1}+i\psi_{2}).$ (9)

Plugging equation (9) into the Lagrangian density (5), the interaction
Lagrangian density, which describes the interaction between the real fields,
in the HF approximation is obtained

${\cal
L}_{int}=\frac{g}{2}\psi_{0}\psi_{1}(\psi_{1}^{2}+\psi_{2}^{2})+\frac{g}{8}(\psi_{1}^{2}+\psi_{2}^{2})^{2}.$
(10)

In the tree approximation one finds the gap equation (8) and the propagator or
Green’s function

$D_{0}(k)=\frac{1}{\omega_{n}^{2}+E_{\text{(tree)}}^{2}(k)}\left(\begin{array}[]{cc}\frac{\hbar^{2}k^{2}}{2m}-\mu+g\psi_{0}^{2}&\omega_{n}\\\
-\omega_{n}&\frac{\hbar^{2}k^{2}}{2m}-\mu+3g\psi_{0}^{2}\\\
\end{array}\right),$ (11)

where $\omega_{n}$ is the $n$th Matsubara frequency for bosons. The latter is
defined as $\omega_{n}=2\pi n/\beta$ where $\beta=1/k_{B}T$ and
$n\in{\mathbb{Z}}$ with $k_{B}$ being the Boltzmann constant. The
$E_{\text{(tree)}}(k)$ is the energy spectrum of the elementary excitation,
which is determined by examining poles of the Green’s function [6, 17, 18].
The result is

$E_{\text{(tree)}}(k)=\sqrt{\frac{\hbar^{2}k^{2}}{2m}\left(\frac{\hbar^{2}k^{2}}{2m}+2g\psi_{0}^{2}\right)}.$
(12)

The spectrum Eq. (12) was first achieved by Bogoliubov [10]. For small
momenta, this equation is gapless and linear in $k$ and indicates the
spontaneous $U(1)$ continuous symmetry breaking, creating a Nambu-Goldstone
boson. To continue our discussion, we introduce the CJT effective potential in
the HF approximation that can be constructed from the interaction Lagrangian
density (10) in the manner that was pointed out in [19, 20],

$\begin{split}V_{\beta}^{\text{(CJT)}}=&-\mu\psi_{0}^{2}+\frac{g}{2}\psi_{0}^{4}+\frac{1}{2}\int_{\beta}\mbox{tr}\left[\ln
G^{-1}(k)+D_{0}^{-1}(k)G(k)-{1\\!\\!1}\right]\\\
&+\frac{3g}{8}(P_{11}^{2}+P_{22}^{2})+\frac{g}{4}P_{11}P_{22}\,,\end{split}$
(13)

for which the functions $P_{11}$ and $P_{22}$ are

$P_{11}=\int_{\beta}G_{11}(k)$ (14a) $P_{22}=\int_{\beta}G_{22}(k)$ (14b)

The Matsubara integrals in these expressions are defined as follows

$\int_{\beta}f(k)=\frac{1}{\beta}\sum_{n=-\infty}^{+\infty}\int\frac{d^{3}\vec{k}}{(2\pi)^{3}}f(\omega_{n},\vec{k})\,.$
(15)

Here $G(k)$ is the propagator or Green’s function in the HF approximation,
which can be obtained by minimizing the CJT effective potential (13) with
respect to the elements of the propagator. Performing these calculations
results in the following expression for the inverse propagator

$G^{-1}(k)=D_{0}^{-1}(k)+\Pi,$ (16)

in which

$\Pi=\left(\begin{array}[]{cc}\Pi_{1}&0\\\ 0&\Pi_{2}\\\ \end{array}\right),$
(17)

with the matrix entries $\Pi_{1}$ and $\Pi_{1}$ being the self-energies that
can be constructed from (14a) and (14b), i.e.,

$\Pi_{1}=\frac{3g}{2}P_{11}+\frac{g}{2}P_{22}$ (18a)
$\Pi_{2}=\frac{g}{2}P_{11}+\frac{3g}{2}P_{22}\,.$ (18b)

The gap equation in the HF approximation can now be found by minimizing the
CJT effective potential (13) with respect to the order parameter $\psi_{0}$,
i.e.,

$-\mu+g\psi_{0}^{2}+\Pi_{1}=0.$ (19)

Combining equations (16)-(19), one has the propagator in the HF approximation

$G(k)=\frac{1}{\omega_{n}^{2}+E_{\text{(HF)}}^{2}(k)}\left(\begin{array}[]{cc}\frac{\hbar^{2}k^{2}}{2m}-\mu+g\psi_{0}^{2}+\Pi_{2}&-\omega_{n}\\\
\omega_{n}&\frac{\hbar^{2}k^{2}}{2m}-\mu+3g\psi_{0}^{2}+\Pi_{1}\\\
\end{array}\right),$ (20)

and consequently the dispersion relation in this approximation is

$E_{\text{(HF)}}(k)=\sqrt{\left(\frac{\hbar^{2}k^{2}}{2m}-\mu+3g\psi_{0}^{2}+\Pi_{1}\right)\left(\frac{\hbar^{2}k^{2}}{2m}-\mu+g\psi_{0}^{2}+\Pi_{2}\right)}.$
(21)

The Goldstone theorem [21] requires a gapless excitation spectrum. In the
physics of BEC, this theorem is referred to as the Hugenholtz-Pines theorem
[22] at zero temperature and a more general proof for all value of temperature
was given by Hohenberg and Martin [23]. More discussion about the gapless
excitation spectrum for the BEC at finite temperature were mentioned in Refs.
[24, 25]. Equations (19) and (21) show a non-gapless spectrum, which means
that the Goldstone theorem is violated in the case of spontaneously broken
symmetry within the HF approximation. To restore the Nambu-Goldstone boson, we
now employ the method developed in [26]. In this way, a phenomenological
symmetry-restoring correction term $\Delta V$ need to be added into CJT
effective potential, which has to contemporaneously satisfy three conditions:

(i) it restores the Goldstone boson in the condensed phase;

(ii) it does not change the HF equations for the mean field and;

(iii) the results in the phase of restored symmetry are not changed.
It is not difficult to check that this term has from

$\Delta V=-\frac{g}{4}(P_{11}^{2}+P_{22}^{2})+\frac{g}{2}P_{11}P_{22},$ (22)

Let the propagator in the IHF approximation, be denoted as
$D_{\text{(IHF)}}(k)$, the CJT effective potential (13) now becomes

$\displaystyle\widetilde{V}_{\beta}^{\text{(CJT)}}$ $\displaystyle=$
$\displaystyle-\mu\psi_{0}^{2}+\frac{g}{2}\psi_{0}^{4}+\frac{1}{2}\int_{\beta}\mbox{tr}\left[\ln
D^{-1}_{\text{(IHF)}}(k)+D_{0}^{-1}(k)D_{\text{(IHF)}}(k)-{1\\!\\!1}\right]$
(23)
$\displaystyle+\frac{g}{8}(P_{11}^{2}+P_{22}^{2})+\frac{3g}{4}P_{11}P_{22}.$

From the CJT effective potential in the IHF approximation (23), one arrives at
the gap equation

$-\mu+g\psi_{0}^{2}+\Sigma_{1}=0,$ (24)

and the SD equation

$M=-\mu+3g\psi_{0}^{2}+\Sigma_{2},$ (25)

in which the self-energies $\Sigma_{1}$ and $\Sigma_{2}$ are

$\Sigma_{1}=\frac{3g}{2}P_{11}+\frac{g}{2}P_{22}$ (26a)
$\Sigma_{2}=\frac{g}{2}P_{11}+\frac{3g}{2}P_{22}\,.$ (26b)

Combining equations (23)-(26b), one can once again calculate the propagator,
now in the IHF approximation, i.e.,

$D_{\text{(IHF)}}(k)=\frac{1}{\omega_{n}^{2}+E_{\text{(IHF)}}^{2}(k)}\left(\begin{array}[]{lr}\frac{\hbar^{2}k^{2}}{2m}&\omega_{n}\\\
-\omega_{n}&\frac{\hbar^{2}k^{2}}{2m}+M\\\ \end{array}\right).$ (27)

Hence, the resulting dispersion relation is

$E_{\rm{(IHF)}}(k)=\sqrt{\frac{\hbar^{2}k^{2}}{2m}\left(\frac{\hbar^{2}k^{2}}{2m}+M\right)}.$
(28)

Clearly, the Goldstone boson is restored in this approximation. This is
precisely the reason why this approximation is called the improved Hartree-
Fock approximation. The gap and SD equations (24) and (25), together with the
momentum integrals (30) form the equations of state, which govern the
variation of all quantities of the system.

To do further, we first deal with the momentum integrals in the gap and SD
equations. Without loss of generality, henceforth we use the abbreviation
$E(k)$ replacing for $E_{\rm(IHF)}(k)$. In the IHF approximation, the momentum
integrals are obtained from equations (14b) after replacing $G(k)$ by $D(k)$.
Using the following formula [27],

$\sum_{n=-\infty}^{+\infty}\frac{1}{\omega_{n}^{2}+E^{2}(k)}=\frac{\beta}{2E(k)}\left[1+\frac{2}{e^{\beta
E(k)}-1}\right],$ (29)

one has

$\displaystyle P_{11}$ $\displaystyle=$
$\displaystyle\int\frac{d^{3}\vec{k}}{(2\pi)^{3}}\frac{1}{2E(k)}\frac{\hbar^{2}k^{2}}{2m}+\int\frac{d^{3}\vec{k}}{(2\pi)^{3}}\frac{1}{E(k)}\frac{1}{e^{\beta
E(k)}-1}\frac{\hbar^{2}k^{2}}{2m},$ (30a) $\displaystyle P_{22}$
$\displaystyle=$
$\displaystyle\int\frac{d^{3}\vec{k}}{(2\pi)^{3}}\frac{1}{2E(k)}\left(\frac{\hbar^{2}k^{2}}{2m}+M\right)+\int\frac{d^{3}\vec{k}}{(2\pi)^{3}}\frac{1}{E(k)}\frac{1}{e^{\beta
E(k)}-1}\left(\frac{\hbar^{2}k^{2}}{2m}+M\right).$ (30b)

The first terms in right-hand side of (30a) and (30b) do not explicitly depend
on the temperature. These integrals are ultraviolet divergent. This divergence
are avoidable by means of the dimensional regularization [6] and thus, the
integrals can be computed. The integral $I_{m,n}$ is

$\displaystyle I_{m,n}({\cal M})$ $\displaystyle=$
$\displaystyle\int\frac{d^{d}\kappa}{(2\pi)^{d}}\frac{\kappa^{2m-n}}{(\kappa^{2}+{\cal
M}^{2})^{n/2}}$ (31) $\displaystyle=$
$\displaystyle\frac{\Omega_{d}}{(2\pi)^{d}}\Lambda^{2\epsilon}{\cal
M}^{d+2(m-n)}\frac{\Gamma\left(\frac{d-n}{2}+m\right)\Gamma\left(n-m-\frac{d}{2}\right)}{2\Gamma\left(\frac{n}{2}\right)},$

where $\Gamma(x)$ is the gamma function, $\Omega_{d}=2\pi^{d/2}/\Gamma(d/2)$
is the surface area of a $d-$dimensional sphere and $\Lambda$ is a
renormalization scale that ensures the integral has the correct canonical
dimension. This number is usually absorbed into the measure, hence it will not
appear in the results. Applying (31) to (30) with $d=3$, one finds

$\displaystyle P_{110}$ $\displaystyle\equiv$
$\displaystyle\int\frac{d^{3}\vec{k}}{(2\pi)^{3}}\frac{1}{2E(k)}\frac{\hbar^{2}k^{2}}{2m}=\frac{(2m)^{3/2}M^{3/2}}{6\pi^{2}\hbar^{3}},$
(32a) $\displaystyle P_{220}$ $\displaystyle\equiv$
$\displaystyle\int\frac{d^{3}\vec{k}}{(2\pi)^{3}}\frac{1}{2E(k)}\left(\frac{\hbar^{2}k^{2}}{2m}+M\right)=-\frac{(2m)^{3/2}M^{3/2}}{12\pi^{2}\hbar^{3}}.$
(32b)

Generally, the temperature-dependent integrals (30) cannot be calculated
analytically. We can solve by either numerical method or in some limit cases.
We will now look for the quantum and thermal fluctuations and calculate the
non-condensate fraction of the dilute Bose gas.

##  III The non-condensate fraction of a homogeneous dilute Bose gas

Let us now investigate the non-condensate fraction of the dilute weakly
interacting Bose gas in the IHF approximation. To this end, we first consider
the pressure. The pressure is defined as the negative of the CJT effective
potential (23) at the minimum, i.e. satisfying both the gap and SD equations

${\cal
P}=-\widetilde{V}_{\beta}\bigg{|}_{\mbox{minimum}}\equiv-\widetilde{{\cal
V}}_{\beta}^{\text{(CJT)}}.$ (33)

Now, substituting equations (24) and (25) into (23), one has

$\displaystyle\widetilde{{\cal V}}_{\beta}^{\text{(CJT)}}$ $\displaystyle=$
$\displaystyle-\mu\psi_{0}^{2}+\frac{g}{2}\psi_{0}^{4}+\frac{1}{2}\int_{\beta}\mbox{tr}\ln
D^{-1}_{\text{(IHF)}}(k)$ (34)
$\displaystyle+\frac{1}{2}(3g\psi_{0}^{2}-\mu-M^{2})P_{11}+\frac{1}{2}(g\psi_{0}^{2}-\mu)P_{22}^{2}$
$\displaystyle+\frac{g}{8}(P_{11}^{2}+P_{22}^{2})+\frac{3g}{4}P_{11}P_{22}.$

The chemical potential is defined as first derivative of the pressure with
respect to the condensate density [28]

$\displaystyle\mu=\frac{\partial\cal P}{\partial n}.$ (35)

Plugging (34) into (35) one can find the chemical potential beyond the mean-
field theory

$\mu=gn_{0}+gP_{11}.$ (36)

Combining (36), (24) and (25) leads to the gap and SD equations in higher
precision

$\displaystyle-1+x+\frac{1}{2n_{0}}(P_{11}+P_{22})=0,$ $\displaystyle{\cal
M}=-1+3x-\frac{1}{2n_{0}}(P_{11}-3P_{22}),$ (37)

in which we defined ${\cal M}=M/gn_{0}$ and $x=n_{c}/n_{0}$ is the condensate
fraction.

We now study of high-temperature limit, i.e the temperature is not much lower
than the critical temperature. In this limit $\beta M\ll 1$ and in lowest
order approximation, the temperature-dependent parts (30) of the momentum
integrals are

$\displaystyle
P_{11T}=P_{22T}=\frac{(m/\beta)^{3/2}\zeta(3/2)}{2\sqrt{2}\pi^{3/2}\hbar^{3}}.$
(38)

In sum, at high temperature, the momentum integrals (30) can be expressed

$\displaystyle P_{11}$ $\displaystyle=$
$\displaystyle\frac{(2m)^{3/2}M^{3/2}}{6\pi^{2}\hbar^{3}}+\frac{(m/\beta)^{3/2}\zeta(3/2)}{2\sqrt{2}\pi^{3/2}\hbar^{3}},$
$\displaystyle P_{22}$ $\displaystyle=$
$\displaystyle-\frac{(2m)^{3/2}M^{3/2}}{12\pi^{2}\hbar^{3}}+\frac{(m/\beta)^{3/2}\zeta(3/2)}{2\sqrt{2}\pi^{3/2}\hbar^{3}}.$
(39)

Inserting (39) into (37) one can write the gap and SD equations in
dimensionless form

$\displaystyle-1+x+\frac{2\sqrt{2}}{3\sqrt{\pi}}{\cal
M}^{3/2}n_{s}^{1/2}+\frac{(m/\beta)^{3/2}\zeta(3/2)}{2\sqrt{2}\pi^{3/2}\hbar^{3}n_{0}}=0,$
$\displaystyle{\cal M}=-1+3x-\frac{10\sqrt{2}}{3\sqrt{\pi}}{\cal
M}^{3/2}n_{s}^{1/2}+\frac{(m/\beta)^{3/2}\zeta(3/2)}{2\sqrt{2}\pi^{3/2}\hbar^{3}n_{0}}.$
(40)

The solution for Eqs. (40) approximates

$\displaystyle x$ $\displaystyle\approx$ $\displaystyle
1-\frac{8}{3\sqrt{\pi}}n_{s}^{1/2}-\frac{(m/\beta)^{3/2}\zeta(3/2)}{2\sqrt{2}\pi^{3/2}\hbar^{3}n_{0}},$
(41a) $\displaystyle{\cal M}^{1/2}$ $\displaystyle\approx$
$\displaystyle\sqrt{2}-\frac{16\sqrt{2}}{3\sqrt{\pi}}n_{s}^{1/2}-\frac{(m/\beta)^{3/2}\zeta(3/2)}{4\pi^{3/2}\hbar^{3}n_{0}}.$
(41b)

At the critical temperature, the order parameter vanishes. Therefore, Eq.
(41a) gives

$\displaystyle
T_{C}\approx\frac{2\pi\hbar^{2}}{mk_{B}}\left[\frac{n_{0}}{\zeta(3/2)}\right]^{2/3}-\frac{32\sqrt{\pi}\hbar^{2}}{9mk_{B}}\left[\frac{n_{0}}{\zeta(3/2)}\right]^{2/3}n_{s}^{1/2}.$
(42)

It is interesting to see that, for the ideal Bose gas, the gas parameter is
zero so that Eq. (42) remains only the first term in right-hand side, which
exactly coincides with (3). In case of the dilute Bose gas, the atomic
interaction reduces the critical temperature amount that equals the second
term. From Eq. (42) one has

$\displaystyle\frac{\Delta T_{C}}{T_{C}^{(0)}}=c.n_{0}^{1/3}a_{s}.$ (43)

Evidently, Eq. (43) has the same form with (4) and value of constant
$c=-16/9\sqrt{\pi}\approx-1.033$. This result is in good agreement with
corresponding result of Martin et. al. in Ref. [29], in which the authors used
perturbative solution of the crossover equations in the canonical ensemble and
found $c=-0.93$. By far, there are many authors who calculated the shift of
critical temperature of the dilute Bose gas. Several typical results are
listed in Table 1.

Table 1: Value of constant $c$ in Eq. (43). $c$ | Method
---|---
– 0.93 | Perturbative solution of the crossover equations [29]
2.33 | leading order $1/N$ [30]
1.32 | Lattice simulation of O(2)$\varphi^{4}$ theory [31]
1.23 | Exact renormalization group [32]
1.3 | Simulations of classical field theory [33]
0.34 | Path-integral Monte Carlo simulations and finite-size scaling [34]

Now we consider the non-condensate fraction of the dilute BEC at finite
temperature. Inverting Eq. (35) one easily see that the condensate density in
the IHF approximation from differentiating the pressure with respect to the
chemical potential, i.e.,

$n=\frac{\partial{\cal P}}{\partial\mu}.$ (44)

Combining equations (33)-(44), the condensate density is expressed in terms of
the order parameter $\psi_{0}$ and the momentum integrals $P_{11}$ and
$P_{22}$ [20],

$n=\psi_{0}^{2}+\frac{1}{2}(P_{11}+P_{22}).$ (45)

The density of the depletion is defined as the number of particles in excited
states per unit volume [3]. Based on equation (45) one can easily see that the
non-condensate fraction for a homogeneous dilute Bose gas is

$n_{\rm ex}=\frac{1}{2}(P_{11}+P_{22}).$ (46)

Substituting Eqs. (41b) into (39) and then (46) one has

$\displaystyle\frac{n_{\rm
ex}}{n_{0}}\approx\frac{8}{3\sqrt{\pi}}n_{s}^{1/2}+\frac{m^{3/2}\zeta(3/2)}{2\sqrt{2}\pi^{3/2}\hbar^{3}n_{0}}(k_{B}T)^{3/2}-\frac{\sqrt{2}m^{3/2}\zeta(3/2)}{\pi^{2}\hbar^{3}n_{0}}(k_{B}T)^{3/2}n_{s}^{1/2}.$
(47)

There are some important things can be read of from (47): (i) At zero
temperature, right-hand side only remains the first term, which associated
with the quantum fluctuations on top of the ground state. This term was found
by many authors [10, 12, 35], particularly was confirmed in experiment [36];
(ii) in case of the ideal Bose gas, the parameter vanishes, right-hand side
only contains the second term, which the thermal fluctuations of the ideal
Bose gas at finite temperature. This result is in agreement with that of other
authors [cf. Eq. (2.30) in Ref. [3]]. It is also shows that at zero
temperature, quantum fluctuations do not appear in the ideal Bose gas [19];
(iii) The non-condensate fraction of the dilute interacting Bose gas contains
an interferential term, which includes both thermal excitations and the
interaction-induced quantum fluctuations.

##  IV Conclusion and outlook

Beyond the standard Bolgoliubov theory, we have explored the CJT effective
action approach to investigate the properties at finite temperature of
homogeneous dilute interacting Bose gas. After deriving the CJT effective
potential in the IHF approximation, where the Goldstone theorem is satisfied,
the equations of state are obtained. Based on these equations, the important
properties of the dilute Bose gas, namely, the critical temperature and non-
condensate fraction have been studied. Two main results are in order:

\- The critical temperature is found in two-terms: one corresponds to that of
the ideal Bose gas and the other depends on the strength of atomic
interaction. The presence of the atomic interaction makes a shift in the
critical temperature in compared with the one of the ideal Bose gas. This
shift can be expressed in a universal form (4).

\- The non-condensate fraction can be separated into three kind terms: first
one is pure quantum fluctuations, second one described the thermal
fluctuations and the last one includes both the quantum and thermal
fluctuations.

It is also very interesting to explore these calculations for Bose-Einstein
condensate mixtures. The interspecies interaction is expected to produce some
novel results, especially to compare with relevant result in [37].
Furthermore, this procedure can also be employed to investigate the condensed
fraction of a Bose gas confined between two parallel plates and the resulting
critical temperature and non-condensate fraction [15].

## Conflict of interest

All of the authors declare that we have no conflict of interest.

## References

  * Bose [1924] S. N. Bose, Plancks Gesetz und Lichtquantenhypothese, Z. Phys. 26, 178–181 (1924).
  * Einstein [1924] A. Einstein, Quantentheorie des einatomigen idealen Gases, Sitz. Ber. Preuss. Akad. Wiss. 22, 261 (1924).
  * Pethick and Smith [2008] C. J. Pethick and H. Smith,  _Bose–Einstein Condensation in Dilute Gases_, 2nd ed. (Cambridge University Press, 2008).
  * Lieb and Liniger [1963] E. H. Lieb and W. Liniger, Exact analysis of an interacting bose gas. i. the general solution and the ground state, Physical Review 130, 1605 (1963).
  * Lieb [1963] E. H. Lieb, Exact analysis of an interacting bose gas. ii. the excitation spectrum, Physical Review 130, 1616 (1963).
  * Andersen [2004] J. O. Andersen, Theory of the weakly interacting Bose gas, Rev. Mod. Phys. 76, 599 (2004).
  * Pitaevskii and Stringari [2003] L. P. Pitaevskii and S. Stringari,  _Bose-Einstein Condensation_ (Oxford University Press, 2003).
  * Gross [1961] E. P. Gross, Structure of a quantized vortex in boson systems, Nuovo Cim. 20, 454–477 (1961).
  * Pitaevskii [1961] L. P. Pitaevskii, Vortex lines in an imperfect Bose gas, Sov. Phys. JETP 13, 451 (1961).
  * Bogolyubov [1947] N. N. Bogolyubov, On the theory of superfluidity, J. Phys. (USSR) 11, 23 (1947).
  * Dalfovo  _et al._ [1997] F. Dalfovo, S. Giorgini, M. Guilleumas, L. Pitaevskii, and S. Stringari, Collective and single-particle excitations of a trapped Bose gas, Phys. Rev. A 56, 3840 (1997).
  * Stringari [2018] S. Stringari, Quantum fluctuations and Gross-Pitaevskii theory, J. Expr. Theor. Phys. 127, 844 (2018).
  * Carlen  _et al._ [2021a] E. A. Carlen, I. Jauslin, and E. H. Lieb, Analysis of a simple equation for the ground state of the bose gas ii: Monotonicity, convexity, and condensate fraction, SIAM Journal on Mathematical Analysis 53, 5322 (2021a), https://doi.org/10.1137/20M1376820 .
  * Van Thu and Berx [2022] N. Van Thu and J. Berx, The condensed fraction of a homogeneous dilute bose gas within the improved hartree–fock approximation, Journal of Statistical Physics 188, 10.1007/s10955-022-02944-0 (2022).
  * Thu [2023] N. V. Thu, Non-condensate fraction of a weakly interacting bose gas confined between two parallel plates within improved hartree-fock approximation at zero temperature, Physics Letters A 486, 129099 (2023).
  * Cornwall  _et al._ [1974] J. M. Cornwall, R. Jackiw, and E. Tomboulis, Effective Action for Composite Operators, Phys. Rev. D 10, 2428 (1974).
  * Orland [1998] J. W. N. H. Orland,  _Quantum Many-Particle Systems_ , edited by D. Pines (Westview Press, 1998).
  * Shi and Griffin [1998] H. Shi and A. Griffin, Finite-temperature excitations in a dilute bose-condensed gas, Physics Reports 304, 1 (1998).
  * Thu and Song [2020] N. V. Thu and P. T. Song, Casimir effect in a weakly interacting Bose gas confined by a parallel plate geometry in improved Hartree–Fock approximation, Physica A 540, 123018 (2020).
  * Phat  _et al._ [2009] T. H. Phat, L. V. Hoa, N. T. Anh, and N. Van Long, Bose–einstein condensation in binary mixture of Bose gases, Ann. Phys. 324, 2074 (2009).
  * Goldstone [1961] J. Goldstone, Field Theories with Superconductor Solutions, Nuovo Cim. 19, 154 (1961).
  * Hugenholtz and Pines [1959] N. M. Hugenholtz and D. Pines, Ground-state energy and excitation spectrum of a system of interacting bosons, Phys. Rev. 116, 489 (1959).
  * Hohenberg and Martin [1965] P. Hohenberg and P. Martin, Microscopic theory of superfluid helium, Annals of Physics 34, 291 (1965).
  * Hutchinson  _et al._ [1998] D. A. W. Hutchinson, R. J. Dodd, and K. Burnett, Gapless finite- $\mathit{T}$ theory of collective modes of a trapped gas, Phys. Rev. Lett. 81, 2198 (1998).
  * Griffin [1996] A. Griffin, Conserving and gapless approximations for an inhomogeneous bose gas at finite temperatures, Phys. Rev. B 53, 9341 (1996).
  * Ivanov  _et al._ [2005] Y. B. Ivanov, F. Riek, and J. Knoll, Gapless Hartree-Fock resummation scheme for the $O(N)$ model, Phys. Rev. D 71, 105016 (2005).
  * Schmitt [2010] A. Schmitt,  _Dense matter in compact stars_ (Springer-Verlag Berlin Heidelberg, 2010).
  * Landau and Lifshitz [1980] L. D. Landau and E. M. Lifshitz,  _Statistical Physics_ (Pergamon Press, Oxford, 1980).
  * Wilkens  _et al._ [2000] M. Wilkens, F. Illuminati, and M. Krämer, Transition temperature of the weakly interacting bose gas: perturbative solution of the crossover equations in the canonical ensemble, Journal of Physics B: Atomic, Molecular and Optical Physics 33, L779 (2000).
  * Baym  _et al._ [2000] G. Baym, J.-P. Blaizot, and J. Zinn-Justin, The transition temperature of the dilute interacting bose gas for n internal states, Europhysics Letters (EPL) 49, 150 (2000).
  * Arnold and Moore [2001] P. Arnold and G. Moore, Bec transition temperature of a dilute homogeneous imperfect bose gas, Physical Review Letters 87, 120401 (2001).
  * Ledowski  _et al._ [2004] S. Ledowski, N. Hasselmann, and P. Kopietz, Self-energy and critical temperature of weakly interacting bosons, Physical Review A 69, 061601 (2004).
  * Davis and Morgan [2003] M. J. Davis and S. A. Morgan, Microcanonical temperature for a classical field: Application to bose-einstein condensation, Physical Review A 68, 053615 (2003).
  * Grüter  _et al._ [1997] P. Grüter, D. Ceperley, and F. Laloë, Critical temperature of bose-einstein condensation of hard-sphere gases, Physical Review Letters 79, 3549 (1997).
  * Carlen  _et al._ [2021b] E. A. Carlen, M. Holzmann, I. Jauslin, and E. H. Lieb, Simplified approach to the repulsive bose gas from low to high densities and its numerical accuracy, Phys. Rev. A 103, 053309 (2021b).
  * Lopes  _et al._ [2017] R. Lopes, C. Eigen, N. Navon, D. Clément, R. P. Smith, and Z. Hadzibabic, Quantum depletion of a homogeneous Bose-Einstein condensate, Phys. Rev. Lett. 119, 190404 (2017).
  * Shi  _et al._ [2000] H. Shi, W.-M. Zheng, and S.-T. Chui, Phase separation of bose gases at finite temperature, Physical Review A 61, 063613 (2000).


# Task-Optimized Adapters
for an End-to-End Task-Oriented Dialogue System

Namo Bang∗

&Jeehyun Lee∗
Department of Artificial Intelligence, Sogang University, Korea
{namo950815, jhlee22<EMAIL_ADDRESS>

&Myoung-Wan Koo

###### Abstract

Task-Oriented Dialogue (TOD) systems are designed to carry out specific tasks
by tracking dialogue states and generating appropriate responses to help users
achieve defined goals. Recently, end-to-end dialogue models pre-trained based
on large datasets have shown promising performance in the conversational
system. However, they share the same parameters to train tasks of the dialogue
system (NLU, DST, NLG), so debugging each task is challenging. Also, they
require a lot of effort to fine-tune large parameters to create a task-
oriented chatbot, making it difficult for non-experts to handle. Therefore, we
intend to train relatively lightweight and fast models compared to PLM. In
this paper, we propose an End-to-end TOD system with Task-Optimized Adapters
which learn independently per task, adding only small number of parameters
after fixed layers of pre-trained network. We also enhance the performance of
the DST and NLG modules through reinforcement learning, overcoming the
learning curve that has lacked at the adapter learning and enabling the
natural and consistent response generation that is appropriate for the goal.
Our method is a model-agnostic approach and does not require prompt-tuning as
only input data without a prompt. As results of the experiment, our method
shows competitive performance on the MultiWOZ benchmark compared to the
existing end-to-end models. In particular, we attain state-of-the-art
performance on the DST task of 2.2 dataset.111Our code is available at
https://github.com/sogang-isds/TOATOD.git

**footnotetext: These authors contributed equally to this work. Figure 1:
Overview of the Task-Optimized Adapters for an End-to-End Task-Oriented
Dialogue System

##  1 Introduction

Task-oriented dialogue systems are trained to achieve specific goal to enhance
efficiency and convenience in various fields such as customer service centers
and healthcare information retrieval. Task-oriented dialogue systems are
divided into key components: understanding the user’s intent (NLU), tracking
the current dialogue states (DST), and generating responses based on previous
sessions (NLG). Pipeline-based systems separately train each component, so
they have the advantage of optimizing each module and raising the performance
of a given task. User feedback is, however, difficult to propagate to each
module, and inputs to the component is dependent on the result of the previous
module (Chen et al., 2017). Recently, dialogue systems have been trained in an
end-to-end manner with transfer learning or pre-training networks with large
dialogue corpora. However, building efficient end-to-end TOD systems requires
a large amount of data and has some limitations due to parameter sharing. End-
to-end models backpropagate to transfer the gradients of the output and end
back to the entire neural network. They pose an issue of parameter efficiency
as updating all parameters for every downstream scenario. Also, it is
challenging to debug each task and take task-flow characteristics into
account.

Therefore, we propose a simple structure of adding adapters to the core
modules (NLU, DST, NLG) of the TOD system as shown in Figure 1. By using the
adapters, it is possible to optimize each task with only a small amount of
training parameters, remaining the pre-trained model’s parameters fixed.
Additionally, it is safe from the catastrophic forgetting problem (French,
1999), which causes pre-trained models to lose important skills acquired
during the pre-training process. The key is to apply a transfer learning
strategy that yields compact and extensible downstream models in the dialogue
system (Houlsby et al., 2019). This makes it easy for people to train large-
scale end-to-end TOD models. Also, by applying REINFORCE (Sutton et al.,
1999), we attempt to reduce the expected score gap caused by the small
parameters of adapter compared to the full fine-tuning. Specifically, we use
Joint Goal Accuracy, and weighted sum of BLEU and Success rate as rewards for
training DST and NLG adapter. To best of our knowledge, this is first work
that use Joint Goal Accuracy as a reward for E2E TOD system.

To address the aforementioned problems, we propose a Task-Optimized Adapter
for an end-to-end Task-Oriented Dialogue system (TOATOD) applying
reinforcement learning to DST and NLG tasks. In summary, our key contributions
are as follows:

  * •

We present a new architecture that can debug per task of the end-to-end model
using the separated adapters.

  * •

Without updating the original parameters of PLM, we train end-to-end TOD
system efficiently with a few trainable parameters.

  * •

It is a novel approach to design a reward function not just for NLG, but also
for DST task with metric-aware reinforcement learning.

  * •

The performance of the proposed approach outperforms on the DST task of
MultiWOZ 2.2 and shows comparable results to full fine-tuning on the NLU and
NLG tasks.

##  2 Background

Pipeline-based Task-Oriented Dialogue System Conventional task-oriented
dialogue systems usually based on the pipeline method, consisted of language
understanding (NLU), dialogue state tracking (DST), policy learning (POL), and
language generation (NLG). This kind of modularization allows for each
component to be optimized independently, making it easier to update and
understand how the model is working. Pipeline-based systems, however, have
several limitations. Each of the modules train sequentially, so it is hard to
align modules to the common optimization targets (Liu and Lane, 2018). This
makes the system more complex and harder to backpropagate cumulated errors.
The performance of the previous components affects the next modules, so if
upper modules perform poorly, errors that occurred earlier may propagate and
be amplified in downstream components (Liu and Lane, 2018).

End-to-end Task-Oriented Dialogue System End-to-end task-oriented dialogue
systems, on the other hand, are easier to optimize and train to directly map
the input to output in a single model. They can leverage large amounts of data
for robust learning and the entire system can be optimized under end-to-end
settings, which leads to better performance. A general approach for building
end-to-end systems is to fine-tune pre-trained language models (Budzianowski
and Vulić, 2019). This approach utilizes the strength of pre-trained networks,
which help the models to leverage the pre-trained knowledge while also
adapting to task-specific data. For example, SimpleTOD (Hosseini-Asl et al.,
2020) solved task-oriented dialogue as causal language modeling task using
several versions of GPT (Radford and Narasimhan, 2018; Radford et al., ).

Pre-training of Dialogue Language Model Recently, methods with pre-training
dialogue language model (Wu et al., 2020; Zhang et al., 2020b; Peng et al.,
2021; Su et al., 2022; He et al., 2022a), instead of fine-tuning pre-trained
networks have outperformed the previous baselines on the benchmark. For
instance, SPACE-3 (He et al., 2022a) captures the contextualized knowledge
from large-scale dialogue corpora by pre-training the unified language model.
However, there are still some issues with these methods. A large amount of
parameters is required for training backbone models like BERT, T5 (Raffel et
al., 2020), GPT, and UniLM (Dong et al., 2019). As shown in Table 1,
$\text{T5}_{base}$ and $\text{T5}_{small}$ require the trainable parameters
over 220M and 60M respectively. And they disregard the task-flow features of
task-oriented dialogue systems. Also, it’s still hard to debug per module
because the model parameters are shared and jointly optimized. PPTOD (Su et
al., 2022) integrated modules into a unified model with task-specific prompts
and alleviated the error accumulation in plug-and-play way. Still, this method
is not completely free from the interference among tasks due to fully shared
parameters.

Figure 2: Architecture of the sub-modules in TOATOD. A red line indicates the
forward path during DST inference. The DB result, based on the output of DST
inference, is used for NLG inference sequentially. All of the adapter layers
(NLU, DST, NLG) share the same architecture.

Adapter tuning for NLP Since the pre-trained models for NLP tasks have become
mainstream, they are mainly used for transfer learning downstream tasks.
However, a parameter efficiency issue has been raised because updating all PLM
parameters is expensive for every downstream scenario. To address the issue,
the adapter module is proposed to transfer PLM like BERT with parameter-
efficient tuning (Houlsby et al., 2019) and shows comparable performance to
full fine-tuning. The main idea behind the adapter is to train the network to
the downstream task with task-specific parameters while maintaining the
original pre-trained parameters. The module is composed of two feed-forward
layers and a non-linear layer, which can be inserted into the transformer
blocks of an end-to-end model. It projects $d$-dimensional input features into
a smaller dimension $m$, and then projects back into the original dimension,
so the total of parameters add per layer with biases is $2md+d+m$. The number
of additional parameters per task can be restricted by setting $m<d$ (Li et
al., 2021). In most of the previous research, the adapter was used for
efficient learning. But in this approach, we adapt the adapter modules not
just for efficient learning (Stickland and Murray, 2019), but also task-
optimized learning as recent studies that have mainly focused on separating
parameters (Lin et al., 2021; Feng et al., 2022; Bapna and Firat, 2019).

Reinforcement learning for Text Generation Although token-based supervised
learning is a widely adopted training method in text generation tasks, as
highlighted by Ranzato et al., 2016, there are two major problems associated
with this approach. The first problem is the exposure bias problem, where
during training, the model is exposed to the ground-truth outputs, thus
allowing it to learn to generate text that is similar to the training data.
However, during evaluation, the model is not exposed to the ground-truth
outputs and instead, generates texts based on the previous words generated by
the model itself. This can lead to errors and a significant deviation in the
generated text from the training data. The second problem is the token-level
loss problem, where the training objective function is based solely on
individual words’ prediction, neglecting the predicted token sequence’s
overall coherence and fluency.

To address these problems, researchers have applied reinforcement learning to
text generation tasks (Ranzato et al., 2016; Li et al., 2016; Paulus et al.,
2018; Chen et al., 2020; Wang et al., 2021; Ye et al., 2022) using metrics
such as BLEU (Papineni et al., 2002) or ROUGE (Lin, 2004) as rewards for
sequence-level training. In our study, we also use the REINFORCE method and
task-oriented dialogue metrics to continually train our model after supervised
learning, not only to address these challenges but also to mitigate
performance degradation caused by the use of an Adapter.

| Pre-trained | Trainable per task
---|---|---
Model | $\text{T5}_{base}$ | $\text{T5}_{small}$ | $\text{TOATOD}_{base}$ | $\text{TOATOD}_{small}$
# of Prams | 220M | 60M | 36M (14%) | 7.9M (12%)
Table 1: This table shows the size of the pre-trained T5 model (frozen shared
parameters) and trainable Adapter per task in our models. We do not experiment
with a large model, because the trainable parameter size of
$\text{TOATOD}_{large}$ is bigger than $\text{T5}_{base}.$

##  3 Methodology

###  3.1 Adapter for each task (NLU, DST, NLG)

The Adapter module is designed to adapt the pre-trained network for each of
the tasks (NLU, DST, NLG) in a task-oriented dialogue system. As illustrated
in Figure 2, the adapters are inserted after the feed-forward layer following
the multi-head attention layer of the transformer blocks of the end-to-end
model. It enables the model to learn task-specific representations while
preserving the shared parameters learned during pre-training. As described in
Table 1, our model consists of large-size frozen shared parameters per task
and small size of trainable parameters that account for about 14% of the
entire network. While original network’s parameters are frozen, the $j^{th}$
adapter of task $i\in$ {NLU, DST, NLG}, $A_{ij}$ computes as below:

${A}_{ij}=LN\ (W_{up}*\ ReLU\ \left(W_{down}*\ H_{j}\right)+\ H_{j})$ (1)

The output of the $j^{th}$ feed-forward layer with residual connection in the
transformer block is represented as $H_{j}\in\text{R}^{n\times d}$, where n is
the input dimension, and d is the hidden dimension. As shown in Figure 2, the
overall architecture of the adapter module includes multiple feed-forward
projections, referred to as down-projection and up-projection, followed by
layer normalization (LN). The down-projection with
$W_{down}\in\text{R}^{d\times h}$ projects the input $H_{j}$, which is passed
by the ReLU activation function, and then the up-projection with
$W_{up}\in\text{R}^{h\times d}$ projects the output back to the original
dimension. The bottleneck dimension h is a hyperparameter to project the
original input to a smaller dimension. And each adapter has a residual
connection to avoid vanishing gradient (Rebuffi et al., 2017; He et al.,
2016).

###  3.2 Metric-Aware Reinforcement Learning for DST & NLG module

The overall loss function for the metric-aware reinforcement learning is given
by the equation (2):

$J\left(\theta\right)=\alpha\times
J_{policy}\left(\theta\right)+\left(1-\alpha\right)\times
CE\left({y,\hat{y}}\right)$ (2)

The equation describes how the network updates the parameters in order to
maximize both the likelihood of the generated response (token loss) and the
quality of the response (policy loss). The token loss can be described as a
categorical cross-entropy loss. $\hat{y}$ denotes the predicted probability of
ground truth and $y$ is the target probability. The token loss
$CE(y,\hat{y})$, which measures how well a set of predicted token
probabilities match the target tokens for a given context of dialogue when
reinforcement learning. By applying REINFORCE method, the network can update
weights towards the direction that allows the model for getting more rewards
even when the reward function is non-differentiable.

$J_{policy}\left(\theta\right)={{-\ log}P({\hat{y}})\times Reward(y,\
\hat{y})\ }$ (3)

The policy loss $J_{policy}(\theta)$ is introduced to measure of how well the
model can generate a token sequence with high probabilities that result in
high rewards. $\hat{y}$ denotes the predicted token sequence by model. The
policy loss described in the equation (3) is calculated as the negative log
probability of the token sequence that has the highest probability multiplied
by the reward.

The parameter $\alpha$ in the overall loss (2) is a hyperparameter, that is a
scalar value between 0 and 1 to weigh the importance of these losses. In this
way, the model is trained to predict the correct labels (categorical cross-
entropy loss) and to make good decisions that result in high rewards (policy
loss) at the same time. We define the reward functions of the DST and NLG
modules as follows:

${Reward}_{DST}=\ JGA(y,\ \hat{y})\ +\ 1$ (4)

The $Reward_{DST}$ is calculated as the sum of the Joint Goal Accuracy (JGA)
and a constant value of 1. The JGA measures how well the model predicted the
values for every slot in the dialogue turns. Using JGA as a reward, the model
is encouraged to accurately track the state of the dialogue, which is crucial
for generating appropriate responses and improving the performance of task-
oriented dialogue systems.

$\displaystyle{Reward}_{NLG\ }=$
$\displaystyle(1-\beta)\times\mathbb{E}[BLEU(y_{u},\ \hat{y_{u}})]$
$\displaystyle+\ \beta\times Success(y,\hat{y}\ )\ +\ 1$ (5)

In the equation (5), BLEU score and Success rate are used as rewards to guide
the learning process of NLG module. $y_{u}$ denotes the ground truth token
sequence of each utterance, and $\hat{y}_{u}$ denotes the predicted token
sequence. To calculate the success rate, we apply the batch of a session-
level, not an utterance-level. $y$ and $\hat{y}$ without $u$ mean the session
level ground truth and prediction. The weighting factor $\beta$ is adjusted to
balance these two metrics. The hyperparameter $\beta$ may have to be carefully
chosen because there is a trade-off relationship between these two metrics in
the RL setting (Wu et al., 2021), where increasing one metric may come at the
cost of decreasing the other. We experiment to choose $\alpha$ and $\beta$ on
the Section 6.1.3 and 6.1.4.

##  4 Experimental Setup

###  4.1 Datasets

We experiment our method for dialogue state tracking (DST) and end-to-end
response generation (NLG) tasks on the MultiWOZ 2.1 and 2.2 datasets, and the
intent prediction (NLU) task with the Banking77, CLINC150, and HWU64 datasets.

MultiWOZ - 2.1, 2.2 The MultiWOZ (Budzianowski et al., 2018) dataset has been
widely used to evaluate the performance of TOD systems. It consists of 8438,
1000, and 1000 for training, dev, test sets with multi-turn dialogues,
collected through a Wizard-of-Oz (WOZ) setup. The dialogues cover a wide range
of domains and topics. MultiWOZ 2.2 (Zang et al., 2020) is the improved
version of MultiWOZ 2.1 (Eric et al., 2020) that has corrected annotation
errors, inconsistencies, and ontology issues, also added span annotations to
standardize.

Banking 77 (Casanueva et al., 2020) This dataset is a collection of 77 real-
life customer banking service queries. It consists of 13,083 utterances. Each
query is labeled with a single intent, however, it is hard to differentiate
because they correspond to very similar tasks.

CLINC150 (Larson et al., 2019) This dataset is multi-domain dataset which
contains 23,700 utterances that cover 150 intent classes over 10 domains.

HWU64 (Liu et al., 2019) This dataset consists of 25,716 examples. It maps
user utterances to structured, but mode abstract. The data provides annotation
with the 64 intents from 21 different domains.

###  4.2 Baselines & Settings

Model |  | Backbone Model
---
(Trainable Prams)
| MultiWOZ
---
2.1
| MultiWOZ
---
2.2
TRADE | - | 46.0 | 45.4 |
DS-DST |  ${\text{BERT}}_{base}$ (110M) | 51.2 | 51.7 |
DST-Picklist |  ${\text{BERT}}_{base}$ (110M) | 53.3 |  |
TripPy |  ${\text{BERT}}_{base}$ (110M) | 55.3 |  |
| ConvBERT
---
+DG+Multi
${\text{BERT}}_{base}$ (110M) | 58.7 |  |
| Trippy
---
+SaCLog
${\text{BERT}}_{base}$ (110M) | 60.61 |  |
SimpleTOD |  DistilGPT-2 (82M) | 56.45 |  |
UniLM |  UniLM (340M) | 54.25* | 54.25* |
AG-DST |  PLATO-2 (310M) | 57.26 | 57.26 |
SPACE-3 |  UniLM (340M) | 57.50 | 57.50 |
$\text{D3ST}_{base}$ |  ${\text{T5}}_{base}$ (220M) | 54.2 | 56.1 |
$\text{D3ST}_{large}$ |  ${\text{T5}}_{large}$ (770M) | 54.5 | 54.2 |
SDP-DST |  ${\text{T5}}_{base}$ (220M) | 56.66 | 57.60 |
$\text{PPTOD}_{base}$ |  ${\text{T5}}_{base}$ (220M) | 57.10 |  |
$\text{PPTOD}_{large}$ |  ${\text{T5}}_{large}$ (770M) | 57.45 |  |
$\text{D3ST}_{\text{XXL}}$ |  ${\text{T5}}_{\text{XXL}}$ (11B) | 57.80 | 58.7 |
$\text{TOATOD}_{small}$ |  ${\text{T5}}_{small}$ (7.9M) | 53.02 | 61.92 |
TOATODbase |  ${\text{T5}}_{base}$ (36M) | 54.97 | 63.79 |

Table 2: Joint Goal Accuracy for DST results. We use the best models after
hyperparameter tuning and applying REINFORCE. Our models get the slightly
higher score on the 2.2 dataset. Trainable Params denotes the number of
trainable parameters of the network. The values with * are from SPACE-3 (He et
al., 2022a).

Model |  | Backbone Model
---
(Trainable Params)
MultiWOZ 2.1 | MultiWOZ 2.2
|  | Inform | Success | BLEU | Combined | Inform | Success | BLEU | Combined
DoTS |  ${\text{BERT}}_{base}$ (110M) | 86.65 | 74.18 | 15.90 | 96.32 | - | - | - | -
PPTOD |  ${\text{T5}}_{base}$ (220M) | 87.09 | 79.08 | 19.17 | 102.26 | - | - | - | -
UBAR |  GPT2 (1.5B) | 95.70 | 81.80 | 16.50 | 104.94 | 83.4 | 70.3 | 17.6 | 94.4
MTTOD |  ${\text{T5}}_{base}$ (360.9M)* | 90.99 | 82.08 | 19.68 | 106.22 | 85.9 | 76.5 | 19.0 | 100.2
RSTOD |  ${\text{T5}}_{small}$ (105.5M)* | 93.50* | 84.70* | 19.24* | 108.34* | 83.5 | 75.0 | 18.0 | 97.3
GALAXY |  UniLM (340M) | 95.30 | 86.20 | 20.01 | 110.76 | 85.4 | 75.7 | 19.64 | 100.2
MinTL |  ${\text{BART}}_{large}$ (440M) | - | - | - | - | 73.7 | 65.4 | 19.4 | 89.0
SOLOIST |  GPT2 (1.5B) | - | - | - | - | 82.3 | 72.4 | 13.6 | 90.9
BORT |  ${\text{T5}}_{small}$ (60M) | - | - | - | - | 85.5 | 77.4 | 17.9 | 99.4
Mars |  ${\text{T5}}_{small}$ (60M) | - | - | - | - | 88.9 | 78.0 | 19.9 | 103.4
${\text{TOATOD}}_{small}$ |  ${\text{T5}}_{small}$ (7.9M) | 92.10 | 80.40 | 18.29 | 104.54 | 85.80 | 74.00 | 18.00 | 97.90
${\text{TOATOD}}_{base}$ |  ${\text{T5}}_{base}$ (36M) | 97.00 | 87.40 | 17.12 | 109.32 | 90.00 | 79.80 | 17.04 | 101.94

Table 3: Inform, Success, BLEU, Combined Score for NLG results. All results of
other models are cited from the official leaderboard. The values with * are
from RSTOD. For the MultiWOZ 2.2 evaluation, we used our models trained on the
MultiWOZ 2.1 after replacing the DST-optimized adapter with those trained on
MultiWOZ 2.2.

For the DST task, we compare our models with other strong baselines including
TRADE (Wu et al., 2019), DS-DST (Zhang et al., 2020a), DST-Picklist (Zhang et
al., 2020a), TripPy (Heck et al., 2020), ConvBERT+DG+Multi (Mehri et al.,
2020a), TripPy+SaCLog (Dai et al., 2021), SimpleTOD (Hosseini-Asl et al.,
2020), AG-DST (Tian et al., 2021), UniLM, SPACE-3, D3ST (Zhao et al., 2022),
SDP-DST (Lee et al., 2021), and PPTOD. On the NLG task, we choose models
trained on PLM in an end-to-end setting such as DoTS (Jeon and Lee, 2021),
PPTOD, UBAR (Yang et al., 2021), MTTOD (Lee, 2021), RSTOD (Cholakov and Kolev,
2022), GALAXY (He et al., 2022b), MinTL (Lin et al., 2020), SOLOLIST, BORT
(Sun et al., 2022a) and Mars (Sun et al., 2022b). For the NLU task, we compare
TOATOD with existing baselines of each dataset.

We use ${\text{T5}}_{small}$ and ${\text{T5}}_{base}$ as backbone models
initialized with PPTOD (Su et al., 2022)’s pre-trained weights, which are
trained on a large dialogue dataset. As described in the Appendix A, the
bottleneck dimension h of the adapter is 1/2 of the hidden dimension of the T5
model. We use the Adafactor (Shazeer and Stern, 2018) optimizer with 15 epochs
and set batch size as 16, learning rate of 1e-4 during supervised learning of
DST and NLG tasks. We sweep a wide range of learning rates: ${\\{}$1e-5, 1e-6,
1e-7${\\}}$. For reinforcement learning, we train models 10 epochs for DST and
3 epochs for NLG. We do not train NLG-optimized adapters for the MultiWOZ 2.2,
because there are not significant changes of response annotations from
MultiWOZ 2.1 and intend to train robust models with noised dataset.

We follow the preprocessing method from UBAR to delexicalize slot values for
each system responses. We evaluate our models using the older version of the
standardized evaluation script for MultiWOZ 2.1, and the newly opened version
for MultiWOZ 2.2, released by Nekvinda and Dusek, 2021. It has been adopted by
the official MultiWoZ dataset github
222https://github.com/budzianowski/multiwoz. Other implementation details are
described in Appendix C.

##  5 Experimental Results

###  5.1 Dialogue State Tracking

We evaluate our models on the DST task with the MultiWOZ 2.1 & 2.2 datasets.
We compute Joint Goal Accuracy on the test set, which measures how many values
are filled accurately compared to the ground truth states for all slots. Joint
Goal Accuracy is considered as more difficult and important metric in most
research (Zhou and Small, 2019; Dey et al., 2022), because once wrong
prediction has been made, it cannot get the score at that turn.

####  5.1.1 Evaluation Result

We compare our best models, $\text{TOATOD}_{small}$ and $\text{TOATOD}_{base}$
to the models trained with a pre-trained network. Table 2 shows that our
models are competitive to the end-to-end models on the current benchmark. In
the 2.1 dataset, our models show a relatively good performance, despite the
small number of trainable parameters. As shown in the Table 5, the Joint Goal
Accuracy of $\text{TOATOD}_{base}$ only with task-optimized adapter (SL) is
53.33, which is slightly lower than other models using ${\text{T5}}_{base}$ as
backbone. So, we reduce the performance degradation applying metric-aware
REINFORCE and report the final results. Trainable parameters of our models are
less than 1/2 of the models with the smallest parameters. The result implies
that the adapter module helps the network more adaptable to the DST task by
only activating a few parts of the model. Because of relatively small
parameters, our model is more robust to overfitting problem with confused
labels. As mentioned in Section 4.1, MultiWOZ 2.2 is the cleaned version of
2.1 dataset, so the performance is better in the 2.2. Among the top results on
the 2.2 dataset, our models obtain state-of-the-art performance. It
demonstrates that TOATOD optimizes well on the given task remaining the prior
knowledge learned from the pre-trained network.

###  5.2 End-to-End Response Generation

We test our methods with end-to-end response generation (NLG) task on the
MultiWOZ 2.1 & 2.2 as in DST evaluation. Four metrics are used to measure the
quality of generated responses. We measure if the system provides the
appropriate entity (Inform rate), answers all the requested information
(Success rate), and responds fluently (BLEU score). And the Combined Score for
end-to-end response generation is computed as ’BLEU+0.5 $\times$
(Inform+Success)’. Under the end-to-end settings, the models have to predict
proper dialogue states and then generate responses based on the states.

####  5.2.1 Evaluation Result

From Table 3, our models achieve comparable results (1.x point different from
SOTA model) in all datasets. The adapter module helps each part of the model
be fine-tuned independently, therefore we can optimize the DST and NLG task
respectively. Our model performs well on NLG task based on the belief states
from DST modules and base knowledge gained from the large-scale dialog dataset
during pre-training. $\text{TOATOD}_{base}$ attains the best score on the
Inform and Success rate. It shows that the reinforcement learning of our
approach is effective for adjusting trade-off problem between BLEU score and
others.

###  5.3 Intent Classification

Model | Banking77 | CLINC150 | HWU64 |
---|---|---|---|---
BERT-FIXED | 87.19’ | 91.79’ | 85.77’ |
BERT-DG | 91.75* | 95.98* | 90.89* |
cist-dial (mslm) | 91.17* | 95.80* | 91.36* |
USE | 92.81’ | 95.06’ | 91.25’ |
USE+CONVERT | 93.36’ | 97.16’ | 92.62’ |
ConvBERT+Pre+Multi | 93.44* | 92.38* | 97.11* |
SPACE 2.0 | 94.77* | 97.80* | 94.33* |
${\text{TOATOD}}_{small}$ | 92.40 | 98.45 | 90.42 |
${\text{TOATOD}}_{base}$ | 92.17 | 98.01 | 90.79 |

Table 4: Accuracy score (%) on all three NLU task dataset with full training.
The values with ’ are from banking77 paper (Casanueva et al., 2020), and * are
from the leaderboard for DialoGLUE paper (Mehri et al., 2020b) and benchmark
444https://eval.ai/web/challenges/challenge-page/708/leaderboard/1943.
33footnotetext: https://eval.ai/web/challenges/challenge-
page/708/leaderboard/1943

We test our models on the NLU task with Banking77, CLINC150, and HWU64. Intent
prediction is the task to identify the intent behind a given input. The task
is normally framed as a classification problem, so we set the metric as turn
accuracy.

####  5.3.1 Evaluation Result

From Table 4, while our models do not achieve the highest score on NLU task,
it is important to note that they perform well in relation to the size when
compared to other models. This highlights the effectiveness of our task-
optimized adapter approach in achieving a balance between model performance
and efficiency.

##  6 Further Analysis and Discussion

###  6.1 Reinforcement Learning

####  6.1.1 w/o Reinforcement Learning

Task | DST | NLG
---|---|---
Metrics | JGA | Slot F1 | Inform | Success | BLEU | Combined
SL (2.2) | 62.92 | 93.72 | 85.30 | 77.00 | 18.44 | 99.59
RL (2.2) | 63.79 | 93.96 | 90.00 | 79.80 | 17.04 | 101.94
SL (2.1) | 53.33 | 91.68 | 88.90 | 81.40 | 18.73 | 103.88
RL (2.1) | 54.97 | 92.01 | 97.00 | 87.40 | 17.12 | 109.32

Table 5: Task performance of $\text{TOATOD}_{base}$ before and after applying
REINFORCE. SL means supervised learning and RL means reinforcement learning.
The test results of the $\text{TOATOD}_{small}$ is attached to Appendix B.

Described on the Table 5, after applying reinforcement learning, the
performance of DST and NLG modules are enhanced. Models with reinforcement
learning obtain best JGA on the DST task and get the highest Combined Score on
the NLG task. The BLEU scores fall slightly but the Combined Score rise with
Inform and Success, which means that incorporating reinforcement learning into
the training process leads our model to complete tasks more efficiently. We
conduct hyperparameter tuning only for the $\text{TOATOD}_{base}$, so there is
some gap in the degree of performance improvement between
$\text{TOATOD}_{small}$ and $\text{TOATOD}_{base}$.

####  6.1.2 Hyperparameters of REINFORCE

${\alpha}$ | 1.0 | 0.7 | 0.9 | 1.0 | 1.0 | 0.3 | 0.5 | 0.7 | 1.0
---|---|---|---|---|---|---|---|---|---
${\beta}$ | 0.4 | 0.5 | 0.5 | 0.5 | 0.6 | 0.7 | 0.7 | 0.7 | 0.7
Inform | 97.50 | 97.40 | 97.50 | 97.50 | 97.50 | 93.40 | 97.00 | 97.30 | 97.50
Success | 85.70 | 86.40 | 85.90 | 85.70 | 85.80 | 85.40 | 87.40 | 87.10 | 86.10
BLEU | 16.01 | 16.35 | 16.07 | 16.03 | 16.04 | 18.36 | 17.12 | 16.52 | 16.10
Combined | 107.67 | 108.25 | 107.77 | 107.63 | 107.69 | 107.76 | 109.32 | 108.72 | 107.90
Table 6: Hyperparameter experiment with ${\alpha}$ and ${\beta}$ on the NLG
task. We experiment $\text{TOATOD}_{base}$ on the MultiWOZ 2.1.

The hyperparameter ${\alpha}$ is used for balancing the importance between
cross-entropy loss and policy loss. We re-train DST and NLG modules by
optimizing the mixed objective function to get higher rewards. The second
hyperparameter ${\beta}$ is a scaling factor to control the trade-off between
BLEU score and Success rate for NLG module. The BLEU score is calculated based
on the number of matching n-grams between the generated text and the reference
text. A higher score indicates that the generated text is more similar to the
label text. Success rate measures the proportion of dialogues, where the model
successfully completes the task. A higher success rate means the model is
better at achieving the user’s goal of the dialogue. In some cases, however,
the model may generate text that is close to the reference text (high BLEU
score) but not relevant to the current dialogue (low Success rate). So, we aim
to reduce the gap between two metrics via hyperparameter tuning.

####  6.1.3 $\alpha$ of DST-optimized adapter

${\alpha}$ | 1.0 | 0.9 | 0.7
---|---|---|---
JGA | 54.97 | 54.96 | 54.76
Slot F1 | 92.01 | 92.02 | 91.95
Table 7: Hyperparameter experiment with ${\alpha}$ on the DST task. We test
$\text{TOATOD}_{base}$ on the MultiWOZ 2.1.

In the Table 7, the result of ${\alpha}$ experiment on DST task indicates that
1.0 yielded the best performance, suggesting the significance of the policy
loss in the overall loss function. By maximizing policy loss, the model is
encouraged to make decisions that result in high rewards, which improves the
performance of the DST task.

####  6.1.4 $\alpha$ and $\beta$ of NLG-optimized adapter

Figure 3: Effect of hyperparameter ${\alpha}$ when ${\beta}$ is fixed
(${\beta}$=0.7). At the point of ${\alpha}$ is 0.5, trade-off issue occurs on
MultiWOZ 2.1.

We experiment with several combinations of ${\alpha}$ and ${\beta}$ within
${\\{}$${\alpha}$: 0.3, 0.5, 0.7, 0.9, 1.0 / ${\beta}$: 0.4, 0.5, 0.6,
0.7${\\}}$. Regardless of hyperparameters, performances improve after applying
the reinforcement learning. The hyperparameters of ${\alpha}$=0.5 and
${\beta}$=0.7, result the best Combined Score, which is 5.44 point higher than
the performance of the supervised learning. As shown in the Figure 3, on the
experiment after removing impact of CE loss (${\alpha}$=1), we found that the
higher the ${\beta}$, success rate increases compared to BLEU score as we
expected.

As shown in Table 6, when the ${\beta}$ is fixed, the bigger weight (smaller
${\alpha}$) on the CE loss ensures the higher BLEU score. On the other hand,
the success rate started to decline from the certain point. Trade-off issue
appears at this point. Therefore, ${\alpha}$ and ${\beta}$ need to be properly
tuned in NLG task for good performance.

##  7 Conclusion

We propose TOATOD, task-optimized adapters for an end-to-end task dialogue
system. By adapting task-optimized adapters, we utilize the end-to-end models
without updating the pre-trained parameters and enabling debugging per task,
which is different from previous research. In addition, we apply REINFORCE
algorithm with metric-aware reward function directly not only on the NLG task
but also DST task to prevent score degradation. As a result, we attain
comparable performance to the previous SOTA models on every benchmark with
very small number of trainable parameters. For the DST task of MultiWOZ2.2,
our TOATOD model outperforms the current SOTA systems.

## Limitations

We train the task-optimized adapters based on the pre-trained weights of
dialogue LM. Therefore, if applied to other dialogue tasks such as chit-chat
and conversational QA system, the performance could be lower than that shown
in our research. And we need future works to clarify the reason why the
performance was better on the MultiWOZ 2.2 dataset, which is expected that our
model does not overfit to the confused labels. Our model inferences in on the
end-to-end manner, but trains like modular system for each task. End-to-end
learning is currently under study. We could adapt the multi-task end-to-end
learning to our method, which may lead to the better performance. Also, we
could analyze the inner working of task-optimized adapters applying XAI
technologies.

## Ethics Statement

We honor the ethical codes set out in the ACL code of Ethics. All of the
datasets used in our study are from previous studies and do not have privacy
issues.

## Acknowledgements

This work was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) (No.2022-0-00621,Development of artificial intelli- gence
technology that provides dialog-based multi-modal explainability)

## References

  * Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019.  Simple, scalable adaptation for neural machine translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1538–1548, Hong Kong, China. Association for Computational Linguistics.
  * Budzianowski and Vulić (2019) Paweł Budzianowski and Ivan Vulić. 2019.  Hello, it’s GPT-2 - how can I help you? towards the use of pretrained language models for task-oriented dialogue systems.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , pages 15–22, Hong Kong. Association for Computational Linguistics.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020.  Efficient intent detection with dual sentence encoders.  _CoRR_ , abs/2003.04807.
  * Chen et al. (2017) Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017.  A survey on dialogue systems.  _ACM SIGKDD Explorations Newsletter_ , 19.
  * Chen et al. (2020) Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020.  Reinforcement learning based graph-to-sequence model for natural question generation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
  * Cholakov and Kolev (2022) Radostin Cholakov and Todor Kolev. 2022.  Efficient task-oriented dialogue systems with response selection as an auxiliary task.  _CoRR_ , abs/2208.07097.
  * Dai et al. (2021) Yinpei Dai, Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. 2021.  Preview, attend and review: Schema-aware curriculum learning for multi-domain dialogue state tracking.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021_ , pages 879–885. Association for Computational Linguistics.
  * Dey et al. (2022) Suvodip Dey, Ramamohan Kummara, and Maunendra Desarkar. 2022.  Towards fair evaluation of dialogue state tracking by flexible incorporation of turn-level performances.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 318–324, Dublin, Ireland. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13042–13054.
  * Eric et al. (2020) Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020.  MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 422–428, Marseille, France. European Language Resources Association.
  * Feng et al. (2022) Shaoxiong Feng, Xuancheng Ren, Kan Li, and Xu Sun. 2022.  Hierarchical inductive transfer for continual dialogue learning.  In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 693–699. Association for Computational Linguistics.
  * French (1999) Robert French. 1999.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ , 3:128–135.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_ , pages 770–778. IEEE Computer Society.
  * He et al. (2022a) Wanwei He, Yinpei Dai, Min Yang, Jian Sun, Fei Huang, Luo Si, and Yongbin Li. 2022a.  Unified dialog model pre-training for task-oriented dialog understanding and generation.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’22, page 187–200, New York, NY, USA. Association for Computing Machinery.
  * He et al. (2022b) Wanwei He, Yinpei Dai, Yinhe Zheng, Yuchuan Wu, Zheng Cao, Dermot Liu, Peng Jiang, Min Yang, Fei Huang, Luo Si, Jian Sun, and Yongbin Li. 2022b.  GALAXY: A generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection.  In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022_ , pages 10749–10757. AAAI Press.
  * Heck et al. (2020) Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gasic. 2020.  TripPy: A triple copy strategy for value independent neural dialog state tracking.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 35–44, 1st virtual meeting. Association for Computational Linguistics.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A simple language model for task-oriented dialogue.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Jeon and Lee (2021) Hyunmin Jeon and Gary Geunbae Lee. 2021.  Domain state tracking for a simplified dialogue system.  _CoRR_ , abs/2103.06648.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  * Lee et al. (2021) Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. 2021.  Dialogue state tracking with a language model using schema-driven prompting.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 4937–4949. Association for Computational Linguistics.
  * Lee (2021) Yohan Lee. 2021.  Improving end-to-end task-oriented dialog system with a simple auxiliary task.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016.  Deep reinforcement learning for dialogue generation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Austin, Texas. Association for Computational Linguistics.
  * Li et al. (2021) Junyi Li, Tianyi Tang, Wayne Xin Zhao, and Ji-Rong Wen. 2021.  Pretrained language models for text generation: A survey.  _CoRR_ , abs/2105.10311.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2021) Zhaojiang Lin, Andrea Madotto, Yejin Bang, and Pascale Fung. 2021.  The adapter-bot: All-in-one controllable conversational model.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 16081–16083. AAAI Press.
  * Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020.  Mintl: Minimalist transfer learning for task-oriented dialogue systems.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 3391–3405. Association for Computational Linguistics.
  * Liu and Lane (2018) Bing Liu and Ian Lane. 2018.  End-to-end learning of task-oriented dialogs.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop_ , pages 67–73, New Orleans, Louisiana, USA. Association for Computational Linguistics.
  * Liu et al. (2019) Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019.  Benchmarking natural language understanding services for building conversational agents.  In _Increasing Naturalness and Flexibility in Spoken Dialogue Interaction - 10th International Workshop on Spoken Dialogue Systems, IWSDS 2019, Syracuse, Sicily, Italy, 24-26 April 2019_ , volume 714 of _Lecture Notes in Electrical Engineering_ , pages 165–183. Springer.
  * Mehri et al. (2020a) Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tür. 2020a.  Dialoglue: A natural language understanding benchmark for task-oriented dialogue.  _CoRR_ , abs/2009.13570.
  * Mehri et al. (2020b) Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tür. 2020b.  Dialoglue: A natural language understanding benchmark for task-oriented dialogue.  _CoRR_ , abs/2009.13570.
  * Nekvinda and Dusek (2021) Tomás Nekvinda and Ondrej Dusek. 2021.  Shades of bleu, flavours of success: The case of multiwoz.  _CoRR_ , abs/2106.05555.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_.
  * Peng et al. (2021) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2021.  SOLOIST: building task bots at scale with transfer learning and machine teaching.  _Trans. Assoc. Comput. Linguistics_ , 9:907–824.
  * Radford and Narasimhan (2018) Alec Radford and Karthik Narasimhan. 2018.  Improving language understanding by generative pre-training.
  * (38) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017.  Learning multiple visual domains with residual adapters.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 506–516.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4596–4604. PMLR.
  * Stickland and Murray (2019) Asa Cooper Stickland and Iain Murray. 2019.  BERT and pals: Projected attention layers for efficient adaptation in multi-task learning.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 5986–5995. PMLR.
  * Su et al. (2022) Yixuan Su, Lei Shu, Elman Mansimov, Arshit Gupta, Deng Cai, Yi-An Lai, and Yi Zhang. 2022.  Multi-task pre-training for plug-and-play task-oriented dialogue system.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4661–4676, Dublin, Ireland. Association for Computational Linguistics.
  * Sun et al. (2022a) Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2022a.  BORT: back and denoising reconstruction for end-to-end task-oriented dialog.  In _Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , pages 2156–2170. Association for Computational Linguistics.
  * Sun et al. (2022b) Haipeng Sun, Junwei Bao, Youzheng Wu, and Xiaodong He. 2022b.  Mars: Semantic-aware contrastive learning for end-to-end task-oriented dialog.  _CoRR_ , abs/2210.08917.
  * Sutton et al. (1999) Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. 1999\.  Policy gradient methods for reinforcement learning with function approximation.  In _Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999]_ , pages 1057–1063. The MIT Press.
  * Tian et al. (2021) Xin Tian, Liankai Huang, Yingzhan Lin, Siqi Bao, Huang He, Yunyi Yang, Hua Wu, Fan Wang, and Shuqi Sun. 2021.  Amendable generation for dialogue state tracking.  In _Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI_ , pages 80–92, Online. Association for Computational Linguistics.
  * Wang et al. (2021) Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. 2021.  Modelling hierarchical structure between dialogue policy and natural language generator with option framework for task-oriented dialogue system.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_.
  * Wu et al. (2020) Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020.  TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 917–929, Online. Association for Computational Linguistics.
  * Wu et al. (2019) Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019.  Transferable multi-domain state generator for task-oriented dialogue systems.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 808–819, Florence, Italy. Association for Computational Linguistics.
  * Wu et al. (2021) Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2021.  Alternating recurrent dialog model with large-scale pre-trained language models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , pages 1292–1301. Association for Computational Linguistics.
  * Yang et al. (2021) Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2021.  UBAR: towards fully end-to-end task-oriented dialog system with GPT-2.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 14230–14238. AAAI Press.
  * Ye et al. (2022) Chenchen Ye, Lizi Liao, Fuli Feng, Wei Ji, and Tat-Seng Chua. 2022.  Structured and natural responses co-generation for conversational search.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’22, page 155–164, New York, NY, USA. Association for Computing Machinery.
  * Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020.  MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 109–117, Online. Association for Computational Linguistics.
  * Zhang et al. (2020a) Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, Philip S. Yu, Richard Socher, and Caiming Xiong. 2020a.  Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking.  In _Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, *SEM@COLING 2020, Barcelona, Spain (Online), December 12-13, 2020_ , pages 154–167. Association for Computational Linguistics.
  * Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2020, Online, July 5-10, 2020_ , pages 270–278. Association for Computational Linguistics.
  * Zhao et al. (2022) Jeffrey Zhao, Raghav Gupta, Yuan Cao, Dian Yu, Mingqiu Wang, Harrison Lee, Abhinav Rastogi, Izhak Shafran, and Yonghui Wu. 2022.  Description-driven task-oriented dialog modeling.  _CoRR_ , abs/2201.08904.
  * Zhou and Small (2019) Li Zhou and Kevin Small. 2019.  Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering.  _CoRR_ , abs/1911.06192.

## Appendices

##  Appendix A Units of Adapters

Dim | ${1}/{2}$ | ${1}/{4}$ | ${1}/{8}$
---|---|---|---
JGA | 51.59 | 51.11 | 50.43
Slot f1 | 91.02 | 91.07 | 90.40
Trainable Params | 7.9M | 7.2M | 6.6M
Table 8: Adapter units experiment results. We test different bottleneck
dimension of adapter module on the DST task.

In this experiment, we evaluate the performance with several bottleneck
dimensions, h = 256, 128, 64 with $\text{TOATOD}_{small}$, which is 1/2, 1/4,
1/8 size of $\text{T5}_{small}$’s embedding dimension of 512. The main focus
of the experiment is to investigate the effect of the bottleneck dimension h
of the adapter module on the performance of the model. To evaluate the
performance of the model, we use the Joint Goal Accuracy for DST task. We keep
the other hyperparameters constant across all the experiments, including
learning rate of 1e-4 and evaluate the performance on the test set of MultiWOZ
2.1. We report the result on the Table 8.

The result implies that there is a trade-off between the bottleneck dimension
and the performance of the model. As the bottleneck dimension increases, the
performance of the model also improves. The best performance is achieved with
a bottleneck dimension of 256, where the JGA is 51.59. It is important to
carefully choose the bottleneck dimension when using the adapter module in the
task-oriented dialogue system. As described in the Table 1, the trainable
parameters of our model with bottleneck dimension of 256 is still
significantly smaller than the PLM’s parameters, so we choose the size of 1/2.

##  Appendix B w/o Reinforcement Learning of $\text{TOATOD}_{small}$

Task | DST | NLG
---|---|---
Metrics | JGA | Slot F1 | Inform | Success | BLEU | Combined
SL (2.2) | 61.29 | 93.46 | 78.80 | 69.50 | 18.46 | 92.61
RL (2.2) | 61.92 | 93.65 | 85.80 | 74.00 | 18.00 | 97.90
SL (2.1) | 52.58 | 91.31 | 84.30 | 74.40 | 18.82 | 98.17
RL (2.1) | 53.01 | 91.61 | 92.10 | 80.50 | 18.28 | 104.58

Table 9: Task performance of $\text{TOATOD}_{small}$ before and after applying
REINFORCE. SL means supervised learning and RL means reinforcement learning.

##  Appendix C Implementation Details

We used 8 A100 (80G) GPUs, but they were fully used during only reinforcement
learning. During supervised learning, we used 4 GPUs. While reinforcement
learning of the DST task-optimized adapters, we set learning rate as 1e-5, and
batch size as 32 (utterance-level). On the contrary, while reinforcement
training of the NLG task-optimized adapters, we set learning rate as 1e-6 and
batch size as 4 (session-level). During the entire training process, we set
random seed as 42. And for the NLU task, we used 1 RTX A5000 (24GB) GPU and
trained models without reinforcement learning. We set batch size as 64 and
single run with random seeds. When training $\text{T5}_{small}$ and
$\text{T5}_{base}$ for the Banking 77 and CLINC150, we used learning rates of
0.001 and 0.15. And we used learning rates of 0.01 and 0.1 for the
$\text{T5}_{small}$ and $\text{T5}_{base}$ with the HWU64.