[b]Lorenzo Magnea

# IBPs and differential equations in parameter space

Daniele Artico

###### Abstract

We present a projective framework for the construction of Integration by Parts
(IBP) identities and differential equations for Feynman integrals, working in
Feynman-parameter space. This framework originates with very early results
which emerged long before modern techniques for loop calculations were
developed [17, 18, 19, 20, 21, 22]. Adapting and generalising these results to
the modern language, we use simple tools of projective geometry to generate
sets of IBP identities and differential equations in parameter space, with a
technique applicable to any loop order. We test the viability of the method on
simple diagrams at one and two loops, providing a unified viewpoint on several
existing results.

##  1 Introduction

The calculation of high-order Feynman integrals is crucial for the current and
future precision physics programs at particle accelerators [1]: to this end,
modern methods have evolved that go much beyond a direct evaluation. These
developments began with the discovery of Integration-by-Parts (IBP) identities
in dimensional regularization [2, 3], and the introduction of the method of
differential equations [4, 5, 6]: calculations are now typically tackled with
automatic algorithms combining these ideas [7]. Further developments have
involved an enhanced understanding of the role of generalised unitarity, and
of the linear functional spaces where classes of Feynman integrals reside [8,
9, 10, 11], and an optimised use of dimensional regularization [12]. A vast
amount of work along these lines has significantly expanded the range of
processes for which high-order calculations are possible, and has much
deepened our mathematical understanding of Feynman integrals [1, 13].

Interestingly, the historical exploration of Feynman integrals via IBPs and
differential equations predates these recent developments. Indeed, the
projective nature of Feynman parameter integrands and the monodromy properties
of Feynman integrals attracted early attention from mathematicians and
physicists already in the late 1960’s [14, 15, 16, 17]. Around that time, in
particular, Tullio Regge and his collaborators explored the monodromy
properties of various classes of Feynman integrals [17, 18, 19, 20], offering
several insights that align with (and predate) contemporary findings. For
example Regge argued, already at the time [17], that Feynman integrals belong
to a class of generalised hypergeometric functions, and proposed that these
functions should satisfy differential equations of the Picard-Fuchs type.

Although actual computational algorithms didn’t emerge from these studies,
Barucchi and Ponzano [21, 22] constructed an explicit implementation of
Regge’s ideas, applicable to one-loop diagrams. The corresponding Feynman
integrals, in parametric form, were organised into sets connected by
difference equations, akin to currently used IBPs; linear systems of
homogeneous differential equations in the Mandelstam invariants were then
derived, mirroring the known one-loop monodromy structure.

This note, summarising the results presented in [24], builds on the work of
Regge and collaborators, to propose a projective framework for deriving IBP
identities and systems of linear differential equations for Feynman integrals,
directly in parameter space. The framework accommodates dimensional
regularisation, extends to infrared-divergent integrals, and generalises
naturally beyond on loop. Interestingly, our results also underscore the role
of boundary terms in IBP identities within the projective framework: unlike
the momentum-space approach in dimensional regularization, these terms do not
generally vanish in parameter space, and indeed they play a critical role in
connecting complex integrals to simpler ones.

We begin our note by setting up conventions for Feynman integrals in parameter
form, in Section 2. Next, in Section 3, we introduce projective forms, and we
use their properties to show how one can construct systems of difference
equations for generic projective integrals. In Section 4 we focus on Feynman
integrals, and provide a general procedure to construct IBPs in this context,
developing the one-loop case in some detail as an example. Explicit one-loop
examples are presented in Section 5, and two-loop examples in Section 6.
Finally, Section 7 briefly discusses perspectives for future work.

##  2 Notations

Scalar Feynman integrals arise in loop-level perturbative calculations in any
quantum filed theory. In a momentum space formulation they take the form

$\displaystyle I_{G}\left(\nu_{i},d\right)\,=\,(\mu^{2})^{\nu-
ld/2}\int\prod_{r=1}^{l}\frac{d^{d}k_{r}}{{\rm
i}\pi^{d/2}}\,\prod_{i=1}^{n}\frac{1}{\left(-q_{i}^{2}+m_{i}^{2}\right)^{\nu_{i}}}\,,\qquad
q_{i}\,=\,\sum_{r=1}^{l}\alpha_{ir}k_{r}+\sum_{j=1}^{m}\beta_{ij}p_{j}\,,$ (1)

where $q_{i}$ are the momenta flowing in each propagator, $k_{r}$ are the
independent loop momenta, and $p_{j}$ are the external momenta, while $d$ is
the space-time dimension, and the integer exponents $\nu_{i}$ satisfy
$\sum_{i}\nu_{i}=\nu$. The integration over loop momenta in Eq. (1) can be
performed in full generality by means of the Feynman parameter technique.
Using the notations from Refs. [25, 26, 13], the integral becomes

$\displaystyle I_{G}\left(\nu_{i},d\right)\,=\,\frac{\Gamma(\nu-
ld/2)}{\prod_{j=1}^{n}\Gamma(\nu_{j})}\int_{z_{j}\geq
0}d^{n}z\,\delta\left(1-\sum_{j=1}^{n}z_{j}\right)\,\left(\prod_{j=1}^{n}z_{j}^{\nu_{j}-1}\right)\,\frac{\mathcal{U}^{\,\nu-(l+1)d/2}}{\mathcal{F}^{\,\nu-
ld/2}}\,,$ (2)

where the Symanzik polynomials $\cal{U}$ and ${\cal F}$,

$\displaystyle\mathcal{U}\,=\,\sum_{\mathcal{T}_{G}}\prod_{i\in\mathcal{T}_{G}}z_{i}\,,\qquad\quad\mathcal{F}\,=\,\sum_{\mathcal{C}_{G}}\frac{\hat{s}\left(\mathcal{C}_{G}\right)}{\mu^{2}}\,\prod_{i\in\mathcal{C}_{G}}z_{i}\,-\,\mathcal{U}\sum_{i\in\mathcal{I}_{G}}\frac{m_{i}^{2}}{\mu^{2}}\,z_{i}\,,$
(3)

can be defined purely from the graph properties. To this end, let us denote by
$\mathcal{I}_{G}$ the set of the internal lines of $G$, each endowed with a
Feynman parameter $z_{i}$. A co-tree $\mathcal{T}_{G}\subset\mathcal{I}_{G}$
is a set of internal lines of $G$ such that that the lines in its complement
$\overline{\mathcal{T}}_{G}\subset\mathcal{I}_{G}$ form a spanning tree.
Similarly, consider subsets ${\cal C}_{G}\subset{\cal I}_{G}$ with the
property that, upon omitting the lines of ${\cal C}_{G}$ from $G$, the graph
becomes a disjoint union of two connected subgraphs. Each subset ${\cal
C}_{G}$ defines a cut of graph $G$, and contains $l+1$ lines; an invariant
mass $\hat{s}\left({\cal C}_{G}\right)$ can be associated with each cut, by
squaring the sum of the momenta flowing in (or out) of one of the two
subgraphs. The Symanzik polynomial $\mathcal{U}$ is homogeneous of degree $l$,
while the Symanzik polynomial $\mathcal{F}$ is homogeneous of degree $l+1$, so
that the integrand (measure included) is homogeneous of degree $0$.

##  3 A projective framework

A crucial mathematical property of Feynman integrals is that their integrands
are projective forms in the space of Feynman parameters, which can be
identified with $\mathbb{PC}^{n-1}$. This property is crucial for the
characterisation of the function spaces to which the integrals belong, and to
many techniques for their explicit evaluation. The relevance of projective
invariance was understood since the earliest systematic studies of Feynman
diagrams [17]. In this Section, we provide an extremely concise summary of the
relevant ideas.

In order to introduce projective forms, begin by considering a generic subset
$A$, $|A|=a$, of the set $D=\left\\{1,\ldots,N\right\\}$, and define the
$a$-form

$\displaystyle\omega_{A}\,=\,dz_{i_{1}}\wedge...\wedge dz_{i_{a}}\,,$ (4)

with $i_{1}<\ldots<i_{a}$. One can show that $\omega_{A}$ integrates to the
projective $(a-1)$-form

$\displaystyle\eta_{A}\,=\,\sum_{i\in
A}\,\epsilon_{i,A-i}\,z_{i}\,\omega_{A-i}\,,\qquad
d\eta_{A}\,=\,a\omega_{A}\,,$ (5)

where we introduced the signature factor

$\displaystyle\epsilon_{k,B}\,=\,(-1)^{|B_{k}|}\,,\qquad
B_{k}\,=\,\left\\{i\in B,i<k\right\\}\,.$ (6)

As an example, for $A=\\{1,2,3\\}$ one finds

$\displaystyle\eta_{\\{1,2,3\\}}\,=\,z_{1}\,dz_{2}\wedge
dz_{3}-z_{2}\,dz_{1}\wedge dz_{3}+z_{3}\,dz_{1}\wedge dz_{2}\,.$ (7)

Projective forms such as $\eta_{A}$ are homogeneous of degree 1 in each
coordinate $z_{i}$, and they can serve as measures of integration for
projective integrals. Indeed, parametric Feynman integrands can be represented
in the general form

$\displaystyle\alpha_{n-1}\,=\,\eta_{n-1}\,\,\frac{Q\big{(}\left\\{z_{i}\right\\}\big{)}}{D^{P}\big{(}\left\\{z_{i}\right\\}\big{)}}\,,$
(8)

where $D\big{(}\left\\{z_{i}\right\\}\big{)}$ and
$Q\big{(}\left\\{z_{i}\right\\}\big{)}$ are polynomials of degrees such that
the form (measure included) is homogeneous of degree $0$. A well-known example
is the integrand for the massless one-loop box integral, which reads

$\displaystyle\psi_{3}\left(\lambda,r\right)\,=\,\frac{(z_{1}+z_{2}+z_{3}+z_{4})^{\lambda}}{\left(r\,z_{1}z_{3}+z_{2}z_{4}\right)^{2+\lambda/2}}\,\,\eta_{\left\\{1,2,3,4\right\\}}\,.$
(9)

For finite integrals both $P$ in Eq. (8) and $\nu$ in Eq. (2) are integers. In
the presence of divergences, as is the case for the massless box, we can
incorporate dimensional regularisation by allowing for general values of $d$,
and thus of $\lambda$ in Eq. (9).

Two theorems naturally emerge within this projective framework. First of all,
an essential property of projective forms is the following [17].

######  Theorem 1.

The boundary of a projective form is itself projective.

This theorem arises from the properties of the operator

$\displaystyle
p\,:\,\sum_{|A|=q}R_{A}(z_{i})\,\omega_{A}\quad\rightarrow\quad\sum_{|A|=q}R_{A}(z_{i})\,\eta_{A}\,,$
(10)

mapping affine $q$-forms into projective $(q-1)$-forms. It can be shown to
satisfy

$\displaystyle p^{2}=0\>,\qquad d\circ p+p\circ d\,=\,0\,.$ (11)

Based on these properties, a proof of the theorem can be found in [24].

Next, it is possible to show that $\alpha_{n-1}$ is a closed form, while
$\eta_{n-1}$ is null on any surface defined by $z_{i}=0$. A second theorem
then follows

######  Theorem 2.

Given two integration domains, $O,O^{\prime}\in\mathbb{C}^{n}$, if their image
in $\mathbb{PC}^{n-1}$ is the same simplex, then
$\int_{O}\alpha_{n-1}\,=\,\int_{O^{\prime}}\alpha_{n-1}$.

This theorem, also known as Cheng-Wu theorem [23], allows, in practice, to set
to zero any subset of the $n$ parameters $z_{i}$ in the argument of the
$\delta$ function in Eq. (2), providing a useful tool for concrete
calculations.

##  4 Integration by parts in projective space

The correspondence between projective forms and parametric integrals is
obtained from the usual choice of chart in projective space identifying a
coordinate symplex in $\mathbb{R}^{n}$ with the choice
$\sum_{i=1}^{n}z_{i}=1$. With this choice one finds simply

$\displaystyle\int_{S_{n-1}}\,\eta_{n-1}\,\frac{Q(z)}{D^{P}(z)}\,=\,\int_{z_{i}\geq
0}dz_{1}\ldots
dz_{n}\,\,\delta\\!\left(1-\sum_{i=1}^{n}z_{i}\right)\,\frac{Q(z)}{D^{P}(z)}\,.$
(12)

We now show how the projective structure just introduced allows to easily
construct sets of difference equations connecting families of Feynman
integrals, which play the role of the conventional IBP identities usually
derived in momentum space. To this end, consider the projective $(n-2)$-forms

$\displaystyle\omega_{n-2}\,\equiv\,\sum_{i=1}^{n}(-1)^{i}\,\eta_{\\{z\\}-z_{i}}\,\frac{H_{i}(z)}{(P-1)\,\big{(}D(z)\big{)}^{P-1}}\,,$
(13)

where $\eta_{\\{z\\}-z_{i}}$ denotes the projective volume form in
$\mathbb{PC}^{n-2}$, obtained by omitting the coordinate $z_{i}$, and
$H_{i}(z)$ are polynomials with a degree chosen (together with $P$) to ensure
projectivity. Differentiating these forms generates (at the integrand level) a
set of identities among parametric integrals, which correspond to those
obtained via integration by parts. One finds

$\displaystyle
d\omega_{n-2}\,=\,\frac{1}{(P-1)\,\big{(}D(z)\big{)}^{P-1}}\,\,\eta_{\\{z\\}}\,\sum_{i=1}^{n}\frac{\partial
H_{i}(z)}{\partial
z_{i}}\,-\,\frac{\eta_{\\{z\\}}}{\big{(}D(z)\big{)}^{P}}\,\sum_{i=1}^{n}H_{i}\,\frac{\partial
D(z)}{\partial z_{i}}\,.$ (14)

Eq. (14) plays a central role in our method. By suitably choosing the
polynomials $H_{i}(z)$, it allows to close systems of linear differential
equations for Feynman integrals that can be used to compute them, just as
usually done in the momentum space approach. We emphasise that these
identities apply for any number of loops or external legs. In the remainder of
this section, we will discuss a concrete implementation at the one-loop level
developing the ideas of Ref. [21].

At one loop, parametric integrals have the general form

$\displaystyle
I_{G}(\nu_{i},d)\,=\,\frac{\Gamma(\nu-d/2)}{\prod_{j=1}^{n}\Gamma(\nu_{j})}\int_{z_{j}\geq
0}\\!d^{n}z\,\delta\bigg{(}1-z_{n+1}\bigg{)}\,\frac{\prod_{j=1}^{n+1}z_{j}^{\nu_{j}-1}}{\bigg{[}\sum_{i=1}^{n+1}\sum_{j=1}^{i-1}s_{ij}z_{i}z_{j}\bigg{]}^{\nu-d/2}}\,,$
(15)

where we introduced the notations

$\displaystyle
z_{n+1}\,\equiv\,\sum_{i=1}^{n}z_{i}\,,\qquad\quad\nu_{n+1}\,\equiv\,\nu-d+1\,,$
(16)

and for the Mandelstam invariants we use

$\displaystyle
s_{ij}\,=\,\frac{(q_{j}-q_{i})^{2}}{\mu^{2}}\quad(i,j=1,\ldots,n)\,,\qquad\quad
s_{i,n+1}\,=\,s_{n+1,i}\,\equiv\,-\frac{m_{i}^{2}}{\mu^{2}}\,.$ (17)

We now make the simplest and natural choice in Eq. (14), picking the
polynomials $H_{i}(z)$ to coincide with the numerator of the relevant
integral, for each value of $i$. Thus we pick

$\displaystyle
H_{i}\,=\,\delta_{ih}\left(\prod_{j=1}^{n}z_{j}^{\nu_{j}-1}\right)\left(\sum_{k=1}^{n}z_{k}\right)^{\nu-d}\,=\,\,\delta_{ih}\,\prod_{j=1}^{n+1}z_{j}^{\nu_{j}-1}\,,$
(18)

for $h=1,\ldots,n$. Applying this choice produces a one-loop ‘integration-by-
parts’ identity that can be written as follows [21]:

$\displaystyle
d\omega_{n-2}+\sum_{k=1}^{n+1}(s_{kh}+s_{k,n+1})\,f\big{(}\\!\left\\{\mathcal{R}-k\right\\}_{0},\left\\{k\right\\}_{1}\\!\big{)}\\!$
$\displaystyle=$
$\displaystyle\\!\frac{\nu_{h}-1}{\nu-(d+1)/2}\,f\big{(}\\!\left\\{h\right\\}_{-1},\left\\{\mathcal{R}-h\right\\}_{0}\\!\big{)}$
$\displaystyle+$
$\displaystyle\\!\frac{\nu-d}{\nu-(d+1)/2}\,f\big{(}\\!\left\\{n+1\right\\}_{-1},\left\\{\mathcal{R}-\left\\{n+1\right\\}\right\\}_{0}\\!\big{)}\,,$

where, following Ref. [21], we introduced an index notation such that, for the
function

$\displaystyle
f\big{(}\\!\left\\{\nu_{1},\ldots,\nu_{n+1}\right\\}\\!\big{)}\,\equiv\,f\big{(}\\!\left\\{{\cal
R}\right\\}\\!\big{)}\,=\,\eta_{\\{z\\}}\,\frac{\prod_{j=1}^{n+1}z_{j}^{\nu_{j}-1}}{\left(\sum_{i=1}^{n+1}\sum_{j=1}^{i-1}s_{ij}z_{i}z_{j}\right)^{\nu-d/2}}\,,$
(20)

we can raise or lower the exponents $\nu_{i}$ by adding $\\{-1,0,1\\}$ in the
subsets $\cal I$, $\cal J$ and $\cal K$ of set ${\cal R}$ respectively, and we
denote the resulting function by

$\displaystyle f\big{(}\\!\left\\{{\cal I}\right\\}_{-1},\left\\{{\cal
J}\right\\}_{0},\left\\{{\cal K}\right\\}_{1}\\!\big{)}\,.$ (21)

Note that the exponent of the denominator in Eq. (20) is adjusted accordingly,
to maintain projective properties. Note also that the action of raising and
lowering exponents according to the convention in Eq. (21) is subject to a
constraint, arising from the definition of $\mathcal{U}$ in Eq. (3).
Specifically, the following sum rule holds

$\displaystyle\sum_{i=1}^{n}f\big{(}\\!\left\\{\mathcal{R}-i\right\\}_{0},\left\\{i\right\\}_{1}\\!\big{)}\,=\,f\big{(}\\!\left\\{\mathcal{R}-\left\\{n+1\right\\}\right\\}_{0},\left\\{n+1\right\\}_{1}\\!\big{)}\,.$
(22)

As we will see in the next section, Eq. (4) and Eq. (22) can be used to close
systems of differential equations, leading to the determination of the one-
loop Feynman integrals under study. Two-loop examples will be discussed in
Section 6.

##  5 One-loop examples

In this section, we present two explicit examples of the use of Eq. (4).
Consider first the massless one-loop box integral, setting $t/s\equiv r$ and
with all momenta incoming. Using dimensional regularisation, we define

$\displaystyle
I_{\textrm{box}}\,\equiv\,\Gamma(2+\epsilon)\,\int_{S_{n-1}}\eta_{\\{z\\}}\,\frac{(z_{1}+z_{2}+z_{3}+z_{4})^{2\epsilon}}{\left(rz_{1}z_{3}+z_{2}z_{4}\right)^{2+\epsilon}}\,\equiv\,\Gamma(2+\epsilon)\,I(1,1,1,1;2\epsilon)\,,$
(23)

where for the box family of integrals we use the notation
$I(\nu_{1},\nu_{2},\nu_{3},\nu_{4};\nu_{5})$. Differentiating with respect to
$r$ raises two indices by one unit, as in

$\displaystyle\partial_{r}I(1,1,1,1;2\epsilon)\,=\,-(2+\epsilon)\,I(2,1,2,1;2\epsilon)\,$
(24)
$\displaystyle\partial_{r}I(2,1,2,1;2\epsilon)\,=\,-(3+\epsilon)\,I(3,1,3,1;2\epsilon)\,.$
(25)

According to a theorem by Barucchi and Ponzano [21], for any one-loop diagram
a system of differential equation can be set up, involving the desired
integral, plus the ones obtained by lifting an even number of propagators by
1. For the massless box, following this construction we find that a closed
system of differential equations can be obtained for the integrals111It is
well-known that a basis of master integrals for the massless box requires only
three integrals. Here we are simply illustrating the Barucchi-Ponzano
construction, which in this case yields an over-complete basis, and we have
not attempted optimisations. On the other hand, the method correctly predicts
the size of the basis for the most general one-loop diagram, as recently
confirmed by Refs. [28, 29, 30].

$\displaystyle\Big{\\{}I(1,1,1,1;2\epsilon),I(2,1,2,1;2\epsilon),I(1,2,1,2;2\epsilon),I(2,2,2,2;2\epsilon)\Big{\\}}\,.$
(26)

The system is obtained by using identities generated by Eq. (4), such as, for
example,

$\displaystyle rI(3,1,3,1;2\epsilon)+\int
d\omega_{n-2}\,=\,\frac{2}{3+\epsilon}I(2,1,2,1;2\epsilon)+\frac{2\epsilon}{3+\epsilon}I(3,1,3,1;-1+2\epsilon)\,.$
(27)

where the integral of $d\omega_{n-2}$ gives a vanishing boundary term, since

$\displaystyle\frac{z_{1}^{2}z_{3}\left(z_{1}+z_{2}+z_{3}+z_{4}\right)^{2\epsilon}}{(3+\epsilon)(rz_{1}z_{3}+z_{2}z_{4})^{3+\epsilon}}\big{(}z_{2}dz_{3}\wedge
dz_{4}-z_{3}dz_{2}\wedge dz_{4}+z_{4}dz_{2}\wedge
dz_{3}\big{)}\Bigg{|}_{\partial S_{n-1}}\,=\,0\,.$ (28)

The system of differential equations obtained in this way can be written as

$\displaystyle\partial_{r}\textbf{b}\,\equiv\,\partial_{r}\left(\begin{array}[]{c}I(1,1,1,1;2\epsilon)\\\
I(2,1,2,1;2\epsilon)\\\ I(1,2,1,2;2\epsilon)\\\
I(2,2,2,2;2\epsilon)\end{array}\right)=\left(\begin{array}[]{c c c
c}0&-(2+\epsilon)&0&0\\\ 0&-\frac{3+\epsilon}{r}&0&-\frac{3+\epsilon}{r}\\\
0&0&0&-(3+\epsilon)\\\
0&-\frac{1}{(3+\epsilon)r(1+r)}&\frac{1}{(3+\epsilon)r(1+r)}&-\frac{1+\epsilon+3r}{(3+\epsilon)r(1+r)}\end{array}\right)\textbf{b}\,.$
(37)

This system can be brought to canonical form by using (for example) the
technique of Magnus exponentiation [31]. It can then be solved by iteration,
and the solution, consistently with the literature [27], is given by

$\displaystyle I_{\rm box}$ $\displaystyle=$
$\displaystyle\frac{k(\epsilon)}{r}\bigg{[}\frac{1}{\epsilon^{2}}-\frac{\log
r}{2\epsilon}-\frac{\pi^{2}}{4}+\epsilon\,\bigg{(}\frac{1}{2}\,{\rm
Li}_{3}(-r)-\frac{1}{2}\,{\rm Li}_{2}(-r)\log r+\frac{1}{12}\log^{3}r$ (38)
$\displaystyle\hskip
56.9055pt-\,\frac{1}{4}\log(1+r)\left(\log^{2}r+\pi^{2}\right)+\frac{1}{4}\,\pi^{2}\log
r+\frac{1}{2}\zeta(3)\bigg{)}\,+\,{\cal O}(\epsilon^{2})\bigg{]}\,,$

with
$k(\epsilon)\,=\,4-\frac{\pi^{2}}{3}\epsilon^{2}-\frac{40\zeta(3)}{3}\epsilon^{3}$.

The difference equations generated in parameter space by Eq. (4) effectively
include also dimensional-shift identities, and they connect the desired
integrals to lower-point integrals through non-vanishing boundary terms. As an
example, consider the following identity for five-point integrals:

$\displaystyle\int_{S_{\left\\{1,2,3,4,5\right\\}}}\\!d\omega_{3}+s_{13}\,I(1,1,2,1,1;2\epsilon)+s_{14}\,I(1,1,1,2,1;2\epsilon)\,=\,\frac{2\epsilon}{2+\epsilon}I(1,1,1,1,1;-1+2\epsilon)\,,\qquad$
(39)

with

$\displaystyle
d\omega_{3}\,=\,d\left[-\,\eta_{\\{2,3,4,5\\}}\,\frac{(z_{1}+z_{2}+z_{3}+z_{4}+z_{5})^{2\epsilon}}{(2+\epsilon)\left(s_{13}z_{1}z_{3}+s_{14}z_{1}z_{4}+s_{24}z_{2}z_{4}+s_{25}z_{2}z_{5}+s_{35}z_{3}z_{5}\right)^{2+\epsilon}}\right]\,.$
(40)

The integration of this form using Stokes theorem produces a non-vanishing
boundary term, corresponding to the one-loop box integral with one external
leg off-shell. Specifically, one finds

$\displaystyle\int_{S_{\\{2,3,4,5\\}}}\\!\eta_{\\{2,3,4,5\\}}\,\frac{(z_{2}+z_{3}+z_{4}+z_{5})^{2\epsilon}}{\left(s_{24}z_{2}z_{4}+s_{25}z_{2}z_{5}+s_{35}z_{3}z_{5}\right)^{2+\epsilon}}\,=\,I_{\rm
box}^{(1)}(s_{25})\,,$ (41)

where in this case $s_{25}$ is the mass of the off-shell leg. Using similar
identities, dimensional-shift relations for the one-loop pentagon [32] can
easily be reproduced. One finds

$\displaystyle 2(2+\epsilon)\,I(1,1,1,1,1;1+2\epsilon)$ $\displaystyle=$
$\displaystyle\Bigg{\\{}\frac{s_{13}s_{24}-s_{13}s_{25}-s_{14}s_{25}+s_{14}s_{35}-s_{24}s_{35}}{s_{13}s_{14}s_{25}}\,I_{\rm
box}^{(1)}(s_{25})$ (42)
$\displaystyle-\,\frac{s_{13}s_{24}+s_{13}s_{25}-s_{14}s_{25}+s_{14}s_{35}-s_{24}s_{35}}{s_{13}s_{24}s_{25}}\,I_{\rm
box}^{(2)}(s_{13})$
$\displaystyle-\,\frac{s_{13}s_{24}-s_{13}s_{25}+s_{14}s_{25}-s_{14}s_{35}+s_{24}s_{35}}{s_{13}s_{24}s_{35}}\,I_{\rm
box}^{(3)}(s_{24})$
$\displaystyle+\,\frac{s_{13}s_{24}-s_{13}s_{25}+s_{14}s_{25}-s_{14}s_{35}-s_{24}s_{35}}{s_{14}s_{24}s_{35}}\,I_{\rm
box}^{(4)}(s_{35})$
$\displaystyle-\,\frac{s_{13}s_{24}-s_{13}s_{25}+s_{14}s_{25}+s_{14}s_{35}-s_{24}s_{35})}{s_{14}s_{25}s_{35}}\,I_{\rm
box}^{(5)}(s_{14})\Bigg{\\}}$
$\displaystyle+\,2\epsilon\,I(1,1,1,1,1;-1+2\epsilon)\,.$

Since the integral in the last line is finite in $d=4$, this gives the well-
known result stating that the massless pentagon integral is given by a liner
combination of box integrals, up to corrections vanishing in four dimensions.

##  6 Two-loop examples

We now very briefly discuss the application of the method beyond one loop. A
first interesting case is given by the family of $l$-loop sunrise diagrams,
i.e. diagrams contributing to a two-point function, with two vertices
connected by $l+1$ propagators, illustrated in Fig. 1. These integrals have
been extensively studied in recent years, since they provide a natural
laboratory for multi-loop calculation, and in particular, with massive legs,
provide the simplest example of integrals involving elliptic curves, and thus
yielding functions beyond polylogarithms (see, for example, [33, 34, 35, 36,
37, 38, 39, 40, 41, 42, 43] and references therein).

The first Symanzik polynomial for $l$-loop sunrise integrals is given by

$\displaystyle\mathcal{U}_{l}\,=\,\sum_{i=1}^{l+1}z_{1}\ldots\hat{z_{i}}\ldots
z_{l+1}\,,$ (43)

where $\hat{z_{i}}$ denotes the omission of $z_{i}$. Eq. (43) displays the
high degree of symmetry of the graph, while the second Symanzik polynomial
${\cal F}$ depends on the configuration of masses on the internal legs.

$z_{1}$$z_{2}$....$z_{n+1}$ Figure 1:  Sunrise diagram at $l$ loops.

In the specific case of $l=2$ and equal internal masses, the Feynman
parametric integral is

$\displaystyle
I\big{(}\nu_{1},\nu_{2},\nu_{3};\lambda_{4}\big{)}\,=\\!\int_{S_{\\{1,2,3\\}}}\frac{\eta_{3}\,z_{1}^{\nu_{1}-1}z_{2}^{\nu_{2}-1}z_{3}^{\nu_{3}-1}\,\big{(}z_{1}z_{2}+z_{2}z_{3}+z_{3}z_{1})^{\lambda_{4}}}{\Big{[}r\,z_{1}z_{2}z_{3}-(z_{1}+z_{2}+z_{3})\big{(}z_{1}z_{2}+z_{2}z_{3}+z_{3}z_{1}\big{)}\Big{]}^{\frac{2\lambda_{4}+\nu}{3}}}\quad.$
(44)

By choosing a suitable numerator in our master identity, Eq. (14),

$\displaystyle
H(z)\,=\,z_{1}^{\nu_{1}-1}z_{2}^{\nu_{2}-1}z_{3}^{\nu_{3}-1}\big{(}z_{1}z_{2}+z_{2}z_{3}+z_{3}z_{1}\big{)}^{\lambda_{4}}\,,\
$ (45)

one easily derives integration by parts identities, and one can build a linear
system of differential equations that closes (as expected) on the three master
integrals, $I(1,1,1;3\epsilon)$, $I(2,1,1;1+3\epsilon)$, and the one-loop
tadpole integral $I(2,2;1+3\epsilon)$. Interestingly, also in this case the
non-vanishing boundary term provides an inhomogeneous contribution to the
system. It arises form the basis integral

$\displaystyle\int
d\omega_{1}\,=\,\frac{1}{2(1+\epsilon)}\int_{S_{\\{1,2\\}}}\\!\eta_{\\{1,2\\}}\,\frac{(z_{1}z_{2})^{\epsilon}}{\big{[}-(z_{1}+z_{2})\big{]}^{2+2\epsilon}}\,=\,\frac{(-1)^{2\epsilon}}{2+2\epsilon}\,\frac{\Gamma^{2}(1+\epsilon)}{\Gamma(2+2\epsilon)}\,,$
(46)

corresponding to the massive one-loop tadpole. In two space-time dimensions,
the sunrise integral is finite and the linear system can be analysed for
$\epsilon=0$. More precisely, as discussed in more detail in [24], the first-
order differential equations can be combined into a single second-order
equation for the equal-mass sunrise, which has long been known to be of
elliptic type [33, 44, 45]. We find

$\displaystyle\frac{r}{3}\,\frac{d^{2}}{dr^{2}}I(1,1,1;0)\,+\,\left(\frac{1}{3}+\frac{3}{r-9}+\frac{1}{3(r-1)}\right)\frac{d}{dr}I(1,1,1;0)$
$\displaystyle\hskip
73.97733pt\,-\,\left(\frac{1}{4(r-9)}+\frac{1}{12(r-1)}\right)I(1,1,1;0)\,=\,\frac{2}{(r-1)(r-9)}\,.$
(47)

It is important to note that the procedure we followed is not expected to
generalise smoothly to the two-loop sunrise diagram with unequal masses, since
a straightforward application of Stokes’ theorem in that case must take into
account the presence of singularities at the simplex boundaries: the
difference between the two cases is discussed in detail in Ref. [46]. We leave
the analysis of the general case to future work. On the other hand, we note
that our method readily reproduces the classic results of Ref. [3] for two-
point, five-propagator integrals, which can be systematically reduced to four-
propagator integrals yielding simple combinations of $\Gamma$ functions. Once
again, boundary terms play a distinctive role in parameter space, as discussed
in detail in [24].

##  7 Perspectives

In this note, we have summarised the results of Ref. [24], where we introduced
a projective framework for deriving IBP identities and differential equations
for Feynman integrals directly in parameter space, building upon very early
work by Tullio Regge and collaborators [17, 18, 19, 20, 21, 22]. Specifically,
we showed how these early techniques can be adapted to include dimensional
regularization, and how they can be generalised beyond one loop.

Comparing the parameter-space method to momentum-space approaches, it’s clear
that the organisation of calculations differs significantly. The integral
bases and the resulting differential equations generated by the projective
framework are in general distinct from the conventional ones. One notable
aspect of this framework is the role played by boundary terms, which vanish in
the momentum-space approach. In this case, instead, they play a crucial role,
linking complex integrals to simpler ones. We note also that parameter-space
integrands closely mirror the graph symmetries, and circumvent issues related
to loop-momentum routing and irreducible numerators, which can complicate
momentum-space algorithms. Importantly, the projective framework aligns
closely with the algebraic structures underpinning Feynman integrals, which
may provide direction for future progress.

The present work is largely a feasibility study: for the future, the goal is
clearly to extend these techniques to more complex integrals, including
higher-loop and multi-scale examples, possibly developing automated tools.
Besides the obvious interest in direct physics applications, this will allow
for a necessary detailed comparison of parameter-space and momentum-space
approaches, including computational aspects.

## Acknowledgments

Research supported in part by the Italian Ministry of University and Research
(MIUR), under grant PRIN 20172LNEEZ. DA is funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) - Projektnummer
417533893/GRK2575 "Rethinking Quantum Field Theory". We thank ChatGPT for
efficiently summarising some of our lengthy arguments, leading to a more
compact note.

## References

  * [1] G. Heinrich, Phys. Rept. 922 (2021), 1-69, [arXiv:2009.00516 [hep-ph]].
  * [2] F. V. Tkachov, Phys. Lett. B 100 (1981), 65-68.
  * [3] K. G. Chetyrkin and F. V. Tkachov, Nucl. Phys. B 192 (1981), 159-204.
  * [4] A. V. Kotikov, Phys. Lett. B 254 (1991), 158-164.
  * [5] E. Remiddi, Nuovo Cim. A 110 (1997), 1435-1452, [arXiv:hep-th/9711188 [hep-th]].
  * [6] T. Gehrmann and E. Remiddi, Nucl. Phys. B 580 (2000), 485-518, [arXiv:hep-ph/9912329 [hep-ph]].
  * [7] S. Laporta, Int. J. Mod. Phys. A 15 (2000), 5087-5159, [arXiv:hep-ph/0102033 [hep-ph]].
  * [8] C. Duhr, H. Gangl and J. R. Rhodes, JHEP 10 (2012), 075, [arXiv:1110.0458 [math-ph]].
  * [9] C. Duhr, Ann. Rev. Nucl. Part. Sci. 69 (2019), 15-39.
  * [10] S. Abreu, R. Britto and C. Duhr, J. Phys. A 55 (2022) no.44, 443004, [arXiv:2203.13014 [hep-th]].
  * [11] R. Britto, Phys. Rev. Lett. 131 (2023) no.9, 9, [arXiv:2305.15369 [hep-th]].
  * [12] J. M. Henn, Phys. Rev. Lett. 110 (2013), 251601, [arXiv:1304.1806 [hep-th]].
  * [13] S. Weinzierl, [arXiv:2201.03593 [hep-th]].
  * [14] R. J. Eden, P. V. Landshoff, D. I. Olive and J. C. Polkinghorne, Cambridge Univ. Press, 1966, ISBN 978-0-521-04869-9
  * [15] F. Pham, “Introduction à l’étude topologique des singularités de Landau”, Mémorial des Sciences Mathématiques, Fascicule 164 (Gauthier-Villars, Paris, 1967).
  * [16] J. Lascoux, “Perturbation theory in quantum field theory and homology”, in “Battelle Rencontres 1967”, Lectures in Mathematics and Physics, edited by C.M. Dewitt and J.A. Wheeler (Benjamin, New York, 1968).
  * [17] T. Regge, “Algebraic Topology Methods in the Theory of Feynman Relativistic Amplitudes”, in “Battelle Rencontres 1967”, Lectures in Mathematics and Physics, edited by C.M. Dewitt and J.A. Wheeler (Benjamin, New York, 1968).
  * [18] G. Ponzano, T. Regge, E. R. Speer and M. J. Westwater, Commun. Math. Phys. 15 (1969), 83-132.
  * [19] G. Ponzano, T. Regge, E. R. Speer and M. J. Westwater, Commun. Math. Phys. 18 (1970), 1-64.
  * [20] T. Regge, E. R. Speer and M. J. Westwater, Fortsch. Phys. 20 (1972), 365-420.
  * [21] G. Barucchi and G. Ponzano, J. Math. Phys. 14 (1973), 396-401.
  * [22] G. Barucchi and G. Ponzano, Nuovo Cim. A 23 (1974), 733-742.
  * [23] H. Cheng and T. T. Wu, “Expanding Protons: Scattering at High Energies”, MIT Press, Cambridge, MA, 1987.
  * [24] D. Artico and L. Magnea, [arXiv:2310.03939 [hep-ph]].
  * [25] C. Bogner and S. Weinzierl, J. Math. Phys. 50 (2009), 042302, [arXiv:0711.4863 [hep-th]].
  * [26] C. Bogner and S. Weinzierl, Int. J. Mod. Phys. A 25 (2010), 2585-2618, [arXiv:1002.3458 [hep-ph]].
  * [27] J. M. Henn, J. Phys. A 48 (2015), 153001, [arXiv:1412.2296 [hep-ph]].
  * [28] T. Bitoun, C. Bogner, R. P. Klausen and E. Panzer, Lett. Math. Phys. 109 (2019) no.3, 497-564, [arXiv:1712.09215 [hep-th]].
  * [29] T. Bitoun, C. Bogner, R. P. Klausen and E. Panzer, PoS LL2018 (2018), 065, [arXiv:1809.03399 [hep-th]].
  * [30] S. Mizera and S. Telen, JHEP 08 (2022), 200, [arXiv:2109.08036 [math-ph]].
  * [31] W. Magnus, Commun. Pure Appl. Math. 7 (1954), 649-673.
  * [32] Z. Bern, L. J. Dixon and D. A. Kosower, Nucl. Phys. B 412 (1994), 751-816, [arXiv:hep-ph/9306240 [hep-ph]].
  * [33] D. J. Broadhurst, J. Fleischer and O. V. Tarasov, Z. Phys. C 60 (1993), 287-302, [arXiv:hep-ph/9304303 [hep-ph]].
  * [34] S. Müller-Stach, S. Weinzierl and R. Zayadeh, Commun. Num. Theor. Phys. 6 (2012), 203-222, [arXiv:1112.4360 [hep-ph]].
  * [35] L. Adams, C. Bogner and S. Weinzierl, J. Math. Phys. 54 (2013), 052303, [arXiv:1302.7004 [hep-ph]].
  * [36] S. Bloch and P. Vanhove, J. Number Theor. 148 (2015), 328-364, [arXiv:1309.5865 [hep-th]].
  * [37] L. Adams, C. Bogner and S. Weinzierl, J. Math. Phys. 57 (2016) n. 3, 032304, [arXiv:1512.05630 [hep-ph]].
  * [38] J. Ablinger, J. Blümlein, A. De Freitas, M. van Hoeij, E. Imamoglu, C. G. Raab, C. S. Radu and C. Schneider, J. Math. Phys. 59 (2018) no.6, 062305, [arXiv:1706.01299 [hep-th]].
  * [39] C. Bogner, S. Müller-Stach and S. Weinzierl, Nucl. Phys. B 954 (2020), 114991, [arXiv:1907.01251 [hep-th]].
  * [40] J. L. Bourjaily, A. J. McLeod, M. von Hippel and M. Wilhelm, Phys. Rev. Lett. 122 (2019) n. 3, 031601, [arXiv:1810.07689 [hep-th]].
  * [41] J. Broedel, C. Duhr and N. Matthes, JHEP 02 (2022), 184, [arXiv:2109.15251 [hep-th]].
  * [42] K. Bönisch, C. Duhr, F. Fischbach, A. Klemm and C. Nega, JHEP 09 (2022), 156, [arXiv:2108.05310 [hep-th]].
  * [43] J. L. Bourjaily, J. Broedel, E. Chaubey, C. Duhr, H. Frellesvig, M. Hidding, R. Marzucca, A. J. McLeod, M. Spradlin and L. Tancredi, et al., contribution to Snowmass 2021, [arXiv:2203.07088 [hep-ph]].
  * [44] S. Laporta and E. Remiddi, Nucl. Phys. B 704 (2005), 349-386, [arXiv:hep-ph/0406160 [hep-ph]].
  * [45] P. Lairez and P. Vanhove, Lett. Math. Phys. 113 (2023) n. 2, 37, [arXiv:2209.10962 [hep-th]].
  * [46] S. Weinzierl, J. Math. Phys. 62 (2021) no.7, 072301, [arXiv:2002.01930 [math-ph]].


#  On the Higgs-Confinement Complementarity

Erhard Seilera**$*$email<EMAIL_ADDRESS>

aMax-Planck-Institut für Physik (Werner-Heisenberg-Institut)
München, Germany

###### Abstract

It has been noticed long ago that in Higgs models with ‘complete symmetry
breaking’ one can move from the confinement to the Higgs regime without
crossing a phase boundary, a fact sometimes called referred to as
‘complementarity’. In a recent paper some doubt was raised about the
correctness of the mathematics underlying this fact and it was claimed that
the supposed ‘flaw’ would resolve the ‘paradox’ seen in this complementarity.
Here we briefly revisit the facts both from a mathematical and a physical
point of view and point out that (a) there is no paradox and (b) there is no
flaw in the mathematical reasoning.

##  1 Introduction

After K. Wilson [1] created lattice theory and thereby found a well-defined
nonperturbative formulation of gauge theories (albeit with an ultraviolet
cutoff) that did not require gauge fixing, it became possible to address
questions not answerable in perturbation theory. The first and most important
such question, the question of confinement of quarks, was answered by Wilson
himself (with later mathematical refinement in [2]) for strong enough bare
coupling.

Very soon the interest turned to the Higgs mechanism, which in perturbation
theory with gauge fixing is conventionally described as spontaneous symmetry
breaking (SSB). In [2] it was shown using a cluster expansion that a Yang-
Mills-Higgs model having a property conventionally denoted as ‘complete
breakdown of symmery’ has a mass gap at arbitrarily weak gauge coupling,
provided the Higgs potential is sufficiently strong. This result was analyzed
in [3] and physically interpreted as the absence of a phase boundary between
‘confinement’ and ‘Higgs’ regions, a property that became known as
‘complementarity’.

Lately is has been been claimed by Grady [4] that the proof of the relevant
theorem in [2] contains a flaw, that the theorem is invalid and thereby a
‘paradox’ is solved. Here we will first remind the reader why there is no
paradox; to set the record straight we then briefly revisit the arguments
entering the proof and show why there is no flaw.

##  2 There is no paradox

G. ’t Hooft [5] in 1979 explained confinement-Higgs complementarity in
physical terms. He showed that the Higgs mechanism may just as well be
interpreted as confinement: bare Yang-Mills quanta and bare Higgs quanta may
be combined into gauge invariant, permanently bound compounds forming the
gauge invariant physical ‘W’ and Higgs bosons, just as quarks and gluons are
bound into hadrons. This is gratifying, because from Elitzur’s theorem [6] we
know that gauge invariance cannot be broken spontaneously if no gauge fixing
term is present. This fact is in some sense obvious, because the local
invariance means that there is no preference for aligning even nearby degrees
of freedom (i. e. there is not even short range order).

A little later Fröhlich, Morchio, and Strocchi [7] set out to formulate
perturbation theory for such models in terms of gauge invariant composite
fields without the usual assumption of SSB via a nonvanishing expectation
value of the Higgs field. The procedure outlined there is not necessarily
useful for computational purposes, but it is important to recognize that it is
possible in principle. Fröhlich et al also showed in that paper that SSB is
absent even in the presence of certain gauge fixes.

Another important, unfortunately much too little recognized result is due to
Kennedy and King [8]. They considered abelian lattice gauge theories in 4
dimensions with a covariant gauge fixing term, as employed usually in
continuum perturbation theory. There finding was: SSB occurs only in the
Landau gauge, i.e. the gauge with minimal infrared fluctuations. In all other
covariant gauges infrared fluctuations destroy SSB, much like infrared
fluctuations destroy long range ordering in 2D ferromagnets (a fact known as
the Mermin-Wagner theorem [9]).

Provocately one might say that SSB in Higgs models, if it occurs at all, is a
gauge fixing artefact.

##  3 There is no flaw in the proof of analyticity

Here want to briefly review the structure of the proof of analyticity given in
[2], in order to clear up some misconceptions. It proceeds by a convergent
cluster expansion in the form developed by Glimm, Jaffe and Spencer [10] in
the context of Constructive Quantum Field Theory. This expansion expresses
expectation values first as a sum of finite volume quantities which are
clearly analytic in the parameters; then it is shown that the expansion
converges uniformly in the volume in a certain complex domain of parameters.
This by itself is, however, not sufficient to prove analyticity of the
infinite volume limit, as correctly remarked by Grady (it is also not
sufficient to establish the existence of that limit). Grady goes on to suggest
that the limits of the individual terms might be nonanalytic.

But in [2] it is remarked that existence of the infinite volume limit and its
analyticity follow by standard applications of the cluster expansion, for
which the authors refer to [10]. Obviously the authors of [2] did not want to
repeat the arguments given in [10], as they were considered ‘standard’ in
their community. So let me repeat the structure of the argument for
analyticity of the infinite volume limit:

(1) The cluster expansion converges uniformly (in the volume) in a certain
complex domain in the parameters of the model (the conditions necessary for
lattice gauge theories are given in OS).

(2) From this follows uniform (in the volume) exponential clustering for pairs
of observables.

(3) This clustering in turn implies convergence to the thermodynamic limit
because boundary effects decouple exponentially.

(4) Vitali’s convergence theorem for holomorphic functions then implies
analyticity of the limit in the domain specified before.

It should also be said that there are other, possibly more transparent ways of
arriving at the results; in particular so-called polymer expansions described
in [11] and later in the textbook [12] produce a somewhat different series;
the main advantage of this approach is the fact that every term in the series
reaches its thermodynamic limit already at a certain finite volume, so
analyticity of each term is obvious, because it is equal to a finite volume
quantity.

##  4 Conclusions

On the lattice there is, at least in the case of ‘complete symmetry
breakdown’, no phase boundary completely separating the ‘Higgs’ and
‘Confinement’ regimes (just as in the liquid-gas case); the old proofs are
valid. This is, however, not a paradox; the phenomenological explanation
provided by ’Hooft makes the fact intuitively plausible.

## References

  * [1] K. G. Wilson, Phys. Rev. D 10 (1974) 2445.
  * [2] K. Osterwalder and E. Seiler, In: *Rome 1977, Proceedings, Mathematical Problems In Theoretical Physics*, Berlin: Springer 1978, 26-36; Annals Phys. 110 (1978) 440.
  * [3] E. H. Fradkin and S. H. Shenker, Phys. Rev. D 19 (1979) 3682.
  * [4] M. Grady, “Solution to the gauge-Higgs analyticity paradox,” arXiv:1502.04362 [hep-lat].
  * [5] G. ’t Hooft, “Which Topological Features of a Gauge Theory Can Be Responsible for Permanent Confinement?,” Lecture given at Cargese Summer Inst.1979, NATO Sci. Ser. B 59 (1980) 117.
  * [6] S. Elitzur, Phys. Rev. D 12 (1975) 3978.
  * [7] J. Fröhlich, G. Morchio and F. Strocchi, Nucl. Phys. B 190 (1981) 553; Phys. Lett. B 97 (1980) 249.
  * [8] T. Kennedy and C. King, Phys. Rev. Lett. 55 (1985) 776; Commun. Math. Phys. 104 (1986) 327.
  * [9] N. D. Mermin and H. Wagner, two-dimensional isotropic Heisenberg models,” Phys. Rev. Lett. 17 (1966) 1133.
  * [10] J. Glimm, A. Jaffe, T. Spencer, in: “Constructive Quantum Field Theory”, G. Velo, A. Wightman (Eds.), Springer Lecture Notes in Physics, vol. 25, Berlin/New York: Springer(1973); Ann. of Math. (2) 100 (1974), 585.
  * [11] E. Seiler, “Gauge Theories as a Problem of Constructive Quantum Field Theory and Statistical Mechanics,” Lect. Notes Phys. 159 (1982) 1.
  * [12] J. Glimm and A. M. Jaffe, “Quantum Physics. A Functional Integral Point Of View,” New York: Springer (1987) 535p


# Defining Problem from Solutions: Inverse Reinforcement Learning (IRL) and
Its Applications for Next-Generation Networking

Yinqiu Liu, Ruichen Zhang, Hongyang Du, Dusit Niyato, , Jiawen Kang, Zehui
Xiong, and Dong In Kim Y. Liu, R. Zhang, H. Du, and D. Niyato are with the
School of Computer Science and Engineering, Nanyang Technological University,
Singapore (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and dniyato@ntu.edu.sg).J. Kang is with the School
of Automation, Guangdong University of Technology, China (e-mail:
kavinkang@gdut.edu.cn).Z. Xiong is with the Pillar of Information Systems
Technology and Design, Singapore University of Technology and Design,
Singapore (e-mail: zehui xiong@sutd.edu.sg).D. I. Kim is with the Department
of Electrical and Computer Engineering, Sungkyunkwan University, Suwon 16419,
South Korea (e-mail:dikim@skku.ac.kr).

###### Abstract

Performance optimization is a critical concern in networking, on which Deep
Reinforcement Learning (DRL) has achieved great success. Nonetheless, DRL
training relies on precisely defined reward functions, which formulate the
optimization objective and indicate the positive/negative progress towards the
optimal. With the ever-increasing environmental complexity and human
participation in Next-Generation Networking (NGN), defining appropriate reward
functions become challenging. In this article, we explore the applications of
Inverse Reinforcement Learning (IRL) in NGN. Particularly, if DRL aims to find
optimal solutions to the problem, IRL finds a problem from the optimal
solutions, where the optimal solutions are collected from experts, and the
problem is defined by reward inference. Specifically, we first formally
introduce the IRL technique, including its fundamentals, workflow, and
difference from DRL. Afterward, we present the motivations of IRL applications
in NGN and survey existing studies. Furthermore, to demonstrate the process of
applying IRL in NGN, we perform a case study about human-centric prompt
engineering in Generative AI-enabled networks. We demonstrate the
effectiveness of using both DRL and IRL techniques and prove the superiority
of IRL.

###### Index Terms:

Inverse Reinforcement Learning (IRL), Next-Generation Networking (NGN),
Generative AI (GAI), Reward Engineering, Deep Reinforcement Learning (DRL).

##  I Introduction

Deep Reinforcement Learning (DRL) has become indispensable in many fields,
such as networking, robotics, and finance [1]. Following the Reinforcement
Learning (RL) principle, an agent can optimize its decision-making capability
by iteratively interacting with an environment, aiming to maximize cumulative
rewards. Meanwhile, Deep Neural Networks (DNNs) enhance RL by enabling agents
to represent complex environments and learn sophisticated policies. Although
such a paradigm demonstrates remarkable success, explicitly defining or even
determining rewards can be challenging in many real-world scenarios [2] due to
environmental complexity, human participation, or information asymmetry. Take
task offloading in mobile edge networks as an example, where users select edge
servers to maximize their Quality of Experience (QoE). We know that several
factors are related to QoE, such as service latency and fees. However, the
weight and relationship of observed factors, as well as how to fuse them
appropriately to model user-side experience, are unknown. Moreover, the
unobserved subjectivity of users, such as personal inner and behavioral
preferences, should also be considered. Without knowing what constitutes the
reward, a trained DRL policy might exhibit severe sub-optimality and bias.

Fortunately, Inverse Reinforcement Learning (IRL) emerges as a pivotal
solution to overcome the obstacles caused by reward inaccessibility [3].
Specifically, IRL enhances DRL by introducing reward inferences. In the above
example, instead of manually defining a reward function without any accurate
prior knowledge and precision guarantee, IRL utilizes DNNs to infer the
rewards that can effectively explain user behaviors. Such behaviors are called
expert trajectories and should be collected before training IRL. Successful
IRL applications include ChatGPT, which involves large-scale human feedback to
fine-tune model generation111https://cgnarendiran.github.io/blog/chatgpt-
future-of-conversational-ai/. Meanwhile, IRL has been widely adopted in human-
in-the-loop networking and systems, such as autonomous driving assistance [4].
We conclude that IRL owns the following benefits.

  * •

Environment Exploration: IRL provides a means to break the information
asymmetry and explore complex or adversarial environments. By leveraging the
inferred reward functions, agents are not only guided toward optimal policies
but are also encouraged to explore uncharted territories within the
environment. For instance, users collect malicious servers’ behaviors to infer
their objectives and patterns, thus taking corresponding defenses [3].

  * •

Behavior Understanding: By inferring reward functions from observed expert
behaviors, IRL offers profound insights into the underlying motivations of
agents, enabling a deeper comprehension of complex behaviors. For example,
human driving patterns are complicated since actions taken by drivers
according to different traffic conditions are based on their empirical
experience, driving skills, and preferences. Leveraging DNNs to represent such
non-linear features, IRL can learn a reward function that aligns with the
driving behaviors222https://meixinzhu.github.io/project/irl/.

  * •

Policy Imitation: IRL excels in distilling policies from demonstrations,
allowing the agents to imitate desired behaviors. This capability is
especially beneficial in scenarios where the desired outcome is known but the
path to achieving it is not. In the driving example, after acquiring a reward
function, an autonomous driving agent can be trained to imitate the expert
drivers’ behaviors.

In this article, we delve into the applications of IRL in Next-Generation
Networking (NGN), which is the foundation of numerous advanced technologies,
e.g., 6G communications, Metaverse, and Internet of Everything. According to
AT&T333https://www.business.att.com/learn/tech-advice/what-is-next-generation-
network-ngn-technology-explained.html, NGN is projected to accommodate massive
devices, support diverse communication/network protocols, and provide
immersive services to users. Such explosive growth in networking scale,
topology, and complexity greatly increases the difficulty of defining
optimization objectives and rewards precisely. Noticing such issues, several
studies [2, 5] adopted IRL to solve certain networking problems, such as
workload balancing, routing schedules, and attack detection, while an in-depth
analysis of IRL-enabled NGN is missing. We conclude that three key questions
are still yet to be answered.

  * •

Q1: How can IRL help NGN, and in which ways?

  * •

Q2: How to design IRL algorithms to serve specific NGN scenarios?

  * •

Q3: What are the open issues and future directions in this topic?

To this end, we provide forward-looking research on IRL-enabled NGN, as well
as conducting a case study for demonstration. To the best of our knowledge,
this is the first work answering why IRL is essential for NGN and showing how
to deploy IRL algorithms to solve real-world NGN problems. Our main
contributions are summarized as follows.

  * •

We comprehensively discuss the fundamentals of IRL. Specifically, based on
Markov Decision Process (MDP) and RL principles, we introduce the basics of
IRL and several representative IRL algorithms. Additionally, we analyze the
benefits of enhancing DRL by IRL.

  * •

We explore the applications of IRL in NGN. To do so, we first illustrate the
driving forces existing in NGN to promote IRL adoption. Afterward, we review
the existing literature on IRL-enabled NGN, as well as the limitations of the
current IRL techniques.

  * •

We conduct a case study to demonstrate the process of applying IRL in NGN. In
particular, we showcase a human-centric prompt engineering scheme for
Generative AI (GAI)-enabled networks. Experimental results prove that IRL can
effectively infer unobserved rewards of human users and achieve higher
experience.

Figure 1: The fundamentals of IRL (a): The cases of defining reward functions.
(b): The illustration of MDP. (c): The workflow of DRL algorithm. (d) The
workflow of IRL. Note that we illustrate four representative approaches to
infer reward, namely maximum margin, maximum entropy, GAIL, and RLHF.

##  II Fundamentals of Inverse Reinforcement Learning

In this section, we comprehensively discuss the fundamentals of IRL, including
MDP, conventional DRL, the basics of IRL, representative IRL algorithms, and
IRL’s advantages.

###  II-A Preliminaries

####  II-A1 Reward Engineering

To precisely formulate an optimization problem, the reward function should be
defined appropriately. The term “reward” indicates the desirability of each
action. By assigning positive/higher and negative/lower reward values to the
actions that make positive and negative progress, respectively, the
optimization objective can be described and the optimal can be gradually
approached. Finally, the inputs of reward functions are observed/unobserved
factors of the environments and stakeholders.

As shown in Fig. 1(a), users may encounter different situations when defining
reward functions. First, if their objective is straightforward, such as
minimizing the throughput, it can be directly adopted as the reward. If
multiple physical factors, e.g., throughput, latency, and package loss, are
involved, the reward function should fuse them in linear/non-linear manners
and acquire a combined expression. Note that the difficulty and potential
errors grow dramatically with the increasing number of factors. Furthermore,
in human-in-the-loop networks, the unobserved subjective factors of users
should be modeled and considered as well, which is challenging. The precision
of reward functions is measured by how well they can describe optimization
objectives and elicit desired behaviors.

####  II-A2 Markov Decision Process

Fig. 1(b) illustrates an MDP [1], the building block for DRL and IRL. As a
discrete-time stochastic control process, MDP presents a mathematical
framework for modeling sequential decision-making in environments with
uncertainty. The basic MDP is constructed by a four-element tuple
$\langle$states, actions, transition probabilities, rewards$\rangle$. These
components describe the environment in which an agent performs certain
actions, acquires rewards, and transits from the current state to the next
state.

  * •

States: The set of all possible situations, each of which encapsulates various
configurations or factors that describe the current environment.

  * •

Actions: For each state, there are actions available to the agent that can
change the state.

  * •

Transition Probabilities: The likelihood of moving from one state to another,
given an action taken by the agent.

  * •

Rewards: After taking an action and moving to a new state, the agent will
receive a reward, whose meaning is discussed before. The reward function can
be defined by mathematical expressions or be inferred by DNNs, which
correspond to DRL and IRL, respectively.

In the NGN context, most of the optimization problems can be modeled as an
MDP, such as resource allocation, routing, attacks/defenses, and sub-carrier
selection [1]. Hence, acquiring a reliable and effective method for MDP
optimization is of significant importance in promoting NGN.

###  II-B Deep Reinforcement Learning

Combining reinforcement learning (RL) principles with the representational
capabilities of DNNs, DRL is the most famous and efficient approach to solving
complex decision-making tasks formulated as MDP. Next, we introduce the basic
idea, architecture, and process of DRL.

####  II-B1 Basic Idea

The fundamental goal of DRL is to derive an optimal policy that maximizes the
cumulative reward in an MDP setup [1]. Through iterative interaction with the
environment, an agent learns to refine its decision-making strategy based on
received feedback. DNNs play a crucial role in this process, approximating the
functions that predict the future rewards of actions, thereby guiding the
agent toward optimal decision-making.

####  II-B2 General Architecture

As shown in Fig. 1(c), the general DRL framework consists of several essential
components:

  * •

Agent: The decision-maker that interacts with the environment, i.e., taking
actions, transiting to the next state, and receiving rewards.

  * •

Environment: The system that defines the state and action spaces and allows
agents to perform MDP.

  * •

Policy Network: A neural network that maps states to actions, defining the
agent’s behavior.

  * •

Value Network: A neural network that estimates future rewards from states or
state-action pairs.

  * •

Replay Buffer: A repository of numerous past trajectories, enabling the agent
to reuse historical training data and reduce training costs.

  * •

Optimizer: The mechanism that adjusts neural network parameters to maximize
expected cumulative rewards under the current policy.

Figure 2: The motivation of applying IRL in NGN, using QoE optimization for
task offloading as an example. A: The attackers’ information is hidden from
users. B: Subjectivity factors can hardly be represented precisely. C: The
immediate reward of each action is unobserved. Action 1 and 1’ choose servers
$H$ and $G$ for task offloading, respectively. D: SAGIN communications involve
numerous physical factors from diverse devices, causing huge difficulty in QoE
modeling. E: The human driving behaviors contain complicated physiological
processes that can hardly be described by manually designed rewards.

####  II-B3 Algorithm Process

To illustrate the DRL process, we show an example of leveraging Proximal
Policy Optimization (PPO) [6] to offload tasks. In this case, agent, action,
and state refer to the user, the selection of task offloading server, and the
edge network, respectively. In addition, reward can be defined by fusing
multiple factors, such as latency and success rate, thereby indicating user
QoE. Afterward, the PPO process is streamlined into three main stages, namely
data collection, advantage estimation, and policy optimization. Initially, the
agent interacts with the edge network using the current server selection
policy and gathers $\langle$action, state, reward$\rangle$ records. Such
records are buffered to calculate advantage estimates, which assess the
relative value of actions. Then, PPO optimizes a specially designed objective
function that includes a clipping mechanism. This mechanism prevents the new
policy from straying too far from the old one, ensuring small, stable updates
and avoiding drastic performance fluctuations. By iteratively cycling through
these stages, PPO gradually fine-tunes the policy, striking a balance between
exploring new strategies and exploiting known rewards, making it adept at
navigating complex environments.

###  II-C Inverse Reinforcement Learning

Despite the success of DRL, in many network scenarios, precisely defining the
reward function is impossible. In the above example, although manually-defined
QoE can reflect users’ preference for low latency and high success rate, the
subjectivity factors are ignored. For instance, users with time-sensitive
tasks may emphasize latency, while secure computing tasks should put success
rate at the highest priority. Following biased reward functions, the policy
training may deviate from the real optimization objective. Accordingly, IRL is
presented to infer such sophisticated reward functions by observing and
analyzing users’ behaviors [2]. As shown in Fig. 1(d), in the beginning, the
uses’ selections of edge servers in different network states are collected,
called expert datasets/trajectories. Afterward, instead of simply learning the
action policy, IRL first infers the form of the reward function that best
explains the demonstration behaviors. Moreover, it can optimize the agent’s
policy towards closely aligning with or mimicking the expert policy according
to the inferred reward function. Given the strong representation and learning
ability, the reward function is generally inferred by a DNN.

###  II-D Evolution of Inverse Reinforcement Learning

The inference of reward functions can be implemented following different
approaches. As illustrated in Fig. 1(c), we introduce four representative and
widely adopted IRL algorithms, from basic maximum margin to advanced
generative imitation learning.

####  II-D1 Maximum Margin

This algorithm [4] focuses on distinguishing the expert’s policy from all
other policies by a margin that depends on the difference in the accumulated
reward. It operates under the principle that the correct reward function
should make the expert behavior appear significantly better than all other
possible behaviors. This method is particularly useful in scenarios where
clear demarcation between optimal and suboptimal policies is possible.

####  II-D2 Maximum Entropy

This method [4] introduces the concept of entropy to address the ambiguity in
the reward function that could explain observed behaviors. It assumes that
among all possible reward functions, the one that leads to the observed
behavior while also maximizing entropy, or in other words, promoting diverse
but consistent behaviors, is the most appropriate. Compared with maximum
margin, this approach is adept at handling the inherent uncertainty in
determining why an agent acts a certain way by favoring reward functions that
support a wide range of plausible actions.

####  II-D3 Generative Imitation Learning

Recall that reward functions serve as an indicator to guide policy refinement.
Apart from inferring an explicit reward value for each action and maximizing
the reward expectation, the policy can be refined in an imitation manner. For
instance, GAIL [4] leverages the Generative Adversarial Networks (GANs)
framework, where a DNN-based generator aims to approach expert behavior.
Meanwhile, another DNN acts as the discriminator, trying to differentiate
between the expert’s actions and those of the generator. The two modules are
trained alternately, i.e., freezing one module and adjusting the parameters of
the other module to optimize its behavior imitation/differentiation
performance. Afterward, the generator can effectively mimic the expert policy
and solve MDP problems without requiring reward inference. Accordingly, the
computational efficiency can be significantly improved.

####  II-D4 Reinforcement Learning with Human Feedback

In many cases, the reward function is unavailable since reward values are
given by humans. Human perception and evaluation is a complex physiological
process and is affected by various subjective factors such as preference,
personality, environment, and strictness. As a variant of IRL, RLHF combines
the RL principle with direct feedback from humans, integrating subjective
insights into the learning process. Specifically, a reward model will be first
trained based on action-score pairs, where scores are manually annotated by
humans. Then, the DRL can be leveraged for policy optimization, ensuring the
policy aligns with human preferences.

###  II-E Comparison and Summary

Based on the above descriptions, the difference between DRL and IRL can be
summarized as follows. First, DRL operates within a well-defined MDP
environment with available four-element tuples. IRL, conversely, applies to
incomplete MDPs, where the reward function is unknown. Accordingly, the
primary goal in DRL is to learn an optimal policy that maximizes cumulative
rewards. IRL aims to infer the reward function based on the expert’s
demonstration to understand the objectives the agent is implicitly striving
for. The inferred reward then guides the policy refinement. Finally, the DRL
policies are optimized by iteratively interacting with the environment. In
contrast, IRL relies on offline expert demonstrations, learning indirectly
from optimal behaviors and inferring the rewards that can elicit such
behaviors. Overall, DRL is used to find a solution to the problem while IRL is
used to find a problem from the solutions.

The most significant advantage of IRL is its ability to infer unobserved
reward functions precisely. Such capability greatly enhances conventional DRL
since if the reward is defined inappropriately, the trained policy cannot
solve the optimization efficiently. In addition, IRL, especially RLHF, excels
in understanding and mimicking human-like decision-making behaviors, making it
ideal for applications requiring nuanced behavior modeling. Last but not
least, IRL can provide insights into the underlying motivations and objectives
behind observed actions, contributing to understanding environments.

TABLE I: Summary of applications of IRL in networking.

##  III Inverse Reinforcement Learning in Next-Generation Networking

###  III-A Motivations from Next-Generation Networking

According to AT&T444https://www.business.att.com/learn/tech-advice/what-is-
next-generation-network-ngn-technology-explained.html, NGN refers to the
evolution and migration of fixed and mobile networking infrastructures from
distinct and proprietary networks to converged networks with high efficiency,
security, and experience. Hence, the manners of networking organization,
management, and service provisioning will undergo significant changes [7]. In
this part, we utilize QoE maximization for task offloading as an example to
demonstrate the driven force in NGN that promotes the further development of
IRL (see Fig. 2).

  * •

Reward Unavailability: In some NGN scenarios, rewards are unavailable to
decision-makers. As shown in Fig. 2(a), users may encounter malicious
offloading servers, which steal sensitive information for attacks. The
intelligent defense against such malicious behaviors relies on detecting
attackers’ objectives, which are hidden from users. In addition, the
complicated topology of NGN leads to sophisticated RL state spaces [7]. For
instance, Fig. 2(b) illustrates a case where offloading servers are structured
as a tree and users choose an offloading service chain from the leaf to the
root. In this case, the immediate reward of performing each action is
inaccessible since the association between the current state and the final
result is unknown. Finally, NGN exhibits human-in-the-loop features since it
aims to support diverse advanced applications and ensures high human-
perceptual experiences [7]. Take the offloading of image generation tasks as
an example (see Fig. 2(c)), user QoE depends not only on objective factors
such as service latency and fee but also on users’ preference for the painting
style, whose modeling is intricate.

  * •

Environmental Complexity: Given massive devices, as well as the advancement of
access protocols and communication diagrams, NGN exhibits environmental
complexity. For instance, in Space-Air-Ground Integrated Network (SAGIN),
multiple physical factors of the devices in each layer explicitly or
implicitly contribute to QoE, while the specific contribution is unknown (see
Fig. 2(d)). In this case, the annually defined reward function may suffer from
incomprehensiveness and bias.

  * •

Policy Optimization: If expert behaviors that maximize QoE are available,
training the agent to mimic the expert can lead to the highest efficiency for
QoE optimization. However, manually designing a reward function can hardly
realize the action imitation. Fig. 2(e) demonstrates an example of autonomous
driving. Contributed to the strong capability of DNNs, IRL can mimic the
expert driver with perfect QoE effectively by inferring the reward that
elicits desired behaviors.

###  III-B Applications of Inverse Reinforcement Learning in Next-Generation
Networking

In this part, we review related works about IRL in networking, as shown in
TABLE I. Particularly, our survey is organized from the three perspectives
shown above.

####  III-B1 Infer Unobserved Rewards

As mentioned in Section II, IRL is effective in inferring unobserved rewards
caused by adversarial relationships or human participation in the networks.
For instance, Snow et al. [5] applied multi-agent IRL to identify coordination
in cognitive radar networks, i.e., multiple radars collaborate to track one
target. Since the radars are adversarial, the target cannot identify their
numbers, configurations, and individual utility functions. Leveraging the IRL
concept, the target first determines if collaboration exists based on whether
the emissions satisfy Pareto optimally [5]. If so, it then uses the emissions
as the expert trajectories to infer the individual utility of each radar.
Likewise, Parras et al. [3] leveraged GAIL to enhance IoT security. This is
because, nowadays, DRL is widely adopted by attackers to create new attacks.
Specifically, they select the attack objective, adopt DRL to train the optimal
attack strategy and launch the attacks on the victims. Accordingly, defenders
can adopt IRL methods, such as GAIL, to infer the objective of the attackers
and thus refine the defense policies.

In addition to adversarial networks, Wang et al. [8] discussed the
applications of IRL in cases where the reward of each action cannot be
mathematically calculated. Specifically, they utilize the branch and bound
(B&B) algorithm to realize full-dimensional task offloading in edge computing.
Although the B&B process can be modeled as an MDP, it is very challenging to
design an appropriate reward function since the B&B algorithm. Before
obtaining an optimal solution, the branching order of variables cannot be
known in advance. That is because the B&B algorithm solves the problem by
breaking it down into smaller sub-problems and using a bounding function to
eliminate sub-problems with limited upper bounds, forming a tree-structured
solution space. In this case, we cannot know that branching which variable can
generate a smaller enumeration tree. Therefore, the authors present a Graph-
based IRL (GIRL), which uses a Graph Neural Network (GNN) to infer the
immediate reward of each action based on the structure of the tree.

Finally, the concept of IRL has been widely adopted in human-in-the-loop
networks. For instance, the training of ChatGPT, the most famous multimodal
AIGC model, involves self-reinforcement from large-scale human feedback,
thereby aligning the model output with human preferences.

####  III-B2 Overcome Complex Network Environments

Owing to complex network environments, even if manually defining some vague
reward functions is available, it can hardly precisely represent the
environmental change due to the agent’s actions and efficiently indicate the
desirability. In [9], the authors presented an IRL-based approach to predict
QoS in dynamic and time-variant networks. Due to large state spaces and
complex projections between routes and QoS values, it is difficult to define a
precise reward function artificially. Likewise, Konar et al. [10] presented
the trajectory extrapolation algorithm to model the quality of experience
(QoE) for communication load balancing. Particularly, they consider the
difficulty of gathering exert trajectories in practice. Hence, their proposed
algorithm first sorts the gathered trajectories according to some objective
and well-established metrics. Then, the trajectories are sampled to facilitate
the reward inference and policy learning. In UAV networks, Shamsoshoara et al.
[11] developed the interference-aware joint path planning and power allocation
scheme to minimize the interference that UAVs cause to the existing ground
user equipment. IRL is applied to capture all physical factors, including the
UAV’s transmission power, terrestrial user density, imposed interference, and
the UAV’s task completion capability, thereby inferring the optimization
objective.

####  III-B3 Guide Policy Optimizations

In some cases where the optimal trajectories are available, directly inferring
the reward function that facilitates the agent to mimic the expert can lead to
the best policy optimization efficiency. For instance, Zhang et al. [12]
adopted maximum entropy to optimize the power allocation schemes in multi-user
cellular networks. Experiments show that IRL greatly outperforms manually
defined reward functions based on objective metrics, such as weighted minimum
mean square error. Similarly, Tian et al. [13] applied this principle to
multi-cell multi-user networks to maximize the sum rate. Furthermore, in [14],
they adopted the GAIL method to minimize power usage in multiple-input-single-
output networks with multiple users. Compared with conventional DRL, the
proposed method can reduce power consumption by over 50%.

Lesson Learned: From the above brief survey, we learn that IRL enhances the
capabilities of DRL by introducing a reward inference step. Contributed to
DNN’s stronger representation and learning capabilities than humans, IRL can
well overcome complex environments, actions, and reward perception processes
(such as human perception of AI-generated images). Such advantages make IRL
inherently suitable for scenarios that are complex while existing expert
trajectories for demonstration. Nonetheless, IRL also exhibits certain
drawbacks, such as the additional computational costs for reward inferences
and the dependency on expert trajectories, which are subject to further
research.

Figure 3: The illustration of our case study. A: The system model of GAI-
enabled network. B: The efficacy of prompt engineering. We can observe that
the image generated from the crafted prompt excels in object rendering and
image composition. C: The illustration of DRL-based prompt engineering. D: The
illustration of IRL-based prompt engineering.

##  IV Case Study: Human-oriented Prompt Engineering in Generative AI-Enabled
Networks

To illustrate how to solve practical NGN problems by IRL, this section
performs a case study about human-oriented prompt engineering. Particularly,
we simultaneously present the DRL- and IRL-based solutions to the same task,
helping readers compare these two methods. The effectiveness of DRL and IRL is
also discussed.

###  IV-A System Model

With the rapid advancement of GAI, generative tasks, such as text-to-image
generation, automatic conversation, and 3D avatar renderings, play an
important role in NGN. As shown in Fig. 3(A), we consider a GAI-enabled
network with users and service providers. Users describe their requests by
natural language (so-called prompt) and submit them to the service providers.
Operating professional GAI models, the service providers perform generative
inferences and generate required content for users.

Nonetheless, due to the lack of professionality/experience, user prompts may
suffer from information insufficiency and ambiguity, causing low generation
quality. To this end, service providers can strategically craft prompts that
maintain semantic equivalence with raw prompts while being informative, clear,
and suitable for the specific GAI model [15]. Such a process is called prompt
engineering [15]. Taking text-to-image generation as an example, Fig. 3(B)
depicts the impact of prompt engineering on the generated image. We can
observe that the image generated by the crafted prompt outperforms in
exquisiteness and composition. Despite such advantages, service providers may
own multiple prompt engineering approaches since the prompts are crafted by
open vocabularies. In this case, how to select the optimal prompt engineering
approach according to the specific request is a remaining challenge. Next, we
solve such an optimization problem following DRL and IRL paradigms,
respectively.

###  IV-B DRL- and IRL-based Prompt Engineering

####  IV-B1 DRL Workflow

First, we solve this problem following the DRL paradigm. Specifically, we
adopt the PPO algorithm discussed in Section II-B. As shown in Fig. 3(C), the
service provider acts as agent, whose actions include all available operations
to refine raw prompts. The raw prompts from users are accommodated by
environment. Finally, reward evaluates the efficacy of applying the selected
prompt engineering operation on the current raw prompt. To this end, we
leverage Image-reward555https://github.com/THUDM/ImageReward, a learning-based
metric for measuring the aesthetic quality of AI-generated images, as the
reward function. Correspondingly, the quality score of the image generated
from the crafted prompt is utilized as the reward. The DRL architecture and
algorithm workflow follow the description in Section II-B, with the goal of
optimizing the policy for selecting prompt engineering operations.

####  IV-B2 IRL Workflow

Then, we apply IRL to solve the same problem. Different from DRL, as shown in
Fig. 3(d), IRL include the following three steps.

  * •

Expert Dataset Collection: First, we construct an expert dataset to indicate
human preference for AI-generated images. Particularly, inspired by the
outstanding capability of Large Language Models (LLMs) in multimodal
understanding [15], we leverage an LLM-empowered agent to mimic real users.
Then, we randomly compose 50 raw prompts and perform all the available prompt
engineering operations on each of them. The agent is utilized to evaluate the
generated images by scores. Note that we adopt two strategies for training
LLM, ensuring it generates rationale scores. First, we instruct LLM to act as
experienced users, recalling its pre-trained knowledge of image quality
assessment. Additionally, we feed ten materials regarding computer vision,
image generation, and painting to the LLM, thus enhancing its expertise. All
the $\langle$raw prompt, crafted prompt, score$\rangle$ pairs construct the
expert dataset, showing which kind of prompt engineering is preferred by
humans for the specific image generation task.

  * •

Policy Optimization: Different from defining rewards via an existing metric,
IRL utilizes DNNs to infer reward functions that align with expert decisions.
To improve the learning efficiency, we adopt GAIL explained in Section II-D.
Specifically, the discriminator distinguishes between the selections of expert
prompt engineering policy and those of the agent’s policy, thereby self-
calibrating rewards to align with the expert decision-making mode. Meanwhile,
the generator aims to learn a prompt engineering policy that mimics the expert
behaviors, driven by feedback from the discriminator. In this way, the
generator can gradually imitate humans in judging AI-generated images and
perform prompt engineering accordingly.

  * •

MDP Design: Finally, we configure the MDP for IRL. Similar to DRL, the action
space contains all candidate prompt engineering operations. The state space is
represented by the LLM-assigned scores of the image generated by the crafted
prompt. This allows GAIL to evaluate the efficacy of each action by evaluating
its impact on the resulting image quality. Unlike traditional DRL which relies
on manually designing reward functions, GAIL leverages the discriminator
network to infer rewards from our expert dataset.

###  IV-C Experiments

####  IV-C1 Experimental Setting

We take text-to-image generation as an example. The service providers adopt
Stable Diffusion v2 to generate images. Raw prompts from users take the form
of A [A] with [B], e.g., “a city with a car” and “a garden with a fountain.”
Leveraging the GPT-4 model, seven kinds of operations to craft prompts can be
performed on each raw prompt, as discussed in [15]. The LLM-based agent is
also implemented by GPT-4. For IRL, the discriminator consists of a four-layer
fully connected DNN and two intermediate layers, each containing 100 neurons.
Meanwhile, the generator containing actors and critics employs a similar four-
layer, fully connected DNN architecture with 64 neurons in each of the two
hidden layers. The hyperbolic tangent function serves as the activation
mechanism for all hidden layers. The learning rate and gamma are set to
$3\times 10^{-4}$ and 0, respectively.

####  IV-C2 Result Analysis

Fig. 4 shows the trend of cumulative rewards of different algorithms during
the training process. It shows that as the number of episodes increases, the
proposed IRL method achieves good convergence and significantly outperforms
the baseline, i.e., the service provider selects prompt engineering operation
randomly. Meanwhile, DRL also converges rapidly. With well-trained DRL and IRL
policies, Fig. 5 evaluates their efficacy in selecting prompt engineering
operations. Note that the quality scores are also assessed by the LLM-
empowered agent. As shown in Fig. 5, IRL can increase the image quality by
0.33 on average, while DRL can only achieve 0.1 increment. This is because IRL
is trained on expert datasets, which can effectively indicate the human
preference for assessing AI-generated images. Consequently, the prompt
engineering operations selected by IRL can better align with human desire,
resulting in high-score images.

Discussion: Contributed to the increasing image quality, the service
experience of humans can be increased drastically. Meanwhile, the re-
generation caused by unqualified outputs can be reduced, which greatly
decreases service latency and bandwidth consumption [15].

Figure 4: The training curves of DRL and IRL. Figure 5: The efficacy of DRL
and IRL in prompt engineering. Each case corresponds to one randomly selected
raw prompt.

##  V Future Directions

###  V-A Mixture-of-Experts

Recall that IRL heavily relies on expert trajectories, while in some complex
scenarios, collecting the optimal trajectories that achieve the objective with
the lowest cost is impossible. Instead, there may only exist multiple local
optimal trajectories owned by distributed experts, each of which reaches the
optimum in certain aspects/stages. To this end, the mixture-of-experts
principle can be leveraged, which utilizes a learning-based gating network to
dynamically select/combine different trajectories in different training
stages.

###  V-B IRL with Human Feedback

Human feedback has been successfully integrated into DRL (e.g., ChatGPT) to
make policies align with human preferences. Likewise, humans can participate
in the IRL process to further enhance the expert dataset/human annotation. For
example, in our case study, the efficient calibration of inferred rewards
based on human feedback is worth researching, thereby ensuring the reward
function represents the real human judgment of AI-generated images precisely.

###  V-C Security of IRL

The reliance on expert trajectories may cause security issues since attackers
can easily mislead policy training by data poisoning. To this end, strict
access control and privacy protection are urgently required for deploying IRL
in practical NGN scenarios. Zero-trust can be a potential technique to
dynamically manage data access and usage, thereby preventing privacy leakage.

##  VI Conclusion

In this article, we have explored the applications of IRL in NGN.
Specifically, we have comprehensively introduced the IRL technique, including
its fundamentals, representative algorithms, and advantages. Then, we have
discussed the vision of NGN, as well as the motivations for adopting IRL.
Afterward, we have surveyed existing literature about IRL proposals for
solving networking problems. Furthermore, we have performed a case study on
human-centric prompt engineering in GAI-enabled networks, comparing the
workflow and effectiveness of both DRL and IRL. Finally, the future directions
to promote the further development of IRL in NGN have been summarized.

## References

  * [1] H. Du, R. Zhang, Y. Liu, J. Wang, Y. Lin, Z. Li, D. Niyato, J. Kang, Z. Xiong, S. Cui, B. Ai, H. Zhou, and D. I. Kim, “Beyond deep reinforcement learning: A tutorial on generative diffusion models in network optimization,” _ArXiv preprint: ArXiv:2308.05384_ , 2023.
  * [2] G. Wang, P. Cheng, Z. Chen, W. Xiang, B. Vucetic, and Y. Li, “Inverse reinforcement learning with graph neural networks for iot resource allocation,” in _ICASSP_ , 2023, pp. 1–5.
  * [3] J. Parras, A. Almodóvar, P. A. Apellániz, and S. Zazo, “Inverse reinforcement learning: A new framework to mitigate an intelligent backoff attack,” _IEEE Internet of Things Journal_ , vol. 9, no. 24, pp. 24 790–24 799, 2022.
  * [4] S. Arora and P. Doshi, “A survey of inverse reinforcement learning: Challenges, methods and progress,” _Artificial Intelligence_ , vol. 297, pp. 103 500–103 548, 2021.
  * [5] L. Snow, V. Krishnamurthy, and B. M. Sadler, “Identifying coordination in a cognitive radar network - a multi-objective inverse reinforcement learning approach,” in _ICASSP_ , 2023, pp. 1–5.
  * [6] R. Zhang, K. Xiong, Y. Lu, P. Fan, D. W. K. Ng, and K. B. Letaief, “Energy efficiency maximization in ris-assisted swipt networks with rsma: A ppo-based approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 41, no. 5, pp. 1413–1430, 2023.
  * [7] V. S. Vaishnavi, Y. M. Roopa, and P. L. Srinivasa Murthy, “A survey on next generation networks,” in _ICCNCT_ , 2020, pp. 162–171.
  * [8] G. Wang, P. Cheng, Z. Chen, B. Vucetic, and Y. Li, “Inverse reinforcement learning with graph neural networks for full-dimensional task offloading in edge computing,” _IEEE Transactions on Mobile Computing_ , pp. 1–18, 2023.
  * [9] J. Li, H. Wu, Q. He, Y. Zhao, and X. Wang, “Dynamic qos prediction with intelligent route estimation via inverse reinforcement learning,” _IEEE Transactions on Services Computing_ , pp. 1–18, 2023.
  * [10] A. Konar, D. Wu, Y. T. Xu, S. Jang, S. Liu, and G. Dudek, “Communication load balancing via efficient inverse reinforcement learning,” in _ICC_ , 2023, pp. 472–478.
  * [11] A. Shamsoshoara, F. Lotfi, S. Mousavi, F. Afghah, and I. Guvenc, “Joint path planning and power allocation of a cellular-connected uav using apprenticeship learning via deep inverse reinforcement learning,” _ArXiv preprint: ArXiv:2306.10071_ , 2023.
  * [12] R. Zhang, K. Xiong, X. Tian, Y. Lu, P. Fan, and K. B. Letaief, “Inverse reinforcement learning meets power allocation in multi-user cellular networks,” in _INFOCOM WKSHPS_ , 2022, pp. 1–2.
  * [13] X. Tian, K. Xiong, R. Zhang, P. Fan, D. Niyato, and K. B. Letaief, “Sum rate maximization in multi-cell multi-user networks: An inverse reinforcement learning-based approach,” _IEEE Wireless Communications Letters_ , vol. 13, no. 1, pp. 4–8, 2024.
  * [14] X. Tian, B. Gao, M. Liu, K. Xiong, P. Fan, and K. Ben Letaief, “Irl-pm: An inverse reinforcement learning-based power minimization in multi-user miso networks,” in _ICCCS_ , 2023, pp. 332–336.
  * [15] Y. Liu, H. Du, D. Niyato, J. Kang, S. Cui, X. Shen, and P. Zhang, “Optimizing mobile-edge ai-generated everything (aigx) services by prompt engineering: Fundamental, framework, and case study,” _IEEE Network_ , pp. 1–1, 2024.