Generative Networks for Precision Enthusiasts

Anja Butter1, Theo Heimel1, Sander Hummerich1, Tobias Krebs1,

Tilman Plehn1, Armand Rousselot2, and Sophia Vent1

1 Institut für Theoretische Physik, Universität Heidelberg, Germany

2 Heidelberg Collaboratory for Image Processing, Universität Heidelberg,
Germany

## Abstract

Generative networks are opening new avenues in fast event generation for the
LHC. We show how generative flow networks can reach percent-level precision
for kinematic distributions, how they can be trained jointly with a
discriminator, and how this discriminator improves the generation. Our joint
training relies on a novel coupling of the two networks which does not require
a Nash equilibrium. We then estimate the generation uncertainties through a
Bayesian network setup and through conditional data augmentation, while the
discriminator ensures that there are no systematic inconsistencies compared to
the training data.

###### Content

  1. 1 Introduction
  2. 2 Precision generator
    1. 2.1 Data set
    2. 2.2 INN generator
  3. 3 DiscFlow generator
    1. 3.1 Discriminator reweighting
    2. 3.2 Joint training
  4. 4 Uncertainties and control
    1. 4.1 Bayesian network
    2. 4.2 Conditional augmentations
    3. 4.3 Discriminator for consistency
  5. 5 Outlook

##  1 Introduction

Precise first-principle simulations provided by the theory community are a
defining feature of LHC physics. They are based on perturbative quantum field
theory with fundamental Lagrangians as their physics input, and they provide
the simulated events necessary for modern LHC analyses and inference. Because
of the close correlation of complexity and precision in perturbative
calculations, precision and speed are, largely, two sides of the same medal.
Both of these sides are facing major challenges for the LHC Runs 3 and 4, and
the hope is that machine learning and its modern numerics toolbox allow us to
provide the simulations needed for a 25-fold increase of LHC data as compared
to Run 2.

In recent years, modern machine learning has shown great potential to improve
LHC simulations [1]. Underlying techniques include GANs [2, 3, 4], VAEs [5,
6], normalizing flows [7, 8, 9, 10, 11], and their invertible network (INN)
variant [12, 13, 14]. As part of the standard LHC simulation chain, modern
neural networks can be applied to the full range of phase space integration
[15, 16], phase space sampling [17, 18, 19, 20], amplitude computations [21,
22], event subtraction [23], event unweighting [24, 25], parton showering [26,
27, 28, 29, 30], or super-resolution enhancement [31, 32]. In essence, there
is no aspect of the standard event generation chain that cannot be improved
through modern machine learning.

Conceptionally new developments are, for instance, based on fully NN-based
event generators [33, 34, 35, 36, 37] or detector simulations [38, 39, 40, 41,
42, 43, 44, 45, 46, 47, 48]. Unlike forward Monte Carlo simulations, these
network generators can be inverted. Conditional GANs and INNs allow us to
invert the simulation chain to unfold detector effects [49, 50] and extract
the hard scattering process at parton level in a statistically consistent
manner [51]. Because of their superior statistical properties, the same
conditional INNs can be used for simulation-based inference based on high-
dimensional and low-level data [52]. Finally, normalizing-flow or INN
generators provide new opportunities when we combine them with Bayesian
network concepts[53, 54, 55, 56, 57, 58] to construct uncertainty-controlled
generative networks [59].

In this paper we combine the full range of ML-concepts to build an NN-based
LHC event generator which meets the requirements in terms of phase space
coverage, precision, and control of different uncertainties. We first present
a precision INN generator in Sec. 2 which learns underlying phase space
densities such that kinematic distributions are reproduced at the percent
level, consistent with the statistical limitations of the training data. Next,
our inspiration by GANs leads us to construct the DiscFlow
discriminator–generator architecture to control the consistency of training
data and generative network in in Sec. 3. Finally, in Sec. 4 we illustrate
three ways to control the network training and estimate remaining
uncertainties (i) through a Bayesian generative network, (ii) using
conditional augmentations for systematic or theory uncertainties, and (iii)
using the DiscFlow discriminator for controlled reweighting. While we employ
forward event generation to illustrate these different concepts, our results
can be directly transferred to inverted simulation, unfolding, or inference
problems.

##  2 Precision generator

As we will show in this paper, generative networks using normalizing flows
have significant advantages over other network architectures, including GANs,
when it comes to LHC event generation. As a starting point, we show how flow-
based invertible networks can be trained to generate events and reproduce
phase space densities with high precision. Our network architecture accounts
for the complication of a variable number of particles in the final state.

###  2.1 Data set

The kind of NN-generators we discuss in this paper are trained on unweighted
events at the hadronization level. We exclude detector effects because they
soften sharp phase space features, so simulations without them tend to be more
challenging and their results are more interesting from a technical
perspective. The production of leptonically decaying $Z$-bosons with a
variable number of jets is the ideal benchmark process. First, the network has
to learn an extremely sharp $Z$-resonance peak. Second, QCD forces us to apply
a geometric separation between jets, inducing a non-trivial topology of phase
space. Finally, again because of QCD it does not make sense to define final
states with a fixed number of jets, so our generative network has to cover a
final state with a variable number of dimensions. Given these considerations
we work with the process

$\displaystyle pp\to Z_{\mu\mu}+\\{1,2,3\\}~{}\text{jets}\;,$ (1)

simulated with Sherpa2.2.10 [60] at 13 TeV. We use CKKW merging [61] to
generate a merged sample with up to three hard jets including ISR, parton
shower, and hadronization, but no pile-up. The final state of the training
sample is defined by Fastjet3.3.4 [62] in terms of anti-$k_{T}$ jets [63] with

$\displaystyle p_{T,j}>20~{}\text{GeV}\qquad\qquad\text{and}\qquad\qquad\Delta
R_{jj}>R_{\text{min}}=0.4\;.$ (2)

The jets and muons are ordered in $p_{T}$. Because jets have a finite
invariant mass, our final state dimensionality is three for each muon plus
four degrees of freedom per jet, giving us phase space dimensionalities 10,
14, and 18. Momentum conservation does not further reduce the dimensionality,
as not every generated hadron is captured by the three leading jets. However,
we will reduce this dimensionality by one by removing the symmetry on the
choice of global azimuthal angle. Our combined sample size is 5.4M events,
divided into 4.0M one-jet events, 1.1M two-jet events, and 300k three-jet
events. This different training statistics will be discussed in more detail in
Sec. 4.1.

To define a representation which makes it easier for an INN to learn the
kinematic patterns we apply a standard pre-processing. First, each particle is
represented by

$\displaystyle\\{\;p_{T},\eta,\phi,m\;\\}\;.$ (3)

Because we can extract a global threshold in the jet $p_{T}$ we represent the
events in terms of the variable $\tilde{p}_{T}=\log(p_{T}-p_{T,\text{min}})$.
This form leads to an approximately Gaussian distribution, matching the
Gaussian latent-space distribution of the INN. Second, the choice of the
global azimuthal angle is a symmetry of LHC events, so we instead train on
azimuthal angles relative to the muon with larger transverse momentum in the
range $\Delta\phi\in[-\pi,\pi]$. A transformation into
$\widetilde{\Delta\phi}=\text{atanh}(\Delta\phi/\pi)$ again leads to an
approximately Gaussian distribution. For all phase space variables $q$ we
apply a centralization and normalization step

$\displaystyle\tilde{q}_{i}=\frac{q_{i}-\overline{q_{i}}}{\sigma(q_{i})}\;.$
(4)

Finally, we apply a whitening/PCA transformation separately for each jet
multiplicity.

###  2.2 INN generator

cINNcINNcINN$N_{\mathrm{jets}}$$z_{1\ldots 9}$$z_{10\ldots 13}$$z_{14\ldots 17}$(1-hot)$\mu_{1},\mu_{2},j_{1}$$j_{2}$$j_{3}$ Figure 1: Generative flow architecture for events with two muons and one to three jets. The INNs relate the latent space (left) to the physical phase space (right). hyperparameter |  INN (Sec. 2.2) |  INN (Sec. 3.1) |  DiscFlow (Sec. 3.2) |  BINN (Sec. 4.1)
---|---|---|---|---
LR scheduling | one-cycle | same | same | same
Starter LR | $10^{-4}$ | $4\cdot 10^{-4}$ | $2\cdot 10^{-4}$ | $10^{-5}$
Maximum LR | $10^{-3}$ | $4\cdot 10^{-3}$ | $2\cdot 10^{-3}$ | $10^{-4}$
Epochs | 100 | 200 | 200 | 100
Batch size | 1024 | 2048 | 2048 | 3072
Adam $\beta_{1}$, $\beta_{2}$ |  $0.9$, $0.99$ |  $0.9$, $0.99$ |  $0.5$, $0.9$ | same
Coupling block | cubic spline | same | same | same
# spline bins | 60 | same | same | same
# coupling blocks | 25 | 25 | 25 | 20
Layers per block | 3 | 3 | 3 | 6
# generated events | 2M | 2M | 2M | 1M
Table 1: Training setup and hyperparameters for the INN generators used in our
different setups.

For a fixed final-state dimensionality we can use a completely standard INN
[12, 59] to generate LHC events, especially after the preprocessing step
defined above. Our technical challenge is to expand the INN architecture to
generate final states with 9, 13, and 17 phase space dimensions. Of course, we
could just split the training sample into different multiplicities and train a
set of individual networks. However, in this case each of these networks has
to learn the basic QCD patterns, making this naive approach inefficient and
unstable.

To increase the efficiency, we stick to the same network for the common
$\mu_{1,2}$ and $j_{1}$ momenta and add small additional networks for each
additional jet, as illustrated in Fig. 1. Because the basic kinematic of the
muons and the first jet does depend on possible additional jets, the base
network gets the one-hot encoded number of jets as condition. Each of the
additional networks is conditioned on the training observables of the previous
networks and on the number of jets. Because the $\mu\mu j$ and $\mu\mu jj$
networks are trained on events with mixed multiplicities, we guarantee a
balanced training by drawing a random subset of the training data set at the
beginning of each epoch containing equal numbers of events from all different
multiplicities. While all three networks are trained separately, they are
combined as a generator.

Our network is implemented using PyTorch [64] with the Adam optimizer [65],
and a one-cycle learning-rate scheduler [66]. The affine coupling blocks of
the standard conditional INN setup [67, 51] are replaced by cubic spline
coupling blocks [68], which are more efficient in learning complex phase space
patterns precisely and reliably. The coupling block splits the target space
into bins of variable width based on trainable support points, which are
connected with a cubic function. They are combined with random but fixed
rotations to ensure interaction between all input variables. The parameter
ranges of input, output and intermediate spaces are limited to $[-10,10]$ on
both sides of the coupling blocks, numbers outside this range are mapped onto
themselves. The individual coupling blocks split their input vector in two
halves $(u_{i},v_{i})$ and transforms $v_{i}$ as

${v_{i}}^{\prime}=s(v_{i};\chi(u_{i},c_{i}))\;.$ (5)

The $c_{i}$ are the conditional inputs of the network. The function $\chi$ is
a fully connected sub-network with $2n_{\text{bins}}+2$ outputs, where
$n_{\text{bins}}$ is the number of spline bins. They encode the horizontal and
vertical positions of the spline knots and its slope at the boundaries. The
loss function for a cINN can most easily be defined in terms of the ratio of
the reference density $P_{\text{ref}}(x)$ and the learned density $P(x)$,

$\displaystyle\mathcal{L}_{G}=-\sum_{i=1}^{B}\log\frac{P(x_{i};c_{i})}{P_{\text{ref}}(x_{i};c_{i})}\;,$
(6)

for a batch of $B$ inputs $x_{i}$ and conditions $c_{i}$. The denominator in
the logarithm just shifts the loss function such that its minimum is at zero.
In the continuum limit this turns into

$\displaystyle\mathcal{L}_{G}=-\int
dx\;P_{\text{ref}}(x,c)\;\log\frac{P(x;c)}{P_{\text{ref}}(x;c)}\;.$ (7)

Changing variables between phase space and latent space we can split
$P(x_{i};c_{i})$ into an approximate latent-space Gaussian in terms of the
network $\psi$ and a Jacobian $J_{i}$, giving us

$\displaystyle\mathcal{L}_{G}\approx\sum_{i=1}^{B}\left(\frac{\psi(x_{i};c_{i})^{2}}{2}-\log
J_{i}\right)\;.$ (8)

In this form we ignore the normalization $\log P_{\text{ref}}(x_{i};c_{i})$,
because it does not affect the network training. We list all hyperparameters
in Tab. 1.

Figure 2: Jet-jet correlations for events with two jets. We show truth (left)
and INN-generated events (right).

#### Magic transformation

Figure 3: Left: $\Delta R_{j_{1}j_{2}}$-distribution for $Z+2$ jets events
before and after the transformation of Eq.(10). Right: histogram of the
weights of the generated events.

A major challenge of the $Z+$ jets final state is illustrated in Fig. 2, where
we show the $\Delta\phi$ vs $\Delta\eta$ correlations for the exclusive 2-jet
sample. We see that most events prefer a back-to-back topology, but a small
number of events features two jets recoiling against the $Z$, cut off by the
requirement $\Delta R_{jj}>0.4$. The ring around the corresponding small
circle is a local maximum, and inside the ring the phase space density drops
to zero. Because this entire structure lives in a poorly populated phase space
region, the INN typically ignores the local maximum and smoothly interpolates
over the entire ring-hole structure. We emphasize that in our case this is not
a problem with the non-trivial phase space topology [69], because the network
is able to interpolate smoothly through these holes, but a problem of the
precision with which the network learns features just around these holes.

To map out the local maximum structure we make use of the fact that our
network is extremely efficient at interpolating smooth functions. To exploit
this property we define a $\Delta R_{jj}$-dependent transformation which turns
the actual phase space pattern into a smoothly dropping curve, let the network
learn this smooth function extremely well, and then undo the transformation to
re-build the local maximum pattern. A simple smoothing function for our case
is

$\displaystyle f(\Delta R)=\begin{cases}0&\text{for }\Delta
R<R_{-}\\\\[5.69054pt] \dfrac{\Delta R-R_{-}}{R_{+}-R_{-}}&\text{for }\Delta
R\in[R_{-},R_{+}]\\\\[5.69054pt] 1&\text{for }\Delta R>R_{+}\;.\end{cases}$
(9)

The transition region is defined such that it includes the cutoff to ensure
non-vanishing weights, $R_{-}<R_{\text{min}}=0.4$, and its upper boundary is
in a stable phase space regime. In our case we use $R_{-}=0.2$ and $R_{+}=1.5$
without much fine-tuning. We also apply this transformation to the 3-jet
sample, where all $\Delta R_{jj}$-distribution have similar challenges,
through additional event weights

$\displaystyle w^{\text{(1-jet)}}$ $\displaystyle=1$ $\displaystyle
w^{\text{(2-jet)}}$ $\displaystyle=f(\Delta R_{j_{1},j_{2}})$ $\displaystyle
w^{\text{(3-jet)}}$ $\displaystyle=f(\Delta R_{j_{1},j_{2}})f(\Delta
R_{j_{2},j_{3}})f(\Delta R_{j_{1},j_{3}})\;.$ (10)

After training the INN generator on these modified events we also enforce the
jet separation and set all event weights with $\Delta R_{jj}<\Delta
R_{\text{min}}$ to zero. The inverse factor compensating for our magic
transformation is then

$\displaystyle\tilde{f}(\Delta R)=\begin{cases}0&\text{for }\Delta
R<R_{\text{min}}\\\\[5.69054pt] \dfrac{R_{+}-R_{-}}{\Delta R-R_{-}}&\text{for
}\Delta R\in[R_{\text{min}},R_{+}]\\\\[5.69054pt] 1&\text{for }\Delta
R>R_{+}\end{cases}\;.$ (11)

To train the INN generator on weighted data the loss function of Eq.(8) has to
be changed to

$\displaystyle\mathcal{L}_{G}=\sum_{i=1}^{B}\left(\frac{\psi(x_{i};c_{i})}{2}-J(x_{i})\right)\;\dfrac{w(x_{i})}{\sum_{i=1}^{B}w(x_{i})}\;,$
(12)

per batch with size $B$. Here, the weights are defined in Eq.(10), $x_{i}$ are
the latent space vectors, and $J_{i}$ are the corresponding logarithms of the
Jacobian. In the right panel of Fig. 2 we see that out network architecture
indeed captures the intricate structure in the jet-jet correlations. The
distribution of the resulting event weights is shown in Fig. 3. By
construction all finite event weights are above one, and hardly any of them
reach values for more than seven, which implies that these weights can be
easily removed by standard reweighting techniques.

Our magic transformation is similar to a refinement, defined as per-event
modifications of phase space distributions [70], whereas reweighting uses
weights for individual phase space points or events to improve the agreement
between generator output and truth [71]. However, our transformation is, by
standard phase-space mapping arguments, counter-intuitive***As a matter of
fact, our magic transformation of the density is the exact opposite of the
standard phase space mapping for Monte Carlo integration.. Instead of removing
a leading dependence from a curve and learning a small but non-trivial
difference, we smooth out a subtle pattern and rely on an outstanding network
interpolation to learn the smoothed-out function better than the original
pattern. This is motivated by the way flow networks learn distributions, which
is more similar to a fit than a combination of local patterns [59]. The
technical disadvantage of the smoothing transformation is that the generated
events are now weighted, its advantage is that it is very versatile.

#### INN-generator benchmark

Figure 4: INN distributions for $Z+1$ jet (upper), $Z+2$ jets (middle), and
$Z+3$ jets (lower) from a combined $Z+$ jets generation. We show weighted
events using the magic transformation of Eq.(10) to improve the $\Delta R$
distributions.

In Fig. 4 we show a set of kinematic distributions for our training data,
truth defined as a statistically independent version of the training sample,
and the output of the INN-generator with the magic transformation of Eq.(10).
We show distributions for exclusive $Z+\\{1,2,3\\}$ jets samples and define
the relative deviation for binned kinematic distributions as

$\displaystyle\delta[\%]=100\;\frac{|\text{Model}-\text{Truth}|}{\text{Truth}}\;.$
(13)

In the top row the final state consists of the $Z$-decay products and one
recoil jet, and we see that the recoil spectrum as well as the sharp $Z$-mass
are learned with high precision. That remains true when we add a second jet,
including the critical $\Delta R_{j_{1}j_{2}}$ correlation discussed above.
Finally, adding yet another jet the network learns the complete set of angular
jet-jet correlations. Looking a the precision of the training sample, which
consists of half of our full data set, we see that at least in the bulk of the
kinematic distribution, the training data set agrees with truth at the percent
level or better. This changes in the kinematic tails, where the statistical
precision of the training data drops continuously. The level of agreement
between the INN-generated events and truth also reaches the percent level in
densely populated phase space regions, but it is slightly worse than the
precision of the training sample. Also the $Z$-peak even in the 1-jet sample
is not perfectly learned by the INN, which leaves us a little bit of work to
do on the precision side.

##  3 DiscFlow generator

One way to systemically improve and control a precision INN-generator is to
combine it with a discriminator. It is inspired by incredibly successful GAN
applications also in LHC simulations [72, 3, 4]. In our preliminary studies we
never reached a sufficient precision with established GAN architectures [36],
while INN-generators proved very promising [59]. Compared to reweighting and
refinement methods, a GAN-like setup has the advantage that the generator and
discriminator networks already communicate during the joint training. We will
show how such a discriminator network can be used to improve precision event
generation and then show how a discriminator can be coupled to our INN
generator in a new DiscFlow architecture.

###  3.1 Discriminator reweighting

hyper-parameter | value
---|---
LR scheduling | Reduce-on-plateau
Starter LR | $1\cdot 10^{-2}$
Epochs | 200
Batch size | 2048
Adam $\beta_{1}$, $\beta_{2}$ |  $0.5$, $0.9$
Layer type | Dense
Number layers | 8
Internal size | 256
Table 2: Training setup and hyperparameters for the discriminator.

Figure 5: Discriminator-reweighted INN distributions for $Z+1$ jet (upper),
$Z+2$ jets (middle), and $Z+3$ jets (lower) from a combined $Z+$ jets
generation. The bottom panels show the average correction factor obtained from
the discriminator output, the INN results without reweighting are the same as
in Fig. 4, except for slightly longer training.

Before we train our INN-generator jointly with a discriminator, we illustrate
the power of such a discriminator by training it independently and reweighting
events with the discriminator output [71]. This requires that our
discriminator output can eventually be transformed into a probabilistic
correction. We train a simple network described in Tab. 2 by minimizing the
cross entropy to extract a probability $D(x_{i})\to 0(1)$ for an identified
generator (truth) event $x_{i}$. For a perfect generated sample the
discriminator cannot tell generated events from true events, and the output
becomes $D(x_{i})=0.5$ everywhere. Using this discriminator output we define
the event weight

$\displaystyle
w_{D}(x_{i})=\frac{D(x_{i})}{1-D(x_{i})}\to\frac{P_{\text{ref}}(x_{i})}{P(x_{i})}\;.$
(14)

In the conventions of Eq.(7) $w_{D}$ approximates the ratio of true over
generated phase space densities, so we can use it to reweight each event such
that it reproduces the true kinematic distributions at the level they are
encoded in the discriminator.

To see how precisely this kind of discriminator works we use the standard INN
generator from Sec. 2.2. We omit the magic transformation described in
Eq.(10), to define a challenge for the discriminator. For each jet-
multiplicity of the cINN model, we train a discriminative model in parallel to
the generative model, but for now without the two networks communicating with
each other. The input to the three distinct discriminator networks, one per
multiplicity, are the usual observables $p_{T},\eta,\phi$, and $m$ of Eq.(3)
for each final-state particle. We explicitly include a set of correlations
known to challenge our naive INN generator and train the discriminator

$\displaystyle\mathcal{L}_{D}=-\sum_{i}^{B}\log(1-D(x_{i}))-\sum_{i}^{B}\log(D(y_{i}))$
(15)

with generated vectors extended depending on the jet multiplicity

$\displaystyle
x_{i}=\\{p_{T,j},\eta_{j},\phi_{j},M_{j}\\}\cup\\{M_{\mu\mu}\\}\cup\\{\Delta
R_{2,3}\\}\cup\\{\Delta R_{2,4},\Delta R_{3,4}\\}\;.$ (16)

and corresponding training vectors $y_{i}$.

In Fig. 5 we show sample kinematic distributions for the $Z+\\{1,2,3\\}$ jet
final states. Truth is defined as the high-statistics limit of the training
data. The INN events are generated with the default generator, without the
magic transformation of Eq.(10), so they are unweighted events. The reweighted
events are post-processed INN events with the average weight per bin shown in
the second panel. While for some of the shown distribution a flat dependence
$w_{D}=1$ indicates that the generator has learned to reproduce the training
data to the best knowledge of the discriminator, our more challenging
distributions are significantly improved by the discriminator. That includes
the reconstructed $Z$-mass as well as the different $\Delta
R_{jj}$-distributions.

Comparing the discriminator-reweighted performance to the magic transformation
results in Fig. 4 we see that especially distributions like $\Delta
R_{j_{1}j_{2}}$ or $\Delta R_{j_{1}j_{3}}$ are both reproduced extremely well.
For the comparably flat $p_{T}$-distributions the precision of the reweighted
events is becoming comparable to the training statistics, both for the bulk of
the distribution and for the sparsely populated tails. Of all kinematic
distributions we checked, the vector sum of all hard transverse momenta of the
5-jet final state is the only distribution where the naive INN-generator only
learns the phase space distribution only at the 10% level. Also those are
corrected fine by the discriminator reweighting.

While the discriminator reweighting provides us with an architecture that
learns complex LHC events at the percent level or at the level of the training
statistics, it comes with the disadvantage of generating weighted events and
does not use the opportunity for the generator and discriminator to improve
each other. Both of these open questions will be discussed in the next
architecture.

###  3.2 Joint training

Figure 6: Illustration of the DiscFlow method. Weights computed by the
discriminator shift the reference (true) density downwards whenever the
generator (fake) distribution overshoots and vice-versa. This way the
deviations of the to-be-trained generator density are over-exaggerated.

After observing the benefits from an additional discriminator network, the
question is how we can make use of this second network most efficiently. If it
is possible to train the discriminator and generator network in parallel and
give them access to each other, a joint GAN-like setup could be very efficient
[73]. Unfortunately, we have not been able to reach the required Nash
equilibrium in an adversarial training for our specific INN setup. Instead,
one of the two players was always able to overpower the other.

Figure 7: DiscFlow distributions for $Z+1$ jet, $Z+2$ jets, and $Z+3$ jets,
from a combined $Z+$ jets generation after joint generator–discriminator
training.

Instead of relying on a Nash equilibrium between the two competing network
architectures we can avoid a two-part loss functions entirely and incorporate
the discriminator information into the generator loss of Eq.(6) through the
event weight function $w_{D}(x)$ of Eq.(14),

$\displaystyle\mathcal{L}_{\text{DiscFlow}}$
$\displaystyle=-\sum_{i=1}^{B}\;w_{D}(x_{i})^{\alpha}\;\log\frac{P(x_{i};c_{i})}{P_{\text{ref}}(x_{i};c_{i})}$
$\displaystyle\approx-\int
dx\;\frac{P_{\text{ref}}^{\alpha+1}(x)}{P^{\alpha}(x)}\;\log\frac{P(x)}{P_{\text{ref}}(x)}$
$\displaystyle=-\int
dx\;\left(\frac{P_{\text{ref}}(x)}{P(x)}\right)^{\alpha+1}\;P(x)\log P(x)+\int
dx\;\left(\frac{P_{\text{ref}}(x)}{P(x)}\right)^{\alpha}\;P_{\text{ref}}(x)\log
P_{\text{ref}}(x)$
$\displaystyle=-\Big{\langle}\left(\frac{P_{\text{ref}}(x)}{P(x)}\right)^{\alpha+1}\log
P(x)\Big{\rangle}_{P}+\Big{\langle}\left(\frac{P_{\text{ref}}(x)}{P(x)}\right)^{\alpha}\log
P_{\text{ref}}(x)\Big{\rangle}_{P_{\text{ref}}}\;,$ (17)

with an appropriately defined expectation value. For the continuum limit we
omit the conditional argument and assume a perfectly trained discriminator.
Note that in our simple DiscFlow setup the discriminator weights
$\omega_{D}\approx P_{\text{ref}}(x)/P(x)$ do not have gradients with respect
to the generative model parameters, so only the first term in the last line
contributes to the optimization. This term corresponds to the negative log-
likelihood of training samples drawn from the weighted truth distribution. The
hyperparameter $\alpha$ determines the impact of the discriminator output, and
we introduce an additional discriminator dependence as

$\displaystyle\alpha=\alpha_{0}\;\left|\frac{1}{2}-D(x)\right|\;.$ (18)

Furthermore, starting at zero, we increase $\alpha_{0}$ linearly during
training.

From Eq.(17) we see that our modified loss is equivalent to training on a
shifted reference distribution. In Fig. 6 we illustrate what happens if the
generator populates a phase space region too densely and we reduce the weight
of the training events there. Conversely, if a region is too sparsely
populated by the generator, increased loss weights amplify the effect of the
training events. Our new discriminator–generator coupling through weights has
the advantage that it does not require a Nash equilibrium between two
competing networks, so the discriminator can no longer overpower the
generator. As the generator converges towards the true distribution, the
discriminator will stabilize as $w_{D}(x)\rightarrow 1$, and the generator
loss will approach its unweighted global minimum.

Figure 8: Discriminator-reweighted DiscFlow distributions for $Z+1$ jet
(upper), $Z+2$ jets (middle), and $Z+3$ jets (lower) from a combined $Z+$ jets
generation. The bottom panels show the average correction factor obtained from
the discriminator output, the DiscFlow results for joint
generator–discriminator training and without reweighting are the same as in
Fig. 7.

When training the two DiscFlow networks jointly, we split the batches per
epoch equally between both networks, training each network on a separate
subset of the training data. To increase the stability, we start by training
the generator and the separate discriminators for the different jet
multiplicities separately and only reweight the latent loss objective
afterwards.

In Fig. 7 we show the performance of the DiscFlow setup to our $Z$+jets
benchmark process. First, we see that in the bulk of the flat distributions
like $p_{T,j}$ the generator reproduces the correct phase space density almost
at the level of the training statistics. We know that the standard INN
generator fails when it comes to the precision of the $M_{\mu\mu}$ and $\Delta
R$ distributions, and in Fig. 7 we confirm that the joint training of the
generator with a discriminator corrects the invariant mass almost to the level
of the training statistics. The non-negligible density of generated events
below the cut at $\Delta R=0.4$ shows that the DiscFlow method is only
effective in phase space regions populated by training data.

In the ideal AI-world we assume that after successful joint training the
discriminator will have transferred all of its information into the generator,
such that $D(x)=0.5$ at any point of phase space. In reality, this is not at
all guaranteed. We know from Fig. 5 that the discriminator can learn the
$\Delta R$ features very well, so we combine the joint training and
discriminator reweighting strategies to ensure that we extract the full
performance of both networks. In Fig. 8 we show the same training results as
in Fig. 7, but reweighted with $w_{D}$. We see that the reweighting leads to a
small correction of the $M_{\mu\mu}$-distribution and a sizeable correction to
the $\Delta R_{jj}$ features close to the jet separation cut. Because of the
way we provide the event input, we note that the transverse momentum
conservation would become the next challenge after mastering $M_{\mu\mu}$ and
$\Delta R_{jj}$. For all other observables our reweighted DiscFlow network
indeed reproduces the true kinematic distributions at the percent level
provided by the training statistics.

Figure 9: Distribution over the weights $w_{D}$ computed over the entire, not
marginalized phase space.

While in Fig. 8 we see that the correction factor obtained from the
discriminator shows the agreement of training events and simulated events, it
is crucial that we search the fully exclusive phase space for systematic
deviations between training and simulated events. In Fig. 9 we histogram all
event weights $w_{D}(x_{i})$ for $Z+$ jets production. For the high-statistics
$Z+1$ jet sample the correction weights are at most at the percent level. The
fact that our generator only learns the phase space density and not the total
rates allows for a slight bias in the event weight distributions. For the bulk
of the kinematic distributions the bin-wise correction in Fig. 8 is still
slightly smaller than the weights shown here, which means that some of the
corrections are simply noise. The width of the weight distribution increases
for higher jet multiplicities, simply reflecting the drop in training
statistics. Combining Fig. 9 and Fig. 8 allows us to trace the large weights
$w_{D}$ to critical phase space regions, like the lower tail of the
$M_{\mu\mu}$-distribution for $Z+1$ jet or $\Delta R_{jj}\lesssim 0.5$ for
$Z+2/3$ jets.

##  4 Uncertainties and control

After introducing our precision generator architecture in Sec. 2 and extending
it to a discriminator–generator architecture for control in Sec. 3, the last
item on our list of LHC tasks is a comprehensive treatment of uncertainties. A
proper uncertainty treatment has been discussed for instance for regression or
classification networks [57, 58, 74], while for generative networks there
exists only a first study on how to use and interpret Bayesian INNs [59]. In
this final section we discuss how different uncertainties on generated events
can be extracted using a Bayesian generator network, a conditional sampling
using simulated uncertainties, and the discriminator introduced in the
previous section. Each of these handles allows us to control certain kinds of
uncertainties, and in combination they allow us to extract a meaningful
uncertainty map over phase space.

###  4.1 Bayesian network

The simple idea behind Bayesian networks is to replace trained network weights
by trained distributions of network weights. If we evaluate the network by
sampling over these distributions, the network output will be a central value
of the numerically defined function and an uncertainty distribution [53, 54,
55]. Because general MCMC-methods become expensive for larger networks, we
rely on variational inference to generate the weight distributions [75]. More
specifically, we rely on a Gaussian approximation for the network weight
distribution and learn the mean and the standard deviation instead of just one
value in a deterministic network. Because of the non-linear nature of the
network the output does not have a Gaussian uncertainty distribution [58]. Our
Bayesian INN (BINN) follows the same setup as our deterministic INN-generator
in Sec. 2.2, converted to the Bayesian setup following Ref. [59].

For a Bayesian generative network we supplement the phase space density
$p(x)$, encoded in the density of unweighted events, with an uncertainty map
$\sigma(x)$ over the same phase space. To extract the density we bin events in
a histogram for a given observable and with finite statistics. Focussing on
one histogram and omitting the corresponding phase space argument $x$ the
expected number of events per bin is

$\displaystyle\langle n\rangle=\sum_{n}nP_{N}(n)\;,$ (19)

with $P_{N}(n)$ given by the binomial or Poisson probability of observing $n$
events in this bin. This event count should be the mean of the BINN
distribution, defined by sampling from the distribution $q(\theta)$ over the
network weights $\theta$,

$\displaystyle\langle n\rangle=\int
d\theta\;q(\theta)\sum_{n}nP_{N}(n|\theta)\equiv\int
d\theta\;q(\theta)\;\langle n\rangle_{\theta}\;.$ (20)

Following the same argument as in Ref. [58] we can compute the standard
deviation of this sampled event count and split it into two terms,

$\displaystyle\sigma_{\text{tot}}^{2}$ $\displaystyle=\langle(n-\langle
n\rangle)^{2}\rangle$ $\displaystyle=\int d\theta\>q(\theta)\left[\langle
n^{2}\rangle_{\theta}-2\langle n\rangle_{\theta}\langle n\rangle+\langle
n\rangle^{2}\right]$ $\displaystyle=\int d\theta\>q(\theta)\left[\langle
n^{2}\rangle_{\theta}-\langle n\rangle_{\theta}^{2}+(\langle
n\rangle_{\theta}-\langle
n\rangle)^{2}\right]\equiv\sigma_{\text{stoch}}^{2}+\sigma_{\text{pred}}^{2}\;.$
(21)

The first contribution to the uncertainty is the variance of the Poisson
distribution,

$\displaystyle\sigma_{\text{stoch}}^{2}=\int d\theta\>q(\theta)\left[\langle
n^{2}\rangle_{\theta}-\langle n\rangle_{\theta}^{2}\right]=\langle
n\rangle\;.$ (22)

Even if the network is perfectly trained and $q(\theta)$ turns into a delta
distribution, it does not vanish, because it describes the stochastic nature
of our binned data set. The second term,

$\displaystyle\sigma_{\text{pred}}^{2}=\int d\theta\>q(\theta)\left[\langle
n\rangle_{\theta}-\langle n\rangle\right]^{2}\;,$ (23)

captures the deviation of our network from a perfectly trained network, where
the widths of the network weights vanish.

Moving from a binned to a continuous distribution we can transform our results
into the density and uncertainty maps over phase space, as introduced in Ref.
[59]. Assuming $\langle n\rangle\propto p(x)$, with an appropriate
proportionality factor and a continuous phase space variable $x$, Eqs.(20) and
(23) turn into

$\displaystyle p(x)$ $\displaystyle=\int d\theta\;q(\theta)\;p(x|\theta)$
$\displaystyle\sigma_{\text{pred}}^{2}(x)$ $\displaystyle=\int
d\theta\;q(\theta)\;\left[p(x|\theta)-p(x)\right]^{2}\;.$ (24)

To estimate $\sigma_{\text{tot}}$, we sample $\theta$ and $n$ from their
underlying distributions and compute $\langle n\rangle$. In practice, we draw
weights $\theta$, generate $N$ events with those weights, histogram them for
the observable of interest, extract $n$ per bin. Because the INN-generator is
very fast, we can repeat this process to compute the standard deviation. To
see the effect of the different contributions to the BINN uncertainty we
illustrate the correlation between the event count and $\sigma_{\text{tot}}$
for $Z+1$ jet events in Figure 10, with the $p_{T,j}$-distribution described
by 60 bins. Each of these bins corresponds to a dot in the figure. As long as
our sampling is limited by the statistics of the generated events we find the
expected Poisson scaling $\sigma\propto\sqrt{\mu}$, corresponding to the
contribution $\sigma_{\text{stoch}}$. For larger statistics,
$\sigma_{\text{stoch}}$ becomes relatively less important, and the actual
predictive uncertainty of the BINN takes over,
$\sigma_{\text{tot}}\approx\sigma_{\text{pred}}$.

Figure 10: Correlation between event count and BINN uncertainty for 1000
(left) and 1M (right) generated events. The diagonal like defines the Gaussian
scaling for a statistically limited sample.

Figure 11: BINN densities and uncertainties for $Z+1$ jet (upper), $Z+2$ jets
(middle), and $Z+3$ jets (lower) from a combined $Z+$ jets generation. The
architecture and training data correspond to the deterministic network results
shown in Fig. 4, including the magic transformation of Eq.(10).

#### Sources of uncertainties

By construction, Bayesian networks capture the effects of limited training
statistics and non-perfect training. If we control the truth information and
can augment the training data, a Bayesian network can also propagate the
effects of systematic biases, systematic uncertainties, or noise into the
network output [57, 58]. For generative networks, the Bayesian network is
ideally suited to understand the way the network learns the phase space
density by following the density map it learns in parallel [59]. As a side
remark, we can use this information to track the learning of the BINN for our
$Z$+jets events. We find that the network first learns the
$p_{T}$-distributions of the different final-state particles quite precisely,
before it targets the angular correlations. This explains why small features
of the $\Delta R$-distributions are the hardest to learn, because they arise
only for the correlation of the $\Delta\eta$ and $\Delta\phi$ correlations.
Correspondingly, we find that one way of improving the performance on the
angular correlation is to apply noise specifically to the
$p_{T}$-distributions. On the other hand, the magic transformation of Eq.(10)
turns out to be the more efficient solution to this problem, so we also apply
it to the BINN.

When modelling different uncertainties, the problem with augmented training
data for generative networks is that their training is, strictly speaking,
unsupervised. We do not have access to the true density distribution and have
to extract it by binning event samples. This means that additional noise will
only be visible in the BINN uncertainty if it destabilizes the training
altogether. Other data augmentation will simply lead to a different target
density, overriding the density encoded in the original set of events. This is
why in the following we will discuss training statistics and stability, and
postpone the description of systematics in generative network training to Sec.
4.2.

In Fig. 11 we show the uncertainty
$\sigma_{\text{tot}}\approx\sigma_{\text{pred}}$ given by the BINN for a
Bayesian version of the network introduced in Sec. 2.2, including the magic
transformation for the $\Delta R$-distributions. As before, we see that the
network learns the phase space density very precisely for simple kinematic
distributions like $p_{T,j_{1}}$. The slightly worse performance compared to
the deterministic network in Fig. 11 is due to the increased training effort
required by the larger network. The extracted uncertainties for $p_{T,j_{1}}$
and $p_{T,j_{2}}$ for instance in the bulk reflect the lower statistics of the
$Z+2$ jet training sample compared to $Z+1$ jet. The narrow
$M_{\mu\mu}$-distribution challenges the uncertainty estimate in that the
network learns neither the density nor the uncertainty very precisely [59].
This limitation will be overcome once the network learns the feature in the
density properly. For the different $\Delta R$-distributions we see that the
network learns the density well, thanks to the magic transformation of
Eq.(10). Therefore, the network also reports a comparably large uncertainty in
the critical phase space regions around $\Delta R_{ij}=0.4~{}...~{}1$.

#### Effect of training statistics

Figure 12: Relative uncertainty from the BINN for the $Z+1$ jet sample, as a
function of the size of the training sample.

From the above discussion it is clear that one way to test the BINN
uncertainties is to train the same network the same way, but on training
samples of different size. We start with one batch size, 3072 events, and
increase the training sample to the maximum of 2.7M. For $Z+1$ jet we show the
relative uncertainty as a function of transverse momenta, for instance, in
Fig. 12. In both cases we see that over most the distribution the uncertainty
improves with the training statistics. However, we also see that in the low-
statistics tail of $p_{T,\mu_{1}}$ the low-statistics trainings systematically
underestimate the uncertainty. Again, this reflects the fact that the network
does not even have enough data to estimate a reasonable uncertainty. For
$p_{T,j_{1}}$ this effect does not occur, even in the tails where the network
has to extrapolate eventually.

###  4.2 Conditional augmentations

Figure 13: BINN densities for $Z+$ jets and conditional training with an
enhanced-tail augmentation in $p_{T,_{1}}$, as defined in Eq.(25).

As discussed above, Bayesian generative networks will not capture typical
systematic or theory uncertainties, because we cannot augment the input truth.
Instead, we can augment our data via a nuisance parameter describing the
nature and size of a systematic or theory uncertainty and train the network
conditionally on this parameter. For illustration purpose, we introduce a
theory uncertainty proportional to a transverse momentum scale. As a function
of a nuisance parameter $a$ we shift the unit weights in our training sample
to

$\displaystyle
w=1+a\;\left(\frac{p_{T,j_{1}}-15~{}\text{GeV}}{100~{}\text{GeV}}\right)^{2}\;,$
(25)

where the transverse momentum is given in GeV, we account for a threshold at
15 GeV, and we choose a quadratic scaling to enhance the effects of this shift
in the tails.

We then train the Bayesian INN conditionally on values $a=0~{}...~{}30$ in
steps of one. For the event generation incorporating the theory uncertainty we
can sample kinematic distributions for different $a$-values. In Fig. 13 we
show generated distributions for different values of $a$. To model the
conditional parameter similar to phase space and allow for an uncertainty on
the conditional nuisance parameter, we sample $a$ with a Gaussian around its
central value and a standard deviation of $\min(a/10,0.1)$. The two panels
show the modified $p_{T,j_{1}}$-distribution and its impact on $p_{T,j_{2}}$
through correlations. As expected, the effects are similar, but the multi-
particle recoil washes out the effects on $p_{T,j_{2}}$. In the upper panels
we compare the effect of the theory uncertainty $a=0~{}...~{}12$ to the
statistical training uncertainty given by the BINN. We see that our method
traces the additional theory or systematic uncertainty, and allows us to
reliably estimate its sub-leading nature for $p_{T,j_{2}}$. While we show
ranges of $a$, corresponding to the typical flat likelihood used for theory
uncertainties, we could obviously sample the different $a$-values during event
generation. In the lower panels we show the relative BINN uncertainties, to
ensure that the training for the different $a$-values is stable. For
$p_{T,j_{1}}$ the data augmentation has a slight chilling effect on the high-
precision training around the maximum of the distribution. In the
statistically limited tails towards larger $p_{T}$ the BINN training without
and with augmentations behaves the same. Looking at the recoil correlation,
the BINN reports a slightly larger uncertainty for the augmented training,
correctly reflecting the fact that the network now has to learn an additional
source of correlations. At least for the range of shown $a$-values this BINN
uncertainty is independent of the size of the augmentation.

###  4.3 Discriminator for consistency

Figure 14: Illustration of uncertainty-controlled DiscFlow simulation. We show
the reweighted $p_{T,j_{1}}$-distribution for the inclusive $Z$+jets sample,
combined with the discriminator $D$, the BINN uncertainty, and the sampled
systematic uncertainty defined through the data augmentation of Eq.(25).

After introducing two ways of tracing specific uncertainties for generative
networks and controlling their precision, we come back to the joint DiscFlow
generator–discriminator training. In complete analogy to, for instance,
higher-order perturbative corrections, we can use the jointly trained
discriminator to improve the network precision and at the same time guide us
to significant differences between training data and generated data. Because
the discriminator is a simpler network than the INN-generator, it is well
suited to search for deviations which the BINN misses in its density and
uncertainty maps.

In Fig. 14 we illustrate the different aspects of our uncertainty-controlled
precision generator. First, we see that the INN generator indeed learns and
reproduces the phase space density at the level of the training statistics. In
the remaining panels we show three ways to control possible uncertainty, using
the discriminator, a BINN, and a BINN combined with augmented training data.
Each aspect is described in detail in this paper:

  * $\cdot$

joint discriminator–generator training (DiscFlow) for precision generation —
Fig. 7;

  * $\cdot$

discriminator to control inconsistencies between training and generated events
— Fig. 8;

  * $\cdot$

BINN to track uncertainty on the learned phase space density — Fig. 11;

  * $\cdot$

conditional augmentation for systematic or theory uncertainties — Fig. 13.

##  5 Outlook

A crucial step in establishing generative networks as event generation tools
for the LHC is the required precision in estimating the phase space density
and full control of uncertainties in generated samples.

In the first part of this paper, we have shown how INN-generators can be
trained on $Z$+jets events with a variable number of particles in the final
state, to reproduce the true phase space density at the percent level, almost
on par with the statistical uncertainty of the training sample. If we are
willing to work with weighted events, with event weights of order one, we can
either use a magic variable transformation or an additional discriminator
network to achieve high precision all over phase space. Alternatively, we can
train the discriminator jointly with the generator and use our novel DiscFlow
architecture to provide unweighted events with high precision (Fig. 7). This
joint training does not involve a Nash equilibrium and is especially stable.
Any information that the discriminator has not transferred to the generator
training can eventually be included through reweighting, giving our NN-event
generator high precision combined with a high level of control (Fig. 8).

In the second part of this paper we have established three methods to control
the precision INN-generator and its uncertainties. First, for unsupervised
generative training we can use a Bayesian INN to estimate uncertainties from
limited training statistics or sub-optimal network training (Fig. 11). Second,
we can augment the training data conditionally on a nuisance parameter and
sample this parameter to account for systematic or theory uncertainties
including the full phase space correlations (Fig. 13). A reliable estimate of
the different uncertainties allows us to compare the numerical impact of the
different uncertainties. Finally, we can use the jointly trained discriminator
to identify phase space regions where the BINN lacks the necessary precision
in its density and uncertainty maps over phase space.

All these aspects of our uncertainty-controlled precision generator are
illustrated in Fig. 14. With this level of precision and control, INN-
generators should be ready to be used as extremely efficient tools to generate
LHC events.

Acknowledgments

We would like to thank Ben Nachman and Jan Pawlowski for very helpful
discussions on the DiskFlow loss function. In addition, we would like to thank
Michel Luchmann and Manuel Haußmann for help with Bayesian networks and Luca
Mantani and Ramon Winterhalder for their work on an earlier incarnation of
this project. We are also very grateful to Ulli Köthe and Lynton Ardizzone for
their expert advice on many aspects of this paper. The research of AB and TP
is supported by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) under grant 396021762 – TRR 257 Particle Physics Phenomenology
after the Higgs Discovery. TH is supported by the DFG Research Training Group
GK-1940, Particle Physics Beyond the Standard Model. The authors acknowledge
support by the state of Baden-Württemberg through bwHPC and the German
Research Foundation (DFG) through grant no INST 39/963-1 FUGG (bwForCluster
NEMO).

## References

  * [1] A. Butter and T. Plehn, Generative Networks for LHC events, arXiv:2008.08558 [hep-ph].
  * [2] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, Generative adversarial networks, arXiv:1406.2661 [stat.ML].
  * [3] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, Generative adversarial networks: An overview, IEEE Signal Processing Magazine 35 (Jan, 2018) 53–65.
  * [4] A. Butter, S. Diefenbacher, G. Kasieczka, B. Nachman, and T. Plehn, GANplifying event samples, SciPost Phys. 10 (2021) 6, 139, arXiv:2008.06545 [hep-ph].
  * [5] D. P. Kingma and M. Welling, Auto-encoding variational bayes, arXiv:1312.6114 [stat.ML].
  * [6] D. P. Kingma and M. Welling, An introduction to variational autoencoders, Foundations and Trends® in Machine Learning 12 (2019) 4, 307–392.
  * [7] D. Rezende and S. Mohamed, Variational inference with normalizing flows, in Proceedings of the 32nd International Conference on Machine Learning, F. Bach and D. Blei, eds.  PMLR, Lille, France, 07–09 Jul, 2015.  arXiv:1505.05770 [stat.ML].
  * [8] I. Kobyzev, S. Prince, and M. Brubaker, Normalizing flows: An introduction and review of current methods, IEEE Transactions on Pattern Analysis and Machine Intelligence (2020) 1–1.
  * [9] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan, Normalizing flows for probabilistic modeling and inference, arXiv:1912.02762 [stat.ML].
  * [10] I. Kobyzev, S. Prince, and M. A. Brubaker, Normalizing flows: An introduction and review of current methods, arXiv:1908.09257 [stat.ML].
  * [11] T. Müller, B. McWilliams, F. Rousselle, M. Gross, and J. Novák, Neural importance sampling, arXiv:1808.03856 [cs.LG].
  * [12] L. Ardizzone, J. Kruse, S. Wirkert, D. Rahner, E. W. Pellegrini, R. S. Klessen, L. Maier-Hein, C. Rother, and U. Köthe, Analyzing inverse problems with invertible neural networks, arXiv:1808.04730 [cs.LG].
  * [13] L. Dinh, J. Sohl-Dickstein, and S. Bengio, Density estimation using real nvp, arXiv:1605.08803 [cs.LG].
  * [14] D. P. Kingma and P. Dhariwal, Glow: Generative flow with invertible 1x1 convolutions, arXiv:1807.03039 [stat.ML].
  * [15] M. D. Klimek and M. Perelstein, Neural Network-Based Approach to Phase Space Integration, arXiv:1810.11509 [hep-ph].
  * [16] I.-K. Chen, M. D. Klimek, and M. Perelstein, Improved Neural Network Monte Carlo Simulation, SciPost Phys. 10 (2021) 023, arXiv:2009.07819 [hep-ph].
  * [17] E. Bothmann, T. Janßen, M. Knobbe, T. Schmale, and S. Schumann, Exploring phase space with Neural Importance Sampling, SciPost Phys. 8 (2020) 4, 069, arXiv:2001.05478 [hep-ph].
  * [18] C. Gao, J. Isaacson, and C. Krause, i-flow: High-dimensional Integration and Sampling with Normalizing Flows, Mach. Learn. Sci. Tech. 1 (2020) 4, 045023, arXiv:2001.05486 [physics.comp-ph].
  * [19] C. Gao, S. Höche, J. Isaacson, C. Krause, and H. Schulz, Event Generation with Normalizing Flows, Phys. Rev. D 101 (2020) 7, 076002, arXiv:2001.10028 [hep-ph].
  * [20] K. Danziger, T. Janßen, S. Schumann, and F. Siegert, Accelerating Monte Carlo event generation – rejection sampling using neural network event-weight estimates, arXiv:2109.11964 [hep-ph].
  * [21] F. Bishara and M. Montull, (Machine) Learning Amplitudes for Faster Event Generation, arXiv:1912.11055 [hep-ph].
  * [22] S. Badger and J. Bullock, Using neural networks for efficient evaluation of high multiplicity scattering amplitudes, JHEP 06 (2020) 114, arXiv:2002.07516 [hep-ph].
  * [23] A. Butter, T. Plehn, and R. Winterhalder, How to GAN Event Subtraction, SciPost Phys. Core 3 (2020) 009, arXiv:1912.08824 [hep-ph].
  * [24] B. Stienen and R. Verheyen, Phase Space Sampling and Inference from Weighted Events with Autoregressive Flows, SciPost Phys. 10 (2021) 038, arXiv:2011.13445 [hep-ph].
  * [25] M. Backes, A. Butter, T. Plehn, and R. Winterhalder, How to GAN Event Unweighting, SciPost Phys. 10 (2021) 4, 089, arXiv:2012.07873 [hep-ph].
  * [26] E. Bothmann and L. Debbio, Reweighting a parton shower using a neural network: the final-state case, JHEP 01 (2019) 033, arXiv:1808.07802 [hep-ph].
  * [27] L. de Oliveira, M. Paganini, and B. Nachman, Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis, Comput. Softw. Big Sci. 1 (2017) 1, 4, arXiv:1701.05927 [stat.ML].
  * [28] J. W. Monk, Deep Learning as a Parton Shower, JHEP 12 (2018) 021, arXiv:1807.03685 [hep-ph].
  * [29] A. Andreassen, I. Feige, C. Frye, and M. D. Schwartz, JUNIPR: a Framework for Unsupervised Machine Learning in Particle Physics, Eur. Phys. J. C79 (2019) 2, 102, arXiv:1804.09720 [hep-ph].
  * [30] K. Dohi, Variational Autoencoders for Jet Simulation, arXiv:2009.04842 [hep-ph].
  * [31] F. A. Di Bello, S. Ganguly, E. Gross, M. Kado, M. Pitt, L. Santi, and J. Shlomi, Towards a Computer Vision Particle Flow, Eur. Phys. J. C 81 (2021) 2, 107, arXiv:2003.08863 [physics.data-an].
  * [32] P. Baldi, L. Blecher, A. Butter, J. Collado, J. N. Howard, F. Keilbach, T. Plehn, G. Kasieczka, and D. Whiteson, How to GAN Higher Jet Resolution, arXiv:2012.11944 [hep-ph].
  * [33] S. Otten, S. Caron, W. de Swart, M. van Beekveld, L. Hendriks, C. van Leeuwen, D. Podareanu, R. Ruiz de Austri, and R. Verheyen, Event Generation and Statistical Sampling for Physics with Deep Generative Models and a Density Information Buffer, Nature Commun. 12 (2021) 1, 2985, arXiv:1901.00875 [hep-ph].
  * [34] B. Hashemi, N. Amin, K. Datta, D. Olivito, and M. Pierini, LHC analysis-specific datasets with Generative Adversarial Networks, arXiv:1901.05282 [hep-ex].
  * [35] R. Di Sipio, M. Faucci Giannelli, S. Ketabchi Haghighat, and S. Palazzo, DijetGAN: A Generative-Adversarial Network Approach for the Simulation of QCD Dijet Events at the LHC, JHEP 08 (2020) 110, arXiv:1903.02433 [hep-ex].
  * [36] A. Butter, T. Plehn, and R. Winterhalder, How to GAN LHC Events, SciPost Phys. 7 (2019) 6, 075, arXiv:1907.03764 [hep-ph].
  * [37] Y. Alanazi, N. Sato, T. Liu, W. Melnitchouk, M. P. Kuchera, E. Pritchard, M. Robertson, R. Strauss, L. Velasco, and Y. Li, Simulation of electron-proton scattering events by a Feature-Augmented and Transformed Generative Adversarial Network (FAT-GAN), arXiv:2001.11103 [hep-ph].
  * [38] M. Paganini, L. de Oliveira, and B. Nachman, Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multilayer Calorimeters, Phys. Rev. Lett. 120 (2018) 4, 042003, arXiv:1705.02355 [hep-ex].
  * [39] M. Paganini, L. de Oliveira, and B. Nachman, CaloGAN : Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks, Phys. Rev. D97 (2018) 1, 014021, arXiv:1712.10321 [hep-ex].
  * [40] P. Musella and F. Pandolfi, Fast and Accurate Simulation of Particle Detectors Using Generative Adversarial Networks, Comput. Softw. Big Sci. 2 (2018) 1, 8, arXiv:1805.00850 [hep-ex].
  * [41] M. Erdmann, L. Geiger, J. Glombitza, and D. Schmidt, Generating and refining particle detector simulations using the Wasserstein distance in adversarial networks, Comput. Softw. Big Sci. 2 (2018) 1, 4, arXiv:1802.03325 [astro-ph.IM].
  * [42] M. Erdmann, J. Glombitza, and T. Quast, Precise simulation of electromagnetic calorimeter showers using a Wasserstein Generative Adversarial Network, Comput. Softw. Big Sci. 3 (2019) 4, arXiv:1807.01954 [physics.ins-det].
  * [43] ATLAS Collaboration, “Energy resolution with a GAN for Fast Shower Simulation in ATLAS.” ATLAS-SIM-2019-004, 2019.  https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PLOTS/SIM-2019-004/.
  * [44] D. Belayneh et al., Calorimetry with Deep Learning: Particle Simulation and Reconstruction for Collider Physics, Eur. Phys. J. C 80 (2020) 7, 688, arXiv:1912.06794 [physics.ins-det].
  * [45] E. Buhmann, S. Diefenbacher, E. Eren, F. Gaede, G. Kasieczka, A. Korol, and K. Krüger, Getting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed, Comput. Softw. Big Sci. 5 (2021) 1, 13, arXiv:2005.05334 [physics.ins-det].
  * [46] E. Buhmann, S. Diefenbacher, E. Eren, F. Gaede, G. Kasieczka, A. Korol, and K. Krüger, Decoding Photons: Physics in the Latent Space of a BIB-AE Generative Network, arXiv:2102.12491 [physics.ins-det].
  * [47] C. Chen, O. Cerri, T. Q. Nguyen, J. R. Vlimant, and M. Pierini, Analysis-Specific Fast Simulation at the LHC with Deep Learning, Comput. Softw. Big Sci. 5 (2021) 1, 15.
  * [48] C. Krause and D. Shih, CaloFlow: Fast and Accurate Generation of Calorimeter Showers with Normalizing Flows, arXiv:2106.05285 [physics.ins-det].
  * [49] K. Datta, D. Kar, and D. Roy, Unfolding with Generative Adversarial Networks, arXiv:1806.00433 [physics.data-an].
  * [50] M. Bellagente, A. Butter, G. Kasieczka, T. Plehn, and R. Winterhalder, How to GAN away Detector Effects, SciPost Phys. 8 (2020) 4, 070, arXiv:1912.00477 [hep-ph].
  * [51] M. Bellagente, A. Butter, G. Kasieczka, T. Plehn, A. Rousselot, R. Winterhalder, L. Ardizzone, and U. Köthe, Invertible Networks or Partons to Detector and Back Again, SciPost Phys. 9 (2020) 074, arXiv:2006.06685 [hep-ph].
  * [52] S. Bieringer, A. Butter, T. Heimel, S. Höche, U. Köthe, T. Plehn, and S. T. Radev, Measuring QCD Splittings with Invertible Networks, SciPost Phys. 10 (2021) 6, 126, arXiv:2012.09873 [hep-ph].
  * [53] D. MacKay, Probable Networks and Plausible Predictions – A Review of Practical Bayesian Methods for Supervised Neural Networks, Comp. in Neural Systems 6 (1995) 4679.
  * [54] R. M. Neal, Bayesian learning for neural networks.  PhD thesis, Toronto,  1995.
  * [55] Y. Gal, Uncertainty in Deep Learning.  PhD thesis, Cambridge,  2016.
  * [56] A. Kendall and Y. Gal, What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?, Proc. NIPS (2017) , arXiv:1703.04977 [cs.CV].
  * [57] S. Bollweg, M. Haußmann, G. Kasieczka, M. Luchmann, T. Plehn, and J. Thompson, Deep-Learning Jets with Uncertainties and More, SciPost Phys. 8 (2020) 1, 006, arXiv:1904.10004 [hep-ph].
  * [58] G. Kasieczka, M. Luchmann, F. Otterpohl, and T. Plehn, Per-Object Systematics using Deep-Learned Calibration, SciPost Phys. 9 (2020) 089, arXiv:2003.11099 [hep-ph].
  * [59] M. Bellagente, M. Haußmann, M. Luchmann, and T. Plehn, Understanding Event-Generation Networks via Uncertainties, arXiv:2104.04543 [hep-ph].
  * [60] Sherpa, E. Bothmann et al., Event Generation with Sherpa 2.2, SciPost Phys. 7 (2019) 3, 034, arXiv:1905.09127 [hep-ph].
  * [61] S. Catani, F. Krauss, R. Kuhn, and B. R. Webber, QCD matrix elements + parton showers, JHEP 11 (2001) 063, arXiv:hep-ph/0109231.
  * [62] M. Cacciari, G. P. Salam, and G. Soyez, FastJet User Manual, Eur. Phys. J. C 72 (2012) 1896, arXiv:1111.6097 [hep-ph].
  * [63] M. Cacciari, G. P. Salam, and G. Soyez, The anti-$k_{t}$ jet clustering algorithm, JHEP 04 (2008) 063, arXiv:0802.1189 [hep-ph].
  * [64] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, eds., pp. 8024–8035.  Curran Associates, Inc., 2019.  arXiv:1912.01703 [cs.LG].
  * [65] D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, arXiv:1412.6980 [cs.LG].
  * [66] L. N. Smith and N. Topin, Super-convergence: Very fast training of neural networks using large learning rates, in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, International Society for Optics and Photonics.  2019\.
  * [67] L. Ardizzone, C. Lüth, J. Kruse, C. Rother, and U. Köthe, Guided image generation with conditional invertible neural networks, arXiv:1907.02392 [cs.CV].
  * [68] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios, Cubic-spline flows, arXiv preprint arXiv:1906.02145 (2019) .
  * [69] R. Winterhalder, M. Bellagente, and B. Nachman, Latent Space Refinement for Deep Generative Models, arXiv:2106.00792 [stat.ML].
  * [70] M. Erdmann, L. Geiger, J. Glombitza, and D. Schmidt, Generating and refining particle detector simulations using the Wasserstein distance in adversarial networks, Comput. Softw. Big Sci. 2 (2018) 1, 4, arXiv:1802.03325 [astro-ph.IM].
  * [71] S. Diefenbacher, E. Eren, G. Kasieczka, A. Korol, B. Nachman, and D. Shih, DCTRGAN: Improving the Precision of Generative Models with Reweighting, JINST 15 (2020) 11, P11004, arXiv:2009.03796 [hep-ph].
  * [72] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, Generative adversarial nets, in Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14.  MIT Press, Cambridge, MA, USA,  2014.
  * [73] A. Grover, M. Dhar, and S. Ermon, Flow-gan: Combining maximum likelihood and adversarial learning in generative models, in AAAI Conference on Artificial Intelligence.  2018\.
  * [74] B. Nachman, A guide for deploying Deep Learning in LHC searches: How to achieve optimality and account for uncertainty, SciPost Phys. 8 (2020) 090, arXiv:1909.03081 [hep-ph].
  * [75] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, Variational inference: A review for statisticians, Journal of the American statistical Association 112 (2017) 518, 859.