# A Diffusion-based Method for Multi-turn Compositional Image Generation

Chao Wang
Toronto AI Lab
LG Electronics
<EMAIL_ADDRESS>
###### Abstract

Multi-turn compositional image generation (M-CIG) is a challenging task that
aims to iteratively manipulate a reference image given a modification text.
While most of the existing methods for M-CIG are based on generative
adversarial networks (GANs), recent advances in image generation have
demonstrated the superiority of diffusion models over GANs. In this paper, we
propose a diffusion-based method for M-CIG named conditional denoising
diffusion with image compositional matching (CDD-ICM). We leverage CLIP as the
backbone of image and text encoders, and incorporate a gated fusion mechanism,
originally proposed for question answering, to compositionally fuse the
reference image and the modification text at each turn of M-CIG. We introduce
a conditioning scheme to generate the target image based on the fusion
results. To prioritize the semantic quality of the generated target image, we
learn an auxiliary image compositional match (ICM) objective, along with the
conditional denoising diffusion (CDD) objective in a multi-task learning
framework. Additionally, we also perform ICM guidance and classifier-free
guidance to improve performance. Experimental results show that CDD-ICM
achieves state-of-the-art results on two benchmark datasets for M-CIG, i.e.,
CoDraw and i-CLEVR.

##  1 Introduction

Figure 1: A three-turn example of multi-turn compositional image generation
(M-CIG).

Image generation is a hot topic in computer vision, which has many
applications in a wide range of areas, such as art, education, and
entertainment. The generation of an image often needs to follow a text prompt.
Additionally, sometimes the generation also needs to be based on an existing
image rather than starting from scratch. Combining the above two requirements
brings about compositional image generation (CIG), which is to generate a
target image by changing a reference image according to a modification text.
Addressing this cross-modal task is useful in computer-aided design (CAD), as
it enables a computer system to generate images given verbal instructions from
users.

In this paper, we focus on multi-turn compositional image generation (M-CIG),
which is to perform CIG in an iterative manner. As shown in Figure 1, M-CIG
can be described as a sequence of CIG turns, where the initial reference image
is a background canvas, and the target image generated at each turn will be
used as the reference image at the next turn. Compared with CIG, M-CIG is more
challenging due to the iterative setting. Meanwhile, M-CIG is also more
practical than CIG, as in the real world, a user usually needs to go through a
series of incremental interactions with a computer system before achieving a
final goal.

To the best of our knowledge, the existing methods for M-CIG [12, 13, 34] are
mostly based on generative adversarial networks (GANs) [15], which are
currently the dominant family of techniques in image generation. According to
some theoretical and empirical studies [54, 2, 6, 36, 4], although GANs can
generate high-quality images, they are usually difficult to train, and the
diversity of the generated images is also limited. Recently, diffusion models
[58, 62, 16, 59], which are another family of generative modeling techniques,
have gained great popularity in image generation. Compared with GANs,
diffusion models are easier to train due to the straightforward definition of
objectives, and can also generate more diverse images due to the explicit
modeling of data distribution. As for the quality of the generated images, it
has been demonstrated that diffusion models are comparable to or even better
than GANs [63, 40, 8, 17]. Therefore, we apply diffusion models to M-CIG.

Diffusion models rely on a denoising diffusion mechanism [16] to generate
images, which can be conditional so that only desired images are generated.
Therefore, the key to addressing M-CIG using diffusion models is to learn
conditional denoising diffusion (CDD), where the condition for generating the
target image at each turn comes from the reference image and the modification
text. However, this raises the following two problems:

The lack of an appropriate conditioning scheme. Although many conditioning
schemes have been proposed for diffusion models, most of these works only deal
with uni-modal cases, where the condition comes from either an image [51, 53,
49] or a text [39, 47, 52]. The conditioning scheme proposed by [22] is aimed
at a multi-modal case, where the condition comes from an image-text pair, but
this work assumes that the text just describes the semantics of the image
rather than the desired change to it. In a word, the above conditioning
schemes cannot support the application of diffusion models to M-CIG.

The concern about the semantic quality of the generated target image. For the
generated target image, we are not only concerned with its visual quality, but
also its semantic quality, which refers to whether it contains the desired
objects and whether the contained objects constitute the desired topology.
Actually, we believe that the semantic quality deserves more concern than the
visual quality. The reason is two-fold. On the one hand, a high semantic
quality implies a high visual quality, but the reverse is not true. On the
other hand, due to the iterative nature of M-CIG, semantic mistakes are likely
to accumulate from turn to turn, which may corrupt the rear turns.

To solve these problems, we propose a diffusion-based method for M-CIG named
conditional denoising diffusion with image compositional matching (CDD-ICM),
which features a novel conditioning scheme equipped with a multi-task learning
framework. Specifically, we use CLIP [45] as the backbone to encode images and
texts. On this basis, we borrow a gated fusion mechanism from a question
answering (QA) method [67] to perform compositional fusion between the
reference image and the modification text at each turn of M-CIG, and use the
result as the condition of the denoising diffusion mechanism to generate the
target image. To guarantee the semantic quality of the generated target image,
we learn image compositional matching (ICM) as an auxiliary objective of CDD
to explicitly enhance the conditon, where the compositional fusion result is
aligned with the representation of the target image through contrastive
learning. Moreover, we also perform ICM guidance and classifier-free guidance
[18] to boost performance. Experimental results show that CDD-ICM achieves
state-of-the-art (SOTA) performance on two benchmark datasets for M-CIG,
namely CoDraw [24] and i-CLEVR [12].

The contribution of this paper is three-fold. First, we creatively apply
diffusion models to M-CIG, where a novel conditioning scheme is developed to
handle a compositional image-text pair, integrating the denoising diffusion
mechanism with CLIP and a gated fusion mechanism. Second, to prioritize the
semantic quality of the generated target image, we establish a multi-task
learning framework for the conditioning scheme, where ICM serves as an
auxiliary objective of CDD to explicitly enhance the condition. Third, our
diffusion-based method outperforms the existing GAN-based methods on two M-CIG
benchmark datasets.

##  2 Method

Figure 2: An overview of CDD-ICM. Colored components are trainable, and those
of the same color share their trainable parameters.

In this section, we elaborate CDD-ICM, which is our diffusion-based method for
M-CIG. We begin by providing a task formulation of M-CIG, which is followed by
a detailed introduction to the design of CDD-ICM.

###  2.1 Task Formulation

Given an initial reference image $a^{(1)}$, which is a background canvas, and
a sequence of $k$ modification texts $\\{m^{(1)},\ldots,m^{(k)}\\}$, which
describe the desired changes to be successively made to $a^{(1)}$, M-CIG is a
$k$-turn iterative process, where at each turn $i\in\\{1,\ldots,k\\}$, it is
required to generate a target image $z^{(i)}$ by changing the current
reference image $a^{(i)}$ according to $m^{(i)}$, and if $i<k$, $z^{(i)}$ will
be used as the next reference image $a^{(i+1)}$.

###  2.2 Encoding

Considering the cross-modal nature of M-CIG, we map images and texts into a
joint representation space through encoding. As shown in Figure 2, we include
an image encoder and a text encoder in CDD-ICM, where the former is used to
encode reference images and target images, and the latter is used to encode
modification texts. Since M-CIG is a vision-and-language (V&L) task, we take
advantage of large-scale V&L pre-training by using CLIP, which is pre-trained
on 400M image-text pairs, as the backbone of both encoders. Specifically, we
use the vision part of CLIP, which is a vision transformer (ViT) [11], as the
backbone of the image encoder, and use the language part of CLIP, which is a
GPT-like [46] auto-regressive language model, as the backbone of the text
encoder. In each encoder, we append a linear projection layer after the
backbone, which finally yields the representations. For both linear projection
layers, we set their output dimensionality to the same value $d$, which is a
hyper-parameter denoting the dimensionality of the joint representation space.

Additionally, as shown in Figure 2, we also include a noisy image encoder in
CDD-ICM, which is used to encode the noisy target images obtained in the
denoising diffusion mechanism. It has the same structure as the image encoder,
but holds different trainable parameters.

###  2.3 Compositional Fusion

At each turn of M-CIG, to extract clues for the generation of the target
image, we perform compositional fusion between the reference image and the
modification text. As shown in Figure 2, we include a fusion module in CDD-
ICM, which fuses the representation of the reference image with that of the
modification text. Actually, we can interpret each turn of M-CIG from the
perspective of QA. Specifically, we can regard the reference image as a
context, the modification text as a relevant question, and the target image as
the corresponding answer. In this way, to implement the fusion module, we
borrow the following gated fusion mechanism from a QA method [67]:

$\small\begin{split}&\mathrm{f}(u,v)=g\odot h+(1-g)\odot u\\\
&g=\mathrm{sigmoid}(W_{g}[u;v;u\odot v;u-v]+b_{g})\\\
&h=\mathrm{gelu}(W_{h}[u;v;u\odot v;u-v]+b_{h})\end{split}$ (1)

where $W_{g}$ and $W_{h}$ are trainable weight matrices, $b_{g}$ and $b_{h}$
are trainable bias vectors, $\odot$ denotes element-wise multiplication, and
$[;]$ denotes vector concatenation. In the fusion module, we set $u$ to the
representation of the reference image, set $v$ to that of the modification
text, and thereby obtain $\mathrm{f}(u,v)$ as the compositional fusion result.

###  2.4 Conditional Denoising Diffusion

To generate the target image at each turn of M-CIG, we learn conditional
denoising diffusion (CDD), which is to perform the generation using a
denoising diffusion mechanism conditioned on the compositional fusion result
between the reference image and the modification text. As proposed by [16],
the denoising diffusion mechanism consists of a forward diffusion process,
which gradually injects noises to the target image, and a reverse denoising
process, which gradually erases the injected noises.

With the target image denoted as $x_{0}\sim q(x_{0})$, the forward diffusion
process is a pre-defined Markov chain of $T$ time steps $\langle
1,\ldots,T\rangle$, where the state transfers from $x_{0}$ all the way to
$x_{T}$. Specifically, at each time step $t$, we obtain $x_{t}$ by injecting a
Gaussian noise $\epsilon_{t}\sim N(0,\mathit{I})$ to $x_{t-1}$:

$\small\begin{split}&q(x_{t}|x_{t-1})=N(\sqrt{\alpha_{t}}x_{t-1},(1-\alpha_{t})\mathit{I})\\\
\Leftrightarrow\quad&x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}}\epsilon_{t}\end{split}$
(2)

where $\alpha_{t}$ is a hyper-parameter used to control the noise scale. It is
easy to derive that with $\prod_{i=1}^{t}\alpha_{i}$ denoted as
$\bar{\alpha}_{t}$, we can actually obtain $x_{t}$ by directly injecting
$\epsilon_{t}$ to $x_{0}$:

$\small\begin{split}&q(x_{t}|x_{0})=N(\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\mathit{I})\\\
\Leftrightarrow\quad&x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon_{t}\end{split}$
(3)

As shown in Figure 2, we include a noise injector in CDD-ICM, which executes
Equation 3 at an arbitrary time step $t$ to obtain $x_{t}$ as a noisy target
image. To implement the noise injector, we set each
$\alpha_{t}\in\\{\alpha_{1},\ldots,\alpha_{T}\\}$ in the following way
proposed by [40]:

$\small\begin{split}&\alpha_{t}=\frac{s(t)}{s(t-1)}\\\
&s(t)=\mathrm{cos}^{2}(\frac{t/T+0.008}{1.008}\cdot\frac{\pi}{2})\end{split}$
(4)

With the compositional fusion result between the reference image and the
modification text denoted as $c$, the reverse denoising process is a
parameterized Markov chain of $T$ time steps $\langle T,\ldots,1\rangle$,
where the state transfers from $x_{T}$ all the way back to $x_{0}$ conditioned
on $c$. Specifically, at each time step $t$, given $x_{t}$ and the condition
$c$, we obtain $x_{t-1}$ using a neural network $\theta$:

$\small\begin{split}&p_{\theta}(x_{t-1}|x_{t},c)=N\big{(}\mu_{\theta}(x_{t},t,c),\Sigma_{\theta}(x_{t},t,c)\big{)}\\\
\Leftrightarrow\quad&x_{t-1}=\mu_{\theta}(x_{t},t,c)+\Sigma_{\theta}^{\frac{1}{2}}(x_{t},t,c)\xi_{t}\end{split}$
(5)

where $\xi_{t}\sim N(0,\mathit{I})$. According to [16], we can learn $\theta$
by minimizing the following variational lower-bound (VLB) loss:

$\small\begin{split}&L^{vlb}=E_{x_{0}\sim q(x_{0}),t\sim
U\\{1,T\\}}[L^{vlb}_{t}]\\\ &L^{vlb}_{t}=\begin{cases}-\mathrm{log}\
p_{\theta}(x_{t-1}|x_{t},c),&\text{if}\ t=1\\\
D_{KL}\big{(}q(x_{t-1}|x_{t},x_{0})||p_{\theta}(x_{t-1}|x_{t},c)\big{)},&\text{else}\end{cases}\end{split}$
(6)

Using Bayes’ theorem, it can be derived that with $1-\alpha_{t}$ denoted as
$\beta_{t}$, $q(x_{t-1}|x_{t},x_{0})$ is the following Gaussian distribution:

$\small\begin{split}&q(x_{t-1}|x_{t},x_{0})=N\big{(}\tilde{\mu}(x_{t},x_{0}),\tilde{\beta}_{t}\mathit{I}\big{)}\\\
&\tilde{\mu}(x_{t},x_{0})=\frac{\sqrt{\bar{\alpha}_{t-1}\beta_{t}}}{1-\bar{\alpha}_{t}}x_{0}+\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_{t}}x_{t}\\\
&\tilde{\beta}_{t}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}\end{split}$
(7)

Based on Equation 3 and Equation 7, we parameterize $\mu_{\theta}(x_{t},t,c)$
in the following way proposed by [16]:

$\small\mu_{\theta}(x_{t},t,c)=\frac{1}{\sqrt{\alpha_{t}}}\big{(}x_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t,c)\big{)}$
(8)

where $\epsilon_{\theta}(x_{t},t,c)$ is a prediction to $\epsilon_{t}$.
Besides, we also parameterize $\Sigma_{\theta}(x_{t},t,c)$ in the following
way proposed by [40]:

$\small\Sigma_{\theta}(x_{t},t,c)=\mathrm{exp}\big{(}\mathrm{log}\frac{\beta_{t}}{\tilde{\beta}_{t}}\rho_{\theta}(x_{t},t,c)+\mathrm{log}\tilde{\beta}_{t}\big{)}$
(9)

where $\rho_{\theta}(x_{t},t,c)$ is a fraction used to interpolate between
$\mathrm{log}\beta_{t}$ and $\mathrm{log}\tilde{\beta}_{t}$. As shown in
Figure 2, we include a denoising U-Net in CDD-ICM, which performs the above
parameterizations and thus can be seen as $\theta$. To implement the denoising
U-Net, we make three changes to the U-Net structure used by [8]. First, we
replace the class embedding with the condition $c$. Second, we concatenate
$x_{t}$ with the reference image along the channel dimension, and thereby use
the result as the input. Third, we concatenate the patch representations of
the reference image with the token representations of the modification text,
and thereby use the result to augment each attention layer as suggested by
[39]. The output of the denoising U-Net is divided into two parts along the
channel dimension, which are separately used as $\epsilon_{\theta}(x_{t},t,c)$
and $\rho_{\theta}(x_{t},t,c)$.

According to [16], to learn the above reverse denoising process, instead of
minimizing $L^{vlb}$, we can actually minimize the following mean squared
error (MSE) loss:

$\small L^{mse}=E_{x_{0}\sim q(x_{0}),t\sim
U\\{1,T\\}}[||\epsilon_{t}-\epsilon_{\theta}(x_{t},t,c)||^{2}]$ (10)

Compared with $L^{vlb}$, $L^{mse}$ is not only simpler but also more
effective. However, minimizing $L^{mse}$ cannot bring any learning signal to
$\Sigma_{\theta}(x_{t},t,c)$. To benefit from learning
$\Sigma_{\theta}(x_{t},t,c)$, we combine $L^{mse}$ with $L^{vlb}$ as suggested
by [40]. Specifically, we minimize a CDD loss $L^{cdd}$, which is calculated
as follows:

$\small L^{cdd}=L^{mse}+\gamma L^{vlb}$ (11)

where $\gamma$ is a hyper-parameter used to control the weight of $L^{vlb}$.
Additionally, we also stop the gradients of $L^{vlb}$ from flowing to
$\epsilon_{\theta}(x_{t},t,c)$.

###  2.5 Image Compositional Matching

At each turn of M-CIG, the semantic quality of the generated target image
depends on the condition of the denoising diffusion mechanism, which is the
compositional fusion result between the reference image and the modification
text. Although learning CDD ensures that the condition is learned, this effect
is implicit. To explicitly enhance the condition so that it embodies more
clues about the target image, we learn image compositional matching (ICM) as
an auxiliary objective of CDD, which is to align the compositional fusion
result with the representation of the target image.

To learn ICM, we adopt the InfoNCE loss [41] used in the contrastive pre-
training of CLIP, and apply it to individual turns constituting M-CIG samples.
Specifically, given a mini-batch of $n$ (reference image, modification text,
target image) triples $\\{(a_{1},m_{1},z_{1}),\ldots,(a_{n},m_{n},z_{n})\\}$,
each of which denotes an individual turn picked from an M-CIG sample, we treat
them as positive samples, and generate $n^{2}-n$ negative samples by replacing
the target image $z_{i}$ in each positive sample $(a_{i},m_{i},z_{i})$
separately with the other $n-1$ target images
$\\{z_{1},\ldots,z_{n}\\}-\\{z_{i}\\}$. For each of the positive and negative
samples, suppose that we have already obtained the compositional fusion result
between the reference image and the modification text and the representation
of the target image, then we calculate the cosine similarity between them. As
a result, we construct a similarity matrix $S\in\mathbb{R}^{n\times n}$, where
the element at the $i$-th row and the $j$-th column corresponds to the sample
$(a_{i},m_{i},z_{j})$. It is easy to see that the diagonal elements in $S$
correspond to the positive samples, while the other elements correspond to the
negative samples. Based on $S$, we minimize an ICM loss $L^{icm}$, which is
calculated as follows:

$\small\begin{split}L^{icm}=&\frac{1}{n}\mathrm{tr}\Big{(}\mathrm{-log}\big{(}\mathrm{softmax}(\frac{S}{\tau})\big{)}\Big{)}+\\\
&\frac{1}{n}\mathrm{tr}\Big{(}\mathrm{-log}\big{(}\mathrm{softmax}(\frac{S^{\top}}{\tau})\big{)}\Big{)}\end{split}$
(12)

where $\tau$ is a trainable temperature scalar, $\mathrm{tr}(\cdot)$ denotes
calculating matrix trace, and $\mathrm{softmax}(\cdot)$ is calculated along
the row dimension. In this way, the compositional fusion result between a
(reference image, modification text) pair will be close to the representation
of the real target image, while apart from those of the fake ones.

Besides, to enable ICM guidance, which will be introduced later, we also learn
noise-aware image compositional matching (N-ICM). Specifically, we replace the
above target images with their noisy variants, which are obtained using the
noise injector, and encode these noisy target images using the noisy image
encoder. On this basis, we minimize an N-ICM loss $L^{n-icm}$, which is
calculated in the same way as we calculate $L^{icm}$.

###  2.6 Training and Inference

For the training, we disassemble M-CIG samples into individual turns and
thereby apply teacher forcing. On this basis, we divide the training into
three stages, where in each stage, we minimize a different loss through mini-
batch gradient descent to update the corresponding trainable components of
CDD-ICM. Specifically, in the first stage, we minimize $L^{icm}$ to update the
image encoder, the text encoder, and the fusion module. In the second stage,
we minimize the following joint loss, which is a combination of $L^{cdd}$ and
$L^{icm}$, to update the image encoder, the text encoder, the fusion module,
and the denoising U-Net:

$\small L^{joint}=L^{cdd}+\delta L^{icm}$ (13)

where $\delta$ is a hyper-parameter used to control the weight of $L^{icm}$.
In the third stage, we freeze the image encoder, the text encoder, and the
fusion module, and minimize $L^{n-icm}$ to update the noisy image encoder. To
effectively fine-tune CLIP, we set the backbone learning rate as a product of
the global learning rate and a backbone activity ratio $\eta$, which is a
hyper-parameter. From the perspective of transfer learning, $\eta$ controls
the trade-off between the knowledge transferred from CLIP and that embodied in
the training data.

For the inference, at each turn of M-CIG, we use the image encoder to encode
the reference image, use the text encoder to encode the modification text, use
the fusion module to perform compositional fusion based on the encoding
results, and use the denoising U-Net to iteratively execute Equation 5 from
the time step $T$ until the time step $1$, where the condition $c$ is set to
the compositional fusion result. From Equation 3 and Equation 4, it can be
derived that if $T$ is large enough, then $q(x_{T})\approx N(0,I)$, thus we
sample $x_{T}$ from $N(0,I)$ at the time step $T$. To accelerate the
inference, we traverse only a part of the time steps, which are uniformly
distributed among all of them, and make this process deterministic as
suggested by [60]. Finally, we obtain $x_{0}$ as the generated target image at
the time step $1$.

Moreover, we also perform ICM guidance and classifier-free guidance to boost
performance. Our ICM guidance is similar to the CLIP guidance of [39].
Specifically, suppose that we have minimized $L^{n-icm}$ in the training, then
in the inference, instead of using $\mu_{\theta}(x_{t},t,c)$ in Equation 5, we
use $\hat{\mu}_{\theta}(x_{t},t,c)$, which is obtained by perturbing
$\mu_{\theta}(x_{t},t,c)$ using the gradient of $L^{n-icm}$ with respect to
$x_{t}$:

$\small\hat{\mu}_{\theta}(x_{t},t,c)=\mu_{\theta}(x_{t},t,c)+\psi\Sigma_{\theta}(x_{t},t,c)\nabla_{x_{t}}L^{n-icm}$
(14)

where $\psi$ is a hyper-parameter used to control the perturbation scale. Our
classifier-free guidance is similar to that of [39]. Specifically, in the
training, when calculating $\epsilon_{\theta}(x_{t},t,c)$ in Equation 10, we
set the condition $c$ to $\vec{0}$ with a probability of $\lambda$, which is a
hyper-parameter. On this basis, in the inference, instead of using
$\epsilon_{\theta}(x_{t},t,c)$ in Equation 8, we use
$\hat{\epsilon}_{\theta}(x_{t},t,c)$, which is obtained by perturbing
$\epsilon_{\theta}(x_{t},t,c)$ using $\epsilon_{\theta}(x_{t},t,\vec{0})$:

$\small\hat{\epsilon}_{\theta}(x_{t},t,c)=\phi\epsilon_{\theta}(x_{t},t,c)+(1-\phi)\epsilon_{\theta}(x_{t},t,\vec{0})$
(15)

where $\phi$ is a hyper-parameter used to control the perturbation scale.

##  3 Related Works

###  3.1 Image Manipulation

The goal of image manipulation is to modify specific attributes of an image
while avoiding unintended changes or generating a completely new image.
Existing works can be split into two main categories: image-to-image
translation and text-conditioned image manipulation.

Image-to-Image Translation. The image-to-image translation aims to generate an
output image only conditioning on an input image, i.e., uni-modal condition.
Image inpainting and image super-resolution are two typical image-to-image
translation tasks. In recent years, deep learning has achieved great success
in image inpainting. Context Encoders [44] first explores to utilize
conditional GANs. Multiple variants [69, 72, 73, 29] of U-Net [50] have been
proposed for image inpainting. Some works explore multi-stage generation by
taking object edges [38], structures [48], or semantic segmentation maps [64]
as intermediate clues. In terms of super-resolution, most early works are
regression-based and trained with MSE loss [9, 10, 70, 23]. Auto-regressive
models [7, 42] and GAN-based methods [19, 26, 68, 35, 32] have also shown high
quality results.

Text-Conditioned Image Manipulation. The text-conditioned image manipulation
targets generating an output image conditioned on both the input image and
text, i.e., multi-modal condition. The input text can be a caption-like
description of the target image, and the editing is usually single-turn. TAGAN
[37] employs word-level local discriminators to preserve text-irrelevant
content. ManiGAN [28] first selects image regions and then correlates the
regions with semantic words. DiffusionCLIP [22] is a robust framework that
utilizes the pre-trained diffusion models and CLIP loss for image
manipulation.

The input text can also be user-provided text instructions that describe
desired modifications, such as adding, changing, or removing the objects in
images. Generating an image based on provided instructions and an input image
is dubbed as the compositional image generation (CIG) task in this paper. [1]
achieve great performance on the benchmarks CSS [65] and Fashion Synthesis
[75] by designing an improved image & text composition layer and a multi-modal
similarity module. [74] propose a GAN-based method to locally modify image
features and show remarkable results on both CSS and Abstract Scene [76].
Afterward, the M-CIG task presents a more challenging setting compared to the
above single-turn CIG task. [12] first propose the M-CIG task known as
Generative Neural Visual Artist (GeNeVA) task, which requires iteratively
generating an image according to ongoing linguistic input. [14] introduce the
self-supervised counterfactual reasoning (SSCR) framework to tackle the data
scarcity problem. LatteGAN [34] improves desired object generation by
introducing a Latte module and a text-conditioned U-Net discriminator. Our
research work targets the M-CIG task, following [12], we conducted experiments
on CoDraw [25] and i-CLEVR [12].

###  3.2 Diffusion Models

Diffusion models (DMs) [58], which formulate the data sampling process as an
iterative denoising procedure, are closely related to a large family of
methods for learning generative models as transition operators of Markov
chains [3, 58, 56, 61, 27]. Many research works concentrate on improving the
diffusion process of DMs. [62] propose to estimate the gradients of data
distribution via score matching and produce samples via Langevin dynamics.
Denoising diffusion probabilistic models (DDPMs) [16], which optimize a
variational lower bound to the log-likelihood, can achieve comparable sample
quality as GANs [5, 20]. Denoising diffusion implicit models (DDIMs) [59]
speed up the sampling process while enabling near-perfect inversion [8]. The
improved DDPM [40] introduces several modifications to achieve competitive
likelihoods without sacrificing sample quality. The latent diffusion [49]
model is applied in latent space instead of pixel space to enable an efficient
diffusion process. Although GANs have achieved plausible results in image
synthesis, they are usually difficult to train and tend to limit the diversity
of the generated images [57, 66]. DMs are more stable during training and
demonstrate comparable or even better performance for image synthesis [8, 63,
17].

Motivated by the progress in developing DMs, some research works explore text
or image conditional diffusion mechanisms. While certain diffusion models
solely utilize an input image for conditioning, such as PALETTE [51] and SR3
[53], those that condition on both an input image and text are more pertinent
to our work. GLIDE [39] is a text-guided diffusion model, where classifier-
free guidance yields higher-quality images than CLIP guidance. DALL-E 2 [47]
initially generates a CLIP [45] image embedding given a text caption and then
generates an image conditioned on the image embedding. ImageGen [52] exhibits
a deep level of language understanding which enables high-fidelity image
generation. Stable Diffusion [49] is an efficient latent diffusion model and
has achieved superior image synthesis performance. [30] compose pre-trained
text-guided diffusion models to improve structured generalization for image
generation. However, in the context of this paper, it should be noted that the
textual input in the aforementioned models refers to a caption-like
description, rather than iterative instructions on image manipulations.
Designing diffusion-based methods for the M-CIG task remains a relatively
underexplored area, presenting challenges in iteratively modifying images with
instructions and conditioning the denoising diffusion mechanism on multi-
modalities. To address this gap in the literature, we propose a diffusion-
based approach coupled with auxiliary ICM objectives to enhance the visual and
semantic fidelity of generated images in the M-CIG task.

##  4 Experiments

Method | CoDraw | i-CLEVR
---|---|---
Precision | Recall | F1 | RSIM | Precision | Recall | F1 | RSIM
GeNeVA-GAN [12] | 66.64 | 52.66 | 58.83 | 35.41 | 92.39 | 84.72 | 88.39 | 74.02
SSCR [13] | 58.17 | 56.61 | 57.38 | 39.11 | 73.75 | 46.39 | 56.96 | 34.54
TIRG [21] | 76.56 | 73.40 | 72.40 | 46.64 | 94.30 | 92.96 | 93.71 | 77.55
LatteGAN [34] | 81.50 | 78.37 | 77.51 | 54.16 | 97.72 | 96.93 | 97.26 | 83.21
CDD-ICM (ours) | 90.61 | 87.55 | 89.05 | 57.39 | 99.99 | 99.94 | 99.96 | 85.66
Table 1: Performance comparison on CoDraw and i-CLEVR.

###  4.1 Datasets

To verify the effectiveness of CDD-ICM, we conduct experiments on the
following two M-CIG benchmark datasets:

CoDraw. CoDraw contains $8$K M-CIG samples for training, $1$K for validation,
and $1$K for test. The number of turns per M-CIG sample varies between $1$ and
$14$ with an average of $4.25$. The reference images and the target images
contain $58$ classes of clip-art-style objects, such as boys, girls, and
trees. The modification texts are conversations between a teller and a drawer.

i-CLEVER. i-CLEVER contains $6$K M-CIG samples for training, $2$K for
validation, and $2$K for test. The number of turns per M-CIG sample is always
$5$. The reference images and the target images contain $24$ classes of
colored geometric objects, such as red spheres, yellow cubes, and blue
cylinders. The modification texts are sentences indicating the addition of a
new object.

Both datasets adopt four metrics to evaluate the semantic quality of the
generated target images: precision, recall, F1 score, and relational
similarity (RSIM). Specifically, each dataset comes with an object detector,
which is trained on the dataset by [12]. In the evaluation, the object
detector is applied to both the generated target images and the ground-truth
target images. On this basis, precision, recall, and F1 score are calculated
for each turn of M-CIG by comparing the object presence in the generated
target image with that in the ground-truth target image:

$\small\begin{split}&\mathrm{precision}=\frac{|O_{gen}\cap
O_{gt}|}{|O_{gen}|}\\\ &\mathrm{recall}=\frac{|O_{gen}\cap
O_{gt}|}{|O_{gt}|}\\\
&\mathrm{F1}=2\cdot\frac{\mathrm{precision}\cdot\mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}\end{split}$
(16)

where $O_{gen}$ and $O_{gt}$ denote the objects in the generated target image
and the ground-truth target image, respectively. RSIM is calculated on the
last turn of M-CIG by comparing the object topology in the generated target
image with that in the ground-truth target image:

$\small\begin{split}&\mathrm{RSIM}=\mathrm{recall}\cdot\frac{|E_{gen}\cap
E_{gt}|}{|E_{gt}|}\end{split}$ (17)

where $E_{gen}$ and $E_{gt}$ denote the edges interconnecting $O_{gen}\cap
O_{gt}$ in the generated target image and the ground-truth target image,
respectively.

###  4.2 Implementation Details

We use PyTorch [43] to implement CDD-ICM, and use HuggingFace’s Transformers
[71] to load CLIP. In the three CLIP-based encoders, we adopt the basic
version of CLIP (i.e. CLIP-ViT-B/32) as the backbone, set the backbone
activity ratio $\eta$ to $0.001$, and set the output dimensionality $d$ of all
the linear projection layers to $512$. In the denoising diffusion mechanism,
we use $1000$ time steps for the training (i.e. $T=1000$), and use the $250$
time steps uniformly distributed among them for the inference. For ICM
guidance, we set $\psi$ to $2$. For classifier-free guidance, we set the value
of $\lambda$ to $0.2$ and $\phi$ to $3$. To calculate $L^{cdd}$, we set
$\gamma$ to $1.5$. To calculate $L^{icm}$ and $L^{n-icm}$, we initialize
$\tau$ to $e^{-1}$. To calculate $L^{joint}$ in the second training stage, we
set $\delta$ to $0.1$. For the optimization in each training stage, we apply
an AdamW optimizer [31] with an initial learning rate of $0.0001$ and a weight
decay factor of $0.01$. We perform the optimization on $8$ NVIDIA V100 $16$GB
GPUs in parallel, and set the mini-batch size on each GPU to $32$. We
calculate the average loss on the validation subset after every $10$ epochs.
If the resulting number is reduced, then we save the current CDD-ICM model,
otherwise we restore the CDD-ICM model to the previous saved version. We decay
the learning rate by $50\%$ after each restoration, and terminate the
optimization after the $5$th restoration.

###  4.3 Experimental Results

On each dataset, we train a CDD-ICM model by using the training set for
optimization and using the validation set for model selection. To compare CDD-
ICM with the existing M-CIG methods, we use the test set for evaluation, and
finally report the precision, recall, F1 score, and RSIM over all the M-CIG
samples in the test set. As shown in Table 1, we achieve SOTA performance on
both datasets. Specifically, on CoDraw, CDD-ICM outperforms the existing M-CIG
methods by a large margin, where the advantage in precision, recall, and F1
score is relatively larger than that in RSIM. On i-CLEVR, although the
existing M-CIG methods did not leave too much room for improvement in
precision, recall, and F1 score, CDD-ICM is still better than them in these
metrics, reaching almost perfect numbers, and also outperforms them in RSIM.

From all the M-CIG methods in Table 1, we observe two regularities. On the one
hand, the performance of these methods on CoDraw is generally worse than that
on i-CLEVR. By comparing CoDraw with i-CLEVR, we speculate that this is mainly
because the modification texts in CoDraw are commonly longer and more
complicated than those in i-CLEVR, which makes it more difficult to generate
the desired target images. On the other hand, the performance of these methods
in object presence, which is reflected by precision, recall, and F1 score, is
generally better than that in object topology, which is reflected by RSIM. By
investigating both datasets, we speculate that this is mainly because good
performance in object presence just requires correctly identifying the names
and attributes of objects from the modification texts, while that in object
topology usually requires comprehensively understanding the modification
texts.

###  4.4 Case Study

Figure 3: Demo cases from CoDraw and i-CLEVR. For the convenience of display,
we only include the utterances of the drawer in the modification texts of
CoDraw.

To visually demonstrate the capability of CDD-ICM, we select several
representative M-CIG samples from the test set of both datasets, and use the
inference results of CDD-ICM on them as demo cases. A demo case from CoDraw
and another one from i-CLEVR are shown in Figure 3, and more demo cases are
available in the appendix. In the demo cases from CoDraw, the generated target
images contain most of the desired objects, but there are still some extra and
missing objects. Besides, it also shows that on CoDraw, CDD-ICM struggles with
accurately positioning and orienting objects. In the demo cases from i-CLEVR,
the generated target images are very similar to the ground-truth target
images, where only a few objects are misaligned.

###  4.5 Ablation Study

Method | CoDraw | i-CLEVR
---|---|---
F1 | RSIM | F1 | RSIM
CDD-ICM | 89.05 | 57.39 | 99.96 | 85.66
w/o ICM | 75.86 | 50.58 | 89.92 | 74.49
w/o ICM Guidance | 87.69 | 56.94 | 96.27 | 81.32
w/o Classifier-free |  |  |  |
Guidance | 86.34 | 56.11 | 94.91 | 80.28
Fine-tuning of CLIP: |  |  |  |
Frozen | 80.63 | 54.89 | 87.24 | 72.44
Fine-tuning of CLIP: |  |  |  |
Fully-Trainable | 58.52 | 39.97 | 65.83 | 44.76
w/o Iterative Setting | 92.51 | 75.93 | 100.00 | 96.20
Table 2: Results of ablation experiments.

To probe the performance contribution from each design point of CDD-ICM, we
conduct the following five ablation experiments. As shown in Table 2, in each
ablation experiment, we change a corresponding design point, and report the
resulting F1 score and RSIM on each dataset.

ICM. We disable ICM by skipping the first training stage and setting $\delta$
to $0$ when calculating $L^{joint}$ in the second training stage. As a result,
we observe a significant drop in F1 score and RSIM. This verifies the
effectiveness of learning ICM as an auxiliary objective of CDD. Besides, we
also observe a significant drop in the converging speed of the second training
stage. This further verifies that learning ICM is beneficial for learning CDD.

ICM Guidance. We disable ICM guidance by skipping the third training stage and
setting $\psi$ to $0$. As a result, we observe a drop in F1 score and RSIM.
This verifies the effectiveness of ICM guidance.

Classifier-free Guidance. We disable classifier-free guidance by setting
$\lambda$ to $0$ and setting $\phi$ to $1$. As a result, we observe a drop in
F1 score and RSIM. This verifies the effectiveness of classifier-free
guidance.

Fine-tuning of CLIP. For the fine-tuning of CLIP, which is controlled by the
backbone activity ratio $\eta$, we examine two extreme cases. On the one hand,
we make CLIP frozen by setting $\eta$ to $0$. On the other hand, we make CLIP
fully-trainable by setting $\eta$ to $1$. As a result, we observe a
significant drop in F1 score and RSIM in both cases. This verifies the
necessity of applying $\eta$.

Iterative Setting. In the evaluation, we disable the iterative setting by
using the ground-truth target image at each turn of M-CIG as the reference
image of the next turn, which actually downgrades M-CIG to CIG. As a result,
we observe a significant rise in F1 score and RSIM. This verifies that M-CIG
is more challenging than CIG.

##  5 Conclusion and Limitation

In this paper, we focus on M-CIG, which is a challenging and practical image
generation task, and propose a diffusion-based method named CDD-ICM, which
achieves SOTA performance on CoDraw and i-CLEVR. The limitation of CDD-ICM
mainly lies in its inference efficiency. Although we have accelerated the
inference of CDD-ICM by traversing only a part of the time steps in a
deterministic manner, it still takes $3$ GPU seconds for CDD-ICM to generate a
target image, which is much slower than the GAN-based methods. In the future,
we plan to further accelerate the inference of CDD-ICM by applying latent
diffusion models [49] and knowledge distillation methods [33, 55].

## References

  * [1] Kenan E. Ak, Ying Sun, and Joo-Hwee Lim.  Learning cross-modal representations for language-based image manipulation.  2020 IEEE International Conference on Image Processing (ICIP), pages 1601–1605, 2020.
  * [2] Martin Arjovsky and Leon Bottou.  Towards principled methods for training generative adversarial networks.  In ICLR, 2017.
  * [3] Yoshua Bengio, Eric Laufer, Guillaume Alain, and Jason Yosinski.  Deep generative stochastic networks trainable by backprop.  In ICML, pages 226–234. PMLR, 2014.
  * [4] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In ICLR, 2019.
  * [5] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  ICLR, 2019.
  * [6] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston.  Neural photo editing with introspective adversarial networks.  In ICLR, 2017.
  * [7] Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens.  Pixel recursive super resolution.  In ICCV, pages 5439–5448, 2017.
  * [8] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In NeurIPS, 2021.
  * [9] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Learning a deep convolutional network for image super-resolution.  In ECCV, pages 184–199. Springer, 2014.
  * [10] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [12] Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor.  Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction.  In ICCV, 2019.
  * [13] Tsu-Jui Fu, Xin Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang.  SSCR: Iterative language-based image editing via self-supervised counterfactual reasoning.  In EMNLP, 2020.
  * [14] Tsu-Jui Fu, Xin Wang, Scott Grafton, Miguel Eckstein, and William Yang Wang.  Sscr: Iterative language-based image editing via self-supervised counterfactual reasoning.  In EMNLP, pages 4413–4422, 2020.
  * [15] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In NeurIPS, 2014.
  * [16] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [17] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  Journal of Machine Learning Research, 2022.
  * [18] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [19] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  In ICLR, 2018.
  * [20] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, pages 4401–4410, 2019.
  * [21] E Ak Kenan, Ying Sun, and Joo Hwee Lim.  Learning cross-modal representations for language-based image manipulation.  In 2020 IEEE International Conference on Image Processing (ICIP), 2020.
  * [22] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In CVPR, 2022.
  * [23] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In CVPR, pages 1646–1654, 2016.
  * [24] Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh.  CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication.  In ACL, 2019.
  * [25] Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh.  Codraw: Collaborative drawing as a testbed for grounded goal-driven communication.  In ACL, pages 6495–6513, 2019.
  * [26] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In CVPR, pages 4681–4690, 2017.
  * [27] Daniel Levy, Matthew D. Hoffman, and Jascha Sohl-Dickstein.  Generalizing hamiltonian monte carlo with neural networks.  In ICLR, 2018.
  * [28] Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr.  Manigan: Text-guided image manipulation.  In CVPR, pages 7880–7889, 2020.
  * [29] Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang.  Rethinking image inpainting via a mutual encoder-decoder with feature equalizations.  In ECCV, pages 725–741. Springer, 2020.
  * [30] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In ECCV, pages 423–439. Springer, 2022.
  * [31] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [32] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Srflow: Learning the super-resolution space with normalizing flow.  In ECCV, pages 715–732. Springer, 2020.
  * [33] Eric Luhman and Troy Luhman.  Knowledge distillation in iterative generative models for improved sampling speed.  arXiv preprint arXiv:2101.02388, 2021.
  * [34] Shoya Matsumori, Yuki Abe, Kosuke Shingyouchi, Komei Sugiura, and Michita Imai.  Lattegan: Visually guided language attention for multi-turn text-conditioned image manipulation.  IEEE Access, 2021.
  * [35] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin.  Pulse: Self-supervised photo upsampling via latent space exploration of generative models.  In CVPR, pages 2437–2445, 2020.
  * [36] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In ICLR, 2018.
  * [37] Seonghyeon Nam, Yunji Kim, and Seon Joo Kim.  Text-adaptive generative adversarial networks: manipulating images with natural language.  NeurIPS, 2018.
  * [38] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and Mehran Ebrahimi.  Edgeconnect: Generative image inpainting with adversarial edge learning.  arXiv preprint arXiv:1901.00212, 2019.
  * [39] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In ICML, 2022.
  * [40] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In ICML, 2021.
  * [41] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [42] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In ICML, pages 4055–4064. PMLR, 2018.
  * [43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [44] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In CVPR, pages 2536–2544, 2016.
  * [45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [46] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  OpenAI, 2018.
  * [47] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [48] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li.  Structureflow: Image inpainting via structure-aware appearance flow.  In ICCV, pages 181–190, 2019.
  * [49] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [50] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [51] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
  * [52] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [53] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [54] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In NeurIPS, 2016.
  * [55] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In ICLR, 2022.
  * [56] Tim Salimans, Diederik Kingma, and Max Welling.  Markov chain monte carlo and variational inference: Bridging the gap.  In ICML, pages 1218–1226. PMLR, 2015.
  * [57] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari.  How good is my gan?  In ECCV, pages 213–229, 2018.
  * [58] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In ICML, 2015.
  * [59] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [60] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [61] Jiaming Song, Shengjia Zhao, and Stefano Ermon.  A-nice-mc: Adversarial training for mcmc.  NeurIPS, 30, 2017.
  * [62] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  In NeurIPS, 2019.
  * [63] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In ICLR, 2021.
  * [64] Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C.-C. Jay Kuo.  Spg-net: Segmentation prediction and guidance network for image inpainting.  In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 97. BMVA Press, 2018.
  * [65] Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays.  Composing text and image for image retrieval-an empirical odyssey.  In CVPR, pages 6439–6448, 2019.
  * [66] Hao Wang, Guosheng Lin, Steven CH Hoi, and Chunyan Miao.  Cycle-consistent inverse gan for text-to-image synthesis.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 630–638, 2021.
  * [67] Wei Wang, Ming Yan, and Chen Wu.  Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering.  In ACL, 2018.
  * [68] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy.  Esrgan: Enhanced super-resolution generative adversarial networks.  In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018.
  * [69] Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia.  Image inpainting via generative multi-column convolutional neural networks.  NeurIPS, 2018.
  * [70] Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, and Thomas Huang.  Deep networks for image super-resolution with sparse prior.  In ICCV, pages 370–378, 2015.
  * [71] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Huggingface’s transformers: State-of-the-art natural language processing.  ArXiv, 2019.
  * [72] Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan.  Shift-net: Image inpainting via deep feature rearrangement.  In ECCV, pages 1–17, 2018.
  * [73] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo.  Learning pyramid-context encoder network for high-quality image inpainting.  In CVPR, pages 1486–1494, 2019.
  * [74] Tianhao Zhang, Hung-Yu Tseng, Lu Jiang, Weilong Yang, Honglak Lee, and Irfan Essa.  Text as neural operator: Image manipulation by text instruction.  In Proceedings of the 29th ACM International Conference on Multimedia, pages 1893–1902, 2021.
  * [75] Shizhan Zhu, Raquel Urtasun, Sanja Fidler, Dahua Lin, and Chen Change Loy.  Be your own prada: Fashion synthesis with structural coherence.  In ICCV, pages 1680–1688, 2017.
  * [76] C Lawrence Zitnick and Devi Parikh.  Bringing semantics into focus using visual abstraction.  In CVPR, pages 3009–3016, 2013.

##  Appendix A Demo Cases from CoDraw

Figure 4:  Figure 5:  Figure 6:  Figure 7:

##  Appendix B Demo Cases from i-CLEVR

Figure 8:  Figure 9:  Figure 10:  Figure 11:


# Can Neutrino Self-interactions Save Sterile Neutrino Dark Matter?

Rui An  Department of Physics and Astronomy, University of Southern
California, Los Angeles, CA 90089, USA  Vera Gluscevic  Department of Physics
and Astronomy, University of Southern California, Los Angeles, CA 90089, USA
TAPIR, Mailcode 350-17, California Institute of Technology, Pasadena, CA
91125, USA  Ethan O. Nadler  Carnegie Observatories, 813 Santa Barbara Street,
Pasadena, CA 91101, USA  Department of Physics and Astronomy, University of
Southern California, Los Angeles, CA 90089, USA  Yue Zhang  Department of
Physics, Carleton University, Ottawa, ON K1S 5B6, Canada

###### Abstract

Sterile neutrinos only interact with the Standard Model through the neutrino
sector, and thus represent a simple dark matter (DM) candidate with many
potential astrophysical and cosmological signatures. Recently, sterile
neutrinos produced through self-interactions of active neutrinos have received
attention as a particle candidate that can yield the entire observed DM relic
abundance without violating the most stringent constraints from X-ray
observations. We examine consistency of this production mechanism with the
abundance of small-scale structure in the universe, as captured by the
population of ultra-faint dwarf galaxies orbiting the Milky Way, and derive a
lower bound on the sterile-neutrino particle mass of 37 keV. Combining these
results with previous collider and X-ray limits excludes $100\%$ sterile
neutrino DM produced by strong neutrino self-coupling, mediated by a heavy
($\gtrsim 1~{}\mathrm{GeV}$) scalar; however, data permits sterile-neutrino DM
production via a light mediator.

Particle astrophysics (96); Dark matter (353); Galaxy abundances (574)

##  1 Introduction

A neutrino–like particle featuring feeble interactions with the Standard Model
has long been an appealing candidate for dark matter (DM) (Gunn et al. 1978).
However, the three known active neutrino states $\nu_{a}$, produced from the
thermal bath in the early universe, are too hot to allow formation of DM
structure consistent with observations (e.g., White et al. 1983). A fourth
neutrino $\nu_{4}$, with mass $m_{4}$ in the keV range, was thus proposed as a
combination of an active component and a new _sterile_ neutrino $\nu_{s}$
(with no Standard-Model interactions), such that

$|\nu_{4}\rangle=\cos\theta|\nu_{s}\rangle+\sin\theta|\nu_{a}\rangle,$ (1)

with a small mixing angle $\theta$ (Drewes et al. 2017; Boyarsky et al. 2019).

Dodelson and Widrow (Dodelson & Widrow 1994) have shown that $\nu_{4}$ can be
produced non-thermally with the correct relic abundance to constitute all of
DM. In their proposal, each active neutrino produced through standard
interactions could experience active–sterile oscillations, with a small
probability of conversion to a nearly-sterile mass eigenstate, i.e.,
$\nu_{4}$. Oscillations would continue until weak interaction decoupling at a
temperature of $\sim$1 MeV, at which point the final relic abundance of
$\nu_{4}$ is set; the relic abundance is thus proportional to the strength of
the weak interaction and the mixing angle $\theta$.

The original Dodelson–Widrow (DW) mechanism is, however, in tension with
observational constraints. In particular, the above active-sterile mixing that
leads to sterile-neutrino DM production would also lead to a late-time decay
of sterile to active neutrinos, accompanied by X-ray photon production in
high-density regions of the universe. Non-detection of an X-ray signal in
indirect DM searches excludes all mixing angles necessary to produce the full
DM relic abundance via the original DW mechanism (Boyarsky et al. 2006;
Horiuchi et al. 2014; Perez et al. 2017; Dessert et al. 2020).

New proposals for sterile–neutrino production have emerged (Shi & Fuller 1999;
Shaposhnikov & Tkachev 2006; Asaka et al. 2006; Bezrukov et al. 2010; Nemevsek
et al. 2012; Roland et al. 2015; Dror et al. 2020; Nemevšek & Zhang 2023),
that avoid the radiative decay bounds from X-ray observations. One such model
invokes active neutrino self-interactions (de Gouvêa et al. 2020; Kelly et al.
2020, 2021; Benso et al. 2022), which have mainly been explored indirectly
through the $Z$-boson invisible width (Schael et al. 2006), red and flavor-
dependent experiments (Berryman et al. 2018, 2022; Esteban et al. 2021). A
strong neutrino self-interaction is a viable and simple extension of the
Standard Model, and could occur via an additional mediator $\phi$ below the
weak scale (e.g. Berryman et al. 2022). This would lead to enhanced out-of-
equilibrium production of sterile neutrinos, resulting in sterile-neutrino DM
characterized by a phase space distribution that depends on its particle mass,
the interaction channel, and the mediator mass.

For concreteness, we consider the sterile-neutrino production model proposed
in de Gouvêa et al. (2020), which can populate 100% of DM in the universe
while evading all current constraints. Within this model, a complex scalar
$\phi$ with mass $m_{\phi}$ mediates new interactions between active
neutrinos; here, $G_{\chi}\equiv\lambda_{\phi}^{2}/m_{\phi}^{2}$ is the
counterpart of the Fermi constant, and $\lambda_{\phi}$ is the Yukawa coupling
between $\phi$ and active neutrinos. Following de Gouvêa et al. (2020), we
distinguish three DM production channels (Figure 1): production in the
presence of neutrino self-interactions through a heavy mediator (Case A); or
through an (on-shell) light mediator, with the effective active-sterile mixing
angle close to the vacuum mixing angle (Case C) or with a suppressed in-medium
mixing (Case B). A number of laboratory and astrophysical constraints on the
model were recently explored (Chen et al. 2022; Kelly et al. 2022; Kelly &
Zhang 2019; Berryman et al. 2018).

We confront all three production channels with recent Milky Way (MW) satellite
galaxy observations, which place stringent limits on the DM free-streaming
scale (Schneider 2016; Cherry & Horiuchi 2017; Nadler et al. 2021; Zelko et
al. 2022). For the first time, we perform probabilistic inference of the free-
streaming scale in this class of sterile neutrino DM models using the observed
abundance of MW satellite galaxies. Furthermore, we combine this inference
with the upper bound on the active-sterile neutrino mixing angle from X-ray
observations and the lower bound on the mixing angle from the invisible
Z-boson decay width. It results in a lower bound of 37 keV on the sterile-
neutrino DM mass. Our analysis rules out strong neutrino self-interactions
(Case A) as a mechanism to produce the entire DM relic abundance.

Figure 1: Sterile-neutrino $\nu_{s}$ production channels, in the presence of
active-neutrino self-interactions, in the cases of a heavy (left) or a light
(right) scalar mediator. Oscillation of an active into a sterile neutrino is
denoted with an “x”. Sterile neutrinos are populated via neutrino self-
interactions and mediator decay, respectively, in these two cases.

##  2 Model

We denote sterile-neutrino DM as $|\nu_{4}\rangle$ and assume that it is a
linear combination of active neutrino states $|\nu_{a}\rangle$ and a new
gauge-singlet fermion $|\nu_{s}\rangle$, such that $\theta$ is the vacuum
active–sterile neutrino mixing angle (see Equation 1). We further assume that
active neutrinos feature self-interactions of the form
$\lambda_{\phi}\nu_{a}\nu_{a}\phi$.

The key quantity that controls observables in this model is the primordial
phase space distribution of sterile-neutrinos $f_{\nu_{s}}$, which satisfies
the following Boltzmann equation (de Gouvêa et al. 2020)

$\begin{split}\frac{\mathrm{d}f_{\nu_{s}}}{\mathrm{d}\widetilde{z}}&=\frac{\Gamma}{4H\widetilde{z}}\sin^{2}2\theta_{\mathrm{eff}}f_{\nu_{a}},\\\
\sin^{2}2\theta_{\mathrm{eff}}&\simeq\frac{\Delta^{2}\sin^{2}2\theta}{\Delta^{2}\sin^{2}2\theta+\Gamma^{2}/4+(\Delta\cos
2\theta-V_{T})^{2}}.\end{split}$ (2)

We emphasize that $f_{\nu_{s}}$ needs not be a thermal distribution; however,
for the parameter space we explore, DM is mostly produced before neutrino
decoupling, ensuring that the active neutrinos and photons share the same
temperature, $T_{\gamma}=T_{\nu_{a}}$. In the above equations,
$\widetilde{z}\equiv 1\mathrm{MeV}/T_{\gamma}\propto 1/z$, is inversely
proportional to cosmological redshift $z$, $H$ is the Hubble expansion
parameter, $f_{\nu_{a}}$ is the Fermi-Dirac distribution of active neutrinos,
$\Gamma$ is the interaction rate of active neutrinos in the early universe
that receives contributions from the standard weak interactions and the
$\phi$-mediated self-interactions, and $\theta_{\mathrm{eff}}$ is the reduced
(effective) mixing angle within the plasma that features neutrino self-
interactions. Note that the vacuum oscillation frequency is given by
$\Delta=m_{4}^{2}/(2E)$, where $E$ is the neutrino energy; $V_{T}$ is the
thermal potential experienced by active neutrinos, and $m_{4}$ is the DM mass.

At high temperatures (early times), when $\Gamma,V_{T}\gg\Delta$, DM is not
produced due to collisional damping of the effective mixing angle
$\theta_{\mathrm{eff}}$; in other words, neutrino interactions, which only
involve the active species, “reset the clock” too frequently to allow for an
active neutrino to convert into a sterile neutrino between consecutive
scatterings. Similarly, at low temperatures (late times), when $m_{\phi}\gg
T_{\gamma}$, the production also ceases because all neutrino interactions drop
out of equilibrium, and DM abundances are unchanged. The DM phase space
distribution $f_{\nu_{s}}$ is thus calculated by integrating the above
equations over an intermediate range of redshift, where DM production occurs
most efficiently.

Figure 2: Left panel: Combinations of neutrino self-interaction strength
$\lambda_{\phi}$ and mediator mass $m_{\phi}$ that produce sterile-neutrino DM
at the correct relic abundance, for a DM mass of $m_{4}=40.3~{}\mathrm{keV}$
and vacuum mixing angle of $\sin^{2}2\theta=8.6\times 10^{-13}$. Points in the
three DM production regimes are labeled with A, B, and C, corresponding to the
Cases described in the text. Middle panel: The corresponding DM phase space
distributions. For comparison, we also show the original DW model with no
neutrino interactions as a dotted line, and the thermal-relic WDM model as a
dot-dashed line (evaluated at the $95\%$ confidence $6.5~{}\mathrm{keV}$ WDM
mass bound from Nadler et al. (2021). Right panel: The corresponding
suppression of DM clustering is illustrated as the ratio of the linear matter
power spectrum in sterile-neutrino DM models to that of CDM. The vertical line
indicates the half-mode scale $k_{\mathrm{hm}}$ of the ruled-out WDM model.

In the left panel of Figure 2, the $\mathcal{S}$-shaped gray curve illustrates
the relationship between the strength of the neutrino self-interactions and
the mediator mass, for models that yield the entire DM relic abundance, at a
fixed DM particle mass of $m_{4}=40.3~{}\mathrm{keV}$ and a fixed mixing angle
of $\sin^{2}2\theta=8.6\times 10^{-13}$. The middle panel of Figure 2 shows
the DM phase space distributions corresponding to the three DM production
cases: in Case A (i.e., points with $\lambda_{\phi}\approx 1$ and
$m_{\phi}\gtrsim 1~{}\mathrm{GeV}$, to the right of the right-most “knee” in
the $\mathcal{S}$-shaped curve), DM is produced through scattering of active
neutrinos; in Cases B and C (i.e., points with $\lambda_{\phi}\ll 1$ and
$m_{\phi}\lesssim 1~{}\mathrm{GeV}$ that lie along the roughly vertical
segment and along the left-most portion of the $\mathcal{S}$-shaped curve,
respectively), DM is produced through the decay of on-shell $\phi$ (de Gouvêa
et al. 2020).

Neutrino interactions affect the phase-space distribution differently in the
three cases, which generally results in different free-streaming lengths. The
specifics of the free-streaming in individual models lead to notable
differences in DM clustering on small physical scales; the resulting
suppression of the linear matter power spectrum, relative to cold DM (CDM)
cosmology is illustrated in the right panel of Figure 2. Below, we use this
suppression of small-scale power to assess the viability of self-interacting
sterile-neutrino production regimes.

##  3 Analysis

To confront the sterile neutrino model with recent small-scale structure
constraints, we use the latest public version of the Boltzmann solver CLASS
(Lesgourgues 2011; Blas et al. 2011) to generate the linear matter power
spectrum $P(k)$ consistent with a given sterile-neutrino cosmology (where $k$
is the cosmological wavenumber), and compare it to standard CDM cosmology. For
standard cosmological parameters, we use the values consistent with Nadler et
al. (2021) (see also Maamari et al. 2021): Hubble constant $h=0.6932$,
present-day DM density parameter $\Omega_{\mathrm{dm}}h^{2}=0.1153$, present-
day baryon density parameter $\Omega_{\mathrm{b}}h^{2}=0.02223$, amplitude of
the scalar perturbations $A_{s}=2.464\times 10^{-9}$, scalar spectral index
$n_{s}=0.9608$, and optical depth to reionization
$\tau_{\mathrm{reio}}=0.081$. To account for the effects of sterile-neutrino
free-streaming on $P(k)$, we provide the correct initial phase-space
distribution (when DM is still relativistic) as an input to CLASS, for each
model parameter combination $\\{m_{4},\theta,\lambda_{\phi},m_{\phi}\\}$. We
sample over pairs of DM mass and mixing angle $\\{m_{4},\theta\\}$, and choose
the interaction strength and the mediator mass $\\{\lambda_{\phi},m_{\phi}\\}$
that yield a DM relic density consistent with Planck.

The key output from CLASS used in our analysis is the ratio of the linear
power spectrum for sterile-neutrino DM and CDM cosmologies,
$P(k)/P_{\mathrm{CDM}}(k)$. This ratio is set at early times and evolved non-
linearly as in CDM cosmology; thus, the effects of sterile-neutrino DM on
structure formation amount to a change in the initial conditions, shown in the
right panel of Figure 2. From this figure, we can see that all three
production cases resemble CDM at large scales, $k\lesssim
10~{}h/\mathrm{Mpc}$, and therefore successfully reproduce large-scale
structure observations. On small scales, power is strongly suppressed due to
particle free streaming, leading to fewer low-mass DM halos compared to CDM.

We use the population of satellite galaxies orbiting the MW, as observed by
the Dark Energy Survey (DES; Abbott et al. 2018) and Pan-STARRS1 (PS1;
Chambers et al. 2016); these galaxies inhabit DM halos with masses down to
$\sim 10^{8}~{}M_{\mathrm{\odot}}$ (Jethwa et al. 2018; Nadler et al. 2020),
which grew from modes $k\gtrsim 30~{}h/\mathrm{Mpc}$ (Nadler et al. 2019a).
Converting these observations into constraints on any new physics that affects
the linear matter power spectrum requires forward-modeling of the satellite
population starting from $P(k)$. Previous studies performed high-resolution
zoom-in simulations of MW-like halos using $P(k)$ for thermal-relic WDM as an
initial condition, thus predicting subhalo and galaxy populations in
cosmologies with power suppression (Schneider et al. 2012; Lovell et al. 2014;
Bose et al. 2017; Nadler et al. 2021; Lovell et al. 2014); these results were
used to bound the thermal-relic WDM particle mass (Drlica-Wagner et al. 2020;
Nadler et al. 2020, 2021).

To obtain sterile-neutrino bounds, we map linear power spectra for our model,
illustrated in the right panel of Figure 2, to the thermal-relic WDM
constraint from MW satellite galaxies of $m_{\mathrm{WDM}}>6.5~{}\mathrm{keV}$
at $95\%$ confidence (Nadler et al. 2021). We parameterize the $P(k)$ cutoff
for each evaluation of our model using the half-mode scale $k_{\rm{hm}}$,
defined as the wavenumber where $P(k)$ drops to 25% of its CDM value, and
match to a thermal-relic WDM model with the same $k_{\rm{hm}}$. This procedure
is accurate because the shape of $P(k)$ in our model is extremely similar to
that in WDM (Figure 2, right panel). We then translate the WDM posterior
distribution from the MW satellite analysis in Nadler et al. (2021) into a
marginalized posterior probability distribution on $m_{4}$, $\sin^{2}2\theta$,
and $\lambda_{\phi}$; the remaining degree of freedom, $m_{\phi}$, is fixed by
the DM relic abundance constraint. The results we leverage here conservatively
marginalize over uncertainties in the MW host halo mass and the galaxy–halo
connection, and carefully account for the recent merger history of the MW and
observational biases of the surveys we consider (see the Appendix and Nadler
et al. (2021) for details).

Figure 3:  Neutrino self-interaction parameters that can populate sterile
neutrinos with the entire DM relic abundance, for several combinations of
$m_{4}$ and $\sin^{2}2\theta$ (gray curves). The dark green hatched region
shows the constraint from invisible $Z$ width; light green shaded regions show
experimental flavor-dependent constraints (Berryman et al. 2018, 2022; Esteban
et al. 2021). Diagonal black lines divide this parameter space into production
regimes corresponding to Cases A, B, and C.

Figure 4:  _Left panel_ : 95% confidence exclusion for sterile neutrino DM
from our analysis of the MW satellite galaxy population combined with particle
collider bounds (colored shaded regions) is compared with X-ray annihilation
bounds (light gray; Watson et al. 2012; Horiuchi et al. 2014; Bulbul et al.
2014; Boyarsky et al. 2014; Perez et al. 2017; Ng et al. 2019; Dessert et al.
2020) and the bounds from dwarf galaxy phase-space densities (dark gray; Alvey
et al. 2021). Galaxy–halo connection and neutrino self-interaction parameters
($m_{\phi}$ and $\lambda_{\phi}$) are marginalized over. The blue–green region
with solid edges corresponds to the case where we only consider the limits
from invisible Z-decays on $\lambda_{\phi}$. The dashed, dash-dot or dotted
lines correspond to bounds on specific mediator flavor-coupling scenarios. DW
production mechanism corresponds to the solid black line. Middle and right
panels: Our sterile neutrino DM constraints divided according to the
production scenario. Case A (red) is the heavy-mediator scenario ruled out by
our analysis. Case B (yellow) and Case C (blue) correspond to light-mediator
scenarios.

We combine the resulting lower limit on sterile-neutrino DM mass with existing
constraints on the neutrino self-interaction strength from experiments
involving charged $K,\pi$ meson decays, neutrinoless double-beta decay, the
$Z$-boson invisible width (Berryman et al. 2018, 2022), and the astrophysical
neutrino observations from the IceCube experiment (Esteban et al. 2021). To
avoid impacting Big Bang Nucleosynthesis, we only focus on mediator masses
above $1\mathrm{MeV}$ scale (Escudero & Witte 2020; Pitrou et al. 2018). The
regions in the $\lambda_{\phi}$–$m_{\phi}$ plane excluded by these experiments
are shaded green in Figure 3; note that some of the constraints are neutrino-
flavor dependent, as labeled. The $\mathcal{S}$-shaped gray curves in Figure 3
illustrate the loci of the correct DM relic abundance, for several choices of
$m_{4}$ and $\sin^{2}2\theta$. We label the three distinctive production
regimes (de Gouvêa et al. 2020), which are roughly divided by the diagonal
black lines. These correspond to a heavy and strongly-coupled mediator (Case
A), light and feebly-coupled mediator (Case C), and an intermediate regime
(Case B), from top to bottom.

##  4 Results

One of our key results is a lower bound on the mass of sterile-neutrino DM,
derived mainly from DES and PS1 MW satellite observations, in the presence of
neutrino self-interactions. By marginalizing over all other model parameters
(see the Appendix for more details), we obtain a $95\%$ confidence limit of

$m_{4}>37~{}\mathrm{keV},$ (3)

regardless of the production mechanism.

The left panel of Figure 4 shows experimental and observational constraints in
the $\sin^{2}2\theta$–$m_{4}$ plane, where all other parameters are
marginalized over; the blue–green shaded regions are ruled out by our combined
analysis, while the gray shaded regions are excluded by X-ray observations
(Watson et al. 2012; Horiuchi et al. 2014; Bulbul et al. 2014; Boyarsky et al.
2014; Perez et al. 2017; Ng et al. 2019; Dessert et al. 2020; also see Roach
et al. 2020; Foster et al. 2021; Sicilian et al. 2022) and DM phase-space
density measurements in dwarf galaxies (Alvey et al. 2021). Neutrino self-
interactions allow the DM relic abundance to be saturated at lower mixing
angles than the original DW production mechanism.

The vertical part of each parameter boundary (i.e., the lower bound on
$m_{4}$) is driven by the observed MW satellite abundance, whereas the
horizontal part (i.e., the lower bound on $\theta$) is dominated by the
experimental bounds from Figure 3. In the left panel, we allow all three
sterile-neutrino production channels by marginalizing the posterior over all
mediator masses and coupling strengths; we also show the impact of the three
flavor-specific experimental bounds on neutrino coupling with $\phi$, for
electron ($\nu_{e}$), muon ($\nu_{\mu}$), and tau ($\nu_{\tau}$) neutrinos as
the dashed, dot-dashed, and dotted boundaries, respectively. We find that the
coupling to $\nu_{e}$ most strongly constrains the mixing angle for the
parameter space that allows the standard neutrino to constitute all of DM.

In the middle and right panels of Figure 4, we show the combined observational
and experimental constraints in the same parameter space, but this time
showing the individual production channels, A, B, and C, separately, as
labeled. This allows us to assess the validity of individual production
channels. In these two panels, we only incorporate the experimental bounds on
the invisible $Z$ width, which is flavor-independent and thus conservative. We
find that the parameter space for Case A—sterile-neutrino DM production in the
presence of a heavy mediator with $m_{\phi}\gtrsim\,$GeV and a large coupling,
$\lambda_{\phi}\gtrsim\mathcal{O}(0.1)$—is entirely ruled out. In contrast,
Cases B and C allow for smaller $\lambda_{\phi}$, where the corresponding
$Z$-decay constraints are much weaker. A small portion of this parameter space
(i.e., the white triangle-shaped region in the right panel of Figure 4)
remains unconstrained for DM masses from 37 keV up to several MeV, depending
on the other interaction parameters.

##  5 Discussion and Conclusions.

Our analysis conservatively marginalizes over astrophysical uncertainties that
govern the connection between MW satellite galaxies and DM halos,
observational uncertainties related to the known population of MW satellite
galaxies, and the properties of the MW system, and accounts for the specific
phase-space distribution of sterile-neutrino DM produced by active neutrino
self-interactions. The fact that we consider active neutrino self-interactions
is crucial because the physics we probe is also directly tested by particle
collider experiments, unlike many other sterile neutrino DM scenarios (Hansen
& Vogl 2017; Bringmann et al. 2023; Johns & Fuller 2019). Our results rule out
sterile-neutrino DM produced through neutrino interactions mediated by a heavy
scalar particle, at $\gg 95\%$ confidence, and constrains sterile-neutrino DM
masses between 37 keV to several MeV, regardless of the other production
parameters. These are the first bounds derived from a combined analysis of
structure formation and collider data and represent the most stringent
sterile-neutrino DM constraints to date.

Importantly, unlike previous small-scale structure constraints on other
sterile neutrino production mechanisms (e.g., Zelko et al. 2022; Schneider
2016), we do not simply construct a one-dimensional mapping to thermal-relic
WDM limits; rather, we cast our self-interacting sterile neutrino DM model
into the full posterior distribution from the MW satellite analysis in Nadler
et al. (2021), which allows us to carefully examine degeneracies among sterile
neutrino DM parameters and with astrophysical parameters governing the
galaxy–halo connection. Furthermore, we extract information for specific
physical scenarios from the posterior distribution and thus differentiate the
mediator flavor-coupling and production regimes shown in the panels of Figure
4.

Our study informs the interpretation of astrophysical and particle accelerator
searches for neutrino self-interactions and sterile neutrino DM. For example,
if a heavy, neutrinophilic scalar is discovered in future accelerator neutrino
experiments, following recent proposals (e.g., Kelly & Zhang 2019; Kelly et
al. 2022), our results imply that such a scalar _cannot_ produce sterile
neutrinos that saturate the DM relic abundance. Meanwhile, if future
astrophysical searches discover an indirect detection signal consistent with
sterile neutrino DM that possesses a small mixing angle, then light neutrino
self-interaction mediators should be targeted by terrestrial experiments as a
potential “smoking gun” of the DM production mechanism.

We thank Alex Drlica-Wagner for comments on the manuscript. VG and RA
acknowledge the support from NASA through the Astrophysics Theory Program,
Award Number 21-ATP21-0135. VG also acknowledges support from the National
Science Foundation under Grant No. PHY-2013951. YZ is supported by the Arthur
B. McDonald Canadian Astroparticle Physics Research Institute.

## References

  * Abbott et al. (2018) Abbott, T. M. C., Abdalla, F. B., Allam, S., et al. 2018, ApJS, 239, 18
  * Alvey et al. (2021) Alvey, J., Sabti, N., Tiki, V., et al. 2021, MNRAS, 501, 1188
  * Asaka et al. (2006) Asaka, T., Shaposhnikov, M., & Kusenko, A. 2006, Phys. Rev. B, 638, 401
  * Benso et al. (2022) Benso, C., Rodejohann, W., Sen, M., & Ramachandran, A. U. 2022, Phys. Rev. D, 105, 055016
  * Berryman et al. (2018) Berryman, J. M., de Gouvêa, A., Kelly, K. J., & Zhang, Y. 2018, Phys. Rev. D, 97, 075030
  * Berryman et al. (2022) Berryman, J. M. et al. 2022, in 2022 Snowmass Summer Study
  * Bezrukov et al. (2010) Bezrukov, F., Hettmansperger, H., & Lindner, M. 2010, Phys. Rev. D, 81, 085032
  * Blas et al. (2011) Blas, D., Lesgourgues, J., & Tram, T. 2011, J. Cosmology Astropart. Phys, 2011, 034
  * Bose et al. (2017) Bose, S., Hellwing, W. A., Frenk, C. S., et al. 2017, MNRAS, 464, 4520
  * Boyarsky et al. (2019) Boyarsky, A., Drewes, M., Lasserre, T., Mertens, S., & Ruchayskiy, O. 2019, Prog. Part. Nucl. Phys., 104, 1
  * Boyarsky et al. (2006) Boyarsky, A., Neronov, A., Ruchayskiy, O., & Shaposhnikov, M. 2006, MNRAS, 370, 213
  * Boyarsky et al. (2014) Boyarsky, A., Ruchayskiy, O., Iakubovskyi, D., & Franse, J. 2014, Phys. Rev. Lett., 113, 251301
  * Bringmann et al. (2023) Bringmann, T., Depta, P. F., Hufnagel, M., et al. 2023, Phys. Rev. D, 107, L071702
  * Bulbul et al. (2014) Bulbul, E., Markevitch, M., Foster, A., et al. 2014, ApJ, 789, 13
  * Chambers et al. (2016) Chambers, K. C., Magnier, E. A., Metcalfe, N., et al. 2016, arXiv e-prints, arXiv:1612.05560
  * Chen et al. (2022) Chen, Y.-M., Sen, M., Tangarife, W., Tuckler, D., & Zhang, Y. 2022, JCAP, 11, 014
  * Cherry & Horiuchi (2017) Cherry, J. F. & Horiuchi, S. 2017, Phys. Rev. D, 95, 083015
  * de Gouvêa et al. (2020) de Gouvêa, A., Sen, M., Tangarife, W., & Zhang, Y. 2020, Phys. Rev. Lett., 124, 081802
  * Dessert et al. (2020) Dessert, C., Rodd, N. L., & Safdi, B. R. 2020, Science, 367, 1465
  * Dodelson & Widrow (1994) Dodelson, S. & Widrow, L. M. 1994, Phys. Rev. Lett., 72, 17
  * Drewes et al. (2017) Drewes, M. et al. 2017, JCAP, 01, 025
  * Drlica-Wagner et al. (2020) Drlica-Wagner, A. et al. 2020, ApJ, 893, 1
  * Dror et al. (2020) Dror, J. A., Dunsky, D., Hall, L. J., & Harigaya, K. 2020, JHEP, 07, 168
  * Escudero & Witte (2020) Escudero, M. & Witte, S. J. 2020, European Physical Journal C, 80, 294
  * Esteban et al. (2021) Esteban, I., Pandey, S., Brdar, V., & Beacom, J. F. 2021, Phys. Rev. D, 104, 123014
  * Foster et al. (2021) Foster, J. W., Kongsore, M., Dessert, C., et al. 2021, Phys. Rev. Lett., 127, 051101
  * Garrison-Kimmel et al. (2017) Garrison-Kimmel, S. et al. 2017, MNRAS, 471, 1709
  * Gunn et al. (1978) Gunn, J. E., Lee, B. W., Lerche, I., Schramm, D. N., & Steigman, G. 1978, ApJ, 223, 1015
  * Hansen & Vogl (2017) Hansen, R. S. L. & Vogl, S. 2017, Phys. Rev. Lett., 119, 251305
  * Horiuchi et al. (2014) Horiuchi, S., Humphrey, P. J., Oñorbe, J., et al. 2014, Phys. Rev. D, 89, 025017
  * Horiuchi et al. (2014) Horiuchi, S., Humphrey, P. J., Onorbe, J., et al. 2014, Phys. Rev. D, 89, 025017
  * Jethwa et al. (2018) Jethwa, P., Erkal, D., & Belokurov, V. 2018, MNRAS, 473, 2060
  * Johns & Fuller (2019) Johns, L. & Fuller, G. M. 2019, Phys. Rev. D, 100, 023533
  * Kelly et al. (2022) Kelly, K. J., Kling, F., Tuckler, D., & Zhang, Y. 2022, Phys. Rev. D, 105, 075026
  * Kelly et al. (2020) Kelly, K. J., Sen, M., Tangarife, W., & Zhang, Y. 2020, Phys. Rev. D, 101, 115031
  * Kelly et al. (2021) Kelly, K. J., Sen, M., & Zhang, Y. 2021, Phys. Rev. Lett., 127, 041101
  * Kelly & Zhang (2019) Kelly, K. J. & Zhang, Y. 2019, Phys. Rev. D, 99, 055034
  * Lesgourgues (2011) Lesgourgues, J. 2011, arXiv e-prints, arXiv:1104.2932
  * Lovell et al. (2014) Lovell, M. R., Frenk, C. S., Eke, V. R., et al. 2014, MNRAS, 439, 300
  * Maamari et al. (2021) Maamari, K., Gluscevic, V., Boddy, K. K., Nadler, E. O., & Wechsler, R. H. 2021, ApJ, 907, L46
  * Nadler et al. (2019a) Nadler, E. O., Gluscevic, V., Boddy, K. K., & Wechsler, R. H. 2019a, ApJ, 878, 32
  * Nadler et al. (2019b) Nadler, E. O., Mao, Y.-Y., Green, G. M., & Wechsler, R. H. 2019b, ApJ, 873, 34
  * Nadler et al. (2018) Nadler, E. O., Mao, Y.-Y., Wechsler, R. H., Garrison-Kimmel, S., & Wetzel, A. 2018, ApJ, 859, 129
  * Nadler et al. (2020) Nadler, E. O. et al. 2020, ApJ, 893, 48
  * Nadler et al. (2021) Nadler, E. O. et al. 2021, Phys. Rev. Lett., 126, 091101
  * Nemevsek et al. (2012) Nemevsek, M., Senjanovic, G., & Zhang, Y. 2012, JCAP, 07, 006
  * Nemevšek & Zhang (2023) Nemevšek, M. & Zhang, Y. 2023, Phys. Rev. Lett., 130, 121002
  * Ng et al. (2019) Ng, K. C. Y., Roach, B. M., Perez, K., et al. 2019, Phys. Rev. D, 99, 083005
  * Perez et al. (2017) Perez, K., Ng, K. C. Y., Beacom, J. F., et al. 2017, Phys. Rev. D, 95, 123002
  * Perez et al. (2017) Perez, K., Ng, K. C. Y., Beacom, J. F., et al. 2017, Phys. Rev. D, 95, 123002
  * Pitrou et al. (2018) Pitrou, C., Coc, A., Uzan, J.-P., & Vangioni, E. 2018, Phys. Rep., 754, 1
  * Roach et al. (2020) Roach, B. M., Ng, K. C. Y., Perez, K., et al. 2020, Phys. Rev. D, 101, 103011
  * Roland et al. (2015) Roland, S. B., Shakya, B., & Wells, J. D. 2015, Phys. Rev. D, 92, 113009
  * Schael et al. (2006) Schael, S. et al. 2006, Phys. Rept., 427, 257
  * Schneider (2016) Schneider, A. 2016, JCAP, 04, 059
  * Schneider et al. (2012) Schneider, A., Smith, R. E., Maccio, A. V., & Moore, B. 2012, MNRAS, 424, 684
  * Shaposhnikov & Tkachev (2006) Shaposhnikov, M. & Tkachev, I. 2006, Phys. Rev. B, 639, 414
  * Shi & Fuller (1999) Shi, X.-D. & Fuller, G. M. 1999, Phys. Rev. Lett., 82, 2832
  * Sicilian et al. (2022) Sicilian, D., Lopez, D., Moscetti, M., Bulbul, E., & Cappelluti, N. 2022, ApJ, 941, 2
  * Watson et al. (2012) Watson, C. R., Li, Z., & Polley, N. K. 2012, J. Cosmology Astropart. Phys, 2012, 018
  * White et al. (1983) White, S. D. M., Frenk, C. S., & Davis, M. 1983, ApJ, 274, L1
  * Zelko et al. (2022) Zelko, I. A., Treu, T., Abazajian, K. N., et al. 2022, Phys. Rev. Lett., 129, 191301

Here, we present and discuss the posterior distribution resulting from our
mapping of the self-interacting sterile neutrino DM model to a thermal-relic
WDM model that was constrained by the MW satellite population in Nadler et al.
(2021). Figure 5 shows the posterior distribution over the eight galaxy–halo
connection parameters, which are described in Nadler et al. (2021) and
summarized in Nadler et al. (2020), and our three self-interacting sterile
neutrino DM parameters. The degeneracies between sterile neutrino DM
parameters were discussed in the main text; here, we focus on degeneracies
between sterile neutrino DM and the following galaxy–halo connection
parameters:

  1. 1.

$\alpha$, the faint-end slope of the galaxy luminosity function. More negative
values of $\alpha$ correspond to steeper luminosity functions, resulting in a
larger number of predicted dwarf galaxies and thus a shallower stellar
mass–halo mass relation in the context of abundance matching;

  2. 2.

$\sigma_{M}$, the scatter in satellite galaxy luminosity at fixed subhalo peak
maximum circular velocity. Larger values of $\sigma_{M}$ cause more faint
galaxies to up-scatter to observable luminosities and thus decreases the
masses of the smallest halos inferred to host observed MW satellite galaxies;

  3. 3.

$\mathcal{M}_{50}$, the peak virial halo mass at which $50\%$ of halos host
potentially observable satellite galaxies. In Nadler et al. (2021), the galaxy
occupation fraction is assumed to be a monotonic function of peak halo mass;
thus, smaller values of $\mathcal{M}_{50}$ combined with small values of
$\sigma_{M}$ ensure that the most massive subhalos host observable MW
satellite galaxies (note that $\mathcal{M}_{50}$ and $\sigma_{M}$ are
significantly anti-correlated in the Figure 5 posterior);

  4. 4.

$\mathcal{B}$, the efficiency of subhalo disruption due to the MW disk,
defined relative to predictions from hydrodynamic simulations (Garrison-Kimmel
et al. 2017), such that $\mathcal{B}=1$ corresponds to hydrodynamic
predictions and smaller (larger) $\mathcal{B}$ corresponds to less (more)
efficient disruption (Nadler et al. 2018, 2019b). Variations in $\mathcal{B}$
mostly affect the radial distribution and overall abundance of MW subhalos,
rather than altering the subhalo population in a mass-dependent fashion;

  5. 5.

$\sigma_{\mathrm{gal}}$, the width of the galaxy occupation fraction. At fixed
$\mathcal{M}_{50}$, increasing $\sigma_{\mathrm{gal}}$ results in a more
gradual transition to galaxy-less halos, and thus allows lower-mass halos to
contribute more significantly to the observed MW satellite population;

  6. 6.

$\mathcal{A}$, the amplitude of the relation between galaxy size
(specifically, azimuthally averaged, projected half-light radius) and halo
size (specifically, subhalo virial radius at the time of accretion onto the
MW). For $\mathcal{A}\gtrsim 30~{}\mathrm{pc}$, increasing this parameter
decreases the surface brightnesses of predicted satellites (at fixed
luminosity), which makes bright satellites more difficult to detect—because
they are generally closer to the surface brightness detectability boundary
than dim satellites (Drlica-Wagner et al. 2020)—and forcing lower-mass halos
to host observed systems. However, for sufficiently small values of
$\mathcal{A}$, dim satellites are pushed below the size that Nadler et al.
(2020, 2021) assume to distinguish dwarf galaxies from star clusters (i.e.,
half-light radii of $10~{}\mathrm{pc}$), and thus forces more high-mass halos
to contribute to the observed satellite population in the inference;

  7. 7.

$\sigma_{\log R}$, the scatter in satellite size at fixed halo size.
Increasing $\sigma_{\log R}$ causes more small satellites to up-scatter to
non-observable surface brightnesses, thus increasing the average mass of halos
inferred to host MW satellites;

  8. 8.

$n$, the power-law index of the galaxy–halo size relation. Increasing $n$
yields a steeper size relation, such that low-mass halos host satellites
decrease in size relative to high-mass halos. This shifts the halo population
predicted to host observed MW satellites towards lower masses, because smaller
satellites are generally easier to detect.

Figure 5: Posterior distribution from the thermal-relic WDM fit to DES and PS1
MW satellites, presented in Nadler et al. (2021), cast into the parameter
space of our self-interacting sterile neutrino DM model (i.e., $m_{4}$,
$\sin^{2}2\theta$, and $\lambda_{\phi}$). Dark (light) shaded contours
represent 68% (95%) confidence intervals, respectively. Marginalized
posteriors are shown in the top panel of each column. Note that $\sigma_{M}$,
$\sigma_{\rm{gal}}$, and $\sigma_{\log R}$ are reported in $\rm{dex}$,
$\mathcal{M}_{50}$ is reported as
$\log_{10}(\mathcal{M}_{50}/\mathcal{M}_{\odot})$, $m_{4}$ is reported as
$\log_{10}(m_{4}/\mathrm{keV})$, $\mathcal{A}$ is reported in $\rm{pc}$, and
$\alpha$, $\mathcal{B}$, $n$, $\sin^{2}2\theta$, and $\lambda_{\phi}$ are
dimensionless. For details on the prior distributions assumed for galaxy–halo
connection parameters when forward-modeling the MW satellite population, see
Nadler et al. (2021).

Based on these effects, the degeneracies between sterile neutrino DM and
galaxy–halo connection parameters are readily understood. In particular, any
change to a galaxy–halo connection parameter that _decreases_ the average mass
of halos inferred to host observed MW satellite galaxies results in a more
stringent lower limit on $m_{4}$, because smaller values of $m_{4}$
preferentially suppress the abundance of low-mass halos, which are required to
host observable galaxies in these regions of parameter space. Conversely, any
change to a galaxy–halo connection parameter that _increases_ the average mass
of halos inferred to host observed MW satellite galaxies allows for smaller
values of $m_{4}$, because larger values of $m_{4}$ decrease the abundance of
low-mass halos and thus do not manifest in observable changes to the predicted
satellite population in these regions of parameter space. This reasoning
explains all of the most prominent degeneracies between sterile neutrino DM
and galaxy–halo connection parameters in Figure 5; for example, $m_{4}$ is
noticeably (and positively) correlated with $\sigma_{M}$, $\mathcal{M}_{50}$,
and $\sigma_{\mathrm{gal}}$ because increasing any of these parameters
decreases the average mass of halos that host observed MW satellites based on
the parameter descriptions above.

Because the free-streaming scale is largely set by $m_{4}$ in our self-
interacting sterile neutrino DM model, degeneracies between the remaining
sterile neutrino DM and galaxy–halo connection parameters are much weaker.
Note that $m_{4}$ is anti-correlated with both $\sin^{2}2\theta$ and
$\lambda_{\phi}$ in the posterior because self-interactions enable more
frequent active neutrino scattering, allowing the DM relic density to be
saturated at smaller mixing angles. Thus, these parameters all exhibit similar
degeneracies with the galaxy–halo connection parameters.