**footnotetext: Corresponding author.

# MLCM: Multistep Consistency Distillation of
Latent Diffusion Model

Qingsong Xie11footnotemark: 1
OPPO AI Center
<EMAIL_ADDRESS>
&Zhenyi Liao
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Chen Chen
OPPO AI Center
<EMAIL_ADDRESS>
Zhijie Deng11footnotemark: 1
Shanghai Jiao Tong University
<EMAIL_ADDRESS>
&Shixiang Tang
Chinese University of Hong Kong
<EMAIL_ADDRESS>
&Haonan Lu
OPPO AI Center
<EMAIL_ADDRESS>
###### Abstract

Distilling large latent diffusion models (LDMs) into ones that are fast to
sample from is attracting growing research interest. However, the majority of
existing methods face a dilemma where they either (i) depend on multiple
individual distilled models for different sampling budgets, or (ii) sacrifice
generation quality with limited (e.g., 2-4) and/or moderate (e.g., 5-8)
sampling steps. To address these, we extend the recent multistep consistency
distillation (MCD) strategy to representative LDMs, establishing the Multistep
Latent Consistency Models (MLCMs) approach for low-cost high-quality image
synthesis. MLCM serves as a unified model for various sampling steps due to
the promise of MCD. We further augment MCD with a progressive training
strategy to strengthen inter-segment consistency to boost the quality of few-
step generations. We take the states from the sampling trajectories of the
teacher model as training data for MLCMs to lift the requirements for high-
quality training datasets and to bridge the gap between the training and
inference of the distilled model. MLCM is compatible with preference learning
strategies for further improvement of visual quality and aesthetic appeal.
Empirically, MLCM can generate high-quality, delightful images with only 2-8
sampling steps. On the MSCOCO-2017 5K benchmark, MLCM distilled from SDXL gets
a CLIP Score of 33.30, Aesthetic Score of 6.19, and Image Reward of 1.20 with
only 4 steps, substantially surpassing 4-step LCM luo2023latent , 8-step SDXL-
Lightning lin2024sdxl , and 8-step HyperSD ren2024hyper . We also demonstrate
the versatility of MLCMs in applications including controllable generation,
image style transfer, and Chinese-to-image generation.

Figure 1: $1024\times 1024$ image samples from MLCM, distilled from SDXL-
base-1.0 podell2023sdxl  based on LoRA hu2021lora . From top to bottom, 2, 3,
and 4 sampling steps are adopted, respectively. Apart from such good visual
quality, MLCM can also yield improved metrics compared to strong baselines.

##  1 Introduction

Diffusion models (DMs) have made great advancements in the field of generative
modeling, becoming the go-to approach for image, video, and audio generation
ho2020denoising ; kong2020diffwave ; saharia2022photorealistic . Latent
diffusion models (LDMs) further enhance DMs by operating in the latent image
space, pushing the limit of high-resolution image and video synthesis
ma2024latte ; peebles2023scalable ; podell2023sdxl ; rombach2022high . Despite
the high-quality and realistic samples, LDMs suffer from frustratingly slow
inference–generating a single sample requires multiple (from tens to hundreds)
evaluations of the model, giving rise to a high cost and bad user experience.

There is growing interest in distilling large-scale LDMs into more efficient
ones. Concretely, progressive distillation lin2024sdxl ; meng2023distillation
; salimans2022progressive  reduces the sampling steps by half in each turn but
finally hinges on a set of models for various sampling steps. InstaFlow
liu2023instaflow , UFO-Gen xu2023ufogen , and ADD sauer2024fast ;
sauer2023adversarial  target one-step generation, yet losing or weakening the
ability to benefit from more (e.g., $>4$) sampling steps. Latent consistency
models (LCMs) luo2023latent  apply consistency distillation
song2023consistency  on LDMs’ reverse-time ordinary differential equation
(ODE) trajectories to conjoin one- and multi-step generation, but the image
quality degrades substantially, especially in 1-4 steps. HyperSD ren2024hyper
applies consistency trajectory distillation kim2023consistency  in segments of
the ODE trajectory, but its performance gain is marginal as sampling steps
increase and it requires numerous high-quality text-image pairs for training.

To address these, we extend the multistep consistency distillation (MCD)
heek2024multistep  strategy to representative LDMs, forming Multistep Latent
Consistency Models (MLCMs) for low-cost high-quality image synthesis.
Technically, we divide the latent-space ODE trajectory into multiple segments
and enforce consistency within each. MLCM serves as a unified model for
various sampling steps without suffering from accumulated errors as in LCM. To
further improve the quality of few-step generations, we enhance MLCMs with a
progressive training strategy for inter-segment consistency.

Considering the discrepancy between the training and inference of typical LDMs
like SDXL podell2023sdxl , where the final state of the forward process is not
a pure Gaussian noise sauer2023adversarial , we suggest leveraging the states
of the denoising process of the teacher model as MLCMs’ training data. Apart
from facilitating the alignment between the training and inference, this also
helps alleviate the need for high-quality training datasets as in HyperSD.
Besides, reward scorers, e.g., Image Rerward (IR) xu2024imagereward  and
aesthetic score (AS) AP , can be easily incorporated into MLCMs to improve the
human preference of the outcomes, thanks to the few-step generation capacity
of MLCMs.

We have performed comprehensive empirical studies to inspect the behavior of
MLCMs. We first empirically evaluate the image quality on the MSCOCO-2017 5K
benchmark. We observe that MLCM distilled from SDXL podell2023sdxl  gets 6.19
AS, 1.20 IR, 33.30 CLIP Score (CS) hessel2021clipscore , and 0.233 Pick Score
(PS) kirstain2024pick  with only 4 steps, substantially surpassing 4-step LCM,
8-step SDXL-Lightning lin2024sdxl , and 8-step HyperSD. Additionally, the
performance of MLCMs consistently improves with additional sampling steps. We
also demonstrate the versatility of MLCMs in applications including
controllable generation, image stylization, and Chinese-to-image generation.

We summarize our contributions as follows:

  * •

We propose MLCMs, which accelerate LDMs to generate high-quality outputs
within $2-8$ steps while maintaining the capability for further improvement
with additional steps.

  * •

We improve MLCMs with a progressive training strategy for inter-segment
consistency, render MLCMs image-free during training by exploiting the
denoising process of the teacher model, and incorporate preference learning to
boost rare-step generation quality.

  * •

MLCM achieves a state-of-the-art CS of 33.30, AS of 6.19, IR of 1.20 in 4
steps, surpassing competing baselines, such as 4-step LCM, 8-step SDXL-
Lightning, and 8-step HyperSD.

  * •

MLCMs’ versatility extends to scenarios such as controllable generation, image
stylization, and Chinese-to-image generation, paving the path for extensive
practical applications.

##  2 Preliminary

###  2.1 Diffusion Models

Diffusion models (DMs) ho2020denoising ; sohl2015deep ; song2020score  are
specified by a predefined forward process that progressively distorts the
clean data $x_{0}$ into a pure Gaussian noise with a Gaussian transition
kernel. It is shown that such a process can be described by the following
stochastic differential equation (SDE) karras2022elucidating ; song2020score :

$dx_{t}=f(x,t)x_{t}dt+g(t)dw_{t},$ (1)

where $t\in[0,T]$, $w_{t}$ is the standard Brownian motion, and $f(x,t)$ and
$g(t)$ are the drift and diffusion coefficients respectively. Let
$p_{t}(x_{t})$ denote the corresponding marginal distribution of $x_{t}$.

It has been proven that this forward SDE possesses the identical marginal
distribution as the following probability flow (PF) ordinary differential
equation (ODE) song2020score :

$dx_{t}=\left[f(x,t)x_{t}-\frac{1}{2}g^{2}(t)\nabla{x_{t}}\log
p_{t}(x_{t})\right]dt.$ (2)

As long as we can learn a neural model $\epsilon_{\theta}(x_{t},{t})$ for
estimating the ground-truth score $\nabla{x_{t}}\log p_{t}(x_{t})$, we can
then draw samples that roughly follow the same distribution as the clean data
by solving the diffusion ODE. In practice, the learning of
$\epsilon_{\theta}(x_{t},{t})$ usually boils down to score matching
song2020score .

The ODE formulation is appreciated due to its potential for accelerating
sampling and has sparked a range of works on specialized solvers for diffusion
ODE lu2022dpm ; lu2022dpm++ ; song2020ddim . Let $\Psi$ denote an ODE solver,
e.g., the deterministic diffusion implicit model (DDIM) solver
song2020denoising . The sampling iterates by:

$x_{t_{n-1}}=\Psi(\epsilon_{\theta}(x_{t_{n}},{t_{n}}),t_{n},t_{n-1}),$ (3)

where $\\{t_{n}\\}_{n=0}^{N}$ denotes a set of pre-defined discretization
timesteps and $t_{N}=T,t_{0}=0$.

Latent diffusion models (LDMs) podell2023sdxl ; rombach2022high  extend DMs to
high-resolution image generation by converting pixel-level images into latent-
space representations with an encoder $E$ and a decoder $D$. Typically, $E$
and $D$ are jointly pre-trained under the variational auto-encoding principle
kingma2014auto .

###  2.2 Consistency Models

Consistency model (CM) song2023improved ; song2023consistency  aims at
generating images from Gaussian noise within one sampling step. Its core idea
is to learn a model $f_{\theta}(x_{t},t)$ that directly maps any point $x_{t}$
on the trajectory of the diffusion ODE to its endpoint. To achieve this, CMs
first parameterizes $f_{\theta}(x_{t},t)$ as:

$f_{\theta}(x_{t},t)=c_{skip}(t)x_{t}+c_{out}(t)F_{\theta}(x_{t},t),$ (4)

where $c_{skip}(t),c_{out}(t)$ is pre-defined to guarantee the boundary
condition $f_{\theta}(x_{0},0)=x_{0}$, and $F_{\theta}(x_{t},t)$ is the neural
network (NN) to train.

CM can be learned from a pre-trained DM $\epsilon_{\theta_{0}}$ via
consistency distillation (CD) by minimizing song2023consistency :

$\mathcal{L}_{CD}=d\big{(}f_{\theta}(x_{t_{m}},t_{m}),f_{\theta^{-}}(x_{t_{n}},t_{n})\big{)},$
(5)

where $t_{m}\sim\mathcal{U}[0,T]$, $x_{t_{m}}\sim p_{t_{m}}(x_{t_{m}})$,
$t_{n}\sim\mathcal{U}[0,{t_{m}})$,
$x_{t_{n}}=\Psi(\epsilon_{\theta_{0}}(x_{t_{m}},{t_{m}}),t_{m},t_{n})$,
$d(.,.)$ is some distance function, and $\theta^{-}$ is the exponential moving
average (EMA) of $\theta$. Typically, the solver $\Psi$ only runs for one step
for efficiency. Motivated by the success of CMs, latent CMs (LCMs)
luo2023latent  apply latent consistency distillation (LCD) to LDMs for
efficient high-resolution image synthesis, but suffer from degraded
generations in practice.

###  2.3 Multistep Consistency Models

CMs learn the mapping between an arbitrary state of the entire ODE trajectory
to the endpoint, which can cause significant learning difficulties and hence
compromise generations kim2023consistency . To address this, multistep
consistency models (MCMs) heek2024multistep  generalize CMs by splitting the
entire range $[0,T]$ into multiple segments and performing consistency
distillation individually within each segment. Formally, MCMs first define a
set of milestones $\\{t_{\mathrm{step}}^{s}\\}_{\mathrm{s}=0}^{K}$ ($K$
denotes the number of segments), and minimize the following multistep
consistency distillation (MCD) loss:

$\mathcal{L}_{MCD}=d\big{(}\mathrm{DDIM}(x_{t_{m}},f_{\theta}(x_{t_{m}},t_{m}),t_{m},t_{\mathrm{step}}^{s}),\mathrm{DDIM}(x_{t_{n}},f_{\theta^{-}}(x_{t_{n}},t_{n}),t_{n},t_{\mathrm{step}}^{s})\big{)},$
(6)

where $s$ is uniformly sampled from $\\{0,\dots,K\\}$,
$t_{m}\sim\mathcal{U}[t_{\mathrm{step}}^{s},t_{\mathrm{step}}^{s+1}]$,
$t_{n}\sim\mathcal{U}[t_{\mathrm{step}}^{s},t_{m})$, and
$\mathrm{DDIM}(x_{t_{m}},f_{\theta}(x_{t_{m}},t_{m}),t_{m},t_{\mathrm{step}}^{s})$
means one-step DDIM transformation from state $x_{t_{m}}$ at timestep $t_{m}$
to timestep $t_{\mathrm{step}}^{s}$ based on the estimated clean image
$f_{\theta}(x_{t_{m}},t_{m})$ song2020ddim .

By simplifying the optimization problem, MCMs enjoy better convergence and
higher generation quality. Besides, MCMs can naturally embrace multistep
generation heek2024multistep . However, it is unclear if the MCM strategy can
apply to large-scale LDMs and cope with the typical text-to-image generation.
Our work aims at bridging this gap.

##  3 Methodology

Existing distillation-based acceleration approaches for LDMs either (i) depend
on multiple individual distilled models for different sampling budgets
lin2024sdxl ; meng2023distillation ; salimans2022progressive , or (ii)
sacrifice generation quality with limited (e.g., 2-4) and/or moderate (e.g.,
5-8) sampling steps liu2023instaflow ; luo2023latent ; ren2024hyper ;
sauer2024fast ; sauer2023adversarial ; xu2023ufogen . We address these by
incorporating the promising MCD strategy into LDMs. Besides, there are three
technical contributions in the proposed Multistep Latent Consistency Models
(MLCMs): 1) we extend MCD to embrace classifier-guidance (CFG)
ho2022classifier  strategy and develop a progressive training strategy to
ensure inter-segment consistency. 2) We propose denoising consistency
distillation which removes the reliance on training images and avoids the
disparity between the training and inference of the model. 3) We incorporate
reward learning techniques into MLCMs to further enhance human preference.

###  3.1 Multistep Latent Consistency Models

We consider distilling representative pre-trained LDMs, e.g., SDXL
podell2023sdxl . Let $z_{t}$ denote the states at timestep $t$ in the latent
space. We abuse $\epsilon_{\theta_{0}}(z_{t},c,t)$ and $f_{\theta}(z_{t},c,t)$
to denote the pre-trained LDM and target model respectively, where $c$ refers
to the generation condition. We formulate the multistep latent consistency
distillation (MLCD) loss for MLCMs as:

$\small\mathcal{L}_{MLCD}=\|\mathrm{DDIM}\big{(}z_{t_{m}},f_{\theta}(z_{t_{m}},c,t_{m}),t_{m},t_{\mathrm{step}}^{s}\big{)}-\text{nograd}(\mathrm{DDIM}\big{(}z_{t_{n}},f_{\theta}(z_{t_{n}},c,t_{n}),t_{n},t_{\mathrm{step}}^{s})\big{)}\|_{2}^{2},$
(7)

where we make some specifications to Equation (6): 1) Instantiate the distance
measure $d$ as the $L^{2}$ distance for simplicity. 2) Replace the EMA model
with a stop-gradient variant of the current model to minimize memory cost
during training. 3) Fix $t_{n}=t_{m}-k$ with $k$ as a skipping interval,
which, unless specified otherwise, is set to $20$ to balance convergence speed
and generation quality.

Given CFG ho2022classifier  is critical for high-quality text-to-image
generation, we integrate it to MLCM by:

$z_{t_{n}}=\Psi(\hat{\epsilon}_{\theta_{0}}(z_{t_{m}},c,w,t_{m}),t_{m},t_{n}),$
(8)

where
$\hat{\epsilon}_{\theta_{0}}(z_{t},c,w,t):=\epsilon_{\theta_{0}}(z_{t},\emptyset,t)+w(\epsilon_{\theta_{0}}(z_{t},c,t)-\epsilon_{\theta_{0}}(z_{t},\emptyset,t))$
with $w$ as the guidance scale.

Better teacher brings further benefits. We empirically notice that the
generation quality of MLCM can be further boosted by incorporating a better
teacher $\epsilon_{\theta_{0}}$ for estimating the ODE trajectory. In this
sense, we advocate employing protovision-xl-v6.6 (PVXL) protovision-xl-v6.6 ,
a variant of SDXL fine-tuned on high-quality images, as
$\epsilon_{\theta_{0}}$ while remaining the model $f_{\theta}$ initialized
from SDXL for improved results. Other strategies for a better estimation of
the teacher trajectory liu2024scott  are left as future work.

###  3.2 Progressive MLCD

After a round of distillation on the $K$ segments, MLCMs can naturally produce
high-quality samples through $K$-step sampling. However, it is empirically
observed that the performance decreases when using fewer steps, shown in
Appendix A.4, which is probably because of the larger discretization error
caused by longer sampling step sizes. To alleviate this, we advocate
explicitly teaching MLCMs to capture the mapping between the states that cross
segments. This implies a progressive training procedure, where the gap between
the timesteps $t_{m}$ and $t_{n}$ increases continually. Given this, we opt to
use the MLCM trained in the previous stage, instead of the original pre-
trained model, as the teacher to more precisely estimate $z_{t_{n}}$ in only
one solving step for distillation. This differentiates MLCM from HyperSD
ren2024hyper  which uses a fixed teacher. In practice, we reduce $K$ by half
in each new distillation stage and multiply the skipping interval $k$ by 2
accordingly.

###  3.3 Image-free MCD with Samples from the Teacher

A common issue of the noise schedulers of LDMs is that the final noisy state
$z_{T}$ is not pure noise. For example, in SDXL, there is:
$z_{T}=0.068265z_{0}+0.99767\epsilon$ luo2024you . As a result, there exists a
discrepancy between the training and inference of such DMs and the models that
are distilled from them. We argue such a discrepancy can cause performance
degradation during inference, especially when the sampling steps are few
lin2024common .

Inspired by the success of Imagine Flash kohler2024imagine , we propose to
draw samples from the teacher models as training data to address this issue.
This also helps to relieve the requirements for extensive high-quality text-
image pairs. Specifically, instead of obtaining $z_{t_{m}}$ via adding noise
to $z_{0}$ as in MCM and HyperSD, we initialize $z_{T}$ as pure Gaussian noise
and perform denoising with off-the-shelf ODE solvers based on the teacher
model $\epsilon_{\theta_{0}}$ to obtain $z_{t_{m}}$. Intuitively, with this
strategy, we leverage and distill only the denoising ODE trajectory of the
teacher without concerning the forward one.

The latent state $z_{t_{m}}$ can be acquired by one-step sampling, but we
empirically find this naive strategy is unable to enhance the performance of
MLCD. This is because this method easily generates blurry and semantically
inconsistent samples. Theoretically, $z_{t_{m}}$ contains less noise for
smaller ${t_{m}}$. Therefore, we design a multistep denoising strategy (MDS)
to predict $z_{t_{m}}$, which executes more iterations for smaller ${t_{m}}$
to get cleaner $z_{t_{m}}$. For the first stage of MLCD to calculate
$z_{t_{m}}$, we present the details in Algorithm 1 in the appendix A.3. For
the progressive training stage, the procedure to compute $z_{t_{m}}$ is
described in Algorithm 2 in the appendix A.3. Figure 2 illustrates image-free
MLCD in the first stage and progressive stage, where we take initial segment
number $K=4$ as an example. In the first stage, DDIM sampler is used to
estimate ODE trajectory and calculate from pure Gaussian noise. Based on the
previous MLCD results, image-free progressive training is used to enforce
consistency on the states that cross previous segments with reduced segment
number, where the trained MLCM is used to estimate trajectory and $z_{t_{m}}$.

Figure 2: An illustration of the image-free MLCD in the first stage and
progressive stage. The first stage divides the entire timestep range $[0,T]$
into $K=4$ segments. Segment-wise LCD is performed within each segment
$[t_{\mathrm{step}}^{s},t_{\mathrm{step}}^{s+1}]$ by mapping each point to
$z_{t_{{\mathrm{step}}}^{s}}$. The latent codes $z_{t_{m}}$, e.g.,
$z_{t_{8}},z_{t_{6}}$, are calculated by denoising pure Gaussian noise $z_{T}$
with one-step and two-step DDIM sampler. In the progressive stage, the segment
number decreases by half. The trained MLCM is used to compute $z_{t_{m}}$ and
the teacher trajectory.

###  3.4 Reward Consistency and Feedback Learning

Barely relying on the $L^{2}$ loss in Equation (7) for distillation makes the
distilled model unaware of human preference. To bridge the gap, we propose to
incorporate human preference scorers $s(x,c)$, e.g., IR xu2024imagereward  and
AS AP , to define extra regularization for improving the visual quality.
Specifically, we define the following reward consistency loss, where the
latent codes are projected back to pixel space and a consistent score of them
is demanded:

$\displaystyle\mathcal{L}_{RC}$
$\displaystyle=\|s(D(\mathrm{DDIM}\left(z_{t_{m}},f_{\theta}(z_{t_{m}},c,t_{m}),t_{m},t_{\mathrm{step}}^{s}\right)),c)$
(9)
$\displaystyle\quad\quad\quad-s(D(\text{nograd}(\mathrm{DDIM}\left(z_{t_{n}},f_{\theta}(z_{t_{n}},c,t_{n}),t_{n},t_{\mathrm{step}}^{s}\right))),c)\|_{2}^{2}.$

Given that in the late stages of training, MLCMs can generate realistic
samples $\hat{x}_{0}$ in one or two steps, we can naturally directly maximize
the feedback of the scorer on the sample $s(\hat{x}_{0},c)$. Formally, we
optimize the following hinge loss with a margin $s_{0}$:

$\mathcal{L}_{FL}=\max(s_{0}-s(x_{0},c),0).$ (10)

The gradients are directly back-propagated from the scorer to model parameters
$\theta$ for optimization.

Table 1: Quantitative comparisons on MSCOCO-2017 5K validation datasets. All models adopts SDXL architecture.  Method | Step | CS | AS | IR | PS
---|---|---|---|---|---
DDIM song2020ddim  | 25 | 33.36 | 5.54 | 0.87 | 0.229
LCM luo2023latent  | 4 | 32.53 | 5.42 | 0.48 | 0.224
SDXL-Turbo sauer2023adversarial  | 4 | 33.30 | 5.64 | 0.83 | 0.226
SDXL-Lightning lin2024sdxl  | 4 | 32.40 | 5.63 | 0.72 | 0.229
SDXL-Lightning lin2024sdxl  | 8 | 32.73 | 5.95 | 0.71 | 0.227
HyperSD ren2024hyper  | 4 | 32.64 | 5.52 | 1.15 | 0.234
HyperSD ren2024hyper  | 8 | 32.41 | 5.83 | 1.14 | 0.233
MLCM | 2 | 33.02 | 6.10 | 1.10 | 0.227
MLCM | 3 | 33.24 | 6.17 | 1.18 | 0.232
MLCM | 4 | 33.30 | 6.19 | 1.20 | 0.233
MLCM | 8 | 33.48 | 6.20 | 1.22 | 0.233

##  4 Experiments

###  4.1 Main Results

We quantitatively compare our method with both the DDIM song2020ddim  baseline
and acceleration approaches including LCM luo2023latent , SDXL-Turbo
sauer2023adversarial , SDXL-Lightning lin2024sdxl , HyperSD ren2024hyper . CS
hessel2021clipscore  with ViT-g/14 backbone, AS AP , IR xu2024imagereward , PS
kirstain2024pick , are exploited as objective metrics. The evaluation is
performed on MSCOCO-2017 5K validation dataset lin2014microsoft . All methods
perform zero-shot validation except for HyperSD since it utilizes the
MSCOCO-2017 dataset for training.

The metrics of various methods are listed in Table 1. Among all the methods,
our 2-step MLCM presents superior CS, AS, IR, and PS than 4-step LCM
luo2023latent , 8-step SDXL-Lightning lin2024sdxl , 8-step HyperSD
ren2024hyper . These results indicate our MLCM’s synthetic images are better
aligned with texts and the human preference than LCM, SDXL-Lightning. Although
2-MLCM shows slightly lower CS than 4-step SDXL-Turbo, 2-MLCM enjoys much
higher AS, IR, and PS. Compared to 8-step HyperSD, 4-step MLCM obtains large
improvements in terms of CS, AS, IR, and comparable PS, although HyperSD uses
MSCOCO for training, which validates that our method is more powerful to
generate high-quality images with few-step inference. Moreover, We can observe
from Table 1 that our MLCM can improve the quality of the generated images
with additional steps.

We present the visual comparisons in Figure 3. Under the same conditions, we
observe that the images generated by MLCM have backgrounds with more intricate
details and maintain higher semantic consistency on more challenging prompts,
which also leads to greater human preference.

SDXL-Base NFE=25

LCM NFE=4

SDXL-Turbo NFE=4

SDXL-Lightning NFE=4

HyperSD NFE=4

MLCM (Ours) NFE=4

A girl, ethereal soft fluffy, soft landscape forest snowavatar, Pastel pink
sky

A blue wall with a large framed watercolor painting of a mountain

A coffee mug with an ankh symbol on it

A cat looking out of a window at a squirrel on a fence

A Christmas tree on a toy train

A giraffe with an owl on its head

A bat landing on a baseball bat

Figure 3: Visual comparison between our MLCM and baseline methods. The prompts
are basically from Parti-Prompts yu2022scaling . We can observe that our MLCM
can exhibit a richness in detail (the first three rows) and better semantic
coherence (the latter four rows), leading to better alignment with human
preference. Zoom in for more details.

Besides, MLCM can be leveraged as a plug-and-play module to provide
acceleration capabilities for other tasks, e.g. controllable generation. We
provide some examples generated by ControlNet zhang2023adding  with MLCM for
illustration in Figure 4. We include several illustrative examples using MLCM
on tasks like image style transfer in Appendix A.2.1 and Chinese-to-image in
Appendix A.2.2 to demonstrate the generality of MLCM.

Source

Canny edge

A dog in the winter

A black dog in the autumn

A beautiful dog in the garden

Source

Depth map

A black bird in the forest

A brown bird under the stars

A gray bird in the room

Figure 4: MLCM with ControlNet. Our MLCM can be incorporated into ControlNet
pipeline and produce satisfactory results with 2 steps sampling.

###  4.2 Ablation Study

To analyze the key components of our method, we make a thorough ablation study
to verify the effectiveness of the proposed MLCM. Table 2 depicts the
performance of MLCM’s variants. All the models use PVXL as the initial teacher
except where specified to use SDXL instead.

Multistep latent consistency distillation. It can be seen from Table 2 that
MLCD (SDXL) outperforms LCD (SDXL) by a large margin, which verifies that MLCD
has a stronger capability to accelerate LDM than LCD. This is because it is
hard for LCD to enforce consistency across the entire timestep range while
MLCD alleviates this by performing latent consistency distillation within each
segment.

Table 2: Ablation study of MLCM with respect to MLCD, single denoising iteration (SDI), multistep denoising iterations (MDI), reward consistency (RC), feedback learning (FL), progressive distillation with fixed teacher (PDF), progressive distillation with updated teacher (PDU). LCD (SDXL) represents the teacher is SDXL during LCD, otherwise, the teacher is PVXL. Method | Step | CS | AS | IR | PS
---|---|---|---|---|---
LCD (SDXL) | 4 | 32.53 | 5.42 | 0.48 | 0.224
MLCD (SDXL) | 4 | 32.74 | 5.88 | 0.92 | 0.224
MLCD | 4 | 32.68 | 6.01 | 0.94 | 0.225
MLCD+ SDI | 4 | 32.67 | 5.96 | 0.91 | 0.224
MLCD+ MDI | 4 | 33.00 | 6.03 | 0.95 | 0.225
MLCD+ MDI+ RC | 4 | 33.08 | 6.06 | 0.97 | 0.225
MLCD+ MDI+ RC+ PDF | 4 | 33.22 | 6.07 | 0.97 | 0.225
MLCD+ MDI+ RC+ PDU | 4 | 33.28 | 6.10 | 1.05 | 0.230
MLCM (SDXL) | 4 | 33.02 | 6.17 | 1.15 | 0.232
MLCM | 4 | 33.30 | 6.19 | 1.20 | 0.233

Image-free MLCD. As outlined in Table 2, MLCD + SDI achieves comparable
performance to MLCD, indicating our denoising MLCD is capable of training MLCM
without requiring any training images. The performance is substantially
enhanced by substituting SDI with MDI in terms of CS, AS, IR, verifying that
the proposed multistep denoising iterations is critical to optimize MLCM.

Reward consistency. As depicted in Table 2, the CS, AS, and IR values are
improved by adding RC to MLCD+ MDI, demonstrating the effectiveness of the
reward consistency.

Progressive MLCD. We can observe that both PDF and PDU introduce gains to
MLCD+ MDI+ RC, especially for CS value. This is because progressive MLCD
reduces segment number and further boosts segment-wise consistency along pre-
defined points at the inference stage. Furthermore, Table 2 demonstrates that
using trained MLCM in the previous stage significantly surpasses the fixed raw
teacher with respect to human preference metrics. The reason behind this is
the trained MLCM is much more powerful in predicting consistency trajectory
for large skipping time steps.

Feedback learning. As shown in Table 2, our MLCM obtains remarkable
improvements over MLCD+ MDI+ RC+ PD on all metrics via the proposed feedback
learning, because it steers MLCM toward the higher human preference
generation.

Teacher trajectory. As listed in Table 1, using different teacher models to
estimate the trajectory can influence the performance of MLCM. MLCM achieves
better performance than MLCM (SDXL) using SDXL as the teacher since PVXL
provides a more accurate trajectory in MLCD. Also, we can observe from Table 1
and Table 2 that our 4-step MLCM (SDXL) again significantly outperforms 4-step
LCM, 8-step SDXL-Lightning, in terms of CS, AS, IR. Meanwhile, 4-step MLCM
(SDXL) shows much higher CS and AS values than 8-step HyperSD, keeping
comparable IR and PS values. These results demonstrate the superiority of this
method over other works on accelerating LDMs.

##  5 Related works

Diffusion models (DMs) ho2020denoising ; sohl2015deep ; song2019generative ;
song2020improved ; song2020score  progressively add Gaussian noise to perturb
the data, then are trained to denoise noise-corrupted data. In the inference
stage, DMs sample from a Gaussian distribution and perform sequential
denoising steps to reconstruct the data. As a type of generative model, they
have demonstrated impressive capabilities in generating realistic and high-
quality outputs in text-to-image generation podell2023sdxl ; rombach2022high ;
saharia2022photorealistic , video generation peebles2023scalable , and audio
generation kong2020diffwave . To enhance the condition awareness in
conditional DMs, the classifier-free guidance (CFG) ho2022classifier
technique is proposed to trade off diversity and fidelity.

Diffusion acceleration. The primary challenges that hinder the practical
adoption of DMs is the slow inference involving tens to hundreds of
evaluations of the model. Early works like Progressive Distillation (PD)
salimans2022progressive  and Classifier-aware Distillation (CAD)
meng2023distillation  explore the approaches of progressive knowledge
distillation to compress sampling steps but lead to blurry samples within four
sampling steps. Consistency models (CMs) song2023consistency , consistency
trajectory models (CTMs) kim2023consistency  and Diff-Instruct luo2023diff
distill a pre-trained DM into a single-step generator, but they do not verify
the effectiveness on large-scale text-to-image generation.

Recently, the distillation of large-scale text-to-image DMs has gained
significant attention. LCM luo2023latent  extends CM to text-to-image
generation with few-step inference but synthesizes blurry images in four
steps. InstaFlow liu2023instaflow , UFOGen xu2023ufogen , SwiftBrush
nguyen2023swiftbrush  and DMD yin2023one  propose one-step sampling for text-
to-image generation but are unable to extend their sampler to multiple steps
for better image quality. More recently, SDXL-Turbo sauer2023adversarial ,
SDXL-Lighting lin2024sdxl  and HyperSD ren2024hyper  are proposed to further
enhance the image quality with few-step sampling but their training depends on
huge high-quality text-image pairs. Our method not only enables the generation
of high-quality samples using a 2 4 steps sampler but also enhances model
performance as the number of steps increases. Furthermore, our training
strategy is resource-efficient and does not require any images.

Feedback learning. While text-to-image DMs are capable of generating plausible
images in the majority of cases, they still suffer from inferior visual
generation quality and misalignment with human aesthetic preferences.
Supervised Learning lee2023aligning ; wu2023better , RLHF-based
black2023training ; fan2024reinforcement  and DPO-based wallace2023diffusion
are popular for better image quality. Among the various fine-tuning methods
proposed, reward feedback learning (ReFL) clark2023directly ;
xu2024imagereward  stands out as a widely used approach that calculates the
gradient of rewards. In our framework, we can leverage few-step generations of
MLCMs to directly guide model tuning, improving the performance of MLCM
without raising significant computation overheads.

##  6 Conclusion & Limitation

In this paper, we propose Multistep Latent Consistency Model (MLCM), a novel
approach for accelerating text-to-image latent diffusion models without
requiring any text-image paired data. MLCM can generate high-quality,
delightful images with only 2-8 sampling steps and achieve better image
quality than baseline methods while being compatible with controllable
generation, image style transfer, and Chinese-to-image generation.

Limitation. In our experiments, we observe that MLCM still has room for
improvement in single-step generation, which we plan to focus on in our future
work.

## References

  * (1) Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.  Training diffusion models with reinforcement learning.  arXiv preprint arXiv:2305.13301, 2023.
  * (2) Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet.  Directly fine-tuning diffusion models on differentiable rewards.  In The Twelfth International Conference on Learning Representations, 2023.
  * (3) Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Reinforcement learning for fine-tuning text-to-image diffusion models.  Advances in Neural Information Processing Systems, 36, 2024.
  * (4) Jonathan Heek, Emiel Hoogeboom, and Tim Salimans.  Multistep consistency models.  arXiv preprint arXiv:2403.06807, 2024.
  * (5) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: a reference-free evaluation metric for image captioning.  In EMNLP, 2021.
  * (6) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * (7) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * (8) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2021.
  * (9) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  Advances in Neural Information Processing Systems, 35:26565–26577, 2022.
  * (10) Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon.  Consistency trajectory models: Learning probability flow ode trajectory of diffusion.  arXiv preprint arXiv:2310.02279, 2023.
  * (11) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  stat, 1050:1, 2014.
  * (12) Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  Advances in Neural Information Processing Systems, 36, 2024.
  * (13) Jonas Kohler, Albert Pumarola, Edgar Schönfeld, Artsiom Sanakoyeu, Roshan Sumbaly, Peter Vajda, and Ali Thabet.  Imagine flash: Accelerating emu diffusion models with backward distillation.  arXiv preprint arXiv:2405.05224, 2024.
  * (14) Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * (15) Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu.  Aligning text-to-image models using human feedback.  arXiv preprint arXiv:2302.12192, 2023.
  * (16) Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang.  Common diffusion noise schedules and sample steps are flawed.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5404–5411, 2024.
  * (17) Shanchuan Lin, Anran Wang, and Xiao Yang.  Sdxl-lightning: Progressive adversarial diffusion distillation.  arXiv preprint arXiv:2402.13929, 2024.
  * (18) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * (19) Hongjian Liu, Qingsong Xie, Zhijie Deng, Chen Chen, Shixiang Tang, Fueyang Fu, Zheng-jun Zha, and Haonan Lu.  Scott: Accelerating diffusion models with stochastic consistency distillation.  arXiv preprint arXiv:2403.01505, 2024.
  * (20) Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu.  Instaflow: One step is enough for high-quality diffusion-based text-to-image generation.  arXiv preprint arXiv:2309.06380, 2023.
  * (21) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  Advances in Neural Information Processing Systems, 35:5775–5787, 2022.
  * (22) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  arXiv preprint arXiv:2211.01095, 2022.
  * (23) Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  arXiv preprint arXiv:2310.04378, 2023.
  * (24) Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang.  Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models.  arXiv preprint arXiv:2305.18455, 2023.
  * (25) Yihong Luo, Xiaolong Chen, and Jing Tang.  You only sample once: Taming one-step text-to-image synthesis by self-cooperative diffusion gans.  arXiv preprint arXiv:2403.12931, 2024.
  * (26) Jian Ma, Chen Chen, Qingsong Xie, and Haonan Lu.  Pea-diffusion: Parameter-efficient adapter with knowledge distillation in non-english text-to-image generation.  arXiv preprint arXiv:2311.17086, 2023.
  * (27) Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao.  Latte: Latent diffusion transformer for video generation.  arXiv preprint arXiv:2401.03048, 2024.
  * (28) Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297–14306, 2023.
  * (29) Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296–4304, 2024.
  * (30) Thuan Hoang Nguyen and Anh Tran.  Swiftbrush: One-step text-to-image diffusion model with variational score distillation.  arXiv preprint arXiv:2312.05239, 2023.
  * (31) William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023.
  * (32) Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  In The Twelfth International Conference on Learning Representations, 2023.
  * (33) Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao.  Hyper-sd: Trajectory segmented consistency model for efficient image synthesis.  arXiv preprint arXiv:2404.13686, 2024.
  * (34) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * (35) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in neural information processing systems, 35:36479–36494, 2022.
  * (36) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In International Conference on Learning Representations, 2022.
  * (37) Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach.  Fast high-resolution image synthesis with latent adversarial diffusion distillation.  arXiv preprint arXiv:2403.12015, 2024.
  * (38) Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation.  arXiv preprint arXiv:2311.17042, 2023.
  * (39) Christoph Schuhmann.  Clip+mlp aesthetic score predictor.  https://github.com/christophschuhmann/improved-aesthetic-predictor.  Accessed: 2024-05-20.
  * (40) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * (41) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  * (42) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In International Conference on Learning Representations, 2020.
  * (43) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * (44) Yang Song and Prafulla Dhariwal.  Improved techniques for training consistency models.  arXiv preprint arXiv:2310.14189, 2023.
  * (45) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  arXiv preprint arXiv:2303.01469, 2023.
  * (46) Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in neural information processing systems, 32, 2019.
  * (47) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  Advances in neural information processing systems, 33:12438–12448, 2020.
  * (48) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2020.
  * (49) stablediffusionapi.  protovision-xl-v6.6.  https://huggingface.co/stablediffusionapi/protovision-xl-v6.6.  Accessed: 2024-05-20.
  * (50) Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik.  Diffusion model alignment using direct preference optimization.  arXiv preprint arXiv:2311.12908, 2023.
  * (51) Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li.  Better aligning text-to-image models with human preference.  arXiv preprint arXiv:2303.14420, 2023.
  * (52) Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  Advances in Neural Information Processing Systems, 36, 2024.
  * (53) Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou.  Ufogen: You forward once large scale text-to-image generation via diffusion gans.  arXiv preprint arXiv:2311.09257, 2023.
  * (54) Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park.  One-step diffusion with distribution matching distillation.  arXiv preprint arXiv:2311.18828, 2023.
  * (55) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, et al.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv preprint arXiv:2206.10789, 2(3):5, 2022.
  * (56) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023.

##  Appendix A Appendix / supplemental material

###  A.1 Implementation Details

We use the prompts from LAION-Aesthetics- 6+ subset of LAION-5B [40] to train
our model. We train the model with 4 A100 and a batch size of 24 for 40,000
iterations. The learning rate is 1e-5 with Adam optimizer. The initial segment
number $K$ is 8 and $s_{0}$ for IR and AS are 2.0 and 8.0, respectively. The
total loss function to train MLCM is:

$\mathcal{L}=\mathcal{L}_{MLCD}+\mathcal{L}_{RC}+w_{FL}\mathcal{L}_{FL},$ (11)

where $w_{FL}$ is 1e-3. We set the guidance scale $w$ in CFG as 8.0. As for
model configuration, we use PVXL [49] as the default teacher to estimate
trajectory while student model $f_{\theta}$ is initialized by SDXL [17]. We
also conduct the experiment adopting SDXL as a teacher for fair comparisons
with other methods. We train a unified Lora instead of UNet in all the
distillation stages for convenient transfer to downstream applications.

###  A.2 Application

####  A.2.1 Acceleration of Image Style Transfer

Our MLCM LoRA is compatible with the pipeline of image style transfer [29]. We
present some examples in Figure 5 with only 2-step sampling.

Source

Japanese comics

Ink and wash style

Pixar dreamworks

Van Gogh’s paintings

Figure 5: MLCM with image style transfer. The styles are presented at the top,
and we apply image style transfer on the source image with our MLCM. Two-step
sampling can produce highly stylized images with excellent results.

####  A.2.2 Acceleration of Chinese-to-image Generation

Our MLCM can accelerate the generation speed of Chinese-to-image diffusion
model [26]. We present some examples in Figure 6.

Figure 6: MLCM for Chinese-to-image generation. With 3 steps sampling, our
MLCM model can produce images that align with Chinese semantic meaning. The
first line presents images in general Chinese contexts, while the second line
showcases images in specific Chinese cultural settings.

###  A.3 Algorithms

Input: Gaussian noise $\epsilon$, timestep ${t_{m}}$, segment index $s$,
teacher model $\epsilon_{\theta_{0}}$, text condition $c$, segment number $K$

Initialize $z_{T}$ with $\epsilon$, calculate denoising steps $L=K-s$, time
interval $\triangle T=(T-t_{m})/L$

for  _$i\quad\text{in}\quad\\{0,1,\cdots,L-1\\}$ _ do

Calculate $t=T-i*\triangle T,\quad t_{m^{\prime}}=t-\triangle T$

Calculate
$z_{t_{m^{\prime}}}=\Psi(\hat{\epsilon}_{\theta_{0}}(z_{t},c,w,t),t,t_{m^{\prime}})$

end for

Algorithm 1 Denoising iterations of teacher generating latent code in the
first stage of MLCD

Input: Gaussian noise $\epsilon$, timestep ${t_{m}}$, MLCM $f_{\theta}^{t}$,
text condition $c$, segment number $K$

Initialize $z_{T}$ with $\epsilon$, calculate time interval $\triangle
T=T/(2K)$ and denoising steps $L=(T-t_{m})/\triangle T$

for  _$i\quad\text{in}\quad\\{0,1,\cdots,L-1\\}$ _ do

Calculate $t=T-i*\triangle T,\quad t_{m^{\prime}}=t-\triangle T$

Calculate
$z_{t_{m^{\prime}}}=\mathrm{DDIM}(z_{t},f_{\theta}^{t}(z_{t},c,t),t,t_{m^{\prime}})$

end for

Algorithm 2 Denoising iterations of MLCM to generate latent code in the
progressive stage

###  A.4 The first stage of training

After the first stage of training, the MLCM can generate clear images with
$K$-step sampling. However, the results from sampling with fewer than $K$
steps are still unsatisfactory. We provide some intermediate samples in Figure
7, which also demonstrate the importance of progressive training.

A train traveling down tracks next to lights

A yellow taxi cab sitting below tall buildings

A fluffy white chair that faces away from a television

A person sitting at a desk with a keyboard and monitor

The shiny motorcycle has been put on display

Figure 7: Examples after the first stage of MLCM training. From top to bottom,
2, 3, 4 and 8 sampling steps are adopted, respectively. We extract prompts
from MSCOCO dataset but omit them for simplicity. We observe that 4-step
sampling can achieve satisfactory results, while 2&3-step sampling still
produces blurry images.

###  A.5 Broader Impact

Our work has the potential to lead to significant societal consequences.
Similar to other generative models, DPMs could be utilized to create
misleading or even harmful content. Furthermore, our method may exacerbate the
negative impact of malicious uses of generative AI.