capbtabboxtable[][] capbtabboxtable[][]

#  Live2Diff: Live Stream Translation via Uni-directional Attention in Video
Diffusion Models

Zhening Xing1 Gereon Fox2 Yanhong Zeng1 Xingang Pan3
Mohamed Elgharib2 Christian Theobalt2 Kai Chen1,$\dagger$

1Shanghai Artificial Intelligence Laboratory
2Saarland Informatics Campus, Max Planck Institute for Informatics
3S-Lab, Nanyang Technological University

https://live2diff.github.io

###### Abstract

Large Language Models have shown remarkable efficacy in generating streaming
data such as text and audio, thanks to their temporally uni-directional
attention mechanism, which models correlations between the current token and
_previous_ tokens. However, video streaming remains much less explored,
despite a growing need for live video processing. State-of-the-art video
diffusion models leverage _bi_ -directional temporal attention to model the
correlations between the current frame and all the _surrounding_ (i.e.
including _future_) frames, which hinders them from processing streaming
videos. To address this problem, we present Live2Diff, the first attempt at
designing a video diffusion model with uni-directional temporal attention,
specifically targeting live streaming video translation. Compared to previous
works, our approach ensures temporal consistency and smoothness by correlating
the current frame with its predecessors and a few initial warmup frames,
without any future frames. Additionally, we use a highly efficient denoising
scheme featuring a $K{}V{}$-cache mechanism and pipelining, to facilitate
streaming video translation at interactive framerates. Extensive experiments
demonstrate the effectiveness of the proposed attention mechanism and
pipeline, outperforming previous methods in terms of temporal smoothness
and/or efficiency.

##  1 Introduction

Large Language Models (LLMs) have recently been very successful in natural
language processing and various other domains jiang2023mistral;
jiang2024mixtral; touvron2023llama; touvron2023llama2; vaswani2017attention.
At the core of LLMs is the autoregressive next-token prediction, seamlessly
enabling real-time streaming data generation. Such a mechanism processes data
continuously as it streams in, bypassing the need for batch storage and
delayed processing. This token prediction mode has been widely used in many
applications such as dialog systems claude; openaiChat, text-to-speech
wang2023neural; zhang2023speak, audio generation copet2024simple;
huang2023make; kreuk2022audiogen, etc. Despite the recent success of this
streamed generation of sequential data like text and audio, it has not been
fully explored for another very common sequential data type: videos. However,
generating videos in a streaming manner is clearly worth investigating given
the growing practical demand, particularly in live video processing, where the
original input frames need to be translated into a target style on the fly.

Motivated by this, we study next-frame-prediction for producing streaming
videos, with streaming video-to-video translation as the target application.
Most existing video diffusion models exploit temporal self-attention modeling
in a _bi_ -directional manner, where the models are trained on sequences of
input frames to capture the pairwise correlations between all frames in a
sequence blattmann2023stable; blattmann2023align; geyer2023tokenflow;
guo2023animatediff; gupta2023photorealistic; yang2023rerender. Despite some
promising results, these models have limitations for streaming video. Early
frames in a sequence rely on information from later frames, and vice versa for
those at the end, which impedes efficient real-time processing of each frame
as it streams in.

To address this issue, we redesign the attention mechanism of video diffusion
models for streaming video translation, ensuring both high efficacy and
temporal consistency. First, we make temporal self-attention _uni_
-directional via an attention mask, akin to attention in LLMs
jiang2023mistral; vaswani2017attention. This removes the dependency of early
frames on later frames in both training and inference, making the model
applicable to streaming videos. However, ensuring both high inference
efficiency and performance on long video streams using unidirectional
attention is non-trivial. An approach is to use dense attention with all
previous frames for next-frame prediction. However, it increases time
complexity and decreases performance when the video length exceeds the
attention window size used during training. Another option is to limit
temporal self-attention to a smaller fixed window size during inference.
Unfortunately, unlike the sufficient context attention from user input tokens
in LLM jiang2023mistral; vaswani2017attention; xiao2023streamingllm, it is
difficult to generate satisfactory frames with limited context attention at
the beginning of the video stream, which further results in artifacts in later
frames. To tackle this, we introduce warmup area in the unidirectional
attention mask, which incorporates bi-directional self-attention modeling to
compensate for the limited context attention at the beginning of the stream.
During inference, we include the attention from a few warmup frames at the
start of the stream to the current frame. Such a tailored attention design
ensures both stream processing efficacy and temporal consistency modeling.

Building upon our tailored attention mechanism, we present Live2Diff, a
pipeline that processes LIVE video streams by a uni-directional video
DIFFUSION model while ensuring high efficacy and temporal consistency. First,
our attention modeling mechanism removes the influence of later frames on
previous frames, allowing for the reuse of $K$ and $V$ maps from previously
generated frames. This eliminates the need for recomputation when processing
subsequent frames. We carefully designed a $K{}V{}$-cache feature in the
diffusion pipeline to cache and reuse K/V maps, resulting in significant
computation time savings. Second, we further include a lightweight depth prior
in the input, ensuring structural consistency with the conditioning stream.
Finally, Live2Diff uses the batch denoising strategy to further improve stream
processing efficacy, achieving 16FPS for $512\times 512$ videos on an RTX 4090
GPU. We conduct extensive experiments to validate the superiority of Live2Diff
in terms of temporal smoothness and/or efficiency. We summarize our main
contributions as follows,

  * •

To the best of our knowledge, we are the first to incorporate uni-directional
temporal attention modeling into video diffusion models for video stream
translation.

  * •

We introduce a new pipeline Live2Diff, which aims at achieving live stream
video translation with both high efficacy (16FPS on an RTX 4090 GPU) and
temporal consistency.

  * •

We conduct extensive experiments including both quantitative and qualitative
evaluation to verify the effectiveness of Live2Diff.

Figure 1:  We visualize different types of temporal self-attention when the
number of frames ($F=8$) exceeds the length of the context window ($L=4$). The
$j$-th cell of the $i$-th row is highlighted if the output for frame $i$ may
contain information from frame $j$. The red square delineates the attention
mask used during training. (a) shows temporal self-attention in current video
diffusion models, which is bi-directional within the context window without
overlap between chunks. (b) uses a sliding window with overlap $L_{s}$ (three
subsequent positions of which are highlighted in different colors, for
clarity) and fuses the output of overlap regions. (c) denotes the uni-
directional attention widely used in LLMs. (d) shows the attention proposed by
our method. We set the initial $L_{w}$ frames as warmup frames and apply bi-
directional attention to them, while using uni-directional attention for the
subsequent frames. The initial warmup frames also contribute to the output for
all future frames.

##  2 Related Work

Attention. LLMs jiang2023mistral; jiang2024mixtral; touvron2023llama;
touvron2023llama2 owe their success largely to the remarkable efficacy of the
attention mechanism vaswani2017attention. In order to support the auto-
regressive prediction of the next token, they use a uni-directional (or
“masked”) attention mechanism, restricting the model to learning the
dependence of _later_ tokens on _earlier_ ones, but no dependence of earlier
tokens on later ones. However, for tasks with very long token sequences,
relating the current token to all previous tokens becomes intractable. To
address this, StreamingLLM xiao2023streamingllm proposes to relate the current
token to several initial tokens and a number of most recent tokens, which
improves efficiency in handling long tokens. While such kind of uni-
directional attention is widely used in generating text and audio, video
generation has not yet followed this trend: Bi-directional attention without
masks is commonly used in video diffusion models blattmann2023stable;
blattmann2023align; guo2023animatediff; gupta2023photorealistic to generate
video chunks. In this work, we study the use of uni-directional temporal
attention in video diffusion models. While our method draws inspiration from
StreamingLLM, it is the first time that such a design is studied in the video
domain.

Video Diffusion Models. The multitude of possible conditioning modalities has
made diffusion models the basis for image editing approaches meng2021sdedit;
kawar2023imagic, as well as video generation models guo2023animatediff;
liang2023flowvid; kodaira2023streamdiffusion. For example, AnimateDiff
guo2023animatediff extends StableDiffusion by a so-called “motion module”,
enabling the denoising of entire video chunks based on temporal self-attention
vaswani2017attention. FreeNoise qiu2023freenoise is a method based on
pretrained video diffusion models (e.g. AnimateDiff guo2023animatediff) for
long video generation. This method carefully selects and schedules the latent
noise for every time step in order to improve temporal smoothness. However,
FreeNoise, according to their experiments section produces frames at under
3FPS on an NVIDIA A100 GPU, which is not acceptable in the kinds of live
streaming scenarios that we aim at (see Section 1). FlowVid liang2023flowvid
and Rerender yang2023rerender produce frames at even lower rates, albeit with
acceptably smooth results.

Accelerating Diffusion Models. Some recent diffusion-based methods luo2023lcm;
luo2023lcmlora; song2023consistency; kodaira2023streamdiffusion have
prioritized low latency and/or high throughput: (Latent) consistency models
(LCMs) song2023consistency; luo2023lcm have reduced the number of denoising
steps from 50 (the default in StableDiffusion) to as low as 4, leading to
large speed ups without too much loss in quality. This principle has even been
combined with the use of low-rank matrices for fine-tuning luo2023lcmlora,
allowing further speedup. A work that very specifically targets the streaming
frame-by-frame translation setting is StreamDiffusion
kodaira2023streamdiffusion: Not only is this technique utilizing the
aforementioned low-rank-adapted LCMs, but also it denoises video frames in a
“pipelined” manner for the streaming scenario, i.e. the batch of images to be
denoised can contain different levels of remaining noise, allowing new frames
to be added to the batch before previous frames in the batch have been
completely denoised, which makes optimal use of GPU parallelization. However,
StreamDiffusion renders videos frame-by-frame without any temporal modeling,
leading to significant temporal discontinuity, which our method avoids due to
the temporal correlations learned during training.

##  3 Method

##  4 Method

Our method, Live2Diff (see Fig. 2), takes as input a stream of video frames,
along with a matching text prompt. It produces a stream of output video
frames, the spatial structure of which is similar to that of the input frames,
but the appearance/style of which conforms to a specified target style,
captured by DreamBooth ruiz2022dreambooth. To achieve this, we replace the
bidirectional temporal attention used in previous approaches by _uni_
directional attention (Section 4.3). This allows us to cache $K$ and $V$ maps
from previous frames, leading to increased throughput (Section 4.4).
Furthermore we accelerate generation by pipelined denoising, i.e. multiple
time steps with different levels of residual noise are denoised in parallel.
By employing LCM-LoRAluo2023lcmlora we can drastically reduce the number of
necessary denoising steps, which also helps meet framerate criteria. We
stabilize the spatial structure of frames with lightweight depth injection.

Figure 2:  The overview of Live2Diff. (a) During training, our model takes as
inputs $L$ frames of noisy latents $z_{t}^{f:f+L}$ and depth conditioning
$y^{f:f+L}$, where $f:f+L$ delimits the frame interval in a video stream, $t$
is the denoising timestep, $\oplus$ denotes point-wise addition. (b) During
inference, frame $z^{f+1}$ is incorporated into the processing batch as it
streams in, often before earlier frames are fully denoised. This results in a
batch that includes frames at various denoising timesteps (e.g., $t_{1}$ and
$t_{2}$). We employ a $K{}V{}$-cache mechanism to effectively reuse $K$ and
$V$ maps from previous frames, significantly improving inference efficiency
while ensuring temporal consistency.

###  4.1 Preliminaries

Diffusion Models. Diffusion models ho2020ddpm; dhariwal2021ADM aim at undoing
the so-called “forward process”, that iteratively adds Gaussian noise to the
representation of a sample from a distribution. To achieve this,
StableDiffuion rombach2022high trains a U-Net ronneberger2015unet to estimate
the noise component of a noisy latent representation of any given image. By
repeatedly estimating remaining noise and removing (some of) this noise from
the latent code, a purely Gaussian noise vector can iteratively be denoised to
obtain a clean sample as follows: Given a noisy latent code $z_{t}$, the U-Net
parametrized by weights $\theta$ computes the estimated noise
$\epsilon_{\theta}(z_{t},t,\mathcal{T}(c))$, where $\mathcal{T}(c)$ is the
CLIP encoding radford2021clip of a conditioning text string $c$. The less
noisy latent code $z_{t-1}$ can then be approximated as

$z_{t-1}\approx\lambda\cdot
z_{t}+\mu\cdot\epsilon_{\theta}(z_{t},t,\mathcal{T}(c))$ (1)

where $\lambda,\mu\in\mathbb{R}$ are constants derived from the noise schedule
of the forward process song2020ddim. The U-Net is trained by sampling images
$x$ from the training distribution, mapping them to latent codes
$z_{0}=\mathcal{E}(x)$ and then adding varying amounts of Gaussian noise to
obtain $z_{t}$, such that the U-Net output can be supervised by $L1$ distance
to the known noise. Like in StableDiffusion, we use this as our main loss, but
with $x$ holding not single images, but chunks of consecutive video frames.

Bidirectional Attention in Video Generation. Several video diffusion models
guo2023animatediff; blattmann2023align; wang2023modelscope;
chen2024videocrafter2 use temporal self-attention layers to improve temporal
smoothness of the output, essentially encouraging the model to learn temporal
correlations. A temporal self-attention layer computes its output as

$f_{\text{out}}:=\operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{C}}\right)\cdot
V$ (2)

where $Q:=\mathcal{W}_{Q}\cdot f_{\text{in}}$, $K:=\mathcal{W}_{K}\cdot
f_{\text{in}}$, and $V:=\mathcal{W}_{V}\cdot f_{\text{in}}$ are linear
projections of the input features$f_{\text{in}}$ and $C$ is the number of
feature channels. AnimateDiff adds sinusoidal position encoding to
$f_{\text{in}}$ before computing Eq. 2, to give the layer access to the
temporal position of each feature vector. Note that $f_{\text{in}}$ contains
information about an entire chunk of frames, as does $f_{\text{out}}$. Since
previous works use such temporal attention layers without masking
vaswani2017attention, $f_{\text{out}}$ can thus base its information about a
particular frame on frames before and _after_ that frame. Exploiting temporal
correlations in this bidirectional way helps produce temporally smooth output,
but is counter-productive for the streaming setting, as a prefix of
$f_{\text{out}}$ will often need to be computed before the full
$f_{\text{in}}$ is even available. In FreeNoise qiu2023freenoise this “direct”
bidirectionality was constrained in the sense that only frames from a
“temporal neighborhood” can influence each other _directly_ (see Fig. 1 (b)),
but since the U-Net contains multiple temporal attention layers and because
denoising requires multiple U-Net applications, the distance over which
information can be transferred backwards in time is still considerable. Fig. 3
(b) and (c) show the artifacts of using this attention mode for streaming.

Figure 3:  The X-T slice shows how the pixel values at the same X-coordinate
change over time T. The position of the horizontal lines in the video
corresponds to the X-coordinate positions visualized in the X-T slice. The
color of each line represents the time in the X-T plot. Red dashed boxes
denote regions suffering from flickering and structural inconsistency, while
blue boxes indicate areas where these issues are resolved. Flickering and
gradual change in the background region can be observed in (b), (c) and (d),
which use the first three attention modes illustrated in Fig. 1 respectively.
In case (e), with the last attention mode from Fig. 1 (see also Section 4.3,
background flickering is reduced. The depth conditioning in (f) improves
structure consistency further.

###  4.2 Input & Structural prior

Following AnimateDiffguo2023animatediff, we insert temporal self-attention
modules (TSA) in-between the StableDiffusionrombach2022high U-Net layers, see
Fig. 2. In order to condition our generated video on the input video similarly
to SDEdit meng2021sdedit we add a certain amount of Gaussian noise to the
input frames. The strength of this noise is chosen such that the high level
spatial structure of the image is still recognizable, but the appearance
information is largely erased at finer scales. This requires the denoising
U-Net to fill in appearance information according to the target style. To
facilitate the preservation of spatial structure we use an additional depth
input: We use MiDas ranftl2022midas; ranftl2021midas for frame-wise depth
estimation. The depth frames are then encoded by StableDiffusion’s encoder
$\mathcal{E}$, with the results being fed into a lightweight convolutional
module $E^{cond}$ (see Fig. 2). Finally, we add the output of the conditional
module to the first convolution layer and pass it through the U-Net. We found
this explicit structural prior to help the transferred video maintain better
structural consistency with the source video. Evidence can be found in Fig. 3
and Section 6.

###  4.3 Uni-directional Temporal Self-Attention with Warmup

To turn bi-directional attention, as shown in Fig. 1 (a), in which each frame
inside a chunk can be based on information from all other time steps in the
chunk, into uni-directional attention, where each frame can only depend on
_earlier_ frames, we use masked attention vaswani2017attention;
jiang2023mistral. Fig. 1 (c) presents a solution using the uni-directional
attention mask commonly used in LLMtouvron2023llama; touvron2023llama2;
vaswani2017attention; jiang2024mixtral. However, as shown in red dashed box in
Fig. 3 (d), the spatial structure of the first frame’s output is inconsistent
with the input, and the character identity differs from subsequent frames.
Flickering in the background region can be observed in the X-T slice as well.
We believe that the reason for this attention mode being less effective for
video transfer than for LLMs is rooted in the fact that in LLMs, attention
operations rely on user prompts as initial tokens for uni-directional
attention, while in video transfer such initial tokens first need to be
generated by the model.

This is why we choose the attention mask in Fig. 1 (d): We apply bi-
directional attention in the corner case of the first $L_{w}$ “warmup” frames,
and unidirectional attention afterwards. In Fig. 3 (e) we observe that the
flickering issue in the red dashed region has been resolved. The remaining
structure inconsistencies between the red and blue dashed regions compared to
the input are resolved by the structural prior (see Section 4.2).

Figure 4:  (a) - (c) depict the usage of our $K{}V{}$-cache for the first
steps of a stream, with $L_{w}=T=2$. The colors of the squares indicate which
frame they belong to. $Q,K,V$ are the matrices used in Eq. 2, with subscripts
indicating which denoising step they are used in and superscripts indicating
which frame they belong to. Each row belongs to one of the two denoising
steps. Red arrows are overwrite operations. For a step by step walkthrough see
Section 4.4.

###  4.4 High Efficiency Inference Pipeline

$K{}V{}$-cache. As described in Section 4.3 our warmup-based temporal self-
attention makes sure that when we compute the attention for a certain frame,
the attention for all previous frames has already been computed. This means
that those parts of the matrices $K$ and $V$ in Eq. 2 that concern the
previous frames do not need to be computed again, but can be retrieved from a
cache, our $K{}V{}$-cache. Note that the diffusion U-Net has multiple temporal
attention layers, and that the U-Net needs to be applied $T$ times in order to
fully denoise a frame. This means that for every combination of layer and
denoising step, we keep a separate $K{}V{}$-cache. More details in Section
A.2.

Warmup stage. At the start of the stream, we feed $L_{w}$ frames into the
U-Net and denoise them completely, with bidirectional attention as introduced
in Section 4.3. This gives us the $K$ and $V$ matrices highlighted in green in
Fig. 4. They are used for _all_ future frames for temporal consistency.

Pipelined denoising. Similarly to StreamDiffusion kodaira2023streamdiffusion,
we denoise frames in a pipelined manner, i.e. as soon as the next input frame
becomes available, we add it to the batch of frames to be denoised, even
though it may contain a much higher amount of noise than previous frames in
the batch that have already undergone multiple denoising steps. This way we
utilize our GPU capacity most efficiently, increasing throughput.

Fig. 4 illustrates the bookkeeping in our $K{}V{}$-cache for the simple case
$L_{w}=T=2$, just after warmup: In the first step, to process the red feature
vector, temporal self attention according to Eq. 2 is computed only using the
green $K$ and $V$ maps and the red one. The $K$ and $V$ for the input token
are then written into the cache (red arrow) and one denoising step can be
computed. In the second step, the orange frame arrives, which in addition to
the green warmup tokens we also correlate with the previously cached $K$ and
$V$ for the red frame. In parallel with the first denoising step for the
orange frame, we compute the second denoising step for the red frame. Now all
caches are filled, for the third step, in which the blue input frame arrives.

###  4.5 Training

To train our model we use data collected from Shutterstock shutterstock2024,
resized to resolution $256\times 256$. We choose $L=16$ and $L_{w}=8$. We
train our model as follows: We initialize the weights of our temporal self-
attention modules with the weights from AnimateDiff and fine-tune them for
3000 iterations using our uni-directional attention (Section 4.3). Then we add
$E^{cond}$ (Section 4.2), with the last layer initialized with zeros
zhang2023controlnet and train all weights jointly for 6000 iterations. We use
the Adam kingma2014adam optimizer with a learning rate of $1e-4$ and train on
batches of 4 samples per GPU, on 8 GPUs. Accumulation of 32 gradients leads to
an effective batch size of 1024.

##  5 Results

##  6 Results

###  6.1 Evaluation Setup

Dataset. We evaluate on the DAVIS-2017 pont2017davis dataset, which contains
90 object-centric videos. We resize all frames to $512\times 768$ via bilinear
interpolation and use CogVLM wang2023cogvlm to caption the middle frame of
each video clip. To specify the target style, we add the corresponding trigger
words of DreamBooth and LoRA as suffix. For more information about the
prompts, see Section A.3.

Metrics. We evaluate three aspects of the generated videos: _structure
consistency_ (Output frames should have similar spatial structure as input
frames), _temporal smoothness_ (no sudden jumps in the motion) and _inference
latency_. We measure _structure consistency_ as the mean squared difference
between the depth maps estimated ranftl2021midas for the input and output
frames. As in previous work wu2023tune; khachatryan2023text2video;
guo2023animatediff we measure _temporal smoothness_ by CLIP score
radford2021clip, i.e. by the cosine similarity of the CLIP embeddings of pairs
of adjacent frames. In addition we compute the so-called “warp error”
lai2018warpError for pairs of adjacent frames, i.e. we compute the optical
flow teed2020 between the frames and then warp the predecessor frame
accordingly, to compute a weighted MSE between the warping result and the
successor. We also conduct a user study to evaluate _structure consistency_
and _temporal smoothness_ : Each participant is given triplets of videos
(original input video, result from our method, result from a random different
method) and is asked to identify the result with the best quality. Then we
calculate the rate of our method winning compared to other methods. A higher
win-rate indicates that users perceive our method to be better in the
corresponding evaluation direction. Please refers to Section A.3 for detailed
information. We measure _inference speed_ as the total amount of time it takes
each method to process an input stream of 100 frames at resolution $512\times
512$, on a consumer GPU (NVIDIA RTX 4090).

###  6.2 Comparisons

We compare our method to three previous works, all based on StableDiffusion
rombach2022high and compatible with DreamBooth ruiz2022dreambooth and LoRA
hu2021lora: StreamDiffusionkodaira2023streamdiffusion applies SDEdit on a
frame-by-frame basis. The same noise vector is used for all the frames, to
improve consistency and smoothness. To achieve interactive framerates,
StreamDiffusion uses LCM-LoRAluo2023lcmlora and Tiny-VAEollin2022taesd, with
the latent consistency model scheduling luo2023lcm. We apply the same
acceleration techniques in our method. Rerenderyang2023rerender first inverts
input key frames into noisy latent codes. During denoising, temporal coherence
and spatial structure stabilization are achieved by using cross-frame
attention, flow-based warping and ControlNet zhang2023controlnet. We select
all frames as key frames for the purposes of our evaluation, but otherwise use
the default settings. FreeNoise qiu2023freenoise does not natively support an
input video as conditioning, but by adding a sufficient amount of noise to the
input, similar to our method and SDEdit, we can nevertheless use it for our
video-to-video translation task. The amount of noise we add is equivalent to
half of the entire denoising process. FreeNoise uses a technique called
window-based attention fusion, that (similar to bidirectional temporal
attention) leads to frames incorporating information from future frames. This
actually makes it unsuitable for the streaming setting, which we mitigate by
giving FreeNoise access to _all_ frames, not expecting to receive the first
output frames after we have given the last input frames. In this sense we are
giving FreeNoise a considerable advantage. Qualitative Comparison. Fig. 5
compares outputs of all methods: While part (a) shows two consecutive output
frames, part (b) shows frames that are further apart.
StreamDiffusionkodaira2023streamdiffusion exhibits strong flickering in the
background (red box in (a)). When foreground and background are difficult to
distinguish (box and shelf in (a), dog and table in (b)), works other than
ours struggle to produce satisfactory results: StreamDiffusion generates
inconsistent results with low quality. Rerender generates strong artifacts in
the first frame (see (a)) and propagates them to later frames. FreeNoise fails
to adhere to the input frame and generates elements unrelated to prompt and
input. In contrast, our method leverages depth information to ensure the
structural accuracy of the generated results (e.g. the box in Fig. 5 (a)) and
maintains consistency over longer duration (dog in Fig. 5 (b)).

Quantitative Comparison. In Table 1 we evaluate structure consistency and
temporal smoothness: While our method outperforms the others in structure
consistency, we observe that FreeNoise achieves a better CLIP score and warp
error for temporal smoothness. This is not surprising, as the way that time
steps are correlated in FreeNoise allows information to flow bidirectionally
along the temporal axis, which, unlike for all other methods, required
FreeNoise to be given access to _all_ frames at once (see first paragraph of
Section 6.2). This is an unfair advantage to FreeNoise, violating some
assumptions of the streaming scenario, as it allows FreeNoise to correlate its
output frames to input frames that would likely not be available at the time
the output frame needs to be produced. In our user study, all our win rates
are way above 50% for both structural consistency and temporal smoothness,
confirming that our results are the most convincing.
StreamDiffusionkodaira2023streamdiffusion scores second-best in structure
consistency, likely because it applies only a moderate amount of noise to its
input, but this limits its ability to conform with the target style (see also
Fig. 5). Table 1 also compares inference latencies, i.e. the average time that
goes by between receiving an input frame and producing the corresponding
output frame. As is to be expected for a method that first consumes all the
frames before producing any output, FreeNoise has by far the largest latency,
which makes it unusable for the live streaming scenario. Only StreamDiffusion
has a better latency than our method, which we attribute to it making a
different tradeoff between performance and quality. This is confirmed by the
MSE , CLIP scores, warp error, as well as the user study results, that
consistently indicate higher output quality for our method.

Method | Structure Consistency | Temporal Smoothness | Latency
---|---|---|---
Depth | Ours | CLIP | Warp | Ours
$\downarrow$ MSE $\downarrow$ |  $\downarrow$ Win Rate $\uparrow$ |  $\uparrow$ Score $\uparrow$ |  $\downarrow$ Error $\downarrow$ |  $\downarrow$ Win Rate $\uparrow$ |
StreamDiffusion | 1.72 | 70.4% | 94.35 | 0.1994 | 79.1% |  0.03s ($\pm 0.01$)
Rerender | 5.50 | 86.7% | 95.05 | 0.1327 | 86.7% | 7.72s ($\pm 0.18$)
FreeNoise | 1.94 | 80.5% | 96.49 | 0.0809 | 80.5% | 58.67s ($\pm 0.08$)
Ours | 1.12 | - | 95.77 | 0.0967 | - |  0.06s ($\pm 0.02$)
Table 1:  To compare our method to previous work, we averaged scores over 90
sequences from the DAVIS-2017 pont2017davis dataset. Our method scores highest
in Depth MSE and second in terms of temporal smoothness. However, because
FreeNoise is actually unable to produce output frames before having seen a
number of future input frames, we had to give it an unfair advantage by having
it consume _all_ input frames before producing its first output frame, leading
to extreme latency and explaining why it can achieve better temporal
smoothness than all other methods. More details of the metrics in Section 6.1.
Our user study win rates confirm that our method produces the best quality for
both aspects (i.e. all win rates over 50%). Only StreamDiffusion, which puts
more emphasis on speed than on output quality (see also Fig. 5) can beat our
method in terms of latency.  Figure 5:  We compare the output quality of our
method to a number of previous approaches: (a) shows temporally adjacent
frames, while (b) shows frames temporally further apart. While our method
preserves the spatial structure of the input well, producing the desired
output styles, previous methods tend to change even the semantic content of
the frames. See more discussions in Section 6.2 Figure 6: In this ablation
study, model (a) was trained with an attention mask like in Fig. 1 (c). Models
(b) and (c) were trained with the attention masks like in Fig. 1 (d), but in
(b) we filled the warmup slots (see Fig. 4) with further close-by predecessor
frames instead of initial frames of the stream. Only (d), our full method,
uses the depth prior. The models mostly agree on Frame 1, but all ablated
versions deviate from the spatial structure of the input for later frames.
More analysis in Section 6.3.

###  6.3 Ablation Study

We include quality and quantity results of model with different setting in
Fig. 6 and Table 2. We employ a noise strength of 0.5 for more apparent
comparison. The model (a), which uses uni-directional attention (see Fig. 1
(c)) fails to be consistent with the input from the first frame. Columns (b),
(c) and (d) are trained with our uni-directional attention with warmup (red
square of Fig. 1 (d)). But the model in (b) fills the warmup area with further
predecessor frames instead of initial frames (see also Fig. 4). This does
improve the output for the first frame (as the predecessor frames happen to be
initial), but leads to deviation from the spatial structure of the input in
later frames.

Setting | Structure Consistency | Temporal Smoothness
---|---|---
|  Train with warmup |  Inference with warmup |  Use depth | Depth MSE $\downarrow$ | CLIP Score $\uparrow$ | Warp error $\downarrow$
A | $\times$ | $\times$ | $\times$ | 2.29 | 95.43 | 0.0968
B | ✓ | $\times$ | $\times$ | 2.39 | 95.28 | 0.1125
C | ✓ | ✓ | $\times$ | 2.22 | 95.80 | 0.0966
D | ✓ | ✓ | ✓ | 1.67 | 95.78 | 0.0768
Table 2:  Ablation study of the model design. Our full method (D) achieves the
optimal in both structure consistency and temporal smoothness warp error. As
is to be expected, training with warmup, but filling the warmup area with
immediate predecessor frames at test time (B) makes quality worse, but using
the warmup area correctly (C) does lead to slight improvements over no warmup
at all. The depth prior leads to a strong improvement again (D).

In column (c) we do use the warmup area properly, but omit the depth prior.
The identity of the subject is now maintained better, but several details in
the background, such as the highlighted table region are still inconsistent
with the input. Only our full method (column (d)) maintains consistency beyond
initial frames. Table 2 confirms these findings: Without the depth prior,
configurations A, B and C fail to be structurally consistent with the input.
And with further predecessor frames instead of initial frames in the warmup
area at inference time, configuration B does not achieve as much temporal
consistency as the others. We also found that removing the warmup cache from
configuration D will decrease the temporal smoothness CLIP score by $0.09$.
The depth prior seems to improve both structure consistency and temporal
smoothness a lot, although the temporal CLIP score fails to show that in Table
2. We interpret this failure as a consequence of the content of subsequent
frames being usually very similar, such that the CLIP embeddings can be
similar despite certain abrupt changes, for example in the background, being
present.

As reported in Table 3, omitting our $K{}V{}$-cache leads to our method having
to re-compute the $K$ and $V$ maps of previous frames multiple times, which
dramatically increases the per-frame latency to a degree that is not
acceptable in streaming use cases.

Method | Latency $\downarrow$
---|---
wo $K{}V{}$-cache | 20.43 ($\pm 0.068$)
Ours | 0.06s ($\pm 0.002$)
Table 3:  Removing the $K{}V{}$-cache from our method drastically increases
latency.

##  7 Conclusion

We have presented Live2Diff, a method to translate video streams to a desired
target style at interactive framerates. Based on our novel unidirectional
attention approach, that allows us to reduce computational cost by means of
our $K{}V{}$-cache, we are able to not only meet the criterion of sufficient
framerate, but also outperform previous approaches in terms of consistency
with the input video and temporal smoothness. We have thus demonstrated that
the unidirectional temporal attention mode, that is an important component of
state of the art LLMs, can beneficially be used for the editing of videos as
well. A method like ours could be of great use in a number of video streaming
use cases, such as in the recent trend of “Virtual YouTubers”, in which online
content producers control stylized virtual avatars and interact with their
audience in a live stream.

##  Appendix A Appendix / Supplemental material

###  A.1 Limitations & Societal impact

Although our method improves upon the state of the art, we sometimes observe
flickering in background regions, especially when the camera is moving fast,
which we hypothesize to be the result of imperfect depth estimation. Also,
editing the video by applying Gaussian noise and then denoising it limits our
method to translation tasks in which the output is structurally similar to the
input, ruling out, for example, Pose-to-Person conditioning.

Like other editing works our method could in principle be misused to
manipulate video data, which weakens the value of such data as a means of
verifying that an event in question actually took place. Because of said
limitation to structure-preserving transfers this danger is somewhat smaller
than for other methods, but given the right kind of target style it could
still be used for example to alter the identity of depicted persons.

###  A.2 Implementation details of our inference pipeline

In this section, we provide the implementation of our temporal self-attention
module with $K{}V{}$-cache. We also describe how we apply streaming inference.

$K{}V{}$-cache. In model initialization, we pre-compute the shape of the
$K{}V{}$-cache for each temporal self-attention module. For temporal attention
with max window size $L$, and input feature size $H\times W\times C$, for $T$
denoising steps, the shape of the $K{}V{}$-cache should be $(T,H\times
W,L,C)$, see LABEL:lst:cachesize.

Listing 1: $K{}V{}$-cache sizing

⬇

def set_cache(T, H, W, L, C):

k_cache = zeros(T, H * W, L, C)

v_cache = zeros(T, H * W, L, C)

register_buffer("k_cache", k_cache)

register_buffer("v_cache", v_cache)

Previous video diffusion modelsguo2023animatediff; chen2024videocrafter2;
wang2023modelscope apply absolute positional encoding $\mathtt{PE}$, which is
added to the input features before the mapping layers
$\mathtt{to\\_q},\mathtt{to\\_k},\mathtt{to\\_v}$, which can be formulated as

$\displaystyle\mathtt{Q}$
$\displaystyle=\mathtt{to\\_q}(\mathtt{PE}+\mathtt{feat})$
$\displaystyle\mathtt{K}$
$\displaystyle=\mathtt{to\\_k}(\mathtt{PE}+\mathtt{feat})$
$\displaystyle\mathtt{V}$
$\displaystyle=\mathtt{to\\_v}(\mathtt{PE}+\mathtt{feat})$

We thus cannot directly cache $\mathtt{K{},V{}}$ since they contain positional
information. Instead, we pre-compute
$\mathtt{to\\_q}(\mathtt{PE}),\mathtt{to\\_k}(\mathtt{PE}),\mathtt{to\\_v}(\mathtt{PE})$
(see LABEL:lst:pecomp), and cache only
$\mathtt{to\\_k}(\mathtt{feat}),\mathtt{to\\_v}(\mathtt{feat})$.

Listing 2: We precompute
$\mathtt{to\\_k}(\mathtt{PE}),\mathtt{to\\_v}(\mathtt{PE})$.

⬇

def prepare_pe_buffer():

pe_full = pos_encoder.pe

q_pe = F.linear(pe_full, to_q.weight)

k_pe = F.linear(pe_full, to_k.weight)

v_pe = F.linear(pe_full, to_v.weight)

register_buffer("q_pe", q_pe)

register_buffer("k_pe", k_pe)

register_buffer("v_pe", v_pe)

In the warmup stage, we use bi-directional attention over all warmup frames,
and cache their $\mathtt{K/V}$, see LABEL:lst:warmup.

Listing 3: Warmup frames are processed with _bi_ -directional attention.

⬇

def temporal_self_attn_warmup(feat, timestep):

"""

feat: [HW, L, C_in]

"""

q = to_q(feat) # [HW, L, C]

k = to_k(feat) # [HW, L, C]

v = to_v(feat) # [HW, L, C]

# cache warmup frames before positional encoding

k_cache[timestep, :, :warmup_size] = k

v_cache[timestep, :, :warmup_size] = v

pe_idx = list(range(k.shape[1]))

pe_q = q_pe[:, pe_idx]

pe_k = k_pe[:, pe_idx]

pe_v = v_pe[:, pe_idx]

q_full = q + pe_q

k_full = k + pe_k

v_full = v + pe_v

# do not use attention mask

feat = scaled_dot_product_attention(

q_full,

key_full,

value_full,

attention_mask=None)

feat = to_out(feat)

return feat

During streaming inference we process up to $T$ samples with different noise
levels at once. For each frame we write to and read from the $K{}V{}$-cache
corresponding to its noise level, and add the mapped positional information.
At the beginning of the stream, the length of context window is incrementally
approaching the max window size $L$. We pass an attention mask to specify
which token should take part in attention. For details see LABEL:lst:attention
and LABEL:lst:streaming.

Listing 4: Our implementation of streaming inference uses the uni-directional
attention approach, see Fig. 1 (d).

⬇

def temporal_self_attn_streaming(feat, attn_mask):

"""

feat: [THW, L, C_in]

attn_mask: [T, L], 0 for attention, -inf for no attention

"""

q_layer = rearrange(q_layer, "(nhw) f c -> n hw f c", n=T)

k_layer = rearrange(k_layer, "(nhw) f c -> n hw f c", n=T)

v_layer = rearrange(v_layer, "(nhw) f c -> n hw f c", n=T)

# handle prev frames, roll back

k_cache[:, :, warmup_size:] = k_cache[:, :, warmup_size:] \

.roll(shifts=-1, dims=2)

v_cache[:, :, warmup_size:] = v_cache[:, :, warmup_size:] \

.roll(shifts=-1, dims=2)

# write curr frame

k_cache[:, :, -1:] = k_layer

v_cache[:, :, -1:] = v_layer

k_full = k_cache

v_full = v_cache

# attn_mask:

# [[0, 0, 0, 0, -inf, -inf, 0, 0],

# [0, 0, 0, 0, -inf, -inf, -inf, 0]]

# then pe for each element shoule be

# [[0, 1, 2, 3, 3, 3, 4, 5],

# [0, 1, 2, 3, 3, 3, 3, 4]]

kv_idx = (attn_mask == 0).cumsum(dim=1) - 1 # [T, L]

q_idx = kv_idx[:, -q_layer.shape[2]:] # [T, 1]

# [n, window_size, c]

pe_k = concatenate([

k_pe.index_select(1, kv_idx[idx])

for idx in range(T)], dim=0)

pe_v = concatenate([

v_pe.index_select(1, kv_idx[idx])

for idx in range(T)], dim=0)

pe_q = concatenate([

q_pe.index_select(1, q_idx[idx])

for idx in range(T)], dim=0)

q_layer = q_layer + pe_q.unsqueeze(1)

k_full = k_full + pe_k.unsqueeze(1)

v_full = v_full + pe_v.unsqueeze(1)

q_layer = rearrange(q_layer, "n hw f c -> (n hw) f c")

k_full = rearrange(k_full, "n hw f c -> (n hw) f c")

v_full = rearrange(v_full, "n hw f c -> (n hw) f c")

attn_mask_ = attn_mask[:, None, None, :].repeat(

1, h * w, q_layer.shape[1], 1)

attn_mask_ = rearrange(attn_mask_, "n hw Q KV -> (n hw) Q KV")

attn_mask_ = attn_mask_.repeat_interleave(heads, dim=0)

feat = scaled_dot_product_attention(

q_full,

key_full,

value_full,

attention_mask=attention_mask_)

feat = to_out(feat)

return feat

Listing 5: During inference we process $T$ frames simultaneously to make full
use of GPU parallelization.

⬇

def streaming_v2v(frame):

"""

frame: [1, 3, H, W]

"""

latent = vae.encode(frame)

depth_latent = vae.encode(depth_detector(frame))

noisy_latent = add_noise(latent) # add noise based on SDEdit

if prev_latent is None:

prev_latent = randn([T-1, ch, h, w])

if attn_mask is None:

attn_mask = zeros(T, L)

attn_mask[:, :warmup_size] = 1

attn_mask[0, -1] = 1 # curr frame participate attention

attn_mask.masked_fill_(attn_mask == 0, float("-inf"))

latent_batch = concatenate([noisy_latent, prev_latent])

noise_pred = UNet(latent_batch, depth_latent,

t, text_embedding, attn_mask)

denoised_latent = scheduler.step(noise_pred, latent_batch, t)

out_latent = denoised_latent[-1]

prev_latent = denoised_latent[1:]

out_frame = vae.decode(out_latent)

return out_frame

Models | Trigger Words
---|---
Flat-2D Animerge 111https://civitai.com/models/35960/flat2danimerge | cartoon style
zaum 222https://civitai.com/models/16048/or-disco-elysium-style-lora | zaum, elysiumchar
vangogh 333https://civitai.com/models/91/van-gogh-diffusion | Starry Night by Van Gogh, lvngvncnt
Table 4: Community models used for evaluation. Each model captures a different
target style. Figure 7: In our user study, the participant is given triplets
of videos: The “Reference” is the input video, that videos $1$ and $2$ should
be structurally consistent with, in addition to being temporally smooth. For
each of these two aspects the user chooses which of the two videos fulfills
this aspect best.

###  A.3 Evaluation

Models for evaluation. We use three Dreambooth and LoRA settings for
evaluation, the model name and trigger words are shown in Table 4. For the
evaluation, we use the trigger word as the prefix of our prompt.

User study. Our user study involved 15 participants who collectively watched a
total of 93 video clips. The video clips were the same as those used in the
quantitative evaluation. Fig. 7 illustrates the user interface of our user
study system: Participants are shown the input video as reference, as well as
an output from our method and an output from one random baseline method. They
are asked to select the output with better temporal smoothness and structure
consistency to the input. For each baseline method, we compute the win rate of
our method as

$\mathtt{ours\\_win\\_rate}=1-\frac{\mathtt{baseline\\_voted}}{\mathtt{baseline\\_shown}}$
(3)

Data captioning. We caption the DAVIS dataset with CogVLMwang2023cogvlm, which
is a state-of-the-art visual language model. For each video clip, we feed the
middle frame together with the following prompt:

Please caption the given image. The caption should focus on the main object in
image and describe the motion of the object.

Figure 8:  Our method translates an input video stream (black boundaries) into
an output video stream (blue boundaries) that conforms to a desired target
style. Each prompt is composed of the caption obtained for the input video
(see Section A.3 followed by target style.

###  A.4 Application

Fig. 8 shows another application of our method, demonstrating its potential in
virtual-liver cases. We transfer the input videos to different styles at 10 -
15 FPS on an NVIDIA RTX 4090.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: We explicitly list contributions in the introduction.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: See Section A.1.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [N/A]

  14. Justification: All our claims and the evidence we give are empirical, see Section 6.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: Section 4 clearly states what our inputs and outputs are, what architectures from the literature we employ and how we modified them. The novel components of our approach are described in dedicated sections (unidirectional self attention in Section 4.3 and $K{}V{}$-cache in Section 4.4). Section 4.5 gives instruction for training our model. Section 6.1 describes how we obtained and preprocessed the evaluation dataset and defines the metrics we use for evaluation.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: The instructions we give in Section 4 are comprehensive. For critical parts of our novel components we are giving code in Section A.2.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: Section 4.5 gives instruction for training our model. Section 6.1 describes how we obtained and preprocessed the evaluation dataset and defines the metrics we use for evaluation.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [Yes]

  34. Justification: All numbers in Section 6 are annotated with standard deviations.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: Section 6 states which GPU types we computed timings on and Section 4.5 states what hardware we used for training our model.

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification:

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [Yes]

  49. Justification: See Section A.1

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [No]

  54. Justification: All the data we use in our work is coming from prior scientific publications, so we assume it has been curated. StableDiffusion, which our method is partially based on does include safeguards against inappropriate output images, but anyone with basic programming skills can disable them easily.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: All assets that we build on are parts of scientific publications, which we properly cited.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes]

  64. Justification: In terms of assets, we are providing critical code sections in Section A.2 and a description of our overall method. Readers are free to use them in their work provided they appropriately cite our article.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [Yes]

  69. Justification: Section A.3 gives this information.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [No]

  74. Justification: The only “research with human participants” that we conducted was our user study, in which subjects were not giving any personal information.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.