# Transfer Learning for Diffusion Models

Yidong Ouyang  School of Data Science, The Chinese University of Hong Kong,
Shenzhen Liyan Xie  School of Data Science, The Chinese University of Hong
Kong, Shenzhen Hongyuan Zha  School of Data Science, The Chinese University of
Hong Kong, Shenzhen Guang Cheng  Department of Statistics, University of
California, Los Angeles

###### Abstract

Diffusion models, a specific type of generative model, have achieved
unprecedented performance in recent years and consistently produce high-
quality synthetic samples. A critical prerequisite for their notable success
lies in the presence of a substantial number of training samples, which can be
impractical in real-world applications due to high collection costs or
associated risks. Consequently, various finetuning and regularization
approaches have been proposed to transfer knowledge from existing pre-trained
models to specific target domains with limited data. This paper introduces the
Transfer Guided Diffusion Process (TGDP), a novel approach distinct from
conventional finetuning and regularization methods. We prove that the optimal
diffusion model for the target domain integrates pre-trained diffusion models
on the source domain with additional guidance from a domain classifier. We
further extend TGDP to a conditional version for modeling the joint
distribution of data and its corresponding labels, together with two
additional regularization terms to enhance the model performance. We validate
the effectiveness of TGDP on Gaussian mixture simulations and on real
electrocardiogram (ECG) datasets.

##  1 Introduction

Diffusion models have achieved remarkable success in modeling data
distributions and generating various types of synthetic data, such as images
[13; 36; 17], videos [14], vision language [32; 33; 30], and time series [39].
However, their success heavily relies on the availability of a large number of
training samples. In real-world applications, acquiring ample samples for
specific tasks can be challenging due to the high costs associated with data
collection or labeling, or the potential risks involved. Therefore, an
important research question is how to effectively transfer knowledge from a
pre-trained generative model in the source domain (using existing large-scale
datasets) to a target domain (for specific tasks) where data is limited.

Training a generative model directly or finetuning a pre-trained generative
model on limited data from the target domain often results in significant
performance degradation due to overfitting and memorization. To address these
issues, numerous studies have proposed methods in generative domain
adaptation, including the GAN-based models [46; 45; 48; 1; 50; 28; 47; 9; 15;
43; 22; 49], diffusion-based model [25; 51; 44], etc. Specifically, approaches
using diffusion models can be divided into two categories: finetuning
lightweight adapters [25; 44] and finetuning with regularization [51].
Approaches involving finetuning lightweight adapters focus on adjusting only a
subset of parameters in a pre-trained model. The primary challenge here is
identifying which parameters to finetune. This process is typically heuristic
and requires preliminary experiments to identify the most efficient parameters
for adjustment. Additionally, the specific parameters to be finetuned can vary
across different neural network architectures. On the other hand, the
challenge in incorporating regularization during the finetuning process is the
heuristic design of the regularization term, which can significantly alter the
optimization landscape. We refer to Appendix A for a more detailed discussion
of existing literature.

In this work, we introduce a new approach, termed Transfer Guided Diffusion
Process (TGDP), to transfer knowledge in the source domain generative model to
the target domain with limited samples. Unlike finetuning-based methods that
primarily use the pre-trained model as an initialization point, TGDP leverages
the pre-trained model as a plug-and-play prior. We show that the score
function for the diffusion model on the target domain is the score function on
the source domain (which can be pre-trained) with additional guidance as shown
by Theorem 3.1 and Theorem 3.3. The guidance network is related to the density
ratio of the target and source domain data distributions. Consequently, we
convert the original optimization problem for a diffusion model on the target
domain into estimating the density ratio.

We utilize a domain classifier (binary classifier) along with samples from
both domains to efficiently estimate the density ratio. Furthermore, we
introduce two additional regularization terms for better training and
calibration of the guidance network. These regularization terms are equivalent
forms that the optimal guidance network should satisfy, ensuring they do not
alter the original optimization problem. We validate the effectiveness of our
approach through experiments on Gaussian mixture simulations and real
electrocardiogram (ECG) data. Under both fidelity and utility evaluation
criteria, TGDP consistently outperforms finetuning-based methods.

Our contributions can be summarized as follows.

  * •

We introduce a new framework, the Transfer Guided Diffusion Process (TGDP),
for transferring a pre-trained diffusion model from the source domain to the
target domain.

  * •

We extend TGDP to a conditional version for modeling the joint distribution of
data and its corresponding labels, along with two additional regularization
terms, which are important for practical applications and downstream tasks.

  * •

TGDP demonstrates superior performance over finetuning-based methods on
Gaussian mixture simulations and on benchmark electrocardiogram (ECG) data.

The rest of the paper is organized as follows. Section 2 reviews the setup of
generative domain adaptation and the diffusion model. Section 3 introduces the
proposed method and theoretically characterizes its effectiveness. Numerical
results are given in Section 4. We conclude the paper in Section 5. All proofs
and additional numerical experiments are deferred to the Appendix.

##  2 Problem Formulation and Preliminaries

###  2.1 Transfer Learning Problem Setup

Let $\mathcal{X}$ denote the data space and $\mathcal{Y}$ the label space. A
domain corresponds to a joint distribution over $\mathcal{X}$ and
$\mathcal{Y}$, denoted as $p_{XY}$ for the source domain and $q_{XY}$ for the
target domain. The marginal distribution of data in the source and target
domains are $p_{X}$ and $q_{X}$, respectively. Suppose we have access to $m$
(labeled) samples from the source domain
$\mathcal{S}=\left\\{\left(\mathbf{x}_{i},y_{i}\right)\right\\}_{i=1}^{m}\sim
p_{XY}$ and $n$ (labeled) samples from the target domain
$\mathcal{T}=\left\\{\left(\mathbf{x}^{\prime}_{i},y_{i}^{\prime}\right)\right\\}_{i=1}^{n}\sim
q_{XY}$. Typically, the source domain contains significantly more samples than
the target domain, i.e., $n\ll m$. This setup reflects the common scenario
where there is limited data available for specific tasks in the target domain,
while abundant data is readily accessible and stored in the source domain.

The problem of interest is as follows. Given a pre-trained generative model
$p_{\theta}$ for the data distribution $p_{X}$ in the source domain, and a
relatively small number of samples from the target domain, generative domain
adaptation approaches aim to obtain a generative model that can generate
synthetic samples following the target data distribution $q_{X}$. We will
focus on diffusion generative models, given their great success in synthetic
data generation. We first present the key idea of a carefully designed
guidance network for the generation of $\mathbf{x}$ values only. Then, we
extend the method to facilitate conditional generations so that we can
generate paired samples with labels, $(\mathbf{x},y)$, and can incorporate
downstream classification tasks on the target domain.

###  2.2 Preliminaries of Diffusion Model

Diffusion models are characterized by their forward and backward processes.
For illustrative purposes, we discuss the diffusion model trained on the
source domain. The forward process involves perturbing the data distribution
$p_{X}(\mathbf{x})$ by injecting Gaussian noise, as described by the following
continuous-time equation [36]:

$\mathrm{d}\mathbf{x}_{t}=\mathbf{f}(\mathbf{x}_{t},t)\mathrm{d}t+g(t)\mathrm{d}\mathbf{w},\
t\in[0,T],$ (1)

where $\mathbf{w}$ is the standard Brownian motion,
$\mathbf{f}(\cdot,t):\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ is a drift
coefficient, and $g(\cdot):\mathbb{R}\rightarrow\mathbb{R}$ is a diffusion
coefficient. The marginal distribution of $\mathbf{x}_{t}$ at time $t$ is
denoted as $p_{t}(\mathbf{x}_{t})$, and $p_{0}$ is the distribution of the
initial value $\mathbf{x}_{0}$, which equals the true data distribution
$p_{X}(\mathbf{x})$. For notational simplicity and provided it does not cause
further confusion, we will refer to this diffusion process as $p$ in the
following, and we define $p(\mathbf{x}_{t}|\mathbf{x}_{s})$, $\forall s,t$, as
the conditional distribution of $\mathbf{x}_{t}$ given the value
$\mathbf{x}_{s}$. Similarly, for initial value $\mathbf{x}$ following the
target domain distribution, we denote the corresponding probability measure
induced by the above diffusion process (1) as $q$.

Then, we can reverse the forward process (1) for generation, defined as:

$\mathrm{d}\mathbf{x}_{t}=\left[\mathbf{f}(\mathbf{x}_{t},t)-g(t)^{2}\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x})\right]\mathrm{d}t+g(t)\mathrm{d}\overline{\mathbf{w}},$ (2)

where $\overline{\mathbf{w}}$ is a standard Brownian motion when time flows
backwards from $T$ to 0, and $\mathrm{d}t$ is an infinitesimal negative time
step. The key of the backward process is to estimate the score function of
each marginal distribution, $\nabla_{\mathbf{x}}\log p_{t}(\mathbf{x})$, then
the generation can be performed by discretizations of (2) [13; 36]. Score
Matching [16; 40; 35] are proposed to train a neural network
$\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)$ (parameterized by
$\boldsymbol{\phi}$) to estimate the score:

$\boldsymbol{\phi}^{*}=\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{p_{t}(\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})\right\|_{2}^{2}\right]\right\\},$ (3)

where $\lambda(t):[0,T]\rightarrow\mathbb{R}_{>0}$ is a positive weighting
function, $t$ is uniformly sampled over $[0,T]$. One commonly adopted forward
process is choosing an affine
$\mathbf{f}(\mathbf{x},t)=-\frac{1}{2}\beta(t)\mathbf{x}$ and
$g(t)=\sqrt{\beta(t)}$, which yields the Gaussian transition distribution
$p(\mathbf{x}_{t}|\mathbf{x}_{s})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta(t)}\mathbf{x}_{s},\beta(t)\mathbb{I})$,
$t>s$, with $\beta(t):[0,T]\rightarrow(0,1)$ as a variance schedule. This is
the Variance Preserving Stochastic Differential Equation (VP SDE) that we use
in the numerical Section 4.

Several works on image generation [4; 5] and inverse problem [7] extends Score
Matching to Conditional Score Matching, i.e.,

$\boldsymbol{\phi}^{*}=\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{p_{t}(\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y)\right\|_{2}^{2}\right]\right\\},$ (4)

where $p_{t}(\mathbf{x}_{t}|y)$ is the conditional distribution of perturbed
data $\mathbf{x}_{t}$ given corresponding label $y$.

##  3 Transfer Guided Diffusion Process

In this section, we introduce the proposed Transfer Guided Diffusion Process
(TGDP) that leverages a pre-trained diffusion model – trained on the source
domain data – to generate data in the target domain. The proposed approach is
orthogonal to and different from the existing fine-tuning type methods. We
introduce the additional guidance in Section 3.1. The methods for calculating
the guidance are provided in Section 3.2. We extend our framework to the
conditional diffusion model in Section 3.3 and we propose two regularization
terms for enhancing the performance of our method in Section 3.4. All proofs
are deferred to Appendix B.

###  3.1 Methodology Formulation

This subsection outlines the process of transferring knowledge from a
diffusion generative model pre-trained using the source domain data
$\mathcal{S}$ for generating samples that match the underlying distribution of
target domain sample $\mathcal{T}$. The simplest non-transfer type approach
involves directly training a diffusion model on samples $\mathcal{T}$ from the
target domain by denoising Score Matching as described by Eq (3) or Eq (4).
However, since we assume only a limited amount of data is accessible on the
target domain, directly learning from the target domain is unlikely to yield
an effective generative model.

Several studies propose to finetune the pre-trained diffusion model to
alleviate the challenges caused by limited data and make use of acquired
knowledge [25; 42; 52]. These methods typically design different strategies,
such as adapters, to avoid finetuning all weights in a pre-trained model.
However, these approaches generally use the pre-trained diffusion model from
the source domain only as initial weights. Our method offers a different way
for better utilization of the acquired knowledge.

Our proposed method is inspired by the key observation detailed in the
following Theorem 3.1. Intuitively, the score function
$\nabla_{\mathbf{x}_{t}}\log q_{t}(\mathbf{x}_{t})$ for the target domain
differs from the score function $\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})$ of the source domain by a term related to the density
ratio function ${q_{X}}/{p_{X}}$. We refer to this differing term as a
guidance term in the following Theorem.

######  Theorem 3.1.

Consider two diffusion models on the source and target domain, denoted as $p$
and $q$, respectively. Let the forward process on the target domain be
identical to that on the source domain,
$q(\mathbf{x}_{t}|\mathbf{x}_{0})=p(\mathbf{x}_{t}|\mathbf{x}_{0})$, and
$\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},t)$ is the score estimator
in the target domain:

$\boldsymbol{\phi}^{*}=\underset{\boldsymbol{\phi}}{\arg\min}~{}\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q_{t}(\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
q_{t}(\mathbf{x}_{t})\right\|_{2}^{2}\right]\right\\},$ (5)

then we have

$\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},t)=\underbrace{\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})}_{\begin{subarray}{c}\text{pre-trained model}\\\
\text{on
source}\end{subarray}}+\underbrace{\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}_{\text{guidance}}.$
(6)

Based on Eq (6), instead of solving $\mathbf{s}_{\boldsymbol{\phi}^{*}}$ from
the limited training samples on the target domain, we construct
$\mathbf{s}_{\boldsymbol{\phi}^{*}}$ by combing the pre-training score
estimator and the guidance based on a binary classifier of source and target
domain samples (detailed in Section 3.2). We comment on some potential
advantages of this simple yet effective idea. First of all, we do not need to
fine-tune the pre-trained diffusion model on the source domain, with the
corresponding computation shifted to training the guidance network which is
essentially a classifier. Second, the guidance network can be effectively
estimated by a domain classifier using data from both the source and target
domains. There is also great flexibility in constructing this guidance network
due to the extensive literature on classification problems and density ratio
estimation approaches. Additionally, the sample complexity for training a
generative model could be much larger than a discriminative model, since the
generative model needs to recover the full spectrum of target data
distribution, while a domain classifier only needs to distinguish whether the
sample is from the source or target distribution.

###  3.2 Learning Guidance Network

We calculate the guidance for the diffusion model on the target domain as
defined in the second term of Eq (6) via two steps. In the first step, we
estimate the density ratio ${q(\mathbf{x}_{0})}/{p(\mathbf{x}_{0})}$ by
training a classifier
$c_{\boldsymbol{\omega}}(\mathbf{x}):\mathcal{X}\to[0,1]$ to distinguish
samples from the source and target domains. We adopt the typical logistic loss
as follows:

$\boldsymbol{\omega}^{*}=\underset{\boldsymbol{\omega}}{\arg\min}\left\\{-\frac{1}{m}\sum_{\mathbf{x}_{i}\sim
p}\log
c_{\boldsymbol{\omega}}(\mathbf{x}_{i})-\frac{1}{n}\sum_{\mathbf{x}^{\prime}_{i}\sim
q}\log(1-c_{\boldsymbol{\omega}}(\mathbf{x}^{\prime}_{i}))\right\\}.$ (7)

Then, the density ratio ${q(\mathbf{x}_{0})}/{p(\mathbf{x}_{0})}$ can be
estimated as
${(1-c_{\boldsymbol{\omega}^{*}}(\mathbf{x}_{0}))}/{c_{\boldsymbol{\omega}^{*}}(\mathbf{x}_{0})}$,
and it can be shown that the optimal solution to the population counterpart of
Eq (7) is exactly the true likelihood ratio [38]. It is worthwhile mentioning
that we may only use a subset of source domain samples to learn the classifier
$c_{\boldsymbol{\omega}}$ to alleviate the unbalanced sample sizes, and we
could also adopt modern density ratio estimators to improve the accuracy [31].
After learning the density ratio ${q(\mathbf{x}_{0})}/{p(\mathbf{x}_{0})}$,
the second step is to calculate the expectation
$\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}[q(\mathbf{x}_{0})/p(\mathbf{x}_{0})]$
using Monte Carlo simulation. Since it is hard to sample from
$q(\mathbf{x}_{0}|\mathbf{x}_{t})$, we use the following equivalent
formulation to get the value instead. This trick has also been used in
previous work such as the Appendix H in [23].

######  Lemma 3.2.

For a neural network $h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)$
parameterized by $\boldsymbol{\psi}$, define the objective

$\mathcal{L}_{\text{guidance}}(\boldsymbol{\psi}):=\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\left\|h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)-\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right\|_{2}^{2}\right],$
(8)

then its minimizer
$\boldsymbol{\psi}^{*}=\underset{\boldsymbol{\psi}}{\arg\min}\
\mathcal{L}_{\text{guidance}}(\boldsymbol{\psi})$ satisfies:

$h_{\boldsymbol{\psi}^{*}}\left(\mathbf{x}_{t},t\right)=\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[{q(\mathbf{x}_{0})}/{p(\mathbf{x}_{0})}\right].$

By Lemma 3.2, we estimate the value
$\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[{q(\mathbf{x}_{0})}/{p(\mathbf{x}_{0})}\right]$
using the guidance network $h_{\boldsymbol{\psi}^{*}}$ solved by minimizing
the objective function $\mathcal{L}_{\text{guidance}}(\boldsymbol{\psi})$,
which can be approximated by easy sampling from the joint distribution
$p(\mathbf{x}_{0},\mathbf{x}_{t})$. Combine the above steps together, the
estimated score function for the diffusion generative model on target domain
$q_{X}$ can be calculated as follows:

$\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},t)=\underbrace{\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t})}_{\begin{subarray}{c}\text{pre-trained model}\\\ \text{on
source}\end{subarray}}+\underbrace{\nabla_{\mathbf{x}_{t}}\log
h_{\boldsymbol{\psi}^{*}}\left(\mathbf{x}_{t},t\right)}_{\text{guidance
network}}.$ (9)

###  3.3 Extension to the Conditional Version

The approach outlined above is for generating the sample $\mathbf{x}$ in the
target domain. In this section, we extend the idea to the conditional
generation task. Such extension is essential when the label sets in the source
and target domain are different since, in such cases, we usually rely on the
conditional diffusion model for sampling [18; 21]. We first present the
following theorem, which is an analog to Theorem 3.1 within the context of
conditional score matching.

######  Theorem 3.3.

Assume $\mathbf{x}_{t}$ and $y$ are conditional independent given
$\mathbf{x}_{0}$ in the forward process, i.e.,
$p(\mathbf{x}_{t}|\mathbf{x}_{0},y)=p(\mathbf{x}_{t}|\mathbf{x}_{0})$,
$\forall t\in[0,T]$, and let the forward process on the target domain be
identical to that on the source domain
$q(\mathbf{x}_{t}|\mathbf{x}_{0})=p(\mathbf{x}_{t}|\mathbf{x}_{0})$, and
$\boldsymbol{\phi}^{*}$ is the optimal solution for the conditional diffusion
model trained on target domain $q(\mathbf{x}_{0},y)$, i.e.,

$\boldsymbol{\phi}^{*}=\underset{\boldsymbol{\phi}}{\arg\min}~{}\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q_{t}(\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
q_{t}(\mathbf{x}_{t}|y)\right\|_{2}^{2}\right]\right\\},$ (10)

then

$\displaystyle\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},y,t)=\underbrace{\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y)}_{\begin{subarray}{c}\text{pre-trained conditional
model}\\\ \text{on
source}\end{subarray}}+\underbrace{\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}_{\text{conditional
guidance}}.$ (11)

The key difference is we need to estimate the joint density ratio between the
source and target domain. We can extend the density ratio estimator in Section
3.2 for estimating joint density ratio, i.e., also feed the label $y$ into the
classifier $c_{\boldsymbol{\omega}}(\mathbf{x},y)$.

###  3.4 Additional Regularizations in Practical Implementations

In this subsection, we provide two additional regularization terms in our
final objective function, to enhance the performance of the proposed scheme.

##### Cycle Regularization

In the approaches described above, after obtaining the classifier network
$c_{\boldsymbol{\omega}^{*}}$, calculation of the additional guidance
$\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}[{q(\mathbf{x}_{0})}/{p(\mathbf{x}_{0})}]$(or
$\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}[{q(\mathbf{x}_{0},y)}/{p(\mathbf{x}_{0},y)}]$
for conditional generation) only utilizes the data from source domain. In this
section, we provide an enhancement in which the limited data from the target
domain can also be utilized to improve the training of the guidance network
$h_{\boldsymbol{\psi}}$.

Notice that (with detailed derivation given in Appendix B.4)

$\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]=\mathbb{E}_{q(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q_{t}(\mathbf{x}_{t})}{p_{t}(\mathbf{x}_{t})}\right],$
(12)

where recall $p_{t}(\mathbf{x}_{t})$ and $q_{t}(\mathbf{x}_{t})$ are the
marginal distribution at time $t$ for source and target distributions,
respectively. A similar idea to Theorem 3.2 implies that we can learn the
guidance network by solving the following optimization problem as well:

$\boldsymbol{\psi}^{*}=\underset{\boldsymbol{\psi}}{\arg\min}\
\mathcal{L}_{\text{cycle}}:=\mathbb{E}_{q\left(\mathbf{x}_{0},\mathbf{x}_{t}\right)}\left[\left\|h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)-\frac{q_{t}(\mathbf{x}_{t})}{p_{t}(\mathbf{x}_{t})}\right\|_{2}^{2}\right].$
(13)

Moreover, in order to estimate the density ratio for marginal distributions at
time $t$ between the target and source data distribution, we train a time-
dependent classifier $c_{\boldsymbol{\omega}}(\mathbf{x},t)$ to distinguish
samples from source domain $p_{t}(\mathbf{x})$ and target domain
$q_{t}(\mathbf{x})$ by the logistic loss as follow:

$\boldsymbol{\omega}^{*}=\underset{\boldsymbol{\omega}}{\arg\min}\left\\{-\frac{1}{m}\sum_{\mathbf{x}_{0}\sim
p}\sum_{\mathbf{x}_{t}|\mathbf{x}_{0}}\log
c_{\boldsymbol{\omega}}(\mathbf{x}_{t},t)-\frac{1}{n}\sum_{\mathbf{x}_{0}\sim
q}\sum_{\mathbf{x}_{t}|\mathbf{x}_{0}}\log(1-c_{\boldsymbol{\omega}}(\mathbf{x}_{t},t))\right\\},$

where $m,n$ are the number of training samples in source and target domains.
The density ratio $q_{t}(\mathbf{x}_{t})/p_{t}(\mathbf{x}_{t})$ can then be
calculated by
$(1-c_{\boldsymbol{\omega}^{*}}(\mathbf{x}_{t},t))/(c_{\boldsymbol{\omega}^{*}}(\mathbf{x}_{t},t))$.

##### Consistency Regularization

Motivated by the fact that an optimal guidance network should recover the
score in the target domain, we further use score matching in the target domain
as the Consistency Regularization $\mathcal{L}_{\text{consistence}}$ to learn
the guidance network better.

$\displaystyle\boldsymbol{\psi}^{*}$
$\displaystyle=\underset{\boldsymbol{\psi}}{\arg\min}\
\mathcal{L}_{\text{consistence}}$
$\displaystyle:=\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q(\mathbf{x}_{0})}\mathbb{E}_{q(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})+\nabla_{\mathbf{x}_{t}}\log
h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)-\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\right]\right\\}.$ (14)

Combining these two additional regularization terms together with the original
guidance loss (8), the final learning objective for the guidance network can
be described as follows:

$\boldsymbol{\psi}^{*}=\underset{\boldsymbol{\psi}}{\arg\min}\
\\{\mathcal{L}_{\text{guidance}}+\eta_{1}\
\mathcal{L}_{\text{cycle}}+\eta_{2}\ \mathcal{L}_{\text{consistence}}\\},$
(15)

where $\eta_{1},\eta_{2}\geq 0$ are hyperparameters that control the strength
of additional regularization, which also enhances the flexibility of our
solution scheme. We summarize the Algorithm of TGDP in Appendix C.1. We
provide the ablation studies that demonstrate the effectiveness of these two
regularizations in Appendix C.2 and we also empirically show only adopt $\
\mathcal{L}_{\text{consistence}}$ to optimize the guidance network is not good
enough because of the limited data from the target distribution.

######  Remark 3.4 (Discussion about related guidance).

Classifier guidance has become a common trick in recent research [36; 8; 3;
6]. We highlight that, under the transfer learning framework, the guidance
proposed in our work is the optimal guidance since the resulting score
function matches the oracle score on the target domain. On the contrary,
vanilla versions of classifier guidance utilizing a domain classifier cannot
generate samples that exactly follow target distribution. Indeed, for a pre-
trained domain classifier $c_{\boldsymbol{\omega}}$, vanilla domain classifier
guidance formulates the source for generation as follows:

$\displaystyle\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},t)$
$\displaystyle=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t})-\nabla_{\mathbf{x}_{t}}\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\log(1-c_{\boldsymbol{\omega}}(\mathbf{x}_{0}))\right]$
$\displaystyle\neq\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t})+\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]\quad\text{(correct
form proven in Theorem \ref{thm:IS_guidance})}$
$\displaystyle=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t})+\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{1-c_{\boldsymbol{\omega}}(\mathbf{x}_{0}))}{c_{\boldsymbol{\omega}}(\mathbf{x}_{0})}\right].$

##  4 Experiments

In this section, we present empirical evidence demonstrating the efficacy of
the proposed Transfer Guided Diffusion Process (TGDP) on limited data from a
target domain. In Section 4.1, we conduct proof-of-concept experiments using a
Gaussian mixture model to showcase that the guidance network of TGDP can
successfully steer the pre-trained diffusion model toward the target domain.
In Section 4.2, we illustrate the effectiveness of TGDP using a real-world
electrocardiogram (ECG) dataset.

###  4.1 Simulation Results

##### Experimental setup

We begin with a Gaussian mixture model where $\mathcal{X}=\mathbb{R}^{d}$ and
$\mathcal{Y}=\\{-1,1\\}$. On both domains, the marginal distribution for label
$y$ is uniform in $\mathcal{Y}$, and the conditional distribution of features
is $\mathbf{x}|y\sim\mathcal{N}(y\boldsymbol{\mu},\sigma^{2}\mathbb{I}_{d})$,
where $\boldsymbol{\mu}\in\mathbb{R}^{d}$ is non-zero, and $\mathbb{I}_{d}$ is
the $d$ dimensional identity covariance matrix. We let
$\boldsymbol{\mu}=\boldsymbol{\mu}_{\rm S}$ on the source domain and
$\boldsymbol{\mu}=\boldsymbol{\mu}_{\rm T}$ on the target domain, with
$(\boldsymbol{\mu}_{\rm S})\top\boldsymbol{\mu}_{\rm T}=0$. Under such case,
the marginal distribution of $\mathbf{x}$ on the source domain $p_{X}$ is a
Gaussian mixture, for convenience we denote it as
$0.5\mathcal{N}({\boldsymbol{\mu}_{\rm
S}},\sigma^{2}\mathbb{I})+0.5\mathcal{N}(-{\boldsymbol{\mu}_{\rm
S}},\sigma^{2}\mathbb{I})$, and the marginal feature of target distribution
$q_{X}$ is $0.5\mathcal{N}({\boldsymbol{\mu}_{\rm
T}},\sigma^{2}\mathbb{I})+0.5\mathcal{N}(-{\boldsymbol{\mu}_{\rm
T}},\sigma^{2}\mathbb{I})$. We let $d=2,\boldsymbol{\mu}_{\rm
S}=[0.5,0.5],\boldsymbol{\mu}_{\rm T}=[0.5,-0.5],\sigma^{2}=0.1$, and draw
$m=10000$ samples from source domain $p_{X}$, and $n=10$, $100$, $1000$
samples from target domain $q_{X}$, respectively.

##### Implementation details and Baselines

We adopt the default Variance Preserving (VP) SDE in [36] with a linear
schedule, i.e.,
$q(\mathbf{x}_{t}|\mathbf{x}_{0})=p(\mathbf{x}_{t}|\mathbf{x}_{0})=\mathcal{N}\left(\mathbf{x}_{t}|\alpha_{t}\mathbf{x}_{0},\sigma_{t}^{2}\boldsymbol{I}\right)$
with $\alpha_{t}$ and $\sigma_{t}$ being:

$\alpha_{t}=-\frac{\beta_{1}-\beta_{0}}{4}t^{2}-\frac{\beta_{0}}{2}t,\quad\sigma_{t}=\sqrt{1-\alpha_{t}^{2}},$

with $\beta_{0}=0.1,\beta_{1}=20$. We adopt 5-layer MLP with hidden sizes of
$[512,512,512,512,256]$ and SiLU activation function as the diffusion model.
We train the diffusion model on data from the source domain for 100 epochs
using the Adam optimizer with a learning rate of $1\mathrm{e}^{-4}$ and batch
size of 4096. The guidance network is a 4-layer MLP with 512 hidden units and
SiLU activation function. We train the guidance network 20 epochs for our TGDP
and train a vanilla diffusion model or finetune the diffusion model target
domain 50 epochs. For generation, we adopt DPM-Solver [24] with a second-order
sampler and a diffusion step of 25. We compare TGDP with the following
baseline methods: 1) Vanilla Diffusion: directly training from target domain;
2) Finetune Diffusion: finetuning all weights of a pre-trained diffusion model
on target distribution 111It is worthwhile mentioning that the reason we do
not compared with the works that finetune partial weights in a pre-trained
diffusion model [44] is their results are usually worse or comparable with
method that finetunes all weights, the implementation of [25] are not
available, and the regularization proposed by [51] is only valid for image
data..

##### Experimental results

We first demonstrate the effectiveness of guidance in Figure 1 under the above
setup. Figure 1 (a) plots the source samples, while Figure 1 (b) shows the
target samples under different sample sizes $n=10$, $100$, $1000$. Figure 1
(c-e) illustrates the generated target samples via different methods,
respectively. It can be seen that the samples generated via the proposed TGDP
approach share similar patterns with the target distribution and two mixture
components are more obvious as compared with other baseline methods.
Furthermore, since the true data distribution of the target domain is known,
we calculate the average likelihood of samples generated by each method as
demonstrated in Table 1 for quantitative evaluation and comparison.

Figure 1: An illustration of the effectiveness of TGDP on simulations with 10/100/1000 target samples, respectively. Table 1: Quantitative evaluation of TGDP on simulations. Training on 10K samples from the source domain and $n=10,100,1000$ numbers on the target domain, respectively. TGDP achieves the highest average likelihood under target distribution. | Average likelihood
---|---
| n=10 | n=100 | n=1000
Vanilla Diffusion | 0.145 | 0.253 | 0.328
Finetune | 0.290 | 0.329 | 0.335
TGDP | 0.417 | 0.627 | 0.673

As a sanity check, we also look at the sensitivity of the learned density
ratio estimator (through the classifier network (7)) regarding different sizes
of target samples. As shown in Figure 2, even with only 10 samples from the
target domain (and 10 samples from the source domain for class balance
sampling), we can accurately estimate the landscape of density ratio (although
the magnitude of the estimated ratio is not entirely accurate when the number
of target samples equal 10).

(a) Oracle

(b) 10 target

(c) 100 target

(d) 1000 target

Figure 2: An ablation study of the sensitivity of density ratio estimator.

###  4.2 ECG Data

In this section, we demonstrate the effectiveness of the proposed guidance on
the benchmark of electrocardiogram (ECG) data. We first provide the standard
synthetic quality and diversity evaluation in Section 4.2.1. Then, we utilize
downstream classification tasks to further evaluate the effectiveness of TGDP
in Section 4.2.2. We follow the setup of existing benchmarks on biomedical
signal processing [37] that regard PTB-XL dataset [41] as the source domain
and ICBEB2018 dataset [27] as the target domain. PTB-XL dataset contains
21,837 clinical 12-lead ECG recordings of 10 seconds length from 18,885 unique
patients. Moreover, the PTB-XL dataset is a multi-label dataset with 71
different statements (label). ICBEB2018 dataset [27] comprises 6877 12-lead
ECGs lasting between 6 and 60 seconds. Each ECG record is categorized into one
of nine classes, which is a subset of labels in the PTB-XL dataset. We
randomly select 10% samples as limited target distribution by stratified
sampling preserving the overall label distribution in each fold following
[41]. We include more implementation details in Appendix C.3.

####  4.2.1 Synthetic Quality and Diversity Evaluation

##### Baseline method

We compare TGDP with the following baseline methods to demonstrate the
effectiveness of TGDP. 1) Learn a generative model directly (Vanilla
Diffusion): The vanilla way is to learn a generative model directly on limited
samples from the target domain. 2) Leveraging the pre-trained generative model
from source domain (Finetune Generator): Since the label set of the target
domain is a subset of that in the source domain, a preliminary solution is to
utilize the pre-trained diffusion model to generate samples with labels in the
target domain.

##### Experimental results

In Table 2, we compare the generation performance on the target domain using
two metrics. The first criterion is the widely used Frechet Inception Distance
(FID) [12] to evaluate the quality of synthetic data, which calculates the
Wasserstein-2 distance between the real data and the synthetic data on the
feature space. We use the pre-trained classifier on the target domain as the
feature extractor, i.e., xresnet1d50 [37]. The second metric is the coverage
[26] that evaluates the diversity of the synthetic data. It is defined as the
ratio of real records that have at least one fake (synthetic) record in its
sphere. The higher the coverage is, the more diverse the synthetic data are.

From Table 2, we see that TGDP achieves better performance than baseline
methods on two criteria, which demonstrates the effectiveness of TGDP on
generative transfer learning in scenarios with limited data. Moreover, TGDP
has fewer parameters to be trained and less training time. We also demonstrate
the T-SNE of the generated ECG data in Figure 3.

(a) Vanilla Diffusion

(b) Finetune Generator

(c) TGDP

(d) Real

Figure 3: T-SNE of the generated ECG data. Table 2: The effectiveness of TGDP on ECG benchmark under synthetic quality and diversity criteria. Method |  Diversity ($\uparrow$) |  FID ($\downarrow$) | Number of Parameters | Training Time
---|---|---|---|---
Vanilla Diffusion | 0.37 | 11.01 | 50.2M | 1h
Finetune Generator | 0.47 | 12.26 | 50.2M | 40min
TGDP | 0.53 | 10.46 | 2.8M | 30min

####  4.2.2 TGDP for Downstream Task

In Section 4.2.1, we illustrate that TGDP is capable of generating samples
that adhere to the joint distribution of data and labels in the target domain
and is diverse enough. In this subsection, we further investigate whether
utilizing TGDP to acquire a generative model for the target domain yields
superior performance compared to existing transfer learning pipelines.

##### Baseline method

First of all, we can utilize the generative model learned in Section 4.2.1 to
generate sufficient samples. Incorporated with the original limited sample
from the target domain, we can train the classifier, which we still denoted as
Vanilla Diffusion, Finetune Generator, and TGDP, respectively. Moreover, we
have the following baseline methods. Directly train a classifier on target
domain (Vanilla Classifier): Utilizing the limited data from the target
domain, a vanilla classifier can be obtained. Finetune pre-trained classifier
(Finetune Classifier): Instead of training a classifier from scratch on the
target domain, the parameters of the classifier trained on the source domain
are adjusted by using the limited data from the target domain.

##### Experimental results

We adopt the same evaluation criteria as ECG benchmark [37], i.e., Macro-
averaged area under the receiver operating characteristic curve (AUC), Macro-
averaged $F_{\beta}$-score $(\beta=2)$, where
$F_{\beta}=\frac{\left(1+\beta^{2}\right)\cdot\text{
TP}}{\left(1+\beta^{2}\right)\cdot\text{ TP }+\beta^{2}\cdot\text{ FN }+\text{
FP }}$, and Macro-averaged $G_{\beta}$-score with $\beta=2$, where
$G_{\beta}=$ $\frac{\text{ TP}}{\text{ TP}+\text{ FP }+\beta\cdot\text{ FN
}}$. In Table 3, TGDP outperforms baseline methods across three evaluation
criteria, showcasing its effectiveness in transfer for diffusion model with
limited data.

Table 3: The effectiveness of TGDP on ECG benchmark for downstream task. We provide 95% confidence intervals via empirical bootstrapping used by [37]. 0.906(03) stands for 0.906 $\pm$ 0.003. Method | AUC | $F_{\beta=2}$ | $G_{\beta=2}$
---|---|---|---
Vanilla Classifier | 0.906(03) | 0.674(06) | 0.433(06)
Finetune Classifier | 0.941(05) | 0.747(08) | 0.521(10)
Vanilla Diffusion | 0.932(05) | 0.718(09) | 0.464(09)
Finetune Generator | 0.941(04) | 0.761(10) | 0.528(12)
TGDP |  0.953(05) |  0.773(11) |  0.534(11)

##  5 Conclusion

In this work, we propose a novel framework, Transfer Guided Diffusion Process
(TGDP), for transferring a source-domain diffusion model to the target domain
which consists of limited data. Instead of reducing the finetuning parameters
or adding regularization for finetuning, TGDP proves the optimal diffusion
model on the target domain is the pre-trained diffusion model on the source
domain with additional guidance. TGDP outperforms existing methods on Gaussian
mixture simulations and electrocardiogram (ECG) data benchmarks.

##### Limitations and broader impact

Overall, this research presents a promising direction for leveraging pre-
trained diffusion models to tackle new tasks. The proposed method, TGDP, has
potential applications in a wide range of tasks where domain shift exists. A
limitation of this study is the lack of empirical validation regarding TGDP’s
performance on language vision tasks, which we have earmarked for future
exploration. Since we propose a generic algorithm for transferring knowledge
to new tasks, this technique could enable people to train Deepfakes for
disinformation better. Our approach hinges on the efficacy of detection
methods in mitigating negative societal consequences.

## References

  * [1] Aibek Alanov, Vadim Titov, and Dmitry P. Vetrov.  Hyperdomainnet: Universal domain adaptation for generative adversarial networks.  ArXiv, abs/2210.08884, 2022.
  * [2] Juan Miguel Lopez Alcaraz and Nils Strodthoff.  Diffusion-based conditional ecg generation with structured state space models.  Computers in biology and medicine, 163:107115, 2023.
  * [3] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 843–852, 2023.
  * [4] Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schonlieb, and Christian Etmann.  Conditional image generation with score-based diffusion models.  ArXiv, abs/2111.13606, 2021.
  * [5] Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun-Yi Lee.  Denoising likelihood score matching for conditional score-based data generation.  In ICLR, 2022.
  * [6] Hyungjin Chung, Jeongsol Kim, Michael T. McCann, Marc Louis Klasky, and J. C. Ye.  Diffusion posterior sampling for general noisy inverse problems.  ArXiv, abs/2209.14687, 2022.
  * [7] Agnimitra Dasgupta, Javier Murgoitio-Esandi, Deep Ray, and Assad Oberai.  Conditional score-based generative models for solving physics-based inverse problems.  In NIPS workshop, Deep Learning and Inverse Problems, 2023.
  * [8] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  ArXiv, abs/2105.05233, 2021.
  * [9] Yuxuan Duan, Li Niu, Y. Hong, and Liqing Zhang.  Weditgan: Few-shot image generation via latent space relocation.  ArXiv, abs/2305.06671, 2023.
  * [10] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, P. Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.  ArXiv, abs/2305.16381, 2023.
  * [11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  ArXiv, abs/2208.01618, 2022.
  * [12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In Neural Information Processing Systems, 2017.
  * [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [14] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.  Video diffusion models.  ArXiv, abs/2204.03458, 2022.
  * [15] Xingzhong Hou, B. Liu, Shuai Zhang, Lulin Shi, Zite Jiang, and Haihang You.  Dynamic weighted semantic correspondence for few-shot image generative adaptation.  Proceedings of the 30th ACM International Conference on Multimedia, 2022.
  * [16] Aapo Hyvärinen.  Estimation of non-normalized statistical models by score matching.  J. Mach. Learn. Res., 6:695–709, 2005.
  * [17] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  ArXiv, abs/2206.00364, 2022.
  * [18] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  ArXiv, abs/2303.13439, 2023.
  * [19] Nupur Kumari, Bin Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1931–1941, 2022.
  * [20] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, P. Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu.  Aligning text-to-image models using human feedback.  ArXiv, abs/2302.12192, 2023.
  * [21] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis L Brown, and Deepak Pathak.  Your diffusion model is secretly a zero-shot classifier.  ArXiv, abs/2303.16203, 2023.
  * [22] Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman.  Few-shot image generation with elastic weight consolidation.  ArXiv, abs/2012.02780, 2020.
  * [23] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu.  Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning.  In International Conference on Machine Learning, 2023.
  * [24] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  ArXiv, abs/2206.00927, 2022.
  * [25] Taehong Moon, Moonseok Choi, Gayoung Lee, Jung-Woo Ha, Juho Lee, AI Kaist, Naver AI Lab, and Aitrics.  Fine-tuning diffusion models with limited data.  2022\.
  * [26] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo.  Reliable fidelity and diversity metrics for generative models.  ArXiv, abs/2002.09797, 2020.
  * [27] Eddie Y. K. Ng, Feifei Liu, Chengyu Liu, Lina Zhao, X. Zhang, Xiaoling Wu, Xiaoyan Xu, Yulin Liu, Caiyun Ma, Shoushui Wei, Zhiqiang He, and Jianqing Li.  An open access database for evaluating the algorithms of electrocardiogram rhythm and morphology abnormality detection.  Journal of Medical Imaging and Health Informatics, 2018.
  * [28] Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10738–10747, 2021.
  * [29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, 2021.
  * [30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  ArXiv, abs/2204.06125, 2022.
  * [31] Benjamin Rhodes, Kai Xu, and Michael U. Gutmann.  Telescoping density-ratio estimation.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4905–4916. Curran Associates, Inc., 2020.
  * [32] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22500–22510, 2022.
  * [33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  ArXiv, abs/2205.11487, 2022.
  * [34] Kunpeng Song, Ligong Han, Bingchen Liu, Dimitris N. Metaxas, and A. Elgammal.  Diffusion guided domain adaptation of image generators.  ArXiv, abs/2212.04473, 2022.
  * [35] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon.  Sliced score matching: A scalable approach to density and score estimation.  In Conference on Uncertainty in Artificial Intelligence, 2019.
  * [36] Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  ArXiv, abs/2011.13456, 2021.
  * [37] Nils Strodthoff, Patrick Wagner, Tobias Schaeffter, and Wojciech Samek.  Deep learning for ecg analysis: Benchmarks and insights from ptb-xl.  IEEE Journal of Biomedical and Health Informatics, 25:1519–1528, 2020.
  * [38] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori.  Density Ratio Estimation in Machine Learning.  Cambridge University Press, 2012.
  * [39] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon.  CSDI: Conditional score-based diffusion models for probabilistic time series imputation.  ArXiv, abs/2107.03502, 2021.
  * [40] Pascal Vincent.  A connection between score matching and denoising autoencoders.  Neural Computation, 23:1661–1674, 2011.
  * [41] Patrick Wagner, Nils Strodthoff, Ralf Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter.  Ptb-xl, a large publicly available electrocardiography dataset.  Scientific Data, 7, 2020.
  * [42] Chendong Xiang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  A closer look at parameter-efficient tuning in diffusion models.  ArXiv, abs/2303.18181, 2023.
  * [43] Jiayu Xiao, Liang Li, Chaofei Wang, Zhengjun Zha, and Qingming Huang.  Few shot generative model adaption via relaxed spatial structural alignment.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11194–11203, 2022.
  * [44] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li.  Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning.  2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4207–4216, 2023.
  * [45] Ceyuan Yang, Yujun Shen, Zhiyi Zhang, Yinghao Xu, Jiapeng Zhu, Zhirong Wu, and Bolei Zhou.  One-shot generative domain adaptation.  2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7699–7708, 2021.
  * [46] Mengping Yang and Zhe Wang.  Image synthesis under limited data: A survey and taxonomy.  ArXiv, abs/2307.16879, 2023.
  * [47] Yabo Zhang, Mingshuai Yao, Yuxiang Wei, Zhilong Ji, Jinfeng Bai, and Wangmeng Zuo.  Towards diverse and faithful one-shot adaption of generative adversarial networks.  ArXiv, abs/2207.08736, 2022.
  * [48] Zicheng Zhang, Yinglu Liu, Congying Han, Tiande Guo, Ting Yao, and Tao Mei.  Generalized one-shot domain adaptation of generative adversarial networks.  In Neural Information Processing Systems, 2022.
  * [49] Miaoyun Zhao, Yulai Cong, and Lawrence Carin.  On leveraging pretrained gans for generation with limited data.  In International Conference on Machine Learning, 2020.
  * [50] Yunqing Zhao, Henghui Ding, Houjing Huang, and Ngai-Man Cheung.  A closer look at few-shot image generation.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9130–9140, 2022.
  * [51] Jin Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan.  Few-shot image generation with diffusion models.  ArXiv, abs/2211.03264, 2022.
  * [52] Jin Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan.  Domainstudio: Fine-tuning diffusion models for domain-driven image generation using limited data.  ArXiv, abs/2306.14153, 2023.

##  Appendix A More Discussion on Related Work

##### Finetune diffusion model on limited data

Directly finetuning the pre-trained generative model on limited data from the
target domain may suffer from overfitting and diversity degradation. Moon et.
al. [25] introduce a time-aware adapter inside the attention block. Since the
attention modules take about 10% of parameters in the entire diffusion model,
they significantly reduced the turning parameters and alleviated the
overfitting. While in [44], the authors only finetune specific parameters
related to bias, class embedding, normalization, and scale factor. Zhu et. al.
[51] found out the images generated by directly finetuned diffusion models
share similar features like facial expressions and lack ample high-frequency
details. Therefore, they introduce two regularization terms, pairwise
similarity loss for diversity and high-frequency components loss to enhance
the high-frequency feature.

The main drawback of finetuning the pre-trained diffusion model is the sample
complexity is larger compared with training a classifier since modeling the
distribution is a tougher task. In our work, we decompose the diffusion model
on the target domain as the diffusion model on the source domain plus a
guidance network. Since training a guidance network (essential as a classifier
demonstrated in section 3) requires smaller sample complexity, we believe this
novel framework might provide a new way for diffusion-based domain adaptation.

##### Text-to-image diffusion model and learning with human feedback

Numerous studies on Text-to-Image diffusion models focus on optimizing the
diffusion model to align with human preferences and personalize its
performance for specific tasks. These endeavors commonly involve strategies
such as text-guided zero-shot finetuning [34; 29] or finetuning diffusion
model (or its adaptor) through reward-weighted objectives [32; 19; 11; 20;
10]. We acknowledge the significant potential in these approaches, given that
language models inherently encapsulate rich semantic information, thereby
endowing text-to-image diffusion models with zero-shot transferability.
However, it is noteworthy that in domains lacking a substantial amount of
paired data for learning semantic mappings, such as biomedical signal
processing and electrocardiogram (ECG) data, we refrain from considering these
methods as the primary benchmarks in our comparative analysis.

##### Non-diffusion based approaches in generative domain adaptation

Numerous works in generative domain adaptation (or few-shot generative
adaptation) study how to improve the transferability of the generative model
on limited data from the target domain. Since we mainly focus on the diffusion
model, we summarize the primary GAN-based domain adaptation there. They mainly
propose to add different kinds of regularization to avoid model collapse [28;
50; 48; 47; 9; 15; 43] or finetune subset of the parameter (adaptor) [1; 45;
22; 49].

##  Appendix B Theoretical Details for Section 3

###  B.1 Proof of Theorem 3.1

###### Proof.

To prove Eq (6), we first build the connection between Score Matching on the
target domain and Importance Weighted Denoising Score Matching on the source
domain in the following Lemma.

######  Lemma B.1.

Score Matching on the target domain is equivalent to Importance Weighted
Denoising Score Matching on the source domain, i.e.,

$\displaystyle\boldsymbol{\phi}^{*}=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q_{t}(\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
q_{t}(\mathbf{x}_{t})\right\|_{2}^{2}\right]\right\\}$ (16) $\displaystyle=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{p(\mathbf{x}_{0})}\mathbb{E}_{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]\right\\}.$

###### Proof of Lemma B.1.

We first connect Score Matching objective in the target domain to Denoising
Score Matching objective in target distribution, which is proven by [40],
i.e.,

$\displaystyle\boldsymbol{\phi}^{*}=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q_{t}(\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
q_{t}(\mathbf{x}_{t})\right\|_{2}^{2}\right]\right\\}$ $\displaystyle=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q(\mathbf{x}_{0})}\mathbb{E}_{q(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\right]\right\\}.$

Then, we split the mean squared error of Denoising Score Matching objective on
target distribution into three terms as follows:

$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0})}\mathbb{E}_{q(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\right]$ $\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}\right]-2\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\right]+C_{1},$ (17)

where
$C_{1}=\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\left\|\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\right]$ is a constant
independent with $\boldsymbol{\phi}$. We can similarly split the objective
function in the right-hand side (RHS) of Eq (16) as follows:

$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0})}\mathbb{E}_{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]-2\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]+C_{2},$
(18)

where $C_{2}$ is a constant independent with $\boldsymbol{\phi}$. It is easy
to show that the first term in Eq (B.1) is equal to the first term in Eq
(B.1), i.e.,

$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}p(\mathbf{x}_{0})p(\mathbf{x}_{t}|\mathbf{x}_{0})\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}d\mathbf{x}_{t}$
$\displaystyle\overset{(i)}{=}$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}p(\mathbf{x}_{0})q(\mathbf{x}_{t}|\mathbf{x}_{0})\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}d\mathbf{x}_{t}$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}q(\mathbf{x}_{0},\mathbf{x}_{t})\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}d\mathbf{x}_{0}d\mathbf{x}_{t}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}\right],$

where the equality $(i)$ is due to
$q(\mathbf{x}_{t}|\mathbf{x}_{0})=p(\mathbf{x}_{t}|\mathbf{x}_{0})$.

Next, we prove the second terms in Eq (B.1) and Eq (B.1) are also equivalent:

$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}p(\mathbf{x}_{0},\mathbf{x}_{t})\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\frac{\nabla_{\mathbf{x}_{t}}p(\mathbf{x}_{t}|\mathbf{x}_{0})}{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\rangle\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}d\mathbf{x}_{t}$
$\displaystyle\overset{(i)}{=}$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}p(\mathbf{x}_{0})p(\mathbf{x}_{t}|\mathbf{x}_{0})\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\frac{\nabla_{\mathbf{x}_{t}}q(\mathbf{x}_{t}|\mathbf{x}_{0})}{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\rangle\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}d\mathbf{x}_{t}$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\nabla_{\mathbf{x}_{t}}q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle
q(\mathbf{x}_{0})d\mathbf{x}_{0}d\mathbf{x}_{t}$ $\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle
q(\mathbf{x}_{t}|\mathbf{x}_{0})q(\mathbf{x}_{0})d\mathbf{x}_{0}d\mathbf{x}_{t}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t),\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\right],$

where the equality $(i)$ is again due to
$q(\mathbf{x}_{t}|\mathbf{x}_{0})=p(\mathbf{x}_{t}|\mathbf{x}_{0})$. Thereby
we prove Eq 16. ∎

According to Lemma B.1,

$\boldsymbol{\phi}^{*}=\underset{\boldsymbol{\phi}}{\arg\min}~{}\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{p(\mathbf{x}_{0})}\mathbb{E}_{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},t)-\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]\right\\}.$

Based on this, we may use Importance Weighted Denoising Score Matching on the
source domain to get the analytic form of $\mathbf{s}_{\boldsymbol{\phi}^{*}}$
as follows:

$\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},t)=\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}.$

The RHS of Eq (6) can be rewritten as follows:

$\displaystyle\text{RHS}=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})+\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]=\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})+\frac{\nabla_{\mathbf{x}_{t}}\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}$
$\displaystyle=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})+\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{0}|\mathbf{x}_{t})\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}.$

Since

$\displaystyle\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{0}|\mathbf{x}_{t})$
$\displaystyle=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})+\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{0})-\nabla_{\mathbf{x}_{t}}\log p_{t}(\mathbf{x}_{t})$
$\displaystyle=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})-\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}),$

we can further rewrite the RHS of Eq (6) as follows:

$\displaystyle\text{RHS}=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})+\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}-\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t})$ $\displaystyle=$
$\displaystyle\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]}$
$\displaystyle=$
$\displaystyle\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},t).$

Thereby we complete the proof. ∎

###  B.2 Proof of Lemma 3.2

###### Proof.

The proof is straightforward and we include it below for completeness. Note
that the objective function can be rewritten as

$\displaystyle\mathcal{L}_{\text{guidance}}(\boldsymbol{\psi}):=$
$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t})}\left[\left\|h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)-\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right\|_{2}^{2}\right]$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{t}}\left\\{\int_{\mathbf{x}_{0}}p(\mathbf{x}_{0}|\mathbf{x}_{t})\left\|h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)-\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right\|_{2}^{2}d\mathbf{x}_{0}\right\\}p(\mathbf{x}_{t})d\mathbf{x}_{t}$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{t}}\left\\{\left\|h_{\boldsymbol{\psi}}(\mathbf{x}_{t},t)\right\|_{2}^{2}-2\langle
h_{\boldsymbol{\psi}}(\mathbf{x}_{t},t),\int_{\mathbf{x}_{0}}p(\mathbf{x}_{0}|\mathbf{x}_{t})\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}\rangle\right\\}p(\mathbf{x}_{t})d\mathbf{x}_{t}+C$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{t}}\left\|h_{\boldsymbol{\psi}}(\mathbf{x}_{t},t)-\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]\right\|_{2}^{2}p(\mathbf{x}_{t})d\mathbf{x}_{t},$

where $C$ is a constant independent of $\boldsymbol{\psi}$. Thus we have the
minimizer $\boldsymbol{\psi}^{*}=\underset{\boldsymbol{\psi}}{\arg\min}\
\mathcal{L}_{\text{guidance}}(\boldsymbol{\psi})$ satisfies
$h_{\boldsymbol{\psi}^{*}}\left(\mathbf{x}_{t},t\right)=\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[{q(\mathbf{x}_{0})}/{p(\mathbf{x}_{0})}\right]$.
∎

###  B.3 Proof of Theorem 3.3

###### Proof.

The proof is similar to the proof of Theorem 3.1. To prove Eq 11, we first
build the connection between the Conditional Score Matching on the target
domain and Importance Weighted Conditional Denoising Score Matching on the
source domain in the following Lemma:

######  Lemma B.2.

Conditional Score Matching on the target domain is equivalent to Importance
Weighted Denoising Score Matching on the source domain, i.e.,

$\displaystyle\boldsymbol{\phi}^{*}=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q_{t}(\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
q_{t}(\mathbf{x}_{t}|y)\right\|_{2}^{2}\right]\right\\}$ $\displaystyle=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{p(\mathbf{x}_{0},y)}\mathbb{E}_{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]\right\\}.$

###### Proof of Lemma B.2.

We first connect the Conditional Score Matching objective in the target domain
to the Conditional Denoising Score Matching objective in target distribution,
which is proven by [4, Theorem 1], i.e.,

$\displaystyle\boldsymbol{\phi}^{*}=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q_{t}(\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
q_{t}(\mathbf{x}_{t}|y)\right\|_{2}^{2}\right]\right\\}$ $\displaystyle=$
$\displaystyle\underset{\boldsymbol{\phi}}{\arg\min}\
\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{q(\mathbf{x}_{0},y)}\mathbb{E}_{q(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\right]\right\\}.$

Then we split the mean squared error of the Conditional Denoising Score
Matching objective on target distribution into three terms as follows:

$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0},y)}\mathbb{E}_{q(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\right]$ $\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)\right\|_{2}^{2}\right]-2\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\right]+C_{1},$ (19)

where
$C_{1}=\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\left\|\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\right]$ is a constant
independent with $\boldsymbol{\phi}$, and
$q(\mathbf{x}_{t}|\mathbf{x}_{0},y)=q(\mathbf{x}_{t}|\mathbf{x}_{0})$ because
of conditional independent of $\mathbf{x}_{t}$ and $y$ given $\mathbf{x}_{0}$
by assumption. We can similarly split the mean squared error of Denoising
Score Matching on the source domain into three terms as follows:

$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},y)}\mathbb{E}_{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]$
(20) $\displaystyle=$
$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]-2\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]$
$\displaystyle+C_{2},$

where $C_{2}$ is a constant independent with $\boldsymbol{\phi}$.

It is obvious to show that the first term in Eq (B.3) is equal to the first
term in Eq (20), i.e.,

$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\int_{y}p(\mathbf{x}_{0},y)p(\mathbf{x}_{t}|\mathbf{x}_{0})\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}d\mathbf{x}_{0}d\mathbf{x}_{t}dy$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\int_{y}p(\mathbf{x}_{0},y)q(\mathbf{x}_{t}|\mathbf{x}_{0})\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)\right\|_{2}^{2}\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}d\mathbf{x}_{0}d\mathbf{x}_{t}dy$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\int_{y}q(\mathbf{x}_{0},\mathbf{x}_{t},y)\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)\right\|_{2}^{2}d\mathbf{x}_{0}d\mathbf{x}_{t}dy$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)\right\|_{2}^{2}\right].$

And the second term is also equivalent:

$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\int_{y}p(\mathbf{x}_{0},\mathbf{x}_{t},y)\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\frac{\nabla_{\mathbf{x}_{t}}p(\mathbf{x}_{t}|\mathbf{x}_{0})}{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\rangle\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}d\mathbf{x}_{0}d\mathbf{x}_{t}dy$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\int_{y}p(\mathbf{x}_{0},\mathbf{x}_{t},y)\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\frac{\nabla_{\mathbf{x}_{t}}q(\mathbf{x}_{t}|\mathbf{x}_{0})}{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\rangle\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}d\mathbf{x}_{0}d\mathbf{x}_{t}dy$
$\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\int_{y}\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\nabla_{\mathbf{x}_{t}}q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle
q(\mathbf{x}_{0},y)d\mathbf{x}_{0}d\mathbf{x}_{t}dy$ $\displaystyle=$
$\displaystyle\int_{\mathbf{x}_{0}}\int_{\mathbf{x}_{t}}\int_{y}\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle
q(\mathbf{x}_{t}|\mathbf{x}_{0})q(\mathbf{x}_{0},y)d\mathbf{x}_{0}d\mathbf{x}_{t}dy$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0},\mathbf{x}_{t},y)}\left[\langle\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t),\nabla_{\mathbf{x}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}_{0})\rangle\right].$

∎

According to Lemma B.2, the optimal solution satisfies

$\boldsymbol{\phi}^{*}=\underset{\boldsymbol{\phi}}{\arg\min}~{}\mathbb{E}_{t}\left\\{\lambda(t)\mathbb{E}_{p(\mathbf{x}_{0},y)}\mathbb{E}_{p(\mathbf{x}_{t}|\mathbf{x}_{0})}\left[\left\|\mathbf{s}_{\boldsymbol{\phi}}(\mathbf{x}_{t},y,t)-\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right\|_{2}^{2}\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]\right\\},$

Then, we use Importance Weighted Conditional Denoising Score Matching on the
source domain to get the analytic form of $\mathbf{s}_{\boldsymbol{\phi}^{*}}$
as follows:

$\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},y,t)=\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}.$

Moreover, the RHS of Eq (11) can be rewritten as:

$\displaystyle\text{RHS}=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y)+\nabla_{\mathbf{x}_{t}}\log\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]$
$\displaystyle=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y)+\frac{\nabla_{\mathbf{x}_{t}}\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}$
$\displaystyle=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y)+\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{0}|\mathbf{x}_{t},y)\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}.$

Since

$\displaystyle\nabla_{\mathbf{x}_{t}}\log p(\mathbf{x}_{0}|\mathbf{x}_{t},y)$
$\displaystyle=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0},y)+\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{0}|y)-\nabla_{\mathbf{x}_{t}}\log p_{t}(\mathbf{x}_{t}|y)$
$\displaystyle=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0},y)-\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y),$ $\displaystyle=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})-\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y),$

we can further simplify the RHS of Eq (11) as follows:

$\displaystyle\text{RHS}=$ $\displaystyle\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y)+\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}-\nabla_{\mathbf{x}_{t}}\log
p_{t}(\mathbf{x}_{t}|y)$ $\displaystyle=$
$\displaystyle\frac{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t}|\mathbf{x}_{0})\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}{\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t},y)}\left[\frac{q(\mathbf{x}_{0},y)}{p(\mathbf{x}_{0},y)}\right]}$
$\displaystyle=$
$\displaystyle\mathbf{s}_{\boldsymbol{\phi}^{*}}(\mathbf{x}_{t},t).$

Thereby, we finish the proof. ∎

###  B.4 Proof for Cycle Regularization

###### Proof of Eq (12).

$\displaystyle\mathbb{E}_{p(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}\right]=$
$\displaystyle\int
p(\mathbf{x}_{0}|\mathbf{x}_{t})\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}=\int\frac{p(\mathbf{x}_{t}|\mathbf{x}_{0})p(\mathbf{x}_{0})}{p_{t}(\mathbf{x}_{t})}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}$
$\displaystyle=$
$\displaystyle\int\frac{q(\mathbf{x}_{t}|\mathbf{x}_{0})p(\mathbf{x}_{0})}{p_{t}(\mathbf{x}_{t})}\frac{q(\mathbf{x}_{0})}{p(\mathbf{x}_{0})}d\mathbf{x}_{0}=\int
q(\mathbf{x}_{t}|\mathbf{x}_{0})\frac{q(\mathbf{x}_{0})}{p_{t}(\mathbf{x}_{t})}d\mathbf{x}_{0}$
$\displaystyle=$
$\displaystyle\int\frac{q(\mathbf{x}_{0}|\mathbf{x}_{t})q_{t}(\mathbf{x}_{t})}{q(\mathbf{x}_{0})}\frac{q(\mathbf{x}_{0})}{p_{t}(\mathbf{x}_{t})}d\mathbf{x}_{0}=\int
q(\mathbf{x}_{0}|\mathbf{x}_{t})\frac{q_{t}(\mathbf{x}_{t})}{p_{t}(\mathbf{x}_{t})}d\mathbf{x}_{0}$
$\displaystyle=$
$\displaystyle\mathbb{E}_{q(\mathbf{x}_{0}|\mathbf{x}_{t})}\left[\frac{q_{t}(\mathbf{x}_{t})}{p_{t}(\mathbf{x}_{t})}\right],$

where $p_{t}(\mathbf{x}_{t})=\int
p(\mathbf{x}_{0})p(\mathbf{x}_{t}|\mathbf{x}_{0})d\mathbf{x}_{0}$ and
$q_{t}(\mathbf{x}_{t})=\int
q(\mathbf{x}_{0})q(\mathbf{x}_{t}|\mathbf{x}_{0})d\mathbf{x}_{0}$ are the
marginal distributions at time $t$ of source and target distributions,
respectively. ∎

##  Appendix C More Details on Experiments

###  C.1 Algorithms for TGDP

TGDP adopts Algorithm 1 and 2 for training a domain classifier and Algorithm 3
and 4 for training the guidance network.

Algorithm 1 Training a domain classifier

0: Samples from the marginal distribution of the source domain $p(\mathbf{x})$
and target domain $q(\mathbf{x})$, and initial weights of domain classifier
$\boldsymbol{\omega}$.

1: repeat

2: Sample mini-batch data from source distribution and target distribution
respectively with batch size $b$.

3: Take gradient descent step on
$\nabla_{\boldsymbol{\omega}}\left\\{-\frac{1}{b}\sum_{\mathbf{x}_{i}\in
p}\left[\log
c_{\boldsymbol{\omega}}(\mathbf{x}_{i})\right]-\frac{1}{b}\sum_{\mathbf{x}^{\prime}_{i}\in
q}\left[\log(1-c_{\boldsymbol{\omega}}(\mathbf{x}^{\prime}_{i}))\right]\right\\}.$

4: until converged.

5: return weights of domain classifier $\boldsymbol{\omega}$.

Algorithm 2 Training a time-dependent domain classifier

0: Samples from the marginal distribution of the source domain $p(\mathbf{x})$
and target domain $q(\mathbf{x})$, pre-defined forward transition
$p(\mathbf{x}_{t}|\mathbf{x}_{0})$, and initial weights of domain classifier
$\boldsymbol{\omega}$.

1: repeat

2: Sample mini-batch data from source distribution and target distribution
respectively with batch size $b$.

3: Sample time $t\sim\operatorname{Uniform}(\\{1,\ldots,T\\})$ and perturb
$\mathbf{x}_{0}$ by forward transition $p(\mathbf{x}_{t}|\mathbf{x}_{0})$.

4: Take gradient descent step on
$\nabla_{\boldsymbol{\omega}}\left\\{-\frac{1}{b}\sum_{\mathbf{x}_{0}\sim
p}\sum_{\mathbf{x}_{t}|\mathbf{x}_{0}}\left[\log
c_{\boldsymbol{\omega}}(\mathbf{x}_{t},t)\right]-\frac{1}{b}\sum_{\mathbf{x}_{0}\sim
q}\sum_{\mathbf{x}_{t}|\mathbf{x}_{0}}\left[\log(1-c_{\boldsymbol{\omega}}(\mathbf{x}_{t},t))\right]\right\\}.$

5: until converged.

6: return weights of time-dependent domain classifier $\boldsymbol{\omega}$.

Algorithm 3 Training a guidance network (without regularization)

0: Samples from the marginal distribution of the source domain
$p(\mathbf{x})$, pre-defined forward transition
$p(\mathbf{x}_{t}|\mathbf{x}_{0})$, pre-trained domain classifier
$c_{\boldsymbol{\omega}}$, and initial weights of guidance network
$\boldsymbol{\psi}$.

1: repeat

2: Sample mini-batch data from source distribution $\mathbf{x}_{0}$ with batch
size $b$.

3: Sample time $t\sim\operatorname{Uniform}(\\{1,\ldots,T\\})$ and perturb
$\mathbf{x}_{0}$ by forward transition $p(\mathbf{x}_{t}|\mathbf{x}_{0})$.

4: Take gradient descent step on
$\nabla_{\boldsymbol{\psi}}\left\\{\frac{1}{b}\sum\limits_{\mathbf{x}_{0},\mathbf{x}_{t}}\left[\left\|h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)-c_{\boldsymbol{\omega}}(\mathbf{x}_{0})\right\|_{2}^{2}\right]\right\\}.$

5: until converged.

6: return weights of guidance network $\boldsymbol{\psi}$.

Algorithm 4 Training a guidance network (with regularization)

0: Samples from the marginal distribution of the source domain $p(\mathbf{x})$
and target domain $q(\mathbf{x})$, pre-trained diffusion model on source
distribution $s_{\text{source}}(\mathbf{x}_{t},t)$, pre-defined forward
transition
$q(\mathbf{x}_{t}|\mathbf{x}_{0}),p(\mathbf{x}_{t}|\mathbf{x}_{0})$, pre-
trained domain classifier $c_{\boldsymbol{\omega}}(\mathbf{x}_{0})$ and time
dependent domain classifier
$c^{\prime}_{\boldsymbol{\omega}}(\mathbf{x}_{0},t)$, hyperparameter
$\eta_{1},\eta_{2}$, and initial weights of guidance network
$\boldsymbol{\psi}$.

1: repeat

2: Sample mini-batch data from source distribution $\mathbf{x}_{0}$ with batch
size $b$.

3: Perturb $\mathbf{x}_{0}$ by forward transition
$p(\mathbf{x}_{t}|\mathbf{x}_{0})$.

4:
$\mathcal{L}_{\text{guidance}}(\boldsymbol{\psi})=\frac{1}{b}\sum\limits_{\mathbf{x}_{0},\mathbf{x}_{t},t}\left[\left\|h_{\boldsymbol{\psi}}\left(\mathbf{x}_{t},t\right)-(1-c_{\boldsymbol{\omega}}(\mathbf{x}_{0}))/c_{\boldsymbol{\omega}}(\mathbf{x}_{0})\right\|_{2}^{2}\right]$

5: Sample mini-batch data from target distribution $\mathbf{x}^{\prime}_{0}$
with batch size $b$.

6: Sample time $t\sim\operatorname{Uniform}(\\{1,\ldots,T\\})$ and perturb
$\mathbf{x}^{\prime}_{0}$ by forward transition
$q(\mathbf{x}^{\prime}_{t}|\mathbf{x}^{\prime}_{0})$.

7:
$\mathcal{L}_{\text{cycle}}=\frac{1}{b}\sum\limits_{\mathbf{x}^{\prime}_{0},\mathbf{x}^{\prime}_{t},t}\left[\left\|h_{\boldsymbol{\psi}}\left(\mathbf{x}^{\prime}_{t},t\right)-c^{\prime}_{\boldsymbol{\omega}}(\mathbf{x}^{\prime}_{0},t)\right\|_{2}^{2}\right].$

8:
$\mathcal{L}_{\text{consistence}}=\frac{1}{b}\sum\limits_{\mathbf{x}^{\prime}_{0},\mathbf{x}^{\prime}_{t},t}\left[\left\|s_{\text{source}}(\mathbf{x}^{\prime}_{t},t)+\nabla_{\mathbf{x}^{\prime}_{t}}\log
h_{\boldsymbol{\psi}}\left(\mathbf{x}^{\prime}_{t},t\right)-\nabla_{\mathbf{x}^{\prime}_{t}}\log
q(\mathbf{x}_{t}|\mathbf{x}^{\prime}_{0})\right\|_{2}^{2}\right].$

9: Take gradient descent step on
$\nabla_{\boldsymbol{\psi}}\left\\{\mathcal{L}_{\text{guidance}}+\eta_{1}\
\mathcal{L}_{\text{cycle}}+\eta_{2}\
\mathcal{L}_{\text{consistence}}\right\\}.$

10: until converged.

11: return weights of guidance network $\boldsymbol{\psi}$.

###  C.2 Ablation Studies on simulations

In Figure 4, we demonstrate the ablation studies on simulations. We can see
that only using the consistency regularization term (Figure 4 (b)) is not able
to recover the true distribution in the target domain. Our guidance loss
together with cycle regularization can learn a good approximation of target
distribution while adding consistency regularization can achieve better
performance.

(a) Source diffusion

(b) Only $\mathcal{L}_{\text{consistence}}$

(c) $\mathcal{L}_{\text{guidance}}+\mathcal{L}_{\text{cycle}}$

(d)
$\mathcal{L}_{\text{guidance}}+\mathcal{L}_{\text{cycle}}+\mathcal{L}_{\text{consistence}}$

(e) Target

Figure 4: An illustration of the effectiveness of cycle regularization and
consistency regularization proposed in Section 3.4.

###  C.3 Implementation details for ECG Benchmark

For TGDP and all of the baseline methods, we utilize the same architecture as
the conditional generative models for ECG data, SSSM-ECG [2]. For Vanilla
Diffusion, we train the diffusion model for 100k iterations by Adam optimizer
with a learning rate 2$\mathrm{e}^{-4}$. For Finetune Generator, we finetune
the pre-trained diffusion model for 50k iterations by Adam optimizer with a
learning rate 2$\mathrm{e}^{-5}$. For TGDP, we adopt a 4-layer MLP with 512
hidden units and SiLU activation function as the backbone of the guidance
network. We train the guidance network for 50k iterations by Adam optimizer
with a learning rate 2$\mathrm{e}^{-4}$. For utility evaluation, we adopt the
same architecture, xresnet1d50 [37], as the backbone. We train the classifier
from sketch for 50 epochs with with a learning rate 1e-2. For Finetune
Classifier, we finetune a pre-trained classifier for 30 epochs with with a
learning rate 1e-3.