# Discriminative Estimation of Total Variation Distance: A Fidelity Auditor
for Generative Data

Lan Tao1
<EMAIL_ADDRESS>&Shirong Xu 111footnotemark: 1
<EMAIL_ADDRESS>&Chi-Hua Wang1
<EMAIL_ADDRESS>&Namjoon Suh1
<EMAIL_ADDRESS>&Guang Cheng1
<EMAIL_ADDRESS>
Equal contributions. 1 Department of Statistics & Data Science, University of
California, Los Angeles

###### Abstract

With the proliferation of generative AI and the increasing volume of
generative data (also called as synthetic data), assessing the fidelity of
generative data has become a critical concern. In this paper, we propose a
discriminative approach to estimate the total variation (TV) distance between
two distributions as an effective measure of generative data fidelity. Our
method quantitatively characterizes the relation between the Bayes risk in
classifying two distributions and their TV distance. Therefore, the estimation
of total variation distance reduces to that of the Bayes risk. In particular,
this paper establishes theoretical results regarding the convergence rate of
the estimation error of TV distance between two Gaussian distributions. We
demonstrate that, with a specific choice of hypothesis class in
classification, a fast convergence rate in estimating the TV distance can be
achieved. Specifically, the estimation accuracy of the TV distance is proven
to inherently depend on the separation of two Gaussian distributions: smaller
estimation errors are achieved when the two Gaussian distributions are farther
apart. This phenomenon is also validated empirically through extensive
simulations. In the end, we apply this discriminative estimation method to
rank fidelity of synthetic image data using the MNIST dataset.

##  1 Introduction

Evaluating the discrepancy between distributions has been a prominent research
topic in the statistics and machine learning communities, as evidenced by its
extensive applications in hypothesis testing [8, 28] and generative data
evaluation [20, 23]. Particularly in recent years, considerable research
efforts have been dedicated to the development of generative models, resulting
in a boom in generative data. Within this context, assessing the fidelity of
generative data to real data is vital for ensuring the significance of
downstream tasks trained on these generative data.

In practice, the fidelity of generative data can be measured via some
statistical divergences, such as Kullback-Leibler divergence, Jensen-Shannon
divergence, and Total Variation (TV) distance. However, estimating these
statistical divergences faces significant hurdles due to the high-dimensional
complexity and intricate correlations within the data. These challenges partly
explain why the existing frameworks for fidelity evaluation [11] predominantly
rely on low-dimensional surrogate metrics, such as marginal distributions [29]
and correlation plots. To avoid directly computing distributional distances in
high dimensions, researchers have proposed several approaches to audit
fidelity. These include comparing the density of synthetic and real
distributions only over random subsets of datasets [4], or quantifying the
similarity between real and synthetic data using precision (quality of
synthetic samples) and recall (diversity of synthetic samples) [20].

To have a more comprehensive auditing, we realize the necessity and importance
of distance estimation at the distributional level. To develop an effective
approach to estimate the (particularly high dimensional) distributional
distance, we start with the TV distance as the metric to compare two
distributions, which stands out as the premier metric for evaluating
generative data quality in the literature [25, 29]. Our key insight is to
frame the TV distance between two distributions as the Bayes risk in a
classification task for distinguishing between them. Thus, the problem of
estimating TV distance can be transformed into estimating Bayes risk in a
classification problem.

We establish theoretical results regarding the convergence rate of the
estimation error of TV distance between two Gaussian distributions, which is
further extended to the exponential family. Specifically, we show that the
proposed estimator converges to the true TV distance in probability at a
faster convergence rate compared with results in [19, 24]. Interestingly, our
theory (one-dimensional Gaussian case (Theorem 3.5)) confirms a phenomenon
that the estimation of TV distance inherently depends on the level of
separation between two distributions: the farther apart the two distributions
are, the easier the estimation task becomes. This phenomenon is validated in
extensive simulations (Figure 2). Our theory is developed under the Gaussian
assumption that is supported by the normality of generative data embeddings
found in images [14] and text data [5].

In numerical experiments, we employ our proposed method to compare images
generated by generative adversarial networks (GANs; [9]), showing that our
method accurately ranks data fidelity based on the different types of
embeddings (Table 4).

###  1.1 Related Work

There are two closely related lines of work, including the estimation of
statistical divergences and the TV distance between two Gaussian
distributions. Next we provide an overview of relevant studies and discuss
their distinctions from our own work.

Statistical divergence estimation. Contemporary methodologies for estimating
divergence metrics predominantly rely on employing plug-in density estimators
as surrogates for the densities within these metrics. Moon et al. [16] employ
a kernel density estimator to estimate the density ratio within the
$f$-divergence family. Similarly, Noshad et al. [18] propose using $k$-nearest
neighbor to approximate the continuous density function ratio within the
$f$-divergence family. Rubenstein et al. [19] introduce a random mixture
estimator to approximate the $f$-divergence between two probability
distributions. Additionally, Sreekumar et al. [24] establish non-asymptotic
absolute error bounds for the use of neural networks in approximating
$f$-divergences. Existing methods primarily nonparametric estimation based,
which are hindered by the curse of dimensionality and often overlook the
separation between two distributions. Interestingly, our developed method
frames the divergence estimation problem as a classification problem that
takes into account of the separation gap closely connected with the classic
low-noise assumption in the literature of classification.

TV distance between Gaussian distributions. Devroye et al. [7] investigate the
total variation distance between two high-dimensional Gaussians with the same
mean, providing both lower and upper bounds for their total variation
distance. Davies et al. [6] derive new lower bounds on the total variation
distance between two-component Gaussian mixtures with a shared covariance
matrix by examining the characteristic function of the mixture. Building upon
the work of [7], Barabesi et al. [2] improve the results by providing a
tighter bound for the total variation distance between two high-dimensional
Gaussian distributions based on a more delicate bound for the cumulative
distribution function of Gaussians. Existing works on the TV distance between
Gaussian distributions primarily focus on deriving upper and lower bounds
rather than establishing effective estimation methods based on finite samples.
Nevertheless, our developed method can also be utilized to establish
probabilistic bounds for the true TV distance between two Gaussians.

###  1.2 Preliminaries

For a random variable $X$, we let $\mathbb{E}_{X}(\cdot)$ denote the
expectation taken with respect to the randomness of $X$. For a random sequence
$\\{X_{n}\\}_{n=1}^{\infty}$, $X_{n}\stackrel{{\scriptstyle
p}}{{\rightarrow}}X$ indicates that $X_{n}$ converges to $X$ in probability.
We use bold symbols to represent multivariate objects. In binary
classification, the objective is to learn a classifier
$f:\mathcal{X}\rightarrow\\{0,1\\}$ for capturing the functional relationship
between the feature vector $\bm{X}\in\mathcal{X}$ and its associated label
$Y\in\\{0,1\\}$. The performance of $f$ is usually measured by the 0-1 risk as
$R(f)=P\left(f(\bm{X})\neq Y\right)$, where the expectation is taken with
respect to the joint distribution of $(\bm{X},Y)$. The optimal classifier
$f^{\star}=\text{argmin}_{f}R(f)$ refers to the Bayes decision rule, which is
obtained by minimizing $R(f)$ in a point-wise manner and given as
$f^{\star}(\bm{X})=I\left(\eta(\bm{X})\geq\frac{1}{2}\right)$, where
$\eta(\bm{X})=P(Y=1|\bm{X})$ and $I(\cdot)$ is the indicator function.

##  2 Discriminative Estimation of Total Variation Distance

In this section, we present an effective classification-based approach to
estimate the underlying total variation (TV) distance between two
distributions using two sets of their realizations. Our key insight is to
conceptualize the total variation distance as a lower bound of the Bayes Risk
for a real-synthetic data classifier. By leveraging the duality between total
variation distance and Bayes Risk, we establish a lower bound on the total
variation distance. This method can serve as a “Fidelity Auditor" for
comparing real and synthetic data, and is directly applicable to arbitrary
data synthesizers.

###  2.1 Framing Total Variation Distance as Classification Problem.

We denote the sets of real data and synthetic data as
$\left\\{\bm{x}_{i}\right\\}_{i=1}^{n}$ and
$\left\\{\widetilde{\bm{x}}_{i}\right\\}_{i=1}^{n}$, respectively, where
$\bm{x}_{i},\widetilde{\bm{x}}_{i}\in$ $\mathbb{R}^{p}$ are $p$-dimensional
continuous vectors. Let $\mathbb{P}(\bm{x})$ and $\mathbb{Q}(\bm{x})$ denote
the density functions of real and synthetic data, respectively. The total
variation (TV) distance between $\mathbb{P}(\bm{x})$ and $\mathbb{Q}(\bm{x})$
is given as

$\text{TV}(\mathbb{P},\mathbb{Q})=\frac{1}{2}\int_{\mathbb{R}^{p}}|\mathbb{P}(\bm{x})-\mathbb{Q}(\bm{x})|d\bm{x}.$

For the mixed dataset
$\mathcal{D}=\left\\{\bm{x}_{i}\right\\}_{i=1}^{n}\cup\left\\{\widetilde{\bm{x}}_{i}\right\\}_{i=1}^{n}$,
the underlying density function can be written as

$\mathbb{D}(\bm{x})=\frac{\mathbb{P}(\bm{x})+\mathbb{Q}(\bm{x})}{2}.$

As elaborated in the work of [17], estimating $f$-divergences can be
equivalently transformed to seek the optimal classifier capable of
distinguishing real data from synthetic data. Specifically, we set the labels
of real and synthetic samples as 1 and 0, respectively. For any sample
$\bm{x}$, the probability of $\bm{x}$ being real is given as
$\eta(\bm{x})=\frac{\mathbb{P}(\bm{x})}{\mathbb{P}(\bm{x})+\mathbb{Q}(\bm{x})}$.
Let $f:\mathbb{R}^{p}\rightarrow\\{0,1\\}$ be a classifier used to
discriminate real and synthetic samples. The expected classification error can
be written as

$\displaystyle R(f)=$
$\displaystyle\mathbb{E}_{\bm{X}}\left[I(f(\bm{X})=1)\frac{\mathbb{Q}(\bm{X})}{\mathbb{P}(\bm{X})+\mathbb{Q}(\bm{X})}+I(f(\bm{X})=0)\frac{\mathbb{P}(\bm{X})}{\mathbb{P}(\bm{X})+\mathbb{Q}(\bm{X})}\right],$
(1)

where $\bm{X}\sim\mathbb{D}$. Therefore, the minimal risk $R(f^{\star})$ is
then given as

$\displaystyle
R(f^{\star})=\frac{1}{2}\int_{\mathbb{R}^{p}}\min\\{\mathbb{P}(\bm{x}),\mathbb{Q}(\bm{x})\\}d\bm{x}=\frac{1}{2}-\frac{1}{2}\text{TV}\left(\mathbb{P},\mathbb{Q}\right).$
(2)

It is clear from (2) that the estimation of the total variation between
$\mathbb{P}$ and $\mathbb{Q}$ is equivalent to that of the Bayes risk
$R(f^{\star})$ for the task of discriminating between real and synthetic data.

###  2.2 Total Variation Distance Lower Bound via Classification

Given an estimator $\widehat{f}$ of the optimal classifier $f^{\star}$, we
always have

$R(\widehat{f})\geq
R(f^{\star})=\frac{1}{2}-\frac{1}{2}\text{TV}\left(\mathbb{P},\mathbb{Q}\right).$

This inequality suggests

$\text{TV}(\mathbb{P},\mathbb{Q})\geq
1-2R(\widehat{f})\triangleq\widehat{\text{TV}}(\mathbb{P},\mathbb{Q})$ (3)

for any feasible classifier $\widehat{f}$. Therefore, $\widehat{f}$ provides a
means to establish a lower bound for the total variation distance between the
distributions of real and synthetic data distributions. Each specific
classifier $\widehat{f}$ yields a lower bound on the indistinguishability
between $\mathbb{P}$ and $\mathbb{Q}$. Intuitively, if none of classifiers
yields a large lower bound, then the synthetic data $\mathbb{Q}$ can be
considered similar to the real data $\mathbb{P}$, indicating that their total
variation distance is small.

If the chosen classifier $\widehat{f}$ is consistent for achieving minimal
risk, that is $\mathcal{E}(\widehat{f})=R(\widehat{f})-R(f^{\star})=0$, where
$\mathcal{E}(\widehat{f})$ is known as the excess risk, then
$\widehat{\text{TV}}(\mathbb{P},\mathbb{Q})$ appears as a consistent estimator
of the real total variation $\text{TV}(\mathbb{P},\mathbb{Q})$, that is

$\displaystyle\mathcal{E}(\widehat{f})=R(\widehat{f})-R(f^{\star})\stackrel{{\scriptstyle
p}}{{\rightarrow}}0\Leftrightarrow\text{TV}(\mathbb{P},\mathbb{Q})-\widehat{\text{TV}}(\mathbb{P},\mathbb{Q})\stackrel{{\scriptstyle
p}}{{\rightarrow}}0.$

Here the equivalence of these two convergence in probability is supported by
the quantitative relation
$\text{TV}(\mathbb{P},\mathbb{Q})-\widehat{\text{TV}}(\mathbb{P},\mathbb{Q})=2\mathcal{E}(\widehat{f})$.
In the literature, there has been various research efforts devoted to
establishing the convergence of $\mathcal{E}(\widehat{f})$ [1, 3].

##  3 Optimal Estimation of Total Variation Distance

In this section, we present several examples where achieving an optimal
classifier is feasible by choosing a proper hypothesis class. For
illustration, we primarily examine a scenario where both real and synthetic
data are generated from multivariate Gaussian distributions. Subsequently, we
offer an extension to encompass the general exponential family. To establish
the tightest convergence rate for the empirical fidelity auditor, we adopt the
following low noise assumption in the classification literature [1, 3].

######  Assumption 3.1 (Low-Noise Condition).

There exist some positive constants $C_{0}$ and $\gamma$ such that
$P(|\eta(\bm{x})-1/2|\leq t)\leq C_{0}t^{\gamma}$ for any $t>0$, where
$\gamma$ is referred to as the noise exponent.

Assumption 3.1 characterizes the behavior of the regression function $\eta$ in
the vicinity of the level $\eta(\bm{x})=1/2$, which is paramount for
convergence of classifiers. Particularly, a larger value of $\gamma$ indicates
smaller noise in the labels, resulting in a faster convergence rate to the
optimal classifier.

###  3.1 Multivariate Gaussian Distribution

We start with delving into a scenario where both real and synthetic data
follow multivariate normal distributions. Our primary aim is to delineate the
optimal function class for training an empirical classifier and assess its
convergence towards the optimal classifier. This assumption finds particular
prevalence in the domain of generative data, owing to the widespread practice
of assuming embeddings of generative data to be normally distributed, such as
images [14] and text data [5].

Specifically, we assume $\mathbb{P}$ and $\mathbb{Q}$ are two different
Gaussian density functions parametrized by $(\bm{\mu}_{1},\bm{\Sigma}_{1})$
and $(\bm{\mu}_{2},\bm{\Sigma}_{2})$, respectively. Under this assumption, the
underlying distribution of the mixed dataset $\mathcal{D}$ is
$\frac{1}{2}N(\bm{\mu}_{1},\bm{\Sigma}_{1})+\frac{1}{2}(\bm{\mu}_{2},\bm{\Sigma}_{2})$.

######  Lemma 3.2.

Given that
$\mathcal{D}\sim\frac{1}{2}N(\bm{\mu}_{1},\bm{\Sigma}_{1})+\frac{1}{2}N(\bm{\mu}_{2},\bm{\Sigma}_{2})$,
the Bayes decision rule (optimal classifier) for determining the true
distribution of a given sample $\bm{x}$ is

$\displaystyle
f^{\star}(\bm{x})=I\left(\log\left(\frac{\mathrm{det}(\bm{\Sigma}_{2})}{\mathrm{det}(\bm{\Sigma}_{1})}\right)+(\bm{x}-\bm{\mu}_{2})^{T}\bm{\Sigma}_{2}^{-1}(\bm{x}-\bm{\mu}_{2})-(\bm{x}-\bm{\mu}_{1})^{T}\bm{\Sigma}_{1}^{-1}(\bm{x}-\bm{\mu}_{1})>0\right),$

where $\mathrm{det}(\cdot)$ denotes the determinant of a matrix.

Lemma 3.2 specifies the optimal classifier for discriminating between two
multivariate Gaussian distributions. However, directly learning $f^{\star}$ is
often computationally infeasible in practical scenarios. As an alternative
approach, we consider employing a plug-in classifier, where we aim to estimate
$\eta(\bm{X})=\frac{\mathbb{P}(\bm{X})}{\mathbb{P}(\bm{X})+\mathbb{Q}(\bm{X})}$
through the following optimization task:

$\displaystyle\widehat{\bm{\beta}}=\arg\min_{\bm{\beta}\in\mathbb{R}^{d}}\frac{1}{2n}\sum_{i=1}^{n}\left\\{\left(1-\frac{\exp(\bm{\beta}^{T}\psi(\bm{x}_{i}))}{1+\exp(\bm{\beta}^{T}\psi(\bm{x}_{i}))}\right)^{2}+\left(\frac{\exp(\bm{\beta}^{T}\psi(\widetilde{\bm{x}}_{i}))}{1+\exp(\bm{\beta}^{T}\psi(\widetilde{\bm{x}}_{i}))}\right)^{2}\right\\}+\lambda\|\bm{\beta}\|_{2}^{2},$
(4)

where
$\psi(\bm{x})=(1,x_{1},\ldots,x_{p},x_{1}^{2},x_{1}x_{2},\ldots,x_{p-1}x_{p},x_{p}^{2})$
being a feature transformation of original features $\bm{x}$ with
$d=(p+2)(p+1)/2$.

Next we denote
$\mathcal{H}=\left\\{h(\bm{x})=\bm{\beta}^{T}\psi(\bm{x}):\bm{\beta}\in\mathbb{R}^{d}\right\\}$
and $\widehat{h}(\bm{x})=\widehat{\bm{\beta}}^{T}\psi(\bm{x})$. As long as
$\widehat{h}$ is obtained, the plug-in classifier can be obtained as

$\displaystyle\text{Plug-in
Classifier:}\quad\widehat{f}(\bm{x})=I\left(\frac{\exp(\widehat{h}(\bm{x}))}{1+\exp(\widehat{h}(\bm{x}))}>\frac{1}{2}\right)=I\left(\widehat{h}(\bm{x})>0\right).$
(5)

Here, $\widehat{f}$ represents an empirical classifier estimated from
$\mathcal{D}$, capable of discerning between real and synthetic data
originating from two distinct Gaussian distributions.

######  Lemma 3.3.

Define
$h_{\phi}^{\star}=\arg\min_{h}\mathbb{E}\left[(\phi(h(\bm{X}))-Y)^{2}\right]$
with $\phi(x)=\frac{1}{1+\exp(-x)}$. Given that
$\bm{X}\sim\frac{1}{2}N(\bm{\mu}_{1},\bm{\Sigma}_{1})+\frac{1}{2}(\bm{\mu}_{2},\bm{\Sigma}_{2})$
and
$P(Y=1|\bm{X})=\frac{\mathbb{P}(\bm{X})}{\mathbb{P}(\bm{X})+\mathbb{Q}(\bm{X})}$,
we have

$h_{\phi}^{\star}(\bm{x})=\log\left(\frac{\mathrm{det}(\bm{\Sigma}_{2})}{\mathrm{det}(\bm{\Sigma}_{1})}\right)+(\bm{x}-\bm{\mu}_{2})^{T}\bm{\Sigma}_{2}^{-1}(\bm{x}-\bm{\mu}_{2})-(\bm{x}-\bm{\mu}_{1})^{T}\bm{\Sigma}_{1}^{-1}(\bm{x}-\bm{\mu}_{1}).$

Lemma 3.3 validates the effectiveness of (4) in obtaining an empirical
classifier. Specifically, as the sample size tends towards infinity,
$\widehat{h}$ becomes consistent with $f^{\star}$ in sign. Therefore, the
plug-in classifier $\widehat{f}$ can be used as a surrogate for $f^{\star}$ to
calculate the total variation between $\mathbb{P}$ and $\mathbb{Q}$. To
theoretically validate this claim, we demonstrate in Theorem 3.4 that our
developed discriminative estimation of the total variation between two
Gaussian distributions exhibits a fast convergence rate of
$O\left(\left(d\log(n)/n\right)^{\frac{\gamma+1}{\gamma+2}}\right)$. This
result aligns with the optimal convergence rate in classification under the
same assumptions as presented in [3, 26].

Moreover, our theoretical result unveils two intriguing phenomena:

  1. 1

When an appropriate function class is chosen for classification, the
estimation of the total variation between two Gaussian distributions remains
robust against data dimension compared to nonparametric density estimation and
neural estimation approaches [24];

  2. 2

The estimation error of total variation inherently depends on the difference
between $\mathbb{P}$ and $\mathbb{Q}$, such that a faster convergence rate is
achieved when the real total variation distance between $\mathbb{P}$ and
$\mathbb{Q}$ is larger (larger values of $\gamma$ or smaller values of $C_{0}$
in Assumption 3.1).

The second phenomenon is striking because it suggests that the difficulty of
estimating total variation diminishes significantly when the true variation is
substantial. Despite lacking theoretical validation in existing literature,
this result is intuitively comprehensible. In Figure 1, we provide a toy
example illustrating that $\mathbb{P}$ and $\mathbb{Q}$ have completely
disjoint supports, resulting in a true total variation of one. It can be
observed that regardless of the number of samples used to compute the
empirical total variation, the estimated total variation is consistent with
zero estimation error.

Figure 1: In this case, the supports of $\mathbb{P}$ and $\mathbb{Q}$ are
completely non-overlapping, and hence Assumption 3.1 holds with $C_{0}=0$ and
any $\gamma>0$. It is evident that the estimation error in (6) is zero due to
the disjoint nature of the histograms for any value of $n$ in this example.

######  Theorem 3.4.

If $\mathbb{P}$ and $\mathbb{Q}$ are two different Gaussian density functions
parametrized by $(\bm{\mu}_{1},\bm{\Sigma}_{1})$ and
$(\bm{\mu}_{2},\bm{\Sigma}_{2})$, respectively. Under Assumption 3.1, we have

$\displaystyle\mathbb{E}_{\mathcal{D}}\left\\{\mathrm{TV}(\mathbb{P},\mathbb{Q})-\widehat{\mathrm{TV}}(\mathbb{P},\mathbb{Q})\right\\}\lesssim
C_{0}^{\frac{1}{\gamma+2}}\left(\frac{d\log
n}{2n}\right)^{\frac{\gamma+1}{\gamma+2}},$ (6)

where $\widehat{\mathrm{TV}}(\mathbb{P},\mathbb{Q})=1-2R(\widehat{f})$ with
$\widehat{f}$ being the plug-in classifier given by (5) with $\lambda\asymp
d\log(n)/n$ and $C_{0}$ and $\gamma$ are as defined in Assumption 3.1.

In Theorem 3.5, we present a detailed analysis of (6) specifically tailored to
the one-dimensional Gaussian scenario. This analysis includes the explicit
determination of the constants $C_{0}$ and $\gamma$ as defined in Assumption
3.1. Our findings illustrate that the proposed discriminative estimation
method achieves a rapid convergence rate of
$O\left(\mu^{-1/3}n^{-\frac{2}{3}}\right)$, accompanied by a logarithmic
factor. Notably, as $\mu$ tends towards infinity, the convergence rate
accelerates, consistent with our second observation mentioned earlier.

######  Theorem 3.5.

Suppose $X\sim\frac{1}{2}N(\mu,1)+\frac{1}{2}N(-\mu,1)$ with $\mu\geq 0$. For
any $c\in(0,\frac{1}{2})$, we have
$P(|\eta(X)-1/2|<t)\leq\sqrt{\frac{8}{(1-2c)^{2}\pi e\mu^{2}}}t$ for any
$t\in(0,c)$. With this, (6) becomes

$\displaystyle\mathbb{E}_{\mathcal{D}}\left\\{\mathrm{TV}(\mathbb{P},\mathbb{Q})-\widehat{\mathrm{TV}}(\mathbb{P},\mathbb{Q})\right\\}\lesssim\mu^{-\frac{1}{3}}\left(\frac{3\log
n}{2n}\right)^{\frac{2}{3}}.$

###  3.2 Extension to Exponential Family

We extend our Gaussian result to encompass the broader exponential family.
Specifically, we address the question of determining the appropriate function
class for estimating the total variation between two exponential-type random
variables. With the appropriate choice of function classes, similar results
for estimating the total variation can be derived, building upon the risk of
the resulting classifier.

For any exponential-type random variable $\bm{X}$, the associated probability
density function can typically be expressed in the general form

$f_{\bm{X}}(\bm{x}|\bm{\theta})=h(\bm{x})\cdot\exp\left[\bm{\eta}(\bm{\theta})\cdot\bm{T}(\bm{x})-A(\bm{\theta})\right],$

where $h(\cdot)$, $\bm{T}(\cdot)$, $\bm{\eta}(\cdot)$, and $A(\cdot)$ are
functions that uniquely depend on the type of $\bm{X}$.

######  Theorem 3.6.

Let $\mathbb{P}(\bm{x})$ and $\mathbb{Q}(\bm{x})$ be the density functions of
two different random variables from the exponential family:

$\displaystyle\mathbb{P}(\bm{x})=h_{1}(\bm{x})\cdot\exp\left[\bm{\eta}_{1}(\bm{\theta}_{1})\cdot\bm{T}_{1}(\bm{x})-A_{1}(\bm{\theta}_{1})\right],$
$\displaystyle\mathbb{Q}(\bm{x})=h_{2}(\bm{x})\cdot\exp\left[\bm{\eta}_{2}(\bm{\theta_{2}})\cdot\bm{T}_{2}(\bm{x})-A_{2}(\bm{\theta}_{2})\right].$

Then the optimal classifier for minimizing (1) is given as

$\displaystyle
f^{\star}(x)=I\left(\log\left(\frac{h_{1}(\bm{x})}{h_{2}(\bm{x})}\right)+A_{2}(\bm{\theta}_{2})-A_{1}(\bm{\theta}_{1})+\bm{T}_{1}(\bm{x})\bm{\eta}_{1}(\bm{\theta_{1}})-\bm{T}_{2}(\bm{x})\bm{\eta}(\bm{\theta_{2}})>0\right).$
(7)

Furthermore, the total variation between $\mathbb{P}(\bm{x})$ and
$\mathbb{Q}(\bm{x})$ is given as
$\mathrm{TV}(\mathbb{P},\mathbb{Q})=2R(f^{\star})-1$.

Theorem 3.6 elucidates the optimal classifier for discriminating between two
random variables from the exponential family, providing a method to calculate
the total variation between their underlying distributions. Furthermore,
Theorem 3.6 also explicates the appropriate class of margin classifiers when
the underlying distributions are from exponential family. For illustration, in
the following, we outline the appropriate selection of function classes for
different combinations between four exponential-type univariate random
variables, as summarized in Table 1. The extension to other exponential-type
random variables and multivariate cases can be derived analytically.

Table 1: The choice of function class takes the form as $\mathcal{H}=\\{f(\bm{x})=\bm{\beta}^{T}\psi(\bm{x}):\bm{\beta}\in\mathbb{R}^{d}\\}$. Below presents the explicit form of $\psi(\bm{x})$ under different combinations of types of $\mathbb{P}$ and $\mathbb{Q}$. Due to the symmetry between $\mathbb{P}$ and $\mathbb{Q}$, we display only the upper triangular results in this table.  $\mathbb{Q}$ $\mathbb{P}$ | Gaussian | Exponential | Gamma | Beta
---|---|---|---|---
Gaussian | $(1,x,x^{2})$ | $(1,x,x^{2})$ | $(1,x,x^{2},\log x)$ | $(1,x,x^{2},\log x,\log(1-x))$
Exponential | - | $(1,x)$ | $(1,x,\log x)$ | $(1,x,\log x,\log(1-x))$
Gamma | - | - | $(1,x,\log x)$ | $(1,x,\log x,\log(1-x))$
Beta | - | - | - | $(1,\log x,\log(1-x))$

##  4 Experiments

(a) $(n,p)=(10^{3},5)$

(b) $(n,p)=(10^{3},10)$

(c) $(n,p)=(10^{4},5)$

(d) $(n,p)=(10^{4},10)$

Figure 2: True total variation ($x$-axis) versus estimated total variation
($y$-axis) in cases $(n,p)\in\\{10^{3},10^{4}\\}\times\\{5,10\\}$ under
varying disparity between two Gaussian distributions.

In this section, we showcase the superior performance of the developed
discriminative method (DisE) for estimating the total variation between two
Gaussian distributions. For each simulated setting, we report the average
results for all simulation settings, accompanied by their respective standard
deviations calculated over $20$ replications, presented in parentheses.

Comparison Methods and Evaluation Metrics. Existing methods for estimating
divergence metrics predominantly rely on a plug-in estimation approach,
typically applied to either two separate density functions or their density
ratio. In this experiment, we consider kernel density estimation (KDE; [21])
for the former type of estimator. For the latter, we explore two nearest
neighbor type estimators, including the ensemble estimation (EE; [16]) and
nearest neighbor ratio estimation (NNRE; [18]). Furthermore, we incorporate a
parameter estimation (PE) approach, which entails approximating the total
variation through the Monte Carlo method based on sample mean and covariance
matrix. As a baseline, we utilize the Monte Carlo method to calculate the true
total variation based on true means and covariance matrices. The performance
of all methods are evaluated in three aspects, including robustness,
computational time, and estimation error measured in absolute error.

Experimental Setting. We conduct a comprehensive analysis of the impact of
sample size and data dimension on the performance of various estimators.
Specifically, we consider $\mathbb{P}$ as a Gaussian distribution with mean
$\bm{\mu}_{1}=\bm{0}_{p}$ and covariance matrix
$\bm{\Sigma}_{1}=\bm{I}_{p\times p}$. In contrast, $\mathbb{Q}$ is a Gaussian
distribution with mean $\bm{\mu}_{2}$ uniformly generated from $[0,1]^{p}$ and
covariance matrix $\bm{\Sigma}_{2}=\bm{I}_{p\times p}+\bm{E}$, where $\bm{E}$
is a symmetric noise matrix. We compare the performance of our proposed method
with that of existing estimation methods across different data dimensions,
sample sizes, and differences between the means of two distributions. For each
fixed setting, we conduct 20 replications to calculate the standard
deviations, which serve as a measure of the robustness of the estimation
accuracy.

Experimental Result. Figure 2 shows that the DisE and PE methods provide the
most accurate estimates of the true total variation distance across all
scenarios. The KDE approach tends to overestimate the total variation in cases
of smaller disparity, while the NNRE and EE approaches tend to underestimate
it in cases of larger disparity. Notably, as the true total variation
increases, the accuracy of our proposed DisE method improves, which aligns
perfectly with the theoretical results established in Theorem 3.4.
Furthermore, compared to other methods, our proposed method is less sensitive
to data dimensionality.

Robustness Study. To further validate the robustness of our proposed method,
we repeatedly compare the estimation results across different dimensions
ranging from $2$ to $12$, and examine the estimation results under different
levels of noise added to data. The average estimation errors under varying
disparities between two distributions are reported in Figure 3 and Table 2.
Clearly, both DisE and PE consistently exhibit smaller estimation errors,
while the other approaches show increasing errors as the dimension expands.
Table 2 demonstrates that the DisE approach achieves higher accuracy and lower
variance compared to the PE approach. Figure 4 and Table 3 show the average
estimation errors under varying levels of variances of noise added to data.
The estimation errors of all approaches show a growing pattern with the
increase of noise level, and the proposed DisE approach has a relatively lower
estimation error compared with other methods. Overall, these findings confirm
the superior robustness and accuracy of the DisE approach in estimating total
variation distance under varying dimensions and noise levels.

Figure 3: The robustness of estimation errors of all methods with respect to
data dimensionality.

Method | dim = $2$ | dim = $4$ | dim = $6$
---|---|---|---
DisE |  0.002(0.002) |  0.003(0.002) |  0.002(0.001)
PE | 0.003(0.002) | 0.005(0.004) | 0.004(0.003)
KDE | 0.008(0.005) | 0.015(0.017) | 0.062(0.045)
NNRE | 0.013(0.015) | 0.048(0.027) | 0.085(0.059)
EE | 0.038(0.019) | 0.086(0.052) | 0.108(0.074)
Method | dim = $8$ | dim = $10$ | dim = $12$
DisE |  0.003(0.002) |  0.002 (0.002) |  0.003(0.002)
PE | 0.004(0.004) | 0.004(0.003) | 0.004(0.003)
KDE | 0.115(0.089) | 0.151(0.121) | 0.202(0.154)
NNRE | 0.154(0.091) | 0.221(0.118) | 0.246(0.124)
EE | 0.165(0.099) | 0.221(0.125) | 0.233(0.124)

Table 2: The averaged estimation errors (standard deviations) of total
variation estimation of all methods across various data dimensions.

Figure 4: The robustness of estimation errors of all methods with respect to
noise added to data (dimension = $5$).

Method | noise = $0.1$ | noise = $0.5$ | noise = $1.0$
---|---|---|---
DisE |  0.003(0.002) |  0.032(0.027) |  0.159(0.117)
PE | 0.005(0.004) | 0.033(0.029) | 0.176(0.120)
KDE | 0.052(0.043) | 0.074(0.060) | 0.203(0.131)
NNRE | 0.054(0.041) | 0.054(0.035) | 0.129(0.100)
EE | 0.079(0.063) | 0.069(0.051) | 0.129(0.092)
Method | noise = $1.5$ | noise = $2.0$ | noise = $2.5$
DisE |  0.310(0.174) |  0.423 (0.207) |  0.494(0.225)
PE | 0.350(0.169) | 0.478(0.187) | 0.557(0.195)
KDE | 0.376(0.173) | 0.501(0.189) | 0.577(0.196)
NNRE | 0.294(0.153) | 0.437(0.179) | 0.524(0.198)
EE | 0.294(0.149) | 0.452(0.179) | 0.569(0.200)

Table 3: The averaged estimation errors (standard deviations) of total
variation estimation of all methods across different noise variances.

##  5 Real Application - Consistent Fidelity Comparison of Generative Data.

Experimental Setting. We evaluate the effectiveness of the DisE, PE, and KDE
methods in measuring the fidelity of synthetic data. Using the MNIST dataset
[15], we train GANs for 100, 300, and 500 epochs, subsequently generating
images with each of these models, as illustrated in Figure 5. Due to the high
dimensionality and sparsity of image data, we employ pretrained ResNet18 [10]
and random projection [27] to obtain embeddings of both real and synthetic
images. Following the literature, which commonly assumes the normality of
embeddings of generative data [14, 5], we then estimate the total variation
between each generated dataset and the original MNIST dataset using the DisE,
PE, and KDE methods. As illustrated in Figure 5, GANs trained for more epochs
generate images of greater fidelity. Consequently, the total variation between
real images and synthetic images generated after 100, 300, and 500 epochs
should follow a decreasing pattern. Hence, in this experiment, we aim to
consistently compare all methods in terms of their ability to provide a
correct ranking of fidelity.

Experimental Result. In Table 4, we present the fidelity of images generated
by GANs trained over varying epochs, measured using total variation distance
estimated by three methods. The total variation distance between the
embeddings of real images and synthetic images generated after 100, 300, and
500 epochs estimated by DisE approach presents a decreasing pattern across all
cases, aligning with the expected quality ranking of the generated models.
However, the fidelity measured by the PE approach deviates from the expected
ranking when the embedding dimension is 50. Similarly, the fidelity measured
by the KDE approach fails to align with the correct ranking when the embedding
dimension is 35. This study demonstrates the effectiveness of the proposed
DisE method in measuring the fidelity of synthetic data, providing a correct
ranking of quality of generative data.

Figure 5: 25 synthetic images generated by GANs after 100, 300, and 500 epochs
of training are displayed from left to right. Table 4: Fidelity rankings of
images generated by GANs trained after varying epochs: Fidelity is measured
using the total variation estimated by different methods. The dimension of
embeddings is set to 20, 35, and 50 for ResNet18 and 35 for the random
projection method.

Method | Embedding-dim | 100 epochs | 300 epochs | 500 epochs | Correct Ranking
---|---|---|---|---|---
DisE | Resnet18-20 | 0.342 (0.068) | 0.153 (0.038) | 0.148 (0.055) | ✓
Resnet18-35 | 0.412 (0.074) | 0.187(0.059) | 0.146 (0.050) | ✓
Resnet18-50 | 0.436 (0.074) | 0.193 (0.072) | 0.186 (0.041) | ✓
RandProj-35 | 0.881 (0.042) | 0.569 (0.083) | 0.539 (0.090) | ✓
PE | Resnet18-20 | 0.483 (0.073) | 0.301 (0.051) | 0.286 (0.063) | ✓
Resnet18-35 | 0.627 (0.076) | 0.436 (0.065) | 0.431 (0.087) | ✓
Resnet18-50 | 0.767 (0.044) | 0.561 (0.061) | 0.563 (0.077) | ✗
| RandProj-35 | 0.894 (0.038) | 0.649 (0.092) | 0.628 (0.101) | ✓
KDE | Resnet18-20 | 0.768 (0.025) | 0.707 (0.017) | 0.703 (0.026) | ✓
Resnet18-35 | 0.907 (0.014) | 0.871 (0.013) | 0.872 (0.020) | ✗
Resnet18-50 | 0.967 (0.005) | 0.944 (0.007) | 0.943 (0.010) | ✓
RandProj-35 | 0.999 (0.002) | 0.998 (0.008) | 0.997 (0.011) | ✓

##  6 Conclusion

In this paper, we propose a novel approach to estimate the TV distance between
two distributions using a classification-based method. This method leverages
the quantitative relationship between Bayes risk and TV distance.
Specifically, we examine a scenario where both distributions are Gaussian,
establishing theoretical results regarding the convergence of our approach.
Our findings reveal an intriguing phenomenon: the estimation error of the TV
distance is dependent on the true separation between the distributions. In
other words, the TV distance is easier to estimate when the distributions are
farther apart. The experimental results demonstrate the superior performance
of our proposed discriminative estimation approach over several existing
methods in estimating total variation distance. While currently confined to
this particular metric, our discriminative approach holds promise for broader
applications in estimating various divergence metrics. Future endeavors will
focus on extending our method to encompass other divergence metrics and
establishing statistical assurances for estimation accuracy.

##  7 Experiments Compute Resources

All simulation and real application experiments in this work were conducted on
a computer equipped with Apple M1 Pro chip, 10-core CPU and 16-core GPU. The
training epochs in real application experiments were $500$.

##  8 Societal Impacts

This work introduces a novel discriminative method for estimating the total
variation distance between two distributions, specifically targeting Gaussian
distributions and the exponential family. This approach can act as a fidelity
auditor for generative data, allowing researchers to directly assess the
fidelity of synthetic data from generative models for model selection.
Consequently, this approach can improve the training of machine learning
models, resulting in more realistic, robust, and reliable models.

## References

  * [1] Jean-Yves Audibert and Alexandre B. Tsybakov.  Fast learning rates for plug-in classifiers.  The Annals of Statistics, 35(2):608–633, 2007.
  * [2] Lucio Barabesi and Luca Pratelli.  An inequality for the total variation distance between high-dimensional centered gaussian laws.  Statistics & Probability Letters, page 110148, 2024.
  * [3] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe.  Convexity, classification, and risk bounds.  Journal of the American Statistical Association, 101(473):138–156, 2006.
  * [4] Claire McKay Bowen and Joshua Snoke.  Comparative study of differentially private synthetic data algorithms from the nist pscr differential privacy synthetic data challenge.  J. Priv. Confidentiality, 11, 2019.
  * [5] Sanghyuk Chun.  Improved probabilistic image-text representations.  In The Twelfth International Conference on Learning Representations, 2024.
  * [6] Sami Davies, Arya Mazumdar, Soumyabrata Pal, and Cyrus Rashtchian.  Lower bounds on the total variation distance between mixtures of two gaussians.  In International Conference on Algorithmic Learning Theory, pages 319–341. PMLR, 2022.
  * [7] Luc Devroye, Abbas Mehrabian, and Tommy Reddad.  The total variation distance between high-dimensional gaussians with the same mean.  arXiv preprint arXiv:1810.08693, 2018.
  * [8] Patrik R Gerber, Yanjun Han, and Yury Polyanskiy.  Minimax optimal testing by classification.  In The Thirty Sixth Annual Conference on Learning Theory, pages 5395–5432. PMLR, 2023.
  * [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [11] James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N. Cohen, and Adrian Weller.  Synthetic data - what, why and how?  ArXiv, abs/2205.03257, 2022.
  * [12] T. Klein and E. Rio.  Concentration around the mean for maxima of empirical processes.  The Annals of Probability, 33(3):1060 – 1077, 2005.
  * [13] Vladimir Koltchinskii.  Oracle inequalities in empirical risk minimization and sparse recovery problems: École D’Été de Probabilités de Saint-Flour XXXVIII-2008, volume 2033.  Springer Science & Business Media, 2011.
  * [14] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen.  The role of imagenet classes in fréchet inception distance.  In The Eleventh International Conference on Learning Representations, 2023.
  * [15] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [16] Kevin R Moon and Alfred O Hero.  Ensemble estimation of multivariate f-divergence.  In 2014 IEEE International Symposium on Information Theory, pages 356–360. IEEE, 2014.
  * [17] XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan.  On surrogate loss functions and $f$-divergences.  Ann. Statist., 37(2):876–904, 2009.
  * [18] Morteza Noshad, Kevin R Moon, Salimeh Yasaei Sekeh, and Alfred O Hero.  Direct estimation of information divergence using nearest neighbor ratios.  In 2017 IEEE International Symposium on Information Theory (ISIT), pages 903–907. IEEE, 2017.
  * [19] Paul Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and Ilya O Tolstikhin.  Practical and consistent estimation of f-divergences.  Advances in Neural Information Processing Systems, 32, 2019.
  * [20] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly.  Assessing generative models via precision and recall.  ArXiv, abs/1806.00035, 2018.
  * [21] Hiroaki Sasaki, Yung-Kyun Noh, and Masashi Sugiyama.  Direct density-derivative estimation and its application in kl-divergence approximation.  In Artificial Intelligence and Statistics, pages 809–818. PMLR, 2015.
  * [22] Bernard W Silverman.  Density estimation for statistics and data analysis.  Routledge, 2018.
  * [23] Joshua Snoke, Gillian M Raab, Beata Nowok, Chris Dibben, and Aleksandra Slavkovic.  General and specific utility measures for synthetic data.  Journal of the Royal Statistical Society Series A: Statistics in Society, 181(3):663–688, 2018.
  * [24] Sreejith Sreekumar and Ziv Goldfeld.  Neural estimation of statistical divergences.  Journal of machine learning research, 23(126):1–75, 2022.
  * [25] Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau.  Benchmarking differentially private synthetic data generation algorithms.  arXiv preprint arXiv:2112.09238, 2021.
  * [26] Alexander B Tsybakov.  Optimal aggregation of classifiers in statistical learning.  The Annals of Statistics, 32(1):135–166, 2004.
  * [27] Santosh S Vempala.  The random projection method, volume 65.  American Mathematical Soc., 2005.
  * [28] Jiasen Yang, Qiang Liu, Vinayak Rao, and Jennifer Neville.  Goodness-of-fit testing for discrete distributions via stein discrepancy.  In International Conference on Machine Learning, pages 5561–5570. PMLR, 2018.
  * [29] Jun Zhang, Graham Cormode, Cecilia M. Procopiuc, Divesh Srivastava, and Xiaokui Xiao.  Privbayes: private data release via bayesian networks.  Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, 2014.

##  Appendix A Proof of Lemmas

###  A.1 Proof of Lemma 3.2

Given that $\mathcal{D}$ is the mixture of two Gaussian distribution
$\mathbb{P}(\bm{x})$ and $\mathbb{Q}(\bm{x})$, where

$\displaystyle\mathbb{P}(\bm{x})=(2\pi)^{-\frac{p}{2}}\mathrm{det}(\bm{\Sigma}_{1})^{-\frac{1}{2}}\exp\left(-\frac{1}{2}(\bm{x}-\bm{\mu}_{1})^{T}\bm{\Sigma}_{1}^{-1}(\bm{x}-\bm{\mu}_{1})\right),$
$\displaystyle\mathbb{Q}(\bm{x})=(2\pi)^{-\frac{p}{2}}\mathrm{det}(\bm{\Sigma}_{2})^{-\frac{1}{2}}\exp\left(-\frac{1}{2}(\bm{x}-\bm{\mu}_{2})^{T}\bm{\Sigma}_{2}^{-1}(\bm{x}-\bm{\mu}_{2})\right),$

and $\mathbb{D}(\bm{x})=\frac{\mathbb{P}(\bm{x})+\mathbb{Q}(\bm{x})}{2}$. For
a classifier $f:\mathbb{R}^{p}\rightarrow\\{0,1\\}$, its risk is given as

$R(f)=\int_{\mathbb{R}^{p}}\mathbb{D}(\bm{x})\big{[}P\left(Y=1|\bm{X}\right)\cdot
I(f(\bm{x})=0)+P\left(Y=0|\bm{X}\right)\cdot I(f(\bm{x})=1)\big{]}d\bm{x}.$

The term
$P\left(\bm{Y}=1|\bm{X}\right)=\frac{\mathbb{P}(\bm{X})}{\mathbb{P}(\bm{X})+\mathbb{Q}(\bm{X})}$
and
$P\left(\bm{Y}=0|\bm{X}\right)=\frac{\mathbb{Q}(\bm{X})}{\mathbb{P}(\bm{X})+\mathbb{Q}(\bm{X})}$,
thus we have

$R(f)=\int_{\bm{X}}\mathbb{D}(\bm{x})\left[\eta(\bm{x})\cdot
I(f(\bm{x})=0)+(1-\eta(\bm{x}))\cdot I(f(\bm{x})=1)\right]d\bm{x}.$

To minimize the risk, the optimal classifier is

$f^{\star}(x)=I\left(\eta(\bm{x})>\frac{1}{2}\right)=I\big{(}\mathbb{P}(\bm{x})>\mathbb{Q}(\bm{x})\big{)}=I\left(\log\frac{\mathbb{P}(\bm{x})}{\mathbb{Q}(\bm{x})}>0\right)$

Next,

$\displaystyle\frac{\mathbb{P}(\bm{x})}{\mathbb{Q}(\bm{x})}$
$\displaystyle=\frac{\mathrm{det}(\bm{\Sigma}_{1})^{-\frac{1}{2}}\exp\left(-\frac{1}{2}(\bm{x}-\bm{\mu}_{1})^{T}\bm{\Sigma}_{1}^{-1}(\bm{x}-\bm{\mu}_{1})\right)}{\mathrm{det}(\bm{\Sigma}_{2})^{-\frac{1}{2}}\exp\left(-\frac{1}{2}(\bm{x}-\bm{\mu}_{2})^{T}\bm{\Sigma}_{2}^{-1}(\bm{x}-\bm{\mu}_{2})\right)}$
$\displaystyle=\frac{\mathrm{det}(\bm{\Sigma}_{2})}{\mathrm{det}(\bm{\Sigma}_{1})}\cdot\exp\left(\frac{1}{2}(\bm{x}-\bm{\mu}_{2})^{T}\bm{\Sigma}_{2}^{-1}(\bm{x}-\bm{\mu}_{2})-\frac{1}{2}(\bm{x}-\bm{\mu}_{1})^{T}\bm{\Sigma}_{1}^{-1}(\bm{x}-\bm{\mu}_{1})\right).$

Considering that
$\text{sign}(\mathbb{P}(\bm{x})-\mathbb{Q}(\bm{x}))=\text{sign}(\log\mathbb{P}(\bm{x})-\log\mathbb{Q}(\bm{x}))$.
Therefore, the Bayes classifier can be written as

$\displaystyle
f^{\star}(\bm{x})=I\left(\log\left(\frac{\mathrm{det}(\bm{\Sigma}_{2})}{\mathrm{det}(\bm{\Sigma}_{1})}\right)+(\bm{x}-\bm{\mu}_{2})^{T}\bm{\Sigma}_{2}^{-1}(\bm{x}-\bm{\mu}_{2})-(\bm{x}-\bm{\mu}_{1})^{T}\bm{\Sigma}_{1}^{-1}(\bm{x}-\bm{\mu}_{1})>0\right).$

This completes the proof. ∎

###  A.2 Proof of Lemma 3.3

We first define $R_{\phi}(h)=\mathbb{E}\left[(\phi(h(\bm{X}))-Y)^{2}\right]$,
which can be expressed as

$\displaystyle\mathbb{E}\left[(\phi(h(\bm{X}))-Y)^{2}\right]=$
$\displaystyle\int_{\mathbb{R}^{p}}\mathbb{D}(\bm{x})\Big{[}\eta(\bm{x})(\phi(h(\bm{x}))-1)^{2}+(1-\eta(\bm{x}))\phi^{2}(h(\bm{x}))\Big{]}d\bm{x}.$

Here $\phi(x)=1/(1+\exp(-x))$. For each $\bm{x}$, we have

$\displaystyle\eta(\bm{x})(\phi(h(\bm{x}))-1)^{2}+(1-\eta(\bm{x}))\phi^{2}(h(\bm{x}))$
$\displaystyle=$
$\displaystyle\eta(\bm{x})\left(\frac{1}{1+\exp(h(\bm{x}))}\right)^{2}+(1-\eta(\bm{x}))\left(\frac{\exp(h(\bm{x}))}{1+\exp(h(\bm{x}))}\right)^{2}.$
(8)

Clearly, (A.2) is minimized when $\phi(h(\bm{x}))=\eta(\bm{x})$, leading to

$\displaystyle
h^{\star}_{\phi}(\bm{x})=\log\left(\frac{\eta(\bm{x})}{1-\eta(\bm{x})}\right)=\log\left(\frac{\mathbb{P}(\bm{x})}{\mathbb{Q}(\bm{x})}\right).$

Finally, we have

$\displaystyle h_{\phi}^{\star}(\bm{x})$
$\displaystyle=\log\left(\frac{\mathbb{P}(\bm{x})}{\mathbb{Q}(\bm{x})}\right)=\log\left(\frac{\mathrm{det}(\bm{\Sigma}_{2})}{\mathrm{det}(\bm{\Sigma}_{1})}\right)+(\bm{x}-\bm{\mu}_{2})^{T}\bm{\Sigma}_{2}^{-1}(\bm{x}-\bm{\mu}_{2})-(\bm{x}-\bm{\mu}_{1})^{T}\bm{\Sigma}_{1}^{-1}(\bm{x}-\bm{\mu}_{1}).$

This completes the proof. ∎

##  Appendix B Proof of Theorems

###  B.1 Proof of Theorem 3.4

First, the convergence of $\widehat{\mathrm{TV}}(\mathbb{P},\mathbb{Q})$ to
$\mathrm{TV}(\mathbb{P},\mathbb{Q})$ is implied by the convergence of
$R(\widehat{f})-R(f^{\star})$, where $\widehat{f}$ be the plug-in classifier
defined in (5). Specifically,

$\displaystyle\widehat{f}(\bm{x})=I\left(\phi(\widehat{h}(\bm{x}))>1/2\right)=I\left(\frac{\exp(\widehat{h}(\bm{x}))}{1+\exp(\widehat{h}(\bm{x}))}>1/2\right).$

To simplify notation, we denote
$\widehat{\eta}(\bm{x})=\phi(\widehat{h}(\bm{x}))$.

Step 1: Establishing the connection between $R(\widehat{f})-R(f^{\star})$ and
$\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}$

Specifically, we first decompose $R(\widehat{f})-R(f^{\star})$ into two parts:

$\displaystyle
R(\widehat{f})-R(f^{\star})=\mathbb{E}\left[I(\widehat{f}(\bm{X})\neq
f^{\star}(\bm{X}))|2\eta(\bm{X})-1|\right]$ $\displaystyle=$ $\displaystyle
2\mathbb{E}\Big{[}I(\widehat{f}(\bm{X})\neq
f^{\star}(\bm{X}))|\eta(\bm{X})-1/2|I(|\eta(\bm{X})-1/2|\leq t)\Big{]}$
$\displaystyle+$ $\displaystyle 2\mathbb{E}\Big{[}I(\widehat{f}(\bm{X})\neq
f^{\star}(\bm{X}))|\eta(\bm{X})-1/2|I(|\eta(\bm{X})-1/2|>t)\Big{]}\triangleq
I_{1}+I_{2},$

for any positive constant $t>0$.

Next, we turn to bound $I_{1}$ and $I_{2}$ separately. Following from the fact
that $|\eta(\bm{x})-1/2|\leq|\eta(\bm{x})-\widehat{\eta}(\bm{x})|$ when
$\widehat{f}(\bm{x})\neq f^{\star}(\bm{x})$, we have

$\displaystyle I_{1}=$ $\displaystyle
2\mathbb{E}\Big{[}I(\widehat{f}(\bm{X})\neq
f^{\star}(\bm{X}))|\eta(\bm{X})-1/2|I(|\eta(\bm{X})-1/2|\leq t)\Big{]}$
$\displaystyle\leq$ $\displaystyle 2\mathbb{E}\Big{[}I(\widehat{f}(\bm{X})\neq
f^{\star}(\bm{X}))|\eta(\bm{X})-\widehat{\eta}(\bm{x})|I(|\eta(\bm{X})-1/2|\leq
t)\Big{]}$ $\displaystyle\leq$ $\displaystyle
2\sqrt{\mathbb{E}\Big{[}(\eta(\bm{X})-\widehat{\eta}(\bm{x}))^{2}\Big{]}}\cdot\sqrt{\mathbb{P}(|\eta(\bm{X})-1/2|\leq
t)}\leq
2\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}C_{0}^{1/2}t^{\gamma/2},$
(9)

where the last inequality follows from the Cauchy–Schwarz inequality.

Next, $I_{2}$ can be bounded as

$\displaystyle I_{2}=$ $\displaystyle
2\mathbb{E}\Big{[}I(\widehat{f}(\bm{X})\neq
f^{\star}(\bm{X}))|\eta(\bm{X})-1/2|I(|\eta(\bm{X})-1/2|>t)\Big{]}$
$\displaystyle\leq$ $\displaystyle 2\mathbb{E}\Big{[}I(\widehat{f}(\bm{X})\neq
f^{\star}(\bm{X}))|\eta(\bm{X})-\widehat{\eta}(\bm{x})|I(|\eta(\bm{X})-1/2|>t)\Big{]}$
$\displaystyle\leq$ $\displaystyle
2\mathbb{E}\Big{[}\left(\eta(\bm{X})-\widehat{\eta}(\bm{x})\right)^{2}\Big{]}t^{-1}=2t^{-1}\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}.$
(10)

Combining (B.1) and (B.1) yields

$\displaystyle R(\widehat{f})-R(f^{\star})\leq$ $\displaystyle
2\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}C_{0}^{1/2}t^{\gamma/2}+2t^{-1}\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}.$

Setting
$t=C_{0}^{-\frac{1}{\gamma+2}}\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{\frac{2}{\gamma+2}}$
yields

$\displaystyle R(\widehat{f})-R(f^{\star})\leq
4C_{0}^{\frac{1}{\gamma+2}}\left(\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}\right)^{\frac{\gamma+1}{\gamma+2}}.$

Step 2. Establish the convergence of
$\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}$

For the mixed dataset
$\mathcal{D}=\\{\bm{x}_{i}\\}_{i=1}^{n}\cup\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{n}$,
we introduce a dataset
$\mathcal{D}_{0}=\\{(\bm{x}_{i}^{(0)},y_{i}^{(0)})\\}_{i=1}^{2n}$ with
$(\bm{x}_{i}^{(0)},y_{i}^{(0)})=(\bm{x}_{i},1)$ and
$(\bm{x}_{n+i}^{(0)},y_{n+i}^{(0)})=(\widetilde{\bm{x}}_{i},0)$. Here
$\mathcal{D}_{0}$ can be understood as a set of i.i.d. realizations of
$(\bm{X},Y)$ with
$\bm{X}\sim\frac{1}{2}N(\bm{\mu}_{1},\bm{\Sigma}_{1})+\frac{1}{2}N(\bm{\mu}_{2},\bm{\Sigma}_{2})$
and
$P(Y=1|\bm{X})=\frac{\mathbb{P}(\bm{X})}{\mathbb{P}(\bm{X})+\mathbb{Q}(\bm{X})}$.
Under the distribution of $(\bm{X},Y)$, we first define
$R_{\phi}(h)=\mathbb{E}\left[(\phi(h(\bm{X}))-Y)^{2}\right]$ as

$\displaystyle
R_{\phi}(h)-R_{\phi}(h^{\star}_{\phi})=\mathbb{E}\left[(\phi(h(\bm{X}))-Y)^{2}\right]-\mathbb{E}\left[(\phi(h^{\star}_{\phi}(\bm{X}))-Y)^{2}\right]$
$\displaystyle=$
$\displaystyle\mathbb{E}\left\\{\eta(\bm{X})[\phi(h(\bm{X}))-1]^{2}+[1-\eta(\bm{X})]\phi^{2}(h(\bm{X}))-\eta(\bm{X})(1-\eta(\bm{X}))\right\\}$
$\displaystyle=$
$\displaystyle\mathbb{E}\left[\left(\eta(\bm{X})-\phi(h(\bm{X})\right)^{2}\right]=\|\eta-\eta_{h}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}.$

Next, we define $\widehat{R}_{\phi}(h)$ as an empirical version of
$R_{\phi}(h)$.

$\displaystyle\widehat{R}_{\phi}(h)=\frac{1}{2n}\sum_{i=1}^{2n}\left(\phi(h(\bm{x}_{i}^{(0)}))-y_{i}^{(0)}\right)^{2}.$

Here
$\widehat{h}=\arg\min_{h\in\mathcal{H}}\widehat{R}_{\phi}(h)+\lambda\|\bm{\beta}\|_{2}^{2}$
and $\widehat{\eta}(\bm{x})=\phi(\widehat{h}(\bm{x}))$. Denote
$\mathcal{A}=\\{\mathcal{D}:\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}>\delta\\}$
and let
$\mathcal{H}_{0}=\\{h\in\mathcal{H}:\|\eta-\eta_{h}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}>\delta\\}$
be a subset of the function class $\mathcal{H}$. First, if the dataset
$\mathcal{D}_{0}\in\mathcal{A}$, then we have $\widehat{h}\in\mathcal{H}_{0}$,
implying
$\sup_{h\in\mathcal{H}_{0}}\widehat{R}_{\phi}(h_{\phi}^{\star})-\widehat{R}_{\phi}(h)+\lambda(\|\bm{\beta}^{\star}\|_{2}^{2}-\|\bm{\beta}\|_{2}^{2})\geq
0$ due to the optimality of $\widehat{h}$ in minimizing
$\widehat{R}_{\phi}(h)$ within $\mathcal{H}$. Therefore, we have

$\displaystyle P(\mathcal{A})\leq
P\left(\sup_{f\in\mathcal{H}_{0}}\widehat{R}_{\phi}(h_{\phi}^{\star})-\widehat{R}(h)+\lambda\|\bm{\beta}^{\star}\|_{2}^{2}-\lambda\|\bm{\beta}\|_{2}^{2}\geq
0\right).$ (11)

Next we can decompose $\mathcal{H}_{0}$ as
$\mathcal{H}_{0}=\cup_{i=1}^{\infty}\mathcal{H}_{0}^{(i)}$ with
$\mathcal{H}_{0}^{(i)}$ being defined as

$\displaystyle\mathcal{H}_{0}^{(i)}=\\{h\in\mathcal{H}_{0}:2^{i-1}\delta\leq
R_{\phi}(h)-R_{\phi}(h^{\star}_{\phi})\leq 2^{i}\delta\\}$

Therefore, (11) can be equivalently written as

$\displaystyle P(\mathcal{A})\leq$ $\displaystyle
P\left(\sup_{\cup_{i=1}^{\infty}\mathcal{H}_{0}^{(i)}}\widehat{R}_{\phi}(h_{\phi}^{\star})-\widehat{R}(h)+\lambda\|\bm{\beta}^{\star}\|_{2}^{2}-\lambda\|\bm{\beta}\|_{2}^{2}\geq
0\right)$ $\displaystyle\leq$
$\displaystyle\sum_{i=1}^{\infty}P\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\widehat{R}_{\phi}(h^{\star})-\widehat{R}_{\phi}(h)+\lambda\|\bm{\beta}^{\star}\|_{2}^{2}-\lambda\|\bm{\beta}\|_{2}^{2}>0\right)$
$\displaystyle\leq$
$\displaystyle\sum_{i=1}^{\infty}P\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\widehat{R}_{\phi}(h^{\star}_{\phi})-R_{\phi}(f_{\phi}^{\star})-\widehat{R}_{\phi}(h)+R_{\phi}(h)>\inf_{h\in\mathcal{H}_{0}^{(i)}}R_{\phi}(h)-R_{\phi}(h_{\phi}^{\star})+\lambda\|\bm{\beta}\|_{2}^{2}-\lambda\|\bm{\beta}^{\star}\|_{2}^{2}\right)$
$\displaystyle\leq$
$\displaystyle\sum_{i=1}^{\infty}P\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\widehat{R}_{\phi}(h^{\star}_{\phi})-R_{\phi}(h_{\phi}^{\star})-\widehat{R}_{\phi}(h)+R_{\phi}(h)>2^{i-1}\delta-\lambda\|\bm{\beta}^{\star}\|_{2}^{2}\right)$
$\displaystyle\leq$
$\displaystyle\sum_{i=1}^{\infty}P\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\widehat{R}_{\phi}(h^{\star}_{\phi})-R_{\phi}(h_{\phi}^{\star})-\widehat{R}_{\phi}(h)+R_{\phi}(h)>2^{i-2}\delta-\lambda\|\bm{\beta}^{\star}\|_{2}^{2}\right)\triangleq\sum_{i=1}^{\infty}I_{i}.$

where the last inequality by choosing
$\lambda=\delta/(2\|\bm{\beta}^{\star}\|_{2}^{2})$.

Step 3. Bounding $I_{i}$

First, we define

$\displaystyle
D_{i}(h)=\left(\phi(h_{\phi}^{\star}(\bm{x}_{i}^{(0)}))-y_{i}^{(0)}\right)^{2}-\left(\phi(h(\bm{x}_{i}^{(0)}))-y_{i}^{(0)}\right)^{2},$
$\displaystyle
D(h)=\mathbb{E}\left[(\phi(h_{\phi}^{\star}(\bm{X}))-Y)^{2}-(\phi(h(\bm{X}))-Y)^{2}\right].$

Then $I_{i}$ can be rewritten as

$\displaystyle I_{i}=$ $\displaystyle
P\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\frac{1}{2n}\sum_{i=1}^{2n}[D_{i}(h)-D(h)]>2^{i-2}\delta\right)$
$\displaystyle=$ $\displaystyle
P\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\frac{1}{2n}\sum_{i=1}^{2n}[D_{i}(h)-D(h)]-\nu_{i}(\mathcal{D}_{0})>2^{i-2}\delta-\nu_{i}(\mathcal{D}_{0})\right),$

where
$\nu_{i}(\mathcal{D}_{0})=\mathbb{E}\left[\sup_{h\in\mathcal{H}_{0}^{(i)}}\frac{1}{2n}\sum_{i=1}^{2n}[D_{i}(h)-D(h)]\right]$.
Here we assume $\nu_{i}(\mathcal{D}_{0})\leq 2^{i-3}\delta$ and then we have

$\displaystyle I_{i}\leq
P\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\frac{1}{2n}\sum_{i=1}^{2n}[D_{i}(h)-D(h)]-\nu_{i}(\mathcal{D}_{0})>2^{i-3}\delta\right).$

Step 4. Verifying $\nu_{i}(\mathcal{D}_{0})\leq 2^{i-3}\delta$ for $i\geq 1$

Next, we intend to present the conditions under which
$\nu_{i}(\mathcal{D}_{0})\leq 2^{i-2}\delta$. Let $\mathcal{D}^{\prime}$ be an
independent copy of $\mathcal{D}_{0}$ and $(\tau_{i})_{i=1}^{2n}$ be
independent Rademacher random variables. Then we have

$\displaystyle\nu_{i}(\mathcal{D}_{0})$
$\displaystyle=\frac{1}{2n}\mathbb{E}_{\mathcal{D}_{0}}\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\sum_{i=1}^{2n}[D_{i}(h)-D(h)]\right)\leq\frac{1}{2n}\mathbb{E}_{\mathcal{D}_{0}}\left\\{\mathbb{E}_{\mathcal{D}^{\prime}}\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\sum_{i=1}^{2n}[D_{i}(h)-D_{i}^{\prime}(h)]\Big{|}\mathcal{D}_{0}\right)\right\\}$
$\displaystyle=\frac{1}{2n}\mathbb{E}_{\mathcal{D}_{0},\mathcal{D}^{\prime}}\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\sum_{i=1}^{2n}\tau_{i}[D_{i}(h)-D_{i}^{\prime}(h)]\right)$
$\displaystyle=\frac{1}{2n}\mathbb{E}_{\mathcal{D}_{0},\mathcal{D}^{\prime}}\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\sum_{i=1}^{2n}\tau_{i}[D_{i}(h)-D_{i}(h_{0})+D_{i}^{\prime}(h_{0})-D_{i}^{\prime}(h)]\right)$
$\displaystyle\leq\frac{1}{n}\mathbb{E}\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\sum_{i=1}^{2n}\tau_{i}[D_{i}(h)-D_{i}(h_{0})]\right),$

for any $h_{0}\in\mathcal{H}_{0}^{(i)}$. Here the first inequality follows
from the Jensen’s inequality, and the second equality follows from the
standard symmetrization argument.

Note that conditional on $\mathcal{D}_{0}$,
$\frac{1}{\sqrt{2n}}\sum_{i=1}^{2n}\tau_{i}D_{i}(h)$ is a sub-Gaussian process
with respect to $d$, where

$\displaystyle\rho^{2}\big{(}h_{1},h_{2}\big{)}=\frac{1}{2n}\sum_{i=1}^{2n}\big{(}D_{i}(h_{1})-D_{i}(h_{2})\big{)}^{2},$

for any $h_{1},h_{2}\in\mathcal{H}_{0}^{(i)}$. It then follows from Theorem
3.1 of [13] that

$\displaystyle\frac{1}{\sqrt{2n}}\mathbb{E}_{\mathcal{D}_{0}}\left(\sup_{h\in\mathcal{H}_{0}^{(i)}}\sum_{i=1}^{2n}\tau_{i}[D_{i}(h)-D_{i}(h_{0})]\right)\lesssim\mathbb{E}\left(\int_{0}^{D(\mathcal{H}_{0}^{(i)})}H^{1/2}\big{(}\mathcal{H}_{0}^{(i)},\rho,\eta\big{)}d\eta\right),$

where $D(\mathcal{H}_{0}^{(i)})$ is the diameter of $\mathcal{H}_{0}^{(i)}$
with respect to $\rho$, and $H(\mathcal{H}_{0}^{(i)},\rho,\eta)$ is the
$\eta$-entropy of $(\mathcal{H}_{0}^{(i)},\rho)$. For any
$h_{1},h_{2}\in\mathcal{H}_{0}^{(i)}$, it follows that

$\displaystyle\mathbb{E}\rho^{2}\big{(}h_{1},h_{2}\big{)}=$
$\displaystyle\frac{1}{2n}\sum_{i=1}^{2n}\mathbb{E}\big{(}D_{i}(h_{1})-D_{i}(h_{2})\big{)}^{2}\leq\frac{1}{2n}\sum_{i=1}^{2n}\mathbb{E}\big{(}D_{i}^{2}(h_{1})\big{)}+\frac{1}{2n}\sum_{i=1}^{2n}\mathbb{E}\big{(}D_{i}^{2}(h_{2})\big{)}$
$\displaystyle\leq$ $\displaystyle
8\|\eta_{h_{1}}-\eta\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}+8\|\eta_{h_{2}}-\eta\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}.$

Therefore, we get

$\displaystyle\mathbb{E}D(\mathcal{H}_{0}^{(i)})\leq
4\sup_{h\in\mathcal{H}_{0}^{(i)}}\|\eta_{h}-\eta\|_{L_{2}(\mathbb{P}_{\bm{X}})}\leq\sqrt{2^{i+4}\delta}.$
(12)

Moreover,

$\displaystyle d^{2}\big{(}h_{1},h_{2}\big{)}=$
$\displaystyle\frac{1}{2n}\sum_{i=1}^{2n}\big{(}D_{i}(h_{1})-D_{i}(h_{2})\big{)}^{2}=\frac{1}{2n}\sum_{i=1}^{2n}\left((\phi(h_{1}(\bm{x}_{i}^{(0)}))-y_{i}^{(0)})^{2}-(\phi(h_{2}(\bm{x}_{i}^{(0)}))-y_{i}^{(0)})^{2}\right)^{2}$
$\displaystyle\leq$
$\displaystyle\frac{2}{n}\sum_{i=1}^{2n}\left(\phi\left(h_{1}(\bm{x}_{i}^{(0)})\right)-\phi\left((h_{2}(\bm{x}_{i}^{(0)}\right)\right)^{2}\leq\frac{1}{8n}\sum_{i=1}^{2n}\left(h_{1}(\bm{x}_{i}^{(0)})-h_{2}(\bm{x}_{i}^{(0)})\right)^{2}$
$\displaystyle\leq$
$\displaystyle\frac{1}{4}\|\bm{\beta}_{1}-\bm{\beta}_{2}\|_{2}^{2}\frac{1}{2n}\sum_{i=1}^{2n}\|\psi(\bm{x}_{i}^{(0)})\|_{2}^{2}\triangleq\frac{M(\mathcal{D}_{0})}{4}\|\bm{\beta}_{1}-\bm{\beta}_{2}\|_{2}^{2}$

where $h_{1}(\bm{x})=\bm{\beta}_{1}^{T}\psi(\bm{x})$,
$h_{2}(\bm{x})=\bm{\beta}_{2}^{T}\psi(\bm{x})$, the second inequality follows
from the fact that $\phi(x)$ is a $1/4$-Lipschitz function, and
$M(\mathcal{D}_{0})=\frac{1}{2n}\sum_{i=1}^{2n}\|\psi(\bm{x}_{i}^{(0)})\|_{2}^{2}$.
Thus, $\rho^{2}\big{(}h_{1},h_{2}\big{)}\leq\eta^{2}$ if
$\|\bm{\beta}_{1}-\bm{\beta}_{2}\|_{2}^{2}\leq\frac{M(\mathcal{D}_{0})\eta^{2}}{4}$.
This further leads to

$\displaystyle H(\mathcal{H}_{0}^{(i)},\rho,\eta)\leq
H\left(B_{2}(d),\|\cdot\|_{2},\frac{C_{\mathcal{H}}\sqrt{M(\mathcal{D}_{0})}\eta}{2}\right)\leq
d\log\left(\frac{6}{C_{\mathcal{H}}\sqrt{M(\mathcal{D}_{0})}\eta}\right),$

where $B_{2}(d)$ is the unit $l_{2}$-ball in $\mathbb{R}^{d}$ and the last
inequality follows by setting
$\frac{6}{C_{\mathcal{H}}\sqrt{M(\mathcal{D}_{0})}\eta}\leq 1$.

Then, applying the Dudley’s integral entropy bound [13], we have

$\displaystyle\nu_{i}(\mathcal{D}_{0})\lesssim$
$\displaystyle\frac{1}{\sqrt{n}}\mathbb{E}\left(\int_{0}^{D(\mathcal{H}_{0}^{(i)})}H^{1/2}\big{(}\mathcal{H}_{0}^{(i)},d,\eta\big{)}d\eta\right)$
$\displaystyle\lesssim$
$\displaystyle\mathbb{E}\left(\frac{1}{\sqrt{n}}\int_{0}^{D(\mathcal{H}_{0}^{(i)})}\sqrt{d\log\left(\frac{6}{C_{\mathcal{H}}\sqrt{M(\mathcal{D}_{0})}\eta}\right)}d\eta\right).$

For ease of notation, we let
$C_{1}=\frac{C_{\mathcal{H}}\sqrt{M(\mathcal{D}_{0})}}{6}$. Next,

$\displaystyle\sqrt{\frac{d}{n}}\int_{0}^{D(\mathcal{H}_{0}^{(i)})}\sqrt{\log\left(\frac{1}{C_{1}\eta}\right)}d\eta=\frac{\sqrt{d}}{C_{1}\sqrt{n}}\int_{0}^{C_{1}D(\mathcal{H}_{0}^{(i)})}\sqrt{\log\left(\frac{1}{\eta}\right)}d\eta$
$\displaystyle=$
$\displaystyle\frac{\sqrt{d}}{C_{1}\sqrt{n}}\int_{0}^{C_{1}D(\mathcal{H}_{0}^{(i)})}\sqrt{\log\left(\frac{1}{\eta}\right)}d\eta=\frac{\sqrt{d}}{C_{1}\sqrt{n}}\int_{\frac{1}{C_{1}D(\mathcal{H}_{0}^{(i)})}}^{+\infty}\frac{\sqrt{\log\left(t\right)}}{t^{2}}dt$
$\displaystyle=$
$\displaystyle\frac{1}{C_{1}}\sqrt{\frac{d}{n\log\left(\frac{1}{C_{1}D(\mathcal{H}_{0}^{(i)})}\right)}}\int_{\frac{1}{C_{1}D(\mathcal{H}_{0}^{(i)})}}^{+\infty}\frac{\log\left(t\right)}{t^{2}}dt.$
(13)

By the fact that $\int_{a}^{\infty}\log(x)/x^{2}dx=(\log(a)+1)/a$, we further
have

$\displaystyle(\ref{RR})\lesssim\frac{\sqrt{d}D(\mathcal{H}_{0}^{(i)})}{\sqrt{n}}\sqrt{\log\left(\frac{1}{C_{1}D(\mathcal{H}_{0}^{(i)})}\right)},$

where the inequality follows from the fact that $1/x+x\leq 2x$ for $x\geq 1$.
Next, by the fact that $f(x,y)=x\sqrt{\log(\frac{1}{xy})}$ is a concave
function, we further have

$\displaystyle\mathbb{E}\left(\frac{\sqrt{d}D(\mathcal{H}_{0}^{(i)})}{\sqrt{n}}\sqrt{\log\left(\frac{1}{C_{1}D(\mathcal{H}_{0}^{(i)})}\right)}\right)$
$\displaystyle\leq$
$\displaystyle\mathbb{E}\left(D(\mathcal{H}_{0}^{(i)})\right)\sqrt{\frac{d}{n}}\sqrt{\log\left(\frac{1}{\mathbb{E}\left(C_{1}D(\mathcal{H}_{0}^{(i)})\right)}\right)}\lesssim\sqrt{\frac{d2^{i+4}\delta}{n}}\sqrt{\log\left(\frac{1}{2^{i+4}\delta}\right)}.$

If
$\sqrt{\frac{d2^{i+4}\delta}{n}}\sqrt{\log\left(\frac{1}{2^{i+4}\delta}\right)}\leq
2^{i}\delta$, we have $\frac{d}{n}\log(n/d)\lesssim\delta$.

Step 5. Bounding $\sum_{i=1}^{\infty}I_{i}$

Applying Theorem 1.1 of [12] to $I_{i}$, we have

$\displaystyle
I_{1}\leq\exp\left(-\frac{2^{2i-4}\delta^{2}n^{2}}{8\nu_{i}(\mathcal{D}_{0})n+2V_{i}n+3\cdot
2^{i-3}\delta
n}\right)\leq\exp\left(-\frac{2^{2i-4}\delta^{2}n^{2}}{8V_{i}n+7\cdot
2^{i-2}\delta n}\right)$ (14)

where
$V_{i}=\sup_{h\in\mathcal{H}_{0}^{(i)}}\text{Var}\left[(\phi(h_{\phi}^{\star}(\bm{X}))-Y)^{2}-(\phi(h(\bm{X}))-Y)^{2}\right]$.
Next, we establish the relation betwen $V$ and $\delta$.

$\displaystyle V_{i}=$
$\displaystyle\sup_{h\in\mathcal{H}_{0}^{(i)}}\text{Var}\left[(\phi(h_{\phi}^{\star}(\bm{X}))-Y)^{2}-(\phi(h(\bm{X}))-Y)^{2}\right]$
$\displaystyle\leq$
$\displaystyle\sup_{h\in\mathcal{H}_{0}^{(i)}}\mathbb{E}\left[(\phi(h_{\phi}^{\star}(\bm{X}))-Y)^{2}-(\phi(h(\bm{X}))-Y)^{2}\right]^{2}$
$\displaystyle\leq$
$\displaystyle\sup_{h\in\mathcal{H}_{0}^{(i)}}\mathbb{E}\left[(\phi(h_{\phi}^{\star}(\bm{X}))-\phi(h(\bm{X})))\cdot(\phi(h_{\phi}^{\star}(\bm{X}))+\phi(h(\bm{X}))-2Y)\right]^{2}$
$\displaystyle\leq$ $\displaystyle
4\sup_{h\in\mathcal{H}_{0}^{(i)}}\mathbb{E}\left[\phi(h_{\phi}^{\star}(\bm{X}))-\phi(h(\bm{X}))\right]^{2}\leq
4\sup_{h\in\mathcal{H}_{0}^{(i)}}\|\eta-\eta_{h}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}\leq
2^{i+2}\delta.$

Therefore, (14) can be further bounded as $I_{i}\leq\exp\left(-C2^{i}\delta
n\right)\leq\exp\left(-Ci\delta n\right)$ for some positive constant $C$.

$\displaystyle\sum_{i=1}^{\infty}I_{i}\leq\sum_{i=1}^{\infty}\exp\left(-Ci\delta
n\right)\leq\frac{\exp(-C\delta n)}{1-\exp(-C\delta n)}\lesssim\exp(-C\delta
n).$

Since $\frac{d}{2n}\log(n)\lesssim\delta$, we further have
$\sum_{i=1}^{\infty}I_{i}\lesssim n^{-C}$ for some positive constant $C$.
Finally, we have

$\displaystyle\mathbb{P}\left(R(\widehat{f})-R(f^{\star})\geq
4C_{0}^{\frac{1}{\gamma+2}}\left(\frac{d\log
n}{2n}\right)^{\frac{\gamma+1}{\gamma+2}}\right)\leq\mathbb{P}\left(\|\eta-\widehat{\eta}\|_{L_{2}(\mathbb{P}_{\bm{X}})}^{2}\geq\frac{d\log
n}{2n}\right)\lesssim n^{-C},$

for some positive constant $C$. Therefore

$\displaystyle\mathbb{E}_{\mathcal{D}}\left\\{\mathrm{TV}(\mathbb{P},\mathbb{Q})-\widehat{\mathrm{TV}}(\mathbb{P},\mathbb{Q})\right\\}\leq
2\mathbb{E}\left(R(\widehat{f})-R(f^{\star})\right)\lesssim
C_{0}^{\frac{1}{\gamma+2}}\left(\frac{d\log
n}{2n}\right)^{\frac{\gamma+1}{\gamma+2}}.$

This completes the proof. ∎

###  B.2 Proof of Theorem 3.5

To prove Theorem 3.5, we first verify Assumption 3.1 for one-dimensional
Gaussian case. Suppose
$\mathbb{P}(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{(x-\mu)^{2}}{2}}$ and
$\mathbb{Q}(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{(x+\mu)^{2}}{2}}$ with
$\mu_{1}>0$.

$\displaystyle\mathbb{D}(x)=\frac{1}{2\sqrt{2\pi}}\left(e^{-\frac{(x-\mu)^{2}}{2}}+e^{-\frac{(x+\mu_{1})^{2}}{2}}\right)$

In this example, $\eta(x)$ is given as $\eta(x)=\frac{\exp(2\mu
x)}{1+\exp(2\mu x)}$. Let $\mathcal{B}(t)=\\{x\in\mathbb{R}:|\eta(x)-1/2|\leq
t\\}$ with $t\in[0,1/2]$. Using the definition of $\eta(x)$, $\mathcal{B}(t)$
can be equivalently written as

$\displaystyle\mathcal{B}(t)=\left\\{x\in\mathbb{R}:\frac{1}{2\mu}\log\left(\frac{1-2t}{1+2t}\right)\leq
x\leq\frac{1}{2\mu}\log\left(\frac{1+2t}{1-2t}\right)\right\\}.$

For ease of notations, we let
$S(t,\mu)=\frac{1}{2\mu}\log\left(\frac{1+2t}{1-2t}\right)$. By the symmetry
of $\mathbb{P}$ and $\mathbb{Q}$, we have

$\displaystyle P_{X}(\mathcal{B}(t))\leq\sqrt{\frac{1}{2\pi
e\mu^{2}}}\log\left(\frac{1+2t}{1-2t}\right),$

where the inequality follows from the fact that $d\mathbb{P}(x)/dx$ attains
its maximum value when $x=\mu-1$ Using the fact that
$\frac{x}{1+x}\leq\log(1+x)\leq x$ for $x>-1$, we have
$\log\left(\frac{1+2t}{1-2t}\right)\leq\frac{4t}{1-2t}$ for any $t\in[0,1/2)$.
Finally, it follows that

$\displaystyle P_{X}(\mathcal{B}(t))\leq\sqrt{\frac{8}{(1-2t)^{2}\pi
e\mu^{2}}}t.$

Therefore, for any $t\in[0,c)$ with $c<1/2$, we have

$\displaystyle P_{X}(\mathcal{B}(t))\leq\sqrt{\frac{8}{(1-2t)^{2}\pi
e\mu^{2}}}t\leq\sqrt{\frac{8}{(1-2c)^{2}\pi e\mu^{2}}}t.$

To derived the desired results on top of Theorem 3.4, it is necessary to
ensure $c>\left(\sqrt{\frac{8}{(1-2c)^{2}\pi
e\mu^{2}}}\right)^{-\frac{1}{3}}$, which then leads to $c^{3}>(1-2c)\sqrt{2\pi
e}\mu/4$. It can be verified that there exists $c_{0}\in(0,1/2)$ such that
$c_{0}^{3}>(1-2c_{0})\sqrt{2\pi e}\mu/4$. Therefore, for one-dimensional
Gaussian case, we have

$\displaystyle\mathbb{E}_{\mathcal{D}}\left\\{\mathrm{TV}(\mathbb{P},\mathbb{Q})-\widehat{\mathrm{TV}}(\mathbb{P},\mathbb{Q})\right\\}\lesssim\left(\sqrt{\frac{8}{(1-2c_{0})^{2}\pi
e\mu^{2}}}\right)^{\frac{1}{3}}\left(\frac{3\log
n}{2n}\right)^{\frac{2}{3}}\lesssim\mu^{-\frac{1}{3}}\left(\frac{\log
n}{2n}\right)^{\frac{2}{3}}.$

This completes the proof. ∎

###  B.3 Proof of Theorem 3.6

Proof of Theorem 3.6: Let $\mathbb{P}(\bm{x})$ and $\mathbb{Q}(\bm{x})$ be the
density functions of two different random variables from the exponential
family:

$\displaystyle\mathbb{P}(\bm{x})=h_{1}(\bm{x})\cdot\exp\left[\bm{\eta}_{1}(\bm{\theta_{1}})\cdot\bm{T}_{1}(\bm{x})-A_{1}(\bm{\theta_{1}})\right],$
$\displaystyle\mathbb{Q}(\bm{x})=h_{2}(\bm{x})\cdot\exp\left[\bm{\eta}_{2}(\bm{\theta_{2}})\cdot\bm{T}_{2}(\bm{x})-A_{2}(\bm{\theta_{2}})\right].$

According to proof of Lemma 3.2, the optimal classifier is

$f^{\star}(x)=\text{sign}\big{(}\mathbb{P}(\bm{x})-\mathbb{Q}(\bm{x})\big{)}.$

Observing that

$\displaystyle\frac{\mathbb{P}(\bm{x})}{\mathbb{Q}(\bm{x})}$
$\displaystyle=\frac{h_{1}(\bm{x})\cdot\exp\left[\bm{\eta}_{1}(\bm{\theta_{1}})\cdot\bm{T}_{1}(\bm{x})-A_{1}(\bm{\theta_{1}})\right]}{h_{2}(\bm{x})\cdot\exp\left[\bm{\eta}_{2}(\bm{\theta_{2}})\cdot\bm{T}_{2}(\bm{x})-A_{2}(\bm{\theta_{2}})\right]}$
$\displaystyle=\frac{h_{1}(\bm{x})}{h_{2}(\bm{x})}\cdot\exp\left[A_{2}(\bm{\theta_{2}})-A_{1}(\bm{\theta_{1}})+\bm{\eta}_{1}(\bm{\theta_{1}})\cdot\bm{T}_{1}(\bm{x})-\bm{\eta}_{1}(\bm{\theta_{2}})\cdot\bm{T}_{2}(\bm{x})\right],$

and that
$\text{sign}(\mathbb{P}(\bm{x})-\mathbb{Q}(\bm{x}))=\text{sign}\left(\log\left(\frac{\mathbb{P}(\bm{x})}{\mathbb{Q}(\bm{x})}\right)\right)$,
thus the optimal classifier is given as

$\displaystyle
f^{\star}(x)=I\left(\log\left(\frac{h_{1}(\bm{x})}{h_{2}(\bm{x})}\right)+A_{2}(\bm{\theta}_{2})-A_{1}(\bm{\theta}_{1})+\bm{T}_{1}(\bm{x})\bm{\eta}_{1}(\bm{\theta_{1}})-\bm{T}_{2}(\bm{x})\bm{\eta}(\bm{\theta_{2}})>0\right).$

This completes the proof. ∎

##  Appendix C Experimental Setting

###  C.1 Simulation Study

Two groups of Gaussian mixture
($\mathbb{D}(\bm{x})=\frac{\mathbb{P}(\bm{x})+\mathbb{Q}(\bm{x})}{2}$) samples
are generated as training data and testing data respectively. Training data
size is set to either $1000$ or $10,000$, while test data size is $50,000$.

Discriminative estimation (DisE): A classifier in the corresponding function
class $\widehat{f}$ is trained with the use of transformed training data, and
its classification error in test data can be used to estimate total variation
via $\widehat{\text{TV}}(\mathbb{P},\mathbb{Q})=1-2R(\widehat{f})$.

True total variation:  Since there is no closed form of TV distance between
two Gaussian distributions, as the standard, we employ the Monte Carlo method
to approximate the true total variation via
$\text{TV}(\mathbb{P},\mathbb{Q})\approx\frac{1}{N}\sum_{i=1}^{N}\left|\frac{\mathbb{Q}(\bm{x}_{i}^{\prime})-\mathbb{P}(\bm{x}_{i}^{\prime})}{\mathbb{P}(\bm{x}_{i}^{\prime})+\mathbb{Q}(\bm{x}_{i}^{\prime})}\right|$,
where
$\\{\bm{x}_{i}^{\prime}\\}_{i=1}^{N}\overset{\text{i.i.d.}}{\sim}\frac{\mathbb{P}+\mathbb{Q}}{2}$.

Parameter estimation (PE):
$\text{TV}(\widehat{\mathbb{P}},\widehat{\mathbb{Q}})\approx\frac{1}{N}\sum_{i=1}^{N}\left|\frac{\widehat{\mathbb{Q}}(\bm{x}_{i}^{\prime})-\widehat{\mathbb{P}}(\bm{x}_{i}^{\prime})}{\widehat{\mathbb{P}}(\bm{x}_{i}^{\prime})+\widehat{\mathbb{Q}}(\bm{x}_{i}^{\prime})}\right|$,
$\widehat{\mathbb{P}}$ and $\widehat{\mathbb{Q}}$ denote the multivariate
Gaussian distribution with parameters estimated based on
$\\{\bm{x}_{i}\\}_{i=1}^{n}$ and $\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{n}$,
respectively.

Kernel density estimation (KDE):
$\text{TV}(\widetilde{\mathbb{P}},\widetilde{\mathbb{Q}})\approx\frac{1}{N}\sum_{i=1}^{N}\left|\frac{\widetilde{\mathbb{Q}}(\bm{x}_{i}^{\prime})-\widetilde{\mathbb{P}}(\bm{x}_{i}^{\prime})}{\widetilde{\mathbb{P}}(\bm{x}_{i}^{\prime})+\widetilde{\mathbb{Q}}(\bm{x}_{i}^{\prime})}\right|$,
where $\widetilde{\mathbb{P}}$ and $\widetilde{\mathbb{Q}}$ denote the kernel
density estimation based on $\\{\bm{x}_{i}\\}_{i=1}^{n}$ and
$\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{n}$, respectively. We select the optimal
bandwidth based on Silverman’s rule of thumb [22].

Nearest neighbor ratio estimation (NNRE):
$\text{TV}(\mathbb{P},\mathbb{Q})\approx\frac{1}{M}\sum_{i=1}^{M}\tilde{g}\left(\frac{\eta
N_{i}}{M_{i}+1}\right)$, where $\tilde{g}(x)=\frac{1}{2}|x-1|$,
$\eta=\frac{M}{N}$ is ratio of samples from $\mathbb{P}$ and $\mathbb{Q}$. For
each sample $\bm{x}_{i}^{\prime}$ from
$\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{M}$, find out the $k$ nearest neighbors
in $\\{\bm{x}_{i}\\}_{i=1}^{N}\cup\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{M}$,
among which $N_{i}$ points from $\\{\bm{x}_{i}\\}_{i=1}^{N}$ and $M_{i}$
points from $\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{M}$. We select the optimal
choice of $k=\sqrt{M}$ [18]

Ensemble estimation (EE):
$\text{TV}(\mathbb{P},\mathbb{Q})\approx\frac{1}{n}\sum_{i=1}^{N}\tilde{g}\left(\frac{M_{2}(\rho_{2,k}(i))^{p}}{M_{1}(\rho_{1,k}(i))^{p}}\right)$,
where $\tilde{g}(x)=\frac{1}{2}|x-1|$. All samples in
$\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{n}$ are divided into two sets
$\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{N}$ and
$\\{\widetilde{\bm{x}}_{i}\\}_{i=N+1}^{N+M_{2}}$, and $M_{1}$ samples are
drawn from $\\{\bm{x}_{i}\\}_{i=1}^{n}$. For each sample $\bm{x}_{i}^{\prime}$
from $\\{\widetilde{\bm{x}}_{i}\\}_{i=1}^{N}$, find out the distance of $k$\-
nearest neighbor of $\bm{x}_{i}^{\prime}$ in $\\{\bm{x}_{i}\\}_{i=1}^{M_{1}}$,
denoted by $\rho_{1,k}(i)$, and the distance of $k$\- nearest neighbor of
$\bm{x}_{i}^{\prime}$ in $\\{\widetilde{\bm{x}}_{i}\\}_{i=N+1}^{n}$, denoted
by $\rho_{2,k}(i)$. The optimal choice of $k=\sqrt{N}$ [16].

Table 5: Competing Total Variation Estimation Methods Methods | Estimator | Samples
---|---|---
True Total Variation | $\frac{1}{N}\sum_{i=1}^{N}\left|\frac{\mathbb{Q}(\bm{x}_{i}^{\prime})-\mathbb{P}(\bm{x}_{i}^{\prime})}{\mathbb{P}(\bm{x}_{i}^{\prime})+\mathbb{Q}(\bm{x}_{i}^{\prime})}\right|$ | $\\{\bm{x}_{i}^{\prime}\\}_{i=1}^{N}\overset{\text{i.i.d.}}{\sim}\frac{\mathbb{P}+\mathbb{Q}}{2}$
Parameter Estimation | $\frac{1}{N}\sum_{i=1}^{N}\left|\frac{\widehat{\mathbb{Q}}(\bm{x}_{i}^{\prime})-\widehat{\mathbb{P}}(\bm{x}_{i}^{\prime})}{\widehat{\mathbb{P}}(\bm{x}_{i}^{\prime})+\widehat{\mathbb{Q}}(\bm{x}_{i}^{\prime})}\right|$ | $\\{\bm{x}_{i}^{\prime}\\}_{i=1}^{N}\overset{\text{i.i.d.}}{\sim}\frac{\widehat{\mathbb{P}}+\widehat{\mathbb{Q}}}{2}$
Kernel Density Estimation [21] | $\frac{1}{N}\sum_{i=1}^{N}\left|\frac{\widetilde{\mathbb{Q}}_{\text{kde}}(\bm{x}_{i}^{\prime})-\widetilde{\mathbb{P}}_{\text{kde}}(\bm{x}_{i}^{\prime})}{\widetilde{\mathbb{P}}_{\text{kde}}(\bm{x}_{i}^{\prime})+\widetilde{\mathbb{Q}}_{\text{kde}}(\bm{x}_{i}^{\prime})}\right|$ | $\\{\bm{x}_{i}^{\prime}\\}_{i=1}^{N}\overset{\text{i.i.d.}}{\sim}\frac{\widetilde{\mathbb{P}}_{\text{kde}}+\widetilde{\mathbb{Q}}_{\text{kde}}}{2}$
Nearest Neighbor Ratio Estimation [18] | $\frac{1}{n}\sum_{i=1}^{M}\frac{1}{2}\left|\frac{\eta N_{i}}{M_{i}+1}-1\right|$ |  $\\{\bm{x}_{i}^{\prime}\\}_{i=1}^{N}\overset{\text{i.i.d.}}{\sim}\mathbb{P}$, $\\{\widetilde{\bm{x}}_{i}^{\prime}\\}_{i=1}^{M}\overset{\text{i.i.d.}}{\sim}\mathbb{Q}$
Esemble Estimation [16] | $\frac{1}{n}\sum_{i=1}^{N}\frac{1}{2}\left|\frac{M_{2}(\rho_{2,k}(i))^{p}}{M_{1}(\rho_{1,k}(i))^{p}}-1\right|$ |  $\\{\bm{x}_{i}^{\prime}\\}_{i=1}^{M_{1}}\overset{\text{i.i.d.}}{\sim}\mathbb{P}$, $\\{\widetilde{\bm{x}}_{i}^{\prime}\\}_{i=1}^{n}\overset{\text{i.i.d.}}{\sim}\mathbb{Q}$

###  C.2 Real Application

With the use of MNIST dataset, we train Generative Adversarial Network models
for 100, 300, and 500 epochs, subsequently generating images with each of
these models.

MNIST database: This dataset contains $70,000$ grayscale images of handwritten
digits. The dataset is divided into a training set of $60,000$ images and a
test set of $10,000$ images. Each image is $28\times 28$ pixels in size, and
each pixel value ranges from $0$ to $255$, representing the intensity of the
pixel. The dataset includes ten classes, corresponding to the digits $0$
through $9$.

For each class/digit, we generate the same number of training (and testing)
samples with each model as there are images in the MNIST training (and
testing) dataset.

Generative adversarial network models (GANs) settings:

generator ={ | Linear(100, 128),
---|---
| LeakyReLU(0.2, inplace=True),
| Linear(128, 256),
| BatchNorm1d(256),
| LeakyReLU(0.2, inplace=True),
| Linear(256, 784),
| Tanh() }

discriminator ={ | Linear(784, 256),
---|---
| LeakyReLU(0.2, inplace=True),
| Linear(256, 128),
| BatchNorm1d(256),
| LeakyReLU(0.2, inplace=True),
| Linear(128, 1),
| Sigmoid() }

Adam optimizer is used for both networks with learning rate $0.0002$ and the
loss function is defined to be binary cross-entropy loss.

Then we use pretrained ResNet-18 model or random projection method to find out
the embedding of each image. We adjust the first layer of ResNet-18 model from
$3$-channel to $1$-channel so that MNIST images can fit this model. We also
modify the output size in last fully-connected layer of this model to the
desired dimension of embeddings {20, 35, 50}. In random projection random
method, we generate a random matrix of size $784\times 35$, where each element
is drawn from $\mathcal{N}(0,0.1)$. After obtaining the embedding, we estimate
TV distance between embedding of original images and generated images for each
class using different approaches. Finally, we calculate mean values and
standard deviation of all classes.