# Assessing uncertainty in Gaussian
mixtures-based entropy estimation

Luca Scrucca
Department of Economics, Università degli Studi di Perugia
0000-0003-3826-0484

###### Abstract

Entropy estimation plays a crucial role in various fields, such as information
theory, statistical data science, and machine learning. However, traditional
entropy estimation methods often struggle with complex data distributions.
Mixture-based estimation of entropy has been recently proposed and gained
attention due to its ease of use and accuracy. This paper presents a novel
approach to quantify the uncertainty associated with this mixture-based
entropy estimation method using weighted likelihood bootstrap. Unlike standard
methods, our approach leverages the underlying mixture structure by assigning
random weights to observations in a weighted likelihood bootstrap procedure,
leading to more accurate uncertainty estimation. The generation of weights is
also investigated, leading to the proposal of using weights obtained from a
Dirichlet distribution with parameter $\alpha=0.8137$ instead of the usual
$\alpha=1$. Furthermore, the use of centered percentile intervals emerges as
the preferred choice to ensure empirical coverage close to the nominal level.
Extensive simulation studies comparing different resampling strategies are
presented and results discussed. The proposed approach is illustrated by
analyzing the log-returns of daily Gold prices at COMEX for the years
2014–2022, and the Net Rating scores, an advanced statistic used in basketball
analytics, for NBA teams with reference to the 2022/23 regular season.

Keywords: differential entropy estimation; Gaussian mixture models;
uncertainty quantification; weighted likelihood bootstrap; Dirichlet weights;
bootstrap percentile intervals; empirical coverage.

###### Contents

  1. 1 Introduction
  2. 2 Mixture-based entropy estimation
  3. 3 Uncertainty assessment via weighted likelihood bootstrap
  4. 4 On the generation of weights in WLB
  5. 5 Simulation studies
    1. 5.1 Resampling methods under comparison
    2. 5.2 Data generation settings
    3. 5.3 Results
  6. 6 Data applications
    1. 6.1 Gold prices data
    2. 6.2 NBA 2022-23 net rating
  7. 7 Conclusion
  8. A Simulation results

##  1 Introduction

Entropy is a core concept in information theory (Shannon,, 1948) providing a
measure of uncertainty or randomness in a random variable (Cover and Thomas,,
2006). The (differential) entropy of a continuous random variable
$X\in\mathbb{R}^{d}$ with probability density function (pdf)
$f(\boldsymbol{x})$ is defined as

$H(X)=-\operatorname{\mathbb{E}}_{X}[\log
f(X)]=-\int_{\mathcal{X}}f(\boldsymbol{x})\log
f(\boldsymbol{x})\,\text{d}\boldsymbol{x},$

where $\mathcal{X}=\\{\boldsymbol{x}:f(\boldsymbol{x})>0\\}$ is the support of
the random variable.

Closed-form expressions for the entropy of several univariate distributions
are available in the literature (Michalowicz et al.,, 2014), but only few
multivariate distributions admit a solution in analytic form. For parametric
pdfs
$\\{f(\boldsymbol{x};\boldsymbol{\theta}):\boldsymbol{\theta}\in\boldsymbol{\Theta}\\}$,
when a closed-form expression is available, using the MLE
$\widehat{\boldsymbol{\theta}}$ guarantees that the estimate of the entropy is
both asymptotically unbiased and efficient (Kay,, 1993, Theorem 7.3). If we
cannot assume a specific parametric distribution, an estimate of the entropy
is often obtained using nonparametric estimation of $f(\boldsymbol{x})$ based
on histogram, kernel density, or nearest neighbour estimators. This latter
approach is, however, limited to low-dimensional cases, often $d\leq 2$.
Recently, a simple and intuitive mixture-based estimator has been proposed by
Robin and Scrucca, (2023) using finite mixture models (FMMs). The authors
showed with extensive simulation studies the accuracy and efficiency of the
proposal, which compares favorably against other estimators often used in
practice.

The remainder of this paper is structured as follows: Section 2 reviews the
mixture-based approach for entropy estimation and its advantages over
traditional methods. Section 3 introduces our proposal for quantifying
uncertainty in mixture-based entropy estimation. Specifically, we propose to
adopt a weighted likelihood bootstrap (WLB) approach by assigning random
weights drawn from a Dirichlet distribution when refitting mixture models.
Generation of weights is discussed in Section 4, where we introduce a novel
proposal instead of the usually adopted uniform Dirichlet over the simplex.
Extensive simulation studies are presented in Section 5, where the performance
of different resampling strategies and percentile bootstrap intervals are
compared. Section 6 illustrate the methodology by analyzing entropy in two
real-world datasets. The final section concludes with a summary of findings
and potential future research directions.

##  2 Mixture-based entropy estimation

Assume that the probability density function of interest can be approximated
by a finite mixture models (FMMs) with $G$ components, i.e.

$f(\boldsymbol{x})\approx
f(\boldsymbol{x};\boldsymbol{\theta})=\sum_{k=1}^{G}\pi_{k}\,f_{k}(\boldsymbol{x};\boldsymbol{\theta}_{k}),$
(1)

where
$\boldsymbol{\theta}=\\{\boldsymbol{\pi},\boldsymbol{\theta}_{1},\dots,\boldsymbol{\theta}_{G}\\}$
are the unknown parameters, with $\boldsymbol{\pi}=(\pi_{1},\dots,\pi_{G})$
the mixing weights (s.t. $\pi_{k}>0,\sum_{k=1}^{G}\pi_{k}=1)$, and
$\boldsymbol{\theta}_{k}$ the parameters associated to the density
$f_{k}(\cdot)$ of the $k$th component of the mixture ($k=1,\dots,G$). Often,
the densities $f_{k}(\cdot)$ are taken to belong to the same parametric family
of distributions, but with different parameters $\boldsymbol{\theta}_{k}$.

Wang and Madiman, (2014, Lemma 11.2) showed that the upper bound on the
entropy $H(X)$ of the FMM distribution in (1) is given by

$H(X)\leq-\sum_{k=1}^{G}\pi_{k}\log\pi_{k}+\sum_{k=1}^{G}\pi_{k}\,H(f_{k}(\boldsymbol{x};\boldsymbol{\theta}_{k})).$

Moreover, when the FMM is made up of Gaussian components, as introduced later
in (3), Huber et al., (2008) showed that both a lower and an upper bounds can
be obtained, respectively, as

$\displaystyle H_{\ell}(X)$
$\displaystyle=-\sum_{k=1}^{G}\pi_{k}\log\left(\sum_{j=1}^{G}\pi_{j}\phi(\boldsymbol{\mu}_{k};\boldsymbol{\mu}_{j},\boldsymbol{\Sigma}_{k}+\boldsymbol{\Sigma}_{j})\right),$
$\displaystyle H_{u}(X)$
$\displaystyle=-\sum_{k=1}^{G}\pi_{k}\log\pi_{k}+\sum_{k=1}^{G}\pi_{k}\left(\frac{d}{2}(1+\log(2\pi))+\frac{1}{2}\log|\boldsymbol{\Sigma}_{k}|\right).$

In practice, following a plug-in principle, lower and upper bound estimates
are easily obtained by replacing unknown quantities with their estimates.

Recalling that, from a generative point of view, FMMs in (1) can be re-
expressed as a hierarchical model by introducing the multinomial discrete
latent variable $Z\in\\{1,\dots,G\\}$ and writing

$\displaystyle Z$ $\displaystyle\sim\mathcal{M}(1,\boldsymbol{\pi})$
$\displaystyle X\mid(Z=k)$ $\displaystyle\sim
f_{k}(\boldsymbol{x};\boldsymbol{\theta}_{k}),$

then by the law of total expectation, the entropy can be written as

$\displaystyle H(X)$
$\displaystyle=-\operatorname{\mathbb{E}}_{Z}\left[\operatorname{\mathbb{E}}_{X\mid
Z}\left[\log
f(X)\right]\right]=-\sum_{k=1}^{G}\pi_{k}\operatorname{\mathbb{E}}_{X}\left[\log
f(X)\mid Z=k\right]$
$\displaystyle=-\sum_{k=1}^{G}\pi_{k}\int_{\mathcal{X}}f_{k}(\boldsymbol{x};\boldsymbol{\theta}_{k})\,\log
f_{k}(\boldsymbol{x};\boldsymbol{\theta}_{k})\,\text{d}\boldsymbol{x}.$

Based on this formulation, (Robin and Scrucca,, 2023) showed that a simple
closed-formula mixture-based estimator for samples of size $n$ can be
calculated as follows:

$\widehat{H}(X)=-\frac{1}{n}\sum_{i=1}^{n}\log\widehat{f}(\boldsymbol{x}_{i};\widehat{\boldsymbol{\theta}}),$
(2)

where
$\widehat{f}(\boldsymbol{x}_{i};\widehat{\boldsymbol{\theta}})=\sum_{k=1}^{G}\widehat{\pi}_{k}f_{k}(\boldsymbol{x}_{i};\widehat{\boldsymbol{\theta}}_{k})$
is the mixture-based density estimate evaluated on the same data points used
for fitting the model via EM algorithm (Dempster et al.,, 1977; McLachlan and
Krishnan,, 2008).

Gaussian mixture models (GMMs) correspond to the case where the general FMM in
(1) is written as

$f(\boldsymbol{x};\boldsymbol{\theta})=\sum_{k=1}^{G}\pi_{k}\,\phi(\boldsymbol{x};\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k}),$
(3)

where
$\boldsymbol{\theta}=\\{\boldsymbol{\pi},\boldsymbol{\mu}_{1},\dots,\boldsymbol{\mu}_{G},\boldsymbol{\Sigma}_{1},\dots,\boldsymbol{\Sigma}_{G}\\}$
are the unknown parameters of the mixture, with
$\boldsymbol{\pi}=(\pi_{1},\pi_{2},\dots,\pi_{G})$ the mixing weights, and
$\phi(\boldsymbol{x};\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})$ the
underlying multivariate Gaussian pdf of the $k$th component with mean vector
$\boldsymbol{\mu}_{k}$ and covariance matrix $\boldsymbol{\Sigma}_{k}$. The
entropy estimate from (2) can then be written as

$\displaystyle\widehat{H}(X)$
$\displaystyle=-\frac{1}{n}\sum_{i=1}^{n}\log\left\\{\sum_{k=1}^{G}\widehat{\pi}_{k}\,\phi(\boldsymbol{x}_{i};\widehat{\boldsymbol{\mu}}_{k},\widehat{\boldsymbol{\Sigma}}_{k})\right\\},$
(4)

where
$(\widehat{\pi}_{k},\widehat{\boldsymbol{\mu}}_{k},\widehat{\boldsymbol{\Sigma}}_{k})$
are the estimated parameters of $k$th component of the GMM obtained using the
EM algorithm. For details see Scrucca et al., (2023, Sec. 2.2).

The estimation procedure described above is available in the R package
mclustAddons (Scrucca,, 2024).

##  3 Uncertainty assessment via weighted likelihood bootstrap

Uncertainty assessment in entropy estimation is crucial for reliable
interpretation of estimates. However, existing literature lacks methodologies
specifically designed for this purpose. To fill this gap, we propose adopting
a resampling approach based on the bootstrap (Efron and Tibshirani,, 1993;
Davison and Hinkley,, 1997) to quantify uncertainty effectively.

Standard nonparametric bootstrap (BS) is a resampling technique that relies on
repeatedly drawing samples with replacement from the observed data (Efron,,
1982). This procedure is equivalent to the repeated generation of random
weights from a multinomial distribution with equal probabilities, i.e.
$\mathcal{M}(1;1/n,...,1/n)$. In contrast, Weighted Likelihood Bootstrap (WLB)
is a resampling procedure that generate bootstrap samples by assigning random
weights to observations generated from a Dirichlet distribution over the
$(n-1)$-simplex, i.e.

$(w_{1},\dots,w_{n})\sim\mathcal{D}(\boldsymbol{\alpha}),$

where $\boldsymbol{\alpha}=(\alpha_{1},\dots,\alpha_{n}){}^{\\!\top}$.
Therefore, the generated weights satisfy $0\leq w_{i}\leq 1$ for all $i$ and
$\sum_{i=1}^{n}w_{i}=1$.

WLB was originally proposed by Newton and Raftery, (1994) as a method for
approximately sampling from a posterior distribution. In its essence, WLB
combines weighted likelihood estimation with bootstrap resampling using random
weights as in the Bayesian bootstrap (BB) approach (Rubin,, 1981). Recently,
Weighted Bayesian Bootstrap (WBB) has been proposed, enabling the inclusion of
a prior distribution into statistical and machine learning models (Newton et
al.,, 2021).

WLB requires the optimization of a weighted likelihood function, which for
Gaussian mixtures is given by

$\ell^{W}(\boldsymbol{\theta})=\sum_{i=1}^{n}w_{i}\log\left\\{\sum_{k=1}^{G}\pi_{k}\,\phi(\boldsymbol{x}_{i};\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})\right\\}.$

EM algorithm can be used for estimation by defining the weighted complete-data
log-likelihood:

$\ell_{C}^{W}(\boldsymbol{\theta})=\sum_{i=1}^{n}\sum_{k=1}^{G}z_{ig}\,w_{i}\log\left\\{\pi_{k}\,\phi(\boldsymbol{x};\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})\right\\},$
(5)

where
$\boldsymbol{\theta}=(\pi_{1},\dots,\pi_{G-1},\boldsymbol{\mu}_{1},\dots,\boldsymbol{\mu}_{G},\boldsymbol{\Sigma}_{1},\dots,\boldsymbol{\Sigma}_{G})$
are the parameters to be estimated.

Let GMM$(\mathcal{M},G)$ be the Gaussian mixture model $\mathcal{M}$ defined
by a parsimonious covariance decomposition (Scrucca et al.,, 2023, Sec. 2.2.1)
with $G$ number of mixture components. These represent tuning parameters that
can be set by the researcher or determined using a model selection criterion,
such as the Bayesian information criterion (Schwarz,, 1978, BIC) or the
integrated complete-data likelihood criterion (Biernacki et al.,, 2000, ICL).

The proposed WLB procedure for a GMM$(\mathcal{M},G)$ can be described as
follows:

  1. 1.

generate weights as $(w_{1},\dots,w_{n})\sim\mathcal{D}(\boldsymbol{\alpha})$;

  2. 2.

fit GMM$(\mathcal{M},G)$ using the data
$\\{\boldsymbol{x}_{i},w_{i}\\}_{i=1}^{n}$ by maximizing (5) to get the
bootstrap estimate of parameters
$\widehat{\boldsymbol{\theta}}^{\,*}_{b}=\left\\{\widehat{\pi}^{\,*}_{k},\widehat{\boldsymbol{\mu}}^{\,*}_{k},\widehat{\boldsymbol{\Sigma}}^{\,*}_{k}\right\\}_{k=1}^{G}$;

  3. 3.

compute the bootstrap estimate of entropy as

$\widehat{H}^{*}_{b}(X)=-\frac{1}{n\overline{w}}\sum_{i=1}^{n}w_{i}\log\left\\{\sum_{k=1}^{G}\widehat{\pi}^{\,*}_{k}\phi(\boldsymbol{x}_{i};\widehat{\boldsymbol{\mu}}^{\,*}_{k},\widehat{\boldsymbol{\Sigma}}^{\,*}_{k})\right\\},$

where $\overline{w}=n^{-1}\sum_{i=1}^{n}w_{i}$;

  4. 4.

replicate steps 1–3 a large number of times, say $B$, to obtain the WLB
distribution $\left\\{\widehat{H}^{*}_{b}(X)\right\\}_{b=1}^{B}$ as an
approximate posterior sampling.

Once the bootstrap distribution is obtained, various measures can be derived
to assess the GMM-based estimate $\widehat{H}(X)$ of entropy. For instance, an
estimate of the bias can be computed as

$\text{Bias}(\widehat{H}(X))=\overline{\widehat{H}^{*}}(X)-\widehat{H}(X),$

where
$\overline{\widehat{H}^{*}}(X)=B^{-1}\sum_{b=1}^{B}\widehat{H}^{*}_{b}(X)$,
while a bootstrap estimate of the standard error is obtained as

$\text{SE}(\widehat{H}(X))=\sqrt{\frac{1}{B-1}\sum_{b=1}^{B}\left(\widehat{H}^{*}_{b}(X)-\overline{\widehat{H}^{*}}(X)\right)^{2}}.$

WLB intervals at the level $(1-\alpha)100\%$ can be derived using the
_bootstrap percentile_ method, i.e. by computing

$\left[\widehat{H}^{*}_{\alpha/2},\;\widehat{H}^{*}_{1-\alpha/2}\right],$ (6)

where $\widehat{H}^{*}_{q}$ is the 100$q$th percentile of the WLB
distribution. Percentile intervals are straightforward to compute and have the
advantage of being range-preserving, ensuring the interval lies within the
proper bounds of the parameter space. However, despite their simplicity and
intuitive reasonableness, percentile intervals have been criticized because
they approximate the sampling distribution of $\widehat{H}(X)-H(X)$ with
$\widehat{H}(X)-\widehat{H}^{*}(X)$ and not with
$\widehat{H}^{*}(X)-\widehat{H}(X)$, which is what the bootstrap prescribes
(Hall,, 1988).

According to the bootstrap principle, we should aim at computing an interval
such that

$\displaystyle\Pr\left(\widehat{H}^{*}_{\alpha/2}-\widehat{H}(X)\leq\widehat{H}(X)-H(X)\leq\widehat{H}^{*}_{1-\alpha/2}-\widehat{H}(X)\right)\approx
1-\alpha$
$\displaystyle\Pr\left(2\widehat{H}(X)-\widehat{H}^{*}_{1-\alpha/2}\leq
H(X)\leq 2\widehat{H}(X)+\widehat{H}^{*}_{\alpha/2}\right)\approx 1-\alpha.$

Notice that, if $\widehat{H}^{*}_{b}(X)-\widehat{H}(X)$ is the bias for the
$b$th bootstrap sample, then a bias-corrected estimate is given by

$\widehat{H}^{*c}_{b}(X)=\widehat{H}(X)-(\widehat{H}^{*}_{b}(X)-\widehat{H}(X))=2\widehat{H}(X)-\widehat{H}^{*}_{b}(X).$

The aforementioned arguments result in a simple bias-corrected version known
as the _centered bootstrap percentile_ method (Manly,, 2006, Sec. 3.3; Singh
and Xie,, 2010) or _basic bootstrap intervals_ (Davison and Hinkley,, 1997,
Sec. 5.2). Given the differing roles of the bootstrap distribution in the two
percentile methods, the resulting intervals will not be equal, especially when
the bootstrap distribution is skewed or biased.

Thus, a WLB centered percentile interval at the level $(1-\alpha)100\%$ can be
computed using the percentiles from the distribution of bias-corrected entropy
estimates $\left\\{\widehat{H}^{*c}_{b}(X)\right\\}_{b=1}^{B}$, i.e. by
computing

$\left[\widehat{H}^{*c}_{\alpha/2},\;\widehat{H}^{*c}_{1-\alpha/2}\right]=\left[2\widehat{H}(X)-\widehat{H}^{*}_{1-\alpha/2},\;2\widehat{H}(X)-\widehat{H}^{*}_{\alpha/2}\right],$
(7)

where $\widehat{H}^{*c}_{q}$ is the 100$q$th percentile of the bias-corrected
WLB distribution.

##  4 On the generation of weights in WLB

In WLB, as well as in BB and WBB methods, the weights associated with each
observation are generated from a Dirichlet distribution, i.e.
$\boldsymbol{w}\sim\mathcal{D}(\boldsymbol{\alpha})$ with
$\boldsymbol{\alpha}=\alpha\boldsymbol{1}_{n}$, where $\boldsymbol{1}_{n}$ is
a vector of ones of length $n$. In general, generating random values from a
$\mathcal{D}(\boldsymbol{\alpha})$ distribution is equivalent to generating
them from independent Gamma distributions $\mathcal{G}(\alpha,\beta=1)$,
resulting in both the expected value and the variance being equal to $\alpha$.
It’s worth noting that when $\alpha=1$, an Exponential distribution with a
rate parameter of 1 is obtained.

Weights generated independently according to
$w_{i}\sim\mathcal{G}(\alpha,\beta=1)$, for $i=1,\dots,n$, can be rescaled in
various ways. Often, weights are scaled such that they sum to one or, more
conveniently, such that their sum is equal to the number of observations $n$.
In the latter case, the rescaling is obtained as
$\widetilde{w}_{i}=w_{i}/\overline{w}$, where
$\overline{w}=n^{-1}\sum_{i=1}^{n}w_{i}$, so
$\sum_{i=1}^{n}\widetilde{w}_{i}=n$. Using this rescaling, weights can be seen
as the contribution of each observation to the overall sample information.
Note that in the unweighted case, each observation brings a unit contribution.
In contrast, in the nonparametric bootstrap, an observation contributes 0 if
it is never sampled, 1 if it is sampled once, 2 if it is sampled twice, and so
forth.

As previously mentioned, a typical choice is
$\boldsymbol{\alpha}=(1,1,\dots,1){}^{\\!\top}=\boldsymbol{1}_{n}$, which
results in random weights being generated uniformly over the simplex. However,
this may not always represent the optimal setting, and few studies have been
dedicated to exploring this aspect. A notable exception is the recommendation
by Shao and Tu, (1995, Sec. 10.2.1) to set
$\boldsymbol{\alpha}=(4,4,\dots,4){}^{\\!\top}=4\boldsymbol{1}_{n}$.

Here, we propose using $\alpha=0.8137$, a value derived by mimicking a well-
known result in standard nonparametric bootstrap. Recalling that the
probability of an observation being included at least once in a nonparametric
bootstrap sample is $1-\exp(-1)\approx 0.6321$, we aim to determine the value
of $\alpha$ that ensures, on average, a median weight of $0.6321$ is assigned
to each observation.

Suppose weights are generated as
$\left\\{w_{i}\sim\mathcal{G}(\alpha,\beta=1)\right\\}_{i=1}^{n}$, thus
$\operatorname{\mathbb{E}}(w_{i})=\alpha$, and rescale the weights by their
average to obtain $\widetilde{w}_{i}=w_{i}/\alpha$. We aim at finding the
value of $\alpha$ such that
$\operatorname{\mathbb{E}}[\operatorname{\text{Median}}(\widetilde{w})]=0.6321$,
i.e the value at which, on average, the median weight assigned to an
observation equals the probability of inclusion in standard nonparametric
bootstrap. The use of the median is justified as a robust measure of
centrality, which is required due to the highly skewed distributions involved.

Panels (a) and (b) of Figure 1 show the distribution of scaled weights
$\widetilde{w}_{i}$ corresponding to the typical value of $\alpha=1$ and
$\alpha=4$, the latter being the value suggested by Shao and Tu, (1995, Sec.
10.2.1). In the former case, the distribution exhibits significant skewness,
with most data points having small weights and only a few receiving larger
weights. Conversely, in the latter case, the weights tend to be more similar,
thereby reducing diversity across resamples. Figure 1d displays the curve of
$\operatorname{\mathbb{E}}[\operatorname{\text{Median}}(\widetilde{w})]$ as a
function of $\alpha$, revealing the sought value of $\alpha=0.8137$ obtained
by root finding. Figure 1c illustrates the distribution of weights
corresponding to the identified value of $\alpha=0.8137$ for the random
generation of weights in a WLB procedure. The distribution of scaled weights
exhibits larger skewness compared to the standard case, thereby increasing the
variability of the generated weights. Contrary to the suggestion by Shao and
Tu, (1995), the value of $\alpha$ is reduced, thereby increasing the diversity
of generated resamples. As will be shown in Section 5, this has beneficial
effects on the empirical coverage of credible intervals for the entropy.

Figure 1: Panels (a)-(c) show the distribution of scaled weights
$\widetilde{w}_{i}$ obtained by varying the parameter
$\boldsymbol{\alpha}=\alpha\boldsymbol{1}_{n}$ of a Dirichlet distribution.
Panel (d) displays the curve of
$\operatorname{\mathbb{E}}[\operatorname{\text{Median}}(\widetilde{w})]$ as a
function of $\alpha$, with the highlighted value $\alpha=0.8137$ indicating
the parameter value that ensures, on average, the median weight assigned to an
observation matches the probability of inclusion in standard nonparametric
bootstrap.

##  5 Simulation studies

In this section, we present the results of a series of comprehensive
simulation studies designed to examine the performance of different WLB
strategies and to compare them with standard nonparametric and parametric
bootstrap approaches, as described in Section 5.1. The simulation settings
examined coincide in part with those analyzed in Robin and Scrucca, (2023),
and are described in detail in Section 5.2. A summary of the main findings of
the simulation studies is presented and discussed in Section 5.3.

###  5.1 Resampling methods under comparison

The bootstrap approaches we aim at comparing are the following:

  * •

nonparametric bootstrap (BS), with samples obtained by resampling with
replacement from and with the same size as the original data;

  * •

parametric bootstrap (PB), with samples obtained by generating synthetic data
of the same size from the fitted GMM on the original data;

  * •

weighted likelihood bootstrap (WLB) with samples obtained by assigning to the
original simulated data a set of weights generated from a Dirichlet
distribution with parameter $\boldsymbol{\alpha}=\alpha\boldsymbol{1}_{n}$,
where $\alpha=\\{1,4,0.8137\\}$.

###  5.2 Data generation settings

The distributions investigated in the simulation study are the following:

  * •

Gaussian distribution with parameters $\mu=0$ and $\sigma=1$, whose entropy is

$H(X)=\frac{1}{2}(1+\log(2\pi)+\log(\sigma^{2}))\approx 1.4189.$

  * •

$t$ distribution with $\text{df}=3$ degrees of freedom, whose entropy is

$H(X)=\frac{\text{df}+1}{2}\left[\psi\left(\frac{\text{df}+1}{2}\right)-\psi\left(\frac{\text{df}}{2}\right)\right]+\log\left[\sqrt{\text{df}}\;\text{B}\left(\frac{\text{df}}{2},\frac{1}{2}\right)\right]\approx
1.7735,$

where $\psi()$ is the digamma function, and $\text{B}()$ the beta function.

  * •

Mixed-Gaussian distribution with parameters $\mu=2$ and $\sigma=1$; the
distribution is bimodal with modes at $\pm 2$ and entropy (Michalowicz et
al.,, 2014)

$H(X)=\frac{1}{2}(1+\log(2\pi)+\log(\sigma^{2}))+0.633\approx 2.0519.$

  * •

Laplace distribution with parameters $\mu=0$ and $\beta=2$, whose entropy is

$H(X)=1+\log(2\beta)\approx 2.3863.$

  * •

Bivariate ($d=2$) Gaussian distribution with parameters
$\mu=[0,0]{}^{\\!\top}$ and
${\boldsymbol{\Sigma}=\bigl{[}\begin{smallmatrix}1.0&0.8\\\
0.8&2.0\end{smallmatrix}\bigr{]}}$, whose entropy is given by

$H(X)=\frac{d}{2}(1+\log(2\pi))+\frac{1}{2}\log|\boldsymbol{\Sigma}|\approx
2.9916.$

  * •

Multivariate ($d=10$) independent $\chi^{2}$ distributions with $\text{df}=5$
degrees of freedom for each dimension, thus having entropy equal to

$H(X)=\sum_{j=1}^{10}H(X_{j})=d\left[\log(2)+\log\Gamma(\text{df}/2)+\text{df}/2+(1-\text{df}/2)\psi(\text{df}/2)\right]\approx
24.2309,$

where $\Gamma()$ is the gamma function, and $\psi()$ the digamma function.

Data were simulated from the aforementioned distributions for sample sizes
$n=\\{100,200,500,1000\\}$. For each setting, 1,000 replications were
generated. A GMM was then fitted to each synthetic dataset using the mclust R
package (Fraley et al.,, 2024), with number of mixture components and
parsimonious variances (or covariance matrices in case of multidimensional
data) selected using the Bayesian information criterion (BIC Schwarz,, 1978).

###  5.3 Results

Tables 1 and 3–7 in A present, for each investigated distribution, summaries
of the simulation results for the bootstrap methods under comparison. For the
same settings, the associated Figures 1 and 10–14 in A show the empirical
coverage of different bootstrap intervals (panel a) and the ratio of bootstrap
absolute bias over the bootstrap standard error (panel b).

Across all scenarios examined in our simulation studies, the nonparametric
bootstrap, the parametric bootstrap, and the WLB with $\alpha=1$ exhibit
remarkably similar coverage behavior. As expected, empirical coverage improves
with larger sample sizes. Conversely, the WLB procedure with $\alpha=0.8137$
provides empirical coverage values that are always closer to the nominal
level, whereas the WLB with $\alpha=4$ consistently produces coverage levels
significantly lower than the nominal level across all tested settings.
Moreover, centered bootstrap percentile intervals outperformed standard
percentile bootstrap intervals in terms of empirical coverage. In fact,
centered percentile intervals demonstrated empirical coverage levels
consistently meeting or exceeding the nominal level. Finally, when deciding
between bootstrap percentile interval types, the absolute bias-to-standard
error ratio provides valuable insight. A ratio exceeding 0.2 indicates a
preference for centered percentile intervals, as these intervals tend to
achieve empirical coverage levels equal to or greater than the nominal level,
although with slightly wider intervals (not shown here).

To summarize the main results of our investigation, the WLB with
$\alpha=0.8137$ emerges as the recommended method for assessing uncertainty in
GMM-based entropy estimation. At the same time, the adoption of centered
percentile intervals seems to offer the most prudent choice to ensure
empirical coverage close to the nominal level.

Table 1: Simulation results for data generated from a $t$ distribution with $\text{df}=3$ degrees of freedom. For each bootstrap method and sample size, the average estimate of entropy, the bootstrap estimates of bias and standard error, and the associated bootstrap percentile intervals are reported. |  |  |  | 95% percentile interval | 95% centered perc. interval
---|---|---|---|---|---
Sample size | Estimate | Bias | SE | lower | upper | coverage | lower | upper | coverage
Nonparametric bootstrap
100 | 1.7565 | 0.0378 | 0.1160 | 1.4725 | 1.9256 | 0.890 | 1.5877 | 2.0413 | 0.927
200 | 1.7621 | 0.0193 | 0.0775 | 1.5866 | 1.8890 | 0.921 | 1.6351 | 1.9370 | 0.939
500 | 1.7703 | 0.0092 | 0.0488 | 1.6651 | 1.8554 | 0.936 | 1.6851 | 1.8754 | 0.953
1000 | 1.7734 | 0.0047 | 0.0344 | 1.7015 | 1.8355 | 0.933 | 1.7112 | 1.8452 | 0.933
Parametric bootstrap
100 | 1.7565 | 0.0238 | 0.0991 | 1.5311 | 1.9181 | 0.859 | 1.5945 | 1.9818 | 0.908
200 | 1.7621 | 0.0105 | 0.0727 | 1.6073 | 1.8909 | 0.899 | 1.6332 | 1.9168 | 0.926
500 | 1.7703 | 0.0032 | 0.0472 | 1.6757 | 1.8599 | 0.948 | 1.6809 | 1.8647 | 0.949
1000 | 1.7734 | 0.0020 | 0.0336 | 1.7086 | 1.8396 | 0.926 | 1.7072 | 1.8384 | 0.925
Weighted likelihood bootstrap ($\alpha=1$)
100 | 1.7565 | 0.0261 | 0.1013 | 1.5349 | 1.9286 | 0.890 | 1.5837 | 1.9784 | 0.914
200 | 1.7621 | 0.0151 | 0.0723 | 1.6092 | 1.8914 | 0.924 | 1.6333 | 1.9154 | 0.927
500 | 1.7703 | 0.0073 | 0.0471 | 1.6730 | 1.8568 | 0.941 | 1.6836 | 1.8675 | 0.947
1000 | 1.7734 | 0.0041 | 0.0337 | 1.7047 | 1.8362 | 0.928 | 1.7105 | 1.8421 | 0.925
Weighted likelihood bootstrap ($\alpha=4$)
100 | 1.7565 | 0.0063 | 0.0507 | 1.6523 | 1.8499 | 0.630 | 1.6632 | 1.8611 | 0.621
200 | 1.7621 | 0.0039 | 0.0366 | 1.6878 | 1.8305 | 0.629 | 1.6937 | 1.8364 | 0.633
500 | 1.7703 | 0.0019 | 0.0238 | 1.7224 | 1.8153 | 0.657 | 1.7252 | 1.8181 | 0.658
1000 | 1.7734 | 0.0011 | 0.0170 | 1.7394 | 1.8058 | 0.662 | 1.7410 | 1.8073 | 0.663
Weighted likelihood bootstrap ($\alpha=0.8137$)
100 | 1.7565 | 0.0319 | 0.1124 | 1.5080 | 1.9437 | 0.917 | 1.5690 | 2.0056 | 0.939
200 | 1.7621 | 0.0184 | 0.0798 | 1.5916 | 1.9028 | 0.938 | 1.6212 | 1.9324 | 0.951
500 | 1.7703 | 0.0088 | 0.0520 | 1.6622 | 1.8654 | 0.963 | 1.6753 | 1.8782 | 0.966
1000 | 1.7734 | 0.0048 | 0.0373 | 1.6971 | 1.8427 | 0.954 | 1.7042 | 1.8495 | 0.953
Figure 2: Simulation results for data generated from a $t$ distribution with
$\text{df}=3$ degrees of freedom. Panel (a) shows the empirical coverage for
1000 replications as function of the sample size for different bootstrap
methods (BS = nonparametric; PB = parametric; WLB = weigthed likelihood
bootstrap with weights generated from a Dirichlet with different values of
$\alpha$) and two type of bootstrap intervals (perc = percentile; cperc =
centered percentile). Panel (b) show the ratio of bias (in absolute value)
over the standard error for each bootstrap procedure as function of sample
size.

##  6 Data applications

###  6.1 Gold prices data

Gold is often seen as a secure asset during periods of political and economic
instability. While the price of gold has generally trended upwards over the
long term, short-term fluctuations of gold price can be substantial. Given its
propensity to appreciate along with the general increase in prices of other
goods and services, it is considered a valuable safeguard against inflation.
For these reasons, gold is often regarded as a reliable investment that can
protect investors’ wealth in times of economic turmoil.

Here we analyze the differential entropy of the daily log returns of gold
prices from 2014 to 2022. By computing and comparing the differential entropy
for each year, the relative uncertainty in gold prices over time can be
assessed. Higher entropy values indicate greater uncertainty, suggesting more
variability and potentially more unpredictable price movements. Conversely,
lower entropy values imply lower uncertainty and more stable price behavior.

Figure 3 shows the daily Gold prices (per troy ounce) at Commodity Exchange
Inc. (COMEX), the primary futures and options market for trading metals such
as gold, silver, copper, and aluminum. The daily gold prices shown on the top
panel exhibit significant fluctuations throughout the period. Starting from
around $1,300 in 2014, the prices gradually increased, reaching a peak of
around $1,900 in 2020. After this peak, the prices experienced a decline but
remained above $1,500 for most of the remaining period until the end of 2022.
The log-returns shown in the bottom panel of Figure 3 display a high degree of
volatility, with frequent fluctuations above and below zero, suggesting
significant price movements in either direction throughout the period.

Figure 3: Daily gold prices at COMEX for the years 2014–2022 (top panel) and
the corresponding log-returns over the same period (bottom panel).

To investigate the volatility in the gold market during the selected time
frame we fitted Gaussian mixtures to the log-return daily prices for each
year. Figure 4 shows the histograms and fitted GMM densities for the years
2019–2022. The volatility in the gold market, as captured by the distribution
of daily log-returns, varied significantly over these four years, with 2020
exhibiting a higher volatility characterized by several larger positive and
negative returns, while year 2019, and to a lesser extend 2021 and 2022,
showed relatively lower volatility.

Figure 4: Distribution of gold price log-returns for each year from 2019 to
2022, with histograms representing the empirical distributions and lines
indicating the density estimates obtained by fitting Gaussian Mixture Models
(GMMs) to the observed data.

Figure 5 displays the trend of GMM-based entropy estimates from 2014 to 2022,
along with their associated 95% centered percentile intervals obtained using
various bootstrap procedures. Despite being computed using different bootstrap
techniques, the intervals represented by the vertical bars appear remarkably
similar over the entire time period. This suggests that in this case the
uncertainty associated with the entropy estimates is consistent, regardless of
the specific bootstrap method employed.

Figure 5: Gaussian mixtures-based entropy estimates for each year from 2014 to
20122, with the associated 95% centered percentile intervals obtained using
various bootstrap methods (BS = nonparametric; PB = parametric; WLB = weigthed
likelihood bootstrap with weights generated from a Dirichlet with $\alpha=1$
and $\alpha=0.8137$).

In Figure 6 each panel shows a histogram representing the bootstrap
distribution of entropy obtained by different bootstrap methods for the year
2019. Across all panels, the bootstrap distributions appear to be relatively
symmetric and spread over a similar range, between -3.8 and -3.4. This
similarity further confirms the robustness of the entropy estimates and the
consistency among the various bootstrap techniques in quantifying the
associated uncertainty.

Figure 6: Bootstrap distributions of entropy obtained using various bootstrap
procedures for the year 2019. In each panel, filled triangles depict the GMM-
based estimate of entropy, while horizontal segments represent the
corresponding centered percentile intervals.

Overall, the analysis reinforces the interpretation that the entropy estimates
and their associated uncertainties are reliable and consistent across the
different bootstrap methods employed, providing confidence in the analysis of
volatility patterns in the gold market.

###  6.2 NBA 2022-23 net rating

Advanced basketball statistics provide deeper insights into the performance of
players and teams (Oliver,, 2004; Zuccolotto and Manisera,, 2020). These
metrics aim to extract meaningful patterns and relationships from data
collected during basketball games. While traditional statistics such as
points, rebounds, and assists provide a basic measure of a player’s or team’s
performance, efficiency metrics, such as Effective Field Goal, True Shooting
Percentage, Offensive and Defensive Rating, aim to assess the efficiency of
play and the overall impact on team success.

A comprehensive measure of a team’s overall game performance relative to its
opponent is provided by the _Net Rating_ (Kubatko et al.,, 2007), defined as
the difference between _Offensive Rating_ (ORtg) and _Defensive Rating_
(DRtg), i.e.

$\text{NetRtg}=\text{ORtg}-\text{DRtg}.$

The last two measures are advanced basketball statistics used to evaluate the
efficiency of teams on offense and defense, respectively. The Offensive Rating
(ORtg) is computed as:

$\text{ORtg}=\frac{\text{Pts}}{\text{Poss}}\times 100,$

where Pts represents the total points scored by a team, and Poss refers to the
total number of possessions. The latter is calculated as:

$\text{Poss}=\text{FGA}+0.44\times\text{FTA}+\text{TOV}-\text{OReb},$

where FGA is the number of (either 2-point and 3-point) field goal attempts,
FTA is the number of free throw attempts, TOV the number of turnovers, and
OReb the number of offensive rebounds. Thus, ORtg is a measure of a team
offensive efficiency, representing the number of points scored by a team per
100 possessions. A higher ORtg indicates a more efficient offensive phase,
while a lower ORtg suggests less efficient offensive performance. Defensive
Rating (DRtg) is calculated similarly but refers to the opposing team.
Therefore, it is a measure of a team defensive efficiency, representing the
number of points allowed by the team per 100 possessions.

We consider data collected during the 2022-23 NBA Regular Season and available
through the API provided by the R package hoopR (Gilani,, 2023). The ORtg and
DRtg values on each game were computed, so net rating (NetRtg) scores were
obtained for each NBA team on the 82 games of the regular season. Figure 7
shows the histograms of net rating scores for the Boston Celtics and the
Indiana Pacers, with the corresponding GMM-based density estimates. The
Celtics had the largest average net rating score, much larger than that of the
Pacers, but also a larger entropy reflecting the higher uncertainty in the
difference between team’s offensive and defensive ratings.

Figure 7: Histogram of net rating (NetRtg) scores for two selected NBA teams
with the corresponding GMM-based density estimates. Average and entropy values
are also reported.

Table 2 reports the average net rating for all the NBA teams, with the
corresponding 95% confidence intervals obtained using the WLB approach as
described in Scrucca et al., (2016) but with $\alpha=0.8137$ as described in
Section 4. The table also includes the estimated entropy and the 95%
confidence intervals computed according to the WLB with $\alpha=0.8137$. The
same information is also shown graphically in Figure 8.

The NBA teams with the lowest net rating averages are those worst ranked in
the final 2022-23 regular season standings. In contrast, the Boston Celtics
(BOS) and Milwaukee Bucks (MIL) show the highest averages and were the top two
teams in the final regular season rankings. The San Antonio Spurs (SA) have
both the lowest average and one of the highest entropy, indicating great
fluctuation in their NetRtg statistics. The team with the highest entropy is
the Golden State Warriors (GS), which showed large fluctuations, with net
ratings going from -50 to +50. This is twice the values observed for the
Indiana Pacers (IND), which on the contrary have the smallest entropy.

Table 2: Average and entropy values of net rating for NBA teams on the 2022-23 regular season, with corresponding 95% credible intervals obtained using the WLB approach with $\alpha=0.8137$. Team | Average | 95% CI | Entropy | 95% CI
---|---|---|---|---
Atlanta Hawks | -0.6568 | $[-3.9290,+2.5045]$ | 4.0469 | $[+3.8827,+4.1820]$
Boston Celtics | 6.6137 | $[+3.0671,+10.1934]$ | 4.1073 | $[+3.9033,+4.2879]$
Brooklyn Nets | -0.2344 | $[-3.8125,+3.4127]$ | 4.1757 | $[+3.9719,+4.3500]$
Charlotte Bobcats | -5.2925 | $[-8.4168,-1.9242]$ | 4.0099 | $[+3.8367,+4.1573]$
Chicago Bulls | 1.6153 | $[-1.7602,+5.2525]$ | 4.1046 | $[+3.9374,+4.2509]$
Cleveland Cavaliers | 4.9818 | $[+1.7502,+8.7335]$ | 4.0994 | $[+3.9419,+4.2187]$
Dallas Mavericks | -0.3520 | $[-3.2899,+2.8415]$ | 3.9602 | $[+3.7321,+4.1800]$
Denver Nuggets | 3.4118 | $[-0.1471,+6.6683]$ | 4.1011 | $[+3.9228,+4.2544]$
Detroit Pistons | -8.4807 | $[-11.4003,-5.2130]$ | 3.9897 | $[+3.8319,+4.1230]$
Golden State Warriors | 0.7551 | $[-3.4145,+4.6851]$ | 4.2258 | $[+4.0118,+4.4048]$
Houston Rockets | -6.4060 | $[-9.4581,-3.3076]$ | 4.0143 | $[+3.8274,+4.1719]$
Indiana Pacers | -3.9495 | $[-7.1604,-0.9001]$ | 3.8108 | $[+3.6811,+3.8880]$
LA Clippers | 0.7368 | $[-2.4117,+4.0755]$ | 4.0442 | $[+3.8849,+4.1700]$
Los Angeles Lakers | 0.9280 | $[-2.0730,+3.9052]$ | 3.9679 | $[+3.7996,+4.1262]$
Memphis Grizzlies | 5.0635 | $[+1.5102,+8.6959]$ | 4.1319 | $[+3.9694,+4.2848]$
Miami Heat | -0.3530 | $[-3.2717,+2.5170]$ | 3.8992 | $[+3.6811,+4.0630]$
Milwaukee Bucks | 5.2019 | $[+0.0575,+8.7758]$ | 4.1030 | $[+3.9004,+4.2525]$
Minnesota Timberwolves | -0.5680 | $[-3.6957,+2.4405]$ | 3.9862 | $[+3.8350,+4.1170]$
New Orleans Hornets | 2.0084 | $[-1.4134,+5.7065]$ | 4.1758 | $[+4.0204,+4.3205]$
New York Knicks | 3.8174 | $[+0.1951,+7.3190]$ | 3.9624 | $[+3.7467,+4.1145]$
Oklahoma City Thunder | 0.2574 | $[-2.9938,+3.5134]$ | 4.0478 | $[+3.8801,+4.1798]$
Orlando Magic | -2.0689 | $[-5.0916,+1.1678]$ | 3.9747 | $[+3.8219,+4.1131]$
Philadelphia 76ers | 3.6097 | $[+0.2767,+7.2501]$ | 4.0897 | $[+3.9346,+4.2499]$
Phoenix Suns | 1.8343 | $[-1.8363,+5.6421]$ | 4.2162 | $[+4.0527,+4.3465]$
Portland Trail Blazers | -4.7156 | $[-8.8471,-1.0848]$ | 4.1898 | $[+4.0327,+4.3369]$
Sacramento Kings | 3.1652 | $[-0.4250,+6.4799]$ | 4.0896 | $[+3.9118,+4.2613]$
San Antonio Spurs | -10.4663 | $[-14.1456,-6.9153]$ | 4.1707 | $[+4.0208,+4.2829]$
Toronto Raptors | 1.3072 | $[-1.7371,+4.7026]$ | 3.9851 | $[+3.8263,+4.1479]$
Utah Jazz | -1.1229 | $[-4.1182,+1.8446]$ | 3.9444 | $[+3.7549,+4.0888]$
Washington Wizards | -0.6406 | $[-3.9708,+2.6205]$ | 4.0346 | $[+3.8766,+4.1670]$
Figure 8: Dotcharts of net rating (NetRtg) average and entropy for NBA teams
on the 2022-23 regular season, with corresponding 95% credible intervals
obtained using the WLB approach with $\alpha=0.8137$.

The joint distribution of average net rating and net rating entropy for NBA
teams in the 2022-23 regular season is shown in Figure 9. The average and
entropy values are plotted along the x-axis and y-axis, respectively, with
horizontal and vertical bars indicating the 95% WLB credible intervals. The
net rating average describes the overall quality and performance level of an
NBA team, with higher values indicating a better team. On the other hand, the
net rating entropy quantifies the consistency of a team’s performance across
the season. Higher entropy values suggest a less constant level of play, with
more fluctuation in performance from game to game.

Ideally, teams should aim to have both a large positive average net rating,
implying they outscored opponents by a considerable margin over the season, as
well as a low net rating entropy, indicating they played at a steady,
consistent level throughout the year. The most successful teams are typically
those that can maintain an excellent level of play while also minimizing
fluctuations and inconsistencies over the season.

Figure 9: Scatterplot of net rating (NetRtg) average and entropy for NBA teams
on the 2022-23 regular season, with corresponding 95% credible intervals
obtained using the WLB approach with $\alpha=0.8137$.

##  7 Conclusion

In this paper, we presented a novel framework for assessing the uncertainty
associated with mixture-based entropy estimation. Our proposal exploits the
underlying mixture structure by assigning random weights to observations in a
weighted likelihood bootstrap (WLB) procedure, leading to more accurate
uncertainty quantification. Through extensive simulation studies, we compared
the performance of different resampling strategies, including nonparametric
bootstrap, parametric bootstrap, and WLB with varying weight generation
schemes. The results demonstrated that the WLB approach with weights generated
from a Dirichlet distribution with parameter $\alpha=0.8137$ consistently
provided empirical coverage levels closest to the nominal level across a wide
range of scenarios. Additionally, the use of centered percentile intervals
emerged as the preferred choice to ensure reliable empirical coverage. We
illustrated the practical utility of our proposed method by analyzing two
real-world datasets: daily log-returns of gold prices at COMEX from 2014 to
2022, and the Net Rating scores for NBA teams during the 2022/23 regular
season. These applications have shown the effectiveness of our approach in
quantifying uncertainty in entropy estimation and providing insights into the
volatility patterns and performance consistency of these datasets.

Overall, our work contributes to the field of entropy estimation by proposing
a novel methodology that addresses the crucial need for reliable uncertainty
assessment in mixture-based entropy estimation. The proposed WLB approach with
optimized weight generation and the use of centered percentile intervals offer
a robust and accurate framework for quantifying uncertainty, enabling more
informed decision-making and deeper insights into the underlying data
distributions.

Future research directions could focus on extending the WLB approach for
uncertainty assessment of entropy estimation in more complex data structures
beyond mixture models, such as time series, spatial processes, and data with
hierarchical or multilevel dependencies.

## Acknowledgements

I would like to express my gratitude to the Istituto Nazionale di Fisica
Nucleare (INFN) for providing the scientific computing infrastructure used for
running the simulations presented in this work.

## References

  * Biernacki et al., (2000) Biernacki, C., Celeux, G., and Govaert, G. (2000).  Assessing a mixture model for clustering with the integrated completed likelihood.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(7):719–725.
  * Cover and Thomas, (2006) Cover, T. M. and Thomas, J. A. (2006).  Elements of Information Theory.  John Wiley & Sons, Hoboken, NJ, 2nd edition.
  * Davison and Hinkley, (1997) Davison, A. and Hinkley, D. (1997).  Bootstrap Methods and Their Applications.  Cambridge University Press.
  * Dempster et al., (1977) Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977).  Maximum likelihood from incomplete data via the EM algorithm (with discussion).  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 39:1–38.
  * Efron, (1982) Efron, B. (1982).  The Jackknife, the Bootstrap and Other Resampling Plans, volume 38.  SIAM, Philadelphia, PA.
  * Efron and Tibshirani, (1993) Efron, B. and Tibshirani, R. J. (1993).  An Introduction to the Bootstrap.  Chapman & Hall/CRC, New York.
  * Fraley et al., (2024) Fraley, C., Raftery, A. E., and Scrucca, L. (2024).  mclust: Gaussian Mixture Modelling for Model-Based Clustering, Classification, and Density Estimation.  R package version 6.1.1.
  * Gilani, (2023) Gilani, S. (2023).  hoopR: Access Men’s Basketball Play by Play Data.  R package version 2.1.0.
  * Hall, (1988) Hall, P. (1988).  Theoretical comparison of bootstrap confidence intervals (with discussions).  The Annals of Statistics, 16:927–953.
  * Huber et al., (2008) Huber, M. F., Bailey, T., Durrant-Whyte, H., and Hanebeck, U. D. (2008).  On entropy approximation for Gaussian mixture random vectors.  In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181–188. IEEE.
  * Kay, (1993) Kay, S. M. (1993).  Fundamentals of Statistical Signal Processing.  Prentice Hall, Upper Saddle River, NJ.
  * Kubatko et al., (2007) Kubatko, J., Oliver, D., Pelton, K., and Rosenbaum, D. T. (2007).  A starting point for analyzing basketball statistics.  Journal of Quantitative Analysis in Sports, 3(3):1–22.
  * Manly, (2006) Manly, B. F. (2006).  Randomization, Bootstrap and Monte Carlo Methods in Biology.  Chapman & Hall/CRC, Boca Raton, FL.
  * McLachlan and Krishnan, (2008) McLachlan, G. and Krishnan, T. (2008).  The EM Algorithm and Extensions.  Wiley-Interscience, Hoboken, New Jersey, 2nd edition.
  * Michalowicz et al., (2014) Michalowicz, J. V., Nichols, J. M., and Bucholtz, F. (2014).  Handbook of Differential Entropy.  Chapman & Hall/CRC, Boca Raton, FL.
  * Newton et al., (2021) Newton, M. A., Polson, N. G., and Xu, J. (2021).  Weighted bayesian bootstrap for scalable posterior distributions.  Canadian Journal of Statistics, 49(2):421–437.
  * Newton and Raftery, (1994) Newton, M. A. and Raftery, A. E. (1994).  Approximate bayesian inference with the weighted likelihood bootstrap (with discussion).  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 56:3–48.
  * Oliver, (2004) Oliver, D. (2004).  Basketball on Paper: Rules and Tools for Performance Analysis.  Potomac Books, Inc, Washington, D.C.
  * Robin and Scrucca, (2023) Robin, S. and Scrucca, L. (2023).  Mixture-based estimation of entropy.  Computational Statistics & Data Analysis, 177:107582.
  * Rubin, (1981) Rubin, D. B. (1981).  The bayesian bootstrap.  The Annals of Statistics, 9(1):130–134.
  * Schwarz, (1978) Schwarz, G. (1978).  Estimating the dimension of a model.  The Annals of Statistics, 6(2):461–464.
  * Scrucca, (2024) Scrucca, L. (2024).  mclustAddons: Addons for the ’mclust’ Package.  R package version 0.8.
  * Scrucca et al., (2016) Scrucca, L., Fop, M., Murphy, T. B., and Raftery, A. E. (2016).  mclust 5: Clustering, classification and density estimation using Gaussian finite mixture models.  The R Journal, 8(1):205–233.
  * Scrucca et al., (2023) Scrucca, L., Fraley, C., Murphy, T. B., and Raftery, A. E. (2023).  Model-Based Clustering, Classification, and Density Estimation Using mclust in R.  Chapman & Hall/CRC, Boca Raton, FL.
  * Shannon, (1948) Shannon, C. E. (1948).  A mathematical theory of communication.  The Bell System Technical Journal, 27(3):379–423.
  * Shao and Tu, (1995) Shao, J. and Tu, D. (1995).  The Jackknife and Bootstrap.  Springer Science & Business Media, New York, NY.
  * Singh and Xie, (2010) Singh, K. and Xie, M. (2010).  Bootstrap method.  In Peterson, P., Baker, E., and McGaw, B., editors, International Encyclopedia of Education, volume 7, pages 46–51. Elsevier, Oxford, third edition edition.
  * Wang and Madiman, (2014) Wang, L. and Madiman, M. (2014).  Beyond the entropy power inequality, via rearrangements.  IEEE Transactions on Information Theory, 60(9):5116–5137.
  * Zuccolotto and Manisera, (2020) Zuccolotto, P. and Manisera, M. (2020).  Basketball Data Science: With Applications in R.  Chapman & Hall/CRC, Boca Raton, FL.

## Appendix

##  Appendix A Simulation results

Table 3: Simulation results for data generated from a Gaussian distribution with $\mu=0$ and $\sigma=1$. For each bootstrap method and sample size, the average estimate of entropy, the bootstrap estimates of bias and standard error, and the associated bootstrap percentile intervals are reported. |  |  |  | 95% percentile interval | 95% centered perc. interval
---|---|---|---|---|---
Sample size | Estimate | Bias | SE | lower | upper | coverage | lower | upper | coverage
Nonparametric bootstrap
100 | 1.4096 | 0.0104 | 0.0706 | 1.2556 | 1.5305 | 0.925 | 1.2886 | 1.5632 | 0.945
200 | 1.4141 | 0.0050 | 0.0498 | 1.3092 | 1.5035 | 0.942 | 1.3248 | 1.5191 | 0.945
500 | 1.4163 | 0.0020 | 0.0316 | 1.3517 | 1.4750 | 0.936 | 1.3575 | 1.4810 | 0.944
1000 | 1.4170 | 0.0011 | 0.0224 | 1.3719 | 1.4592 | 0.955 | 1.3749 | 1.4621 | 0.957
Parametric bootstrap
100 | 1.4096 | 0.0105 | 0.0715 | 1.2548 | 1.5338 | 0.939 | 1.2853 | 1.5637 | 0.951
200 | 1.4141 | 0.0050 | 0.0502 | 1.3088 | 1.5047 | 0.943 | 1.3236 | 1.5195 | 0.947
500 | 1.4163 | 0.0020 | 0.0316 | 1.3516 | 1.4751 | 0.944 | 1.3574 | 1.4809 | 0.947
1000 | 1.4170 | 0.0010 | 0.0224 | 1.3718 | 1.4593 | 0.957 | 1.3748 | 1.4621 | 0.962
Weighted likelihood bootstrap ($\alpha=1$)
100 | 1.4096 | 0.0101 | 0.0684 | 1.2675 | 1.5345 | 0.930 | 1.2851 | 1.5516 | 0.948
200 | 1.4141 | 0.0049 | 0.0490 | 1.3148 | 1.5062 | 0.939 | 1.3219 | 1.5138 | 0.943
500 | 1.4163 | 0.0020 | 0.0314 | 1.3537 | 1.4762 | 0.946 | 1.3564 | 1.4788 | 0.939
1000 | 1.4170 | 0.0010 | 0.0223 | 1.3729 | 1.4599 | 0.960 | 1.3742 | 1.4612 | 0.957
Weighted likelihood bootstrap ($\alpha=4$)
100 | 1.4096 | 0.0025 | 0.0346 | 1.3401 | 1.4753 | 0.654 | 1.3441 | 1.4792 | 0.650
200 | 1.4141 | 0.0013 | 0.0247 | 1.3650 | 1.4615 | 0.694 | 1.3668 | 1.4633 | 0.689
500 | 1.4163 | 0.0006 | 0.0157 | 1.3854 | 1.4467 | 0.664 | 1.3859 | 1.4474 | 0.667
1000 | 1.4170 | 0.0003 | 0.0111 | 1.3951 | 1.4387 | 0.681 | 1.3955 | 1.4389 | 0.684
Weighted likelihood bootstrap ($\alpha=0.8137$)
100 | 1.4096 | 0.0121 | 0.0754 | 1.2517 | 1.5460 | 0.954 | 1.2729 | 1.5670 | 0.962
200 | 1.4141 | 0.0061 | 0.0541 | 1.3038 | 1.5152 | 0.963 | 1.3129 | 1.5244 | 0.966
500 | 1.4163 | 0.0024 | 0.0347 | 1.3471 | 1.4825 | 0.958 | 1.3503 | 1.4855 | 0.965
1000 | 1.4170 | 0.0013 | 0.0246 | 1.3684 | 1.4644 | 0.973 | 1.3698 | 1.4658 | 0.971
Figure 10: Simulation results for data generated from a Gaussian distribution with $\mu=0$ and $\sigma=1$. Panel (a) shows the empirical coverage for 1000 replications as function of the sample size for different bootstrap methods (BS = nonparametric; PB = parametric; WLB = weigthed likelihood bootstrap with weights generated from a Dirichlet with different values of $\alpha$) and two type of bootstrap intervals (perc = percentile; cperc = centered percentile). Panel (b) show the ratio of bias (in absolute value) over the standard error for each bootstrap procedure as function of sample size. Table 4: Simulation results for data generated from a Mixed Gaussian distribution with $\mu=2$ and $\sigma=1$. For each bootstrap method and sample size, the average estimate of entropy, the bootstrap estimates of bias and standard error, and the associated bootstrap percentile intervals are reported. |  |  |  | 95% percentile interval | 95% centered perc. interval
---|---|---|---|---|---
Sample size | Estimate | Bias | SE | lower | upper | coverage | lower | upper | coverage
Nonparametric bootstrap
100 | 2.0273 | 0.0205 | 0.0601 | 1.8821 | 2.1158 | 0.861 | 1.9388 | 2.1728 | 0.936
200 | 2.0436 | 0.0100 | 0.0413 | 1.9495 | 2.1105 | 0.901 | 1.9765 | 2.1379 | 0.924
500 | 2.0474 | 0.0040 | 0.0259 | 1.9915 | 2.0926 | 0.941 | 2.0021 | 2.1034 | 0.961
1000 | 2.0511 | 0.0020 | 0.0183 | 2.0128 | 2.0841 | 0.955 | 2.0181 | 2.0895 | 0.962
Parametric bootstrap
100 | 2.0273 | 0.0206 | 0.0613 | 1.8805 | 2.1198 | 0.880 | 1.9350 | 2.1736 | 0.943
200 | 2.0436 | 0.0101 | 0.0420 | 1.9484 | 2.1123 | 0.911 | 1.9750 | 2.1386 | 0.934
500 | 2.0474 | 0.0042 | 0.0261 | 1.9911 | 2.0929 | 0.949 | 2.0018 | 2.1036 | 0.961
1000 | 2.0511 | 0.0024 | 0.0183 | 2.0128 | 2.0841 | 0.953 | 2.0180 | 2.0895 | 0.955
Weighted likelihood bootstrap ($\alpha=1$)
100 | 2.0273 | 0.0199 | 0.0577 | 1.8920 | 2.1175 | 0.865 | 1.9368 | 2.1625 | 0.934
200 | 2.0436 | 0.0099 | 0.0405 | 1.9542 | 2.1124 | 0.896 | 1.9749 | 2.1332 | 0.923
500 | 2.0474 | 0.0040 | 0.0257 | 1.9933 | 2.0935 | 0.942 | 2.0010 | 2.1014 | 0.957
1000 | 2.0511 | 0.0020 | 0.0182 | 2.0137 | 2.0847 | 0.954 | 2.0175 | 2.0885 | 0.957
Weighted likelihood bootstrap ($\alpha=4$)
100 | 2.0273 | 0.0050 | 0.0290 | 1.9652 | 2.0783 | 0.623 | 1.9764 | 2.0894 | 0.666
200 | 2.0436 | 0.0025 | 0.0203 | 2.0014 | 2.0806 | 0.627 | 2.0064 | 2.0859 | 0.637
500 | 2.0474 | 0.0010 | 0.0129 | 2.0213 | 2.0716 | 0.697 | 2.0232 | 2.0734 | 0.692
1000 | 2.0511 | 0.0005 | 0.0091 | 2.0329 | 2.0684 | 0.698 | 2.0337 | 2.0694 | 0.696
Weighted likelihood bootstrap ($\alpha=0.8137$)
100 | 2.0273 | 0.0243 | 0.0639 | 1.8751 | 2.1247 | 0.885 | 1.9305 | 2.1793 | 0.947
200 | 2.0436 | 0.0122 | 0.0448 | 1.9430 | 2.1182 | 0.924 | 1.9690 | 2.1440 | 0.952
500 | 2.0474 | 0.0049 | 0.0284 | 1.9870 | 2.0979 | 0.959 | 1.9967 | 2.1077 | 0.976
1000 | 2.0511 | 0.0024 | 0.0201 | 2.0095 | 2.0881 | 0.973 | 2.0141 | 2.0927 | 0.968
Figure 11: Simulation results for data generated from a Mixed Gaussian distribution with $\mu=2$ and $\sigma=1$. Panel (a) shows the empirical coverage for 1000 replications as function of the sample size for different bootstrap methods (BS = nonparametric; PB = parametric; WLB = weigthed likelihood bootstrap with weights generated from a Dirichlet with different values of $\alpha$) and two type of bootstrap intervals (perc = percentile; cperc = centered percentile). Panel (b) show the ratio of bias (in absolute value) over the standard error for each bootstrap procedure as function of sample size. Table 5: Simulation results for data generated from a Laplace distribution with $\mu=0$ and $\beta=2$. For each bootstrap method and sample size, the average estimate of entropy, the bootstrap estimates of bias and standard error, and the associated bootstrap percentile intervals are reported. |  |  |  | 95% percentile interval | 95% centered perc. interval
---|---|---|---|---|---
Sample size | Estimate | Bias | SE | lower | upper | coverage | lower | upper | coverage
Nonparametric bootstrap
100 | 2.3742 | 0.0259 | 0.1020 | 2.1380 | 2.5351 | 0.904 | 2.2134 | 2.6105 | 0.919
200 | 2.3767 | 0.0154 | 0.0709 | 2.2193 | 2.4955 | 0.926 | 2.2581 | 2.5345 | 0.945
500 | 2.3858 | 0.0066 | 0.0447 | 2.2909 | 2.4652 | 0.945 | 2.3066 | 2.4809 | 0.948
1000 | 2.3870 | 0.0035 | 0.0317 | 2.3212 | 2.4449 | 0.947 | 2.3292 | 2.4530 | 0.941
Parametric bootstrap
100 | 2.3742 | 0.0201 | 0.0899 | 2.1725 | 2.5234 | 0.870 | 2.2249 | 2.5758 | 0.883
200 | 2.3767 | 0.0118 | 0.0676 | 2.2302 | 2.4938 | 0.918 | 2.2592 | 2.5231 | 0.930
500 | 2.3858 | 0.0042 | 0.0436 | 2.2955 | 2.4655 | 0.947 | 2.3061 | 2.4762 | 0.948
1000 | 2.3870 | 0.0025 | 0.0310 | 2.3243 | 2.4450 | 0.941 | 2.3291 | 2.4500 | 0.942
Weighted likelihood bootstrap ($\alpha=1$)
100 | 2.3742 | 0.0226 | 0.0958 | 2.1659 | 2.5384 | 0.899 | 2.2101 | 2.5829 | 0.917
200 | 2.3767 | 0.0144 | 0.0689 | 2.2287 | 2.4976 | 0.922 | 2.2560 | 2.5246 | 0.938
500 | 2.3858 | 0.0065 | 0.0441 | 2.2940 | 2.4661 | 0.944 | 2.3055 | 2.4778 | 0.948
1000 | 2.3870 | 0.0034 | 0.0315 | 2.3226 | 2.4454 | 0.950 | 2.3285 | 2.4515 | 0.946
Weighted likelihood bootstrap ($\alpha=4$)
100 | 2.3742 | 0.0057 | 0.0481 | 2.2750 | 2.4627 | 0.616 | 2.2853 | 2.4731 | 0.639
200 | 2.3767 | 0.0038 | 0.0346 | 2.3057 | 2.4408 | 0.629 | 2.3125 | 2.4478 | 0.635
500 | 2.3858 | 0.0017 | 0.0222 | 2.3412 | 2.4276 | 0.675 | 2.3439 | 2.4305 | 0.682
1000 | 2.3870 | 0.0009 | 0.0158 | 2.3554 | 2.4170 | 0.677 | 2.3569 | 2.4186 | 0.679
Weighted likelihood bootstrap ($\alpha=0.8137$)
100 | 2.3742 | 0.0276 | 0.1060 | 2.1403 | 2.5520 | 0.919 | 2.1956 | 2.6082 | 0.935
200 | 2.3767 | 0.0175 | 0.0761 | 2.2113 | 2.5083 | 0.951 | 2.2451 | 2.5418 | 0.961
500 | 2.3858 | 0.0079 | 0.0487 | 2.2838 | 2.4737 | 0.964 | 2.2977 | 2.4878 | 0.971
1000 | 2.3870 | 0.0042 | 0.0348 | 2.3153 | 2.4512 | 0.968 | 2.3226 | 2.4584 | 0.967
Figure 12: Simulation results for data generated from a Laplace distribution with $\mu=0$ and $\beta=2$. Panel (a) shows the empirical coverage for 1000 replications as function of the sample size for different bootstrap methods (BS = nonparametric; PB = parametric; WLB = weigthed likelihood bootstrap with weights generated from a Dirichlet with different values of $\alpha$) and two type of bootstrap intervals (perc = percentile; cperc = centered percentile). Panel (b) show the ratio of bias (in absolute value) over the standard error for each bootstrap procedure as function of sample size. Table 6: Simulation results for data generated from a bivariate Gaussian distribution with $\mu=[0,0]{}^{\\!\top}$ and ${\boldsymbol{\Sigma}=\bigl{[}\begin{smallmatrix}1.0&0.8\\\ 0.8&2.0\end{smallmatrix}\bigr{]}}$. For each bootstrap method and sample size, the average estimate of entropy, the bootstrap estimates of bias and standard error, and the associated bootstrap percentile intervals are reported. |  |  |  | 95% percentile interval | 95% centered perc. interval
---|---|---|---|---|---
Sample size | Estimate | Bias | SE | lower | upper | coverage | lower | upper | coverage
Nonparametric bootstrap
100 | 2.9634 | 0.0258 | 0.0992 | 2.7375 | 3.1241 | 0.896 | 2.8029 | 3.1894 | 0.940
200 | 2.9779 | 0.0126 | 0.0702 | 2.8254 | 3.0994 | 0.915 | 2.8565 | 3.1303 | 0.923
500 | 2.9829 | 0.0050 | 0.0447 | 2.8898 | 3.0641 | 0.940 | 2.9019 | 3.0761 | 0.958
1000 | 2.9896 | 0.0025 | 0.0315 | 2.9252 | 3.0481 | 0.928 | 2.9312 | 3.0541 | 0.936
Parametric bootstrap
100 | 2.9634 | 0.0260 | 0.1014 | 2.7354 | 3.1305 | 0.910 | 2.7964 | 3.1918 | 0.943
200 | 2.9779 | 0.0125 | 0.0712 | 2.8242 | 3.1021 | 0.923 | 2.8541 | 3.1315 | 0.933
500 | 2.9829 | 0.0051 | 0.0448 | 2.8897 | 3.0644 | 0.943 | 2.9014 | 3.0765 | 0.950
1000 | 2.9896 | 0.0025 | 0.0316 | 2.9249 | 3.0485 | 0.934 | 2.9309 | 3.0545 | 0.940
Weighted likelihood bootstrap ($\alpha=1$)
100 | 2.9634 | 0.0250 | 0.0956 | 2.7517 | 3.1248 | 0.896 | 2.8022 | 3.1750 | 0.927
200 | 2.9779 | 0.0122 | 0.0687 | 2.8325 | 3.1011 | 0.915 | 2.8547 | 3.1233 | 0.923
500 | 2.9829 | 0.0049 | 0.0443 | 2.8924 | 3.0652 | 0.942 | 2.9008 | 3.0738 | 0.951
1000 | 2.9896 | 0.0025 | 0.0314 | 2.9263 | 3.0487 | 0.927 | 2.9304 | 3.0531 | 0.937
Weighted likelihood bootstrap ($\alpha=4$)
100 | 2.9634 | 0.0062 | 0.0484 | 2.8630 | 3.0519 | 0.595 | 2.8752 | 3.0637 | 0.616
200 | 2.9779 | 0.0031 | 0.0347 | 2.9075 | 3.0428 | 0.635 | 2.9127 | 3.0485 | 0.640
500 | 2.9829 | 0.0013 | 0.0223 | 2.9384 | 3.0253 | 0.664 | 2.9405 | 3.0274 | 0.660
1000 | 2.9896 | 0.0007 | 0.0157 | 2.9585 | 3.0198 | 0.671 | 2.9595 | 3.0208 | 0.667
Weighted likelihood bootstrap ($\alpha=0.8137$)
100 | 2.9634 | 0.0302 | 0.1057 | 2.7261 | 3.1381 | 0.918 | 2.7884 | 3.2002 | 0.953
200 | 2.9779 | 0.0149 | 0.0758 | 2.8162 | 3.1124 | 0.937 | 2.8432 | 3.1401 | 0.950
500 | 2.9829 | 0.0061 | 0.0489 | 2.8821 | 3.0732 | 0.961 | 2.8927 | 3.0839 | 0.970
1000 | 2.9896 | 0.0030 | 0.0347 | 2.9194 | 3.0548 | 0.953 | 2.9244 | 3.0599 | 0.950
Figure 13: Simulation results for data generated from a bivariate Gaussian distribution with $\mu=[0,0]{}^{\\!\top}$ and ${\boldsymbol{\Sigma}=\bigl{[}\begin{smallmatrix}1.0&0.8\\\ 0.8&2.0\end{smallmatrix}\bigr{]}}$. Panel (a) shows the empirical coverage for 1000 replications as function of the sample size for different bootstrap methods (BS = nonparametric; PB = parametric; WLB = weigthed likelihood bootstrap with weights generated from a Dirichlet with different values of $\alpha$) and two type of bootstrap intervals (perc = percentile; cperc = centered percentile). Panel (b) show the ratio of bias (in absolute value) over the standard error for each bootstrap procedure as function of sample size. Table 7: Simulation results for data generated from 10-dimensional independent $\chi^{2}$ distributions with $\text{df}=5$ degrees of freedom each. For each bootstrap method and sample size, the average estimate of entropy, the bootstrap estimates of bias and standard error, and the associated bootstrap percentile intervals are reported. |  |  |  | 95% percentile interval | 95% centered perc. interval
---|---|---|---|---|---
Sample size | Estimate | Bias | SE | lower | upper | coverage | lower | upper | coverage
Nonparametric bootstrap
100 | 24.1444 | 0.0819 | 0.2494 | 23.5687 | 24.5447 | 0.865 | 23.7527 | 24.7167 | 0.971
200 | 24.2166 | 0.0406 | 0.1719 | 23.8388 | 24.5103 | 0.940 | 23.9262 | 24.5912 | 0.914
500 | 24.2210 | 0.0154 | 0.1103 | 23.9901 | 24.4206 | 0.925 | 24.0227 | 24.4534 | 0.937
1000 | 24.2423 | 0.0073 | 0.0778 | 24.0830 | 24.3868 | 0.913 | 24.0986 | 24.4027 | 0.897
Parametric bootstrap
100 | 24.1444 | 0.0767 | 0.2575 | 23.5638 | 24.5660 | 0.868 | 23.7192 | 24.7280 | 0.985
200 | 24.2166 | 0.0344 | 0.1800 | 23.8258 | 24.5328 | 0.954 | 23.8997 | 24.6038 | 0.970
500 | 24.2210 | 0.0110 | 0.1133 | 23.9881 | 24.4290 | 0.953 | 24.0106 | 24.4513 | 0.937
1000 | 24.2423 | 0.0041 | 0.0800 | 24.0827 | 24.3945 | 0.926 | 24.0901 | 24.4028 | 0.912
Weighted likelihood bootstrap ($\alpha=1$)
100 | 24.1444 | 0.0804 | 0.2448 | 23.5838 | 24.5416 | 0.864 | 23.7513 | 24.7028 | 0.969
200 | 24.2166 | 0.0400 | 0.1693 | 23.8457 | 24.5068 | 0.939 | 23.9232 | 24.5859 | 0.927
500 | 24.2210 | 0.0150 | 0.1105 | 23.9909 | 24.4222 | 0.923 | 24.0211 | 24.4495 | 0.951
1000 | 24.2423 | 0.0073 | 0.0777 | 24.0845 | 24.3877 | 0.913 | 24.0973 | 24.4004 | 0.897
Weighted likelihood bootstrap ($\alpha=4$)
100 | 24.1444 | 0.0216 | 0.1233 | 23.8821 | 24.3635 | 0.542 | 23.9248 | 24.4058 | 0.588
200 | 24.2166 | 0.0095 | 0.0852 | 24.0422 | 24.3745 | 0.728 | 24.0592 | 24.3934 | 0.700
500 | 24.2210 | 0.0038 | 0.0552 | 24.1100 | 24.3258 | 0.615 | 24.1166 | 24.3320 | 0.628
1000 | 24.2423 | 0.0016 | 0.0390 | 24.1650 | 24.3175 | 0.547 | 24.1674 | 24.3197 | 0.564
Weighted likelihood bootstrap ($\alpha=0.8137$)
100 | 24.1444 | 0.0990 | 0.2690 | 23.5188 | 24.5698 | 0.894 | 23.7160 | 24.7718 | 0.985
200 | 24.2166 | 0.0471 | 0.1879 | 23.8032 | 24.5396 | 0.955 | 23.8983 | 24.6327 | 0.956
500 | 24.2210 | 0.0179 | 0.1216 | 23.9673 | 24.4413 | 0.953 | 24.0003 | 24.4758 | 0.954
1000 | 24.2423 | 0.0084 | 0.0865 | 24.0662 | 24.4029 | 0.955 | 24.0830 | 24.4170 | 0.941
Figure 14: Simulation results for data generated from 10-dimensional
independent $\chi^{2}$ distributions with $\text{df}=5$ degrees of freedom
each. Panel (a) shows the empirical coverage for 1000 replications as function
of the sample size for different bootstrap methods (BS = nonparametric; PB =
parametric; WLB = weigthed likelihood bootstrap with weights generated from a
Dirichlet with different values of $\alpha$) and two type of bootstrap
intervals (perc = percentile; cperc = centered percentile). Panel (b) show the
ratio of bias (in absolute value) over the standard error for each bootstrap
procedure as function of sample size.