###### Abstract

A discrete system’s heterogeneity is measured by the Rényi heterogeneity
family of indices (also known as Hill numbers or Hannah–Kay indices), whose
units are the numbers equivalent. Unfortunately, numbers equivalent
heterogeneity measures for non-categorical data require a priori (A)
categorical partitioning and (B) pairwise distance measurement on the
observable data space, thereby precluding application to problems with ill-
defined categories or where semantically relevant features must be learned as
abstractions from some data. We thus introduce representational Rényi
heterogeneity (RRH), which transforms an observable domain onto a latent space
upon which the Rényi heterogeneity is both tractable and semantically
relevant. This method requires neither a priori binning nor definition of a
distance function on the observable space. We show that RRH can generalize
existing biodiversity and economic equality indices. Compared with existing
indices on a beta-mixture distribution, we show that RRH responds more
appropriately to changes in mixture component separation and weighting.
Finally, we demonstrate the measurement of RRH in a set of natural images,
with respect to abstract representations learned by a deep neural network. The
RRH approach will further enable heterogeneity measurement in disciplines
whose data do not easily conform to the assumptions of existing indices.

###### keywords:

heterogeneity; diversity; Rényi heterogeneity; representation learning;
variational autoencoder; functional diversity indices; Hill numbers;
Leinster–Cobbold Index; Rao’s quadratic entropy

xx 1 5 Received: 26 March 2020; Accepted: 4 April 2020; Published: date
highlight HIGHLIGHThighlight Representational Rényi Heterogeneity Abraham
Nunes 1,2,†,∗, Martin Alda 1, Timothy Bardouille 3 and Thomas Trappenberg
2,*Abraham Nunes, Martin Alda, Timothy Bardouille and Thomas Trappenberg
Correspondence<EMAIL_ADDRESS>(A.N<EMAIL_ADDRESS>(T.T) Current address: 5909
Veterans Memorial Lane (8th Floor), Abbie J. Lane Memorial Building, QE II.
Health Sciences Centre, Halifax, NS B3H 2E2, Canada

##  1 Introduction

Measuring heterogeneity is of broad scientific importance, such as in studies
of biodiversity (ecology and microbiology) (Jost, 2006; Prehn-Kristensen et
al., 2018), resource concentration (economics) (Cowell, 2011), and consistency
of clinical trial results (biostatistics) (Higgins et al., 2003), to name a
few. In most of these cases, one measures the heterogeneity of a discrete
system equipped with a probability mass function.

Discrete systems assume that all observations of a given state are identical
(zero distance), and that all pairwise distances between states are
permutation invariant. This assumption is violated when relative distances
between states are important. For example, an ecosystem is not biodiverse if
all species serve the same functional role (Hooper et al., 2005). Although
species are categorical labels, their pairwise differences in terms of
ecological functions differ and thus violate the discrete space assumptions.
Mathematical ecologists have thus developed heterogeneity measures for non-
categorical systems, which they generally call “functional diversity indices”
(Botta-Dukát, 2018; Mouchet et al., 2010; Chiu and Chao, 2014; Petchey and
Gaston, 2002; Leinster and Cobbold, 2012; Chao et al., 2014). These indices
typically require a priori discretization and specification of a distance
function on the observable space.

The requirement for defining the state space a priori is problematic when the
states are incompletely observable: that is, when they may be noisy,
unreliable, or invalid. For example, consider sampling a patient from a
population of individuals with psychiatric disorders and assigning a
categorical state label corresponding to his or her diagnosis according to
standard definitions (American Psychiatric Association, 2015). Given that
psychiatric conditions are not defined by objective biomarkers, the
individual’s diagnostic state will be uncertain. Indeed, many of these
conditions are inconsistently diagnosed across raters (Regier et al., 2013),
and there is no guarantee that they correspond to valid biological processes.
Alternatively, it is possible that variation within some categorical
diagnostic groups is simply related to diagnostic “noise,” or nuisance
variation, but that variation within other diagnostic groups constitutes the
presence of sub-strata. Appropriate measurement of heterogeneity in such
disciplines requires freedom from the discretization requirement of existing
non-categorical heterogeneity indices.

Pre-specified distance functions may fail to capture semantically relevant
geometry in the raw feature space. For example, the Euclidean distance between
Edmonton and Johannesburg is relatively useless since the straight-line path
cannot be traversed. Rather, the appropriate distances between points must
account for the data’s underlying manifold of support. Representation learning
addresses this problem by learning a latent embedding upon which distances are
of greater semantic relevance (Bengio et al., 2013). Indeed, we have observed
superior clustering of natural images embedded on Riemannian manifolds
(Arvanitidis et al., 2018) (but also see Shao et al. (2018)), and preservation
of semantic hierarchies when linguistic data are embedded on a hyperbolic
space (Nickel and Kiela, 2017).

Therefore, we seek non-categorical heterogeneity indices without requisite a
priori definition of categorical state labels or a distance function. The
present study proposes a solution to these problems based on the measurement
of heterogeneity on learned latent representations, rather than on raw
observable data. Our method, representational Rényi heterogeneity (RRH),
involves learning a mapping from the space of observable data to a latent
space upon which an existing measure (the Rényi heterogeneity (Rényi, 1961),
also known as the Hill numbers (Hill, 1973) or Hannah–Kay indices (Hannah and
Kay, 1977)) is meaningful and tractable.

The paper is structured as follows. Section 2 introduces the original
categorical formulation of Rényi heterogeneity and various approaches by which
it has been generalized for application on non-categorical spaces (Ricotta and
Szeidl, 2009; Leinster and Cobbold, 2012; Chiu and Chao, 2014). Limitations of
these indices are highlighted, thereby motivating Section 3, which introduces
the theory of Representational Rényi Heterogeneity (RRH), which generalizes
the process for computing many indices of biodiversity and economic equality.
Section 4 provides an illustration of how RRH may be measured in various
analytical contexts. We provide an exact comparison of RRH to existing non-
categorical heterogeneity indices under a tractable mixture of beta
distributions. To highlight the generalizability of our approach to complex
latent variable models, we also provide an evaluation of RRH applied to the
latent representations of a handwritten image dataset (LeCun et al., 1998)
learned by a variational autoencoder (Kingma and Welling, 2014, 2019).
Finally, in Section 5 we provide a summary of our findings and discuss avenues
for future work.

##  2 Existing Heterogeneity Indices

###  2.1 Rényi Heterogeneity in Categorical Systems

There are many approaches to derive Rényi heterogeneity (Rényi, 1961; Hill,
1973; Hannah and Kay, 1977). Here, we loosely follow the presentation of
Eliazar and Sokolov (2012) by using the metaphor of repeated sampling from a
discrete system $X$ with event space
$\mathcal{X}=\left\\{1,2,\ldots,n\right\\}$ and probability distribution
$\mathbf{p}=\left(p_{i}\right)_{i=1,2,\ldots,n}$. The probability that
$q\in\mathbb{N}_{>1}$ independent and identically distributed (i.i.d.)
realizations of $X$, sampled with replacement, will be identical is

$\mathbb{P}_{X}\left[x_{1}=x_{2}=\cdots=x_{q}\right]=\sum_{i=1}^{n}p_{i}^{q}.$
(1)

Now let $X_{\ast}$ be an idealized reference system with a uniform probability
distribution over $n_{\ast}$ categorical states,
$\mathbf{p}_{\ast}=\left(n_{\ast}^{-1}\right)_{i=1,2,\ldots,n_{\ast}}$, and
let $\left(x_{\ast 1},x_{\ast 2},\ldots,x_{\ast q}\right)$ be a sample of $q$
i.i.d. realizations of $X_{\ast}$ such that

$\mathbb{P}_{X}\left[x_{1}=x_{2}=\cdots=x_{q}\right]=\mathbb{P}_{X_{\ast}}\left[x_{\ast
1}=x_{\ast 2}=\cdots=x_{\ast q}\right]=\sum_{i=1}^{n_{\ast}}n_{\ast}^{-q}.$
(2)

We call $X_{\ast}$ an “idealized” categorical system because its probability
distribution is uniform, and it is a “reference” system for $X$ in that the
probability of drawing homogeneous samples of $q$ observations from both
systems is identical. Substituting Equation 2 into Equation 1 and solving for
$n_{\ast}$ yields the Rényi heterogeneity of order $q$,

$\Pi_{q}\left(\mathbf{p}\right)=\left(\sum_{i=1}^{n}p_{i}^{q}\right)^{\frac{1}{1-q}}=n_{\ast},$
(3)

whose units are the numbers equivalent of system $X$ (Patil and Taillie, 1982;
Adelman, 1969; Jost, 2007, 2006), insofar as $n_{\ast}$ is the number of
states in an “equivalent” (idealized reference) system $X_{\ast}$. Thus far,
we have restricted the parameter $q$ to take integer values greater than 1
solely to facilitate this intuitive derivation in a concise fashion. However,
the elasticity parameter $q$ in Equation 3 can be any real number (but $q\neq
1$), although in the context of heterogeneity measurement only $q\geq 0$ are
used (Eliazar and Sokolov, 2012; Jost, 2006). Despite Equation 3 being
udefined at $q=1$ directly, L’Hôpital’s rule can be used to show that the
limit $q\to 1$ exists, wherein it corresponds to the exponential of Shannon’s
entropy (Shannon, 1948; Jost, 2007), known as perplexity (Eliazar, 2015).

Equation 3 is the exponential of Rényi’s entropy (Rényi, 1961), and is
alternatively known as the Hill numbers in ecology (Hill, 1973; Jost, 2006),
Hannah–Kay indices in economics (Hannah and Kay, 1977), and generalized
inverse participation ratio in physics (Eliazar and Sokolov, 2012).
Interestingly, it generalizes or can be transformed into several heterogeneity
indices that are commonly employed across scientific disciplines (Table 1).

Table 1: Relationships between Rényi heterogeneity and various diversity or inequality indices for a system $X$ with event space $\mathcal{X}=\\{1,2,\ldots,n\\}$ and probability distribution $\mathbf{p}=\left(p_{i}\right)_{i=1,2,\ldots,n}$. The function $\mathbbm{1}[\cdot]$ is an indicator function that evaluates to 1 if its argument is true or to 0 otherwise. Index |  Expression
---|---
Observed richness (Gotelli and Chao, 2013) |  $\Pi_{0}\left(\mathbf{p}\right)=\sum_{i=1}^{n}\mathbbm{1}[p_{i}>0]$
Perplexity (Eliazar, 2015) |  $\Pi_{1}\left(\mathbf{p}\right)=\exp\left\\{-\sum_{i=1}^{n}p_{i}\log p_{i}\right\\}$
Inverse Simpson concentration (Jost, 2006) |  $\Pi_{2}\left(\mathbf{p}\right)=\left(\sum_{i=1}^{n}p_{i}^{2}\right)^{-1}$
Berger-Parker Diversity Index (Berger and Parker, 1970; Daly et al., 2018) |  $\Pi_{\infty}\left(\mathbf{p}\right)=\left(\max_{i}p_{i}\right)^{-1}$
Rényi entropy (Rényi, 1961) |  $R_{q}\left(\mathbf{p}\right)=\log\Pi_{q}\left(\mathbf{p}\right)$
Shannon entropy (Shannon, 1948) |  $H\left(\mathbf{p}\right)=\log\Pi_{1}\left(\mathbf{p}\right)$
Tsallis entropy (Tsallis, 1988) |  $T_{q}\left(\mathbf{p}\right)=\frac{1}{q-1}\left(1-\Pi_{q}\left(\mathbf{p}\right)^{1-q}\right)$
Simpson concentration (Simpson, 1949) |  $\mathrm{Simpson}(\mathbf{p})=\left(\Pi_{2}\left(\mathbf{p}\right)\right)^{-1}$
Gini-Simpson index Gini (1912) |  $\mathrm{GSI}(\mathbf{p})=1-\mathrm{Simpson}(\mathbf{p})$
Generalized entropy index (Shorrocks, 1980; Cowell, 2011) |  $\mathrm{GEI}\left(\mathbf{p}\right)=\frac{1}{q(q-1)}\left[\left(\frac{1}{n}\Pi_{q}\left(\mathbf{p}\right)\right)^{1-q}-1\right]$

####  2.1.1 Properties of the Rényi Heterogeneity

Equation 3 satisfies several properties that render it a preferable measure of
heterogeneity. These have been detailed elsewhere (Hannah and Kay, 1977;
Eliazar and Sokolov, 2012; Jost, 2006, 2007, 2009; Daly et al., 2018), but we
focus on three properties that are of particular relevance for the remainder
of this paper.

First, $\Pi_{q}$ satisfies the principle of transfers (Pigou, 1912; Dalton,
1920) which states that any equality-increasing transfer of probability
between states must increase the heterogeneity. The maximal value of $\Pi_{q}$
is attained if and only if $p_{i}=p_{j}$ for all $(i,j)\in\\{1,2,\ldots,n\\}$.
This property follows from Schur-concavity of Equation 3 (Hannah and Kay,
1977).

Second, $\Pi_{q}$ satisfies the replication principle (Macarthur, 1965; Jost,
2006, 2009), which is equivalent to stating that Equation 3 scales linearly
with the number of equally probable states in an idealized categorical system
(Eliazar and Sokolov, 2012). More formally, consider a set of systems
$X_{1},X_{2},\ldots,X_{N}$ with probability distributions
$\mathbf{p}_{1},\mathbf{p}_{2},\ldots,\mathbf{p}_{N}$ over respective discrete
event spaces $\mathcal{X}_{1},\mathcal{X}_{2},\ldots,\mathcal{X}_{N}$. These
systems are also assumed to satisfy the following properties:

  1. 1.

Event spaces are disjoint: $\mathcal{X}_{i}\cap\mathcal{X}_{j}=\emptyset$ for
all $(i,j)\in\\{1,2,\ldots,N\\}$ where $i\neq j$

  2. 2.

All systems have equal heterogeneity:
$\Pi_{q}\left(\mathbf{p}_{1}\right)=\Pi_{q}\left(\mathbf{p}_{2}\right)=\cdots=\Pi_{q}\left(\mathbf{p}_{i}\right)=\cdots=\Pi_{q}\left(\mathbf{p}_{N}\right)$

The replication principle states that if we combine $X_{1},X_{2},\ldots,X_{N}$
into a pooled system $X$ with probability distribution $\bar{\mathbf{p}}$,
then

$\Pi_{q}\left(\bar{\mathbf{p}}\right)=N\Pi_{q}\left(\mathbf{p}_{i}\right)$ (4)

must hold (see Appendix A for proof that Rényi heterogeneity satisfies the
replication principle).

The replication principle suggests that Equation 3 satisfies a property known
as decomposability, in that the heterogeneity of a pooled system can be
decomposed into that arising from variation within and between component
subsystems. However, we require that this property be satisfied when either
(A) subsystems’ event spaces are overlapping, or (B) subsystems do not have
equal heterogeneity. The decomposability property will be particularly
important for Section 3, and so we detail it further in Section 2.1.2.

####  2.1.2 Decomposition of Categorical Rényi Heterogeneity

Consider a system $X$ defined by pooling subsystems $X_{1},X_{2},\ldots,X_{N}$
with potentially overlapping event spaces
$\mathcal{X}_{1},\mathcal{X}_{2},\ldots,\mathcal{X}_{N}$, respectively. The
event space of the pooled system is defined as

$\mathcal{X}=\cup_{i=1}^{N}\mathcal{X}_{i}=\left\\{1,2,\ldots,n\right\\}.$ (5)

Furthermore, we define the matrix
$\mathbf{P}=\left(p_{ij}\right)_{i=1,2,\ldots,N}^{j=1,2,\ldots,n}$ whose $i$th
row is the probability of system $X_{i}$ being observed in each state
$j\in\\{1,2,\ldots,n\\}$.

It may be the case that some subsystems comprise a larger proportion of $X$
than others. For instance, if the probability distribution for subsystem
$X_{i}$ was estimated based on a larger sample size than that of $X_{j}$, one
may want to weight the contribution of $X_{i}$ higher. Thus, we define a
column vector of weights $\mathbf{w}=\left(w_{i}\right)_{i=1,2,\ldots,N}$ over
the $N$ subsystems such that $\sum_{i=1}^{N}w_{i}=1$ and $w_{i}\geq 0$ for all
$i$. The probability distribution over states in the pooled system $X$ may
thus be computed as $\bar{\mathbf{p}}=\sum_{i=1}^{N}w_{i}\mathbf{p}_{i}$, from
which the definition of pooled heterogeneity follows:

$\Pi_{q}^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)=\left[\sum_{j=1}^{n}\left(\sum_{i=1}^{N}w_{i}p_{ij}\right)^{q}\right]^{\frac{1}{1-q}}.$
(6)

One can interpret $\Pi_{q}^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)$ as
the effective number of states in the pooled categorical system $X$.

Jost (2007) showed that the within-group heterogeneity, which is the effective
number of unique states arising from individual component systems, can be
defined as

$\Pi_{q}^{\mathrm{W}}\left(\mathbf{P},\mathbf{w}\right)=\left[\frac{\sum_{i=1}^{N}w_{i}^{q}\left(\sum_{j=1}^{n}p_{ij}^{q}\right)}{\sum_{k=1}^{N}w_{k}^{q}}\right]^{\frac{1}{1-q}},$
(7)

For example, in the case where all subsystems have disjoint event spaces with
heterogeneity equal to constant $\nu$, then they each contribute $\nu$ unique
states to the pooled system $X$.

Deriving the between-group heterogeneity
$\Pi_{q}^{\mathrm{B}}\left(\mathbf{P},\mathbf{w}\right)$, is thus
straightforward. If the effective total number of states in the pooled system
is $\Pi_{q}^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)$, and the effective
number of unique states contributed by distinct subsystems is
$\Pi_{q}^{\mathrm{W}}\left(\mathbf{P},\mathbf{w}\right)$, then

$\Pi_{q}^{\mathrm{B}}\left(\mathbf{P},\mathbf{w}\right)=\frac{\Pi_{q}^{\mathrm{P}}\left(\mathbf{P},\mathbf{w}\right)}{\Pi_{q}^{\mathrm{W}}\left(\mathbf{P},\mathbf{w}\right)}$
(8)

is the effective number of completely distinct subsystems in the pooled system
$X$. A word of caution is warranted. If we require that within-group
heterogeneity is a lower bound on pooled heterogeneity (Lande, 1996), then
Jost (2007, see Proofs 2 and 3) showed that Equation 8 will hold (A) at any
value of $q$ when weights are equal (i.e., $w_{i}=1/N$ for all
$i\in\\{1,2,\ldots,N\\}$), or (B) only at $q=0$ and $q=1$ if weights are
unequal.

####  2.1.3 Limitations of Categorical Rényi Heterogeneity

The chief limitation of Rényi heterogeneity (Equation 3) is its assumption
that all states in a system $X$ (with event space
$\mathcal{X}=\\{1,2,\ldots,n\\}$ and probability distribution
$\mathbf{p}=\left(p_{i}\right)_{i=1,2,\ldots,n}$) are categorical. More
formally, the dissimilarity between a pair of observations
$(x,y)\in\mathcal{X}$ from this system is defined by the discrete metric

$d^{\ast}(x,y)=1-\delta_{xy},$ (9)

where $\delta_{xy}$ is Kronecker’s delta, which takes a value of 1 if $x=y$
and 0 otherwise. Since the discrete metric assumption is an idealization, we
have continued to use the asterisk to qualify an arbitrary distance function
$d(\cdot,\cdot)$ as categorical in nature. The resulting expected pairwise
distance matrix between states in $X$ is

$\mathbf{D}^{\ast}=\left[d^{\ast}(i,j)\right]_{i=1,2,\ldots,n}^{j=1,2,\ldots,n}=\mathbf{1}\
\mathbf{1}^{\top}-\mathbf{I},$ (10)

where $\mathbf{1}=\left(1\right)_{i=1,2,\ldots,n}$ is a column vector of ones,
and $\mathbf{I}=\left(\delta_{ij}\right)_{i=1,2,\ldots,n}^{j=1,2,\ldots,n}$ is
the $n\times n$ identity matrix.

Clearly, many systems of interest in the real world are not categorical. For
example, although we may label a sample of organisms according to their
respective species, there may be differences between these taxonomic classes
that are relevant to the functioning of the ecosystem as a whole (Hooper et
al., 2005). It is also possible that no valid and reliable set of categorical
labels is known a priori for a system whose event space is naturally non-
categorical.

###  2.2 Non-Categorical Heterogeneity Indices

Consider a system $X$ with probability distribution
$\mathbf{p}=\left(p_{i}\right)_{i=1,2,\ldots,n}$ defined over event space
$\mathcal{X}=\\{1,2,\ldots,n\\}$ and equipped with dissimilarity function
$d_{X}(\cdot,\cdot)$. We assume that $d_{X}$ is more general than the discrete
metric (Equation 9), and further still need not be a true (metric) distance.
For such systems, there are three heterogeneity indices whose units are
numbers equivalent, and respect the replication principle (Botta-Dukát, 2018;
Chiu and Chao, 2014; Chao et al., 2014; Ricotta and Szeidl, 2009; Leinster and
Cobbold, 2012). Much like our derivation of the Rényi heterogeneity in Section
2.1, these indices quantify the heterogeneity of a non-categorical system as
the number of states in an idealized reference system, but differ primarily in
how the idealized reference is defined. We begin with a discussion of the
Numbers-Equivalent Quadratic Entropy (Section 2.2.1), followed by the
Functional Hill Numbers (Section 2.2.2) and the Leinster–Cobbold index
(Leinster and Cobbold, 2012) (Section 2.2.3).

####  2.2.1 Numbers Equivalent Quadratic Entropy

Rao (1982) introduced the diversity index commonly known as Rao’s quadratic
entropy (RQE),

$Q_{1}\left(\mathbf{D},\mathbf{p}\right)=\sum_{i=1}^{n}\sum_{j=1}^{n}D_{ij}p_{i}p_{j}$
(11)

where $\mathbf{D}$ is an $n\times n$ matrix where $D_{ij}=d_{X}(i,j)$ for
states $(i,j)\in\mathcal{X}$.

Ricotta and Szeidl (2009) assume that $D_{ij}=1$ means that states $i$ and $j$
are maximally dissimilar (i.e., categorically different), and that $D_{ij}=0$
means $i=j$, which occurs when $\mathcal{X}$ is a categorical system. An
arbitrary dissimilarity matrix $\mathbf{D}$ can be rescaled to respect this
assumption by applying the following transformation:

$\tilde{\mathbf{D}}=\frac{\mathbf{D}-\min_{ij}D_{ij}}{\max_{ij}D_{ij}-\min_{ij}D_{ij}}.$
(12)

Under this transformation, Ricotta and Szeidl (2009) search for an idealized
categorical reference system $X_{\ast}$ with event space
$\mathcal{X}_{\ast}=\\{1,2,\ldots,n_{\ast}\\}$, probability distribution
$\mathbf{p}_{\ast}=\left(n_{\ast}^{-1}\right)_{i=1,2,\ldots,n_{\ast}}$, and
RQE equal to that of $X$. For a column vector of ones,
$\mathbf{1}=\left(1\right)_{i=1,2,\ldots,n_{\ast}}$, and the identity matrix
$\mathbf{I}=\left(\delta_{ij}\right)_{i=1,2,\ldots,n_{\ast}}^{j=1,2,\ldots,n_{\ast}}$,
this is

$Q_{1}\left(\tilde{\mathbf{D}},\mathbf{p}\right)=Q_{1}\left(\mathbf{1}\mathbf{1}^{\top}-\mathbf{I},\mathbf{p}_{\ast}\right).$
(13)

Expanding the right-hand side, we have

$Q_{1}\left(\tilde{\mathbf{D}},\mathbf{p}\right)=\sum_{i=1}^{n_{\ast}}\sum_{j=1}^{n_{\ast}}n_{\ast}^{-2}\left(1-\delta_{ij}\right)=1-\frac{1}{n_{\ast}}.$
(14)

Recalling that $\Pi_{q}\left(\mathbf{p}_{\ast}\right)=n_{\ast}$ and
substituting into Equation 14 yields

$\Pi_{q}\left(\mathbf{p}_{\ast}\right)=\left[1-Q_{1}\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right]^{-1},$
(15)

which establishes the units of
$\left[1-Q_{1}\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right]^{-1}$ as
numbers equivalent.

For consistency, we require that
$\Pi_{q}\left(\mathbf{p}_{\ast}\right)=\Pi_{q}\left(\mathbf{p}\right)$ if
$\tilde{\mathbf{D}}$ were categorical. This only holds at $q=2$:

$\left[1-Q_{1}\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right]^{-1}=\left[1-\sum_{i=1}^{n}\sum_{j=1}^{n}p_{i}p_{j}\left(1-\delta_{ij}\right)\right]^{-1}=\left(\sum_{i=1}^{n}p_{i}^{2}\right)^{-1}=\Pi_{2}\left(\mathbf{p}_{\ast}\right).$
(16)

Based on this result, Ricotta and Szeidl (2009) define the numbers equivalent
quadratic entropy $\hat{Q}_{e}$ as

$\hat{Q}_{e}\left(\tilde{\mathbf{D}},\mathbf{p}\right)=\left(1-Q_{1}\left(\tilde{\mathbf{D}},\mathbf{p}\right)\right)^{-1}.$
(17)

This can be interpreted as the inverse Simpson concentration of an idealized
categorical reference system whose average pairwise distance between states is
equal to $Q_{1}\left(\tilde{\mathbf{D}},\mathbf{p}\right)$.

####  2.2.2 Functional Hill Numbers

Chiu and Chao (2014) derived the Functional Hill Numbers, denoted $F_{q}$,
based on a similar procedure to that of Ricotta and Szeidl (2009). However,
whereas $\hat{Q}_{e}$ uses a purely categorical system as the idealized
reference, $F_{q}$ requires only that

$Q_{1}\left(\mathbf{D},\mathbf{p}\right)=\sum_{i=1}^{n_{\ast}}\sum_{j=1}^{n_{\ast}}Q_{1}\left(\mathbf{D},\mathbf{p}\right)p_{\ast
i}p_{\ast
j}=\sum_{i=1}^{n_{\ast}}\sum_{j=1}^{n_{\ast}}Q_{1}\left(\mathbf{D},\mathbf{p}\right)n_{\ast}^{-2},$
(18)

which means that the idealized reference system is one for which the between-
state distance matrix is set to $Q_{1}\left(\mathbf{D},\mathbf{p}\right)$
everywhere (or to 0 along the leading diagonal and
$Q_{1}\left(\mathbf{D},\mathbf{p}\right)n_{\ast}/(n_{\ast}-1)$ on the off
diagonals).

Chiu and Chao (2014) generalized Rao’s quadratic entropy to include the
elasticity parameter $q\geq 0$

$Q_{q}\left(\mathbf{D},\mathbf{p}\right)=\sum_{i=1}^{n}\sum_{j=1}^{n}D_{ij}\left(p_{i}p_{j}\right)^{q},$
(19)

and sought to find $n_{\ast}$ for the idealized reference system satisfying
Equation 18 and the following:

$Q_{q}\left(\mathbf{D},\mathbf{p}\right)=\sum_{i=1}^{n_{\ast}}\sum_{j=1}^{n_{\ast}}Q_{1}\left(\mathbf{D},\mathbf{p}\right)\left(\frac{1}{n_{\ast}}\frac{1}{n_{\ast}}\right)^{q}.$
(20)

Solving Equation 20 for $n_{\ast}$ yields the functional Hill numbers of order
$q$:

$F_{q}\left(\mathbf{D},\mathbf{p}\right)=\left(\frac{Q_{q}\left(\mathbf{D},\mathbf{p}\right)}{Q_{1}\left(\mathbf{D},\mathbf{p}\right)}\right)^{\frac{1}{2(1-q)}}=n_{\ast},$
(21)

which is the effective number of states in an idealized categorical reference
system whose distance function is scaled by a factor of
$Q_{1}\left(\mathbf{D},\mathbf{p}\right)n_{\ast}/(n_{\ast}-1)$.

####  2.2.3 Leinster–Cobbold Index

The index derived by Leinster and Cobbold (2012), denoted $L_{q}$, is distinct
from $\hat{Q}_{e}$ and $F_{q}$ in two ways. First, for a given system $X$, the
$L_{q}$ is not derived based on finding an idealized reference system
$X_{\ast}$ whose average between-state dissimilarity is equal to that of $X$.
Second, it does not use a dissimilarity matrix; rather, it uses a measure of
similarity or affinity.

The Leinster–Cobbold index may be derived by simple extension of Equation 3.
Assuming $X$ has state space $\mathcal{X}=\\{1,2,\ldots,n\\}$ with probability
distribution $\mathbf{p}=\left(p_{i}\right)_{i=1,2,\ldots,n}$, we note that

$\Pi_{q}\left(\mathbf{p}\right)=\left(\sum_{i=1}^{n}p_{i}^{q}\right)^{\frac{1}{1-q}}=\left[\sum_{i=1}^{n}p_{i}\left(\mathbf{I}\mathbf{p}\right)_{i}^{q-1}\right]^{\frac{1}{1-q}}.$
(22)

Here, $\mathbf{I}$ is the $n\times n$ identity matrix representing the
pairwise similarities between states in $X$. The Leinster–Cobbold index
generalizes $\mathbf{I}$ to be any $n\times n$ similarity matrix $\mathbf{S}$,
yielding the following formula:

$L_{q}\left(\mathbf{S},\mathbf{p}\right)=\left[\sum_{i=1}^{n}p_{i}\left(\sum_{j=1}^{n}S_{ij}p_{j}\right)^{q-1}\right]^{\frac{1}{1-q}}.$
(23)

The similarity matrix can be obtained from a dissimilarity matrix by the
transformation $S_{ij}=e^{-uD_{ij}}$, where $u\geq 0$ is a scaling factor.
When $u=0$, then $\mathbf{S}$ is 1 everywhere. Conversely, when $u\to\infty$,
then $\mathbf{S}$ approaches $\mathbf{I}$. The Leinster–Cobbold index can thus
be interpreted as an effective number if the states are in an idealized
reference system (i.e., one with uniform probabilities over states) whose
topology is also governed by the similarity matrix $\mathbf{S}$.

####  2.2.4 Limitations of Existing Non-Categorical Heterogeneity Indices

We illustrate several limitations of the $\hat{Q}_{e}$, $F_{q}$, and $L_{q}$
indices using a simple 3-state system $X$ with event space
$\mathcal{X}=\\{1,2,3\\}$ over which we specify a probability distribution

$\mathbf{p}(\kappa)=\left\\{\begin{array}[]{ll}\left(1,0,0\right)^{\top}&\kappa=0\\\
\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)^{\top}&\kappa=1\\\
\left(0,0,1\right)^{\top}&\kappa=\infty\\\
\left(\frac{1}{1+\sqrt{\kappa}+\kappa},\frac{\sqrt{\kappa}}{1+\sqrt{\kappa}+\kappa},\frac{\kappa}{1+\sqrt{\kappa}+\kappa}\right)^{\top}&\text{Otherwise}\end{array}\right.$
(24)

where $0\leq\kappa$ is a parameter that smoothly varies the level of
inequality. When $\kappa=1$ the distribution is perfectly even (Figure 1A).
Since an undirected graph of the system is arranged in a triangle with height
$h$ and base $b$, we also specify the following parametric distance matrix,

$\mathbf{D}(h,b)=\left(\begin{array}[]{ccc}0&b&\sqrt{\frac{b^{2}}{4}+h^{2}}\\\
b&0&\sqrt{\frac{b^{2}}{4}+h^{2}}\\\
\sqrt{\frac{b^{2}}{4}+h^{2}}&\sqrt{\frac{b^{2}}{4}+h^{2}}&0\\\
\end{array}\right),$ (25)

which allows us to smoothly vary the level of dissimilarity between states in
$X$. Importantly, Equation 25 allows us to generate distance matrices that are
either metric (when $h<b\sqrt{3}/2$; Definition 1) or ultrametric (when $h\geq
b\sqrt{3}/2$; Definition 1). This is illustrated in Figure 1B.

Figure 1: Illustration of simple three-state system under which we compare
existing non-categorical heterogeneity indices. Panel A depicts a three state
system $X$ as an undirected graph, with node sizes corresponding to state
probabilities governed by Equation 24. As $0\leq\kappa$ diverges further from
$\kappa=1$, the probability distribution over states becomes more unequal.
Panel B visually represents the parametric pairwise distance matrix
$\mathbf{D}(h,b)$ of Equation 25 ($h$ is height, $b$ is base length, $D_{ij}$
is distance between states $i$ and $j$). In the examples shown in Panels B and
C, we set $b=1$. Specifically, we provide visual illustration of settings for
which the distance function on $X$ is a metric (Definition 1; when
$h<b\sqrt{3}/2$) or ultrametric (Definition 1; when $h\geq b\sqrt{3}/2$).
Panel C compares the numbers equivalent quadratic entropy (solid lines marked
$\hat{Q}_{e}$; Section 2.2.1), functional Hill numbers (at $q=1$, dashed lines
marked $F_{1}$; Section 2.2.2), and the Leinster–Cobbold Index (at $q=1$,
dotted lines marked $L_{1}$; Section 2.2.3) for reporting the heterogeneity of
$X$. The y-axis reports the value of respective indices. The x-axis plots the
height parameter for the distance matrix $\mathbf{D}(h,1)$ (Equation 25 and
Panel B). The range of $h$ at which $\mathbf{D}(h,1)$ is only a metric is
depicted by the gray shaded background. The range of $h$ shown with a white
background is that for which $\mathbf{D}(h,1)$ is ultrametric. For each index,
we plot values for a probability distribution over states that is perfectly
even ($\kappa=1$; dotted markers) or skewed ($\kappa=10$; vertical line
markers). Panel D shows the sensitivity of the Leinster–Cobbold index
($L_{1}$; y-axis) to the scaling parameter $0\leq u$ (x-axis) used to
transform a distance matrix into a similarity matrix ($S_{ij}=e^{-uD_{ij}}$).
This is shown for three levels of skewness for the probability distribution
over states (no skewness at $\kappa=1$, dotted markers; significant skewness
at $\kappa=10$, vertical line markers; extreme skewness at $\kappa=100$,
square markers).

[Metric distance] A function
$d:\mathcal{X}\times\mathcal{X}\to\mathbb{R}_{\geq 0}$ on a set $\mathcal{X}$
is a metric if and only if all of the following conditions are satisfied for
all $(x,y,z)\in\mathcal{X}$:

  1. 1.

Non-negativity: $d(x,y)\geq 0$

  2. 2.

Identity of indiscernibles: $d(x,y)=0\iff x=y$

  3. 3.

Symmetry: $d(x,y)=d(y,x)$

  4. 4.

Triangle inequality: $d(x,z)\leq d(x,y)+d(y,z)$

[Ultrametric distance] A function
$d:\mathcal{X}\times\mathcal{X}\to\mathbb{R}_{\geq 0}$ on a set $\mathcal{X}$
is ultrametric if and only if, for all $(x,y,z)\in\mathcal{X}$, criteria 1-3
for a metric are satisfied (Definition 1), in addition to the ultrametric
triangle inequality:

$d(x,z)\leq\max\left\\{d(x,y),d(y,z)\right\\}$ (26)

Figure 1C compares the $\hat{Q}_{e},F_{q}$, and $L_{q}$ indices when applied
to $X$ across variation in between-state distances (via Equation 25) and
skewness in the probability distribution over states (Equation 24). With
respect to the numbers equivalent quadratic entropy ($\hat{Q}_{e}$; Section
2.2.1), we note that its behavior is categorically different with respect to
whether the distance matrix is ultrametric. That is $\hat{Q}_{e}$ increases
with the triangle height parameter $h$ (Equation 25) until it passes the
ultrametric threshold, after which it decreases monotonically with $h$. The
behavior of $\hat{Q}_{e}$ is sensible in the ultrametric range. When the
distance matrix is scaled, as in Equation 12, pulling one of the three states
in $X$ further away from the remaining two should function similarly to
progressively merging the latter states. Thus, the behavior of $\hat{Q}_{e}$
is highly sensitive to whether a given distance matrix is ultrametric (which
will often not be the case in real-world applications).

With respect to $F_{q}$, a notable benefit in comparison to $\hat{Q}_{e}$ is
that $F_{q}$ behaves consistently regardless of whether distance is
ultrametric. However, Figure 1 shows other drawbacks. First, we can see that
$F_{q}$ becomes insensitive to $\mathbf{D}(h,1)$ when $\mathbf{p}(\kappa)$ is
perfectly even (shown analytically in Appendix A). Second, $F_{q}$ can
paradoxically estimate a greater number of states than the theoretical maximum
allows. That this occurs when the state probability distribution is more
unequal violates the principle of transfers (Pigou, 1912; Dalton, 1920; Hannah
and Kay, 1977; Daly et al., 2018) (Section 2.1.1). This is made more
problematic since Figure 1C shows it occurs when one state is being pushed
closer to the others (i.e., with smaller values of $h$). To summarize, the
functional Hill numbers are estimating more states than are really present
despite the reduction in between-state distances and greater inequality in the
probability mass function.

Figure 1C shows that the Leinster-Cobbold index compares favorably to $F_{q}$
because the former does not lose sensitivity to dissimilarity when
$\mathbf{p}(\kappa)$ is perfectly even. However, Figure 1D shows that the
Leinster-Cobbold index is particularly sensitive to the form of similarity
transformation. In the present case, the maximal value of the $L_{q}$
gradually approaches 3 as $u$ grows (and only when $u\to\infty$ does it reach
3), while progressively losing sensitivity to distance. As mentioned by
Leinster and Cobbold (2012), the choice of $u$ or other similarity
transformation is dependent on the importance assigned to functional
differences between states. However, it is not clear how a given similarity
transformation (e.g., $u$), and therefore the idealized reference system of
$L_{q}$, should be validated.

Above all of the idiosyncratic limitations of existing numbers equivalent
heterogeneity indices, we must highlight two basic assumptions they all share.
First, they continue to assume that some valid and reliable categorical
partitioning on $X$ is known a priori. Second, they assume that a distance
function specified a priori describes semantically relevant geometry of the
system in question. These two limitations are not independent, since an
unreliable categorical partitioning of the state space will lead to erroneous
estimates of the pairwise distances between states. Thus, we seek an approach
for measuring heterogeneity that has neither these limitations, nor those
shown above to be specific to the other numbers equivalent heterogeneity
indices for non-categorical systems.

##  3 Representational Rényi Heterogeneity

In this section, we propose an alternative approach to the indices of Section
2.2 that we call representational Rényi heterogeneity (RRH). It involves
transforming $X$ into a representation $Z$, defined on an unobservable or
latent event space $\mathcal{Z}$, that satisfies two criteria:

  1. 1.

The representation $Z$ captures the semantically relevant variation in $X$

  2. 2.

Rényi heterogeneity can be directly computed on $Z$

Satisfaction of the first criterion can only be ascertained in a domain-
specific fashion. Since $Z$ is essentially a model of $X$, investigators must
justify that this model is appropriate for the scientific question at hand.
For example, an investigator may evaluate the ability of $X$ to be
reconstructed from representation $Z$ under cross-validation. The second
criterion simply means that the transformation of $X\to Z$ must specify a
probability distribution on $\mathcal{Z}$ upon which the Rényi heterogeneity
can be directly computed.

Figure 2 illustrates the basic idea of RRH. However, the specifics of this
framework differ based on the topology of the representation $Z$. Thus, the
remainder of this section discusses the following approaches:

  1. A.

Application of standard Rényi heterogeneity (Section 2.1) when $Z$ is a
categorical representation

  2. B.

Deriving parametric forms for Rényi heterogeneity when $Z$ is a non-
categorical representation

Figure 2: Graphical illustration of the two main approaches for computing
representational Rényi heterogeneity. In both cases, we map sampled points on
an observable space $\mathcal{X}$ onto a latent space $\mathcal{Z}$, upon
which we apply the Rényi heterogeneity measure. The mapping is illustrated by
the curved arrows, and should yield a posterior distribution over the latent
space. Panel A shows the case in which the latent space is categorical (for
example, discrete components of a mixture distribution on a continuous space).
Panel B illustrates the case in which the latent space has non-categorical
topology. A special case of the latter mapping may include probabilistic
principal components analysis. When the latent space is continuous, we must
derive a parametric form for the Rényi heterogeneity.

###  3.1 Rényi Heterogeneity on Categorical Representations

Let $X$ be a system defined on an observable space $\mathcal{X}$ that is non-
categorical and $n_{x}$-dimensional. Consider the scenario in which the
semantically relevant variation in $X$ is categorical: for instance, images of
different object categories stored in raw form as real-valued vectors. An
investigator may be interested in measuring the effective number of states in
$X$ with respect to this categorical variation. This requires transforming $X$
into a semantically relevant categorical representation $Z$ upon which
Equation 3 can be applied.

Assume we have a large random sample of $N$ points
$\mathbf{X}=\left(\mathbf{x}_{i}\right)_{i=1,2,\ldots,N}$ from system $X$. We
can conceptualize each discrete observation $\mathbf{x}_{i}$ in this sample as
the single point in the event space of a perfectly homogeneous subsystem
$X_{i}$. When pooled, the subsystems $\left\\{X_{i}\right\\}_{i=1,2,\ldots,N}$
constitute $X$. The contribution weights of each subsystem to $X$ as a whole
are denoted $\mathbf{w}=\left(w_{i}\right)_{i=1,2,\ldots,N}$, where
$\sum_{i=1}^{N}w_{i}=1$ and $w_{i}\geq 0$.

We now specify a vector-valued function
$\mathbf{f}:\mathcal{X}\to\mathcal{P}(\mathcal{Z})$ such that
$\mathbf{x}\mapsto\mathbf{f}(\mathbf{x})=\left[f_{j}(\mathbf{x})\right]_{j=1,2,\ldots,n_{z}}$
is a mapping from $n_{x}$-dimensional coordinates on the observable space,
$\mathbf{x}\in\mathcal{X}$, onto an $n_{z}$-dimensional discrete probability
distribution over $\mathcal{Z}=\\{1,2,\ldots,n_{z}\\}$. Thus,
$\mathbf{f}(\mathbf{x}_{i})$ can be conceptualized as mapping subsystem
$X_{i}$ onto its categorical representation $Z_{i}$. After defining
$\mathbf{f}$, the effective number of states in the latent representation of
$X_{i}$ can be computed as

$\Pi_{q}\left(\mathbf{x}_{i}\right)=\left(\sum_{j=1}^{n_{z}}f_{j}^{q}(\mathbf{x}_{i})\right)^{\frac{1}{1-q}}.$
(27)

When $\Pi_{q}\left(\mathbf{x}_{i}\right)=1$, then $\mathbf{f}$ assigns
$\mathbf{x}$ to a single category with perfect certainty. Conversely, when
$\Pi_{q}\left(\mathbf{x}_{i}\right)=n_{z}$, then either $\mathbf{x}_{i}$
belongs to all categorical states with equal probability, or $\mathbf{f}$ is
maximally uncertain about the mapping of point $\mathbf{x}_{i}$.

Mapping all points $\mathbf{X}$ onto the categorical latent space yields a
collection of subsystems $\left\\{Z_{i}\right\\}_{i=1,2,\ldots,N}$, which
generate $Z$ when pooled. Using Equation 6, we can compute the effective
number of total states in $Z$ as the pooled heterogeneity:

$\Pi_{q}^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)=\left[\sum_{j=1}^{n_{z}}\left(\sum_{i=1}^{N}w_{i}f_{j}(\mathbf{x}_{i})\right)^{q}\right]^{\frac{1}{1-q}},$
(28)

Unfortunately, $\Pi_{q}^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)$ counts
some heterogeneity that is due to uncertainty in the model (i.e., that
quantified by Equation 27). We, therefore, compute the effective number of
states in $Z$ per point $\mathbf{x}\in\mathcal{X}$ using the within-group
heterogeneity formula (Equation 7):

$\Pi_{q}^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)=\left[\frac{\sum_{i=1}^{N}w_{i}^{q}\left(\sum_{j=1}^{n_{z}}f_{j}^{q}(\mathbf{x}_{i})\right)}{\sum_{k=1}^{N}w_{k}^{q}}\right]^{\frac{1}{1-q}}.$
(29)

Finally, the effective number of states (points) in $X$—with respect to the
categorical variation modeled by $Z$—can then be computed using the between-
group heterogeneity formula (Equation 8):

$\Pi_{q}^{\mathrm{B}}\left(\mathbf{X},\mathbf{w}\right)=\frac{\Pi_{q}^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)}{\Pi_{q}^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)}.$
(30)

Example 3.1 demonstrates that current methods of measuring biodiversity and
wealth concentration can be viewed as special cases of categorical RRH.

[Classical measurement of biodiversity and economic equality as categorical
RRH]

Definitions necessary for this example are shown in Table 2. The traditional
analysis of species diversity and economic equality can be recovered from an
RRH-based formulation when $\mathbf{f}$ is assumed to be deterministic and
$\mathbf{w}=\left(N^{-1}\right)_{i=1,2,\ldots,N}$. In this case within-group
heterogeneity can be shown to reduce to 1:

$\begin{split}\Pi_{q}^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)&=\left[\sum_{i=1}^{N}\frac{N^{-q}}{\sum_{k=1}^{N}N^{-q}}\left(\sum_{j=1}^{n_{z}}f_{j}^{q}(\mathbf{x}_{i})\right)\right]^{\frac{1}{1-q}}\\\
&=\left[\sum_{i=1}^{N}N^{-1}\left(1\right)\right]^{\frac{1}{1-q}}\\\ &=1.\\\
\end{split}$ (31)

Thus, we have

$\begin{split}\Pi_{q}^{\mathrm{B}}\left(\mathbf{X},\mathbf{w}\right)&=\Pi_{q}^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)\\\
&=\left[\sum_{j=1}^{n_{z}}\left(\sum_{i=1}^{N}N^{-1}f_{j}(\mathbf{x}_{i})\right)^{q}\right]^{\frac{1}{1-q}}\\\
&=\left[\sum_{j=1}^{n_{z}}\left(\frac{N_{j}}{N}\right)^{q}\right]^{\frac{1}{1-q}},\end{split}$
(32)

which yields the categorical Rényi heterogeneity (Hill numbers for
biodiversity analysis and Hannah–Kay indices in the economic setting (Hill,
1973; Hannah and Kay, 1977)), and by extension many diversity indices to which
it is connected (Table 1). Thus, traditional analysis of species biodiversity
and economic equality are special cases of representational Rényi
heterogeneity where the representation is specified by a mapping onto
degenerate distributions over categorical labels. The only differences lie in
the definition of observable and latent spaces, and the representational
models.

In the case of biodiversity analysis, the model $\mathbf{f}$ in real-world
practice may simply be a human expert assigning species labels to a sample of
organisms from a field study. In the economic setting, one may speculate that
$\mathbf{f}$ would essentially reduce to contracts specifying ownership of
assets, whose value is deemed by market forces.

Table 2: Definitions in formulation of classical biodiversity and economic equality analysis as categorical representational Rényi heterogeneity. Superscripted indexing on $\mathbf{x}=\left(x_{i}\right)^{i=1,\ldots,n_{x}}$ denotes that this is a row vector. | Analytical Context
---|---
Symbol |  Biodiversity |  Economic Equality
$X$ |  Ecosystem, whose observation yields an organism denoted by vector $\mathbf{x}=\left(x_{i}\right)^{i=1,\ldots,n_{x}}\in\mathcal{X}$ |  A system of resources, whose observation yields an asset denoted by vector $\mathbf{x}=\left(x_{i}\right)^{i=1,\ldots,n_{x}}\in\mathcal{X}$
$\mathcal{X}\subseteq\mathbb{R}^{n_{x}}$ |  $n_{x}$-dimensional feature space of organisms in the ecosystem |  $n_{x}$-dimensional feature space of assets in the economy, whose topology is such that the “economic” or monetary value is equal at each coordinate $\mathbf{x}\in\mathcal{X}$
$\mathcal{Z}=\left\\{\mathbf{z}\in\left\\{0,1\right\\}^{n_{z}}:\sum_{i=1}^{n_{z}}z_{i}=1\right\\}$ |  $n_{z}$-dimensional space of one-hot species labels |  $n_{z}$-dimensional space of one-hot labels over wealth-owning agents
$\mathbf{f}:\mathcal{X}\to\mathcal{P}(\mathcal{Z})$ |  A model that performs the mapping $\mathbf{x}\mapsto\mathbf{f}(\mathbf{x})$ of organisms to discrete probability distributions over $\mathcal{Z}$ |  A model that performs the mapping $\mathbf{x}\mapsto\mathbf{f}(\mathbf{x})$ of assets to discrete probability distributions over $\mathcal{Z}$
$N_{i}\in\mathbb{N}_{+}$ |  The number of organisms observed belonging to species $i\in\left\\{1,\ldots,n_{z}\right\\}$ |  The number of equal valued assets belonging to agent $i\in\left\\{1,\ldots,n_{z}\right\\}$
$N=\sum_{i=1}^{n_{z}}N_{i}$ |  The total number of organisms observed |  The total quantity of assets observed
$\mathbf{X}=\left(x_{ij}\right)_{i=1,\ldots,N}^{j=1,\ldots,n_{x}}$ |  A sample of $N$ organisms |  A sample of $N$ assets
$\mathbf{w}=\left(w_{i}\right)_{i=1,\ldots,N}$ | Sample weights, such that $w_{i}\geq 0$ and $\sum_{i=1}^{N}w_{i}=1$

###  3.2 Rényi Heterogeneity on Non-Categorical Representations

In Section 3.1, we dealt with instances in which semantically relevant
variation in $X$ is categorical, such as when object categories are embedded
in images stored as real-valued vectors. Here, we consider scenarios in which
the semantically relevant information in an observable system $X$ is non-
categorical: for instance, where a piece of text contains information about
semantic concepts best represented as real-valued “word vectors” Mikolov et
al. (2013); Pennington et al. (2014). Measuring the effective number of
distinct states in $X$ with respect to this continuous variation requires
transforming $X$ into a semantically relevant continuous representation $Z$
upon which procedures analogous to those of Section 3.1 may be undertaken.

Let $Z$ be defined on an $n_{z}$-dimensional event space
$\mathcal{Z}\subseteq\mathbb{R}^{n_{z}}$ over which there exists a family of
parametric probability distributions $\mathcal{P}(\mathcal{Z})$ of a form
chosen by the experimenter. Let $f:\mathcal{X}\to\mathcal{P}(\mathcal{Z})$ be
a model that performs the mapping $\mathbf{x}\mapsto f(\cdot|\mathbf{x})$ from
a point $\mathbf{x}\in\mathcal{X}$ on the observable space to a probability
density on $\mathcal{Z}$. For example, if $\mathcal{P}(\mathcal{Z})$ is the
family of multivariate Gaussians, then
$f(\mathbf{z}|\mathbf{x}_{i})=\mathcal{N}(\mathbf{z}|\bm{\mu}_{i},\bm{\Sigma}_{i})$,
where $\bm{\mu}_{i}$ and $\bm{\Sigma}_{i}$ are the Gaussian mean and
covariance functions at $\mathbf{x}_{i}$, respectively. Given a sample
$\mathbf{X}=\left(\mathbf{x}_{i}\right)_{i=1,2,\ldots,N}$, as in Section 3.1,
we compute the continuous analogue of Equation 27 as follows

$\Pi_{q}\left(\mathbf{x}_{i}\right)=\left(\int_{\mathcal{Z}}f^{q}(\mathbf{z}|\mathbf{x}_{i})\
\mathrm{d}\mathbf{z}\right)^{\frac{1}{1-q}}.$ (33)

This formula yields the effective size of the domain of a uniform distribution
on $\mathbb{R}^{n_{z}}$ whose Rényi heterogeneity is equal to
$\Pi_{q}\left(\mathbf{x}_{i}\right)$ (proof is given in Appendix A). Thus, it
is possible for $\Pi_{q}\left(\mathbf{x}_{i}\right)$ to be less than 1, though
it will remain non-negative.

Similar to the procedure in Section 3.1, we now define a continuous version of
the within-observation heterogeneity

$\Pi_{q}^{\mathrm{W}}\left(\mathbf{X},\mathbf{w}\right)=\left[\sum_{i=1}^{N}\frac{w_{i}^{q}}{\sum_{j=1}^{N}w_{j}^{q}}\int_{\mathcal{Z}}f^{q}(\mathbf{z}|\mathbf{x}_{i})\
\mathrm{d}\mathbf{z}\right]^{\frac{1}{1-q}},$ (34)

which estimates the effective size of the latent space occupied per observable
point $\mathbf{x}\in\mathcal{X}$.

In order to compute the pooled heterogeneity
$\Pi_{q}^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)$, the experimenter
must specify the form of the pooled distribution, here denoted
$\bar{f}_{\mathbf{w}}$. The conceptually most simple approach is non-
parametric, using a model average,

$\bar{f}_{\mathbf{w}}\left(\mathbf{z}|\mathbf{X}\right)=\sum_{i=1}^{N}w_{i}f(\mathbf{z}|\mathbf{x}_{i}),$
(35)

whereby the pooled heterogeneity would be

$\Pi_{q}^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)=\left[\int_{\mathcal{Z}}\left(\sum_{i=1}^{N}w_{i}f(\mathbf{z}|\mathbf{x}_{i})\right)^{q}\
\mathrm{d}\mathbf{z}\right]^{\frac{1}{1-q}}.$ (36)

The integral in Equation 36 may often be analytically intractable and
potentially difficult to solve accurately in high dimensions with numerical
methods. Furthermore, some areas of $\mathcal{Z}$ may be assigned low
probability by $f(\mathbf{z}|\mathbf{x}_{i})$ for all
$i\in\\{1,2,\ldots,N\\}$. This is not a problem as the sample $\mathbf{X}$
becomes infinitely large. However, with finite samples, it may be the case
that some representational states in $\mathcal{Z}$ are unlikely simply because
we have not sampled from the corresponding regions of $\mathcal{X}$. An
alternative to Equation 35 is therefore to specify a parametric pooled
distribution

$\bar{f}_{\mathbf{w}}\left(\cdot|\mathbf{X}\right)=\Xi_{f}\left(\mathbf{X},\mathbf{w}\right),$
(37)

where $\Xi_{f}$ is a deterministic function that combines
$f(\cdot|\mathbf{x}_{i})$ for $i\in\\{1,2,\ldots,N\\}$ into a valid
probability density on $\mathcal{Z}$. In this case, the pooled Rényi
heterogeneity is simply

$\Pi_{q}^{\mathrm{P}}\left(\mathbf{X},\mathbf{w}\right)=\left(\int_{\mathcal{Z}}\bar{f}_{\mathbf{w}}^{q}(\mathbf{z}|\mathbf{X})\
\mathrm{d}\mathbf{z}\right)^{\frac{1}{1-q}}.$ (38)

Using either Equation 36 or 38 as the pooled heterogeneity and Equation 34 as
the within-group heterogeneity, the effective number of distinct states in
$X$—with respect to the non-categorical representation $Z$—can then be
computed using Equation 30.

Figure 3 demonstrates the difference between the parametric and non-parametric
approaches to pooling for non-categorical RRH, and Example 3 demonstrates one
approach to parametric pooling for a mixture of multivariate Gaussians.

Figure 3: Illustration of approaches to computing the pooled distribution on a
simple representational space $\mathcal{Z}=\mathbb{R}$. In this example, two
points on the observable space,
$(\mathbf{x}_{1},\mathbf{x}_{2})\in\mathcal{X}$, are mapped onto the latent
space via model $f(\cdot|\mathbf{x}_{i})$ for $i\in\\{1,2\\}$, which indexes
univariate Gaussians over $\mathcal{Z}$ (depicted as hatched patterns for
$\mathbf{x}_{1}$ and $\mathbf{x}_{2}$, respectively). A pooled distribution
computed non-parametrically by model-averaging (Equation 35) is depicted as
the solid black line. The parametrically pooled distribution (see Example 3)
is depicted as the dashed black line. The parametric approach implies the
assumption that further samples from $\mathcal{X}$ would yield latent space
projections in some regions assigned low probability by $f(z|\mathbf{x}_{1})$
and $f(z|\mathbf{x}_{2})$.

[Parametric pooling of multivariate Gaussian distributions] Let
$\mathbf{X}=\left(\mathbf{x}_{i}\right)_{i=1,2,\ldots,N}$ be a sample of
$n_{x}$-dimensional vectors from a system $X$ with event space
$\mathcal{X}\subseteq\mathbb{R}^{n_{x}}$. Let $Z$ be a latent representation
of $X$ with $n_{z}$-dimensional event space $\mathcal{Z}=\mathbb{R}^{n_{z}}$.
Let

$f(\mathbf{z}|\mathbf{x}_{i})=\mathcal{N}\left(\mathbf{z}|\bm{\mu}_{i},\bm{\Sigma}_{i}\right)$
(39)

be a model that returns a multivariate Gaussian density with mean
$\bm{\mu}_{i}$ and covariance $\bm{\Sigma}_{i}$ given point
$\mathbf{x}_{i}\in\mathcal{X}$. Finally, let
$\mathbf{w}=\left(w_{i}\right)_{i=1,2,\ldots,N}$ be weights assigned to each
sample in $\mathbf{X}$ such that $w_{i}\geq 0$ and $\sum_{i=1}^{N}w_{i}=1$.

If one assumes that the pooled distribution over $\mathcal{Z}$ given the set
of components
$f(\mathbf{z}|\mathbf{x}_{1}),f(\mathbf{z}|\mathbf{x}_{2}),\ldots,f(\mathbf{z}|\mathbf{x}_{N})$
is itself a multivariate Gaussian,

$\bar{f}_{\mathbf{w}}\left(\mathbf{z}|\mathbf{X}\right)=\mathcal{N}(\mathbf{z}|\bm{\mu}_{\ast},\bm{\Sigma}_{\ast})$
(40)

with $n_{z}\times 1$ pooled mean,

$\bm{\mu}_{\ast}=\sum_{i=1}^{N}w_{i}\bm{\mu}_{i}$ (41)

and $n_{z}\times n_{z}$ pooled covariance matrix

$\bm{\Sigma}_{\ast}=-\bm{\mu}_{\ast}\bm{\mu}_{\ast}^{\top}+\sum_{i=1}^{N}w_{i}\left[\bm{\Sigma}_{i}+\bm{\mu}_{i}\bm{\mu}_{i}^{\top}\right],$
(42)

then the pooled heterogeneity $\Pi_{q}^{\mathrm{P}}$ is therefore simply the
Rényi heterogeneity of a multivariate Gaussian,

$\Pi_{q}\left(\bm{\Sigma}\right)=\left\\{\begin{array}[]{ll}\mathrm{Undefined}&q=0\\\
\left(2\pi e\right)^{\frac{n_{z}}{2}}\sqrt{\left|\bm{\Sigma}\right|}&q=1\\\
\left(2\pi\right)^{\frac{n_{z}}{2}}\sqrt{\left|\bm{\Sigma}\right|}&q=\infty\\\
\left(2\pi\right)^{\frac{n_{z}}{2}}q^{\frac{n_{z}}{2(q-1)}}\sqrt{\left|\bm{\Sigma}\right|}&\mathrm{Otherwise}\end{array}\right.$
(43)

evaluated at $\bm{\Sigma}_{\ast}$. The derivation is provided in Appendix A
Nunes et al. (2020). Equation 43 at $\bm{\Sigma}_{\ast}$ is interpreted as the
effective size of space $\mathcal{Z}$ occupied by the complete latent
representation of $X$ under model $f$.

The within-group heterogeneity can be obtained for the set of components
$\left[f(\mathbf{z}|\mathbf{x}_{i})\right]_{i=1,2,\ldots,N}$ by solving
Equation 34 for the Gaussian densities, yielding:

$\Pi_{q}^{\mathrm{W}}\left(\bm{\Sigma}_{1:N},\mathbf{w}\right)=\left\\{\begin{array}[]{ll}\mathrm{Undefined}&q=0\\\
\exp\left\\{\frac{1}{2}\left(n_{z}+\sum_{i=1}^{N}w_{i}\log\left|2\pi\bm{\Sigma}_{i}\right|\right)\right\\}&q=1\\\
0&q=\infty\\\
\left(2\pi\right)^{\frac{n_{z}}{2}}\left(\sum_{i=1}^{N}\frac{\bar{w}_{i}^{q}\left|\bm{\Sigma}_{i}\right|^{\frac{1}{2}}}{q^{\frac{n_{z}}{2}}}\right)^{\frac{1}{1-q}}&\mathrm{Otherwise}\end{array}\right.,$
(44)

where we denote
$\bm{\Sigma}_{1:N}=\left\\{\bm{\Sigma}_{i}\right\\}_{i=1,2,\ldots,N}$ for
parsimony, and $\bar{w}_{i}=w_{i}\left(\sum_{j=1}^{N}w_{j}^{q}\right)^{-1/q}$.
Equation 44 estimates the effective size of the $n_{z}$-dimensional
representational space occupied per state $\mathbf{x}\in\mathcal{X}$.

The effective number of states in $X$ with respect to the continuous
representation $Z$ is thus the between-group heterogeneity
$\Pi_{q}^{\mathrm{B}}$ which can be computed as the ratio
$\Pi_{q}\left(\bm{\Sigma}_{\ast}\right)/\Pi_{q}^{\mathrm{W}}\left(\bm{\Sigma}_{1:N},\mathbf{w}\right)$.
The properties of this decomposition—specifically the conditions under which
$\Pi_{q}^{\mathrm{B}}\geq 1$ (Lande’s requirement (Lande, 1996; Jost,
2007))—are discussed further elsewhere Nunes et al. (2020).

##  4  Empirical Applications of Representational Rényi Heterogeneity

In this section, we demonstrate two applications of RRH under assumptions of
categorical (Section 4.1) and continuous (Section 4.2) latent spaces. First,
Section 4.1, uses a simple closed-form system consisting of a mixture of two
beta distributions on the (0,1) interval to give exact comparisons of the
behavior of RRH against that of existing non-categorical heterogeneity indices
(Section 2.2). This experiment provides evidence that existing non-categorical
heterogeneity indices can demonstrate counterintuitive behavior under various
circumstances. Second, Section 4.2 demonstrates that RRH can yield
heterogeneity measurements that are sensible and tractably computed, even for
highly complex mappings $f:\mathcal{X}\to\mathcal{P}(\mathcal{Z})$. There, we
use a deep neural network to compute the effective number of observations in a
database of handwritten images with respect to compressed latent
representations on a continuous space.

###  4.1 Comparison of Heterogeneity Indices Under a Mixture of Beta
Distributions

Consider a system $X$ with event space $\mathcal{X}$ on the open interval
$(0,1)$, containing an embedded, unobservable, categorical structure
represented by the latent system $Z$ with event space
$\mathcal{Z}=\left\\{1,2\right\\}$. The systems’ collective behavior is
governed by the joint distribution of a beta mixture model (BMM),

$p(x,z)=\mathbbm{1}[z=1](1-\theta_{1})\mathrm{Beta}_{\theta_{2},\theta_{3}}\left(x\right)+\mathbbm{1}[z=2]\theta_{1}\mathrm{Beta}_{\theta_{3},\theta_{2}}\left(x\right),$
(45)

where $\mathrm{Beta}_{\alpha,\beta}\left(x\right)$ is the probability density
function for a beta distribution with shape parameters $\alpha,\beta$, and
$\bm{\theta}=\left(\theta_{1},\theta_{2},\theta_{3}\right)$ are parameters.
The indicator function $\mathbbm{1}[\cdot]$ evaluates to 1 if its argument is
true, and to 0 otherwise. The prior distribution is

$p(z)=\mathbbm{1}[z=1](1-\theta_{1})+\mathbbm{1}[z=2]\theta_{1},$ (46)

and marginal probability of observable data is as follows (see Figure 4 for
illustrations):

$p(x)=(1-\theta_{1})\mathrm{Beta}_{\theta_{2},\theta_{3}}\left(x\right)+\theta_{1}\mathrm{Beta}_{\theta_{3},\theta_{2}}\left(x\right).$
(47)

Figure 4: Demonstration of data-generating distribution (top row; Equations
45-47), and relationship between the representational model’s decision
threshold (Equations 48, and 50) and categorical representational Rényi
heterogeneity (bottom row). The optimal decision boundary (Equation 50) is
shown as a gray vertical dashed line in all plots. Each column depicts a
specific parameterization of the data-generating system (parameters are stated
above the top row). Top Row: Probability density functions for data-generating
distributions. Shaded regions correspond to the two mixture components. Solid
black lines denote the marginal distribution (Equation 47). The x-axis
represents the observable domain, which is the (0,1) interval. Bottom Row:
Effect of varying categorical representational Rényi heterogeneity (RRH) for
$q\in\\{1,2,\infty\\}$ across different category assignment thresholds for the
beta-mixture models shown in the top row. Varying levels of decision boundary
are plotted on the x-axis. The y-axis shows the resulting between-observation
RRH. Black dots highlight the RRH computed at the optimal decision boundary.

To facilitate exact comparisons between heterogeneity indices, below, let us
assume we have a model $f:\mathcal{X}\to\mathcal{P}(\mathcal{Z})$ that maps an
observation $x\in\mathcal{X}$ onto a degenerate distribution over
$\mathcal{Z}$:

$f_{\bm{\theta}}(z|x)=\mathbbm{1}[z=1]\mathbbm{1}[x\leq\tau(\bm{\theta})]+\mathbbm{1}[z=2]\mathbbm{1}[x>\tau(\bm{\theta})].$
(48)

The subscripting of $f_{\bm{\theta}}$ denotes that the model is optimized such
that the threshold $0\leq\tau(\bm{\theta})\leq 1$ is the solution to

$p\left(z=1|x=\tau(\bm{\theta})\right)=p\left(z=2|x=\tau(\bm{\theta})\right),$
(49)

which is

$\tau(\bm{\theta})=\left\\{\begin{array}[]{ll}\left[\left(\theta_{1}^{-1}-1\right)^{\frac{1}{2(\theta_{2}-\theta_{3})}}\left(1-\theta_{1}\right)^{\frac{1}{2(\theta_{2}-\theta_{3})}}\theta_{1}^{-\frac{1}{2(\theta_{2}-\theta_{3})}}+1\right]^{-1}&\theta_{2}-\theta_{3}\neq
0\\\ 0&\left((\theta_{2}=\theta_{3})\wedge(\theta_{1}>\frac{1}{2})\right)\\\
1&\text{Otherwise}\end{array}\right.$ (50)

Under this model, the categorical RRH at point $x\in\mathcal{X}$ is

$\Pi_{q}\left(x\right)=\left(\sum_{i=1}^{2}f_{\bm{\theta}}^{q}\left(z=i|x\right)\right)^{\frac{1}{1-q}}=\left(\mathbbm{1}^{q}\left[x\leq\tau(\bm{\theta})\right]+\mathbbm{1}^{q}\left[x>\tau(\bm{\theta})\right]\right)^{\frac{1}{1-q}}=1.$
(51)

The expected value of $f_{\bm{\theta}}(z=2|x)$ with respect to the data
generating distribution (Equation 47) is

$\begin{split}\bar{f}_{\bm{\theta}}(z=2)&=\mathbb{E}_{x\sim
p(x)}\left[f_{\bm{\theta}}(z=2|x)\right]\\\
&=\int_{0}^{1}p(x)\mathbbm{1}\left[x>\tau(\bm{\theta})\right]\ \mathrm{d}x\\\
&=\int_{\tau(\bm{\theta})}^{1}p(x)\ \mathrm{d}x\\\
&=(1-\theta_{1})I_{x}^{1}\left(\theta_{2},\theta_{3}\right)+\theta_{1}I_{x}^{1}\left(\theta_{3},\theta_{2}\right),\end{split}$
(52)

where $I_{x_{0}}^{x_{1}}(a,b)$ is the generalized regularized incomplete beta
function (BetaRegularized[$x_{0},x_{1},a,b$] command in the Wolfram language
and betainc($a,b,x_{0},x_{1}$,regularized=True) in Python’s mpmath package).
Equation 52 implies that
$\bar{f}_{\bm{\theta}}(z=1)=1-\bar{f}_{\bm{\theta}}(z=2)$. The pooled
heterogeneity is thus expressed as a function of $\bm{\theta}$ as follows:

$\Pi_{q}^{\mathrm{P}}\left(\bm{\theta}\right)=\left\\{\begin{array}[]{ll}\sum_{i=1}^{2}\mathbbm{1}[\bar{f}_{\bm{\theta}}(z=i)>0]&q=0\\\
\exp\left\\{-\sum_{i=1}^{2}\bar{f}_{\bm{\theta}}(z=i)\log\bar{f}_{\bm{\theta}}(z=i)\right\\}&q=1\\\
\left(\max_{i}\bar{f}_{\bm{\theta}}(z=i)\right)^{-1}&q=\infty\\\
\left(\sum_{i=1}^{2}\bar{f}_{\bm{\theta}}^{q}(z=i)\right)^{\frac{1}{1-q}}&\text{Otherwise}\\\
\end{array}\right..$ (53)

As a function of $\bm{\theta}$, the within-group heterogeneity is

$\begin{split}\Pi_{q}^{\mathrm{W}}\left(\bm{\theta}\right)&=\left[\int_{0}^{1}\frac{p^{q}(x)}{\int_{0}^{1}p^{q}(u)\
\mathrm{d}u}\left(\sum_{i=1}^{2}f_{\bm{\theta}}(z=i|x)\right)^{q}\
\mathrm{d}x\right]^{\frac{1}{1-q}}\\\
&=\left[\int_{0}^{1}\frac{p^{q}(x)}{\int_{0}^{1}p^{q}(u)\
\mathrm{d}u}\left(1\right)\ \mathrm{d}x\right]^{\frac{1}{1-q}}\\\
&=1,\end{split}$ (54)

and therefore the between-group heterogeneity is
$\Pi_{q}^{\mathrm{B}}\left(\bm{\theta}\right)=\Pi_{q}^{\mathrm{P}}\left(\bm{\theta}\right)$.

Analytic expressions for the existing non-categorical heterogeneity indices
$\hat{Q}_{e}$ (Equation 17), $F_{q}$ (Equation 21), and $L_{q}$ (Equation 23)
were computed as “best-case” scenarios, as follows. First, the probability
distributions over states for all expressions was the true prior distribution
(Equation 46). Distance matrices—and by extension, the similarity matrix for
$L_{q}$—were computed using the closed-form expectation of the absolute
distance between two beta-distributed random variables (see Appendix B and the
Supplementary Materials).

Figure 5 compares the categorical RRH against $\hat{Q}_{e}$, $F_{q}$, and
$L_{q}$ for BMM distributions of varying degrees of separation, and across
different mixture component weights ($0.5\leq\theta_{1}<1$). Without
significant loss of generality, we show only those comparisons at $q=1$ (which
excludes the numbers equivalent quadratic entropy), and $q=2$.

The most salient differences between these indices occur when the BMM mixture
components completely overlap (i.e., at $\theta_{2}=\theta_{3}$). The RRH
correctly identifies that there is effectively only one component, regardless
of mixture weights. Only the Leinster–Cobbold index showed invariance to the
mixture weights when $\theta_{2}=\theta_{3}$, but it could not correctly
identify that data were effectively unimodal.

The other stark difference arose when the mixture components were furthest
apart (here when $\theta_{2}=5$ and $\theta_{3}=20$). At this setting, the
functional Hill numbers showed a paradoxical increase in the heterogeneity
estimate as the prior distribution on components was skewed. The
Leinster–Cobbold index was appropriately concave throughout the range of prior
weights, but it never reached a value of 2 at its peak (as expected based on
the predictions outlined in Section 2.2.3). Conversely, the RRH was always
concave and reached a peak of 2 when both mixture components were equally
probable.

Figure 5: Comparison of categorical representational Rényi heterogeneity
($\Pi_{q}$), the functional Hill numbers ($F_{q}$), the numbers equivalent
quadratic entropy ($\hat{Q}_{e}$), and the Leinster–Cobbold index ($L_{q}$)
within the beta mixture model. Each row of plots corresponds to a given
separation between the beta mixture components. Column 1 illustrates the beta
mixture distributions upon which indices were compared. The x-axis plots the
domain of the distribution (open interval between 0 and 1). The y-axis shows
the corresponding probability density. Different line styles in Column 1
provides visual examples of the effect of changing the $\theta_{1}$ parameter
over the range [0.5,1]. Column 2 compares $\Pi_{q}$ (solid line), $F_{q}$
(dashed line), and $L_{q}$ (dotted line), each at elasticity $q=1$. The x-axis
shows the value of the $0.5\leq\theta_{1}<1$ parameter at which the indices
were compared. Index values are plotted along the y-axis. Column 3 compares
the indices shown in Column 2, as well as $\hat{Q}_{e}$ (dot-dashed line).

###  4.2 Representational Rényi Heterogeneity is Scalable to Deep Learning
Models

In this example, the observable system $X$ is that of images of handwritten
digits defined on an event space $\mathcal{X}=[0,1]^{784}$ of dimension
$n_{x}=784$ (the black and white images are flattened from $28\times 28$ pixel
matrices into 784-dimensional vectors). Our sample
$\mathbf{X}=\left(x_{ij}\right)_{i=1,2,\ldots,N}^{j=1,2,\ldots,784}$ from this
space is the familiar MNIST training dataset (LeCun et al., 1998) (Figure 6),
which consists of $N=60,000$ images roughly evenly distributed across digits
$\\{0,1,\ldots,9\\}$, and where approximately 10% of all images come from each
class. We assume each image carries equal importance, given by a weight vector
$\mathbf{w}=\left(N^{-1}\right)_{i=1,2,\ldots,N}$. We are interested in
measuring the heterogeneity of $X$ with respect to a continuous latent
representation $Z$ defined on event space $\mathcal{Z}=\mathbb{R}^{2}$. In the
present example, this space is simply the continuous 2-dimensional compression
of an image that best facilitates its reconstruction. We choose a
dimensionality of $n_{z}=2$ for the latent space in order to facilitate a
pedagogically useful visualization of the latent feature representation,
below. Unlike Section 4.1, in the present case we have no explicit
representation of the true marginal distribution over the data,
$p(\mathbf{x})$.

Figure 6: Sample images from the MNIST dataset (LeCun et al., 1998).

Having defined the observable and latent spaces, measuring RRH now requires
defining a model $f:\mathcal{X}\to\mathcal{P}(\mathcal{Z})$ that maps a
(flattened) image vector $\mathbf{x}_{i}\in\mathcal{X}$ onto a probability
distribution over the latent space. Our chosen model is the encoder module of
a pre-trained convolutional variational autoencoder (cVAE) provided by the
Smart Geometry Processing Group at University College London (Figure 7)
(Kingma and Welling, 2014, 2019):

(a) Schematic of the model architecture.

(b) Visualization of the two-dimensional latent space.

Figure 7: Panel A: Illustration of the convolutional variational autoencoder
(cVAE) (Kingma and Welling, 2014). The computational graph is depicted from
top to bottom. An $n_{x}$-dimensional input data $\mathbf{X}_{i}$ (white
rectangle) is passed through an encoder (in our experiment this is a
convolutional neural network, CNN) which parameterizes an $n_{z}$-dimensional
multivariate Gaussian over the coordinates $\mathbf{z}_{i}$ for the image’s
embedding on the latent space $\mathcal{Z}=\mathbb{R}^{n_{z}}$. The latent
embedding can then be passed through a decoder (blue rectangle) which is a
neural network employing transposed convolutions (here denoted
$\mathrm{CNN}^{\top}$) to yield a reconstruction $\hat{\mathbf{X}}_{i}$ of the
original input data. The objective function for this network is a variational
lower bound on the model evidence of the input data (see Kingma and Welling
(2014) for details). Panel B: Depiction of the latent space learned by the
cVAE. This model was a pre-trained model from the Smart Geometry Processing
Group at University College London.

$f_{\bm{\phi}}(\mathbf{z}|\mathbf{x}_{i})=\mathcal{N}\left(\mathbf{z}|\mathbf{m}(\mathbf{x}_{i}),\mathbf{C}(\mathbf{x}_{i})\right)$
(55)

where $\bm{\phi}$ are the encoder’s parameters, which specify a convolutional
neural network (CNN) whose output layer returns a $2\times 1$ mean vector
$\mathbf{m}(\mathbf{x}_{i})$ and a $2\times 1$ log-variance vector
$\mathbf{s}(\mathbf{x}_{i})$ given $\mathbf{x}_{i}$. For simplicity, we denote
the latter as the $2\times 2$ diagonal covariance matrix
$\mathbf{C}(\mathbf{x}_{i})=\left(e^{s_{j}(\mathbf{x}_{i})}\delta_{jk}\right)_{j=1,2}^{k=1,2}$.
Further details of the cVAE and its training can be found in Kingma and
Welling (Kingma and Welling, 2014, 2019), although the specific implementation
in this paper was a pre-trained implementation by the Smart Geometry
Processing Group at University College London. Briefly, the cVAE learns to
generate a compressed latent representation (via encoder $f_{\bm{\phi}}$,
which is an approximate posterior distribution) that contains enough
information about the input $\mathbf{x}_{i}$ to facilitate its reconstruction
by a “decoder” module. The objective function is a lower bound on the model
evidence $p(\mathbf{x})$, which if maximized is equivalent to minimizing the
Kullback–Leibler divergence between the approximate and true (but unknown)
posteriors $f_{\bm{\phi}}$ and $p(\mathbf{z}|\mathbf{x})$, respectively.

The continuous RRH under the model in Equation 55 for a single example
$\mathbf{x}_{i}\in\mathcal{X}$ can be computed by merely evaluating the Rényi
heterogeneity of a multivariate Gaussian (Equation 43 in Example 3) for the
covariance matrix given by $\mathbf{C}(\mathbf{x}_{i})$. This is interpreted
as the effective area of the 2-dimensional latent space consumed by
representation of $\mathbf{x}_{i}$.

Since the handwritten digit images belong to groups of “Zeros, Ones, Twos, …,
Nines,” this section will call the quantity $\Pi_{q}^{\mathrm{W}}$ the within-
observation heterogeneity (rather than the “within-group” heterogeneity) in
order to avoid its interpretation as measuring the heterogeneity of a group of
digits. Rather, it is interpreted as the effective area of latent space
consumed by representation of a single observation $\mathbf{x}\in\mathcal{X}$
on average. It is computed by evaluation of Equation 44 at
$\mathbf{C}(\mathbf{X})=\left\\{\mathbf{C}(\mathbf{x}_{i})\right\\}_{i=1,2,\ldots,N}$,
given uniform weights on samples.

Finally, to compute the pooled heterogeneity $\Pi_{q}^{\mathrm{P}}$, we use
the parametric pooling approach detailed in Example 3, wherein the pooled
distribution is a multivariate Gaussian with mean and covariance given by
Equations 41 and 42, respectively. The pooled heterogeneity is then merely
Equation 43 evaluated at $\mathbf{C}_{\ast}(\mathbf{X})$, and represents the
total amount of area in the latent space consumed by the representation of $X$
under $f_{\bm{\phi}}$. The effective number of observations in $X$ with
respect to the continuous latent representation $Z$ is, therefore, given by
the between-observation heterogeneity:

$\Pi_{q}^{\mathrm{B}}\left(\mathbf{C}(\mathbf{X}),\mathbf{w}\right)=\frac{\Pi_{q}^{\mathrm{P}}\left(\mathbf{C}_{\ast}(\mathbf{X})\right)}{\Pi_{q}^{\mathrm{W}}\left(\mathbf{C}(\mathbf{X}),\mathbf{w}\right)}.$
(56)

Equation 56 gives the effective number of observations in $X$ because it uses
the entire sample $\mathbf{X}$ (of course, assuming $\mathbf{X}$ provides
adequate coverage of the observable event space). However, one could compute
the effective number of observations in a subset of $\mathbf{X}$, if
necessary. Let
$\mathbf{X}^{(j)}=\left(\mathbf{x}_{k}\right)_{k=1,2,\ldots,N_{j}}$ be the
subset of $N_{j}$ points in $\mathbf{X}$ found in the observable subspace
$\mathcal{X}_{j}\subset\mathcal{X}$ (such as the subspace of MNIST digits
corresponding to a given digit class). Given corresponding weights
$\mathbf{w}^{(j)}=\left(N_{j}^{-1}\right)_{k=1,2,\ldots,N_{j}}$, Equation 56
is then simply

$\Pi_{q}^{\mathrm{B}}\left(\mathbf{C}(\mathbf{X}^{(j)}),\mathbf{w}^{(j)}\right)=\frac{\Pi_{q}^{\mathrm{P}}\left(\mathbf{C}_{\ast}(\mathbf{X}^{(j)})\right)}{\Pi_{q}^{\mathrm{W}}\left(\mathbf{C}(\mathbf{X}),\mathbf{w}^{(j)}\right)}.$
(57)

Figure 8 shows the effective number of observations in the subsets of MNIST
images belonging to each image class, under the continuous representation
learned by the cVAE. One can appreciate that the MNIST class of “Ones” (in the
training set) has the smallest effective number of observations. Subjective
visual inspection of the MNIST samples in Figure 6 may suggest that the Ones
are indeed relatively more homogeneous as a group than the other digits (this
claim is given further objective support in Appendix C based on deep
similarity metric learning (Bromley et al., 1994; Hadsell et al., 2006)).

Figure 8: Heterogeneity for the subset of MNIST training data belonging to
each digit class respectively projected onto the latent space of the
convolutional variational autoencoder (cVAE). The leftmost plot shows the
pooled heterogeneity for each digit class (the effective total area of latent
space occupied by encoding each digit class). The middle plot shows the
within-observation heterogeneity (the effective total area of latent space per
encoded observation of each digit class, respectively). The rightmost plot
shows the between-observation heterogeneity (the effective number of
observations per digit class). Recall that Rényi heterogeneity on a continuous
distribution gives the effective size of the domain of an equally
heterogeneous uniform distribution on the same space, which explains why the
within-observation heterogeneity values here are less than 1.

Figure 9 demonstrates the correspondence of between-observation heterogeneity
(i.e., the effective number of observations) and the visual diversity of
different samples from the latent space of our cVAE model. For each image in
the MNIST training dataset, we computed the effective location of its latent
representation: $\mathbf{m}(\mathbf{x}_{i})$ for $i\in\\{1,2,\ldots,N\\}$. For
each of these image representations, we defined a “neighborhood” including the
49 other images whose latent coordinates were closest in Euclidean distance
(which is sensible on the latent space given the Gaussian prior). For all such
neighbourhoods defined, we then reconstructed the corresponding images on
$\mathcal{X}$, whose between-observation heterogeneity was then computed using
Equation 57. Figure 9(b) shows the estimated effective number of observations
for the latent neighborhoods with the greatest and least heterogeneity. One
can appreciate that neighborhoods with $\Pi_{q}^{\mathrm{B}}$ close to 1
include images with considerably less diversity than neighborhoods with
$\Pi_{q}^{\mathrm{B}}$ closer to the upper limit of 49. These data suggest
that the between-observation heterogeneity—which is the effective number of
observations in $X$ with respect to the latent features learned by a cVAE—can
indeed correspond to visually appreciable sample diversity.

(a) Illustration of analysis.

(b) Heterogeneity of patches in the latent space.

Figure 9: Visual illustration of MNIST image samples corresponding to
different levels of representational Rényi heterogeneity under the
convolutional variational autoencoder (cVAE). Panel (a) illustrates the
approach to this analysis. Here, the surface $\mathcal{Z}$ shows hypothetical
contours of a probability distribution over the 2-dimensional latent feature
space. The surface $\mathcal{X}$ represents the observable space, upon which
we have projected an “image” of the latent space $\mathcal{Z}$ for
illustrative purposes. We first compute the expected latent locations
$\mathbf{m}(\mathbf{x}_{i})$ for each image $\mathbf{x}_{i}\in\mathcal{X}$.
(A1) We then define the latent neighbourhood of image $\mathbf{x}_{i}$ as the
49 images whose latent locations are closest to $\mathbf{m}(\mathbf{x}_{i})$
in Euclidean distance. (A2) Each coordinate in the neighbourhood of
$\mathbf{m}(\mathbf{x}_{i})$ is then projected onto a corresponding patch on
the observable space of images. (A3) These images are then projected as a
group back onto the latent space, where Equation 57 can be applied, given
equal weights over images, to compute the effective number of observations in
the neighbourhood of $\mathbf{x}_{i}$. Panel (b) plots the most and least
heterogeneous neighbourhoods so that we may compare the estimated effective
number of observations with the visually appreciable sample diversity.

##  5 Discussion

This paper introduced representational Rényi heterogeneity, a measurement
approach that satisfies the replication principle (Jost, 2006, 2009;
Macarthur, 1965) and is decomposable (Jost, 2007) while requiring neither a
priori (A) categorical partitioning nor (B) specification of a distance
function on the input space. Rather, the experimenter is free to define a
model that maps observable data onto a semantically relevant domain upon which
Rényi heterogeneity may be tractably computed, and where a distance function
need not be explicitly manipulated. These properties facilitate heterogeneity
measurements for several new applications. Compared to state-of-the-art
comparator indices under a beta mixture distribution, RRH more reliably
quantified the number of unique mixture components (Section 4.1), and under a
deep generative model of image data, RRH was able to measure the effective
number of distinct images with respect to latent continuous representations
(Section 4.2). In this section, we further synthesize our conclusions, discuss
their implications, and highlight open questions for future research.

The main problem we set out to address was that all state of the art numbers
equivalent heterogeneity measures (Section 2.2) require a priori specification
of a distance function and categorical partitioning on the observable space.
To this end, we showed that RRH does not require categorical partitioning of
the input space (Section 3). Although our analysis under the two-component BMM
assumed that the number of components was known, RRH was the only index able
to accurately identify an effectively singular cluster (i.e., where mixture
components overlapped; Figure 5). We also showed that the categorical RRH did
not violate the principle of transfers (Pigou, 1912; Dalton, 1920) (i.e., it
was strictly concave with respect to mixture component weights), unlike the
functional Hill numbers (Figure 5). Future studies should extend this
evaluation to mixtures of other distributional forms in order to better
characterize the generalizability of our conclusions.

Sections 3.1 and 3.2 both showed that RRH does not require specification of a
distance function on the observable space. Instead, one must specify a model
that maps the observable space onto a probability distribution over the latent
representation. This is beneficial since input space distances are often
irrelevant or misleading. For example, latent representations of image data
learned by a convolutional neural network will be robust to translations of
the inputs since convolution is translation invariant. However, pairwise
distances on the observable space will be exquisitely sensitive to
semantically irrelevant translations of input data. Furthermore, semantically
relevant information must often be learned from raw data using hierarchical
abstraction. Ultimately, when (A) pre-defined distance metrics are sensitive
to noisy perturbations of the input space, or (B) the relevant semantic
content of some input data is best captured by a latent abstraction, the RRH
measure will be particularly useful.

The requirement of specifying a representational model
$f:\mathcal{X}\to\mathcal{P}(\mathcal{Z})$ implies the additional problem of
model selection. In Section 3, we noted that the determination of whether a
model is appropriate must be made in a domain-specific fashion. For instance,
the method by which ecologists assign species labels prior to measurement of
species diversity implies the use of a mapping from the observable space of
organisms to a degenerate distribution over species labels (Example 3.1). In
Section 4.2, we used the encoder module of a cVAE (a generative model based on
a convolutional neural network architecture (Kingma and Welling, 2014, 2019))
to represent images as 2-dimensional real-valued vectors in order to
demonstrate our ability to capture variation in digits’ written forms (see
Figures 7B and 9). Someone concerned with measuring heterogeneity of image
batches in terms of the digit-class distribution could choose a categorical
latent representation corresponding to the digit classes (this would return
the effective number of digit classes per sample). Regardless, the model used
to map between observations and the latent space should be validated using
either explanatory power (e.g., maximization of a lower bound on the model
evidence), generalizability (e.g., out of sample predictive power), or another
approach that is justifiable within the investigator’s scientific domain of
interest.

In addition to the results of empirical applications of RRH in Section 4, we
were also able to show that RRH generalizes the process by which species
diversity and indices of economic equality are computed (Example 3.1). In
doing so, we are able to clarify some of the assumptions inherent in those
indices. Specifically, that assignment of species or ownership labels (in
ecological and economic settings, respectively) corresponds to mapping from an
observable space, such as the space of organisms’ identifiable features or the
space of economic resources, onto a degenerate distribution over the
categorical labels (Table 2). It is possible that altering the form of that
mapping may yield new insights about ecological and economic diversity.

In conclusion, we have introduced an approach for measuring heterogeneity that
requires neither (A) categorical partitioning nor (B) distance measure on the
observable space. Our RRH method enables measurement of heterogeneity in
disciplines where categorical entities are unreliably defined, or where
relevant semantic content of some data is best captured by a hierarchical
abstraction. Furthermore, our approach includes many existing heterogeneity
indices as special cases, while facilitating clarification of many of their
assumptions. Future work should evaluate the RRH in practice and under a
broader array of models.

Supplementary materials include code for Sections 2–4 and Appendix B
(RRH$\\_$Supplement$\\_$3State$\\_$BMM$\\_$CVAE.ipynb), and Appendix C
(RRH$\\_$Supplement$\\_$Siamese.ipynb).

conceptualization, A.N.; methodology, A.N.; validation, A.N.; formal analysis,
A.N.; investigation, A.N.; resources, T.T.; data curation, A.N.;
writing—original draft preparation, A.N.; writing—review and editing, A.N.,
M.A., T.B., T.T.; visualization, A.N.; supervision, M.A., T.B., T.T.; project
administration, A.N.; funding acquisition, A.N., M.A.

This research was funded by Genome Canada (A.N., M.A.), the Nova Scotia Health
Research Foundation (A.N.), the Killam Trusts (A.N.), and the Ruth Wagner
Memorial Fund (A.N.).

The authors declare no conflict of interest. The funders had no role in the
design of the study; in the collection, analyses, or interpretation of data;
in the writing of the manuscript, or in the decision to publish the results.

yes

##  Appendix A Mathematical Appendix

Rényi heterogeneity (Equation 3) obeys the replication principle.

###### Proof.

The Rényi heterogeneity for a single distribution
$\mathbf{p}_{i}=(p_{ij})_{j=1,2,\ldots,n_{i}}$, where $n_{i}\in\mathbb{N}_{+}$
is the size of the state space in system $i$, is

$\Pi_{q}(\mathbf{p}_{i})=\left(\sum_{j=1}^{n_{i}}p_{ij}^{q}\right)^{\frac{1}{1-q}}$
(58)

and for the aggregation of $N$ subsystems is

$\Pi_{q}(\bar{\mathbf{p}}_{i})=\left(\sum_{i=1}^{N}\sum_{j=1}^{n_{i}}\left(\frac{p_{ij}}{N}\right)^{q}\right)^{\frac{1}{1-q}}.$
(59)

The replication principle asserts that

$\Pi_{q}(\bar{\mathbf{p}})=N\Pi_{q}(\mathbf{p}_{i}).$ (60)

Let $\lambda_{i}=\sum_{j=1}^{n_{i}}p_{ij}^{q}$ and recall that
$\lambda_{i}=\lambda_{k}$ for all $(i,k)\in\\{1,2,\ldots,N\\}$. Then,

$\begin{split}\left(N^{-q}\sum_{i=1}^{N}\sum_{j=1}^{n_{i}}p_{ij}^{q}\right)^{\frac{1}{1-q}}&=N\left(\sum_{j=1}^{n_{i}}p_{ij}^{q}\right)^{\frac{1}{1-q}}\\\
\left(N^{-q}\sum_{i=1}^{N}\lambda_{i}\right)^{\frac{1}{1-q}}&=N\lambda_{i}^{\frac{1}{1-q}}\\\
\left(N^{1-q}\lambda_{i}\right)^{\frac{1}{1-q}}&=N\lambda_{i}^{\frac{1}{1-q}}\\\
N\lambda_{i}^{\frac{1}{1-q}}&=N\lambda_{i}^{\frac{1}{1-q}}.\\\ \end{split}$
(61)

Since $\lim_{q\to 1}\lambda_{i}^{\frac{1}{1-q}}$ exists (it is the perplexity
index), the result also holds at $q=1$. ∎

For a system $X$ with probability mass function represented by the vector
$\mathbf{p}=\left(p_{i}\right)_{i=1,2,\ldots,n}$ on event space
$\mathcal{X}=\\{1,2,\ldots,n\\}$, with distance function
$d_{X}:\mathcal{X}\times\mathcal{X}\to\mathbb{R}_{\geq 0}$ represented by the
$n\times n$ matrix
$\mathbf{D}=\left[d_{X}(i,j)\right]_{i=1,2,\ldots,n}^{j=1,2,\ldots,n}$, the
functional Hill numbers family of indices

$F_{q}\left(\mathbf{D},\mathbf{p}\right)=\left(\frac{Q_{q}\left(\mathbf{D},\mathbf{p}\right)}{Q_{1}\left(\mathbf{D},\mathbf{p}\right)}\right)^{\frac{1}{2(1-q)}}$
(62)

is insensitive to $d_{X}(i,j)$ for all $(i,j)\in\mathcal{X}$ when $\mathbf{p}$
is uniform.

###### Proof.

The proof is direct given substitution of
$\mathbf{p}=\left(n^{-1}\right)_{i=1,2,\ldots,n}$ into Equation 62.

$F_{q}\left(\mathbf{D},\mathbf{p}\right)=\left(\frac{Q_{q}\left(\mathbf{D},\mathbf{p}\right)}{Q_{1}\left(\mathbf{D},\mathbf{p}\right)}\right)^{\frac{1}{2(1-q)}}=\left(\frac{n^{-2q}\sum_{i=1}^{n}\sum_{j=1}^{n}d_{X}(i,j)}{n^{-2}\sum_{i=1}^{n}\sum_{j=1}^{n}d_{X}(i,j)}\right)^{\frac{1}{2(1-q)}}=n$
(63)

∎

[Rényi Heterogeneity of a Continuous System] The Rényi heterogeneity of a
system $X$ with event space $\mathcal{X}\subseteq\mathbb{R}^{n}$ and pdf
$f\in\mathcal{P}(\mathcal{X})$ is equal to the magnitude of the volume of an
$n$-cube over which there is a uniform probability density with the same Rényi
heterogeneity as that given by $f$.

###### Proof.

Let the basic integral of $X$ be defined as
$\int_{\mathcal{X}}f^{q}(\mathbf{x})\ \mathrm{d}\mathbf{x}$. Furthermore, let
$X_{\ast}$ be an idealized reference system with a uniform probability density
$f_{\ast}$ on $\mathcal{X}$ with lower bounds
$\mathbf{0}=\left(0\right)_{i=1,\ldots,n}$ and upper bounds
$\mathbf{u}=\left(u_{\ast}\right)_{i=1,\ldots,n}$ where $u_{\ast}\geq 0$ is
the side length of an $n$-cube. We assume that $X_{\ast}$ has basic integral
$\int_{\mathcal{X}}f_{\ast}^{q}(\mathbf{x})\ \mathrm{d}\mathbf{x}$ such that

$\begin{split}\int_{\mathcal{X}}f^{q}(\mathbf{x})\
\mathrm{d}\mathbf{x}&=\int_{\mathcal{X}}f_{\ast}^{q}(\mathbf{x})\
\mathrm{d}\mathbf{x}\\\ &=\prod_{i=1}^{n}u_{\ast}^{1-q}\\\
&=u_{\ast}^{n(1-q)}.\end{split}$ (64)

Solving Equation 64 for $u_{\ast}^{n}$ gives the Rényi heterogeneity of order
$q$. At $q\neq 1$,

$u_{\ast}^{n}=\left(\int_{\mathcal{X}}f^{q}(\mathbf{x})\
\mathrm{d}\mathbf{x}\right)^{\frac{1}{1-q}}$ (65)

and in the limit of $q\to 1$, Equation 65 becomes the exponential of the
Shannon (differential) entropy. Thus, $\Pi_{q}$ is interpreted as the volume
of an $n$-cube of side length $u_{\ast}$, over which there is a uniform
distribution giving the same heterogeneity as $X$. ∎

[Rényi heterogeneity of a multivariate Gaussian] The Rényi heterogeneity of an
$n$-dimensional multivariate Gaussian with probability density function (pdf)

$f(\mathbf{x}|\bm{\mu},\bm{\Sigma})=\left(2\pi\right)^{-\frac{n}{2}}\left|\bm{\Sigma}\right|^{-\frac{1}{2}}e^{-\frac{1}{2}\left(\mathbf{x}-\bm{\mu}\right)^{\top}\bm{\Sigma}^{-1}\left(\mathbf{x}-\bm{\mu}\right)},$
(66)

with mean $\bm{\mu}=\left(\mu_{i}\right)_{i=1,2,\ldots,n}$ and covariance
matrix
$\bm{\Sigma}=\left(\Sigma_{ij}\right)_{i=1,2,\ldots,n}^{j=1,2,\ldots,n}$ is

$\Pi_{q}\left(\bm{\Sigma}\right)=\left\\{\begin{array}[]{ll}\mathrm{Undefined}&q=0\\\
\left(2\pi e\right)^{\frac{n}{2}}\sqrt{\left|\bm{\Sigma}\right|}&q=1\\\
\left(2\pi\right)^{\frac{n}{2}}\sqrt{\left|\bm{\Sigma}\right|}&q=\infty\\\
\left(2\pi\right)^{\frac{n}{2}}q^{\frac{n}{2(q-1)}}\sqrt{\left|\bm{\Sigma}\right|}&\mathrm{Otherwise}\end{array}\right..$
(67)

###### Proof.

Let $\bm{\Sigma}^{-1}=\mathbf{U}\bm{\Lambda}\mathbf{U}^{-1}$ be the
eigendecomposition of the inverse covariance matrix into an orthonormal matrix
of eigenvectors $\mathbf{U}$ and $n\times n$ diagonal matrix $\bm{\Lambda}$
with eigenvalues $\left(\lambda_{i}\right)_{i=1,2,\ldots,n}$ down the leading
diagonal. Furthermore, let $\frac{\ \mathrm{d}x_{i}}{\
\mathrm{d}y_{j}}=U_{ij}$ and use the substitution
$\mathbf{y}=\mathbf{U}^{-1}\left(\mathbf{x}-\bm{\mu}\right)$ to proceed as
follows:

$\begin{split}\Pi_{q}\left(\bm{\Sigma}\right)&=\left[\left(2\pi\right)^{-\frac{qn}{2}}\left|\bm{\Sigma}\right|^{-\frac{q}{2}}\int
e^{-\frac{q}{2}\left(\mathbf{x}-\bm{\mu}\right)^{\top}\bm{\Sigma}^{-1}\left(\mathbf{x}-\bm{\mu}\right)}\
\mathrm{d}\mathbf{x}\right]^{\frac{1}{1-q}}\\\
&=\left(\left(2\pi\right)^{-\frac{qn}{2}}\left|\bm{\Sigma}\right|^{-\frac{q}{2}}\int
e^{-\frac{q}{2}\mathbf{y}^{\top}\bm{\Lambda}\mathbf{y}}\
\mathrm{d}\mathbf{y}\right)^{\frac{1}{1-q}}\\\
&=\left(\left(2\pi\right)^{-\frac{qn}{2}}\left|\bm{\Sigma}\right|^{-\frac{q}{2}}\left(\frac{(2\pi)^{n}}{q^{n}\prod_{i=1}^{n}\lambda_{i}}\right)^{\frac{1}{2}}\right)^{\frac{1}{1-q}}\\\
&=\left(\left(2\pi\right)^{-\frac{qn}{2}}\left|\bm{\Sigma}\right|^{-\frac{q}{2}}\left(\frac{(2\pi)^{n}}{q^{n}\left|\bm{\Lambda}\right|}\right)^{\frac{1}{2}}\right)^{\frac{1}{1-q}}\\\
&=q^{\frac{n}{2(q-1)}}(2\pi)^{\frac{n}{2}}\sqrt{\left|\bm{\Sigma}\right|}\\\
\end{split}$ (68)

which holds only at $q\notin\\{0,1,\infty\\}$. At $q=1$, we have

$\begin{split}\lim_{q\to 1}\log\Pi_{q}\left(\bm{\Sigma}\right)&=\lim_{q\to
1}\left(\frac{n}{2(q-1)}\log
q\right)+\frac{n}{2}\log(2\pi)+\frac{1}{2}\log\left|\bm{\Sigma}\right|\\\
&=\frac{n}{2}+\frac{n}{2}\log(2\pi)+\frac{1}{2}\log\left|\bm{\Sigma}\right|,\\\
\end{split}$ (69)

and therefore,

$\Pi_{1}\left(\bm{\Sigma}\right)=\left(2\pi
e\right)^{\frac{n}{2}}\sqrt{|\bm{\Sigma}|}.$ (70)

One can then easily show that $\Pi_{0}(\bm{\Sigma})$ is undefined and that as
$q\to\infty$,

$\Pi_{\infty}\left(\bm{\Sigma}\right)=\left(2\pi\right)^{\frac{n}{2}}\sqrt{|\bm{\Sigma}|}.$
(71)

∎

##  Appendix B Expected Distance Between two Beta-Distributed Random Variables

To compute the numbers equivalent RQE $\hat{Q}_{e}$, the functional Hill
numbers $F_{q}$, and the Leinster-Cobbold index $L_{q}$ under the beta mixture
model, we must derive an analytical expression for the distance matrix. This
involves the following integral:

$d(x,y)=\int_{0}^{1}\int_{0}^{1}|x-y|f(x)g(y)\ \mathrm{d}x\ \mathrm{d}y,$ (72)

where $f(x)=\mathrm{Beta}_{\alpha_{1},\beta_{1}}(x)$ and
$g(y)=\mathrm{Beta}_{\alpha_{2},\beta_{2}}(y)$. By exploiting the identity

$|x-y|=x+y-2\min\\{x,y\\},$ (73)

and expanding, the integral is greatly simplified and gives the following
closed-form solution:

$d(x,y)=\langle x\rangle-\langle
y\rangle+\eta\left(\Phi_{a}-\alpha_{1}\Phi_{b}\right),$ (74)

where

$\eta=\frac{2\Gamma(\alpha_{1})\Gamma(\beta_{2})\Gamma(\alpha_{1}+\alpha_{2}+1)}{B(\alpha_{1},\beta_{1})B(\alpha_{2},\beta_{2})},$
(75)

and where $\langle y\rangle=\frac{\alpha_{2}}{\alpha_{2}+\beta_{2}}$, $\langle
x\rangle=\frac{\alpha_{1}}{\alpha_{1}+\beta_{1}}$, and the $\Phi$’s are
regularized hypergeometric functions:

$\Phi_{a}=\prescript{}{3}{\tilde{F}_{2}}\left[\begin{array}[]{c}\alpha_{1},\alpha_{1}+\alpha_{2}+1,1-\beta_{1}\\\
\alpha_{1}+1,\alpha_{1}+\alpha_{2}+\beta_{2}+1\end{array},1\right]$ (76)

$\Phi_{b}=\prescript{}{3}{\tilde{F}_{2}}\left[\begin{array}[]{c}\alpha_{1}+1,\alpha_{1}+\alpha_{2}+1,1-\beta_{1}\\\
\alpha_{1}+2,\alpha_{1}+\alpha_{2}+\beta_{2}+1\end{array},1\right]$ (77)

Figure 10: Numerical verification of the analytical expression for the
expected absolute distance between two Beta-distributed random variables.
Solid lines are the theoretical predictions. Ribbons show the bounds between
25th-75th percentiles (the interquartile range, IQR) of the simulated values.

Figure 10 provides numerical verification of this result. One simply uses
Equation 74 to compute the analytic distance matrix

$\mathbf{D}(\alpha_{1},\beta_{1},\alpha_{2},\beta_{2})=\left(\begin{array}[]{cc}d(x,x)&d(x,y)\\\
d(y,x)&d(y,y)\end{array}\right),$ (78)

which, with the component probabilities (Equation 46), can be used to compute
$\hat{Q}_{e},F_{q}$, and $L_{q}$ using the formulas shown in the main body.

##  Appendix C Evidence Supporting Relative Homogeneity of MNIST “Ones”

In our evaluation of non-categorical RRH using the MNIST data, we asserted
that the class of handwritten Ones were relatively more homogeneous than other
digits. Our initial statement was based simply on visual inspection of samples
from the dataset, wherein the Ones ostensibly demonstrate fewer relevant
feature variations than other classes. However, to test this hypothesis more
objectively, we conducted an empirical evaluation using similarity metric
learning.

We implemented a deep neural network architecture known as a “siamese network”
(Bromley et al., 1994) to learn a latent distance metric on the MNIST classes.
Our siamese network architecture is depicted in Figure 11(a). Training is
conducted by sampling batches of 10,000 image pairs from the MNIST test set,
where 5000 pairs are drawn from the same class (i.e., a pair of Fives or a
pair of Threes), and 5000 pairs are drawn from different classes (i.e., the
pairs [2,3] or [1,7]). The siamese network is then optimized using gradient-
based methods over 100 epochs using the contrastive loss function (Hadsell et
al., 2006) (Figure 11(a)). This analysis may be reproduced in the
Supplementary Materials.

After training, we sampled same-class pairs (n=25,000) and different-class
pairs (n = 25,000) from the MNIST training set (which contains 60,000 images).
Pairwise distances for each sample were computed using the trained siamese
network. If the “ones” are indeed the most homogeneous class, they should
demonstrate a generally smaller pairwise distance than other digit class
pairs. We evaluated this hypothesis by comparing empirical cumulative
distribution functions (CDF) on the class-pair distances (Figure 11(b)). Our
results show that the empirical CDF for “1–1” image pairs dominate that of all
other class pairs (where the distance between pairs of “ones” is lower).

(a) Depiction of a siamese network architecture. At iteration $k$, each of two
samples, $X_{A}^{(k)}$ and $X_{B}^{(k)}$, are passed through a convolutional
neural network to yield embeddings $z_{A}$ and $z_{B}$, respectively. The
class label for samples A and B are denoted $y_{A}$ and $y_{B}$, respectively.
The L2-norm of these embeddings is computed as $D_{AB}$. The network is
optimized on the contrastive loss (Hadsell et al., 2006) $\mathcal{L}$. Here
$\mathbb{I}[\cdot]$ is an indicator function.

(b) Empirical cumulative distribution functions (CDF) for pairwise distances
between images of the listed classes under the siamese network model. The
x-axis plots the L2-norm between embedding vectors produced by the siamese
network. The y-axis shows the proportion of samples in the respective group
(by line color) whose embedded L2 norms were less than the specified threshold
on the x-axis. Class groups are denoted by different line colors. For
instance, “0-0” refers to pairs where each image is a “zero.” We combine all
disjoint class pairs, for example “0–8” or “3–4,” into a single empirical CDF
denoted as “A$\neq$B.”

Figure 11: Depiction of the siamese network architecture and the empirical
cumulative distribution function for pairwise distances between digit classes.

References

## References

  * Jost (2006) Jost, L.  Entropy and diversity.  Oikos 2006, 113, 363–375.
  * Prehn-Kristensen et al. (2018) Prehn-Kristensen, A.; Zimmermann, A.; Tittmann, L.; Lieb, W.; Schreiber, S.; Baving, L.; Fischer, A.  Reduced microbiome alpha diversity in young patients with ADHD.  PLoS ONE 2018, 13, e0200728.
  * Cowell (2011) Cowell, F.  Measuring Inequality, 2nd ed.; Oxford University Press: Oxford, UK, 2011.
  * Higgins et al. (2003) Higgins, J.P.T.; Thompson, S.G.; Deeks, J.J.; Altman, D.G.  Measuring inconsistency in meta-analyses.  BMJ Br. Med. J. 2003, 327, 557–560.
  * Hooper et al. (2005) Hooper, D.U.; Chapin, F.S.; Ewel, J.J.; Hector, A.; Inchausti, P.; Lavorel, S.; Lawton, J.H.; Lodge, D.M.; Loreau, M.; Naeem, S.; et al.  Effects of biodiversity on ecosystem functioning: A consensus of current knowledge.  Ecol. Monogr. 2005, 75, 3–35.
  * Botta-Dukát (2018) Botta-Dukát, Z.  The generalized replication principle and the partitioning of functional diversity into independent alpha and beta components.  Ecography 2018, 41, 40–50.
  * Mouchet et al. (2010) Mouchet, M.A.; Villéger, S.; Mason, N.W.; Mouillot, D.  Functional diversity measures: An overview of their redundancy and their ability to discriminate community assembly rules.  Funct. Ecol. 2010, 24, 867–876.
  * Chiu and Chao (2014) Chiu, C.H.; Chao, A.  Distance-based functional diversity measures and their decomposition: A framework based on hill numbers.  PLoS ONE 2014, 9, e113561.
  * Petchey and Gaston (2002) Petchey, O.L.; Gaston, K.J.  Functional diversity (FD), species richness and community composition.  Ecol. Lett. 2002.
  * Leinster and Cobbold (2012) Leinster, T.; Cobbold, C.A.  Measuring diversity: The importance of species similarity.  Ecology 2012, 93, 477–489.
  * Chao et al. (2014) Chao, A.; Chiu, C.H.; Jost, L.  Unifying Species Diversity, Phylogenetic Diversity, Functional Diversity, and Related Similarity and Differentiation Measures Through Hill Numbers.  Annu. Rev. Ecol. Evol. Syst. 2014, 45, 297–324.
  * American Psychiatric Association (2015) American Psychiatric Association.  Diagnostic and Statistical Manual of Mental Disorders (5th Ed.); American Psychiatric Publishing: Washington, DC, USA 2013
  * Regier et al. (2013) Regier, D.A.; Narrow, W.E.; Clarke, D.E.; Kraemer, H.C.; Kuramoto, S.J.; Kuhl, E.A.; Kupfer, D.J.  DSM-5 field trials in the United States and Canada, part II: Test-retest reliability of selected categorical diagnoses.  Am. J. Psychiatr. 2013, 170, 59–70.
  * Bengio et al. (2013) Bengio, Y.; Courville, A.; Vincent, P.  Representation learning: A review and new perspectives.  IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1798–1828.
  * Arvanitidis et al. (2018) Arvanitidis, G.; Hansen, L.K.; Hauberg, S.  Latent Space Oddity: On the Curvature of Deep Generative Models.  In Proceedings of International Conference on Learning Representations, Vancouver, BC, Canada, 30 April–3 May 2018; pp. 1–15, [1710.11379].
  * Shao et al. (2018) Shao, H.; Kumar, A.; Thomas Fletcher, P.  The Riemannian geometry of deep generative models.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–22 June 2018.
  * Nickel and Kiela (2017) Nickel, M.; Kiela, D.  Poincaré embeddings for learning hierarchical representations.  In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; Vol. 2017–Decem, pp. 6339–6348.
  * Rényi (1961) Rényi, A.  On measures of information and entropy.  Proc. Fourth Berkeley Symp. Math. Stat. Probab. 1961, 114, 547–561.
  * Hill (1973) Hill, M.O.  Diversity and Evenness: A Unifying Notation and Its Consequences.  Ecology 1973, 54, 427–432.
  * Hannah and Kay (1977) Hannah, L.; Kay, J.A.  Concentration in Modern Industry: Theory, measurement and the U.K. experience; The MacMillan Press, Ltd.: London, UK, 1977.
  * Ricotta and Szeidl (2009) Ricotta, C.; Szeidl, L.  Diversity partitioning of Rao’s quadratic entropy.  Theor. Popul. Biol. 2009, 76, 299–302.
  * LeCun et al. (1998) LeCun, Y.; Bottou, L.; Bengio, Y.; Haffner, P.  Gradient-based learning applied to document recognition.  Proc. IEEE 1998, 86, 2278–2324.
  * Kingma and Welling (2014) Kingma, D.P.; Welling, M.  Auto-Encoding Variational Bayes.  ICLR 2014 2014, [arXiv:1312.6114v10].
  * Kingma and Welling (2019) Kingma, D.P.; Welling, M.  An Introduction to Variational Autoencoders.  Found. Trend. Mach. Learn. 2019, 12, 307–392.
  * Eliazar and Sokolov (2012) Eliazar, I.I.; Sokolov, I.M.  Measuring statistical evenness: A panoramic overview.  Phys. A Stat. Mech. Its Appl. 2012, 391, 1323–1353.
  * Patil and Taillie (1982) Patil, A.G.P.; Taillie, C.  Diversity as a Concept and its Measurement.  J. Am. Stat. Assoc. 1982, 77, 548–561.
  * Adelman (1969) Adelman, M.A.  Comment on the "H" Concentration Measure as a Numbers-Equivalent.  Rev. Econ. Stat. 1969, 51, 99–101.
  * Jost (2007) Jost, L.  Partitioning Diversity into Independent Alpha and Beta Components.  Ecology 2007, 88, 2427–2439.
  * Shannon (1948) Shannon, C.E.  A mathematical theory of communication.  Bell Syst. Tech. J. 1948, 27, 379–423.
  * Eliazar (2015) Eliazar, I.  How random is a random vector?  Ann. Phys. 2015, 363, 164–184.  doi:black10.1016/j.aop.2015.09.023.
  * Gotelli and Chao (2013) Gotelli, N.J.; Chao, A.  Measuring and Estimating Species Richness, Species Diversity, and Biotic Similarity from Sampling Data. In Encyclopedia of Biodiversity, 2nd ed.; Levin, S.A. Ed.; Academic Press: Waltham, MA, USA 2013; pp. 195–211.
  * Berger and Parker (1970) Berger, W.H.; Parker, F.L.  Diversity of planktonic foraminifera in deep-sea sediments.  Science 1970, 168, 1345–1347.  doi:black10.1126/science.168.3937.1345.
  * Daly et al. (2018) Daly, A.; Baetens, J.; De Baets, B.  Ecological Diversity: Measuring the Unmeasurable.  Mathematics 2018, 6, 119.
  * Tsallis (1988) Tsallis, C.  Possible generalization of Boltzmann-Gibbs statistics.  J. Stat. Phys, 1988, 52, 479–487.
  * Simpson (1949) Simpson, E.H.  Measurement of Diversity.  Nature 1949, 163, 688.
  * Gini (1912) Gini, C.  Variabilità e mutabilità. Contributo allo Studio delle Distribuzioni e delle Relazioni Statistiche; C. Cuppini: Bologna, Italy, 1912\.
  * Shorrocks (1980) Shorrocks, A.F.  The Class of Additively Decomposable Inequality Measures.  Econometrica 1980, 48, 613–625.
  * Jost (2009) Jost, L.  Mismeasuring biological diversity: Response to Hoffmann and Hoffmann (2008).  Ecol. Econ. 2009, 68, 925–928.
  * Pigou (1912) Pigou, A.C.  Wealth and Welfare; MacMillan and Co., Ltd: London, England, 1912\.
  * Dalton (1920) Dalton, H.  The Measurement of the Inequality of Incomes.  Econ. J. 1920, 30, 348.
  * Macarthur (1965) Macarthur, R.H.  Patterns of species diversity.  Biol. Rev. 1965, 40, 510–533.
  * Lande (1996) Lande, R.  Statistics and partitioning of species diversity and similarity among multiple communities.  Oikos 1996, 76, 5–13.
  * Rao (1982) Rao, C.R.  Diversity and dissimilarity coefficients: A unified approach.  Theor. Popul. Biol. 1982, 21, 24–43.
  * Mikolov et al. (2013) Mikolov, T.; Chen, K.; Corrado, G.; Dean, J.  Distributed representations of words and hrases and their compositionality.  In Proceedings of the NIPS 2013, Lake Tahoe, NV, USA, 5–10 December 2013; pp. 1–9, [1310.4546].
  * Pennington et al. (2014) Pennington, J.; Socher, R.; Manning, C.  Glove: Global Vectors for Word Representation.  Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, 25–29 October 2014; pp. 1532–1543, [1504.06654].  doi:black10.3115/v1/D14-1162.
  * Nunes et al. (2020) Nunes, A.; Alda, M.; Trappenberg, T.  On the Multiplicative Decomposition of Heterogeneity in Continuous Assemblages.  arXiv 2020, arXiv:2002.09734.
  * Bromley et al. (1994) Bromley, J.; Guyon, I.; LeCun, Y.; Säckinger, E.; Shah, R.  Signature verification using a “siamese” time delay neural network.  In Proceedings of the Advances in Neural Information Processing Systems 6, Denver, CO, USA, 29 November to 2 December 1993; pp. 737–744.
  * Hadsell et al. (2006) Hadsell, R.; Chopra, S.; LeCun, Y.  Dimensionality Reduction by Learning an Invariant Mapping.  In Proceedings of the CVPR 2006, New York, NY, USA, 17–22 June 2006; pp. 1735–1742.