Version:

###

Total Empiricism: Learning from Data

Orestis Loukas111E-mail<EMAIL_ADDRESS>Ho Ryun
Chung222E-mail<EMAIL_ADDRESS>

Institute for Medical Bioinformatics and Biostatistics
Philipps-Universität Marburg
Hans-Meerwein-Straße 6, 35032 Germany

###### Abstract

Statistical analysis is an important tool to distinguish systematic from
chance findings. Current statistical analyses rely on distributional
assumptions reflecting the structure of some underlying model, which if not
met lead to problems in the analysis and interpretation of the results.
Instead of trying to fix the model or “correct” the data, we here describe a
totally empirical statistical approach that does not rely on ad hoc
distributional assumptions in order to overcome many problems in contemporary
statistics. Starting from elementary combinatorics, we motivate an
information-guided formalism to quantify knowledge extracted from the given
data. Subsequently, we derive model-agnostic methods to identify patterns that
are solely evidenced by the data based on our prior knowledge. The data-
centric character of empiricism allows for its universal applicability,
particularly as sample size grows larger. In this comprehensive framework, we
re-interpret and extend model distributions, scores and statistical tests used
in different schools of statistics.

##  1 Introduction

Statistical analysis helps distinguish “true” from chance findings. Statistics
derives much of its power and usefulness from some distributional assumptions
on the data. Model-centric learning essentially consists of verifying or
slightly improving correctly suspected distributional assumptions that
accurately capture underlying mechanisms which generate data. In such
convenient scenario, we are mostly aware (even if it is not readily
acknowledged) of statistics that are sufficient to ideally describe the data,
every deviation usually attributed to sampling noise [1]. However, this
attitude towards modeling does not exclude that our model might be plagued by
both intentional and unintentional misconceptions [2] introducing tension with
the reality of data.

As datasets grow larger and more versatile, they refuse to adhere to the
distributional assumptions made by statisticians. The violation of
distributional assumptions [3], which increasingly manifests in any
statistical analysis, points towards what is often described as model
misspecification [4]. In fact, there exists a large body of literature [5, 6]
concerned about correcting the adverse effects of unmet distributional
assumptions in order to rescue the statistical analysis based on them. In this
work, we take an alternative route. Instead of “repairing” the adverse effects
of unmet distributional assumptions, we remove them altogether and focus on
the observed data itself. Given a dataset or a series of datasets on some
subjects, the question then arises regarding “good” statistics that adequately
summarize the data [7].

Abandoning distributional assumptions to answer this question paves the way
for a novel approach to statistics based solely on data, which we refer to as
Total Empiricism (TotEm). Starting from the concept of the _reality of data_
gives rise to the notion that all attributes multivariately describing the
studied subjects inevitably take a finite number of manifestations bounded by
the sample size $N$. In particular, metric attributes would only take a finite
number of manifestations, even if we believed that they descend from
(in)finite interval on the real line. Instead of thus augmenting observations
with infinitely many unobserved values to approximate the distribution of an
attribute by e.g. a normal distribution, we concentrate on the observed
manifestations. The finite and countable number of manifestations of
attributes – the so-called entities – in a dataset defines the resolution
power of our empirical analysis. Any statement beyond entities could lead to
daydreaming away from data.

Starting from the entities always allows to compute their frequencies in a
dataset, which we summarize by the empirical distribution $\mathfrak{f}$.
Therefore, the subjects in the dataset form a closed system where all
information about them lies within the empirical distribution assigning a
weight to each entity. More generally, a linear operator $\hat{X}$ assigns
$c$-numbers to entities. In this way, $\hat{X}$ extracts statistical
information about the subjects through its expectation on the data (summarized
by $\mathfrak{f}$) which represents a collective characteristic. Depending on
the application, collective characteristics can correspond to marginal
distributions of subsets of attributes, moments of metric attributes or
generalizations thereof. Statistics is generally concerned with demarcating
those characteristics that are required to describe the subjects in the
dataset from those collective characteristics that capture random noise in the
dataset. Any demarcating statement inevitably refers back to our prior
knowledge about the investigated subjects, which we encode by a distribution
$\boldsymbol{\upsilon}$.

To foster the aforementioned demarcation, we propose to consider probability
distributions with the property that the collective characteristics extracted
by some linear operators exactly match the corresponding expectations from the
empirical distribution $\mathfrak{f}$. Such TotEm construction allows for an
infinite set of probability distributions that all share with $\mathfrak{f}$
the same expectations about the chosen characteristics. We refer to this set
of probability distributions as the TotEmplex. Subsequently, we select one,
special probability distribution $\mathfrak{q}$ from the TotEm, which can be
understood as the $I$-projection [8] of the reference $\boldsymbol{\upsilon}$
onto the TotEmplex. Finally, we derive the probability density over
distributions in the TotEm, as the universal result of a large $N$ expansion
around the $I$-projection. From this probability density over the TotEm, we
formulate the $I$-score and the $I$-test, which allow for demarcating those
collective characteristics that are required to describe the subjects in a
dataset from those that – at the current data size – cannot be distinguished
from sample noise.

All in all, we provide a novel approach for the statistical analysis of data
that does not require distributional assumptions based on and generalizing the
fruitful path laid out in [9, 10]. In our data-centric treatment, we do not
need to commit to any particular model or try to be near some (potentially
unknown and unreachable) ground truth [11]. Instead of regarding continuous
probability densities as fundamental, we derive them – at most – as an
approximation to the fundamentally discrete problem of combinatorics given $N$
data points. Mainly, our study is guided by empiricism [12, 13, 14] without
excluding though, testing and extending of established models. The
universality of empiricism at larger sample sizes ensures that the derived
methodology would self-consistently report on “good” observations, i.e.
summary statistics, that are needed to enhance our prior knowledge about
investigated subjects, no matter how misinformed this prior knowledge might
be.

#### Outline of the paper.

The paper is structured as follows: First, we introduce in Section 2 an
operator formalism to deal with probability distributions and summarize the
phenomenology of data via expectations in a parameter-agnostic way.
Subsequently, we develop in Section 3 the totally empirical theory of sampling
at large sample size $N$. After variationally motivating the effective
description of the large-$N$ combinatorics through a probability density over
distributions that comply with phenomenology, we exemplify (Section 3.2) the
special role of the the $I$-projection in empirical studies. Taking advantage
of this special distribution enables to derive the universal $N$-leading
contribution to the effective description of sampling in Section 3.4. Using
the introduced probability density in Section 4 helps classify any additional
knowledge gained by the phenomenology of data from the viewpoint of prior
knowledge. Eventually, we show in Section 5 how to apply the proposed methods
to uncover trends in the phenomenology of familiar settings. Appendix A
discusses covariant regularization in probability space and its (potential)
issues. Appendix B provides lemmas and proves supporting the stated theorems.
Appendix C relates our findings to standard score and test in the model-
centric literature.

##  2 The Algebra of Empiricism

In a fully data-centric approach, the very concept of a model is critically
challenged, since observations can change not only our prior knowledge about
the strength of suspected associations in the model, but the very model
structure itself. Hence, we need a flexible formalism that would keep track of
such structural variations. In this paragraph, we provide the algebraic
characterization of a quantitative analysis that is solely based on
empiricism.

#### The entities

An empirical description about some subjects comprises a finite set of $L$
attributes that distinctly (e.g. a person cannot be minor and adult at the
same time) characterize these subjects. Each attribute $i\in\\{1,\ldots,L\\}$
admits a finite number of countably many categories or levels; either
unordered (e.g. black, white and yellow) or ordered (e.g. detector counts). In
this spirit, a metric variable over a finite support like human height
resolved up to centimeter scale can be thought of as an ordered categorical
attribute. The set of possible categories for an attribute constitutes its
domain $\mathcal{D}_{i}$. Its specification is based on plausibility or prior
knowledge about the investigated subjects.

Any subject is represented by some configuration where each attribute assumes
a definite category from its respective domain, $e_{i}\in\mathcal{D}_{i}$ for
all $i=1,\ldots,L$. For ease of reference, we can think of a subject as an
entity333Given a well-defined notion of a system, its entities would be called
microstates in statistical physics. labeled by an $L$-tuple
$\left(e_{1},\ldots,e_{L}\right)$. In more algebraic terms, an entity can be
represented by an element

$\displaystyle|{\boldsymbol{e}}\rangle\equiv|e_{1},\ldots,e_{L}\rangle$ (1)

living in an abstract vector space. Two subjects represented by the same
entity cannot be empirically distinguished. Hence, the maximal resolution
power of empiricism after specifying attributes is dictated by the Cartesian
product of their domains,

$\displaystyle\boldsymbol{\mathcal{D}}=\mathcal{D}_{1}\times\cdots\times\mathcal{D}_{L}=\left\\{\left(e_{1},\ldots,e_{L}\right)~{}|~{}e_{1}\in\mathcal{D}_{1},\ldots,e_{L}\in\mathcal{D}_{L}\right\\}~{},$
(2)

which indexes the basis of our vector space,

$\displaystyle\mathcal{E}=\left\\{|e_{1},\ldots,e_{L}\rangle~{}|~{}\left(e_{1},\ldots,e_{L}\right)\in\boldsymbol{\mathcal{D}}\right\\}~{}.$
(3)

This basis includes all entities both observed and unobserveded, which are
maximally anticipated to be distinguishable in the attribute-based approach.
In principle, an unobserved entity that belongs to a subset
$\mathcal{E}_{0}\subset\mathcal{E}$ could be attributed either to finite-
sampling effects on rare realizations (e.g. a desert receiving more than 25
centimeters of precipitation a year) or deterministically impossible
situations (e.g. being male and pregnant). In statistics literature [15],
former entity is referred to as a sampling zero and latter as a structural
zero. We shall call both types nullentities.

Let ${\boldsymbol{\mathcal{D}}_{\star}}$ denote manifestations of the
Cartesian product Eq. (2) that retain some stochasticity. A sampled record
from the database would be always a distinct entity
$|{\boldsymbol{e}}\rangle\in\mathcal{E}_{\star}$ where

$\displaystyle\mathcal{E}_{\star}\equiv\mathcal{E}\setminus\mathcal{E}_{0}=\\{|{\boldsymbol{e}}\rangle\in\mathcal{E}~{}|~{}{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}\\}$
(4)

signifies the subspace spanned by admissible entities. After (arbitrarily)
enumerating possible entities via
${enum}:\mathcal{E}_{\star}\rightarrow\mathbb{N}$, one can intuitively
identify them with the canonical basis in
$\mathbb{R}^{|\mathcal{E}|-|\mathcal{E}_{0}|}$:

$\displaystyle|{\boldsymbol{e}}\rangle\overset{\cdot}{=}\begin{pmatrix}0\\\\[-4.30554pt]
\vdots\\\\[-2.58334pt] 1\\\\[-3.44444pt] \vdots\\\\[-2.15277pt]
0\end{pmatrix}\quad\leftarrow\text{ row }{enum}({\boldsymbol{e}})~{}.$ (5)

In that way, entities can be thought of as column vectors in
$\mathbb{R}^{|\mathcal{E}|-|\mathcal{E}_{0}|}$. Dual to entities we introduce
row vectors $\langle e|$ so that we can define a scalar product with

$\displaystyle\langle{\boldsymbol{e}}^{\prime}\;|\;{\boldsymbol{e}}\rangle=\prod_{i=1}^{L}\delta(e_{i},e^{\prime}_{i})\quad\text{for}\quad|{\boldsymbol{e}}\rangle\in\mathcal{E}\quad\text{and}\quad\langle{\boldsymbol{e}}^{\prime}|\in\mathcal{E}^{*}$
(6)

where $\delta$ denotes $L$ indicator maps
$\mathcal{D}_{i}\times\mathcal{D}_{i}\rightarrow\\{0,1\\}$ in the respective
feature domains.

#### The data

In any setting, there exists (even if we do not have direct access to) a data
matrix X with the records from a sample of size $N$ in the rows and $L$
features along the columns. In the picture of Eq. (5), the $r$-th row of X
corresponds to an entity from $\mathcal{E}$. The total number of records
corresponding to rows of X defines the only (hopefully large) scale we are
going to use in our analysis, the sample size $N$. From the data matrix, we
can further calculate over all distinct configurations in Eq. (2) relative
frequencies

$\displaystyle
f_{\boldsymbol{e}}=\frac{1}{N}\sum_{r=1}^{N}\delta(e_{1},\texttt{X}_{r,1})\cdots\delta(e_{L},\texttt{X}_{r,L})\quad\text{for}\quad{\boldsymbol{e}}\in\mathcal{D}_{1}\times\dots\times\mathcal{D}_{L}~{}.$
(7)

#### Probabilities

We quantify our algebraic characterization of inference problems by only
incorporating physically invariant, i.e. parameter-agnostic information. As
explained in the previous paragraph, an empirical analysis can at most draw
conclusions on subjects from the abundance of entities, so that (relative)
frequencies in Eq. (7) become the cornerstone of data-centric studies.
Algebraically, our quantitative perception of distinct entities can be fully
described by a diagonal operator

$\displaystyle\mathfrak{p}=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}p_{\boldsymbol{e}}\,|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|~{}\in\,\mathcal{B}(\mathcal{E}_{\star})~{},$
(8)

acting on $\text{span}(\mathcal{E}_{\star})$, the space spanned by the set of
relevant entities. By construction, its eigenvalues

$\displaystyle\mathfrak{p}|{\boldsymbol{e}}\rangle=p_{{\boldsymbol{e}}}|{\boldsymbol{e}}\rangle\quad\text{for}\quad|{\boldsymbol{e}}\rangle\in\mathcal{E}_{\star}$
(9)

give the expansion coefficients $p_{\boldsymbol{e}}\geq 0$ which weigh our
anticipation about each admissible entity. Such operators are sufficient to
characterize our knowledge about subjects for any practical purpose.

At given sample size $N$, the smallest possible variation of a relative
frequency

$\displaystyle p_{\boldsymbol{e}}\quad\rightarrow\quad
p_{\boldsymbol{e}}+\delta p_{\boldsymbol{e}}$ (10)

is naturally of order $1/N$ corresponding to a change of one data entry.
Statistics as an effective description is expected to become sensible only at
sufficiently large $N$. In anticipation of the large-$N$ expansion motivated
in Section 3.1, we can thus approximate relative frequencies with non-negative
real numbers, the error being variationally suppressed by $N$. Working at
fixed sample size immediately leads to normalization condition

$\displaystyle\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}p_{\boldsymbol{e}}\overset{!}{=}1~{}.$
(11)

Normalization and non-negativity imply in turn, $p_{\boldsymbol{e}}\in[0,1]$.

Consequently, the expansion coefficients of $\mathfrak{p}$ can be regarded at
large $N$ as probabilities making $\mathfrak{p}$ a probability operator. In
statistics, such map from $\mathcal{E}_{\star}$ to
$[0,1]^{|\mathcal{E}|-|\mathcal{E}_{0}|}$ which assigns to each admissible
entity $|{\boldsymbol{e}}\rangle$ a weight $p_{\boldsymbol{e}}$ gives rise to
the joint probability distribution on the simplex

$\displaystyle\mathcal{P}(\mathcal{E}_{\star})=\left\\{\mathfrak{p}~{}\big{|}~{}p_{\boldsymbol{e}}\in\mathbb{R}^{+}_{0}~{}\text{
for
}~{}{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}\quad\text{and}\quad\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}p_{\boldsymbol{e}}=1\right\\}~{}.$
(12)

For the simplex over $\mathcal{E}$, we simply write
$\mathcal{P}\equiv\mathcal{P}(\mathcal{E})$.

#### The empirical distribution

At the center of our algebra, there always exists the empirical distribution
${\mathfrak{f}}$ over the full $\mathcal{E}$ with components given in Eq. (7)
that can be easily extracted from $\mathfrak{f}$ via

$\displaystyle
f_{e}=\langle{\boldsymbol{e}}\;|\mathfrak{f}|\;{\boldsymbol{e}}\rangle$ (13)

using the orthonormality of Eq. (6). The empirical set of unobserved entities

$\displaystyle\left\\{|{\boldsymbol{e}}\rangle\in\mathcal{E}~{}|~{}f_{\boldsymbol{e}}=0\right\\}$
(14)

dictates the maximal set of nullentities that is logically allowed by the
provided data. Evidently, nullentities from a declared $\mathcal{E}_{0}$ must
be compatible with the empirical perception, meaning that they form a subset
of the empirical nullentities,

$\displaystyle\forall\,|{\boldsymbol{e}}\rangle\in\mathcal{E}_{0}\,:\quad
f_{\boldsymbol{e}}=0~{}.$ (15)

Consequently, ${\mathfrak{f}}$ can be accommodated within
$\mathcal{B}(\mathcal{E}_{\star})$ by appropriately tuning
$p_{\boldsymbol{e}}=f_{\boldsymbol{e}}\geq 0$.

#### Priors and reference distribution

In addition to the provided dataset X, we can express any prior knowledge on
the subjects under investigation through a reference distribution
$\boldsymbol{\upsilon}\in\mathcal{P}(\mathcal{E})$. For example,
${\boldsymbol{\upsilon}}$ could incorporate deterministic statements
reflecting structural zeros or more generally fundamental laws governing the
subjects under investigation. Falsely declaring structural zeros that are
later observed in the data would lead to infinities in the information-
theoretic metrics. Hence,

######  Definition 2.1.

A data-compatible reference distribution ${\boldsymbol{\upsilon}}$ fulfills

$\displaystyle
f_{\boldsymbol{e}}>0\quad\Rightarrow\quad\upsilon_{\boldsymbol{e}}>0~{},$

i.e. its nullentities are a subset of the empirically observed nullentities.

Throughout the statistical analysis in Sections 3 and 4,
${\boldsymbol{\upsilon}}$ remains the reference state of knowledge to
benchmark empirical descriptions and select the most adequate for a given
problem.

#### Minimal prior knowledge

A very symmetric situation arises when we are most ignorant about subjects and
their relations. Lacking any prior knowledge about the underlying mechanisms
that generate data, each attribute domain $\mathcal{D}_{i}$ enjoys a
permutation symmetry (mere re-ordering) of its levels ${\boldsymbol{e}}_{i}$
as discussed e.g. in [16]. At first, such symmetry might seem counter-
intuitive for counts and metric variables where a natural order exists.
However, such physical order is later re-instated via moment observations and
must not be confused with the arbitrary order of the labels $e_{i}$.

Since the set of entities inherits the aforementioned permutation symmetry,
minimal knowledge about $\mathcal{E}$ is necessarily described by uniformly
assigning probabilities to all entities via

$\displaystyle{\mathfrak{u}}=\frac{1}{|\mathcal{E}|}\,\mathbb{I}_{\mathcal{E}}=\frac{1}{|\mathcal{E}|}\sum_{{\boldsymbol{e}}\in\boldsymbol{\mathcal{D}}}|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|~{}.$
(16)

The most symmetric probability assignment reflects the same degree of
ignorance for all entities, as any other distribution would discriminate
against some entity, hence breaking permutation symmetry.

#### Characterizing entities

To quantitatively describe subjects, the associated entities must be related
to some observable quantity that characterizes them. At the algebraic level, a
sensible characteristic can be inferred using a suitable operator
$\,\hat{M}\in\mathcal{B}(\mathcal{E}_{\star})$ which assigns a definite value
$m({\boldsymbol{e}})\in\mathbb{R}$ to each admissible entity via a well-
defined eigenvalue equation

$\displaystyle\forall\,|{\boldsymbol{e}}\rangle\in\mathcal{E}_{\star}:\quad\,\hat{M}|{\boldsymbol{e}}\rangle=m({\boldsymbol{e}})|{\boldsymbol{e}}\rangle~{}.$
(17)

Colloquially, we also refer to $\,\hat{M}$ as the characteristic. Notice that
linear action Eq. (17) does not need to uniquely determine an entity. Most
notably the resolution of identity

$\displaystyle\,\mathbb{I}_{\mathcal{E}_{\star}}=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|$
(18)

assigns the same value to all admissible entities. In contrast, projector
operator

$\displaystyle\,\mathbb{I}_{{\boldsymbol{e}}}=|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|$
(19)

fully adheres to the unique characteristic of entity
$|{\boldsymbol{e}}\rangle$, which is unambiguously defined by specifying the
levels $e_{i}$ for all attributes, as already discussed.

Most interestingly, characteristics which non-trivially group subjects under
common eigenvalues enable a quantitative investigation of the induced
collective effect. When a distribution $\mathfrak{p}$ describes the knowledge
about entities, the statistics of $\,\hat{M}$ – which collectively
characterize subjects – can be extracted as the $\mathfrak{p}$-expectation

$\displaystyle\langle\mathfrak{p}\,\hat{M}\rangle\equiv\langle\mathcal{E}_{\star}\;|\mathfrak{p}\,\hat{M}|\;\mathcal{E}_{\star}\rangle=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}p_{\boldsymbol{e}}\,m({\boldsymbol{e}})=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}\langle{\boldsymbol{e}}\;|\mathfrak{p}\,\hat{M}|\;{\boldsymbol{e}}\rangle$
(20)

where

$\displaystyle|\mathcal{E}_{\star}\rangle=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}|{\boldsymbol{e}}\rangle~{}\in~{}\text{span}(\mathcal{E}_{\star})~{}.$
(21)

Whenever clear from context or if $\mathcal{E}_{0}=\emptyset$, we shall drop
$|\mathcal{E}_{\star}\rangle$ from expectations and use the compact notation
$\langle\mathfrak{p}\,\hat{M}\rangle$. Trivially, normalization condition Eq.
(11) can be written as the $\mathfrak{p}$-expectation of identity Eq. (18)

$\displaystyle\langle\mathfrak{p}\,\mathbb{I}_{\mathcal{E}}\rangle=\langle\mathfrak{p}\rangle\overset{!}{=}1~{},$
(22)

while the probability of entity $|{\boldsymbol{e}}\rangle$ may be understood
as the $\mathfrak{p}$-expectation of the corresponding projector Eq. (19)

$\displaystyle\langle\mathfrak{p}\,\mathbb{I}_{\boldsymbol{e}}\rangle=\langle{\boldsymbol{e}}\;|\mathfrak{p}|\;{\boldsymbol{e}}\rangle=p_{\boldsymbol{e}}~{}.$
(23)

In principle, there exist various – possibly unaccountably many, not
necessarily distinct – ways to characterize entities. This results in a
multitude of statistics. Collective characterizations like the average weight
in a group of pupils or the prevalence of single-author papers in a field,
usually offer a concise way to succinctly describe trends and patterns among
attributes without committing to a certain entity. Any quantity that is easily
observed like net magnetization (against orientation of constituent spins) or
easily accessed like prevalence of a disease (against patients’ microdata)
could sensibly serve as a collective characterization of entities.

Different types of characteristics can be analyzed contingent on the form of
$m({\boldsymbol{e}})$ in Eq. (17). For instance, the eigenvalue problem

$\displaystyle\,\mathbb{I}_{z_{i}}|{\boldsymbol{e}}\rangle=\delta(e_{i},z_{i})|{\boldsymbol{e}}\rangle\quad,\quad
z_{i}\in\mathcal{D}_{i}$ (24)

would simply result in the marginal probability
$\langle\mathfrak{p}\,\mathbb{I}_{z_{i}}\rangle$ of feature $i$ being on state
$e_{i}=z_{i}$. Also, moment measurements are easily realized, e.g. the $n$-th
moment of feature $j$ can be extracted via

$\displaystyle\,\hat{L}_{j}^{n}|{\boldsymbol{e}}\rangle=e_{j}^{n}|{\boldsymbol{e}}\rangle~{},$
(25)

invoking an operator which gets the level of desired attribute,
$\,\hat{L}_{j}|{\boldsymbol{e}}\rangle=e_{j}|{\boldsymbol{e}}\rangle$. Any
combination of marginal and moment operators is evidently admissible.

Considering multiple characteristics $\,\hat{M}_{\alpha}$ provides a summary
of entities. To obtain the summary statistics, i.e. the statistics of each
characteristic, we use:

######  Definition 2.2 (constructing elements).

An irreducible constructing element (also thought as a vector operator)

$\displaystyle\mathbf{\,\hat{M}}\equiv\text{span}\left\\{\,\hat{M}_{1},\ldots,\,\hat{M}_{D}\right\\}$
(26)

is a collection of $D$ diagonalizable operators $\,\hat{M}_{\alpha}$ defined
by linearly independent actions on $\text{span}(\mathcal{E}_{\star})$, i.e.
fulfill as an operator identity:

$\displaystyle\sum_{\alpha=1}^{D}c_{\alpha}\,\hat{M}_{\alpha}=0\quad\Rightarrow\quad
c_{\alpha}=0~{}\,\forall\,\alpha=1,\ldots,D~{}.$ (27)

Within maximally diagonalizable operators in
$\mathcal{B}(\mathcal{E}_{\star})$, we express through irreducible vector
operator $\mathbf{\,\hat{N}}$ satisfying

$\displaystyle\langle\,\hat{N}_{a}\,\hat{M}_{\alpha}\rangle=0\quad\forall~{}\alpha=1,\ldots,D\quad\text{and}\quad
a=1,\ldots|\mathcal{E}|-|\mathcal{E}_{0}|-D$ (28)

the orthogonal complement to the space spanned by $\mathbf{\,\hat{M}}$. In
other words, $\mathbf{\,\hat{N}}$ spans the kernel of $\mathbf{\,\hat{M}}$,
$\ker\mathbf{\,\hat{M}}=\text{span}(\mathbf{\,\hat{N}})$. By definition Eq.
(27), the eigenvalues of the constructing element on admissible entities,

$\displaystyle\alpha=1,\ldots
D\quad\text{and}\quad|{\boldsymbol{e}}\rangle\in\mathcal{E}_{\star}\,:\quad\,\hat{M}_{\alpha}|{\boldsymbol{e}}\rangle=m_{\alpha}({\boldsymbol{e}})|{\boldsymbol{e}}\rangle~{},$
(29)

can be viewed as elements of a
$D\times\left(|\mathcal{E}|-|\mathcal{E}_{0}|\right)$ matrix of full row-rank.
In that way, the $\mathfrak{p}$-expectation of $\mathbf{\,\hat{M}}$ induces a
linear map

$\displaystyle\langle\mathcal{E}_{\star}\;|\mathfrak{p}\mathbf{\,\hat{M}}|\;\mathcal{E}_{\star}\rangle:~{}~{}[0,1]^{|\mathcal{E}|-|\mathcal{E}_{0}|}\rightarrow\mathbb{R}^{D}$
(30)

from the probabilities of distinct entities to the observable world of
collective characteristics.

Of course, nothing prevents us from learning the data X “by heart”, when
$\mathbf{\,\hat{M}}$ is equivalent to a collection of
$|\mathcal{E}|-|\mathcal{E}_{0}|$ projector operators Eq. (19),

$\displaystyle\text{\small
1}\text{1}=\left\\{|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|\right\\}_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}~{},$
(31)

represented by identity matrix in
$\mathbb{R}^{|\mathcal{E}|-|\mathcal{E}_{0}|}$ (viz. identification of Eq.
(5)). In that way, we recover the relative frequencies $f_{\boldsymbol{e}}$
from X. Committing to empirical distribution ${\mathfrak{f}}$, trace action
Eq. (20) can be understood as an abstraction for measurement (e.g. in the lab)
or observation (e.g. in astronomy). As both are formally realized by the same
algebraic operation as an $\mathfrak{f}$-expectation of characteristics, we
interchangeably use the notions of measurement and observation. Due to Eq.
(15), the empirical statistics
$\langle\mathfrak{f}\,\hat{M}_{\alpha}\rangle\equiv\langle\mathcal{E}\;|\mathfrak{f}\,\hat{M}_{\alpha}|\;\mathcal{E}\rangle=\langle\mathcal{E}_{\star}\;|\mathfrak{f}\,\hat{M}_{\alpha}|\;\mathcal{E}_{\star}\rangle$
of any operator $\,\hat{M}_{\alpha}\in\mathcal{B}(\mathcal{E}_{\star})$ remain
insensitive to nullentities.

######  Definition 2.3 (phenomenology).

Using irreducible vector operator $\mathbf{\,\hat{M}}$ to infer summary
statistics, the set of $D$ empirical expectations
$\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$ that are calculated from data
constitutes a phenomenology.

Defined as a set of observed values referring back to some distinct entities,
phenomenology naturally induces an equivalence orbit on
$\mathcal{P}(\mathcal{E}_{\star})$,

$\displaystyle{\mathfrak{p}}\quad\overset{\mathbf{\,\hat{M}}}{\sim}\quad{\mathfrak{f}}$

that contains all distributions with expectations
$\langle\mathcal{E}_{\star}\;|\mathfrak{p}\mathbf{\,\hat{M}}|\;\mathcal{E}_{\star}\rangle$
dictated by the empirical estimates
$\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$. Hence, phenomenology becomes a
set of guidelines to select adequate distributions based on the empirical
summary statistics, such as the macroscopic effects of pressure and
temperature hint towards the kinematic distribution of molecules in an atomic
gas. In most interesting applications, linear map Eq. (30) – being usually
rank-deficit – is not one-to-one, so that phenomenology
$\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$ cannot fix the probabilities of
all entities. For example, there could exist many atomic gases with different
microscopic properties all exhibiting the same pressure at the same
temperature.

Ultimately, we arrive at

######  Definition 2.4 (TotEm).

After declaring nullentities in $\mathcal{E}_{0}$ compatible with prior
knowledge,

$\displaystyle|{\boldsymbol{e}}\rangle\in\mathcal{E}_{\star}\,:\quad\boldsymbol{\upsilon}_{\boldsymbol{e}}>0~{},$
(32)

a TotEm of the data is defined by selecting distributions
$\mathfrak{p}\in\mathcal{P}(\mathcal{E}_{\star})$ which support a
phenomenology $\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$:

$\displaystyle\langle\mathcal{E}_{\star}\;|\mathfrak{p}\mathbf{\,\hat{M}}|\;\mathcal{E}_{\star}\rangle\overset{!}{=}\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle~{}.$
(33)

The induced equivalence class $[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ which
comprises all distributions on $\mathcal{P}(\mathcal{E}_{\star})$ that satisfy
the phenomenology of the TotEm description forms the TotEmplex.

Two irreducible vector operators related by full-rank444$\mathbf{T}$ must be a
$D\times D$ matrix of full rank, otherwise the transformed vector operator
would be reducible. transformation matrix $\mathbf{T}$ via

$\displaystyle\,\hat{M}_{\alpha}\quad\rightarrow\quad\sum_{\beta=1}^{D}T_{\alpha\beta}\,\,\hat{M}_{\beta}\quad\text{for}\quad\alpha=1,\ldots,D$
(34)

construct the same TotEmplex being for all practical purposes (fapp)
equivalent.

Normalization condition Eq. (11) implemented by
$\,\mathbb{I}_{\mathcal{E}_{\star}}$ is automatically taken to be part of any
$\mathbf{\,\hat{M}}$ constructing a TotEm, either explicitly or implicitly
e.g. by fixing two complementary marginals which empirically sum to one. In
absence of observations besides the specification of $\mathcal{E}_{0}$, the
TotEmplex thus reduces to the simplex Eq. (12),
$[\mathfrak{f};{\,\mathbb{I}_{\mathcal{E}_{\star}}}]=\mathcal{P}(\mathcal{E}_{\star})$
where joint distributions are still empirically indistinguishable points. In
the opposite limit, whenever $|\mathcal{E}|-|\mathcal{E}_{0}|$ independent
observations are specified so that $\mathbf{\,\hat{M}}$ is fapp equivalent to
Eq. (31), we have $[\mathfrak{f};{\text{\small
1}\text{1}}]=\\{\mathfrak{f}\\}$.

Trivially, phenomenological constraints Eq. (33) always admit at least one
solution, the empirical distribution $\mathfrak{f}$ itself. As long as
$D<|\mathcal{E}|-|\mathcal{E}_{0}|$, so that
$|\ker\mathbf{\,\hat{M}}|=|\mathcal{E}|-|\mathcal{E}_{0}|-D>0$, there exist
infinitely many distributions on the TotEmplex generated by convexity through

$\displaystyle\lambda{\mathfrak{p}}+\left(1-\lambda\right){\mathfrak{p}^{\prime}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]\quad\text{for}\quad{\mathfrak{p}},{\mathfrak{p}^{\prime}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]\subseteq\mathcal{P}(\mathcal{E}_{\star})\quad\text{and}\quad\lambda\in[0,1]~{},$
(35)

where ${\mathfrak{p}}-{\mathfrak{p}^{\prime}}\in\ker\mathbf{\,\hat{M}}$. In
Section 3.4, we come back to the study of such kernel fluctuations.

Using linear action Eq. (17) of operators on admissible entities, the defining
operation Eq. (33) of a TotEm can be written in the spirit of Eq. (30) as a
linear program

$\displaystyle p_{\boldsymbol{e}}\geq
0~{}\forall\,{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}\quad\text{and}\quad\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}m_{\alpha}({\boldsymbol{e}})\left(p_{\boldsymbol{e}}-f_{\boldsymbol{e}}\right)\overset{!}{=}0\quad\forall\,\alpha=1,\ldots
D$ (36)

in the non-negative probabilities $p_{\boldsymbol{e}}$ with the components of
coefficient matrix given by the eigenvalues $m_{\alpha}({\boldsymbol{e}})$. In
particular, full-rank linear transformation Eq. (34) is realized by elementary
row operations on the matrix of eigenvalues. Relating fapp equivalent
constructing elements, such transformations do not change the solution space
of linear program Eq. (36). To robustly classify inequivalent TotEm
descriptions the reduced row-echelon form $\mathbf{R}$ of the coefficient
matrix in Eq. (36) could be invoked, leading to generalized expectations. In
particular, normalization on the simplex Eq. (12) implies

$\displaystyle\sum_{\alpha=1}^{D}R_{\alpha,{\boldsymbol{e}}}=1~{}~{}\forall~{}{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}~{},$
(37)

since there are by construction $D=\text{rank}\,\mathbf{\mathbf{\,\hat{M}}}$
columns in $\mathbf{R}$ with all zeros but the leading one.

###  2.1 Information-theoretic metrics

To algebraically compare different states of knowledge about entities we need
a notion of distance. On simplex $\mathcal{P}(\mathcal{E}_{\star})$, an
information-compatible, non-commutative form is given by the negative
$\mathfrak{p}$-expectation of the logarithm of probability operator
$\mathfrak{q}$

$\displaystyle\ell(\mathfrak{p};\mathfrak{q})\equiv-\langle\mathfrak{p}\log\mathfrak{q}\rangle=-\langle\mathcal{E}_{\star}\;|\mathfrak{p}\log\mathfrak{q}|\;\mathcal{E}_{\star}\rangle=-\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}p_{\boldsymbol{e}}\log
q_{\boldsymbol{e}}~{}.$ (38)

Evidently, it is linear in its first argument. In particular, the series
definition of logarithm implies due to orthonormality Eq. (6) the expression

$\displaystyle\log\mathfrak{q}=$
$\displaystyle\,\,\log\left(\,\mathbb{I}_{\mathcal{E}_{\star}}+\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}(q_{\boldsymbol{e}}-1)|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|\right)=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}(q_{\boldsymbol{e}}-1)|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|-\mbox{$\frac{1}{2}$}\left(\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}(\mathfrak{q}_{\boldsymbol{e}}-1)|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|\right)^{2}+\ldots$
$\displaystyle=$
$\displaystyle\,\,\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}\left[(q_{\boldsymbol{e}}-1)-\mbox{$\frac{1}{2}$}\left(\mathfrak{q}_{e}-1\right)^{2}+\ldots\right]|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}\log
q_{\boldsymbol{e}}|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|~{},$

where the resolution of identity Eq. (18) has been used in rewriting the
argument within the logarithm. The induced norm squared

$\displaystyle
H[\mathfrak{p}]\equiv\ell(\mathfrak{p};\mathfrak{p})=-\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}p_{\boldsymbol{e}}\log
p_{\boldsymbol{e}}$ (39)

coincides [17] with Shannon’s differential entropy and is semi-positive
definite.

Notice that

$\displaystyle\exp\left(-\langle\mathcal{E}_{\star}\;|\mathfrak{p}\log\mathfrak{q}|\;\mathcal{E}_{\star}\rangle\right)$
(40)

determines the likelihood of ${\mathfrak{p}}$ under ${\mathfrak{q}}$. Any
entity with $p_{\boldsymbol{e}}=0$ does not contribute to the
$\log$-likelihood leaving thus the likelihood invariant. On the other hand, an
entity with $p_{\boldsymbol{e}}\neq 0$ for which $q_{\boldsymbol{e}}=0$ would
result into vanishing likelihood as contradictory to logic. In particular, a
compatible in the sense of 2.1 reference distribution asserts that
information-theoretic measures like Eq. (38) remain finite throughout our
explorations. A safe strategy to avoid breakdowns of the information-theoretic
description is to select a reference distribution which assigns555The opposite
direction that $\boldsymbol{\upsilon}_{\boldsymbol{e}}>0$ implies
$\mathfrak{f}_{\boldsymbol{e}}>0$ does not need to hold; and will not for many
realistic datasets. non-zero probability to all possible entities, but (at
most) structural zeros.

Based on the $\log$-likelihood, we now introduce a “distance” via

$\displaystyle D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q}):=-\ell(\mathfrak{p};\mathfrak{p})+\ell(\mathfrak{p};\mathfrak{q})=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}p_{\boldsymbol{e}}\log\frac{p_{\boldsymbol{e}}}{q_{\boldsymbol{e}}}$
(41)

to measure [18] the information divergence (in short $I$-divergence) between
two distributions on $\mathcal{P}(\mathcal{E}_{\star})$. Noting that
$0\cdot\log 0=0$, the $I$-divergence can be lifted on the ambient simplex
$\mathcal{P}$ to become the well-known [19, 20] Kullback-Leibler divergence or
relative entropy. This information-theoretic metric remains [8] non-negative
and vanishes iff ${\mathfrak{p}}={\mathfrak{q}}$. On the other hand, it is
neither symmetric nor linear in its arguments.

##  3 Knowledge given phenomenology

In this Section, we systematize the study of trends and patterns given a TotEm
2.4 of the data triggered by phenomenology 2.3. Starting from a sensible
reference distribution, we show the existence and uniqueness of a special
distribution compatible with the phenomenology. Subsequently, we show how to
estimate the probabilities of distributions on the TotEmplex as a large-$N$
expansion around this special distribution.

###  3.1 From discrete to continuous

So far, we have not discussed the total number of entries in X, the sample
size $N>0$. In fact, $N$ plays a crucial role in any TotEm, as it becomes the
controlling parameter of the effective theory that assigns probabilities to
distributions on the TotEmplex. To appreciate the significance of this scale
we emphasize the discrete nature of Eq. (36). The actual empirical problem at
hand consists of the Diophantine system of linear equations

$\displaystyle\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}m_{\alpha}({\boldsymbol{e}})\left(Np_{\boldsymbol{e}}-Nf_{\boldsymbol{e}}\right)=0\quad\text{with}\quad
Nf_{\boldsymbol{e}}\,,\,Np_{\boldsymbol{e}}\in\mathbb{N}_{0}^{+}\quad\forall\,{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}$
(42)

to determine appropriate counts $Np_{\boldsymbol{e}}$ satisfying the
phenomenology $\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$. For the purposes
of this paragraph, we recycle operator notation $\mathfrak{p}$ to represent
estimates about relative frequencies. The solution set of Eq. (42) is an
integral lattice polytope, i.e. a convex bounded polyhedron with vertices over
the integer lattice.

Notice that the reference distribution itself does not need to belong to the
polytope (or the TotEmplex), since it would generically neither share the same
nullentities with the TotEm nor adhere to the defining expectations of
phenomenology 2.3. Nevertheless, $\boldsymbol{\upsilon}\in\mathcal{P}$ as part
of prior knowledge expresses all we know about the subjects when we receive
the dataset. Consequently, counts compatible with the phenomenology are
naturally666For finite but sufficiently large populations, it is permissible
[21] to use instead of the hypergeometric the multinomial distribution.
sampled [22, 23, 24] from this reference point on the simplex via
($a!=\Gamma(a+1)$ denotes factorial)

$\displaystyle\text{mult}(N\mathfrak{p};\boldsymbol{\upsilon})=\exp\left\\{\log
N!+\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}\left[Np_{\boldsymbol{e}}\log\upsilon_{\boldsymbol{e}}-\log\left(Np_{\boldsymbol{e}}\right)!\right]\right\\}~{}.$
(43)

#### The large-$N$ expansion

Appealing to large-$N$ and large-deviation techniques [25, 26], we can
smoothly extend Eq. (43) from relative counts to more general probabilities.
At the operational level, this means that rational relative frequencies in the
Diophantine system are lifted over to the non-negative real numbers roughly
approximating the lattice polytope by the TotEmplex to deduce probabilities
over phenomenology-compatible distributions.

At sufficiently large sample size $N$, the probability density to select a
distribution in the differential region around $\mathfrak{p}$ becomes

$\displaystyle\text{mult}\left(N\mathfrak{p};\boldsymbol{\upsilon}\right)=\left(2\pi
N\right)^{-\frac{|\mathcal{E}_{\star}|-1}{2}}\exp\left\\{-ND\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})+\frac{|\mathcal{E}|-|\mathcal{E}_{0}|}{2}\,\ell(\mathfrak{u};\mathfrak{p})+\mathcal{O}\left(N^{-1}\right)\right\\}$
(44)

using Stirling’s formula. The large-$N$ expansion of the multinomial
distribution thus reveals that $\boldsymbol{\upsilon}$-based sampling is
governed by the $I$-divergence Eq. (41) of phenomenology-compatible
distribution $\mathfrak{p}$ from reference distribution
$\boldsymbol{\upsilon}$,

$\displaystyle\log\text{mult}\left(N\mathfrak{p};\boldsymbol{\upsilon}\right)\approx-
ND\hskip 0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})~{}.$ (45)

To demonstrate the logic of continuum limit, we consider the trivial setting
in absence of phenomenological constraints. Taking for simplicity
$\mathcal{E}_{0}=\emptyset$, we can replace discrete summation over counts
$Np_{\boldsymbol{e}}\in\\{0,1,\ldots,N\\}$ with integration on the simplex,

$\displaystyle\prod_{{\boldsymbol{e}}\in\boldsymbol{\mathcal{D}}}\sum_{Np_{\boldsymbol{e}}}\delta\left(\sum_{{\boldsymbol{e}}^{\prime}\in\boldsymbol{\mathcal{D}}}Np_{{\boldsymbol{e}}^{\prime}},N\right)\quad\rightarrow\quad
N^{|\mathcal{E}|-1}\int_{\mathcal{P}}~{},$ (46)

the relative mistake generically being of $\mathcal{O}\left(1/N\right)$ for
variations Eq. (10). The $\delta$ symbol on the l.h.s. stands for the
Kronecker Delta. Along these lines, the volume of the $N$-dilation of the
simplex $\mathcal{P}$,

$\displaystyle\text{vol}\left(N\mathcal{P}\right)=N^{|\mathcal{E}|-1}\,\text{vol}\left(\mathcal{P}\right)=\frac{N^{|\mathcal{E}|-1}}{\left(|\mathcal{E}|-1\right)!}~{},$
(47)

indeed shares the same large-$N$ scaling

$\displaystyle\frac{\text{vol}\left(N\mathcal{P}\right)}{\text{vol}\left(\mathcal{P}\cap\mathbb{N}_{0}^{|\mathcal{E}|}\right)}=1-\frac{(L-1)L}{2N}+\mathcal{O}\left(N^{-2}\right)$
(48)

with the volume [27] of its discrete counterpart

$\displaystyle\text{vol}\left(\mathcal{P}\cap\mathbb{N}_{0}^{|\mathcal{E}|}\right)=\begin{pmatrix}N+|\mathcal{E}|-1\\\
|\mathcal{E}|-1\end{pmatrix}~{},$ (49)

which counts the non-negative solutions of Diophantine system Eq. (42) in the
trivial case of $\mathbf{\,\hat{M}}=\\{\,\mathbb{I}_{\mathcal{E}}\\}$. One can
explicitly verify (see Appendix E) integrating order-by-order the large-$N$
expansion of Eq. (44) the consistency of the suggested continuum approximation
Eq. (46).

The bounded convex polytope associated [28] to a more generic Diophantine
linear system Eq. (42) over $|\mathcal{E}_{\star}|$ counts becomes complicated
to deal with. Already the computation of its volume is np-complete [29]. At
large $N$ and in absence of many low-frequency entities nevertheless, we
anticipate the continuum limit to give stable relative estimates about the
scaling and tendencies of the discrete problem. In this spirit, we formally
write based on Eq. (45) the leading probability density

$\displaystyle f(\mathfrak{p})=Z^{-1}\exp\left\\{-ND\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})\right\\}$ (50)

to select a distribution ${\mathfrak{p}}$ from the TotEmplex
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ normalized by the cumulative probability

$\displaystyle Z=\int_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}e^{-ND\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})}$ (51)

to sample any distribution on $[\mathfrak{f};{\mathbf{\,\hat{M}}}]$. Notice
that the probabilities of distributions on the TotEmplex depend on operator
$\boldsymbol{\upsilon}\,\mathbb{I}_{\mathcal{E}_{\star}}$ – not necessarily a
distribution – due to $D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon}\,\mathbb{I}_{\mathcal{E}_{\star}})=D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})$.

At this point, there a priori exists no clear integral prescription over
generic polytopes. However, the $N$-scaling in the exponential of the
integrand invites us to perform Laplace-type approximations of increased
quality as the sample size grows. In Section 3.4, we argue how to perform the
integration order-by-order in large $N$ around a stable saddle point to
estimate probability density Eq. (50).

###  3.2 The minimum information divergence given phenomenology

Clearly, leading probability density Eq. (50) of distributions on a TotEmplex
and cumulative probability Eq. (51) are dominated with increasing $N$ by the
minimum $I$-divergence of phenomenology-compatible distribution from the
reference. In any realistic problem (proof given in B.1),

######  Theorem 3.1 (the $I$-projection).

th:iProjection Among distributions
${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ satisfying the summary
statistics of phenomenology $\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$,
there always exists a unique distribution signified by
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}\equiv\mathfrak{q}$
of minimum $I$-divergence from reference distribution
${\boldsymbol{\upsilon}}\in\mathcal{P}$:

$\displaystyle D\hskip 0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})\leq
D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})\quad\forall~{}{\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$
(52)

In information geometry, the point on the
simplex777$\langle\mathfrak{p}\rangle=1$ for all probability operators
$\mathfrak{p}$, thus also $\langle\mathfrak{q}\rangle=1$. corresponding to
$\mathfrak{q}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ best assimilates [30] the
properties of a projection of reference point ${\boldsymbol{\upsilon}}$ onto
the TotEmplex, when the $I$-divergence plays the role of “distance”. Hence, we
shall call
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}$ the
$I$-projection of ${\boldsymbol{\upsilon}}$ onto
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$.

Let us now focus on an “extreme” distribution ${\mathfrak{b}}$ from the
boundaries of the TotEmplex $[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ where at
least one $\mathfrak{b}_{\boldsymbol{e}}$ vanishes, saturating the lower bound
in Eq. (36). Using any other distribution
${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ we introduce –
exploiting the convexity Eq. (35) – an one-parameter family of distributions

$\displaystyle\mathfrak{l}={\mathfrak{b}}+\lambda\left({\mathfrak{p}}-{\mathfrak{b}}\right)\quad\text{for}\quad\lambda\in[0,1]~{},$
(53)

living on the investigated TotEmplex. Next, we look at the $I$-divergence of
$\mathfrak{l}$ from ${\boldsymbol{\upsilon}}$ as a convex function of
$\lambda$, whose right-derivative at the boundary equals in terms of Eq. (38)

$\displaystyle\lim_{\lambda\rightarrow 0^{+}}\frac{\text{d}D\hskip
0.28453pt(\mathfrak{l}\;\|\;\boldsymbol{\upsilon})}{\text{d}\lambda}=\sum_{\mathcal{E}_{\star}}\left(p_{\boldsymbol{e}}-b_{\boldsymbol{e}}\right)\log\frac{b_{\boldsymbol{e}}}{\upsilon_{\boldsymbol{e}}}=-\ell(\mathfrak{p};\mathfrak{b})+\ell(\mathfrak{p};\boldsymbol{\upsilon})-D\hskip
0.28453pt(\mathfrak{b}\;\|\;\boldsymbol{\upsilon})~{}.$ (54)

It is readily recognized that $-\ell(\mathfrak{p};\mathfrak{b})$ diverges as
$-\infty$, if any $b_{\boldsymbol{e}}$ vanishes whenever
$p_{\boldsymbol{e}}>0$, while the rest of the metrics remain finite (for
compatible reference distribution). Hence, the rate of change of the
$I$-divergence from ${\boldsymbol{\upsilon}}$ while leaving the boundary
becomes arbitrarily large and negative. This forces us to conclude that the
minimum must correspond to a distribution well in the interior of the
TotEmplex where all of the admissible probabilities are non-vanishing.

Having established that the $I$-projection cannot live on the boundaries of
the TotEmplex, i.e. its probabilities must be strictly positive, we can apply
variational techniques (well established in fields such as Bayesian inference
[31]) to find its form.

######  Theorem 3.2.

The $I$-projection of ${\boldsymbol{\upsilon}}$ onto
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ defined by 2.4, which lives in the
interior of the TotEmplex, can be parametrized as

$\displaystyle\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}=\exp\left(\boldsymbol{\theta}\cdot\mathbf{\,\hat{M}}\right)\boldsymbol{\upsilon}=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}\upsilon_{\boldsymbol{e}}\exp\left(\sum\limits_{\alpha=1}^{D}\theta_{\alpha}\,m_{\alpha}({\boldsymbol{e}})\right)|{\boldsymbol{e}}\rangle\langle{\boldsymbol{e}}|~{},$
(55)

given the observed values in Eq. (36) and a vector
$\boldsymbol{\theta}\in\mathbb{R}^{|\mathcal{E}|-|\mathcal{E}_{0}|}$ of
appropriately fixed Lagrange multipliers. The converse also holds: any
distribution on the TotEmplex of this form is the $I$-projection of
$\boldsymbol{\upsilon}$ onto the TotEmplex.

As before with Eq. (38), we use the series definition to resolve the
exponential of an operator by

$\displaystyle\exp\left(\boldsymbol{\theta}\cdot\mathbf{\,\hat{M}}\right)=\,\mathbb{I}_{\mathcal{E}_{\star}}+\boldsymbol{\theta}\cdot\mathbf{\,\hat{M}}+\mbox{$\frac{1}{2}$}\left(\boldsymbol{\theta}\cdot\mathbf{\,\hat{M}}\right)^{2}+\ldots~{}.$
(56)

The details about the exponential form of $I$-projections are given in B.2.

From the variational treatment of the linear program, we verify that the
$I$-projection does not introduce any zero probabilities, besides the declared
nullentities associated to $\mathcal{E}_{0}$ in the TotEm. Despite that the
$I$-projection itself is unique given a phenomenology
$\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$ and prior knowledge
${\boldsymbol{\upsilon}}$, its parameterized form Eq. (55) is not generally
unique, since the parametrization in terms of Lagrange multipliers depends on
physically equivalent observations realized as linear transformations of
coupled system Eq. (33). Using the derived parametric form Eq. (55) the full
version of “Pythagorean” relation is easily checked:

######  Corollary 3.3.

For any distribution ${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$
the equivalence applies:

$\displaystyle D\hskip 0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})\geq
D\hskip
0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})\quad\Leftrightarrow\quad
D\hskip 0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})=D\hskip
0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})+D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})~{},$ (57)

$\mathfrak{q}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ being the $I$-projection
of $\boldsymbol{\upsilon}$ onto the TotEmplex.

#### The emergence of entropy maximization

Starting from minimal prior knowledge realized [32] by the uniform
distribution, ${\boldsymbol{\upsilon}}={\mathfrak{u}}$, where

$\displaystyle\text{mult}(N\mathfrak{p};\mathfrak{u})=|\mathcal{E}|^{-N}\,W[N\mathfrak{p}]~{},$
(58)

the combinatorics of multinomial coefficient $W[N\mathfrak{p}]$ determines the
probability density of distributions on the TotEmplex. $W[N\mathfrak{p}]$
counts the combinatoric ways to partition $N$ identities into $|\mathcal{E}|$
teams such that the counts $N\mathfrak{p}_{\boldsymbol{e}}$ are
thermodynamically reproduced. At sufficiently large $N$, this is governed by
the norm squared Eq. (39) coinciding with the entropy of ${\mathfrak{p}}$, so
that distributions of higher entropy, conversely closer to ${\mathfrak{u}}$,

$\displaystyle H[\mathfrak{p}]=-D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{u})+\log|\mathcal{E}|~{},$ (59)

contribute more to the cumulative probability Eq. (51) over the TotEmplex. In
this special case that we have started from minimal prior knowledge over
$\mathcal{E}$, we recover [33, 34, 35] the Principle of Maximum Entropy
(maxent).

Despite that the maxent principle has been extensively studied in data-driven
literature and beyond [36], there is no mathematical prior in favor of the
uniform distribution as a reference distribution, besides axiomatic
formulation(s) [37, 38, 39]. In fact, any sensible
${\boldsymbol{\upsilon}}\in\mathcal{P}$ that respects 2.1 could mathematically
serve as a valid starting point to apply the theoretical framework developed
in the following. In [40] and future work, we show by applying TotEm
methodology to versatile problems the utility of reference distributions well
away from the maxent prior.

###  3.3 Iterative minimization of information divergence

As we have shown in the previous paragraph, the linear program Eq. (33)
supplemented by the objective of $I$-divergence minimization (entropy
maximization in the case of uniform prior knowledge) poses a well-defined
optimization problem. Given phenomenological constraints induced by the
$\mathfrak{f}$-expectation of $\mathbf{\,\hat{M}}$, we now want to obtain the
$I$-projection of $\boldsymbol{\upsilon}$ on
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ in a covariant way that does not commit
to any simplifying conditions of a particular problem.

In the context of statistical learning, the procedure of inferring from data
values that parameterize a certain distribution is known [41] as the inverse
problem. In presence of (exponentially) many entities, the inverse – similar
to the forward – problem can quickly become intractable. In practice however,
the np-complete bound usually lies far above a pragmatic bound, as noted in
[42]. Latter bound is posed by the size of realistic datasets (in turn bounded
by finite populations of species in the universe) which severely constrains
the ability to formulate statistical tests concerning the structure of more
complex [43] relationships among subjects.

For small kernels Eq. (28), one could well parameterize
$\ker\mathbf{\,\hat{M}}$ to analytically minimize the $I$-divergence in the
solution space of linear system Eq. (33). In the maxent context, a wealth of
perturbative methods [44, 45, 46, 47] might prove useful to approximate
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\mathfrak{u}$. In special
setups where (usually marginal) constraints relate attributes according to the
topology of a tree, there even exists [48] a closed-form solution for the
maxent distribution. Beyond tree topologies iterative algorithms such as
(generalized) belief propagation [49] have been used in the maxent context for
approximate inference in higher dimensional problems.

More generally, starting from some reference distribution
${\boldsymbol{\upsilon}}$ we iteratively impose constraints Eq. (36) that we
summarize in a vector function $\mathbf{F}$ with

$\displaystyle
F_{\alpha}[\mathfrak{p}]=\langle(\mathfrak{p}-\mathfrak{f})\,\hat{M}_{\alpha}\rangle=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}m_{\alpha}({\boldsymbol{e}})\left(p_{{\boldsymbol{e}}}-f_{{\boldsymbol{e}}}\right)\quad\text{for}\quad\alpha=1,\ldots
D~{},$ (60)

where $p_{\boldsymbol{e}}$ are parametrized according to Eq. (55). A well-
known numerical method to find the root in the resulting non-linear system of
coupled equations is the Newton-Raphson algorithm, which has inspired many
iterative methods [50] for approximate inference. In probability space, we can
covariantly formulate (convergence discussed in B.3)

######  Theorem 3.4.

The iterative algorithm to covariantly approach the probabilities of the
$I$-projection of ${\boldsymbol{\upsilon}}$ onto
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$, expressed for $n\in\mathbb{N}$ in
matrix-operator notation

$\displaystyle\mathfrak{p}^{(0)}=\boldsymbol{\upsilon}\,\mathbb{I}_{\mathcal{E}_{\star}}\quad,\quad\mathbf{J}^{(n)}=\langle\mathbf{\,\hat{M}}\,\mathfrak{p}^{(n)}\mathbf{\,\hat{M}^{T}}\rangle\quad\text{and}\quad\mathfrak{p}^{(n+1)}=\exp\left\\{-\mathbf{\,\hat{M}^{T}}(\mathbf{J}^{(n)})^{-1}\mathbf{F}[\mathfrak{p}^{(n)}]\right\\}\mathfrak{p}^{(n)}$

or component-wise as

$\displaystyle p^{(0)}_{\boldsymbol{e}}=$
$\displaystyle\,\,\upsilon_{\boldsymbol{e}}$ for
$\displaystyle{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}$ (61)
$\displaystyle J_{\alpha\beta}=$
$\displaystyle\,\,\langle\,\hat{M}_{\alpha}\,\mathfrak{p}^{(n)}\,\hat{M}_{\beta}\rangle=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}m_{\alpha}({\boldsymbol{e}})\,p_{\boldsymbol{e}}^{(n)}\,m_{\beta}({\boldsymbol{e}})$
for $\displaystyle\alpha,\beta=1,\ldots,D$ $\displaystyle
p^{(n+1)}_{\boldsymbol{e}}=$
$\displaystyle\,\,p^{(n)}_{\boldsymbol{e}}\exp\left\\{-\sum_{\alpha,\beta=1}^{D}m_{\alpha}({\boldsymbol{e}})\left(\mathbf{J}^{-1}\right)_{\alpha\beta}F_{\beta}[\mathfrak{p}^{(n)}]\right\\}$
for $\displaystyle{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}$

is well-behaved, whenever
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}$ is
reasonbly close to reference distribution ${\boldsymbol{\upsilon}}$.

The corresponding update rule to numerically determine the Lagrange
multipliers in Eq. (55) reads

$\displaystyle\theta^{(n+1)}_{\alpha}=\theta^{(n)}_{\alpha}-\sum_{\beta=1}^{D}\left(\mathbf{J}^{-1}\right)_{\alpha\beta}F_{\beta}[\mathfrak{p}^{(n)}]~{}.$
(62)

Concerning the value $F^{(n)}_{\alpha}\equiv F_{\alpha}[\mathfrak{p}^{(n)}]$
of the vector function Eq. (60) whose unique root we are after, we recognize
after $n$ iterations by the scalar product

$\displaystyle\sum_{\alpha=1}^{D}F_{\alpha}^{(n)}\left(\theta^{(n+1)}_{\alpha}-\theta^{(n)}_{\alpha}\right)=-\sum_{\alpha,\beta=1}^{D}F_{\alpha}^{(n)}\left(\mathbf{J}^{-1}\right)_{\alpha\beta}F_{\beta}^{(n)}<0\quad\forall\,n\quad\text{s.t.}\quad
F^{(n)}_{\alpha}\neq 0$

using Eq. (62) and positive-definiteness of $\mathbf{J}^{-1}$ that the
procedure Eq. (61) moves along a descent direction until the desired
convergence.

The covariance of algorithm 3.4 manifests due to the affine invariance [51] of
Newton-Raphson method. A physically equivalent re-parametrization of measuring
operations performed via $\mathbf{T}$ in Eq. (34) relates two fapp-equivalent
vector operators constructing the same TotEmplex, hence leading to the same
$I$-projection of $\boldsymbol{\upsilon}$. Accordingly, constraining function
Eq. (60) covariantly transforms under $\mathbf{T}$ as a vector,

$\displaystyle
F_{\alpha}\quad\rightarrow\quad\sum_{\beta=1}^{D}T_{\alpha\beta}F_{\beta}~{},$
(63)

while the Jacobian $\mathbf{J}$ as a matrix,

$\displaystyle\mathbf{J}_{\alpha\beta}\quad\rightarrow\quad\sum_{\alpha^{\prime}\beta^{\prime}=1}^{D}T_{\alpha,\alpha^{\prime}}\,J_{\alpha^{\prime}\beta^{\prime}}\,T_{\beta,\beta^{\prime}}~{}.$
(64)

In total, we readily verify that the update rule of Eq. (61) remains invariant
under re-parametrization of measuring operations Eq. (34) exemplifying the
merits of working in probability space. Not only this property
programmatically streamlines the generation of model distributions, but helps
setup approximate algorithms [48, 42] – independently of the current framework
– for inference in higher-dimensional problems where the np-completeness
manifests.

#### Chain of $I$-projections in nested descriptions

Since the Newton-Raphson estimate of 3.4 remains agnostic to the concrete
parametrization of Theorem 3.2, convergence of the iterative procedure
solely888At the technical level, the inversion of the Jacobian at each Newton-
Raphson step might be poorly conditioned for some parametrizations of
constraints eventually de-railing the iterative estimate. depends on the
physical setup. If the reference distribution ${\boldsymbol{\upsilon}}$ poorly
captures phenomenology $\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$
resulting into higher $I$-divergence $D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})$,
the algorithm might fail to converge to the $I$-projection, as generally
expected by Newton-based methods [51, 52].

######  Definition 3.5 (nested descriptions).

Consider two irreducible constructing elements $\mathbf{\,\hat{M}}$ and
$\mathbf{\,\hat{M}^{\prime}}$ over $\mathcal{E}_{\star}$ related for
$D<D^{\prime}$ through

$\displaystyle\,\hat{M}_{\alpha}=\sum_{\alpha^{\prime}=1}^{D^{\prime}}S_{\alpha\alpha^{\prime}}\,\hat{M}^{\prime}_{\alpha^{\prime}}\quad\text{for}\quad\alpha=1,\ldots,D$
(65)

where $\text{rank}\,\mathbf{S}=D$ (otherwise $\mathbf{\,\hat{M}}$ would be
row-rank deficit). The TotEms defined by phenomenologies
$\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$ and
$\langle\mathfrak{f}\mathbf{\,\hat{M}^{\prime}}\rangle$ and the resulting
TotEmlices
$[\mathfrak{f};{\mathbf{\,\hat{M}^{\prime}}}]\subseteq[\mathfrak{f};{\mathbf{\,\hat{M}}}]$
are called nested.

The fact that the induced TotEmplices are subsets can be readily seen from
linearity of expectations:

$\displaystyle{\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}^{\prime}}}]:\quad\langle\left({\mathfrak{p}}-{\mathfrak{f}}\right)\,\hat{M}_{\alpha}\rangle=\sum_{\alpha^{\prime}=1}^{D^{\prime}}S_{\alpha\alpha^{\prime}}\underbrace{\langle\left({\mathfrak{p}}-{\mathfrak{f}}\right)\,\hat{M}^{\prime}_{\alpha^{\prime}}\rangle}_{=0}=0\quad\Rightarrow\quad{\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]~{}.$

The more knowledgeable we become about the provided data via additional
measurements performed as linearly independent $\mathfrak{f}$-expectations,
the further we generically move away from ${\boldsymbol{\upsilon}}$, since

$\displaystyle D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};\mathbf{\,\hat{M}}^{\prime}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})\geq
D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})$
(66)

by construction 3.1 of the $I$-projection of ${\boldsymbol{\upsilon}}$ onto
ambient TotEmplex $[\mathfrak{f};\mathbf{\,\hat{M}}]$ where
$\mathbb{P}_{[\mathfrak{f};\mathbf{\,\hat{M}}^{\prime}]}\boldsymbol{\upsilon}$
also belongs to.

Exploiting this fact, we might still try to numerically determine the
$I$-projection of ${\boldsymbol{\upsilon}}$ onto
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ constructed by some vector operator
$\mathbf{\,\hat{M}}$, for which the iterative algorithm of 3.4 failed to
converge, by progressively projecting onto nested TotEmplices. Specifically,

######  Theorem 3.6.

Partition $\mathbf{\,\hat{M}}$ into $d\leq D$ constructing elements
$\mathbf{\,\hat{M}^{(\nu)}}$ inducing nested TotEms in the sense of 3.5,

$\displaystyle[\mathfrak{f};{\mathbf{\,\hat{M}}}]\subseteq\ldots[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu)}}}]\subseteq\ldots[\mathfrak{f};{\mathbf{\hat{M}}^{(1)}}]\subseteq\mathcal{P}(\mathcal{E}_{\star})~{},$
(67)

where eventually $\mathbf{\hat{M}}^{(d)}$ is fapp equivalent to the given
$\mathbf{\,\hat{M}}$. Starting from reference
$\boldsymbol{\upsilon}^{(0)}=\boldsymbol{\upsilon}\,\mathbb{I}_{\mathcal{E}_{\star}}\in\mathcal{P}(\mathcal{E}_{\star})$,
we successively use iterative algorithm 3.4 to determine the $I$-projection of

$\displaystyle\boldsymbol{\upsilon}^{(\nu-1)}=\mathbb{P}_{[\mathfrak{f};\mathbf{\,\hat{M}^{(\nu-1)}}]}\cdots\mathbb{P}_{[\mathfrak{f};\mathbf{\hat{M}}^{(1)}]}\boldsymbol{\upsilon}\,\in[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu-1)}}}]$
(68)

onto inner TotEmplex $[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu)}}}]$ for
$\nu=2,\ldots,d$. Eventually, the last estimate coincides with the desired
$I$-projection,
$\boldsymbol{\upsilon}^{(d)}=\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}$.

After $\nu$ steps, trying to numerically find the distribution on
$[\mathfrak{f};{\mathbf{\hat{M}}^{(\nu+1)}}]$ closest to updated reference
distribution ${\boldsymbol{\upsilon}^{(\nu)}}$ (itself living on the ambient
$[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu)}}}]$) is expected to exhibit better
convergence properties than directly projecting from
${\boldsymbol{\upsilon}^{(\nu-1)}}$ onto inner TotEmplex
$[\mathfrak{f};{\mathbf{\hat{M}}^{(\nu+1)}}]$. By merit of Corollary 3.3
applied within $[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu)}}}]$ where
$\boldsymbol{\upsilon}^{(\nu)}=\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(\nu)}}]}\boldsymbol{\upsilon}^{(\nu-1)}$
is the $I$-projection of $\boldsymbol{\upsilon}^{(\nu-1)}$ and
$\boldsymbol{\upsilon}^{(\nu+1)}=\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu+1)}}}]}\boldsymbol{\upsilon}^{(\nu)}\in[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu+1)}}}]\subset[\mathfrak{f};{\mathbf{\,\hat{M}^{(\nu)}}}]$
we recognize that

$\displaystyle D\hskip
0.28453pt(\boldsymbol{\upsilon}^{(\nu+1)}\;\|\;\boldsymbol{\upsilon}^{(\nu-1)})=D\hskip
0.28453pt(\boldsymbol{\upsilon}^{(\nu)}\;\|\;\boldsymbol{\upsilon}^{(\nu-1)})+D\hskip
0.28453pt(\boldsymbol{\upsilon}^{(\nu+1)}\;\|\;\boldsymbol{\upsilon}^{(\nu)})\geq
D\hskip
0.28453pt(\boldsymbol{\upsilon}^{(\nu+1)}\;\|\;\boldsymbol{\upsilon}^{(\nu)})~{}.$

Pipeline Eq. (68) breaks the original problem of constrained minimization of
the $I$-divergence into root-finding sub-routines whose target lies closer to
initial estimate. This only makes sense iff the consecutive determination of
$I$-projections onto nested TotEmplices is equivalent to directly taking the
$I$-projection of outermost reference distribution onto the inner-most
TotEmplex. Indeed, this is asserted in B.4.

#### Iterative Proportional Fitting

There exist many variations of the main iterative scheme. Suppose that $M$
marginal, not necessarily linearly independent operators
$\,\mathbb{I}_{\alpha}$ with eigenvalues like in Eq. (24) are provided. Their
combined action on entities can always be encoded by a binary matrix
$\mathbf{C}$, such that

$\displaystyle\langle\mathfrak{p}\,\mathbb{I}_{\alpha}\rangle=\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}C_{\alpha,{\boldsymbol{e}}}\,p_{\boldsymbol{e}}~{}.$
(69)

Cycling through each row depicting a marginal constraint for $\alpha=1,\ldots
M$ to iteratively update probabilities via one-dimensional Newton method (as
the multidimensional Newton method would be plagued by the singularity of
redundancies) gives after $n$ cycles the rule

$\displaystyle
p^{(nM+\alpha)}_{\boldsymbol{e}}=p^{(nM+\alpha-1)}_{\boldsymbol{e}}\exp\left\\{C_{\alpha,{\boldsymbol{e}}}\left(\frac{\langle\mathfrak{f}\,\mathbb{I}_{\alpha}\rangle}{\langle\mathfrak{p}^{(nM+\alpha-1)}\,\mathbb{I}_{\alpha}\rangle}-1\right)\right\\}\quad\forall\,{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}~{}.$
(70)

Using $C_{\alpha,{\boldsymbol{e}}}^{2}=C_{\alpha,{\boldsymbol{e}}}$ in the
series expansion of the exponential up to first order999Equivalently, we could
set
$\langle\mathfrak{p}^{(nM+\alpha-1)}\,\mathbb{I}_{\alpha}\rangle=\langle\mathfrak{f}\,\mathbb{I}_{\alpha}\rangle/x$
to Taylor-expand around $x=1$ up to first order. gives

$\displaystyle p^{(nM+\alpha)}_{\boldsymbol{e}}\approx
p^{(nM+\alpha-1)}_{\boldsymbol{e}}\left(\frac{\langle\mathfrak{f}\,\mathbb{I}_{\alpha}\rangle}{\langle\mathfrak{p}^{(nM+\alpha-1)}\,\mathbb{I}_{\alpha}\rangle}\right)^{C_{\alpha,{\boldsymbol{e}}}}~{}.$
(71)

One can show [53, 54] that the Taylor-expanded version always converges to the
desired $I$-projection starting from a sensible
$\mathfrak{p}^{(0)}=\boldsymbol{\upsilon}$. This approach leads to the
algorithm of iterative proportional fitting [55] with its own long history
[56, 57, 58, 15, 59, 9].

###  3.4 The distribution of phenomenology-compatible probability fluctuations

After establishing existence and uniqueness of the $I$-projection
$\mathfrak{q}\equiv\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}$
given prior knowledge and a TotEm of the data, we estimate the probability to
sample distributions on the TotEmplex by performing a large-$N$ expansion in
the probability fluctuations around the $I$-projection.

First, let us examine – again in the discrete problem – the relative abundance
of a count vector predominantly sampled from $\boldsymbol{\upsilon}$ over
another count vector in the Diophantine class. Following the logic of Section
3.1, such ratio can be well represented by

$\displaystyle\max_{\mathfrak{p}^{\prime}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\log\frac{{\text{mult}}(N\mathfrak{p}^{\prime};\boldsymbol{\upsilon})}{{\text{mult}}(N\mathfrak{p};\boldsymbol{\upsilon})}\overset{N\gg
1}{\approx}N\left[D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})-D\hskip
0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})\right]\geq 0~{}.$ (72)

This clearly demonstrates that counts close to $N\mathfrak{q}$ should be
overwhelmingly anticipated given the empirical constraints from phenomenology
2.3 and compatible prior knowledge 2.1, both at finite101010The finite-$N$
combinatorics of the Diophantine problem and the associated notion of typical
count vector(s) are explored in [9]. and large $N$ and the latest
asymptotically as $N\rightarrow\infty$. The probability of any other member
vector $N\mathfrak{p}$ whose $I$-divergence from $\boldsymbol{\upsilon}$
differs by more than $1/N$ from $D\hskip
0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})$ would be exponentially
suppressed.

Intuitively, the dominating role of $\mathfrak{q}$ in the equivalence class
explains why the $I$-projection is the most typical of the TotEm based on our
prior knowledge. Intuitively, this is perhaps best understood starting from
minimal prior knowledge, when the minimization of $I$-divergence reduces to
the maxent. In that case, the ratio of multinomial coefficients for some count
vector in the Diophantine class,

$\displaystyle\log\frac{W[N\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\mathfrak{u})]}{W[N\mathfrak{p}]}\overset{N\gg
1}{\approx}N\left(H[\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\mathfrak{u}]-H[\mathfrak{p}]\right)~{},$

clearly shows that any count vector closest to
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\mathfrak{u}$, which is the
least biased given the constraints [60], can be realized in the most
combinatorial ways, as the entropy difference will be by definition positive.

Regarding the probability density at some
$\mathfrak{p}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ in the continuum, the
combinatorics of Section 3.1 have (approximately) motivated Eq. (50), whose
$N$-leading behaviour can be rewritten by Corollary 3.3 as

$\displaystyle f(\mathfrak{p})=Z^{-1}\,\exp\left\\{-ND\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})\right\\}+\ldots$ (73)

with redefined partition function

$\displaystyle
Z=\int_{{\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\exp\left\\{-ND\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})\right\\}~{}.$ (74)

To systematically investigate this probability density function
$(\textsc{pdf})$ over member distributions on the TotEmplex constructed by
$\mathbf{\,\hat{M}}$ on data described by $\mathfrak{f}$, we exploit the moral
of the discrete observations above.

#### Probability fluctuations

Controlled deviations around the special point on the TotEmplex represented by
the $I$-projection would prove particularly useful in estimating the
aforementioned pdf. Such probability fluctuations $\boldsymbol{\pi}$ around
the $I$-projection of ${\boldsymbol{\upsilon}}$ onto the TotEmplex span the
appropriate polyhedral cone such that

$\displaystyle\mathfrak{p}=\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}+{\boldsymbol{\pi}}$
(75)

describes a valid distribution from $\mathcal{P}(\mathcal{E}_{\star})$ with
non-negative probabilities $p_{\boldsymbol{e}}\geq 0$. Since by definition
$\langle\boldsymbol{\pi}\mathbf{\,\hat{M}}\rangle=0$, fluctuation operator
$\boldsymbol{\pi}\in\ker\mathbf{\,\hat{M}}$ can be accommodated along the
“normal” basis of Eq. (28),

$\displaystyle\boldsymbol{\pi}=\sum_{a=1}^{|\ker\mathbf{\,\hat{M}}|}\boldsymbol{\pi}_{a}\,\hat{N}_{a}~{}.$
(76)

As we are going to assert below, a large-$N$ expansion in the neighbourhood of
the $I$-projection enables us to extract the universal behavior of metrics in
Section 2.1 without worrying about the conic boundaries. Since the
$I$-projection lives well in the interior of the TotEmplex (cf. Theorem 3.2),
we anticipate to always find some parametrization that does not degenerate,
thus ensuring that probability fluctuations
$\boldsymbol{\pi}_{a}=\mathcal{O}\left(1\right)$.

Due to convexity, any distribution on the TotEmplex can be reached by
fluctuating around some base point. Consequently, the study of probability
density Eq. (73) becomes equivalent to investigating how fluctuations
distribute, $f(\mathfrak{p})\equiv f(\boldsymbol{\pi})$. Studying fluctuations
around some arbitrary base distribution on the TotEmplex however, would not
help much in computing the pdf. On the other hand, fluctuating around the
$I$-projection of our reference distribution onto the TotEmplex enables to
formulate

######  Theorem 3.7 (Gaussian approximation).

Consider a TotEm constructed by $\mathbf{\,\hat{M}}$ over
$\mathcal{E}_{\star}$ and the $I$-projection of reference distribution
$\boldsymbol{\upsilon}$ onto the TotEmplex. At large $N$, probability
fluctuations in $\ker\mathbf{\,\hat{M}}$ around the $I$-projection of
$\boldsymbol{\upsilon}$ are controlled by Gaussian approximation in the sense
of predominantly following a $|\ker\mathbf{\,\hat{M}}|$-dimensional normal
distribution of zero mean and $\mathcal{O}\left(1/N\right)$ variance.

#### Proof

After deriving the Gaussian kernel that drives probability fluctuations around
the $I$-projection, an integral representation for the partition function Eq.
(74) of the $N$-leading pdf would be naturally motivated. Subsequently, the
universal character of the expansion would be self-consistently guaranteed by
showing suppression at large $N$ of any higher-order corrections to the pdf
over the TotEmplex. For that, we need (proof in B.5)

######  Lemma 3.8.

The $I$-divergence of any distribution
${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ from the $I$-projection
of ${\boldsymbol{\upsilon}}$ onto $[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ is
universally (i.e. independently of the phenomenological problem) determined in
the vicinity of the $I$-projection by a quadratic form $\mathcal{K}$ over the
kernel of $\mathbf{\,\hat{M}}$:

$\displaystyle D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon})=\mbox{$\frac{1}{2}$}\mathcal{K}(\boldsymbol{\pi})+\mathcal{O}\left(\boldsymbol{\pi}^{3}\right)\quad\text{with}\quad\mathcal{K}(\boldsymbol{\pi})=\langle\boldsymbol{\pi}\mathfrak{q}^{-1}\boldsymbol{\pi}\rangle$
(77)

The resulting precision matrix $\mathcal{K}(\boldsymbol{\pi})$ is positive
definite for the reasons outlined in Lemma B.3 given irreducible vector
operator $\mathbf{\,\hat{N}}$ spanning the kernel of $\mathbf{\,\hat{M}}$.
Hence, a Gaussian approximation to $f(\mathfrak{p})\equiv f(\boldsymbol{\pi})$
in kernel space around the $I$-projection is always well-defined. In turn,
this motivates the evaluation of the partition function $Z$ over the TotEmplex
via a multidimensional Gaussian integral whose limits can be extended to
infinity, the error being exponentially suppressed in $N$.

Furthermore, Laplace approximation in Eq. (73) ensures that higher-order
fluctuations are polynomially suppressed in $N$, turning the expansion in
fluctuations Eq. (77) around the $I$-projection into a large-$N$ expansion.
The same applies for higher probability-dependent terms in the large-$N$
expansion Eq. (44) of multinomial distribution. Generically, these are
expected to appear at $\mathcal{O}\left(N^{-1/2}\right)$. In total, we can
self-consistently verify order-by-order in large-$N$ the universal form

$\displaystyle
f(\boldsymbol{\pi})=Z^{-1}\exp\left\\{-N\mathcal{K}(\boldsymbol{\pi})/2\right\\}+\mathcal{O}\left(N^{-1/2}\right)$
(78)

of $N$-leading pdf of probability fluctuations around the $I$-projection on
the TotEmplex. ∎

As perhaps anticipated by the law of large numbers, probability fluctuations
in a TotEm become at sufficiently large sample size almost normally
distributed, once expanded around the $I$-projection of our reference
distribution onto the TotEmplex. The dominating role of the $I$-projection
discussed in Eq. (72) is quantitatively reflected on the variance of the
multivariate normal distribution which quickly penalizes distributions whose
$I$-divergence from
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}$ is
larger than $\mathcal{O}\left(1/N\right)$. In other words using Lemma 3.8
together with Eq. (78) and Eq. (203), the expected $I$-divergence of
distributions on the TotEmplex from the $I$-projection scales as

$\displaystyle\int_{\mathfrak{p}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]}f(\mathfrak{p})D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon})\approx\prod_{a=1}^{|\ker\mathbf{\,\hat{M}}|}\int\text{d}\boldsymbol{\pi}_{a}f(\boldsymbol{\pi})\,\mbox{$\frac{1}{2}$}\mathcal{K}(\boldsymbol{\pi})=\mathcal{O}\left(1/N\right)~{}.$
(79)

In the context of maxent principle, this observation translated by Corollary
3.3 and Eq. (59) into an entropy difference,

$\displaystyle D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\mathfrak{u})=D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{u})-D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\mathfrak{u}\;\|\;\mathfrak{u})=H[\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\mathfrak{u}]-H[\mathfrak{p}]~{},$
(80)

led authors talk [61] about the concentration of entropies with increasing $N$
around the maxent distribution.

According to Lemma 3.8, the quadratic form which controls probability
fluctuations around the $I$-projection only depends on the $I$-divergence of
TotEm distributions from the $I$-projection. This universal fact always
becomes relevant as the sample size grows. In the reminder of this paragraph,
we exploit the geometric symmetry of the Gaussian approximation to translate
Theorem 3.7 into operationally useful prescriptions to eventually select
appropriate operators in the following section.

######  Corollary 3.9.

On the TotEmplex, consider the probability of a distribution
${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ with definite
$I$-divergence from the $I$-projection
$\mathfrak{q}\equiv\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}$
given prior knowledge ${\boldsymbol{\upsilon}}$. To leading order in the
large-$N$ expansion, the probability density enjoys spherical symmetry in
$k\equiv|\ker\mathbf{\,\hat{M}}|$ dimensions with radial density

$\displaystyle\rho(\mathfrak{p})\equiv\rho(D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q}))=\frac{1}{2}\left(\frac{N}{\pi}\right)^{k/2}D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})^{k/2-1}\exp\left\\{-ND\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})\right\\}~{},$ (81)

dictated by the specified $I$-divergence $D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})$ which plays the role of radius
squared in $\ker\mathbf{\,\hat{M}}$.

The way to read off the radial density from Eq. (73) is outlined in B.6.
According to probability density $\rho$ any distribution on the TotEmplex
living on the $(k-1)$-sphere of squared radius $D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})$ will be assigned the same weight
from the perspective of reference distribution.

To connect Corollary 3.9 with more established conventions in the literature,
we trivially execute the angular integration in Eq. (81) (cf. Eq. (134)) to
obtain a cumulative radial probability

$\displaystyle\text{d}D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})\,\frac{N^{k/2}}{\Gamma(k/2)}D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})^{k/2-1}\,\exp\left\\{-ND\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})\right\\}$ (82)

on the $(k-1)$-sphere. Subsequently, we set

$\displaystyle Q=2ND\hskip 0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})$ (83)

to immediately arrive at the chi-squared probability in $k$ dimensions

$\displaystyle\text{d}Q\,\frac{1}{2^{k/2}\Gamma(k/2)}Q^{k/2-1}\,e^{-Q/2}\quad\text{for}\quad
Q\geq 0~{},$ (84)

which selects any distribution
${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ whose $I$-divergence
from the $I$-projection lies on a spherical shell of radius $\frac{Q}{2N}$ and
thickness $\text{d}Q$. This describes a chi-squared distribution with
$k\equiv|\ker\mathbf{\,\hat{M}}|$ degrees of freedom and statistics given by
Eq. (83).

Due to the non-linear optimization problem leading to Theorem 3.2, it becomes
evident that the $I$-projection, which crucially depends on reference
distribution ${\boldsymbol{\upsilon}}$, does not generically coincide with the
notion of a mean distribution over the TotEmplex. Already with
$|\mathcal{E}|=3$ entities, it is easy to find counter-examples both in the
discrete and continuous case. Consequently, the developed large-$N$ expansion
cannot be understood as mere application of central limit theorem [62] on
probabilities – treated themselves as $|\mathcal{E}|$ random variables. By the
same token, the variance of probability distributions implied by Eq. (79) does
not coincide with the “naive” variance from a mean probability on the
TotEmplex.

##  4 The classification of phenomenologies

Starting from prior knowledge about subjects, we have so far motivated – given
some phenomenology 2.3 of the data – the distribution of compatible
probability fluctuations around the $I$-projection of reference distribution
onto the TotEmplex. Performing however different measurements on the same
dataset, i.e. taking $\mathfrak{f}$-expectations of different vector operators
$\mathbf{\,\hat{M}}$ would yield a different phenomenology, generically
constructing a distinct TotEm. In this Section, we describe a way to select
appropriate operators inducing optimally descriptive phenomenologies given
prior knowledge and the data. Ultimately, we relate to familiar notions of
statistical testing in our unified framework of large-$N$ description.

#### Quotient space

An often encountered task in data analysis is to assess the amount of
knowledge gained by performing additional measurements through
$\mathfrak{f}$-expectations. In other words, we wish to decide whether a more
detailed phenomenology $\langle\mathfrak{f}\mathbf{\,\hat{M}^{\prime}}\rangle$
extracts from the data additional valuable knowledge on the investigated
subjects compared to more plain phenomenology
$\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$. From the perspective of our
prior knowledge, we would have to compare the ambient TotEmplex constructed by
vector operator $\mathbf{\,\hat{M}}$ to the nested TotEmplex constructed by
$\mathbf{\,\hat{M}^{\prime}}$ which fapp implies the former in the sense of
definition 3.5.

Under the action of $\mathbf{\,\hat{M}^{\prime}}$ probability operators in the
ambient TotEmplex $[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ get identified
according to

$\displaystyle{\mathfrak{p}},{\mathfrak{p}^{\prime}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]~{}:\quad{\mathfrak{p}^{\prime}}\sim{\mathfrak{p}}\quad\Longleftrightarrow\quad{\mathfrak{p}^{\prime}}\in[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]\subset[\mathfrak{f};{\mathbf{\,\hat{M}}}].$
(85)

This means that we should not distinguish between two distributions that both
satisfy the phenomenology $\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$, if
they also agree on the higher statistics captured by the phenomenology
$\langle\mathfrak{f}\mathbf{\,\hat{M}^{\prime}}\rangle$. Conversely, two
distributions that differ by some expectations in
$\langle\mathfrak{f}\mathbf{\,\hat{M}^{\prime}}\rangle$ are called
$\mathbf{\,\hat{M}^{\prime}}$-distinct. The set of
$\mathbf{\,\hat{M}^{\prime}}$-induced equivalence classes defines the quotient
space $[\mathfrak{f};{\mathbf{\,\hat{M}}}]/\mathbf{\,\hat{M}^{\prime}}$ of the
ambient TotEmplex. Theorem 3.2 motivates a parametrization of the set of
$\mathbf{\,\hat{M}^{\prime}}$-distinct operators where Lagrange multipliers
continuously vary over representatives in the equivalence classes of the
quotient space.

Having access to the full data X we can construct orbits of nested TotEms over
$\mathcal{E}_{\star}$,

$\displaystyle\\{\mathfrak{f}\\}\equiv[\mathfrak{f};{\text{\small
1}\text{1}}]\subseteq\ldots\subseteq[\mathfrak{f};{\mathbf{\,\hat{M}^{\prime}}}]\subseteq[\mathfrak{f};{\mathbf{\,\hat{M}}}]\subseteq\ldots\subseteq[\mathfrak{f};{\hat{I}}]\equiv\mathcal{P}(\mathcal{E}_{\star})~{},$
(86)

which interpolate between learning empirical frequencies by heart via the
identity operator Eq. (18) and not learning anything from the data except
normalization condition Eq. (11). Traversing the chain of nested TotEmplices
from right to left, the amount of valuable knowledge that each phenomenology
2.3 at fixed sample size $N$ additionally conveys depends on the underlying
(usually unknown) laws that govern the investigated subjects as well as on our
prior knowledge about them.

#### The information score

Given the data, we first motivate a score to systematically benchmark from the
perspective of prior knowledge distinct TotEm descriptions within an orbit
like Eq. (86), but also among different not necessarily nested TotEms. For
that, we proceed with

######  Definition 4.1.

To each TotEm we associate at sufficiently large sample size $N$ the
information score

$\displaystyle\textsc{s}core_{I}=-ND\hskip
0.28453pt(\mathfrak{f}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon})+\frac{|\ker\mathbf{\,\hat{M}}|}{2}\log
N+\mathcal{O}\left(1\right)~{},$ (87)

in terms of the $I$-projection
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}$ onto
the TotEmplex $[\mathfrak{f};{\mathbf{\,\hat{M}}}]$.

Clearly, this information score corresponds to the logarithm of the weight
assigned to empirical distribution $\mathfrak{f}$ by the radial density 3.9
over $\ker\mathbf{\,\hat{M}}$ that governs probability fluctuations around the
$I$-projection onto the TotEmplex. Concretely, the two leading terms determine
how radial probability Eq. (82) would change by an infinitesimal variation in
the vicinity of $D\hskip
0.28453pt(\mathfrak{f}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon})$.

In our data-centric approach, the information criterion implied by 4.1
crucially depends on the $I$-projection as well as the degrees of freedom on
the TotEmplex to covariantly assign weights to TotEms, i.e. to operator
collections given $\mathfrak{f}$ and $\boldsymbol{\upsilon}$. Pragmatically,
this becomes useful by

######  Proposition 4.1 ($I$-score).

Over a set of fapp inequivalent vector operators 2.2, the principle of Maximum
Likelihood dictates to select the vector operator with highest information
score,

$\displaystyle\mathbf{\,\hat{M}}_{\textsc{mle}}=\max_{\mathbf{\,\hat{M}}}\textsc{s}core_{I}$

which maximizes the likelihood of the data in the respective TotEm.

#### The distribution over the quotient space

Comparing two nested (in the sense of Definition 3.5) TotEms constructed by
distinct phenomenologies $\langle\mathfrak{f}\mathbf{\,\hat{M}}\rangle$ and
$\langle\mathfrak{f}\mathbf{\,\hat{M}^{\prime}}\rangle$ can be additionally
performed via hypothesis testing. For that, we need to compute the probability
of a TotEmplex, i.e. the cumulative probability to sample any distribution on
the TotEmplex. In particular, we consider inner TotEmplex
$[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]\subset[\mathfrak{f};{\mathbf{\,\hat{M}}}]$
represented by $\mathbf{\,\hat{M}^{\prime}}$-distinct distribution
$\mathfrak{p}$. Emphasizing the importance of the $I$-projection to leading
order in $N$, the probability to sample
$[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]$ in the quotient space
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]/\mathbf{\,\hat{M}^{\prime}}$ is dictated
by

$\displaystyle Z^{-1}\exp\left\\{-ND\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]}\boldsymbol{\upsilon}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon})\right\\}~{},$
(88)

as derived in Lemma B.5. The exponential factor can also be understood as a
ratio of $N$-leading probability mass Eq. (213) of inner TotEmplex to the
probability mass of ambient TotEmplex upon using Corollary 3.3.

In accordance with the quotient action, probability fluctuations in Eq. (76)
along $\mathbf{\,\hat{N}}$ (spanning the ambient kernel of
$\mathbf{\,\hat{M}}$) can be always decomposed into
$|\ker\mathbf{\,\hat{M}^{\prime}}|$ directions $\,\hat{N}_{a^{\prime}}$ along
inner $\ker\mathbf{\,\hat{M}^{\prime}}$ and remaining normal directions
$\,\hat{N}_{a}$ with $\langle\,\hat{N}_{a}\,\hat{N}_{a^{\prime}}\rangle=0$.
Investigating the distribution of probability fluctuations along latter
directions it is straight-forward to verify Eq. (88) and formulate (proven in
B.7)

######  Theorem 4.2.

In the quotient space induced by $\mathbf{\,\hat{M}^{\prime}}$ over
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ consider inner TotEmplex
$[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]$ with $I$-projection
$\mathbb{P}_{[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]}\boldsymbol{\upsilon}$.
We examine the cumulative probability to sample from $\boldsymbol{\upsilon}$
any inner TotEmplex whose $I$-projection has a fixed $I$-divergence $D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]}\boldsymbol{\upsilon}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon})$
from the ambient $I$-projection. At large $N$, this probability is dictated by
the chi-squared distribution of $2ND\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{p};{\mathbf{\,\hat{M}^{\prime}}}]}\boldsymbol{\upsilon}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon})$
with $D^{\prime}-D$ degrees of freedom.

Whenever $\mathbf{\,\hat{M}^{\prime}}=\text{\small 1}\text{1}$,
$[\mathfrak{p};{\text{\small 1}\text{1}}]=\\{\mathfrak{p}\\}$ so that
probability density Eq. (88) reduces back to Eq. (73) and Theorem 4.2 the chi-
squared distribution Eq. (84) below Corollary 3.9. Again, we are interested in
the weight assigned within $[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ to the
empirical equivalence class $[\mathfrak{f};{\mathbf{\,\hat{M}^{\prime}}}]$, so
that

######  Proposition 4.2 ($I$-test).

At significance level $\alpha$, we declare nested
$\mathbf{\,\hat{M}^{\prime}}$ to extract significantly new information from
data over constructing element $\mathbf{\,\hat{M}}$, when

$\displaystyle 1-\textsc{cdf}_{\chi^{2}_{D^{\prime}-D}}(2ND\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}^{\prime}}}]}\boldsymbol{\upsilon}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}))<\alpha~{}.$
(89)

###  4.1 The (self-)consistency of large-$N$ expansion

The developed formalism needs neither to assume that a reference distribution
is close to the true distribution underlying the generation of data, nor
relies on appropriate measurements to unveil the truth. In fact, it does not
need to adhere to the very notion of some ground truth. In that sense, it is a
purely effective description driven by data after deciding about attributes
and their domains. Propositions 4.1 and 4.2 simply indicate those measurements
that inform us about systematic trends – beyond our prior knowledge – as
evidenced by the data at a given sample size.

To further support this model- and truth-agnostic formulation of operator
selection at large-$N$, we now adopt the more familiar viewpoint that there
exists a well-defined system – governed by some stochastic laws – which
generates data. In the following, we denote by ${\mathfrak{t}}$ the target
distribution that underlies data generation. Furthermore, we signify by
$\mathcal{T}_{0}$ the set of non-realizable entities so that
$\mathcal{T}_{\star}=\mathcal{E}\setminus\mathcal{T}_{0}$. Given a compatible
($\boldsymbol{\upsilon}_{\boldsymbol{e}}=0$ means
$|{\boldsymbol{e}}\rangle\in\mathcal{T}_{0}$) reference distribution
${\boldsymbol{\upsilon}}\in\mathcal{P}$, we can always move from
$\boldsymbol{\upsilon}$ to $\mathfrak{t}$ by applying the translation operator
(c.f. Eq. (3.2)),

$\displaystyle\mathfrak{t}=\exp\left\\{\boldsymbol{\tau}\cdot\mathbf{\,\hat{T}}\right\\}\boldsymbol{\upsilon}=\left(\,\mathbb{I}_{\mathcal{T}_{\star}}+\boldsymbol{\tau}\cdot\mathbf{\,\hat{T}}+\mbox{$\frac{1}{2}$}(\boldsymbol{\tau}\cdot\mathbf{\,\hat{T}})^{2}+\ldots\right)\boldsymbol{\upsilon}~{}.$
(90)

Irreducible vector operator
$\mathbf{\,\hat{T}}\in\mathcal{B}(\mathcal{T}_{\star})$ describes the number
of measurements $\Delta=\text{rank}\,\mathbf{\,\hat{T}}$ that are required to
learn the target on top of reference distribution
$\boldsymbol{\upsilon}\in\mathcal{P}$.
$\boldsymbol{\tau}\in\mathbb{R}^{\Delta}$ encode displacement parameters fixed
by the details of the underlying system.

The rank $\Delta$ of the vector operator demonstrates the significance of
reference distribution in inferring laws from data. If we know beforehand or
correctly guess the law, then $\Delta=1$ corresponding to normalization on
$\mathcal{P}(\mathcal{T}_{\star})$. A TotEm over properly declared entities
with $\mathcal{E}_{0}=\mathcal{T}_{0}$ (so that
$\mathcal{E}_{\star}=\mathcal{T}_{\star}$) is already sufficient after
projecting prior knowledge onto
$[\mathfrak{f};{\,\mathbb{I}_{\mathcal{E}_{\star}}}]$,
$\mathbb{P}_{[\mathfrak{f};{\,\mathbb{I}_{\mathcal{E}_{\star}}}]}\boldsymbol{\upsilon}=e^{\tau}\,\mathbb{I}_{\mathcal{T}_{\star}}\boldsymbol{\upsilon}=\mathfrak{t}$,
i.e. the data provides no additional valuable information. On the other hand,
if our prior knowledge fails to capture aspects of the underlying law
governing subjects, $1<\Delta\leq|\mathcal{E}|$. In the poorest state of prior
knowledge when $\Delta=|\mathcal{E}|$, a TotEm would need to infer the
probability of every entity from measurements on the data (cf. expectations
$\langle{\boldsymbol{e}}\;|\mathfrak{f}|\;{\boldsymbol{e}}\rangle$) whipping
out all probabilities assumed by $\boldsymbol{\upsilon}$.

Ultimately, the efficiency of mle estimate is assured by (proof in B.8)

######  Theorem 4.3.

Among various vector operators
$\mathbf{\,\hat{M}}\in\mathcal{B}(\mathcal{E}_{\star})$ including one that is
fapp-equivalent to $\mathbf{\,\hat{T}}$ from Eq. (90), we benchmark the
constructed TotEms using information score 4.1. With increasing sample size,
$\mathbf{\,\hat{M}}_{\textsc{mle}}$ is expected to be fapp-equivalent to
$\mathbf{\,\hat{T}}$.

##  5 Illustrative examples

###  5.1 Logistic regression

As a straight-forward application, consider a regression setting with $m$
predictors $x_{i}\in\mathcal{D}_{\texttt{X}_{i}}$ (summarized by vector
$\mathbf{x}$) and one response variable $y\in\mathcal{D}_{\texttt{Y}}$. A
TotEm constructed by

$\displaystyle\mathbf{\,\hat{R}}=\left\\{\,\mathbb{I}_{\mathcal{E}},\,\mathbb{I}_{y},\,\mathbb{I}_{\mathbf{x}},\,\mathbb{I}_{y,\mathbf{x}}\right\\}$
(91)

learns the prevalence of response, all $m$ pairwise marginals
$\langle\mathfrak{f}\,\mathbb{I}_{y,x_{i}}\rangle$ alongside the joint
predictor distribution $\langle\mathfrak{f}\,\mathbb{I}_{\mathbf{x}}\rangle$
through linear system Eq. (30). Assuming (to avoid regularizing as in A) that
none of the aforementioned marginals vanishes and declaring
$\mathcal{E}_{0}=\emptyset$ unambiguously leads to maxent distribution
according to the exponential form in Theorem 3.2:

$\displaystyle\mathbb{P}_{\mathbf{\,\hat{R}}}\mathfrak{u}=\prod_{i=1}^{m}\sum_{x_{i}\in\mathcal{D}_{\texttt{X}_{i}}}Z^{-1}\sum_{y\in\mathcal{D}_{\texttt{Y}}}\langle\mathfrak{f}\,\mathbb{I}_{\mathbf{x}}\rangle\exp\left\\{\theta_{\texttt{Y}}(y)+\sum_{i=1}^{m}\theta^{i}_{\texttt{Y},\texttt{X}}(y,x_{i})\right\\}|y,\mathbf{x}\rangle\langle
y,\mathbf{x}|$ (92)

with logistic partition function

$\displaystyle Z\equiv
Z(\mathbf{x})=\sum_{y\in\mathcal{D}_{\texttt{Y}}}\exp\left\\{\theta_{\texttt{Y}}(y)+\theta_{i}(y,x_{i})\right\\}~{}.$
(93)

As explicitly shown in B.10 for the case of binary variables, when Lagrange
multipliers are promoted to parameters, the maxent distribution associated to
constructing element Eq. (91) is structurally identical to the model
distribution of logistic regression – up to reparametrization symmetry.
Exclusively considering marginal constraints, we do not need to worry about
such redundancies (which get enhanced for categorical variables with multiple
levels) when running ipf algorithm Eq. (70). In fact, we are free to specify
the (finite) domain of our variables. Most importantly,
$D=\text{rank}\,\mathbf{\,\hat{R}}$ robustly informs us about the degrees of
freedom in classifying regression architectures via Propositions 4.1 and 4.2.

###  5.2 Binomial distribution - revisited

To illustrate the insights unveiled by TotEm we consider the coin experiment
under the logic of [60], where $N$ agents perform $L$ Bernoulli experiments
and we obtain (portions of) data to decide about the coin(s) used. The outcome
of a Bernoulli experiment is unambiguously described by a binary variable
$s_{i}\in\mathcal{D}_{i}=\\{\texttt{head},\texttt{tail}\\}$, so that entity
$|s_{1},\ldots,s_{L}\rangle$ results from fully specifying the sequence
$\boldsymbol{s}\equiv(s_{1},\ldots,s_{L})$ of outcomes from the Cartesian
product $\mathcal{S}=\\{\texttt{head},\texttt{tail}\\}^{L}$.

Let us declare head as “success” and define111111As usual, $\delta$ denotes
the indicator map in the $i$-th domain $\mathcal{D}_{i}$. a success operator

$\displaystyle\,\hat{H}|\boldsymbol{s}\rangle=\frac{l(\boldsymbol{s})}{L}|\boldsymbol{s}\rangle\quad\text{with}\quad
l(\boldsymbol{s})=\sum_{i=1}^{L}\delta(s_{i},\texttt{head})~{},$ (94)

whose eigenvalues are individually given by the success rates of each
Bernoulli realization. Based on success operator Eq. (94), average moments
$\langle\mathfrak{f}\,\hat{H}^{n}\rangle$ for $n\in\mathbb{N}$ can be
initiated. Compatible with this information is the marginal distribution over
the number of successes $k$. Recording in the experiment only the frequency of
number of head can be realized by a $k$-marginal operator with eigenvalue
problem

$\displaystyle\,\mathbb{I}_{k}|\boldsymbol{s}\rangle=\delta_{l(\boldsymbol{s}),k}|\boldsymbol{s}\rangle~{}.$
(95)

Here, $\delta$ stands for the Kronecker delta. $\,\mathbb{I}_{k}$ is an
indicator operator that projects onto the subspace of Bernoulli realizations
with fixed number of head, i.e. success number $k$. Specifying phenomenology
$\phi_{k}\equiv\langle\mathfrak{f}\,\mathbb{I}_{k}\rangle$ for $k=0,\ldots,L$
partially breaks permutation symmetry over the $2^{L}$ Bernoulli entities into
degenerate subspaces with fixed $k$ whose dimension is given by binomial
coefficient

$\displaystyle b_{L,k}\equiv\begin{pmatrix}L\\\ k\end{pmatrix}~{}.$ (96)

#### The perception of one coin

In the case that only success rate
$\eta\equiv\langle\mathfrak{f}\hat{H}\rangle$ is specified from data, there is
a particularly simple form (derived in B.11) for the maxent distribution,

$\displaystyle\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{H}}}]}\mathfrak{u}=\sum_{\boldsymbol{s}\in\mathcal{S}}\eta^{l(\boldsymbol{s})}\left(1-\eta\right)^{L-l(\boldsymbol{s})}|\boldsymbol{s}\rangle\langle\boldsymbol{s}|~{},$
(97)

associated to constructing element
$\mathbf{\,\hat{H}}=\\{\,\mathbb{I}_{\mathcal{E}},\,\hat{H}\\}$. Eventually,
it is straight-forward to take the expectation w.r.t. maxent distribution of
marginal operator Eq. (95),

$\displaystyle\langle\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{H}}}]}\mathfrak{u}\,\,\mathbb{I}_{k}\rangle=b_{L,k}\,\eta^{k}\left(1-\eta\right)^{L-k}~{}.$
(98)

This coincides with the probability estimate for event $k$ from the familiar
binomial distribution $B(L,\eta)$.

Following the same steps, we can determine the $I$-projection of
$\mathfrak{u}$ onto the $k$-marginal TotEmplex constructed by
$\mathbf{\,\hat{K}}=\\{\,\mathbb{I}_{0},\ldots,\,\mathbb{I}_{L}\\}$,

$\displaystyle\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{K}}}]}\mathfrak{u}=\prod_{k=0}^{L}\exp\left\\{\log\frac{\phi_{k}}{b_{L,k}}\,\,\mathbb{I}_{k}\right\\}\,\mathbb{I}_{\mathcal{S}}=\sum_{\boldsymbol{s}\in\mathcal{S}}\frac{\phi_{l(\boldsymbol{s})}}{b_{L,l(\boldsymbol{s})}}|\boldsymbol{s}\rangle\langle\boldsymbol{s}|~{}.$
(99)

that reproduces the empirical relative frequencies
$\phi_{k}\equiv\langle\mathfrak{f}\,\mathbb{I}_{k}\rangle$ of the possible
success outcomes $k=0,\ldots,L$. Normalization is implied by
$\sum_{k=0}^{L}\phi_{k}=1$. Incidentally, success operator Eq. (94) can be
written as the superposition of $k$-marginal operators Eq. (95):

$\displaystyle\,\hat{H}=L^{-1}\sum_{k=0}^{L}k\,\,\mathbb{I}_{k}~{}.$ (100)

Hence, TotEmplex
$[\mathfrak{f};{\mathbf{\,\hat{K}}}]\subseteq[\mathfrak{f};{\mathbf{\,\hat{H}}}]$
is nested in the sense of Definition 3.5.

Using the distribution over the quotient space
$[\mathfrak{f};{\mathbf{\,\hat{H}}}]/\mathbf{\,\hat{K}}$ in Theorem 4.2, we
can compare how representative is the inner TotEmplex of $k$-marginals within
the ambient TotEmplex mean success rate. At large $N$, the $I$-test results
into a chi-squared distribution with
$\text{rank}\,\mathbf{\,\hat{K}}-\text{rank}\,\mathbf{\,\hat{H}}=L-1$ degrees
of freedom and statistics dictated by $2N$ times

$\displaystyle D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{K}}}]}\mathfrak{u}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{H}}}]}\mathfrak{u})=\sum_{k=0}^{L}\phi_{k}\log\frac{\phi_{k}}{b_{L,k}\eta^{k}\left(1-\eta\right)^{L-k}}~{}.$
(101)

As intuitively anticipated, the $I$-test statistics boils down to the kl
distance of the binomial distribution with empirical mean success rate
$\eta=\langle\mathfrak{f}\,\hat{H}\rangle$ from the empirical marginal
distribution $\phi_{k}=\langle\mathfrak{f}\,\mathbb{I}_{k}\rangle$ of success
number $k$.

To perform the $I$-test, we do not need to assume anything more concrete about
the experimental setup or the empirical distribution, beyond specifying the
entity space of $L$ Bernoulli trials. In particular, we have never assumed
independence of those Bernoulli trials. Therefore, it is straight-forward to
apply our TotEm formalism to explore the significance of higher
phenomenologies $\langle\mathfrak{f}\,\hat{H}^{n}\rangle$ for $n>1$ against
the $k$-marginal TotEm.

#### Updating prior knowledge

After deciding in favor of a TotEm on day1 constructed say by $\,\hat{H}$, we
can use the $I$-projection of $\mathfrak{u}$ onto
$[\mathfrak{f}_{\texttt{day1}};{\,\hat{H}}]$ as our updated reference
distribution:

$\displaystyle\boldsymbol{\upsilon}_{1}=\mathbb{P}_{[\mathfrak{f}_{\texttt{day1}};{\mathbf{\,\hat{H}}}]}\mathfrak{u}~{}.$
(102)

On the following day, the $I$-projection of $\boldsymbol{\upsilon}_{1}$ onto
the TotEmplex constructed by $\,\hat{H}$ on the new data boils down to simply
updating average success rate
$\eta_{1}=\langle\mathfrak{f}_{\texttt{day1}}\,\hat{H}\rangle$ in Eq. (97)
with the new estimate
$\eta_{2}=\langle\mathfrak{f}_{\texttt{day2}}\,\hat{H}\rangle$. This is easily
recognized by the form of $I$-projections in Theorem 3.2,

$\displaystyle\mathbb{P}_{[\mathfrak{f}_{\texttt{day2}};{\mathbf{\,\hat{H}}}]}\boldsymbol{\upsilon}_{1}=\mathbb{P}_{[\mathfrak{f}_{\texttt{day2}};{\mathbf{\,\hat{H}}}]}\mathbb{P}_{[\mathfrak{f}_{\texttt{day1}};{\mathbf{\,\hat{H}}}]}\mathfrak{u}=\exp\\{(\theta_{1}+\theta_{2})\,\hat{H}-\log(Z_{1}Z_{2})\,\mathbb{I}_{\mathcal{E}}\\}\mathfrak{u}~{}.$
(103)

which leads to the same structure Eq. (166) by redefining Lagrange multipliers
$\theta=\theta_{1}+\theta_{2}$ and $Z=Z_{1}Z_{2}$. Similarly, the
$I$-projection of $\boldsymbol{\upsilon}_{1}$ onto
$[\mathfrak{f}_{\texttt{day2}};{\mathbf{\,\hat{K}}}]$ now learns the new
$k$-marginals by heart, since the form

$\displaystyle\mathbb{P}_{[\mathfrak{f}_{\texttt{day2}};{\mathbf{\,\hat{K}}}]}\boldsymbol{\upsilon}_{1}=\mathbb{P}_{[\mathfrak{f}_{\texttt{day2}};{\mathbf{\,\hat{K}}}]}\mathbb{P}_{[\mathfrak{f}_{\texttt{day1}};{\mathbf{\,\hat{H}}}]}\mathfrak{u}=\exp\left\\{\sum_{k}\vartheta_{k}\,\mathbb{I}_{k}+\theta_{1}\,\hat{H}-\log
Z\right\\}\mathfrak{u}~{},$ (104)

upon using Eq. (100) and redefining
$\tilde{\vartheta}_{k}=\vartheta_{k}+\theta_{1}k/L-\log Z$, remains
structurally the same leading to Eq. (99) with
$\phi_{k}=\langle\mathfrak{f}_{\texttt{day2}}\,\mathbb{I}_{k}\rangle$. The
test statistics of $D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f}_{\texttt{day2}};{\mathbf{\,\hat{K}}}]}\boldsymbol{\upsilon}_{1}\;\|\;\mathbb{P}_{[\mathfrak{f}_{\texttt{day2}};{\mathbf{\,\hat{H}}}]}\boldsymbol{\upsilon}_{1})$
inform us whereas we need to update the mean success rate or not given the new
$k$-marginals.

Furthermore, the covariance of Algorithm 3.4 enables to numerically deduce the
$I$-projection associated to any higher-degree moment $\,\hat{H}^{n}$ beyond
the mean. Consequently, we can decide about updating not only the value of the
mean, but also the very structure of our prior knowledge from day1, augmenting
it with the value of higher moments estimated on day2.

#### The perception of two coins

As a little twist, let us assume that there is a confounding attribute Group
$g\in\mathcal{D}_{\texttt{G}}=\\{\texttt{A,B}\\}$ to which we suspect that
slightly different coins are assigned. Accordingly, the entity space is
augmented by this binary variable,
$(g;\boldsymbol{s})\equiv(g,s_{1},\ldots,s_{L})\in\mathcal{D}_{\texttt{G}}\times\mathcal{S}$.
In particular, success operator Eq. (94) is trivially lifted onto the
augmented entity space,

$\displaystyle\,\hat{H}|{\boldsymbol{e}}\rangle=L^{-1}l(\boldsymbol{s})|{\boldsymbol{e}}\rangle\quad\text{where}\quad|{\boldsymbol{e}}\rangle\equiv|g;\boldsymbol{s}\rangle~{}.$
(105)

To measure Group prevalence of team A within the entirety of $N$ agents, we
use indicator operator

$\displaystyle\,\mathbb{I}_{\texttt{A}}|{\boldsymbol{e}}\rangle=\delta(A,g)|{\boldsymbol{e}}\rangle~{},$
(106)

and similarly for Group prevalence of team B. Analogously to Eq. (97), we find
(details in B.11) the $I$-projection of $\mathfrak{u}$ onto a TotEmplex
constructed by121212Since
$\,\mathbb{I}_{\texttt{A}}+\,\mathbb{I}_{\texttt{B}}=\,\mathbb{I}_{\mathcal{E}}$
(reflecting that $\Phi_{\texttt{A}}+\Phi_{\texttt{B}}=1$),
$\\{\,\mathbb{I}_{\mathcal{E}},\,\mathbb{I}_{\texttt{A}},\,\hat{H}\\}$ and
$\\{\,\mathbb{I}_{\texttt{A}},\,\mathbb{I}_{\texttt{B}},\,\hat{H}\\}$ are fapp
equivalent.
$\mathbf{\,\hat{H}}=\\{\,\mathbb{I}_{\mathcal{E}},\,\mathbb{I}_{\texttt{A}},\,\hat{H}\\}$,

$\displaystyle\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{H}}}]}\mathfrak{u}=\sum_{g\in\\{\texttt{A,B}\\}}\Phi_{\texttt{g}}\sum_{\boldsymbol{s}\in\mathcal{S}}\eta^{l(\boldsymbol{s})}(1-\eta)^{L-l(\boldsymbol{s})}|g;\boldsymbol{s}\rangle\langle
g;\boldsymbol{s}|$ (107)

to be a straight-forward generalization of preceding discussion on the global
mean success rate additionally taking Group prevalence
$\Phi_{g}\equiv\langle\mathfrak{f}\,\mathbb{I}_{g}\rangle$ for
$g=\texttt{A,B}$ into account.

The empirical success rate in Group $g$ can be estimated by the conditional
mean

$\displaystyle\eta_{g}\equiv\frac{\langle\mathfrak{f}\,\hat{H}\,\mathbb{I}_{g}\rangle}{\langle\mathfrak{f}\,\mathbb{I}_{g}\rangle}\quad\text{for}\quad
g=\texttt{A,B}~{},$ (108)

using success operator Eq. (105). In addition to the phenomenology of global
mean Eq. (107), the maxent distribution that incorporates knowledge from Group
prevalences and Group mean success rates via linear system Eq. (33),

$\displaystyle\langle\mathfrak{p}\,\mathbb{I}_{g}\rangle\overset{!}{=}\langle\mathfrak{f}\,\mathbb{I}_{g}\rangle=\Phi_{g}\quad\text{and}\quad\langle\mathfrak{p}\,\hat{H}\,\mathbb{I}_{g}\rangle\overset{!}{=}\langle\mathfrak{f}\,\hat{H}\,\mathbb{I}_{g}\rangle=\Phi_{g}\eta_{g}\quad\text{for}\quad
g=\texttt{A,B}$ (109)

is given by the $I$-projection

$\displaystyle\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{H}^{\prime}}}]}\mathfrak{u}=\sum_{g\in\\{\texttt{A,B}\\}}\Phi_{g}\sum_{\boldsymbol{s}\in\mathcal{S}}\eta_{g}^{l(\boldsymbol{s})}\left(1-\eta_{g}\right)^{L-l(\boldsymbol{s})}|g;\boldsymbol{s}\rangle\langle
g;\boldsymbol{s}|~{}.$ (110)

By merit of
$\,\mathbb{I}_{\texttt{A}}+\,\mathbb{I}_{\texttt{B}}=\,\mathbb{I}_{\mathcal{E}}$
and
$\,\mathbb{I}_{\texttt{A}}\,\hat{H}+\,\mathbb{I}_{\texttt{B}}\,\hat{H}=\,\hat{H}$,
vector operator
$\mathbf{\,\hat{H}^{\prime}}=\\{\,\mathbb{I}_{\texttt{A}},\,\mathbb{I}_{\texttt{B}},\,\hat{H}\,\mathbb{I}_{\texttt{A}},\,\hat{H}\,\mathbb{I}_{\texttt{B}}\\}$
(which treats the two teams A,B symmetrically) is fapp equivalent to
$\\{\,\mathbb{I}_{\mathcal{E}},\,\mathbb{I}_{\texttt{A}},\,\hat{H},\,\hat{H}\,\mathbb{I}_{\texttt{A}}\\}$.
Since the latter form readily implies $\mathbf{\,\hat{H}}$, we could
immediately appeal to the $I$-test 4.2. At large $N$, we use chi-squared
distribution with one degree of freedom and test statistics given by
$2ND\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{H}^{\prime}}}]}\mathfrak{u}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{H}}}]}\mathfrak{u})$
to compare whether $\eta_{\texttt{A}}$ is significantly different from
$\eta_{\texttt{B}}$ or if both are well described by global $\eta$.

Again, our model-agnostic formalism with minimal assumptions allows the
straight-forward comparison of higher phenomenologies
$\langle\mathfrak{f}\,\hat{H}^{n}\,\mathbb{I}_{g}\rangle$ constructing nested
TotEmplices. Whenever $n>1$ or multiple categorical attributes act as
confounders quickly leading to analytically intractable expressions for the
$I$-projection, the iterative minimization of $I$-divergence (entropy
maximization in the case of $\boldsymbol{\upsilon}=\mathfrak{u}$) presented in
3.3 turns out to be practically beneficial.

##  Appendix A The curse of undersampling

Many realistic datasets suffer from sampling zeros which reduce the
statistical power of any inferential method, usually whenever
$N\leq|\mathcal{E}|$. At the level of contingency tables and histograms (used
to deduce empirical distribution Eq. (7)), sampling zeros are manifested by a
wealth of vanishing entries131313In particular, $f_{\boldsymbol{e}}=0$ for any
structural zero otherwise the provided dataset would have violated known
deterministic relationships.. To counteract “annoying” undersampling effects,
regularization has become a common practice in the literature [63].

Working in probability space, any sensible form of regularization boils down
to the method [64] of pseudocounts. The core idea relies on augmenting the
data matrix X by adding “unobserved” rows

$\displaystyle{\boldsymbol{e}}\in\mathcal{E}\,:\quad\texttt{X}_{{\boldsymbol{e}},i}=e_{i}\quad\text{for
}~{}i=1,\ldots L~{}.$ (111)

This would amount to modifying e.g. in a uniform manner the empirical
distribution $\mathfrak{f}\rightarrow\mathfrak{f}_{\lambda}$ such that
$\langle\mathfrak{f}_{\lambda}\rangle=1$ according to

$\displaystyle{f_{\boldsymbol{e}}}\quad\rightarrow\quad\frac{{f_{\boldsymbol{e}}}+\lambda/N}{1+|\mathcal{E}_{\star}|\cdot\lambda/N}\quad\text{for}\quad{\boldsymbol{e}}\in\mathcal{E}_{\star}~{}.$
(112)

The introduced hyper-parameter $\lambda>0$, often dumped regularization
strength helps interpolate from the original towards regularized data.

Despite that regularization methods lift sampling zeros and related
singularity problems for parametrizations of certain phenomenologies, the –
even uniform – modification of input data X in critical regions where $N$ and
$|\mathcal{E}|$ are of comparable order usually introduces artifacts that
undermine the statistical methods presented in Section 4. Besides the extreme
scenario of “strong” regularization which erases all data information in favor
of full agnosticism,
$\mathfrak{f}_{\lambda}\overset{\lambda\rightarrow\infty}{\rightarrow}\mathfrak{u}$,
there exist less evident, serious situations. Starting for concreteness from
uniform prior, the test statistics of Corollary 3.9 are determined by the
deviation of empirical entropy from the maximum entropy (entropy of the
$I$-projection of $\mathfrak{u}$). Whenever regularization Eq. (112) is
applied, the critical entropy difference

$\displaystyle\Delta H_{c}(\lambda)\equiv
H[\mathbb{P}_{[\mathfrak{f}_{\lambda};{\mathbf{\,\hat{M}}}]}\mathfrak{u}]-H[\mathfrak{f}_{\lambda}]$
(113)

is distorted, especially if
$\mathcal{O}\left(|\mathcal{E}_{\star}|\right)=\mathcal{O}\left(N\right)$.
Unless $\lambda=0$, the distortion in that regime cannot be generically
remedied even for “mild” regularization schemes where

$\displaystyle
H[\mathfrak{f}_{\lambda}]=H[\mathfrak{f}]+\lambda\frac{|\mathcal{E}_{\star}|}{N}\left(1-H[\mathfrak{f}]\right)+\mathcal{O}\left(\lambda^{2}\right)~{}.$
(114)

Most alarmingly, even if both entropies in Eq. (113) smoothly go to their non-
regularized counterparts as $\lambda\rightarrow 0$, the critical statistics
governing Eq. (83),

$\displaystyle 2N\Delta H_{c}(\lambda)=2N\Delta
H_{c}(0)+\mathcal{O}\left(N\lambda\right)$ (115)

imply that any finite $\lambda$ could control (at fixed threshold $\alpha$ in
Proposition 4.2) alongside the physical scale $N$ the significance outcome in
experimentally decisive settings where $\Delta H_{c}(0)$ is below
$\mathcal{O}\left(1/N\right)$.

To avoid such artifacts and stay data-centric retaining full control over the
information flow in our framework, we do not modify the data at any stage of
the statistical analysis. Therefore, we must learn to tolerate (nearly) non-
realizable entities.

##  Appendix B Detailed derivations

###  B.1 The information projection

######  Theorem B.1.

th:iProjection

#### Proof

In our data-centric approach, we recycle familiar [18] notions of information
geometry. The uniqueness of a minimum of $I$-divergence immediately follows in
probability space by the convexity of the feasible region
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ of the phenomenological problem Eq. (33)
at hand, combined with strict convexity of the $I$-divergence [65, 66] thought
as a function
$[0,1]^{|\mathcal{E}|-|\mathcal{E}_{0}|}\rightarrow\mathbb{R}_{0}^{+}$. Hence,
the $I$-divergence Eq. (41) possesses one global minimum in
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$, at most.

At the same time, a TotEmplex in the identification of Eq. (5) is represented
by a non-empty, convex, bounded and closed – hence compact – subset of
$[0,1]^{|\mathcal{E}|-|\mathcal{E}_{0}|}$, viz. Eq. (36), over which any
continuous function necessarily attains a minimum by the extreme value
theorem. In total, we conclude that the $I$-divergence must attain its global
minimum in the TotEmplex.

###  B.2 Form of the information projection

See 3.2

#### Proof

Following an established procedure in optimization theory, we incorporate
phenomenological constraints on expectations Eq. (33) via Lagrange multipliers
$\boldsymbol{\theta}$ in the Lagrangian

$\displaystyle\mathcal{L}=D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})-\sum_{\alpha=1}^{D}\theta_{\alpha}\sum_{{\boldsymbol{e}}\in{\boldsymbol{\mathcal{D}}_{\star}}}m_{\alpha}({\boldsymbol{e}})\left(p_{\boldsymbol{e}}-f_{\boldsymbol{e}}\right)~{}.$
(116)

Subsequently, we extremize the Lagrangian in the probabilities
$\mathfrak{p}_{\boldsymbol{e}}>0$ and the parameters
$\boldsymbol{\theta}\in\mathbb{R}^{D}$. Using exactly
$D=\text{rank}\,\mathbf{\,\hat{M}}$ Lagrange multipliers – in accordance with
Definition 2.2 – ensures the independence of the incorporated constraints,
which becomes essential for the convergence of many iterative algorithms and
important for their efficiency.

In the basis of admissible entities Eq. (5), we directly arrive at the
exponential expression

$\displaystyle\langle{\boldsymbol{e}}\;|\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}|\;{\boldsymbol{e}}\rangle=\boldsymbol{\upsilon}_{\boldsymbol{e}}\exp\left\\{\sum_{\alpha=1}^{D}\theta_{\alpha}m_{\alpha}({\boldsymbol{e}})-1\right\\}$
(117)

for some vector of Lagrange multipliers such that Eq. (33) is fulfilled.
Noting that our constructing elements 2.2 always imply normalization Eq. (11)
means that we can always transform via Eq. (34) performing a “rotation” in the
space of expectations, so that

$\displaystyle\exists\,T_{0\alpha}~{}:\quad\sum_{\alpha=1}^{D}T_{0\alpha}m_{\alpha}({\boldsymbol{e}})=1~{}.$
(118)

Shifting accordingly

$\displaystyle\theta_{\alpha}\quad\rightarrow\quad\theta_{\alpha}+T_{0\alpha}$
(119)

identically results into the r.h.s. of Eq. (55).

To show the converse of the Theorem we need a

######  Lemma B.2.

If the relation

$\displaystyle D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})=D\hskip
0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})+D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})\quad\text{for
all}\quad{\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ (120)

for a distribution ${\mathfrak{q}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$
holds, then this probability operator is the $I$-projection of
${\boldsymbol{\upsilon}}$ onto the TotEmplex
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$.

In fact, this is an analogue of Pythagorean theorem for $I$-divergences, see
e.g. [67, 30].

#### Proof.

Due to the non-negativity of the $I$-divergence Eq. (41), the forward
direction of the Lemma follows from Eq. (120) rewritten as

$\displaystyle D\hskip
0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})=D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})-D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})\leq D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon}),$ (121)

for all distributions ${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$,
which reproduces the definition of the $I$-projection in the TotEmplex. ∎

Back to the proof of the Theorem, substituting the assumed exponential form
into the r.h.s. of Eq. (120) we see that

$\displaystyle D\hskip
0.28453pt(\mathfrak{q}\;\|\;\boldsymbol{\upsilon})+D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})=D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})+\sum_{\alpha=1}^{D}\theta_{\alpha}\underbrace{\sum_{{\boldsymbol{e}}\in\mathcal{E}_{\star}}\left(q_{\boldsymbol{e}}-p_{\boldsymbol{e}}\right)m_{\alpha}({\boldsymbol{e}})}_{=0\text{
in }[\mathfrak{f};\mathbf{\,\hat{M}}]}$ (122)

on any probability operator
${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$. Hence, the
variationally optimal solution Eq. (55) fulfills the relation of Lemma B.2 and
is the unique $I$-projection of ${\boldsymbol{\upsilon}}$ onto
$[\mathfrak{f};{\mathbf{\,\hat{M}}}]$.

###  B.3 Constrained minimization of information divergence

See 3.4

#### Proof

At the $n$-th step, the $D\times D$ Jacobian $\mathbf{J}$ is invertible –
which is a necessary condition for convergence – due to following

######  Lemma B.3.

Let $\mathbf{P}$ be a real symmetric positive definite matrix (signified by
$\mathbf{P}\succ 0$) and $\mathbf{C}$ a real matrix of full-row rank with a
column-structure compatible with $\mathbf{P}$. Then, the adjoint map from the
column to the row space of $\mathbf{C}$ is also given by a symmetric positive
definite and thus invertible matrix

$\displaystyle\mathbf{C}\mathbf{P}\mathbf{C}^{T}\textdegree.$ (123)

#### Proof

For linearly independent rows in $\mathbf{C}$, it is easy to show that the
adjoint map is positive definite (and thus invertible):

$\displaystyle\forall\,\mathbf{u}\in\mathbb{R}^{\text{rank}\,\mathbf{C}}\,:0\overset{!}{=}$
$\displaystyle\,\,\mathbf{u}^{T}\left(\mathbf{C}\mathbf{P}\mathbf{C}^{T}\right)\mathbf{u}=\left(\mathbf{C}^{T}\mathbf{u}\right)^{T}\mathbf{P}\left(\mathbf{C}^{T}\mathbf{u}\right)$
(124) $\displaystyle\overset{\mathbf{P}\succ
0}{\Longrightarrow}\quad~{}\mathbf{u}^{T}\mathbf{C}=\mathbf{0}_{\text{rank}\,\mathbf{P}}\quad\overset{\mathbf{C}\succ
0}{\Longrightarrow}\quad\mathbf{u}=\mathbf{0}_{\text{rank}\,\mathbf{C}}~{}.\phantom{...}\qed$

Since an irreducible vector operator 2.2 that measures collective
characteristics constructs a TotEm, the eigenvalues
$m_{\alpha}({\boldsymbol{e}})$ on admissible microstates seen as a linear map
Eq. (30) constitute a matrix of full row-rank equaling $D$. A diagonal
operator with elements $\mathfrak{p}^{(n)}_{e}>0$ can be thought of in the
representation of Eq. (5) as a diagonal matrix in
$\mathbb{R}^{|\mathcal{E}|-|\mathcal{E}_{0}|}$ that remains positive definite
throughout the optimization. Consequently, Lemma B.3 applies and the Jacobian
is indeed at each Newton-Raphson step positive-definite and thus invertible.

###  B.4 The information divergence in nested descriptions

See 3.6 To assert that successive determination of $I$-projections on nested
TotEmplices is equivalent to the $I$-projection of referrence distribution to
the innermost, target TotEmplex, we need

######  Lemma B.4.

For nested TotEmlices
$[\mathfrak{f};{\mathbf{\,\hat{B}}}]\subseteq[\mathfrak{f};{\mathbf{\,\hat{A}}}]$
spanned by irreducible vector operators ${\mathbf{\,\hat{A}}}$ and
${\mathbf{\,\hat{B}}}$ the $I$-projection of ${\boldsymbol{\upsilon}}$ onto
the inner TotEmplex $[\mathfrak{f};{\mathbf{\,\hat{B}}}]$ satisfies chain rule

$\displaystyle\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\boldsymbol{\upsilon}=\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}$
(125)

coinciding with the $I$-projection onto the inner TotEmplex of reference
distribution
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}$ which
is in turn the $I$-projection of ${\boldsymbol{\upsilon}}$ onto the ambient
TotEmplex $[\mathfrak{f};{\mathbf{\,\hat{A}}}]$.

Reading Eq. (125) from right to left reveals that the theorem can help
eliminate the intermediate $I$-projection of ${\boldsymbol{\upsilon}}$ onto
the ambient TotEmplex. Repeated application of this notion starting at the
terminating step $\nu=d$ of Eq. (68) to eliminate intermediate $I$-projections
from inner towards outer TotEmplices,

$\displaystyle\boldsymbol{\upsilon}^{(d)}=$
$\displaystyle\,\,\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(d)}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(d-1)}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(d-2)}}]}\cdots\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(1)}}]}\boldsymbol{\upsilon}$
(126) $\displaystyle=$
$\displaystyle\,\,\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(d)}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(d-2)}}]}\cdots\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(1)}}]}\boldsymbol{\upsilon}=\ldots=\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}^{(d)}}]}\boldsymbol{\upsilon}~{},$

we verify that the suggested pipeline is equivalent to directly determining
the $I$-projection of ${\boldsymbol{\upsilon}}$ onto the TotEmplex defined by
$\mathbf{\,\hat{M}}^{(d)}\equiv\mathbf{\,\hat{M}}$.

#### Proof

We start from the definition of the $I$-projection of
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}$ onto
the inner TotEmplex $[\mathfrak{f};\mathbf{\,\hat{B}}]$:

$\displaystyle\forall\,\,{\mathfrak{p}}\in[\mathfrak{f};\mathbf{\,\hat{B}}]\,:\quad
D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon})\geq
D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon})~{}.$
(127)

Applying Corollary 3.3 in the ambient TotEmplex, onto which
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}$ is the
$I$-projection of ${\boldsymbol{\upsilon}}$, for distributions
${\mathfrak{p}}~{},~{}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\in[\mathfrak{f}]_{\hat{\mathbf{A}}}$

$\displaystyle D\hskip
0.28453pt(\mathfrak{p}\;\|\;\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon})=$
$\displaystyle\,\,D\hskip
0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})-D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})\quad\text{and}$
$\displaystyle D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\;\|\;\mathbb{P}_{\boldsymbol{\upsilon}}[{[\mathfrak{f}]_{\hat{\mathbf{A}}}}])=$
$\displaystyle\,\,D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})-D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})~{},$

cancels the common term $D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})$
on both sides of defining inequality Eq. (127) leading to

$\displaystyle D\hskip 0.28453pt(\mathfrak{p}\;\|\;\boldsymbol{\upsilon})\geq
D\hskip
0.28453pt(\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}\;\|\;\boldsymbol{\upsilon})~{}.$
(128)

Since this applies for all ${\mathfrak{p}}$ in the inner TotEmplex
$[\mathfrak{f};{\mathbf{\,\hat{B}}}]$, we conclude by uniqueness (Theorem 3.1)
that the distribution
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{B}}}]}\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}$
defined as the $I$-projection of
$\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{A}}}]}\boldsymbol{\upsilon}$ onto
$[\mathfrak{f};{\mathbf{\,\hat{B}}}]$ is also the $I$-projection of
${\boldsymbol{\upsilon}}$ onto $[\mathfrak{f};{\mathbf{\,\hat{B}}}]$. …∎

###  B.5 The Gaussian kernel

See 3.8

#### Proof

First, we pass from perception of entities in the TotEmplex to fluctuations of
perception which appropriately live in the kernel of $\mathbf{\,\hat{M}}$, as
described above Eq. (75). Next, we perform a Taylor expansion of the
$I$-divergence of generic
${\mathfrak{p}}\in[\mathfrak{f};{\mathbf{\,\hat{M}}}]$ from the $I$-projection
in the TotEmplex around base distribution
${\mathfrak{p}}=\mathbb{P}_{[\mathfrak{f};{\mathbf{\,\hat{M}}}]}\boldsymbol{\upsilon}\equiv\mathfrak{q}$
in “small” fluctuations,

$\displaystyle D\hskip 0.28453pt(\mathfrak{p}\;\|\;\mathfrak{q})=$
$\displaystyle\,\,\langle\left(\mathfrak{q}+\boldsymbol{\pi}\right)\left(\log\left(\mathfrak{q}+\boldsymbol{\pi}\right)-\log\mathfrak{q}\right)\rangle$
(129) $\displaystyle=$
$\displaystyle\,\,\langle\left(\mathfrak{q}+\boldsymbol{\pi}\right)\log\left(1+\mathfrak{q}^{-1}\boldsymbol{\pi}\right)\rangle=\langle\boldsymbol{\pi}\rangle+\mbox{$\frac{1}{2}$}\langle\boldsymbol{\pi}^{2}q^{-1}\rangle+\mathcal{O}\left(\boldsymbol{\pi}^{3}\right)~{}.$

The linear term vanishes, since we immediately recognize from Eq. (75) that
$\langle\mathfrak{p}\rangle=1$ implies $\langle\boldsymbol{\pi}\rangle=0$.
This is expected, as we expand around the minimum of the $I$-divergence in the
convex space of the TotEmplex. As indicated in Eq. (77), there thus remains to
leading order in the fluctuations a quadratic form

$\displaystyle\mathcal{K}(\boldsymbol{\pi})=\langle\boldsymbol{\pi}\mathfrak{q}^{-1}\boldsymbol{\pi}\rangle=\sum_{a,b=1}^{|\ker\mathbf{\,\hat{M}}|}\pi_{a}\langle\,\hat{N}_{a}\mathfrak{q}^{-1}\,\hat{N}_{b}\rangle\pi_{b}~{}.$
(130)

###  B.6 Spherical symmetry in probability fluctuations

See 3.9

#### Proof

Any distribution on the TotEmplex is sampled from ${\boldsymbol{\upsilon}}$ as
a fluctuation away from its $I$-projection by the probability

$\displaystyle f(\boldsymbol{\pi})\prod_{a=1}^{k}\text{d}\pi_{a}~{},$ (131)