# Generalized Information Bottleneck for Gaussian Variables

Vudtiwat Ngampruetikorn  David J. Schwab  Initiative for the Theoretical
Sciences, The Graduate Center, CUNY, New York, NY 10016, USA

###### Abstract

The information bottleneck (IB) method offers an attractive framework for
understanding representation learning, however its applications are often
limited by its computational intractability. Analytical characterization of
the IB method is not only of practical interest, but it can also lead to new
insights into learning phenomena. Here we consider a generalized IB problem,
in which the mutual information in the original IB method is replaced by
correlation measures based on Rényi and Jeffreys divergences. We derive an
exact analytical IB solution for the case of Gaussian correlated variables.
Our analysis reveals a series of structural transitions, similar to those
previously observed in the original IB case. We find further that although
solving the original, Rényi and Jeffreys IB problems yields different
representations in general, the structural transitions occur at the same
critical tradeoff parameters, and the Rényi and Jeffreys IB solutions perform
well under the original IB objective. Our results suggest that formulating the
IB method with alternative correlation measures could offer a strategy for
obtaining an approximate solution to the original IB problem.

##  I Information Bottleneck

Effective representation of data is key to generalizable learning.
Characterizing what makes such representation good and how it emerges is
crucial to understanding the success of modern machine learning. The
information bottleneck (IB) method—an information-theoretic formulation for
representation learning [1]—has proved a particularly useful conceptual
framework for this question, and has led to a deeper understanding of
representation learning in both supervised and self-supervised learning [2, 3,
4]. Investigating this abstraction of representation learning has the
potential to yield new insights that are applicable to learning problems.

Quantifying the goodness of a representation requires the knowledge of what is
to be learned from data. Information bottleneck theory exploits the fact that,
in many settings, we can define relevant information through an additional
variable; for example, it could be the label of each image in a classification
task. This notion of relevance allows for a precise definition of
optimality—an IB optimal representation $T$ is maximally predictive of the
relevance variable $Y$ while minimizing the number of bits extracted from the
data $X$. The IB method formulates this principle as an optimization problem
[1],

$\min\nolimits_{Q_{T|X}}\;I(T;X)-\beta I(T;Y).$ (1)

Here the optimization is over the encoders $Q_{T|X}$ which provide a
(stochastic) mapping from $X$ to $T$. Maximizing the mutual information
$I(T;Y)$ [second term in Eq (1)] encourages a representation $T$ to encode
more relevant information while minimizing $I(T;X)$ [first term in Eq (1)]
discourages it from encoding irrelevant bits. The parameter $\beta>0$ controls
the fundamental tradeoff between the two information terms.

The IB method has proved successful in a number of applications, including
neural coding [5, 6], statistical physics [7, 8, 9], clustering [10], deep
learning [11, 2, 3], reinforcement learning [12] and learning theory [13, 14,
15, 16]. However the nonlinear nature of the IB problem makes it
computationally costly. Although scalable learning methods based on the IB
principle are possible thanks to variational bounds of mutual information [11,
17, 18], the choice of such bounds as well as specific details on their
implementations can introduce strong inductive bias that competes with the
original objective [19].

While large-scale applications of the IB method in its exact form are
generally intractable, special cases exist. For example, the limit of low
information—i.e., when both terms in Eq (1) are small—can be described by a
perturbation theory, which provides a recipe for identifying a representation
that yields maximum relevant information per extracted bit [20, 21]. But
perhaps the most important special case is when the source $X$ and the target
$Y$ are Gaussian correlated random variables. In this case, an exact
_analytical_ solution exists [22].

Although originally formulated with Shannon mutual information, the
fundamental tradeoff in the IB method applies more generally: the IB
optimization [Eq (1)] remains well-defined when the information terms are
replaced by appropriate mutual dependence measures. In this work, we consider
generalized IB problems based on two important correlation measures. The first
is a parametric generalization of Shannon information, based on Rényi
divergence [23]. Rényi-based generalizations of mutual information and entropy
are central in quantifying quantum entanglement [24, 25] and have proved a
powerful tool in Monte-Carlo simulations [26, 27, 28] as well as in
experiments [29, 30, 31]. The second mutual dependence measure we consider is
based on Jeffreys divergence [32]. The resulting Jeffreys information is (up
to a constant prefactor) equal to the generalization gap of a broad family of
learning algorithms, known as Gibbs algorithms [33].

We derive an analytical IB solution for the case in which $X$ and $Y$ are
Gaussian correlated, generalizing the result of Ref [22] to a class of
information-theoretic mutual dependence measures which includes Shannon
information as a limiting case. We show that, for both Rényi and Jeffreys
cases, an optimal encoder can be constructed from the eigenmodes of the
normalized regression matrix $\Sigma_{X|Y}\Sigma_{X}^{-1}$. Our solution
reveals a series of phase transitions, similar to those observed in the
Gaussian IB method [22]. In both Rényi and Jeffreys cases, we find that
although the optimal encoders depend on information measures, the phase
transitions occur at the critical tradeoff parameters $\beta_{c}^{(i)}$ that
coincide with that of the Shannon case, independent of the order of Rényi
information.

##  II Divergence-based Correlation measure

When two random variables $X$ and $Y$ are uncorrelated, their joint
distribution $P_{XY}$ is equal to the product of their marginals $P_{X}$ and
$P_{Y}$. As a result, we can quantify the mutual dependence between $X$ and
$Y$ by the difference between $P_{XY}$ and $P_{X}\otimes P_{Y}$,

$\Omega(X;Y)\equiv\mathcal{D}(P_{XY}\parallel P_{X}\otimes P_{Y})\geq 0.$ (2)

Here $\mathcal{D}(P\,\|\,Q)$ denotes a statistical divergence which, by
definition, is nonnegative and vanishes if and only if $P=Q$. When defined
with the Kullback–Leibler (KL) divergence, the above measure becomes Shannon
information, $I(X;Y)=\operatorname{KL}(P_{XY}\,\|\,P_{X}\otimes P_{Y})$.

##  III Rényi q–information

We consider a correlation measure, based on Rényi divergence [23]. More
precisely, we define _Rényi $q$-information_ as

$I_{q}(X;Y)\equiv\mathcal{R}_{q}(P_{XY}\parallel P_{X}\otimes P_{Y}),$ (3)

where $\mathcal{R}_{q}$ denotes Rényi divergence of order $q$,

$\mathcal{R}_{q}(P\parallel
Q)=\frac{1}{q-1}\ln\int\\!\\!dQ\,\left(\frac{dP}{dQ}\right)^{q}$ (4)

for $q\in(0,1)\cup(1,\infty)$. This definition extends to $q=0,1$ and $\infty$
via continuity in $q$. In particular,
$\mathcal{R}_{1}(P\|Q)=\operatorname{KL}(P\|Q)$ [34, Thm 5], and as a result
$I_{1}(X;Y)=I(X;Y)$. Rényi divergences, and thus $q$-information, satisfy the
data processing inequality since they have a strictly increasing relationship
with an $f$-divergence [with $f(t)=(t^{q}-1)/(q-1)$] which exhibits this
property, see, e.g., Ref [35].

###  III.1 Gaussian variables

For Gaussian correlated variables $X$ and $Y$, the $q$-information is given by
(see Appendix A for derivation)

$I_{q}(X;Y)=\frac{-1}{2\bar{q}}\ln\frac{|\Sigma_{X|Y}\Sigma_{X}^{-1}|^{\bar{q}}}{|I-\bar{q}^{2}(I-\Sigma_{X|Y}\Sigma_{X}^{-1})|}\;\;\text{with}\;\;\bar{q}=1-q,$
(5)

where $I$ denotes the identity matrix in compatible dimensions. We see that
this information depends on the covariance matrices only through the
normalized regression matrix $\Sigma_{X|Y}\Sigma_{X}^{-1}$. We note also that
this information can diverge when $q>2$ since the eigenvalues of
$\Sigma_{X|Y}\Sigma_{X}^{-1}$ range from zero to one [22, Lemma B.1]. It is
easy to verify that Shannon information corresponds to the limit $q\to 1$,

$\displaystyle I(X;Y)=\lim_{q\to
1}I_{q}(X;Y)=-\frac{1}{2}\ln|\Sigma_{X|Y}\Sigma_{X}^{-1}|.$ (6)

In addition, we note that for Gaussian variables $I_{2}(X;Y)=2I(X;Y)$ and
$I_{q}(X;Y)$ increases with $q$ from zero at $q=0$.

Note that alternative definitions of Rényi mutual information exist. In
physics literature, a frequently used definition is
$I_{q}(X;Y)=S_{q}(X)+S_{q}(Y)-S_{q}(X,Y)$ where $S_{q}(X)=(1-q)^{-1}\ln\int
dx\,p_{X}(x)^{q}$ is Rényi (differential) entropy of order $q$. However, for
Gaussian variables, this definition leads to Rényi information that is equal
to Shannon information regardless of $q$; the resulting Rényi IB problem is
therefore identical to the original IB problem.

Figure 1: a The optimal frontiers of the generalized IB methods based on
generalized correlation measures $\Omega(A;B)$, including Shannon
($\Omega\\!=\\!I$), Jeffreys ($\Omega\\!=\\!J$) and Rényi
($\Omega\\!=\\!I_{q}$) informations (see legend). For the Rényi case, we
depict the results for a range of Rényi orders $q$ (see color bar). We
emphasize that while Shannon information is equivalent to Rényi information of
order one ($q\\!=\\!1$), Jeffreys information is not a special case of Rényi
information. The relevant information $\Omega(T;Y)$ is bounded by the data
processing inequality (DPI, black dotted line),
$\Omega(T;Y)\\!\leq\\!\Omega(T;X)$. We also depict the tight, data-dependent
version of DPI, the strong data processing inequality (SDPI, blue dahsed
line), $\Omega(T;Y)\\!\leq\\!(1-\lambda_{\text{min}})\Omega(T;X)$, where
$\lambda_{\text{min}}$ is the smallest eigenvalue of the normalized regression
matrix $\Sigma_{X|Y}\Sigma_{X}^{-1}$. Note that the DPI and SDPI are the same
for all information measures shown. b-c The extracted and relevant bits,
$\Omega(T;X)$ and $\Omega(T;Y)$ respectively, increase with the tradeoff
parameter $\beta$ and vanish below the critical value
$\beta_{c}\\!=\\!1/(1-\lambda_{\text{min}})$. The vertical lines mark the
location of the critical tradeoff parameters [Eqs (22) & (35)]. Here the
eigenvalues of $\Sigma_{X|Y}\Sigma_{X}^{-1}$ are
$\lambda_{i}=0.1,0.2,0.3,0.5,0.7,0.8$.

##  IV Rényi Information Bottleneck for Gaussian variables

Replacing the mutual information in the original IB objective [Eq (1)] with
$q$-information yields

$\mathcal{L}_{q}[Q_{T|X}]=I_{q}(T;X)-\beta I_{q}(T;Y)$ (7)

where $X$ denotes the source data, $Y$ the target variable and $T$ the
representation of $X$. The loss function varies with the encoder $Q_{T|X}$
which provides a stochastic mapping from $X$ to $T$. In general, the
$q$-information terms need not be of the same order but the data processing
inequality $I_{q}(T;X)\geq I_{q^{\prime}}(T;Y)$ is guaranteed only when
$q=q^{\prime}$.

We specialize to the case where $X$ and $Y$ are Gaussian correlated and
consider a family of noisy linear encoders,

$T=AX+\xi\quad\text{with}\quad\xi\sim N(0,\Sigma_{\xi}).$ (8)

Since Rényi divergences are invariant under an invertible transformation of
random variables [see Eq (4)], we can transform $T$ such that $\Sigma_{\xi}$
becomes the identity matrix without changing the information content. In the
following analysis, we set $\Sigma_{\xi}=I$ without loss of generality. That
is, the encoder becomes a Gaussian channel, parametrized only by the matrix
$A$,

$T\mid X\sim N(AX,I).$ (9)

To compute the information in Eq (7), we first marginalize out $X$ from the
above equation, yielding

$\displaystyle T$ $\displaystyle\sim
N(A\mu_{X},I+A\Sigma_{X}A^{\mkern-1.5mu\mathsf{T}})$ (10) $\displaystyle T\mid
Y$ $\displaystyle\sim
N(A\mu_{X|Y},I+A\Sigma_{X|Y}A^{\mkern-1.5mu\mathsf{T}}),$ (11)

where we use $X\sim N(\mu_{X},\Sigma_{X})$ and $X\,|\,Y\sim
N(\mu_{X|Y},\Sigma_{X|Y})$. Substituting the covariance matrices in the above
equations into Eq (5) results in

$\displaystyle I_{q}(T;X)$
$\displaystyle=\frac{-1}{2\bar{q}}\ln\frac{|I+A\Sigma_{X}A^{\mkern-1.5mu\mathsf{T}}|^{q}}{|I+(1-\bar{q}^{2})A\Sigma_{X}A^{\mkern-1.5mu\mathsf{T}}|}$
(12) $\displaystyle I_{q}(T;Y)$
$\displaystyle=\frac{-1}{2\bar{q}}\ln\frac{|I+A\Sigma_{X|Y}A^{\mkern-1.5mu\mathsf{T}}|^{\bar{q}}|I+A\Sigma_{X}A^{\mkern-1.5mu\mathsf{T}}|^{q}}{|I+A[I-\bar{q}^{2}(I-\Sigma_{X|Y}\Sigma_{X}^{-1})]\Sigma_{X}A^{\mkern-1.5mu\mathsf{T}}|}.$
(13)

Following the analysis of Ref [22], we define the _mixing matrix_ $W$ such
that

$A=WV,$ (14)

where $V$ is a matrix of left (row) eigenvectors of the normalized regression
matrix $\Sigma_{X|Y}\Sigma_{X}^{-1}$, i.e.,

$V\Sigma_{X|Y}\Sigma_{X}^{-1}=\Lambda
V\quad\text{with}\quad\Lambda=\operatorname{diag}(\lambda_{1},\lambda_{2},\cdots).$
(15)

We note that $V\Sigma_{X}^{1/2}$ is orthogonal and thus
$V\Sigma_{X}V^{\mkern-1.5mu\mathsf{T}}$ is a diagonal matrix [22, Lemma B.1],
i.e.,

$V\Sigma_{X}V^{\mkern-1.5mu\mathsf{T}}=R\quad\text{with}\quad
R=\operatorname{diag}(r_{1},r_{2},\cdots).$ (16)

Writing Eqs (12-13) in terms of $W$, $\Lambda$ and $R$ leads to

$\displaystyle I_{q}(T;X)$
$\displaystyle=\frac{-1}{2\bar{q}}\ln\frac{|I+WRW^{\mkern-1.5mu\mathsf{T}}|^{q}}{|I+(1-\bar{q}^{2})WRW^{\mkern-1.5mu\mathsf{T}}|}$
(17) $\displaystyle I_{q}(T;Y)$
$\displaystyle=\frac{-1}{2\bar{q}}\ln\frac{|I+W\Lambda
RW^{\mkern-1.5mu\mathsf{T}}|^{\bar{q}}|I+WRW^{\mkern-1.5mu\mathsf{T}}|^{q}}{|I+W[I-\bar{q}^{2}(I-\Lambda)]RW^{\mkern-1.5mu\mathsf{T}}|}.$
(18)

Substituting Eqs (17-18) into Eq (7) and setting its first order derivative
with respect to the mixing matrix $W$ to zero yields the first order condition

$\displaystyle\frac{q}{I+RW^{\mkern-1.5mu\mathsf{T}}W}-\frac{1-\bar{q}^{2}}{I+(1-\bar{q}^{2})RW^{\mkern-1.5mu\mathsf{T}}W}$
(19)
$\displaystyle\quad=\beta\bigg{(}\frac{q}{I+RW^{\mkern-1.5mu\mathsf{T}}W}+\frac{\bar{q}}{I+\Lambda
RW^{\mkern-1.5mu\mathsf{T}}W}\Lambda$
$\displaystyle\quad\qquad-\frac{1}{I+[I-\bar{q}^{2}(I-\Lambda)]RW^{\mkern-1.5mu\mathsf{T}}W}[I-\bar{q}^{2}(I-\Lambda)]\bigg{)}.\\!$

In deriving the above, we use the identity
$d\ln|I+WCW^{\mkern-1.5mu\mathsf{T}}|/dW=2W(I+CW^{\mkern-1.5mu\mathsf{T}}W)^{-1}C$
for any compatible square matrix $C$ and assume that $R$ and $W$ are
invertible.

We seek a solution of the form

$RW^{\mkern-1.5mu\mathsf{T}}W=\operatorname{diag}(r_{1}w_{1}^{2},r_{2}w_{2}^{2},\cdots)\equiv\operatorname{diag}(u_{1},u_{2},\cdots).$
(20)

Substituting this ansatz into Eq (19) results in

$\frac{1}{\beta}=g_{q}(u_{i},\lambda_{i})=\frac{1-\lambda_{i}}{1+u_{i}\lambda_{i}}\frac{1+\frac{\bar{q}(1+\bar{q})(1-\lambda_{i})u_{i}}{1+(1-\bar{q}^{2}(1-\lambda_{i}))u_{i}}}{1+\frac{\bar{q}(1+\bar{q})u_{i}}{1+(1-\bar{q}^{2})u_{i}}}.$
(21)

We see that the contributions from the eigenmodes of
$\Sigma_{X|Y}\Sigma_{X}^{-1}$ decouple from one another and the reduced mixing
weight, $u_{i}=r_{i}w_{i}^{2}$, for each mode depends only on the eigenvalue
of that mode $\lambda_{i}$, the IB tradeoff parameter $\beta$ and the order of
Rényi information $q$. For $\lambda\in(0,1)$ and $q\in[0,2]$, the function
$g_{q}(u,\lambda)$ is strictly decreasing in $u$ for $u\geq 0$ and approaches
zero as $u\to\infty$ (see Appendix C). As a result, Eq (21) has exactly one
positive solution $u_{i}>0$ when $1/\beta<g_{q}(0,\lambda_{i})$. That is, the
eigenmode with eigenvalue $\lambda_{i}$ contributes to the Rényi IB encoder
only when $\beta$ exceeds the critical value

$\beta_{c}^{(i)}=\frac{1}{g_{q}(0,\lambda_{i})}=\frac{1}{1-\lambda_{i}}.$ (22)

Note that $\beta_{c}^{(i)}$ does not depend on $q$. To obtain $u_{i}$, we can
either directly solve Eq (21) or use the analytical formula for the roots of
the equivalent cubic equation (omitting the eigenmode indices)

$0=au^{3}+bu^{2}+cu+d,$ (23)

where the coefficients are given by

$\displaystyle a$ $\displaystyle=\lambda(1+\bar{q})(1-(1-\lambda)\bar{q}^{2})$
$\displaystyle b$
$\displaystyle=\lambda(2+2\bar{q}+\lambda\bar{q}^{2})+d(1+\bar{q})(1-\lambda\bar{q}-(1-\lambda)\bar{q}^{2})$
$\displaystyle c$
$\displaystyle=\lambda(1+\bar{q}+\bar{q}^{2})+d(2+(1-\lambda)\bar{q}-\bar{q}^{2})$
$\displaystyle d$ $\displaystyle=1-\beta(1-\lambda).$

Although the above calculation does not uniquely determine the mixing matrix
$W$, we can obtain a valid IB encoder by taking
$W=\operatorname{diag}(w_{1},w_{2},\cdots)$ where $w_{i}=\sqrt{u_{i}/r_{i}}$
since the Rényi-IB loss depends on $W$ only through the diagonal entries of
$W^{\mkern-1.5mu\mathsf{T}}W$. To see this, we substitute Eq (20) into Eqs
(17-18) and write down

$\displaystyle I_{q}(T;X)$
$\displaystyle=\frac{-1}{2\bar{q}}\sum_{i}^{\;\;\beta>\beta_{c}^{(i)}}\ln\frac{(1+u_{i})^{q}}{1+(1-\bar{q}^{2})u_{i}}$
(24) $\displaystyle I_{q}(T;Y)$
$\displaystyle=\frac{-1}{2\bar{q}}\sum_{i}^{\;\;\beta>\beta_{c}^{(i)}}\ln\frac{(1+\lambda_{i}u_{i})^{\bar{q}}(1+u_{i})^{q}}{1+[1-\bar{q}^{2}(1-\lambda_{i})]u_{i}},$
(25)

where the summations are restricted to the eigenmodes that contribute the IB
encoder, i.e., those with $u_{i}>0$. We depict the optimal frontiers of Rényi
IB in Fig 1.

To complete our analysis of Rényi IB, we note that the analytical solution of
Ref [22] is a limiting case of our results. In the limit $q\to 1$, Eq (21)
reads

$\frac{1}{\beta}=g_{q=1}(u_{i},\lambda_{i})=\frac{1-\lambda_{i}}{1+u_{i}\lambda_{i}}\implies
u_{i}^{(q=1)}=\frac{\beta(1-\lambda_{i})-1}{\lambda_{i}}.$ (26)

Recalling that $u_{i}=r_{i}w_{i}^{2}$, we see immediately that this solution
is identical to that in Ref [22, Lemma 4.1].

Figure 2: Solving Rényi and Jeffreys IB problems yields representations that
are close to Shannon IB optimal. Plotted on the Shannon information plane, the
solutions to Shannon (dotted), Jeffreys (dashed) and Rényi (solid) IB problems
are nearly indistinguishable. For the Rényi case, we depict the results for a
range of Rényi orders $q$ (see color bar). Inset: We depict the gap between
the maximum achievable and encoded relevant Shannon informations,
$I^{\text{max}}(T;Y)$ and $I(T;Y)$ respectively, as a function of the
extracted Shannon information $I(T;X)$. This gap vanishes in the low and high-
information limits, $I(T;X)\\!\to\\!0$ and $I(T;X)\\!\to\\!\infty$. The
satellite peaks result from the fact that the solutions to Shannon, Jeffreys
and Rényi IB problems go through structural transitions at different values of
$I(T;X)$ even though these transitions occur at the same set of critical
tradeoff parameters. Here the eigenvalues of $\Sigma_{X|Y}\Sigma_{X}^{-1}$ are
$\lambda_{i}=0.1,0.2,0.3,0.5,0.7,0.8$.

##  V Jeffreys Information Bottleneck for Gaussian variables

The technique in the previous section applies also to the IB problems, based
on other statistical divergences. In this section, we consider Jeffreys IB,
defined by the loss function

$\mathcal{L}_{\mathrm{J}}[Q_{T|X}]=J(T;X)-\beta J(T;Y).$ (27)

Here $J(X;Y)$ denotes Jeffreys information which is a mutual dependence
measure, defined by

$J(X;Y)\equiv\operatorname{D}_{\mathrm{J}}(P_{XY}\parallel P_{X}\otimes
P_{Y}),$ (28)

where $\operatorname{D}_{\mathrm{J}}$ is Jeffreys divergence [32],

$\operatorname{D}_{\mathrm{J}}(P\parallel
Q)=\frac{1}{2}[\operatorname{KL}(P\parallel Q)+\operatorname{KL}(Q\parallel
P)].$ (29)

For Gaussian correlated random variables, Jeffreys information takes a simple
form (see Appendix B for derivation)

$J(X;Y)=\frac{1}{2}\operatorname{tr}\left(\Sigma_{X}\Sigma_{X|Y}^{-1}-I\right).$
(30)

Using the linear encoder from Eq (9), the information terms in Eq (27) read

$\displaystyle J(T;X)$
$\displaystyle=\frac{1}{2}\operatorname{tr}\left(WRW^{\mkern-1.5mu\mathsf{T}}\right)$
(31) $\displaystyle J(T;Y)$
$\displaystyle=\frac{1}{2}\operatorname{tr}\left((I+WRW^{\mkern-1.5mu\mathsf{T}})\frac{1}{I+W\Lambda
RW^{\mkern-1.5mu\mathsf{T}}}-I\right).$ (32)

where $W$, $\Lambda$ and $R$ are defined in Eqs (14-16).

To solve the IB optimization, we differentiate of the loss function with
respect to the mixing matrix $W$ and set the resulting derivative to zero,
yielding

$I=\beta\frac{1}{I+\Lambda
RW^{\mkern-1.5mu\mathsf{T}}W}\left(I-(I+RW^{\mkern-1.5mu\mathsf{T}}W)\frac{1}{I+\Lambda
RW^{\mkern-1.5mu\mathsf{T}}W}\Lambda\right),$ (33)

where we use the identities $\partial A^{-1}/\partial a=-A^{-1}(\partial
A/\partial a)A^{-1}$ and
$\partial\operatorname{tr}(CABA^{\mkern-1.5mu\mathsf{T}})/\partial A=2CAB$ for
symmetric matrices $B$ and $C$. We see that this equation is solvable by
taking $W^{\mkern-1.5mu\mathsf{T}}W$ to be diagonal. Substituting Eq (20) into
the above equation and solving for $u_{i}$ gives

$u_{i}=\frac{\sqrt{\beta(1-\lambda_{i})}-1}{\lambda_{i}}.$ (34)

Since $u_{i}=r_{i}w_{i}^{2}\geq 0$, we see that this solution is valid only
when the tradeoff parameter $\beta$ in greater than the critical value

$\beta_{c}^{(i)}=\frac{1}{1-\lambda_{i}}.$ (35)

We note that this critical value is identical to that of Rényi IB [Eq (22)].
Finally substituting Eq (34) into Eqs (31-32) leads to

$\displaystyle J(T;X)$
$\displaystyle=\frac{1}{2}\sum_{i}^{\;\;\beta>\beta_{c}^{(i)}}\frac{\sqrt{\beta(1-\lambda_{i})}-1}{\lambda_{i}}$
(36) $\displaystyle J(T;Y)$
$\displaystyle=\frac{1}{2}\sum_{i}^{\;\;\beta>\beta_{c}^{(i)}}\frac{1-\lambda_{i}}{\lambda_{i}}\frac{\sqrt{\beta(1-\lambda_{i})}-1}{\sqrt{\beta(1-\lambda_{i})}}.$
(37)

where the summations are limited to the modes that contribute to the encoder,
i.e., those with $\beta_{c}^{(i)}<\beta$. In Fig 1, we depict an example of
the Jeffreys IB optimal frontier, computed from the above equations. We
emphasize that while Shannon information is equivalent to Rényi information
with $q=1$, Jeffreys information is not a special case of Rényi information.

##  VI Discussion & Conclusion

In Fig 2, we depict the solutions to the original, Rényi and Jeffreys IB
problems on the Shannon information plane. We see that these solutions are
very close to the optimal frontier, characterized by the Shannon IB solutions.
This result suggests that formulating and solving an IB problem, defined with
alternative correlation measures other than Shannon information, could offer a
strategy for obtaining an approximate solution to the original IB problem. To
better illustrate the differences between the solutions to the original, Rényi
and Jeffreys IB problems, the inset shows how much less relevant Shannon
information the optimal representations of Rényi and Jeffreys IB encode,
compared to the Shannon IB optimal representation. We see that the differences
are maximum at intermediate information and vanish in the low and high-
information limits. In addition, the Shannon information gaps exhibit
satellite peaks, resulting from structural the transition of the IB solutions.
We note that although these transitions occur at the same critical tradeoff
parameters [Eqs (22) & (35)], they generally correspond to different values of
extracted Shannon information.

To sum up, we consider generalized IB problems in which the mutual information
is replaced by mutual dependence measures, based on Rényi and Jeffreys
divergences. We obtain exact analytical solutions for the case of Gaussian
correlated random variables, generalizing the results of Ref [22]. We show
that the fundamental IB tradeoff between relevance and compression holds also
for correlation measures other than Shannon information. Our analyses reveal
structural transitions in the optimal representations, similar to that in the
original IB method [22]. Interestingly the critical tradeoff parameters are
the same for original, Rényi and Jeffreys IB problems, even though the
solutions are distinct.

We anticipate that our work will find application in physics of correlated
components which relies on Rényi-generalization of entropy and information to
quantify entanglement. In addition, our characterization of Jeffreys IB could
have implications for understanding the generalization properties of Gibbs
learning algorithms of which the generalization gap is proportional to
Jeffreys information between fitted models and training data. Finally we note
that the conditional IB problem, in which the compression term $I(T;X)$ is
replaced by $I(T;X\,|\,Y)$, becomes non-trivial for generalized information
measures since the chain rule does not hold for Rényi and Jeffreys
information—that is, given the Markov constraint $T$–$X$–$Y$, we have
$I(T;X\,|\,Y)=I(T;X)-I(T;Y)$ for Shannon information, but in general,
$\Omega(T;X\,|\,Y)\neq\Omega(T;X)-\Omega(T;Y)$. The logical steps in our
analyses are readily generalizable to conditional IB problems.

###### Acknowledgements.

This work was supported in part by the National Science Foundation, through
the Center for the Physics of Biological Function (PHY-1734030), the Simons
Foundation and the Sloan Foundation.

##  Appendix A Rényi information for Gaussian variables

In this appendix, we derive Rényi mutual information for Gaussian correlated
variables. Using the definition from Eqs (3-4), we write down Rényi mutual
information for continuous random variables,

$I_{q}(X;Y)=\frac{1}{q-1}\ln\int\\!dxdy\,p_{X}(x)p_{Y}(y)\left(\frac{p_{XY}(x,y)}{p_{X}(x)p_{Y}(y)}\right)^{q}.$
(38)

where $p_{X}$, $p_{Y}$ and $p_{XY}$ denote the probability density functions
of $X$, $Y$ and $(X,Y)$, respectively. We consider Gaussian correlated random
variables

$\genfrac{[}{]}{0.0pt}{0}{X}{Y}\sim
N(\mu,\Sigma)\;\;\text{with}\;\;\mu=\genfrac{[}{]}{0.0pt}{0}{\mu_{X}}{\mu_{Y}}\;\;\text{and}\;\;\Sigma=\left[\begin{matrix}\Sigma_{X}&\Sigma_{XY}\\\
\Sigma_{YX}&\Sigma_{Y}\end{matrix}\right].$ (39)

In this case, the joint probability density is given by

$p_{XY}(x,y)=\frac{\exp\left\\{-([\begin{smallmatrix}x\\\
y\end{smallmatrix}]-\mu)^{\mkern-1.5mu\mathsf{T}}\Sigma^{-1}([\begin{smallmatrix}x\\\
y\end{smallmatrix}]-\mu)\right\\}}{|2\pi\Sigma|^{1/2}}$ (40)

The product of the marginal distributions is equal to a joint distribution but
with $\Sigma_{XY}$ and $\Sigma_{YX}$ set to zero, i.e.,

$p_{X}(x)p_{Y}(y)=\frac{\exp\left\\{-([\begin{smallmatrix}x\\\
y\end{smallmatrix}]-\mu)^{\mkern-1.5mu\mathsf{T}}\bar{\Sigma}^{-1}([\begin{smallmatrix}x\\\
y\end{smallmatrix}]-\mu)\right\\}}{|2\pi\bar{\Sigma}|^{1/2}}$ (41)

where $\bar{\Sigma}=\left[\begin{smallmatrix}\Sigma_{X}&\cdot\\\
\cdot&\Sigma_{Y}\end{smallmatrix}\right]$. Substituting the above densities
into Eq (38) and performing the resulting Gaussian integration over $x$ and
$y$ gives

$I_{q}(X;Y)=\frac{1}{q-1}\ln\frac{|q\Sigma^{-1}+(1-q)\bar{\Sigma}^{-1}|^{-1/2}}{|\Sigma|^{q/2}|\bar{\Sigma}|^{(1-q)/2}}.$
(42)

The determinants of the covariance matrices are given by

$|\Sigma|=|\Sigma_{Y}|\times|\Sigma_{X|Y}|\quad\text{and}\quad|\bar{\Sigma}|=|\Sigma_{Y}|\times|\Sigma_{X}|,$
(43)

where $\Sigma_{X|Y}=\Sigma_{X}-\Sigma_{XY}\Sigma_{Y}^{-1}\Sigma_{YX}$ and we
use the identity

$\left|\begin{matrix}A&B\\\ C&D\end{matrix}\right|=|D|\times|A-BD^{-1}C|.$
(44)

We now consider the numerator in Eq (42),

$\displaystyle\left|q\Sigma^{-1}+(1-q)\bar{\Sigma}^{-1}\right|$
$\displaystyle=\left|\Sigma^{-1}\left(q\bar{\Sigma}+(1-q)\Sigma\right)\bar{\Sigma}^{-1}\right|$
$\displaystyle=\frac{1}{|\Sigma|\times|\bar{\Sigma}|}\left|\begin{matrix}\Sigma_{X}&(1-q)\Sigma_{XY}\\\
(1-q)\Sigma_{YX}&\Sigma_{Y}\end{matrix}\right|$
$\displaystyle=\frac{\left|I-(1-q)^{2}(I-\Sigma_{X|Y}\Sigma_{X}^{-1})\right|}{|\Sigma_{Y}|\times|\Sigma_{X|Y}|},$
(45)

where the last equality follows from Eqs (43-44). Finally we write down the
Rényi information for Gaussian variables

$I_{q}(X;Y)=\frac{1/2}{q-1}\ln\frac{|\Sigma_{X|Y}\Sigma_{X}^{-1}|^{1-q}}{\left|I-(1-q)^{2}(I-\Sigma_{X|Y}\Sigma_{X}^{-1})\right|}.$
(46)

This expression is identical to Eq (5) (with $\bar{q}=1-q$).

##  Appendix B Jeffreys information for Gaussian variables

The Jeffreys information is defined via

$J(X;Y)\equiv\operatorname{D}_{\mathrm{J}}(P_{XY}\parallel P_{X}\otimes
P_{Y}),$ (47)

where $\operatorname{D}_{\mathrm{J}}$ is Jeffreys divergence [32],

$\operatorname{D}_{\mathrm{J}}(P\parallel
Q)=\frac{1}{2}[\operatorname{KL}(P\parallel Q)+\operatorname{KL}(Q\parallel
P)].$ (48)

For Gaussian correlated $X$ and $Y$, the Jeffreys information follows
immediately from the KL divergence between two multivariate Gaussian
distributions

$\displaystyle\operatorname{KL}(N(\mu_{0},\Sigma_{0})\parallel
N(\mu_{1},\Sigma_{1}))=\frac{1}{2}\bigg{(}\operatorname{tr}(\Sigma_{1}^{-1}\Sigma_{0}-I)$
$\displaystyle\hskip
56.9055pt+(\mu_{1}-\mu_{0})^{\mkern-1.5mu\mathsf{T}}\Sigma_{1}^{-1}(\mu_{1}-\mu_{0})+\ln\frac{|\Sigma_{1}|}{|\Sigma_{0}|}\bigg{)}.$
(49)

For $X$ and $Y$ described by Eq (39), we have $P_{XY}=N(\mu,\Sigma)$ and
$P_{X}\otimes P_{Y}=N(\mu,\bar{\Sigma})$, where
$\Sigma=\left[\begin{smallmatrix}\Sigma_{X}&\Sigma_{XY}\\\
\Sigma_{YX}&\Sigma_{Y}\end{smallmatrix}\right]$ and
$\bar{\Sigma}=\left[\begin{smallmatrix}\Sigma_{X}&\cdot\\\
\cdot&\Sigma_{Y}\end{smallmatrix}\right]$. As a result, we have

$\displaystyle\operatorname{KL}(P_{XY}\parallel P_{X}\otimes P_{Y})$
$\displaystyle=\frac{1}{2}\left(\operatorname{tr}(\bar{\Sigma}^{-1}\Sigma-I)+\ln\frac{|\bar{\Sigma}|}{|\Sigma|}\right)$
(50) $\displaystyle\operatorname{KL}(P_{X}\otimes P_{Y}\parallel P_{XY})$
$\displaystyle=\frac{1}{2}\left(\operatorname{tr}(\Sigma^{-1}\bar{\Sigma}-I)+\ln\frac{|\Sigma|}{|\bar{\Sigma}|}\right).$
(51)

We see that the logarithmic term drops out upon symmetrization [Eq (48)].
Substituting
$\bar{\Sigma}^{-1}=\left[\begin{smallmatrix}\Sigma_{X}^{-1}&\cdot\\\
\cdot&\Sigma_{Y}^{-1}\end{smallmatrix}\right]$ and the determinant formula in
Eq (43) into Eq (50) gives

$\operatorname{KL}(P_{XY}\parallel P_{X}\otimes
P_{Y})=\frac{1}{2}\ln\frac{|\bar{\Sigma}|}{|\Sigma|}=-\frac{1}{2}\ln|\Sigma_{X|Y}\Sigma_{X}^{-1}|$
(52)

which is the usual mutual information, as expected. To compute trace in Eq
(51), we write down the inverse of the covariance matrix,

$\displaystyle\Sigma^{-1}=\begin{pmatrix}\Sigma_{X|Y}^{-1}&-\Sigma_{X|Y}^{-1}\Sigma_{XY}\Sigma_{Y}^{-1}\\\
-\Sigma_{Y}^{-1}\Sigma_{YX}\Sigma_{X|Y}^{-1}&\Sigma_{Y|X}^{-1}\end{pmatrix}.$
(53)

Therefore we have

$\displaystyle\operatorname{tr}(\Sigma^{-1}\bar{\Sigma}-I)$
$\displaystyle=\operatorname{tr}(\Sigma^{-1}(\bar{\Sigma}-\Sigma))$
$\displaystyle=\operatorname{tr}\left(\left[\begin{smallmatrix}\Sigma_{X|Y}^{-1}&-\Sigma_{X|Y}^{-1}\Sigma_{XY}\Sigma_{Y}^{-1}\\\
-\Sigma_{Y}^{-1}\Sigma_{YX}\Sigma_{X|Y}^{-1}&\Sigma_{Y|X}^{-1}\end{smallmatrix}\right]\left[\begin{smallmatrix}\cdot&-\Sigma_{XY}\\\
-\Sigma_{YX}&\cdot\end{smallmatrix}\right]\right)$
$\displaystyle=\operatorname{tr}(\Sigma_{X|Y}^{-1}\Sigma_{XY}\Sigma_{Y}^{-1}\Sigma_{YX})+\operatorname{tr}(\Sigma_{Y}^{-1}\Sigma_{YX}\Sigma_{X|Y}^{-1}\Sigma_{XY})$
$\displaystyle=2\operatorname{tr}(\Sigma_{XY}\Sigma_{Y}^{-1}\Sigma_{YX}\Sigma_{X|Y}^{-1})$
$\displaystyle=2\operatorname{tr}(\Sigma_{X}\Sigma_{X|Y}^{-1}-I),$ (54)

where the last equality follows from the identity
$\Sigma_{X|Y}=\Sigma_{X}-\Sigma_{XY}\Sigma_{Y}^{-1}\Sigma_{YX}$. Substituting
the above result into Eq (51) yields

$\operatorname{KL}(P_{X}\otimes P_{Y}\parallel
P_{XY})=\operatorname{tr}(\Sigma_{X}\Sigma_{X|Y}^{-1}-I)+\frac{1}{2}\ln|\Sigma_{X|Y}\Sigma_{X}^{-1}|.$
(55)

Finally eliminating the logarithmic term with Eq (52) leads to

$\displaystyle J(X;Y)$
$\displaystyle=\frac{1}{2}[\operatorname{KL}(P_{X}\otimes P_{Y}\parallel
P_{XY})+\operatorname{KL}(P_{XY}\parallel P_{X}\otimes P_{Y})]$
$\displaystyle=\frac{1}{2}\operatorname{tr}\left(\Sigma_{X}\Sigma_{X|Y}^{-1}-I\right).$
(56)

##  Appendix C Supplementary figure

Figure C1: The function $g_{q}(u,\lambda)$ [Eq (21)] decreases with $u$ from
$1\\!-\\!\lambda$ at $u\\!=\\!0$ and approaches zero as $u\\!\to\\!\infty$. As
a result, the equation $\beta^{-1}\\!=\\!g_{q}(u,\lambda)$ always has a unique
positive solution when $\beta\\!>\\!1/(1-\lambda)$. We consider only
$0\\!\leq\\!q\\!\leq\\!2$ since Rényi information for Gaussian variables can
diverge for $q\\!>\\!2$ [see Eq (5)].

## References

  * Tishby  _et al._ [1999] N. Tishby, F. C. N. Pereira, and W. Bialek, The information bottleneck method, in  _37th Allerton Conference on Communication, Control and Computing_, edited by B. Hajek and R. S. Sreenivas (University of Illinois, 1999) pp. 368–377.
  * Achille and Soatto [2018] A. Achille and S. Soatto, Information dropout: Learning optimal representations through noisy computation, IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 2897 (2018).
  * Achille and Soatto [2018] A. Achille and S. Soatto, Emergence of invariance and disentanglement in deep representations, Journal of Machine Learning Research 19, 1 (2018).
  * Tian  _et al._ [2020] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, What makes for good views for contrastive learning?, in  _Advances in Neural Information Processing Systems_, Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 6827–6839.
  * Palmer  _et al._ [2015] S. E. Palmer, O. Marre, M. J. Berry, and W. Bialek, Predictive information in a sensory population, Proceedings of the National Academy of Sciences 112, 6908 (2015).
  * Wang  _et al._ [2021] S. Wang, I. Segev, A. Borst, and S. Palmer, Maximally efficient prediction in the early fly visual system may support evasive flight maneuvers, PLOS Computational Biology 17, e1008965 (2021).
  * Still  _et al._ [2012] S. Still, D. A. Sivak, A. J. Bell, and G. E. Crooks, Thermodynamics of prediction, Physical Review Letters 109, 120604 (2012).
  * Gordon  _et al._ [2021] A. Gordon, A. Banerjee, M. Koch-Janusz, and Z. Ringel, Relevance in the renormalization group and in information theory, Physical Review Letters 126, 240601 (2021).
  * Kline and Palmer [2022] A. G. Kline and S. E. Palmer, Gaussian information bottleneck and the non-perturbative renormalization group, New Journal of Physics 24, 033007 (2022).
  * Strouse and Schwab [2019] D. Strouse and D. J. Schwab, The information bottleneck and geometric clustering, Neural Computation 31, 596 (2019).
  * Alemi  _et al._ [2017] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, Deep variational information bottleneck, in  _International Conference on Learning Representations_ (2017).
  * Goyal  _et al._ [2019] A. Goyal, R. Islam, D. Strouse, Z. Ahmed, H. Larochelle, M. Botvinick, S. Levine, and Y. Bengio, Transfer and exploration via the information bottleneck, in  _International Conference on Learning Representations_ (2019).
  * Bialek  _et al._ [2001] W. Bialek, I. Nemenman, and N. Tishby, Predictability, Complexity, and Learning, Neural Computation 13, 2409 (2001).
  * Shamir  _et al._ [2010] O. Shamir, S. Sabato, and N. Tishby, Learning and generalization with the information bottleneck, Theoretical Computer Science 411, 2696 (2010), Algorithmic Learning Theory (ALT 2008).
  * Bialek  _et al._ [2020] W. Bialek, S. E. Palmer, and D. J. Schwab, What makes it possible to learn probability distributions in the natural world? (2020), arXiv:2008.12279 [cond-mat.stat-mech] .
  * Ngampruetikorn and Schwab [2022] V. Ngampruetikorn and D. J. Schwab, Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality, in  _Advances in Neural Information Processing Systems_, Vol. 35, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc., 2022) pp. 9784–9796.
  * Chalk  _et al._ [2016] M. Chalk, O. Marre, and G. Tkacik, Relevant sparse codes with variational information bottleneck, in  _Advances in Neural Information Processing Systems_, Vol. 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Inc., 2016) pp. 1957–1965.
  * Poole  _et al._ [2019] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, On variational bounds of mutual information, in  _Proceedings of the 36th International Conference on Machine Learning_, Proceedings of Machine Learning Research, Vol. 97, edited by K. Chaudhuri and R. Salakhutdinov (PMLR, 2019) pp. 5171–5180.
  * Tschannen  _et al._ [2020] M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic, On mutual information maximization for representation learning, in  _International Conference on Learning Representations_ (2020).
  * Wu  _et al._ [2019] T. Wu, I. Fischer, I. L. Chuang, and M. Tegmark, Learnability for the information bottleneck, Entropy 21, 924 (2019).
  * Ngampruetikorn and Schwab [2021] V. Ngampruetikorn and D. J. Schwab, Perturbation theory for the information bottleneck, in  _Advances in Neural Information Processing Systems_, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 21008–21018.
  * Chechik  _et al._ [2005] G. Chechik, A. Globerson, N. Tishby, and Y. Weiss, Information bottleneck for Gaussian variables, Journal of Machine Learning Research 6, 165 (2005).
  * Rényi [1961] A. Rényi, On measures of entropy and information, in  _Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability_ , Vol. 1, edited by J. Neyman (1961) pp. 547–561.
  * Horodecki  _et al._ [2009] R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Quantum entanglement, Reviews of Modern Physics 81, 865 (2009).
  * Eisert  _et al._ [2010] J. Eisert, M. Cramer, and M. B. Plenio, Colloquium: Area laws for the entanglement entropy, Reviews of Modern Physics 82, 277 (2010).
  * Hastings  _et al._ [2010] M. B. Hastings, I. González, A. B. Kallin, and R. G. Melko, Measuring renyi entanglement entropy in quantum monte carlo simulations, Physical Review Letters 104, 157201 (2010).
  * Singh  _et al._ [2011] R. R. P. Singh, M. B. Hastings, A. B. Kallin, and R. G. Melko, Finite-temperature critical behavior of mutual information, Physical Review Letters 106, 135701 (2011).
  * Herdman  _et al._ [2017] C. M. Herdman, P. N. Roy, R. G. Melko, and A. D. Maestro, Entanglement area law in superfluid 4He, Nature Physics 13, 556 (2017).
  * Islam  _et al._ [2015] R. Islam, R. Ma, P. M. Preiss, M. Eric Tai, A. Lukin, M. Rispoli, and M. Greiner, Measuring entanglement entropy in a quantum many-body system, Nature 528, 77 (2015).
  * Bergschneider  _et al._ [2019] A. Bergschneider, V. M. Klinkhamer, J. H. Becher, R. Klemt, L. Palm, G. Zürn, S. Jochim, and P. M. Preiss, Experimental characterization of two-particle entanglement through position and momentum correlations, Nature Physics 15, 640 (2019).
  * Brydges  _et al._ [2019] T. Brydges, A. Elben, P. Jurcevic, B. Vermersch, C. Maier, B. P. Lanyon, P. Zoller, R. Blatt, and C. F. Roos, Probing Rényi entanglement entropy via randomized measurements, Science 364, 260 (2019).
  * Jeffreys [1946] H. Jeffreys, An invariant form for the prior probability in estimation problems, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences 186, 453 (1946).
  * Aminian  _et al._ [2021] G. Aminian, Y. Bu, L. Toni, M. Rodrigues, and G. Wornell, An exact characterization of the generalization error for the gibbs algorithm, in  _Advances in Neural Information Processing Systems_, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 8106–8118.
  * van Erven and HarremoÃ«s [2014] T. van Erven and P. HarremoÃ«s, RÃ©nyi divergence and kullback-leibler divergence, IEEE Transactions on Information Theory 60, 3797 (2014).
  * Liese and Vajda [2006] F. Liese and I. Vajda, On divergences and informations in statistics and information theory, IEEE Transactions on Information Theory 52, 4394 (2006).