Feature Learning and Signal Propagation in Deep Neural Networks

Yizhang Loustjc
Chris Mingarddc,comp
Yoonsoo Namcomp
Soufiane Hayousing

stjcSt John's College, University of Oxford, Oxford, UK
dcPTCL, University of Oxford, Oxford, UK
compDepartment of Physics, University of Oxford, UK
singDepartment of Mathematics, National University of Singapore

Yizhang<EMAIL_ADDRESS>Soufiane<EMAIL_ADDRESS>
Machine Learning, ICML

Recent work by <cit.> sheds light on an intriguing pattern that occurs during the training of deep neural networks: some layers align

much more with data compared to other layers
(where the alignment is defined as the euclidean product of the tangent features matrix and the data labels matrix).
The curve of the alignment as a function of layer index (generally) exhibits an ascent-descent pattern where the maximum is reached for some hidden layer.

In this work, we provide the first explanation for this phenomenon. We introduce the Equilibrium Hypothesis which connects this alignment pattern to signal propagation in deep neural networks. Our experiments demonstrate an excellent match with the theoretical predictions.

§ INTRODUCTION

The empirical success of modern Deep Neural Networks (DNNs) has sparked a growing interest in the theoretical understanding of these models. An important development in this direction was the introduction of the Neural Tangent Kernel (NTK) framework by <cit.>, which provides a dual view of Gradient Descent (GD) in function space. The NTK is the dot product kernel of Tangent Features (gradient features), given by

K(x, x') = \nabla_{\theta} f(x) \nabla_{\theta} f(x')^T,

where $f$ is the network output and $\theta$ is the vector of model parameters. The NTK has been the subject of an extensive body of literature, both in the NTK regime where the NTK remains constant during training, e.g. <cit.>, and when the NTK changes during training, e.g <cit.>. The latter is called the feature learning regime since the tangent features $\nabla_{\theta} f(x)^T$ evolve during training in some data-dependent directions; we say that tangent features and the NTK adapt to the data. A simple way to quantify how much these features adapt to data is by measuring the alignment between the tangent features and data labels <cit.>, which is given by the normalized euclidean product between the tangent kernel matrix $\hat{K}$ and data labels matrix $YY^T$ (see <ref>).

Example Alignment Hierarchies for 20 layer FFNNs trained on Fashion MNIST (L) and CIFAR10 (R).

Role of hidden layers.
In the context of DNNs, hidden layers act as successive embeddings that evolve in data-dependent directions during training. However, it is still unclear how different layers contribute to model performance.
A possible approach in this direction is the study of feature learning in each layer, i.e. how features adapt to data as we train the model. A simple way to do this is by measuring the change in the values of pre-activations as we train the network; this was used by <cit.> to engineer a network parameterization that maximizes feature learning. To capture the dynamics of GD, <cit.> studied the change in tangent features instead of pre-activations. For each layer $l$, they measured the alignment between the layer tangent kernel matrix $\hat{K}_l$ (where $K_l(x,x') = \nabla_{\theta_l} f(x) \nabla_{\theta_l} f(x')^T$, $\theta_l$ being the vector of parameters in the $l^{th}$ layer). The kernel $\hat{K}_l$ can be seen as the NTK of the network if only the parameters of the $l^{th}$ layer are allowed to change (other layers are frozen, see <ref> for a detailed discussion). The authors demonstrated the existence of an alignment pattern in which the tangent features of some hidden layers are significantly more aligned with data labels compared to other layers. We call this pattern the Alignment Hierarchy (illustrated in <ref>). In general, the alignment reaches its maximum for some hidden layer and tends to be minimal in the external layers (first and last).
To the best of our knowledge, no explanation has been provided for this phenomenon in the literature.
In this paper, we propose an explanation based on the theory of signal propagation in randomly initialized neural networks. Our intuition is based on the observation that tangent kernels can be decomposed as an Hadamard product of two quantities linked to how signal propagates in DNNs.

Forward-BackwardDecomposition. We show in <ref> that for a depth $L$ neural network, with proper normalization, the $l^{th}$ layer tangent kernel matrix can be written as

\begin{equation}\label{eq:intro_decomposition}
\hat{K}_l =  \overrightarrow{K}_l \circ \overleftarrow{K}_l
\end{equation}

where $\overrightarrow{K}_l$ is a kernel that depends only on the $l$ first layers, and $\overleftarrow{K}_l$ is a kernel that is essentially governed by the last $L-l+1$ layers. The $\circ$ denotes the Hadamard product. Intuitively, this decomposition suggests that tangent features are the result of the collaboration between the forward kernel $\overrightarrow{K}_l$ and the backward kernel $\overleftarrow{K}_l$. For DNNs, it is expected that depth has some layer-dependent effect on the kernels $\overrightarrow{K}_l$ and $\overleftarrow{K}_l$ and therefore on $K_l$. Understanding the effect of depth on how information propagates has been the subject of a large body of work which constitute what is known as the theory of signal propagation in DNNs. We briefly introduce this theory in the next paragraph (a more detailed discussion is provided in <ref>).

Signal propagation in DNNs. A recent line of research (e.g <cit.>) studied the dynamics of signal propagation in randomly initialized DNNs. Under mild conditions, in the infinite (layer) width limit, each neuron $y(.)$ in the network converges in distribution to a Gaussian process (GP) <cit.>. Hence, in this limit, the covariance kernel captures all the properties of the neurons at initialization <cit.>[GPs are fully characterized by their covariance kernel.]. From a geometric perspective, the correlation/covariance measures the angular distortion between vectors. Hence, the covariance represents a geometric information[Here, the information refers purely to the covariance between two vectors, and is different from the information-theoretic definition of information.]. As shown in <ref>, the kernels $\overrightarrow{K}_l$ and $\overleftarrow{K}_l$ are covariance kernels; $\overrightarrow{K}_l$, resp. $\overleftarrow{K}_l$, represents a notion of forward, resp. backward, geometric information (covariance). Inspired by this observation, we provide an explanation of the feature alignment pattern based on how the geometric information, encoded in $\overrightarrow{K}_l$ and $\overleftarrow{K}_l$, changes with depth. Notably, we prove that these geometric information become degenerate in the limit of large depth which translates to the information being lost as we increase depth. We say that there is a information loss as we increase depth and we characterize this loss in the case of fully-connected DNNs. Could this information loss be the reason behind the observed alignment pattern? More precisely, could some notion of balance between information loss in kernels $\overrightarrow{K}_l$ and $\overleftarrow{K}_l$ explain the alignment pattern? We formulate this intuition as the Equilibrium hypothesis (EH) which we introduce in <ref>.

Our contributions are three-fold. Firstly, we introduce and empirically validate the Equilibrium Hypothesis, which provides an explanation for the alignment pattern. More precisely, we give an explanation for the fact that the alignment peaks at some hidden layer. Secondly, we provide a comprehensive analysis of this hypothesis in the case of fully-connected neural networks. Most notably, we prove that layers with indices $\mathbf{l = \Theta(L^{3/5})}$ achieve a notion of equilibrium in geometric information in the limit of large depth $L$. Our experiments yield excellent match between theoretical and empirical results. Finally, we provide an empirical analysis of the connection between the Alignment Hierarchy (illustrated in <ref>) and the generalization error.

§ FEATURE LEARNING IN DNNS

Consider a neural network model consisting of $L$ layers of widths $(N_l)_{1 \leq l \leq L}$, $N_0 = d$, and let $\theta = (\theta_l)_{1 \leq l \leq L}$ be the flattened vector of weights indexed by the layer's index, and $P$ be the dimension of $\theta$. Given an input $x \in \reals^d$, the network is described by the set of equations

z_l(x) = \mathcal{F}_l(\theta_l, z_{l-1}(x)), \quad 1\leq l\leq L,

where $\mathcal{F}_l$ is a mapping that defines the $l^{th}$ layer, e.g. fully-connected, convolutional, etc.

The network output function $f$ is given by $f_\theta(x) = \nu(z_L(x)) \in \reals^o$ where $\nu : \mathbb{R}^{N_L} \rightarrow \mathbb{R}^o$ is a mapping of choice, and $o$ is the dimension of the output, e.g. the number of classes for a classification problem.

We consider a loss function $\mathbb{L}: \mathbb{R}^{o} \times \mathbb{R}^{o} \rightarrow \mathbb{R}$ and a dataset $\dataset = \{(x_1,y_1), \ldots, (x_{n},y_n)\}$. The network is then trained by minimizing the empirical loss $\loss: \mathbb{R}^P \to \mathbb{R}$ given by

\begin{align*}
\loss(\theta) = \frac{1}{n} \sum_{i =1}^{n} \mathbb{L}(f_{\theta}(x_i), y_i).
\end{align*}

Tangent Features. <cit.> introduced the Neural Tangent Kernel (NTK), which provides a dual view of the training procedure; it links gradient updates in parameter space to a kernel gradient descent in function space. The NTK is given by

\begin{equation}\label{eq:ntk_decomposition}
\begin{aligned}
K^L_{\theta}(x, x') &= \nabla_{\theta} f_\theta(x) \nabla_{\theta} f_\theta(x')^T\\ &= \sum_{l=1}^L \nabla_{\theta_l} f_\theta(x) \nabla_{\theta_l} f_\theta(x')^T \in \mathbb R^{o \times o}.
\end{aligned}
\end{equation}

The tangent features are the feature maps of the NTK, given by the output gradients w.r.t the network parameters, namely

\begin{equation}
\begin{aligned}
\Psi_{\theta}(x):=\nabla_{\theta} f_{\theta}(x)^T \in \mathbb{R}^{P \times o}.
\end{aligned}
\end{equation}

For the sake of simplicity, we remove  $\theta$ and $L$ in the kernel notation and define $\mathbf{\Psi} \in \mathbb{R}^{P \times on}$, the tangent feature matrix over the training dataset $\mathcal{D}$. $\mathbf{\Psi}$ is the horizontal concatenation of $\Psi(x_1), \ldots, \Psi(x_{n})$. The corresponding tangent kernel matrix is given by $\hat{\mathbf{K}}=\mathbf{\Psi}^T  \mathbf{\Psi} \in \mathbb{R}^{on \times on}$.

§.§ Quantifying the role of each layer

<cit.> demonstrated that neural network based kernel methods (e.g. infinite width NTK regime) can achieve near parity with finite width networks, and have near identical posterior distributions, over a range of architectures (e.g. LSTMs, WideResNet) and datasets (e.g. Cifar10).
However, optimizer hyperparameters are known to affect generalization, suggesting an extra layer of complexity <cit.>.
Furthermore, <cit.> proved that the large depth limit of the NTK regime is trivial in the sense that the limiting NTK has rank 1. This suggests that this kernel regime, where tangent features are fixed at initialization, cannot explain the inductive bias of ultra deep neural networks, and that feature learning (tangent features evolve during training) could be the backbone of generalization in very deep networks.
Finite width CNNs operating in the feature learning regime have also been shown to generalise better than their infinite width counterparts <cit.>.
A question that arises is that of the role of each layer in feature learning – unfortunately there is no consensus on how feature learning should best be measured. One way to approach this question is by analyzing the behaviour of a network where only the parameters in a given layer are allowed to change with gradient updates. We call this approach `parameter freezing'.

Parameter freezing.
The derivations of the results in this paragraph are provided in <ref>. We omitted the details in the main text to meet space constraints.

Consider a classification task with $o=k$ classes and assume that the dataset is balanced, i.e. $\frac{1}{n} \sum_{i=1}^n y_i \approx \frac{1}{k} \mathbf{1},$ where $\mathbf{1} \in \reals^k$ is the vector of ones.

Intuition: One way to measure feature learning in the $l^{th}$ layer is by freezing the parameters in the other $L-1$ layers and tracking the change in network output when we update the parameters of the $l^{th}$ layer.

Given a layer index $l$, suppose that we freeze all the weights in the other $L-1$ layers and allow the parameters in the $l^{th}$ layer to be updated with a gradient step. Consider the vector $f_t(X) \in \reals^{kn}$ which consists of the concatenation of the sequence $(f_t(x_i))_{1 \leq i \leq n}$ (here $X$ refers to the concatenation of $x_1, x_2, \dots, x_n$). Then, with one gradient step, the update $\delta f_t(X)$ is given by

$$\delta f_t(X) = -\eta \hat{\mathbf{K}}_l (Z_t - Y),$$

where $\hat{\mathbf{K}}_l$ is the tangent kernel matrix for layer $l$, $Z_t: = (\textup{softmax}(f_t(x_i)))_{1 \leq i \leq n}, Y \in \reals^{on}$, and $\eta$ is the normalized learning rate (i.e. $\eta = \textup{LR}/n$). Hence, the kernel matrix $\hat{\mathbf{K}}_l$ controls the change in the vector $f_t(X)$.

To understand the interaction between $\hat{\mathbf{K}}_l$ and $Y$, let us see what happens at the first step of gradient descent. At initialization, the output function $f$ is random and has an average accuracy of an random classifier, i.e. a random guess with uniform probability $1/k$ for each class. In this case, the average update is given by

\begin{equation}\label{eq:dynamics_classif}
\delta f_t(X) \approx -\eta \hat{\mathbf{K}}_l \left(\frac{1}{k} \mathbf{1} - Y\right) \approx \eta \hat{\mathbf{K}}_l \tilde{Y} ,
\end{equation}

where $\tilde{Y} = \left(I - \frac{1}{kn} \mathbf{1} \mathbf{1}^T\right) Y$. $\tilde{Y}$ is a centered version of $Y$. Using <ref>, we obtain

\|\delta f_t(X) \| \leq \eta \textup{Tr}(\hat{\mathbf{K}}_l) \|\tilde{Y}\|,

with equality if and only if $\hat{\mathbf{K}}_l$ and $\tilde{Y}$ are aligned, i.e. $\hat{\mathbf{K}}_l \propto \tilde{Y} \tilde{Y}^T$. Hence, the maximum update of the network output is induced by a perfect alignment between $\hat{\mathbf{K}}_l$ and $\tilde{Y} \tilde{Y}^T$.

Conclusion: Assume that only parameters in the $l^{th}$ layer are updated with gradient descent. Then, the alignment between the tangent kernel matrix $\hat{\mathbf{K}}_l$ and the centered data labels matrix $\tilde{Y} \tilde{Y}^T$ controls the magnitude of change in $f_t(X)$ at early training.

Although this analysis is performed at early training, we hypothesize that this alignment quantifies feature learning in each layer during training[ <ref> shows that the increase in alignments occurs mostly during the first few epochs.]. Hence, we propose to use this alignment to quantify the role of each layer. This alignment can also be interpreted as a measure of how informative each layer's gradient update is. In fact, if we update all the layers (no parameter freezing), the change in function update projected on data labels vector is approximately

\begin{equation}\label{eq:dynamics_align}
\langle\tilde{Y}, \delta f_t(X)\rangle \approx \eta \tilde{Y}^T \hat{\mathbf{K}} \tilde{Y} = \sum_{l} \eta \textup{Tr}(\hat{\mathbf{K}}_l \tilde{Y} \tilde{Y}^T),
\end{equation}

Hence, $\textup{Tr}(\hat{\mathbf{K}}_l \tilde{Y} \tilde{Y}^T)$ quantifies how gradient updates in each layer contribute to output function moving in the direction of the training target.

§.§ Centered Kernel Alignment (CKA)
From the analysis in <ref>, we define the centered kernel alignment between two kernel matrices $\mathbf{K}, \mathbf{K}^{\prime} \in \mathbb{R}^{on \times on}$ by

\begin{equation}
\begin{aligned}
A\left(\mathbf{K}, \mathbf{K}^{\prime}\right)=\frac{\operatorname{Tr}\left[\mathbf{K}_{c} \mathbf{K}_{c}^{\prime}\right]}{\left\|\mathbf{K}_{c}\right\|_{F}\left\|\mathbf{K}_{c}^{\prime}\right\|_{F}}
\end{aligned}
\end{equation}

where $\mathbf{K}_{c} = \mathbf{C}\mathbf{K}\mathbf{C}$, $\mathbf{C} = \mathbf{I} - \frac{1}{on} \mathbf{1}\mathbf{1}^T$ is the centering matrix ($\mathbf{1}$ is a vector with all entries being $1$), and $\|.\|_F$ is the Frobenius norm. The CKA was used by <cit.> as a measure of feature learning.

Remark 1. For all kernels $\mathbf{K}, \mathbf{K}^{\prime}$, we have $A\left(\mathbf{K}, \mathbf{K}^{\prime}\right) \in [0,1]$ with $A\left(\mathbf{K}, \mathbf{K}^{\prime}\right) = 1$ if and only if the kernel matrices are colinear.

To quantify the role of the $l^{th}$ layer ($1\leq l \leq L$), we use $\mathbf{K} = \hat{\mathbf{K}}_l = \mathbf{\Psi}^T_l \mathbf{\Psi}_l$ where $\mathbf{\Psi}_l \in \reals^{P_l \times on}$ are the tangent features of layer $l$ (the horizontal concatenation of the tangent features $(\Psi_l(x_i) = \nabla_{\theta_l}f_\theta(x_i))_{1\leq i \leq n}$), $P_l$ is the dimension of $\theta_l$, and $\mathbf{K}' = YY^T$ where $Y \in  \mathbb{R}^{on}$ is the horizontal concatenation of output vectors in the dataset $ \mathcal{D}$.

\begin{equation}
\begin{aligned}
A_l:= A\left(\mathbf{K}_l, \mathbf{K}_l^{\prime}\right)=\frac{\tilde{Y}^T \mathbf{\Psi}_l^T \mathbf{\Psi}_l \tilde{Y}}{\left\|\mathbf{C} \mathbf{\Psi}_l^T \mathbf{\Psi}_l \mathbf{C}\right\|_{F}\left\|\tilde{Y}\right\|^2}
\end{aligned}
\end{equation}

Layerwise alignment hierarchy for the MNIST and CIFAR10 datasets when trained on an FFNN with depth 10 and width 256. Left hand panels show progression of loss and layer alignment with iterations of SGD. Right hand panel shows layer alignment at the end of training.

§.§ Alignment Hierarchy
The alignment $A_l$ acts as a measure of how much layers contribute to the performance of the network (<ref>). <cit.> observed an interesting hierarchical structure in the alignments $A_l$ for different neural network architectures. During the course of training, the increase in $A_l$ for some middle layers is sharp and significantly larger than the alignments of other layers. We illustrate this pattern in
<ref> on MNIST and CIFAR10 datasets with fully-connected networks. It appears that alignments of some layers increase much more effectively with gradient updates over others [This also suggests the existence of an implicit layer selection phenomenon during training].
We call this pattern the Alignment Hierarchy, and aim to understand the reason why the alignment peaks at some hidden layer. For both datasets in <ref>, the pattern is similar and shows large alignments for some middle layers. Further empirical results on K-MNIST and FashionMNIST datasets and VGG19/ResNet18 architectures are provided in <ref>. Motivated by this empirical findings, we formulate the Equilibrium Hypothesis in the next section, where we give an explanation of the Alignment Hierarchy using tools from the theory of signal propagation at initialization.

§ THE EQUILIBRIUM HYPOTHESIS

In this section, we aim to understand a specific aspect of the AH: why does the alignment peak in some intermediate layer? We argue that this is a result of the dynamics of signal propagation in DNNs at initialization. For the sake of simplicity, we restrict our theoretical analysis to fully-connected DNNs, although our results can in principle be extended to other architectures.

Fully-connected Feedforward Neural Network (FFNN).
Given an input $x \in \reals^d$, and a set of weights $(W_l)_{1 \leq L}$, we consider the following neural network model

\begin{equation}
\begin{aligned}
z_1(x) &= W_1 x\\
z_l(x) &= W_l \phi(z_{l-1}(x)), \quad 2 \leq l \leq L,
\end{aligned}
\label{eq:ffnn_1}
\end{equation}

where $W_{l} \in \mathbb{R}^{N \times N}$ with $o = N_{L} = 1$[For simplicity, we restrict our analysis to rectangular networks with 1D output networks.], $W_1 \in \mathbb{R}^{N \times d}$, $N$ is the network width, and $\phi$ is the ReLU activation function given by $\phi(v) = (\max(v_i,0))_{1\leq i \leq p}$ for $v \in \reals^p$.

For each layer, the weights are initialized with i.i.d Gaussian variables $W \sim \mathcal{N}(0,\frac{2}{\textup{fan\_in}})$, where `fan_in' refers to the dimension of the previous layer. This standard initialization scheme is known as the He initialization <cit.> or the Edge of Chaos initialization <cit.>.

§.§ Tangent Kernel decomposition

The tangent kernel at hidden layer $l$ can be expressed as

\begin{align*}
K_l(x,x') &= \nabla_{\theta_l}f(x) \cdot \nabla_{\theta_l}f(x')\\
&= \sum_{i,j} \phi(z_{l-1}^j(x))\phi(z_{l-1}^j(x')) \, \fracpartial{f}{z^{i}_l}(x)  \fracpartial{f}{z^{i}_l}(x').
\end{align*}

Since $K_l$ is a sum over $N^2$ terms, we consider the average kernel $\bar{K}_l$ given by
\bar{K}_l = \frac{1}{N^2} K_l.
In matrix form[Bold characters in <ref> refer to kernel matrices and not kernels], $\bar{K}_l$ can be written as the Hadamard product of two kernels

\begin{equation}\label{eq:kernel_forward_backward_decomposition}
\bar{\mathbf{K}}_l = \overrightarrow{\mathbf{K}}_l \circ \overleftarrow{\mathbf{K}}_l,
\end{equation}

where $\overrightarrow{K}_l(x,x') = \frac{1}{N}\phi(z_{l-1}(x)) \cdot \phi(z_{l-1}(x'))$ is the forward features kernel and $\overleftarrow{K}_l(x,x') = \frac{1}{N} \fracpartial{f_{l:L}}{z}(z_l(x)) \cdot \fracpartial{f_{l:L}}{z}(z_l(x'))$ is the backward tangent features kernel, where $f_{l:L}$ is the function that maps the $l^{th}$ layer to the network output. The above decomposition illustrates the collaborative roles played by kernels $\overrightarrow{K}$ and $\overleftarrow{K}$ in constructing the tangent features at layer $l$. To depict the role of each kernel, we use some tools from the theory of signal propagation at initialization.

§.§ Signal propagation at initialization

Consider an FFNN of type (<ref>). The weights $W$ are randomly initialized. Hence, the network neurons and output are random processes at initialization. Understanding the properties of such processes is crucial for both training and generalization <cit.>.
It turns out that in the limit of infinite width $N \to \infty$, the neurons act as Gaussian processes. To see this, consider the simple case of a two layers FFNN. Since the weights are i.i.d, neurons $\{z_1^i(.)\}_{i\in[1:N]}$ are also iid Gaussian processes with covariance kernel given by
$\E_W[z_1^i(x) z_1^i(x')] = \frac{2 x \cdot x'}{d} $. Using the Central Limit Theorem, as $N \rightarrow \infty$, $z^i_{2}(x)$ is a Gaussian variable for any input $x$ and index $i \in [1:o]$. Moreover, the random variables $\{z^i_2(x)\}_{i \in [1:o]}$ are iid. Hence, the processes $z^i_{2}(.)$ can be seen as independent (across $i$), centered Gaussian processes with some covariance kernel $q_2$. This Gaussian process limit of FFNNs was first proposed by <cit.> in the single layer case and was extended to  multi-layer networks by <cit.> where the authors showed that in $l^{th}$ layer, neurons become i.i.d Gaussian processes with covariance kernel $q_l$ in the limit $N \to \infty$. This result holds for all standard neural network architectures <cit.>. A more complete review of this theory is provided in <ref>. The covariance kernel $q_l(x,x')$ is a measure of the angular distortion between the vectors $z_l^i(x)$ and $z_l^i(x')$. Thus, the covariance kernel carries some information on how inputs propagate within the network. We formalize this notion of information in the next definition.

Given random weights $W$, we say that a kernel function $k$ is a geometric information if it can be expressed as $k(x,x') = \mathbb{E}_W[g(W,x)g(W,x')]$ for some function $g:\reals\times \reals^d \to \reals$.

Hereafter, we simply use `information'[This is different from the information-theoretic definition of information.] to refer to the geometric information in <ref>. Recall the kernel decomposition given by <ref>

\begin{equation*}
\bar{\mathbf{K}}_l = \overrightarrow{\mathbf{K}}_l \circ \overleftarrow{\mathbf{K}}_l.
\end{equation*}

The kernels $\overrightarrow{K}_l$ and $\overleftarrow{K}_l$ depend on random weights $W$ and thus are random. We propose to study the average behaviour instead, where we consider the average kernels. For $\overrightarrow{K}_l$, the average kernel is given  $\E_W[\overrightarrow{K}_l(x,x')] = \E_W[\phi(z^1_{l-1}(x)) \phi(z^1_{l-1}(x'))]$ (since the neurons $z_l^j$ are identically distributed) which represents a geometric information as per <ref>. We call this average kernel the forward information. A standard result in signal propagation is that kernels $\overrightarrow{K}_l$ and $\overleftarrow{K}_l$ converge to their corresponding expected value in the limit of infinite width <cit.> which justifies our choice of the average kernel $\E_W[\overrightarrow{K}_l(x,x')]$. A similar result holds for $\overleftarrow{K}_l$. Let us formalize these definitions.

Given a layer index $l$, we define the forward information $I^f_{l,N}$ by

\begin{align*}
I^f_{l, N}(x,x') = \mathbb{E}_W\left[\phi(z_{l-1}^1(x))\phi(z_{l-1}^1(x'))\right],
\end{align*}

where the expectation is taken w.r.t $W$. Similarly, the backward information $I^b_{l,N}(x,x')$ is defined by

\begin{align*}
I^b_{l, N}(x,x') = \mathbb{E}_W\left[\fracpartial{f_{l:L}}{z^1}(z_l(x))  \fracpartial{f_{l:L}}{z^1}(z_l(x'))\right].
\end{align*}

$I^f_{l, N}$ and $I^b_{l, N}$ are geometric information in accordance with <ref>.

§.§ Information loss in the large depth limit

A classical result in the theory of signal propagation is that the information deteriorates with depth $L$ <cit.> in the sense that the covariance kernels converge to trivial kernels (e.g. constant kernels) in the limit of infinite depth. This is a natural result of the randomness that adds to the neurons with each additional layer. This deterioration occurs with some rate (convergence rate to the trivial kernel w.r.t to $L$) which we call the information loss in the following definition. For two non-negative sequences $(a_n)_{n\geq 0}, (b_n)_{n\geq 0}$, we write $a_n = \Theta_n(b_n)$ if there exists two constant $M_1, M_2 > 0$ such that for all $n$, $M_1 \, b_n \leq a_n \leq M_2 \, b_n$.

Let $(g_n(.))_{n \geq 0}$ be a sequence of real-valued functions defined on some set $\mathcal{C} \subset \mathbb{R}^m, m\geq 1$. Assume that $g_n$ converges uniformly to some constant $\kappa$ as $n \to \infty$ and that there exists a non-negative sequence $(r_n)_{n\geq 0}$ such that $\sup_{t \in \mathcal{C}}|g_n(t) - \kappa| = \Theta_n(r_n)$. We say that $g_n$ has an information loss of order $r_n$.

The $\IL$ characterizes the rate at which the sequence $(g_n(t))_{n\geq 1}$ `forgets' the input $t$ since, by definition, the limiting value $\kappa$ is independent of $t$[Note that $\IL$ is unique up to a $\Theta$ factor, e.g. an $\IL$ of $n^{-1}$ is the same as an $\IL$ of $n^{-1} \times (2 + n^{-1})$].
In our case, we would expect similar behaviour of the forward information $I^f_{l, N}$ in the limit $l \to \infty$[Note that $I^f_{l, N}$ does not depend on the network depth $L$.], and the backward information $I^b_{l, N}$ in some $(l,L)-$dependent limit(see <ref>). For instance, assume that $L$ is large and consider a small $l$. Then, the forward information has minimal information loss (forward information loss occurs in the limit $l \rightarrow \infty$) while the backward information suffers from deterioration as it depends on the $L-l+1$ last layers, and thus it suffers from the accumulated randomness as it travels back through the network. The opposite happens when $l$ is large, e.g. $l \sim L$. This antagonistic roles of the forward/backward information is key in understanding the behaviour of the tangent kernel $K_l$: there exists a layer index $l_0$ for which the information loss for forward and backward information is comparable. By <ref>, we expect $K_l$ to suffer from the deterioration that affects either the forward/backward information. Hence, we hypothesize that a layer with comparable forward/backward information loss is better conditioned to align with data.

The Equilibrium Hypothesis (EH). Let $\IL^f_{l,N}$, resp. $\IL^b_{l,N}$, be the information loss of the sequence $(I^f_{l,N})_{l\geq 1}$, resp. $(I^b_{l,N})_{l\geq 1}$. The layers with the highest alignments with data labels are the ones that satisfy the equilibrium property
$$\IL^f_{l,N} = \Theta\left(\IL^b_{l,N}\right).$$

The EH conjectures that balanced information loss between forward and backward information is related to high alignment with data. Our intuition is as follows: balance between forward and backward information at layer $l$ relates to informative updates of layer $l$'s parameter $\theta_l$, which corresponds to informative updates in the tangent feature $\mathbf{\Psi}_l$ leading to greater alignment with data. To see this for an FFNN, consider function update:

\begin{align*}
\delta f_t(X) &= -\eta \hat{\mathbf{K}} \nabla_{f}\mathbb{L}(f(X), Y) \\
&= -\eta \sum_{l} \hat{\mathbf{K}_l} \nabla_{f}\mathbb{L}(f(X), Y),
\end{align*}

where the contribution of updating $\theta_l$ to the overall change in $f_t(X)$ is  $- \eta \hat{\mathbf{K}_l} \nabla_{f}\mathbb{L}(f(X), Y)$, where $\eta =\textup{LR}/n$ is the normalized learning rate. Since the kernel matrices satisfy $\bar{\mathbf{K}}_l = \overrightarrow{\mathbf{K}}_l \circ \overleftarrow{\mathbf{K}}_l$, we expect that any deterioration of the kernels $\overrightarrow{K}_l$ and $\overleftarrow{K}_l$ would affect $K_l$. Our intuition is that the parameter update in layer $l$ benefits from the equilibrium property which guarantees that none of the kernels is more deteriorated than the other. This ensures that the function space update benefits from both forward and backward geometric information.  Note that at initialization, with the same analysis of <ref>, the function update due to $\theta_l$ can be approximated by $\delta f_t(X) \approx \eta \hat{\mathbf{K}_l} \tilde{Y}$. This suggest that the high alignment between tangent features $\mathbf{\Psi}_{l}(X)$ and data labels $Y$ is associated with informative parameter update. A more in-depth discussion of this result is provided in <ref>. In addition, informative parameter update is associated with informative tangent feature update (<ref>).

Data with $x=10j$ in the plot corresponds to layer alignments for a FFNN with depth $10j$ trained on the MNIST/FashionMNIST/CIFAR10 datasets. The brighter the color, the closer the corresponding layer's alignment is to the maximum alignment across all layers. x indicates the layer where largest alignment occurs. See <ref> for further experiments on Fashion MNIST showing larger learning rates decrease the $y$-intercept (alignment peaks in earlier layers).

§.§ The Equilibrium in infinite width FFNNs

For FFNNs, we provide a comprehensive analysis of the Equilibrium property in the infinite width limit. We characterize the layers where the equilibrium is achieved and we confirm our theoretical findings with empirical results. For the sake of simplicity, we restrict our theoretical analysis to the sphere $\Sphere=\{ x \in \reals^d, \|x\|=\sqrt{d}\}$ where $\|.\|$ is the euclidean norm. The generalization to $\reals^d$ is straightforward. To avoid issues with col-linearity in the dataset, we consider the set $E_\epsilon$, parameterized by $\epsilon \in (0,1)$, defined by

\begin{equation}\label{eq:set_E}
E_\epsilon = \{(x,x') \in (\Sphere)^2: \frac{1}{d} x\cdot x' < 1 - \epsilon\}
\end{equation}

The next result characterizes the information loss $\IL$ of the forward/backward information defined above in the limit of infinite width ($N \to \infty$). In this limit, the forward information has an information loss of $l^{-2}$.

Let $\epsilon \in (0,1)$, and consider the set $E_\epsilon$ as in <ref>. Define $I^f_{l,\infty}(x,x') := \lim\limits_{N \to \infty} I^f_{l,N}(x,x')$ for all $x,x' \in \reals^d$. We have that

\sup_{(x,x') \in E_\epsilon}|I^f_{l,\infty}(x,x') - 1/2| = \Theta_l(l^{-2}).

<ref> is a corollary of a previous result that appeared in <cit.>. The proof of the latter relies on an asymptotic analysis of the forward covariance kernel in the limit of large $l$, coupled with a uniform bounding of the convergence rate (See <ref> for more details).

The forward information $I^f_{l,N}$ does not depend on depth $L$. On the other hand, the backward information $I^b_{l,N}$ depends both on $l$ and the depth $L$. Therefore, in order to study the asymptotic information loss, we should specify how $l$ grows relatively to $L$. In the next result, we study the two cases where $l \ll L$ or $l =\lfloor \alpha L \rfloor$.

Let $\epsilon \in (0,1)$, and consider the set $E_\epsilon$ as in <ref>. Define $I^b_{l,\infty}(x,x') := \lim\limits_{N \to \infty} I^b_{l,N}(x,x')$ for all $x,x' \in \reals^d$. Then,

* If $l = \lfloor \alpha L \rfloor$ where $\alpha \in (0,1)$ is a constant, then there exists a constant $\mu$ such that in the limit $L \to \infty$,

\sup_{(x,x')\in E_\epsilon} |I^b_{l,\infty}(x,x') - \mu| = \Theta_L(\log(L) L^{-1})

* In the limit $l, L \to \infty$ with $l/L \to 0$,

\sup_{(x,x')\in E_\epsilon} |I^b_{l,\infty}(x,x')| = \Theta_{l,L}(\left(L/l\right)^{-3}).

A key ingredient in the proof of <ref> in the so-called Gradient Independence assumption. In the literature on signal propagation at initialization (e.g. <cit.>), results on gradient backpropagation rely on the assumption that the weights used for backpropagation are independent from the ones used for forward propagation. <cit.> showed that this assumption yields exact computations of gradient covariance and NTK in the infinite width limit.

We refer the reader to <ref> for more details.

In the case of infinite width FFNN, using Theorems <ref> and <ref>, we show which layers satisfy the equilibrium property.

Under the conditions of Theorems <ref> and <ref>, the equilibrium for an FFNN is achieved for layers with index

l = \Theta_L(L^{3/5})

where $L$ is the network depth.

<ref> indicates that layers that satisfy the equilibrium property verify $l = \Theta_L(L^{3/5})$. In logarithmic scale, this implies  $\log(l) \in \left[\frac{3}{5} \log(L) + C_1,  \frac{3}{5} \log(L) + C_2\right] $ where $C_1, C_2$ are constants that depends on $\epsilon$ from Theorems <ref> and <ref>. <ref> shows the layer alignments $A_l$'s after training an FFNN (width 256) on different datasets. We fit the line $\log(l)=3/5 \log(L)+C$ by finding the constant $C$ that minimizes the squared error. We also perform a simple linear regression to see if the slope is close to $3/5$ (line of best fit). All experiments show an excellent match with the theoretical line $3/5 \log{L}+C$ (which was derived for infinite width networks).

§ ALIGNMENT AND HYPERPARAMETERS

Generalization and feature learning have been linked to optimizer hyperparameter choices (e.g. <cit.>). While this paper largely focuses on the $L^{3/5}$ scaling law in the EH ($l = \Theta(L^{3/5})$) it would be incomplete without a brief discussion of the constant term in this equation (the y-intercept in <ref>).
In this section, we observe that the layer index with the greatest alignment can be significantly affected by choice of optimizer hyperparameters, and good generalization is associated with a well defined peak away from the last layer. See <ref> for clear demonstration of the change in $C$ with learning rate.

To understand (1) the effect of optimizer hyperparameters on the alignment and (2) the impact of the Alignment Hierarchy on the generalization error, we trained multiple models with different datasets with a selection of batch sizes, learning rates and optimizers. We show the corresponding AH pattern, generalization error and generalization gap for CIFAR10 on Resnet18 and VGG19 in <ref>.
The colour corresponds to the final test loss, and the number to the loss gap.
<ref> weakly suggests that large alignments with data labels, especially for the middle layers, correlate with good generalization properties.
Note that the peak for the FFNN occurs in the early layers; VGG the middle layers, and for Resnet18 very near the last layer. If the EH is also valid for resnets, the peak should move towards the middle layers as depth increases. This is left for future work.
For further experiments, including on random labels, see <ref>.

FFNN on Fashion MNIST

Resnet18 on CIFAR10

VGG19 on CIFAR10

Alignment hierarchy as a function of batch size and learning rate for (a) Fashion MNIST on a FFNN (b) CIFAR10 on a Resnet18 and (c) CIFAR10 on a VGG19. The location of the peak is different for each dataset, architecture and hyperparameter combination – note that AH cannot be seen here, as it predicts a scaling law with layers, but not the constants (see <ref>). The background colour denotes the test loss, and the number in the top left corner is the generalisation loss gap. Each model was trained stopping after a train loss of 0.1 or 500 epochs, whichever was sooner. There is a clear positive correlation between the height of the peak and the overall generalisation error for each example given here.

Batch size and learning rate dependence
<ref> shows that the AH is strongly affected by both batch size and learning rate. Typically, smaller batch sizes and larger learning rates lead to better generalisation where convergence is possible.
They also lead to a larger peak in the AH, and the peak shifts towards the center. Although it is expected that learning rate and batch size affect training, we do not currently have a theoretical explanation for these effects on the AH.

Connection to Stable Rank, Alignment & Fisher Information

<cit.> observed a correlation between improved generalisation and (1) an increasing majority of the singular values of $\mathbf{\Psi}$ are very small with a few very large (2) the label vector $Y$ is increasingly aligned with the large singular directions in $\mathbf{\Psi}$. These are the conditions for maximsing CKA. In <ref>, we extend these results to Layerwise CKA.
The layerwise CKA $A_l$ can be decomposed into a product of the inverse square root of its stable rank and the correlation term between the eigenvectors and the label vector $Y$. The stable rank measures an effective dimension of the internal representations of the neural network.
CIFAR10 with degrees of data randomisation was trained on VGG19, and showed a lower overall layerwise stable rank for the better generalising models. The alignment term varied the most across generalisation errors, with much greater alignment (in the later layers) for the best generalising model than the others.
Larger overall alignment therefore correlates with lower stable rank; and an earlier peak appears to coincide with earlier alignment between $Y$ and large singular directions in $\mathbf{\Psi}$.
We might expect lower dimensional internal representations and earlier, larger alignment to correlate with good generalisation.
We also investigated Fisher Information Matrix $\mathbf{J}=\mathbf{\Psi}\mathbf{\Psi}^T$ (for MSE loss), and $\mathbf{K}$.
We use observations from <cit.> to link properties of $\mathbf{J}$ that coincide with larger CKA to explain why more alignment is likely to correlate with better generalisation.

§ RELATED WORK

To the best of our knowledge, the AH has only been discussed in <cit.>. However, the significance of AH goes beyond a simple feature learning pattern. The AH can be seen a structural implicit regularization effect, i.e. a regularization effect that is purely induced by the depth.
Traditionally, implicit regularization refers to any hidden regularization effect induced by the training algorithm. For example, it is widely believed that the implicit regularization effect of Stochastic Gradient Descent (SGD) <cit.> is mainly driven by the small-batch sampling noise <cit.>.
However, recent empirical findings such as <cit.> demonstrated that DNNs can still achieve high accuracy on some image datasets with full-batch GD. <cit.> demonstrated that increasing batch size by several orders of magnitude on ImageNet does not affect generalization error significantly, suggesting that classical implicit regularization theories that rely on SGD noise to explain generalization are not sufficient to explain why neural networks generalize at all (further backed up by the near parity achieved by kernel methods <cit.>).
These results suggest that implicit regularization might occur via other mechanisms than previously thought, one of them could this purely structural effect that results in the Alignment Hierarchy.
Given interest in understanding how the amount of feature learning affects generalization and transfer learning, we believe this is a promising topic for future work.

§ CONCLUSION AND LIMITATIONS

In this paper, we introduced the Equilibrium Hypothesis (EH) which connects information flow at initialization to tangent features alignment with data labels. the EH explains the alignment hierarchy, illustrated in <ref>. Our empirical results showed an excellent match with the theoretical prediction $l = \Theta(L^{3/5})$ for FFNN on different datasets in <ref>, and presented empirical evidence that earlier alignment correlates with better generalisation <ref>.
Finally, we used connections between layerwise CKA, the stable rank and Fisher Information to present a theoretical case for this observation.
There are still multiple open questions to answer, e.g. the impact of the architecture on the alignment hierarchy. As demonstrated in <ref>, it seems that the alignment pattern is sensitive to the choice of the architecture. We leave this topic for future work.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

titleImplicit Regularization via Neural Feature Alignment

booktitleAdvances in Neural Information Processing Systems
titleNeural Tangent Kernel: Convergence and Generalization in Neural Networks

titleThe asymptotic spectrum of the Hessian of DNN throughout training

journaltitlearXiv preprint arXiv:1905.13654
titleMean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks

booktitleInternational Conference on Machine Learning
titleAn investigation into neural net optimization via hessian eigenvalue density

journaltitlearXiv preprint arXiv:2009.10685
titleTensor Programs III: Neural Matrix Laws

journaltitlearXiv preprint arXiv:1910.12478
titleTensor programs i: Wide feedforward or recurrent neural networks of any architecture are Gaussian processes

journaltitleICML 2021
titleTensor Programs IV: Feature Learning in Infinite-Width Neural Networks

journaltitle30th Conference on Neural Information Processing Systems
titleExponential expressivity in deep neural networks through transient chaos

booktitleInternational Conference on Learning Representations
titleDeep Information Propagation

booktitleInternational Conference on Learning Representations
titleDeep Neural Networks as Gaussian Processes

booktitleInternational Conference on Machine Learning
titleOn the Impact of the Activation Function on Deep Neural Networks Training

Springer Science & Business Media

titleBayesian Learning for Neural Networks

booktitleInternational Conference on Learning Representations
titleGaussian Process Behaviour in Wide Deep Neural Networks

journaltitlearXiv preprint arXiv:2007.15801
titleFinite versus infinite neural networks: an empirical study

journaltitleAdvances in neural information processing systems
titleWide neural networks of any depth evolve as linear models under gradient descent

journaltitlearXiv preprint arXiv:1805.08522
titleDeep learning generalizes because the parameter-function map is biased towards simple functions

journaltitleJournal of Machine Learning Research
titleIs SGD a Bayesian sampler? Well, almost

titleOn the Implicit Biases of Architecture & Gradient Descent

titleDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

booktitleAdvances in neural information processing systems
titleMean field residual networks: On the edge of chaos

journaltitlearXiv preprint arXiv:1902.04760
titleScaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation

journaltitleICML 2018
titleDynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

journaltitlearXiv preprint arXiv:1609.04836
titleOn large-batch training for deep learning: Generalization gap and sharp minima

journaltitlearXiv preprint arXiv:1906.05392
titleGeneralization guarantees for neural networks via harnessing the low-rank structure of the jacobian

journaltitlearXiv preprint arXiv:2003.02139
titleRethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

journaltitleProceedings of the IEEE
titleGradient-based learning applied to document recognition

journaltitleAdvances in Neural Information Processing Systems
titleImageNet Classification with Deep Convolutional Neural Networks

booktitleICLR (Workshop)
titleFinding flatter minima with sgd

journaltitlearXiv preprint arXiv:1706.10239
titleTowards understanding generalization of deep learning: Perspective of loss landscapes

titleStochastic Training is Not Necessary for Generalization

journaltitlearXiv preprint arXiv:1706.02677
titleAccurate, large minibatch sgd: Training imagenet in 1 hour

journaltitleNature Communications
titleRandom synaptic feedback weights support error backpropagation for deep learning

booktitleThe 22nd International Conference on Artificial Intelligence and Statistics
titleUniversal statistics of fisher information in deep neural networks: Mean field approach

journaltitlePhysical Review E
titleJamming transition as a paradigm to understand the loss landscape of deep neural networks

titleRapid Feature Evolution Accelerates Learning in Neural Networks

journaltitlearXiv preprint arXiv:1703.07115
titleLayer-wise training of deep networks using kernel similarity

booktitleNeurIPS Workshop on Advances in Approximate Bayesian Inference
titleSample-then-optimize posterior sampling for bayesian linear models

ACM New York, NY, USA

journaltitleJournal of the ACM (JACM)
titleSampling from large matrices: An approach through geometric functional analysis

booktitle2007 15th European signal processing conference
titleThe effective rank: A measure of effective dimensionality

American Mathematical Soc.

titleMethods of information geometry

journaltitlearXiv preprint arXiv:1707.09564
titleA pac-bayesian approach to spectrally-normalized margin bounds for neural networks

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

titleImplicit Regularization via Neural Feature Alignment

booktitleAdvances in Neural Information Processing Systems
titleNeural Tangent Kernel: Convergence and Generalization in Neural Networks

titleThe asymptotic spectrum of the Hessian of DNN throughout training

journaltitlearXiv preprint arXiv:1905.13654
titleMean-field Behaviour of Neural Tangent Kernel for Deep Neural Networks

booktitleInternational Conference on Machine Learning
titleAn investigation into neural net optimization via hessian eigenvalue density

journaltitlearXiv preprint arXiv:2009.10685
titleTensor Programs III: Neural Matrix Laws

journaltitlearXiv preprint arXiv:1910.12478
titleTensor programs i: Wide feedforward or recurrent neural networks of any architecture are Gaussian processes

journaltitleICML 2021
titleTensor Programs IV: Feature Learning in Infinite-Width Neural Networks

journaltitle30th Conference on Neural Information Processing Systems
titleExponential expressivity in deep neural networks through transient chaos

booktitleInternational Conference on Learning Representations
titleDeep Information Propagation

booktitleInternational Conference on Learning Representations
titleDeep Neural Networks as Gaussian Processes

booktitleInternational Conference on Machine Learning
titleOn the Impact of the Activation Function on Deep Neural Networks Training

Springer Science & Business Media

titleBayesian Learning for Neural Networks

booktitleInternational Conference on Learning Representations
titleGaussian Process Behaviour in Wide Deep Neural Networks

journaltitlearXiv preprint arXiv:2007.15801
titleFinite versus infinite neural networks: an empirical study

journaltitleAdvances in neural information processing systems
titleWide neural networks of any depth evolve as linear models under gradient descent

journaltitlearXiv preprint arXiv:1805.08522
titleDeep learning generalizes because the parameter-function map is biased towards simple functions

journaltitleJournal of Machine Learning Research
titleIs SGD a Bayesian sampler? Well, almost

titleOn the Implicit Biases of Architecture & Gradient Descent

titleDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

booktitleAdvances in neural information processing systems
titleMean field residual networks: On the edge of chaos

journaltitlearXiv preprint arXiv:1902.04760
titleScaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation

journaltitleICML 2018
titleDynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

journaltitlearXiv preprint arXiv:1609.04836
titleOn large-batch training for deep learning: Generalization gap and sharp minima

journaltitlearXiv preprint arXiv:1906.05392
titleGeneralization guarantees for neural networks via harnessing the low-rank structure of the jacobian

journaltitlearXiv preprint arXiv:2003.02139
titleRethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

journaltitleProceedings of the IEEE
titleGradient-based learning applied to document recognition

journaltitleAdvances in Neural Information Processing Systems
titleImageNet Classification with Deep Convolutional Neural Networks

booktitleICLR (Workshop)
titleFinding flatter minima with sgd

journaltitlearXiv preprint arXiv:1706.10239
titleTowards understanding generalization of deep learning: Perspective of loss landscapes

titleStochastic Training is Not Necessary for Generalization

journaltitlearXiv preprint arXiv:1706.02677
titleAccurate, large minibatch sgd: Training imagenet in 1 hour

journaltitleNature Communications
titleRandom synaptic feedback weights support error backpropagation for deep learning

booktitleThe 22nd International Conference on Artificial Intelligence and Statistics
titleUniversal statistics of fisher information in deep neural networks: Mean field approach

journaltitlePhysical Review E
titleJamming transition as a paradigm to understand the loss landscape of deep neural networks

titleRapid Feature Evolution Accelerates Learning in Neural Networks

journaltitlearXiv preprint arXiv:1703.07115
titleLayer-wise training of deep networks using kernel similarity

booktitleNeurIPS Workshop on Advances in Approximate Bayesian Inference
titleSample-then-optimize posterior sampling for bayesian linear models

ACM New York, NY, USA

journaltitleJournal of the ACM (JACM)
titleSampling from large matrices: An approach through geometric functional analysis

booktitle2007 15th European signal processing conference
titleThe effective rank: A measure of effective dimensionality

American Mathematical Soc.

titleMethods of information geometry

journaltitlearXiv preprint arXiv:1707.09564
titleA pac-bayesian approach to spectrally-normalized margin bounds for neural networks