# The rigidity of biconservative surfaces in $\operatorname{Sol_{3}}$

Dorel Fetcu  Department of Mathematics and Informatics
Gh. Asachi Technical University of Iasi
Bd. Carol I, 11A
700506 Iasi, Romania<EMAIL_ADDRESS>
###### Abstract.

We consider biconservative surfaces in $\operatorname{Sol_{3}}$, find their
local equations, and then show that all biharmonic surfaces in this space are
minimal.

###### Key words and phrases:

Biconservative Surfaces, Biharmonic Surfaces, Thurston Geometries

###### 2020 Mathematics Subject Classification:

53C40, 31B30, 53C42

##  1\. Introduction

Almost forty years ago, B.-Y. Chen [3] introduced biharmonic submanifolds of
Euclidean spaces $\mathbb{E}^{n}$, as isometric immersions with harmonic mean
curvature vector field. Then, in [4, 14] it was proved that biharmonic
surfaces in $\mathbb{E}^{3}$ are minimal, a result that led to the still open
Chen’s Conjecture that biharmonic submanifolds of Euclidean spaces are minimal
[3].

In the same era, independently and in a more abstract way, biharmonic maps
were defined by G.-Y. Jiang [12] as critical points of the $L^{2}-$norm of the
tension field. This type of a variational problem was suggested back in the
1964 by J. Eells and J. H. Sampson in their seminal paper [8]. We note that
the two definitions agree in Euclidean spaces $\mathbb{E}^{n}$.

Since ambient spaces with non-positive curvatures do not allow for interesting
compact examples, most of the research has been done on biharmonic
submanifolds of Euclidean spheres and other spaces with convenient curvature
properties (for detailed accounts see [10, 26]).

The notion of biconservative submanifolds was derived from the theory of
biharmonic submanifolds by only requiring the vanishing of the tangent part of
the bitension field. Although a rather new one, this subject is already well
established. If we are to illustrate only the literature on biconservative
surfaces, we refer to papers like [2, 9, 11, 17, 21, 22, 23, 25] to gain a
satisfactory (if fairly incomplete) imagine.

Studying the geometry of surfaces in $\operatorname{Sol_{3}}$ seems to be the
most challenging among the eight Thurston geometries. The lack of some
powerful tools used in other homogeneous $3$-manifolds to describe the
geometry of immersed surfaces lead to many difficulties that do not appear in
the case of the remaining seven geometries (see [6]). However, in spite of
these difficulties, there are important results on the existence and
uniqueness of constant mean curvature (CMC) spheres [6, 19, 20], on totally
umbilic surfaces [28], half-space theorems for minimal surfaces [5], as well
as Jenkins-Serrin type results for minimal [24] and CMC surfaces [15]. In a
local approach, one classified constant angle surfaces [16]. Also CMC
biharmonic surfaces were studied in [27] and it turned out that in
$\operatorname{Sol_{3}}$ they are actually minimal.

In our paper, we first prove that CMC biconservative surfaces in
$\operatorname{Sol_{3}}$ are minimal. Next, we show that non-CMC
biconservative surfaces are constant angle surfaces and find their local
equations. We conclude with the fact that all biharmonic surfaces in
$\operatorname{Sol_{3}}$ are minimal. This last result is similar to that for
biharmonic surfaces in $\mathbb{E}^{3}$ (see [3, 7]).

Conventions. Throughout the paper surfaces are oriented, we will use the
following sign conventions

$R(X,Y)Z=\nabla_{X}\nabla_{Y}Z-\nabla_{Y}\nabla_{X}Z-\nabla_{[X,Y]}Z,\quad\Delta=\operatorname{trace}\nabla^{2}=\operatorname{trace}(\nabla\nabla-\nabla_{\nabla}),$

and objects on $\operatorname{Sol_{3}}$ will be indicated by
$\overline{(\cdot)}$.

##  2\. Preliminaries

Biharmonic maps $\phi:M^{m}\to N^{n}$ between two Riemannian manifolds are
critical points of the bienergy functional

$E_{2}:C^{\infty}(M,N)\to\mathbb{R},\quad
E_{2}(\phi)=\frac{1}{2}\int_{M}|\tau(\phi)|^{2}dv,$

where $\tau(\phi)=\operatorname{trace}\nabla d\phi$ is the tension field of
$\phi$. The Euler-Lagrange equation, also called the biharmonic equation, was
derived by G.-Y. Jiang [12]

(2.1)
$\displaystyle\tau_{2}(\phi)=\Delta\tau(\phi)-\operatorname{trace}R^{N}(d\phi(\cdot),\tau(\phi))d\phi(\cdot)=0,$

where $\tau_{2}(\phi)$ is the bitension field of $\phi$.

Since any harmonic map is biharmonic, the important case, from the point of
view of biharmonicity, is that of non-harmonic biharmonic ones.

Next, if we consider a fixed map $\phi$ and let the domain metric vary, one
obtains a functional on the set $\mathcal{G}$ of Riemannian metrics on $M$

$\mathcal{F}_{2}:\mathcal{G}\to\mathbb{R},\quad\mathcal{F}_{2}(g)=E_{2}(\phi).$

Critical points of this functional are characterized by the vanishing of the
stress-energy tensor $S_{2}$ of the bienergy (see [17]). This tensor was
introduced in [13] as

$\displaystyle S_{2}(X,Y)$ $\displaystyle=$
$\displaystyle\frac{1}{2}|\tau(\phi)|^{2}\langle X,Y\rangle+\langle
d\phi,\nabla\tau(\phi)\rangle\langle X,Y\rangle-\langle
d\phi(X),\nabla_{Y}\tau(\phi)\rangle$ $\displaystyle-\langle
d\phi(Y),\nabla_{X}\tau(\phi)\rangle,$

and it satisfies

$\operatorname{div}S_{2}=-\langle\tau_{2}(\phi),d\phi\rangle.$

For isometric immersions,
$(\operatorname{div}S_{2})^{\sharp}=-\tau_{2}(\phi)^{\top}$, where
$\tau_{2}(\phi)^{\top}$ is the tangent part of the bitension field.

######  Definition 2.1.

A submanifold $\phi:M^{m}\to N^{n}$ of a Riemannian manifold $N^{n}$ is called
biconservative if $\operatorname{div}S_{2}=0$.

From the definition, it is easy to see that a submanifold is biconservative if
and only if the tangent part of the bitension field vanishes.

Now, let us consider a surface $\Sigma^{2}$ in a Riemannian manifold $N^{n}$,
with the unit normal vector field $\xi$. The Gauss and the Weingarten
equations of the surface

$\nabla^{N}_{X}Y=\nabla_{X}Y+\sigma(X,Y)\quad\textnormal{and}\quad\nabla^{N}_{X}\xi=-AX,$

hold for all vector fields $X$ and $Y$ tangent to the surface, where $\nabla$
is the induced connection on $\Sigma^{2}$, $\sigma$ is its second fundamental
form, and $A$ its shape operator. The mean curvature vector field of
$\Sigma^{2}$ is given by $H=f\xi$, where $f=(1/2)\operatorname{trace}A$ is the
mean curvature function.

If $f$ is constant, then $\Sigma^{2}$ is called a constant mean curvature
(CMC) surface.

The Codazzi equation of the surface reads, for any vector fields $X$, $Y$, $Z$
tangent to $\Sigma^{2}$, and any normal vector field $V$,

(2.2)
$\begin{array}[]{cl}\langle\bar{R}(X,Y)Z,V\rangle=&\langle(\nabla^{\perp}_{X}\sigma)(Y,Z),V\rangle-\langle(\nabla^{\perp}_{Y}\sigma)(X,Z),V\rangle,\end{array}$

where
$(\nabla^{\perp}_{X}\sigma)(Y,Z)=\nabla^{\perp}_{X}\sigma(Y,Z)-\sigma(\nabla_{X}Y,Z)-\sigma(Y,\nabla_{X}Z)$.

The Gauss equation of the surface is

(2.3) $\langle R(X,Y)Z,W\rangle=\langle\bar{R}(X,Y)Z,W\rangle+\langle
AY,Z\rangle\langle AX,W\rangle-\langle AX,Z\rangle\langle AY,W\rangle,$

for all tangent vector fields $X$, $Y$, $Z$, and $W$, where $R$ is the
curvature tensor of $\Sigma^{2}$.

Specializing a general result in [17] in the case of surfaces, we have the
following splitting of biharmonic equation theorem.

######  Theorem 2.2 ([17]).

A surface $\Sigma^{2}$ in a Riemannian manifold $N^{n}$ is biharmonic if and
only if

$\begin{cases}\Delta
f-f|A|^{2}-f\langle\operatorname{trace}(R^{N}(\cdot,\xi)\cdot),\xi\rangle=0\\\
A(\nabla f)+f\nabla
f+f\operatorname{trace}(R^{N}(\cdot,\xi)\cdot)^{\top}=0.\end{cases}$

######  Corollary 2.3.

A surface $\Sigma^{2}$ in a Riemannian manifold $N^{n}$ is biconservative if
and only if

(2.4) $A(\nabla f)+f\nabla
f+f\operatorname{trace}(R^{N}(\cdot,\xi)\cdot)^{\top}=0.$

Next, we shall briefly recall some basic facts on $\operatorname{Sol_{3}}$.
This Lie group is $\mathbb{R}^{3}$ with the Riemannian metric

$\langle,\rangle=e^{2z}dx^{2}+e^{-2z}dy^{2}+dz^{2},$

where $(x,y,z)$ are the canonical coordinates of $\mathbb{R}^{3}$.

A left-invariant orthonormal frame field $\\{E_{1},E_{2},E_{3}\\}$ with
respect to this metric, called the canonical frame, is defined by

$E_{1}=e^{-z}\frac{\partial}{\partial x},\quad
E_{2}=e^{z}\frac{\partial}{\partial y},\quad E_{3}=\frac{\partial}{\partial
z}.$

The Levi-Civita connection of $\operatorname{Sol_{3}}$ is then the following

(2.5)
$\begin{array}[]{lll}\bar{\nabla}_{E_{1}}E_{1}=-E_{3},&\bar{\nabla}_{E_{1}}E_{2}=0,&\bar{\nabla}_{E_{1}}E_{3}=E_{1}\\\
\\\
\bar{\nabla}_{E_{2}}E_{1}=0,&\bar{\nabla}_{E_{2}}E_{2}=E_{3},&\bar{\nabla}_{E_{2}}E_{3}=-E_{2}\\\
\\\
\bar{\nabla}_{E_{3}}E_{1}=0,&\bar{\nabla}_{E_{3}}E_{2}=0,&\bar{\nabla}_{E_{3}}E_{3}=0.\end{array}$

One can see that the vertical vector field $E_{3}$ foliates
$\operatorname{Sol_{3}}$ by vertical geodesics. Moreover, we get that the
sectional curvatures of the vertical plane fields $(E_{1},E_{3})$ and
$(E_{2},E_{3})$ are equal to $-1$, while that of the horizontal plane field
$(E_{1},E_{2})$ is equal to $1$.

The only totally geodesic surfaces in $\operatorname{Sol_{3}}$ are the leaves
of the first two canonical foliations

$\mathcal{F}_{1}\equiv\\{x=\operatorname{constant}\\}\quad\textnormal{and}\quad\mathcal{F}_{2}\equiv\\{y=\operatorname{constant}\\},$

which are isometric to the hyperbolic plane $\mathbb{H}^{2}$ (see [28]).

The leaves of the third canonical foliation
$\mathcal{F}_{3}\equiv\\{z=\operatorname{constant}\\}$ are isometric to
$\mathbb{R}^{2}$ with its usual flat metric and are minimal.

The curvature tensor $\bar{R}$ of $\operatorname{Sol_{3}}$ is given by (see
[28])

$\displaystyle\bar{R}(X,Y)Z$ $\displaystyle=$ $\displaystyle\langle Y,Z\rangle
X-\langle X,Z\rangle Y+2\langle Z,E_{3}\rangle(\langle X,E_{3}\rangle
Y-\langle Y,E_{3}\rangle X)$ $\displaystyle+2(\langle X,Z\rangle\langle
Y,E_{3}\rangle-\langle Y,Z\rangle\langle X,E_{3}\rangle)E_{3}.$

##  3\. Biconservative surfaces in $\operatorname{Sol_{3}}$

As we have seen, CMC biharmonic surfaces in $\operatorname{Sol_{3}}$ are
minimal ([27]). The following result shows that this stands true when we only
keep the weaker biconservative condition.

######  Proposition 3.1.

A CMC biconservative surface in $\operatorname{Sol_{3}}$ is minimal.

###### Proof.

From Corrolary 2.3 and the expression (2) of the curvature tensor of
$\operatorname{Sol_{3}}$, one can easily see that a CMC surface $\Sigma^{2}$
is biconservative if and only if either the vector field $E_{3}$ is either
normal or tangent to the surface.

If $E_{3}$ is normal, then $\Sigma^{2}$ is minimal and therefore we only have
to study the case when $E_{3}$ is tangent. In this situation, we consider an
orthonormal frame field $\\{X_{1}=E_{3},X_{2}\\}$ on the surface and, since
$\bar{\nabla}_{E_{3}}E_{3}=0$, it follows that $AX_{1}=0$ and
$AX_{2}=2fX_{2}$. Also, we can write $X_{2}$ as

$X_{2}=\cos\alpha E_{1}+\sin\alpha E_{2},$

and then

$E_{1}=\cos\alpha X_{2}+\sin\alpha\xi,\quad\quad E_{2}=\sin\alpha
X_{2}-\cos\alpha\xi,$

along the surface, where $\xi=\sin\alpha E_{1}-\cos\alpha E_{2}$ is a unit
vector field normal to $\Sigma^{2}$.

Next, using formulas (2.5) and the Gauss and Weingarten equations of the
surface, one obtains

$\displaystyle\bar{\nabla}_{X_{2}}E_{1}$ $\displaystyle=$
$\displaystyle-\cos\alpha E_{3}$ $\displaystyle=$ $\displaystyle
X_{2}(\cos\alpha)X_{2}+\cos\alpha\nabla_{X_{2}}X_{2}+2f\cos\alpha\xi+X_{2}(\sin\alpha)\xi-2f\sin\alpha
X_{2},$

and

$\displaystyle\bar{\nabla}_{X_{2}}E_{2}$ $\displaystyle=$
$\displaystyle\sin\alpha E_{3}$ $\displaystyle=$ $\displaystyle
X_{2}(\sin\alpha)X_{2}+\sin\alpha\nabla_{X_{2}}X_{2}+2f\sin\alpha\xi-
X_{2}(\cos\alpha)\xi+2f\cos\alpha X_{2}.$

Taking the inner product with $X_{1}$ in both equations, we get

$\cos\alpha\langle\nabla_{X_{2}}X_{2},X_{1}\rangle=-\cos\alpha\quad\textnormal{and}\quad\sin\alpha\langle\nabla_{X_{2}}X_{2},X_{1}\rangle=\sin\alpha,$

which leads to $\sin\alpha\cos\alpha=0$. This means that either $E_{1}$ or
$E_{2}$ is normal to $\Sigma_{2}$ and therefore the surface is totally
geodesic. ∎

######  Remark 3.2.

In contrast to our situation, any CMC surface in a $3$-dimensional space form
is biconservative [2] and also in the other $3$-dimensional BCV-spaces (that
are local models for six of the Thurston geometries, the exceptions being
$\mathbb{H}^{3}$ and $\operatorname{Sol_{3}}$) non-minimal CMC biconservative
surfaces do exist [23].

We will henceforth focus on non-CMC biconservative surfaces. Let $\Sigma^{2}$
be a biconservative surface in $\operatorname{Sol_{3}}$ such that $\nabla
f\neq 0$ at a point $p\in\Sigma^{2}$. Therefore, there exists a neighborhood
$V$ of $p$ with $\nabla f\neq 0$ on $V$. Since $f$ cannot vanish on $V$, there
exists on open subset $U$ of $V$ such that $f\neq 0$ at each point of $U$.
Moreover, we can assume that $f>0$ on $U$. Next, we consider an orthonormal
frame field $\\{X_{1},X_{2}\\}$ on $U$, where

$X_{1}=\frac{\nabla f}{|\nabla f|},\quad X_{2}\perp X_{1},\quad|X_{2}|=1.$

We now have

$\nabla f=X_{1}(f)X_{1}+X_{2}(f)X_{2},$

which leads to

(3.1) $X_{1}(f)=|\nabla f|>0\quad\textnormal{and}\quad X_{2}(f)=0.$

Next, let $H=f\xi$ be the mean curvature vector field of $\Sigma^{2}$, where
$\xi$ is a unit vector field normal to the surface. By a straightforward
computation, using isothermal coordinates $(x,y)$ on the surface and since
$\Sigma^{2}$ is biconservative, one obtains (see also [18]),

$E=\left(\nabla_{\frac{\partial}{\partial
x}}A_{H}\right)\frac{\partial}{\partial
y}-\left(\nabla_{\frac{\partial}{\partial
y}}A_{H}\right)\frac{\partial}{\partial x}=3f\left(-\frac{\partial f}{\partial
y}\frac{\partial}{\partial x}+\frac{\partial f}{\partial
x}\frac{\partial}{\partial y}\right),$

which shows that $X_{1}\perp E$ on $U$, i.e., $E$ and $X_{2}$ are collinear.

On the other hand, from the Codazzi equation (2.2) and the expression of the
curvature tensor (2), we get

$E=\left(\nabla_{\frac{\partial}{\partial
x}}A_{H}\right)\frac{\partial}{\partial
y}-\left(\nabla_{\frac{\partial}{\partial
y}}A_{H}\right)\frac{\partial}{\partial
x}=2f\langle\xi,E_{3}\rangle\left(\left\langle\frac{\partial}{\partial
y},E_{3}\right\rangle\frac{\partial}{\partial
x}-\left\langle\frac{\partial}{\partial
x},E_{3}\right\rangle\frac{\partial}{\partial y}\right),$

and this readily shows that $E$ is orthogonal to $E_{3}$. Therefore, the two
vector fields $X_{2}$ and $E_{3}$ are orthogonal and, along $U$, we can write

(3.2) $\begin{cases}E_{1}=\sin\beta\sin\theta X_{1}+\cos\beta
X_{2}-\sin\beta\cos\theta\xi\\\ E_{2}=-\cos\beta\sin\theta X_{1}+\sin\beta
X_{2}+\cos\beta\cos\theta\xi\\\ E_{3}=\cos\theta
X_{1}+\sin\theta\xi,\end{cases}$

or, equivalently,

(3.3) $\begin{cases}X_{1}=\sin\beta\sin\theta E_{1}-\cos\beta\sin\theta
E_{2}+\cos\theta E_{3}\\\ X_{2}=\cos\beta E_{1}+\sin\beta E_{2}\\\
\xi=-\sin\beta\cos\theta E_{1}+\cos\beta\cos\theta E_{2}+\sin\theta
E_{3},\end{cases}$

where $\beta$ and $\theta$ are real valued functions on $U$.

From Corrolary 2.3 and formula (2) we easily get that the biconservative
condition is equivalent to

(3.4) $AX_{1}=-\left(f+\frac{f}{|\nabla
f|}\sin(2\theta)\right)X_{1}=\lambda_{1}X_{1},$

and therefore $X_{1}$ is a principal direction with the corresponding
eigenfunction $\lambda_{1}$. It follows that also $X_{2}$ is a principal
direction and

(3.5) $AX_{2}=\left(3f+\frac{f}{|\nabla
f|}\sin(2\theta)\right)X_{2}=\lambda_{2}X_{2}.$

In the following, working on $U$, we will prove a sequence of three lemmas
that will eventually lead to our first main result.

######  Lemma 3.3.

On the set $U$ the following identities hold:

  1. (1)

$X_{1}(\theta)=-\lambda_{1}+\cos(2\beta)\sin\theta$;

  2. (2)

$X_{2}(\theta)=-\sin(2\beta)$;

  3. (3)

$\cos\theta\langle\nabla_{X_{1}}X_{1},X_{2}\rangle=\sin(2\beta)\sin\theta$;

  4. (4)

$\cos\theta\langle\nabla_{X_{2}}X_{1},X_{2}\rangle=\lambda_{2}\sin\theta+\cos(2\beta)$;

  5. (5)

$(X_{1}(\beta)-\langle\nabla_{X_{1}}X_{1},X_{2}\rangle\sin\theta)\sin\beta=0$;

  6. (6)

$X_{1}(\beta)\sin\beta\cos\theta=2\sin^{2}\beta\cos\beta\sin^{2}\theta$;

  7. (7)

$X_{2}(\beta)\cos\theta=\lambda_{2}+\cos(2\beta)\sin\theta$;

  8. (8)

$X_{2}(\beta)\sin\theta-\langle\nabla_{X_{2}}X_{1},X_{2}\rangle=-\cos(2\beta)\cos\theta$.

###### Proof.

From (2.5), (3.2), and (3.3), we have

$\displaystyle\bar{\nabla}_{X_{1}}E_{3}$ $\displaystyle=$
$\displaystyle\sin\beta\sin\theta E_{1}+\cos\beta\sin\theta E_{2}$
$\displaystyle=$ $\displaystyle-X_{1}(\theta)\sin\theta
X_{1}+\cos\theta\nabla_{X_{1}}X_{1}+\cos\theta\sigma(X_{1},X_{1})$
$\displaystyle+X_{1}(\theta)\cos\theta\xi-\sin\theta\lambda_{1}X_{1},$

and

$\displaystyle\bar{\nabla}_{X_{2}}E_{3}$ $\displaystyle=$
$\displaystyle\cos\beta E_{1}-\sin\beta E_{2}$ $\displaystyle=$
$\displaystyle-X_{2}(\theta)\sin\theta
X_{1}+\cos\theta\nabla_{X_{2}}X_{1}+X_{2}(\theta)\cos\theta\xi-\sin\theta\lambda_{2}X_{2}.$

Considering the inner product with $X_{1}$, $X_{2}$, and $\xi$, one obtains
the first four equations.

Next, again using (2.5), (3.2), and (3.3), it follows

$\displaystyle\bar{\nabla}_{X_{1}}E_{1}$ $\displaystyle=$
$\displaystyle-\sin\beta\sin\theta E_{3}$ $\displaystyle=$ $\displaystyle
X_{1}(\theta)\sin\beta\cos\theta X_{1}+X_{1}(\beta)\cos\beta\sin\theta
X_{1}+\sin\beta\sin\theta\nabla_{X_{1}}X_{1}$
$\displaystyle+\sin\beta\sin\theta\sigma(X_{1},X_{1})-X_{1}(\beta)\sin\beta
X_{2}+\cos\beta\nabla_{X_{1}}X_{2}$
$\displaystyle+X_{1}(\theta)\sin\beta\sin\theta\xi-
X_{1}(\beta)\cos\beta\cos\theta\xi+\sin\beta\cos\theta\lambda_{1}X_{1}.$

We take the inner product with $X_{2}$ and obtain the fifth identity. From
this and the third identity, the sixth one follows immediately.

Finally, we have

$\displaystyle\bar{\nabla}_{X_{2}}E_{1}$ $\displaystyle=$
$\displaystyle-\cos\beta E_{3}$ $\displaystyle=$ $\displaystyle
X_{2}(\theta)\sin\beta\cos\theta X_{1}+X_{2}(\beta)\cos\beta\sin\theta
X_{1}+\sin\beta\sin\theta\nabla_{X_{2}}X_{1}$
$\displaystyle+\cos\beta\sigma(X_{2},X_{2})-X_{2}(\beta)\sin\beta
X_{2}+\cos\beta\nabla_{X_{2}}X_{2}$
$\displaystyle+X_{2}(\theta)\sin\beta\sin\theta\xi-
X_{2}(\beta)\cos\beta\cos\theta\xi+\sin\beta\cos\theta\lambda_{2}X_{2},$

and, taking the inner product with each $X_{1}$, $X_{2}$, and $\xi$, also
using the third and the fourth identities, we prove the last two items of the
lemma. ∎

######  Remark 3.4.

Any two of the third, fifth, and sixth items of Lemma 3.3 imply the other one.
This also happens with the fourth, seventh, and eighth items. Therefore only
six of the identities are independent. It is easy to verify that this is the
maximum number of independent equations of this type that can be derived from
the expressions of $\bar{\nabla}_{X_{i}}E_{j}$, $i,j\in\\{1,2,3\\}$.

The next result shows that one of the two vector fields $E_{1}$ or $E_{2}$ is
tangent to the surface $\Sigma^{2}$ at any point of $U$.

######  Lemma 3.5.

On the set $U$ we have $\sin(2\beta)=0$.

###### Proof.

From the Codazzi equation (2.2) and (2), one obtains

$\left(\nabla_{X_{1}}A\right)X_{2}-\left(\nabla_{X_{2}}A\right)X_{1}=-\sin(2\theta)X_{2},$

which can be written as

(3.6)
$\begin{cases}X_{1}(\lambda_{2})+(\lambda_{2}-\lambda_{1})\langle\nabla_{X_{2}}X_{1},X_{2}\rangle+\sin(2\theta)=0\\\
X_{2}(\lambda_{1})-(\lambda_{2}-\lambda_{1})\langle\nabla_{X_{1}}X_{2},X_{1}\rangle=0.\end{cases}$

Using Lemma 3.3, the second equation become

(3.7)
$X_{2}(\lambda_{1})\cos\theta+(\lambda_{2}-\lambda_{1})\sin(2\beta)\sin\theta=0.$

In the following, we will compute $X_{2}(\lambda_{1})$. Since $X_{2}(f)=0$ and
$X_{1}(f)=|\nabla f|$, we have, also using Lemma 3.3,

$\displaystyle X_{2}(\lambda_{1})$ $\displaystyle=$ $\displaystyle-
X_{2}\left(f+\frac{f}{|\nabla f|}\sin(2\theta)\right)$ $\displaystyle=$
$\displaystyle\frac{f}{|\nabla f|}\left(\frac{X_{2}(X_{1}(f))}{|\nabla
f|}\sin(2\theta)+2\sin(2\beta)\cos(2\theta)\right).$

Now, since
$X_{2}(X_{1}(f))=X_{1}(X_{2}(f))-[X_{1},X_{2}](f)=-[X_{1},X_{2}](f)$ and, from
Lemma 3.3, we know that

$\displaystyle\cos\theta[X_{1},X_{2}]$ $\displaystyle=$
$\displaystyle\cos\theta(\nabla_{X_{1}}X_{2}-\nabla_{X_{2}}X_{1})$
$\displaystyle=$ $\displaystyle-\sin(2\beta)\sin\theta
X_{1}-(\lambda_{2}\sin\theta+\cos(2\beta))X_{2},$

one obtains

(3.9) $\cos\theta X_{2}(X_{1}(f))=\sin(2\beta)\sin\theta
X_{1}(f)=\sin(2\beta)\sin\theta|\nabla f|.$

Replacing in (3) we get

$X_{2}(\lambda_{1})=\frac{2f}{|\nabla f|}\sin(2\beta)\cos^{2}\theta,$

and then, from (3.7), since

$\lambda_{2}-\lambda_{1}=4f+\frac{2f}{|\nabla f|}\sin(2\theta),$

one obtains

$\frac{2\sin(2\beta)}{|\nabla f|}\left(2|\nabla
f|\sin\theta+\cos\theta\left(1+\sin^{2}\theta\right)\right)=0.$

Therefore, at each point of $U$, either

$\sin(2\beta)=0\quad\textnormal{or}\quad 2|\nabla
f|\sin\theta+\cos\theta\left(1+\sin^{2}\theta\right)=0.$

Assume that $\sin(2\beta)\neq 0$ at a point $q\in U$. Then, there exists a
neighborhood $W\subset U$ of $q$ such that $\sin(2\beta)\neq 0$ at all points
of $W$, which means that

(3.10) $2|\nabla f|\sin\theta+\cos\theta\left(1+\sin^{2}\theta\right)=0$

on $W$. From here, it follows that

$2X_{2}(X_{1}(f))\sin\theta+(2|\nabla
f|\cos\theta-\sin\theta-\sin^{3}\theta+2\sin\theta\cos^{2}\theta)X_{2}(\theta)=0,$

on $W$. We then multiply by $\cos\theta$, use (3.9), Lemma 3.3, and again
(3.10), to obtain, after a straightforward computation, that $\cos\theta=0$ on
the set $W$. But, from (3.10), this implies that also $\nabla f=0$ on $W$,
which is a contradiction and we conclude. ∎

######  Lemma 3.6.

Throughout the set $U$ we have

$\cos\theta\neq 0,\quad\sin\theta\neq 0,\quad X_{1}(\theta)=-2f,\quad
X_{2}(\theta)=0,\quad\nabla_{X_{1}}X_{1}=0,\quad X_{2}(X_{1}(f))=0,$

and either

  1. (1)

$\nabla_{X_{2}}X_{1}=\cos\theta X_{2}$;

  2. (2)

$\lambda_{2}=-\sin\theta$,

or

  1. (1)

$\nabla_{X_{2}}X_{1}=-\cos\theta X_{2}$;

  2. (2)

$\lambda_{2}=\sin\theta$,

as $\beta=0$ or $\beta=\pi/2$ on $U$.

###### Proof.

The claims on $\nabla_{X_{2}}X_{1}$, $X_{2}(\theta)$, and $\lambda_{2}$ follow
directly from Lemmas 3.3 and 3.5. Then, using the expression of $\lambda_{2}$
and again the two previous lemmas, one obtains that $X_{1}(\theta)=-2f$.

If $\sin\theta=0$ at a point $q\in U$, then, at this point, $\lambda_{2}=0$
and $\lambda_{1}=2f$. But, from the biconservative condition (3.4), it follows
that $\lambda_{1}=-f$ which implies that $f=0$, at $q$. Therefore,
$\sin\theta\neq 0$ at any poiny of $U$.

Next, from the first equation of (3.6), one can see that, if we assume
$\cos\theta=0$ at a point $q\in U$, then, at that point, we have
$X_{1}(\lambda_{2})=0$. But, in general, on $U$,

$X_{1}(\lambda_{2})=3X_{1}(f)+X_{1}\left(\frac{f}{|\nabla
f|}\right)\sin(2\theta)+\frac{2f}{|\nabla f|}X_{1}(\theta)\cos(2\theta),$

and therefore, at $q$,

$X_{1}(\lambda_{2})=3|\nabla f|+\frac{4f^{2}}{|\nabla f|},$

as $X_{1}(\theta)=-2f$. Since $f>0$ and $|\nabla f|\ 0$ on $U$, this is a
contradiction, i.e., $\cos\theta\neq 0$ throughout $U$. To conclude, we note
that $\nabla_{X_{1}}X_{1}=0$ now follows immediately from Lemma 3.3, and
$X_{2}(X_{1}(f))=0$ from equation (3.9). ∎

Now, we can state the rigidity of biconservative surfaces result.

######  Theorem 3.7.

Let $\Sigma^{2}$ be a biconservative surface in $\operatorname{Sol_{3}}$ with
$f>0$ and $\nabla f\neq 0$ at any point. Then, locally, $\Sigma^{2}$ can be
parametrized as either

(3.11)
$x_{1}(u,v)=ve^{\Psi(u)}E_{1}+\Phi_{1}(u)e^{-\Psi(u)}E_{2}+\Psi(u)E_{3}=(v,\Phi_{1}(u),\Psi(u)),$

or

(3.12)
$x_{2}(u,v)=\Phi_{2}(u)e^{-\Psi(u)}E_{1}+ve^{-\Psi(u)}E_{2}+\Psi(u)E_{3}=(\Phi_{2}(u),v,\Psi(u)),$

where

$\Psi(u)=\int_{u_{0}}^{u}\cos\theta(s)ds,\Phi_{1}(u)=-\int_{u_{0}}^{u}\sin\theta(s)e^{\Psi(s)}ds,\Phi_{2}(u)=\int_{u_{0}}^{u}\sin\theta(s)e^{-\Psi(s)}ds.$

The function $\theta=\theta(u)$ is given either by

(3.13) $\theta(u)=2\arctan e^{(1-\sqrt{13})u/3},\quad u<0,$

or by the following first order ODE

(3.14)
$(\theta^{\prime}+2a_{1}\sin\theta)^{6a_{2}}=c(\theta^{\prime}+2a_{2}\sin\theta)^{6a_{1}},$

such that $\theta^{\prime}<0$ and $\theta^{\prime\prime}<0$, where
$a_{1,2}=(-1\pm\sqrt{13})/6$ and $c>0$ is a positive constant.

###### Proof.

From Lemma 3.5, we know that either $E_{1}$ or $E_{2}$ is tangent to our
surface and therefore $\Sigma^{2}$ is a constant angle surface as defined in
[16], where equations (3.11) and (3.12) of such surfaces were found.

We will consider only the case of the first surface, that with our notations
correspond to $\beta=0$, meaning that $X_{2}=E_{1}$. The computations in the
second case are similar and therefore we will omit them.

In [16], the local coordinates $(u,v)$ were chosen such that

$X_{1}=\frac{\partial}{\partial u}\quad\textnormal{and}\quad
X_{2}=\alpha(u,v)\frac{\partial}{\partial v},$

for some function $\alpha$, which is allowed by the fact that

$[X_{1},X_{2}]=-\nabla_{X_{1}}X_{2}=-\cos\theta X_{2}.$

Therefore, in this setting, $X_{2}(f)=0$ and $X_{2}(\theta)=0$ imply that
$f=f(u)$ and $\theta=\theta(u)$. It is now straightforward to verify that
surfaces defined by (3.11) and (3.12) satisfy all identities in Lemma 3.6.

Thus, we only have to find the function $\theta=\theta(u)$ by solving the
biconservative equation, which, for the first surface, taking into account
Lemma 3.6, can be written as

$\lambda_{1}=-\left(f+\frac{f}{f^{\prime}}\sin(2\theta)\right)=2f+\sin\theta,$

or, equivalently,

(3.15) $3ff^{\prime}+f^{\prime}\sin\theta+f\sin(2\theta)=0,$

where $\theta^{\prime}=-2f$. Denote $y=\sin\theta$ and then we can see that

$2yy^{\prime}=2\theta^{\prime}\sin\theta\cos\theta=-2f\sin(2\theta),$

and equation (3.15) becomes

$3ff^{\prime}+yf^{\prime}-yy^{\prime}=0.$

We can write $f^{\prime}=y^{\prime}(df/dy)$ and, replacing in the last
equation, we have

$3fy^{\prime}\frac{df}{dy}+yy^{\prime}\frac{df}{dy}-yy^{\prime}=0.$

Since from Lemma 3.6 we know that $\cos\theta\neq 0$ and we also have
$\theta^{\prime}=-2f\neq 0$, it follows that
$y^{\prime}=\theta^{\prime}\cos\theta\neq 0$, which shows that the above
equation is equivalent to

$3f\frac{df}{dy}+y\frac{df}{dy}-y=0.$

We denote $f=yg$ and, after a straightforward computation, this equation reads
as

(3.16) $y(3g+1)\frac{dg}{dy}+3g^{2}+g-1=0.$

We now have two cases as $3g^{2}+g-1=0$ or $3g^{2}+g-1\neq 0$. In the first
one, $g=a_{1,2}=(-1\pm\sqrt{13})/6$ is a constant and $f=a\sin\theta$, where
$a$ stands either for $a_{1}$ or $a_{2}$. Since $\theta^{\prime}=-2f$, it
follows that

$\theta(u)=2\arctan e^{-2au},\quad
f(u)=\frac{2ae^{-2au}}{1+e^{-4au}},\quad\textnormal{and}\quad
f^{\prime}(u)=\frac{4a^{2}e^{-2au}(e^{-4au}-1)}{(1+e^{-4au})^{2}}.$

Since $f>0$, we see that $a=(\sqrt{13}-1)/6$, and, from
$f^{\prime}(u)=X_{1}(f)=|\nabla f|>0$, it follows that $u$ must be negative.

In the second case, a simple computation shows that equation (3.16) has the
implicit solution

(3.17) $(y(g-a_{1}))^{6a_{2}}=c(y(g-a_{2}))^{6a_{1}},$

where $c\in\mathbb{R}$ is a positive constant. Hence, we have

$(f-a_{1}\sin\theta)^{6a_{2}}=c(f-a_{2}\sin\theta)^{6a_{1}},$

which is just equation (3.14). ∎

######  Remark 3.8.

In the case when $\theta=\theta(u)$ is given by (3.13), one can compute the
explicit expression of the function $\Psi=\Psi(u)$ as

$\Psi(u)=\int_{u_{0}}^{u}\cos\theta(s)ds=\frac{1}{2a}\ln\left(e^{-4au}+1\right)+u+c_{0},$

with $a=(-1+\sqrt{13})/6$, $c_{0}=\operatorname{constant}\in\mathbb{R}$.

In this case, we can also compute, using equation (2.3), the Gaussian
curvature of the surface

$K=-\cos^{2}\theta-2f\sin\theta=-\frac{4ae^{-4au}+(1-e^{-4au})^{2}}{(1+e^{-4au})^{2}}<0.$

######  Remark 3.9.

The only possible solution of the biconservative equation (3.15) of the form
$f=a\sin\theta$, where $a\in\mathbb{R}$ is a constant, is the one with
$a=(-1+\sqrt{13})/6$. Otherwise, equation (3.14) implies that $\theta$ is a
constant and therefore $f=0$.

######  Theorem 3.10.

Any biharmonic surface in $\operatorname{Sol_{3}}$ is minimal.

###### Proof.

The case of CMC biharmonic surfaces was studied in [18] so we only have to
show that if $\Sigma^{2}$ is a biharmonic surface in $\operatorname{Sol_{3}}$
such that there exists a point $p\in\Sigma^{2}$ at which $\nabla f\neq 0$,
then the surface is minimal.

As before, we consider an open neighborhood $U\subset\Sigma^{2}$ of the point
$p$ such that $\nabla f\neq 0$ and $f>0$ on $U$. As in the proof of Theorem
3.7, suffices treating only the case of a surface given by (3.11), the other
one being similar. We will also use the same setting and notations as in the
proof of Theorem 3.7.

From Theorem 2.2, we know that the normal part of the biharmonic equation is

$\Delta
f-f|A|^{2}-f\langle\operatorname{trace}(\bar{R}(\cdot,\xi)\cdot),\xi\rangle=0.$

A direct computation, using Lemma 3.6, shows that

$|A|^{2}=4f^{2}+4f\sin\theta+2\sin^{2}\theta.$

From formula (2) we find

$\langle\operatorname{trace}(\bar{R}(\cdot,\xi)\cdot),\xi\rangle=2\sin^{2}\theta.$

It follows that the biconservative surface $\Sigma^{2}$ given by (3.11) is
biharmonic if and only if

(3.18) $\Delta f=4f(f^{2}+f\sin\theta+\sin^{2}\theta)>0.$

Now, let us assume that $\theta$ is given by (3.13) and, therefore,

$f(u)=\frac{2ae^{-2au}}{1+e^{-4au}},\quad a=\frac{-1+\sqrt{13}}{6}.$

From Lemma 3.6, we have that

$\Delta f=f^{\prime\prime}+\cos\theta f^{\prime},$

and then, a direct computation leads to

$\Delta
f=\frac{4e^{-6au}\left(e^{-4au}(2a^{3}-a^{2})+e^{4au}(2a^{3}-a^{2})+2a^{2}-12a^{3}\right)}{\left(1+e^{-4au}\right)^{3}}.$

It is easy to verify that $\Delta f<0$, in this case, which is a
contradiction.

Next, let the function $\theta$ be a solution of equation (3.14) and again
denote $g=f/\sin\theta$, as in the proof of Theorem 3.7. From Remark 3.9, we
have that $g\neq 0$ throughout $U$.

The first derivative of $f$ is given by the biconservative equation as

$f^{\prime}=-\frac{f\sin(2\theta)}{3f+\sin\theta},$

where we have also used $\theta^{\prime}=-2f$. Taking into account Remark 3.9
(and possibly restricting to an open subset $V\subset U$) we can see that
$f^{\prime}$ is well defined. Using this formula, and again
$\theta^{\prime}=-2f$, it is straightforward to compute $f^{\prime\prime}$ and
then

$\displaystyle\Delta f$ $\displaystyle=$
$\displaystyle\frac{f}{(3f+\sin\theta)^{3}}\Big{(}36f^{3}(1-2\sin^{2}\theta)-f^{2}(6\sin\theta+18\sin^{3}\theta)$
$\displaystyle-f(12\sin^{2}\theta-8\sin^{4}\theta)+2\sin^{3}\theta-2\sin^{5}\theta\Big{)}.$

Replacing in (3.18), and taking into account that $\sin\theta\neq 0$ on $U$,
one obtains the following equation

(3.19) $\sin^{2}\theta P_{1}(g)-P_{2}(g)=0,$

where $P_{1}$ and $P_{2}$ are two polynomial functions of $u$, given by

$P_{1}(g)=100g^{5}+216g^{4}+324g^{3}+166g^{2}+32g+6$

and

$P_{2}(g)=36g^{3}-6g^{2}-12g+2.$

Since $g$ satisfies equation (3.17), we can write

$\sin^{2}\theta=\frac{(g-a_{2})^{b_{1}}}{c(g-a_{1})^{b_{2}}},$

where $b_{1,2}=(6a_{1,2})/\sqrt{13}$ and $c\in\mathbb{R}$ is a positive
constant. Then, equation (3.19) can be written as

$(g-a_{2})^{b_{1}}P_{1}(g)=c(g-a_{1})^{b_{2}}P_{2}(g).$

Differentiating with respect to $u$ and using that $g^{\prime}(u)\neq 0$ at
any point of $U$, we obtain

$(g-a_{2})^{b_{1}}(g-a_{1})\left(b_{1}P_{1}+(g-a_{2})\frac{dP_{1}}{dg}\right)=c(g-a_{1})^{b_{1}}(g-a_{2})\left(b_{2}P_{2}+(g-a_{1})\frac{dP_{2}}{dg}\right)$

The last two equations imply

$2(3g+1)P_{1}P_{2}+(3g^{2}+g-1)\left(P_{1}\frac{dP_{2}}{dg}-P_{2}\frac{dP_{1}}{dg}\right)=0.$

After a long but simple computation, this polynomial equation reads as

$25128g^{8}+92760g^{7}+85632g^{6}+15840g^{5}-19352g^{4}-13224g^{3}-1872g^{2}+656g+160=0,$

and shows that $g$ is a constant, which is a contradiction. We conclude that
there are no biharmonic surfaces in $\operatorname{Sol_{3}}$ other than the
minimal ones. ∎

######  Remark 3.11.

This last result is similar to that in [4, 7, 14] on biharmonic surfaces of
the Euclidean space $\mathbb{E}^{3}$ and even to the one in [1] about
biharmonic surfaces in the Euclidean sphere $\mathbb{S}^{3}$. Although in this
last case non-minimal biharmonic surfaces do exist, they are also CMC, and,
therefore, there are no non-CMC such surfaces in $\mathbb{S}^{3}$.

## References

  * [1] R. Caddeo, S. Montaldo, and C. Oniciuc, Biharmonic submanifolds of $\mathbb{S}^{3}$, Internat. J. Math. 12 (2001), 867–876.
  * [2] R. Caddeo, S. Montaldo, C. Oniciuc, and P. Piu, Surfaces in three-dimensional space forms with divergence-free stress-bienergy tensor, Ann. Mat. Pura Appl. (4) 193 (2014), 529–550.
  * [3] B. Y. Chen, Some open problems and conjectures on submanifolds of finite type, Soochow J. Math. 17 (1991), 169–188.
  * [4] B. Y. Chen and S. Ishikawa, Biharmonic surfaces in pseudo-Euclidean spaces, Mem. Fac. Sci. Kyushu Univ. Ser. A 45 (1991), 323–347.
  * [5] B. Daniel, W. H. Meeks III, and H. Rosenberg, Half-space theorems for minimal surfaces in ${\rm Nil}_{3}$ and ${\rm Sol}_{3}$, J. Differential Geom. 88 (2011), 41–59.
  * [6] B. Daniel and P. Mira, Existence and uniqueness of constant mean curvature spheres in ${\rm Sol}_{3}$, J. Reine Angew. Math. 685 (2013), 1–32.
  * [7] I. Dimitrić, Submanifolds of $\mathbb{E}^{m}$ with harmonic mean curvature vector, Bull. Inst. Math. Acad. Sinica 20 (1992), 53–65.
  * [8] J. Eells, J. H. Sampson, Harmonic mappings of Riemannian manifolds, Amer. J. Math. 86 (1964), 109–160.
  * [9] D. Fetcu, S. Nistor, and C. Oniciuc, On biconservative surfaces in 3-dimensional space forms, Comm. Anal. Geom. 24 (2016), 1027–1045.
  * [10] D. Fetcu and C. Oniciuc, Biharmonic and biconservative hypersurfaces in space forms, Differential geometry and global analysis–in honor of Tadashi Nagano, 65–90, Contemp. Math. 777, Amer. Math. Soc., [Providence], RI, 2022.
  * [11] T. Hasanis and T. Vlachos, Hypersurfaces in $\mathbb{E}^{4}$ with harmonic mean curvature vector field, Math. Nachr. 172 (1995), 145–169.
  * [12] G. Y. Jiang, 2-harmonic maps and their first and second variational formulas, Chinese Ann. Math. Ser. A 7 (1986), 389–402.
  * [13] G. Y. Jiang, The conservation law for 2-harmonic maps between Riemannian manifolds, Acta Math. Sinica 30 (1987), 220–225.
  * [14] G. Y. Jiang, Some nonexistence theorems on 2-harmonic and isometric immersions in Euclidean space, Chinese Ann. Math. Ser. B 8 (1987), 377–383.
  * [15] P. Klaser and A. Menezes, The Dirichlet problem for the constant mean curvature equation in $Sol_{3}$, Illinois J. Math. 63 (2019), 259–297.
  * [16] R. López and M. I. Munteanu, On the geometry of constant angle surfaces in $Sol_{3}$, Kyushu J. Math. 65 (2011), 237–249.
  * [17] E. Loubeau, S., Montaldo, and C. Oniciuc, The stress-energy tensor for biharmonic maps, Math. Z. 259 (2008), 503–524.
  * [18] E. Loubeau, C. Oniciuc, Biharmonic surfaces of constant mean curvature, Pacific J. Math. 271 (2014), 213–230.
  * [19] W. H. Meeks III, Constant mean curvature spheres in $\operatorname{Sol_{3}}$, Amer. J. Math. 135 (2013), 763–775.
  * [20] W. H. Meeks III, P. Mira, J. Pérez, and A. Ros, Constant mean curvature spheres in homogeneous three-manifolds, Invent. Math. 224 (2021), 147–244.
  * [21] S. Montaldo, C. Oniciuc, and A. Ratto, Proper biconservative immersions into the Euclidean space, Ann. Mat. Pura Appl. (4) 195 (2016), 403–422.
  * [22] S. Montaldo, C. Oniciuc, and A. Ratto, Biconservative surfaces, J. Geom. Anal. 26 (2016), 313–329.
  * [23] S. Montaldo, I. I. Onnis, and A. P. Passamani, Biconservative surfaces in BCV-spaces, Math. Nachr. 290 (2017), 2661–2672.
  * [24] M. H. Nguyen, The Dirichlet problem for the minimal surface equation in $Sol_{3}$, with possible infinite boundary data, Illinois J. Math. 58 (2014), 891–937.
  * [25] S. Nistor, On biconservative surfaces, Differential Geom. Appl. 54 (2017), 490–502.
  * [26] Y. L. Ou and B. Y. Chen, Biharmonic submanifolds and biharmonic maps in Riemannian geometry, World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2020.
  * [27] Y. L. Ou and Z. P. Wang, Constant mean curvature and totally umbilical biharmonic surfaces in 3-dimensional geometries, J. Geom. Phys. 61 (2011), 1845–1853.
  * [28] R. Souam and E. Toubiana, Totally umbilic surfaces in homogeneous $3$-manifolds, Comment. Math. Helv. 84 (2009), 673–704.


# Why do small language models underperform?
Studying LM Saturation via the Softmax Bottleneck

Nathan Godey1,2, Éric de la Clergerie1 & Benoît Sagot1
1 Inria Paris, 2 Sorbonne Université
Paris, France
<EMAIL_ADDRESS>

###### Abstract

Recent advances in language modeling consist in pretraining highly
parameterized neural networks on extremely large web-mined text corpora.
Training and inference with such models can be costly in practice, which
incentivizes the use of smaller counterparts. However, it has been observed
that smaller models can suffer from saturation, characterized as a drop in
performance at some advanced point in training followed by a plateau. In this
paper, we find that such saturation can be explained by a mismatch between the
hidden dimension of smaller models and the high rank of the target contextual
probability distribution. This mismatch affects the performance of the linear
prediction head used in such models through the well-known softmax bottleneck
phenomenon. We measure the effect of the softmax bottleneck in various
settings and find that models based on less than 1000 hidden dimensions tend
to adopt degenerate latent representations in late pretraining, which leads to
reduced evaluation performance.

##  1 Introduction

The representation degeneration problem is a common phenomenon that affects
self-supervised learning methods used for textual data (Gao et al., 2019; Lai
et al., 2023), among other modalities (Jing et al., 2022; Godey et al., 2024).
Many observations on the intermediate representations of Language Models (LMs)
have shed light on their low angular variability (or anisotropy) (Zhou et al.,
2021; Rajaee & Pilehvar, 2022) or on outlier dimensions that emerged during
training (Puccetti et al., 2022). However, these observations were mostly made
on relatively small-scale models of dimensions comparable to BERT (Devlin et
al., 2019) or models from the GPT-2 family (Radford et al., 2019).

These models are usually composed of a neural network $f_{\theta}$ that takes
sequences of tokens $(y_{<i})\in[1,V]^{i-1}$ as inputs and produces a
relatively low-dimensional contextual representation in $\mathbb{R}^{d}$,
where $d$ is the hidden dimension of the model. They then rely on a language
modeling head that produces logits for contextual token probabilities. A
common choice for the language modeling head is a linear layer with parameter
$W\in\mathbb{R}^{V\times d}$, where $V$ is the number of possible tokens. The
resulting next-token probability distribution is then given by:

$p(y_{i})=\sigma(Wf_{\theta}(y_{<i}))$

where $\sigma$ is the softmax function.

In the language modeling field, the current trend consists in scaling up the
generative pretraining approach introduced with GPT-2, which implies training
neural models made of several billions of parameters on gigantic web-mined
text corpora (Brown et al., 2020; Touvron et al., 2023; Almazrouei et al.,
2023; Jiang et al., 2023). However, training and serving such highly
parameterized models raises energy and hardware-related problematics, which
motivates for looking into achieving similar performance levels with smaller
models (Sardana & Frankle, 2023).

Nevertheless, the evaluation of the Pythia model suite (Biderman et al., 2023)
has shown that training small models on very large corpora could lead to
saturation, in the form of a performance degradation in late pretraining. In
this paper, we explore this saturation phenomenon through the lens of
representation degeneration, and find that both phenomena strongly correlate.
We further demonstrate that representation degeneration strongly occurs in the
language modeling head of small models, and we theoretically and empirically
show how a linear language modeling head can represent a performance
bottleneck for architectures based on small hidden dimensions.

Overall, our contributions can be summarized as:

  * •

We characterize the performance saturation of small language models through
evaluation and extrapolation of the scaling laws;

  * •

We find that the representations of smaller models degenerate concurrently
with this saturation. We shed light on rank saturation, i.e. the explosion of
the entropy of singular value distributions of small LM prediction heads;

  * •

We empirically verify that the rank of the target contextual distribution is
usually high. Moreover, we observe that regardless of the expressiveness of
the output representations of a model, a linear head $W$ substantially affects
performance when $rank(W)<1000$;

  * •

We theoretically quantify the performance limitation induced by a low-rank
linear language modeling head.

##  2 Related Works

#### Small LMs & Saturation

Biderman et al. (2023) train Pythia, a suite of models of various sizes on
300B tokens from the Pile (Gao et al., 2020), and release the weights for an
exhaustive number of checkpoints during training. They notice that smaller
models suffer a performance decrease on the Lambada dataset (Paperno et al.,
2016) in late training. The scaling laws (Kaplan et al., 2020; Hoffmann et
al., 2022) predict that training smaller models on large corpora is suboptimal
in terms of compute. However, recent initiatives (Zhang et al., 2024; Faysse
et al., 2024; Team et al., 2024) have pretrained smaller language models on
large datasets, motivated by inference cost reduction (Sardana & Frankle,
2023).

#### Softmax Bottleneck

The concept of softmax bottleneck was introduced in Yang et al. (2018), where
the authors show that a model using a hidden dimension inferior to the rank of
the contextual probability matrix cannot predict correctly in every context.
They then hypothesize that this rank is relatively high in natural language
and propose an alternative method for the predictive layer of language models.
Subsequent works have explored negative effects of the softmax linear layer on
language modeling performance (Chang & McCallum, 2022) and possible
alternatives (Lin, 2021; Kanai et al., 2018). We extend this line of work by
quantifying the critical dimensionalities involved in the softmax bottleneck.

#### Representation Degeneration

is a phenomenon in which pretrained models tend to adopt low-entropy singular
value distributions (Jing et al., 2022). In language modeling, representation
degeneration takes the form of anisotropy (Ethayarajh, 2019; Rajaee &
Pilehvar, 2021) and was proven to be related with the Zipfian shape of token
distribution (Gao et al., 2019; Biś et al., 2021). We study this phenomenon
along training and its relation with saturation.

#### Data Dimensionality and Performance

Sharma & Kaplan (2022) link the scaling laws observed across pretrained models
to data dimensionality, through the lens of Intrinsic Dimension (Camastra &
Staiano, 2016). While they show that Singular Value Decomposition (SVD) is not
suited for studying the dimensionality of the data manifold in the universal
approximation paradigm, we argue that it is well-suited, to a certain extent,
when studying the performance of a linear classifier limited by the
dimensionality of input representations.

##  3 Language Model Saturation

We first verify that we can indeed observe and quantify performance saturation
for the Pythia checkpoints, as they are the only released intermediate
checkpoints for a wide range of model sizes. We measure the cross-entropy of
Pythia checkpoints on 50k tokens randomly sampled from their pretraining
dataset, i.e. The Pile (Gao et al., 2020).

(a) Loss saturation

(b) Loss extrapolation

Figure 1: Performance of Pythia models on the Pile. On the left, we compare
training dynamics of models from 14M (top) to 410M (bottom) parameters,
displaying darker shades as we approach the minimal value. On the right, we
fit a power law on larger models and find that final checkpoints of smaller
models underperform compared to predictions.

In Figure 1(a), we clearly see that models up to 410M parameters suffer from
the saturation phenomenon, characterized as an increase of the in-domain loss
in advanced training stages.

In Figure 1(b), we fit a scaling law in the style of Hoffmann et al. (2022) on
data points from models ranging from 410M parameters, only optimizing for
model-related constants ($A$ and $\alpha$) while reusing all other values
($B=410.7$, $\beta=0.28$, $E=1.69$). We recall the relation between parameter
count $N$ and token count $T$ given in Hoffmann et al. (2022):

$L(N,T)=\frac{A}{N^{\alpha}}+\frac{B}{T^{\beta}}+E$

We find that optimal parameters are $A=119.09$ and $\alpha=0.246$. We display
the fitted curves for token counts that correspond to best and final
checkpoints. We observe that the final checkpoints underperform the
extrapolation by 8% in average. The loss-minimizing (best) checkpoints, which
are expected to fall short of the extrapolation due to their incomplete
learning rate cooldown, only underperform it by roughly 4%.

A similar performance saturation is also observed on datasets used for
evaluation in the LM Evaluation Harness (Gao et al., 2023), as shown in Table
1.

Checkpoint | Lambada (ppl.) $\downarrow$ | Lambada $\uparrow$ | StoryCloze $\uparrow$ | WikiText (ppl.) $\downarrow$ | SciQ $\uparrow$ | ARC-e $\uparrow$
---|---|---|---|---|---|---
Best | 24.6 | 40.3 | 59.6 | 30.47 | 79.6 | 46.5
Final | 32.9 | 38 | 57.2 | 33.4 | 73.4 | 43.2

Table 1: Zero-shot performance of Pythia-160M best and final checkpoints on
evaluation datasets. Unless specified, we report accuracy for all tasks.

##  4 Performance Saturation is Rank Saturation

###  4.1 Anisotropy at Scale

Anisotropy is a common form of representation degeneration that has been
observed among various small language models. It consists in reduced angular
variability of the representation distribution at a given layer. Previous
works (Ethayarajh, 2019; Godey et al., 2024) notice that almost all layers of
small Transformers language models are anisotropic. A common measure for
anisotropy in a set $H$ of vector representations is the average cosine-
similarity:

$\mathcal{A}(H)=\frac{1}{|H|^{2}-|H|}\sum_{h_{i},h_{j}\in H,i\neq
j}\frac{h_{i}^{T}h_{j}}{||h_{i}||_{2}\cdot||h_{j}||_{2}}$

However, it remains unclear whether anisotropy affects models with over 1
billion parameters. In order to address this question, we compute average
cosine-similarity of intermediate representations across layers in suites of
models; namely GPT-2 (Radford et al., 2019), OPT (Zhang et al., 2022), Pythia
(Biderman et al., 2023), and Gemma (Team et al., 2024). We use a subsample of
The Pile (Gao et al., 2020), as we hypothesize that the domain of this dataset
includes or matches the domain of the pretraining datasets used in these
suites.

(a) Pythia

(b) GPT-2

(c) Gemma

(d) OPT

Figure 2: Anisotropy in function of layer depth (i.e. order in the forward
pass).

In Figure 2, we observe that most layers of Transformers models are
anisotropic to some extent, regardless of the scale. Nevertheless, there seems
to be a dichotomy in the last layer, where models are either nearly isotropic
or highly anisotropic. Interestingly, we notice that the dichotomy aligns with
the one of the saturation phenomenon for the Pythia suite, where only models
containing 160M or fewer parameters seem affected by last-layer anisotropy.

We thus decide to study the training dynamics of anisotropy for the Pythia
suite, and compare them with the saturation phenomenon in Figure 3.

(a) 14M

(b) 31M

(c) 70M

(d) 160M

(e) 410M

Figure 3: Evolution of the language modeling performance on the Wikipedia test
set from the LM Evaluation Harness (Gao et al., 2023) and last-layer
anisotropy of Pythia models along training (color).

Figure 3 illustrates a neat correlation between the emergence of the
performance saturation phenomenon and the appearance of anisotropy in the
last-layer representations of the models. It also shows that anisotropy
increases abruptly around the saturation point during training. Moreover, we
see here that on a specific in-domain corpus, the models quickly lose
performance at saturation and never seem to fully recover from this explosion.

###  4.2 Singular Values Saturation

Average cosine-similarity is a valuable measure of the uniformity of a
distribution, but including other metrics can help to better capture the
complexity of some manifolds (Rudman et al., 2022). Moreover, it only focuses
on the output embeddings of the language models, and not on their weights. In
this section, we extend our analysis by studying the singular value
distributions of the language modeling heads, to link our empirical
observations to our theoretical findings. In Figure 4, we display the singular
value distributions of the final predictive layer weights $W$ along training.

(a) 14M

(b) 31M

(c) 70M

(d) 160M

(e) 410M

Figure 4: Evolution of the singular value distributions of the LM heads of
Pythia models during training, normalized by the maximum singular value.

Figure 4 sheds light on a specific pattern of spectral saturation, roughly co-
occurring with the performance saturation phenomenon. It shows that the
singular value distribution progressively flattens during training, and nearly
reaches uniformity before abruptly evolving towards a spiked distribution with
a high maximal singular value, relatively to the other ones.

Figure 5: Training dynamics of the singular entropy, for different Pythia
models.

In order to quantify this behavior more accurately, we use a singular entropy
metric, computed as the Kullback-Leibler divergence between the normalized
singular value distribution and the uniform distribution.

Figure 5 shows that singular distributions evolve differently for models using
less than 410M parameters than for the larger ones. The heads of small models
see their singular value distributions become increasingly uniform, up to a
point where they degenerate abruptly, which again correlates with the LM
performance drop. The singular value distributions of larger models tend to be
more stable, and do not display clear monotonic patterns throughout training.

##  5 The Softmax Bottleneck & Language Dimensionality

###  5.1 Inherent Dimensionality of Natural Language

Intuitively, the saturation of the singular values distribution observed only
for smaller models in Section 4.2 questions the dimensionalities involved in
the optimization of the LM head. In this section, we propose to empirically
measure a critical value for the rank of the LM head, and to estimate the
dimensionality of the contextual probability distribution the head’s outputs
are supposed to match.

In order to empirically measure the effect of the rank of the linear head, we
propose to train rank-constrained heads on pretrained contextual
representations from highly-parameterized language models. In order to control
the maximum rank $r$, we consider heads of the form
$W=AB\in\mathbb{R}^{V\times d}$, where the coefficients of
$A\in\mathbb{R}^{V\times r}$ and $B\in\mathbb{R}^{r\times d}$ are drawn from
$\mathcal{N}(0,1)$ ($d$ being the hidden dimension of the model). The rank of
such $W$ matrices is limited by the parameter $r\in[1,d]$, which we sweep over
a wide range of values.

We freeze the language models and train the rank-constrained heads on their
output representations on roughly 150M tokens, while adjusting the learning
rate to the trainable parameter count (more details in Appendix B).

(a) Accuracy

(b) Cross-entropy

Figure 6: Performance of several models as the bottleneck dimension of the
head increases.

In Figure 6, we observe that perplexity starts to noticeably decrease when the
rank of the language modeling head $W$ is inferior to 1000, regardless of the
model size. This hints that the head is not a major performance bottleneck for
models with greater hidden dimensions, but that it may hurt performance for
models with smaller ones independently of the quality of the output
representations.

Another interesting factor to estimate is the dimensionality inherent to the
data itself. To avoid possible effects related to specific inductive biases,
we train naive 5-gram language models on several datasets of varying coverage
(IMDb (Maas et al., 2011), Wikitext (Merity et al., 2016), and The Pile (Gao
et al., 2020)), using two tokenizers of varying vocabulary sizes (30k tokens
for Llama-2 and 50k tokens for Pythia). Given $C$ observed 5-grams, we
consider the matrices $W\in\mathbb{R}^{C\times V}$ where each row is a
probability distribution over possible tokens in a given 4-token context, and
compute their singular value distributions, as in Terashima et al. (2003). In
Figure 7, we report $W$-error, the minimal approximation error on $W$ for a
matrix of rank $d$ as predicted by the Eckart-Young-Mirsky theorem (see Lemma
5.2), normalized by the Frobenius norm of $W$:

$W\text{-error}(d)=\frac{||\sigma_{d+1:}||_{2}}{||W||_{F}}$

(a) Llama-2 tokenizer

(b) Pythia tokenizer

Figure 7: $W$-error as $d$ increases, for different tokenizers and datasets.
We observe that while W-error can be halved using 1000 or 2000 dimensions, it
only becomes negligible after 10,000-15,000 dimensions.

We find that the estimated rank of $W$ is non-negligible with respect to the
usual magnitude of hidden dimensions. In the next section, we analyze the
connection between the dimensionality of an ideal linear language modeling
head and performance from a theoretical perspective.

###  5.2 A Theoretical Bottleneck

In this section, we aim at identifying a formal link between the inherent
dimensionality of the contextual distribution and the performance bottleneck
that can be attributed to the lower dimensionality of the output
representations of a language model. To that end, we conceptualize a language
modeling head optimized on ideal contextual representations, and we explore
the relationship between its spectral properties and the performance gap
induced when training a low-rank head on the same representations.

Let’s consider a set $\mathcal{T}$ of sequences $(y_{i})_{i\in[1,|y|]}$ of
elements taken from a vocabulary of size $V$, representing the pretraining
data. We consider a function $\phi^{*}$ that perfectly (e.g. in a bijective
way) represents a given context $y_{<i}$ as a single real vector of infinite
dimension. As we do not focus on $\phi^{*}$, we can simplify the notations by
introducing the contextual representations $x^{*}_{i}=\phi^{*}(y_{<i})$.

The task of the linear language modeling head can be formalized as an
optimization problem on the matrix $W$:

$W^{*}=\operatorname*{argmin}_{W\in\mathbb{R}^{V\times\infty}}\sum_{y\in\mathcal{T}}\sum_{i}\mathcal{L}(W,x^{*}_{i},y_{i})$
(1)

where $\mathcal{L}$ is the cross-entropy objective defined using the softmax
function $\sigma$ as:

$\mathcal{L}(W,x,y)=-\log(\sigma(Wx)_{y})$

In practice, a neural language model $\phi_{\theta}$ produces contextual
representations $x_{i}=\phi_{\theta}(y_{<i})$ of dimension
$d\in\mathbb{N}^{*}$. The linear language modeling head
$W_{\theta}\in\mathcal{R}^{V\times d}$ is trained concurrently with
$\phi_{\theta}$ with the same objective as in Equation 1.

We focus on the maximal expressiveness of a lower-dimensional head: when
provided with perfect contextual representations $x^{*}_{i}$, what is the
maximal performance level of a linear language modeling head of maximal rank
$d$? This question can be put in mathematical terms:

$W_{d}^{*}=\operatorname*{argmin}_{W\in\mathbb{R}^{V\times\infty}}\sum_{y\in\mathcal{T}}\sum_{i}\mathcal{L}(W,x^{*}_{i},y_{i})\text{
s.t. }rank(W)\leq d$ (2)

Lemma 5.1 shows that by approaching $W^{*}$ directly, we can asymptotically
expect to close the performance gap.

###### Lemma 5.1

(proof in Section A.1) Let’s consider
$W\in\mathbb{R}^{V\times\infty},M\in\mathcal{H}^{V\times\infty}$ the matrix
unit sphere for the Frobenius norm $||\cdot||_{F}$, and
$\varepsilon\in\mathbb{R}^{*}_{+}$ such that $W=W^{*}+\varepsilon M$ . When
$\epsilon\rightarrow 0$:

$|\mathcal{L}(W,x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})|=O(\varepsilon)$

Hence, our problem is linked to a low-rank matrix approximation (Kumar &
Schneider, 2017), which has direct connections with spectral theory. In our
case, we can use the Eckart–Young–Mirsky theorem.

###### Lemma 5.2

(Eckart–Young–Mirsky theorem) Let’s consider $(\sigma_{i})$ the singular
values of $W^{*}$ in decreasing order, and $\mathcal{M}_{d}$ the set of
matrices in $\mathbb{R}^{V\times\infty}$ of rank $d<V=rank(W^{*})$. Then:

$\min_{W_{d}\in\mathcal{M}_{d}}||W_{d}-W^{*}||_{F}=\sqrt{\sum_{i=d+1}^{V}\sigma_{i}^{2}}$

Combining all of the above yields Theorem 5.3.

###### Theorem 5.3

(proof in Section A.2) Let’s consider $(\sigma_{i})$ the singular values of
$W^{*}$ in decreasing order. Then, when $d\rightarrow V$, the loss gap induced
by a $d$-dimensional bottleneck on the linear LM head follows:

$\sum_{y\in\mathcal{T}}\sum_{i}\mathcal{L}(W_{d}^{*},x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})=O\left(\sqrt{\sum_{i=d+1}^{V}\sigma_{i}^{2}}\right)$

These properties shed light on how the dimensionality of the ideal language
modeling head impacts the performance when the LM head is low-rank. However,
the relation obtained in Theorem 5.3 is not particularly strong, as discussed
in Section A.2.

In Figure 8, we compare the results of the head bottleneck experiment of the
Pythia-1B model in Section 5.1 to the $W$-error on the head of the same model
as the bottleneck dimension $d$ evolves. It shows that the loss gap grows
slowly with the $W$-error, implying that even when the allowed rank would lead
to a poor approximation of $W$, the performance can still remain acceptable.
We notice that the performance starts decreasing when the $W$-error outgrows
0.6.

Figure 8: Final loss with trained rank-constrained heads (mimicking
$W_{d}^{*}$), as a function of the theoretical $W$-error for rank $d$ on the
head of the Pythia-1B model.

##  6 Discussion

One way to address the problem at hand could be to train shallow small
language models, increasing hidden dimension at the expense of other
hyperparameters, such as layer count or feed-forward dimension. However, we
believe that such research directions may not be promising in this context.
Previous works have extensively explored and optimized the hyperparameter
choices for various architecture sizes. The impact of width and depth has been
extensively studied (Merrill et al., 2022; Tay et al., 2022; Petty et al.,
2023), often showcasing the importance of depth in final performance and
generalization capabilities.

Another possible way forward consists in implementing more expressive softmax
alternatives (Yang et al., 2018; Chang & McCallum, 2022) in the context of
pretraining small language models on large datasets. We leave the exploration
of such techniques for future work.

We also believe that further exploration of the specific nature of the
singular components after the collapse we describe in Section 4.2 could
improve our understanding of LM saturation. We hypothesize that the resulting
dominating components are correlated with token frequency, based on previous
works that link anisotropy with token frequency (Gao et al., 2019; Ethayarajh,
2019; Biś et al., 2021) and show the importance of token frequency in the LM
head mechanism (Meister et al., 2023).

Beyond the scope of this article, we argue that our work demonstrates that
last-layer anisotropy is symptomatic of performance saturation, and is thus
likely not a desirable property of language models. We also advocate that this
work paves the way towards a better understanding of the structure of the
contextual probability distribution, which could also enhance our
interpretation of the scaling laws.

## Conclusion

Small language models can be affected by performance saturation during
training. We find that this phenomenon can be explained by an inherent
difficulty in mapping a low-dimensional output representation space to a high-
rank contextual probability distribution through a linear language modeling
head. Indeed, we show a theoretical link between the performance gap induced
by a smaller hidden dimension and the spectral properties of the contextual
probability distribution.

We empirically confirm that the rank of such a mapping can be expected to be
relatively high compared to regular hidden dimension choices. Moreover, we
conduct experiments to measure the impact of constraining the rank of the LM
head on the performance of a large language model. Our results show that
performance noticeably drops when using a hidden dimension below 1000. We
further analyze the saturation phenomenon through the lens of spectral
analysis and find that the emergence of last-layer anisotropy that only
affects small models can be correlated with saturation. We also show that the
LM heads of small models concurrently suffer from spectral saturation, i.e. a
uniformization of singular values that leads to a degenerated state.

Our work paves the way for a better understanding of the consequences of the
softmax bottleneck on language modeling, and for the conception of language
models that better embrace the complexity of the target probability
distribution.

## Limitations

The main limitation of this article is the relatively small amount of
saturated language models we studied. As it is the only suite of language
models trained in the range of interest to release an extensive amount of
intermediate checkpoints, we could only observe the training dynamics of small
Pythia models. Although we observe strong last-layer anisotropy for the
smallest GPT-2 model, we cannot tell with certainty whether it suffered from
saturation. The OPT-125m model does not display a strong last-layer
anisotropy, which could indicate that it was not affected by the saturation
phenomenon.

Nevertheless, we argue that this paper does not show that all small models
should suffer from saturation, but rather that the saturation of small
language models is symptomatic of a limitation that may affect language models
that are based on a relatively small hidden dimension.

Another limitation of this work is the loose nature of the mathematical
connection that we establish between the dimensionality of the ideal language
modeling head and the rank-constrained performance (cf. Theorem 5.3).
Moreover, it can also be argued that considering ideal $x_{i}^{*}$
representations is an ill-defined notion. We argue that the reasoning behind
Theorem 5.3 could be applied to any contextual representations, as the ideal
nature of $x_{i}^{*}$ is not necessary in the demonstrations. The word ideal
reflects that our observations hold for $x_{i}^{*}$ representations obtained
from any underlying model, to an extent that depends on the structure that
these representations impose on the $W^{*}$ matrix for a given training set
$\mathcal{T}$.

## Acknowledgements

We thank Song Duong for carefully reviewing this article and for his valuable
suggestions.

This work was funded by the last author’s chair in the PRAIRIE institute
funded by the French national agency ANR as part of the “Investissements
d’avenir” programme under the reference ANR-19-P3IA-0001.

This work was granted access to the HPC resources of GENCI operated by IDRIS
(allocation 2023-AD011013680R1).

## References

  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  The falcon series of open language models, 2023.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pp. 2397–2430. PMLR, 2023.
  * Biś et al. (2021) Daniel Biś, Maksim Podkorytov, and Xiuwen Liu.  Too much in common: Shifting of embeddings in transformer language models and its implications.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 5117–5130, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.403.  URL https://aclanthology.org/2021.naacl-main.403.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Camastra & Staiano (2016) Francesco Camastra and Antonino Staiano.  Intrinsic dimension estimation: Advances and open problems.  _Information Sciences_ , 328:26–41, 2016.  ISSN 0020-0255.  doi: https://doi.org/10.1016/j.ins.2015.08.029.  URL https://www.sciencedirect.com/science/article/pii/S0020025515006179.
  * Chang & McCallum (2022) Haw-Shiuan Chang and Andrew McCallum.  Softmax bottleneck makes language models unable to represent multi-mode word distributions.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8048–8073, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.554.  URL https://aclanthology.org/2022.acl-long.554.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Ethayarajh (2019) Kawin Ethayarajh.  How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 55–65, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1006.  URL https://aclanthology.org/D19-1006.
  * Faysse et al. (2024) Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo.  Croissantllm: A truly bilingual french-english language model, 2024.
  * Gao et al. (2019) Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu.  Representation degeneration problem in training natural language generation models.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=SkEYojRqtm.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.  The pile: An 800gb dataset of diverse text for language modeling, 2020.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, 12 2023.  URL https://zenodo.org/records/10256836.
  * Godey et al. (2024) Nathan Godey, Éric de la Clergerie, and Benoît Sagot.  Anisotropy is inherent to self-attention in transformers, 2024.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023.
  * Jing et al. (2022) Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian.  Understanding dimensional collapse in contrastive self-supervised learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=YevsQ05DEN7.
  * Kanai et al. (2018) Sekitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, and Shuichi Adachi.  Sigsoftmax: Reanalysis of the softmax bottleneck.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.  URL https://proceedings.neurips.cc/paper_files/paper/2018/file/9dcb88e0137649590b755372b040afad-Paper.pdf.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei.  Scaling laws for neural language models.  _ArXiv_ , abs/2001.08361, 2020.  URL https://api.semanticscholar.org/CorpusID:210861095.
  * Kumar & Schneider (2017) N. Kishore Kumar and J. Schneider.  Literature survey on low rank approximation of matrices.  _Linear and Multilinear Algebra_ , 65(11):2212–2244, 2017.  doi: 10.1080/03081087.2016.1267104.  URL https://doi.org/10.1080/03081087.2016.1267104.
  * Lai et al. (2023) Wen Lai, Alexandra Chronopoulou, and Alexander Fraser.  Mitigating data imbalance and representation degeneration in multilingual machine translation.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pp. 14279–14294, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.953.  URL https://aclanthology.org/2023.findings-emnlp.953.
  * Lin (2021) Ying-Chen Lin.  Breaking the softmax bottleneck for sequential recommender systems with dropout and decoupling, 2021.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pp. 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.  URL http://www.aclweb.org/anthology/P11-1015.
  * Meister et al. (2023) Clara Meister, Wojciech Stokowiec, Tiago Pimentel, Lei Yu, Laura Rimell, and Adhiguna Kuncoro.  A natural bias for language generation models.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pp. 243–255, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-short.22.  URL https://aclanthology.org/2023.acl-short.22.
  * Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models, 2016.
  * Merrill et al. (2022) William Merrill, Ashish Sabharwal, and Noah A. Smith.  Saturated transformers are constant-depth threshold circuits.  _Transactions of the Association for Computational Linguistics_ , 10:843–856, 2022.  doi: 10.1162/tacl˙a˙00493.  URL https://aclanthology.org/2022.tacl-1.49.
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez.  The LAMBADA dataset: Word prediction requiring a broad discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1525–1534, Berlin, Germany, August 2016. Association for Computational Linguistics.  URL http://www.aclweb.org/anthology/P16-1144.
  * Petty et al. (2023) Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, and Tal Linzen.  The impact of depth and width on transformer language model generalization, 2023.
  * Puccetti et al. (2022) Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell’Orletta.  Outlier dimensions that disrupt transformers are driven by frequency.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pp. 1286–1304, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.findings-emnlp.93.
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Rajaee & Pilehvar (2021) Sara Rajaee and Mohammad Taher Pilehvar.  A cluster-based approach for improving isotropy in contextual embedding space.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 575–584, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-short.73.  URL https://aclanthology.org/2021.acl-short.73.
  * Rajaee & Pilehvar (2022) Sara Rajaee and Mohammad Taher Pilehvar.  An isotropy analysis in the multilingual BERT embedding space.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 1309–1316, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.103.  URL https://aclanthology.org/2022.findings-acl.103.
  * Rudman et al. (2022) William Rudman, Nate Gillman, Taylor Rayne, and Carsten Eickhoff.  IsoScore: Measuring the uniformity of embedding space utilization.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 3325–3339, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.262.  URL https://aclanthology.org/2022.findings-acl.262.
  * Sardana & Frankle (2023) Nikhil Sardana and Jonathan Frankle.  Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2023.
  * Sharma & Kaplan (2022) Utkarsh Sharma and Jared Kaplan.  Scaling laws from the data manifold dimension.  _Journal of Machine Learning Research_ , 23(9):1–34, 2022.  URL http://jmlr.org/papers/v23/20-1111.html.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler.  Scale efficiently: Insights from pretraining and finetuning transformers.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=f2OYVDyfIB.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy.  Gemma: Open models based on gemini research and technology, 2024.
  * Terashima et al. (2003) Shiro Terashima, Kazuya Takeda, and Fumitada Itakura.  A linear space representation of language probability through svd of n-gram matrix.  _Electronics and Communications in Japan (Part III: Fundamental Electronic Science)_ , 86(8):61–70, 2003.  doi: https://doi.org/10.1002/ecjc.10106.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/ecjc.10106.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Yang et al. (2018) Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen.  Breaking the softmax bottleneck: A high-rank RNN language model.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=HkwZSG-CZ.
  * Zhang et al. (2024) Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu.  Tinyllama: An open-source small language model, 2024.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.
  * Zhou et al. (2021) Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky.  Frequency-based distortions in contextualized word embeddings.  _CoRR_ , abs/2104.08465, 2021.  URL https://arxiv.org/abs/2104.08465.

##  Appendix A Proofs

###  A.1 Lemma 5.1

The proof is mainly based on calculations and limited development:

$\displaystyle|\mathcal{L}(W,x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})|$
$\displaystyle=\left|-\log\frac{\exp((Wx^{*}_{i})_{y_{i}})}{\sum_{j\in
V}\exp((Wx^{*}_{i})_{j})}+\log\frac{\exp((W^{*}x^{*}_{i})_{y_{i}})}{\sum_{j\in
V}\exp((W^{*}x^{*}_{i})_{j})}\right|$ $\displaystyle=\left|-(\varepsilon
Mx^{*}_{i})_{y_{i}}+\log\frac{\sum_{j\in
V}\exp((W^{*}x^{*}_{i})_{j})\exp((\varepsilon Mx^{*}_{i})_{j})}{\sum_{j\in
V}\exp((W^{*}x^{*}_{i})_{j})}\right|$
$\displaystyle=\left|-\varepsilon(Mx^{*}_{i})_{y_{i}}+\log\left(1+\frac{\sum_{j\in
V}\varepsilon\exp((Mx^{*}_{i})_{j})}{\sum_{j\in
V}\exp((W^{*}x^{*}_{i})_{j})}+o(\varepsilon)\right)\right|$
$\displaystyle=\left|-\varepsilon(Mx^{*}_{i})_{y_{i}}+\varepsilon\frac{\sum_{j\in
V}\exp((Mx^{*}_{i})_{j})}{\sum_{j\in
V}\exp((W^{*}x^{*}_{i})_{j})}\right|+o(\varepsilon)$
$\displaystyle=\varepsilon\left|-(Mx^{*}_{i})_{y_{i}}+\frac{\sum_{j\in
V}\exp((Mx^{*}_{i})_{j})}{\sum_{j\in
V}\exp((W^{*}x^{*}_{i})_{j})}\right|+o(\varepsilon)$

The continuous function
$M\longrightarrow\left|-(Mx^{*}_{i})_{y_{i}}+\frac{\sum_{j\in
V}\exp((Mx^{*}_{i})_{j})}{\sum_{j\in V}\exp((W^{*}x^{*}_{i})_{j})}\right|$ is
bounded on the compact matrix unit sphere (i.e. where $||M||_{F}=1$), which
ends the proof.

###  A.2 Theorem 5.3

Let us note $W_{d}$ the best approximation of $W^{*}$ of rank $d$ with respect
to the Frobenius norm. By definition of $W_{d}^{*}$, we have that:

$\left|\sum_{y\in\mathcal{T}}\sum_{i}\mathcal{L}(W_{d}^{*},x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})\right|\leq\sum_{y\in\mathcal{T}}\sum_{i}\left|\mathcal{L}(W_{d},x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})\right|$
(3)

The Eckart-Young-Mirsky theorem tells us that when $d\rightarrow V$,

$||W_{d}-W^{*}||_{F}=\sqrt{\sum_{i=d+1}^{V}\sigma_{i}^{2}}\rightarrow 0$

By defining $\varepsilon=W_{d}-W^{*}$, we can apply Lemma 5.1 and show that:

$\left|\mathcal{L}(W_{d},x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})\right|=O(||W_{d}-W^{*}||_{F})=O\left(\sqrt{\sum_{i=d+1}^{V}\sigma_{i}^{2}}\right)$

From Equation 3, we have that:

$\left|\sum_{y\in\mathcal{T}}\sum_{i}\mathcal{L}(W_{d}^{*},x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})\right|=O\left(\sqrt{\sum_{i=d+1}^{V}\sigma_{i}^{2}}\right)$

By definition of $W^{*}$ and $W_{d}^{*}$, we also have that:

$0\leq\sum_{y\in\mathcal{T}}\sum_{i}\mathcal{L}(W_{d}^{*},x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})=\left|\sum_{y\in\mathcal{T}}\sum_{i}\mathcal{L}(W_{d}^{*},x^{*}_{i},y_{i})-\mathcal{L}(W^{*},x^{*}_{i},y_{i})\right|$

which ends the proof.

#### Remark

The bound used in Equation 3 can be rather loose in practice. We can think of
no particular reason why approaching $W^{*}$ directly should be the optimal
way to minimize the loss on $\mathcal{T}$. Hence, the presented result should
be taken carefully, and we leave the refinement of such an analysis for future
work.

##  Appendix B Hyperparameters

###  B.1 Constrained head experiments (Figure 6)

We freeze the pretrained weights in the Transformer layers, and we train each
rank-constrained head (i.e. in the form $W=AB$ with $r$ as the inner dimension
of the matrix product) for various values of $r$ on 150M tokens sampled from
The Pile using 4 V100 GPUs for the Pythia models and 4 A100 GPUs for Llama-7B.
We use the hyperparameters from Biderman et al. (2023), except for the batch
size which we set to 256 as it fits our hardware setup better. As the
trainable parameter count evolves with $r$, we search for the best-performing
learning rates among values ranging from $1\cdot 10^{-3}$ to $5\cdot 10^{-2}$.

We report the chosen learning rates in Figure 9.

Figure 9: Chosen peak learning rates used for the rank-constrained head
experiments for each model.