# Phonon-phonon coupling in bismuth vanadate over a large temperature range
across the monoclinic phase

Christina Hill  Materials Research and Technology Department, Luxembourg
Institute of Science and Technology, 41 rue du Brill, L-4422 Belvaux,
Luxembourg  University of Luxembourg, 41 rue du Brill, L-4422 Belvaux,
Luxembourg  Georgy Gordeev  Mael Guennou  University of Luxembourg, 41 rue du
Brill, L-4422 Belvaux, Luxembourg

(August 28, 2024)

###### Abstract

In this work we study phonon-phonon coupling in bismuth vanadate (BiVO4),
known for its second-order transition involving a variety of coupling
mechanisms. Using Raman spectroscopy as a probe, we identify two optical
coupled phonon modes of the VO4 tetrahedron and study them by varying light
polarization and temperature. The coupling manifests in non-Lorentzian line-
shapes of Raman peaks and frequency shifts. We use theoretical framework of
coupled damped harmonic oscillators to model the coupling and capture the
phenomena in the temperature evolution of the coupling parameters. The
coupling is negligible at temperatures below 100 K and later increases in
magnitude with temperature until 400 K. The sign of the coupling parameter
depends on the light polarization direction, causing either phonon attraction
or repulsion. After 400 K the phonon-phonon coupling diminishes when
approaching phase transition at which the phonon modes change their symmetry
and the coupling is no longer allowed.

Suggested keywords

††preprint: AIP/123-QED

## Introduction

Raman spectroscopy is a powerful technique to study lattice dynamics in
ferroic oxides. In these materials, the phonon frequency strongly depends on
external parameters such as temperature or pressure as well as on the order
parameter for example the strain in the material. In order to determine the
phonon frequency precisely a proper model to describe the line-shape of the
phonon mode should be used. Typically, Raman bands are modelled by a
Lorentzian function derived from the classical damped harmonic oscillator
model in the approximation of low damping. The eigenfrequency is an important
parameter to follow one particular phonon mode across the phase transition. In
practice, various phenomena may lead to a deviation from a perfect Lorentzian
line-shape: Gaussian like broadening due to instrumental effects [1, 2], the
Doppler broadening in gases [3], the superimposition of Raman bands as in
silicate glass[4], a strong anharmonic potential of atomic bands[5] as in
ZnO[6] and in PbTiO3 [7] or sample related effects observed for example in
nanoparticales [8, 9, 10, 5], graphene and related systems [11, 12] or in
polymers[13]. Another source for deviations from Lorentzian line-shape are
coupling effects of various kind, i.e. when the system cannot be reduced to as
set of independent oscillators. Coupling between two phonon bands is discussed
in literature for quartz [14],BiTO3 [15] and AlPO4[16, 17]. In ferroelectric
KPD (KH2PO4) [18] and in the arsenates CsH2AsO4 and KH2AsO4 [19], proton-
phonon coupling causes the deviation from a Lorentzian line-shape. Another
widely discussed phenomenon is the coupling between one phonon and a broad
continuum [20, 21, 22, 23] in literature referred to as Auger-like resonant
interference[20], Fano-interference [22, 23] or Breit-Wigner-Fano coupling
[24, 25, 26, 22, 27]. Coupling effects typically cause an asymmetric line-
shape. In that case, the eigenfrequency no longer converges with central
frequency of the Raman mode. Modeling correctly the Raman line-shape is key to
extract the eigenfrequency and therefore identify the Raman mode correctly.

The coupled damped harmonic oscillator (coupled oscillator) model has first
been introduced by Barker and Hopfield.[28] They successfully applied it to
explain the dielectric properties of various perovskites. Later, Scott[16],
Zawadowski and Ruvalds [29], used the Green’s formalism to derive a more
generalized form to model phonon-phonon coupling in AlPO4. Katiyar et al. [19]
and She et al. [18] then applied the Greens formalism for proton-phonon
coupling in KH2AsO4 (KDA) and KH2PO4 (KDP), respectively. The coupling between
phonon modes is a temperature induced effect and Chaves  _et al._ [15] showed
that only modes of same symmetry can couple. While the coupled oscillators
model is known, its implementation can be challenging and misleading. In many
cases, multiple solutions provide a good fit to the data leading to very
different physical conclusions. The implementation of physical meaningful
constraints on the modeling parameters is a key to overcome this problem. In
the literature, constraining the complex coupling parameter assuming either
pure real [19, 15, 18, 30] or pure imaginary [31, 32] coupling has been widely
used. However, the physical correctness of the coupling parameters needs to be
carefully revised. This can be done by fitting the line-shape of data measured
at many temperatures (or pressures or electric fields). However there are not
many examples in the literature where this has been done.BiVO4 is a nice
example to study effects of phonon-phonon coupling since it is well known for
the phase transition driven by a Bg optical soft mode (tetragonal) coupled to
an acoustic mode.[33, 34] The phase transition in BiVO4 is classified as 2nd
order from high temperature tetragonal phase (space group $I4_{1}/a$) to a low
temperature monoclinic phase (space group $C27c$, most commonly described in
non-standard setting $I2/a$), also referred to as Scheelite and Fergusonite
structures respectively. The Raman active modes are known from standard
symmetry analysis and they have been successfully assigned.[34, 35] Their
frequency shifts with temperature and pressure across the phase transition
have been reported.[35] However, we observed line-shape asymmetries of the
vibration modes of the VO4 tetrahedra that strongly depends on temperature and
on the polarization conditions of the experiment.

In this work we measured the Raman modes corresponding to the vibration modes
of the VO4 tetrahedron in BiVO4 over large temperature range at different
light polarization geometries. We observed anomalies in the line-shape and in
the frequency position of these Raman modes that suggest coupling between
phonons of the same symmetry. We applied the coupled damped harmonic
oscillators model to describe the observed phonon-phonon coupling for two
different polarization conditions. Finally We studied the phonon-phonon
coupling in a high temperature range from 10 K to 510 K across the phase
transition.

## Experimental Methods

We investigated Raman spectra of BiVO4 single crystals over a large
temperature range. The Raman spectrometer InVia from Renishaw in
backscattering configuration with a microscope, 50x objective was used. For
all measurements, we used $633\text{\,}\mathrm{nm}$
($1.96\text{\,}\mathrm{eV}$) laser excitation. We use the Porto’s notation
i(jk)l to label the different polarization geometries, where i and l
correspond to the propagation direction and j and k the polarization direction
with respect to the crystallographic axis. Linkam temperature cell THMS600 was
used to cover temperatures from $80\text{\,}\mathrm{K}$ to
$500\text{\,}\mathrm{K}$. The helium flow cryostat Microstat from Oxford
Instruments was used to cover the low temperature range from
$10\text{\,}\mathrm{K}$ to $300\text{\,}\mathrm{K}$.

The crystal structure of BiVO4 is tetragonal at high temperature with a phase
transition at $523\text{\,}\mathrm{K}$ to monoclinic. Fig. 1 (a) shows the
tetragonal unit cell with 4-fold rotational axis along $[001]$. In the low-
temperature monoclinic phase, the 4-fold rotational axis becomes a 2-fold
rotational axis, the so called principle axis. For this study we used a single
with the surface orientation $(110)$. The view on the $(110)$ surface is given
in Fig. 1 (b) in which the principal axis is labeled with $c$ and the
$[1\bar{1}0]$-axis with $b^{\prime}$. The direction $[110]$ pointing out of
the surface plane is labeled with $a^{\prime}$. From standard symmetry
analysis, we expect 8 phonon modes of $A_{g}$ symmetry and 10 phonon modes of
$B_{g}$ symmetry in the monoclinic phase.[35, 34]

Figure 1: Schematic representation of (a) the tetragonal unit cell of bismuth
vanadate and (b) view on the $(001)$ surface. Visualization of the symmetries
of the atomic vibrations of the oxygen tetrahedron corresponding to (c) the
$A_{g}^{7}$ and (d) the $A_{g}^{8}$.

The Raman (Stokes) scattering intensity is correlated to the imaginary part of
the susceptibility $\chi^{\prime\prime}$

$S(\omega)\propto\chi^{\prime\prime}(\omega)[n(\omega)+1],$ (1a)

where $n$ is the phonon population factor given by the Bose thermal factor
$n(\omega)=(e^{\hbar\omega/kT}-1)^{-1}$ and T is the absolute temperature of
the sample. All presented data have been divided by the thermal Bose factor
${n(\omega)+1}$ so that they are directly proportional to the imaginary part
of the susceptibility. The description of the Raman scattering intensity
strongly depends on the model chosen for the susceptibility.

## Theoretical Methods

In the following the coupled damped harmonic oscillators (coupled oscillators)
model is explained. The basic concept has been developed by Barker and
Hopfield.[28] Fig. 2 shows the mechanical model for a system of two coupled
phonons. Each phonon is described as a damped harmonic oscillator with an
effective mass $m$, spring constant $k$ and damping coefficient $\Gamma$. The
two masses $\alpha$ and $\beta$ are mechanically connected by an additional
spring $k_{\alpha\beta}$ and dashpot $\Gamma_{\alpha\beta}$.

Figure 2: Mechanical model of two damped harmonic oscillators $\alpha$ and
$\beta$ described by spring constants $k_{\alpha}$, $k_{\beta}$ and the
damping coefficients $\Gamma_{\alpha}$,$\Gamma_{\beta}$. The coupling is
complex and given by the coupling spring constant $k_{\alpha\beta}$ and the
damping coefficient $\Gamma_{\alpha\beta}$.

The Green’s function may be written as
$G^{-1}=g=-\omega^{2}\bm{I_{2}}+i\omega\bm{\Gamma}+\bm{K}$ in which
$\bm{I_{2}}$ being the unit matrix and $\bm{K}$ and $\bm{\Gamma}$ being
restoring force and damping matrices. Using the Green’s function formalism a
system of two coupled damped harmonic oscillators[19, 18] is defined by

$\begin{bmatrix}\omega_{\alpha}^{2}-\omega^{2}+i\omega\Gamma_{\alpha}&\Delta^{2}+i\omega\Gamma_{\alpha\beta}\\\
\Delta^{2}+i\omega\Gamma_{\alpha\beta}&\omega_{\beta}^{2}-\omega^{2}+i\omega\Gamma_{\beta}\end{bmatrix}\begin{bmatrix}G_{\alpha\alpha}&G_{\alpha\beta}\\\
G_{\alpha\beta}&G_{\beta\beta}\end{bmatrix}=\begin{bmatrix}1&0\\\
0&1\end{bmatrix}$ (1b)

.

The parameters $\Delta$, $\Gamma_{\alpha}$, $\Gamma_{\beta}$ and
$\Gamma_{\alpha\beta}$ are frequency independent. The effective mass $m$ is
set to unity. The coupling is described by $\Delta$ and $\Gamma_{\alpha\beta}$
which is the force constant of the spring and the damping coefficient of the
dashpot connecting the two oscillators. $\Delta$ and $\Gamma_{\alpha\beta}$
must be constraint by $det(K),det(\Gamma)\geq 0$.[36] This means that the
coupling cannot be arbitrary large. The frequencies $\omega_{\alpha}$ and
$\omega_{\beta}$ correspond to the eigenfrequencies of the uncoupled phonons.
The eigenfrequency squared is equal to the spring constant divided by the
effective mass $m$,
$\omega_{\alpha/\beta}^{2}=k_{\alpha/\beta}/m_{\alpha/\beta}$.
$\Gamma_{\alpha}$ and $\Gamma_{\beta}$ are the damping coefficient of the
uncoupled vibration modes $\alpha$ and $\beta$.

Equation 1b is solved to give the following Green’s functions coefficients:

$G_{\alpha\alpha}=(\omega_{\beta}^{2}-\omega^{2}+i\omega\Gamma_{\beta})/D$
(1c) $G_{\alpha\beta}=-(\Delta^{2}+i\omega\Gamma_{\alpha\beta})/D$ (1d)
$G_{\beta\beta}=(\omega_{\alpha}^{2}-\omega^{2}+i\omega\Gamma_{\alpha})/D$
(1e)

where
$D=(\omega_{\alpha}^{2}-\omega^{2}+i\omega\Gamma_{\alpha})(\omega_{\beta}^{2}-\omega^{2}+i\omega\Gamma_{\beta})-(\Delta^{2}+i\omega\Gamma_{\alpha\beta})^{2}$.

Equation 1b allows arbitrary choice of diagonalization and can produce an
infinite number of solutions.[37] Some physical assumptions have to be made to
filter reasonable solutions. Many authors[38, 36] proposed to make the
assumptions on the complex coupling of modes in which the two limiting cases
occur for only real coupling ($\Gamma_{\alpha\beta}=0$) and only imaginary
coupling ($\Delta=0$). The parameters obtained in the two cases are entirely
different as well as the physical interpretation, see supplementary
information. In this work, we suggest to make use of the temperature dependent
data as well as the polarization dependent data to find an unique solution to
the coupled oscillators model.

Using matrix notation, the complex susceptibility $\chi(\omega)$ for coupled
oscillators model is given by $\chi(\omega)=\tilde{P}\cdot G\cdot P$ in which
$P$ and $\tilde{P}$ are one column and one row vectors. The imaginary part of
the complex susceptibility $\chi^{\prime\prime}(\omega)$ is then given by

$\chi^{\prime\prime}(\omega)=-Im[P_{\alpha}^{2}G_{\alpha\alpha}(\omega)+2P_{\alpha}P_{\beta}G_{\alpha\beta}(\omega)+P_{\beta}^{2}G_{\beta\beta}(\omega)],$
(1f)

$P_{\alpha}$, $P_{\beta}$ being the oscillator strengths of the vibration mode
$\alpha$ and $\beta$ and $\chi^{\prime\prime}$ is proportional to the
intensity of the Raman spectrum.

## Experimental results

First, we discuss the evidences for coupling phenomena in the high-frequency
Raman spectrum of BiVO4. Raman spectra showing all 8 Raman modes of $A_{g}$
symmetry are shown in the supplementary information. For most of these Raman
modes, the eigenfrequency does not dependent on the polarization conditions
chosen in the experiment, however this is different for the high-frequency
$A_{g}^{7}$ mode that shows coupling signatures with the $A_{g}^{8}$ mode.
Fig. 1 (b) shows $A_{g}^{7}$ near $705\text{\,}{\mathrm{cm}}^{-1}$ and
$A_{g}^{8}$ at $830\text{\,}{\mathrm{cm}}^{-1}$ measured at room temperature.
The symmetries of the atomic vibrations corresponding to the polar $A_{g}^{7}$
and the non-polar $A_{g}^{8}$ mode are shown in Figs. 1 (c) and (d),
respectively.

Fig. 3 compares the phonon modes $A_{g}^{7}$ and $A_{g}^{8}$ measured at (a)
$200\text{\,}\mathrm{K}$ and (b) room temperature for two polarization
geometries. The first observation we made on the raw data is that the central
frequency of the $A_{g}^{7}$ depends on the polarization geometry,
$\Delta\omega=8$ and $10\text{\,}{\mathrm{cm}}^{-1}$ for 200 and 300 K,
respectively. Second, at $300\text{\,}\mathrm{K}$ and for the polarization
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ (brown spectrum in Fig.3
(b)), we see additional intensity in between the two Raman modes.

Lorentz model fails to reproduce the experimental data, see dotted line. At
$200\text{\,}\mathrm{K}$ the Lorenz model only partly converges with the data
by assuming non physical behavior: different $A_{g}^{7}$ eigenfrequency for
different polarization conditions. The fitting parameters are given in Table
1. At $300\text{\,}\mathrm{K}$ the two Lorentzian peaks fit even fails to
identify the $A_{g}^{7}$ phonon mode, if the frequency is not fixed. There are
two reasons for that; first the line-shape of the $A_{g}^{7}$ is more
asymmetric at higher temperatures and there is significant intensity
background in between the two modes. Overall it is clear that the Lorentz
model cannot be used for the Raman interpretation since it delivers falsified
results. While the eigenfrequency of the $A_{g}^{8}$ mode almost matches for
both polarization geometries, the eigenfrequency of the $A_{g}^{7}$ mode is
different from one polarization geometry to the other by almost 1.2 $\%$. This
is rather unexpected, since mode frequency should be independent upon
polarization and can only be explained by the coupled oscillator model.

Now we will discuss the fits using the coupled oscillators model. For the
coupled oscillators model we imposed the same eigenfrequency and the damping
coefficient for both polarization geometries. Further, we assumed the real
part of the coupling parameter $\Delta$ to be zero. For the $A_{g}^{8}$ phonon
mode, there is little difference between the classical Lorentzian and the
coupled oscillators models. The overall line-shape is well described by both
models and the frequency position as well as the damping coefficient of the
$A_{g}^{8}$ mode is nearly identical for the both models. The major
improvement is found for the $A_{g}^{7}$ mode at $300\text{\,}\mathrm{K}$. The
coupled oscillators model succeeds to describe the asymmetry in the line-shape
of the $A_{g}^{7}$ mode as well as the increased background intensity in
between the two modes. This is especially true for the
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ polarization geometry. The
coupling coefficients are listed in Table 1, they vary from one polarization
geometry to the other. For $b^{\prime}(cc)\bar{b}^{\prime}$, the imaginary
part of the coupling parameter $\Gamma_{78}$ is positive which correspond to a
repulsion of the two phonon modes involved in the coupling. In contrast, for
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$, the imaginary part
$\Gamma_{78}$ is negative leading to an attraction of the two phonon modes.

Figure 3: Raman spectra of the high-frequency $A_{g}^{7}$ and $A_{g}^{8}$ modes measured at different polarization geometries $b^{\prime}(cc)\bar{b}^{\prime}$ and $b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ at (a) $200\text{\,}\mathrm{K}$ and b) $300\text{\,}\mathrm{K}$ with a $1.96\text{\,}\mathrm{eV}$ excitation. Dashed lines show the fit assuming two Lorentzian oscillators in black and coupled oscillators model in blue. The fitting parameters are given in Table 1. Table 1: Comparison of fitting parameters between the classical Lorentzian model and coupled oscillators model at $200\text{\,}\mathrm{K}$ and $300\text{\,}\mathrm{K}$. Here, the real part of the coupling $\Delta$ is fixed to zero. The eigenfrequencies $\omega$ and damping coefficients $\Gamma$ are given in units of $\text{\,}{\mathrm{cm}}^{-1}$. | 200K | 300K
---|---|---
| $b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ | $b^{\prime}(cc)\bar{b}^{\prime}$ | $b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ | $b^{\prime}(cc)\bar{b}^{\prime}$
Lorentzian model
$\omega_{7}$ | 708 | 700 | 711 (fixed) | 701 (fixed)
$\Gamma_{7}$ | 24 | 22 | 54 | 31
$\omega_{8}$ | 834 | 835 | 830 | 830
$\Gamma_{8}$ | 26 | 26 | 36 | 34
Coupled oscillators model
$\omega_{7}$ | 706 | 712
$\Gamma_{7}$ | 25.5 | 47.0
$\omega_{8}$ | 835 | 831
$\Gamma_{8}$ | 26.2 | 34.9
$\Gamma_{78}$ | -5.3 | 12.3 | -15.0 | 22.4

We further study the temperature dependence of phonon-phonon more
systematically. Fig. 4 (a) shows the $A_{g}^{7}$ and the $A_{g}^{8}$ mode
measured at temperatures ranging from $10\text{\,}\mathrm{K}$ up to phase
transition at $523\text{\,}\mathrm{K}$ for the polarization geometries
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$. By increasing the
temperature we observe typical effects: the modes shift and broadening with
temperature. The $A_{g}^{8}$ mode clearly shifts to lower frequencies with
increasing temperatures whereas the $A_{g}^{7}$ seemingly shift to higher
frequencies. Also we can directly see the evolution of phonon-phonon coupling.
At low temperatures, the two Raman modes are well isolated whereas at higher
temperature additional intensity develops in between the modes. The additional
intensity increases with increasing temperatures with respect to the intensity
contribution from the individual modes. That is why, we expect a strong
temperature dependence of the coupling parameter.

Fig. 4 (b) shows a zoomed spectral range with the $A_{g}^{7}$ mode measured
with $b^{\prime}(cc)\bar{b}^{\prime}$ polarization. In this light polarization
the $A_{g}^{7}$ mode becomes extremely weak compared to the $A_{g}^{8}$ mode
while approaching the phase transitions. At temperatures above
$470\text{\,}\mathrm{K}$, the $A_{g}^{7}$ mode is no longer visible in the raw
data, see also the supplementary information. Surprisingly, the $A_{g}^{7}$
mode shifts to lower frequencies with increasing temperatures which
contradicts the measurement for
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ shown in Fig. 4 (a). A
phonon-phonon coupling model is required to explain this unconventional
behavior.

The black dashed lines in Fig. 4 are fits based on the coupled oscillators
model. For both polarization geometries we find a good agreement between the
model and experiment. As mentioned in the methods section, the coupled
oscillators model is overparameterized so that assumptions on the fitting
parameters have to be made. We assumed i) the coupling parameter $\Delta$ to
be zero, ii) the individual damping $\Gamma_{7}$, $\Gamma_{8}$ to increase
linearly with temperature at a constant slope of $0.1\text{\,}$ and
$0.08\text{\,}{\mathrm{cm}}^{-1}\text{\,}{\mathrm{K}}^{-1}$, respectively as
expected for two-phonon decay and iii) the eigenfrequency of the $A_{g}^{7}$
mode to be constant as a function of temperature for the light polarization
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$. As a final element we
implemented a $weight$ equal to the inverse of the intensity in the
minimization algorithm. The implementation of a $weight$ compensates for the
huge intensity difference between the $A_{g}^{7}$ and the $A_{g}^{8}$ mode and
is introduced as a multiplication factor so that $weight\cdot(data-fit)^{2}$
is minimized . This non-standard implementation in the minimization algorithm
was key to find fitting solutions for the data measured at
$b^{\prime}(cc)\bar{b}^{\prime}$ polarization and at high temperatures. The
intensity difference is not as drastic for the
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ light polarization, so that
in this case the $weight$ was set to one.

Figure 4: Coupled oscillators model applied to the Raman modes $A_{g}^{7}$ and
$A_{g}^{8}$ at different temperatures for the (a) polarization geometry
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ and (b)
$b^{\prime}(cc)\bar{b}^{\prime}$ . Here, the real part of the coupling
$\Delta$ is assumed to be zero. The damping of the individual oscillators
$\Gamma_{7}$ and $\Gamma_{8}$ is assumed to increase linear with temperature.
Only for polarization geometry
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$, eigenfrequency
$\omega_{7}$ of the $A_{g}^{7}$ mode is assumed to be constant.

Fig. 5 (a) shows the eigenfrequency $\omega_{8}$ of the $A_{g}^{8}$ mode
extracted from the fits shown in Fig.4 as a function of temperature for the
two polarization geometries. The data points above the phase transition are
included and symbolized with a cross. The eigenfrequency $\omega_{8}$ strongly
decreases with increasing temperature as it is visible in the raw data shown
in Fig.4. Across the phase transition at $T_{c}=510K$, the change in the
eigenfrequency is continuous. This is expected for the second-order phase
transition, but the slope clearly changes at the phase transition.

Fig. 5 (b) shows the eigenfrequency $\omega_{7}$ of the $A_{g}^{7}$ as a
function of temperature for the two polarization geometries. For the
polarization geometry $b^{\prime}(cc)\bar{b}^{\prime}$, the eigenfrequency of
the $A_{g}^{7}$ is constant over a large temperature range, before the values
slightly increases while approaching the phase transition. A constant
temperature trend of eigenfrequency $\omega_{7}$ is only obtained by the
coupled oscillators model, see also Lorentzian fitting as a function of
temperature in the supplementary information. Even though we do not have data
at temperatures above $470\text{\,}\mathrm{K}$, the change in the
eigenfrequency seems to be continuous similar to the discussion for the
eigenfrequency of the $A_{g}^{8}$ mode.

To find a reliable temperature trend for the coupling parameter we assumed the
eigenfrequency $\omega_{7}$ to be independent on temperature for the second
polarization geometry $b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$. The
intensity of the mode is relatively low therefore constraining it’s frequency
substantially improved the reliability of the fit. This is fully consistent
with lattice dynamics where in non-polar crystals the $\omega_{7}$ should be
polarization independent. Therefore the $\omega_{7}$ was set to
$705\text{\,}{\mathrm{cm}}^{-1}$, same as the average frequency found in the
$b^{\prime}(cc)\bar{b}^{\prime}$ polarization direction. While the
eigenfrequency of the $A_{g}^{8}$ mode is robust against different modeling
approaches, coupled oscillators model succeeds to deliver physical results for
the $A_{g}^{7}$ mode, where experimental data are reproduced with the same
eigenfrequency for both polarization directions in the entire measured
temperature range.

Figure 5: Fitting parameters of the coupled oscillators model applied to the
Raman modes $A_{g}^{7}$ and $A_{g}^{8}$ as a function of temperature:
eigenfrequency for the two polarization geometries of (a) the low intensity
$A_{g}^{7}$ mode and (b) of the $A_{g}^{8}$ mode.

Fig. 6 shows the coupling parameter $\Gamma_{78}$ as a function of temperature
for the polarization geometries $b^{\prime}(cc)\bar{b}^{\prime}$ and
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$, respectively. In the
polarization geometry $b^{\prime}(cc)\bar{b}^{\prime}$, the coupling parameter
is positive and increases continuously with increasing temperatures below 400
K, see red circles in Figure 6. On the other hand the $\Gamma_{78}$ for
$b^{\prime}(cc)\bar{b}^{\prime}$ has an opposite trend in the same temperature
range. The overall magnitude of the coupling increases, but the sign is
negative, see green triangles in Figure 6. The data between
$160\text{\,}\mathrm{K}$ and $300\text{\,}\mathrm{K}$ have been measured at a
later experiment and show a small offset to overall temperature trend. Overall
we have a precise control over the phonon-phonon coupling, where the coupling
strength is set by temperature and the coupling sign by the light
polarization.

Now we discus the coupling behavior near the phase transition. When going from
monoclic to tetragonal the phonon symmetries change, along with the selection
rules. The $A_{g}^{7}$ changes its symmetry to a $B_{g}$ symmetry while the
$A_{g}^{8}$ remains of $A_{g}$ symmetry. As a consequence, we expect no
coupling between these two phonons in the high-temperature tetragonal phase.
This is reflected in change of the temperature trend found the coupling
parameter for the light polarization
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ in which the coupling
parameter decreases while approaching the phase transition.

The situation is slightly different for the other light polarization
$b^{\prime}(cc)\bar{b}^{\prime}$, where only modes of $A_{g}$ symmetry are
allowed in the tetragonal phase. [35] Therefore, the Raman intensity of the
$A_{g}^{7}$ mode gradually decreases with temperature until it disappears in
the tetragonal phase. This is nicely observed in the waterfall plot shown in
Fig 4 (b). The ratio of the relative intensities is plotted in the
supplementary information. Therefore the symbols for $\Gamma_{78}$ at
$b^{\prime}(cc)\bar{b}^{\prime}$ are missing in Figure 6. Overall we in BiVO4
we find a beautiful demonstration how the change of phonon symmetries
influences phonon-phonon coupling. We confirm that only phonon modes of the
same symmetry can couple as predicted by Chaves  _et al._ [15].

Figure 6: Coupling parameter $\Gamma_{78}$ as a function of temperature for
the polarization geometry $b^{\prime}(cc)\bar{b^{\prime}}$ and
$b^{\prime}(a^{\prime}a^{\prime})\bar{b^{\prime}}$. Because the intensity of
the $A_{g}^{7}$ mode diminishes while approaching the phase transition, there
are no data for $\Gamma_{78}$ at temperatures above $470\text{\,}\mathrm{K}$
in case of the $b^{\prime}(cc)\bar{b^{\prime}}$ polarization.

## Conclusion

In BiVO4 we studied coupling between two optical phonons with the $A_{g}$
symmetry. We show that the phonon-phonon coupling causes the asymmetric line-
shapes of the $A_{g}^{7}$ and $A_{g}^{8}$ mode at temperatures above 100 K.
Depending on the polarization geometry, the sign of coupling parameter is
either positive or negative referring to either repulsion or attraction of
phonon modes. The coupling strength, expressed in the absolute value of the
coupling parameter, increases with increasing temperatures. At temperatures
above $400\text{\,}\mathrm{K}$, the coupling strength decreases for the
$b^{\prime}(a^{\prime}a^{\prime})\bar{b}^{\prime}$ polarization geometry. This
is explained by the change in symmetry of the $A_{g}^{7}$ mode, which leads to
decoupling of the two phonon modes above the phase transition temperature.

## Acknowledgments

CH acknowledges funding from the Fond National de la Recherche under Project
PRIDE/15/10935404. This research was funded in whole, or in part, by the
Luxembourg National Research Fund (FNR), grant reference
[INTER/MERA20/14995416/SWIPE]. For the purpose of open access, and in
fulfilment of the obligations arising from the grant agreement, the author has
applied a Creative Commons Attribution 4.0 International (CC BY 4.0) license
to any Author Accepted Manuscript version arising from this submission.

## References

  * Meier [2005a] R. J. Meier, On art and science in curve-fitting vibrational spectra, Vibrational Spectroscopy 39, 266 (2005a).
  * Chen and Dai [2016] Y. Chen and L. Dai, Automated decomposition algorithm for Raman spectra based on a Voigt line profile model, Applied Optics 55, 4085 (2016).
  * Alsmeyer and Marquardt [2004] F. Alsmeyer and W. Marquardt, Automatic generation of peak-shaped models, Applied Spectroscopy 58, 986 (2004).
  * Mysen  _et al._ [1982] B. O. Mysen, L. W. Finger, D. Virgo, and S. F. A., Curve-fitting of raman spectra of silicate glasses, American Mineralogist 67, 686 (1982).
  * Gouadec and Colomban [2007] G. Gouadec and P. Colomban, Raman spectroscopy of nanomaterials: How spectra relate to disorder, particle size and mechanical properties, Progress in Crystal Growth and Characterization of Materials 53, 1 (2007).
  * Cuscó  _et al._ [2007] R. Cuscó, E. Alarcón-Lladó, J. Ibáñez, L. Artús, J. Jiménez, B. Wang, and M. J. Callahan, Temperature dependence of raman scattering in zno, Physical Review B 75, 165202 (2007).
  * Schwarz and Maier [5 01] U. T. Schwarz and M. Maier, Asymmetric raman lines caused by an anharmonic lattice potential in lithium niobate, Physical Review B 55, 11041 (1997-05-01).
  * Adu  _et al._ [2006] K. Adu, Q. Xiong, H. Gutierrez, G. Chen, and P. Eklund, Raman scattering as a probe of phonon confinement and surface optical modes in semiconducting nanowires, Applied Physics A 85, 287 (2006).
  * Nemanich  _et al._ [1981] R. J. Nemanich, S. A. Solin, and R. M. Martin, Light scattering study of boron nitride microcrystals, Physical Review B 23, 6348 (1981).
  * Korepanov and Hamaguchi [2017] V. I. Korepanov and H. O. Hamaguchi, Quantum-chemical perspective of nanoscale Raman spectroscopy with the three-dimensional phonon confinement model, Journal of Raman Spectroscopy 48, 842 (2017).
  * Martins Ferreira  _et al._ [2010] E. H. Martins Ferreira, M. V. O. Moutinho, F. Stavale, M. M. Lucchese, R. B. Capaz, C. A. Achete, and A. Jorio, Evolution of the raman spectra from single-, few-, and many-layer graphene with increasing disorder, Physical Review B 82, 125429 (2010).
  * Huang  _et al._ [2009] M. Huang, H. Yan, C. Chen, D. Song, T. F. Heinz, and J. Hone, Phonon softening and crystallographic orientation of strained graphene studied by Raman spectroscopy, Proceedings of the National Academy of Sciences of the United States of America 106, 7304 (2009).
  * Lehnert  _et al._ [1997] R. Lehnert, P. Hendra, N. Everall, and N. Clayden, Comparative quantitative study on the crystallinity of poly(tetrafluoroethylene) including raman, infra-red and 19F nuclear magnetic resonance spectroscopy, Polymer 38, 1521 (1997).
  * Scott [1968] J. F. Scott, Evidence of coupling between one- and two-phonon excitations in quartz, Physical Review Letters 21, 907 (1968).
  * Chaves  _et al._ [1974a] A. Chaves, R. S. Katiyar, and S. P. Porto, Coupled modes with A1 symmetry in tetragonal BaTiO3, Physical Review B 10, 3522 (1974a).
  * Scott [1970] J. F. Scott, Hybrid Phonons and Anharmonic Interactions in AlPO4, Physical Review Letters 24, 1107 (1970).
  * Scott [1974] J. F. Scott, Soft-mode spectroscopy: Experimental studies of structural phase transitions, Reviews of Modern Physics 46, 83 (1974).
  * She  _et al._ [1972] C. Y. She, T. W. Broberg, L. S. Wall, and D. F. Edwards, Effect of proton-phonon coupling on the ferroelectric mode in $\mathrm{KH_{2}PO_{4}}$, Physical Review B 6, 1847 (1972).
  * Katiyar  _et al._ [1971] R. S. Katiyar, J. F. Ryan, and J. F. Scott, Proton-phonon coupling in $\mathrm{CsH_{2}AsO_{4}}$ and $\mathrm{KH_{2}AsO_{4}}$, Physical Review B 4, 2635 (1971).
  * Rousseau and Porto [1968] D. L. Rousseau and S. P. S. Porto, Auger-like resonant interference in raman scattering from one- and two-phonon states of $\mathrm{BaTiO_{3}}$, Physical Review Letters 20, 1354 (1968).
  * Ager  _et al._ [1995] J. W. Ager, W. Walukiewicz, M. McCluskey, M. A. Plano, and M. I. Landstrass, Fano interference of the raman phonon in heavily boron-doped diamond films grown by chemical vapor deposition, Applied Physics Letters 66, 616 (1995).
  * Burke  _et al._ [2010] B. G. Burke, J. Chan, K. A. Williams, Z. Wu, A. A. Puretzky, and D. B. Geohegan, Raman study of fano interference in p-type doped silicon, Journal of Raman Spectroscopy 41, 1759 (2010), 0910.5244 [cond-mat, physics:physics] .
  * Magidson and Beserman [2002] V. Magidson and R. Beserman, Fano-type interference in the raman spectrum of photoexcited si, Physical Review B 66, 195206 (2002).
  * Fano [1961] U. Fano, Effects of configuration interaction on intensities and phase shifts, Physical Review 124, 1866 (1961).
  * Breit and Wigner [1936] G. Breit and E. Wigner, Capture of slow neutrons, Physical Review 49, 519 (1936).
  * Brown  _et al._ [2001] S. D. M. Brown, A. Jorio, P. Corio, M. S. Dresselhaus, G. Dresselhaus, R. Saito, and K. Kneipp, Origin of the breit-wigner-fano lineshape of the tangential G -band feature of metallic carbon nanotubes, Physical Review B 63, 155414 (2001).
  * Gordeev  _et al._ [2016] G. Gordeev, A. Setaro, M. Glaeske, S. Jürgensen, and S. Reich, Doping in covalently functionalized carbon nanotubes: A Raman scattering study, Physica Status Solidi (B) 253, 2461 (2016).
  * Barker and Hopfield [1964] A. S. Barker and J. J. Hopfield, Coupled-optical-phonon-mode theory of the infrared dispersion in $\mathrm{BaTiO_{3}}$ , $\mathrm{SrTiO_{3}}$ , and $\mathrm{KTaO_{3}}$, Physical Review 135, A1732 (1964).
  * Zawadowski and Ruvalds [1970] A. Zawadowski and J. Ruvalds, Indirect Coupling and Antiresonance of Two Optic Phonons, Physical Review Letters 24, 1111 (1970).
  * Scalabrin  _et al._ [1977] A. Scalabrin, A. S. Chaves, D. S. Shim, and S. P. Porto, Temperature dependence of the A1 and E optical phonons in BaTiO3, Physica Status Solidi (B) 79, 731 (1977).
  * Lagakos and Cummins [1974] N. Lagakos and H. Z. Cummins, Preliminary observation of a central peak in the light-scattering spectrum of $kh_{2}po_{4}$, Physical Review B 10, 1063 (1974).
  * Bozinis and Hurrell [1976] D. G. Bozinis and J. P. Hurrell, Optical modes and dielectric properties of ferroelectric orthorhombic $\mathrm{KNbO_{3}}$, Physical Review B 13, 3109 (1976).
  * Benyuan  _et al._ [1981] G. Benyuan, M. Copic, and H. Z. Cummins, Soft acoustic mode in ferroelastic $\mathrm{BiVO_{4}}$, Physical Review B 24, 4098 (1981).
  * Avakyants  _et al._ [2004] L. P. Avakyants, A. V. Chervyakov, V. S. Gorelik, and P. P. Sverbil’, Inelastic light scattering near the ferroelectric phase-transition point in bismuth vanadate crystals, Journal of Russian Laser Research 25, 535 (2004).
  * Pellicer-Porres  _et al._ [2018] J. Pellicer-Porres, D. Vázquez-Socorro, S. López-Moreno, A. Muñoz, P. Rodríguez-Hernández, D. Martínez-García, S. N. Achary, A. J. E. Rettie, and C. B. Mullins, Phase transition systematics in $\mathrm{BiVO_{4}}$ by means of high-pressure–high-temperature raman experiments, Physical Review B 98, 214109 (2018).
  * Takagi [1983] Y. Takagi, Coupled modes analysis of raman spectra: Application to the case of a $\mathrm{KH2PO4}$ crystal in the paraelectric phase, Ferroelectrics 46, 245 (1983).
  * Lowndes  _et al._ [1974] R. P. Lowndes, N. E. Tornberg, and R. C. Leung, Ferroelectric mode and molecular structure in the hydrogen-bonded ferroelectric arsenates and their deuterated isomorphs, Physical Review B 10, 911 (1974).
  * Mazzacurati  _et al._ [1976] V. Mazzacurati, G. Signorelli, and M. Sampoli, Indirect coupling of the soft optic phonon in $\mathrm{KH_{2}PO_{4}}$, Journal of Raman Spectroscopy 4, 339 (1976).
  * Scarparo  _et al._ [1978] M. A. F. Scarparo, R. S. Katiyar, R. Srivastava, and S. P. S. Porto, Evidence of non-linear behaviour of soft modes in crystals with kdp structure, Physica Status Solidi (b) 90, 543 (1978).
  * Kaminow and Damen [1968] I. P. Kaminow and T. C. Damen, Temperature dependence of the ferroelectric mode in KH2PO4, Physical Review Letters 20, 1105 (1968).
  * Burns and Scott [1973] G. Burns and B. A. Scott, Lattice modes in ferroelectric perovskites: PbTiO3, Physical Review B 7, 3088 (1973).
  * Chaves  _et al._ [1974b] A. Chaves, R. S. Katiyar, and S. P. S. Porto, Coupled modes with a1 symmetry in tetragonal $\mathrm{BaTiO_{3}}$, Physical Review B 10, 3522 (1974b).
  * Liu  _et al._ [2000] M. S. Liu, L. A. Bursill, S. Prawer, and R. Beserman, Temperature dependence of the first-order raman phonon line of diamond, Physical Review B 61, 3391 (2000).
  * Klemens [1966] P. G. Klemens, Anharmonic decay of optical phonons, Physical Review 148, 845 (1966).
  * Zhu  _et al._ [2005] K.-R. Zhu, M.-S. Zhang, Q. Chen, and Z. Yin, Size and phonon-confinement effects on low-frequency raman mode of anatase $\mathrm{TiO_{2}}$ nanocrystal, Physics Letters A 340, 220 (2005).
  * Spanier  _et al._ [2001] J. E. Spanier, R. D. Robinson, F. Zhang, S.-W. Chan, and I. P. Herman, Size-dependent properties of $\mathrm{CeO_{2-y}}$ nanoparticles as studied by raman scattering, Physical Review B 64, 245407 (2001).
  * Wang  _et al._ [2007] D. Wang, B. Chen, and J. Zhao, Lattice vibration fundamentals of nanocrystalline anatase: Temperature-dependent study using micro-raman scattering spectroscopy, Journal of Applied Physics 101, 113501 (2007).
  * Menéndez and Cardona [1984] J. Menéndez and M. Cardona, Temperature dependence of the first-order raman scattering by phonons in si, ge, and $\alpha$-sn : Anharmonic effects, Physical Review B 29, 2051 (1984).
  * Meier [2005b] R. J. Meier, On art and science in curve-fitting vibrational spectra, Vibrational Spectroscopy 39, 266 (2005b).
  * Zhou  _et al._ [1992] P. Zhou, K.-A. Wang, A. M. Rao, P. C. Eklund, G. Dresselhaus, and M. S. Dresselhaus, Raman-scattering studies of homogeneous $\mathrm{K_{3}C_{6}0}$ films, Physical Review B 45, 10838 (1992).


# Mamba-PTQ: Outlier Channels in Recurrent Large Language Models

Alessandro Pierro  Steven Abreu

###### Abstract

Modern recurrent layers are emerging as a promising path toward edge
deployment of foundation models, especially in the context of large language
models (LLMs). Compressing the whole input sequence in a finite-dimensional
representation enables recurrent layers to model long-range dependencies while
maintaining a constant inference cost for each token and a fixed memory
requirement. However, the practical deployment of LLMs in resource-limited
environments often requires further model compression, such as quantization
and pruning. While these techniques are well-established for attention-based
models, their effects on recurrent layers remain underexplored.

In this preliminary work, we focus on post-training quantization for recurrent
LLMs and show that Mamba models exhibit the same pattern of outlier channels
observed in attention-based LLMs. We show that the reason for difficulty of
quantizing SSMs is caused by activation outliers, similar to those observed in
transformer-based LLMs. We report baseline results for post-training
quantization of Mamba that do not take into account the activation outliers
and suggest first steps for outlier-aware quantization.

Recurrent Neural Networks, State Space Models, Language Modeling, Quantization

##  1 Introduction

Attention-based models, also known as Transformers (Vaswani et al., 2023),
constitute the current state-of-the-art backbone for large language models
(LLMs) (Brown et al., 2020). However, their powerful modeling capabilities
come with significant computational requirements, resulting in high inference
costs and limiting the deployment on edge and low-power devices. Novel
recurrent neural network (RNN) architectures, informed mainly by recent work
on state space models (SSMs) (Gu et al., 2020, 2022), are now emerging as
promising alternatives for sequence modeling tasks, either in isolation (Poli
et al., 2023; Gu & Dao, 2023; Peng et al., 2023) or as hybrid models
interleaving recurrent and attention blocks (De et al., 2024; Lieber et al.,
2024; Botev et al., 2024). In particular, RNNs compress the input sequence
into a finite-dimensional representation, decoupling the computational and
memory cost of each token’s forward pass from the sequence’s length. Hence,
they provide better scalability to long context scenarios than vanilla self-
attention, which scales quadratically with sequence length.

However, similarly to Transformers, deploying recurrence-based LLMs at scale
or in resource-constrained environments requires advanced model optimization
techniques, such as quantization, pruning, and knowledge distillation. While
applying these techniques starts to be well understood in the context of
attention-based LLMs, model optimization for recurrent and hybrid
architectures remains an important yet underexplored topic. In this paper, we
focus on quantization and analyze its impact on Mamba (Gu & Dao, 2023) model
family, drawing connections to previous work on quantized LLMs.

##  2 Quantization and outlier channels in LLMs

Figure 1: Architecture diagram of the Mamba block and details on the absolute
maximum activation (on the y-axis) across channels (x-axis), measured on a
subset of WikiText-2 (Merity et al., 2016) for Mamba-130m. Shaded regions
account for six standard deviations.

Quantization is a compression technique that reduces the numerical precision
of a model’s weights and activations to integer datatypes in order to
facilitate inference (Jacob et al., 2017). We adopt symmetric per-tensor
quantization: given a tensor $\mathbf{x}$ and a bit precision $n$, its
quantized representation is computed as:

$\bar{\mathbf{x}}_{n}=\left\lfloor\frac{(2^{n-1}-1)\mathbf{x}}{\max|\mathbf{x}|}\right\rceil=\left\lfloor
s_{x}\mathbf{x}\right\rceil$ (1)

where the quantization scale $s_{x}$ is a scalar. The benefits of quantization
for inference efficiency are twofold. Firstly, weight quantization reduces the
memory footprint of the model, which is especially beneficial in the memory-
bound regime of autoregressive generation. Secondly, when both weights and
activations are quantized, matrix multiplications can be offloaded to the
integer processing units, which typically offer higher throughput and energy
efficiency than floating point units.

Most state-of-the-art techniques for quantizing LLMs are based on the
empirical observation of outlier channels (Bondarenko et al., 2021), a small
percentage of model dimensions with a dynamic range that is consistently
larger than the rest. This phenomenon complicates activation quantization
since the large abs max values from the outlier channels deteriorate the
effective bit precision of the remaining channels. A possible solution would
be maintaining a different quantization scale for each channel, which is not
hardware-friendly on current GPU architectures (Xiao et al., 2024). Various
strategies have been proposed to circumvent this issue. For instance, some
methods treat outlier channels separately, either by maintaining them in
floating point format (Dettmers et al., 2022) or by representing them with two
integer channels each (Zhang et al., 2024). Other approaches modify the
transformer architecture to prevent the emergence of outliers (Bondarenko et
al., 2023), while some partially shift the quantization difficulty to the
weights, thereby mitigating the impact of outliers (Xiao et al., 2024).

We make the first steps towards post-training quantization for recurrent LLMs,
focusing on the Mamba (Gu & Dao, 2023) model family. We analyse the activation
patterns of Mamba to assess the presence of outliers, which we define as those
channels having an absolute maximum activation beyond six standard deviations
from the layer mean, following prior practice (Bondarenko et al., 2021).
Figure 1 reports the pre-activations of the linear block of a layer from
Mamba-130m (similar results were observed for the other model sizes), measured
running the model on a subset of WikiText-2 (Merity et al., 2016). We observe
distinct outlier patterns. The pre-activations of the three largest linear
layers (in, x, and out), consistently with what was observed for attention-
based LLMs, show outliers accounting for less than $1\%$ of channels. However,
while the outliers of the first linear block are mostly consistent across
layers, the remaining two blocks exhibit no regular behavior. The linear layer
projecting the SSM’s time steps (dt) shows almost no outliers. Similarly to
(Dettmers et al., 2022), we further assess the importance of the outlier
channels for the model’s predictions by evaluating the impact of zeroing out
the outliers on downstream accuracy. For Mamba-130m and Mamba-2.8b, we observe
a drop in average accuracy of $12.61\%$ and $17.49\%$, respectively,
suggesting that these channels play a significant role in the model dynamics.
Extended results are available in the Appendix in Table 2.

##  3 Method

###  3.1 Mamba model

Existing state space models are described by the following dynamics:

$\displaystyle x^{k}$ $\displaystyle=Ax^{k-1}+Bu^{k}$ (2) $\displaystyle
y^{k}$ $\displaystyle=Cx^{k}+Du^{k}$ (3)

where $A$ is the recurrent matrix, $B$ is the input matrix, $C$ is the output
matrix, and $D$ is the residual matrix from the input to the output. State
space models of this form must treat every input token equally, as the input
matrix $B$ is fixed, such that the SSM cannot focus on or ignore specific
tokens. This is a major shortcoming compared to transformer architectures,
whose attention mechanism allows for such interactions and thus limits the
performance of SSMs, especially on language tasks.

The key innovation of Mamba over previous SSMs is its ability to perform such
content-based reasoning. By letting Mamba’s parameters depend on the input,
the model effectively gains the ability to filter out irrelevant information
so that the relevant context can be compressed more efficiently into the
hidden state. Specifically, the parameters for Mamba’s SSM block are obtained
by:

$\displaystyle B_{t},\Delta_{t},C_{t}$ $\displaystyle=W^{proj}u_{t}$ (4)
$\displaystyle\overline{\Delta_{t}}$
$\displaystyle=\sigma^{+}(W^{dt}\Delta_{t})$ (5)
$\displaystyle\overline{A_{t}}$
$\displaystyle=\exp\left(-\exp\left(A^{log}\overline{\Delta_{t}}\right)\right)$
(6) $\displaystyle\overline{B_{t}}$ $\displaystyle=\overline{\Delta_{t}}\odot
B_{t}$ (7)

where $u_{t}$ is the input to the SSM block, $W^{proj}$, $A^{log}$ and
$W^{dt}$ are time-invariant weight matrices, $\sigma^{+}$ denotes the softplus
function and $\odot$ denotes element-wise multiplication. The weight matrices
$B_{t}$ and $C_{t}$ are thus directly dependent on the input $u_{t}$, whereas
the recurrent matrix $A_{t}$ is dependent on the input $u_{t}$ only through
the input-dependent timescale parameter $\Delta_{t}$. The hidden state $h_{t}$
and output $y_{t}$ of the SSM block is then computed as:

$\displaystyle h_{t}$ $\displaystyle=\overline{A_{t}}\odot
h_{t-1}+\overline{B_{t}}\odot u_{t}$ (8) $\displaystyle y_{t}$
$\displaystyle=C_{t}h_{t}+D_{t}\odot u_{t}$ (9)

As shown in Figure 1, each layer in the Mamba architecture also includes
additional gating, nonlinearities, normalization, causal convolution, and
linear blocks.

###  3.2 Baseline quantization

In order to quantize Mamba, we distinguish between Mamba’s pre-trained weights
and its activations. Importantly, due to the input-dependent parameterization,
we consider only input-independent parameters as weights, such as $A^{log}$,
while we consider input-dependent parameters like $\overline{A_{t}}$ as
activations.

We adopt symmetric, per-tensor quantization for weights and activations as
described in section 2, using the absolute maximum (absmax) of the tensor for
calibration.

For our experiments using naive quantization on the activations, we quantize
the output from all linear layers (including the matrices $B_{t}$,
$\Delta_{t}$, $C_{t}$ from Equation 4), but we do not quantize the effective
weight matrices $\overline{A_{t}},\overline{\Delta_{t}},\overline{B_{t}}$. We
further do not quantize the output from the SSM block $y_{t}$ but only
quantize the output from the downstream out projection linear layer.

We use standard notation to denote quantization with $n$-bit integers for
weights as W$n$ and quantization with $n$-bit integers for activations as
A$n$. For example, 8-bit weight quantization and 4-bit activation quantization
is denoted by W8A8.

###  3.3 Outlier-aware quantization (e.g., SmoothQuant)

The naive absmax quantization is sensitive to outliers. A large value in the
tensor $\mathbf{x}$ will yield a small scale
$s_{x}=\frac{2^{n-1}-1}{\max|\mathbf{x}|}$, thus leading to larger rounding
errors for the same $n$-bit quantization precision. As discussed in section 2,
outliers (particularly in activations) are the subject of research in LLM
quantization.

Most notably, the SmoothQuant method proposed by Xiao _et al._ (Xiao et al.,
2024) exploits the fact that outliers exist in activations but not in the
weights. SmoothQuant smooths the activation outliers by partially taking them
into the preceding weights. Because activation outliers typically persist in
the same activation channels, a weight matrix with per-channel quantization
can absorb part of the quantization difficulty from the subsequent
activations. As such, SmoothQuant introduces a per-channel smoothing factor
$s\in\mathbb{R}^{C_{i}}$ where $C_{i}$ is the dimension of the activations $X$
and, equivalently, the number of output channels of the weight matrix $W$.
This smpoothing factor is used to scale the weights and activations:

$\displaystyle(\mathbf{X}\text{diag}(s)^{-1})\cdot(\text{diag}(s)\textbf{W})=\hat{\mathbf{X}}\hat{\mathbf{W}}$
(10)

The aim is to choose a smoothing factor $s$ so that
$\hat{\mathbf{X}}=\mathbf{X}\text{diag}(s)^{-1}$ is easy to quantize. However,
simply choosing $s_{j}=\max(|\mathbf{X}_{j}|)$ where $j=1,\ldots,C_{i}$ to
minimize the difficulty in quantization activations, will push all these
difficulties into the weights. On the other hand, we can choose
$s_{j}=1/\max(|\mathbf{W}_{j}|)$ to move all the quantization difficulty from
the weights into the activations. The authors propose a new hyperparameter,
the migration strength $\alpha$, to control how much difficulty we want to
migrate from activations to weights, using the equation:

$\displaystyle
s_{j}=\frac{\max(|\mathbf{X}_{j}|)^{\alpha}}{\max(|\mathbf{W}_{j}|)^{1-\alpha}}$
(11)

where a smaller $\alpha$ will leave more difficulty with the activations, and
a larger $\alpha$ will migrate more difficulty to the weights. The authors
suggest to use a default value of $\alpha=0.5$ and a larger $\alpha$ for
models where activation outliers are more significant such that more
quantization difficulty is moved into the weights.

Table 1: One-shot accuracy on downstream tasks for Mamba-1.4b across different quantization configurations. | LAMBADA | HellaSwag | PIQA | WinoGrande | RTE | COPA
---|---|---|---|---|---|---
Baseline | $64.95\%$ | $59.11\%$ | $74.16\%$ | $61.4\%$ | $48.01\%$ | $79\%$
W8 (mlp) | $64.43\%$ | $44.91\%$ | $74.32\%$ | $60.06\%$ | $48.01\%$ | $77.00\%$
W8 (all) | $63.01\%$ | $44.71\%$ | $73.07\%$ | $60.06\%$ | $51.62\%$ | $76.00\%$
W4 (mlp) | $0.02\%$ | $25.70\%$ | $52.29\%$ | $51.54\%$ | $52.35\%$ | $56.00\%$
W4 (all) | $0.00\%$ | $25.72\%$ | $52.39\%$ | $50.99\%$ | $55.23\%$ | $64.00\%$
W8A8 (mlp) | $63.11\%$ | $44.42\%$ | $73.01\%$ | $60.06\%$ | $51.62\%$ | $76\%$
W8A8 (all) | $55.35\%$ | $43.84\%$ | $70.24\%$ | $54.3\%$ | $52.71\%$ | $75\%$

##  4 Experiments

Figure 2: Average one-shot accuracy on downtream tasks across model sizes for
Mamba with different quantization configurations. The accuracy is averaged
over all tasks shown in Table 1.

###  4.1 Experimental setup

We assess the impact of different quantization configurations on the zero-shot
accuracy of six downstream tasks: LAMBADA (Paperno et al., 2016), HellaSwag
(Zellers et al., 2019), PIQA (Bisk et al., 2019), WinoGrande (Sakaguchi et
al., 2019), RTE (Wang et al., 2019), and COPA (Roemmele et al., ). We
explicitly neglect perplexity benchmarking, since prior work noted how it may
not be informative of the actual task performance degradation (Sun et al.,
2021).

We run three different experimental conditions:

  1. 1.

To assess the importance of outlier channels, we analyze the impact of
removing outlier channels on downstream task accuracy.

  2. 2.

We then analyze the effect of naive quantization of only the pre-trained
weights on downstream task accuracy.

  3. 3.

Finally, we analyze the effect of quantization on the pre-trained weights, as
well as the activations, without accounting for activation outliers.

Full results for the effect of Experiment 1 for removing outlier channels are
found in Table 2 in the Appendix. We present an overview of the findings from
Experiment 2 and 3 in Table 1 on the Mamba-1.4b model, while results for all
other model sizes are presented in Table 3 in the Appendix.

##  5 Discussion

In this preliminary work, we make the first steps towards post-training
quantization of Mamba, in order to inform future edge deployments of recurrent
LLMs based on selective state space models such as Mamba. We have shown that
the difficulty of quantizing Mamba is caused by activation outliers, similar
to those observed in transformer-based LLMs. We presented baseline results for
post-training quantization of Mamba that does not take into account the
activation outliers and a first proposal for outlier-aware quantization of
Mamba.

###  5.1 Future work

As this area is under rapid development, several opportunities exist to extend
this work. Firstly, a similar analysis could be performed on other recurrent
LLMs, such as the RWKV family (Peng et al., 2023), the novel Mamba-2
architecture (Dao & Gu, 2024), or hybrid models such as Griffin (De et al.,
2024) and RecurrentGemma (Botev et al., 2024). Secondly, additional work
should be done to convert the SSM dynamics fully to integer operations, as
previously demonstrated by (Blouw et al., 2021), and explore the use of
quantized activations. Lastly, it will be interesting to see how quantized
recurrent LLMs perform at the edge in energy-constrained scenarios for real-
time multimodal processing (Shrestha et al., 2024), as the specific of the
hardware architecture could provide additional guidance on model compression
requirements.

## Acknowledgements

We thank Jonathan Timcheck, Philipp Stratmann, and Sumit Shrestha for helpful
comments and discussions.

## References

  * Bisk et al. (2019) Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y.  PIQA: Reasoning about Physical Commonsense in Natural Language, November 2019.  URL http://arxiv.org/abs/1911.11641.  arXiv:1911.11641 [cs].
  * Blouw et al. (2021) Blouw, P., Malik, G., Morcos, B., Voelker, A., and Eliasmith, C.  Hardware aware training for efficient keyword spotting on general purpose and specialized hardware.  In _Research Symposium on Tiny Machine Learning_ , 2021.  URL https://openreview.net/forum?id=iqcQPOrPcb7.
  * Bondarenko et al. (2021) Bondarenko, Y., Nagel, M., and Blankevoort, T.  Understanding and Overcoming the Challenges of Efficient Transformer Quantization, September 2021.  URL http://arxiv.org/abs/2109.12948.  arXiv:2109.12948 [cs].
  * Bondarenko et al. (2023) Bondarenko, Y., Nagel, M., and Blankevoort, T.  Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing, November 2023.  URL http://arxiv.org/abs/2306.12929.  arXiv:2306.12929 [cs].
  * Botev et al. (2024) Botev, A., De, S., Smith, S. L., Fernando, A., Muraru, G.-C., Haroun, R., Berrada, L., Pascanu, R., Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Girgin, S., Bachem, O., Andreev, A., Kenealy, K., Mesnard, T., Hardin, C., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., Joulin, A., Fiedel, N., Senter, E., Chen, Y., Srinivasan, S., Desjardins, G., Budden, D., Doucet, A., Vikram, S., Paszke, A., Gale, T., Borgeaud, S., Chen, C., Brock, A., Paterson, A., Brennan, J., Risdal, M., Gundluru, R., Devanathan, N., Mooney, P., Chauhan, N., Culliton, P., Martins, L. G., Bandy, E., Huntsperger, D., Cameron, G., Zucker, A., Warkentin, T., Peran, L., Giang, M., Ghahramani, Z., Farabet, C., Kavukcuoglu, K., Hassabis, D., Hadsell, R., Teh, Y. W., and de Frietas, N.  RecurrentGemma: Moving Past Transformers for Efficient Open Language Models, April 2024.  URL http://arxiv.org/abs/2404.07839.  arXiv:2404.07839 [cs].
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language Models are Few-Shot Learners, July 2020.  URL http://arxiv.org/abs/2005.14165.  arXiv:2005.14165 [cs].
  * Dao & Gu (2024) Dao, T. and Gu, A.  Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024.
  * De et al. (2024) De, S., Smith, S. L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., Desjardins, G., Doucet, A., Budden, D., Teh, Y. W., Pascanu, R., De Freitas, N., and Gulcehre, C.  Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models, February 2024.  URL http://arxiv.org/abs/2402.19427.  arXiv:2402.19427 [cs].
  * Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.  LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, November 2022.  URL http://arxiv.org/abs/2208.07339.  arXiv:2208.07339 [cs].
  * Gu & Dao (2023) Gu, A. and Dao, T.  Mamba: Linear-Time Sequence Modeling with Selective State Spaces, December 2023.  URL http://arxiv.org/abs/2312.00752.  arXiv:2312.00752 [cs].
  * Gu et al. (2020) Gu, A., Dao, T., Ermon, S., Rudra, A., and Re, C.  HiPPO: Recurrent Memory with Optimal Polynomial Projections, October 2020.  URL http://arxiv.org/abs/2008.07669.  arXiv:2008.07669 [cs, stat].
  * Gu et al. (2022) Gu, A., Goel, K., and Ré, C.  Efficiently Modeling Long Sequences with Structured State Spaces, August 2022.  URL http://arxiv.org/abs/2111.00396.  arXiv:2111.00396 [cs].
  * Jacob et al. (2017) Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D.  Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, December 2017.  URL http://arxiv.org/abs/1712.05877.  arXiv:1712.05877 [cs, stat].
  * Lieber et al. (2024) Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., Abend, O., Alon, R., Asida, T., Bergman, A., Glozman, R., Gokhman, M., Manevich, A., Ratner, N., Rozen, N., Shwartz, E., Zusman, M., and Shoham, Y.  Jamba: A Hybrid Transformer-Mamba Language Model, March 2024.  URL http://arxiv.org/abs/2403.19887.  arXiv:2403.19887 [cs].
  * Merity et al. (2016) Merity, S., Xiong, C., Bradbury, J., and Socher, R.  Pointer Sentinel Mixture Models, September 2016.  URL http://arxiv.org/abs/1609.07843.  arXiv:1609.07843 [cs].
  * Paperno et al. (2016) Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R.  The LAMBADA dataset: Word prediction requiring a broad discourse context, June 2016.  URL http://arxiv.org/abs/1606.06031.  arXiv:1606.06031 [cs].
  * Peng et al. (2023) Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., GV, K. K., He, X., Hou, H., Lin, J., Kazienko, P., Kocon, J., Kong, J., Koptyra, B., Lau, H., Mantri, K. S. I., Mom, F., Saito, A., Song, G., Tang, X., Wang, B., Wind, J. S., Wozniak, S., Zhang, R., Zhang, Z., Zhao, Q., Zhou, P., Zhou, Q., Zhu, J., and Zhu, R.-J.  RWKV: Reinventing RNNs for the Transformer Era, December 2023.  URL http://arxiv.org/abs/2305.13048.  arXiv:2305.13048 [cs].
  * Poli et al. (2023) Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C.  Hyena Hierarchy: Towards Larger Convolutional Language Models, April 2023.  URL http://arxiv.org/abs/2302.10866.  arXiv:2302.10866 [cs].
  * (19) Roemmele, M., Bejan, C. A., and Gordon, A. S.  Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning.
  * Sakaguchi et al. (2019) Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.  WinoGrande: An Adversarial Winograd Schema Challenge at Scale, November 2019.  URL http://arxiv.org/abs/1907.10641.  arXiv:1907.10641 [cs].
  * Shrestha et al. (2024) Shrestha, S. B., Timcheck, J., Frady, P., Campos-Macias, L., and Davies, M.  Efficient Video and Audio Processing with Loihi 2.  In _ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 13481–13485, April 2024.  doi: 10.1109/ICASSP48485.2024.10448003.  URL https://ieeexplore.ieee.org/abstract/document/10448003.
  * Sun et al. (2021) Sun, S., Krishna, K., Mattarella-Micke, A., and Iyyer, M.  Do long-range language models actually use long-range context?, 2021.
  * Vaswani et al. (2023) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention Is All You Need, August 2023.  URL http://arxiv.org/abs/1706.03762.  arXiv:1706.03762 [cs].
  * Wang et al. (2019) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, February 2019.  URL http://arxiv.org/abs/1804.07461.  arXiv:1804.07461 [cs].
  * Xiao et al. (2024) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.  SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, March 2024.  URL http://arxiv.org/abs/2211.10438.  arXiv:2211.10438 [cs].
  * Zellers et al. (2019) Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y.  HellaSwag: Can a Machine Really Finish Your Sentence?, May 2019.  URL http://arxiv.org/abs/1905.07830.  arXiv:1905.07830 [cs].
  * Zhang et al. (2024) Zhang, Y., Yang, F., Peng, S., Wang, F., and Pan, A.  FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization, February 2024.  URL http://arxiv.org/abs/2402.17985.  arXiv:2402.17985 [cs].

##  Appendix A Additional results

Herein we present all additional experimental results from the experiments
presented in this paper.

###  A.1 Impact of removing outlier channels

Table 2 shows the accuracy on all evaluated tasks for the Mamba-130m model and
Mamba-2.8B model, with different rows indicating the outlier removal specific
to particular layers, or across the entire model.

Table 2: Impact of removing outlier channels on downstream task accuracy. Model | LAMBADA | HellaSwag | PIQA | WinoGrande | RTE | COPA | Avg.
---|---|---|---|---|---|---|---
Mamba-130m |  |  |  |  |  |  |
Baseline | $44.25\%$ | $35.25\%$ | $64.47\%$ | $52\%$ | $54.87\%$ | $65\%$ | $52.64\%$
Linear(in) only | $0\%$ | $25.61\%$ | $53.75\%$ | $51.14\%$ | $52.71\%$ | $53\%$ | $39.37\%$
Linear(x) only | $26.88\%$ | $26.73\%$ | $58.49\%$ | $49.57\%$ | $55.6\%$ | $63\%$ | $46.71\%$
Linear(dt) only | $44.25\%$ | $30.80\%$ | $64.47\%$ | $52.09\%$ | $54.87\%$ | $65\%$ | $51.91\%$
Linear(out) only | $32.45\%$ | $30.42\%$ | $65.34\%$ | $53.35\%$ | $53.43\%$ | $70\%$ | $50.83\%$
All | $0\%$ | $25.83\%$ | $54.52\%$ | $53.12\%$ | $52.71\%$ | $54\%$ | $40.03\%$
Mamba-2.8B |  |  |  |  |  |  |
Baseline | $69.24\%$ | $66.16\%$ | $75,24\%$ | $63.46\%$ | $52.71\%$ | $84\%$ | $68.46\%$
All | $7.45\%$ | $49.43\%$ | $66.92\%$ | $58.25\%$ | $53.79\%$ | $70\%$ | $50.97\%$

###  A.2 Impact of quantization on downstream task accuracy

Table 3 shows the accuracy on all evaluated tasks for all Mamba models and all
quantization configurations.

Table 3: One-shot accuracy on downstream tasks for the Mamba model family across different quantization configurations. Model | LAMBADA | HellaSwag | PIQA | WinoGrande | RTE | COPA
---|---|---|---|---|---|---
Mamba-130m |  |  |  |  |  |
Baseline | $44.25\%$ | $35.25\%$ | $64.47\%$ | $52\%$ | $54.87\%$ | $65\%$
W8 (mlp) | $42.48\%$ | $30.71\%$ | $64.09\%$ | $52.88\%$ | $52.71\%$ | $63.00\%$
W8 (all) | $5.53\%$ | $28.03\%$ | $58.11\%$ | $50.59\%$ | $47.29\%$ | $64.00\%$
W4 (mlp) | $0.00\%$ | $25.80\%$ | $51.74\%$ | $50.75\%$ | $49.82\%$ | $57.00\%$
W4 (all) | $0.00\%$ | $25.34\%$ | $53.43\%$ | $50.36\%$ | $52.35\%$ | $53.00\%$
W8A8 (mlp) | $5.72\%$ | $28.09\%$ | $57.83\%$ | $51.3\%$ | $47.65\%$ | $65\%$
W8A8 (all) | $4.31\%$ | $27.72\%$ | $56.47\%$ | $51.07\%$ | $50.18\%$ | $65\%$
Mamba-370m |  |  |  |  |  |
Baseline | $55.62\%$ | $46.48\%$ | $69.48\%$ | $55.49\%$ | $53.07\%$ | $70\%$
W8 (mlp) | $54.86\%$ | $37.17\%$ | $69.04\%$ | $55.80\%$ | $53.79\%$ | $70.00\%$
W8 (all) | $16.61\%$ | $31.58\%$ | $61.37\%$ | $51.38\%$ | $49.10\%$ | $72.00\%$
W4 (mlp) | $0.00\%$ | $25.23\%$ | $53.81\%$ | $50.28\%$ | $52.35\%$ | $55.00\%$
W4 (all) | $0.00\%$ | $25.72\%$ | $53.10\%$ | $51.38\%$ | $52.71\%$ | $55.00\%$
W8A8 (mlp) | $16.61\%$ | $36.87\%$ | $61.53\%$ | $51.22\%$ | $48.74\%$ | $70\%$
W8A8 (all) | $10.29\%$ | $31.04\%$ | $58.76\%$ | $50.99\%$ | $45.49\%$ | $63\%$
Mamba-790m |  |  |  |  |  |
Baseline | $61.71\%$ | $55.07\%$ | $72.14\%$ | $55.96\%$ | $55.23\%$ | $72\%$
W8 (mlp) | $2.64\%$ | $25.30\%$ | $54.13\%$ | $50.83\%$ | $53.07\%$ | $52.00\%$
W8 (all) | $1.20\%$ | $25.84\%$ | $54.08\%$ | $51.07\%$ | $55.96\%$ | $53.00\%$
W4 (mlp) | $0.00\%$ | $25.93\%$ | $53.05\%$ | $48.22\%$ | $46.21\%$ | $60.00\%$
W4 (all) | $0.00\%$ | $25.29\%$ | $50.98\%$ | $47.67\%$ | $46.57\%$ | $56.00\%$
W8A8 (mlp) | $1.44\%$ | $25.37\%$ | $54.30\%$ | $50.2\%$ | $53.43\%$ | $53\%$
W8A8 (all) | $0.95\%$ | $25.77\%$ | $54.73\%$ | $50.99\%$ | $55.23\%$ | $57\%$
Mamba-1.4B |  |  |  |  |  |
Baseline | $64.95\%$ | $59.11\%$ | $74.16\%$ | $61.4\%$ | $48.01\%$ | $79\%$
W8 (mlp) | $64.43\%$ | $44.91\%$ | $74.32\%$ | $60.06\%$ | $48.01\%$ | $77.00\%$
W8 (all) | $63.01\%$ | $44.71\%$ | $73.07\%$ | $60.06\%$ | $51.62\%$ | $76.00\%$
W4 (mlp) | $0.02\%$ | $25.70\%$ | $52.29\%$ | $51.54\%$ | $52.35\%$ | $56.00\%$
W4 (all) | $0.00\%$ | $25.72\%$ | $52.39\%$ | $50.99\%$ | $55.23\%$ | $64.00\%$
W8A8 (mlp) | $63.11\%$ | $44.42\%$ | $73.01\%$ | $60.06\%$ | $51.62\%$ | $76\%$
W8A8 (all) | $55.35\%$ | $43.84\%$ | $70.24\%$ | $54.3\%$ | $52.71\%$ | $75\%$
Mamba-2.8B |  |  |  |  |  |
Baseline | $69.24\%$ | $66.16\%$ | $75.24\%$ | $63.46\%$ | $52.71\%$ | $84\%$
W8A8 (mlp) | $64.64\%$ | $48.74\%$ | $73.72\%$ | $64.33\%$ | $56.32\%$ | $78\%$
W8A8 (all) | $51.39\%$ | $47.64\%$ | $70.24\%$ | $57.62\%$ | $54.51\%$ | $79\%$