# INT-FP-QSim: Mixed Precision and Formats For Large Language Models and
Vision Transformers

Lakshmi Nair*, Mikhail Bernadskiy, Arulselvan Madhavan, Craig Chan, Ayon
Basumallik, Darius Bunandar *Email<EMAIL_ADDRESS>Lightmatter Inc.,
100 Summer Street,
Boston MA 02110

###### Abstract

The recent rise of large language models (LLMs) has resulted in increased
efforts towards running LLMs at reduced precision. Running LLMs at lower
precision supports resource constraints and furthers their democratization,
enabling users to run billion-parameter LLMs on their personal devices. To
supplement this ongoing effort, we propose INT-FP-QSim: an open-source
simulator that enables flexible evaluation of LLMs and vision transformers at
various numerical precisions and formats. INT-FP-QSim leverages existing open-
source repositories such as TensorRT, QPytorch and AIMET for a combined
simulator that supports various floating point and integer formats. With the
help of our simulator, we survey the impact of different numerical formats on
the performance of LLMs and vision transformers at 4-bit weights and 4-bit or
8-bit activations. We also compare recently proposed methods like Adaptive
Block Floating Point, SmoothQuant, GPTQ and RPTQ on the model performances. We
hope INT-FP-QSim will enable researchers to flexibly simulate models at
various precisions to support further research in quantization of LLMs and
vision transformers.

###### Index Terms:

Large language models, vision transformers, quantization, simulation

##  I Introduction

The recent rise in popularity of large language models (LLMs) has prompted
significant ongoing research into running LLMs at reduced precision, to
support resource constraints and democratize their access. Prior work has
looked at running the weights and activations of LLMs in 8-bit precision [1,
2]. The most recent techniques focus on enabling 4-bit integer quantization of
weights while retaining FP16 activations [3], and 4-bit to 3-bit quantization
of both weights and activations [4]. Beyond the recent advancements that have
focused specifically on LLMs, numerous techniques have been developed for
efficiently running convolutional models and smaller scale language models
such as BERT at low precision [5, 6, 7]. However, the latter techniques have
not been evaluated in the context of modern LLMs and vision transformers.
Given the recent interest in quantization of LLMs, this paper presents an
open-source simulator, INT-FP-QSim111https://github.com/lightmatter-ai/INT-FP-
QSim for flexible evaluation of LLMs and vision transformers at various
numerical formats. INT-FP-QSim combines resources from existing open-source
repositories, such as TensorRT [8], QPytorch [9] and AIMET [10, 11] for a
combined simulator that enables flexible investigations with a variety of
numerical data formats and precisions. With the help of the simulator, we
survey the impact of different numerical formats (floating point, integer,
mixed floating point and integer) on LLMs and vision transformers, including
the application of post-training quantization (PTQ) methods and quantization-
aware training (QAT) for improving model performance. In contrast to prior
work investigating different PTQ methods specifically for LLMs [12], we cover
a range of models (See Figure 1), and newer PTQ techniques such as Adaptive
Block Floating Point (ABFP) [6], SmoothQuant [1], GPTQ [3] and RPTQ [4]. In
all the cases, we investigate 4-bit weights (either integer or floating
point), along with 4-bit or 8-bit activations.

Figure 1: Performances of a range of models, relative to FP32, with 4-bit
integer weights and activations using Adaptive Block Floating Point (ABFP).

We summarize our contributions as follows: a) we present INT-FP-QSim, an open-
source flexible simulator for simulating different numerical formats; b) we
investigate the impact of using mixed data formats with 4-bit integer weights,
and 8-bit floating point activations; and c) we investigate the impact of
mixed and low precision on a variety of models and task domains ranging from
conventional LLMs like OPT, to transformers for computer vision and text-to-
image generation that are often less explored in this context. With INT-FP-
QSim and the findings of this work, we hope to recognize insights for future
work in this space.

Figure 2: High level overview of INT-FP-QSim. The user specifies the input
model and the quantizer functions. The matrix multiplication layers (e.g.,
conv, linear) within the model are replaced with versions of the layers that
have the quantizers attached. During forward pass, the quantizers are applied
to the inputs, weights and outputs of each layer to simulate quantization.

##  II Background and Related Work

We describe the different numerical formats along with the accuracy recovery
methods that we investigate in this work.

###  II-A Numerical Formats

In this work, we fix weights to 4-bit precision, and explore different
precisions for activations. Specifically, with weights kept either in 4-bit
integer or 4-bit floating point, we explore activation quantization in the
following formats: 1) 4-bit integer; 2) 8-bit integer; 3) 4-bit floating
point; and 4) 8-bit floating point [13]. For the 4-bit floating point formats,
we explore both 2-bit exponent (with 1-bit mantissa, i.e., E2M1) and 1-bit
exponent formats (E1M2). For 8-bit floating point, we use the format proposed
in [13], namely, 4-bit exponent and 3-bit mantissa (E4M3). We perform integer
quantization as follows, using nearest rounding:

$\displaystyle s=\frac{2^{b}-1}{\alpha}$ (1) $\displaystyle
x_{q}=Q(x;s,b)=clip(round(s\cdot x)),-2^{b}-1,2^{b}-1)$ (2)

We de-quantize the input as follows:

$\displaystyle\hat{x}=DQ(x_{q})=\frac{1}{s}x_{q}$ (3)

Performing quantization to lower precision often results in a drop in the
model accuracy, requiring the application of special techniques to recover
comparably to higher precision levels.

###  II-B Accuracy Recovery With Post-Training Quantization

We investigate the following methods for accuracy recovery from quantization:
a) static quantization with Mean Squared Error (MSE) calibration [7], b)
Adaptive Block Floating Point (ABFP) [6], c) SmoothQuant (SQ) [1], d) GPTQ
[3], and e) RPTQ [4]. We briefly describe each method here, and refer readers
to the corresponding papers for more details.

####  II-B1 Static Quantization with MSE Calibration

Calibration is the process of choosing the scales $\alpha$ (Eqn (1)) for
quantizing the model weights and activations [7]. The calibrated scales are
then used to perform quantization as shown in Eqn (2). In this work, we follow
the approach laid out in [7] and use per-channel max calibration for weights,
with mean-squared error (MSE) calibration for activations. MSE calibration
computes the scale (i.e., $\alpha$) that minimizes MSE between the quantized
and unquantized outputs of each layer, and has been shown to perform well for
minimizing quantization error [14].

####  II-B2 Adaptive Block Floating Point

Recent approaches like VS-Quant [5] and Adaptive Block Floating Point (ABFP)
[6] have looked at dynamically scaling vectors of length $n$, as a potential
solution to maintaining accuracy at low precision. In this work, we focus on
ABFP and evaluate its performance for different numerical formats. ABFP
computes scale factors as max over vectors of length $n$, wherein the scales
themselves are left in BF16. Given an $M\times N$ matrix of input activations
$X\in\mathbb{R}^{M\times N}$, let $x^{i}$ represent the $i$-th column of $X$.
Then scale factors are computed for vectors of length $n$ in each column:

$\displaystyle s^{i}_{j}=max(x^{i}_{j})\text{ where
}x^{i}_{j}\in\\{x^{i}_{1},x^{i}_{2},\dots,x^{i}_{\left\lceil\frac{|x^{i}|}{n}\right\rceil}\\}$
(4)

The scale $s^{i}_{j}$ is used to quantize the $n$-length vector $x^{i}_{j}$ of
column $x_{i}$. For weights, Eqn (4) is repeated except the scales are
computed over rows instead of columns. In contrast to computing the max value
over an entire matrix, the computation of max over vectors of length $n$,
helps minimize information loss due to quantization. We explore the
performance of ABFP for vectors of length $n\in\\{64,128\\}$. A second-level
quantization for the scale factors could be utilized to achieve further
compression [5]. However, we do not explore the second-level quantization in
this work in order to focus our analysis on the best-case performance of the
different models.

####  II-B3 SmoothQuant

SmoothQuant is a post-training quantization technique shown to work well with
8-bit integer weights and 8-bit integer activations in LLMs [1]. SmoothQuant
involves migrating the quantization difficulty from activations to weights,
based on a smoothing factor. It is motivated by the observation that LLM
activations are harder to quantize than weights owing to the presence of a
significant amount of outliers. In this work, we explore the application of
SmoothQuant (denoted as SQ) for 4-bit weights and 4-bit or 8-bit activations.
In our experiments, the migration or smoothing factor is set to 0.5 for all
layers within the model based on prior work [1].

####  II-B4 GPTQ

GPTQ is a post-training quantization approach that utilizes approximate
second-order information of the weights in order to compress it to lower
precision [3]. GPTQ achieves improved performance at 4-bit weights and FP16
activations, compared to FP16 baselines. The authors highlight that
activations do not pose a significant bottleneck, enabling the 175-billion
parameter OPT to be run on a single GPU.

####  II-B5 RPTQ

RPTQ [4] is a post-training quantization technique that rearranges the
channels in activations, and quantizes them in clusters, motivated by the
observation that in addition to presence of outliers, varying ranges across
channels makes activation quantization difficult. RPTQ demonstrates improved
results for 4 and 3-bit quantization of weights and activations.

###  II-C Accuracy Recovery With Training

Although post-training quantization methods achieve good results, significant
improvement is observed with the use of training or fine-tuning methods such
as quantization-aware training (QAT). While QAT can be computationally
intensive, especially for really large language models, we explore the
performance benefits of applying QAT in conjunction with ABFP for models up to
OPT 2.7B, in order to assess the degree of performance improvements obtained.
During fine-tuning, the forward pass is executed using ABFP, with a Piecewise-
linear (PWL) estimator in the backward pass:

$\displaystyle\frac{\partial Q(x,s,b)}{\partial x}=\mathbbm{1}_{\\{|x|\leq
s\\}}$ (5)

###  II-D Other Related Methods Not Explored In This Work

In addition to the methods discussed above for low precision execution of
LLMs, prior work called LLM-INT8() has looked at running specific channels
with a high incidence of outliers in higher precision (FP16), while running
inliers in INT8 [2]. In this work, we do not explore LLM-INT8(), instead
focusing on consistently running all the activations and weights in lower
precision (4-bit and 8-bit).

##  III Simulator Overview

In this section, we describe the components of our simulator, INT-FP-QSim,
where the high-level workflow is shown in Fig 2. To begin, the user provides
the input model and specifies the input and weight quantizer functions. These
functions couple quantization and de-quantization together, to perform
simulated quantization [7] of inputs and weights. The simulator optionally
supports quantization of outputs for accommodating alternate hardware
configurations. For instance, photonics hardware can involve output
quantization operations [15]. The quantizer functions perform the following:

$\displaystyle\hat{w}=f^{w}_{q}(w)=DQ(Q(w;s_{w},b_{w}))$ (6)
$\displaystyle\hat{x}=f^{x}_{q}(x)=DQ(Q(x;s_{x},b_{x}))$ (7) $\displaystyle
y=\hat{w}*\hat{x}$ (8)
$\displaystyle\hat{y}=f^{y}_{q}(y)=DQ(Q(y;s_{y},b_{y}))$ (9)

Once the quantizers and input model is specified, the simulator then replaces
each layer that performs matrix multiplications (i.e., linear, attention, and
convolution) within the model, with versions of the layers that have the
quantizers attached to them. The replaced versions of the layers apply the
quantizers to the inputs, weights and outputs during the forward pass to
simulate quantization. The data-types are maintained in float32, while the
data is quantized to lower precision and then de-quantized, as specified in
the quantizer configuration by the user. Hence, data is stored in lower
precision, while all the non-linear operations and matmuls are performed in
higher precision after de-quantization (similar to prior work [3, 13]). This
gives us an upper-bound on the performance of the models. Our future work will
seek to add support for different matmul precisions. The quantizer functions
($f^{w}_{q}(w),f^{x}_{q}(x),f^{y}_{q}(y)$) can be flexibly specified to
perform integer or floating point quantization. For instance, the quantizer
functions can be specified using TensorRT to perform integer quantization.
Alternatively, the quantizer function can be specified using QPytorch in order
to perform low precision floating point quantization.

Modified versions of the layers for the models discussed in this work (e.g.,
OPTAttention, CodegenAttention) are provided within INT-FP-QSim, to take the
quantizer functions that are specified and to perform the simulated
quantization operations (Eqns (6)-(9)) during forward pass.

For accuracy recovery, INT-FP-QSim provides support for QAT and static
calibration of activations and weights. INT-FP-QSim also includes support for
static max calibration of activations, where the max value is pre-calibrated
over a subset of data, and statically reused during quantization (in contrast
to dynamic max computation). Once the calibration is performed, the quantizer
functions use the calibrated scale to perform simulated quantization during
the forward pass of the model. Future iterations of our simulator will add
ABFP support. PTQ methods such as SmoothQuant, GPTQ and RPTQ are already
available as open-source repositories222SQ: https://github.com/mit-han-
lab/smoothquant; GPTQ: https://github.com/IST-DASLab/gptq; RPTQ:
https://github.com/hahnyuan/RPTQ4LLM.

##  IV Experiments and Results

In this section, we investigate the performance of a variety of models on
different numerical formats, including the accuracy recovery methods discussed
previously. We begin by carrying out a subset of the experiments on four OPT
models from OPT 125M - OPT 2.7B on the Wikitext2 dataset, and present more
extensive results on additional models for the promising approaches. Note that
we are focusing on the best-case outcomes for the models at different
precisions, with different techniques. Hence, in all our experiments we do not
explore the impact of low-precision output quantization, instead leaving the
output activations in FP16. Further, we denote the sequence lengths used for
the models in each case. Given the resource constraints for fine-tuning LLMs,
we use smaller sequence length of 256 for fine-tuning OPT 1.3B-2.7B.

Note on practical implementations of SQ and RPTQ: When applying SQ and RPTQ
from their open source repositories, we note that running SQ for the OPT 350M
raised errors owing to slight differences in the model architecture333See
Github issue reported here:
https://github.com/huggingface/transformers/issues/17653. Hence for the SQ
experiments, we report results for OPT 125M, OPT 1.3B and OPT 2.7B. Similarly,
the RPTQ implementation does not support OPT 350M and OPT 2.7B, hence RPTQ
results for these models are also exempt from the experiments.

###  IV-A 4-bit weights and 4-bit activations

In this section, we explore the performance of OPT models for 4-bit weights
and 4-bit activations in integer and floating point formats. Specifically, a)
we compare static MSE calibration vs. ABFP for 4-bit integer weights and 4-bit
integer activations; b) we compare performance of INT4 weights and activations
to FP4 weights and activations (in E2M1 and E1M2 formats). Since floating
point formats have recently garnered interest [13] we hope to investigate the
applicability of 4-bit floating point formats here; c) we look at performance
of ABFP, ABFP-QAT (ABFP along with QAT) and ABFP-SQ (ABFP with SmoothQuant)
for the different formats.

####  IV-A1 Static MSE calibration vs. ABFP

Table I shows the results comparing Static MSE calibration and ABFP for 4-bit
integer weights and 4-bit integer activations (W4A4) for OPT 125M and OPT
350M. Given the low bit precisions for the two models, MSE calibration
significantly under-performs ABFP, and does not yield a useable perplexity
(PPL). This could be correlated to the nature of outliers within these models,
wherein the MSE values would have to clip most outliers to be effective, even
if the outliers themselves could be critical to model performance. In
contrast, ABFP enables the outliers to be represented, while at the same time
minimizes information loss due to them by quantizing only vectors of length
$n$.

####  IV-A2 4-bit integer vs. floating point for weights and activations

Table II shows results comparing 4-bit integer weights and activations to
4-bit floating point weights and activations in E2M1 (2-bit Exponent, 1-bit
Mantissa) and E1M2 formats. Both formats use ABFP for $n=64$. We see that the
4-bit floating point formats do not offer any significant advantages over
4-bit integers (with the exception of OPT 350M). In particular, the
performances of W4A4 and E1M2 seem to be quite similar, while E1M2 under-
performs the two. The results also show that ABFP is a viable method that can
be applied with either integer or floating point formats.

In Figure 3, we denote the impact of varying the vector lengths $n$ from 64 to
128. We observe that larger vector lengths can impact model performance, but
the differences between model PPL from $n=64$ to $n=128$ seem to reduce as
model size increases to 1.3B and 2.7B indicating that larger vector lengths
could possibly be viable for larger models.

TABLE I: Perplexity (PPL) values for W4A4 configuration using static MSE calibration and ABFP (n=64). Lower PPL is better. Model (Seq len) | FP32 | MSE | ABFP
---|---|---|---
OPT 125M (1024) | 25.94 | 1130 | 33.14
OPT 350M (1024) | 16.84 | 927 | 32.28
TABLE II: Comparison of 4-bit integer and 4-bit floating point formats with ABFP (n=64). |  |  | With ABFP |
---|---|---|---|---
Model (Seq len) | FP32 | W4A4 | E2M1 | E1M2
OPT 125M (1024) | 25.94 | 33.14 | 37.59 | 33.12
OPT 350M (1024) | 16.84 | 32.28 | 28.55 | 32.41
OPT 1.3B (256) | 14.33 | 19.97 | 51.82 | 20.06
OPT 2.7B (256) | 13.35 | 16.39 | 32.77 | 16.41

####  IV-A3 Accuracy recovery for 4-bit integer weights and activations

For the training experiments in this section, the backward pass for the
function is defined as given in Eqn (5). The results are shown in Table III.
We see that using ABPF-QAT and ABFP-SQ can both improve performance over ABFP
alone. In particular, ABFP-QAT helps achieve PPL values that are close to the
baseline model. However, QAT can be highly resource intensive and further
investigations into improved fine-tuning methods could be useful. To support
the training experiments in our work, we use sequence lengths of 256 for the
larger models (OPT 1.3B and OPT 2.7B). For larger models, we see that using
ABFP along with SQ can achieve performances that are closer to QAT, indicating
that the combination of ABFP and SQ could be a useful technique.

We further assess the impact of applying QAT with ABFP for different vector
lengths in Figure 4. In all cases, fine-tuning with QAT helps improve model
performance compared to using vanilla ABFP. Additionally, we see that for the
different models, applying QAT at larger vector lengths ($n=128$) can achieve
performances that are close to smaller vector lengths.

Figure 3: Performance of the models with E1M2 weights and activations for different vector lengths $(n=64,128)$. TABLE III: Accuracy recovery methods on W4A4 representation to augment ABFP for $n=64$. Model (Seq len) | FP32 | ABFP | ABFP-QAT | ABFP-SQ
---|---|---|---|---
OPT 125M (1024) | 25.94 | 33.14 | 22.38 | 32.77
OPT 350M (1024) | 16.84 | 32.28 | 22.17 | -
OPT 1.3B (256) | 14.33 | 19.97 | 18.97 | 19.57
OPT 2.7B (256) | 13.35 | 16.39 | 15.22 | 15.78
Figure 4: Performance of the models with ABFP and QAT for different vector
lengths $(n=64,128)$ in the W4A4 format.

###  IV-B 4-bit weights and 8-bit activations

In this section, we explore the performance of OPT models for 4-bit integer
weights and 8-bit activations in the integer and floating point formats.
Specifically, a) we compare static MSE calibration vs. ABFP for 4-bit integer
weights and 8-bit integer activations; b) we compare performance of 8-bit
integer activations to FP8 activations (in E4M3 format) with 4-bit integer
weights in each case. We also compare the results obtained here to GPTQ
baseline; c) we look at performance of ABFP, ABFP-QAT (ABFP along with QAT)
and ABFP-SQ (ABFP with SmoothQuant) for the different formats.

####  IV-B1 Static MSE calibration vs. ABFP

Table IV shows the results comparing Static MSE calibration and ABFP for 4-bit
integer weights and 8-bit integer activations (W4A8) for OPT 125M - OPT 2.7B.
Compared to 4-bit activations, MSE calibration yields a usable PPL with 8-bit
activations. Using 8-bit activations enables a wider representation of the
outliers, and possibly enables MSE calibration to perform better. However,
ABFP outperforms MSE static calibration for all the models, achieving close to
the baseline model PPL values.

TABLE IV: W4A8 configuration using static MSE calibration and ABFP (n=64). Model (Seq len) | FP32 | MSE | ABFP
---|---|---|---
OPT 125M (1024) | 25.94 | 108.98 | 29.21
OPT 350M (1024) | 16.84 | 29.81 | 21.67
OPT 1.3B (256) | 14.33 | 36.8 | 15.54
OPT 2.7B (256) | 13.35 | 38.11 | 14.34

####  IV-B2 8-bit integer vs floating point activations

Table V shows results of using 4-bit integer weights with 8-bit floating point
activations (E4M3 format) with ABFP. We also show results of combining ABFP
with SmoothQuant and compare it to the GPTQ results (that use FP16 activations
and 4-bit weights). We see that using the E4M3 format for activations with
ABFP, can help achieve model PPL close to float32. Combining ABFP with
SmoothQuant can further improve the model PPL, achieving significantly
improved results for the OPT models (particularly for OPT 1.3B and OPT 2.7B),
compared to the corresponding GPTQ results. However, as shown in Table VI,
using E4M3 representation for input activations does not result in any
significant improvements over using 8-bit integers.

TABLE V: Comparison of E4M3 activations and INT4 weights with ABFP ($n=64$) to GPTQ that uses FP16 activations and INT4 weights. |  | W4-AE4M3 |  | W4A16
---|---|---|---|---
Model (Seq len) | FP32 | ABFP | ABFP-SQ | GPTQ
OPT 125M (1024) | 25.94 | 29.99 | 29.63 | 30.95
OPT 350M (1024) | 16.84 | 21.99 | - | 19.87
OPT 1.3B (1024) | 11.78 | 13.04 | 12.54 | 15.13
OPT 2.7B (1024) | 11 | 11.87 | 11.44 | 14.16
TABLE VI: Comparison of E4M3 activations vs. 8-bit integer activations with ABFP ($n=64$) and ABFP with SmoothQuant. | W4-AE4M3 |  | W4A8 |
---|---|---|---|---
Model (Seq len) | ABFP | ABFP-SQ | ABFP | ABFP-SQ
OPT 125M (1024) | 29.99 | 29.63 | 29.21 | 28.47
OPT 350M (1024) | 21.99 | - | 21.67 | -
OPT 1.3B (1024) | 13.04 | 12.54 | 12.73 | 12.26
OPT 2.7B (1024) | 11.87 | 11.44 | 11.78 | 11.37
TABLE VII: Accuracy recovery methods on W4A8 representation to augment ABFP for $n=64$. |  | W4A8 |  | W4A16
---|---|---|---|---
Model (Seq len) | ABFP | ABFP-QAT | ABFP-SQ | GPTQ
OPT 125M (1024) | 29.21 | 21.22 | 28.47 | 30.95
OPT 350M (1024) | 21.67 | 18.95 | - | 19.87
OPT 1.3B (256) | 15.54 | 14.79 | 14.95 | 18.68
OPT 2.7B (256) | 14.34 | 13.68 | 13.81 | 17.48
Figure 5: Performance of the models with ABFP and QAT for different vector
lengths $(n=64,128)$ in the W4A8 format.

####  IV-B3 Accuracy recovery for 4-bit integer weights and 8-bit integer
activations

Results for the experiments in this section are shown in Table VII. Similar to
the case with W4A4 in the previous sections, we see that using ABPF-QAT and
ABFP-SQ can both improve performance over ABFP alone. ABFP-QAT helps achieve
significantly improved PPL values that are close to the baseline models. In
the case of 8-bit activations, ABFP with SmoothQuant also achieves
performances that are very close to the baseline models, indicating that QAT
does not provide significant advantages here.

We further assess the impact of applying QAT with ABFP for different vector
lengths in Figure 5. In all cases, fine-tuning with QAT helps improve model
performance compared to using vanilla ABFP. Similar to the case with W4A4, we
see that for the different models, applying QAT at larger vector lengths
($n=128$) can achieve performances that are very close to smaller vector
lengths ($n=64$). Especially in the case of OPT 1.3B and OPT 2.7B the
resultant PPL values are very close to the baseline float32 models.

###  IV-C RPTQ vs. ABFP

In Table VIII we show results comparing RPTQ to ABFP for W4A4 and W4A8
formats. The open-source RPTQ repository does not include support for OPT 2.7B
and OPT 350M, hence we report results for OPT 125M and OPT 1.3B. For W4A8, we
observe that RPTQ performs slightly better than ABFP for OPT 1.3B, but worse
for OPT 125M. However, for W4A4 we see that ABFP performs better than RPTQ.
Nevertheless, there is a trade-off regarding storage overhead of the scales
for ABFP, which can be further mitigated through a second-order quantization
of the scales themselves [5].

TABLE VIII: Comparison of ABFP ($n=64$) and RPTQ for W4A4 and W4A8 formats. |  | W4A4 |  | W4A8 |
---|---|---|---|---|---
Model (Seq len) | FP32 | RPTQ | ABFP | RPTQ | ABFP
OPT 125M (1024) | 25.94 | 41.48 | 33.14 | 37.61 | 29.21
OPT 350M (1024) | 16.84 | - | 32.28 | - | 21.67
OPT 1.3B (1024) | 11.78 | 15.25 | 14.13 | 12.54 | 12.73
OPT 2.7B (1024) | 11 | - | 13.68 | - | 11.78

###  IV-D Results on Additional Models and Domains

INT-FP-QSim supports simulation of the different models shown in Table IX. The
models are shown along with their corresponding tasks, datasets and metrics.
In Table X we show the results of applying ABFP on 4-bit integer weights with
4-bit and 8-bit integer activations for $n=64$. In the case of W4A8, ABFP
enables all the models to achieve good out-of-the-box performance compared to
the float32 baseline. With W4A4, the vision models yield slightly better
performances as compared to the large language models indicating that the
vision models may be inherently easier to quantize. In particular, the Codegen
6B model drops significantly in performance with 4-bit activations. However,
in general, the out-of-the-box performances using ABFP ($n=64$) yields models
that perform well, and quite close to float32 baselines. The performances can
be further boosted by using ABFP with other techniques, such as QAT and
SmoothQuant.

##  V Data resources

In the appendix, we highlight the different baseline model checkpoints used
for the experiments (See Table XI for reference). The source code for INT-FP-
QSim is available here: https://github.com/lightmatter-ai/INT-FP-QSim. We
currently do not include the ABFP implementation in the simulator, although it
will be added in future iterations.

TABLE IX: Additional models, datasets and tasks investigated in this work along with the corresponding metrics. Model | Task | Dataset | Metric
---|---|---|---
OPT | Language modeling | Wikitext2 | PPL
Codegen | Code generation | HumanEval | Pass@1
BERT | Question answering | Squad v1.1 | F1
ViT | Image classification | ImageNet | Accuracy
ImageBind | Image classification | ImageNet | Accuracy
Maskformer | Image segmentation | ADE20k | mIOU
Graphormer | Graph classification | OGBG-Molhiv | Accuracy
Stable Diffusion | Text-to-image | Conceptual | CLIP
v1.4 | generation | Captions |
TABLE X: ABFP results on additional models on a series of different tasks for $n=64$ with the W4A4 and W4A8 formats. Model (Seq len) | FP32 | ABFP; W4A4 | ABFP; W4A8
---|---|---|---
OPT 125M (1024) | 25.94 | 33.14 | 29.21
OPT 350M (1024) | 16.84 | 32.28 | 21.67
OPT 1.3B (1024) | 11.78 | 15.46 | 12.73
OPT 2.7B (1024) | 11 | 13.67 | 11.78
Codegen 2B | 23.78 | 23.17 | 23.78
Codegen 6B | 26.83 | 21.95 | 23.78
BERT-base | 86.81 | 83.57 | 86.12
BERT-large | 93.48 | 92.41 | 93.14
ViT-large-16 | 79.7 | 79 | 79.7
ViT-large-32 | 77 | 75 | 76.7
ImageBind | 72.5 | 70.4 | 71.6
Maskformer-base | 52.85 | 51.29 | 52.71
Maskformer-large | 54.37 | 51.75 | 53.83
Graphormer | 98.27 | 98.18 | 98.27
Stable Diffusion v1.4 | 30.51 | 28.51 | 30.17

##  VI Discussion

In this work, we presented INT-FP-QSim, an open-source simulator for flexible
evaluation of large language models and vision transformers. INT-FP-QSim
leverages existing open-source repositories like TensorRT, QPytorch and AIMET
to provide a single destination for conducting experiments with models for
different numerical formats. With the simulator, we demonstrate results for a
range of models, tasks and precisions, and numerical formats.

With the existing implementation, we currently only analyze the models from an
accuracy perspective. Actual hardware implementations of the different
numerical formats can involve additional challenges in terms of throughput and
latency, that is not covered in this work. Further, matmuls are done in
float32 precision to understand the best-case performance of the different
models. However, INT-FP-QSim currently does not support matmuls in other
formats. Lastly, INT-FP-QSim currently does not support specification of
different quantizers for different layers in the model. Nevertheless, we hope
that access to INT-FP-QSim will enable researchers to easily and flexibly
simulate models at various precision combinations of weights, activations
(including output activations if desired), to encourage further research and
development in quantization of LLMs and vision transformers.

##  VII APPENDIX

In Table XI we provide details of the different checkpoints used in our
experiments. Most of the checkpoints are official HuggingFace checkpoints,
while ViT is obtained from TorchVision. Instead of using the original Facebook
checkpoints for OPT that are not fine-tuned on Wikitext2, we used available
checkpoints of the models that were specifically fine-tuned on Wikitext2 or
OGBG-Molhiv for Graphormer.

TABLE XI: Checkpoint links for the different models used in the experiments. HF denotes HuggingFace checkpoints. Model | Checkpoint Details
---|---
OPT 125M | HF: Aalaa/opt-125m-finetuned-wikitext2
OPT 350M | HF: lnair/opt-350m-wikitext2
OPT 1.3B | HF: lnair/opt-1.3b-wikitext2
OPT 2.7B | HF: lnair/opt-2.7b-wikitext2
Codegen 2B | HF: Salesforce/codegen-2B-mono
Codegen 6B | HF: Salesforce/codegen-6B-mono
BERT-base | HF: deepset/bert-base-cased-squad2
BERT-large | HF: bert-large-cased-whole-word-masking-finetuned-squad
ViT-16, 32 | Torchvision: vit_l_16, vit_l_32
ImageBind | Github: facebookresearch/ImageBind
Maskformer-base | HF: facebook/maskformer-swin-base-ade
Maskformer-large | HF: facebook/maskformer-swin-large-ade
Graphormer | HF: lnair/graphormer-ogbg-molhiv
Stable Diffusion | HF: CompVis/stable-diffusion-v1-4

## References

  * [1] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han.  Smoothquant: Accurate and efficient post-training quantization for large language models.  arXiv preprint arXiv:2211.10438, 2022.
  * [2] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  arXiv preprint arXiv:2208.07339, 2022.
  * [3] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.  Gptq: Accurate post-training quantization for generative pre-trained transformers.  arXiv preprint arXiv:2210.17323, 2022.
  * [4] Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu.  Rptq: Reorder-based post-training quantization for large language models.  arXiv preprint arXiv:2304.01089, 2023.
  * [5] Steve Dai, Rangha Venkatesan, Mark Ren, Brian Zimmer, William Dally, and Brucek Khailany.  Vs-quant: Per-vector scaled quantization for accurate low-precision neural network inference.  Proceedings of Machine Learning and Systems, 3:873–884, 2021.
  * [6] Ayon Basumallik, Darius Bunandar, Nicholas Dronen, Nicholas Harris, Ludmila Levkova, Calvin McCarter, Lakshmi Nair, David Walter, and David Widemann.  Adaptive block floating-point for analog deep learning hardware.  arXiv preprint arXiv:2205.06287, 2022.
  * [7] Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius.  Integer quantization for deep learning inference: Principles and empirical evaluation.  arXiv preprint arXiv:2004.09602, 2020.
  * [8] Tensorrt - pytorch quantization.  https://github.com/NVIDIA/TensorRT/tree/master/tools/pytorch-quantization.  Accessed: 2023-05-18.
  * [9] Tianyi Zhang, Zhiqiu Lin, Guandao Yang, and Christopher De Sa.  Qpytorch: A low-precision arithmetic simulation framework.  In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS), pages 10–13. IEEE, 2019.
  * [10] Ai model efficiency toolkit.  https://github.com/quic/aimet.  Accessed: 2023-05-18.
  * [11] Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort.  Fp8 quantization: The power of the exponent.  arXiv preprint arXiv:2208.09225, 2022.
  * [12] Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He.  A comprehensive study on post-training quantization for large language models.  arXiv preprint arXiv:2303.08302, 2023.
  * [13] Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al.  Fp8 formats for deep learning.  arXiv preprint arXiv:2209.05433, 2022.
  * [14] Charbel Sakr, Steve Dai, Rangha Venkatesan, Brian Zimmer, William Dally, and Brucek Khailany.  Optimal clipping and magnitude-aware differentiation for improved quantization-aware training.  In International Conference on Machine Learning, pages 19123–19138. PMLR, 2022.
  * [15] Cansu Demirkiran, Furkan Eris, Gongyu Wang, Jonathan Elmhurst, Nick Moore, Nicholas C Harris, Ayon Basumallik, Vijay Janapa Reddi, Ajay Joshi, and Darius Bunandar.  An electro-photonic system for accelerating deep neural networks.  arXiv preprint arXiv:2109.01126, 2021.


# Detecting heat leaks with trapped ion qubits

D. Pijn  Institut für Physik, Universität Mainz, Staudingerweg 7, 55128 Mainz,
Germany  O. Onishchenko  Institut für Physik, Universität Mainz, Staudingerweg
7, 55128 Mainz, Germany  J. Hilder  Institut für Physik, Universität Mainz,
Staudingerweg 7, 55128 Mainz, Germany  U. G. Poschinger<EMAIL_ADDRESS>Institut für Physik, Universität Mainz, Staudingerweg 7, 55128 Mainz, Germany
F. Schmidt-Kaler  Institut für Physik, Universität Mainz, Staudingerweg 7,
55128 Mainz, Germany  R. Uzdin  Fritz Haber Research Center for Molecular
Dynamics,Institute of Chemistry, The Hebrew University of Jerusalem, Jerusalem
9190401, Israel

###### Abstract

Recently, the principle of passivity has been used to set bounds on the
evolution of a microscopic quantum system with a thermal initial state. In
this work, we experimentally demonstrate the utility of two passivity based
frameworks: global passivity and passivity deformation, for the detection of a
“hidden” or unaccounted environment. We employ two trapped-ion qubits
undergoing unitary evolution, which may optionally be coupled to an unobserved
environment qubit. Evaluating the measurement data from the system qubits, we
show that global passivity can verify the presence of a coupling to an
unobserved environment - a heat leak - in a case where the second law of
thermodynamics fails. We also show that passivity deformation is even more
sensitive, detecting a heat leak where global passivity fails.

Introduction -  Thermodynamics has originally been conceived as a practical
theory for describing heat flows and efficiencies of heat engines. Taking the
point of view that quantum devices emerge to be the “heat engines” of the 21st
century motivates to explore if and how quantum thermodynamics can contribute
to the maturation of quantum technologies. In stochastic thermodynamics,
fluctuation theorems [1, 2] and thermodynamic uncertainty relations have been
formulated [3, 4], and in quantum thermodynamics, advanced master equations,
resource theory [5, 6], passivity-based frameworks [7, 8], and entropy-based
methods [9, 10] have been developed. Recent years have seen increasing
experimental studies on demonstrations and validations [11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24].

Figure 1: (a) A number of system qubits and one environment qubit are
initialized to thermal states. The system qubits undergo a unitary “black box”
evolution and are measured. In the case of a heat leak, the system is coupled
to the undetected environment qubit. (b) Laser-driven operations transfer the
qubit states between $\ket{0}$ and $\ket{1}$, with respective energy
eigenvalues.

Quantum information processing (QIP) devices, such as quantum computers and
quantum simulators, suffer from the inherent fragility of quantum states with
respect to environmental perturbations. This renders QIP devices to be
susceptible to various error mechanisms. In this work, we show that
thermodynamics-inspired frameworks are relevant for the characterization of
the operation of presently available well-controlled quantum systems. We use a
trapped-ion setup to experimentally study two recently developed passivity-
based frameworks: global passivity [7], and passivity deformation [8].

A wide variety of methods has been developed for benchmarking QIP devices and
their operational building blocks, ranging from low-level techniques such as
quantum process tomography [25] to holistic high-level approaches such as
quantum volume measurement [26, 27]. Methods developed within microscopic
thermodynamics offer complementary approaches for characterizing the
performance of QIP devices. Thermodynamic constraints such as the microscopic
second law [28, 29, 30] set constraints on the allowed dynamics of mixed
states in an isolated system. A violation of these constraints provides
information on undesired interaction with an external environment. Crucially,
these constraints make no assumptions on the tested protocol (a “black box”
test), and are therefore agnostic to the complexity of the evolution.
Moreover, thermodynamics-based tests are insensitive to coherent errors that
arise due to miscalibration. This represents a useful feature in view of the
identification and mitigation of error sources.

Not all thermodynamic constraints are scalable in the sense of providing
realistic measurement protocols for increasing system sizes. The analogue of
Clausius inequality in microscopic systems [28, 29, 30] requires quantum state
tomography for evaluating changes in the von Neumann entropy. The measurement
of trajectories in fluctuation theorems [1, 2] is equivalent to classical
process tomography, and resource theory [5, 6] also requires state tomography
for evaluating the Rényi divergence.
Passivity-based bounds, based on expectation values of observables, provide
polynomial scaling of the number of measurements with respect to the system
size [22]. The potential practical use of a thermodynamic bound is also
determined by its tightness. For example, global passivity [7] and the second
law set intrinsically loose bounds when the thermal environment is small,
therefore their accuracy and predictive power are inherently limited. However,
passivity deformation [8] provides increased sensitivity, since the
constraints are tight by construction, independently of the sizes of the
system constituents.

In this work, a trapped-ion based quantum computer is used to demonstrate that
global passivity can be more sensitive to a heat leak - a spurious energy
exchange channel to environmental degrees of freedom - as compared to the
second law, and that passivity deformation is more sensitive than global
passivity. To achieve this, we use an unobserved thermal qubit as a
controllable environment. Two system qubits constitute the “visible” part of
the system, i.e. qubits employed for the execution of a QIP protocol. The goal
is to detect the interaction with the environment qubit by measuring only the
system qubits.

Theory -  We consider a system consisting of degrees of freedom on which
measurements can be performed. If the system is isolated in the sense that
only classical (possibly noisy) external driving fields are applied, an
initial state $\hat{\rho}_{0}$ of the system will evolve into a final state

$\hat{\rho}_{f}=\sum_{k}p_{k}\hat{U}_{k}\hat{\rho}_{0}\hat{U}_{k}^{\dagger}.$
(1)

This evolution is determined by a mixture of unitary transforms $\hat{U}_{k}$.
As such evolutions are unital (i.e. a fully mixed state is invariant under
such transformation), the entropy of the system will always increase [31]. A
unital evolution can be interpreted as the result of a classical noisy driving
field. Quantum evolution that cannot be written as (1) ultimately requires an
interaction of the system with some external environment, e.g. an ancilla or a
thermal bath. Therefore, a verification that the evolution is not of the form
(1) via observations on the system confirms the presence of a heat leak.
The frameworks employed in this work for the detection of heat leaks rely on
the notion of passivity. In general, an operator $\hat{A}$ is passive with
respect to another operator $\hat{B}$, if $[\hat{A},\hat{B}]=0$, i.e. a common
set of eigenvectors exists, and if decreasingly ordered eigenvalues of
$\hat{A}$ correspond to increasingly ordered eigenvalues of $\hat{B}$. For
example, if a density operator $\hat{\rho}$ is passive with respect to the
system Hamiltonian $\hat{H}$, eigenstates are less populated for increasing
energy eigenvalues. The most common example is a thermal (Gibbs) state
expressed in the energy eigenbasis, where the occupation probabilities
monotonically decrease with the energy eigenvalue. Physically, this has the
important consequence that no energy can be extracted from a passive state via
unitary coupling to an external work body [32]. In the next section, we
briefly outline the global passivity and passivity deformation frameworks.

Global passivity -  The global passivity inequalities impose bounds for
changes of expectation values of a certain class of observables. We consider
any unital process (Eq. 1) taking initial state $\hat{\rho}_{0}$ to the final
state $\hat{\rho}_{f}$, and a function $F(x)$ which is monotonically
decreasing on $\min[\text{eig}(\hat{\rho}_{0})]\leq
x\leq\max[\text{eig}(\hat{\rho}_{0})]$. By construction, $F(\hat{\rho}_{0})$
is passive with respect to $\hat{\rho}_{0}$, and the global passivity
inequalities assume the form:

$\displaystyle\delta\langle F(\hat{\rho}_{0})\rangle$ $\displaystyle=$
$\displaystyle\langle F(\hat{\rho}_{0})\rangle_{f}-\langle
F(\hat{\rho}_{0})\rangle_{0}$ (2) $\displaystyle=$
$\displaystyle\text{tr}[F(\hat{\rho}_{0})(\hat{\rho}_{f}-\hat{\rho}_{0})]\geq
0$

Any violation of this inequality is a sufficient condition for the evolution
$\hat{\rho}_{0}\rightarrow\hat{\rho}_{f}$ to be not of the form (1) and
therefore indicates the presence of a heat leak.

Equation 2 leads to a simple form of the second law in a microscopic setup. We
consider a setup comprised of a cold object $c$ and a hot object $h$, each
described by Hamiltonian $\hat{H}_{c,h}$, and initial thermal states

$\hat{\rho}_{j}^{(0)}=e^{-\beta_{j}\hat{H}_{j}}/Z_{j}\qquad j=c,h$ (3)

where $Z_{j}=\text{tr}[e^{-\beta_{j}\hat{H}_{j}}]$. The initial state of the
joint cold/hot system is uncorrelated

$\hat{\rho}_{0}=\hat{\rho}_{c}^{(0)}\otimes\hat{\rho}_{h}^{(0)}=\frac{e^{-\beta_{c}\hat{H}_{c}\otimes\mathbb{1}_{h}-\beta_{h}\mathbb{1}_{c}\otimes\hat{H}_{h}}}{Z_{c}Z_{h}}$
(4)

Setting $F(x)=-\ln x$ yields

$F(\hat{\rho}_{0})=\beta_{c}H_{c}\otimes\mathbb{1}_{h}+\beta_{h}\mathbb{1}_{c}\otimes
H_{h}-\ln(Z_{c}Z_{h})\mathbb{1}_{ch}$ (5)

where $\mathbb{1}_{ch}$ is the identity operator on the product Hilbert space
of systems $c$ and $h$. The term $-\ln(Z_{c}Z_{h})\mathbb{1}_{ch}$ ensures
positivity of $F(\hat{\rho}_{0})$. However, upon taking the difference
$\langle F(\hat{\rho}_{0})\rangle_{f}-\langle F(\hat{\rho}_{0})\rangle_{0}$,
this term cancels out.

Now Eq. 2 reads

$\beta_{c}\;\delta\left\langle\hat{H}_{c}\right\rangle+\beta_{h}\;\delta\left\langle\hat{H}_{h}\right\rangle\geq
0$ (6)

which is one possible form of the second law for microscopic systems, i.e. the
analogue of the classical Clausius theorem $\oint\frac{\delta Q}{T}\geq 0$.

We obtain a more general, parametric set of inequalities by using
$F(x)=\text{sgn}(\alpha)(-\ln x)^{\alpha}$, which is a monotonically
increasing function on $0\leq x\leq 1$. We introduce

$\hat{B}=\beta_{c}\hat{H}_{c}+\beta_{h}\hat{H}_{h}-d\;\mathbb{1}_{ch}$ (7)

and the shorthand notation

$F(\hat{B})=\hat{B}^{\alpha}=\text{sgn}(\alpha)(\beta_{c}\hat{H}_{c}+\beta_{h}\hat{H}_{h}-d_{\epsilon}\mathbb{1}_{ch})^{\alpha}$
(8)

where the choice

$d=\min(\text{eig}(\beta_{c}\hat{H}_{c}+\beta_{h}\hat{H}_{h}))+\epsilon$ (9)

with a sufficiently small $\epsilon$ enforces nonzero and positive eigenvalues
of $\hat{B}$. Invoking Eq. 2, we obtain the global passivity inequalities in a
compact form:

$\delta\langle\hat{B}^{\alpha}\rangle\geq 0.$ (10)

Note that the microscopic form of the second law, Eq. 6, is obtained from Eq.
10 for $\alpha=1$. A violation of the inequality Eq. 10 necessarily implies
that one of the underlying assertions are violated, which is either that the
initial state is not passive or that the system evolution is not of the form
(1) and therefore an interaction with environmental degrees of freedom is
present.

For $\alpha\gg 1$, the largest eigenvalue of $\hat{B}$ dominates, since the it
corresponds to the smallest eigenvalue of $\hat{\rho}_{0}$, as $\hat{B}$ is
obtained from $\hat{\rho}_{0}$ using a monotonically decreasing function.
Thus, as $\alpha\to+\infty$, inequality Eq. 10 states that the probability of
the state that was initially least populated cannot decrease beyond its
initial value. Conversely, for $\alpha\to-\infty$ we learn that the
probability of the state that was initially most populated cannot increase
beyond its initial value. Different values of $\alpha$ put emphasis on
different eigenvalues of the expectation values and therefore can potentially
detect different types of heat leaks.

Passivity deformation -  Passivity deformation [8] is a general and versatile
framework for deriving passivity-based inequalities. It follows from the
observation that a globally passive operator $\hat{B}$ can be used to generate
inequalities involving an observable $\hat{A}$. We introduce the operator

$\hat{B}^{\prime}=\hat{B}+\xi\hat{A}$ (11)

where $\xi$ is a real deformation parameter and $\hat{A}$ is an observable of
interest that satisfies $[\hat{B},\hat{A}]=0$. Hence, $\hat{B}$ and
$\hat{B}^{\prime}$ have the same eigenstates. If, moreover, the eigenvalue
ordering of $\hat{A}$ and $\hat{B}$ is the same, the global passivity of
$\hat{B}$ is inherited by $\hat{B}^{\prime}$ and the inequality

$\delta\langle\hat{B}^{\prime}\rangle\geq 0$ (12)

holds for any unitary evolution. This condition is trivially satisfied for
$\xi=0$, while it is violated for large deformations, since $\hat{A}$ is in
general not globally passive. Thus, there exist extremal values $\xi_{m}\leq
0$ and $\xi_{p}\geq 0$ such that Eq. 12 holds for $\xi_{m}<\xi<\xi_{p}$. If
the eigenvalues of $\hat{B}$ and $\hat{A}$ are known, finding the limit values
$\xi_{m,p}$ analytically or numerically is a simple task. The obtained
passivity deformation inequalities

$\delta\langle\hat{A}\rangle\geq-\frac{1}{\xi}\delta\langle\hat{B}\rangle\qquad\forall\qquad\xi_{m}\leq\xi\leq\xi_{p}$
(13)

First, they may describe observables beyond global energetics (e.g. the ground
state population of a sub-system). Furthermore, unlike global passivity, the
passivity deformation inequalities Eqs. 13 are guaranteed to be tight for some
nontrivial process $\hat{\rho}_{0}\rightarrow\hat{\rho}_{f}$, even when the
thermal environment is small. Consequently, as demonstrated in our experiment,
the passivity deformation inequalities may have stronger sensitivity to
violation of unitality.

Figure 2: Quantum circuits used for demonstrating a violation of (a) the
global passivity inequality (Eq. 6), and (b) the passivity deformation
inequality (Eq. 13). In both cases, all qubits are initialized to thermal
states (before (i)), the system qubits undergo unitary evolution (before
(ii)), and one of the system qubits is optionally swapped with the environment
qubit before measurement (iii). Generation of thermal (mixed) states is
depicted by a flame, with different spin temperatures proportional to the size
of the flame. The temperatures and the particular unitary quantum operations
before the final SWAP gate are chosen to provide optimum sensitivity for the
detection of the heat leak.

Experiment -  The platform we employ in this work for showing the violation of
passivity-based inequalities is based on qubits encoded in trapped atomic
ions. The ions are confined in a microstructured, segmented radio frequency
trap [33], and can be moved between different storage sites via shuttling
operations [34, 35]. Laser beams, directed to a fixed storage site - the laser
interaction zone (LIZ), are employed for initialization, manipulation and
readout of the qubits. This way, any single qubit or a pair of qubits can
undergo a laser-driven operation in the LIZ, without any crosstalk affecting
the remaining qubits stored at different trap sites. The qubits are encoded in
the spin degree of freedom of the valence electron of 40Ca+ ions [36], i.e.
the qubit states correspond to the electronic states
$\ket{0}\equiv\ket{S_{1/2},m_{J}=-1/2}$ and
$\ket{1}\equiv\ket{S_{1/2},m_{J}=+1/2}$. The qubits states feature an energy
splitting of $\omega_{0}\approx 2\pi\times$10 MHz via the Zeeman effect caused
by a static externally applied magnetic field [37]. However, without loss of
generality, we assign the dimensionless energy eigenvalues $E_{0}=0$ and
$E_{1}=1$ in the following. The free Hamiltonian for both qubits thus reads

$\hat{H}^{(0)}_{j}=\ket{1_{j}}\bra{1_{j}}\qquad j=c,h$ (14)

The qubits are read out via laser-driven, selective population transfer to the
metastable $D_{5/2}$ state, followed by detection of state-dependent laser-
induced fluorescence [36]. This way, above-threshold detection of fluorescence
corresponds to the qubit being detected in $\ket{0}$, while below-threshold
detection corresponds to the qubit being detected in $\ket{1}$. Repeated
execution of a given protocol therefore yields estimates of the occupation
probabilities for each logical basis state of the qubit register. The relevant
error sources are given by shot noise for a finite number of shots, yielding
statistical errors, and state preparation and measurement (SPAM) errors,
leading to systematic errors.
Our experimental protocols employ a ‘cold’ qubit $c$, a ‘hot’ qubit $h$ and a
third, unobserved environment qubit $e$. At the beginning of each experimental
sequence, these are successively initialized to thermal states Eq. 3 with
respect to the free Hamiltonian Eq. 14 via incomplete optical pumping [38].
Here, any desired spin temperature can be preset via control of the pump laser
pulse duration, such that the inverse temperatures $\beta_{j}$ for $j=c,h,e$
(in terms of the dimensionless energy eigenvalues) are given from the
Boltzmann weights via

$\beta_{j}=\ln\left(\frac{p_{0}^{(j)}}{1-p_{0}^{(j)}}\right),$ (15)

where $p_{0}^{(j)}$ is the population of state $\ket{0_{j}}$.

Heat leak detection via global passivity -  Qubits $c$, $h$ and $e$ are
successively moved to the LIZ and initialized to thermal state. The inverse
temperatures $\beta_{c}=\ $2.23(4), $\beta_{h}=\ $0.43(2) and $\beta_{e}=\
$2.02(4) are chosen to provide optimum sensitivity for the detection of the
heat leak.
After initialization, the system qubits $c$ and $h$ are stored pairwise at the
LIZ and undergo a laser-driven unitary evolution. For our protocol, this
evolution consists of a two-qubit Ising-type phase gate mediated by light-
shifts [39], described by

$\displaystyle\\{\ket{0_{c}0_{h}},\ket{1_{c}1_{h}}\\}$
$\displaystyle\rightarrow$ $\displaystyle
e^{i\Phi}\\{\ket{0_{c}0_{h}},\ket{1_{c}1_{h}}\\}$
$\displaystyle\\{\ket{0_{c}1_{h}},\ket{1_{c}0_{h}}\\}$
$\displaystyle\rightarrow$
$\displaystyle\\{\ket{0_{c}1_{h}},\ket{1_{c}0_{h}}\\}.$ (16)

We chose $\Phi=3\pi/4$ to provide optimum sensitivity to the heat leak. This
gate is sandwiched between two local qubit rotations by angle $\pi/2$:

$\hat{U}_{y}=\exp\left(-i\frac{\pi}{4}(\hat{\sigma}_{y}^{(c)}\oplus\hat{\sigma}_{y}^{(h)})\right),$
(17)

where $\hat{\sigma}_{y}^{(j)}$ is the Pauli $Y$ operator for qubit $j$. The
quantum circuit for this protocol is depicted in Fig. 2(a). After the coherent
evolution, qubits $c$ and $h$ are separated [40], then qubits $h$ and $e$ are
merged to the LIZ, where they can undergo an optional SWAP gate. The SWAP gate
is executed via physical swapping of the ion positions, which has been shown
to realize a unit-fidelity gate in [41], as the ions are indistinguishable and
the control over the operations exerted via electric fields, which do not
affect the qubit. Finally, only the system qubits $c$ and $h$ are read out as
described above.
Each single shot $k$ yields one of the results
$\\{\ket{0_{c}0_{h}},\ket{0_{c}1_{h}},\ket{1_{c}0_{h}},\ket{1_{c}1_{h}}\\}$,
corresponding to the measured energies $E^{(k)}_{j}=\\{0,+1\\}$. This yields
the single-shot measurement result of operator $\hat{B}^{\alpha}$ Eq. 8 via

$B_{(k)}^{\alpha}=\text{sgn}(\alpha)(\beta_{c}E_{c}^{(k)}+\beta_{h}E_{h}^{(k)}-d_{\epsilon})^{\alpha}.$
(18)

From $N$ acquired shots, we evaluate the expectation values by evaluating
sample averages based on the obtained single-shot measurement results Eq. 18.
We acquire three independent data sets, each consisting of 6700 shots in
total, for the cases where the measurements take place i) after
initialization, ii) after the gates acting on $c$ and $h$, without the SWAP
gate and iii) after the SWAP gate between qubits $h$ and $e$. Expectation
values $\langle\hat{B}^{\alpha}\rangle$ are computed for all three data sets,
by varying $\alpha$. Changes $\delta\langle\hat{B}^{\alpha}\rangle$ with
respect to $\alpha$ are then computed for both the cases with and without SWAP
gate, with respect to the expectation values computed for the initial state.
The results are shown in Fig. 3. Estimates for the statistical error are
computed via non-parametric bootstrapping: Artificial event rates of detecting
$\\{\ket{0_{c}0_{h}},\ket{0_{c}1_{h}},\ket{1_{c}0_{h}},\ket{1_{c}1_{h}}\\}$
are generated by drawing event numbers from a multinomial distribution,
governed by the measured event rates. The artificial rates are used for
computing expectation values $\langle\hat{B}^{\alpha}\rangle$, which are used
in turn to compute a 1$\sigma$ error channel.
For the case without SWAP gate, we observe
$\delta\langle\hat{B}^{\alpha}\rangle\geq 0$ for the entire range of $\alpha$
values, which indicates a unitary evolution of qubits $c$ and $h$. In
contrast, for the case with SWAP gate, we observe
$\delta\langle\hat{B}^{\alpha}\rangle\leq 0$ for values of $\alpha$ below
0.5090(75). This shows a clear violation of the global passivity inequality
Eq. 10. Note that the microscopic form of the second law ($\alpha=1$) Eq. 6
provides $\delta\langle\hat{B}\rangle\geq 0$, which confirms that the
framework of global passivity provides an increased sensitivity for
experimental verification of heat leaks.

Figure 3: For the first protocol, depicted in Fig. 2(a),
$\delta\langle\hat{B}^{\alpha}\rangle$ is shown as a function of $\alpha$. In
(a), the system qubits are coupled to the environment qubit via the final SWAP
gate, and we observe $\delta\langle\hat{B}^{\alpha}\rangle\leq 0$ for
$\alpha\lesssim$ 0.5. In (b), the SWAP gate is not performed, and we observe
no violation of the global passivity inequality (Eq. 10). The experimental
data (brown, dotted) is compared to the theoretical expectation values
(green). The inset shows the measured binary probabilities for system qubits
$c,h$ from which the experimental data was calculated. 6700 shots were used.

Heat leak detection via passivity deformation -  A slightly modified protocol
serves for detecting the heat leak via the passivity deformation approach. For
this case, we choose $\beta_{c}=\ $1.627(7), $\beta_{h}=\ $1.099(8) and
$\beta_{e}=\ $2.232(5). As depicted in Fig. 2(b), the joint local qubit
rotations are replaced by a rotation about 2.5 rad of qubit $h$ only,
described by

$\hat{U}_{y}=\exp\left(-i\;2.5\;\hat{\sigma}_{y}^{(h)}\right),$ (19)

The phase gate between qubit $c$ and $h$ is replaced by a SWAP gate. Similarly
to the global passivity test, qubits $c$ and $h$ are separated after the
unitary evolution, then qubits $h$ and $e$ are merged to the LIZ, where they
can undergo an optional SWAP gate.
We chose the Hamiltonian of the hot qubit
$\hat{H}_{h}=\mathbb{1}_{c}\otimes\ket{1_{h}}\bra{1_{h}}$ to be the
deformation operator $\hat{A}$ (cf. Eq. 11). Both operators $\hat{B}$ and
$\hat{B}^{\prime}$ are diagonal, with eigenvalues

$\displaystyle\text{eig}_{\uparrow}(\hat{B})$ $\displaystyle=$
$\displaystyle\\{0,\beta_{h},\beta_{c},\beta_{h}+\beta_{c}\\}$
$\displaystyle\text{eig}(\hat{B^{\prime}})$ $\displaystyle=$
$\displaystyle\\{0,\beta_{h}+\xi,\beta_{c},\beta_{h}+\beta_{c}+\xi\\}.$ (20)

Note that the eigenvalues of $\hat{B}$ are sorted, while the eigenvalues of
$\hat{B}^{\prime}$ are not. Condition Eq. 12 requires the eigenvalues of
$\hat{B}^{\prime}$ to have the same sorting as for $\hat{B}$, which leads to

$\xi_{m}=-\beta_{h}\leq\xi\leq\beta_{c}-\beta_{h}=\xi_{p}$ (21)

The passivity deformation inequality Eq. 13 then yields

$\delta\langle\hat{H}_{c}\rangle\geq-\frac{\beta_{h}+\xi}{\beta_{c}}\delta\langle\hat{H}_{h}\rangle\quad\forall\
\xi_{m}\leq\xi\leq\xi_{p}$ (22)

The left-hand side of this inequality, computed from the measurement data, is
shown for varying $\xi$ in Fig. 4. For the case with SWAP, we observe a clear
violation of Eq. 22 for $\xi<-0.880(1)$, which is about 5.3 standard
deviations above the bound $\xi_{m}$. From Fig. 4 c), we see that global
passivity fails to detect the heat leak for this scenario, as
$\delta\langle\hat{B}^{\alpha}\rangle\geq 0$ for any $\alpha$. This
demonstrates that passivity deformation based inequalities yield increased
sensitivity to heat leaks as compared to global passivity.

Figure 4: For the second protocol, depicted in Fig. 2(b), we compare the
energy change in the cold bath (blue) to some function of the parameter $\xi$
times the energy change in the hot bath (red), for the cases (a) with final
SWAP gate, and (b) without final SWAP gate. We find a violation for
$-\beta_{h}\leq\xi\leq-0.880(1)$ in the case with SWAP gate. Experimental data
(blue/red, dotted) is compared to theoretical expectation values (green). The
binary probabilities for system qubits $h,c$, from which the data was
calculated, are shown as an inset. 3200 shots were used. Panel (c) shows that
we observe no violation of the global passivity inequality (Eq. 10) in this
case, for the data including final SWAP gate.

Conclusion -  We have used trapped ions to demonstrate the relevance of
passivity-based inequalities for detecting controllable heat leaks, i.e. the
presence of measurable interactions with the environment. While a formulation
of a diagnostics scheme based on these ideas requires further study, this
experiment shows that passivity based constraints are experimentally relevant
and that they are more sensitive to heat leaks as compared to the second law
of thermodynamics.
Future work will aim on using periodically repeating protocols to amplify the
effect of a heat leak and therefore increase the detection sensitivity, in
order to detect genuine heat leaks rather than artificially introduced
environments in quantum devices.

###### Acknowledgements.

FSK and UGP acknowledge funding from DFG with research unit Thermal Machines
in the Quantum World (FOR 2724), from the EUH2020-FETFLAG-2018-03 under Grant
Agreement no.820495 and by the Germany ministry of science and education
(BMBF) within IQuAn. RU is grateful for support from Israel Science Foundation
(Grant No. 2556/20).

## References

  * Speck and Seifert [2007] T. Speck and U. Seifert, The jarzynski relation, fluctuation theorems, and stochastic thermodynamics for non-markovian processes, Journal of Statistical Mechanics: Theory and Experiment 2007, L09002 (2007).
  * Campisi  _et al._ [2011] M. Campisi, P. Hänggi, and P. Talkner, Colloquium: Quantum fluctuation relations: Foundations and applications, Rev. Mod. Phys. 83, 771 (2011).
  * Van Vu and Hasegawa [2020] T. Van Vu and Y. Hasegawa, Thermodynamic uncertainty relations under arbitrary control protocols, Phys. Rev. Research 2, 013060 (2020).
  * Horowitz and Gingrich [2020] J. M. Horowitz and T. R. Gingrich, Thermodynamic uncertainty relations constrain non-equilibrium fluctuations, Nature Physics 16, 15 (2020).
  * Gour  _et al._ [2015] G. Gour, M. P. Müller, V. Narasimhachar, R. W. Spekkens, and N. Y. Halpern, The resource theory of informational nonequilibrium in thermodynamics, Physics Reports 583, 1 (2015).
  * Lostaglio [2019] M. Lostaglio, An introductory review of the resource theory approach to thermodynamics, Reports on Progress in Physics 82, 114001 (2019).
  * Uzdin and Rahav [2018] R. Uzdin and S. Rahav, Global passivity in microscopic thermodynamics, Phys. Rev. X 8, 021064 (2018).
  * Uzdin and Rahav [2021] R. Uzdin and S. Rahav, Passivity deformation approach for the thermodynamics of isolated quantum setups, PRX Quantum 2, 010336 (2021).
  * Strasberg and Winter [2021] P. Strasberg and A. Winter, First and second law of quantum thermodynamics: A consistent derivation based on a microscopic definition of entropy, PRX Quantum 2, 030202 (2021).
  * Bera  _et al._ [2019] M. N. Bera, A. Riera, M. Lewenstein, Z. B. Khanian, and A. Winter, Thermodynamics as a consequence of information conservation, Quantum 3, 121 (2019).
  * Maslennikov  _et al._ [2019] G. Maslennikov, S. Ding, R. Hablützel, J. Gan, A. Roulet, S. Nimmrichter, J. Dai, V. Scarani, and D. Matsukevich, Quantum absorption refrigerator with trapped ions, Nature communications 10, 1 (2019).
  * Cottet  _et al._ [2017] N. Cottet, S. Jezouin, L. Bretheau, P. Campagne-Ibarcq, Q. Ficheux, J. Anders, A. Auffèves, R. Azouit, P. Rouchon, and B. Huard, Observing a quantum maxwell demon at work, Proceedings of the National Academy of Sciences 114, 7561 (2017).
  * von Lindenfels  _et al._ [2019a] D. von Lindenfels, O. Gräb, C. T. Schmiegelow, V. Kaushal, J. Schulz, M. T. Mitchison, J. Goold, F. Schmidt-Kaler, and U. G. Poschinger, Spin heat engine coupled to a harmonic-oscillator flywheel, Phys. Rev. Lett. 123, 080602 (2019a).
  * Elouard  _et al._ [2020] C. Elouard, M. Waegell, B. Huard, and A. N. Jordan, An interaction-free quantum measurement-driven engine, Foundations of Physics 50, 1294 (2020).
  * Naghiloo  _et al._ [2018] M. Naghiloo, J. J. Alonso, A. Romito, E. Lutz, and K. W. Murch, Information gain and loss for a quantum maxwell’s demon, Phys. Rev. Lett. 121, 030604 (2018).
  * Karimi  _et al._ [2020] B. Karimi, F. Brange, P. Samuelsson, and J. P. Pekola, Reaching the ultimate energy resolution of a quantum detector, Nature communications 11, 1 (2020).
  * Klatzow  _et al._ [2019] J. Klatzow, J. N. Becker, P. M. Ledingham, C. Weinzetl, K. T. Kaczmarek, D. J. Saunders, J. Nunn, I. A. Walmsley, R. Uzdin, and E. Poem, Experimental demonstration of quantum effects in the operation of microscopic heat engines, Phys. Rev. Lett. 122, 110601 (2019).
  * Roßnagel  _et al._ [2016] J. Roßnagel, S. T. Dawkins, K. N. Tolazzi, O. Abah, E. Lutz, F. Schmidt-Kaler, and K. Singer, A single-atom heat engine, Science 352, 325 (2016).
  * Peterson  _et al._ [2019] J. P. Peterson, T. B. Batalhão, M. Herrera, A. M. Souza, R. S. Sarthour, I. S. Oliveira, and R. M. Serra, Experimental characterization of a spin quantum heat engine, Physical review letters 123, 240601 (2019).
  * Buffoni and Campisi [2020] L. Buffoni and M. Campisi, Thermodynamics of a quantum annealer, Quantum Science and Technology (2020).
  * Gardas and Deffner [2018] B. Gardas and S. Deffner, Quantum fluctuation theorem for error diagnostics in quantum annealers, Scientific reports 8, 17191 (2018).
  * Henao  _et al._ [2019] I. Henao, R. Uzdin, and N. Katz, Experimental detection of microscopic environments using thermodynamic observables, arXiv preprint arXiv:1908.08968 (2019).
  * Solfanelli  _et al._ [2021] A. Solfanelli, A. Santini, and M. Campisi, Experimental verification of fluctuation relations with a quantum computer, arXiv preprint arXiv:2106.04388 (2021).
  * Henao  _et al._ [2021] I. Henao, R. Uzdin, and N. Katz, Experimental detection of microscopic environments using thermodynamic observables (2021), arXiv:1908.08968 [quant-ph] .
  * Häffner  _et al._ [2005] H. Häffner, W. Hänsel, C. F. Roos, J. Benhelm, D. Chek-al kar, M. Chwalla, T. Körber, U. D. Rapol, M. Riebe, P. O. Schmidt, C. Becher, O. Gühne, W. Dür, and R. Blatt, Scalable multiparticle entanglement of trapped ions, Nature 438, 643 (2005).
  * Cross  _et al._ [2019] A. W. Cross, L. S. Bishop, S. Sheldon, P. D. Nation, and J. M. Gambetta, Validating quantum computers using randomized model circuits, Phys. Rev. A 100, 032328 (2019).
  * Pino  _et al._ [2021] J. M. Pino, J. M. Dreiling, C. Figgatt, J. P. Gaebler, S. A. Moses, M. S. Allman, C. H. Baldwin, M. Foss-Feig, D. Hayes, K. Mayer, C. Ryan-Anderson, and B. Neyenhuis, Demonstration of the trapped-ion quantum ccd computer architecture, Nature 592, 209 (2021).
  * Uzdin [2018] R. Uzdin, The second law and beyond in microscopic quantum setups, As a chapter of: F. Binder, L. A. Correa, C. Gogolin, J. Anders, and G. Adesso (eds.), ”Thermodynamics in the quantum regime - Recent Progress and Outlook”, (Springer International Publishing). arXiv: 1805.02065 (2018).
  * Esposito and Van den Broeck [2011] M. Esposito and C. Van den Broeck, Second law and landauer principle far from equilibrium, EPL (Europhysics Letters) 95, 40004 (2011).
  * Peres [2006] A. Peres,  _Quantum theory: concepts and methods_ , Vol. 57 (Springer Science & Business Media, 2006).
  * Mendl and Wolf [2009] C. B. Mendl and M. M. Wolf, Unital quantum channels–convex structure and revivals of birkhoffs theorem, Communications in Mathematical Physics 289, 1057 (2009).
  * Allahverdyan  _et al._ [2004] A. Allahverdyan, R. Balian, and T. M. Nieuwenhuizen, Maximal work extraction from finite quantum systems, Europhys. Lett. 67, 565 (2004).
  * Kaushal  _et al._ [2020] V. Kaushal, B. Lekitsch, A. Stahl, J. Hilder, D. Pijn, C. Schmiegelow, A. Bermudez, M. Müller, F. Schmidt-Kaler, and U. Poschinger, Shuttling-based trapped-ion quantum information processing, AVS Quantum Science 2, 014101 (2020).
  * Kielpinski  _et al._ [2002] D. Kielpinski, C. Monroe, and D. J. Wineland, Architecture for a large-scale ion-trap quantum computer, Nature 417, 709 (2002).
  * Walther  _et al._ [2012] A. Walther, F. Ziesel, T. Ruster, S. T. Dawkins, K. Ott, M. Hettrich, K. Singer, F. Schmidt-Kaler, and U. G. Poschinger, Controlling fast transport of cold trapped ions, Phys. Rev. Lett. 109, 080501 (2012).
  * Poschinger  _et al._ [2009] U. G. Poschinger, G. Huber, F. Ziesel, M. Deiß, M. Hettrich, S. A. Schulz, K. Singer, G. Poulsen, M. Drewsen, R. J. Hendricks, and F. Schmidt-Kaler, Coherent manipulation of a 40ca+ spin qubit in a micro ion trap, Journal of Physics B: Atomic, Molecular and Optical Physics 42, 154013 (2009).
  * Ruster  _et al._ [2016] T. Ruster, C. T. Schmiegelow, H. Kaufmann, C. Warschburger, F. Schmidt-Kaler, and U. G. Poschinger, A long-lived zeeman trapped-ion qubit, Applied Physics B 122, 254 (2016).
  * von Lindenfels  _et al._ [2019b] D. von Lindenfels, O. Gräb, C. T. Schmiegelow, V. Kaushal, J. Schulz, M. T. Mitchison, J. Goold, F. Schmidt-Kaler, and U. G. Poschinger, Spin heat engine coupled to a harmonic-oscillator flywheel, Phys. Rev. Lett. 123, 080602 (2019b).
  * Leibfried  _et al._ [2003] D. Leibfried, B. DeMarco, V. Meyer, D. Lucas, M. Barrett, J. Britton, W. M. Itano, B. Jelenkovic, C. Langer, T. Rosenband, and D. J. Wineland, Experimental demonstration of a robust, high-fidelity geometric two ion-qubit phase gate, Nature 422, 412 (2003).
  * Ruster  _et al._ [2014] T. Ruster, C. Warschburger, H. Kaufmann, C. T. Schmiegelow, A. Walther, M. Hettrich, A. Pfister, V. Kaushal, F. Schmidt-Kaler, and U. G. Poschinger, Experimental realization of fast ion separation in segmented paul traps, Phys. Rev. A 90, 033410 (2014).
  * Kaufmann  _et al._ [2017] H. Kaufmann, T. Ruster, C. T. Schmiegelow, M. A. Luda, V. Kaushal, J. Schulz, D. von Lindenfels, F. Schmidt-Kaler, and U. G. Poschinger, Fast ion swapping for quantum-information processing, Phys. Rev. A 95, 052319 (2017).