# Effective versus Floquet theory for the Kerr parametric oscillator

Ignacio García-Mata  Instituto de Investigaciones Físicas de Mar del Plata
(IFIMAR), Facultad de Ciencias Exactas y Naturales, Universidad Nacional de
Mar del Plata & CONICET, 7600 Mar del Plata, Argentina  Rodrigo G. Cortiñas
Department of Applied Physics and Physics, Yale University, New Haven,
Connecticut 06520, USA  Yale Quantum Institute, Yale University, New Haven,
Connecticut 06520, USA  Xu Xiao  Department of Applied Physics and Physics,
Yale University, New Haven, Connecticut 06520, USA  Jorge Chávez-Carlos
Department of Physics, University of Connecticut, Storrs, Connecticut, USA
Victor S. Batista  Department of Chemistry, Yale University, P.O. Box 208107,
New Haven, Connecticut 06520-8107, USA  Yale Quantum Institute, Yale
University, New Haven, Connecticut 06520, USA  Lea F. Santos  Department of
Physics, University of Connecticut, Storrs, Connecticut, USA  Diego A.
Wisniacki  Departamento de Física “J. J. Giambiagi” and IFIBA, FCEyN,
Universidad de Buenos Aires, 1428 Buenos Aires, Argentina

###### Abstract

Parametric gates and processes engineered from the perspective of the static
effective Hamiltonian of a driven system are central to quantum technology.
However, the perturbative expansions used to derive static effective models
may not be able to efficiently capture all the relevant physics of the
original system. In this work, we investigate the conditions for the validity
of the usual low-order static effective Hamiltonian used to describe a Kerr
oscillator under a squeezing drive. In this work, we exploit the opportunity
provided by this system, which is sufficiently simple to be built and operated
in the lab, sufficiently complex to require modern calculation techniques for
the exploration of nontrivial parameter regimes, and sufficiently rich to be
of fundamental and technological interest. We compare the low-order static
effective states and energies with the exact Floquet states and quasi-energies
and determine the parameter regime where the descriptions agree. Our work
brings to light the physics that is left out by ordinary static effective
treatments and that can be explored by state-of-the-art experiments.

##  1  Introduction

Driven systems can present unexpected behaviors oftentimes without a static
analog. A typical example is the Kapitza pendulum, where a rapidly driven
rigid pendulum can stabilize against gravity by developing a minimum of
potential energy when pointing upward [1, 2]. In the quantum regime, this
static effective potential was proposed as a way to generate an error-
protected qubit [3] and the electronic analog of this mechanical system was
recently named “Kapitzonium” [4]. Another example are the Paul traps [5, 6],
that use time-dependent electric fields to trap charged particles and
therefore bypass Earnshaw’s theorem, which states that a charge distribution
cannot be stabilized in a stationary equilibrium configuration via its
electrostatic interactions. By changing the electric field faster than the
escaping rate of the particles, an average confining force can be created.
This idea is at the basis of some atomic clocks and trapped-ion quantum
technologies [7, 8].

Usually the perspective of a static effective model, which provides analytical
expressions and simplifications to the time-dependent problem, is used to
study driven systems. Many useful methods to achieve static effective
Hamiltonians have been derived in the last century [9, 10, 11, 12, 13] (see
[3] for a classification), but they are not without limitations. In
particular, in highly nonlinear systems, micromotion caused by the kicks of
the rapidly changing potential can feedback into the dynamics and produce
sizable effects associated with nonlinear mixing and amplifications [14].
These effects ultimately lead to regimes beyond the scopes of the static
effective Hamiltonian. Alternatively, when the system is periodically driven,
it can be studied numerically via Floquet theory. The advantage is that this
method can be carried out with a minimal amount of approximations, yet it
mostly remains a numerical treatment.

In this work, we discuss to what extent the ordinary static effective theory
and the numerical Floquet treatment agree. We direct our attention to a
dynamically rich system that is central to ongoing investigations: the Kerr
parametric oscillator. It consists of a nonlinear oscillator subjected to a
squeezing drive and boasts a storied history. It has served as an exemplary
instance of a parametric oscillator [15, 16, 17], an amplifier [18, 19, 20], a
tool for stabilizing quantum information [21, 22, 23, 24, 25], a framework for
quantum optical tuning [26], and more recently, as a platform to study excited
state phase transitions (ESQPTs) [27] and tunneling [28]. The model has also
been experimentally implemented with superconducting circuits, being employed
to generate Schrödinger’s cat states [29, 30], detect ESQPTs (aka “spectral
kissing”) [31], and analyze tunneling [32, 33]. For the parameters values
used, these experiments were well described by low-order static effective
Hamiltonians. However, steady experimental progress and access to broader
ranges of parameters prompt a deeper investigation beyond this regime. In this
case, the full Floquet numerical analysis becomes useful.

Here, we quantify the proximity of the driven and effective descriptions of
the Kerr parametric oscillator by comparing the Floquet quasienergies and
Floquet states of the time-dependent Hamiltonian with the eigenenergies and
eigenstates of the corresponding low-order effective Hamiltonian. This is done
via numerical simulations covering experimentally relevant parameters regimes.
We find that deviations between the two approaches become evident when the
nonlinear effects are significant. Understanding the parameter region where
the effective low-order Hamiltonian that has been used to describe recent
experiments is valid and reliable is of paramount importance for the design of
quantum technologies, including qubits, gates, and circuits. Analyzing when
low-order expansions may fail can point in the direction of new physics and
new possibilities for applications in quantum information processing.

The paper is structured as follows. In Sec. 2, we introduce the model system,
the time-dependent Hamiltonian and its corresponding static effective
approximation. In Sec. 3, we compare the eigenvalues and eigenvectors of the
effective Hamiltonian with the quasienergies and Floquet states of the time-
dependent system. This allows us to determine the experimentally accessible
parameters regimes, where the two descriptions agree and where they are
expected to diverge. Conclusions are presented in Sec. 4 and additional
results in the appendices.

##  2  Model system: Kerr parametric oscillator

The Hamiltonian of the driven superconducting nonlinear oscillator – the Kerr
parametric oscillator – that we analyze [30, 31, 32] is analogous to a one-
dimensional asymmetric driven quantum pendulum. In terms of dimensionless
coordinates, the Hamiltonian is given by

$\displaystyle\begin{split}\frac{\hat{H}(t)}{\hbar}=\omega_{o}\hat{a}^{\dagger}\hat{a}&+\frac{g_{3}}{3}(\hat{a}+\hat{a}^{\dagger})^{3}+\frac{g_{4}}{4}(\hat{a}+\hat{a}^{\dagger})^{4}\\\
&-i\Omega_{d}(\hat{a}-\hat{a}^{\dagger})\cos\omega_{d}t,\end{split}$ (1)

where $\omega_{o}$ is the bare frequency of the oscillator, $\hat{a}$
($\hat{a}^{\dagger}$) is the bosonic annihilation (creation) operator
satisfying the commutation relation $[\hat{a},\hat{a}^{\dagger}]=1$, the third
and fourth order nonlinearities of the potential energy are
$g_{3},g_{4}\ll\omega_{o}$, and the drive is characterized by its strength
$\Omega_{d}$ and its frequency $\omega_{d}$. The dimensionless coordinate,
$\hat{a}=\frac{1}{2}(\frac{\hat{X}}{X_{\mathrm{zps}}}+i\frac{\hat{P}}{P_{\mathrm{zps}}})$,
is written in terms of the position-like $\hat{X}$ and the momentum-like
$\hat{P}$ coordinates, the zero-point spread in the position-like degree of
freedom of the oscillator $X_{\mathrm{zps}}=\sqrt{\hbar/(2M\omega_{o})}$, and
the zero-point spread in momentum
$P_{\mathrm{zps}}=\hbar/(2X_{\mathrm{zps}})$, where $M$ is the effective mass
of the oscillator (for the mapping to a superconducting quantum system see
[34, 35]). Following the experiments [30, 31, 32], we consider the condition
to create parametric squeezing, that is, the system is driven at twice its
bare oscillation frequency, $\omega_{d}\approx 2\omega_{o}$. Under this
condition, the system undergoes a period-doubling bifurcation [26, 21, 15, 16,
36], whose static effective description corresponds to a double-well system
[15, 16, 17, 27].

In Ref. [31], it was shown that the experiment carried out with the time-
dependent Hamiltonian in Eq. (1) could be described by a low-order static
effective Hamiltonian. To compute the effective Hamiltonian, two
transformations must be applied to Eq. (1). The first one is a displacement
into the linear response of the oscillator, where the effective amplitude of
the displacement is $\Pi\approx\frac{2\Omega_{d}}{3\omega_{d}}$. The second
corresponds to a change into a rotating frame induced by
$\frac{\omega_{d}}{2}\hat{a}^{\dagger}\hat{a}$, transforming Eq. (1) to

$\displaystyle\begin{split}\frac{\hat{\mathcal{H}}(t)}{\hbar}=&-\delta\hat{a}^{\dagger}\hat{a}+\sum_{m=3}^{4}\frac{g_{m}}{m}(\hat{a}e^{-i\omega_{d}t/2}+\\\
&\ \hat{a}^{\dagger}e^{i\omega_{d}t/2}+\Pi
e^{-i\omega_{d}t}+\Pi^{*}e^{i\omega_{d}t})^{m},\end{split}$ (2)

where $\delta=\frac{\omega_{d}}{2}-\omega_{o}\ll\omega_{d}$. This choice of
frame brings the period doubling dynamics into focus (note that the
periodicity of eq.(2) is two times the periodicity of the drive. See [15]
section B.I for a discussion).

The propagator over a period $T=4\pi/\omega_{d}$ induced by eq.(2) is given by

$\displaystyle\hat{U}(T)=\mathcal{T}e^{\frac{-i}{\hbar}\int_{0}^{T}\hat{\mathcal{H}}(t)dt}=e^{\frac{i}{\hbar}\hat{S}(T)}e^{-\frac{i}{\hbar}\hat{H}_{\mathrm{eff}}T}e^{-\frac{i}{\hbar}\hat{S}(0)},$
(3)

where $\mathcal{T}$ is the time ordering operator which does not appear on the
right-hand side of the equation. This defines the operator
$\hat{S}(t)=\hat{S}(t+T_{d})=\hat{S}(t+T)$, whose purpose is to generate a
canonical transformation to a frame where the evolution is ruled by a time-
independent Hamiltonian $\hat{H}_{\mathrm{eff}}$. This provides an important
simplification. As discussed in detail in the next section, to compare the
eigenstates of the effective Hamiltonian with the Floquet states, we have to
take into account the unitary transformation,

$\hat{U}_{S}=e^{-\frac{i}{\hbar}\hat{S}},$ (4)

where $\hat{S}=\hat{S}(0)=\hat{S}(T)$.

So far no approximation has been made, but one can compute
$\hat{H}_{\mathrm{eff}}$ and $\hat{S}$ perturbatively to arbitrary order using
a mutually recursive formula developed in [3]. Following the approach in [3,
14], one uses the zero-point spread of the oscillator, $X_{\mathrm{zps}}$, as
the perturbation parameter and reaches the second-order effective Hamiltonian
[31]

$\frac{\hat{H}^{(2)}_{\mathrm{eff}}}{\hbar}=\epsilon_{2}(\hat{a}^{\dagger
2}+\hat{a}^{2})-K\hat{a}^{\dagger 2}\hat{a}^{2}.$ (5)

This Hamiltonian conserves parity,
$[\hat{H}^{(2)}_{\mathrm{eff}},e^{i\pi\hat{a}^{\dagger}\hat{a}}]=0$. The
driving condition for the period-doubling bifurcation can now be better
specified as $\omega_{d}=2\omega_{a}$, where
$\omega_{a}\approx\omega^{(2)}_{a}=\omega_{o}+3g_{4}-20g_{3}^{2}/3\omega_{o}+$
$(6g_{4}+9g_{3}^{2}/\omega_{o})(2\Omega_{d}/3\omega_{o})^{2}$ includes the
Lamb and Stark shift to the bare frequency $\omega_{o}$. In Eq. (5), the Kerr
nonlinearity to leading order is $K\approx
K^{(2)}=-\frac{3g_{4}}{2}+\frac{10g_{3}^{2}}{3\omega_{o}}$ and
$\epsilon_{2}\approx\epsilon_{2}^{(2)}=g_{3}\frac{2\Omega_{d}}{3\omega_{o}}$.
In these expressions, all nonlinear corrections are kept to order
$X_{\mathrm{zps}}^{2}$. For the parameters ranges that have so far been
considered experimentally [31, 32], the Hamiltonian
$\hat{H}^{(2)}_{\mathrm{eff}}$ matches spectroscopically the experimental
results for the driven systems up to the tenth excited state.

In what follows, we study the limits of applicability of the low-order static
effective theory for a wide range of parameters of experimental interest.
Various combinations of the native parameters $g_{3}$ and $g_{4}$ in Eq. (1)
can yield the same emergent effective Kerr nonlinearity $K$, yet the effective
Hamiltonian in Eq. (5) is not equally good for all choices. This analysis is
important for applications in quantum computing, where the Kerr nonlinearity
is required to be much larger than the decoherence rate of the driven system
[36, 37, 38]. The ability to engineer the correct static effective spectrum
for the highly excited states of the driven nonlinear oscillator is also
paramount for applications in quantum computing based on Kerr-cat qubits,
because the autonomous error-protecting properties of the qubit encoded in the
bifurcated oscillator depend strongly on the dynamics of the excited states
[31, 32, 39, 40].

Going beyond Eq. (5), we write down in appendix A the fourth-order static
effective Hamiltonian. In appendix B, we extend the comparison between the
time-dependent Hamiltonian in Eq. (2) and the effective model in Eq. (5), that
is developed in the main text, to include the fourth- and sixth-order
effective Hamiltonians. The results are qualitatively similar.

##  3  Spectra of the Floquet and the Effective Hamiltonian

We start this section by comparing the solutions of the time-dependent system
described by the periodically driven Hamiltonian $\hat{\mathcal{H}}(t)$ in Eq.
(2) with those of the effective Hamiltonian $\hat{H}^{(2)}_{\mathrm{eff}}$ in
Eq. (5). The time-dependent oscillator in Eq. (2) relies on five parameters:
$\omega_{o},\;g_{3},\;g_{4},\;\omega_{d}$ and $\Omega_{d}$. From now on, we
set $\omega_{o}=\hbar=1$.

We denote the eigenstates and eigenvalues of $\hat{H}^{(2)}_{\mathrm{eff}}$ as
$|\psi_{k}\rangle$ and $E_{k}$, and its ground-state energy as $E_{0}$. The
driven system is described by the Floquet states [41],

$|\Psi_{k}(t)\rangle=e^{-i\varepsilon_{k}t}|\phi_{k}(t)\rangle,$

where $|\phi_{k}(t)\rangle=|\phi_{k}(t+T)\rangle$ are the Floquet modes and
$\varepsilon_{k}$ are the Floquet quasienergies. The Floquet modes are the
eigenstates of the time-evolution operator (Floquet operator) after two
periods of the drive, at $T=2T_{d}$,

$\hat{U}(T)|\phi_{k}\rangle=e^{-i\varepsilon_{k}T}|\phi_{k}\rangle,$

and the quasienergies are obtained by diagonalizing $\hat{U}(T)$. Notice that
at $t=nT$, where $n$ is a natural number, the Floquet states and the Floquet
modes coincide, $|\Psi_{k}\rangle=|\phi_{k}\rangle$. The quasienergies are
uniquely defined modulo $\hbar\omega_{d}/2=2\pi\hbar/T$, that is,

$\varepsilon_{k}\in[0,\hbar\omega_{d}/2].$

###  3.1 Quasienergies vs Eigenvalues

In the top panel of Fig. 1, we show the excitation energies
$\tilde{E}=E-E_{0}$ (black lines) of $\hat{H}^{(2)}_{\mathrm{eff}}$ as a
function of the control parameter $\epsilon_{2}/K$ and compare them with the
quasienergies (orange lines) computed with respect to the ground-state
quasienergy, $\tilde{\varepsilon}=\varepsilon-\varepsilon_{0}$. To be able to
compare $\tilde{E}$ and $\tilde{\varepsilon}$ properly, we plot
$\tilde{E}\equiv(E-E_{0})/K$ and
$\tilde{\varepsilon}=[(\varepsilon-\varepsilon_{0})\mod(\omega_{d}/2)]/K$. For
this figure, we chose the value of the nonlinearities to be
$g_{3}/\omega_{o}=0.00075$, $g_{4}/\omega_{o}=1.27\times 10^{-7}$, and
therefore $K/\omega_{o}=1.685\times 10^{-6}$.

Figure 1:  Top panel: (Rescaled) Excitation energies $\tilde{E}=(E-E_{0})/K$
(black lines) of $\hat{H}^{(2)}_{\mathrm{eff}}$ in Eq. (5) and quasienergies
$\tilde{\varepsilon}=[(\varepsilon-\varepsilon_{0})\mod(\omega_{d}/2)]$
(orange) of $\hat{\mathcal{H}}(t)$ in Eq. (2) as a function of the control
parameter $\epsilon_{2}/K$. Panels (i)-(vi): Wigner functions of the Floquet
states corresponding to the quasienergies at the cross symbols x in the top
panel for $\epsilon_{2}/K=13$ and $\tilde{\varepsilon}_{k}$ equal to (i) 0,
(ii) 51.25, (iii) 97.9, (iv) 170.1 (v) 251.74, and (vi) 364.76. States
(i)-(iii) lie below the ESQPT and state (iv) is at the ESQPT critical energy.
Panels (vii)-(xii): Wigner functions of the Floquet states corresponding to
the circle symbols $\circ$ in the top panel for $\tilde{\varepsilon}_{6}$ and
$\epsilon_{2}/K$ equal to (vii) 2.19, (viii) 4.38, (ix) 8.76, (x) 10.96, (xi)
19.73, and (xii) 35.1. All panels: Basis size $N=200$,
$g_{3}/\omega_{o}=0.00075$, $g_{4}/\omega_{o}=1.27\times 10^{-7}$, and
$K/\omega_{o}=1.685\times 10^{-6}$. In the Wigner functions, red corresponds
to positive values and blue corresponds to negative values.

To determine the ground-state quasienergy $\varepsilon_{0}$, we use the method
developed in [42] to follow the eigenstates with definite localization
properties. We first determine the ground state
$\left|\Psi_{0}(\epsilon_{2}/K=0)\right>$ of the Hamiltonian in Eq. (2) with
$\Omega_{d}=0$. We then turn on the drive with a small increment
$\delta\Omega_{d}$ that results in an increase $\delta(\epsilon_{2}/K)$ of the
control parameter. To determine the new Floquet ground state,
$\left|\Psi_{0}(\delta(\epsilon_{2}/K))\right>$, we search for the Floquet
state that has the largest overlap with
$\left|\Psi_{0}(\epsilon_{2}/K=0)\right>$. The procedure is repeated each time
the drive amplitude is increased so that the Floquet ground state is
recursively updated.

For the small values of $g_{3}$ and $g_{4}$ considered in Fig. 1,
$\hat{H}^{(2)}_{\mathrm{eff}}$ accurately describes the spectrum of
$\hat{\mathcal{H}}(t)$, both Hamiltonians exhibiting an ESQPT. Specifically,
the spacing between pairs of adjacent levels $E_{k}$ belonging to different
parity sectors gets exponentially small at the energy of the ESQPT [27], a
phenomenon that became dubbed as “spectral kissing” in [31]. The critical
energy of the ESQPT grows as $\epsilon_{2}/K$ increases. The coalescence of
the energies is accompanied by the clustering of the energy levels and the
consequent divergence of the density of states at the ESQPT critical energy
[27].

In Figs. 1(i)-(xii), we show the Wigner functions for the Floquet states. They
are visually indistinguishable from the corresponding eigenstates of
$H^{(2)}_{\rm eff}$ in this regime (see below).

The behavior of the states below and above the ESQPT is markedly different.
Below the ESQPT, the system is represented by a double well and the states
exhibit a cat-like structure that can be used in quantum information
processing [43]. This structure is revealed in Figs. 1(i)-(iii) by the Wigner
functions of three Floquet states of $\hat{\mathcal{H}}(t)$ that show two
ellipses (one on the extreme left and the other on the extreme right of each
panel) located at the minima of the effective double well, and in between
them, centered at $q=0$, we see the interference fringes. The state shown in
Fig. 1(iv) is at the ESQPT critical energy, being highly localized at the Fock
state $|0\rangle$, which translates into a state concentrated at the origin of
the classical phase space, where there is an unstable hyperbolic point [27].
The Wigner function of this state in Fig. 1(iv) is visibly localized along the
separatrix (classical stable and unstable manifolds). Above the ESQPT, as the
energy increases, the Floquet states approach the eigenstates of a harmonic
potential, as seen in Figs. 1(v)-(vi).

In Figs. 1(vii)-(xii), we show the Wigner functions of the Floquet state with
quasienergy $\tilde{\varepsilon}_{5}$ for different values of
$\epsilon_{2}/K$. They are marked with circles in the top panel of Fig. 1. For
the value of $\epsilon_{2}/K$ in Fig. 1(vii), this Floquet state is above the
ESQPT and resembles an eigenstate of a harmonic potential, but for the values
of $\epsilon_{2}/K$ in Figs. 1(x)-(xii), the state is below the ESQPT and
turns into a cat-state like.

Figure 2:  Top row (a)-(d): Excitation energies $\tilde{E}=E-E_{0}$ (black
lines) of $\hat{H}^{(2)}_{\mathrm{eff}}$ in Eq. (5) and quasienergies
$\tilde{\varepsilon}=\varepsilon-\varepsilon_{0}$ (orange) of
$\hat{\mathcal{H}}(t)$ in Eq. (2) as a function of the control parameter
$\epsilon_{2}/K$. The gray dots in (c)-(d) correspond to Floquet states that
have, respectively,
$\langle\phi|\hat{a}^{\dagger}\hat{a}|\phi\rangle>28,\,20$. Bottom row
(a)-(d): $\langle E|\hat{a}^{\dagger}\hat{a}|E\rangle$ (black circles)
[$\langle\phi|\hat{a}^{\dagger}\hat{a}|\phi\rangle$ (orange circles)] as a
function of $\tilde{E}$ [$\tilde{e}$] for $\epsilon_{2}/K=20$. The
nonlinearity strengths $(g_{3},g_{4})$ are $(2\times 10^{-5},8\times 10^{-6})$
for (a); $(0.00075,1.27\times 10^{-7})$ for (b); $(0.015,10^{-7})$ for (c);
and $(0.02,10^{-7})$ for (d). Basis size is $N=200$

In Fig. 2, we continue our comparison of the quasienergies (orange lines) of
the driven Hamiltonian in Eq. (2) with the eigenenergies (black lines) of the
effective Hamiltonian $\hat{H}^{(2)}_{\mathrm{eff}}$ in Eq. (5). In the Fig.
2(a) of the top row, the choices of $g_{3}$ and $g_{4}$ lead to $K<0$, while
the Figs. 2(b)-(d) of the top row have $K>0$. Notice that the Fig. 2(b) of the
top row is the same as Fig. 1 and it exhibits almost perfect coincidence
between $\tilde{E}$ and $\tilde{\varepsilon}$. This panel is shown here again
for a good comparison with the other cases.

Spectral kissing is still observed in the Fig. 2(a) of the top row, but the
agreement between $\tilde{E}$ and $\tilde{\varepsilon}$ deteriorates for
larger values of $\epsilon_{2}/K$. In the Figs. 2(c)-(d) of the top row, in
addition to the reduced agreement between the spectrum of
$\hat{H}^{(2)}_{\mathrm{eff}}$ (black lines) and part of the quasienergies of
$\hat{\mathcal{H}}(t)$ (orange points), we also show with gray points the
quasienergies that have no relationship with $\tilde{E}$. These are the
quasienergies that are folded to the first Brillouin zone. Since the
quasienergies are defined modulo $\hbar\omega_{d}/2$, after folding them to
the first Brillouin zone, they get clustered, as seen with those gray points.
This issue becomes more evident in the Figs. 2(c)-(d) of the top row, where
$g_{3}$ is larger than in the Fig. 2(b). To compare $\tilde{\varepsilon}$ and
$\tilde{E}$, visually, we distinguish the quasienergies for the states
associated with a small mean number of photons
$\langle\phi|\hat{a}^{\dagger}\hat{a}|\phi\rangle$ (orange) – smaller than
some maximum value that is set differently depending on $g_{3}$ and $g_{4}$ –
with those that have a large mean number of photons (gray).

Figure 3:  Left panel: Same as Fig. 2(c), but showing also the inverse
participation ratio, ${\cal I}$, of each eigenstate below the ESQPT. The
circle and the square symbols mark the state at $\epsilon_{2}/K=10$ and
$\epsilon_{2}/K=30$, respectively. Middle of the figure: Left (Right) column
gives the Wigner functions for the state marked with a circle (square) on the
left panel; top panel in the column corresponds to the Floquet state, middle
panel in the column corresponds to the Floquet state after the $\hat{U}_{S}$
transformation, and bottom panel in the column corresponds to the eigenstate
of $\hat{H}_{\rm eff}^{(2)}$. Right panel: Average IPR, $\overline{\cal I}$,
for the eigenstates below the ESQPT. All panels: Basis size $N=200$ ,
$g_{3}/\omega_{o}=0.015$ and $g_{4}/\omega_{o}=10^{-7}$.

In addition to the spectral kissing and the divergence of the density of
states, the presence of an ESQPT is also characterized by a discontinuity in
some observables at the ESQPT energy [27, 44]. This can be seen in the bottom
row of Fig. 2, where we show the average number of photons for each excited
state as a function of the rescaled excitation energy (quasienergy)
$\tilde{E}$($\tilde{\varepsilon}$) for $\epsilon_{2}/K=20$ for the same
nonlinearities used in the top row of Fig. 2. The sudden drop in the values of
$\langle\phi|\hat{a}^{\dagger}\hat{a}|\phi\rangle$ at the ESQPT critical
energy is visible in Figs. 2(a)-(b) of the bottom row. This becomes less
evident in the Fig. 2(d) of the bottom row, reflecting the weaker agreement
between eigenvalues and quasienergies in the Fig. 2(d) of the top row.

###  3.2 Floquet States vs Eigenstates

A deeper comparison between the driven system and its corresponding effective
model can be achieved through the analysis of the structure of the Floquet
states of $\hat{\mathcal{H}}(t)$ and the eigenstates of
$\hat{H}^{(2)}_{\mathrm{eff}}$. Due to the importance of the cat-like states
for quantum information processing, we concentrate our analysis on the states
that lie below the ESQPT.

To quantify the proximity of an eigenstate $|E_{k}\rangle$ of the effective
Hamiltonian to a Floquet state, we use the inverse participation ratio (IPR),
which is a measure of the level of delocalization of quantum states (see [45]
and references therein). Here, we define the IPR of state $|E_{k}\rangle$ as
$I_{k}=\sum_{j}|a_{j}|^{4}$, where $a_{j}$ are the coefficients given by the
expansion of the eigenstate in the basis of the Floquet modes, that is,
$|E_{k}\rangle=\sum_{j}a_{j}|\phi_{j}\rangle$. The IPR ranges from $1/N$ for a
completely delocalized state in the given basis ($N$ is the number of basis
states) to 1 for a state that is completely localized in a single basis state.

There is, however, one further aspect that needs to be taken into account.
According to Eq. (3), the Floquet modes $\left|\phi_{j}\right>$ and the
effective Hamiltonian eigenstates $\left|E_{k}\right>$ live in different
reference frames separated by $\hat{S}$. To be able to compare them in the
same frame, we need to compute the IPR defined as

${\cal I}_{k}=\sum_{j}|\langle\phi_{j}|\hat{U}_{S}|E_{k}\rangle|^{4}.$ (6)

In addition to ${\cal I}_{k}$, the other quantity that we use as a figure of
merit is the average

$\overline{{\cal I}}=\frac{1}{n_{b}}\sum_{k=1}^{n_{b}}{\cal I}_{k}$ (7)

where $n_{b}$ $\approx 2\frac{\epsilon_{2}}{\pi K}$ is the number of states of
the effective Hamiltonian below the energy of the ESQPT
($E_{k}\leq\epsilon_{2}^{2}/K$) obtained in [36, 27].

In the left panel of Fig. 3, we show $\overline{{\cal I}}$ (colored lines) as
a function of $\epsilon_{2}/K$ for each eigenstate of $\hat{H}^{(2)}_{\rm
eff}$ with energy below the ESQPT. The parameters are the same as in Fig.
2(c). One sees that as $\tilde{E}$ (black lines) and $\tilde{\varepsilon}$
(orange points) distant themselves in the Fig. 2(c) of the top row, ${\cal I}$
decreases significantly in the left panel of Fig. 3. This separation between
the spectrum of the driven system and of the effective model is more visible
for larger values of the control parameter $\epsilon_{2}/K$ and for larger
energies. In this region, higher orders of the effective Hamiltonian are
needed to get better agreement.

In the two columns of panels in the middle of Fig. 3, we have a closer look at
the structure of two selected states, which are marked with a circle (at
$\epsilon_{2}/K=10$) and a square (at $\epsilon_{2}/K=30$) in the left panel
of Fig. 3. The left column of panels is for Wigner function of the state at
$\epsilon_{2}/K=10$ and the right column for the state at $\epsilon_{2}/K=30$.
The top panels of the columns give the Wigner function for the Floquet state,
the middle panels are for the Floquet state after the $\hat{U}_{S}^{\dagger}$
transformation, and the bottom panels for the eigenstate of $\hat{H}_{\rm
eff}^{(2)}$. After the $\hat{U}_{S}^{\dagger}$ transformation, the Floquet
states approach the eigenstates of $\hat{H}_{\rm eff}^{(2)}$.

Figure 4:  Average IPR, ${\cal I}$, computed for the eigenstates of $H_{\rm
eff}^{(2)}$ in the Floquet basis as a function of $g_{3}/\omega_{o}$ and
$g_{4}/\omega_{o}$, for $\epsilon_{2}/K=10$ (top panel) and
$\epsilon_{2}/K=30$ (bottom panel). The white line in both panels marks $K=0$
and the basis size is $N=150$. The crosses in both panels (labelled a-d in the
top panel) mark the points corresponding to the values of $g_{3}$ and $g_{4}$
used, respectively, in Figs. 2(a)-(d). The red lines in both panels are
constant $K$ lines corresponding to
$K/\omega_{o}=-10^{-5},\,\,10^{5},\,10^{4},\,10^{-3}$, as indicated in the
bottom panel. The dashed orange line in both panels corresponds to the
empirically determined boundary $g_{4}=ag_{3}^{3/4}/(\epsilon_{2}/K)$ with
$a=0.65$. Figure 5:  Trace distance between $\hat{U}_{S}=e^{-i\hat{S}}$ and
the identity operator $\hat{\mathbb{I}}$ as a function of $g_{3}/\omega_{o}$
and $g_{4}/\omega_{o}$ for $\epsilon_{2}/K=10$ (top panel) and
$\epsilon_{2}/K=30$ (bottom panel). $N=100$. The dashed orange line in both
panels corresponds to the empirically determined boundary
$g_{4}=ag_{3}^{3/4}/(\epsilon_{2}/K)$ with $a=0.65$.

In the right panel of Fig. 3, we show the average $\overline{\cal I}$ for the
states below the ESQPT as a function of $\epsilon_{2}/K$. For small
$\epsilon_{2}/K$, reflecting the left panel of Fig. 3, $\overline{\cal
I}\approx 1$. As $\epsilon_{2}/K$ increases, the Floquet states and
eigenstates drift apart, more excited states fall under the well (that is,
$n_{b}$ increases), and $\overline{\cal I}$ gradually decreases.

To gain further insight into the structures of the states and their dependence
on the nonlinearities, we perform in Fig. 4 a systematic study of the mean
value of the IPR, defined in Eq. (7), as a function of $g_{3}$ and $g_{4}$. In
the top panel of Fig. 4, $\epsilon_{2}/K=10$, in the bottom panel,
$\epsilon_{2}/K=30$, and the thick white line in both panels marks the values
of $g_{3}$ and $g_{4}$ that lead to $K=0$. We observe that there is a yellow
region around $K=0$, where ${\cal I}\approx 1$, which indicates that the
eigenstates of $\hat{H}_{\rm eff}^{(2)}$ describe very well the Floquet states
of the driven Hamiltonian in Eq. (2).

Comparing the top and bottom panels of Fig. 4, we see that the yellow region,
indicating the excellent agreement between eigenstates and Floquet states,
decreases as $\epsilon_{2}/K$ increases. This is evident for small values of
$g_{3}$, where the yellow region is bounded by a power-law curve (dashed
orange line) empirically found to be approximately $g_{4}\sim
g_{3}^{3/4}/(\epsilon_{2}/K)$, and for large values of $g_{3}$, where the
region of low values of ${\cal I}$ (blue) grows to the left.

The cross symbols marked with the letters a, b, c, and d in Fig. 4 correspond
toe the values of $g_{3}$ and $g_{4}$ used, respectively, in Figs. 2(a), (b),
(c), and (d). One sees that even though the average IPR in point a, ${\cal
I}\approx 1$, indicates significant localization for $\epsilon_{2}/K=10$ in
the top panel of Fig. 4, the same point shows delocalization (${\cal I}\approx
0$) for $\epsilon_{2}/K=30$ in the bottom panel. This behavior matches what is
observed for the spectrum in the Fig. 2(a) of the top row. The correspondence
with the top row of Fig. 2 holds for the other three points b, c, and d.
Points c and d, in particular, have large values of the $g_{3}$-nonlinearity,
where the agreement between eigenstates and Floquet states are expected to
break down for large $\epsilon_{2}/K$, as indeed verified in the bottom panel
of Fig. 4.

The red lines in Fig. 4 indicate constant values of $K/\omega_{o}$. They serve
as reference for experimental designs, so that one can know for which values
of $g_{3}/\omega_{o}$ and $g_{4}/\omega_{o}$, the constant $K/\omega_{o}$ is
inside a region of agreement between Floquet states and eigenstates.

In the appendix C, we show a figure equivalent to Fig. 4, but for $g_{4}<0$.
This is done, because it can be important for current and future experiments.
In this case, one cannot reach $K=0$, but we verify that the results are very
similar to those found for $g_{4}>0$ in Fig. 4.

The message conveyed by Fig. 4 rich and subtle. One could straightforwardly
infer that inside the yellow region, where $\bar{\cal{I}}\approx 1$ , the
static effective theory describes well the Floquet system. But this is true
provided the transformation $\hat{U}_{S}$ is taken into account. To better
understand this point, we show in Fig. 5 the distance between $\hat{U}_{S}$
and the identity operator $\hat{\mathbb{I}}$, defined as

$d(\hat{U}_{S},\hat{\mathbb{I}})=\frac{1}{2N}\|\hat{U}_{S}-\hat{\mathbb{I}}\|,$
(8)

where $\|\cdot\|$ is the trace norm.

In the regime of parameters where $d(\hat{U}_{S},\hat{\mathbb{I}})\ll 1$ (Fig.
5) we find that all the static effective theory is contained within the
effective Hamiltonian: the complicated frame transformation $\hat{U}_{S}$ away
from the (trivially displaced and rotating) lab frame is negligibly small. In
this regime, a large overlap between states and quasistates
($\bar{\cal{I}}\approx 1$, Fig. 4) is guaranteed.

A sizable $d(\hat{U}_{S},\hat{\mathbb{I}})$ may still allow for a large
overlap, provided the frame transformation modifying the eigenstates of the
effective Hamiltonian is taken into account when comparing them to the Floquet
states. The relevant states in the lab frame may look very different from the
eigenstates of the static effective Hamiltonian. That is, Fig. 5 measures the
distance in between the reference frame in which eq.(2) described the system
and the frame generated by $\hat{S}$ where the static effective description is
valid.

We then remark that Fig. 4 measures the accuracy of the full static effective
theory while Fig. 5 measures the accuracy of the usual static effective
Hamiltonian treatment, which assumes that eq.(2) and the static effective
Hamiltonian describe the system in the same frame [30, 31, 32, 33, 29].

Finally, we observe that the regime of sizable
$d(\hat{U}_{S},\hat{\mathbb{I}})$, the coupling to the environment is itself
modified. Considering the Hamiltonian of the system plus its environment, one
finds that the operator $\hat{U}_{S}$, built iterative to produce a static
effective description of the system alone, forcefully modifies the coupling to
the environment too. This leads to exotic forms of nonlinear-driven
dissipation [46] that are currently being explored as possible culprits for
discrepancies between open quanutm system experiment and theory [47]. Our
analysis provides a different viewpoint on this open problem.

Finally, we observe that the regime of sizable
$d(\hat{U}_{S},\hat{\mathbb{I}})$, the coupling to the environment is itself
modified. Considering the Hamiltonian of the system plus its environment —
whether it’s a measurement device, a coherent control pulse, or a heat
reservoir — it is evident that the operator $\hat{U}_{S}$, which is
iteratively constructed to produce a static effective description of the
system alone, also forcefully modifies the coupling to the environment. This
could lead to measurement infidelity, control anomalies, or exotic forms of
nonlinear-driven dissipation [46]. Presently, this nonlinear-driven
dissipation is under investigation as a potential reason for discrepancies
observed between open quantum system experiments and theoretical models [47].
Our analysis provides a different viewpoint on this open problem.

##  4  Final remarks

We analyzed the conditions under which the time-dependent Hamiltonian
describing a driven superconducting circuit can be approximated by low-order
effective time-independent Hamiltonians. The main takeaway is the observation
that the static effective theory goes beyond the static effective Hamiltonian
treatment, and we introduced a metric to map the parameter space. Our focus
was on the part of the spectrum below the ESQPT, where the states exhibit cat-
like features that are relevant for quantum computing and quantum information
science.

We found that there exists a well-defined region of values of the nonlinearity
parameters $g_{3}$ and $g_{4}$, where the eigenvalues and the eigenstates of
the effective Hamiltonian $\hat{H}_{\textrm{eff}}^{(2)}$ describe correctly
the quasienergies and the Floquet states of the time-dependent system.
However, in the limit of large values of $g_{3}$ or $g_{4}$, the
correspondence breaks down. The phase diagram of the nonlinearities $g_{3}$
and $g_{4}$ that we provided for the analysis of coincidence between the
effective and driven models has practical implications for the design of Kerr
parametric oscillators and other parametric processes that tend to be
overlooked.

An important conclusion of our study is that the effective Hamiltonian
suffices for the comparison between the quasienergies of the driven system and
the eigenvalues of the static effective description, but the comparison
between states requires also the analysis of the unitary transformation
$\hat{U}_{S}$. The static effective theory includes both the effective
Hamiltonian and $\hat{U}_{S}$. Without the latter, one may infer the failure
of a given order of the effective description for parameter values, where it
may actually still hold.

We finish with a brief discussion about the analysis presented in the appendix
B, where we showed that the agreement between the driven and effective model
can hold for larger values of the nonlinearities if one increases the
perturbation order. This raises the question of whether one could expect exact
agreement between the driven and static descriptions for an infinite order.
The answer is negative, because for large nonlinearities and strong drive, the
driven system can develop chaos, as shown elsewhere [48], and in this case,
there is no perturbation order that can lead to agreement with the static
effective Hamiltonian, which is necessarily integrable. Chaos can produce the
collapse of the ESQPT [49], while the static effective description is
integrable by construction.

This work was supported by the NSF CCI grant (Award Number 2124511). D.A.W and
I.G.-M. received support from CONICET (Grant No. PIP 11220200100568CO), UBACyT
(Grant No. 20020170100234BA) and ANCyPT (Grants No. PICT-2020-SERIEA-00740 and
PICT-2020-SERIEA-01082). I.G.-M. received support from CNRS (France) through
the International Research Project (IRP) “Complex Quantum Systems” (CoQSys).

##  Appendix A Fourth-order effective Hamiltonian

Figure 6:  Average IPR, ${\cal I}$, as a function of $g_{3}/\omega_{o}$ and
$g_{4}/\omega_{o}$ computed using $H_{\rm eff}^{(4)}$ for $\epsilon_{2}/K=30$.
The red squares mark the boundary between delocalization and localization set
to $\xi\approx 0.5$. The gray circles and white triangles mark the same
boundary for $H_{\rm eff}^{(2)}$ (as shown in Fig. 4) and $H_{\rm eff}^{(6)}$,
respectively. Basis size is $N=150$.

At fourth order, the effective Hamiltonian includes a four-photon drive and
the first non-squeezing drive term resulting in

$\displaystyle\frac{\hat{H}_{\mathrm{eff}}^{(4)}}{\hbar}=$ $\displaystyle-$
$\displaystyle\Delta^{(4)}\hat{a}^{\dagger}\hat{a}-K^{(4)}\hat{a}^{\dagger
2}\hat{a}^{2}-\lambda^{(4)}\hat{a}^{\dagger 3}\hat{a}^{3}$ (9)
$\displaystyle+$ $\displaystyle\epsilon^{4}_{(4)}\hat{a}^{\dagger
4}+\epsilon_{(4)}^{*}\hat{a}^{4},$

where

$\displaystyle\Delta^{(4)}$
$\displaystyle=\sum_{k=0,1,2}\Delta^{(4)}_{[k]}|\Pi|^{2k},$ (10)
$\displaystyle K^{(4)}$ $\displaystyle=\sum_{k=0,1}K^{(4)}_{[k]}|\Pi|^{2k},$
(11)

and displayed below, are the analytical expressions for all the coefficients
of Eq. (9):

$\displaystyle\begin{split}-\Delta^{(4)}_{[0]}&=\frac{9g_{4}^{2}}{\omega_{a}}+47\frac{g_{3}^{2}g_{4}}{\omega_{a}^{2}}-\frac{6269}{324}\frac{g_{3}^{4}}{\omega_{a}^{3}}\\\
-\Delta^{(4)}_{[1]}&=\frac{54}{5}\frac{g_{4}^{2}}{\omega_{a}}+\frac{671}{10}\frac{g_{3}^{2}g_{4}}{\omega_{a}^{2}}+\frac{113}{360}\frac{g_{3}^{4}}{\omega_{a}^{3}}\\\
-\Delta^{(4)}_{[2]}&=-\frac{9}{2}\frac{g_{4}^{2}}{\omega_{a}}+\frac{15113}{600}\frac{g_{3}^{2}g_{4}}{\omega_{a}^{2}}-\frac{297947}{32400}\frac{g_{3}^{4}}{\omega_{a}^{3}}\\\
-K^{(4)}_{[0]}&=\frac{153}{16}\frac{g_{4}^{2}}{\omega_{a}}+\frac{225}{4}\frac{g_{3}^{2}g_{4}}{\omega_{a}^{2}}+\frac{805}{36}\frac{g_{3}^{4}}{\omega_{a}^{3}}\\\
-K^{(4)}_{[1]}&=\frac{27}{5}\frac{g_{4}^{2}}{\omega_{a}}+\frac{671}{20}\frac{g_{3}^{2}g_{4}}{\omega_{a}^{2}}+\frac{113}{720}\frac{g_{3}^{4}}{\omega_{a}^{3}}\\\
-\lambda^{(4)}&=\frac{17}{8}\frac{g_{4}^{2}}{\omega_{a}}+\frac{25}{2}\frac{g_{3}^{2}g_{4}}{\omega_{a}^{2}}+\frac{805}{162}\frac{g_{3}^{4}}{\omega_{a}^{3}}\\\
\epsilon_{4}^{(4)}&=\left(\frac{33}{8}\frac{g_{4}^{2}}{\omega_{a}}-\frac{101}{96}\frac{g_{3}^{2}g_{4}}{\omega_{a}}-\frac{2009}{1296}\frac{g_{3}^{4}}{\omega_{a}^{3}}\right)\Pi^{2}.\end{split}$
(12)

##  Appendix B Convergence of the static effective theory

In the main text, we focused on the static effective description of the Kerr
parametric oscillator to the first nontrivial order, which corresponds to the
second-order effective Hamiltonian $\hat{H}^{(2)}_{\rm eff}$. In this
appendix, we evaluate the convergence of the high order static effective
theory when applied to the Kerr parametric oscillator. This is done by
comparing the fourth-order (see appendix A) and the sixth-order static
effective Hamiltonians with the driven Hamiltonian in Eq. (2).

Figure 7:  Average IPR, ${\cal I}$, as a function of $g_{3}/\omega_{o}$
computed using $\hat{H}_{\rm eff}^{(2)}$, $\hat{H}_{\rm eff}^{(4)}$, and
$\hat{H}_{\rm eff}^{(6)}$, as indicated, for $N=100$, $\epsilon_{2}/K=30$ and
$g_{4}/\omega_{o}=10^{-7}$.

Motivated by experimental developments in the field of quantum circuits, two
methods to go beyond this first-order theory in a systematic manner were
derived [3, 14]. These approaches, useful to explain experimental data and
numerical simulations [3, 14], requires a symbolic computer program to carry
out the analytical calculation, since the number of terms is far too great to
write down by hand.

Figure 8:  Average IPR, ${\cal I}$, computed for the eigenstates of $H_{\rm
eff}^{(2)}$ in the Floquet basis as a function of $g_{3}/\omega_{o}$ and
$|g_{4}/\omega_{o}|$, ($g_{4}<0$), for $\epsilon_{2}/K=10$ (top panel) and
$\epsilon_{2}/K=30$ (bottom panel). Basis size $N=200$. The red lines in both
panels are constant $K$ lines corresponding to
$K/\omega_{o}=10^{-5},\,10^{-4}$, and $10^{-3}$, as marked in the bottom
panel. The dashed orange line in both panels corresponds to the empirically
determined boundary $g_{4}=ag_{3}^{3/4}/(\epsilon_{2}/K)$ with $a=0.65$.

In Fig. 6, we show ${\cal I}$ computed for the eigenstates of the static
effective Hamiltonian $\hat{H}^{(4)}_{\rm eff}$ (see appendix‘A) in comparison
with the Floquet states obtained from ${\cal H}(t)$. The results are
qualitatively similar to those seen for $\hat{H}^{(2)}_{\rm eff}$ in Fig. 4,
but there are quantitative differences. To illustrate these differences, we
mark in Fig. 6 the place where, for small values of $g_{3}$, we get ${\cal
I}\approx 0.5$ for the second-order static effective Hamiltonian
$\hat{H}^{(2)}_{\rm eff}$ (gray circles), the fourth-order $\hat{H}^{(4)}_{\rm
eff}$ (red squares), and the sixth-order $\hat{H}^{(6)}_{\rm eff}$ sixth order
(white triangles). We see that the region of disagreement between Floquet
states and eigenstates (blue region) decreases as the order increases.

In Fig. 7, we show the average IPR, ${\cal I}$, as a function of $g_{3}$, for
a fixed value $g_{4}=10^{-7}$ (note that for large $g_{3}$, $\tilde{\cal I}$
does not depend on 4 ), for $\hat{H}_{\rm eff}^{(2)}$, $\hat{H}_{\rm
eff}^{(4)}$, and $\hat{H}_{\rm eff}^{(6)}$. The horizontal line marks the
point (circle) in the curves where ${\cal I}\approx 0.5$. One sees that the
value of $g_{3}$ for ${\cal I}\approx 0.5$ gets displaced to the right,
effectively enlarging the area where there is good agreement between the
eigenstates and Floquet states below the ESQPT. However, the rate of
convergence of this expansion is slow and comes at the cost of highly complex
expressions.

##  Appendix C Spectrum for negative $g_{4}$

For completeness and because it can be important for present and future
experiments, we show in Fig. 8 the average IPR, equivalently to what was done
in Fig. 4, but now for $g_{4}<0$. In this case, one cannot reach $K=0$, but
the results are very similar to those found for $g_{4}>0$.

## References

  * Kapitza [1951] P. L. Kapitza, Soviet Phys. JETP 21, 588–592 (1951).
  * Landau and Lifshitz [1976] L. D. Landau and E. M. Lifshitz,  _Mechanics: Volume 1_ , Vol. 1 (Butterworth-Heinemann, 1976).
  * Venkatraman  _et al._ [2022a] J. Venkatraman, X. Xiao, R. G. Cortiñas, A. Eickbusch, and M. H. Devoret, Phys. Rev. Lett. 129, 100601 (2022a).
  * Wang and Safavi-Naeini [2023] Z. Wang and A. H. Safavi-Naeini, “Quantum control and noise protection of a Floquet $0-\pi$ qubit,” (2023), arXiv:2304.05601 [quant-ph] .
  * Paul [1990] W. Paul, Rev. Mod. Phys. 62, 531 (1990).
  * Goldman and Dalibard [2014] N. Goldman and J. Dalibard, Phys. Rev. X 4, 031027 (2014).
  * Wineland [2013] D. J. Wineland, Rev. Mod. Phys. 85, 1103 (2013).
  * Bruzewicz  _et al._ [2019] C. D. Bruzewicz, J. Chiaverini, R. McConnell, and J. M. Sage, Applied Physics Reviews 6, 021314 (2019).
  * Magnus [1954] W. Magnus, Commun Pure Appl Math 7, 649 (1954).
  * Fer [1958] F. Fer, Bull. Classe Sci. Acad. R. Bel. 21, 818 (1958).
  * Ernst  _et al._ [1994] R. R. Ernst, G. Bodenhausen, and A. Wokaun,  _Principles of Nuclear Magnetic Resonance in One and Two Dimensions_ (Oxford University Press, Oxford, 1994).
  * Haeberlen [2012] U. Haeberlen,  _High Resolution NMR in Solids Selective Averaging: Supplement 1 Advances in Magnetic Resonance_, Advances in magnetic resonance. Supplement (Elsevier Science, 2012).
  * Wilcox [1967] R. M. Wilcox, J. Math. Phys. 8, 962 (1967).
  * Xiao  _et al._ [2023] X. Xiao, J. Venkatraman, R. G. Cortiñas, S. Chowdhury, and M. H. Devoret, “A diagrammatic method to compute the effective Hamiltonian of driven nonlinear oscillators,” (2023), arXiv:2304.13656 [quant-ph] .
  * Marthaler and Dykman [2006] M. Marthaler and M. I. Dykman, Phys. Rev. A 73, 042108 (2006).
  * Marthaler and Dykman [2007] M. Marthaler and M. I. Dykman, Phys. Rev. A 76, 010102 (2007).
  * Dykman [2012] M. Dykman,  _Fluctuating nonlinear oscillators: from nanomechanics to quantum superconducting circuits_ (Oxford University Press, 2012).
  * Wustmann and Shumeiko [2013] W. Wustmann and V. Shumeiko, Phys. Rev. B 87, 184501 (2013).
  * Krantz  _et al._ [2016] P. Krantz, A. Bengtsson, M. Simoen, S. Gustavsson, V. Shumeiko, W. Oliver, C. Wilson, P. Delsing, and J. Bylander, Nature communications 7, 11417 (2016).
  * Frattini  _et al._ [2017] N. Frattini, U. Vool, S. Shankar, A. Narla, K. Sliwa, and M. Devoret, App. Phys. Lett. 110, 222603 (2017).
  * Cochrane  _et al._ [1999] P. T. Cochrane, G. J. Milburn, and W. J. Munro, Phys. Rev. A 59, 2631 (1999).
  * Goto [2016] H. Goto, Scientific Reports 6, 21686 (2016).
  * Goto [2019] H. Goto, Journal of the Physical Society of Japan 88, 061015 (2019).
  * Goto and Kanao [2021] H. Goto and T. Kanao, Phys. Rev. Research 3, 043196 (2021).
  * Puri  _et al._ [2020] S. Puri, L. St-Jean, J. A. Gross, A. Grimm, N. E. Frattini, P. S. Iyer, A. Krishna, S. Touzard, L. Jiang, A. Blais, S. T. Flammia, and S. M. Girvin, Sci. Adv. 6, 5901 (2020).
  * Wielinga and Milburn [1993] B. Wielinga and G. J. Milburn, Phys. Rev. A 48, 2494 (1993).
  * Chávez-Carlos  _et al._ [2023] J. Chávez-Carlos, T. L. Lezama, R. G. Cortiñas, J. Venkatraman, M. H. Devoret, V. S. Batista, F. Pérez-Bernal, and L. F. Santos, npj Quantum Information 9, 76 (2023).
  * Reynoso  _et al._ [2023] M. A. P. Reynoso, D. J. Nader, J. Chávez-Carlos, B. E. Ordaz-Mendoza, R. G. Cortiñas, V. S. Batista, S. Lerma-Hernández, F. Pérez-Bernal, and L. F. Santos, “Quantum tunneling and level crossings in the squeeze-driven Kerr oscillator,” (2023), arXiv:2305.10483 [quant-ph] .
  * Wang  _et al._ [2019] Z. Wang, M. Pechal, E. A. Wollack, P. Arrangoiz-Arriola, M. Gao, N. R. Lee, and A. H. Safavi-Naeini, Phys. Rev. X 9, 021049 (2019).
  * Grimm  _et al._ [2020] A. Grimm, N. E. Frattini, S. Puri, S. O. Mundhada, S. Touzard, M. Mirrahimi, S. M. Girvin, S. Shankar, and M. H. Devoret, Nature 584, 205 (2020).
  * Frattini  _et al._ [2022] N. E. Frattini, R. G. Cortiñas, J. Venkatraman, X. Xiao, Q. Su, C. U. Lei, B. J. Chapman, V. R. Joshi, S. Girvin, R. J. Schoelkopf, _et al._ , arXiv preprint arXiv:2209.03934 (2022).
  * Venkatraman  _et al._ [2022b] J. Venkatraman, R. G. Cortinas, N. E. Frattini, X. Xiao, and M. H. Devoret, “Quantum interference of tunneling paths under a double-well barrier,”  (2022b), arXiv:2211.04605 [quant-ph] .
  * Iyama  _et al._ [2023] D. Iyama, T. Kamiya, S. Fujii, H. Mukai, Y. Zhou, T. Nagase, A. Tomonaga, R. Wang, J.-J. Xue, S. Watabe, S. Kwon, and J.-S. Tsai, “Observation and manipulation of quantum interference in a superconducting Kerr parametric oscillator,” (2023), arXiv:2306.12299 [quant-ph] .
  * Koch  _et al._ [2007] J. Koch, T. M. Yu, J. Gambetta, A. A. Houck, D. I. Schuster, J. Majer, A. Blais, M. H. Devoret, S. M. Girvin, and R. J. Schoelkopf, Phys. Rev. A 76, 042319 (2007).
  * Girvin [2014] S. M. Girvin, Quantum machines: measurement and control of engineered quantum systems , 113 (2014).
  * Puri  _et al._ [2017] S. Puri, S. Boutin, and A. Blais, npj Quantum Information 3, 1 (2017).
  * Chamberland  _et al._ [2022] C. Chamberland, K. Noh, P. Arrangoiz-Arriola, E. T. Campbell, C. T. Hann, J. Iverson, H. Putterman, T. C. Bohdanowicz, S. T. Flammia, A. Keller, G. Refael, J. Preskill, L. Jiang, A. H. Safavi-Naeini, O. Painter, and F. G. Brandão, PRX Quantum 3, 010329 (2022), publisher: American Physical Society.
  * Ruiz  _et al._ [2023] D. Ruiz, R. Gautier, J. Guillaud, and M. Mirrahimi, Phys. Rev. A 107, 042407 (2023).
  * Gautier  _et al._ [2022] R. Gautier, A. Sarlette, and M. Mirrahimi, PRX Quantum 3, 020339 (2022).
  * Putterman  _et al._ [2022] H. Putterman, J. Iverson, Q. Xu, L. Jiang, O. Painter, F. G. Brandão, and K. Noh, Phys. Rev. Lett. 128, 110502 (2022), publisher: American Physical Society.
  * Shirley [1965] J. H. Shirley, Phys. Rev. 138, B979 (1965).
  * Wisniacki [2014] D. A. Wisniacki, Europhysics Lett. 106, 60006 (2014).
  * Mirrahimi  _et al._ [2014] M. Mirrahimi, Z. Leghtas, V. V. Albert, S. Touzard, R. J. Schoelkopf, L. Jiang, and M. H. Devoret, New Journal of Physics 16, 045014 (2014).
  * Santos  _et al._ [2016] L. F. Santos, M. Távora, and F. Pérez-Bernal, Phys. Rev. A 94, 012113 (2016).
  * Evers and Mirlin [2008] F. Evers and A. D. Mirlin, Rev. Mod. Phys. 80, 1355 (2008).
  * Dykman and Krivoglaz [1975] M. I. Dykman and M. A. Krivoglaz, Physica Status Solidi (B) 68, 111 (1975).
  * Venkatraman  _et al._ [2022c] J. Venkatraman, X. Xiao, R. G. Cortiñas, and M. H. Devoret, “On the static effective Lindbladian of the squeezed Kerr oscillator,” (2022c), arXiv:2209.11193 [quant-ph] .
  * [48] J. Chávez-Carlos, M. A. P. Reynoso, R. G. Cortiñas, I. García-Mata, M. Devoret, V. S. Batista, F. Pérez-Bernal, D. A. Wisniacki, and L. F. Santos, “Transmons: from qubits to quantum chaos generators,” (in preparation).
  * García-Mata  _et al._ [2021] I. García-Mata, E. Vergini, and D. A. Wisniacki, Phys. Rev. E 104, L062202 (2021).


# Understanding the Effects of Noise in Text-to-SQL:
An Examination of the BIRD-Bench Benchmark

Niklas Wretblad1,* Fredrik Gordh Riseby1,* Rahul Biswas2
Amin Ahmadi2 Oskar Holmström1
1Linköping University, 2Silo AI
<EMAIL_ADDRESS>
###### Abstract

Text-to-SQL, which involves translating natural language into Structured Query
Language (SQL), is crucial for enabling broad access to structured databases
without expert knowledge. However, designing models for such tasks is
challenging due to numerous factors, including the presence of ’noise,’ such
as ambiguous questions and syntactical errors. This study provides an in-depth
analysis of the distribution and types of noise in the widely used BIRD-Bench
benchmark and the impact of noise on models. While BIRD-Bench was created to
model dirty and noisy database values, it was not created to contain noise and
errors in the questions and gold queries. We found that noise in questions and
gold queries are prevalent in the dataset, with varying amounts across
domains, and with an uneven distribution between noise types. The presence of
incorrect gold SQL queries, which then generate incorrect gold answers, has a
significant impact on the benchmark’s reliability. Surprisingly, when
evaluating models on corrected SQL queries, zero-shot baselines surpassed the
performance of state-of-the-art prompting methods. We conclude that
informative noise labels and reliable benchmarks are crucial to developing new
Text-to-SQL methods that can handle varying types of noise.

Understanding the Effects of Noise in Text-to-SQL:
An Examination of the BIRD-Bench Benchmark

Niklas Wretblad1,* Fredrik Gordh Riseby1,* Rahul Biswas2 Amin Ahmadi2 Oskar
Holmström1 1Linköping University, 2Silo AI<EMAIL_ADDRESS>

##  1 Introduction

Text-to-SQL††*Equal Contribution through large language models facilitates
broader access to structured databases without requiring expert knowledge. To
develop such models, high-quality open datasets and benchmarks are essential
resources, and over the years, several benchmarks and datasets have been
created. Early benchmarks, such as WikiSQL Zhong et al. (2017), modeled simple
scenarios, often with single-table queries, and following datasets attempts to
closer approximate real-world scenarios: complex queries with join-statements
over several tables Yu et al. (2018), unseen domain-specific datasets Gan et
al. (2021b) Lee et al. (2021), and noisy questions Gan et al. (2021a). BIRD-
Bench, a recent and challenging benchmark, aims to further close the gap
between text-to-SQL research and real-world applications by for example
containing large and dirty database values and requiring external knowledge Li
et al. (2023).

While BIRD-Bench does not explicitly introduce noise to the questions in the
data, it could be that it is added inadvertently due to human error. For the
same reason, noise is an essential aspect of real-world use cases, as human
inputs often are ambiguous and contain syntactical errors. However, for the
benchmark to be a helpful tool for judging model properties, such as noise
handling, the data must be valid and inform us in what areas a model can be
improved.

Figure 1: Example of an incorrect SQL query that generates the wrong gold
reference answer for the given question. The JOIN operation incorrectly
matches clients and accounts by district_id. Due to the possibility of
multiple clients and accounts in the same district, accounts are incorrectly
associated with the wrong users.

This paper continues the tradition of examining the suitability and
limitations of open datasets and benchmarks. We specifically focus on how
noise is represented in questions and queries in BIRD-Bench. We perform a
qualitative analysis of what types of noise exist in the data and the noise
distribution in specific domains. We then study the effects of noise on
different models and prompting techniques, using both strong baselines and
state-of-the-art methods.

We find that noise in questions and queries is prevalent, noise is unevenly
distributed across domains, and categories of noise types are represented
unequally in the data. Errors in gold SQL queries are also common and decrease
the reliability of BIRD-Bench. When evaluating models on a dataset with
corrected gold queries, the performance gap between zero-shot baselines and
state-of-the-art prompting techniques is closed, questioning how we should
interpret model performance on BIRD-Bench.

##  2 Related Work

#### Datasets

WikiSQL is a large Text-to-SQL dataset containing only simple SELECT and WHERE
operations without nested queries or JOIN operations Zhong et al. (2017).
SPIDER Yu et al. (2018) was later developed to approximate real-life scenarios
more closely, requiring models to construct complex queries and understand the
database schema. While complexity is a critical aspect of real use cases,
variations of SPIDER have been created to contain noisy questions Gan et al.
(2021a) and domain-specific questions Gan et al. (2021b). BIRD-Bench was
created to close the gap between academic research and real-world applications
by introducing large and dirty database values, questions requiring external
knowledge and optimizing SQL execution efficiency Li et al. (2023).

#### Text-to-SQL Methods

The notable gap in accuracy between automated systems (65.45%) and human
experts (92.96%)111BIRD-Bench benchmark as of 2024-02-16 (https://bird-
bench.github.io), highlights the need for ongoing developments in Text-to-SQL
models.

Different approaches have been taken to create models capable of Text-to-SQL
generation. A more traditional approach is to finetune LLMs on Text-to-SQL
examples. While these models offer promising results, there is a performance
gap to instruction-tuned LLMs, in particular GPT-4, that is adapted to the
Text-to-SQL task through prompt engineering Li et al. (2023). Prompts are
often chained, where each prompt is applied to the task sub-problems, such as
schema linking, decomposition of queries, and refinement of model generations
Pourreza and Rafiei (2023); Wang et al. (2023).

#### Noise in Datasets

The contemporaneous work of Wang et al. (2023) shows that ambiguous questions
and incorrect SQL queries exist in BIRD-Bench. However, unlike our work, they
do not study how noise varies across domains or how the identified noise and
errors affect model performance. Katsogiannis-Meimarakis and Koutrika (2023)
points out that database schemas often misalign with data entities, which may
cause lexical or syntactic ambiguities affecting text-to-SQL models.

Statistic | Financial | California Schools | Superhero | Toxicology | Thrombosis Prediction
---|---|---|---|---|---
Data points with noise | 52/106 (49%) | 9/20 (45%) | 3/20 (15%) | 7/20 (35%) | 8/20 (40%)
Noisy questions | 44/106 (41.5%) | 5/20 (25%) | 2/20 (10%) | 6/20 (30%) | 3/20 (15%)
Erroneous gold queries | 22/106 (20.7%) | 8/20 (40%) | 1/20 (5%) | 2/20 (10%) | 6/20 (30%)
Table 1: Statistics of the total amount of data points that contains errors and the amount of errors in questions and gold queries across five datasets. Note that a data point can have errors in both the question and SQL query. Noise Type | Financial | California Schools | Superhero | Toxicology | Thrombosis Prediction
---|---|---|---|---|---
Spelling/Syntactical Errors | 23 | 2 | 1 | 4 | 2
Vague/Ambiguous Questions | 17 | 1 | 1 | 1 | 1
Incorrect SQL | 22 | 8 | 1 | 2 | 6
Synonyms | 2 | 0 | 0 | 0 | 0
String Capitalization | 7 | 0 | 0 | 0 | 0
Question does not map to DB | 1 | 4 | 1 | 0 | 0
Total number of errors | 72 | 15 | 4 | 7 | 9
Table 2: Distribution of different types of noise encountered in the domains.

##  3 Method

###  3.1 Data

The BIRD-Bench dataset Li et al. (2023) is used in this paper as it is a
recent and widely used dataset that is the most similar to real world
scenarios. Since BIRD contains 12,751 samples across many domains and because
of time-consuming human annotation, the main focus of the analysis is on the
financial domain222This was also motivated by the fact this paper was a
collaborative endeavor with the Swedish bank SEB., which includes queries
related to banking operations.

The development set of the financial domain contains 106 question and SQL
query pairs, which represent approximately 7.5% of the data points in the
development set and are structured around eight distinct tables, presented in
full in Appendix A.1. Each question is annotated with a difficulty level
(simple, moderate, and challenging). The specific distribution is found in
Figure 4, in Appendix A.2.

We selected four additional domains to validate our noise analysis of the
financial domain and performed the same analysis on 20 randomly sampled
questions from each domain. The selection was based on question difficulties
and model accuracy of DIN-SQL333Results of DIN-SQL across domains were
provided by the creators of DIN-SQL. on each of the domains, as presented in
Figure 4. We selected California Schools with low accuracy and simple
questions, Superhero with high accuracy and simple questions, Toxicology with
similar accuracy to the financial domain but more complex questions, and
Thrombosis Prediction with low accuracy and moderately difficult questions.

###  3.2 Annotation of Noise

All questions and SQL queries in the selected domains were annotated to
determine whether they contained errors. The annotations were performed
independently by two authors of this paper, fluent in English and experts in
SQL. The identified errors were grouped based on similarity and named after
the errors’ common properties, as shown in Table 2. The annotations were then
used to generate two distinct datasets: one where SQL was corrected, and one
where both SQL queries and noisy questions were corrected.

Annotations and cleaned datasets is published after the anonymity period.

###  3.3 Models and Prompt Techniques

Two models, GPT-3.5 and GPT-4, were used with three different prompting
methods: zero-shot prompting as a baseline and the more advanced DIN-SQL
Pourreza and Rafiei (2023) and MAC-SQL Wang et al. (2023). We used GPT-3.5 and
GPT-4 for zero-shot prompting, but for the advanced prompting techniques, we
only used GPT-3.5 since chaining prompts with GPT-4 was outside of the
available resources for this project. We chose the models and prompting
methods because they were the highest-performing publicly available models on
BIRD-Bench at the time of writing.

Information about the database schema is crucial to generating correct queries
for BIRD-Bench questions. DIN-SQL and MAC-SQL has a predefined format for
adding the database schema. For the zero-shot model, we provide the database
schema in-context in the form of SQL table creation statements, as this has
been shown to improve accuracy compared to other formats Nan et al. (2023).
The prompt template for the zero-shot model is found in Appendix A.3. The code
base is published after the anonymity period.

Figure 2: Accuracy of various models on Bird-Bench’s financial domain. Models
are evaluated on the original data (left), corrected SQL queries (middle), and
corrected SQL queries and corrected noisy questions.

##  4 Qualitative Analysis of Noise

Even though BIRD-Bench is not intended to represent noise in questions and SQL
queries, our analysis reveals that noise exists in all studied domains to
different extents. The financial domain exhibits the highest levels of noise
at 49% closely followed by the California Schools domain at 45%, as shown in
Table 1. In contrast, the Superhero domain demonstrated the lowest noise
levels, with only 15% of data points containing errors. As presented in
Section 3.1 and Figure 4, the Superhero domain had the highest accuracy while
having a similar distribution of question difficulties. This could indicate
that model accuracy across tasks correlates with noise, which implies that
noise in questions and SQL queries need to be carefully considered during
dataset design.

The categories and absolute frequency of noise per dataset are presented in
Table 2, and both examples and descriptions of the noise types are presented
in Appendix A.4. Our analysis shows that spelling/syntactic errors and
incorrect SQL queries were most prevalent in the financial domain. The
presence of noise in questions is not necessarily undesirable, as it more
closely mimics real-life scenarios. However, noise distribution across the
categories is unequal. While this could be the real-world distribution, it
might unfairly bias the benchmark towards models better at handling
syntactical errors. Given the uneven distribution of errors and the lack of
noise labels, the benchmark does not inform which noise types are challenging
for current models and how models should improve.

A more severe issue is that all domains contained incorrect SQL queries, which
are used for generating gold reference answers. An example of an erroneous SQL
query is shown in Figure 1. These types of errors question the reliability of
the benchmark to accurately determine model performance, which is explored in
the next section.

##  5 Impact of Noise on Model Performance

We apply models to the original dataset, a dataset where SQL has been
corrected, and a dataset where both SQL queries and noisy questions have been
corrected. Figure 2 presents the results of a single evaluation for all models
on all datasets.

MAC-SQL slightly outperforms DIN-SQL and the zero-shot baselines on the
original dataset, where noise exists in both questions and queries. However,
correcting SQL queries decreases MAC-SQL’s performance, tying it with DIN-SQL
as the poorest performers. Surprisingly, even the zero-shot GPT-3.5 baseline
outperforms the more advanced DIN-SQL and MAC-SQL. The dataset with corrected
SQL queries could also be considered optimal since gold labels are correct and
noise in questions is represented. Given the drastic re-ranking of models, it
is relevant to question if BIRD-Bench is a reliable assessor of models and a
useful tool to assist researchers in developing new methods for Text-to-SQL.

When evaluating models on the dataset with both questions and SQL queries
corrected, the accuracy of all models increases significantly. While zero-shot
GPT-4 performs the best, the remaining models perform similarly with DIN-SQL
slightly ahead. Compared to the ideal scenario where only SQL queries are
corrected, the presence of noise noticeably impacts all models’ accuracy.
However, models are not equally affected by noise as some models have a more
pronounced increase in accuracy. This might suggests that the types of noise
also affects models differently, which needs to be studied further.

##  6 Conclusions and Future Work

This paper analyzed the quality and distribution of noise in the BIRD-Bench
benchmark for Text-to-SQL. We show that noise in both questions and SQL
queries are prevalent, and noise is unevenly distributed across noise types
and domains. Errors in gold SQL queries were common, decreasing the
reliability of BIRD-Bench. Surprisingly, when evaluating models on corrected
gold queries, zero-shot baselines surpassed more advanced prompting
techniques. These findings highlight the necessity for developing benchmarks
that can guide researchers in designing models that are more resistant to
noise. Therefore, a significant improvement would be to label noise types
across the dataset. In future work, we plan to study how large language models
can be applied to noise classification, a new task that could also be critical
in systems where Text-to-SQL is employed.

Overall, this study provides a deeper understanding of how noise is expressed
in Text-to-SQL tasks and how noise and models interact, pinpointing areas for
improvement in the BIRD-Bench dataset.

## Limitations

While our study provides valuable insights regarding the influence of dataset
noise in Text-to-SQL translation tasks, it has several limitations. As the
analysis was performed mainly on the BIRD-Bench dataset’s financial domain,
our findings’ generalizability may be limited. We only examined a small subset
of other domains to validate our findings, which may represent only some of
the noise distribution across domains.

Additionally, annotators may have introduced subjective bias during noise
annotation, even though we attempt to minimize this by having two independent
annotators. Further, our decision to categorize noise into six specific
classes might have oversimplified the complexity and diversity of noise types
in these benchmarks.

Our choice of models and prompting techniques could also be a potential
limitation. We only employed two models, GPT-3.5 and GPT-4, and three
different prompting methods. Evaluating a more comprehensive array of models
and prompting techniques might have given a more comprehensive understanding
of their performance under the influence of noise.

Lastly, the substantial effort required to correct SQL queries and noisy
questions in the dataset may have introduced errors despite the review
process. This might influence the model performances we report when evaluating
models on the corrected datasets.

## Acknowledgments

We extend our gratitude to Mohammadreza Pourreza for the results from the DIN-
SQL model. We are also grateful to SEBx for their generous support and the
provision of resources. Additionally, this research was partial funded by the
National Graduate School of Computer Science in Sweden (CUGS).

## References

  * Gan et al. (2021a) Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Pengsheng Huang. 2021a.  Towards robustness of text-to-SQL models against synonym substitution.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2505–2515, Online. Association for Computational Linguistics.
  * Gan et al. (2021b) Yujian Gan, Xinyun Chen, and Matthew Purver. 2021b.  Exploring underexplored limitations of cross-domain text-to-SQL generalization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8926–8931, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Katsogiannis-Meimarakis and Koutrika (2023) George Katsogiannis-Meimarakis and Georgia Koutrika. 2023.  A survey on deep learning approaches for text-to-sql.  _The VLDB Journal_ , 32(4):905–936.
  * Lee et al. (2021) Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021.  KaggleDBQA: Realistic evaluation of text-to-SQL parsers.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2261–2273, Online. Association for Computational Linguistics.
  * Li et al. (2023) Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023.  Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.  In _Advances in Neural Information Processing Systems_.  Spotlight Poster.
  * Nan et al. (2023) Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023.  Enhancing text-to-SQL capabilities of large language models: A study on prompt design strategies.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 14935–14956, Singapore. Association for Computational Linguistics.
  * Pourreza and Rafiei (2023) Mohammadreza Pourreza and Davood Rafiei. 2023.  Din-sql: Decomposed in-context learning of text-to-sql with self-correction.  In _Advances in Neural Information Processing Systems 36_.  Accepted for poster presentation, full citation details to be updated.
  * Wang et al. (2023) Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian-Wen Zhang, Zhao Yan, and Zhoujun Li. 2023.  Mac-sql: A multi-agent collaborative framework for text-to-sql.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher. 2017.  Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning.  _arXiv e-prints_ , page arXiv:1709.00103.

##  Appendix A Appendix

###  A.1 Database Schema of the Financial Domain

Figure 3: Database schema of the database in the financial domain of BIRD-
Bench.

Figure 3 displays the database schema for the financial domain. This schema
contains various tables, such as those for loans, transactions, accounts,
cards and clients, all reflecting the financial orientation of the database.
Descriptions of what information these tables contain are presented in Table
3. The database consists of 55 columns distributed across eight distinct
tables. While the majority of the column names are intuitively understandable,
some present interpretative challenges, as evident in the schema. An
illustrative example is the district table, which incorporates 16 unique
columns. This includes a column titled district_ID along with 15 other
columns, ranging from A2 to A16. The latter columns’ names do not readily
convey the nature of the data they hold, making them less intuitive to
understand. In practice, a database schema will often be accompanied by a data
dictionary or documentation that explains each table and column in detail.
Such documentation would typically provide the context needed to fully
understand the meaning of each element in the schema, the range of possible
values for fields with unspecified types, and the business logic underlying
the relationships. Without this additional documentation, fully interpreting
and effectively using the database can be challenging as illustrated by the
column names in the districts table. The BIRD-Bench dataset includes a unique
feature for each question termed hint. This feature is designed to offer
insights or supplementary information corresponding to the specifics detailed
in such database documentation. This feature is provided to all models
described in 3.3 for each question during the experiments.

Table 3: Table descriptions of the tables in the database of the financial domain of BIRD-Bench. Table Name | Description
---|---
loan | Contains details of loans.
order | Holds information about monetary orders.
trans | Represents financial transactions.
account | Contains account information.
disp | Links clients to accounts (dispositions).
card | Contains details about cards issued.
client | Holds client information.
district | Contains details about districts or regions.

Further, the lines in Figure 3 between the tables represent relationships,
where the nature of the relationship is indicated by the shape of the tail end
of the lines where they connect to each table. A one-to-many relationship is
indicated by the line beginning with a single line and the one digit above it,
and then ending in a crow’s foot (three lines) at the opposite end. For
example, an account can have multiple orders, transactions, dispositions, and
loans associated with it, but each of those entities is only linked to one
account. An account can have many loans, but one loan is exclusively only
linked to one account, which makes sense. Further, clients and accounts are
related through the disposition table in a many-to-many relationship. An
account can have many different clients associated with it, for example, one
client listed as the owner of the account and multiple other clients listed as
users for the account. This could for example be practical for sharing an
account in a family, where one parent could be the owner of the account and
then multiple other family members listed as users. A single client can also
be related to many different accounts in the other way around.

###  A.2 Distribution of question difficulty and model accuracy

Figure 4: Distribution of question difficulties and execution accuracy of the
DIN-SQL model on the different domains of the BIRD-Bench development set.

###  A.3 Prompt Templates

⬇

1"""Database schema in the form of CREATE_TABLE statements:

2

3{database_schema}

4

5Using valid SQL, answer the following question based on the

6tables provided above.

7

8Hint helps you to write the correct sqlite SQL query.

9Question: {question}

10Hint: {evidence}

11DO NOT return anything else except the SQL query."""

Listing 1: Zero-Shot Prompting Template.

The prompt template underlying the zero-shot models described in Section 3.3
can be found in Listing 1. The prompt integrates a given question, the
associated database schema, an instruction directing the LLM to generate valid
SQL, and a hint provided by the BIRD-Bench dataset. The hint is designed to
offer insights or supplementary information needed in order to accurately
interpret the database schema and to correctly convert the question into a SQL
query. Note that the other models implemented in this research is also
provided with this feature.

###  A.4 Examples of Errors and Corrections

This section provides examples of erroneous data points and their corrections
from the different error categories found in Table 1.

### Example 1: Spelling/Syntactical Error

In Figure 5, an example question with a syntactical error is provided,
representing the question with ID 125 from the financial domain in the BIRD-
Bench development set. The grammatical structure of the question complicates
the interpretation of its meaning for a human reader and makes it difficult to
understand which information it is asking for. Therefore, there is a chance
that an LLM might also misinterpret the question. A corrected version of the
question can be seen in the figure.

Figure 5: Question with ID 125 from the development set of BIRD-Bench which
contains syntactical errors and a corrected version of the question.

### Example 2: Ambiguous/Vague Question

Figure 6 displays the data point with ID 159 from the financial domain of the
development set of BIRD-Bench. It contains an error which were grouped into
the ambiguous/vague question category. The challenge lies in the natural
language question’s ambiguity, specifically in the phrase “List all the
withdrawals…” This ambiguity revolves around determining which columns to
return when executing the SQL query.

Figure 6: Question, gold SQL query and a corrected version of the question
corresponding to the data point with ID 159 from the development set of BIRD-
Bench, showcasing an error in the ambiguous/vague category.

### Example 3: Incorrect Gold SQL

Figure 7 showcases an incorrect golden SQL query found in the data point with
ID 132 of the financial domain of the development set of BIRD-Bench. The JOIN
operation incorrectly matches clients and accounts by district_id. Due to the
possibility of multiple clients and accounts in the same district, accounts
are incorrectly associated with the wrong users.

Figure 7: Example of an incorrect SQL query that generates the wrong gold
reference answer for the given question. The JOIN operation incorrectly
matches clients and accounts by district_id. Due to the possibility of
multiple clients and accounts in the same district, accounts are incorrectly
associated with the wrong users.

### Example 4: Synonyms

Figure 8 demonstrates how specific wordings can complicate interpretation for
an LLM. The term sum, being both a SQL keyword and a descriptor, led to the
LLM’s literal interpretation and the incorrect summation of a transaction. The
actual intent was to inquire about the transaction’s balance or amount. A
rephrased question resulted in the LLM generating the correct SQL query,
fetching the intended information, as seen in the figure.

Figure 8: Question from data point with ID 177 from the development set of
BIRD-Bench containing a difficult synonym, a corrected version of the question
with the synonym replaced and corresponding predicted SQL queries by the DIN-
SQL (GPT-3.5) model described in Section 3.3. Showcases the difficulty of
synonyms on model predictions.

### Example 5: String Capitalization

As a consequence of SQL being a case-sensitive language when comparing string
values in a query, the way a question is formulated regarding the use of
uppercase or lowercase letters when asking for a specific value affects the
result. This is because the LLM will most likely use the specific entry as
given when generating the query, unless it has knowledge of the case used for
different entries in the database. Therefore, in Figure 9, an example is
provided where the terms "East" and "North" are mentioned with initial capital
letters, as is commonly the case. However, the entries for these column values
are in lowercase in the database, which means the question needs to account
for this for the LLM to be able to generate a correct query. The corrected
question and the SQL query generated from it can also be seen in Figure 9.

Figure 9: Example Ambiguous.

### Example 6: Database Schema Non-Alignment

Table 4: Examples of questions that does not map to the database schema and accompanying descriptions of why they do not. Incorrect Question |  Description
---|---
What is the disposition ID of the client who made $5100 USD transaction on 1998/9/2? |  The question asks for a single disposition ID, which does not reflect that there is a one-to-many relation between client and disposition, and most likely it won’t be possible to return a single ID.
List out the account numbers of clients who are youngest and have highest average salary? |  There is no information about salaries of specific clients in the database.

Table 4 showcases two questions improperly aligned with the database schema’s
structure, meaning the questions cannot be accurately answered with data from
the database. The structure of the database schema can be viewed in Figure 3.
The first question overlooks the possibility of clients having multiple
dispositions, as well as multiple disposition IDs. The second question instead
cannot be converted into a SQL correctly since there is no information in the
database about salaries of specific clients.