# Anomalous 4$f$ fine structure in TmSe1-xTex across the metal-insulator
transition

C.-H. Min<EMAIL_ADDRESS>Institut für Experimentelle und Angewandte
Physik, Christian-Albrechts-Universität zu Kiel and Kiel Nano, Surface and
Interface Science KiNSIS, D–24098 Kiel, Germany  Center for Quantum
Spintronics, Department of Physics, Norwegian University of Science and
Technology, NO7034 Trondheim, Norway  S. Müller  Experimentelle Physik VII and
Würzburg-Dresden Cluster of Excellence ct.qmat, Fakultät für Physik und
Astronomie, Universität Würzburg, D–97074 Würzburg, Germany  W. J. Choi
Department of Physics and Chemistry, DGIST, Daegu 42988, Republic of Korea  L.
Dudy  Synchrotron SOLEIL, Saint-Aubin 91190, France  V. Zabolotnyy
Experimentelle Physik IV, Universität Würzburg, D–97074 Würzburg, Germany  M.
Heber  Deutsches Elektronen-Synchrotron DESY, D–22607 Hamburg, Germany  J. D.
Denlinger  Advanced Light Source, Lawrence Berkeley Laboratory, Berkeley,
California 94720, USA  C.-J. Kang  Department of Physics, Chungnam National
University, Daejeon 34134, Republic of Korea  M. Kalläne  Institut für
Experimentelle und Angewandte Physik, Christian-Albrechts-Universität zu Kiel
and Kiel Nano, Surface and Interface Science KiNSIS, D–24098 Kiel, Germany
Ruprecht Haensel Laboratory, Christian-Albrechts-Universität zu Kiel, D–24098
Kiel, Germany  N. Wind  Institut für Experimentelle und Angewandte Physik,
Christian-Albrechts-Universität zu Kiel and Kiel Nano, Surface and Interface
Science KiNSIS, D–24098 Kiel, Germany  Deutsches Elektronen-Synchrotron DESY,
D–22607 Hamburg, Germany  Institut für Experimentalphysik, Universität
Hamburg, 22761 Hamburg, Germany  M. Scholz  Deutsches Elektronen-Synchrotron
DESY, D–22607 Hamburg, Germany  T. L. Lee  Diamond Light Source, Harwell
Science and Innovation Campus, Didcot, OX11 0DE, UK  C. Schlueter  Deutsches
Elektronen-Synchrotron DESY, D–22607 Hamburg, Germany  A. Gloskovskii
Deutsches Elektronen-Synchrotron DESY, D–22607 Hamburg, Germany  E. D. L.
Rienks  Helmholtz-Zentrum Berlin für Materialien und Energie,
Elektronenspeicherring BESSY II, D–12489 Berlin, Germany.  V. Hinkov
Experimentelle Physik IV, Universität Würzburg, D–97074 Würzburg, Germany  H.
Bentmann  Center for Quantum Spintronics, Department of Physics, Norwegian
University of Science and Technology, NO7034 Trondheim, Norway  Y. S. Kwon
<EMAIL_ADDRESS>Department of Physics and Chemistry, DGIST, Daegu 42988,
Republic of Korea  F. Reinert  Experimentelle Physik VII and Würzburg-Dresden
Cluster of Excellence ct.qmat, Fakultät für Physik und Astronomie, Universität
Würzburg, D–97074 Würzburg, Germany  K. Rossnagel  Institut für Experimentelle
und Angewandte Physik, Christian-Albrechts-Universität zu Kiel and Kiel Nano,
Surface and Interface Science KiNSIS, D–24098 Kiel, Germany  Ruprecht Haensel
Laboratory, Christian-Albrechts-Universität zu Kiel, D–24098 Kiel, Germany
Ruprecht Haensel Laboratory, Deutsches Elektronen-Synchrotron DESY, D–22607
Hamburg, Germany

###### Abstract

Hybridization between localized 4$f$ and itinerant 5$d$6$s$ states in heavy
fermion compounds is a well-studied phenomenon and commonly captured by the
paradigmatic Anderson model. However, the investigation of additional
electronic interactions, beyond the standard Anderson model, has been limited,
despite their predicted important role in the exotic quasiparticle formation
in mixed-valence systems. We investigate the 4$f$ states in TmSe1-xTex
throughout a semimetal-insulator phase transition, which drastically varies
the interactions related to the 4$f$ states. Using synchrotron-based hard
x-ray and extreme ultraviolet photoemission spectroscopy, we resolve subtle
peak splitting in the 4$f$ peaks near the Fermi level in the mixed-valent
semimetal phase. The separation is enhanced by several tens of meV by
increasing the lattice parameter by a few percent. Our results elucidate the
evolving nature of the 4$f$ state across the phase transition, and provide
direct experimental evidence for electronic interactions beyond the standard
Anderson model in mixed-valence systems.

Elucidating the mechanism of quasiparticle formation is essential for gaining
an intuitive understanding of complex electronic states in many-body systems
Landau (1959). Based on the Landau-Fermi liquid theory, complex many-electron
problems are simplified by representing them in terms of itinerant
quasiparticles with modified mass and energy. Recent studies of mixed-valence
(MV) systems have led condensed matter physicists to explore the existence of
exotic composite quasiparticles that are neutrally charged Baskaran (2015);
Knolle and Cooper (2017); Erten  _et al._ (2017); Chowdhury  _et al._ (2018);
Varma (2020). Intriguingly, these MV systems have the potential to produce
Majorana fermions that do not require a superconducting state for their
existence Varma (2020).

These exotic effects arise from a peculiar interaction in MV systems that
competes with the 4$f$$-$5$d$ hybridization in the standard Anderson model
Haldane (1977a). This interaction is described as a screening term, which is
crucial for the fulfillment of the Friedel sum rule when two charge states
coexist for a short time. However, the role of this interaction in MV systems
remains unclear Haldane (1977b); Anderson (1984), and was only considered in a
handful of theoretical studies Perakis  _et al._ (1993); Varma (1994).

More generally, there is a strong motivation to identify the various
interactions between 4$f$, 5$d$, and perhaps the other states from anions in
rare earth compounds Takeshige  _et al._ (1985, 1991); Allen  _et al._ (1986).
The MV compound investigated here, TmSe1-xTex , offers a unique opportunity
to study the evolution of electronic interactions in response to changes in
the lattice parameter using photoemission spectroscopy. The conceptual view of
TmSe1-xTex is as follows: the MV character sits on the Tm ion, which lies
between Tm2+ (4$f^{13}$) and Tm${}^{3+}($4$f^{12}$) (see Supplementary Sec. S
I). When the Se/Te composition is changed, the $fcc$ crystal structure is
retained. Both anions (Se and Te) accept about two electrons from a Tm cation
to form an (almost) closed shell; therefore, replacing Se with Te leads to
lattice expansion without significantly affecting the periodicity of the Tm
ions. This is largely analogous to changing the lattice constant by applying
pressure. Transport and magnetic properties corroborate this analogy, as the
behavior of TmSe1-xTex under hydrostatic pressure smoothly transitions to that
of TmSe at ambient pressure Derr  _et al._ (2006); Tsiok  _et al._ (2014);
Boppart (1985).

To understand what evolution can be expected when the Te/Se ratio is changed,
it is instructive to look at the properties of the two end members, TmSe and
TmTe. At one end, TmSe is the only binary MV system Campagna  _et al._ (1974)
that exhibits both antiferromagnetic and semimetallic properties Møller  _et
al._ (1977); Clayman  _et al._ (1977). At the other end, TmTe is almost an
ionic insulator with a quadrupolar magnetic order Matsumura  _et al._ (1995);
Clementyev  _et al._ (1997). Thus, a semimetal-insulator transition should
occur between TmSe and TmTe, causing a drastic change in the 4$f$ bonding
properties at ambient pressure.

Two macroscopic effects can be observed in the transition that can be
attributed to the difference in the 4$f$ bonding properties. First, there is a
strong change in the compressibility, which is higher in the semimetallic
phase. Second, there is a sudden change in the effective magnetic moments
across the transition, indicating a significant difference in the 4$f$
occupation numbers. However, systematic spectroscopic studies of TmSe1-xTex as
a function of $x$ that reveal the underlying electronic structure changes are
still lacking.

Here we report the detailed evolution of the electronic structure of
TmSe1-xTex across the semimetal-insulator transition as obtained from core and
valence electron emissions using synchrotron-based hard x-ray and extreme
ultraviolet photoelectron spectroscopy (PES) Kaindl  _et al._ (1982); Ufuktepe
_et al._ (1998); Lebègue  _et al._ (2005). We firmly establish a bulk
electronic transition at $x=0.29$ between semimetallic and insulating
electronic states with distinct non-integer valences. In the semimetallic
phase, our results reveal a conspicuous two-component 4$f$-peak structure
indicating a so far omitted interaction of the 4$f$ electrons.

Experiment.—Single-crystalline TmSe1-xTex samples were grown with Tm flux
using the Bridgman method (Supplementary Sec. S II). The Te content ($x$) was
mainly determined by energy-dispersive x-ray spectroscopy (EDXS). Hard x-ray
photoemission spectroscopy (HAXPES) and high-resolution extreme ultraviolet
(XUV) photoemission spectroscopy were performed at ALS MERLIN, BESSY II 13,
DIAMOND I09, PETRA III P22 & P04, and SOLEIL TEMPO, respectively.
Photoemission spectra were taken from cleaved (001) surfaces, typically within
six hours after cleavage under ultrahigh vacuum, in a temperature range from 2
K to 110 K (See also Supplementary Sec. S III).

Figure 1: (a) Lattice parameter $a_{\rm 0}$ as a function of the Te content
$x_{\text{EDXS}}$. Two fitted straight lines with different slopes (black
dashed lines) intersect at $x_{\text{EDXS}}=0.29$, indicating a phase
transition. (b) Temperature- and Te concentration-dependent resistivity,
showing a transition at around $x_{\rm EDXS}=0.29$ from semimetallic behavior
for $x_{\rm EDXS}\leq 0.11$ to semicondcuting behavior for $x_{\rm EDXS}\geq
0.46$.

Electrical properties.—Figure 1(a) shows the relationship between the lattice
constant $a_{\rm 0}$ and the composition ratio $x_{\text{EDXS}}$, which are
measured after the crystal growth. In particular, $x_{\text{EDXS}}$ is
hereafter denoted as $x$ (Supplementary Table I). As the two fitted lines
show, there are two different regimes with different slopes. Since $x$ and
$a_{\rm 0}$ can be considered as external pressure and volume, respectively,
the different slopes indicate different compressibilities. The samples with
lower $x$ values show higher compressibility, which is in agreement with the
previous result reported in Ref. Boppart (1985). Note that the intersection of
the two lines is at $x=0.29$.

The resistivity data of our samples in Fig. 1(b) show the behavior known from
the literature Kaldis  _et al._ (1979); Köbler  _et al._ (1981). For $x\leq
0.11$, the resistivity increases by about an order of magnitude with
decreasing temperature $T$, but remains at a low level. This corresponds to
the behavior of TmSe ($x=0$), which is classified as a semimetal: The absolute
electrical resistivity is low, while the optical reflectivity is high, and the
material exhibits characteristic conduction band plasma effects at room
temperature Batlogg (1981). For $x>0.29$, however, the resistivity increase
with decreasing $T$ is much more pronounced, indicating semiconducting or
insulating behavior. At the transition point $x=0.29$, the temperature
dependence of the resistivity is flat, representing an intermediate behavior.
The data therefore establish a semimetal-insulator transition in TmSe1-xTex at
an actual $x=0.29$.

Figure 2: (a) Bulk-sensitive Se 2$p$ and Tm 3$d$ core-level spectra of
TmSe1-xTex for different Te substitutions $x$. Due to the mixed-valence
properties, all Tm core-level emissions consist of two ionization peaks. (b)
Extracted Tm valence ($V_{\rm Tm}$ ), showing a strong deviation between two
phases, namely around 2.65 and 2.15 (dashed lines), while remaining almost
constant in each phase. For the semimetallic samples ($0\leq x\leq 0.11$), the
spectral weight of Tm2+ and Tm3+ emissions is comparable, while for the
insulating samples (0.29 $<x\leq$ 1.0), that of Tm2+ is dominant, as shown in
the inset of (b) with 3$d_{3/2}$. (c) $V_{\rm Tm}$ of TmTe: By deconvoluting
the loss function from the Te 3$p$ spectrum as explained in Supplementary Sec.
LABEL:sec:VTMestimation, the intrinsic Tm 3$d$ spectrum (orange circles) is
extracted. Inset of (c): With the intrinsic spectrum, the Tm2+ and Tm3+
contributions are identified (green and pink areas, respectively), giving a
non-integer $V_{\rm Tm}$ = 2.12 even in the insulating TmTe.

Mixed valence and valence transition.— In rare-earths systems, the electronic
properties at the surface can substantially deviate from those in the bulk
Reinert  _et al._ (1998); Lutz  _et al._ (2016); Utsumi  _et al._ (2017). We
use HAXPES to probe the bulk valence properties. The insulating samples with
$x\geq 0.46$ were measured at a higher temperature to avoid sample charging.
Figure 2(a) shows representative core-level emissions of Tm 3$d$ and Se 2$p$
with their spin-orbit partners (Supplementary Sec. S III). Both 3$d_{5/2}$ and
3$d_{3/2}$ peaks consist of two ionization peaks of Tm2+ and Tm3+ due to the
MV nature.

The Tm valence $V_{\rm Tm}$ is calculated from the average spectral weight of
Tm2+ and Tm3+. The two Se 2$p$ peaks and the inelastic background must be
removed to estimate the Tm3+ spectral weight, since the latter is strongly
reduced in the samples with $x\geq 0.29$, as shown in the inset of Fig. 2(b).
Figure 2(b) shows the $V_{\rm Tm}$ values obtained after subtracting these
contributions (Supplementary Sec. S IV). The $V_{\rm Tm}$ values differ
between semimetallic and insulating samples, while they remain almost constant
between the samples of the same phase. Such $x$-dependent $V_{\rm Tm}$
behavior is consistent with magnetic susceptibility measurements Köbler  _et
al._ (1981); Boppart (1985), which do not show any integer $V_{\rm Tm}$ in the
Tm monochalcogenide. The $V_{\rm Tm}$ values for the semimetallic and
insulating samples are around 2.65 and 2.15 (indicated by the dashed lines),
respectively, which can be converted to $n_{\rm f}$ $=$ 12.35, and $n_{\rm f}$
$=$ 12.85, respectively. We do not find a clear temperature dependence in
$V_{\rm Tm}$ , as exemplified by the two data points for $x=0.05$ in Fig.
2(b), unlike the case of the Sm valence in SmB6 Mizumaki  _et al._ (2009);
Lutz  _et al._ (2016); Utsumi  _et al._ (2017). Often $V_{\rm Tm}$ in the
insulating phase has been considered to be divalent Launois  _et al._ (1980);
Jarrige  _et al._ (2005), but our estimation clearly shows a non-integer
value. As shown in detail in Supplementary Sec. S IV, we can obtain an
estimate of $V_{\rm Tm}$ for the insulating TmTe [Fig. 2(c)] more easily
because the loss function can be unambiguously determined using Te $3p$ core-
level spectra. As a result, the inelastic background (cyan dashed line) can be
removed and the intrinsic Tm $3d$ spectrum can be extracted (orange circles).
From the intrinsic spectrum [inset of Fig. 2(c)], $V_{\rm Tm}$ is evaluated
using multiplet ligand field theory calculations Haverkort (2016). Our fitting
results clearly yield a non-integer $V_{\rm Tm}$ = 2.12 $\pm$ 0.05 that is
consistent with bulk-sensitive magnetic susceptibility and spectroscopic
measurements Köbler  _et al._ (1981); Kinoshita  _et al._ (1998); Launois  _et
al._ (1980); Brewer  _et al._ (1985). Since our bulk-sensitive data were
obtained on freshly cleaved surfaces, the non-integer $V_{\rm Tm}$ cannot be
attributed to contamination Kaindl  _et al._ (1982).

Based on the fact that the conduction 5$d$ states are shifted above the Fermi
level $E_{\rm F}$ to become insulating Antonov  _et al._ (2001), the non-
integer valence indicates that the hybridization between 4$f$ and non-5$d$
states also exists in the insulating phase. We speculate that the Te 5$p$
states should be involved in the covalent bonding formation. Thus, it implies
that the $f-p$ hybridization plays a role in the insulating phase.

Figure 3: (a) Overall 4$f$ spectra of TmSe recorded at XUV and hard x-ray
photon energies. Due to the mixed-valence nature, both 4$f^{12}$ (pink bars)
and 4$f^{11}$ (green bars) final states appear as multiplet lines. (b)
Enlargement of the 4$f^{12}$ final states in (a) near the Fermi level $E_{\rm
F}$ . The surface contributions (blue bars) in the $h\nu$ $=70$ eV data are
stronger than the bulk peak contributions (green bars). Except for these
surface peaks, both spectra show identical peak structures. (c) Hard x-ray Tm
4$f$ spectra were obtained at the same temperatures as in Fig. 2. In
semimetallic samples, the ${}^{\rm 3}H_{\rm 6}$ peaks are positioned near
$E_{\rm F}$ , while in insulating samples, they appear at lower energies.
The maximum peak energy separation between TmSe and TmTe is approximately 0.6
eV (indicated by black dashed lines). Spectral weight in the gap region
reaches a maximum at the transition point $x=$ 0.29. (d) XUV Tm 4$f$ spectra
exhibit a similar trend to the HAXPES results in (c). The spectral weight in
the gap region and the energy separation of the ${}^{\rm 3}H_{\rm 6}$ peaks
match those in (c).

4f multiplet structure and spectral gap.—We take a closer look at the 4$f$
emissions near $E_{\rm F}$ with both XUV and hard x-ray excitations. Figure
3(a) shows the complete 4$f$ multiplet structure of TmSe. Due to the 4$f^{13}$
and 4$f^{12}$ contributions in the ground-state wave function, both 4$f^{12}$
and 4$f^{11}$ final states are accessible after removing one electron in the
photoemission process. The green and pink bars represent the theoretically
calculated atomic multiplet lines for the corresponding transitions Gerken
(1983). The spectrum taken with XUV light ($h\nu$ = 70 eV) shows a more
complicated peak structure than that at $h\nu$ = 6 keV because of additional
surface 4$f$ peaks (Supplementary Sec. S V). However, we find that both the
XUV and hard x-ray 4$f$ spectra are consistent, except for the presence of
these surface peaks. Figure 3(b) shows the enlarged 4$f$ spectra of the
$f^{13}\rightarrow f^{12}$ transition. The surface peaks (blue bars) are
dominant in the XUV spectra (cyan straight line) and are absent in the HAXPES
data (cyan dashed line). The surface peaks are located at $E-E_{\rm F}$
$\approx-0.4$ eV on the freshly cleaved surface and shift to lower $E-E_{\rm
F}$ with time [Supplementary Fig. LABEL:FigSuBu(b)].

The first two bulk 4$f$ peaks, ${}^{\rm 3}H_{\rm 6}$ and ${}^{\rm 3}F_{\rm 4}$
, in TmSe1-xTex [Figs. 3(c) and 3(d)] do not overlap with the surface peaks,
making it easier to analyze the bulk electronic states. Since the peak
position of ${}^{\rm 3}H_{\rm 6}$ determines the gap opening, the peaks below
the ${}^{\rm 3}H_{\rm 6}$ peaks of semimetallic samples are faintly depicted
to emphasize the ${}^{\rm 3}H_{\rm 6}$ peaks of the insulating samples
(Supplementary Sec. S III). In Fig. 3(c), medium-resolution HAXPES spectra
illustrate the evolution of the 4$f$ states across the semimetal-insulator
transition. For the semimetallic samples ($0\leq x\leq 0.11$), the ${}^{\rm
3}H_{\rm 6}$ appears at $E-E_{\rm F}$ $\approx-50$ meV, while for the
insulating samples ($0.46\leq x\leq 1$), the ${}^{\rm 3}H_{\rm 6}$ peaks are
below $-0.5$ eV. Between TmSe ($x=0$) and TmTe ($x=1$), the energy difference
in ${}^{\rm 3}H_{\rm 6}$ becomes maximal ($\approx$ 0.6 eV), which can be
considered as a fully opened gap. Furthermore, the ${}^{\rm 3}H_{\rm 6}$ peak
becomes broader up to $x=0.29$ and narrower for larger $x>0.29$. Thus, for
$x=0.29$, spectral weight still largely fills the gap region, indicating that
this concentration is indeed close to the semimetal-insulator transition. It
can be seen that the shapes of the ${}^{\rm 3}H_{\rm 6}$ peaks are different
among the three semimetallic samples. Therefore, we now investigate their
behavior with higher energy-resolution using XUV photoemission spectroscopy.

Figure 3(d) shows high-resolution 4$f$ spectra taken at $h\nu$ $=110$ eV near
$E_{\rm F}$ as a function of $x$ (Supplementary Sec. S III). The spectral
features are mostly consistent with those of HAXPES shown in Fig. 3(c): The
energy difference in ${}^{\rm 3}H_{\rm 6}$ between semimetallic and insulating
phases is about 0.6 eV, and the larger spectral weight in the gap region is
present at $x=0.29$. Therefore, we attribute the XUV ${}^{\rm 3}H_{\rm 6}$
peaks to bulk states with higher energy resolution. The results show that the
${}^{\rm 3}H_{\rm 6}$ peak in TmTe is located at about $E-E_{\rm F}$ $=-0.7$
eV; thus, the charge gap is larger than 0.3 eV Suryanarayanan  _et al._
(1975); Boppart (1985). In addition, asymmetric shapes and intensity variation
of ${}^{\rm 3}H_{\rm 6}$ are observed among semimetallic samples.

Figure 4: (a) Low-temperature 4$f$ spectra measured with linear vertical
polarization to highlight the asymmetric peak shape of ${}^{\rm 3}H_{\rm 6}$
and ${}^{\rm 3}F_{\rm 4}$ . With increasing $x$, both peaks become more
asymmetric as indicated by the downward arrows at ${}^{\rm 3}H_{\rm 6}$ and
${}^{\rm 3}F_{\rm 4}$ . (b) Line decomposition of ${}^{\rm 3}H_{\rm 6}$ .
The asymmetric shape is due to the two-peak structure of each bulk multiplet
peak. For example, the ${}^{\rm 3}H_{\rm 6}$ peak for $x=$ 0.11 consists of a
sharp peak (B1) and a broad peak (B2). As $x$ increases, the spectral weight
of the B1 at $E_{\rm F}$ decreases relative to that of the B2, colored in cyan
and black, respectively. (c) Energy difference between B1 and B2, increasing
with $x$. The energy position of B1 remains almost the same, but B2 shifts to
lower energies, as indicated by the arrows in (a), up to $E-E_{\rm F}$ $=-96$
meV. (d) High-resolution HAXPES spectra of ${}^{\rm 3}H_{\rm 6}$  showing
the increasing asymmetric line shape with increasing $x$, thereby supporting
the two-peak structure in the bulk electronic structure.

4f fine structure in the semimetallic phase.— Low-temperature measurements
($T=2$ K) were performed to study the asymmetric shapes at reduced thermal
broadening. In addition, a different light polarization was used to reduce the
intensity of the surface multiplet peaks, as shown in Fig. 4(a) (Supplementary
Sec. S III). We also found that both the ${}^{\rm 3}H_{\rm 6}$ and ${}^{\rm
3}F_{\rm 4}$ emissions become more asymmetric with increasing $x$, as
indicated by the downward arrows. As a result, each peak consists of two
peaks, B1 and B2, as indicated by the green bars.

Figure 4(b) shows that the ${}^{\rm 3}H_{\rm 6}$ emission for $x=0.11$
consists of a sharp peak ($E-E_{\rm F}$ $\approx-14$ meV) and a broad peak
($E-E_{\rm F}$ $\leq-96$ meV) whose splitting is too large to be attributed to
crystal-field splitting in Tm monochalcogenides Furrer  _et al._ (1981);
Matsumura  _et al._ (2002). Note that as $x$ increases, the sharp peak B1 is
suppressed, while the broad peak B2 becomes relatively more intense and shifts
to lower energies, as indicated by the downward arrows [see also Fig. 4(a)].
The peak positions of B1 and B2 as a function of $x$ are plotted in Fig. 4(c).
The energy difference between B1 and B2 increases from 6 meV to 82 meV for a
2% lattice expansion from $x=0$ to $x=0.11$.

If B2 is from a bulk state, the two-peak structure should also be observable
in bulk-sensitive HAXPES. Therefore, as additional evidence, we have taken
high-resolution HAXPES spectra of the ${}^{\rm 3}H_{\rm 6}$ peaks for $x=0.05$
and 0.11, as shown in Fig. 4(d) (Supplementary Sec. S III). These spectra show
asymmetric peaks consistent with the broadening of ${}^{\rm 3}H_{\rm 6}$
observed in Fig. 4(c). Interestingly, the two-peak structure has also been
observed in the other MV system SmB6 Min  _et al._ (2015).

Discussion.—Our comprehensive photoemission experiments on TmSe1-xTex across
the MV semimetal-to-insulator phase transition show good agreement with
resistivity data, providing compelling evidence for the phase transition at
$x=0.29$. More interestingly, we have found strong distinct 4$f$ photoemission
signatures between the semimetallic and insulating phases. The two main
experimental observations are (1) a non-integer $V_{\rm Tm}$ that decreases
significantly across the transition and (2) a two-peak structure of ${}^{\rm
3}H_{\rm 6}$ and ${}^{\rm 3}F_{\rm 4}$ in the semimetallic phase.

First, the non-integer $V_{\rm Tm}$ in the insulating TmTe is discussed.
$V_{\rm Tm}$ $=2.12$ in the insulating states indicates that 0.12 electrons
have been shared from the Tm 4$f$ state. It indicates that some bonding
between 4$f$ and other states still exists in the insulating phase. When the
5$d$ states are occupied, the system may enter a metallic phase. Thus, we can
rule out the existence of the 5$d$ states, and one can naturally speculate
that non-5$d$ states are involved in bonding with the 4$f$ states. A strong
candidate is the anion $p$-band, which spreads 1.7 eV below $E_{\rm F}$ ,
consistent with covalent bonding causing non-integer $V_{\rm Tm}$ .

Second, we expect a clear change in the bonding character as $V_{\rm Tm}$
decreases significantly from semimetallic to insulating phases. The strong
reduction in $V_{\rm Tm}$ across the transition indicates that 0.5 electrons
are mostly transferred from the 5$d$6$s$ states to the 4$f$ states. The
consequence appears in the 4$f$ spectra with a large energy shift of 0.6 eV.
It indicates strong changes in the 4$f$$-$5$d$ hybridization due to the
absence of 5$d$6$s$ states below $E_{\rm F}$ .

Finally, we focus on the constant $V_{\rm Tm}$ ($n_{\rm f}$ ) and the two-
peak structure among the semimetallic samples ($0\leq x\leq 0.11$). In
particular, we have found that the relative intensities of the two peaks vary
with $x$. Since $n_{\rm f}$ remains unchanged (constant $V_{\rm Tm}$ ),
these results indicate that the sharp 4$f$ states (B1) seem to transform into
the broader 4$f$ states (B2) with increasing $x$ (lattice expansion). This
suggests that the bonding properties of the 4$f$ states in TmSe1-xTex evolve
with increasing $x$.

We also performed DMFT+DFT calculations, but the two-peak structure was not
reproduced (Supplementary Sec. S VI). Although more in-depth studies are
needed to understand the evolution from B1 to B2 states in MV systems, our
novel experimental results have revealed an additional 4$f$ state that is
absent in standard Anderson model studies Denlinger  _et al._ (2013). This
finding provides evidence for the need for an additional interaction term,
going beyond the standard Anderson model. For example, the 4$f$-5$d$ screening
term Haldane (1977b); Varma (2020) can provide insight into the interpretation
of the 4$f$ nature. In addition, the evolution of the coupling between 4$f$
electrons and phonons, leading to the formation of a polaronic state, could
play a role in heavy fermion systems Martin-Rodero  _et al._ (2008). While
various interactions could be proposed, our results indicate that the desired
interaction should induce an energy separation between B1 and B2 of up to 76
meV with a 2% increase in the lattice parameter.

Conclusion.— We have investigated the nature of the localized 4$f$ states in
TmSe1-xTex for a wide range of compositions. Our results highlight tunable
interactions among the 4$f$ states as $x$ varies across the semimetal-
insulator phase transition. These interactions across the transition extend
beyond the hybridization with 5$d$6$s$ states, and include other valence
orbitals. We have demonstrated this by resolving large variations in the non-
integer $V_{\rm Tm}$ . Moreover, we observe two distinct 4$f$ states in the
low-energy semimetallic electronic structure. Notably, the population of these
two states varies with $x$, even in the absence of changes in the 4$f$
occupancy. The energy splitting between these two states significantly depends
on $x$, exhibiting a substantial 76 meV shift when the lattice parameter is
expanded by 2%. These observations reveal a rich and complex correlated
electron phase in TmSe1-xTex , well beyond the predictions of the standard
Anderson model that has served as a paradigm for the description of heavy-
fermion compounds. The highly tunable character of the 4$f$ states in
TmSe1-xTex may induce distinct physical phenomena, potentially promoting the
formation of exotic quasiparticles. However, rigorous theoretical studies will
be required to elucidate the underlying nature of the novel 4$f$ low-energy
phenomenology we observe here in the metallic phase of TmSe1-xTex .

###### Acknowledgements.

C.H.M. and L.D. would like to thank J. W. Allen and H.-D. Kim for their
helpful discussions. C.H.M. would also like to thank K. Kissner, T. R. F.
Peixoto, C. Fornari, J. Buck and W.-S. Kyung for their kind assistance. This
work was supported by the German Research Foundation (DFG) in the framework of
the Würzburg–Dresden Cluster of Excellence on Complexity and Topology in
Quantum Matter–ct.qmat (EXC 2147, Project-ID 390858490), Project-ID
258499086–SFB 1170 (projects A01, C06) and BMBF Project-ID 05K19WW2. This
research used resources of the Advanced Light Source supported by a DOE Office
of Science User Facility under Contract No. DE-AC02-05CH11231. We acknowledge
Diamond Light Source for access to beamline I09 (Proposals No. SI22630), which
contributed to the results presented here. We acknowledge DESY (Hamburg,
Germany), a member of the Helmholtz Association HGF, for the provision of
experimental facilities. Parts of this research were carried out at PETRA III
and we would like to thank the staff for their assistance. Funding for the
HAXPES instrument at P22 by the Federal Ministry of Education and Research
(BMBF) under framework program ErUM is gratefully acknowledged. We also thank
the BMBF for funding the photoemission spectroscopy instrument at beamline P04
(contracts 05KS7FK2, 05K10FK1, 05K12FK1, and 05K13FK1 with Kiel University;
05KS7WW1 and 05K10WW2 with Würzburg University). The research leading to the
presented results was supported by the project CALIPSOplus under Grant
Agreement 730872 from the EU Framework Program for Research and Innovation
HORIZON 2020. Y.S.K. was supported by grants from the National Research
Foundation (NRF) of Korea (NRF) funded by the Korean government (MSIP; Grant
No.NRF-2019R1F1A1040989). C.-J.K. was supported by NRF (Grant No.
2022R1C1C1008200) and the KISTI Supercomputing Center (Project No.
KSC-2022-CRE-0438). N.W was supported by the Deutsche Forschungsgemeinschaft
(DFG; German Research Foundation) via CRC 925, Project ID 170620586(Project
B2)

## References

  * Landau (1959) L. Landau, Sov. Phys. JETP 8, 70 (1959).
  * Baskaran (2015) G. Baskaran, arXiv:1507.03477 [cond-mat] (2015), arXiv: 1507.03477.
  * Knolle and Cooper (2017) J. Knolle and N. R. Cooper, Phys. Rev. Lett. 118, 096604 (2017).
  * Erten  _et al._ (2017) O. Erten, P.-Y. Chang, P. Coleman, and A. M. Tsvelik, Phys. Rev. Lett. 119, 057603 (2017).
  * Chowdhury  _et al._ (2018) D. Chowdhury, I. Sodemann, and T. Senthil, Nat. Commun 9, 1 (2018).
  * Varma (2020) C. M. Varma, Phys. Rev. B 102, 155145 (2020).
  * Haldane (1977a) F. D. M. Haldane, Phys. Rev. B 15, 2477 (1977a).
  * Haldane (1977b) F. D. M. Haldane, in  _Valence Instabilities and Related Narrow-Band Phenomena_, edited by R. D. Parks (Springer US, Boston, MA, 1977) pp. 191–200.
  * Anderson (1984) P. W. Anderson, in  _Moment Formation In Solids_, NATO ASI Series, edited by W. J. L. Buyers (Springer US, Boston, MA, 1984) pp. 313–326.
  * Perakis  _et al._ (1993) I. E. Perakis, C. M. Varma, and A. E. Ruckenstein, Phys. Rev. Lett. 70, 3467 (1993).
  * Varma (1994) C. M. Varma, Phys. Rev. B 50, 9952 (1994).
  * Takeshige  _et al._ (1985) M. Takeshige, O. Sakai, and T. Kasuya, J. Magn. Magn. Mater. 52, 363 (1985).
  * Takeshige  _et al._ (1991) M. Takeshige, O. Sakai, and T. Kasuya, J. Phys. Soc. Jpn. 60, 666 (1991).
  * Allen  _et al._ (1986) J. W. Allen, S. J. Oh, O. Gunnarsson, K. Schönhammer, M. B. Maple, M. S. Torikachvili, and I. Lindau, Advances in Physics 35, 275 (1986).
  * Derr  _et al._ (2006) J. Derr, G. Knebel, G. Lapertot, B. Salce, M.-A. Méasson, and J. Flouquet, J. Phys.: Condens. Matter 18, 2089 (2006).
  * Tsiok  _et al._ (2014) O. B. Tsiok, L. G. Khvostantsev, A. V. Golubkov, I. A. Smirnov, and V. V. Brazhkin, Phys. Rev. B 90, 165141 (2014).
  * Boppart (1985) H. Boppart, J. Magn. Magn. Mater. 47-48, 436 (1985).
  * Campagna  _et al._ (1974) M. Campagna, E. Bucher, G. K. Wertheim, D. N. E. Buchanan, and L. D. Longinotti, Phys. Rev. Lett. 32, 885 (1974).
  * Møller  _et al._ (1977) H. B. Møller, S. M. Shapiro, and R. J. Birgeneau, Phys. Rev. Lett. 39, 1021 (1977).
  * Clayman  _et al._ (1977) B. P. Clayman, R. W. Ward, and J. P. Tidman, Phys. Rev. B 16, 3734 (1977).
  * Matsumura  _et al._ (1995) T. Matsumura, Y. Haga, Y. Nemoto, S. Nakamura, T. Goto, and T. Suzuki, Physica B Condens. Matter. 206-207, 380 (1995).
  * Clementyev  _et al._ (1997) E. Clementyev, R. Köhler, M. Braden, J. M. Mignot, C. Vettier, T. Matsumura, and T. Suzuki, Physica B Condens. Matter. 230-232, 735 (1997).
  * Kaindl  _et al._ (1982) G. Kaindl, C. Laubschat, B. Reihl, R. A. Pollak, N. Maartensson, F. Holtzberg, and D. E. Eastman, Phys. Rev. B 26, 1713 (1982).
  * Ufuktepe  _et al._ (1998) Y. Ufuktepe, S.-i. Kimura, T. Kinoshita, K. G. Nath, H. Kumigashira, T. Takahashi, T. Matsumura, T. Suzuki, H. Ogasawara, and A. Kotani, J. Phys. Soc. Jpn. 67, 2018 (1998).
  * Lebègue  _et al._ (2005) S. Lebègue, G. Santi, A. Svane, O. Bengone, M. I. Katsnelson, A. I. Lichtenstein, and O. Eriksson, Phys. Rev. B 72, 245102 (2005).
  * Kaldis  _et al._ (1979) E. Kaldis, B. Fritzler, E. Jilek, and A. Wisard, J. Phys. Colloques 40, C5 (1979).
  * Köbler  _et al._ (1981) U. Köbler, K. Fischer, K. Bickmann, and H. Lustfeld, J. Magn. Magn. Mater. 24, 34 (1981).
  * Batlogg (1981) B. Batlogg, Phys. Rev. B 23, 1827 (1981).
  * Reinert  _et al._ (1998) F. Reinert, R. Claessen, G. Nicolay, D. Ehm, S. Hüfner, W.P. Ellis, G.-H. Gweon, J.W. Allen, B. Kindler, and W. Assmus, Phys. Rev. B 58, 12808 (1998).
  * Lutz  _et al._ (2016) P. Lutz, M. Thees, T. R. F. Peixoto, B. Y. Kang, B. K. Cho, C. H. Min, and F. Reinert, Phil. Mag. 96, 3307 (2016).
  * Utsumi  _et al._ (2017) Y. Utsumi, D. Kasinathan, K.-T. Ko, S. Agrestini, M. W. Haverkort, S. Wirth, Y.-H. Wu, K.-D. Tsuei, D.-J. Kim, Z. Fisk, A. Tanaka, P. Thalmeier, and L. H. Tjeng, Phys. Rev. B 96, 155130 (2017).
  * Mizumaki  _et al._ (2009) M. Mizumaki, S. Tsutsui, and F. Iga, J. Phys.: Conf. Ser. 176, 012034 (2009).
  * Launois  _et al._ (1980) H. Launois, M. Rawiso, E. Holland-Moritz, R. Pott, and D. Wohlleben, Phys. Rev. Lett. 44, 1272 (1980).
  * Jarrige  _et al._ (2005) I. Jarrige, H. Ishii, Y. Q. Cai, J.-P. Rueff, C. Bonnelle, T. Matsumura, and S. R. Shieh, Phys. Rev. B 72, 075122 (2005).
  * Haverkort (2016) M. W. Haverkort, J. Phys. Conf. Ser. 712, 012001 (2016).
  * Kinoshita  _et al._ (1998) T. Kinoshita, Y. Ufuktepe, K. G. Nath, S. Kimura, H. Kumigashira, T. Takahashi, T. Matsumura, T. Suzuki, H. Ogasawara, and A. Kotani, J Electron Spectros. Relat. Phenomena 88-91, 377 (1998).
  * Brewer  _et al._ (1985) W. D. Brewer, G. Kalkowski, G. Kaindl, and F. Holtzberg, Phys. Rev. B 32, 3676 (1985).
  * Antonov  _et al._ (2001) V. N. Antonov, B. N. Harmon, and A. N. Yaresko, Phys. Rev. B 63, 205112 (2001).
  * Gerken (1983) F. Gerken, J. Phys. F: Met. Phys. 13, 703 (1983).
  * Suryanarayanan  _et al._ (1975) R. Suryanarayanan, G. Güntherodt, J. L. Freeouf, and F. Holtzberg, Phys. Rev. B 12, 4215 (1975).
  * Furrer  _et al._ (1981) A. Furrer, W. Bührer, and P. Wachter, Solid State Commun. 40, 1011 (1981).
  * Matsumura  _et al._ (2002) T. Matsumura, K. A. McEwen, R. S. Eccleston, and T. Suzuki, Phys. Rev. B 66, 104410 (2002).
  * Min  _et al._ (2015) C.-H. Min, O. Sommer, B. Y. Kang, B. K. Cho, H. Bentmann, and F. Reinert, J Electron Spectros. Relat. Phenomena 199, 46 (2015).
  * Denlinger  _et al._ (2013) J. D. Denlinger, J. W. Allen, J.-S. Kang, K. Sun, J.-W. Kim, J. H. Shim, B. I. Min, D.-J. Kim, and Z. Fisk, arXiv:1312.6637 [cond-mat] (2013).
  * Martin-Rodero  _et al._ (2008) A. Martin-Rodero, A. Levy Yeyati, F. Flores, and R. C. Monreal, Phys. Rev. B 78, 235112 (2008).