# Gap and magnetic engineering via doping and pressure in tuning the colossal
magnetoresistance in (Mn1-xMgx)3Si2Te6

Chaoxin Huang  Mengwu Huo  Xing Huang  Hui Liu  Lisi Li  Center for Neutron
Science and Technology, Guangdong Provincial Key Laboratory of Magnetoelectric
Physics and Devices, School of Physics, Sun Yat-Sen University, Guangzhou
510275, China  Ziyou Zhang  Zhiqiang Chen  Center for High Pressure Science
and Technology Advanced Research, Shanghai 201203, China  Yifeng Han  Center
for Materials of the University, School of Molecular Sciences, Arizona State
University, Tempe, Arizona 85287, USA  Lan Chen  Feixiang Liang  Center for
Neutron Science and Technology, Guangdong Provincial Key Laboratory of
Magnetoelectric Physics and Devices, School of Physics, Sun Yat-Sen
University, Guangzhou 510275, China  Hongliang Dong  Center for High Pressure
Science and Technology Advanced Research, Shanghai 201203, China  Bing Shen
Center for Neutron Science and Technology, Guangdong Provincial Key Laboratory
of Magnetoelectric Physics and Devices, School of Physics, Sun Yat-Sen
University, Guangzhou 510275, China  Hualei Sun<EMAIL_ADDRESS>School of Science, Sun Yat-Sen University, Shenzhen 518107, China  Meng Wang
<EMAIL_ADDRESS>Center for Neutron Science and Technology,
Guangdong Provincial Key Laboratory of Magnetoelectric Physics and Devices,
School of Physics, Sun Yat-Sen University, Guangzhou 510275, China

###### Abstract

Ferrimagnetic nodal-line semiconductor Mn3Si2Te6 keeps the records of colossal
magnetoresistance (CMR) and angular magnetoresistance (AMR). Here we report
tuning the electronic transport properties via doping and pressure in
(Mn1-xMgx)3Si2Te6. As the substitution of nonmagnetic Mg2+ for magnetic Mn2+,
ferrimagnetic transition temperature $T_{C}$ gradually decreases, while the
resistivity increases significantly. At the same time, the CMR and AMR are
both enhanced for the low-doping compositions (e.g., $x=0.1$ and 0.2), which
can be attributed to doping-induced broadening of the band gap and a larger
variation range of the resistivity when undergoing a metal-insulator
transition by applying a magnetic field along the $c$ axis. On the contrary,
$T_{C}$ rises with increasing pressure due to the enhancement of the magnetic
exchange interactions until a structural transition at $\sim$13 GPa.
Meanwhile, the activation gap is lowered under pressure and the
magnetoresistance is decreased dramatically above 6 GPa where the gap is
closed. The results reveal that doping and pressure are effective methods to
tune the activation gap, and correspondingly, the CMR and AMR in nodal-line
semiconductors, providing an approach to investigate the magnetoresistance
materials for novel spintronic devices.

Since giant magnetoresistance was discovered in magnetic multilayers, the
spin-charge coupling has been widely investigated for decades in a number of
fields, including magnetic storage, magnetic sensor, magnetometer and so
onGrünberg et al. (1986); Baibich et al. (1988); Xiao et al. (1992); Thompson
(2008); Ennen et al. (2016); Diény (1994). CMR is more sensitive to a magnetic
field than giant magnetoresistance, whose variation in resistance can be close
to 100%. CMR was first reported in doped manganese perovskite, where the
double exchange mechanism and dynamic Jahn-Teller effect were demonstrated to
play a key rolevon Helmolt et al. (1993); Jin et al. (1994); Millis et al.
(1996). Other mechanisms were suggested for the later discovered CMR
materials, such as strong magnetic fluctuations in EuCd2P2Wang et al. (2021),
magnetic polarons in Ti2Mn2O7Shimakawa et al. (1996); Majumdar and Littlewood
(1998) and Eu5In2Sb6Rosa et al. (2020), and spin-orbit coupling/spin-texture
driven CMR in EuTe2 and EuMnSb2Yin et al. (2020); Yang et al. (2021); Sun et
al. (2021a). Two-dimensional (2D) layered materials have higher spatial
anisotropy, which is a fertile ground to explore new material systems with CMR
and AMROgasawara et al. (2021); Sun et al. (2021a).

Indeed, the quasi-2D ferrimagnetic (FIM) semiconductor Mn3Si2Te6 was found to
exhibit the largest CMR and AMR, where the resistivity decreases by 107 as
defined by ($\rho$H-$\rho$0)/$\rho$H for $H\|c$ axis, but changes slightly in
resistivity when $H$ is along the $ab$ planeVincent et al. (1986); Ni et al.
(2021); Seo et al. (2021); Sala et al. (2022); Kwon et al. (2023); Mijin et
al. (2023); Lovesey (2023). It was suggested that the nodal-line degeneracy of
Mn3Si2Te6 can be controlled by spin orientation. When the applied field is
along the $c$ axis, spin-orbit coupling (SOC) between Mn and Te will result in
band splitting and one of the bands will shift toward the Fermi level, leading
to the closing of the electric gap and metal-insulator transition (MIT)Seo et
al. (2021). Later, chiral orbital currents (COC) were also revealed
circulating along the edges of MnTe6 octahedra. The coupling between the spin
of Mn2+ and the effective moments of the COC could reduce the electron
scattering when a magnetic field is applied along the $c$ axisZhang et al.
(2022a). However, single-crystal neutron diffraction and magnetization
measurements reveal that the applied field along the $c$ axis smoothly tilts
the magnetic moment during the CMR occurs Ni et al. (2021); Seo et al. (2021);
Ye et al. (2022). The activation gap may be a key gradient in the mechanism.
Thus, further research to improve the magnitudes of the CMR and AMR and
elucidate the mechanism is crucial to promote potential applications in the
future.

Doping and pressure are considered as effective methods to tune quantum
transport properties in electronic correlated materialsZhang et al. (2022b,
2023); Sun et al. (2023a, b, 2021b); Cai et al. (2020). In this work, we
report the successful synthesis of a series of (Mn1-xMgx)3Si2Te6 single
crystals from $x$ = 0 to 1.0 at 0.1 intervals and investigations of the
evolutions of structure, magnetism, and electric transport properties with
doping and pressure. We find the Mg-doping can suppress the FIM evidenced by
the reduced $T_{C}$, increase the resistivity by enlarging the activation gap,
and enhance the CMR and AMR by about two orders of magnitude compared to the
undoped compound. On the other hand, pressure-induced MIT yields a dramatic
change of the electronic transport properties between 6$-$13 GPa. Our results
reveal that the nodal-line topological band gap can be widened or narrowed by
doping or applying pressure, thus tuning the CMR and AMR to a required
magnitude.

Single crystals of (Mn1-xMgx)3Si2Te6 ($0\leq x\leq 1$) were grown by the self-
flux method and characterized by x-ray diffraction (XRD, Empyrean), energy
dispersive x-ray spectroscopy (EDS, EVO Zeiss), Laue diffractometer(Photonic
Science), and physical property measurement system (PPMS, Quantum Design). For
partial samples with resistance exceeding the measuring range (e.g., data for
$x$ = 0.1 at low temperature and low field), an external voltammeter
(Keithley) was employed while necessary. High-pressure electrical transport
measurements of Mn3Si2Te6 single crystals were carried out using a miniature
diamond anvil cell (DAC) made from a Be-Cu alloy on the PPMS. Diamond anvils
with a 300 $\mu$m culet were used. The corresponding sample chamber with a 110
$\mu$m diameter was made in an insulating gasket achieved by cubic boron
nitride and epoxy mixture. KBr powders were employed as the pressure-
transmitting medium, providing a quasi-hydrostatic environment. The pressure
was calibrated by measuring the shift of the fluorescence wavelength of the
ruby sphere, which was loaded in the sample chamber. The standard four-probe
technique was adopted for these measurements. The in situ high-pressure
synchrotron powder XRD patterns were collected at 300 K with an x-ray
wavelength of 0.6199 Å on the Shanghai Synchrotron Radiation Facility. A
symmetric DAC with a pair of 300 $\mu$m diameter culet was used. The sample
chamber was drilled by laser with a diameter of 120 $\mu$m. Daphne oil was
used as the pressure-transmitting medium, and the pressure in the DAC was also
calibrated by the shift of the fluorescence of the ruby sphere.

XRD measurements reveal that the crystal structure of (Mn1-xMgx)3Si2Te6
($0\leq x\leq 1$) preserves the trigonal space group $P\overline{3}1c$. A
small number of impurities of MnTe2 can be identified in the XRD patterns of
$x$ = 0 and 0.4 compoundsXu et al. (2018). The lattice parameters are smoothly
enlarged as the Mg doping. The nominal compositions are close to the EDS-
determined compounds [Supplementary Fig. S1]. We use the nominal $x$ to
indicate the composition of (Mn1-xMgx)3Si2Te6 through out this paper.

Figure 1: (a) Temperature dependence of magnetization of (Mn1-xMgx)3Si2Te6
with the magnetic field parallel to the $c$ axis and $H$ = 1000 Oe. (b)
Temperature dependence of magnetization of (Mn1-xMgx)3Si2Te6 at 2 K with an
applied field parallel to the $c$ axis from -14 to 14 T. The inset shows the
hysteresis loops of the $x$ = 0, 0.1, and 0.3 samples at low fields. (c)
Temperature dependence of resistivity of (Mn1-xMgx)3Si2Te6. The resistivity of
the $x$ = 1.0 sample was measured from 328 to 400 K due to the large magnitude
which exceeded the measurement range at lower temperatures. (d) Temperature
dependence of resistivity of Mn3Si2Te6 with thicknesses of 200 and 80 $\mu$m
measured with various currents.

Figure 1 shows the doping effects on magnetization and resistivity of
(Mn1-xMgx)3Si2Te6. The temperature dependence of magnetization in Fig. 1(a)
reveals a transition at $\sim$78 K for Mn3Si2Te6 that is consistent with the
reported FIM orderMay et al. (2017); Liu et al. (2018); Martinez et al.
(2020). The FIM order is progressively suppressed with increasing doping until
the disappearance for $x>0.5$. In addition to the FIM transition, an abnormal
decline on the magnetization curve can be observed at low temperatures for
$0.1\leq x\leq 0.5$, which could be ascribed to a spin reorientation
transition due to a doping-induced magnetic disorder (Supplementary Fig. S2).
The saturation magnetization decreases with increasing $x$ [Fig. 1(b)]. The
hysteresis loops resulting from the FIM can be observed in the low field range
in both the undoped and doped samples [see inset in Fig. 1(b)].

Figure 2: (a) Resistivity as a function of temperature in different magnetic
fields parallel to the $c$ axis for the undoped sample and (b) for the $x$ =
0.1 sample. (c) Field dependence of $\rho$H/$\rho$0 for the samples of $x$ = 0
and 0.1 at 10 K, (d) 20 K, and for the samples of $x$ = 0, 0.1, and 0.2 at (e)
50 K and (f) 100 K.

Temperature dependence of the resistivity for different compositions is
displayed in Fig. 1(c). Anomalies on the resistivity of $x$ = 0 and 0.1
samples correspond to the $T_{C}$s. When the $x$ exceeds 0.1, the anomaly
associated with $T_{C}$ cannot be observed due to the limit of the measured
resistance range. The values of the resistivity increase from 10-1 to 107
$\Omega$ cm at $\sim$300 K in the doping range of $0\leq x\leq 1$. The
resistivity for Mg3Si2Te6 is too large to be measured below 328 K in our
experimental setupHuang et al. (2022). We fit the resistivity above $T_{C}$
using the activation-energy model $\rho$($T$) = $\rho$0exp($E$a/$k$B$T$),
where $E_{a}$ is the activation energy and $k_{B}$ is the Bolzman constant.
The results reveal linearly increased $E_{a}$ from 60.5 to 378 meV with the
Mg2+ doping [Fig. 5], revealing that the gap changes uniformly as doping,
which may facilitate the electronic structure tuning in real applications.
Figure 1(d) shows the resistivity measurements on Mn3Si2Te6 single crystals
with different currents and thicknesses. The $T_{C}$ can be tuned by the
magnitude of current for the single crystal with a thickness of $\sigma$ = 80
$\mu$m, consistent with the previously proposed COC stateZhang et al. (2022a).
However, the anomalies in resistivity for the $\sigma$ = 200 $\mu$m sample are
hardly changed until the current $I$ is up to 8 mA. According to the
discrepancy of the $T_{C}$s for different currents and samples, the COC state
does not depend on the current magnitude but the current density. The critical
current density for the FIM state at 0 K is estimated to be 95 mA/mm2
[Supplementary Fig. S1(d)].

To reveal the doping effect on the CMR, we show the temperature dependence of
resistivity with magnetic fields parallel to the $c$ axis for the undoped
$x=0$ and doped $x=0.1$ compounds in Figs. 2(a) and 2(b). They all exhibit
large CMR below the $T_{C}$. At 5 K, the resistivity of Mn3Si2Te6 drastically
decreases by about 8 orders of magnitude from 106 $\Omega$cm at zero filed to
10-2 $\Omega$cm at $\mu_{0}H$ = 14 T. The change is smaller for
(Mn0.9Mg0.1)3Si2Te6 at 5 K. However, over a wider temperature range from 10 K
to the $T_{C}$, the drop of resistivity for the doped $x=0.1$ compound under
high magnetic fields is greater. To compare the CMR as doping quantitatively,
we plot $\rho(H)/\rho(0)$ as a function of the magnetic field at 10, 20 and 50
K below the $T_{C}$, and 100 K above the $T_{C}$ in Figs. 2(c-f). Due to the
limitation of the measured resistivity range, the data of the $x=0.2$ compound
at 10 and 20 K are not shown in Figs. 2(c) and 2(d). The change of the
magnitude of $\rho(H)/\rho(0)$ for the doped $x=0.1$ sample is about 2 orders
larger than that of the undoped $x=0$ sample below $T_{C}$ at 14 T. The
magnitudes of $\rho(H)/\rho(0)$ reverse below 6.0 T at 10 K, 2.8 T at 20 K,
and 1.8 T at 50 K, respectively. The CMR effect becomes much weaker above
$T_{C}$ as shown in Fig. 2(f). Thermal fluctuations, SOC, and thermal
activation gap are three ingredients that govern the CMR in (Mn1-xMgx)3Si2Te6.
Increasing the temperature below $T_{C}$ and strengthening the SOC will
promote the conductance of electrons as expected. Our results demonstrate that
substituting Mn2+ by nonmagnetic Mg2+ will enlarge the thermal activation gap,
enhancing the CMR in this system.

Figure 3: (a) Angular dependence of resistivity for Mn3Si2Te6 and (b)
(Mn0.9Mg0.1)3Si2Te6 with $\mu_{0}H$ = 14 T at 20, 50, and 100 K. The inset in
(a) shows the direction of the magnetic field, where $\theta$ is the angle
between $H$ and the $c$ axis.

Figure 3 shows a comparison of the colossal AMR for the undoped ($x=0$) and
doped ($x=0.1$) compounds at 20 and 50 K below $T_{C}$, and 100 K above
$T_{C}$. The AMR exhibits a two-fold rotational symmetry. The resistivity as a
function of the angle of the magnetic field to the $c$ axis evolves close to
linearly, consistent with the scenario that the spins of Mn2+ couple with the
moments of the COC along the edges of the MnTe6 octahedraZhang et al. (2022a).
Upon 10$\%$ Mg doping, the thermal activation gap is enlarged, and the
magnitude of the AMR in (Mn0.9Mg0.1)3Si2Te6 is enhanced an order at the same
measurement conditions. Nonetheless, it is expected that the AMR will be
enhanced in a wider doping range and at lower temperaturesSeo et al. (2021).

Pressure can effectively tune the structure, electronic transport, and
magnetism of layered structural materialsSun et al. (2023a, 2021b); Olmos et
al. (2023). Figure 4(a) displays the high-pressure XRD patterns of Mn3Si2Te6
up to 24.2 GPa. A clear structural transition is identified between 13.2 and
14.6 GPa, consistent with the reported Raman spectroscopy measurementsWang et
al. (2022). The XRD patterns above 14.6 GPa are consistent with the
orthorhombic space group $F222$, where the lattice constants
$a=11.456,b=4.385$, and $c=5.597$ Å at 14.6 GPa. Under pressure, the
resistance measurements reveal a semiconducting to metallic transition [Fig.
4(b)]. Below the pressure of the structural transition, the $T_{C}$ can be
identified as a hump in resistance, yielding a linearly increased $T_{C}$ from
$\sim$78 K at ambient pressure to $\sim$214 K at 10 GPa. The anomalous Hall
effect of Mn3Si2Te6 under pressure is intimately coupled with the magnetism
(Supplementary Fig. S4). The thermal activation gap decreases with increasing
pressure until closing at around 7 GPa, as shown in Fig. 4(d). The resistance
measured at various pressures and temperatures also shows an abrupt drop at
$\sim$7 GPa [Fig. 4(e)]. Figure 4(c) shows typical magnetoresistance ($MR$
defined as ($\rho$H-$\rho$0)/$\rho$${}_{0}\times$100%) measurements at 2.6 GPa
and different temperatures (for more pressures, see supplementary Fig. S5).
The $MR$ shows an abrupt change across $T_{C}=127$ K at 2.6 GPa [Fig. 4 (c)]
and a sharp drop at $\sim$6 GPa [Fig. 4 (f)], revealing that the SOC and
thermal activation gap are the dominating ingredients that govern the $MR$ as
elevated temperature and pressure, respectively.

Figure 4: (a) XRD patterns of Mn3Si2Te6 under pressures from 0.96 to 24.2 GPa.
A structural transition occurs between 13.2 and 14.6 GPa. (b) Temperature
dependence of resistance of Mn3Si2Te6 at various pressures up to 28 GPa. The
Curie transition temperatures $T_{C}$s under pressure are determined from the
humps on resistance marked with arrows. (c) $MR$ defined by
($\rho$H-$\rho$0)/$\rho$${}_{0}\times$100% at various temperatures at 2.6 GPa.
Pressure dependence of (d) the thermal activation gap $E_{a}$ and $T_{C}$, (e)
the resistance with $\mu_{0}H=0$ T, and (f) the $MR$ with $\mu_{0}H=14$ T at
various temperatures.

A comprehensive phase diagram combining the electronic, magnetic, and
structural measurements as functions of doping and pressure is shown in Fig.
5. The $T_{C}$ of the FIM transition decreases as doping but increases as
applying pressure. The evolution of the thermal activation gap, $E_{a}$,
reverses, changing from 378 meV in Mg3Si2Te6 to 60.5 meV in Mn3Si2Te6 at
ambient pressure, and closing at 7 GPa. The magnetoresistance decreases
dramatically as the $E_{a}$ closes, revealing that the CMR of Mn3Si2Te6 is
strongly coupled with $E_{a}$. To see the evolution of magnetoresistance more
clearly, the $MR^{*}$ defined as ($\rho$H-$\rho$0)/$\rho$${}_{H}\times$100% is
shown in the inset of Fig. 5, where the magnitude of $MR^{*}$ changes four
orders. The decrease of the magnitude of $MR^{*}$ is in contrast to the
increased $T_{C}$, revealing that the magnetoresistance is dominated by the
electronic gap. The presence of magnetoresistance requires SOC that is
comparable with the electronic gap. In (Mn1-xMgx)3Si2Te6, the spins of Mn2+
possibly couple with the moments of the COC of the MnTe6 octahedra. In this
scenario, the large CMR and AMR could be understood.

Figure 5: A phase diagram of (Mn1-xMgx)3Si2Te6 aganist doping and pressure.
The paramagnetic (PM), ferrimagnetic (FIM), and spin reorientation (SR)
regions are determined according to the resistance and magnetic measurements.
$MR$ is defined by ($\rho$H-$\rho$0)/$\rho$${}_{0}\times$100%. The inset is a
zoomed-in version of $MR^{*}$ defined by
($\rho$H-$\rho$0)/$\rho$${}_{H}\times$100% in the dashed box area.

In conclusion, we systematically studied the CMR, AMR, FIM, and structures of
(Mn1-xMgx)3Si2Te6 as functions of doping and pressure. As Mn2+ is replaced by
Mg2+ progressively, the FIM order is suppressed gradually until disappearing
for $x>0.5$. For the $x$ is between 0.1 and 0.5, a spin reorientation
transition appears evidenced by a magnetic anomaly on magnetization at low
temperatures. The thermal activation gap is enlarged linearly by the Mg
doping. The CMR and AMR are both enhanced in the low-doped samples compared to
the parent, which can be attributed to a doping-induced broadening of the band
gap and a larger variation range of resistivity. On the contrary, $T_{C}$ is
enhanced with increasing pressure due to the enhanced magnetic exchange
interactions until disappearing when undergoing a structural transition above
13.2 GPa. The gap closes and a MIT occurs at $\sim$6 GPa, where the CMR is
abruptly reduced. Our finding reveals that both doping and pressure are
effective methods to change the band structure and magnetism, consequently
enhancing the CMR and AMR in the FIM (Mn1-xMgx)3Si2Te6.

Work at Sun Yat-Sen University was supported by the National Natural Science
Foundation of China (grant No. 12174454), the Guangdong Basic and Applied
Basic Research Foundation (grant No. 2021B1515120015), the Guangzhou Basic and
Applied Basic Research Funds (grant No. 202201011123), and the Guangdong
Provincial Key Laboratory of Magnetoelectric Physics and Devices (grant No.
2022B1212010008). We also thank the BL15U1 station in Shanghai Synchrotron
Radiation Facility (SSRF) for the help in characterizations.

## References

  * Grünberg et al. (1986) P. Grünberg, R. Schreiber, Y. Pang, M. Brodsky, and H. Sowers, Physical Review Letters 57, 2442 (1986).
  * Baibich et al. (1988) M. N. Baibich, J. M. Broto, A. Fert, F. N. Van Dau, F. Petroff, P. Etienne, G. Creuzet, A. Friederich, and J. Chazelas, Physical Review Letters 61, 2472 (1988).
  * Xiao et al. (1992) J. Q. Xiao, J. S. Jiang, and C. Chien, Physical Review Letters 68, 3749 (1992).
  * Thompson (2008) S. M. Thompson, Journal of Physics D: Applied Physics 41, 093001 (2008).
  * Ennen et al. (2016) I. Ennen, D. Kappe, T. Rempel, C. Glenske, and A. Hütten, Sensors 16, 904 (2016).
  * Diény (1994) B. Diény, Journal of Magnetism and Magnetic Materials 136, 335 (1994).
  * von Helmolt et al. (1993) R. von Helmolt, J. Wecker, B. Holzapfel, L. Schultz, and K. Samwer, Physical Review Letters 71, 2331 (1993).
  * Jin et al. (1994) S. Jin, T. H. Tiefel, M. McCormack, R. Fastnacht, R. Ramesh, and L. Chen, Science 264, 413 (1994).
  * Millis et al. (1996) A. Millis, B. I. Shraiman, and R. Mueller, Physical Review Letters 77, 175 (1996).
  * Wang et al. (2021) Z.-C. Wang, J. D. Rogers, X. Yao, R. Nichols, K. Atay, B. Xu, J. Franklin, I. Sochnikov, P. J. Ryan, D. Haskel, et al., Advanced Materials 33, 2005755 (2021).
  * Shimakawa et al. (1996) Y. Shimakawa, Y. Kubo, and T. Manako, Nature 379, 53 (1996).
  * Majumdar and Littlewood (1998) P. Majumdar and P. Littlewood, Physical Review Letters 81, 1314 (1998).
  * Rosa et al. (2020) P. Rosa, Y. Xu, M. Rahn, J. Souza, S. Kushwaha, L. Veiga, A. Bombardi, S. Thomas, M. Janoschek, E. Bauer, et al., npj Quantum Materials 5, 52 (2020).
  * Yin et al. (2020) J. Yin, C. Wu, L. Li, J. Yu, H. Sun, B. Shen, B. A. Frandsen, D.-X. Yao, and M. Wang, Physical Review Materials 4, 013405 (2020).
  * Yang et al. (2021) H. Yang, Q. Liu, Z. Liao, L. Si, P. Jiang, X. Liu, Y. Guo, J. Yin, M. Wang, Z. Sheng, et al., Physical Review B 104, 214419 (2021).
  * Sun et al. (2021a) Z. Sun, A. Wang, H. Mu, H. Wang, Z. Wang, T. Wu, Z. Wang, X. Zhou, and X. Chen, npj Quantum Materials 6, 94 (2021a).
  * Ogasawara et al. (2021) T. Ogasawara, K.-K. Huynh, T. Tahara, T. Kida, M. Hagiwara, D. Arčon, M. Kimata, S. Y. Matsushita, K. Nagata, and K. Tanigaki, Physical Review B 103, 125108 (2021).
  * Vincent et al. (1986) H. Vincent, D. Leroux, D. Bijaoui, R. Rimet, and C. Schlenker, Journal of Solid State Chemistry 63, 349 (1986).
  * Ni et al. (2021) Y. Ni, H. Zhao, Y. Zhang, B. Hu, I. Kimchi, and G. Cao, Physical Review B 103, L161105 (2021).
  * Seo et al. (2021) J. Seo, C. De, H. Ha, J. E. Lee, S. Park, J. Park, Y. Skourski, E. S. Choi, B. Kim, G. Y. Cho, et al., Nature 599, 576 (2021).
  * Sala et al. (2022) G. Sala, J. Lin, A. Samarakoon, D. Parker, A. May, and M. Stone, Physical Review B 105, 214405 (2022).
  * Kwon et al. (2023) C. I. Kwon, K. Kim, S. Y. Kim, R. A. Susilo, B. Kang, K. Kim, D. Y. Kim, J. Kim, B. Kim, and J. S. Kim, Current Applied Physics 53, 51 (2023).
  * Mijin et al. (2023) S. D. Mijin, A. Šolajić, J. Pešić, Y. Liu, C. Petrovic, M. Bockstedte, A. Bonanni, Z. Popović, and N. Lazarević, Physical Review B 107, 054309 (2023).
  * Lovesey (2023) S. W. Lovesey, Physical Review B 107, 224410 (2023).
  * Zhang et al. (2022a) Y. Zhang, Y. Ni, H. Zhao, S. Hakani, F. Ye, L. DeLong, I. Kimchi, and G. Cao, Nature 611, 467 (2022a).
  * Ye et al. (2022) F. Ye, M. Matsuda, Z. Morgan, T. Sherline, Y. Ni, H. Zhao, and G. Cao, Physical Review B 106, L180402 (2022).
  * Zhang et al. (2022b) Q. Zhang, J. Liu, H. Cao, A. Phelan, D. Graf, J. DiTusa, D. A. Tennant, and Z. Mao, NPG Asia Materials 14, 1884 (2022b).
  * Zhang et al. (2023) Y. Zhang, L.-F. Lin, A. Moreo, and E. Dagotto, Physical Review B 107, 054430 (2023).
  * Sun et al. (2023a) H. Sun, L. Qiu, Y. Han, E. Yi, J. Li, M. Huo, C. Huang, H. Liu, M. Li, W. Wang, et al., Materials Today Physics 36, 101188 (2023a).
  * Sun et al. (2023b) H. Sun, L. Qiu, Y. Han, Y. Zhang, W. Wang, C. Huang, N. Liu, M. Huo, L. Li, H. Liu, et al., Communications Physics 6, 40 (2023b).
  * Sun et al. (2021b) H. Sun, C. Chen, Y. Hou, W. Wang, Y. Gong, M. Huo, L. Li, J. Yu, W. Cai, N. Liu, et al., Science China Physics, Mechanics & Astronomy 64, 118211 (2021b).
  * Cai et al. (2020) W. Cai, H. Sun, W. Xia, C. Wu, Y. Liu, H. Liu, Y. Gong, D.-X. Yao, Y. Guo, and M. Wang, Physical Review B 102, 144525 (2020).
  * Xu et al. (2018) Y. Xu, W. Li, C. Wang, Z. Chen, Y. Wu, X. Zhang, J. Li, S. Lin, Y. Chen, and Y. Pei, Journal of Materiomics 4, 215 (2018).
  * May et al. (2017) A. F. May, Y. Liu, S. Calder, D. S. Parker, T. Pandey, E. Cakmak, H. Cao, J. Yan, and M. A. McGuire, Physical Review B 95, 174440 (2017).
  * Liu et al. (2018) Y. Liu, C. Petrovic, et al., Physical Review B 98, 064423 (2018).
  * Martinez et al. (2020) L. Martinez, H. Iturriaga, R. Olmos, L. Shao, Y. Liu, T. T. Mai, C. Petrovic, A. R. Hight Walker, and S. Singamaneni, Applied Physics Letters 116, 172404 (2020).
  * Huang et al. (2022) C. Huang, B. Cheng, Y. Zhang, L. Jiang, L. Li, M. Huo, H. Liu, X. Huang, F. Liang, L. Chen, et al., Chinese Physics B (2022).
  * Olmos et al. (2023) R. Olmos, P.-H. Chang, P. Mishra, R. R. Zope, T. Baruah, Y. Liu, C. Petrovic, and S. R. Singamaneni, The Journal of Physical Chemistry C 127, 10324 (2023).
  * Wang et al. (2022) J. Wang, S. Wang, X. He, Y. Zhou, C. An, M. Zhang, Y. Zhou, Y. Han, X. Chen, J. Zhou, et al., Physical Review B 106, 045106 (2022).


# Full or Weak annotations?
An adaptive strategy for budget-constrained annotation campaigns

Javier Gamazo Tejero1, Martin S. Zinkernagel2, Sebastian Wolf2
Raphael Sznitman1, Pablo Márquez Neila1
1University of Bern, 2Inselspital Bern, Switzerland
{javier.gamazo-tejero, raphael.sznitman<EMAIL_ADDRESS>
{martin.zinkernagel<EMAIL_ADDRESS>
###### Abstract

Annotating new datasets for machine learning tasks is tedious, time-consuming,
and costly. For segmentation applications, the burden is particularly high as
manual delineations of relevant image content are often extremely expensive or
can only be done by experts with domain-specific knowledge. Thanks to
developments in transfer learning and training with weak supervision,
segmentation models can now also greatly benefit from annotations of different
kinds. However, for any new domain application looking to use weak
supervision, the dataset builder still needs to define a strategy to
distribute full segmentation and other weak annotations. Doing so is
challenging, however, as it is a priori unknown how to distribute an
annotation budget for a given new dataset. To this end, we propose a novel
approach to determine annotation strategies for segmentation datasets, whereby
estimating what proportion of segmentation and classification annotations
should be collected given a fixed budget. To do so, our method sequentially
determines proportions of segmentation and classification annotations to
collect for budget-fractions by modeling the expected improvement of the final
segmentation model. We show in our experiments that our approach yields
annotations that perform very close to the optimal for a number of different
annotation budgets and datasets.

##  1 Introduction

Semantic segmentation is a fundamental computer vision task with applications
in numerous domains such as autonomous driving [11, 43], scene understanding
[45], surveillance [50] and medical diagnosis [9, 18]. As the advent of deep
learning has significantly advanced the state-of-the-art, many new application
areas have come to light and continue to do so too. This growth has brought
and continues to bring exciting domain-specific datasets for segmentation
tasks [19, 29, 6, 32, 52].

Today, the process of establishing machine learning-based segmentation models
for any new application is relatively well understood and standard. Only once
an image dataset is gathered and curated, can machine learning models be
trained and validated. In contrast, building appropriate datasets is known to
be difficult, time-consuming, and yet paramount. Beyond the fact that
collecting images can be tedious, a far more challenging task is producing
ground-truth segmentation annotations to subsequently train (semi) supervised
machine learning models. This is mainly because producing segmentation
annotations often remains a manual task. As reported in [4], generating
segmentation annotations for a single PASCAL image [15] takes over 200 seconds
on average. This implies over 250 hours of annotation time for a dataset
containing a modest 5’000 images. What often further exacerbates the problem
for domain-specific datasets is that only the dataset designer, or a small
group of individuals, have enough expertise to produce the annotations
(_e.g_., doctors, experts, etc.), making crowd-sourcing ill-suited.

Figure 1: Illustration of different semantic segmentation applications; OCT:
Pathologies of the eye in OCT images, SUIM: Underwater scene segmentation
[19], Cityscape: street level scene segmentation [11], PASCAL VOC: natural
object segmentation.

To overcome this challenge, different paradigms have been suggested over the
years. Approaches such as Active Learning [7, 8, 26] aim to iteratively
identify subsets of images to annotate so as to yield highly performing
models. Transfer learning has also proved to be an important tool in reducing
annotation tasks [13, 17, 24, 25, 30, 36]. For instance, [37] show that
training segmentation models from scratch is often inferior to using pre-
training models derived from large image classification datasets, even when
the target application domain differs from the source domain. Finally, weakly-
supervised methods [2, 40] combine pixel-wise annotations with other weak
annotations that are faster to acquire, thereby reducing the annotation
burden. In particular, Papandreou  _et al_. [40] showed that combinations of
strong and weak annotations (_e.g_., bounding boxes, keypoints, or image-level
tags) delivered competitive results with a reduced annotation effort. In this
work, we rely on these observations and focus on the weakly supervised
segmentation setting.

In the frame of designing annotation campaigns, weakly-supervised approaches
present opportunities for efficiency as well. Instead of completely spending a
budget on a few expensive annotations, weakly-supervised methods allow a
proportion of the budget to be allocated to inexpensive, or weak, labels. That
is, one could spend the entire annotation budget to manually segment available
images, but would ultimately lead to relatively few annotations. Conversely,
weak annotations such as image-level labels are roughly 100 times cheaper to
gather than their segmentation counterparts [4]. Thus, a greater number of
weakly-annotated images could be used to train segmentation models at an equal
cost. In fact, under a fixed budget, allocating a proportion of the budget to
inexpensive image-level class labels has been shown to yield superior
performance compared to entirely allocating a budget to segmentation labels
[4].

Yet, allocating how an annotation budget should be distributed among strong
and weak annotations is challenging, and inappropriate allocations may
severely impact the quality of the final segmentation model. For example,
spending the entire budget on image-level annotations will clearly hurt the
performance of a subsequent segmentation model. Instead, a naive solution
would be to segment and classify a fixed proportion of each (_e.g_., say 80% -
20%). Knowing what proportion to use for a given dataset is unclear, however.
Beyond this, there is no reason why the same fixed proportion would be
appropriate across different datasets or application domains. That is, it
would be highly unlikely that the datasets shown in Fig. 1 all require the
same proportion of strong and weak annotations to yield optimal segmentation
models.

Despite its importance, choosing the best proportion of annotation types
remains a largely unexplored research question. Weakly-supervised and
transfer-learning methods generally assume that the annotation campaign and
the model training are independent and that all annotations are simply
available at training time. While active learning methods do alternate between
annotation and training, they focus on choosing optimal samples to annotate
rather than choosing the right type of annotations. Moreover, most active
learning methods ignore constraints imposed by an annotation budget. More
notable, however, is the recent work of Mahmood et. al. [33, 34] which aims to
determine what weak and strong annotation strategy is necessary to achieve a
target performance level. While noteworthy, this objective differs from that
here, whereby given a fixed budget, what strategy is best suited for a given
new dataset?

To this end, we propose a novel method to find an optimal budget allocation
strategy in an online manner. Using a collection of unlabeled images and a
maximum budget, our approach selects strong and weak annotations, constrained
by a given budget, that maximize the performance of the subsequent trained
segmentation model. To do this, our method iteratively alternates between
partial budget allocations, label acquisition, and model training. At each
step, we use the annotations performed so far to train multiple models to
estimate how different proportions of weak and strong annotations affect model
performance. A Gaussian Process models these results and maps the number of
weak and strong annotations to the expected model improvement. Computing the
Pareto optima between expected improvement and costs, we choose a new sub-
budget installment and its associated allocation so to yield the maximum
expected improvement. We show in our experiments that our approach is
beneficial for a broad range of datasets, and illustrate that our dynamic
strategy allows for high performances, close to optimal fixed strategies that
cannot be determined beforehand.

##  2 Related work

###  2.1 Weak annotations for segmentation

Weakly supervised semantic segmentation (WSSS) relies on coarser annotations,
such as bounding boxes [46], scribbles [31, 49] or image-level classification
labels [1], to train a segmentation network. WSSS methods have often employed
saliency maps as weak annotations for segmentation models, as these are
typically obtained from CAM [55], which leverages image-level classification
annotation. These methods then focus on refining the saliency maps with a
variety of techniques [16, 28]. Others make use of attention to achieve coarse
segmentations [20, 23]. Conversely, [54] combined annotations in the form of
bounding boxes and image-level labels to accurately generate image graphs, to
be used by a graph neural network to predict node values corresponding to
pixel labels. In this context, the work in [33] and [34] are close to this
one, whereby their objective is to determine what annotation strategy over
annotation types is likely to yield a target performance level.

###  2.2 Transfer learning

Due to the limited availability of annotated image data in some domains, it is
now common to use neural networks pre-trained on large image classification
tasks [12] for subsequent target tasks. Specifically, in cases where the
target task has limited data or annotations, this has been shown to be
particularly advantageous. Among others, this practice is now widely used in
medical imaging and has been linked to important performance gains after fine-
tuning [13, 14, 36, 48, 24].

Efforts are now pivoting towards the use of in-domain pre-training, avoiding
the leap of faith that is often taken with Imagenet [17, 30]. In [30], the
model is pre-trained on ChestX-ray14 [51] to more accurately detect pneumonia
in chest X-ray images from children. In [17], the authors show that joint
classification and segmentation training, along with pre-training on other
medical datasets that have domain similarity, increases segmentation
performances with respect to the segmentation using Imagenet-based pre-
training.

Alternatively, cross-task methods seek to transfer features learned on one
task (_e.g_. classification, normal estimation, etc.) to another, usually more
complex one. Along this line, Taskonomy [53] explored transfer learning
capabilities among a number of semantic tasks and built a task similarity tree
that provided a clustered view of how much information is available when
transferring to other tasks. Similarly, [37] performed an extensive study of
cross-task transfer capabilities for a variety of datasets, reaching the
conclusion that Imagenet pre-training outperforms random initialization in all
cases, but further training on related tasks or domains also brings additional
benefits.

###  2.3 Active learning

In active learning, the goal is to train a model while querying an oracle to
label new samples that are expected to improve the model’s accuracy. In
computer vision, it has been applied to image classification [22, 41] or
semantic segmentation [3, 5, 44] among others. As a byproduct, Active learning
has also been used as a way to reduce labeling time. For example, [27]
describes a method that couples Reinforcement Learning and Active Learning to
derive the shortest sequence of annotation actions that will lead to object
detection within an image. Others have focused on speeding up this process via
eye-tracking [38] or extreme clicking [39]. As such, Active Learning is
related to the present work in the sense that our approach is adaptive but
differs in that our method determines what annotations types should be
collected under a constrained budget instead of predicting at each time step
which samples should be added to the annotated set.

##  3 Method

Training segmentation models using a combination of expensive pixel-wise
annotations and other types of cheaper annotations, such as image-wise labels
or single-pixel annotations is known to be beneficial, as well as using cross-
task transfer learning techniques [37]. This is motivated by empirical
findings showing that, under a limited annotation budget, allocating a
proportion of the budget to inexpensive image-level class labels led to
superior performance compared to allocating the budget entirely to
segmentation labels [4]. However, the optimal proportion of the budget to
allocate per annotation type is a-priori unknown beforehand and data-
dependent. Thus, the goal of our method is to find this data-specific optimal
budget allocation in an online manner, as it is necessary for any dataset
builder starting off.

We describe our method in the subsequent sections. For clarity, we focus on
image segmentation and assume two kinds of annotations are possible: strong
annotations as segmentation labels and weak annotations as image-level
classification labels. Generalizing this formulation to other tasks or
settings with more than two annotations types should follow directly.

###  3.1 Problem formulation

Let $p_{\textrm{data}}(\mathbf{x})$ be the distribution of training images for
which we have no annotations initially. Each training image $\mathbf{x}$ can
be annotated with a pixel-wise segmentation labeling
$(\mathbf{x},\mathbf{y})\sim{}p_{\textrm{data}}(\mathbf{x})p_{\textrm{sgm}}(\mathbf{y}\mid\mathbf{x})$
or an image-wise classification annotation
$(\mathbf{x},c)\sim{}p_{\textrm{data}}(\mathbf{x})p_{\textrm{cls}}(c\mid\mathbf{x})$Sampling
from the distributions $p_{\textrm{cls}}$ and $p_{\textrm{sgm}}$ represents
the task of manually annotating the image and has associated costs of
$\alpha_{\textrm{c}}>0$ and $\alpha_{\textrm{s}}>0$, respectively. Supported
by previous work [4, 37, 33], we will assume that
$\alpha_{\textrm{s}}\gg\alpha_{\textrm{c}}$.

By sampling $C$ classifications from $p_{\textrm{cls}}$ and $S$ segmentation
from $p_{\textrm{sgm}}$, we can build an annotated training dataset
$\mathcal{T}=(\mathcal{T}_{c},\mathcal{T}_{s})\sim{}(p_{\textrm{cls}}^{C},p_{\textrm{sgm}}^{S})$.
The dataset $\mathcal{T}$ then has an annotation cost,

$\alpha_{\textrm{c}}C+\alpha_{\textrm{s}}S,$ (1)

which we assume to be bounded by an upper limit, or _budget_ , $B$.

To annotate $\mathcal{T}$, however, we can choose different _allocation
strategies_ , or combinations of $C$ and $S$, that have different costs and
that yield different segmentation model performances. The utility $u$ of an
allocation strategy $(C,S)$ is the expected performance of a model trained
with datasets that follow that strategy,

$u(C,S)=\mathbb{E}_{(\mathcal{T}_{c},\mathcal{T}_{s})\sim{}(p_{\textrm{cls}}^{C},p_{\textrm{sgm}}^{S})}\left[m(\mathcal{T}_{c},\mathcal{T}_{s})\right],$
(2)

where $m(\mathcal{T}_{c},\mathcal{T}_{s})$ is the performance score (_e.g_.,
Dice score, IoU) of a segmentation model trained with datasets
($\mathcal{T}_{c},\mathcal{T}_{s}$) and evaluated on a separate fixed test
dataset. Note that in contrast to Active Learning, the utility is defined over
the set of strategies $(C,S)$ and not over the individual samples of a fixed
training set. This is motivated by our aim to estimate the performance of the
annotation strategy $(C,S)$ and not the ensuing specific training dataset.

Figure 2: Illustration of proposed method. At a given iteration $t$, $C_{t}$
and $S_{t}$ classification and segmentation annotations have already been
collected (blue region, left panel) with a budget of $B_{t}$. For the next
annotation phase, the budget is increased to $B_{t+1}$. To determine how many
new classification and segmentation annotations to collect, $M$ combinations
of different quantities $(C^{(i)},S^{(i)})$ are gathered according to Alg. 1
to compute $m(C^{(i)},S^{(i)})$. A Gaussian Process is then trained to
estimate the utility of different combinations of annotation types (light blue
area, left panel). From this, we infer $\Delta C$ and $\Delta S$ to select
next by computing the combination that maximizes the expected improvement
along the Pareto front given by the budget $B_{2}$ (red point, left panel).
The next iteration starts then with the new proportions (red point, right
panel) and follows the same steps (see text and Alg. 2 for details). For
illustration purposes, the costs are set here to $\alpha_{c}=\alpha_{s}=1$.

Our goal then is to find the annotation strategy that maximizes the expected
performance constrained to a budget $B$,

$\displaystyle\max_{(C,S)\in\mathbb{N}^{2}}$ $\displaystyle u(C,S),$ (3) s.t.
$\displaystyle\alpha_{c}C+\alpha_{s}S\leq B.$

In the following, we describe how we optimize Eq. (3).

###  3.2 Utility model

As defined Eq. (2), the utility function, $u$, marginalizes over all possible
training sets, which is intractable to compute in practice. To overcome this
computational challenge, we approximate $u$ with a collection $\mathcal{M}$ of
discrete samples, where each sample $m\in\mathcal{M}$ is a tuple containing an
allocation strategy $(C,S)$ and the estimated score
$m(\mathcal{T}_{c},\mathcal{T}_{s})$ obtained for a dataset sampled with that
allocation strategy. To build $\mathcal{M}$, one could simply sample a random
strategy $(C^{\prime},S^{\prime})$, annotate a dataset
$(\mathcal{T}^{\prime}_{c},\mathcal{T}^{\prime}_{s})\sim{}(p_{\textrm{cls}}^{C^{\prime}},p_{\textrm{sgm}}^{S^{\prime}})$,
and measure its performance. However, this would imply annotating for
different potential budgets and is thus infeasible in practice. Instead, a
practical alternative is to leverage previously annotated data
$(\mathcal{T}_{c},\mathcal{T}_{s})$. For each sampled strategy
$(C^{\prime},S^{\prime})$, we build the corresponding dataset
$(\mathcal{T}^{\prime}_{c},\mathcal{T}^{\prime}_{s})$ by taking random samples
from the already annotated data according to the strategy. While this
procedure, formalized in Alg. 1, leads to biased samples, we empirically found
this bias to have a minor impact on the final strategies compared to
estimations with unbiased sampling.

Algorithm 1 Build utility samples from annotated data

1:function BuildUtilitySamples($\mathcal{T}_{c},\mathcal{T}_{s}$)

2: $C\leftarrow|\mathcal{T}_{c}|$, $S\leftarrow|\mathcal{T}_{s}|$

3: $\mathcal{M}\leftarrow\\{((C,S),m(\mathcal{T}_{c},\mathcal{T}_{s}))\\}$
$\triangleright$ Add sample with all the available data

4: repeat $M-1$ times

5: Sample $(C^{\prime},S^{\prime})\in[0,C]\times[0,S]$

6: $\mathcal{T}^{\prime}_{c}\leftarrow$ {$C^{\prime}$ elements sampled from
$\mathcal{T}_{c}$}

7: $\mathcal{T}^{\prime}_{s}\leftarrow$ {$S^{\prime}$ elements sampled from
$\mathcal{T}_{s}$}

8:
$\mathcal{M}\leftarrow\mathcal{M}\cup((C^{\prime},S^{\prime}),m(\mathcal{T}^{\prime}_{c},\mathcal{T}^{\prime}_{s}))$

9: end repeat

10:end function

While $\mathcal{M}$ provides an estimation of $u$ as a set of discrete
locations, we generalize these estimations to the entire space of strategies
by fitting a Gaussian process (GP) to the samples in $\mathcal{M}$. The
Gaussian process, $\mathcal{GP}(\mu,k)$ is parameterized by a suitable mean
function $\mu$ and covariance function $k$. In our case, we use the mean
function,

$\mu(C,S)=\gamma_{c}\log(\beta_{c}C+1)+\gamma_{s}\log(\beta_{s}S+1),$ (4)

which accounts for the fact that the segmentation performance increases
logarithmically with the volume of the training data [47] and that each
annotation type has a different rate of performance growth. Similarly, the
covariance $k$ is a combination of two RBF kernels with different scales
$\ell_{c}$, $\ell_{s}$ for each annotation type,

$k\left((C,S),(C^{\prime},S^{\prime})\right)=\sigma^{2}e^{-\frac{(C-C^{\prime})^{2}}{2\ell_{c}^{2}}}e^{-\frac{(S-S^{\prime})^{2}}{2\ell_{s}^{2}}}.$
(5)

The values $\gamma_{c}$, $\beta_{c}$, $\gamma_{s}$, $\beta_{s}$ from the mean,
the length scales $\ell_{c}$, $\ell_{s}$ and the amplitude $\sigma$ from the
covariance are trainable parameters of the GP.

The trained GP models a distribution over utility functions,
$u\sim\mathcal{GP}(\mu,k)$, that are plausible under the samples
$\mathcal{M}$. This distribution represents not only the expected utility, but
also its uncertainty in different areas of the strategy space. Sampling just a
single $u$ from the GP to solve Eq. (3) would thus be suboptimal. For this
reason, we substitute the utility $u$ in Eq. (3) by a surrogate function
$\hat{u}$ that trades-off exploitation and exploration, thus incorporating
uncertainty information into the optimization problem. Following a Bayesian
optimization approach [21], we choose $\hat{u}$ to be the expected improvement
(EI),

$\hat{u}(C,S)=\mathbb{E}_{u\sim\mathcal{GP}_{t}}[\max\\{u(C,S)-m^{*},0\\}],$
(6)

where $m^{*}$ is the current maximum point.

###  3.3 Optimization

Training the GP requires annotated data to build the set $\mathcal{M}$, which
in turn relies on an annotation strategy that we are trying to find, whereby
implying a circular dependency. We address this circular dependency by
optimizing Eq. (3) in an iterative manner.

Our algorithm shown in Alg. 2, allocates the available budget $B$ in a fixed
number of adaptive installments, alternating between data annotation with the
current strategy, GP fitting, and strategy selection for the next budget
installment. More specifically, our method starts with an initial strategy
$(C_{0},S_{0})$ with associated cost $B_{0}$. At each iteration $t$, new data
is annotated according to the current strategy $(C_{t},S_{t})$ so that the
sets of annotated data $(\mathcal{T}_{c},\mathcal{T}_{s})$ contain $C_{t}$
classification and $S_{t}$ segmentation annotations, respectively. From the
available annotated data $(\mathcal{T}_{c},\mathcal{T}_{s})$, we extract new
samples for $\mathcal{M}$ and fit the GP, which defines the surrogate function
$\hat{u}_{t}$. The corresponding current maximal point $m^{*}_{t}$ is set to
be the maximum performance found so far, (_i.e_., the performance of the model
trained with all the annotated data available at this iteration),
$m^{*}_{t}=m(\mathcal{T}_{c},\mathcal{T}_{s})$. Finally, this surrogate
function is used to estimate the next best strategy $(C_{t+1},S_{t+1})$. We
find a delta strategy $(\Delta{}C,\Delta{}S)$ that increases the expected
improvement by a fixed fraction of its maximum possible value,

$\displaystyle\operatorname*{arg\,min}_{(\Delta{}C,\Delta{}S)\in\mathbb{N}^{2}}$
$\displaystyle\alpha_{c}(C_{t}+\Delta{}C)+\alpha_{s}(S_{t}+\Delta{}S),$ (7)
s.t.
$\displaystyle\hat{u}_{t}(C_{t}+\Delta{}C,S_{t}+\Delta{}S)\geq\dfrac{1}{T-t}\hat{u}^{*}_{t},$

where $T$ is the desired maximum number of iterations of the algorithm and
$\hat{u}_{t}^{*}$ is the maximum expected improvement that can be reached
using the entire budget $B$ for the current surrogate function $\hat{u}_{t}$
according to Eq. (3). The found delta strategy defines the new strategy
$(C_{t+1},S_{t+1})=(C_{t}+\Delta{}C,S_{t}+\Delta{}S)$ for the next iteration.
The process is depicted in Fig. 2.

Algorithm 2 Proposed approach

1:Number of iterations $T$, initial labelling strategy $(C_{0},S_{0})$

2:$t\leftarrow 0$, $\Delta{}C\leftarrow C_{0}$, $\Delta{}S\leftarrow S_{0}$,
$\mathcal{T}_{c}=\emptyset$, $\mathcal{T}_{s}=\emptyset$,
$\mathcal{M}=\emptyset$

3:while $t<T$ do

4: Annotate new data
$(\Delta{}\mathcal{T}_{c},\Delta{}\mathcal{T}_{s})\sim{}(p_{\textrm{cls}}^{\Delta{}C},p_{\textrm{sgm}}^{\Delta{}S})$

5:
$\mathcal{T}_{c}\leftarrow\mathcal{T}_{c}\cup\Delta{}\mathcal{T}_{c},\quad\mathcal{T}_{s}\leftarrow\mathcal{T}_{s}\cup\Delta{}\mathcal{T}_{s}$
$\triangleright$ Note that $|\mathcal{T}_{c}|=C_{t}$ and
$|\mathcal{T}_{s}|=S_{t}$

6:
$\mathcal{M}\leftarrow\mathcal{M}\,\cup\,$BuildUtilitySamples($\mathcal{T}_{c}$,
$\mathcal{T}_{s}$)

7: Train GP with samples in $\mathcal{M}$

8: Compute $(\Delta{}C,\Delta{}S)$ from Eq. (7)

9: $C_{t+1}\leftarrow C_{t}+\Delta{}C,\quad S_{t+1}\leftarrow S_{t}+\Delta{}S$

10: $t\leftarrow t+1$

11:end while

12:$(C_{T},S_{T})$

Note that solving Eq. (7) requires finding $\hat{u}_{t}^{*}$, which in turn
requires solving Eq. (3). While solving two optimization problems may seem
unnecessary, the solutions of both problems are in the Pareto front of
strategies (_i.e_., the set of non-dominated strategies for which no other
strategy has simultaneously smaller cost and larger or equal expected
improvement). Given that the space of strategies is discrete, the elements of
the Pareto front can be easily found in linear time by enumerating all
possible strategies, computing their costs and expected improvements with
$\hat{u}_{t}$, and discarding the dominated elements. Given the Pareto front,
the strategy with the maximum EI $u^{*}_{t}$ and the strategy of minimum
budget with EI larger than $\frac{1}{T-t}\hat{u}^{*}_{t}$, which correspond to
the solutions of Eq.(3) and Eq. (7), respectively, can be found in linear
time.

##  4 Experimental setup

To validate our approach, we evaluated it on four different datasets, while
comparing its performance to a set of typical fixed budget allocation
strategies. In addition, we explore the impact of different hyper parameters
on the overall performance of the method.

Figure 3:  Performance of our method (orange line) on OCT, PASCAL VOC, SUIM
and Cityscapes datasets. Shaded region is computed from three seeds. Fixed
strategies are shown in blue. Red points show the estimated-best-fixed
strategy with $B_{0}$. Labels expressed as percentage of the budget allocated
to segmentation. Note that the first budget $B$ fulfills $B\gg B_{0}$ in all
cases.  Figure 4: Mean of our method with $\alpha_{s}=\\{5,12,25,50\\}$ on
Cityscapes (orange, line). Shaded region is computed from three seeds. Fixed
strategies are shown in blue. Labels expressed as percentage of the budget
allocated to segmentation.

###  4.1 Datasets

We chose a collection of datasets with different image modalities, including a
medical dataset as they often suffer from data and annotation scarcity. In
this context, they represent a typical new application domain where our method
could be particularly helpful. In each case, we enumerate the number of images
for which classification or segmentation images can be sampled by a method:

Augmented PASCAL VOC 2012 [15]:

5’717 classification and 10’582 segmentation natural images with 21 classes
for training. The validation sets contain 1’449 segmented images.

SUIM [19]:

training set consists of 1’525 underwater images with annotations for 8
classes. For evaluation, we used a separate split of 110 additional images.
The classification labels were estimated from the segmentation ground-truth as
a multi-label problem by setting the class label to 1 if the segmentation map
contained at least one pixel assigned to that class.

Cityscapes [11]:

2’975 annotated images for both classification and segmentation are available
for training. We test on the official Cityscapes validation set, which
contains 500 images.

OCT:

22’723 Optical Coherence Tomography (OCT) images with classification
annotations and 1,002 images with pixel-wise annotations corresponding to 4
different types of retinal fluid for segmentation. We split the data into 902
training images and 100 test images.

###  4.2 Baseline strategies.

We compared our method to ten different fixed budget allocation strategies.
Each of these randomly sample images for classification and segmentation
annotations according to a specified and fixed proportion. We denote these
policies by the percentage dedicated to segmentation annotations: $B_{0}$:
$50\%,55\%,\ldots,95\%$ with increases in 5%. For fair comparison, the
strategies are computed from the budget $B_{0}$.

In addition, we consider an estimated-best-fixed budget allocation strategy,
whereby the method estimates what fixed budget should be used for a given
dataset. This is done by using the initial budget $B_{0}$ to compute the best
performing fixed strategy (mentioned above) and then using this fixed strategy
for the annotation campaign until budget $B$ is reached. This strategy
represents an individual that chooses to explore all fixed strategies for an
initial small budget and then exploit it.

###  4.3 Implementation details.

Weakly supervised segmentation model: To train a segmentation model that uses
both segmentation and classifications, we first train the models with the
weakly-annotated data $\mathcal{T}_{c}$ until convergence and then with the
segmentation data $\mathcal{T}_{s}$. We use the U-Net segmentation model [42]
for OCT, and the DeepLabv3 model [10] with a ResNet50 backbone on the SUIM,
PASCAL, and Cityscapes. For the U-Net, a classification head is appended at
the end of the encoding module for the classification task. For the DeepLab-
like models, we train the entire backbone on the classification task and then
add the ASPP head for segmentation. In all cases, we use the cross-entropy
loss for classification and the average of the Dice loss and the cross-Entropy
loss for segmentation. While we choose this training strategy for its
simplicity, other cross-task or weakly supervised alternatives could have been
used as well [2, 40]. Additional details are provided in the supplementary
materials.

Note that all models are randomly initialized to maximize the impact of
classification labels, as Imagenet-pretraining shares a high resemblance to
images in PASCAL and Cityscapes. Failing to do so would lead to classification
training not adding significant information and may even hurt performance due
to catastrophic forgetting [35].

Hyperparameters:  We measured costs in terms of class-label equivalents
setting $\alpha_{c}=1$ and leaving only $\alpha_{s}$ as a hyperparameter of
our method. We set $\alpha_{s}=12$ for all datasets following previous studies
on crowdsourced annotations [4]. We predict the first GP surface with 8% of
the dataset for both classification and segmentation. This quantity is reduced
for OCT classification and VOC segmentation due to the high number of labels
available. In all cases, we fixed the number of iterative steps to 8 and set
the learning rate of the GP to 0.1.

##  5 Results

Figure 5: Mean of our method when using different numbers of iteration steps
$\\{3,5,8,10\\}$. Results shown with three seeds.  Figure 6: Cityscapes (top)
and SUIM (bottom) ground truth budget-segmentation surfaces. We note that
segmentation performance grows logarithmically with training set size on
Cityscapes (as well as OCT and VOC, see the Supplementary materials). This
trend is not observed on the SUIM dataset.

Main results: Figure 3 compares the performance achieved by our method against
that of the different fixed strategies and the estimated best fixed strategy
when using $\alpha_{s}=12$ across the different datasets. From these results
we can make a number of key observations.

First, we can observe that no single fixed strategy is performing optimally
across the different datasets evaluated. This is coherent with our initial
claims and with the literature. Indeed, for OCT the best strategy appears to
be one that samples 90% of segmentations, while this same policy performs
poorly on the SUIM dataset. This implies that blindly using a fixed policy
would on average not be very effective.

Second, the estimated best-fixed strategy (in red) appears to do well
initially and progressively loses competitiveness as the budget increases.
This behaviour is expected as the estimated fixed strategy is that with
$B_{0}$ (the lowest budget), and becomes increasingly irrelevant as $B$ grows.
This is particularly clear on VOC where the best low-budget strategy allocates
95% of the budget to segmentation and still achieves superior performance up
to $B=12^{\prime}000$. However, that strategy drops below average performance
with budgets greater than $B=25^{\prime}000$. In the case of SUIM, the best-
fixed strategy corresponds to 50% of the budget allocated to segmentation.
Since the dataset contains only 1,525 segmentation samples, this strategy is
not attainable with $B>4000$.

Last, we can observe that our method is consistently able to produce good
performances, across both different budget quantities and datasets. We can
also clearly see that our strategy is not guaranteed to be the top performing
strategy, but that on average it performs well in different cases.

At the same time, we notice that the performance of our approach on SUIM
begins well and then drops after a 3’500 budget. This differs sharply from the
other datasets. By observing the true budget-performance surface of SUIM and
the other datasets (see Fig. 6), we can see that the SUIM surface does not
grow logarithmically with the dataset size, while it does for Cityscapes (and
the other too, see the Supplementary materials). This is relevant as our GP
mean prior (4) assumes this relationship and explains why our approach fails
when the true surface deviates from our GP mean form. While the use of
adaptive, higher-level order priors would be beneficial to deal with such
cases, we leave this as future work to be researched.

###  5.1 Sensitivity to $\alpha_{s}$ and $T$

Different types of annotations or domains may have different ratios of cost.
While we have fixed $\alpha_{s}$ in our experiments across all datasets
regardless of their domain, some datasets such as OCT and VOC require
different expertise and domain knowledge to annotate and thus different
$\alpha_{s}$. In Fig. 4, we three additional values of
$\alpha_{s}=\\{5,12,25,50\\}$ and show the performance implication it has on
our methods and the baselines. For Cityscapes, we see that the method is
robust regardless of the value of $\alpha_{s}$, showing above average
performance especially for $\alpha_{s}=25$ and $\alpha_{s}=50$. This behavior
is reproduced in all four datasets (see Supplementary materials).

Similarly, the number of steps $T$ given to reach the final budget is a
hyperparameter of our approach. While low $T$ values could lead to poor
solutions due to the unreliability of the GP far from the sampled region,
higher $T$ values (_i.e_., therefore smaller steps) may exacerbate the
intrinsic greedy nature of our method. We thus seek a trade-off between
reliability and greediness. To study the sensitivity of the algorithm with
respect to this variable, we show the behaviour of our method with different
number of steps in Fig. 5. We see that lower $T$ values greatly affect the
reliability of the found strategy, especially for OCT and SUIM (blue line).
However, as the number of steps increases, the variance of the strategy
reduces sharply. We can therefore conclude that the method is robust to this
hyperparameter as long as it is kept within reasonable ranges.

##  6 Conclusion

In this paper, we propose a novel approach to determine a dynamic annotation
strategy for building segmentation datasets. We design an iterative process
that identifies efficient dataset-specific combinations of weak annotations in
the form of image-level labels and full segmentations. We show in our
experiments that the best strategies are often dataset and budget-dependent,
and therefore the trivial approaches do not always produce the best results.
Our method however is capable of adapting to different image domains and finds
combinations of annotations that reach high-performance levels. We show our
method is robust to a number of hyperparameters and that it offers a good
option for allocating annotation strategies.

## References

  * [1] Jiwoon Ahn, Sunghyun Cho, and Suha Kwak.  Weakly supervised learning of instance segmentation with inter-pixel relations.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2209–2218, 2019.
  * [2] Jiwoon Ahn and Suha Kwak.  Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4981–4990, 2018.
  * [3] Mykhaylo Andriluka, Jasper R R Uijlings, and Vittorio Ferrari.  Fluid annotation: a human-machine collaboration interface for full image annotation.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1957–1966, 2018.
  * [4] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei.  What’s the point: Semantic segmentation with point supervision.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 549–565, Cham, 2016. Springer International Publishing.
  * [5] Rodrigo Benenson, Stefan Popov, and Vittorio Ferrari.  Large-Scale Interactive Object Segmentation With Human Annotators.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11692–11701, 2019.
  * [6] Sebastian Bodenstedt, Max Allan, Anthony Agustinos, Xiaofei Du, Luis C. García-Peraza-Herrera, Hannes Kenngott, Thomas Kurmann, Beat P. Müller-Stich, Sébastien Ourselin, Daniil Pakhomov, Raphael Sznitman, Marvin Teichmann, Martin Thoma, Tom Vercauteren, Sandrine Voros, Martin Wagner, Pamela Wochner, Lena Maier-Hein, Danail Stoyanov, and Stefanie Speidel.  Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery.  ArXiv, abs/1805.02475, 2018.
  * [7] Lile Cai, Xun Xu, Jun Hao Liew, and Chuan Sheng Foo.  Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10983–10992, 2021.
  * [8] Arantxa Casanova, Pedro O. Pinheiro, Negar Rostamzadeh, and Christopher J. Pal.  Reinforced active learning for image segmentation.  In International Conference on Learning Representations, 2020.
  * [9] Chen Chen, Chen Qin, Huaqi Qiu, Giacomo Tarroni, Jinming Duan, Wenjia Bai, and Daniel Rueckert.  Deep learning for cardiac image segmentation: a review.  Frontiers in Cardiovascular Medicine, page 25, 2020.
  * [10] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587, 2017.
  * [11] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [13] Yiming Ding, Jae Ho Sohn, Michael G Kawczynski, Hari Trivedi, Roy Harnish, Nathaniel W Jenkins, Dmytro Lituiev, Timothy P Copeland, Mariam S Aboian, Carina Mari Aparici, Spencer C Behr, Robert R Flavell, Shih-Ying Huang, Kelly A Zalocusky, Lorenzo Nardo, Youngho Seo, Randall A Hawkins, Miguel Hernandez Pampaloni, Dexter Hadley, and Benjamin L Franc.  A Deep Learning Model to Predict a Diagnosis of Alzheimer Disease by Using 18F-FDG PET of the Brain.  Radiology, 290(2):456–464, 2019.
  * [14] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun.  Dermatologist-level classification of skin cancer with deep neural networks.  Nature, 542(7639):115–118, 2017.
  * [15] M Everingham, L Van~Gool, C K I Williams, J Winn, and A Zisserman.  The PASCAL Visual Object Classes Challenge 2012 VOC2012 Results.  http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  * [16] Junsong Fan, Zhaoxiang Zhang, Chunfeng Song, and Tieniu Tan.  Learning integral objects with intra-class discriminator for weakly-supervised semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4283–4292, 2020.
  * [17] Michal Heker and Hayit Greenspan.  Joint liver lesion segmentation and classification via transfer learning.  arXiv preprint arXiv:2004.12352, 2020.
  * [18] Mohammad Hesam Hesamian, Wenjing Jia, Xiangjian He, and Paul Kennedy.  Deep learning techniques for medical image segmentation: achievements and challenges.  Journal of digital imaging, 32(4):582–596, 2019.
  * [19] Md Jahidul Islam, Chelsey Edge, Yuyang Xiao, Peigen Luo, Muntaqim Mehtaz, Christopher Morse, Sadman Sakib Enan, and Junaed Sattar.  Semantic segmentation of underwater imagery: Dataset and benchmark.  In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1769–1776. IEEE, 2020.
  * [20] Peng-Tao Jiang, Qibin Hou, Yang Cao, Ming-Ming Cheng, Yunchao Wei, and Hong-Kai Xiong.  Integral object mining via online attention accumulation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2070–2079, 2019.
  * [21] Donald R Jones, Matthias Schonlau, and William J Welch.  Efficient global optimization of expensive black-box functions.  Journal of Global optimization, 13(4):455–492, 1998.
  * [22] Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos.  Multi-class active learning for image classification.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2372–2379, 2009.
  * [23] Minsong Ki, Youngjung Uh, Wonyoung Lee, and Hyeran Byun.  Contrastive and consistent feature learning for weakly supervised object localization and semantic segmentation.  Neurocomputing, 445:244–254, jul 2021.
  * [24] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In European conference on computer vision, pages 491–507. Springer, 2020.
  * [25] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  2021\.
  * [26] Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua.  Introducing geometry in active learning for image segmentation.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2974–2982, 2015.
  * [27] Ksenia Konyushkova, Jasper Uijlings, Christoph H Lampert, and Vittorio Ferrari.  Learning Intelligent Dialogs for Bounding Box Annotation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), jun 2018.
  * [28] Jungbeom Lee, Eunji Kim, Sungmin Lee, Jangho Lee, and Sungroh Yoon.  Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5267–5276, 2019.
  * [29] Lin Li, Eric Rigall, Junyu Dong, and Geng Chen.  MAS3K: An Open Dataset for Marine Animal Segmentation.  In International Symposium on Benchmarking, Measuring and Optimization, pages 194–212. Springer, 2020.
  * [30] Gaobo Liang and Lixin Zheng.  A transfer learning method with deep residual network for pediatric pneumonia diagnosis.  Computer Methods and Programs in Biomedicine, 187:104964, apr 2020\.
  * [31] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun.  Scribblesup: Scribble-supervised convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3159–3167, 2016.
  * [32] Shenlan Liu, Xiang Liu, Gao Huang, Lin Feng, Lianyu Hu, Dong Jiang, Aibin Zhang, Yang Liu, and Hong Qiao.  FSD-10: a dataset for competitive sports content analysis.  arXiv preprint arXiv:2002.03312, 2020.
  * [33] Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M Alvarez, Zhiding Yu, Sanja Fidler, and Marc T Law.  How much more data do i need? estimating requirements for downstream tasks.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 275–284, 2022.
  * [34] Rafid Mahmood, James Lucas, Jose M. Alvarez, Sanja Fidler, and Marc T. Law.  Optimizing data collection for machine learning.  Advances in Neural Information Processing Systems (NeurIPS), 10 2022\.
  * [35] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [36] Afonso Menegola, Michel Fornaciali, Ramon Pires, Flávia Vasques Bittencourt, Sandra Avila, and Eduardo Valle.  Knowledge transfer for melanoma screening with deep learning.  In 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), pages 297–300. IEEE, 2017.
  * [37] Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari.  Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types.  arXiv preprint arXiv:2103.13318, 2021.
  * [38] Dim P Papadopoulos, Alasdair D F Clarke, Frank Keller, and Vittorio Ferrari.  Training Object Class Detectors from Eye Tracking Data.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 361–376, Cham, 2014. Springer International Publishing.
  * [39] Dim P Papadopoulos, Jasper R R Uijlings, Frank Keller, and Vittorio Ferrari.  Extreme clicking for efficient object annotation.  In Proceedings of the IEEE international conference on computer vision, pages 4930–4939, 2017.
  * [40] George Papandreou, Liang-Chieh Chen, Kevin P. Murphy, and Alan L. Yuille.  Weakly- and semi-supervised learning of a deep convolutional network for semantic image segmentation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [41] Hiranmayi Ranganathan, Hemanth Venkateswara, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep active learning for image classification.  In 2017 IEEE International Conference on Image Processing (ICIP), pages 3934–3938, 2017.
  * [42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [43] Mennatullah Siam, Sara Elkerdawy, Martin Jagersand, and Senthil Yogamani.  Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges.  In 2017 IEEE 20th international conference on intelligent transportation systems (ITSC), pages 1–8. IEEE, 2017.
  * [44] Yawar Siddiqui, Julien Valentin, and Matthias Nießner.  Viewal: Active learning with viewpoint entropy for semantic segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9433–9443, 2020.
  * [45] Liat Sless, Bat El Shlomo, Gilad Cohen, and Shaul Oron.  Road scene understanding by occupancy grid learning from sparse radar clusters using semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, page 0, 2019.
  * [46] Chunfeng Song, Yan Huang, Wanli Ouyang, and Liang Wang.  Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3136–3145, 2019.
  * [47] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017.
  * [48] Nima Tajbakhsh, Jae Y. Shin, Suryakanth R. Gurudu, R. Todd Hurst, Christopher B. Kendall, Michael B. Gotway, and Jianming Liang.  Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?  IEEE Transactions on Medical Imaging, 35(5):1299–1312, may 2016\.
  * [49] Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, and Christopher Schroers.  Normalized cut loss for weakly-supervised cnn segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1818–1827, 2018.
  * [50] Chien-Hao Tseng, Chia-Chien Hsieh, Dah-Jing Jwo, Jyh-Horng Wu, Ruey-Kai Sheu, and Lun-Chi Chen.  Person Retrieval in Video Surveillance Using Deep Learning–Based Instance Segmentation.  Journal of Sensors, 2021, 2021.
  * [51] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald Summers.  ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 3462–3471, 2017.
  * [52] P Welinder, S Branson, T Mita, C Wah, F Schroff, S Belongie, and P Perona.  Caltech-UCSD Birds 200.  Technical Report CNS-TR-2010-001, California Institute of Technology, 2010\.
  * [53] Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling Task Transfer Learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3712–3722, jun 2018.
  * [54] Bingfeng Zhang, Jimin Xiao, Jianbo Jiao, Yunchao Wei, and Yao Zhao.  Affinity Attention Graph Neural Network for Weakly Supervised Semantic Segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2021.
  * [55] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.

##  Appendix A Implementation details

Weakly-supervised segmentation model hyperparameters:  We used the U-Net
segmentation model [42] for OCT, and the DeepLabV3 model [10] with a ResNet50
backbone on the SUIM, PASCAL, and Cityscapes datasets. For the U-Net, a
classification head with max-avg pooling and one fully connected layer was
appended at the end of the encoding module for the classification task. For
the DeepLab-like models, we trained the entire ResNet-50 backbone on the
classification task and then added the ASPP head for segmentation. In all
cases, we used the cross-entropy loss for classification and the average of
the Dice and cross-entropy losses for segmentation. Tab. 1 contains the
details of batch sizes and optimizers.

Dataset | Model | Optimizer | Batch size
---|---|---|---
OCT | U-Net | Adam | 8
VOC | DeepLabV3 | SGD | 16
SUIM | DeepLabV3 | SGD | 8
Cityscapes | DeepLabV3 | SGD | 16
Table 1: Hyperparameters and conditions for all experiments.

Algorithm hyperparameters:  We measured the costs of annotations in terms of
class-label equivalents setting $\alpha_{c}=1$ and leaving only $\alpha_{s}$
as a hyperparameter of our method. We set to $\alpha_{s}=12$ for all datasets
following previous studies on crowdsourced annotations [4]. We fixed the
number of iterative steps to $T=8$ and the learning rate of the GP to $0.1$.
We set both the initial number of class annotations $C_{0}$ and segmentation
annotations $S_{0}$ to $8\%$ of the available labels for SUIM and Cityscapes.
We reduced $C_{0}$ in OCT and $S_{0}$ in VOC to account for the higher number
of labels available in those datasets, as detailed in Tab. 2.

Dataset | $C_{0}(\%)$ | $S_{0}(\%)$ | $B_{0}$
---|---|---|---
OCT | 4 | 8 | 1’774
VOC | 8 | 6 | 8’076
SUIM | 8 | 8 | 1’586
Cityscapes | 8 | 8 | 1’774
Table 2: Initial conditions for our method. $B_{0}$ calculated with
$\alpha_{s}=12$ and $\alpha_{c}=1$.

##  Appendix B Sensitivity to $\alpha_{s}$

Fig. 7 shows aditional experiments on the sensitivity of our method to
$\alpha_{s}$ in the considered datasets.

Figure 7: Performance our method with $\alpha_{s}=\\{5,12,25,50\\}$ on four
datasets. One-sigma error bars were computed from three seeds. Blue marks show
the performance of fixed strategies, with labels indicating the percentage of
the budget allocated to segmentation annotations.

##  Appendix C Average performance

We report in Table 3 the average relative performance for each dataset and
segmentation split. Our method is best on average and for two datasets (SUIM
and VOC), while it performs on par with the best baseline for Cityscapes and
OCT.

Seg. Split | 95 | 90 | 85 | 80 | 75 | 70 | 65 | 60 | 55 | 50 | Ours
---|---|---|---|---|---|---|---|---|---|---|---
Cityscapes | 73.9 | 73.2 | 73.0 | 73.1 | 73.3 | 73.5 | 73.5 | 72.5 | 72.3 | 71.4 | 73.4
OCT | 95.4 | 94.6 | 93.8 | 93.0 | 92.1 | 91.3 | 90.5 | 89.7 | 88.8 | 87.7 | 92.6
SUIM | 72.1 | 74.1 | 76.7 | 79.2 | 80.9 | 81.3 | 79.7 | 78.1 | 76.7 | 75.1 | 81.9
VOC | 44.3 | 44.0 | 44.3 | 44.9 | 45.6 | 46.0 | 43.1 | 43.3 | 40.9 | 40.8 | 46.0
Average | 71.4 | 71.5 | 71.9 | 72.5 | 73.0 | 73.0 | 71.7 | 70.9 | 69.7 | 68.8 | 73.5
Table 3: Relative performance (in %) against full supervision for each dataset
and segmentation split.

##  Appendix D Surfaces

To explain the drop in performance for SUIM after a budget of 3’500, we
computed the true surfaces for all datasets (Fig. 8). We observed that
segmentation performance grows logarithmically for OCT, VOC, and Cityscapes,
but not for SUIM after a certain number of class annotations. Since our GP
assumes a logarithmic relation between dataset size and performance, this
observation is particularly relevant to explain the decline in performance for
SUIM. Most notably, and confirming our hypothesis, this decline is not seen
with other values of $\alpha_{s}$ for this same dataset (Fig. 7). Due to the
limited dataset size, larger values of $\alpha_{s}$ constrain the area of the
surface that can be reached in our experiments. In the case of SUIM,
$\alpha_{s}=50$ translates to exploring zones where performance grows
logarithmically with dataset size (low data regime).

Figure 8: Segmentation performance grows logarithmically with training set
size on Cityscapes, OCT, and VOC. This trend is not observed in the SUIM
dataset.

##  Appendix E Ground-truth approximation

To address the high computational demands of our experiments, we followed the
procedure of [34] for ground-truth approximation. In particular, we built
subsets of the training dataset by randomly sampling different proportions of
the available annotated samples [33]. The proportions $\rho_{s}$ and
$\rho_{c}$ for segmentation and classification samples, respectively, were
chosen from the set
$\\{2\%,4\%,6\%,8\%,10\%,20\%,30\%,40\%,60\%,80\%,100\%\\}$, for a total of
$11\times{}11=121$ possible training subsets. For each subset, we trained the
weakly-supervised segmentation model and measured its Dice score on a fixed
segmentation test set. We finally interpolated these scores with third-order
splines to obtain a surface of ground-truth Dice scores. This procedure
allowed efficient estimations of the Dice Score values without retraining a
new model for each strategy.

The proportions $\rho_{s}$ and $\rho_{c}$ are relative to the total amounts of
available annotated samples in each dataset, which are shown in Tab. 4.

Dataset | Segmentation | Classification
---|---|---
OCT | 902 | 22’723
VOC | 10’582 | 5’717
SUIM | 1’525 | 1’525
Cityscapes | 2’975 | 2’975
Table 4: Number of training images for each dataset and modality.