Observation of top-quark pair production in proton-lead collisions in ATLAS

Petr Baroň111Contact<EMAIL_ADDRESS>
“Copyright 2023 CERN for the benefit of the ATLAS Collaboration. Reproduction
of this article or parts of it is allowed as specified in the CC-BY-4.0
license.”on behalf of the ATLAS Collaboration

Joint Laboratory of Optics of Palacký University Olomouc and Institute of
Physics of Czech Academy of Sciences, Czech Republic

> Top-quarks and Higgs boson are the only elementary particles that have not
> been observed in heavy-ion collisions in the ATLAS detector yet. In
> particular top quarks, the heaviest elementary particles carrying colour
> charges, have been argued to be attractive candidates for probing the quark-
> gluon plasma produced in heavy-ion collisions. In proton-lead collisions,
> top-quark production is expected to be sensitive to nuclear modifications of
> parton distribution functions (PDF) at high Bjoerken-x values which are hard
> to access experimentally using other probes available so far. In 2016 the
> ATLAS experiment collected proton-lead collisions at centre-of-mass energy
> of 8.16 TeV per nucleon pair. The data sample corresponds to an integrated
> luminosity of 164 nb-1, which allows for the first time in this data set
> with ATLAS, to measure top-quark pair production. In this work, we discuss
> the inclusive cross section measurement for the top-quark pairs production
> in dilepton and lepton+jets decay modes with electrons and muons recorded by
> the ATLAS experiment. The measurement is compared to the NNLO predictions
> for top-quark production using various PDF sets.

> PRESENTED AT
>
>
>
>
> $16^{\mathrm{th}}$ International Workshop on Top Quark Physics
> (Top2023), 24–29 September, 2023

##  1 Introduction

This proceedings showcases the process of $t\bar{t}$ production in $p$+Pb
collisions at $\sqrt{s_{\mathrm{NN}}}=8.16~{}\mathrm{TeV}$ measured by the
ATLAS experiment. A comprehensive exposition is available in the recent
conference note [1]. The study focuses on the reconstruction of top-quark
pairs observed in both $\ell+$jets and dilepton channels, employing electrons
and muons in the final states. While the dilepton mode is less abundant, its
purity significantly surpasses that of the $\ell+$jets channels. The results
are compared with NNLO calculations involving the most recent nPDF sets.

The ATLAS detector [2, 3, 4, 5, 6, 7, 8] collected the data for this
measurement during the 2016 $p$+Pb collision running period, amassing an
integrated luminosity of 165 nb-1. Operation of the LHC at energies of 6.5 TeV
for the proton beams and 2.51 TeV per nucleon for the Pb beams led in a
center-of-mass collision energy of 8.16 TeV. This configuration induces a
rapidity boost of $\pm 0.465$ units w.r.t. the ATLAS laboratory frame,
depending upon the direction of the $p$ beam.

The experiment facilitated two beam-direction configurations: $p$+Pb and
Pb+$p$. The latter configuration, with the Pb beam directed in the $+z$
direction, provided approximately twice the integrated luminosity compared to
the former.

##  2 Data and Monte Carlo Simulation Samples

The ATLAS experiment used data collected during the 2016 $p$+Pb running
period, amounting to an integrated luminosity of 165 nb-1. Events were
selected via single-lepton electron or muon triggers with a minimum transverse
momentum $p_{\mathrm{T}}$ threshold of 15 GeV [9, 10]. These events were
required to possess at least one reconstructed vertex built from a minimum of
two good-quality charged tracks with $p_{\mathrm{T}}>0.1~{}\mathrm{GeV}$.
Electron candidates were identified through a cluster of energy deposits in
the EM calorimeter matched to an ID track. Selection criteria included
’Medium’ likelihood-based requirements [11],
$p_{\mathrm{T}}>18~{}\mathrm{GeV}$, and $|\eta|<2.47$. Varying isolation
requirements were imposed based on track and calorimeter-based isolation, with
efficiency thresholds at different $p_{\mathrm{T}}$ values. Muon candidates
were reconstructed by combining inner detector (ID) and muon spectrometer (MS)
tracks, satisfying ’Medium’ requirements [12] with
$p_{\mathrm{T}}>18~{}\mathrm{GeV}$, $|\eta|<2.5$, and isolation criteria using
a cone size of $\Delta R=0.2$ around the muon and a track-based isolation
requirement with a varying cone size [12]. Jets were reconstructed from
calorimeter energy deposits [13, 14], using the anti-$k_{t}$ algorithm [15,
16] with $R=0.4$. The jet kinematics are corrected event-by-event for the
contribution from underlying event (UE) particles, calibration included
calorimeter response simulations [17], and in situ absolute energy scale
measurements. In situ measurements are carried out in $pp$ collisions and
cross-calibrated to the $p+Pb$ system typical by its low pile-up
($<\mu>=0.18$) enviroment. Jets reconstructed this way are referred to as
Heavy Ion (HI) jets. HI jets and Particle Flow (PF) jets, were matched using a
minimal $\Delta R$ criterion. HI jets lacking PF counterparts were considered
non-$b$-tagged. The analysis relied on HI jets for kinematic calculations, but
inherited $b$-tagging information from matched PF jets.

Monte Carlo (MC) simulated events were utilized for analysis development,
signal estimation, and background evaluation. Samples were processed using the
ATLAS detector simulation based on Geant4 [18]. Top-quark samples were
simulated using various event generators [19, 20, 21], normalized using
NNLO+NNLL predictions [22]. Backgrounds from $W/Z$+jets, single top-quark, and
diboson production were simulated using Sherpa and Powheg event generators
with Pythia8 for particle shower. The fake-lepton and non-prompt background
was estimated using the Matrix Method [23]. Events were categorized into
dilepton and $\ell+$jets channels based on lepton count and $b$-tagged jet
requirements, with specific event yields and signal purities observed.

The dilepton channel comprises events featuring precisely two opposite-sign
leptons. Pairs of same-flavor leptons ($e^{+}e^{-}$ or $\mu^{+}\mu^{-}$)
within an invariant mass range of $80<m_{\ell\ell}<100~{}\mathrm{GeV}$ are
discarded. Additionally, the invariant mass in the $e\mu$ ($ee$ and $\mu\mu$)
channel must exceed 15 (45) GeV, a constraint aimed at aligning with the phase
space of the $Z$+jets simulation samples without significantly impacting the
results. Events with a minimum of two HI jets with at least one being
$b$-tagged form the Signal Region (SR). In total 106 and 104 data events were
observed in the $1b$ and $\geq 2b$ regions, respectively, exhibiting expected
signal purities of 53% and 91% in data. The lepton+jets ($\ell+$jets) channel
is constructed from events featuring precisely one lepton and at least four HI
jets, including at least one $b$-tagged jet, in the SR. This channel comprises
1874 and 1075 data events in the $1b$ and $\geq 2b$ regions, respectively,
exhibiting anticipated signal purities of 20% and 71%. Further division of the
$\ell+$jets SR, based on lepton flavor, produces in total four signal regions:
$1\ell 1b$ ejets, $1\ell 2b\mathrm{incl}$ ejets, $1\ell 1b$ mujets, and
$4j1\ell 2b\mathrm{incl}$ mujets.

##  3 Systematic uncertainties

The measurement is subject to systematic uncertainties stemming from various
sources:

  * •

Lepton reconstruction uncertainties encompass muon momentum scale and
resolution based on existing references [12]. For electrons, reconstruction,
identification, isolation, and low-pileup energy calibration uncertainties are
assessed [24].

  * •

Jet-related uncertainties rely on studies of calorimeter response and
comparisons between different simulation samples [25, 26]. These uncertainties
also cover $b$-tagging and jet matching variations.

  * •

To account for uncertainties in $V$+jets processes, single-top-quark
processes, and the diboson background, the Berends scaling technique, diagram
removal/subtraction variations, and normalization uncertainties are utilized
[27, 28, 29].

  * •

Uncertainties in the non-prompt and fake-lepton background involve statistical
and systematic variations in efficiencies. Shape variations in the $\ell+$jets
channel and normalization uncertainties are imposed based on conservative
estimations in the $0b$ control region.

  * •

Effects from the QCD initial-state and final-state radiation, evaluated via
variations in $\alpha_{\mathrm{s}}$, renormalization/factorization scales,
$h_{\mathrm{damp}}$ parameter in Powheg, and PDF uncertainties, are considered
[21, 30].

  * •

The integrated luminosity uncertainty is estimated at 2.4%. This evaluation is
based on luminosity scale calibration using $x$-$y$ beam-separation scans and
measurements from the LUCID-2 detector [31, 32].

##  4 Analysis

The signal strength parameter, denoted as $\mu_{t\bar{t}}$, represents the
observed signal yield in both the $\ell+$jets and dilepton final states,
relative to the Standard Model (SM) expectation without nuclear parton
distribution function (nPDF) effects. Its determination employs a profile-
likelihood fit [33, 34] to the $H_{\mathrm{T}}^{\ell\mathrm{jet}}$ data
distributions across six Signal Regions, where
$H_{\mathrm{T}}^{\ell\mathrm{jet}}$ denotes the scalar sum of transverse
momenta of leptons and HI jets. Figure in 1 depict the post-fit
$H_{\mathrm{T}}^{\ell\mathrm{jet}}$ distributions for six channels within the
dilepton and $\ell+$jets SRs, distinguishing between exactly one and at least
two $b$-tagged jets. Observed distributions are in good agreement with those
predicted by the fit.

Figure 1: Data post-fit plots representing the $H_{\mathrm{T}}^{\ell,j}$
variable in the six SRs ($e+$jets: $1\ell 1b$ and $1\ell 2b\mathrm{incl}$,
$\mu+$jets: $1\ell 1b$ and $1\ell 2b\mathrm{incl}$, dilepton: $2\ell 1b$ and
$2\ell 2b\mathrm{incl}$.), with uncertainties represented by the hatched area.
The full markers in the bottom panels show a ratio between data and a sum of
predictions. Open triangles indicate bins with entries which are outside the
ratio range.

The fit incorporates roughly 130 nuisance parameters that account for
systematic uncertainties. These uncertainties are collectively treated as
correlated across the Signal Regions. Primary contributors to the pre-fit
uncertainties encompass fake-lepton and $W$+jets normalization in the
$\ell+$jets channel, along with $Z$+jets normalization in the dilepton
channel. The fit procedure mitigates the impact of fake-lepton uncertainties.
The major factors contributing to the overall systematic uncertainty include
signal modeling, jet energy scale, luminosity of the data sample, and fake-
lepton background. A summary outlining grouped systematic uncertainties is
presented in Table 1. Overall, the total relative systematic uncertainty is
estimated at 8%.

Table 1: Breakdown of relative systematic uncertainties on the cross section in data. Owing to rounding effects and small correlations between the different sources of uncertainties, the total systematic uncertainty is different from the sum in quadrature of the individual sources. Source | unc. up | unc. down
---|---|---
Jet energy scale | +0.048 | -0.044
$t\bar{t}$ generator | +0.048 | -0.043
Fake-lepton background | +0.030 | -0.027
Background | +0.030 | -0.025
Luminosity | +0.029 | -0.025
Muon systs. | +0.024 | -0.021
$W+$jets | +0.023 | -0.020
$b$-tagging | +0.022 | -0.021
Electron systs. | +0.018 | -0.017
MC statistical uncertainties | +0.011 | -0.010
Jet energy resolution | +0.005 | -0.004
$t\bar{t}$ PDF | +0.001 | -0.001
Total syst. | +0.088 | -0.081

##  5 Results

The measured value of $\mu_{t\bar{t}}$ is used to derive the inclusive
$t\bar{t}$ production cross section using the formula

$\sigma_{t\bar{t}}=\mu_{t\bar{t}}\cdot
A_{\mathrm{Pb}}\cdot\sigma^{\mathrm{th}}_{t\bar{t}}.$ (1)

Here, $\sigma^{\mathrm{th}}_{t\bar{t}}$ represents the predicted $t\bar{t}$
production cross section in nucleon-nucleon collisions, determined at NNLO
precision, used as normalization for the signal $t\bar{t}$ samples in both
$\ell+$jets and dilepton decay modes. The measured inclusive $t\bar{t}$ cross
section for $p$+Pb collisions is $\sigma_{t\bar{t}}=57.9\pm
2.0~{}\mathrm{(stat.)\;^{+4.9}_{-4.5}~{}\mathrm{(syst.)}}~{}\mathrm{nb}=57.9\;^{+5.3}_{-4.9}~{}\mathrm{(tot.)}~{}\mathrm{nb}$.
The total relative uncertainty stands at 9%, mainly driven by systematic
contributions.

The background-only hypothesis is dismissed with a significance exceeding 5
standard deviations, confirming the observation of the $t\bar{t}$ process in
$p$+Pb collisions by ATLAS. Figure 2 showcases the signal strength
$\mu_{t\bar{t}}$ obtained separately in each region and through the combined
fit. The two $\ell+$jets channels with the highest signal yield (in the $\geq
2b$ region) produce very close results. Slight variations are observed in the
signal strength between the dilepton channels, with a slightly higher
$\mu_{t\bar{t}}$ preferred in the dilepton channels compared to the $\mu+$jets
$1b$ channel, but consistent within uncertainties. The precision of the
$\mu_{t\bar{t}}$ value is limited by systematic uncertainties in the
$\ell+$jets SRs, while statistical uncertainties dominate in the dilepton SRs.
Additionally, separate fits of $\mu_{t\bar{t}}$ to the combined four
$\ell+$jets and two dilepton SRs both exceed $5\sigma$, confirming the
observation of $t\bar{t}$ production also in the individual $\ell+$jets and
dilepton channels. This marks the first report of the latter in $p$+Pb
collisions at the LHC.

Figure 2: Signal strengths with total and statistical uncertainties in
different fit regions and combined.

The combined measurement of the $t\bar{t}$ production cross section
$\sigma_{t\bar{t}}$ showed in Figure 3 is compared with the corresponding
result obtained by CMS [35], both acquired using $p$+Pb collisions at
$\sqrt{s_{\mathrm{NN}}}=8.16~{}\mathrm{TeV}$. These two measurements exhibit
agreement within the range of uncertainties. Additionally, the most precise
$t\bar{t}$ production cross section measured in $pp$ collisions at
$\sqrt{s}=8~{}\mathrm{TeV}$, derived from a combination of ATLAS and CMS data
[36], is presented. This cross-section value is extrapolated to the center-of-
mass energy of this particular measurement using predictions from Top++ v2 and
scaled by $A_{\mathrm{Pb}}$ to the $p$+Pb system. The extrapolated cross
section bears a 2.5% relative uncertainty and remains independent of nPDF
dependencies. The measured cross section is compared with NLO calculations
obtained using the MCFM generator [37], scaled to NNLO precision in QCD using
a k-factor (k=1.139) derived from the Top++ generator. The MCFM calculations
utilize four distinct nPDF sets: EPPS21 [38], nCTEQ15HQ [39], nNNPDF30 [40],
and TUJU21 [41]. The nNNPDF30 nPDF set exhibits the largest discrepancy when
compared with the measured cross-section value. In contrast, the remaining
nPDF sets showcase good alignment with the measured cross-section value.

Figure 3: Comparison between observed and predicted values of
$\sigma_{t\bar{t}}$ as well as with the CMS Collaboration measurement of
$\sigma_{t\bar{t}}$ in $p$+Pb collisions at
$\sqrt{s_{\mathrm{NN}}}=8.16~{}\mathrm{TeV}$ [35], and the combined
measurement of $t\bar{t}$ production cross section in $pp$ collisions at
$\sqrt{s}=8~{}\mathrm{TeV}$ from ATLAS and CMS collaborations [36]. The latter
is extrapolated to the center-of-mass energy of this measurement and also
using the $A_{\mathrm{Pb}}$ factor. Predictions are calculated at NNLO
precision using the MCFM code [37] scaled to the $p$+Pb system and given for
different nPDF sets. The uncertainty on predictions represents the uncertainty
on internal PDF. The solid black line indicates the measured value. The
combined statistical and systematic uncertainty of the measurement is shown in
green while the statistical component is depicted in yellow.

##  6 Conclusion

This proceeding presented measurement of top-quark pair production in $p$+Pb
collisions at a center-of-mass energy of
$\sqrt{s_{\mathrm{NN}}}=8.16~{}\mathrm{TeV}$ per nucleon pair, conducted with
the ATLAS experiment. The observation of top-quark pairs occurs through the
distinct $\ell+$jets and dilepton channels, with final states containing
electrons and muons. The observation of top-quark pair production in the
dilepton channel surpasses 8 standard deviations, marking its first
observation in the $p$+Pb system at the LHC. The total integrated cross
section is measured with a total relative uncertainty of 9%, establishing this
as the most precise determination of the $t\bar{t}$ cross section in nuclear
collisions to date. While the precision in the individual $t\bar{t}$ decay
channels is constrained by systematic uncertainties in the $\ell+$jets
channel, the dilepton region is predominantly limited by statistical
uncertainties.

The measured cross section is compared with the CMS measurement in the $p$+Pb
system and the combined measurement from ATLAS and CMS in $pp$ collisions at
$\sqrt{s}=8~{}\mathrm{TeV}$. The latter is appropriately scaled to the $p$+Pb
system and the energy $\sqrt{s}=8.16~{}\mathrm{TeV}$ of this analysis.
Additionally, the measured cross section is compared against NNLO calculations
derived from various nPDF sets. The measured cross section aligns closely with
previous measurements and Standard Model predictions. With its enhanced
precision, this measurement promises to offer constraints on nPDFs,
particularly in the high-$x$ region.

## Acknowledgement

The author gratefully acknowledge the support from the project
IGA_PrF_2023_005 of Palacky University.

## References

  * [1] ATLAS Collaboration Observation of $t\bar{t}$ production in lepton+jets and dilepton channels in p+Pb collisions at $\sqrt{s}_{\mathrm{NN}}=8.16$ TeV with the ATLAS detector. (CERN,2023), http://cds.cern.ch/record/2873517
  * [2] ATLAS Collaboration The ATLAS Experiment at the CERN Large Hadron Collider. JINST. 3 pp. S08003 (2008)
  * [3] ATLAS Collaboration ATLAS Insertable B-Layer Technical Design Report. (2010), https://cds.cern.ch/record/1291633
  * [4] Abbott, B. & Others Production and Integration of the ATLAS Insertable B-Layer. JINST. 13, T05008 (2018)
  * [5] ATLAS Collaboration Performance of the ATLAS Trigger System in 2015. Eur. Phys. J. C. 77, 317 (2017)
  * [6] ATLAS Collaboration Operation of the ATLAS trigger system in Run 2. JINST. 15, P10004 (2020)
  * [7] ATLAS Collaboration The ATLAS Collaboration Software and Firmware. (CERN,2021), https://cds.cern.ch/record/2767187
  * [8] ATLAS Collaboration ATLAS data quality operations and performance for 2015–2018 data-taking. JINST. 15, P04003 (2020)
  * [9] ATLAS Collaboration Performance of electron and photon triggers in ATLAS during LHC Run 2. Eur. Phys. J. C. 80, 47 (2020)
  * [10] ATLAS Collaboration Performance of the ATLAS muon triggers in Run 2. JINST. 15, P09015 (2020)
  * [11] ATLAS Collaboration Electron and photon performance measurements with the ATLAS detector using the 2015–2017 LHC proton-proton collision data. JINST. 14, P12006 (2019)
  * [12] ATLAS Collaboration Muon reconstruction and identification efficiency in ATLAS using the full Run 2 pp collision data set at $\sqrt{s}$=13 TeV. Eur. Phys. J. C. 81, 578 (2021)
  * [13] ATLAS Collaboration Dijet azimuthal correlations and conditional yields in pp and p+Pb collisions at sNN=5.02TeV with the ATLAS detector. Phys. Rev. C. 100, 034903 (2019)
  * [14] ATLAS Collaboration Charged-hadron production in $pp$, $p$+Pb, Pb+Pb, and Xe+Xe collisions at $\sqrt{s_{{}_{\mathrm{NN}}}}=5$ TeV with the ATLAS detector at the LHC, JHEP 07 (2023), 074 doi:10.1007/JHEP07(2023)074 [arXiv:2211.15257 [hep-ex]].
  * [15] Cacciari, M., Salam, G. & Soyez, G. The anti-$\mathrm{k}_{\mathrm{t}}$ jet clustering algorithm. JHEP. 4 pp. 063 (2008)
  * [16] Cacciari, M., Salam, G. & Soyez, G. FastJet User Manual. Eur. Phys. J. C. 72 pp. 1896 (2012)
  * [17] ATLAS Collaboration Jet energy scale and resolution measured in proton–proton collisions at $\sqrt{s}$=13 TeV with the ATLAS detector. Eur. Phys. J. C. 81, 689 (2021)
  * [18] Agostinelli, S. & Others GEANT4–a simulation toolkit. Nucl. Instrum. Meth. A. 506 pp. 250-303 (2003)
  * [19] Alioli, S., Nason, P., Oleari, C. & Re, E. A general framework for implementing NLO calculations in shower Monte Carlo programs: the POWHEG BOX. JHEP. 6 pp. 043 (2010)
  * [20] Sjöstrand, T., Mrenna, S. & Skands, P. A brief introduction to PYTHIA 8.1. Computer Physics Communications. 178, 852-867 (2008), https://www.sciencedirect.com/science/article/pii/S0010465508000441
  * [21] ATLAS Collaboration ATLAS Pythia 8 tunes to 7 TeV data. (CERN,2014), https://cds.cern.ch/record/1966419
  * [22] Czakon, M. & Mitov, A. Top++: A Program for the Calculation of the Top-Pair Cross-Section at Hadron Colliders. Comput. Phys. Commun.. 185 pp. 2930 (2014)
  * [23] G. Aad et al. ATLAS Collaboration Tools for estimating fake/non-prompt lepton backgrounds with the ATLAS detector at the LHC, JINST 18 (2023) no.11, T11004, doi:10.1088/1748-0221/18/11/T11004 [arXiv:2211.16178 [hep-ex]].
  * [24] ATLAS Collaboration Electron reconstruction and identification in the ATLAS experiment using the 2015 and 2016 LHC proton-proton collision data at $\sqrt{s}$ = 13 TeV. Eur. Phys. J. C. 79, 639 (2019)
  * [25] ATLAS Collaboration Jet energy scale measurements and their systematic uncertainties in proton-proton collisions at $\sqrt{s}$ = 13 TeV with the ATLAS detector. Phys. Rev. D. 96, 072002 (2017)
  * [26] ATLAS Collaboration Jet energy scale and its uncertainty for jets reconstructed using the ATLAS heavy ion jet algorithm. (CERN,2015), https://cds.cern.ch/record/2008677
  * [27] Berger, C., Bern, Z., Dixon, L., Febres Cordero, F., Forde, D., Gleisberg, T., Ita, H., Kosower, D. & Maıt̂re, D. Precise Predictions for W+4-Jet Production at the Large Hadron Collider. Phys. Rev. Lett.. 106, 092001 (2011,3), https://link.aps.org/doi/10.1103/PhysRevLett.106.092001
  * [28] Re, E. Single-top Wt-channel production matched with parton showers using the POWHEG method. Eur. Phys. J. C. 71 pp. 1547 (2011)
  * [29] ATLAS Collaboration Measurement of the $t\bar{t}$ production cross-section in pp collisions at $\sqrt{s}$ = 5.02 TeV with the ATLAS detector. JHEP. 6 pp. 138 (2023)
  * [30] Butterworth, J. & Others PDF4LHC recommendations for LHC Run II. J. Phys. G. 43 pp. 023001 (2016)
  * [31] ATLAS Collaboration Luminosity determination in pp collisions at $\sqrt{s}$ = 13 TeV using the ATLAS detector at the LHC. (2022,12)
  * [32] Avoni, G., Bruschi, M., Cabras, G., Caforio, D., Dehghanian, N., Floderus, A., Giacobbe, B., Giannuzzi, F., Giorgi, F., Grafström, P., Hedberg, V., Manghi, F., Meneghini, S., Pinfold, J., Richards, E., Sbarra, C., Cesari, N., Sbrizzi, A., Soluk, R., Ucchielli, G., Valentinetti, S., Viazlo, O., Villa, M., Vittori, C., Vuillermet, R. & Zoccoli, A. The new LUCID-2 detector for luminosity measurement and monitoring in ATLAS. (IOP Publishing,2018,7), https://doi.org/10.1088/1748-0221/13/07/p07017
  * [33] Cowan, G., Cranmer, K., Gross, E. & Vitells, O. Asymptotic formulae for likelihood-based tests of new physics. Eur. Phys. J. C. 71 pp. 1554 (2011), [Erratum: Eur.Phys.J.C 73, 2501 (2013)]
  * [34] Cranmer, K., Lewis, G., Moneta, L., Shibata, A. & Verkerke, W. HistFactory: A tool for creating statistical models for use with RooFit and RooStats. (New York U.,2012), https://cds.cern.ch/record/1456844
  * [35] CMS Collaboration Observation of top quark production in proton-nucleus collisions. Phys. Rev. Lett.. 119, 242001 (2017)
  * [36] ATLAS, CMS Collaborations Combination of inclusive top-quark pair production cross-section measurements using ATLAS and CMS data at $\sqrt{s}$ = 7 and 8 TeV. (2022,5)
  * [37] Campbell, J., Wackeroth, D. & Zhou, J. Study of weak corrections to Drell-Yan, top-quark pair, and dijet production at high energies with MCFM. Phys. Rev. D. 94, 093009 (2016)
  * [38] Eskola, K., Paakkinen, P., Paukkunen, H. & Salgado, C. Towards EPPS21 nuclear PDFs. SciPost Phys. Proc.. 8 pp. 033 (2022)
  * [39] Kusina, A. & Others Impact of LHC vector boson production in heavy ion collisions on strange PDFs. Eur. Phys. J. C. 80, 968 (2020)
  * [40] Ubiali, M. Towards the NNPDF3.0 parton set for the second LHC run. PoS. DIS2014 pp. 041 (2014)
  * [41] Helenius, I., Walt, M. & Vogelsang, W. TUJU21: nuclear PDFs with electroweak-boson data at NNLO. 29th International Workshop On Deep-Inelastic Scattering And Related Subjects. (2022,7)


there exists an $h\in\widetilde{\mathcal{H}}_{4,3}$ so that the fibre over $h$
represents the map $F$ and $\tau_{i}(h)=p$. Therefore the images of the
$T_{i}$’s under the map $\phi$ coincide and are equal to the union of all
ramification points of all the $g^{1}_{3}$’s.

The map $h$ is simply branched at $E_{0}$ and $E_{3}$. The map $\phi$ factors
via the fibre product
$\mathcal{X}=\overline{\mathcal{M}}_{4,1}\times_{\overline{\mathcal{M}}_{4}}\widetilde{\mathcal{H}}_{4,3}$.
Let

$\textstyle{\mathcal{C}_{\widetilde{\mathcal{H}}_{4,3}}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\rho}$$\scriptstyle{t}$$\textstyle{\mathcal{X}\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\widetilde{\pi}}$$\scriptstyle{\widetilde{h}}$$\textstyle{\overline{\mathcal{M}}_{4,1}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\pi}$$\textstyle{\widetilde{\mathcal{H}}_{4,3}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{h}$$\textstyle{\overline{\mathcal{M}}_{4}}$

the induced diagram, with $\phi=\widetilde{h}\rho$. We have
$\omega_{\widetilde{\pi}}=\widetilde{h}^{*}\omega_{\pi}$. The map $h$ is
ramified at the points $x\in E_{0}$ and therefore the fibre of
$\widetilde{\pi}$ over $x$ has an $A_{1}$ singularity corresponding to the
node over the fibre of $\pi$. The map $\rho$ resolves the singularity and the
corresponding exceptional divisor $\mathcal{C}_{1}$ (the union of the $C_{1}$
from above) consists of $(-2)$-curves. In addition, on the fibres over $E_{0}$
the map $\rho$ contracts an exceptional divisor $\mathcal{R}_{1}$ which
consists of $(-1)$-curves - blow-up along a smooth locus (the union of the
tails $R_{1}$ as above). The map $h$ is also ramified at the points $x\in
E_{3}$ but since the fibre over $x$ maps to a smooth curve, the fibre of
$\widetilde{\pi}$ over $x$ is a smooth curve. The map $\rho$ is the blow-up at
the triple ramification point locus of the fibres of $\widetilde{\pi}$. The
exceptional divisor $\mathcal{S}$ consists of $(-1)$-curves (the curves $S$ as
above). Since the restriction of $\omega_{t}$ on $(-2)$-curves is trivial as
$C_{1}$ gets contracted via $\phi$, we get that
$\omega_{t}=\phi^{*}\omega_{\pi}+B$ with $B=\mathcal{R}_{1}+\mathcal{S}$.

Let $\alpha$ be a divisor class on
$\mathcal{C}_{\widetilde{\mathcal{H}}_{4,3}}$. We then have

$\displaystyle\pi_{*}(\omega_{\pi}\phi_{*}\alpha)$
$\displaystyle=\pi_{*}\phi_{*}(\phi^{*}\omega_{\pi}\,\alpha)$
$\displaystyle=h_{*}t_{*}(\phi^{*}\omega_{\pi}\,\alpha)$
$\displaystyle=h_{*}t_{*}(\omega_{t}\,\alpha-B\alpha)$
$\displaystyle=h_{*}t_{*}(\omega_{t}\,\alpha)-h_{*}t_{*}(B\alpha).$

We have $\omega_{t}=f^{*}\omega_{\pi_{b}}+R_{f}$, with $R_{f}$ the
ramification divisor of the map $f$, that is $R_{f}=\sum_{i=1}^{b}T_{i}$. Then

$\displaystyle h_{*}t_{*}(\omega_{t}\,\alpha)$
$\displaystyle=h_{*}t_{*}(f^{*}\omega_{\pi_{b}}\,\alpha)+h_{*}t_{*}(R_{i}\,\alpha)$
$\displaystyle=h_{*}t_{*}(f^{*}\omega_{\pi_{b}}\,\alpha)+\sum_{i=1}^{b}h_{*}t_{*}(\tau_{*}\tau_{i}^{*}\alpha)$
$\displaystyle=h_{*}t_{*}(f^{*}\omega_{\pi_{b}}\,\alpha)+\sum_{i=1}^{b}h_{*}(\tau_{i}^{*}\alpha).$

Suppose now that $\alpha=R_{f}=\sum_{i=1}^{b}T_{i}$. Note that the degree of
$R$ over $\overline{\mathcal{M}}_{4}$ is 2b so we have

$R={2\over N_{0}}\phi_{*}R_{f}.$

By [vdGK11, Lemma 4.2] we have

$\displaystyle\sum_{i=1}^{b}\tau_{i}^{*}R_{f}$
$\displaystyle=\sum_{i=1}^{b}\tau_{i}^{*}T_{i}$
$\displaystyle=\sum_{i=1}^{b}t_{*}\tau_{i*}\tau_{i}^{*}T_{i}$
$\displaystyle=\sum_{i=1}^{b}t_{*}T_{i}^{2}$
$\displaystyle=t_{*}(R_{f}^{2})=-{1\over 2}j^{*}\psi$

and $t_{*}(f^{*}\omega_{\pi_{b}}\,R_{f})=j^{*}\psi$. Therefore
$h_{*}t_{*}(\omega_{t}\,R_{f})={1\over 2}h_{*}j^{*}\psi$. Also,
$t_{*}(BR_{f})=2E_{3}$. This is because $\mathcal{S}$ has two ramification
points and $\mathcal{R}_{1}$ has none. Therefore

$\pi_{*}(\omega_{\pi}\,\phi_{*}R_{f})={1\over 2}h_{*}j^{*}\psi-2h_{*}E_{3}.$

Following formulas in [vdGK12] we have:

$\displaystyle h_{*}j^{*}\psi$ $\displaystyle=40N_{0}(9\lambda-\delta_{0})$
$\displaystyle 2h_{*}E_{3}$ $\displaystyle=N_{0}(132\lambda-15\delta_{0})$

and so,

$\displaystyle\pi_{*}(\omega_{\pi}\,R)$ $\displaystyle={2\over
N_{0}}\pi_{*}(\omega_{\pi}\,\phi_{*}R_{f})=40(9\lambda-\delta_{0})-2(132\lambda-15\delta_{0})$
$\displaystyle=96\lambda-10\delta_{0}.$

Writing $R=a\omega_{\pi}+b\pi^{*}\lambda+c\pi^{*}\delta_{0}$, we get $a=4$
(because the fibre has 24 points and $24=4\cdot 6$, with $6=2g-2$). This
implies

$\pi_{*}(\omega_{\pi}\,R)=4(12\lambda-\delta_{0})+6b\lambda+6c\delta_{0}=(48+6b)\lambda+(-4+6c)\delta_{0}.$

Hence $48+6b=96$ which implies $b=8$ and $-4+6c=-10$ giving $c=-1$.

Suppose now that $\alpha=A$ is the sum of the residual sections to the
$T_{i}$’s. Then $R^{\prime}={2\over N_{0}}\phi_{*}A$. We have
$h_{*}t_{*}(\omega_{t}\,A)=h_{*}t_{*}(f^{*}\omega_{\pi_{b}}\,A)=h_{*}t_{*}(f^{*}\omega_{\pi_{b}}A)=j^{*}\psi$
and $B=2E_{0}+2E_{3}$. This is because both $\mathcal{R}_{1}$ and
$\mathcal{S}$ contain two points which are residual to the ramification
points. Then

$\pi_{*}(\omega_{\pi}\,\phi_{*}A)=h_{*}j^{*}\psi-2h_{*}E_{0}-2h_{*}E_{3}.$

We have $h_{*}E_{0}={N_{0}\over 2}\delta_{0}$. Therefore
$\pi_{*}(\omega_{\pi}\,R_{1})={2\over
N_{0}}\pi_{*}(\omega_{\pi}\,\phi_{*}A)=80(9\lambda-\delta_{0})-2(132\lambda-15\delta_{0})-2\delta_{0}=456\lambda-52\delta_{0}$.
Writing now $R^{\prime}=4\omega_{\pi}+b\pi^{*}\lambda+c\pi^{*}\delta_{0}$ we
have as above that $48+6b=456$ which implies $b=68$ and $-4+6c=-52$ giving
$c=-8$. ∎

## References

  * [AK77] Allen B. Altman and Steven L. Kleiman.  Foundations of the theory of Fano schemes.  Compositio Math., 34(1):3–47, 1977.
  * [Ame11] Ekaterina Amerik.  A computation of invariants of a rational self-map.  Annales de la faculté des sciences de Toulouse Mathématiques, 18(3):481–493, 2011.
  * [BVdV79] W. Barth and A. Van de Ven.  Fano varieties of lines on hypersurfaces.  Arch. Math. (Basel), 31(1):96–104, 1978/79.
  * [BD85] Arnaud Beauville and Ron Donagi.  La variété des droites d’une hypersurface cubique de dimension $4$.  C. R. Acad. Sci. Paris Sér. I Math., 301(14):703–706, 1985\.
  * [Bui94] Alexandru Buium.  Differential algebra and Diophantine geometry.  Actualités Mathématiques. [Current Mathematical Topics]. Hermann, Paris, 1994.
  * [CML09] Sebastian Casalaina-Martin and Radu Laza.  The moduli space of cubic threefolds via degenerations of the intermediate Jacobian.  J. Reine Angew. Math., 633:29–65, 2009.
  * [CG72] C. Herbert Clemens and Phillip A. Griffiths.  The intermediate Jacobian of the cubic threefold.  Ann. of Math. (2), 95:281–356, 1972.
  * [CS09] Izzet Coskun and Jason Starr.  Rational curves on smooth cubic hypersurfaces.  Int. Math. Res. Not. IMRN, (24):4626–4641, 2009.
  * [DvGK05] I. Dolgachev, B. van Geemen, and S. Kondō.  A complex ball uniformization of the moduli space of cubic surfaces via periods of $K3$ surfaces.  J. Reine Angew. Math., 588:99–148, 2005.
  * [EH87] David Eisenbud and Joe Harris.  The Kodaira dimension of the moduli space of curves of genus $\geqslant 23$.  Invent. Math., 90(2):359–387, 1987.
  * [EH00] David Eisenbud and Joe Harris.  The geometry of schemes, volume 197 of Graduate Texts in Mathematics.  Springer-Verlag, New York, 2000.
  * [EH16] David Eisenbud and Joe Harris.  3264 and all that—a second course in algebraic geometry.  Cambridge University Press, Cambridge, 2016.
  * [Ful98] William Fulton.  Intersection theory, volume 2 of Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge. A Series of Modern Surveys in Mathematics [Results in Mathematics and Related Areas. 3rd Series. A Series of Modern Surveys in Mathematics].  Springer-Verlag, Berlin, second edition, 1998.
  * [GK20] Frank Gounelas and Alexis Kouvidakis.  On some invariants of cubic fourfolds.  arXiv e-prints, page arXiv:2008.05162, August 2020.
  * [GH78] Phillip Griffiths and Joseph Harris.  Principles of algebraic geometry.  Wiley-Interscience [John Wiley & Sons], New York, 1978.  Pure and Applied Mathematics.
  * [Har84] J. Harris.  On the Kodaira dimension of the moduli space of curves. II. The even-genus case.  Invent. Math., 75(3):437–466, 1984.
  * [HM98] Joe Harris and Ian Morrison.  Moduli of curves, volume 187 of Graduate Texts in Mathematics.  Springer-Verlag, New York, 1998.
  * [Huy20] Daniel Huybrechts.  The geometry of cubic hypersurfaces. Lecture notes.  http://www.math.uni-bonn.de/people/huybrech/, 2021.
  * [LSV17] Radu Laza, Giulia Saccà, and Claire Voisin.  A hyper-kähler compactification of the intermediate jacobian fibration associated with a cubic 4-fold.  Acta Mathematica, 218(1):55–135, 2017.
  * [NO21] Denis Nesterov and Georg Oberdieck.  Elliptic curves in hyper-Kähler varieties.  Int. Math. Res. Not. IMRN, (4):2962–2990, 2021.
  * [OSY19] Georg Oberdieck, Junliang Shen, and Qizheng Yin.  Rational curves in holomorphic symplectic varieties and Gromov-Witten invariants.  Adv. Math., 357:106829, 8, 2019.
  * [SV16] Mingmin Shen and Charles Vial.  The Fourier transform for certain hyperkähler fourfolds.  Mem. Amer. Math. Soc., 240(1139):vii+163, 2016.
  * [Sta21] The Stacks project authors.  The stacks project.  https://stacks.math.columbia.edu, 2021.
  * [vdGK11] Gerard van der Geer and Alexis Kouvidakis.  The Hodge bundle on Hurwitz spaces.  Pure Appl. Math. Q., 7(4, Special Issue: In memory of Eckart Viehweg):1297–1307, 2011.
  * [vdGK12] Gerard van der Geer and Alexis Kouvidakis.  The class of a Hurwitz divisor on the moduli of curves of even genus.  Asian J. Math., 16(4):787–805, 2012.
  * [Voi04] Claire Voisin.  Intrinsic pseudo-volume forms and $K$-correspondences.  In The Fano Conference, pages 761–792. Univ. Torino, Turin, 2004.
  * [Voi10] Claire Voisin.  Coniveau 2 complete intersections and effective cones.  Geom. Funct. Anal., 19(5):1494–1513, 2010.
  * [Voi14] Claire Voisin.  Chow rings, decomposition of the diagonal, and the topology of families, volume 187 of Annals of Mathematics Studies.  Princeton University Press, Princeton, NJ, 2014.
  * [W05] C.T.C. Wall.  Sextic curves and quartic surfaces with higher singularities.  Preprint, 2005.


# Attention-Guided Masked Autoencoders For Learning Image Representations

Leon Sick
Ulm University
<EMAIL_ADDRESS>Dominik Engel
Ulm University
<EMAIL_ADDRESS>Pedro Hermosilla
TU Vienna
<EMAIL_ADDRESS>Timo Ropinski
Ulm University
<EMAIL_ADDRESS>
###### Abstract

Masked autoencoders (MAEs) have established themselves as a powerful method
for unsupervised pre-training for computer vision tasks. While vanilla MAEs
put equal emphasis on reconstructing the individual parts of the image, we
propose to inform the reconstruction process through an attention-guided loss
function. By leveraging advances in unsupervised object discovery, we obtain
an attention map of the scene which we employ in the loss function to put
increased emphasis on reconstructing relevant objects, thus effectively
incentivizing the model to learn more object-focused representations without
compromising the established masking strategy. Our evaluations show that our
pre-trained models learn better latent representations than the vanilla MAE,
demonstrated by improved linear probing and k-NN classification results on
several benchmarks while at the same time making ViTs more robust against
varying backgrounds.

##  1 Introduction

Despite their emergence as a promising alternative to CNNs in computer vision,
transformers bring their own training challenges to the table: unstable
training, high computational demand, as well as the need for large datasets to
achieve competitive results [22]. Recent work has shown that self-supervised
Vision Transformer (ViT) pre-training [17, 46, 5, 10, 2] can help mitigate
some of these issues and improve their performance for various vision tasks.
While there have been different pre-training strategies proposed, one idea
that has gained much traction is masking portions of an input image and
training a ViT to reconstruct the missing parts of the image. This task has
been shown to be most effective in the setting of an MAE [17], which
reconstructs the image with equal emphasis on each individual part.

Figure 1: Attention-Guided Reconstruction Loss. We leverage an attention map,
obtained from a self-supervised object discovery network, to inform our guided
reconstruction loss _AttG_ towards reconstructing the relevant objects in the
scene. Through this process, we guide the MAE to learn more effective off-the-
shelf representations.

While MAEs achieve state-of-the-art representations for end-to-end finetuning
tasks, it has also been shown that for linear evaluations, state-of-the-art
results cannot be achieved [17, 1]. With our work, we attempt to make progress
towards closing this performance gap for MAEs on linear evaluations and k-NN
classification. Methods from the field of explainability [35, 37, 45, 7]
suggest that successful image classification models should mainly focus on the
relevant objects in the scene to make their predictions. However, recent
research has shown [28] that many failure cases in state-of-the-art models are
due to the focus of these models on background elements. Therefore, we
hypothesize that inducing semantic knowledge into the MAE pre-training
produces better latent representations, and we propose to induce information
on the relevant object in the scene through an attention-guided reconstruction
loss. While the vanilla MAE does not exhibit a notable deficiency in its
capability to reconstruct objects, we show that guiding the model towards
better reconstruction of the aforementioned allows it to learn representations
more effective for linear evaluations. With our method, we aim to shift the
process of learning representations more to the pre-training phase as to
reduce the need for more expensive adaptation mechanisms like end-to-end
finetuning. Our model therefore learns representations more powerful for
evaluations that harness off-the-shelf features like linear probing and k-NN
classification. By manipulating the reconstruction loss, our model can be
guided by an attention map to put increased emphasis on reconstructing
relevant objects (see Fig. 1), which according to our ablations improves
linear evaluations as compared to changing the established input masking
strategy [25, 20]. To generate the employed attention maps, we explore several
object discovery methods, both ViT- and CNN-based. Our results show that our
attention guidance mechanism _AttG_ narrows the gap for linear evaluation and
k-NN classification tasks for MAEs. Thus, we obtain latent representations
that enable increased top-1 accuracy for k-NN classification, where we see the
biggest improvement, and linear probing in standard and few-shot settings.
Furthermore, our learned, off-the-shelf representations prove to be especially
useful on datasets not seen during pre-training as well as for image retrieval
tasks. Further, by keeping a high input masking ratio, our method effectively
enables our performance gains to be achieved without significantly increasing
computational requirements.

##  2 Related Work

Our work builds upon recent advances in masked modeling, as well as self-
supervised pre-training.

Masked modeling. Exploring a reconstruction task of masked regions in the
input modality has first seen its successes in language processing with
transformer architectures [40, 14] in BERT [13] or GPT [3, 33, 34]. More
recently, these methods have been successfully transferred to the computer
vision domain. In their work on masked autoencoders, He  _et al_. [17] mask a
large number of image patches and feed the remaining tokens to the ViT [14]
encoder. Afterwards, the masked tokens are added and fed through the ViT
decoder with the goal of reconstructing the masked patches. Feichtenhofer  _et
al_. [15] and Tong  _et al_. [39] extend this idea into the spatio-temporal
domain for video data, while Girdhar  _et al_. [16] propose a unified MAE
architecture for both image and video data. Recently, masking strategies have
also been applied for generative image transformers in work done by Chang  _et
al_. [6]. Their model learns to predict the initially masked tokens of the
training image. To generate high-fidelity images at inference time, they
sample the model to iteratively predict the tokens to be reconstructed and
thereby refine the image. Li  _et al_. [25] propose to adapt a semantic-guided
masking strategy for pre-training an MAE. They extract segmentation parts by
refining the multi-head attention of a ViT encoder and use these to mask a
portion or the whole part in the input image. While being effective, their
approach requires additional finetuning on their segmentation model before
extracting the segmentation parts. In contrast, our method does not require
this preparation step and directly works with out of the box object discovery
methods. Furthermore, we do not alter the original masking strategy, as it has
proven to be beneficial by He  _et al_. [17]. AttnMask, a novel masking
strategy proposed by Kakogeorgiou et al. [20], is integrated into a student-
teacher knowledge distillation training process where the teacher network
outputs an attention map that is used to mask the input image for the student
network. They also extend their masking approach to keep some patches of the
attention map visible, giving the model additional ”hints”. In their work on
SimMIM, Xie  _et al_. [44] successfully design an approach for masked image
modelling (MIM), for which they formulate a raw pixel regression task to
predict the reconstruction of all masked pixels using a prediction head on top
of the latent representation output by their encoder. Bao  _et al_. [2]
propose BEiT, which in contrast predicts masked patches in the token space,
which is created from the image patches by a frozen discrete VAE.

Self-supervised pre-training. Recent work [38, 5, 4, 9, 8, 27] has shown
significant advances in self-supervised pre-training for vision models with
various backbone architectures like ResNets [19] and ViTs [14]. Chen  _et al_.
[8] explore pre-training for ResNets and highlight that effective data
augmentation is a critical factor for pre-training on images. Their approach
is formulated so that its performance is dependent on the batch size. Caron
_et al_. [4] built upon these findings and introduced a multi-crop
augmentation strategy as well as a novel learning approach to eliminate the
batch size dependence. Pre-training of ResNets is further pushed by works from
Mitrovic  _et al_. [27] and Tomasev  _et al_. [38]. Their works propose to
formulate a probabilistic learning objective through an invariance
regularization mechanism, built upon the Kullback-Leibler divergence. Research
by Caron  _et al_. [5] puts special emphasis on exploring self-supervised pre-
training with ViTs. They investigate the use of knowledge distillation and
find that their ViTs can provide a segmentation of the scene and also perform
well on k-NN classification. With MoCo v3 [10], Chen  _et al_. also explore
the properties of ViTs for pre-training in an evolution of their initial
methods [18, 9].

##  3 Method

(a) Attention-Guided Masked Autoencoder Architecture. (b) Scaling Function
Figure 2:  _AttG_ Architecture Overview And Scaling Function. Our
architecture, displayed in Figure 2(a), employs two streams. Besides the main
$MAE$ backbone, we facilitate an unsupervised object discovery ($OD$) stream
with fixed weights to obtain a patch-based attention map. This attention map
is then scaled and finally used to inform our attention guided reconstruction
loss _AttG_ towards relevant objects. Gray overlays depict image patches which
are not part of the loss since they have not been masked in the input image.
To make the attention map useful for reconstruction guidance, we scale its
values, as visualized in Figure 2(b). This results in the loss for background
patches not being masked out and patches from the relevant object in the scene
being further emphasized.

In our work, we build upon vanilla MAEs [17]. We enable them to learn more
robust representations by steering their focus on objects of interest, without
affecting the random patch sampling found to be essential by He  _et al_.
[17]. As illustrated in Fig. 2(a), we facilitate unsupervised object discovery
(OD) and use the obtained results within a temperature-controlled guidance
loss to steer the model towards the relevant objects. In the following
subsections, we will detail our learning architecture (Section 3.1), introduce
our attention-guided loss (Section 3.2), and discuss unsupervised OD (Section
3.3).

###  3.1 Learning Architecture

Our learning architecture exploits two input streams as illustrated in Fig.
2(a). As with conventional MAEs, the backbone of our learning architectures is
a ViT-based autoencoder, which is trained to reconstruct an input image based
on a masked version of the same image. To achieve this, the autoencoder first
encodes the input to a latent representation, which is then fed into a decoder
to perform the reconstruction process. To feed the input into the transformer,
we first divide it into non-overlapping patches and apply patch-based masking,
before the remaining visible patches are fed into the encoder. In the latent
space, the masked patches are considered again and the original amount of
tokens is fed into the decoder for reconstruction. The reconstruction loss is
computed on the same per-patch basis, as the masking is obtained. To guide the
reconstruction loss towards the objects of interest, our architecture
facilitates an additional stream for unsupervised OD. Within this stream, a
patch-based attention map, focusing on relevant objects, is obtained from the
input image. As we want to use this attention map to steer the reconstruction
loss, it is important to obtain patches with the same sizes, as the MAE
backbone processes.

###  3.2 Attention-Guided Loss

In order to exploit the attention map obtained in our OD stream, we need to
define a guiding mechanism for the model. While directly applying the
attention map $\mathcal{M}$ to generate an object-based masking on the input
image might seem intuitive, He  _et al_. [17] show that a high masking ratio
in combination with random sampling significantly outperforms a block-wise
masking strategy as proposed by Bao  _et al_. [2] with a lower masking ratio.
Since the latter is similar to masking objects of interest in the input image,
simple object-based patch masking cannot be assumed to be a promising
strategy. Additionally, masking the object in the input image over and over
again will not lead to masking pattern variations, which have been proven to
be essential for robust MAE training. Therefore, we opt to not interfere with
the input masking strategy, and rather steer the learning process by using the
attention maps to guide the reconstruction loss. With this approach, we expect
the autoencoder to put increased emphasis on relevant objects, without
affecting beneficial patch sampling strategies.

To employ a guided loss instead of the vanilla MAE reconstruction loss, we
compute the mean squared error (MSE) between the reconstructed and the
normalized original image patches in pixel space, for which we only consider
masked patches. With this, we obtain the per-patch reconstruction loss, i.e.,
each patch has its individual loss value. Having a loss value per patch,
enables us to not only have a general influence on the learning process, but
to guide the model in a more targeted way by individually weighting different
parts of the image.

Therefore, to make our attention maps effective for guidance, they are first
normalized to $\mathcal{M}_{\text{norm}}\in[0,1]$. With this, all values for
patches that show part of the object are close to 1 while background patches
are close, but never equal, to $0$. Using the map in this form would not be
useful, since background patches with values of $0$ would lead to our
attention maps zeroing-out all loss values for background patches in the per-
patch reconstruction loss, effectively rendering the model unable to learn to
reconstruct the entire image. Our goal is instead, to keep loss values of
background patches, but in a non-scaled manner. To achieve this, we scale our
attention maps with the exponential function and a temperature parameter
$\tau>0$:

$\mathcal{M}_{\text{scaled}}=\exp\left(\frac{\mathcal{M}_{\text{norm}}}{\tau}\right)$
(1)

The exponential function provides the ideal properties since background
patches with values of $0$ are now converted to $1$, so that, when weighting
the reconstruction, loss values for background patches are left unchanged. At
the same time, the attention map values for object patches are scaled, giving
them a stronger impact when weighting the loss. By adding a temperature
parameter $\tau$, we obtain an additional hyperparameter, which gives us fine-
grained control over the strength of weighting.

With this, we arrive at our guided loss

$\mathcal{L}_{\text{AttG}}=\gamma\cdot\mathcal{L}_{\text{Rec}}\cdot\mathcal{M}_{\text{scaled}}$
(2)

with $\gamma$ being the binary mask of all masked tokens and
$\mathcal{L}_{\text{Rec}}$ the reconstruction loss. We schedule $\tau$ to
increase according a half-cycle cosine temperature schedule starting at
$\tau=0.75$, which we find most effective as detailed in Section 5.4. Our full
training process is detailed in Algorithm 1. As mentioned before, the
temperature parameter enables us to control how strongly the MAE is guided. By
decreasing $\tau$, the scaling effect on the attention map is increased,
therefore providing a stronger guidance to the reconstruction loss. On the
other hand, increasing it leads to a softer guidance and less enforced
emphasis on reconstructing the relevant object in the scene.

Algorithm 1 Pseudocode Attention Map Guidance

image $\mathcal{X}$, attention map $\mathcal{M}$,

masking function mask, model MAE

for all $\mathcal{X}$, $\mathcal{M}$ in Dataset do

$\mathcal{X}_{\text{masked}},\gamma\leftarrow\textbf{mask}(\mathcal{X},\text{ratio}=0.75)$

$\hat{\mathcal{X}}\leftarrow{}\textbf{MAE}(\mathcal{X}_{\text{masked}})$

$\mathcal{M}_{\text{norm}}\leftarrow(\mathcal{M}-\mathcal{M}_{\text{min}})/(\mathcal{M}_{\text{max}}-\mathcal{M}_{\text{min}})$

$\mathcal{M}_{\text{scaled}}\leftarrow\exp{(\frac{\mathcal{M}_{\text{norm}}}{\tau})}$
(Eq. 1)

$\mathcal{X}_{\text{norm}}\leftarrow(\mathcal{X}-\mu_{\mathcal{X}})/(\sigma_{\mathcal{X}}+\epsilon)^{0.5}$

$\mathcal{L}_{\text{Rec}}\leftarrow(\hat{\mathcal{X}}-\mathcal{X}_{\text{norm}})^{2}$

$\mathcal{L}_{\text{Guided}}\leftarrow\gamma\cdot\mathcal{L}_{\text{Rec}}\cdot\mathcal{M}_{\text{scaled}}$
(Eq. 2)

end for

(a) Original

(b) TokenCut

(c) DINO

(d) Grad-CAM

Figure 3: Visual Comparison Of Our Attention Maps. We illustrate the
differences between our attention maps. Each has been normalized and scaled in
the manner as we use them for pre-training our models. From these examples,
TokenCut can be perceived to output the best foreground-background separation
maps of all presented methods. We study the effect of the different maps on
our guidance in Section 5.1.

###  3.3 Object Discovery

The effectiveness of our guidance loss introduced above, stands and falls with
the quality of the exploited attention maps. We ensure wide applicability of
our method by exploiting state-of-the-art unsupervised OD approaches. Based on
their reported results as well as their wide-spread use, we have investigated
three unsupervised OD approaches to inform our guided loss: a DINO-pretrained
ViT [5], a TokenCut ViT [42], and a Grad-CAM-based approach [35]. While the
first two are transformer-based approaches, which directly deliver attention
maps, Grad-CAM is often praised for its capabilities to extract structures
highly relevant for a learner, and has therefore also been considered. With
these three approaches, we generate an attention map $\mathcal{M}$ of the
object in the scene that serves as a foreground-background segmentation.

For the pre-trained DINO [5] ViT, the attention map is obtained, by feeding
the image through the network and querying the self-attention module of the
last block. This self-attention module with $H$ heads and consequently
$j=1,...,H$ sequences takes as input the normalized output sequence from the
previous block $Y~{}\in~{}\mathbb{R}^{(n+1)\times d}$ and feeds it through
three linear layers to map it to the sequences for query $Q_{j}$, key $K_{j}$
and value $V_{j}$, all $\in~{}\mathbb{R}^{(n+1)\times d^{\prime}}$. The
attention matrix is then calculated as follows:

$\mathcal{A}_{j}=\text{softmax}(Q_{j}K^{T}_{j}/\sqrt{d^{\prime}})$ (3)

with $d^{\prime}=d/H$, allowing for the maps of all attention heads to be
retrieved. For DINO, we use the attention maps without post processing.

TokenCut ViT [42] post-processes the segmentation-like attention maps
$\mathcal{A}_{j}$ of the pre-trained DINO transformer to obtain a more refined
segmentation. Briefly described, TokenCut constructs an undirected graph of
patches with each patch having assigned a feature vector. Each patch is linked
to neighboring patches through edges, represented by the cosine similarity
between the two vectors. The normalized cuts algorithm [36] is then used to
obtain the final segmentation output $\mathcal{M}$.

The Grad-CAM-based approach [35]visualizes the gradients for a certain
prediction.The result is a segmentation-like attention map. Differently
thought, this approach originally requires labels to inform the process. Since
we do not have any labels available, we select the highest scoring category as
our target. Since the Grad-CAM model will output a pixel-wise heatmap, we
apply average pooling to obtain patches of the same patch-wise format as our
MAE and the other two ViTs. In our approach we use a supervised
ImageNet-1K-pretrained ResNet-50 as the encoder.

Figure 3 shows a visual comparison of the segmentation outputs. From
qualitative observations, TokenCut provides the most semantically accurate
attention maps with evenly valued patches and a clear distinction between
object and background. DINO’s maps also provide a useful semantic layout of
the scene but are lacking the same distinctiveness. Grad-CAM on the other hand
outputs heatmap-like masks that provides a coarser map. In our ablations
(Section 5.1), we explore the effectiveness of these different maps for our
guidance mechanism.

##  4 Experiments

We pre-train our model on ImageNet [12] (IN1K). Our attention maps are
generated by TokenCut, as they perform the best according to our ablations in
Section 5.1. We then evaluate our learned representations with linear probing,
and k-NN classification studies, both on IN1K and other relevant benchmarks,
as well as low-shot finetuning and image retrieval studies.

###  4.1 ImageNet Evaluation

Method | Size | Epochs | k-NN | Linear
---|---|---|---|---
SimMIM | ViT-B/16 | 800 | 9.4 | 56.7
SemMAE | ViT-B/16 | 800 | 45.1 | 65.0
BEiT | ViT-L/16 | 800 | 11.4 | 73.5
MAE | ViT-L/16 | 800 | 52.8 | 73.5
_\+ AttG (Ours)_ | ViT-L/16 | 800 | 56.2 | 74.4
MAE | ViT-L/16 | 1600 | 50.9 | 75.1
_\+ AttG (Ours)_ | ViT-L/16 | 1600 | 59.0 | 75.9
Table 1: $\bm{k}$-NN classification and linear probing on IN1K for purely MIM-based techniques. We show results for linear probing and $k$-NN classification, for IN1K we use $k$=20. For both training durations, our method improves upon the results of the MAE for both $k$-NN and linear probing.  Method | Size | Epochs | 1% | 10%
---|---|---|---|---
SimMIM | ViT-B/16 | 800 | 23.2 | 66.3
SemMAE | ViT-B/16 | 800 | 52.8 | 70.6
BEiT | ViT-L/16 | 800 | 38.3 | 71.6
MAE | ViT-L/16 | 1600 | 67.1 | 78.5
_\+ AttG (Ours)_ | ViT-L/16 | 1600 | 67.5 | 78.1
Table 2: Low-shot finetuning on IN1K. We finetune the pre-trained models with
1% and 10% of IN1K training data and evaluate on the full validation set. Our
model performs especially well in the 1% regime.

We collect several comparable methods in the field and report k-NN
classification and linear probing results on ImageNet-1K in Table 1. For a
leveled comparison, Table 1 evaluates the vanilla MAE and our attention
guidance method, AttG, against other pure MIM-based approaches without a
contrastive objective like SimMIM [44], BEiT [2] and SemMAE [25]. We show that
our method is able to outperform other purely MIM-based techniques. With our
attention guidance, our model improves upon the vanilla MAE on top-1 accuracy
for linear probing by +0.9% after 800 pre-training epochs and +0.8% after 1600
epochs, showcasing the effectiveness of our learned off-the-shelf
representations. We also perform $k$-NN classification with $k$=20 on
ImageNet-1K. At 1600 epochs, we are able to outperform the MAE baseline by
+8.1%. We also evaluate our model on low-shot finetuning with 1% and 10% of
training data and report the results in Table 2. We see that our model
performs best in for the 1% subset.

Additionally, we experiment with k-NN classification in a few-shot setting
with the 1600-epoch models. We evaluate our model with 1, 5, 10 and 20
training images on the full validation dataset. Our results in Table 3 show
our model outperforming for all setups, while the delta increases from +4.4%
to +13.5% with an increasing number of training images, which underlines the
quality of our representations. We also evaluate on IN1K in a few-shot linear
probing setting. As presented in Table 7, our attention guidance improves the
results of the MAE across all subsets.

Method |  # | 1 | 5 | 10 | 20
---|---|---|---|---|---
MAE |  | 6.5 | 12.1 | 15.6 | 20.2
_\+ AttG (Ours)_ |  | 10.9 | 19.8 | 26.8 | 33.7
Table 3: $\bm{k}$-NN classification with few embedded training examples. We
report $k$-NN classification top-1 accuracy for $k$=5 with an IN1K subset of
$\\#\in\\{1,5,10,20\\}$ images. With an increasing number of training images,
the delta by which our model outperforms the vanilla MAE increases. Both
models have been pretrained for 1600 epochs.

###  4.2 Comparison to Attention-Masking Add-On Methods

We also compare our method against other approaches that use attention maps to
build upon a pre-existing baseline, such as AttnMask [21], an extension of
iBOT [46], and SemMAE [25], which also builds upon the vanilla MAE. Since
these methods build upon different baselines, and therefore also start at
varying levels of performance, we attempt to isolate their effects by
reporting the delta in performance these methods exhibit versus their
respective baselines. Therefore, in Table 4, we report the gains of AttnMask
compared to iBOT, whereas for SemMAE and our method, we report additional
performance versus the MAE baseline. We show that our method leads to the
overall greatest improvement on k-NN performance. While the performance gains
for linear probing are below those of AttnMask, our method for guidance with
attention maps is the most effective for MAEs.

###  4.3 Robustness to Background Changes

ImageNet-9 [43] (IN-9) provides a suitable dataset for evaluating robustness
against background variations in image classification. It includes validation
datasets for nine classes from ImageNet-1K [12] where either the background or
the object in the foreground have been altered, resulting in eight different
benchmarks. We perform linear probing on the larger training set, ImageNet-9L
(IN-9L). IN-9 contains four validation subsets which are suitable for
evaluating a model’s robustness to background changes: Only-Foreground (OF),
Mixed-Next (MN), Mixed-Rand (MR), and Mixed-Same (MS). These subsets contain
images with modified backgrounds, of which MN and MR contain out-of-
distribution (OOD) backgrounds. MS is edited with a different in-distribution
backgrounds, while OF only has the background removed. We report top-1
accuracy for linear probing in Table 5. Our model is able to outperform the
MAE on all four tasks.

###  4.4 Transfer to Other Datasets

To evaluate the capability of our model to extract relevant features from not
only ImageNet, but also other benchmarks, we perform k-NN classification and
linear probing on the CIFAR-100 [24], Oxford Flowers [29], and Stanford Cars
[23] datasets. We present our results in Table 6. Across all tasks, our model
outperforms the vanilla MAE. This yet again shows our model’s capability to
extract valuable features, even for datasets other our pre-training dataset.
We especially see strong results on CIFAR-100, where the vanilla MAE is
outperformed by +11.5% on k-NN classification accuracy and +2.3% for linear
probing.

Baseline | Add-On | Epochs | k-NN | Linear
---|---|---|---|---
iBOT | _\+ AttnMask_ | 100 | +1.3 | +1.7
MAE | _\+ SemMAE_ | 800 | - | +0.7
_\+ AttG (Ours)_ | 800 | +3.4 | +0.9
_\+ AttG (Ours)_ | 1600 | +8.1 | +0.7
Table 4: Comparison of performance deltas from attention-informed approaches that are built on top of different baselines. We compare our method to SemMAE, another method that builds upon MAE, and AttnMask, a technique on top of iBOT. We report gains for linear probing and k-NN on IN1K.  |  |  |  |
---|---|---|---|---
Method |  OF |  MS |  MR |  MN
MAE |  93.6 |  92.8 |  81.9 |  78.9
_+AttG(Ours)_ |  93.9 |  93.9 |  83.9 |  81.1
Table 5: Background influence evaluation on ImageNet-9 [43]. Our model has an
edge on top-1 accuracy over the vanilla MAE on three benchmarks with modified
backgrounds (OF, MR, and MN), showcasing the effectiveness of our method for
pre-training robust ViTs. We perform linear probing on IN-9L.

Dataset | Method | Epochs | k-NN | Linear
---|---|---|---|---
CIFAR100 | MAE | 1600 | 48.9 | 74.1
_\+ AttG (Ours)_ | 1600 | 60.4 | 76.4
Flowers | MAE | 1600 | 55.0 | 72.8
_\+ AttG (Ours)_ | 1600 | 66.2 | 75.2
Cars | MAE | 1600 | 10.3 | 24.3
_\+ AttG (Ours)_ | 1600 | 13.2 | 29.3

Table 6: $\bm{k}$-NN classification and linear probing on different datasets.
We evaluate our method on a wide variety of transfer datasets by performing
linear probing and k-NN evaluation and observe consistent performance
improvements with our method. Our evaluation settings are equivalent to those
in Table 1.

Dataset | Method |  # | 10 | 25 | 50
---|---|---|---|---|---
IN1K | MAE |  | 39.3 | 56.0 | 62.3
_\+ AttG (Ours)_ |  | 43.3 | 57.7 | 63.1
CIFAR-100 | MAE |  | 33.0 | 39.6 | 52.0
_\+ AttG (Ours)_ |  | 35.4 | 42.8 | 55.7
Cars | MAE |  | 11.0 | 19.3 | 23.2
_\+ AttG (Ours)_ |  | 12.9 | 23.6 | 29.4
DTD | MAE |  | 46.4 | 53.6 | -*
_\+ AttG (Ours)_ |  | 46.4 | 53.6 | -*
Pets | MAE |  | 59.1 | 69.9 | 71.3
_\+ AttG (Ours)_ |  | 69.9 | 78.9 | 80.0

Table 7: Few-shot classification using linear probing on transfer datasets. We
report top-1 accuracy results for few-shot classification when linear probing
both models on $\\#\in\\{10,25,50\\}$ images per class. Both models have been
pretrained for 1600 epochs. *DTD train split contains only 40 images per
class.

###  4.5 Few-Shot Transfer

We also explore effectiveness of our learned representations in different few-
shot linear probing settings. For both CIFAR-100 and Stanford Cars, we limit
the amount of training images to 10, 25, and 50. To test our methods
robustness against changes in texture, we also evaluate on the Describable
Textures Dataset (DTD) [11] and the Oxford-IIIT Pets dataset [30]. We perform
linear probing on the reduced datasets and evaluate the models’ performance on
the full validation splits. As detailed in Table 7, our approach is able to
outperform the vanilla MAE by substantial margins throughout the entire range
of subset configurations and across multiple datasets. These results
demonstrate once more that our attention-guided loss is highly effective in
enabling MAEs to learn better representations.

###  4.6 Image Retrieval

We evaluate our learned representations with regard to their effectiveness on
the revisited [31] Oxford and Paris image retrieval datasets [32]. Our
evaluation protocol follows Caron  _et al_. [5]. We freeze our model and
retrieve the image pairs using a $k$-NN based on the $1024$-d output
representations from the network. We report the mAP for the Medium (M) and
Hard (H) splits in Table 8. With our representations, we are able to improve
the performance of the vanilla MAE across the board.

##  5 Ablations

In all ablations, we pre-train all models for 400 epochs on ImageNet-100.

###  5.1 Attention Map Quality

We experiment with different approaches for generating our attention maps. An
overview of qualitative differences can be observed in Figure 3. The TokenCut
attention map covers many details, such as the spider legs or the shark fins.
DINO on the other hand outputs an attention map mostly covering the object but
not with equal emphasis. In contrast, the Grad-CAM output generously overlays
the entire animal but also intersects many background patches. In Table 9, we
report the influence of using different OD approaches for deriving the
attention maps informing our reconstruction loss. All models are pre-trained
with an input masking ratio of $0.75$ and a half-cycle cosine temperature
schedule starting at $\tau=0.75$. As expected based on the qualitative
results, TokenCut maps perform best with a clear advantage over the other two
OD approaches. This confirms our intuition that high-quality attention maps
should yield the best results, and proves the effectiveness of our approach,
since the model seems to be able to extract most valuable semantic knowledge
from highest quality maps.

On the other hand, the Grad-CAM attention maps seem to be more effective than
the seemingly higher-quality DINO maps. This finding could point towards the
necessity of enough guidance mass, since the DINO map, while being visually
perceived to be more accurate, generally seems to have less impact on the
reconstruction loss.

Dataset |  $\mathcal{R}$Ox |  $\mathcal{R}$Par
---|---|---
Method | M | H | M | H
MAE | 13.1 | 2.3 | 24.4 | 5.2
_\+ AttG (Ours)_ | 19.1 | 3.0 | 33.0 | 8.4
Table 8: Image retrieval. We evaluate on the medium (M) and hard (H) image retrieval tasks of the Oxford and Paris datasets after pre-training IN1K. We show that our network’s off-the-shelf representations enable improved performance on all tasks. Model | Linear | k-NN
---|---|---
TokenCut [42] | 77.1 | 57.1
DINO [5] | 75.7 | 56.0
Grad-CAM [35] | 76.3 | 57.0
Table 9: Comparison of different OD approaches. The TokenCut maps prove to be most effective for our guidance mechanism. This is in accordance with our qualitative evaluation, where TokenCut seems to provide the highest quality attention maps.  Attention Guidance | k-NN | Linear
---|---|---
Input Masking | 39.2 | 57.1
Foreground-Only | 56.7 | 74.8
Background-Only | 43.5 | 69.2
Inverted Attention Map | 48.7 | 72.6
_AttG_ Reconstruction Guidance (Ours) | 57.0 | 77.1
Table 10: Ablation of different uses of the attention map. Guiding the MAE
through the reconstruction loss yields better results, both for k-NN
classification with $k$=5 and linear probing.

###  5.2 Computational Overhead

Applying our attention maps to the reconstruction of the MAE comes with only
marginally increased computational costs. When pre-training our models, we
observed an increase in runtime of below 1% over the vanilla MAE. Our
attention maps can be generated during training, but to ensure reproducibility
of our results, we decide to generate them once ahead of pre-training and use
the same for all our experiments. Obtaining an attention map requires a single
forward pass of the object discovery network over the dataset, making it a
negligible computational overhead in relation to the total training time. We
will provide an efficient implementation for generating all attention maps.

###  5.3 Guidance Mechanism

One alternative to our proposed attention-guided reconstruction loss would be
to inform the initial _Input Masking_ of the input image through the attention
maps [25, 21]. Therefore, we investigate a modified masking strategy, where we
keep our masking ratio of 75% by sampling the values from the attention map in
a descending fashion, removing patches with the highest attention values. We
assume that in most cases, this masks the relevant object as well as some
additional patches. Using this approach together with the original MAE loss
results in significantly inferior results across multiple of our evaluations,
when compared to our proposed attention-guided reconstruction loss.
Furthermore, we experiment with different alternatives to guiding the
reconstruction loss with our attention maps. We try reconstructing
_Foreground-Only_ by zeroing all patches with attention values below the
10%-quantile threshold. Furthermore, to emphasize background reconstruction,
we use the _Inverted Attention Map_ s to guide the MAE and also mask out the
10%-quantile threshold of this inverted map to reconstruct _Background-Only_.
In Table 10, we report the top-1 accuracy for k-NN classification and linear
probing on IN-100. As can be seen, using our attention loss guidance
outperforms the alternative approaches.

###  5.4 Temperature Parameter Influence

In Table 11, we evaluate the influence of different temperature parameters and
our half-cycle temperature schedule. For fixed values, we observe that
$\tau=0.8$ leads to the best linear probing results. But when slowly
increasing $\tau$ from $0.8$ on, we see a drop in performance, fueling the
belief that further increasing the temperature is ineffective. A decrease of
$\tau$ leads to worse performance in our experiments. We achieve our best
results when increasing the temperature according to a half-cycle temperature
schedule, starting at $\tau=0.75$, outperforming the model with fixed
$\tau=0.75$ by +0.5% top-1 accuracy for linear probing.

##  6 Limitations

As outlined in Section 1, MAEs provide state-of-the-art results for finetuning
tasks, while not being able to do so for linear evaluations. As we show that
our method narrows the gap for linear evaluations, it is important to also
consider its impact on finetuning tasks. Our findings show that while our
method does not outperform the vanilla MAE on finetuning, our results are
still competitive. Also, our approach exhibits dependence on the object
discovery network used to obtain the attention maps.

$\tau$ | With Schedule | Without Schedule
---|---|---
0.7 | 76.1 | 76.3
0.75 | 77.1 | 76.6
0.8 | 76.5 | 77.0
Table 11: Experiments with different temperatures and scheduling. We observe
the best linear probing results when $\tau=0.75$ and is increased with a half-
cycle cosine schedule.

##  7 Conclusion & Discussion

We have proposed an attention-guided reconstruction masked autoencoder, which
introduce an attention-guided reconstruction loss together with attention
maps. Our results have shown that our attention-guided loss yields
representations that enable increased top-1 accuracy for linear probing and
k-NN classification across different datasets after IN1K pre-training.
Further, our models exhibit improved properties for few-shot classification
and have shown to be robust against realistically modified backgrounds. The
presented experiments showcase the power of reconstruction guidance to learn
better representations. Thus, our method presents a significant step towards
enabling MAEs to learn more powerful representations already during pre-
training that improve linear evaluations and k-NN classifications.

Our approach has been designed to be transferable to future MAE frameworks
since we apply our guidance to the reconstruction loss. While our guidance
mechanism has shown its usefulness when pre-trained on object-centric
datasets, we leave it open to future work to design further guidance
approaches that extend to multi-object semantic structures as can be found in
COCO [26]. To apply our attention-guided reconstruction loss, one could
replace TokenCut with CutLER, a recent work by Wang et al. [41]. This method
extends TokenCut for multi-object discovery, which would allow both guiding
individual objects separately or simultaneously.

## References

  * [1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.  Self-supervised learning from images with a joint-embedding predictive architecture, 2023.
  * [2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  In International Conference on Learning Representations, 2021.
  * [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems, 33:9912–9924, 2020.
  * [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660, 2021.
  * [6] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and Bill Freeman, editors.  MaskGIT: Masked Image Generative Transformers, 2022.
  * [7] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839–847. IEEE, 2018.
  * [8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [9] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [10] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021.
  * [11] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [13] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT (1), 2019.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [15] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He.  Masked autoencoders as spatiotemporal learners.  arXiv preprint arXiv:2205.09113, 2022.
  * [16] Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Omnimae: Single model masked pretraining on images and videos.  arXiv preprint arXiv:2206.08356, 2022.
  * [17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis.  What to hide from your students: Attention-guided masked image modeling.  arXiv preprint arXiv:2203.12719, 2022.
  * [21] Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis.  What to hide from your students: Attention-guided masked image modeling.  CoRR, abs/2203.12719, 2022.
  * [22] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  ACM computing surveys (CSUR), 54(10s):1–41, 2022.
  * [23] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
  * [24] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [25] Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng.  Semmae: Semantic-guided masking for learning masked autoencoders.  arXiv preprint arXiv:2206.10207, 2022.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [27] Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell.  Representation learning via invariant causal mechanisms.  In International Conference on Learning Representations, 2020.
  * [28] Mazda Moayeri, Phillip Pope, Yogesh Balaji, and Soheil Feizi.  A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19087–19097, June 2022.
  * [29] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
  * [30] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar.  Cats and dogs.  In IEEE Conference on Computer Vision and Pattern Recognition, 2012\.
  * [31] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman.  Lost in quantization: Improving particular object retrieval in large scale image databases.  In 2008 IEEE conference on computer vision and pattern recognition, pages 1–8. IEEE, 2008.
  * [32] Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum.  Revisiting oxford and paris: Large-scale image retrieval benchmarking.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5706–5715, 2018.
  * [33] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [35] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [36] Jianbo Shi and Jitendra Malik.  Normalized cuts and image segmentation.  IEEE Transactions on pattern analysis and machine intelligence, 22(8):888–905, 2000.
  * [37] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [38] Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic.  Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet?  arXiv preprint arXiv:2201.05119, 2022.
  * [39] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  arXiv preprint arXiv:2203.12602, 2022.
  * [40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [41] Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra.  Cut and learn for unsupervised object detection and instance segmentation.  arXiv preprint arXiv:2301.11320, 2023.
  * [42] Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz.  Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut.  arXiv preprint arXiv:2209.00383, 2022.
  * [43] Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry.  Noise or signal: The role of image backgrounds in object recognition.  In International Conference on Learning Representations, 2020.
  * [44] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663, 2022.
  * [45] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.  Understanding neural networks through deep visualization.  arXiv preprint arXiv:1506.06579, 2015.
  * [46] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  Image bert pre-training with online tokenizer.  In International Conference on Learning Representations, 2021.