# Probing the incompressibility of dense hadronic matter near QCD phase
transition
in relativistic heavy-ion collisions

Zhi-Min Wu1,2 Gao-Chan Yong1,2<EMAIL_ADDRESS>1Institute of Modern
Physics, Chinese Academy of Sciences, Lanzhou 730000, China
2School of Nuclear Science and Technology, University of Chinese Academy of
Sciences, Beijing 100049, China

###### Abstract

Based on the extended hadronic transport model of relativistic heavy-ion
collisions, the incompressibility of dense hadronic matter created in
relativistic Au+Au heavy-ion collisions at $\sqrt{s_{NN}}=3$ GeV is studied.
By comparing experimental proton directed flow, productions of strange hadrons
$\phi$, $K^{-}$ as well as their ratio $\phi/K^{-}$, proton high-order
cumulants to the model calculations, a large incompressibility of dense
hadronic matter is obtained from nucleon observabels while a rather small
incompressibility is needed to fit the data of strange hadrons. This may
indicate hadronic matter possesses different incompressibilities in different
density regions, i.e., the incompressibility may become stiffer from
saturation density to a certain baryon density and then turn to soft before
reaching hadron-quark phase transition. The study also shows that the
incompressibility significantly affects the critical baryon density of hadron-
quark phase transition.

##  I Introduction

One of the main goals of relativistic heavy-ion collisions is to understand
the properties of nuclear matter under conditions of extreme energy and baryon
density or to explore the phase diagram of hot and dense nuclear matters pr1 ;
pr2 ; pr3 . A lot of circumstantial signals in relativistic heavy-ion
collisions have suggested the formation of quark-gluon plasma (QGP) qgp1 ;
qgp2 . Currently it is commonly considered that at small baryon chemical
potential and high temperature, the transition from hadronic phase to QGP
phase is a smooth crossover, whereas a first-order phase transition is
expected at high baryon chemical potential region. Due to its fundamental
importance, the exploration of quantum chromodynamics (QCD) phase diagram of
strongly interacting nuclear matter is the current focus of many research
activities worldwide, both theoretically and experimentally pt1 ; pt2 ; ptko22
; guo2021 ; nara18 . Indeed, mapping the QCD phase diagram is the major
scientific goal of the beam energy scan (BES) program in heavy-ion collisions
pr2 ; bes19 ; besa ; besb . In astronomy, whether there is hadron-quark phase
transition in neutron-stars (NSs) with central densities of several times
nuclear saturation density is also of great interest in the study of neutron
star structure akm1998 ; nature2020 ; liapj2020 ; hu2021 ; kojo2021 ; ran2021
; xie2021  and gravitational-wave (GW) emission gw2019 ; gw2018 ; gw20182 .
The study of the phase transition of QCD matter from earth to heaven is
thought to have crucial implications toward an unprecedented understanding of
the early and present universe ann2006 . It’s worth noting that the
exploration of dense matter theory for heavy-ion collisions and neutron stars
has been one part of the new US Long-Range Plan in Nuclear Physics usplan2023
.

By constructing equation of state (EoS) of nuclear matter created in heavy-ion
collisions and simulating relevant observables and comparing with experimental
data, one can get related information on the stiffness of EoS as well as the
hadron-quark phase transition of nuclear matter lipc2022 ; stj2022 ; dmy2022 .
Previous studies on nucleon directed and elliptic flows by Danielewicz _et
al_. showed the possible change in compressibility with increasing baryon
density dan2002 . Their studies seem to very different from the recent results
by Nara _et al_., the latter explained the colliding energy dependence of the
proton directed flow with a single hadronic EoS nara22 . Fortunately, the
recent STAR experiments clarified some facts on the properties of dense QCD
matter v1flow2021 ; qgpa21 . Theoretically, the compressed baryon density
reached in heavy-ion collisions depends on the EoS used and the effects of the
EoS on relevant observables in heavy-ion collisions also depend on the
compressed baryon density reached. Interplay of the compressed baryon density
reached in heavy-ion collisions and the effects of the EoS without or with the
hadron-quark phase transition at various densities complicates the question.

To probe the boundary of hadron-quark phase transition of nuclear matter,
constraints could be obtained from the recent RHIC Beam Energy Scan-II program
v1flow2021 ; comu2021 ; kpsi2021 . It is shown from different aspects that
hadronic interactions dominate in matter created in relativistic Au+Au
collisions at $\sqrt{s_{NN}}$ = 3 GeV. While it is not straightforward to
obtain the baryon density at which there is no occurrence of hadron-quark
phase transition in the Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV. Our
studies show that, to constrain the boundary of hadron-quark phase transition
of nuclear matter, one first needs to constrain the incompressibility of dense
hadronic matter near hadron-quark phase transition.

##  II The AMPT model with different modes

A multi-phase transport (AMPT) model AMPT2005  is recently extended so that it
can perform not only multi-phase transport simulations with both parton and
hadron degrees of freedom but also pure hadron cascade with hadronic mean-
field potentials cas2021 . As a Monte Carlo parton and hadron transport model,
the AMPT model consists of four components, i.e., a fluctuating initial
condition, partonic interactions, conversion from the partonic to the hadronic
matter, and hadronic interactions AMPT2005 . The model has been extensively
applied to heavy-ion collisions at RHIC and LHC energies nst2021 . In the AMPT
model, $\pi$, $\rho$, $\omega$, $\eta$, $K$, $K^{*}$, $\phi$, $N$, $\Delta$,
$N^{*}(1440)$, $N^{*}(1535)$, $\Lambda$, $\Sigma$, $\Xi$ and $\Omega$ are
included deu2009 . In the pure hadron cascade model (AMPT-HC) cas2021 , the
Woods-Saxon nucleon density distribution and local Thomas-Fermi approximation
are used to initialize the position and momentum of each nucleon in colliding
projectile and target. The parton degree of freedom is switched off. In
addition to the usual elastic and inelastic collisions, hadron potentials with
the test-particle method are applied to nucleons, baryon resonances,
strangenesses as well as their antiparticles cas2021 ; yongrcas2022 . Since
the form of the single nucleon potential at high momenta and high densities is
still less known, to make minimum assumptions, here we use the density-
dependent single nucleon mean-field potential
$U(\rho)=\alpha\frac{\rho}{\rho_{0}}+\beta(\frac{\rho}{\rho_{0}})^{\gamma}$
with $\alpha=(-29.81-46.9\frac{\kappa+44.73}{\kappa-166.32}){~{}\rm MeV}$,
$\beta=23.45\frac{\kappa+255.78}{\kappa-166.32}{~{}\rm MeV}$,
$\gamma=\frac{\kappa+44.73}{211.05}$ ($\rho_{0}$ and $\kappa$ stand for the
saturation density and incompressibility of nuclear matter, respectively) to
model the soft and stiff EoSs guo2021 . As comparisons, the AMPT-SM mode with
quark transport is also used in the present studies.

##  III Results and Discussions

Recent observation of particle directed and elliptic flows in 10-40%
centrality for Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV at RHIC is clear
evidence that predominantly hadronic matter is created in such collisions and
the QCD critical region, if created in heavy-ion collisions, could only exist
at energies higher than 3 GeV v1flow2021 ; comu2021 ; kpsi2021 , for example,
the recent STAR experimental studies on pion and proton elliptic flows show
behavior which hints at constituent quark scaling (i.e., occurrence of quark
matter) at $\sqrt{s_{NN}}$ = 4.5 GeV qgpa21 . The stiffness and the maximum
compressed baryon density of the hadronic matter created in terrestrial
laboratory have significant implications on the studies of the structure of
neutron stars and the QCD phase diagram.

Figure 1: Rapidity dependences of proton directed flow $v_{1}$ in 10-40%
centrality for Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV given by the AMPT-
HC mode with different EoSs and the quark transport AMPT-SM mode. The STAR
data is taken from Ref. v1flow2021 .

The slope of nucleon directed flow in semi-central heavy-ion collisions
reflects the stiffness of nuclear matter created by the collisions of
participant nucleons in target and projectile, thus is frequently used to
probe the EoS or the QCD phase transition guo2021 . Figure 1 shows the proton
directed flow $v_{1}$ as a function of rapidity given by the hadronic
transport model AMPT-HC with different incompressibilities in the mean-field
option. By comparing theoretical calculations with the variety of
incompressibility to the experimental data v1flow2021 , one sees that a larger
value of the incompressibility is needed to fit the data. The proton directed
flow $v_{1}$ thus indicates a large incompressibility of dense hadronic matter
created in Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV. More specifically, an
incompressibility coefficient of $k$ $\sim$ 300 MeV seems to be a good fit for
the stiffness of the created hadronic matter in Au+Au collisions at
$\sqrt{s_{NN}}$ = 3 GeV at RHIC. As comparison, the result of quark transport
on the nucleon directed flow studied here indicates predominantly hadronic
matter is created in such collisions.

Figure 2: The ratios of cumulants $C_{4}/C_{2}$ for proton in 0-5% centrality
for Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV given by the AMPT-HC mode with
different EoSs and the quark transport AMPT-SM mode. The hatched area stands
for the STAR data, taken from Ref. comu2021 .

Non-monotonic variation with $\sqrt{s_{NN}}$ of moments of the net-baryon
number distribution, related to the correlation length and the
susceptibilities of the system, is suggested as a signature for a critical
point ks2 . It is interesting to see if such observable is also affected by
the EoS of dense hadronic matter since the moments of the net-baryon number
distribution reflect the properties of bulk nuclear system. The cumulant
ratios of proton multiplicity distributions in Au+Au collisions at
$\sqrt{s_{NN}}$ = 3 GeV were reported recently comu2021 . A suppression with
respect to the Poisson baseline was observed in proton $C_{4}/C_{2}$ = -0.85
$\pm$ 0.09 $\pm$ 0.82 in the most central 0-5% centrality collisions at 3 GeV.
Since the quadratic variances is proportional to approximate $\xi^{2}$, where
$\xi$ is the correlation length, and the quartic one proportional to $\xi^{7}$
c4c2 , the ratio of cumulants $C_{4}/C_{2}\sim\xi^{5}$. Because the soft EoS
corresponds a larger correlation length $\xi$, the ratio of cumulants
$C_{4}/C_{2}$ with the soft EoS should be larger than that with the stiff one.
Figure 2 shows the ratio of cumulants $C_{4}/C_{2}$ for proton in 0-5%
centrality for Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV given by the AMPT-
HC model with different incompressibilities in the mean-field option. It is
seen that a soft EoS or a small incompressibility coefficient corresponds
higher $C_{4}/C_{2}$ ratio for proton cumulants. To fit the data, one has to
use a stiff EoS or a large incompressibility coefficient. More specifically,
the incompressibility coefficient of the created hadronic matter in Au+Au
collisions at $\sqrt{s_{NN}}$ = 3 GeV at RHIC seems to be $k>300$ MeV. Again,
the result of quark transport on the $C_{4}/C_{2}$ ratio studied here
indicates predominantly hadronic matter is created in such collisions.

Figure 3: Strange $\phi$ and $K^{-}$ productions and their ratios $\phi/K^{-}$
in 0-10% centrality for Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV given by
the AMPT-HC mode with different EoSs and the quark transport AMPT-SM mode. The
data is taken from Ref. kpsi2021 .

Due to strangeness conservation, $K^{-},\phi$ mesons, once produced, are
rarely absorbed by the surrounding matter. The lack of final state
interactions makes strange meson a penetrating probe for the EoS of dense
matter produced in heavy-ion collisions. Also the strange meson is usually
produced at maximum compression stage of nuclear collisions cas2021 . Figure 3
shows the strange $\phi$ and $K^{-}$ productions and their ratio $\phi/K^{-}$
in 0-10% centrality for Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV given by
the AMPT-HC model with different incompressibilities in the mean-field option.
It is seen that the yields of the strange $\phi$ and $K^{-}$ productions are
sensitive to the incompressibility in the mean-field option while their ratio
$\phi/K^{-}$ is not affected by the variety of the incompressibility
evidently. In heavy-ion collisions, the $K^{-},\phi$ productions are mainly
from baryon-baryon and meson-baryon or meson-meson collisions. The larger
compression of the colliding nuclei, the more number of hadronic collisions
occurs, thus more $K^{-}$ and $\phi$ are produced. Since the maximum
compression reached is sensitive to the EoS, it is not surprising to see
$K^{-}$ and $\phi$ productions are sensitive to the EoS. The soft EoS causes
nuclear matter created in heavy-ion collisions to be more compressed, thus
more mesons are produced. For the $K^{-}$’s, in the present study, we used an
attractive $K^{-}$ potential in nuclear matter kaon1 . For the $\phi$ meson,
although the in-medium corrections of the $\phi$ meson production were studied
very recently phi1 ; phi2 , considering currently there is no specific $\phi$
single particle potential in the literature as kaon’s, we did not use specific
$\phi$ potential in the present study. Comparing to the experimental data
kpsi2021 , it is found that a rather soft EoS or a fairly small
incompressibility is needed to fit the data based on the model calculations.
More specifically, the created hadronic matter in Au+Au collisions at
$\sqrt{s_{NN}}$ = 3 GeV at RHIC seems to be soft, with the incompressibility
coefficient of $75<k<150$ MeV. While the results of quark transport on the
$\phi$ and $K^{-}$ productions and their ratio $\phi/K^{-}$ studied here
evidently deviate the data, thus suggests predominantly hadronic matter is
created in such collisions.

Figure 4: Evolutions of the increment of sensitivity of incompressibility
$dS/dt$ for different observables in Au+Au collisions at $\sqrt{s_{NN}}$ = 3
GeV simulated by the AMPT-HC mode. The maximum compressed baryon densities in
a single central cell of 1 cubic Fermi with two different EoSs are also shown.

The obtained incompressibility or EoS of the hadronic matter created in Au+Au
collisions at $\sqrt{s_{NN}}$ = 3 GeV varies with different observables as
shown in Figures 1-3. This indicates complexity of the question of probing the
properties of dense hadronic matter by using different observables in heavy-
ion collisions.

To show why various EoSs or incompressibilities are obtained while using
different observables via comparisons of theoretical results and experimental
data in Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV, specific density regions
probed by different observables are demonstrated for Au+Au collisions at
$\sqrt{s_{NN}}$ = 3 GeV. Figure 4 shows sensitive density region of the
incompressibility for different observables. Here the sensitivity
$S=y(t)_{EoS1}-y(t)_{EoS2}$ with $y(t)_{EoS1}$ and $y(t)_{EoS2}$ being an
observable’s values with two different EoSs. The sensitivity increment is thus
$dS/dt$, which reflecting the situation where and when the EoS plays a major
role. It is seen that the sensitive density region of the incompressibility
for both the proton directed flow $v_{1}$ and the proton multiplicity lies in
the 3-4$\rho_{0}$ density region before maximum compression. Proton emission
is mainly decided by the strength of the gradient force of nuclear mean-field
potential which is proportional to the gradient of baryon density. It is thus
not surprising to see the sensitivity of both the proton directed flow $v_{1}$
and the proton multiplicity to the incompressibility reaches maximum before
the maximum compression of nuclear collisions.

While the strange $K^{-},\phi$ are both secondary particles that are produced
in the process of nucleons’ and (or) mesons’ multi-scatterings. So most
strange mesons are produced around maximum compression of colliding nuclei.
Although the larger gradient force of nuclear mean-field (or the larger
gradient of baryon density) may speed up nucleon motion thus produce more
mesons, these larger density gradients have minor effects compared with the
maximum density compression. Therefore it is understandable that the sensitive
density region of the incompressibility for strange mesons is around
3-5$\rho_{0}$ but still somewhat before the maximum nuclear compression as
shown in the bottom panel of Figure 4. Since proton and strange meson
observables are sensitive to various density regions of hadronic matter formed
in heavy-ion collisions, it is not surprising to see different EoSs or
incompressibilities are obtained from decoding relevant experimental data. In
general, the present studies demonstrate that hadronic matter gradually
softens as the density increases. This is exactly what we expected, since a
hadron-quark phase transition would definitely occur at a certain baryon
density point.

Another crucial implication of the present studies is that the critical
density of hadron-quark phase transition of nuclear matter depends not only on
the beam energy in heavy-ion collisions but on the incompressibility of dense
matter. This is clearly shown in Figure 4. Supposing hadron-quark phase
transition occurs in Au+Au collisions at $\sqrt{s_{NN}}$ = 3 GeV, the critical
density of hadron-quark phase transition may lie in 3-5 times saturation
density which depends on the incompressibility of dense matter near the phase
transition.

##  IV Conclusions

In summary, by comparing the experimental data of nucleon and strange meson
observables, which have been carried out at RHIC, to the theoretical
calculations of hadronic transport model in Au+Au collisions at
$\sqrt{s_{NN}}$ = 3 GeV, various incompressibilities are obtained for hadronic
matter created in heavy-ion collisions. The studies imply that hadronic matter
gradually softens as the density increases from 3 to 5 times saturation
density. Also the critical density of hadron-quark phase transition depends on
the incompressibility of dense hadronic matter.

This work is supported by the National Natural Science Foundation of China
under Grant No. 12275322 and the Strategic Priority Research Program of
Chinese Academy of Sciences with Grant No. XDB34030000.

## References

  * (1) Horst Stöcker, Walter Greiner, Phys. Rep. 137, 277 (1986).
  * (2) M. M. Aggarwal _et al_. (STAR Collaboration), arXiv: 1007.2613
  * (3) Kenji Fukushima, Tetsuo Hatsuda, Rept. Prog. Phys. 74, 014001 (2011).
  * (4) K. Adcox _et al_. (PHENIX Collaboration), Nucl. Phys. A 757, 184 (2005).
  * (5) J. Adams _et al_. (STAR Collaboration), Nucl. Phys. A 757, 102 (2005).
  * (6) Adam Bzdak, ShinIchi Esumi, Volker Koch, Jinfeng Liao, Mikhail Stephanov, Nu Xu, Phys. Rep. 853, 1 (2020).
  * (7) X. Luo, N. Xu, Nucl. Sci. Tech. 28, 112 (2017).
  * (8) Kai-Jia Sun, Wen-Hao Zhou, Lie-Wen Chen, Che Ming Ko, Feng Li, Rui Wang, Jun Xu, arXiv:2205.11010 (2022).
  * (9) Y. F. Guo, G. C. Yong, Phys. Lett. B 815, 136138 (2021).
  * (10) Y. Nara, H. Niemi, A. Ohnishi _et al_., Eur. Phys. J. A 54, 18 (2018).
  * (11) G. Odyniec (for the STAR Collaboration), PoS CORFU2018, 151 (2019).
  * (12) M. A. Stephanov, Prog. Theor. Phys. Suppl. 153, 139 (2004).
  * (13) B. Mohanty, Nucl. Phys. A 830, 899C (2009).
  * (14) A. Akmal, V. R. Pandharipande, and D. G. Ravenhall, Phys. Rev. C 58, 1804 (1998).
  * (15) E. Annala, T. Gorda, A. Kurkela, J. Nättilä, Nat. Phys. 16, 907 (2020).
  * (16) Z. Miao, A. Li, Z. Zhu, and S. Han, Astrophys. J. 904, 103 (2020).
  * (17) Min Ju, Xuhao Wu, Fan Ji, Jinniu Hu, and Hong Shen, Phys. Rev. C 103, 025809 (2021).
  * (18) Toru Kojo, Defu Hou, Jude Okafor, and Hajime Togashi, Phys. Rev. D 104, 063036 (2021).
  * (19) J. Ranjbar and M. Ghazanfari Mojarrad, Phys. Rev. C 104, 045807 (2021).
  * (20) Wen-Jie Xie, Bao-An Li, Phys. Rev. C 103, 035802 (2021).
  * (21) Andreas Bauswein, Niels-Uwe F. Bastian, David B. Blaschke, Katerina Chatziioannou, James A. Clark, Tobias Fischer, and Micaela Oertel, Phys. Rev. Lett. 122, 061102 (2019).
  * (22) Soumi De, Daniel Finstad, James M. Lattimer, Duncan A. Brown, Edo Berger, and Christopher M. Biwer, Phys. Rev. Lett. 121, 091102 (2018).
  * (23) Elias R. Most, L. Jens Papenfort, Veronica Dexheimer, Matthias Hanauske, Stefan Schramm, Horst Stöcker, and Luciano Rezzolla, Phys. Rev. Lett. 122, 061101 (2019).
  * (24) D. Boyanovsky, H. J. de Vega, D. J. Schwarz, Annu. Rev. Nucl. Part. Sci. 56, 44 (2006).
  * (25) Alessandro Lovato _et al_., arXiv:2211.02224 (2022).
  * (26) Pengcheng Li, Jan Steinheimer, Tom Reichert, Apiwit Kittiratpattana, Marcus Bleicher, Qingfeng Li, arXiv:2209.01413 (2022).
  * (27) Jan Steinheimer, Anton Motornenko, Agnieszka Sorensen, Yasushi Nara, Volker Koch, Marcus Bleicher, arXiv:2208.12091 (2022).
  * (28) Dmytro Oliinychenko, Agnieszka Sorensen, Volker Koch, Larry McLerran, arXiv:2208.11996 (2022).
  * (29) P. Danielewicz, R. Lacey, and W. G. Lynch, Science 298, 1592 (2002).
  * (30) Yasushi Nara, Asanosuke Jinno, Koichi Murase, and Akira Ohnishi, Phys. Rev. C 106, 044902 (2022).
  * (31) M. S. Abdallah _et al_. (STAR Collaboration), Phys. Lett. B 827, 137003 (2022).
  * (32) M. S. Abdallah _et al_. (STAR Collaboration), Phys. Rev. C 103, 034908 (2021).
  * (33) M. S. Abdallah _et al_. (STAR Collaboration), Phys. Rev. Lett. 128, 202303 (2022).
  * (34) M. S. Abdallah _et al_. (STAR Collaboration), Phys. Lett. B 831, 137152 (2022).
  * (35) Zi-Wei Lin, Che Ming Ko, Bao-An Li, Bin Zhang, Subrata Pal, Phys. Rev. C 72, 064901 (2005).
  * (36) Gao-Chan Yong, Zhi-Gang Xiao, Yuan Gao, Zi-Wei Lin, Phys. Lett. B 820, 136521 (2021).
  * (37) Zi-Wei Lin, Liang Zheng, Nucl. Sci. Tech. 32, 113 (2021).
  * (38) Yongseok Oh, Zi-Wei Lin, Che Ming Ko, Phys. Rev. C 80, 064902 (2009).
  * (39) Gao-Chan Yong, Bao-An Li, Zhi-Gang Xiao, and Zi-Wei Lin, Phys. Rev. C 106, 024902 (2022).
  * (40) J. Adam _et al_. (STAR Collaboration), Phys. Rev. Lett. 126, 092301 (2021).
  * (41) M. A. Stephanov, Phys. Rev. Lett. 102, 032301 (2009).
  * (42) G. Q. Li, C.-H. Lee, G. E. Brown, Phys. Rev. Lett. 79, 5214 (1997).
  * (43) Taesoo Song, Joerg Aichelin, and Elena Bratkovskaya, Phys. Rev. C 106, 024903 (2022).
  * (44) A. Mishra, S. P. Misra, Eur. Phys. J. A 57, 98 (2021).


# InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene
Understanding

Hanrong Ye and Dan Xu  Hanrong Ye and Dan Xu are with the Department of
Computer Science and Engineering, The Hong Kong University of Science and
Technology (HKUST), Hong Kong. E-mail: {hyeae<EMAIL_ADDRESS>
###### Abstract

Multi-task scene understanding aims to design models that can simultaneously
predict several scene understanding tasks with one versatile model. Previous
studies typically process multi-task features in a more local way, and thus
cannot effectively learn spatially global and cross-task interactions, which
hampers the models’ ability to fully leverage the consistency of various tasks
in multi-task learning. To tackle this problem, we propose an Inverted Pyramid
multi-task Transformer, capable of modeling cross-task interaction among
spatial features of different tasks in a global context. Specifically, we
first utilize a transformer encoder to capture task-generic features for all
tasks. And then, we design a transformer decoder to establish spatial and
cross-task interaction globally, and a novel UP-Transformer block is devised
to increase the resolutions of multi-task features gradually and establish
cross-task interaction at different scales. Furthermore, two types of Cross-
Scale Self-Attention modules, _i.e._ , Fusion Attention and Selective
Attention, are proposed to efficiently facilitate cross-task interaction
across different feature scales. An Encoder Feature Aggregation strategy is
further introduced to better model multi-scale information in the decoder.
Comprehensive experiments on several 2D/3D multi-task benchmarks clearly
demonstrate our proposal’s effectiveness, establishing significant state-of-
the-art performances. The codes are publicly available here.

###### Index Terms:

Multi-task Learning, Transformer, Scene Understanding, Dense Prediction

##  1 Introduction

The concurrent learning and reasoning of multiple interrelated tasks is the
goal of multi-task scene understanding [1]. This technique is applicable to
various AI products including robots, self-driving vehicles, and augmented
reality (AR) headsets. A majority of visual scene understanding tasks (_e.g._
, segmentation, depth estimation, and object detection) require a pixel-level
understanding of visual scenes [2]. These tasks, which involve dense
predictions, inherently possess considerable explicit and implicit
correlations at the pixel level [3]. These correlations, when fully exploited,
can significantly boost the overall performance of multi-task models.
Furthermore, the simultaneous learning and prediction of multiple tasks
inherently offer greater efficiency than the separate training and inference
of several single-task models, as various tasks can share numerous network
modules. Such observations have sparked a growing interest among researchers
to develop advanced multi-task learning methods from different perspectives,
aiming to more effectively tackle the complexities of multi-task scene
understanding.

Figure 1: Effective information exchange among tasks is particularly important
for multi-task dense scene understanding, InvPT++ proposes a pioneering
approach to construct fully-connected global modeling of different task
features along spatial and task dimensions in an efficient manner.

The prevailing approaches for multi-task scene understanding primarily depend
on Convolutional Neural Networks (CNNs), with considerable progress made by
devising multi-task optimization loss functions [4], as well as creating [3]
or exploring [5] multi-task network architectures. While these methods have
demonstrated encouraging results, they remain constrained by the intrinsic
properties of convolution kernels extensively employed in their deep learning
frameworks, _i.e._ , locality as discussed in prior works [6, 7]. These
kernels model essential spatial and task-related contexts within relatively
localized receptive fields. Attention-based techniques have been proposed to
address this limitation [8, 9, 10], but the extent of cross-task interaction
they enable is still narrow. Thus, capturing global dependencies and
relationships of all tasks, which is vital for multi-task learning in dense
scene understanding, remains an open issue.

Recently, transformer models have been introduced to capture long-range
spatial relationships in dense prediction problems [11, 12]. However, these
models primarily focus on single-task learning settings, without an
exploration of joint modeling for multiple dense prediction tasks using
transformers. Developing a unified transformer framework that globally models
both spatial and cross-task correlations is not straightforward and
challenging. Existing transformers face difficulty when adopting higher
feature resolution because of the exponentially larger computational
complexity, therefore many methods choose to drastically reduce the spatial
resolution to mitigate computational overhead [13, 14]. However, this limits
the performance of dense prediction tasks whose performances are significantly
influenced by the resolution of the feature maps generated by the model.
Simultaneous to our work, several studies have investigated multi-task
transformers [15, 16, 17]. However, their attention maps still possess limited
context and lack the capability to model spatial and cross-task interactions
globally among features of different tasks.

To address the aforementioned challenges, we have developed an innovative end-
to-end Inverted Pyramid Multi-task Transformer framework, abbreviated as
“InvPT++”. This framework effectively models long-range dependencies within
spatial and all-task contexts while concurrently learning in high resolution,
producing accurate dense prediction maps for multi-task dense scene
understanding. InvPT++ comprises three primary components. As the first part
of the model, an InvPT++ transformer encoder derives generic visual
representations from input images via spatial long-range relationship
modeling. Following this, an InvPT++ decoder, equipped with several innovative
UP-Transformer blocks, refines multi-task features at progressively increasing
resolutions. More importantly, it is capable of capturing cross-task
interaction among various tasks at different scales within a global and
spatial context. We further propose a Cross-Scale Self-Attention, which is
able to utilize the attention message from the previous transformer layer to
help model cross-task interaction at the current layer, in the designed UP-
Transformer blocks. We develop two types of Cross-Scale Self-Attention: Fusion
Attention and Selective Attention. The Fusion Attention learns a pair of
weights to fuse the attention message from the previous layer and the
attention score at the current layer. Differently, Selective Attention is
designed to further boost computational efficiency. It selects a subset of
important tokens from the original token sequence and employs only these
important tokens in self-attention calculations. The process of identifying
the important tokens is guided by the attention scores of the previous layer.
Additionally, we devise an Encoder Feature Aggregation (EFA) strategy to
augment decoder features by incorporating multi-scale features obtained from
the InvPT++ encoder. InvPT++ clearly outperforms previous state-of-the-art
methods on several commonly used benchmarks for multi-task dense scene
understanding, _i.e._ , PASCAL-Context, NYUD-v2, Cityscapes, and Cityscapes-3D
datasets. The proposed modules are carefully examined by the experimental
results on these competitive benchmarks.

Compared with our previous conference version InvPT [18], in this work, we
further introduce a powerful Selective Attention strategy into the Cross-Scale
Self-Attention module, which greatly reduces redundant computation in the
self-attention module by leveraging cross-scale attention information from the
previous layer. The Selective Attention mechanism enables InvPT++ to discover
important tokens before the computation of self-attention maps and use only
the important tokens in self-attention computation. The selection of important
tokens is effectively guided by the attention message from the previous layer.
This innovation enables InvPT++ to markedly surpass InvPT while utilizing
22.51% fewer FLOPs in the transformer decoder. We remarkably extend our
experiments to investigate the effectiveness of InvPT++ by considering two new
and more challenging 2D/3D multi-task scene understanding benchmarks, _i.e._ ,
the Cityscapes [19] and Cityscapes-3D [20] datasets. Substantial results and
analysis in qualitative and quantitative aspects are further provided in the
experiments.

The contributions of this work are three-fold:

  * •

A powerful transformer model “InvPT++”, designed for multi-task visual scene
understanding, is proposed. To the best of our knowledge, it is the first
multi-task deep framework that enables a spatially global context in the
modeling of cross-task interaction, which empowers the tasks to mutually
enhance their learning within a single model.

  * •

An UP-Transformer block is developed, which effectively facilitates the cross-
task interaction and multi-scale learning of multi-task features. By stacking
multiple UP-Transformer blocks, we construct the InvPT++ decoder that can
generate high-resolution task-specific features for different tasks.

  * •

Inside the UP-Transformer block, we propose two types of Cross-Scale Self-
Attention modules: Fusion Attention and Selective Attention. They both help
model cross-task interaction by passing attention information across different
scales. An Encoder Feature Aggregation method is further designed to help the
learning of multi-scale features in the multi-task decoder.

The effectiveness of different components of the proposed InvPT++ is
extensively validated, and InvPT++ also produces new state-of-the-art
performances on several major 2D/3D multi-task dense scene understanding
benchmarks, including PASCAL-Context, NYUD-v2, Cityscapes, and Cityscapes-3D
datasets. The rest of the paper is organized as follows: Section 2 reviews the
literature in related fields. Section 3 gives a detailed introduction to
different modules in the proposed InvPT++. Section 4 demonstrates our
experimental results to verify the effectiveness of our proposal. Section 5
concludes the paper.

##  2 Related Work

The most related works are organized into two aspects, _i.e._ , multi-task
learning for dense scene understanding, and visual transformers.

Multi-task Learning for Dense Scene Understanding Multi-task learning has two-
fold strengths compared with single-task learning: (i) Firstly, jointly
learning multiple tasks in one model is naturally more efficient than learning
tasks with different models separately, since different tasks can share some
common information captured by task-shared network modules. (ii) Secondly, as
different tasks can help each other via information complementation, they can
outperform single-task models with a proper design of multi-task models. Given
the significant commonalities shared among different dense scene understanding
tasks, the strengths of multi-task learning have inspired a substantial amount
of research in this area.

Figure 2:  Illustration of our InvPT++ framework. The process comprises three
parts: (i) The InvPT++ transformer encoder, shared among all tasks, acquires
task-agnostic visual features from the input image. (ii) A set of task-
specific preliminary decoders generate task-specif feature
$\mathbf{F}^{d}_{t}$ and preliminary predictions $\mathbf{P}_{t}$ for each
task $t$, which are subsequently concatenated on the channel dimension to form
the combined feature $\mathbf{F}^{c}_{t}$. (iii) A unique InvPT++ transformer
decoder processes $\mathbf{F}^{c}_{t}$ of all $T$ tasks as input, enables
global cross-task interaction in spatial and task dimensions, and refines task
features with incrementally increasing resolutions. In the figure, ⓒ
represents concatenation, and ”Proj” refers to the linear projection layer.

Existing deep multi-task learning methods explore two different directions,
_i.e._ , optimization-based and architecture-based. Optimization-based methods
concentrate on enhancing the multi-task optimization process, typically by
designing multi-task balancing techniques for loss functions and gradients of
different tasks [21, 22, 4], or by enforcing cross-task consistency through
contrastive learning regularization loss [23, 24]. Conversely, architecture-
based methods primarily focus on developing effective multi-task model
architectures, either throughout the entire deep model [25, 26, 27, 28, 25] or
specifically in the decoder stage [3, 8, 9, 29, 30, 18].

Significant development has been witnessed in the domain of multi-task dense
scene understanding, as evidenced by the pioneering studies [4, 2, 27, 25, 3,
31, 32]. To design better multi-task model architecture, PAD-Net [3] proposed
an effective cross-task interaction module named “Multi-Modal Distillation”
enabling cross-task information fusion. MTI-Net [30] improved PAD-Net by
incorporating a multi-scale modeling strategy and further boosted the
performance. However, these works can only model cross-task relationships in a
highly spatially local manner because of the usage of CNNs in the cross-task
interaction modules. Therefore, some researchers turn to self-attention [33]
for modeling cross-task interaction. [29], [8], and [9] developed spatially
global or local attention-based methods to learn task-specific representation
for each task, and then propagate the features across the tasks. However,
these works still model cross-task interaction in a local context which
hampers the further improvement of the multi-task models.

Several concurrent works have incorporated self-attention in multi-task
models. ATRC [10] utilizes the neural architecture search technique to
identify suitable attention types for individual task pairs. However, it falls
short in capturing a global context that encompasses all tasks. MTFormer [16]
and MQTransformer [17] develop different cross-task information fusion
modules, yet the multi-task information cannot interact with each other in a
spatially global context. Similar to MTAN [26], the task-specific modules of
MulT [15] directly query the task-shared backbone feature to acquire task-
specific features but are unable to model cross-task interactions. MultiMAE
[34] learns a versatile backbone model but requires fine-tuning on each single
task, making it essentially a series of single-task models instead of one
multi-task model. Unlike these researches on multi-task learning, we introduce
a novel transformer framework in this paper that facilitates spatially global
interaction across features of different tasks that crucially affects multi-
task model performance.

Visual Transformers The study of visual transformers is a blooming research
area. Initially designed for NLP tasks [33], transformers have been widely
used in the study of computer problems [13, 35, 36], including multi-modal
learning [37, 38], 2D [39, 13, 40, 41, 42, 43] and 3D tasks [44, 45, 46].
Regarding the architecture design of visual transformers, previous works can
be categorized into several classes: (i) Exploit useful inductive bias with
self-attention. For instance, Swin Transformer [47] proposes to compute local
attention instead of global attention based on shifted windows, to improve
computational efficiency. Focal Transformer [48] applies a multi-granularity
strategy when defining the context in self-attention motivated by the belief
that adjacent areas are more important. This helps reduce the size of the
attention map and improve computational efficiency. (ii) Marrying transformers
with CNNs [49, 14, 50]. BOTNet [49] embeds a special multi-head self-attention
module in some of the layers of ResNet. PVT [14] and CVT [50], conversely,
improve transformer architecture with CNNs to increase performance and
efficiency. (iii) Inventing special training techniques for transformers. For
example, DEIT [51] devised a special learnable token for knowledge
distillation, which makes it possible to learn from pre-trained CNNs. DRLOC
[52] utilized an auxiliary task to help the transformer learn the relative
location of tokens. DPT [11] put forward a transformer designed for dense
prediction tasks, utilizing transposed convolution layers to obtain high-
resolution feature maps. HRFormer [12] maintains multi-scale features across
different layers of the models via using local attention to reduce overhead.

Based on our survey, this study is the first to concurrently model all-task
interaction within a spatially global context at multiple scales using a
unified transformer model for multi-task scene understanding, which helps it
greatly outperform previous methods on several benchmarks.

##  3 Method

###  3.1 Framework Overview

The overall framework overview of the proposed InvPT++ is depicted in Fig. 2.
It can be divided into three parts: a task-generic InvPT++ transformer
encoder, several task-specific preliminary decoders, and an InvPT++
transformer decoder. The transformer encoder is made to be shared for all the
tasks, acquiring task-shared features from the same input image. Following
this, the preliminary decoders extract task-specific features for all tasks,
which are used to compute the preliminary predictions. The preliminary
predictions are learned via the task labels and the loss functions. For each
task, the corresponding task-specific features and preliminary predictions are
concatenated on the channel dimension, forming the input for the subsequent
InvPT++ decoder. The InvPT++ decoder is designed to model all-task interaction
in a spatially global manner and refine the task-specific features of all
tasks. Task-specific linear projection layers then calculate the final
predictions from the refined task-specific features. The following sections
will provide a detailed description of these modules.

###  3.2 InvPT++ Encoder

The InvPT++ encoder is a transformer encoder designed to extract generic task-
shared features given the image. In each transformer layer, the self-attention
module aids in learning a global feature representation by modeling long-range
spatial dependencies among different spatial locations. The encoder produces a
feature sequence that is then reshaped into a spatial feature map. The shape
of the feature map is $H_{0}\times W_{0}$, where $H_{0}$ and $W_{0}$ represent
the height and width of the feature map, respectively. Assuming we learn $T$
tasks, the task-generic feature obtained from the encoder is then passed to
$T$ preliminary decoders to generate the corresponding $T$ feature maps. In
the experiments, we explore different encoder choices, _e.g._ , ViT [13] with
global attention, and Swin Transformer [47] with window attention.

Figure 3:  Design of the InvPT++ decoder. It is composed of three stages, each
of which globally models the cross-task interaction between features of all
tasks in a spatial global context at different scales via a specially designed
transformer block. The spatial resolutions of multi-task token sequences are
gradually increased at different stages, and the attention message at each
stage is passed to the next stage to enable cross-scale cross-task
interaction. Multi-scale encoder features are utilized to help learn multi-
scale information in the InvPT++ decoder. “$\oplus$” indicates adding and
“Proj” is a linear projection layer.  Figure 4:  Structure of the designed
“UP-Transformer Block”. It learns to upsample multi-task features, and then
establishes global all-task interaction via a novel Cross-Scale Self-Attention
module. An Attention Message Passing strategy is designed to bridge attention
information between two adjacent stages, and an Encoder Feature Aggregation
mechanism helps learn multi-scale information. The block takes a feature
sequence $\mathbf{F}_{s}$, the attention score matrix from the preceding stage
$\mathbf{A}_{s-1}$, and the encoder feature $\mathbf{F}^{e}_{s}$ as inputs. It
finally outputs the refined and upsampled multi-task token sequence
$\mathbf{F}_{s+1}$ and the attention score matrix $\mathbf{A}_{s}^{\prime}$.
They are inputs of the next stage.

###  3.3 Task-Specific Preliminary Decoders

The function of the task-specific preliminary decoders is to generate task-
specific features and coarse predictions for different tasks. To accomplish
this, we construct a base unit composed of a convolutional layer with $3\times
3$ kernel, a batch normalization operation, and a ReLU, denoted as “Conv-BN-
ReLU”. The preliminary decoder of each task consists of two Conv-BN-ReLU
units. It accepts the encoder features as input and delivers the task-specific
feature for the corresponding task.

Suppose there are $T$ tasks to learn, the preliminary decoders generate task-
specific features denoted as $\mathbf{F}_{t}^{d},t\in[1,T]$. The preliminary
prediction for the $t$-th task, $\mathbf{P}_{t}$, is achieved by applying a
task-specific $1\times 1$ convolution to the corresponding task-specific
feature $\mathbf{F}_{t}^{d}$. The preliminary predictions are learned with
supervision from the ground-truth task labels via loss functions.

Next, we integrate $\mathbf{F}_{t}^{d}$ and $\mathbf{P}_{t}$ on the channel
dimension. The resulting tensor is processed by a linear projection layer to
align the channel dimension to $C_{0}$, producing the combined task-specific
feature $\mathbf{F}_{t}^{c}$. We then flatten and concatenate all combined
task-specific features $\mathbf{F}_{t}^{c},t\in[1,T]$ to form a multi-task
token sequence $\mathbf{F}^{c}$, such that
$\mathbf{F}^{c}\in\mathbb{R}^{TH_{0}W_{0}\times C_{0}}$. This multi-task token
sequence, $\mathbf{F}^{c}$, will serve as the input for the subsequent InvPT++
decoder.

###  3.4 Structure of InvPT++ Decoder

The decoder plays a crucial role in InvPT++, facilitating effective
collaboration among diverse tasks. Our design of the InvPT++ transformer
decoder building block is motivated by three key considerations:

(i) Establishing interaction among all tasks’ features within a spatially
global context is crucial for facilitating task synergy in a multi-task model,
thus significantly enhancing its performance.

(ii) Many vision transformers [13, 50, 14] tend to dramatically reduce the
resolutions of feature maps due to the prohibitively high computational cost
of long-range self-attention at high spatial resolutions. However, the feature
map resolution is crucial for the pixel-wise prediction tasks, since the fine-
granularity task label value is assigned to each pixel [53]. Therefore, we
want to refine multi-task features at higher resolutions.

(iii) Various tasks necessitate distinct perspectives on visual information.
Embracing a multi-scale representation proves immensely advantageous as it
offers a holistic understanding of a diverse range of tasks [30, 12].

Taking these motivations into account, we design a transformer decoder to
progressively enlarge the feature resolution, referred to as the “InvPT++
decoder”. The InvPT++ decoder’s architecture is depicted in Fig. 3. The
proposed InvPT++ decoder is composed of three stages, each of which features a
carefully designed transformer block to facilitate cross-task interaction at
varying spatial resolutions. In the first stage of InvPT++ decoder (_i.e._ ,
$s=0$), we establish cross-task interaction and refine multi-task features at
the encoder’s output resolution, _i.e._ , $H_{0}\times W_{0}$. During the
second and third stages (_i.e._ , $s=1,2$), we design a “UP-Transformer block”
to progressively increase the feature resolutions and enable cross-task
interaction at higher scales. A Cross-Scale Self-Attention module is devised
to boost the modeling of cross-task interaction by connecting the attention
messages from different scales, and an Encoder Feature Aggregation approach is
further developed to assist in learning multi-scale representations.

For clearer illustration in the following, in the $s$-th stage, where
$s\in[0,1,2]$, the input feature of is notated as $\mathbf{F}_{s}$, and we
have $\mathbf{F}_{s}\in\mathbb{R}^{TH_{s}W_{s}\times C_{s}}$. Here, $H_{s}$
and $W_{s}$ denote the height and width of the input feature, while $C_{s}$
refers to the number of channels. The input for the initial stage is
$\mathbf{F}^{c}$, as discussed in Sec. 3.3, leading to
$\mathbf{F}_{0}=\mathbf{F}^{c}$.

The proposed InvPT++ decoder outputs refined task features for all tasks. It
combines the features from various decoder stages to incorporate multi-scale
information. Specifically, it obtains the token sequences from the outputs at
different stages of the InvPT++ decoder and subsequently reshapes them into
spatial feature maps. It aligns the resolution of feature maps from different
stages using bilinear interpolation and employs a linear projection layer to
match channel numbers before summing the feature maps in an element-wise
manner. The merged feature map is then fed into a Conv-BN-ReLU unit to
generate the final $T$ refined task features.

###  3.5 UP-Transformer Block

The UP-Transformer block acts as a key building block in the InvPT++ decoder.
It is designed to progressively upsample the multi-task token sequence while
fostering cross-task interaction to refine features for all tasks through
spatially global long-range interactions. The structure of the UP-Transformer
block is illustrated in Fig. 4. It is used in the second and third stages
(_i.e._ , $s=1,2$) of InvPT++ decoder.

####  3.5.1 Reshape & Up: Multi-Task Feature Upsampling

To refine task features at higher resolutions, upsampling is considered in the
UP-Transformer. However, directly interpolating the 2D token sequence can
disrupt the spatial structure of features, which however is crucial for dense
scene understanding problems. As a result, we design a block to reshape and
upsample multi-task token sequences, called the Reshape & UP module, as
depicted in Fig. 5. In the Reshape & UP module, we initially divide the token
sequence of $T$ tasks $\mathbf{F}_{s}\in\mathbb{R}^{TH_{s}W_{s}\times C_{s}}$
along the first dimension evenly into $T$ feature groups using tensor slicing.
We then reshape each group’s token sequence into a spatial feature map with
dimensions $H_{s}\times W_{s}\times C_{s}$. Each group’s feature map is
upsampled using bilinear interpolation, doubling the height and width. The
upsampled feature map of each group is further processed by a Conv-BN-ReLU
unit to adjust the channel dimension. Finally, the $T$ groups of feature maps
are flattened back into token sequences and then concatenated along the first
dimension, yielding an upsampled token sequence that contains features for all
the $T$ tasks.

At the beginning of UP-Transformer, we employ the Reshape & Up module to
enhance the spatial resolution of the multi-task token sequence while reducing
the channel count by half, yielding an upsampled token sequence
$\mathbf{F}^{up}_{s}\in\mathbb{R}^{4TH_{s}W_{s}\times(C_{s}/2)}$. Next,
$\mathbf{F}_{s}^{up}$ is combined with a feature token sequence from the
transformer encoder, which will be discussed in a later section, and processed
by an LN layer[54]. The output,
$\mathbf{F}^{\prime}_{s}\in\mathbb{R}^{4TH_{s}W_{s}\times(C_{s}/2)}$, is then
fed into the subsequent self-attention layer.

####  3.5.2 Spatially Global Modeling of Cross-Task Interactions

In the self-attention block, our goal is to establish spatially global cross-
task interaction. We utilize the self-attention mechanism to model the
relationship between each pair of tokens from the features of all tasks.
However, directly computing the self-attention map based on the high-
resolution multi-task token sequence $\mathbf{F}^{\prime}_{s}$ would lead to
an excessively large memory footprint. Therefore, we downsample
$\mathbf{F}^{\prime}_{s}$ prior to inputting it into the self-attention module
in all transformer blocks in the InvPT++ decoder. Specifically,
$\mathbf{F}^{\prime}_{s}$ is initially split and reshaped into $T$ groups of
spatial maps, where each group has a shape of $\mathbb{R}^{2H_{s}\times
2W_{s}\times(C_{s}/2)}$ and corresponds to a single task. To generate the
query tensor, a $3\times 3$ convolution layer with a stride 2 is employed for
each group, denoted as $\mathrm{Conv}(\cdot)$. For generating the key and
value tensors, an average pooling operation $\mathrm{Pool}(\cdot,k_{s})$ with
a stride of $k_{s}$ ($k_{s}=2^{s+1}$ for the $s$-th stage) is utilized. This
approach significantly enhances the computational efficiency of the self-
attention module, making spatially global cross-task interactions at high
resolutions viable.

Next, the query, key, and value tensors of each group are flattened and
concatenated as query, key, and value token sequences. This process is denoted
as $\mathrm{Flatten}(\cdot)$. Assuming $\mathbf{W}_{s}^{q}$,
$\mathbf{W}_{s}^{k}$, and $\mathbf{W}_{s}^{v}$ are the weight matrices of the
QKV linear projection layers, the computation of query $\mathbf{Q}_{s}$, key
$\mathbf{K}_{s}$, and value $\mathbf{V}_{s}$ can be described as follows:

$\displaystyle\mathbf{Q}_{s}$
$\displaystyle=\mathbf{W}^{q}_{s}\times\mathrm{Flatten}\big{(}\mathrm{Conv}(\mathbf{F}^{\prime}_{s})\big{)},\,\mathbf{Q}_{s}\in\mathbb{R}^{{TH_{s}W_{s}}\times\frac{C_{s}}{2}},$
(1) $\displaystyle\mathbf{K}_{s}$
$\displaystyle=\mathbf{W}^{k}_{s}\times\mathrm{Flatten}\big{(}\mathrm{Pool}(\mathbf{F}^{\prime}_{s},k_{s})\big{)},\,\mathbf{K}_{s}\in\mathbb{R}^{\frac{4TH_{s}W_{s}}{(k_{s})^{2}}\times\frac{C_{s}}{2}},$
$\displaystyle\mathbf{V}_{s}$
$\displaystyle=\mathbf{W}^{v}_{s}\times\mathrm{Flatten}\big{(}\mathrm{Pool}(\mathbf{F}^{\prime}_{s},k_{s})\big{)},\,\mathbf{V}_{s}\in\mathbb{R}^{\frac{4TH_{s}W_{s}}{(k_{s})^{2}}\times\frac{C_{s}}{2}}.$

Subsequently, we can calculate the self-attention score matrix
$\mathbf{A}_{s}$ using $\mathbf{Q}_{s}$ and $\mathbf{K}_{s}$:

$\mathbf{A}_{s}=\frac{\mathbf{Q}_{s}\mathbf{K}^{T}_{s}}{\sqrt{C^{\prime}_{s}}},\,\mathbf{A}_{s}\in\mathbb{R}^{TH_{s}W_{s}\times\frac{4TH_{s}W_{s}}{(k_{s})^{2}}},$
(2)

here $C^{\prime}_{s}=\frac{C_{s}}{2}$ serves as a scaling factor to tackle the
magnitude explosion issue [33]. In the original transformer model, the self-
attention map is directly computed by applying a softmax activation function
to $\mathbf{A}_{s}$.

As multi-scale information is important for visual understanding tasks [30],
we improve the self-attention module by devising a Cross-Scale Self-Attention
module that integrates attention information from the preceding layer into the
self-attention module at the current layer. The subsequent section will
discuss the module in detail.

Figure 5: Illustration of Reshape & Up module. The multi-task token sequence
is initially divided along the first dimension, creating a series of task-
specific token sequences. Each token sequence is then restructured into a
spatial feature map and increased in size through bilinear interpolation.
Following this, a Conv-BN-ReLU unit is utilized to modify the channel
dimension. The final feature maps are then flattened back into token
sequences, which are subsequently concatenated to form an upsampled multi-task
token sequence.  Figure 6: Illustration of two types of Cross-Scale Self-
Attention modules: Fusion Attention and Selective Attention. (a) Fusion
Attention merges cross-scale attention messages from the preceding layer with
the attention map in the present stage. This facilitates the learning process
of self-attention modules across different layers. (b) In contrast, Selective
Attention suggests using the attention message from the earlier scale as
guidance to pinpoint important tokens in self-attention computation, which
significantly cuts down computational redundancy. It chooses the top-$k$
tokens having the highest average values in the attention message and
incorporates only these $k$ tokens in the current stage’s self-attention
calculation.

###  3.6 Cross-Scale Self-Attention: Fusion Attention and Selective Attention

As shown in Fig. 4, to facilitate cross-task interaction modeling of multi-
scale features across different scales, we propose a special self-attention
module named Cross-Scale Self-Attention in the UP-Transformer block. The
Cross-Scale Self-Attention mechanism passes the attention messages from the
previous stage to the current stage to help the learning of cross-task
interaction with cross-scale information.

To obtain an attention message from the previous stage, we first design an
Attention Message Passing mechanism. As shown in Fig. 4, the Attention Message
Passing conveys the attention score matrix from the $(s-1)$-th stage,
$\mathbf{A}_{s-1}\in\mathbb{R}^{\frac{TH_{s}W_{s}}{4}\times\frac{4TH_{s}W_{s}}{(k_{s})^{2}}}$,
to the current stage. Specifically, we first properly adjust the stride
parameters of the average pooling layers $k_{s}$ in order to keep the second
dimension of $\mathbf{A}_{s}$ consistent across various stages. The Reshape &
Up module is utilized to increase the spatial resolution of $\mathbf{A}_{s-1}$
along the first dimension, resulting in an attention message matrix
$\mathbf{M}_{s-1}\in\mathbb{R}^{{TH_{s}W_{s}}\times\frac{4TH_{s}W_{s}}{(k_{s})^{2}}}$
with dimensions that match those of $\mathbf{A}_{s}$. The attention message
matrix $\mathbf{M}_{s-1}$, which embodies the cross-task relationship from the
previous stage, will be employed to assist in modeling cross-task interaction
in the current phase through the implementation of Cross-Scale Self-Attention
as discussed further on.

We develop two different types of Cross-Scale Self-Attention mechanisms,
_i.e._ , Fusion Attention and Selective Attention, as shown in Fig. 6. These
two mechanisms vary in their utilization of the attention message from the
preceding layer. In essence, Fusion Attention employs a linear combination to
merge the attention message from the previous stage and the attention score at
current stage, whereas the Selective Attention introduces a more efficient and
effective strategy for learning cross-task interactions guided by attention
message from the previous stage. We will introduce these two variants in the
following sections.

TABLE I: Details about the shapes of query $\mathbf{Q}$, key $\mathbf{K}$, and
value $\mathbf{V}$ tensors in different stages of InvPT++ decoder.

| $s=0$ | $s=1$ | $s=2$
---|---|---|---
$\mathbf{Q}$ | $\frac{TH_{0}W_{0}}{4},C_{0}$ | $TH_{0}W_{0},\frac{C_{0}}{2}$ | $4TH_{0}W_{0},\frac{C_{0}}{4}$
$\mathbf{K}$ | $\frac{TH_{0}W_{0}}{4},C_{0}$ | $\frac{TH_{0}W_{0}}{4},\frac{C_{0}}{2}$ | $\frac{TH_{0}W_{0}}{4},\frac{C_{0}}{4}$
$\mathbf{V}$ | $\frac{TH_{0}W_{0}}{4},C_{0}$ | $\frac{TH_{0}W_{0}}{4},\frac{C_{0}}{2}$ | $\frac{TH_{0}W_{0}}{4},\frac{C_{0}}{4}$

####  3.6.1 Fusion Attention

Fusion Attention learns a pair of weights for fusing the attention message
$\mathbf{M}_{s-1}$ from the previous stage and the attention score
$\mathbf{A}_{s}$ at the current stage, as shown in Fig. 6. Specifically, after
we obtain the attention message matrix $\mathbf{M}_{s-1}$, Fusion Attention
learns two weights $\alpha^{1}_{s}$ and $\alpha^{2}_{s}$ to combine
$\mathbf{A}_{s}$ and $\mathbf{M}_{s-1}$, and get the combined attention map
$\mathbf{A}_{s}^{m}$ after applying a Softmax function:

$\displaystyle\mathbf{A}^{\prime}_{s}$
$\displaystyle=\alpha^{1}_{s}\mathbf{A}_{s}+\alpha^{2}_{s}\mathbf{M}_{s-1},$
(3) $\displaystyle\mathbf{A}_{s}^{m}$
$\displaystyle=\mathrm{Softmax}(\mathbf{A}^{\prime}_{s}),$

here the $\mathrm{Softmax(\cdot)}$ function applies a row-wise softmax
operation to the composite attention score matrix.

The UP-Transformer block outputs the updated multi-task token sequence
$\mathbf{F}_{s+1}$, which is derived by multiplying $\mathbf{A}^{m}_{s}$ with
the value tensor $\mathbf{V}_{s}$. Subsequently, $\mathbf{F}_{s+1}$ undergoes
upsampling and is added by the skip connection originating from
$\mathbf{F}^{\prime}_{s}$.

####  3.6.2 Selective Attention

Selective Attention takes a different approach from Fusion Attention and
achieves much higher computational efficiency. Its motivation is that, since
the attention message matrix $\mathbf{M}_{s-1}$ encompasses the token-level
cross-task relationships acquired from the preceding layer, we can leverage
this information to find the important tokens and performs self-attention
exclusively among them. This approach reduces computational costs by focusing
only on the essential tokens. Consequently, Selective Attention reduces
redundant token relationships in the attention map, enhancing the
effectiveness and efficiency of cross-task interaction.

As illustrated in Fig. 6, we first carry out a process named “important tokens
discovery” to pinpoint the important tokens. Specifically, suppose the
attention message matrix $\mathbf{M}_{s-1}$ has a dimension of $N_{q}\times
N_{k}$, we first compute the average value of $\mathbf{M}_{s-1}$ along the
first dimension and get $N_{k}$ average values, we denote this process as
“$\mathrm{Avg}(\cdot)$”. These average values represent the significance
levels of the corresponding key tokens in the self-attention of the preceding
scale. Subsequently, the top-$k$ key tokens are selected as important tokens
based highest average values, and their corresponding indexes are recorded as
$\mathbf{I}_{s}\in\mathbb{R}^{k}$. We denote this operation as
“$\mathrm{top}(\textperiodcentered,k)$”, leading to the following:

$\displaystyle\mathbf{I}_{s}=\mathrm{top}(\mathrm{Avg}(\mathbf{M}_{s-1}),k).$
(4)

Figure 7: Qualitative comparison between our predictions and the ground-truth
labels (GT) for joint 2D-3D multi-task scene understanding using the
Cityscapes-3D dataset[20].

We then select the important key tokens of current stage
$\textbf{K}_{s}^{imp}$ from $\textbf{K}_{s}$ based on the indexes
$\mathbf{I}_{s}$. This operation is notated as
“$\mathrm{Select}(\textbf{K}_{s},\mathbf{I}_{s})$”. Subsequently, we compute
the attention map exclusively for the $k$ important key tokens
$\textbf{K}_{s}^{imp}$, which prevents redundant computation of all tokens,
thereby enhancing efficiency and effectiveness. The computation of attention
map $\mathbf{A}_{s}^{m}$ is now formulated as:

$\displaystyle\textbf{K}_{s}^{imp}$
$\displaystyle=\mathrm{Select}(\textbf{K}_{s},\mathbf{I}_{s}),$ (5)
$\displaystyle\mathbf{A}^{\prime}_{s}$
$\displaystyle=\frac{\mathbf{Q}_{s}{\mathbf{K}^{imp}_{s}}^{T}}{\sqrt{C^{\prime}_{s}}},$
$\displaystyle\mathbf{A}_{s}^{m}$
$\displaystyle=\mathrm{Softmax}(\mathbf{A}^{\prime}_{s}).$

In InvPT++, we favor Selective Attention as the standard Cross-Scale Self-
Attention method, due to its superior efficiency and effectiveness. These
merits are further substantiated in the experimental section.

####  3.6.3 Encoder Feature Aggregation

Different scene understanding tasks depend on distinct levels of visual
information. Some tasks, such as object boundary detection, necessitate a
basic semantic comprehension of images, while others, such as semantic
segmentation and human parsing, demand higher-level visual representation.
Consequently, learning features at different scales plays a crucial role in
providing comprehensive information for multi-task prediction. With this
motivation, we introduce an effective Encoder Feature Aggregation approach
that integrates multi-scale features from the encoder into corresponding
decoder stages. As depicted in Fig. 4, within each UP-Transformer block, we
obtain the corresponding encoder token sequence with the same spatial
resolution, denoted as $\mathbf{F}^{e}_{s}$. Assuming the channel count of
$\mathbf{F}^{e}_{s}$ is $C_{s}^{e}$, we reshape it into a spatial feature map
and employ a $3\times 3$ convolution layer to modify the channel count to
$C_{s}$. Subsequently, we flatten the result back into a token sequence and
expand it by $T$ times to match the dimensions of $\mathbf{F}_{s}^{up}$ before
adding to $\mathbf{F}_{s}^{up}$. In doing so, the InvPT++ decoder receives
multi-scale features from the encoder, which ultimately enhances the overall
performance.

Figure 8: Qualitative analysis of the preliminary decoder’s predictions and
the final predictions of the InvPT++ decoder on PASCAL-Context. The final
predictions are notably more accurate across all tasks.

##  4 Experiments

In the experimental section, our goal is to present comprehensive experimental
findings that evaluate the efficacy of our proposed approach.

###  4.1 Implementation of Experiments

####  4.1.1 Datasets

To assess our approach, we use four widely recognized benchmarks in multi-task
learning which include PASCAL-Context [55, 56], NYUD-v2 [57], Cityscapes [19],
and Cityscapes-3D [20] datasets. PASCAL-Context dataset consists of 4,998
training samples and 5,105 testing samples, featuring labels for various
essential visual understanding tasks, _e.g._ , semantic segmentation (Semseg),
human parsing (Parsing), and object boundary detection (Boundary). Earlier
work in multi-task learning generates pseudo labels for two more tasks [2]:
surface normal estimation (Normal) and saliency detection (Saliency). NYUD-v2
dataset contains 1,449 training samples and 795 testing samples, offering
labels for Semseg and Depth. Normal and Boundary labels can be derived from
these annotations. We evaluate our approach using all tasks available in these
datasets. Cityscapes dataset includes images of streets across different
cities, providing pixel labels for Semseg and Depth. This dataset consists of
2,975 training images and 500 validation images with fine annotations.
Cityscapes-3D dataset further supplies 3D bounding boxes for various vehicles
in Cityscapes images, allowing us to perform joint learning of 3D object
detection (3Ddet), Semseg, and Depth.

####  4.1.2 Evaluation

We employ the same set of evaluation metrics as utilized in previous works
[30, 18], including, the mean Intersection over Union (mIoU) for Semseg and
Parsing tasks, the Root Mean Square Error (RMSE) or the absolute error
(absErr) for the Depth task, the mean error (mErr) of angles for the Normal
task, the maximal F-measure (maxF) for the Saliency task, and the optimal-
dataset-scale F-measure (odsF) for the Boundary task. The 3Ddet task is
evaluated by mean detection score (mDS) [20]. We compute the mean relative
difference across all tasks in comparison to the single-task baseline, denoted
as MT Gain $\Delta_{m}$ [2].

(a) Visualization of learned features with t-SNE [58] of all 20 semantic
classes on PASCAL-Context and Cityscapes datasets.

(b) Some examples of learned features for the semantic segmentation task on
PASCAL-Context and Cityscapes datasets.

Figure 9: Qualitative study of the learned features from the baseline model
and our model. We find that InvPT++ generates much more discriminative
features than the baseline model.

####  4.1.3 Implementation Details

For model training, we follow the setting of InvPT [18] and use Adam optimizer
to train the models. The number of training iterations and the batch size are
40,000 and 4, respectively. We adopt the Polynomial learning rate scheduler,
using a learning rate of $2\times 10^{-5}$ and a weight decay rate of $1\times
10^{-6}$. We adopt the design of loss functions for different tasks in MTI-Net
[30] and use the same loss weights. The Swin-T transformer [47] pre-trained on
ImageNet-22K [59] is considered as the encoder for our ablation study. The
preliminary decoders have an output channel count of 768. In experiments on
Cityscapes, we employ ViT-B [13] as the encoder and train the models for
80,000 iterations with a batch size of 8. For experiments on Cityscapes-3D,
due to the higher resolution requirement of $768\times 1536$ for 3D detection,
the Swin-T is used as the encoder and it is trained with 40,000 iterations
using a batch size of 2. For 3Ddet, we employ the final prediction heads of
FCOS-3D [60] to estimate location coordinates, rotation angles, dimensions,
object classes, center-ness, and direction classification. For a comprehensive
understanding of the 3D detection prediction heads, please see the FCOS-3D
paper for more details.

Figure 10: Ablation study of different numbers of stages in the InvPT++
decoder. More stages lead to better performance on all tasks in InvPT++.

####  4.1.4 Implementation of Encoder Feature Aggregation

The design of our Encoder Feature Aggregation modules is tailored to the
distinct attributes of various encoder architectures. For encoder models with
clearly defined hierarchical structures, such as the Swin Transformer [47], we
can directly feed their encoder token sequences from different stages into the
InvPT++ decoder.

In contrast, for models that exhibit a uniform structure across all layers,
such as ViT [13], we adopt a different approach. Here, we manually choose
token sequences from different layers in a uniform distribution. These
selected token sequences are then reshaped into spatial maps and passed
through transposed convolution layers. This operation aligns the spatial
resolution of these sequences with the decoder features at the target stage.

Specifically, in the case of the ViT-B model, which comprises 12 layers in
total, we opt to use the features from the third, sixth, and ninth layers of
the encoder. For the ViT-L model, which has 24 layers, we utilize the features
from the sixth, twelfth, and eighteenth layers of the encoder. For the feature
map of the first scale, we employ a transposed convolution with a kernel size
of 2 and a stride of 2. For the feature map of the second scale, we use a
kernel size of 4 and a stride of 4.

####  4.1.5 Details about QKV in InvPT++ Decoder

As introduced in the method section, the query $\mathbf{Q}$, key $\mathbf{K}$,
and value $\mathbf{V}$ tensors have different shapes at different stages of
InvPT++ decoder. We provide more details about their shapes in Table I. The
notations are defined in Section 3.

####  4.1.6 Model Optimization

For experiments on NYUD-v2 and PASCAL-Context datasets, we take into account a
total of six dense prediction tasks, _i.e._ , Segseg, Depth, Normal, Parsing,
Saliency, and Boundary. For continuous regression tasks (_i.e._ , Depth and
Normal), an $\mathcal{L}1$ loss is employed. For discrete classification tasks
(_i.e._ , Semseg, Parsing, Saliency, and Boundary), a cross-entropy loss is
utilized. To maintain simplicity, we apply the same set of loss functions for
both intermediate and final supervision. The entire model can be optimized
end-to-end. In experiments on Cityscapes, we employ cross-entropy loss for
Semseg and $\mathcal{L}1$ loss for Depth. For 3D object detection, we utilize
a combination of loss functions as described in the FCOS-3D [60].

####  4.1.7 Data Pre-Processing

We adopt the same data pre-processing method as [10]. On PASCAL-Context,
images are padded to a size of $512\times 512$, while on NYUD-v2, input images
are randomly cropped to $448\times 576$, since Swin Transformer [47] requires
even height and width dimensions for patch merging. Common data augmentations,
such as random scaling, cropping, horizontal flipping, and color jittering,
are employed. For experiments on Cityscapes, we adhere to the data pre-
processing of UR [61]. We use the 7-class Semseg annotation and set the image
resolution to $128\times 256$. Random flip and random scale data augmentation
methods are utilized. For experiments on Cityscapes-3D, we use 19-class Semseg
annotation and set the image resolution to $768\times 1536$. Data augmentation
is not applied to Cityscapes-3D.

###  4.2 Model Analysis

####  4.2.1 Baselines and Model Variants

We establish a series of model baselines and variants to validate different
components of the proposed InvPT++, as presented in Table IV. The model
variants are illustrated as follows: (i) InvPT++ Baseline (MT) represents a
robust multi-task model. It employs Swin-T as the transformer encoder and
utilizes 2 Conv-BN-ReLU units with a $3\times 3$ kernel size as the task-
specific decoder for each task, which is the same as the preliminary decoder
used in our method. The encoder feature map is upsampled 8 times before the
decoders for finer-granularity predictions, using a multi-scale feature
aggregation strategy to further enhance performance, similar to multi-task
baselines in previous works [30, 10]. (ii) InvPT++ Baseline (ST) is a
collection of single-task models with a structure akin to InvPT++ Baseline
(MT). (iii) InvPT++ w/ UTB incorporates the UP-Transformer block into InvPT++
Baseline (MT). (iv) InvPT++ w/ UTB + FA further implements Attention Message
Passing with Fusion Attention (FA). (v) InvPT++ w/ UTB + FA + EFA introduces
Encoder Feature Aggregation.

TABLE II: Comparison of Fusion Attention (Fusion) and Selective Attention
(Selective) on PASCAL-Context. Selective Attention attains superior
performance across all tasks while requiring significantly less computational
resources.

Version | Semseg | Parsing | Saliency | Normal | Boundary | Decoder
---|---|---|---|---|---|---
mIoU $\mathbf{\uparrow}$ |  mIoU $\mathbf{\uparrow}$ |  maxF $\mathbf{\uparrow}$ |  mErr $\mathbf{\downarrow}$ |  odsF $\mathbf{\uparrow}$ | FLOPs
Fusion | 79.03 | 67.61 | 84.81 | 14.15 | 73.00 | -
Selective | 80.22 | 69.12 | 84.74 | 13.73 | 74.20 | -22.51%

TABLE III: Comparison of Fusion Attention (Fusion) and Selective Attention
(Selective) on NYUD-v2. Selective Attention achieves higher or equal
performance on all tasks compared with Fusion Attention.

Version | Semseg | Depth | Normal | Boundary
---|---|---|---|---
mIoU $\mathbf{\uparrow}$ | RMSE $\mathbf{\downarrow}$ | mErr $\mathbf{\downarrow}$ | odsF $\mathbf{\uparrow}$
Fusion | 53.56 | 0.5183 | 19.04 | 78.10
Selective | 53.85 | 0.5096 | 18.67 | 78.10

Figure 11: Qualitative analysis of the predicted results on NYUD-v2. Our
InvPT++ generates predictions with higher quality on all tasks.  Figure 12:
Visualization of the generated multi-task predictions by ATRC [10] and
InvPT++. InvPT++ produces notably improved predictions, particularly in
higher-level tasks including semantic segmentation and human parsing.  TABLE
IV: Ablation study of the proposed modules in InvPT++ decoder. Our proposals
demonstrate consistent improvement across various datasets and deliver a clear
overall enhancement for each individual task and multi-task (MT) gain
$\Delta_{m}$. A Swin-T encoder is employed in all experiments.
’$\mathbf{\downarrow}$’ signifies that lower values are better, while
’$\mathbf{\uparrow}$’ indicates higher values are better.

Model | NYUD-v2 | PASCAL-Context
---|---|---
Semseg | Depth | Normal | Boundary | Semseg | Parsing | Saliency | Normal | Boundary
mIoU $\mathbf{\uparrow}$ |  RMSE $\mathbf{\downarrow}$ |  mErr $\mathbf{\downarrow}$ |  odsF $\mathbf{\uparrow}$ |  mIoU $\mathbf{\uparrow}$ |  mIoU $\mathbf{\uparrow}$ |  maxF $\mathbf{\uparrow}$ |  mErr $\mathbf{\downarrow}$ |  odsF $\mathbf{\uparrow}$
InvPT++ Baseline (ST) | 43.29 | 0.5975 | 20.80 | 76.10 | 72.43 | 61.13 | 83.43 | 14.38 | 71.50
InvPT++ Baseline (MT) | 41.06 | 0.6350 | 21.47 | 76.00 | 70.92 | 59.63 | 82.63 | 14.63 | 71.30
InvPT++ w/ UTB | 43.18 | 0.5643 | 21.05 | 76.10 | 72.34 | 61.08 | 83.99 | 14.49 | 71.60
InvPT++ w/ UTB+FA | 43.64 | 0.5617 | 20.87 | 76.10 | 73.29 | 61.78 | 84.03 | 14.37 | 71.80
InvPT++ w/ UTB+FA+EFA | 44.27 | 0.5589 | 20.46 | 76.10 | 73.93 | 62.73 | 84.24 | 14.15 | 72.60
MT Gain $\Delta_{m}$ [2] |  +2.59 |  +1.76

TABLE V: Experiments of using different ratios of selected tokens in Selective
Attention on PASCAL-Context dataset. Selecting more tokens tends to bring
better performance.

Retention | Semseg | Parsing | Saliency | Normal | Boundary
---|---|---|---|---|---
Ratio | mIoU $\mathbf{\uparrow}$ | mIoU $\mathbf{\uparrow}$ | maxF $\mathbf{\uparrow}$ | mErr $\mathbf{\downarrow}$ | odsF $\mathbf{\uparrow}$
25% | 80.01 | 69.06 | 84.71 | 13.71 | 74.10
50% | 80.22 | 69.12 | 84.74 | 13.73 | 74.20
75% | 79.99 | 69.27 | 84.92 | 13.71 | 74.30

####  4.2.2 Ablation Study of InvPT++ Decoder

We conduct a series of experiments on PASCAL-Context and NYUD-v2 to validate
the efficacy of different modules of InvPT++, especially in the decoder, and
show the results in Table IV. Based on the experimental results, we observe
that the proposed UP-Transformer Block (UTB), which is a core contribution,
brings significant performance improvement on Semseg by 2.12, on Depth by
0.0707 and on Normal by 0.42, on the NYUD-v2 dataset. Introducing the Fusion
Attention (FA) version of Cross-Scale Self-Attention results in notable
performance improvement, which is further boosted by applying the proposed
Encoder Feature Aggregation (EFA).

To benchmark against single-task baselines, we compare our model with its
single-task equivalent, termed ”InvPT Baseline (ST)”, on both datasets as
displayed in Table IV. Our model exhibits a substantial performance elevation
compared with the single-task variant, achieving a multi-task gain of 2.59% on
NYUD-v2, and a multi-task gain of 1.76% on PASCAL-Context.

To assess the effectiveness and efficiency of the devised Selective Attention
and Fusion Attention in the Cross-Scale Self-Attention, we implement them
using a larger ViT-L encoder variant of InvPT++ and present the results in
Table II (PASCAL-Context) and Table III (NYUD-v2). By default, we keep the top
50% of key tokens in Selective Attention. Selective Attention achieves better
performance on all tasks while utilizing much fewer FLOPs. As a result, we
choose Selective Attention as the default option for InvPT++.

Additionally, we investigate the impact of varying the proportion of selected
key tokens in Selective Attention on performance. We perform experiments
retaining 25%, 50%, and 75% of key tokens and present the results in Table V.
We observe that maintaining a higher percentage of tokens tends to yield
better performance, although it increases the computational cost.

TABLE VI: Ablation study of the InvPT++ encoders on PASCAL-Context.

Encoder | Semseg | Parsing | Saliency | Normal | Boundary
---|---|---|---|---|---
mIoU $\mathbf{\uparrow}$ |  mIoU $\mathbf{\uparrow}$ |  maxF $\mathbf{\uparrow}$ |  mErr $\mathbf{\downarrow}$ |  odsF $\mathbf{\uparrow}$
Swin-T | 73.93 | 63.09 | 84.25 | 14.03 | 72.60
Swin-B | 78.43 | 67.66 | 84.58 | 14.04 | 74.00
Swin-L | 79.65 | 69.14 | 84.78 | 14.09 | 74.80
Vit-B | 76.95 | 66.89 | 85.12 | 13.54 | 73.30
Vit-L | 80.22 | 69.12 | 84.74 | 13.73 | 74.20

TABLE VII: Performance comparison of using different transformer encoder
structures in InvPT++ on NYUD-v2.

Encoder | Semseg | Depth | Normal | Boundary
---|---|---|---|---
mIoU $\mathbf{\uparrow}$ | RMSE $\mathbf{\downarrow}$ | mErr $\mathbf{\downarrow}$ | odsF $\mathbf{\uparrow}$
Swin-T | 44.93 | 0.5552 | 20.46 | 76.10
Swin-B | 50.98 | 0.5009 | 19.26 | 77.10
Swin-L | 52.35 | 0.4921 | 18.99 | 77.90
Vit-B | 49.79 | 0.5318 | 18.90 | 77.10
Vit-L | 53.85 | 0.5096 | 18.67 | 78.10

TABLE VIII: Comparison with the state-of-the-art methods on NYUD-v2 (left) and
PASCAL-Context (right) datasets. Our InvPT and InvPT++ obtain significantly
superior performance than previous methods.

Model | Semseg | Depth | Normal | Boundary
---|---|---|---|---
mIoU $\mathbf{\uparrow}$ | RMSE $\mathbf{\downarrow}$ | mErr $\mathbf{\downarrow}$ | odsF $\mathbf{\uparrow}$
Cross-Stitch [27] | 36.34 | 0.6290 | 20.88 | 76.38
PAP [8] | 36.72 | 0.6178 | 20.82 | 76.42
PSD [9] | 36.69 | 0.6246 | 20.87 | 76.42
PAD-Net [3] | 36.61 | 0.6270 | 20.85 | 76.38
MTI-Net [30] | 45.97 | 0.5365 | 20.27 | 77.86
ATRC [10] | 46.33 | 0.5363 | 20.18 | 77.94
MQTransformer [17] | 49.18 | 0.5785 | 20.81 | 77.00
InvPT [18] | 53.56 | 0.5183 | 19.04 | 78.10
InvPT++ | 53.85 | 0.5096 | 18.67 | 78.10

Model | Semseg | Parsing | Saliency | Normal | Boundary
---|---|---|---|---|---
mIoU $\mathbf{\uparrow}$ | mIoU $\mathbf{\uparrow}$ | maxF $\mathbf{\uparrow}$ | mErr $\mathbf{\downarrow}$ | odsF $\mathbf{\uparrow}$
ASTMT [2] | 68.00 | 61.10 | 65.70 | 14.70 | 72.40
PAD-Net [3] | 53.60 | 59.60 | 65.80 | 15.30 | 72.50
MTI-Net [30] | 61.70 | 60.18 | 84.78 | 14.23 | 70.80
ATRC [10] | 62.69 | 59.42 | 84.70 | 14.20 | 70.96
ATRC-ASPP [10] | 63.60 | 60.23 | 83.91 | 14.30 | 70.86
ATRC-BMTAS [10] | 67.67 | 62.93 | 82.29 | 14.24 | 72.42
MQTransformer [17] | 71.25 | 60.11 | 84.05 | 14.74 | 71.80
InvPT [18] | 79.03 | 67.61 | 84.81 | 14.15 | 73.00
InvPT++ | 80.22 | 69.12 | 84.74 | 13.73 | 74.20

####  4.2.3 Ablation Study of InvPT++ Encoders

We examine the influence of InvPT++ encoders in this section. We adopt two
categories of transformer encoders, namely Swin Transformer (Swin-T, Swin-B,
and Swin-L)[47] and ViT (ViT-B and ViT-L)[13], as our InvPT++ transformer
encoder. The results for PASCAL-Context are presented in Table VI, while the
outcomes for NYUD-v2 are presented in Table VII. We notice that, within the
same model family, models with higher capacity generally achieve consistent
performance improvements on tasks such as Semseg and Parsing. However, the
enhancement is not as apparent for other lower-level tasks (_e.g._ ,
Boundary). One possible explanation for this discrepancy in performance gains
could be the task competition issue during training, as discussed in previous
studies [62, 4].

####  4.2.4 Ablation Study of Different Numbers of Stages

The UP-Transformer block in our approach generally comprises three stages. In
Fig. 10, we demonstrate the impact of varying the number of stages in the
InvPT++ decoder on the performance of different tasks in the PASCAL-Context
dataset. The model utilizes a ViT-L encoder. We observe that employing
additional stages enhances the InvPT++ decoder’s ability to generate improved
predictions across all tasks. This is due to our InvPT++ decoder can model
cross-task interaction at higher resolutions as the number of stages
increases.

####  4.2.5 Visualization of learned Features by InvPT++

We provide a visualization of the learned final features from the transformer
baseline (_i.e._ , “InvPT++ Baseline (MT)”) and our InvPT++ model in Fig.9. It
further illustrates how our proposed InvPT++ model enhances the quality of the
features. The statistics of the learned feature points are visualized using
t-SNE[63] on all 20 semantic classes from 1000 randomly selected samples per
class in the test split of the PASCAL-Context dataset and Cityscapes dataset.
It is evident that our model aids in learning more discriminative features,
consequently leading to higher quantitative results. Additionally, the
generated task-specific feature maps for semantic segmentation are also
intuitively improved.

####  4.2.6 Visualization of the Preliminary and Final Predictions

Fig. 8 presents a qualitative comparison of the preliminary and final
predictions generated by InvPT++ on PASCAL-Context. We can see that the
InvPT++ decoder effectively refines the preliminary predictions, resulting in
significantly improved outcomes across all dense prediction tasks. This
further verifies the effectiveness of the carefully-designed InvPT++ decoder

Figure 13:  Visualization of the generated multi-task predictions by PAD-Net
[3], ATRC [10], and our method on DAVIS dataset. All the models are trained
with the training split of PASCAL-Context dataset. Our method yields better
generalization performance on all tasks.  TABLE IX: State-of-the-art
comparison with previous multi-task learning methods on Cityscapes. InvPT++
significantly outperforms previous methods.

Model | Backbone | Semseg | Depth
---|---|---|---
mIoU $\mathbf{\uparrow}$ | abs Err $\mathbf{\downarrow}$
XTC [24] | SegNet | 74.23 | 0.0235
MTAN [26] | SegNet | 75.31 | 0.0119
UR [61] | SegNet | 76.42 | 0.0117
PAD-Net [3] | ViT-B | 77.93 | 0.0102
MTI-Net [30] | ViT-B | 78.70 | 0.0097
InvPT++ | ViT-B | 81.97 | 0.0091

TABLE X: Joint 2D-3D Multi-task scene understanding on Cityscapes-3D. InvPT++
achieves much higher performance than the multi-task baseline model.

Model | 3Ddet | Semseg | Disparity
---|---|---|---
mDS $\mathbf{\uparrow}$ | mIoU $\mathbf{\uparrow}$ | RMSE $\mathbf{\downarrow}$
Baseline | 27.49 | 67.94 | 5.33
InvPT++ | 28.57 | 74.50 | 4.86

###  4.3 State-of-the-art Comparison

To showcase the outstanding performance of the proposed InvPT++ model on
multi-task scene understanding, we compare it with many state-of-the-art
methods. Table VIII provides a comparison between the proposed InvPT++ method
and existing state-of-the-art techniques, including PAD-Net [3], MTI-Net [30],
ATRC [10], and the transformer-based MQTransformer [17] on both NYUD-v2 and
PASCAL-Context datasets. InvPT++ consistently surpasses other methods across
all nine task metrics from these two benchmarks, particularly excelling in
higher-level scene understanding tasks such as Semseg and Parsing. Remarkably,
on NYUD-v2, our InvPT++ surpasses MQTransformer by +4.67 (mIoU) on Semseg,
while on PASCAL-Context, InvPT++ outperforms MQTransformer by +8.97 (mIoU) and
+9.01 (mIoU) on Semseg and Parsing, respectively. We implement ATRC, MTI-Net,
and PAD-Net with the ViT-L backbone and compare them against our InvPT++ in
Table XI. We can still observe that InvPT++ consistently outperforms the
others across all tasks while consuming a reduced computational cost,
providing compelling evidence of its superior effectiveness. We further
illustrate the multi-task predictions generated by our model and the
competitive ATRC [10] in Fig.12 (for PASCAL-Context) and Fig.11 (for NYUD-v2).
The visuals clearly indicate that InvPT++ produces significantly more refined
outputs. Both qualitative and quantitative results prove the superiority of
our methods.

To further evaluate the effectiveness of InvPT++, we investigate its
performance on another popular benchmark: Cityscapes. We implement PAD-Net and
MTI-Net on Cityscapes and compare InvPT++ with several methods, including UR
[61], MTAN [26], and XTC [24]. The results are shown in Table IX. InvPT++
achieves the best performance, further demonstrating the effectiveness of our
proposal.

We also implement InvPT++ on the Cityscapes-3D benchmark, where it is required
to simultaneously learn semantic segmentation, 3D object detection, and depth
estimation. For comparison, we construct a strong baseline model that uses
Swin-T as the encoder, two Conv-BN-ReLU units as task-specific decoders, and
FCOS-3D [60] as the 3D detection head. The results are presented in Table X.
InvPT++ surpasses the baseline in all tasks. Additionally, we showcase
generated results on the Cityscapes-3D dataset in Fig. 7, which demonstrate
that InvPT++ can simultaneously generate high-quality predictions for both 2D
and 3D tasks.

####  4.3.1 Generalization Performance

We further investigate our model’s performance with respect to its ability to
adapt to different data distributions. We train our model using the training
split of PASCAL-Context dataset and subsequently generate multi-task
predictions on the DAVIS dataset. Our method is compared with PAD-Net and
ATRC, and the visualizations of the outputs are displayed in Fig. 13. Unlike
other methods, our model successfully generates satisfactory predictions even
when faced with an unseen data distribution. This outcome suggests that our
model exhibits a significantly superior generalization capability compared to
previous models.

TABLE XI: Comparison with the state-of-the-art methods implemented with ViT-L
backbone on PASCAL-Context. Our InvPT++ demonstrates superior performance
across all tasks, utilizing the same backbone but with a reduced computational
cost.

Model | FLOPs | #Param | Semseg | Parsing | Saliency | Normal | Boundary
---|---|---|---|---|---|---|---
mIoU $\mathbf{\uparrow}$ |  mIoU $\mathbf{\uparrow}$ |  maxF $\mathbf{\uparrow}$ |  mErr $\mathbf{\downarrow}$ |  odsF $\mathbf{\uparrow}$
PAD-Net [3] | 773G | 330M | 78.01 | 67.12 | 79.21 | 14.37 | 72.60
MTI-Net [30] | 774G | 851M | 78.31 | 67.40 | 84.75 | 14.67 | 73.00
ATRC [10] | 871G | 340M | 77.11 | 66.84 | 81.20 | 14.23 | 72.10
InvPT++ | 667G | 421M | 80.22 | 69.12 | 84.74 | 13.73 | 74.20

##  5 Conclusion

This study introduces the Inverted Pyramid Multi-task Transformer (InvPT++), a
potent model created for multi-task visual scene understanding. With its
unique ability to model cross-task interactions within a spatially global
context, InvPT++ can refine multi-task representations across diverse feature
scales. We have also proposed a new Cross-Scale Self-Attention mechanism,
designed to connect attention information across different scales, thereby
enhancing the modeling of cross-task interactions. To support the learning of
multi-scale features, we employ a specially designed Encoder Feature
Aggregation strategy, further enhancing the effectiveness of our InvPT++
approach. Through quantitative and qualitative experiments, we demonstrate the
efficacy of various modules within our framework. The results clearly show
that our method outperforms its predecessors and achieves strong state-of-the-
art performances on several competitive multi-task benchmarks.

## Acknowledgements

This research is supported in part by the Early Career Scheme of the Research
Grants Council (RGC) of the Hong Kong SAR under grant No. 26202321 and HKUST
Startup Fund No. R9253.

## References

  * [1] S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool, “Multi-task learning for dense prediction tasks: A survey,” _TPAMI_ , vol. 44, no. 7, pp. 3614–3633, 2021.
  * [2] K.-K. Maninis, I. Radosavovic, and I. Kokkinos, “Attentive single-tasking of multiple tasks,” in _CVPR_ , 2019.
  * [3] D. Xu, W. Ouyang, X. Wang, and N. Sebe, “Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing,” in _CVPR_ , 2018.
  * [4] A. Kendall, Y. Gal, and R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in _CVPR_ , 2018.
  * [5] Y. Gao, H. Bai, Z. Jie, J. Ma, K. Jia, and W. Liu, “Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning,” in _CVPR_ , 2020.
  * [6] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _CVPR_ , 2018.
  * [7] I. Bello, B. Zoph, Q. Le, A. Vaswani, and J. Shlens, “Attention augmented convolutional networks,” in _ICCV_ , 2019.
  * [8] Z. Zhang, Z. Cui, C. Xu, Y. Yan, N. Sebe, and J. Yang, “Pattern-affinitive propagation across depth, surface normal and semantic segmentation,” in _CVPR_ , 2019.
  * [9] L. Zhou, Z. Cui, C. Xu, Z. Zhang, C. Wang, T. Zhang, and J. Yang, “Pattern-structure diffusion for multi-task learning,” in _CVPR_ , 2020\.
  * [10] D. Bruggemann, M. Kanakis, A. Obukhov, S. Georgoulis, and L. Van Gool, “Exploring relational context for multi-task dense prediction,” in _ICCV_ , 2021.
  * [11] R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision transformers for dense prediction,” in _ICCV_ , 2021.
  * [12] Y. Yuan, R. Fu, L. Huang, W. Lin, C. Zhang, X. Chen, and J. Wang, “Hrformer: High-resolution transformer for dense prediction,” in _NeurIPS_ , 2021.
  * [13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2021.
  * [14] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _ICCV_ , 2021.
  * [15] D. Bhattacharjee, T. Zhang, S. Süsstrunk, and M. Salzmann, “Mult: an end-to-end multitask learning transformer,” in _CVPR_ , 2022.
  * [16] X. Xu, H. Zhao, V. Vineet, S.-N. Lim, and A. Torralba, “Mtformer: Multi-task learning via transformer and cross-task reasoning,” in _ECCV_ , 2022.
  * [17] Y. Xu, X. Li, H. Yuan, Y. Yang, J. Zhang, Y. Tong, L. Zhang, and D. Tao, “Multi-task learning with multi-query transformer for dense prediction,” _arXiv preprint arXiv:2205.14354_ , 2022.
  * [18] H. Ye and D. Xu, “Inverted pyramid multi-task transformer for dense scene understanding,” in _ECCV_ , 2022.
  * [19] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _CVPR_ , 2016.
  * [20] N. Gählert, N. Jourdan, M. Cordts, U. Franke, and J. Denzler, “Cityscapes 3d: Dataset and benchmark for 9 dof vehicle detection,” in _CVPR Workshop_ , 2020.
  * [21] Z. Chen, J. Ngiam, Y. Huang, T. Luong, H. Kretzschmar, Y. Chai, and D. Anguelov, “Just pick a sign: Optimizing deep multitask models with gradient sign dropout,” in _NeurIPS_ , 2020.
  * [22] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in _ICML_ , 2018.
  * [23] S. Yang, H. Ye, and D. Xu, “Contrastive multi-task dense prediction,” in _AAAI_ , 2022.
  * [24] W.-H. Li, X. Liu, and H. Bilen, “Learning multiple dense prediction tasks from partially annotated data,” in _CVPR_ , 2022.
  * [25] Y. Gao, J. Ma, M. Zhao, W. Liu, and A. L. Yuille, “Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction,” in _CVPR_ , 2019.
  * [26] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in _CVPR_ , 2019.
  * [27] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in _CVPR_ , 2016.
  * [28] S. Ruder, J. Bingel, I. Augenstein, and A. Søgaard, “Latent multi-task architecture learning,” in _AAAI_ , 2019.
  * [29] X. Zhang, L. Zhou, Y. Li, Z. Cui, J. Xie, and J. Yang, “Transfer vision patterns for multi-task pixel learning,” in _ACMMM_ , 2021.
  * [30] S. Vandenhende, S. Georgoulis, and L. Van Gool, “Mti-net: Multi-scale task interaction networks for multi-task learning,” in _ECCV_ , 2020.
  * [31] D. Kim, J. Kim, S. Cho, C. Luo, and S. Hong, “Universal few-shot learning of dense prediction tasks with visual token matching,” in _ICLR_ , 2023.
  * [32] H. Ye and D. Xu, “Taskprompter: Spatial-channel multi-task prompting for dense scene understanding,” in _ICLR_ , 2023.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NeurIPS_ , 2017.
  * [34] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir, “Multimae: Multi-modal multi-task masked autoencoders,” in _ECCV_ , 2022.
  * [35] S. Bhojanapalli, A. Chakrabarti, D. Glasner, D. Li, T. Unterthiner, and A. Veit, “Understanding robustness of transformers for image classification,” in _ICCV_ , 2021.
  * [36] Y. Bai, J. Mei, A. Yuille, and C. Xie, “Are transformers more robust than cnns?” in _NeurIPS_ , 2021.
  * [37] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” _arXiv preprint arXiv:2103.00020_ , 2021.
  * [38] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _ICML_ , 2021.
  * [39] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran, “Image transformer,” in _ICML_ , 2018.
  * [40] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _ECCV_ , 2020.
  * [41] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in _CVPR_ , 2021.
  * [42] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang, “Transformer in transformer,” in _NeurIPS_ , 2021.
  * [43] T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. Hinton, “Pix2seq: A language modeling framework for object detection,” _arXiv preprint arXiv:2109.10852_ , 2021.
  * [44] X. Pan, Z. Xia, S. Song, L. E. Li, and G. Huang, “3d object detection with pointformer,” in _CVPR_ , 2021.
  * [45] J. Mao, Y. Xue, M. Niu, H. Bai, J. Feng, X. Liang, H. Xu, and C. Xu, “Voxel transformer for 3d object detection,” in _ICCV_ , 2021.
  * [46] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun, “Point transformer,” in _ICCV_ , 2021.
  * [47] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _ICCV_ , 2021.
  * [48] J. Yang, C. Li, P. Zhang, X. Dai, B. Xiao, L. Yuan, and J. Gao, “Focal self-attention for local-global interactions in vision transformers,” in _NeurIPS_ , 2021.
  * [49] A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani, “Bottleneck transformers for visual recognition,” in _CVPR_ , 2021.
  * [50] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang, “Cvt: Introducing convolutions to vision transformers,” _arXiv preprint arXiv:2103.15808_ , 2021.
  * [51] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou, “Training data-efficient image transformers & distillation through attention,” in _ICML_ , 2021.
  * [52] Y. Liu, E. Sangineto, W. Bi, N. Sebe, B. Lepri, and M. De Nadai, “Efficient training of visual transformers with small-size datasets,” _arXiv preprint arXiv:2106.03746_ , 2021.
  * [53] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _CVPR_ , 2017.
  * [54] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [55] X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille, “Detect what you can: Detecting and representing objects using holistic models and body parts,” in _CVPR_ , 2014.
  * [56] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _IJCV_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [57] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in _ECCV_ , 2012.
  * [58] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [59] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009.
  * [60] T. Wang, X. Zhu, J. Pang, and D. Lin, “FCOS3D: Fully convolutional one-stage monocular 3d object detection,” in _ICCV Workshop_ , 2021.
  * [61] W.-H. Li, X. Liu, and H. Bilen, “Universal representations: A unified look at multiple task and domain learning,” _arXiv preprint arXiv:2204.02744_ , 2022\.
  * [62] O. Sener and V. Koltun, “Multi-task learning as multi-objective optimization,” in _NeurIPS_ , 2018.
  * [63] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _JMLR_ , vol. 9, no. 11, 2008.

|  Hanrong Ye is currently a PhD student in the Department of Computer Science
and Engineering at Hong Kong University of Science and Technology. His
research interest mainly focuses on multi-modal multi-task learning.
---|---
|  Dan Xu is an Assistant Professor in the Department of Computer Science and
Engineering at HKUST. He was a Postdoctoral Research Fellow in VGG at the
University of Oxford. He was a Ph.D. in the Department of Computer Science at
the University of Trento. He was also a research assistant of MM Lab at the
Chinese University of Hong Kong. He received the best scientific paper award
at ICPR 2016, and a Best Paper Nominee at ACM MM 2018. He served as Area
Chairs at multiple main-stream conferences including CVPR, AAAI, ACM
Multimedia, WACV, ACCV and ICPR.
---|---