# Measurement of the cluster position resolution of the Belle II Silicon
Vertex Detector

R. Leboucher<EMAIL_ADDRESS>K. Adamczyk  L. Aggarwal  H. Aihara  T.
Aziz  S. Bacher  S. Bahinipati  G. Batignani  J. Baudot  P. K. Behera  S.
Bettarini  T. Bilka  A. Bozek  F. Buchsteiner  G. Casarosa  L. Corona  T.
Czank  S. B. Das  G. Dujany  C. Finck  F. Forti  M. Friedl  A. Gabrielli  E.
Ganiev  B. Gobbo  S. Halder  K. Hara  S. Hazra  T. Higuchi  C. Irmler  A.
Ishikawa  H. B. Jeon  Y. Jin  C. Joo  M. Kaleta  A. B. Kaliyar  J. Kandra  K.
H. Kang  P. Kapusta  P. Kodyš  T. Kohriki  M. Kumar  R. Kumar  C. La Licata
K. Lalwani  S. C. Lee  J. Libby  L. Martel  L. Massaccesi  S. N. Mayekar  G.
B. Mohanty  T. Morii  K. R. Nakamura  Z. Natkaniec  Y. Onuki  W. Ostrowicz  A.
Paladino  E. Paoloni  H. Park  L. Polat  K. K. Rao  I. Ripp-Baudot  G. Rizzo
D. Sahoo  C. Schwanda  J. Serrano  J. Suzuki  S. Tanaka  H. Tanigawa  R.
Thalmeier  R. Tiwary  T. Tsuboyama  Y. Uematsu  O. Verbycka  L. Vitale  K. Wan
Z. Wang  J. Webb  J. Wiechczynski  H. Yin  L. Zani
(Belle-II SVD Collaboration)  School of Physics, University of Melbourne,
Melbourne, Victoria 3010, Australia  Institute of High Energy Physics,
Austrian Academy of Sciences, 1050 Vienna, Austria  Faculty of Mathematics and
Physics, Charles University, 121 16 Prague, Czech Republic  Aix Marseille
Universit$\acute{e}$ , CNRS/IN2P3, CPPM, 13288 Marseille, France  IPHC, UMR
7178, Universit$\acute{e}$ de Strasbourg, CNRS, 67037 Strasbourg, France
Indian Institute of Technology Bhubaneswar, Satya Nagar, India  Indian
Institute of Technology Madras, Chennai 600036, India  Malaviya National
Institute of Technology Jaipur, Jaipur 302017, India  Punjab Agricultural
University, Ludhiana 141004, India  Panjab University, Chandigarh 160014,
India  Tata Institute of Fundamental Research, Mumbai 400005, India
Dipartimento di Fisica, Università di Pisa, I-56127 Pisa, Italy  INFN Sezione
di Pisa, I-56127 Pisa, Italy  Dipartimento di Fisica, Università di Trieste,
I-34127 Trieste, Italy  INFN Sezione di Trieste, I-34127 Trieste, Italy  The
Graduate University for Advanced Studies (SOKENDAI), Hayama 240-0193, Japan
Kavli Institute for the Physics and Mathematics of the Universe (WPI),
University of Tokyo, Kashiwa 277-8583, Japan  Department of Physics,
University of Tokyo, Tokyo 113-0033, Japan  High Energy Accelerator Research
Organization (KEK), Tsukuba 305-0801, Japan  Department of Physics, Kyungpook
National University, Daegu 41566, Korea  H. Niewodniczanski Institute of
Nuclear Physics, Krakow 31-342, Poland

###### Abstract

The Silicon Vertex Detector (SVD), with its four double-sided silicon strip
sensor layers, is one of the two vertex sub-detectors of Belle II operating at
SuperKEKB collider (KEK, Japan). Since 2019 and the start of the data taking,
the SVD has demonstrated a reliable and highly efficient operation, even
running in an environment with harsh beam backgrounds that are induced by the
world’s highest instantaneous luminosity.

In order to provide the best quality track reconstruction with an efficient
pattern recognition and track fit, and to correctly propagate the uncertainty
on the hit’s position to the track parameters, it is crucial to precisely
estimate the resolution of the cluster position measurement. Several methods
for estimating the position resolution directly from the data will be
discussed.

###### keywords:

Belle II , Vertex detector , Cluster position resolution

††journal: Nuclear Instruments and Methods A

## Introduction

The Belle II experiment [1] operates at the high-energy physics intensity
frontier and searches for physics beyond the Standard Model [2] in rare $b$,
charm and tau decays. Belle II is collecting data since March 2019 at the
$e^{+}e^{-}$ asymmetric energy collider SuperKEKB [3] at Tsukuba, in Japan,
mainly at the centre of mass energy of the $\Upsilon(4S)$ resonance,
$10.58\mathrm{\,Ge\kern-1.00006ptV}$. By achieving a design instantaneous
luminosity of $6\times 10^{35}{\rm\,cm}^{-2}{\rm s}^{-1}$, it will collect a
final data set of $50\mbox{\,ab}^{-1}$. Relevant features of the Belle II
detector to attain its main physics goals are the precise and efficient track
reconstruction capabilities, including for those with low momentum, decay
vertex determination and identification of different kinds of charged
particles. The vertex detector plays a crucial role in fulfilling each of
these requirements. It is composed of two layers of DEPFET pixel sensors
(PXD), with the innermost layer at 1.4 cm from the interaction point, and 4
layers of double-sided silicon strip sensors (SVD).

##  1 The Belle II silicon strip detector

The Silicon Vertex Detector SVD [4] is composed of 172 double-sided silicon
strip detectors (DSSD) distributed in four layers of 7, 10, 12 and 16 ladders
with 2, 3, 4 and 5 sensors each, for a total material budget of 0.7% of the
radiation length per layer on average. Along the sensors, strips are arranged
in perpendicular directions on opposite sides and the signal is collected by
the APV25 chips which provide an analog readout: the u/P side measures the
$r\phi$-direction and the v/N side provides information on the $z$-coordinate
along the beam line.

|  Small |  Large |  Trap.
---|---|---|---
No. u/P readout strips |  768 |  768 |  768
No. v/N readout strips |  768 |  512 |  512
Readout pitch u/P strips (${\,\mu\rm m}$) |  50 |  75 |  50-75
Readout pitch v/N strips (${\,\mu\rm m}$) |  160 |  240 |  240
Sensor thickness (${\,\mu\rm m}$) |  320 |  320 |  300
Active Length (${\,m\rm m}$) |  122.90 |  122.90 |  122.76
Active Width (${\,m\rm m}$) |  38.55 |  57.72 |  57.59-38.42
Table 1: Geometrical details of the SVD DSSD sensors. All sensors have one
intermediate floating strip between two readout strips.

Layer 3 is equipped with “small” rectangular sensors, Layer 4-5-6 are build
with “large” rectangular sensors and a “trapezoidal” one. The geometrical
details of the different sensors are shown in Tab. 1.

The SVD plays a crucial role in reconstructing the decay vertex and low-
momentum particles, providing stand-alone tracking capabilities and
contributing to charged particle identification through the ionisation energy-
loss information. Moreover, it contributes to extrapolating the tracks towards
the PXD and defining the region of interests to reduce the PXD data size. An
excellent cluster position resolution is mandatory for SVD reconstruction, and
it is a crucial input for tracking to improve the quality of reconstructed
tracks and vertices, moreover its knowledge is necessary to correctly
propagate the uncertainty on the track extrapolation.

##  2 Cluster position resolution analysis strategy

The tracks traversing the SVD sensors activate adjacent strips that are
gathered into clusters. To each reconstructed cluster we assign (Fig. 1):

  * 1.

the cluster position $m=\frac{\sum_{i}X_{i}S_{i}}{\sum_{i}S_{i}}$ obtained as
the center of gravity of all strips position $X_{i}$ of the given cluster
weighted by the charge collected on each strip $S_{i}$;

  * 2.

an _unbiased_ track position intercept $t$ from the track finding [5] and its
error $\sigma_{t}$, where the track reconstruction is performed excluding the
cluster considered for the resolution measurement;

  * 3.

the true position $x$, only in simulation.

The cluster position resolution is extracted from the residuals $R=m-t$ of the
cluster position with respect to the unbiased track intercept position and the
effect of the track extrapolation error is subtracted.

Figure 1: Schematic view of a sensor plane with the cluster position $m$, the
_unbiased_ track position $t$ and the true position $x$ only available in
simulations, the measured residuals $R$.

We consider three distinct approaches in this paper:

  * 1.

Event by Event (EBE): consists in removing event-by-event the error on track
extrapolation $\sigma_{t}$ from the residual $R$ in quadrature.

$\sigma_{cl}^{\rm EBE}=\sqrt{\langle R^{2}-\sigma_{t}^{2}\rangle}_{trunc}.$
(1)

Here, $trunc$ refers to the truncation of $R^{2}-\sigma_{t}^{2}$ optimized on
the simulation to match the true resolution, defined as the width of the
distribution of $m-x$. The truncation is needed to eliminate the long non-
Gaussian tail of the $R^{2}-\sigma_{t}^{2}$ distribution.

  * 2.

Global: differently from the EBE method, the so-called ”global method” aims at
removing the contribution of $\sigma_{t}$ by subtracting in quadrature from
the width of the residuals the width and central value of the track error. The
resolution is finally extracted by:

$\sigma_{cl}^{\rm
GL}=\sqrt{\texttt{mad}^{2}(R)-\texttt{median}^{2}(\sigma_{t})-\texttt{mad}^{2}(\sigma_{t})}.$
(2)

The median is used as estimator of the central value of $\sigma_{t}$
distribution as is is robust against the outliers. For the same reason the
widths of $R$ and $\sigma_{t}$ distribution are estimated with the median
absolute deviation, which is defined, for variable $y$, as
$\texttt{mad}(y)=1.4826\times\texttt{median}(\left|y-\texttt{median}(y)\right|)$.

Figure 2: Schematic view of the SVD volume in the $r\phi$ direction.

  * 3.

Pair-method: an alternative strategy [6] is implemented in Belle II thanks to
the SVD’s windmill architecture, Fig. 2. The tracks are reconstructed,
accepting only those with two hits in the same layer and on consecutive
ladders and in the fiducial area. The residuals $m-t$ determined on both
overlapping ladders (internal and externals) are then subtracted to define the
double residual $\Delta R$. The double residuals are geometrically corrected
to account for the non-parallel sensors on a same layer. Then $\Delta R$ is
fitted with a Student’s t-distribution $T$ with the parameters: the number of
degrees of freedom $\nu$; the mean of the distribution $\mu$, and the variance
$\sigma^{2}$. The resolution $\sigma_{cl}^{\rm pair}$ is finally defined as
the width of $T$ computed as the sigma-68 111sigma-68 is half distance between
16th and 84th quantiles.:

$\sigma_{cl}^{\rm pair}=\texttt{sigma-68}\left(T(X,\nu,\mu,\sigma)\right).$
(3)

|  L3 |  L4 |  L5 |  L6
---|---|---|---|---
Internal |  $20^{\circ}$;$25^{\circ}$ |  $5^{\circ}$;$15^{\circ}$ |  $5^{\circ}$;$10^{\circ}$ |  $0^{\circ}$;$5^{\circ}$
External |  $-35^{\circ}$;$-30^{\circ}$ |  $-30^{\circ}$;$-20^{\circ}$ |  $-25^{\circ}$;$-20^{\circ}$ |  $-25^{\circ}$;$-15^{\circ}$
Table 2: Incident angle acceptance region for the u/P side in the pair-method.

The double residual has the advantage of being independent from the error on
the extrapolated track intercept position, which is cancelled out in the
double subtraction. The method decouples the contribution of the tracking
precision from the actual cluster position resolution and is only marginally
sensitive to the Coulomb scattering thanks to the small radial distance
between the two overlapped sensors, however the incident angular range is
limited following the u/P side as shown in Tab. 2.

##  3 SVD resolutions measurement

(a) Determined with the Event by Event method

(b) Determined with the Global method

Figure 3: Cluster position resolution (top) and resolution over strip pitch
(bottom) for each sides and layers following the track incident angle.

The cluster position resolution is measured using
$e^{+}e^{-}\rightarrow\mu^{+}\mu^{-}$ data, as a function of the track
incident angle in bins from $-40^{\circ}$ to $40^{\circ}$ with a $5^{\circ}$
step. Results are shown for the EBE method in Fig. 3(a) and for the global
method in Fig. 3(b). In both methods, the measured resolutions has the
expected shape, showing a minimum at the incident angle for which the
projection of the track along the direction perpendicular to the strips on the
detector plane corresponds to two strip pitches. Given the various sensor
pitches with one floating strip, the minimum is expected at $4^{\circ}$
($7^{\circ}$) for the u/P side and at $14^{\circ}$ ($21^{\circ}$) on the v/N
side, for layer 3 (4, 5 and 6).

In addition, the measured resolution for normal incident tracks is in fair
agreement with digital resolution, that with a floating strip is equal to
$pitch/(2\sqrt{12})$, with 11${\,\mu\rm m}$ for u/P layer 4, 5 and 6; and 25
(35) for v/N layer 3 (4, 5 and 6) as shown in Fig. 3(a) and Fig. 3(b). The
digital resolution provides a reference value only under the assumption that a
track activates only one strip. Tracks with larger incident angle activate
more strips and indeed our resolution is better than the digital one thanks to
the analog readout. Layer 3 u/P side, the sensors with the smallest pitch,
represents an exception: the measured resolution is slightly worse than the
digital one for perpendicular tracks and it degrades with larger angles. This
is effect is still under investigation.

Differently from EBE and Global methods, the pair method is sensitive to a
limited incident angle range in u/P side by construction of the overlapping
region in the $r\phi$ direction, therefore the resolutions measurement
reported in Fig. 4 is averaged on the whole accessible angular range. The
resolution measured with the pair method shown in Fig. 4 and detailed in Tab.
3 are higher than the measured resolution with the other methods, except for
the v/N outermost layers. This behaviour is not fully understood, but will be
the subject of future studies.

Figure 4: Cluster position resolution (top) and resolution over strip pitch
(bottom) determined by the pair method for each sides and layers following the
track incident angle.

## Conclusions and Outlooks

Since the start of the data taking the SVD has demonstrated a reliable and
highly efficient operation, with excellent performances confirmed by the
measurement of the cluster position resolution, summarized Tab. 3. Some
studies to investigate the small discrepancies observed on layer 3 u/P side
and to further investigate larger deviation measured on pair overlaps method
are planned.

| Digital | EBE | Global | Pair
---|---|---|---|---
Layer 3 u/P (${\,\mu\rm m}$) | 7 | 7 | 9 | 15
Layer 456 u/P (${\,\mu\rm m}$) | 11 | 10 | 11 | 16-17
Layer 3 v/N (${\,\mu\rm m}$) | 23 | 24 | 23 | 33
Layer 456 v/N (${\,\mu\rm m}$) | 35 | 32 | 35 | 29-36
Table 3: Summary of the digital and measured resolution taken at the normal
incidence for the EBE and Global methods. For the Pair method the average on
the whole accessible angular range is shown.

## Acknowledgements

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under the Marie Sklodowska-Curie grant
agreements No 644294 and 822070 and ERC grant agreement No 819127. This work
is supported by MEXT, WPI, and JSPS (Japan); ARC (Australia); BMBWF (Austria);
MSMT (Czechia); CNRS/IN2P3 (France); AIDA-2020 (Germany); DAE and DST (India);
INFN (Italy); NRF and RSRI (Korea); and MNiSW (Poland).

## References

  * [1] T. Abe, et al., Belle II Technical Design Report (2010).  arXiv:1011.0352.
  * [2] E. Kou, et al., The Belle II Physics Book, Progress of Theoretical and Experimental Physics 2019 (12) (Dec 2019).  doi:10.1093/ptep/ptz106.
  * [3] Y. Ohnishi, et al., Accelerator design at SuperKEKB, Progress of Theoretical and Experimental Physics 2013 (3), 03A011 (03 2013).  doi:10.1093/ptep/pts083.
  * [4] Y. Uematsu, et al., The Silicon Vertex Detector of the Belle II Experiment, Nuclear Instruments and Methods in Physics Research (2021).
  * [5] V. Bertacchi, et al., Track finding at Belle II, Computer Physics Communications 259 (2021) 107610.  doi:10.1016/j.cpc.2020.107610.
  * [6] CMS Tracker Collaboration, Stand-alone Cosmic Muon Reconstruction Before Installation of the CMS Silicon Strip Tracker, Journal of Instrumentation 4 (05) (2009) P05004–P05004, arXiv: 0902.1860.  doi:10.1088/1748-0221/4/05/P05004.


# Med-Query: Steerable Parsing of 9-DoF Medical Anatomies with Query Embedding

Heng Guo, Jianfeng Zhang, Ke Yan, Le Lu, , and Minfeng Xu H. Guo, J. Zhang, K.
Yan, L. Lu and M. Xu are all with DAMO Academy, Alibaba Group
(🖂gh205191@alibaba-inc.com). Corresponding author: Minfeng Xu
(🖂eric.xmf@alibaba-inc.com).

###### Abstract

Automatic parsing of human anatomies at instance-level from 3D computed
tomography (CT) scans is a prerequisite step for many clinical applications.
The presence of pathologies, broken structures or limited field-of-view (FOV)
all can make anatomy parsing algorithms vulnerable. In this work, we explore
how to exploit and conduct the prosperous detection-then-segmentation paradigm
in 3D medical data, and propose a steerable, robust, and efficient computing
framework for detection, identification, and segmentation of anatomies in CT
scans. Considering complicated shapes, sizes and orientations of anatomies,
without lose of generality, we present the nine degrees-of-freedom (9-DoF)
pose estimation solution in full 3D space using a novel single-stage, non-
hierarchical forward representation. Our whole framework is executed in a
steerable manner where any anatomy of interest can be directly retrieved to
further boost the inference efficiency. We have validated the proposed method
on three medical imaging parsing tasks of ribs, spine, and abdominal organs.
For rib parsing, CT scans have been annotated at the rib instance-level for
quantitative evaluation, similarly for spine vertebrae and abdominal organs.
Extensive experiments on 9-DoF box detection and rib instance segmentation
demonstrate the effectiveness of our framework (with the identification rate
of 97.0% and the segmentation Dice score of 90.9%) in high efficiency,
compared favorably against several strong baselines (e.g., CenterNet, FCOS,
and nnU-Net). For spine identification and segmentation, our method achieves a
new state-of-the-art result on the public CTSpine1K dataset. Last, we report
highly competitive results in multi-organ segmentation at FLARE22 competition.
Our annotations, code and models are made publicly available at: Med-Query.

###### Index Terms:

9-DoF Parameterization, Steerable Detection, Detection-then-segmentation,
Instance Query.

##  I Introduction

Figure 1: Illustration of the steerable anatomy parsing concept. (a) Input 3D
CT scan. (b) Target rib identified by query. (c) Target vertebra identified by
query. (d) Identifying pancreas and spleen simultaneously. Best viewed in
color.

Deep learning based methods have been remarkably successful in various medical
imaging tasks [1]. It may be generally believed that models can analyze almost
any anatomy with sufficient well-annotated data at hand, if not considering
computation burdens. It is also desirable to accurately segment and quantify
as many anatomies as possible in clinical practice. For instance, clinicians
may use automatically segmented ribs around the target tumor as organs-at-risk
in stereotactic body radiation therapy (SBRT) [2]. In opportunistic
osteoporosis screening using routine CT, the first lumbar vertebra is the
common anatomy of interest [3]. From the computing side, precisely parsing
ribs or vertebrae into individual instances and assigning each one with its
anatomical label are required, e.g., “5th right rib” or “1st lumbar vertebra”.
Extensive previous work have been proposed for nearly two decades [4, 5, 6, 7,
8, 9, 10, 11, 12, 13], among them complicated heuristic parsing rules were
constructed in order to obtain robust and holistically valid anatomy labeling
results. This motivates us to develop an alternative Transformer [14, 15]
based method that can detect, identify, and segment anatomies in a steerable
way to bypass the daunting procedure. By steerable we mean that we can
directly retrieve any anatomy of interest. If the 5th right rib is expected to
be analyzed, our framework should output the actual localization and pixel-
level segmentation mask of the targeted rib if it exists in the input scan;
otherwise output empty. This is illustrated intuitively in Fig. 1.

To achieve this goal, we need an effective deep feature embedding to encode
high-level anatomical information including semantic label, location, size,
shape, texture, and relation with contextual circumstances. The success of
Transformer on natural language processing and its application on object
detection in computer vision shed light on the embedding learning [14, 15].
DETR, the first Transformer-based object detector, models object detection as
a set prediction problem and uses bipartite matching to bind query predictions
with the ground-truth boxes during training [15]. In this work, we explore to
utilize some intrinsic properties in medical imaging, such as label uniqueness
(e.g., there is only one “5th right rib”), to achieve a steerable detection
scheme. An effective weighted adjacency matrix is proposed to guide the
bipartite matching process. Once converged, each query embedding will be
attached with an anatomically semantic label, and the model can run in a
steerable way. Thus, our model is named Med-Query, i.e., steerable medical
anatomy parsing with query embedding.

Detection in full 3D space is non-trivial, especially for repeated anatomical
sequential structures, such as ribs or vertebrae. For rib parsing, the first
barrier that we have to overcome is how to represent each elongated and
oblique rib using a properly parameterized bounding box. The ordinary axis-
aligned 3D bounding boxes enclosing ribs will have large portions of overlaps
(spatial collisions among nearby boxes), posing additional difficulties for
segmentation. To be effective and keep generality, we choose the fully-
parameterized 9-DoF bounding box to handle 6D pose and 3D scale/size
estimation. This strategy was studied in marginal space learning for heart
chambers detection and aortic valve detection [16, 17], and in incremental
parameter learning for ileo-cecal valve detection [18]. However, previous
methods employed a hierarchical formulation to estimate 9-DoF parameters in
decomposed steps. In contrast, we present a direct and straightforward one-
stage 9-DoF bounding box detection strategy.

9-DoF bounding box detection task poses a huge challenge for modern detectors
if they are based on heuristic anchor assignment[19, 20, 21], because of the
intractable computation expense of intersection-over-union (IoU) in 3D space.
It is hard for the widely used post-processing operation of non-maximum
suppression (NMS)[22] on 9-DoF bounding boxes to work efficiently for the same
reason. However, in our scenario, due to the uniqueness of each anatomy or
query, NMS is not essential. Our Med-Query is built upon DETR, a Transformer-
based anchor-free detector [15], to predict the box parameters directly, thus
circumvents the otherwise necessary IoU and NMS computations. We further
extend another two popular anchor-free detectors of CenterNet [23] and FCOS
[24] into the 9-DoF setting in our framework to investigate the performance
gap between Transformer and convolutional neural network (CNN) architectures.
Experimental results show that Transformer-based method empirically exhibits
advantages on anatomy parsing problems, which might be attributed to its
capability of modeling long-range dependencies and holistic spatial
information.

For a comprehensive anatomy parsing task, only detection and identification is
not sufficient and pixel-level segmentation is needed in many scenarios. As we
know, segmentation task should be addressed in high image resolution to reduce
imaging resampling artifacts [25]. However, keeping the whole 3D image in
(original) high CT resolution is not computationally efficient. Therefore,
detecting to obtain a compact 3D region of interest (ROI) then segmenting to
get the instance mask inside ROI becomes the mainstream practice in
computationally efficient solutions [26, 27, 28, 29, 16, 17], which we also
follow.

In this work, we present a unified computing framework for a variety of
anatomy parsing problems, with ribs, spine, and abdominal organs as examples.
Our method consists of three main stages: task-oriented ROI extraction,
anatomy-specific 9-DoF ROI detection, and anatomy instance segmentation. Input
3D CT scans are operated at different resolutions in different processing
stages. ROI extractor is trained and tested at an isotropic spacing of 3mm,
detector is conducted at 2mm, and the segmentation network is conducted on the
cropped sub-volumes, which have a higher image resolution achieved by
interpolating input sub-volumes to the desired size and voxel resolution. The
whole inference latency is around 3 seconds per CT scan on an NVIDIA V100 GPU,
outperforming several highly optimized methods[25, 9] in the rib parsing task.
If we query only a subset of all ribs instead, the needed computing time can
be further shortened.

In addition, among these three anatomical structures utilized in this work,
ribs relatively lack of high quality publicly available dataset. Thus this
instance parsing problem has not been as extensively studied as the other two
tasks [10]. To address this issue, we curate an elaborately annotated
instance-level rib parsing dataset, named RibInst, substantially built upon a
previously released chest-abdomen CT dataset of rib fracture detection and
classification [30]. RibInst will be made publicly available to the community
by providing a standardized and fair evaluation benchmark for future rib
instance segmentation and labeling methods.

Our main contributions are summarized as follows.

  * •

We present a Transformer-based Med-Query method for simultaneously estimating
and labeling 9-DoF ROI of anatomy in CT scans. To the best of our knowledge,
we are the first to estimate the 9-DoF representation in 3D medical imaging
using a one-stage Transformer.

  * •

A steerable object/anatomy detection model is achieved by proposing an
effective weighted adjacency matrix in the bipartite matching process between
query and ground-truth boxes. This steerable attribute enables a novel medical
image analysis paradigm that permits to directly retrieve any instance of
anatomy of interest and further boost the inference efficiency.

  * •

We propose a unified computing framework that can generalize well for a
variety of anatomy parsing problems. This framework achieves new state-of-the-
art quantitative performance results on detection, identification and
segmentation of anatomies, comparing to other strong baseline methods.

  * •

Last but not least, we have released publicly a comprehensively annotated
instance-level rib parsing dataset of 654 patients, termed RibInst, built on
top of a previous chest-abdomen CT dataset [30].

##  II Related Work

Detection, identification and segmentation. Automated parsing techniques have
been widely adopted for object/anatomy detection, identification and
segmentation in medical imaging domain [31, 32, 1]. From the specific
application perspective, previous work can be roughly categorized into task-
specific and universal methods. In rib parsing, existing methods solve rib
segmentation and labeling relying heavily on seed points detection and
centerline extraction, thus its results can be inaccurate or even fail when
seed points are missing or mis-located [4, 5, 6, 9]. Using auxiliary
representations for parsing ribs are also explored: skeletonized centerlines
[7, 8] or point cloud [10]. Similarly in spine parsing, complicated heuristic
rules or sophisticated processing pipelines [11, 12, 13, 33] are often
constructed in order to obtain robust labeling results. Recent end-to-end
segmentation neural network models demonstrate their versatility so that have
been widely adopted, such as U-Net [34, 35], V-Net [36], and so on.
Particularly, nnU-Net [25] and its variants have achieved cutting-edge
performances in various medical imaging segmentation tasks [37, 38, 39, 40].
All existing techniques do not offer steerable 3D object/anatomy detection as
query retrieval. In this work, we present a unified computing framework that
can generalize well for a variety of anatomy parsing tasks, based on the
detection-then-segmentation paradigm in 3D medical image analysis with
steerable, robust and efficiency benefits.

Detection-then-segmentation paradigm has dominated the instance segmentation
problem in photographic images. He et al.[26] extend Faster R-CNN[19] to Mask
R-CNN by adding a branch for predicting the segmentation mask of each detected
ROI. Several work emerge from Mask R-CNN, e.g., PANet [27], HTC [28] and
QueryInst [29]. There are some other work that do not rely on the explicit
detection stage [41, 42, 43]. The detection-then-segmentation paradigm
deserves to be exploited further in 3D medical imaging data where precision
and computation efficiency both matter. For rib instance segmentation,
previous techniques can not be directly applied since the ribs are orientated
obliquely and are so close to neighbouring ones that these regular axis-
aligned bounding boxes would be largely overlapped. Partially inspired by [16]
and [18], in which 9-DoF parameterized boxes are employed to localize the
target region with a hierarchical workflow to estimate the box parameters
progressively, we parametrically formulate the detection target for each rib
as a 9-DoF bounding box encompassing the rib tightly. This 9-DoF
representation can be generalized to other anatomies existing in 3D medical
imaging volumes. Direct 9-DoF box estimation on point clouds have been
explored recently [44, 45, 46], demonstrating promising results on this
complex pose estimation problem in 3D space. In this work, we tackle the
similar task by detecting the 9-DoF bounding box of anatomy in one-stage using
3D CT scans.

Anchor-free object detection techniques have been well studied [47, 48, 49,
23, 24, 50, 15, 51, 52]. YOLO[47] reports comparable detection performance
close to Faster R-CNN[19] while running faster. Several anchor-free methods
also obtain competitive accuracy with high inference efficiency. Zhou et
al.[23] present CenterNet that models an object as the center point of its
bounding box so that object detection is simplified as key point estimation
and size regression. FCOS[24] takes advantage of all points in a ground-truth
box to supplement positive samples and suppresses the low-quality detected
boxes with its “centerness” branch. Apart from CNN based methods, DETR[15]
comes up with a new representation that views object detection as a direct set
prediction problem with Transformer. It learns to produce unique predictions
via the bipartite matching loss and utilizes global information with the self-
attention mechanism [14], beneficial for the detection of large or elongated
objects. More recently, 3DETR [53] of a detection model for 3D point clouds
has proven that the Transformer-based detection paradigm can be extended to 3D
space with minor modifications and achieve convincing results with good
scalability in dealing with higher dimensional object detection problems.

##  III Approach

###  III-A Problem Definition

Anatomy parsing focuses on distinguishing each instance in a cluster of
similar anatomical structures with a unique anatomical label, e.g.,
recognizing each rib in a ribcage, splitting each vertebra in a spine, and
delineating each organ in the abdominal region in 3D CT scans. Our work
distinguishes itself from existing work on its steerable capability that is
attributed to a Transformer-based 9-DoF box detection module. We briefly
describe our overall detection framework as follows. Giving a CT scan
containing $N(1\leq N\leq C)$ targets, $C$ is the maximum number of target
anatomies in normal scan (e.g., $C$ is set as 24 for rib parsing in our work
according to [54]), the whole anatomy set can be expressed as
$x=\\{x_{i}=(c_{i},x_{i}^{p},x_{i}^{s},x_{i}^{a}),1\leq i\leq N\\}$, where
$c_{i},x_{i}^{p},x_{i}^{s},x_{i}^{a}$ stands for anatomy instance label,
center position component, scale component, and angle component of the
correspondingly parameterized bounding box, respectively. The goal of the
detection task is to find the best prediction set $\hat{x}$ that matches $x$
for each object/organ/anatomy instance. Unless specified otherwise, the term
position in the following context refers to the position of the box center.

Figure 2: An instantiation of Med-Query architecture for rib parsing, which
consists of A: a ribcage ROI extractor, B: a steerable 9-DoF parametric rib
detector, and C: a stand-alone segmentation head, for robust and efficient rib
parsing, i.e., instance segmentation and labeling. In (B) Anatomy Detector, we
use an adapted 3D version of ResNet [55] as feature extractor. The stacked
colored illustrative blocks next to the feature extractor represents the
flattened spatial features.

###  III-B 9-DoF Box Parameterization

To get a compact parameterization for anatomy localization, we compute the
9-DoF bounding box via principle component analysis (PCA)[56], based on the
annotated instance-level 3D segmentation mask. Specifically, three steps are
performed: 1) computing eigenvalues and eigenvectors of covariance matrix,
based on the voxel coordinates of each anatomy mask; 2) sorting eigenvalues to
determine the axes for the local coordinate system with the correspondent
eigenvectors; 3) formulating the 9-DoF box in representation of
$(x,y,z,w,h,d,\alpha,\beta,\gamma)$ based on the anatomy mask and the local
coordinate system. To be more specific, $(x,y,z)$ stands for the box center
and is transformed back into the world coordinate system to keep invariance
whenever any data augmentation applies; $(w,h,d)$ represents for the box scale
along $x$-axis, $y$-axis, and $z$-axis of the local coordinate system,
respectively, and is stored in the unit of millimeter; $(\alpha,\beta,\gamma)$
stands for the box’s Euler angles around $Z$-axis, $Y$-axis, and $X$-axis of
the world coordinate system, respectively. An intuitive visual illustration
can be found in Fig. 1. It is noteworthy that this universal parameterization
can be customized to be degraded or simplified, e.g., only keeping the angle
of pitch as the rotation parameter may be sufficient to define the vertebra
parameterization (Fig. 1(c)), depending on the specific anatomical
characteristics.

###  III-C Med-Query Architecture

In this section, we describe our full architecture in details. Based on the
steerable anatomy detector, we also enhance our algorithm pipeline with a
preceding task-oriented ROI extractor and a subsequent stand-alone
segmentation head. The term “Med-Query” is not limited to the detector itself
but represents the whole processing framework. For ease of illustration, a
schematic flowchart on a concrete application example of rib parsing, is
demonstrated in Fig. 2.

ROI Extractor. The scales and between-slice spacings of chest-abdomen CT scans
vary greatly. To make Med-Query concentrated in the target rib region, a task-
oriented ROI extraction which involves only ribcage area would be helpful.
Simply thresholding to get the ROI is not robust, especially for input scans
with large FOVs, e.g., from neck to abdomen. To obtain an accurate ROI
estimation when handling with input CT scans under various situations, we
train a simplified U-Net[34] model to coarsely identify the rib regions in 3D,
then the proper ROI box can be inferred by calculating the weighted average
coordinates and distribution scope of the predicted rib voxels. At inference
stage, the obtained ROI center is critical and the ROI scale can be adjusted
depending on the data augmentation strategies adopted in training. Our ROI
extractor is computationally efficient while running on an isotropic spacing
of $3mm\times 3mm\times 3mm$.

Steerable Anatomy Detector. DETR [15] handles object detection as a direct set
prediction problem through the conjunction of the bipartite matching loss and
Transformer [14] with parallel decoding of queries. Queries in DETR
demonstrate the preference on object spatial locations and sizes in a
statistical perspective. If there exists a deterministic binding between the
learned query embedding and anatomical structures, a steerable object parsing
paradigm for medical imaging is achievable. Medical imaging differs from
natural images mainly in two aspects: 1) the semantic targets inside medical
imaging scans are relatively more stable with respect to their quantities and
absolute/relative positions, even though there exist some local ambiguities;
2) the anatomical label for each instance is unique. These two intrinsic
properties constitute the cornerstones of developing a steerable anatomy
detector. The remaining obstacle is how to obtain a deterministic/steerable
binding.

We empirically find that the model can be trapped in random bindings between
queries and ground-truth boxes due to different initialization parameters. Two
simple examples are illustrated in Fig. 3(a)(b). We take this one step further
and intend to fix the binding consequence, as in Fig. 3(c). We define the
query set as $q=\\{q_{i},0\leq i\leq Q\\}$. Note that in our experiments, the
total number of queries is set as $C+1$, which is the number of output
channels for the classification branch in each query prediction, with the
channel zero as the background class.

To explicitly guide the bipartite matching process, we propose a weighted
adjacency matrix $M\in\mathbb{R}^{(Q+1)\times(Q+1)}$, which can be interpreted
as an index cost term to penalize the index mismatch between queries and
ground-truth boxes. A matrix instantiation with 10 queries is shown in Fig.
3(d), as can be seen, the greater the index gap, the higher the index cost.
Therefore, assuming a query with index $\sigma(i)$ and its prediction
$\hat{x}_{\sigma(i)}$, our matching cost on
$\left(\hat{x}_{\sigma(i)},x_{i}\right)$ can be defined as:

$\displaystyle\mathcal{C}\left(\hat{x}_{\sigma(i)},x_{i}\right)=$
$\displaystyle-\lambda_{c}\hat{p}_{\sigma(i)}\left(c_{i}\right)+\lambda_{p}\|\hat{x}_{\sigma(i)}^{p}-x_{i}^{p}\|_{1}$
(1)
$\displaystyle+\lambda_{s}\|\hat{x}_{\sigma(i)}^{s}-x_{i}^{s}\|_{1}+\lambda_{a}\|\hat{x}_{\sigma(i)}^{a}-x_{i}^{a}\|_{1}$
$\displaystyle+\lambda_{m}M[\sigma(i),c_{i}],$

where $c_{i}$ is the target class label,
$\hat{p}_{\sigma(i)}\left(c_{i}\right)$ is the probability prediction on class
$c_{i}$, and $\lambda_{c},\lambda_{p},\lambda_{s},\lambda_{a},\lambda_{m}$ are
cost coefficients for classification, position, scale, rotation, and the
preset weighted adjacency matrix, respectively. In our implementation, the
position values and scale values in the image coordinate system are normalized
by the image size $[W,H,D]$, thus they can be merged into the same branch with
a sigmoid function as its activation.

Figure 3: Three types of binding results between queries and ground-truth
boxes: (a), (b) and (c). (d) shows an instantiation of the weighted adjacency
matrix with 10 queries and 10 ground-truth classes. Queries are arranged in
rows while ground-truths are arranged in columns.

The optimal matching $\hat{\sigma}$ is searched using Hungarian algorithm
[57]. Then our loss function is defined similarly as:

$\displaystyle\mathcal{L}(\hat{x},x)=\frac{1}{N}\sum_{i=1}^{N}\left.\Big{[}-\lambda_{c}\log\hat{p}_{\hat{\sigma}(i)}\left(c_{i}\right)+\lambda_{p}\|\hat{x}_{\hat{\sigma}(i)}^{p}-x_{i}^{p}\|_{1}\right.$
(2)
$\displaystyle\phantom{=\;\;}\left.+\lambda_{s}\|\hat{x}_{\hat{\sigma}(i)}^{s}-x_{i}^{s}\|_{1}+\lambda_{a}\|\hat{x}_{\hat{\sigma}(i)}^{a}-x_{i}^{a}\|_{1}\right.\Big{]},$

where the coefficients are kept consistent with those in Eq. 1. For queries
matched to the background class, only the classification loss is accounted.
Note that we merely use L1 Loss for the geometrical 9-DoF box regression. No
IoU-related loss [58] is used in bipartite matching or loss computing, which
is different from [15] and [53]. The proposed preset matrix only takes effects
at the matching stage.

Segmentation Head. The input data for the aforementioned detector is isotropic
with the spacing of $2mm\times 2mm\times 2mm$ per voxel which is finer than
the ROI extraction step. To obtain high accuracy instance segmentation results
with affordable training expense, we adopt a stand-alone U-Net[34] to segment
each anatomy independently with a finer spatial resolution in a locally
cropped FOV. Specifically, each detected 9-DoF bounding box from our steerable
anatomy detector is slightly expanded to ensure that all the anatomical
structures of interest are included, and then used to crop a sub-volume from
the input CT volume. This sub-volume is interpolated as needed to achieve the
desired input size of the segmentation head, which may vary across different
organs. Then the segmentation head performs a binary segmentation for all sub-
volumes, with the anatomy of interest being segmented as foreground and other
tissues (including neighbouring anatomies of the same category) as background.
After this, all predicted binary masks are merged back with their
corresponding labels and spatial locations into the original CT scan
coordinate system to form the final instance segmentation output. Our detector
and segmentation head is designed to be disentangled, so there are good
flexibility and scalability to employ even more powerful segmentation networks
than [34]. Furthermore, benefiting from the steerable nature of the proposed
detector, the segmentation head can be invoked dynamically per detection box
at request, to save computational cost if necessary.

###  III-D Data Augmentation

For better model generalization and training efficiency, we employ both online
and offline data augmentation schemes.

Offline Scheme. We perform RandomCrop along $Z$-axis to imitate (largely)
varying data input FOVs in realistic clinical situations, where FOVs can
contain or cover the targeted object/anatomy region completely or only
partially. Spatial cropping may truncate some obliquely oriented anatomies,
whose 9-DoF parameterization need to be recomputed. Online computation is
time-consuming so that we conduct this operation offline.

Online Schemes. Since the anatomy position, orientation and scale vary case by
case, local spatial ambiguities exist among different CT scans, posing great
challenges for identifying each anatomy correctly and precisely. We perform 3D
RandomTranslate, RandomScale, and RandomRotate to add more input variations
and alleviate this problem. These operations can be conducted efficiently
without 9-DoF box recomputation, so they are performed online during the
training process. According to [54], 5%-8% of normal individuals may have 11
pairs of ribs. This raises an obstacle because we have no sufficient training
data exhibiting this pattern of anatomy variation. Hence, for rib parsing
task, we particularly use RandomErase to remove the bottom pair of ribs in the
training set with a certain probability ratio, to alleviate the possible over-
prediction problem. No additional data augmentation is used otherwise.

##  IV Experiments

###  IV-A Datasets

RibInst dataset is constructed from a public CT dataset that was used as the
rib fracture evaluation benchmark in RibFrac challenge[30]. This original
dataset consists of total 660 chest-abdomen CT scans with 420 for training, 80
for validation and 160 for testing. We comply with the dataset splitting rule
and annotate each rib within each CT scan with spatially ordered and unique
labels. There are 6 cases with extremely incomplete FOV in the validation set,
infeasible to identify the correct rib labels. Therefore RibInst dataset has
the remaining 654 CT scans. Note that a dataset with 490 CT scans annotated
with binary rib segmentation masks and labeled rib centerlines was released
[10]. Compared with it, RibInst is more concise and comprehensive, for
conducting rib instance segmentation evaluation without centerline extraction.

CTSpine1K provides 1,005 CT scans with instance-level vertebra annotation for
spine analysis [59, 39]. The dataset is curated from four previously released
datasets ranging from head-and-neck imaging to colonography scans. It serves
as a benchmark for spine-related image analysis tasks, such as vertebrae
segmentation and labeling. There are 610 scans for training and the remaining
are for validation and testing.

FLARE22 stands for Fast and Low-resource semi-supervised Abdominal oRgan
sEgmentation in CT, a challenge held in MICCAI 2022. The dataset is curated
from three abdomen CT datasets. The segmentation targets contain 13 organs:
liver, spleen, pancreas, right kidney, left kidney, stomach, gallbladder,
esophagus, aorta, inferior vena cava, right adrenal gland, left adrenal gland,
and duodenum. As a semi-supervised task, the training set includes 50 labeled
CT scans with pancreas disease and 2,000 unlabeled CT scans with liver,
kidney, spleen, or pancreas diseases. There are also 50 CT scans in the
validation set and 200 CT scans in the test set with various diseases. For
more information, readers are referred to the challenge
website111https://flare22.grand-challenge.org.

###  IV-B RibInst Curation Details

Iterative Annotation. First of all, we have to get well-annotated binary rib
segmentation mask for each volume even if manually annotating voxels in 3D
space is a time-consuming, labor-intensive task. We adopted an automatic tool
to improve the annotating efficiency, which has an operating pipeline of 1)
separating bones using thresholding and morphology operations, 2) removing the
spine and the sternum, and 3) distinguishing rib from other remaining bones
with a pre-trained machine learning model. Using this tool, we collected 50
coarse masks with hollow components, thus binary closing operation was
performed on each mask to include rib marrow regions. Two radiologists with
over 5 years of experience manually removed the cartilago costalis and refined
the boundaries between the ribs and spine for each scan. Then, we trained one
U-Net [34] and predicted foregrounds (rib masks) for another 50 cases among
unannotated scans using the trained model, followed by a new round of manual
refinement. After several rounds of this iterative annotation process, we
obtained high-quality binary segmentation for all ribs in this dataset.

Instance Labeling. Given the binary segmentation, we wish to efficiently set a
unique label per rib in each CT scan. An interactive annotation tool was
developed to fulfill this goal. The label for each rib within one connected
component is determined by one click and the following entered number. The
maximal number was set to 24, as 12 pairs of ribs may present at most common
scans, 5%-8% of normal individuals may have 11 pairs, while supernumerary ribs
may rarely be seen as normal variant according to [54]. The instance label for
each rib increases in the order of top to bottom and right to left in the
patient view.

Quality Control. Despite its high efficiency of our instance labeling tool, it
is hard (if not impossible) to avoid mistakes when observing and labeling in
3D space, especially for identifying elongated and oblique objects such as
ribs. Thus it is essential to perform quality control to improve and assure
the overall annotation quality. All labeled masks were projected into 2D space
via cylindrical projection [60] to obtain visually intuitive demonstrations.
Rib labels were mapped to common strings for the sake of readability and
attached to the corresponding ribs. Mislabeled ribs or bad segmented
components were easy to be distinguished in this 2D image. When mistakes were
found, we turned back to previous stages to manually repair the segmentation
masks or correct the labels.

TABLE I: Detection and identification results on the test set of RibInst. *Note that CenterNet and FCOS demonstrated here are significantly developed and modified to fit into our task and equipped with the proposed 9-DoF box representation. GFLOPs are computed under the same input size $256\times 256\times 256$. Methods | #params(M) | GFLOPs$\downarrow$ | Id.Rate(%)$\uparrow$ |  $\operatorname{P_{mean}}(\text{mm})$$\downarrow$ |  $\operatorname{S_{mean}}(\text{mm})$$\downarrow$ |  $\operatorname{A_{mean}}(^{\circ})$$\downarrow$ | Latency(s)$\downarrow$
---|---|---|---|---|---|---|---
CenterNet* [23] | 1.3 | 1897.7 | 94.9$\pm$9.4 | 3.032$\pm$0.662 | 2.333$\pm$0.659 | 2.890$\pm$0.651 | 2.575$\pm$0.136
FCOS* [24] | 19.4 | 1848.5 | 94.1$\pm$7.7 | 3.100$\pm$0.792 | 2.516$\pm$0.708 | 1.786$\pm$0.355 | 0.745$\pm$0.025
Med-Query | 9.6 | 90.7 | 97.0$\pm$4.2 | 4.702$\pm$2.107 | 2.784$\pm$0.952 | 2.185$\pm$0.414 | 0.065$\pm$0.008

###  IV-C Training Details

Our implementation uses PyTorch [61] and is partially built upon
MMDetection[62]. The detector in Med-Query is trained with AdamW optimizer
[63] in the initial setting of
$lr=4\mathrm{e}{-4},\beta_{1}=0.9,\beta_{2}=0.999,weight\\_{decay}=0.1$. The
total training process contains 1000 epochs: the first 200 epochs to linearly
warm up and the learning rate $lr$ is reduced by a factor of 10 at the 800th
epoch. It costs about 20 hours on 8 V100 GPUs with a single GPU batch size of
8. We empirically set coefficients
$\lambda_{c}=1,\lambda_{p}=10,\lambda_{s}=10,\lambda_{a}=10,\lambda_{m}=4$.
The network output of the fourth stage (C4), from an adapted 3D ResNet50 is
used as the spatial features for Transformer encoding. The Transformer
component consists of a one-layer encoder and a three-layer decoder, which is
empirically found to be sufficient in our task. We follow [53] to set a
dropout of 0.1 in encoder and 0.3 in decoder. The MultiHeadAttention block of
both have 4 heads. The length of positional encoding in one-axis is set as
128, thus the total embedding dimensions is 384 in our case. We use data
augmentations mentioned in Section III-D to generate diverse training data
offline and on the fly, the maximum translation distance from the reference
center (i.e., anatomy mask center in our experiments) is 20mm, the scaling
range is [0.9, 1.1], and the rotation range is [-15∘, 15∘]. The ROI extractor
and segmentation head are trained separately, for the sake of training
efficiency. All models in our experiments are trained from scratch.

###  IV-D Performance Metrics

Detection and Identification. To evaluate the 9-DoF box detection and
identification performance, we refer to the practice of VerSe [13], a
vertebrae labeling and segmentation benchmark. We extend the Identification
Rate (Id.Rate) computation in VerSe by considering all factors of the label
accuracy, center position deviation ($\operatorname{P_{mean}}$), scale
deviation ($\operatorname{S_{mean}}$) and angle deviation
($\operatorname{A_{mean}}$), to evaluate 9-DoF box predictions from different
methods. To be more specific: giving a CT scan containing $N$ ground-truth
anatomy boxes and the true location of the $i^{th}$ anatomy box denoted as
$x_{i}$ with its predicted box denoted as $\hat{x}_{i}$ ($M$ predictions in
total), the anatomy $i$ is correctly identified if $\hat{x}_{i}$ among
$\\{\hat{x}_{j}\forall j\in\\{1,2,\ldots,M\\}\\}$ is the closest box predicted
to $x_{i}$ and subjects to the following conditions:

$\left\\{\begin{array}[]{r}\|\hat{x}_{i}^{p}-x_{i}^{p}\|_{2}<20\operatorname{mm},\\\
(\sum_{k=1}^{3}\|\hat{x}_{i}^{s_{k}}-x_{i}^{s_{k}}\|_{1})/3<20\operatorname{mm},\\\
(\sum_{k=1}^{3}\|\hat{x}_{i}^{a_{k}}-x_{i}^{a_{k}}\|_{1})/3<10^{\circ},\end{array}\right.$
(3)

where the superscript $p$ denotes the center position component of the box,
the superscript $s_{k}$ denotes the scale component in axis $k$, and the
superscript $a_{k}$ represents the Euler angle component rotated around axis
$k$. Given a CT scan, if there are $R$ anatomies correctly identified, then
Id.Rate is defined as $\operatorname{Id.Rate}=\frac{R}{N}$. The center
position deviation is computed as
$\operatorname{P_{mean}}=(\sum_{i=1}^{R}\|\hat{x}_{i}^{p}-x_{i}^{p}\|_{2})/R$.
Similarly, the scale deviation is computed as
$\operatorname{S_{mean}}=(\sum_{i=1}^{R}\sum_{k=1}^{3}\|\hat{x}_{i}^{s_{k}}-x_{i}^{s_{k}}\|_{1})/3R$,
and the angle deviation is computed as
$\operatorname{A_{mean}}=(\sum_{i=1}^{R}\sum_{k=1}^{3}\|\hat{x}_{i}^{a_{k}}-x_{i}^{a_{k}}\|_{1})/3R$.
Note that we only compute the average deviations of the identified anatomies
where unpredicted anatomies or mislocated predictions have been reflected by
the Id.Rate index.

Segmentation. We employ the widely-used Dice similarity coefficient (DSC), 95%
Hausdorff distance (HD95) and average symmetric surface distance (ASSD) as
segmentation metrics. For missing anatomies, HD and ASSD are not defined. We
follow the practice of [13] to ignore such anatomies when computing the
averages. Those missing anatomies will be reflected on DSC and Id.Rate. We
utilize a publicly available evaluation
toolkit222https://github.com/deepmind/surface-distance to compute the surface
measures.

Figure 4: Detection visualizations show that our 9-DoF predictions enclose the
ground-truth rib masks accurately. (a) Normal results in superior-to-inferior
view. (b) A limited FOV case in posterior-to-anterior view. (c) A case with
rib adhesions. (d) Only odd-number labels are queried. Ground-truth masks are
rendered as visual reference.

###  IV-E Main Results

We report comprehensive evaluation results on RibInst from rib instance
detection and segmentation. Besides, we validate the performance generality of
our framework on CTSpine1K and FLARE22 datasets, and report the segmentation
performances compared with several strong baseline methods.

TABLE II: Segmentation results on the test set of RibInst. ${\dagger}$A centerline-based method proposed in [9]. *Adapted versions, also equipped with ROI extractor and segmentation head. “nnU-Net cas.” means coarse-to-fine cascade version and “nnU-Net” means 3D full resolution version. Methods | DSC(%)$\uparrow$ | HD95(mm)$\downarrow$ | ASSD(mm)$\downarrow$ | Latency(s)$\downarrow$
---|---|---|---|---
Centerline† | 84.6$\pm$21.2 | 13.84$\pm$32.86 | 3.437$\pm$10.52 | 18.50$\pm$3.182
CenterNet* | 89.5$\pm$8.8 | 1.826$\pm$3.031 | 0.725$\pm$1.821 | 4.793$\pm$1.259
FCOS* | 88.1$\pm$7.0 | 1.137$\pm$1.409 | 0.352$\pm$0.947 | 3.241$\pm$1.026
nnU-Net cas. | 89.3$\pm$3.5 | 2.421$\pm$4.117 | 0.497$\pm$0.899 | 183.0 $\pm$ (-)
nnU-Net | 89.7$\pm$7.8 | 4.498$\pm$5.524 | 0.900$\pm$1.658 | 252.0 $\pm$ (-)
Med-Query | 90.9$\pm$7.4 | 1.644$\pm$3.080 | 0.438$\pm$1.486 | 2.591$\pm$0.977
TABLE III: Organ-specific DSC (%) on the validation set of FLARE22.
Abbreviations: “RK”-Right Kidney, “IVC”-Inferior Vena Cava, “RAG”-Right
Adrenal Gland, “LAG”-Left Adrenal Gland, “Gall.”-Gallbladder,
“Eso.”-Esophagus, “Stom.”-Stomach, “Duode.”-Duodenum, “LK”-Left Kidney,
“ens.”-ensemble, “pre.”-pre-training, “mDSC”-mean DSC.

Methods | Liver | RK | Spleen | Pancreas | Aorta | IVC | RAG | LAG | Gall. | Eso. | Stom. | Duode. | LK | mDSC$\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
nnU-Net [25] | 97.7 | 94.1 | 95.8 | 87.2 | 96.8 | 87.8 | 83.0 | 80.1 | 76.5 | 89.2 | 89.9 | 77.1 | 91.1 | 88.2
nnU-Net ens. | 97.9 | 94.8 | 96.0 | 88.6 | 96.9 | 89.7 | 83.8 | 81.9 | 78.7 | 90.1 | 90.7 | 79.2 | 92.0 | 89.2
Swin UNETR [64] | 96.5 | 91.2 | 94.2 | 84.6 | 93.0 | 86.5 | 75.8 | 74.2 | 77.1 | 79.0 | 88.6 | 76.5 | 88.7 | 85.0
Swin UNETR pre. | 96.4 | 92.1 | 95.2 | 88.1 | 93.7 | 86.2 | 79.4 | 79.1 | 79.2 | 81.8 | 89.5 | 79.0 | 87.9 | 86.7
Med-Query | 98.0 | 94.5 | 97.2 | 89.0 | 96.6 | 90.3 | 82.4 | 80.6 | 86.1 | 87.4 | 91.5 | 78.7 | 93.7 | 89.7

Figure 5: An example with broken structures in RibInst. Missing or wrong
labels are marked using golden arrows and dashed circles, respectively.

Detection Evaluation on RibInst. From Table I, Med-Query achieves the best
identification rate of 97.0% with a moderate amount of parameters. The
capability of capturing long-range dependencies and holistic information of
Transformer appears to be suitable at solving this sequence modeling problem.
Notably, compared with pure CNN architectures, Med-Query can infer at least
10x faster in terms of latency. This may benefit from not relying on
additional upsampling layers in [23] or dense prediction heads in [24].
Meanwhile, we have some interesting findings on different characteristics of
Med-Query and traditional object detectors [23, 24] that recognize objects
individually while ignoring relations between objects[65]. In our case, the
main reason why FCOS and CenterNet could fail is that they sometimes assign
different labels to the same rib or same label to different ribs, subsequently
leading to missing ribs as shown in Fig. 5. These traditional detectors did
not explicitly consider the anatomy uniqueness and spatial
relationship/dependency between rib labels. Med-Query behaves in another way
that solves a label assignment problem rather than an unconstrained object
detection task, so that the relations between targets are explicitly
constructed and enforced through the steerable label assignment strategy and
the intrinsic self-attention mechanism in Transformer models [14]. On the
other hand, Med-Query is slightly inferior in local regression indicators as
shown in Table I. Med-Query can correctly identify more ribs with global
information, but including those challenging ribs in the calculation of
deviations may deteriorate the quantitative performances on localization
errors. It is noteworthy that all these optimized models achieved small
regression deviations w.r.t. $\operatorname{P_{mean}}$,
$\operatorname{S_{mean}}$, and $\operatorname{A_{mean}}$. Note that we have
improved FCOS and CenterNet with our one-stage 9-DoF detection strategy. The
above results demonstrate that the challenging 9-DoF parameter estimation
problem can be solved effectively. Some qualitative results of Med-Query in
rib detection and labeling are shown in Fig. 4. Our 3D detection visualization
tool is developed based on vedo [66].

Segmentation Evaluation on RibInst. As shown in Table II, the widely-used
self-configuring nnU-Net [25] achieves a good DSC of 89.7% with a long latency
of 252 seconds. Its coarse-to-fine cascade version also achieves a competitive
DSC of 89.3%. nnU-Net is trained with sampled patches, therefore the lack of
global information leads to noticeable label confusion, significantly revealed
by HD95 of 4.498 mm. We also reimplement a centerline-based method [9] to
investigate how the parsing results relies heavily on centerline extraction
and heuristic rules. Among all listed methods in Table II, Med-Query achieves
the best DSC of 90.9%. Notably, the whole pipeline of Med-Query can be
finished in 2.591 seconds. The efficiency is attributed to our detection-then-
segmentation paradigm which detects on low resolution with global FOV and
segments on high resolution with local FOV. A qualitative comparison on a
representative patient case with rib fractures is shown in Fig. 5,
demonstrating the robustness of Med-Query.

Segmentation Evaluation on CTSpine1K. To be aligned with the evaluation
protocol in [39], we report segmentation metrics on vertebra-level average
scores computed on all scans in the validation and test sets (HD is employed
instead of HD95). As in Table V, we compare Med-Query with several strong
baselines, including nnFormer [67], nnU-Net [25], and CPTM [39]. Med-Query
obtains marked performance improvements on both DSC and HD, achieving a new
state-of-the-art result for this challenging task of CTSpine1K. Note that the
semi-supervised annotation curation procedure of this dataset used nnU-Net in-
the-loop.

TABLE IV: Segmentation results on the validation set and test set of CTSpine1K. The results of nnFormer, nnU-Net, and CPTM are quoted from [39]. Methods | DSC(%)$\uparrow$ | HD(mm)$\downarrow$ | ASSD(mm)$\downarrow$
---|---|---|---
nnFormer [67] | 74.3 | 11.56 | -
nnU-Net [25] | 84.2 | 9.02 | -
CPTM [39] | 84.5 | 9.03 | -
Med-Query | 85.0 | 8.68 | 1.16

TABLE V: Id.Rate evaluations in different operations and different measurement thresholding combinations. RandomTranslate, RandomScale, and RandomRotate are performed jointly, thus are abbreviated as “T.S.R”. “Re.Dist.” represents relative distance constraints. “PE” is short for positional encoding. The values in the top row of the right column represent the different maximum tolerances used for translation, scale and rotation in Eq. 3. No Aug. | Data Aug. | Re.Dist. | w/o PE | 20mm20mm | 10mm10mm | 10mm10mm
---|---|---|---|---|---|---
T.S.R | Crop | Erase | 10∘ | 10∘ | 5∘
✓ |  |  |  |  |  | 83.1$\pm$22.1 | 49.0$\pm$29.6 | 33.7$\pm$25.9
| ✓ |  |  |  |  | 94.4$\pm$11.3 | 84.8$\pm$20.8 | 79.2$\pm$21.8
| ✓ | ✓ |  |  |  | 96.2$\pm$4.1 | 89.0$\pm$16.7 | 84.2$\pm$18.7
| ✓ | ✓ | ✓ |  |  | 96.5$\pm$5.3 | 90.7$\pm$17.0 | 87.0$\pm$17.7
| ✓ | ✓ | ✓ | ✓ |  | 97.0$\pm$4.2 | 91.8$\pm$15.0 | 87.4$\pm$17.6
| ✓ | ✓ | ✓ | ✓ | ✓ | 96.2$\pm$5.9 | 92.0$\pm$14.9 | 87.9$\pm$15.8

Figure 6: Ablation studies on the coefficient of the proposed index cost and
the number of query embeddings at inference.

Segmentation Evaluation on FLARE22. We report the evaluation results on
FLARE22 validation leaderboard that gives the detailed performance evaluation
at organ-level. In Table III, nnU-Net and its ensemble version are merely
trained on labeled data, as shown, they can achieve very good results on the
validation set even with only 50 labeled data. The ensembled nnU-Net (assembly
of 12 models from different settings and different folds in cross-validation)
can obtain a 1% improvement on mean DSC. Considering the semi-supervised
setting of this task, we use nnU-Net as the teacher model to generate pseudo
labels for the unlabeled part of the training set, as a straightforward but
effective semi-supervised strategy. Based on this, we exploit the performance
of a recent published model Swin UNETR [64] on this task. As can be seen, Swin
UNETR with self-supervised pre-training on unlabeled data has a 1.7%
improvement against its vanilla version. Our method compares favorably with
the average DSC of 89.7%, which is a competitive result ranking in top 3%
among all 1,162 entries in the validation leaderboard without heavy model
ensembles.

###  IV-F Ablation Study

We perform ablation study on RibInst to investigate the effectiveness of three
key factors in our framework.

Effects of the Weighted Adjacency Matrix. To validate the efficacy of the
preset weighted adjacency matrix, we conduct an ablation study w.r.t.
$\lambda_{m}$, the scaling factor of the matrix, which is also known as index
cost. Candidate parameter is chosen from $[0,1,2,4,8]$ where 0 means the
vanilla bipartite matching. We should notice that when $\lambda_{m}$ is large
enough, it can be regarded as directly assigning the query to a fixed ground-
truth label. Fig. 6(a) reveals that a proper value of $\lambda_{m}=4$ stands
out with the highest Id.Rate of rib labeling. One explanation is that a
reasonable perturbation term during the matching process could gradually
impose anatomical semantic labels on queries and help build the correct
relations among them.

Steerable at Inference? To further validate the steerable attribute of the
proposed model, we randomly pick some rib labels with varied number from 4 to
24 with a step size of 4 at inference. The task is to retrieve the pre-
selected ribs with query embeddings in the same sequence where Id.Rate for the
given subset of ribs is the evaluation metric. Fig. 6(b) demonstrates the
performances of two different settings of with index cost or without index
cost. Note that in the round of only 4 ribs are expected to be detected, the
model without index cost misses all targets. This is not out of expectation,
because the one-to-one relations between query series and ground-truth
sequence are not explicitly constructed. As the number of queries increases,
the vanilla model can gradually return some correct targets back. On the
contrary, Med-Query has a stable performance when the number of query varies.
In terms of inference efficiency, Med-Query can be further boosted with less
queries required, e.g., if 4 queries are required, it can save 17% of time in
detection stage and the ratio can even expand to 50% in the perspective of
full pipeline, as shown in Fig. 6(c). The efficiency improvement benefits from
less computation in Transformer decoder and segmentation head when fewer
queries are executed. During the above ablation experiments, cropped data is
not used.

Data Augmentation & Beyond. We conduct comprehensive experiments to analyze
the critical factors in our proposed framework as shown in Table V. The joint
operations of RandomTranslate, RandomScale, and RandomRotate in 3D space get
the most significant performance gain. RandomCrop along $Z$-axis for imitating
limited FOVs in clinical practice can further improve the Id.Rate by 1.8%.
RandomErase gets additional performance benefit. We also explore a common
trick of imposing relative distance constraints between neighboring center
points in landmark detection tasks [68], and obtain a gain of 0.5%.
Additionally, we exploit the efficacy of a 3D extended version of the fixed
Sine spatial positional encoding that has proven to be useful in [15]. It can
be seen that cancellation of this leads to 0.8% Id.Rate drop if the
measurement thresholding is set to (20mm, 20mm, 10∘). However, if the
thresholding is set more strictly, drop will disappear and even get the
opposite results. A similar observation that removing positional encoding only
degrades accuracy by a small margin can be found in [69]. It implies that
positional encoding may deserve further investigation especially in 3D
scenarios.

##  V Discussion

###  V-A Technology Limitations

Although the proposed method has achieved appealing performance on a range of
anatomy parsing tasks, there are still several limitations that may be
acknowledged. Firstly, our framework requires each stage to be trained
separately, rather than being easily end-to-end trainable. This can lead to
increased complexity as well as potential difficulties in optimizing the
performance of each individual stage. Secondly, despite our careful
consideration of the trade-off between efficiency and accuracy, it is
difficult to avoid the situation that the final segmentation results are
heavily reliant on the accuracy of the detection stage. This means that the
detection model must achieve a high level of performance in order to obtain
accurate segmentation results. Finally, with regards to steerable inference,
our current framework relies on ordered trained query embeddings, which may
limit the flexibility and adaptability of the system. In our future work, we
will explore the integration of natural language embeddings pre-trained using
CLIP [70], to provide a more intuitive and user-friendly interaction within
the network architecture, as in some most recent interactive segmentation
methods [71, 72].

###  V-B Dataset Limitations

Despite the high quality of our curated RibInst dataset, there are limitations
to its generality. It is curated from a dataset that was originally designed
for rib fracture detection and classification, meaning that the ribs included
in the dataset may have varying degrees of underlying pathologies, limiting
their representativeness of the broader (healthy) population. Therefore,
caution must be taken when applying our model to wider real-world scenarios,
as it may require additional training and testing on more diverse data
distributions, such as more normal/healthy patient CT scans and full-body
scans, to further ensure its performance and generality.

##  VI Conclusion

In this work, we have presented a steerable, robust and efficient Transformer-
based framework for anatomy parsing in CT scans. The pipeline is conducted via
following the detection-then-segmentation paradigm and processing input 3D
scans at different resolutions progressively. To our best knowledge, this work
is the first to estimate the 9-DoF representation for object detection in 3D
medical imaging via one-stage Transformer. The resulted method can be executed
in a steerable way to directly retrieve any anatomy of interest and further
boost the inference efficiency. It is a unified computing framework that can
generalize well for a variety of anatomy parsing tasks and achieves new state-
of-the-art performance on anatomy instance detection, identification and
segmentation. We have released our annotations, code and models, hoping to
benefit the community and facilitate the future development on automatic
parsing of anatomical structures.

## References

  * [1] X. Chen, X. Wang, K. Zhang, K.-M. Fung, T. C. Thai, K. Moore, R. S. Mannel, H. Liu, B. Zheng, and Y. Qiu, “Recent advances and clinical applications of deep learning in medical image analysis,” _Medical Image Analysis_ , p. 102444, 2022.
  * [2] B. Stam, H. Peulen, M. M. Rossi, J. S. Belderbos, and J.-J. Sonke, “Validation of automatic segmentation of ribs for ntcp modeling,” _Radiotherapy and Oncology_ , vol. 118, no. 3, pp. 528–534, 2016.
  * [3] S. Jang, P. M. Graffy, T. J. Ziemlewicz, S. J. Lee, R. M. Summers, and P. J. Pickhardt, “Opportunistic osteoporosis screening at routine abdominal and thoracic ct: normative l1 trabecular attenuation values in more than 20 000 adults,” _Radiology_ , vol. 291, no. 2, pp. 360–367, 2019.
  * [4] H. Shen, L. Liang, M. Shao, and S. Qing, “Tracing based segmentation for the labeling of individual rib structures in chest ct volume data,” in _Proc. Int’l Conf. Medical Image Computing and Computer Assisted Intervention_. Springer, 2004, pp. 967–974.
  * [5] T. Klinder, C. Lorenz, J. v. Berg, S. P. Dries, T. Bülow, and J. Ostermann, “Automated model-based rib cage segmentation and labeling in ct images,” in _Proc. Int’l Conf. Medical Image Computing and Computer Assisted Intervention_. Springer, 2007, pp. 195–202.
  * [6] J. Staal, B. van Ginneken, and M. A. Viergever, “Automatic rib segmentation and labeling in computed tomography scans using a general framework for detection, recognition and segmentation of objects in volumetric data,” _Medical Image Analysis_ , vol. 11, no. 1, pp. 35–46, 2007.
  * [7] S. Ramakrishnan, C. Alvino, L. Grady, and A. Kiraly, “Automatic three-dimensional rib centerline extraction from ct scans for enhanced visualization and anatomical context,” in _Proc. of SPIE Medical Imaging_ , vol. 7962. SPIE, 2011, pp. 922–933.
  * [8] D. Wu, D. Liu, Z. Puskas, C. Lu, A. Wimmer, C. Tietjen, G. Soza, and S. K. Zhou, “A learning based deformable template matching method for automatic rib centerline extraction and labeling in ct images,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_. IEEE, 2012, pp. 980–987.
  * [9] M. Lenga, T. Klinder, C. Bürger, J. von Berg, A. Franz, and C. Lorenz, “Deep learning based rib centerline extraction and labeling,” in _International Workshop on Computational Methods and Clinical Applications in Musculoskeletal Imaging_. Springer, 2018, pp. 99–113.
  * [10] J. Yang, S. Gu, D. Wei, H. Pfister, and B. Ni, “Ribseg dataset and strong point cloud baselines for rib segmentation from ct scans,” in _Proc. Int’l Conf. Medical Image Computing and Computer Assisted Intervention_. Springer, 2021, pp. 611–621.
  * [11] J. Yao, S. D. O’Connor, and R. M. Summers, “Automated spinal column extraction and partitioning,” in _3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 2006._ IEEE, 2006, pp. 390–393.
  * [12] T. Klinder, J. Ostermann, M. Ehm, A. Franz, R. Kneser, and C. Lorenz, “Automated model-based vertebra detection, identification, and segmentation in ct images,” _Medical Image Analysis_ , vol. 13, no. 3, pp. 471–482, 2009\.
  * [13] A. Sekuboyina, M. E. Husseini, A. Bayat, M. Löffler, H. Liebl, H. Li, G. Tetteh, J. Kukačka, C. Payer, D. Štern _et al._ , “Verse: a vertebrae labelling and segmentation benchmark for multi-detector ct images,” _Medical Image Analysis_ , vol. 73, p. 102166, 2021.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [15] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _Proc. European Conf. Computer Vision_. Springer, 2020, pp. 213–229.
  * [16] Y. Zheng, A. Barbu, B. Georgescu, M. Scheuering, and D. Comaniciu, “Fast automatic heart chamber segmentation from 3d ct data using marginal space learning and steerable features,” in _Proc. Int’l Conf. Computer Vision_. IEEE, 2007, pp. 1–8.
  * [17] F. C. Ghesu, E. Krubasik, B. Georgescu, V. Singh, Y. Zheng, J. Hornegger, and D. Comaniciu, “Marginal space deep learning: efficient architecture for volumetric image parsing,” _IEEE Trans. Medical Imaging_ , vol. 35, no. 5, pp. 1217–1228, 2016.
  * [18] L. Lu, A. Barbu, M. Wolf, J. Liang, L. Bogoni, M. Salganicoff, and D. Comaniciu, “Simultaneous detection and registration for ileo-cecal valve detection in 3d ct colonography,” in _Proc. European Conf. Computer Vision_. Springer, 2008, pp. 465–478.
  * [19] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in Neural Information Processing Systems_ , vol. 28, 2015.
  * [20] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _Proc. European Conf. Computer Vision_. Springer, 2016, pp. 21–37.
  * [21] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proc. Int’l Conf. Computer Vision_ , 2017, pp. 2980–2988.
  * [22] R. Girshick, F. Iandola, T. Darrell, and J. Malik, “Deformable part models are convolutional neural networks,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2015, pp. 437–446.
  * [23] X. Zhou, D. Wang, and P. Krähenbühl, “Objects as points,” in _arXiv preprint arXiv:1904.07850_ , 2019.
  * [24] Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” in _Proc. Int’l Conf. Computer Vision_ , 2019, pp. 9627–9636.
  * [25] F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnu-net: a self-configuring method for deep learning-based biomedical image segmentation,” _Nature methods_ , vol. 18, no. 2, pp. 203–211, 2021.
  * [26] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proc. Int’l Conf. Computer Vision_ , 2017, pp. 2961–2969.
  * [27] S. Liu, L. Qi, H. Qin, J. Shi, and J. Jia, “Path aggregation network for instance segmentation,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2018, pp. 8759–8768.
  * [28] K. Chen, J. Pang, J. Wang, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Shi, W. Ouyang _et al._ , “Hybrid task cascade for instance segmentation,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2019, pp. 4974–4983.
  * [29] Y. Fang, S. Yang, X. Wang, Y. Li, C. Fang, Y. Shan, B. Feng, and W. Liu, “Instances as queries,” in _Proc. Int’l Conf. Computer Vision_ , 2021, pp. 6910–6919.
  * [30] L. Jin, J. Yang, K. Kuang, B. Ni, Y. Gao, Y. Sun, P. Gao, W. Ma, M. Tan, H. Kang, J. Chen, and M. Li, “Deep-learning-assisted detection and segmentation of rib fractures from ct scans: Development and validation of fracnet,” _EBioMedicine_ , 2020.
  * [31] N. Sharma, L. M. Aggarwal _et al._ , “Automated medical image segmentation techniques,” _Journal of Medical Physics_ , vol. 35, no. 1, p. 3, 2010.
  * [32] D. Shen, G. Wu, and H.-I. Suk, “Deep learning in medical image analysis,” _Annual Review of Biomedical Engineering_ , vol. 19, p. 221, 2017.
  * [33] F. Wang, K. Zheng, L. Lu, J. Xiao, M. Wu, and S. Miao, “Automatic vertebra localization and identification in ct by spine rectification and anatomically-constrained optimization,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2021, pp. 5280–5288.
  * [34] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Proc. Int’l Conf. Medical Image Computing and Computer Assisted Intervention_. Springer, 2015, pp. 234–241.
  * [35] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in _Proc. Int’l Conf. Medical Image Computing and Computer Assisted Intervention_. Springer, 2016, pp. 424–432.
  * [36] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _Int’l Conf. on 3D Vision (3DV)_. IEEE, 2016, pp. 565–571.
  * [37] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers _et al._ , “The medical segmentation decathlon,” _Nature communications_ , vol. 13, no. 1, pp. 1–13, 2022.
  * [38] J. Ma, Y. Zhang, S. Gu, C. Zhu, C. Ge, Y. Zhang, X. An, C. Wang, Q. Wang, X. Liu _et al._ , “Abdomenct-1k: Is abdominal organ segmentation a solved problem,” _IEEE Trans. Pattern Anal. Machine Intell._ , 2021.
  * [39] P. Liu, Y. Deng, C. Wang, Y. Hui, Q. Li, J. Li, S. Luo, M. Sun, Q. Quan, S. Yang _et al._ , “Universal segmentation of 33 anatomies,” _arXiv preprint arXiv:2203.02098_ , 2022.
  * [40] Y. Bian, Z. Zheng, X. Fang, H. Jiang, M. Zhu, J. Yu, H. Zhao, L. Zhang, J. Yao, L. Lu _et al._ , “Artificial intelligence to predict lymph node metastasis at ct in pancreatic ductal adenocarcinoma,” _Radiology_ , p. 220329, 2022.
  * [41] X. Chen, R. Girshick, K. He, and P. Dollár, “Tensormask: A foundation for dense object segmentation,” in _Proc. Int’l Conf. Computer Vision_ , 2019\.
  * [42] X. Wang, T. Kong, C. Shen, Y. Jiang, and L. Li, “Solo: Segmenting objects by locations,” in _Proc. European Conf. Computer Vision_ , 2020.
  * [43] B. Cheng, A. G. Schwing, and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” _Advances in Neural Information Processing Systems_ , 2021.
  * [44] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas, “Normalized object coordinate space for category-level 6d object pose and size estimation,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2019, pp. 2642–2651.
  * [45] J. Lin, Z. Wei, Z. Li, S. Xu, K. Jia, and Y. Li, “Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency,” in _Proc. Int’l Conf. Computer Vision_ , 2021, pp. 3560–3569.
  * [46] Y. Weng, H. Wang, Q. Zhou, Y. Qin, Y. Duan, Q. Fan, B. Chen, H. Su, and L. J. Guibas, “Captra: Category-level pose tracking for rigid and articulated objects from point clouds,” in _Proc. Int’l Conf. Computer Vision_ , 2021, pp. 13 209–13 218.
  * [47] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2016, pp. 779–788.
  * [48] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” _Int. J. Computer Vision_ , vol. 128, pp. 642–656, 2020.
  * [49] X. Zhou, J. Zhuo, and P. Krähenbühl, “Bottom-up object detection by grouping extreme and center points,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2019.
  * [50] Z. Yang, S. Liu, H. Hu, L. Wang, and S. Lin, “Reppoints: Point set representation for object detection,” in _Proc. Int’l Conf. Computer Vision_ , 2019.
  * [51] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” in _Proc. Int’l Conf. Learning Representations_ , 2021.
  * [52] P. Sun, R. Zhang, Y. Jiang, T. Kong, C. Xu, W. Zhan, M. Tomizuka, L. Li, Z. Yuan, C. Wang, and P. Luo, “Sparse r-cnn: End-to-end object detection with learnable proposals,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2021.
  * [53] I. Misra, R. Girdhar, and A. Joulin, “An end-to-end transformer model for 3d object detection,” in _Proc. Int’l Conf. Computer Vision_ , 2021, pp. 2906–2917.
  * [54] R. B. Glass, K. I. Norton, S. A. Mitre, and E. Kang, “Pediatric ribs: a spectrum of abnormalities,” _Radiographics_ , vol. 22, no. 1, pp. 87–104, 2002.
  * [55] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [56] I. T. Jolliffe and J. Cadima, “Principal component analysis: a review and recent developments,” _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , vol. 374, no. 2065, p. 20150202, 2016.
  * [57] H. W. Kuhn, “The hungarian method for the assignment problem,” _Naval research logistics quarterly_ , vol. 2, no. 1-2, pp. 83–97, 1955.
  * [58] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2019, pp. 658–666.
  * [59] Y. Deng, C. Wang, Y. Hui, Q. Li, J. Li, S. Luo, M. Sun, Q. Quan, S. Yang, Y. Hao _et al._ , “Ctspine1k: A large-scale dataset for spinal vertebrae segmentation in computed tomography,” _arXiv preprint arXiv:2105.14711_ , 2021.
  * [60] C. Tobon-Gomez, T. Stroud, J. Cameron, D. Elcock, A. Murray, D. Wyeth, C. Conway, S. Reynolds, P. A. G. Teixeira, A. Blum _et al._ , “Unfolded cylindrical projection for rib fracture diagnosis,” in _International Workshop on Computational Methods and Clinical Applications in Musculoskeletal Imaging_. Springer, 2017, pp. 36–47.
  * [61] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [62] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu _et al._ , “Mmdetection: Open mmlab detection toolbox and benchmark,” _arXiv preprint arXiv:1906.07155_ , 2019.
  * [63] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [64] Y. Tang, D. Yang, W. Li, H. R. Roth, B. Landman, D. Xu, V. Nath, and A. Hatamizadeh, “Self-supervised pre-training of swin transformers for 3d medical image analysis,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2022, pp. 20 730–20 740.
  * [65] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei, “Relation networks for object detection,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2018, pp. 3588–3597.
  * [66] M. Musy, G. Jacquenot, and G. Dalmasso, “vedo, a python module for scientific analysis and visualization of 3d objects and point clouds,” doi:10.5281/zenodo.5655358, 2021.
  * [67] H.-Y. Zhou, J. Guo, Y. Zhang, L. Yu, L. Wang, and Y. Yu, “nnformer: Interleaved transformer for volumetric segmentation,” _arXiv preprint arXiv:2109.03201_ , 2021.
  * [68] W. Liu, Y. Wang, T. Jiang, Y. Chi, L. Zhang, and X.-S. Hua, “Landmarks detection with anatomical constraints for total hip arthroplasty preoperative measurements,” in _Proc. Int’l Conf. Medical Image Computing and Computer Assisted Intervention_. Springer, 2020, pp. 670–679.
  * [69] X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in _Proc. IEEE Conf. Computer Vision and Pattern Recognition_ , 2021, pp. 9640–9649.
  * [70] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _Proc. Int’l Conf. Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [71] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” 2023.
  * [72] X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y. J. Lee, “Segment everything everywhere all at once,” 2023.