# U-LanD: Uncertainty-Driven Video Landmark Detection

Mohammad H. Jafari  Christina Luong  Michael Tsang  Ang Nan Gu  Nathan Van
Woudenberg  Robert Rohling  Teresa Tsang  Purang Abolmaesumi This work was
supported in part by the Natural Sciences and Engineering Research Council
(NSERC) of Canada and in part by the Canadian Institutes of Health Research
(CIHR). T. Tsang is the Director of the Vancouver General Hospital and
University of British Columbia Echocardiography Laboratories, and Principal
Investigator of the CIHR-NSERC grant supporting this work. P. Abolmaesumi is
Co-Principal Investigator of the grant supporting this work. T. Tsang and P.
Abolmaesumi contributed equally (joint senior authors).M. Jafari, A. Gu, N.
Woudenberg, and P. Abolmaesumi are with the Department of Electrical and
Computer Engineering, The University of British Columbia, Vancouver, BC V6T
1Z4, Canada (e-mail: purang@ece.ubc.ca).R. Rohling is with the Department of
Electrical and Computer Engineering, The University of British Columbia,
Vancouver, BC V6T 1Z4,Canada, and also with the Department of Mechanical
Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4,
Canada.C. Luong, M. Tsang, and T. Tsang are with the Vancouver General
Hospital, Echocardiography Laboratory, Division of Cardiology, Department of
Medicine, The University of British Columbia, Vancouver, BC V5Z 1M9, Canada
(e-mail: t.tsang@ubc.ca).

###### Abstract

This paper presents U-LanD, a framework for joint detection of key frames and
landmarks in videos. We tackle a specifically challenging problem, where
training labels are noisy and highly sparse. U-LanD builds upon a pivotal
observation: a deep Bayesian landmark detector solely trained on key video
frames, has significantly lower predictive uncertainty on those frames vs.
other frames in videos. We use this observation as an unsupervised signal to
automatically recognize key frames on which we detect landmarks. As a test-bed
for our framework, we use ultrasound imaging videos of the heart, where sparse
and noisy clinical labels are only available for a single frame in each video.
Using data from 4,493 patients, we demonstrate that U-LanD can exceedingly
outperform the state-of-the-art non-Bayesian counterpart by a noticeable
absolute margin of 42% in $R^{2}$ score, with almost no overhead imposed on
the model size. Our approach is generic and can be potentially applied to
other challenging data with noisy and sparse training labels.

Video landmark detection, uncertainty estimation, key frame recognition,
sparse training labels, echocardiography.

##  1 Introduction

Figure 1: Key Observation: A Bayesian landmark detector has a significantly
different predictive uncertainty on key frames vs. non-key frames. Our
proposed U-LanD framework exploits this observation to jointly identify key
frames and predict landmarks in videos. We demonstrate the efficacy of U-LanD
for the very challenging problem of analyzing ultrasound videos of the heart,
where only single-frame, noisy labels are available for each training sample.

Accurate detection of the reference points (landmarks) coordinates in images
and videos is a fundamental step in many vision tasks, such as face analysis
[1, 2, 3, 4, 5, 6], pose estimation [7, 8, 9], and medical imaging [10, 11,
12].

The video landmark detection involves additional complexities of recognizing
key frames used to detect the landmarks. The key frames denote moments in time
when a targeted event happens, and an object of interest has the highest
visibility in the video. During data labelling routines, experts usually only
label the landmark points at the selected key frames. As a result, the
available training video datasets suffer from two limitations: 1) videos are
sparsely labelled, _i.e.,_ a small portion of frames in each video have
ground-truth landmark labels; and 2) the labelled frames are extensively
biased towards specific points in time, _i.e.,_ only key frames in each
training video are labelled. Previous work mainly divides the problem of video
object detection on key frames into sub-problems of key frame recognition [13,
14, 15, 16, 17] and object detection. They propose techniques such as self-
supervised learning [18], semi-supervised learning [19], label propagation
[20], registration [21], and temporal cycle-consistency [22]. However, the
inherent predictive uncertainties in video landmark measurements are generally
ignored in the existing literature.

In this paper, we present U-LanD, a novel uncertainty-driven video landmark
detection framework. U-LanD is specifically designed to consider the bias in
sparsely annotated video data and automatically detect the location of
landmarks on the key frames of the video. U-LanD is based on a core
observation: a deep Bayesian encoder-decoder network only trained on labelled
key frames, shows significantly higher prediction confidence (lower
uncertainty) in test time when deployed on key frames vs. non-key frames, as
illustrated in Fig 1. U-LanD exploits the test-time predictive uncertainty as
an unsupervised guiding signature to detect landmark points on key frames.
U-LanD considers several aspects of predictive uncertainty in landmark
detection, namely i) contextual quality control based on landmark heatmaps;
ii) pixel-level epistemic uncertainty with variational dropout; and iii)
pixel-level heteroscedastic aleatoric uncertainty by placing a Gaussian
distribution over the observed labels.

We demonstrate the efficacy of U-LanD on a challenging dataset with sparsely
annotated data, namely echocardiography (echo, heart ultrasound). Echo videos,
aka echo cine series, have a notoriously noisy nature (sample frames can be
seen in Fig. 1), which increases the complication for automated analysis. We
tackle the task of left ventricular outflow tract (LVOT) landmark detection
[23], as a problem that suffers from an extreme limit of label sparsity; in
LVOT training videos, only one frame throughout the entire span of frames has
the ground-truth landmark label. In each training video, the expert clinician
has annotated locations of two points (coordinates of points used to measure
the LVOT diameter), only for one frame (a frame around the mid-systole phase
of the heart), where the targeted object, _i.e.,_ the aorta, has the highest
visibility. For our experiments, we gather a large-scale echo dataset of 4,493
patients, demonstrating that U-LanD can significantly improve the results of
state-of-the-art non-Bayesian counterparts. Also, while U-LanD is fully
automatic, it can surpass the results of semi-automatic key frame landmark
detection involving expert supervision.

##  2 Related Works

###  2.1 Image Landmark Detection

Deep models are the de-facto state-of-the-art for image landmark detection,
which can be categorized into three broad groups: i) Regression networks [24,
25], which use Convolutional Neural Networks (CNNs) to directly regress
coordinate values of the landmark points; ii) Reinforcement Learning (RL)
approaches [26, 27, 28], where the RL agents learn to find an optimal path to
the locations of the landmarks by interacting with the surrounding environment
in the image space; and iii) Fully Convolutional Networks (FCN) [29, 30, 31,
9], which tackle landmark detection as semantic segmentation by predicting
heatmaps (or blobs) denoting the locations of landmarks. Various iterative or
cascaded combinations of the above approaches are used in the literature [32,
33]. The FCN approach provides advantages such as translation invariance,
independence from image size, and interpretability of the predicted heatmaps.
The FCN networks [34] have benefited from many improvements in recent years
through successor architectures, such as the U-Net [35], Deeplab family [36],
and Efficientdet [37]. In this work, we build U-LanD on top of a modified
U-Net model, while the method has no specific assumptions regarding the choice
of the backbone landmark detection network.

###  2.2 Video Landmark Tracking

In video landmark tracking, the goal is to label all the frames of the video
and provide temporally-consistent predictions through time. Variations of the
above image landmark detectors are extended to temporal dimensions using
3D/(2+1)D models [38, 39], attention modules [40], transformers [41], and
models based on optical flow tracking [5]. However, our task requires only to
detect landmarks on the key frames of the video and discard the measurements
on non-key frames; therefore, the stabilized tracking through time, followed
by the above researches, is not the goal here.

###  2.3 Label Propagation

The semi-supervised and self-supervised learning techniques [18, 10, 19] are
used to improve training when data is partly labelled, _e.g.,_ only few frames
of training videos have ground-truth annotation. Methods such as registration
[21] and temporal correspondence by cycle-consistency [42, 43, 22] are used to
propagate labels to unlabelled frames, providing weakly supervised annotations
to reduce the bias in training [44, 45]. Our work is different from this line
of research, where U-LanD leverages the existing training bias, captured via
predictive uncertainty, as a signature to recognize the key frames.

###  2.4 Prediction Quality Control

Safe deployment of deep learning methods requires the model not to report a
result with low confidence (high uncertainty), _i.e.,_ raise an alarm that the
model does not know the prediction. Works of [46, 47, 48] leverage the
predictive uncertainty to detect failed predictions. Also, non-uncertainty
based approaches include scoring the validity of the prediction by comparing
it to the distribution of ground-truth labels, via approaches such as atlas
registration [49], generative adversarial networks [50], and variational
autoencoders [51]. However, in U-LanD, we track the prediction confidence
through time and use it to detect landmarks on the key frames of the video. To
this end, U-LanD measures pixel-level epistemic and aleatoric uncertainty, and
a task-specific contextual validity check based on the predicted landmark
heatmaps.

Figure 2: A block diagram of the proposed U-LanD framework, including
training, statistical calibration, and test phases. U-LanD automatically
predicts landmarks on key frames of videos, where the training videos are only
labelled on one frame. We propose to leverage the predictive uncertainty of a
Bayesian landmark detector (BU-Net) to recognize key frames vs. non-key frames
through time.

##  3 Contributions

In this work, we preset U-LanD, a deep Bayesian framework for landmark
detection on key frames of the videos. To summarize, the major contributions
of the paper are:

  * •

We formulate a Bayesian method for video landmark detection using sparsely
labelled training data; only one key frame of each training video has the
ground-truth label.

  * •

We exploit the inter-frame variations of predictive uncertainty as an
unsupervised signal to recognize key frames. This enables a simple 2D landmark
detector to achieve superior results for video landmark detection.

  * •

We demonstrate U-LanD in a challenging task of echo key frame landmark
detection. We gather a large-scale echo video dataset of 4,493 patients for
our experiments, showing U-LanD can improve the state-of-the-art by a
significant absolute margin of 42% in $R^{2}$ score, and surprisingly surpass
semi-automatic key frame landmark detection by expert human supervision.

##  4 Proposed Method: U-LanD

###  4.1 Problem Formulation

The goal is to automatically detect landmark locations on key frames of the
video, while in training data only one key frame of each video has the ground-
truth landmark, as shown in  in Fig. 2. The inputs at test time are variable-
length videos, with no human supervision regarding location of landmarks or
indices of the key frames, as shown in  in Fig. 2. To define the task
formally, let
$\operatorname{\mathcal{D}}=\\{(f^{k_{i}}_{i},y^{k_{i}}_{i})\\}_{i=1}^{N}$
indicate the $N$ training videos, where pairs of $f^{k_{i}}_{i}$ and
$y^{k_{i}}_{i}$ are the frame and ground-truth labels (coordinates of $\tau$
landmarks) on the key frame $k_{i}$ of the $i^{th}$ training video. Each video
may have one or more key frames, represented by
${\operatorname{\mathcal{K}}_{i}}$ set, but only one is labelled, _i.e.,_
$k_{i}\in{\operatorname{\mathcal{K}}_{i}}$,
$|\operatorname{\mathcal{K}}_{i}|\geq 1$. Sample of test data is denoted by
$\hat{x}_{i}={\hat{f}_{i}}^{\ 1:P_{i}}$, which is the $i^{th}$ test video with
$P_{i}$ frames; test videos can have variable lengths. The U-LanD framework
$U(x)$ aims to obtain the mapping $U({\hat{f}_{i}}^{\
1:P_{i}})\xrightarrow[]{}\hat{y}^{k_{i}}_{i}$, _i.e.,_ the $U(x)$ predicts the
key frame landmark measurements. The key frames ($k$) in videos can occur at
different indices which are unknowns, but all the key frames have specific
common features, such as denoting specific events, when a targeted object has
a high visibility.

###  4.2 Solution Overview

The block diagram of the train, calibration, and test phases of the U-LanD
framework are shown in Figure 2. In U-LanD, a Bayesian 2D image landmark
detector, namely BU-Net, is solely trained on the labelled key frame in
$\operatorname{\mathcal{D}}$, as is shown in  in Fig. 2. The BU-Net is a deep
Bayesian encoder-decoder network, formulated in Section 4.3, predicting both
the heatmaps ($\hat{y}$) that show the locations of landmarks, as well as the
predictive uncertainty maps (epistemic $\hat{u}_{epi}$ and aleatoric
$\hat{u}_{alea}$) that show pixel classification uncertainties.

In training, BU-Net has only been exposed to key frames, therefore, non-key
frames at test will be regarded as out-of-distribution (OOD) frames. In this
setup, the key frame recognition is simplified to discarding OOD frames (non-
key frames) based on the quality of the BU-Net’s prediction. We leverage three
types of prediction quality to recognize key frames vs. non-key frames (OOD
frames): i) Interpretability provided by the prediction heatmaps, detailed in
Section 4.4; ii) Pixel-level epistemic uncertainty, as OOD frames are
generally expected to incur a higher epistemic uncertainty, described in
Section 4.5; and iii) Pixel-level aleatoric uncertainty, as the non-key frames
where the object of interest is obscured or has lower visibility are expected
to have a higher aleatoric uncertainty, detailed in Section 4.6. Subsequently,
a statistical method is used to learn to differentiate between BU-Net’s
uncertainty signatures on key versus non-key frames, in a calibration phase
(shown in  in Fig. 2), explained in Section 4.7. In the test, the landmark
detector (BU-Net) is applied to all the frames of the video and detects
landmarks on key frames based on its prediction confidence.

###  4.3 Landmark Detector

The per-frame landmark detection is formulated as a segmentation problem,
where the BU-Net predicts both the heatmaps highlighting the coordinates of
the landmark points, and uncertainty maps described in Sections 4.5 and 4.6.
The BU-Net is based on a modified U-Net architecture (network details are
given in 5.2).

To generate the ground-truth masks for training, we place a circle with radius
$\delta$ at the coordinates of landmarks, where $\delta$ is a hyperparameter
tuned to account for the inter-observer variability in landmark labels. Here
for the purpose of demonstration, we adapt a circle formation, while more
complex choices of ground-truth genesis based on the underlying object could
be investigated [30]. At test time, the center of gravity (COG) of the
predicted heatmaps are used to obtain the coordinates of the landmarks. The
BU-Net, only trained on key frames, generally produces valid predictions in
vicinity of the key frames, and produces invalid predictions for non-key
frames, where it shows an unseen event or the object of interest does not have
clear visibility. Therefore, the prediction quality based on the
interpretation of the heatmaps could be used as a criterion to detect the key
frames discussed next.

###  4.4 Contextual Quality Control

The predicted heatmaps by the detector are passed through a sanity check to
discard non-valid predictions, which correlate to non-key frames. Each key
frame has $\tau$ landmark points, where each point is marked in the ground-
truth mask with a circle of radius $\delta$. We perform a spatial sanity check
on the predicted heatmap of each frame by locating all blobs with radius
$>\delta$; if $\hat{\tau}$ heatmaps are located in the prediction, and
$\hat{\tau}\neq\tau$, the frame is discarded as a non-key frame. The BU-Net
predictions tend to be invalid the further you move from the key frames,
_i.e.,_ the detector fails to predict enough heatmaps ($\hat{\tau}<\tau$) or
predicts multitude of noisy heatmaps ($\hat{\tau}>\tau$). After this initial
contextual sanity check based on the prediction heatmap, the remaining
candidate key frames are assessed in terms of predictive uncertainty. The two
major sources of predictive uncertainty considered are epistemic and
aleatoric.

###  4.5 Epistemic Uncertainty

The epistemic uncertainty accounts for the ignorance in model parameters,
which can be decreased by collecting more training data. The BU-Net, having
only been trained on key frames, is expected to have a lower epistemic
uncertainty on frames that have characteristics similar to the key frames, and
higher uncertainty as the further you move from the key frames in time (OOD
frames). Therefore, the epistemic uncertainty of BU-Net’s prediction could be
used to distinguish between key and non-key frames.

Bayesian neural networks (BNNs) quantify epistemic uncertainty by learning the
posterior distribution over the network’s parameters ($w$). Given training
frames $F$ and labels $Y$, the Bayes rule gives the posterior distribution by:

$p(w|F,Y)=\dfrac{p(Y|F,w)p(w)}{p(Y|F)},$ (1)

where the denominator is defined as:

$p(Y|F)=\int p(Y|F,w)p(w)dw.$ (2)

The prediction $\hat{y}$ for frame $\hat{f}$ is obtained by marginalizing over
the space of possible weights:

$p(\hat{y}|\hat{f},F,Y)=\int p(\hat{y}|\hat{f},w,F,Y)p(w|F,Y)dw,$ (3)

which provides a distribution over possible outputs, and naturally gives a
measure of epistemic uncertainty. In practice, the integral in Eq. (2) is not
computationally tractable and must therefore be approximated. Various
approximations of BNNs exist that estimate the true posterior $p(w|F,Y)$, with
a variational distribution $q(w)$, which is feasible to train. Popular BNN
approximations include stochastic variational inference [52, 53, 54],
multiplicative normalizing flows [55], stochastic batch normalization [56],
and variational inference by Monte-Carlo (MC) dropout [57, 58]. Also, non-
Bayesian approaches generally train many deep ensemble or bootstrap models
[59, 60] to measure the uncertainty through predictive variance.

In our work, U-LanD adapts the widely used MC dropout BNN approximation;
however, the U-LanD framework is not tied to the specific choice of the
uncertainty estimation method. Dropout is the regularization technique
conventionally used only in the training phase of the deep models, where each
node is randomly ignored (dropped) with the probability $p_{drop}$, in order
to avoid over-fitting by imposing several predictive paths. Gal and Ghahramani
show in [61] that dropout layers give a Bernoulli-approximate variational
inference of BNNs. Dropout is added after all the layers of BU-Net where the
dropout is enabled at both training and test phases. With MC dropout enabled,
each forward pass gives a sample from the approximate posterior distribution
$q(w)$, obtained through a stochastic subset of the network. In the test, we
run the BU-Net for $M_{E}$ times to use the average of the outputs from
stochastic forward passes as our predicted landmark heatmap, and consider the
standard deviation as a measure of epistemic uncertainty (gives
$\hat{u}_{epi}$).

###  4.6 Aleatoric Uncertainty

The aleatoric uncertainty accounts for the ambiguity in the observed data,
such as the inherent noise in the measurements or labels. The Bayesian
landmark detector is expected to have a higher aleatoric uncertainty on noisy
non-key frames, where the object of interest has low visibility. The aleatoric
uncertainty of BU-Net therefore can be used to detect the key frames in time.

Popular approaches to estimate the input-dependent aleatoric uncertainty
include test-time augmentation [62, 63], and learning uncertainty by
parameterizing the observation noise in the likelihood function [64, 65]. In
BU-Net, the input-dependent pixel-level aleatoric uncertainty is learned by
modelling the observation as a sample from a Gaussian distribution. The BU-Net
learns to jointly predict two output maps, namely the aleatoric uncertainty
along with the landmark heatmap.

We place a Gaussian distribution over the logits of the output layer, thus for
the output pixel in row $i$ and column $j$ we have:

$\displaystyle l_{i,j}\sim\mathcal{N}(\mu_{i,j},\sigma^{2}_{i,j}),$ (4)
$\displaystyle p_{i,j}=sigmoid(l_{i,j}),$

where $l$ and $p$ denote the output logit and the landmark probability for
each pixel, respectively, and $\mu$ and $\sigma$ are parameters learned by the
network, where $\sigma$ correlates to the pixel-level aleatoric uncertainty.
The reparametrization trick is used with Eq. (4) to enable learning by
backpropagation:

$l_{i,j}=\mu_{i,j}+\sigma_{i,j}\epsilon.\quad\quad\epsilon\sim\mathcal{N}(0,1).$
(5)

The network is trained with two loss functions, namely, soft Dice and the
weighted binary cross-entropy (WBCE), comparing the predicted heatmap ($p$ in
Eq. (4)) to the ground-truth landmark masks. The weights of the WBCE are
assigned inversely proportionate to the number of pixels in landmark versus
background, normalizing the effect of imbalanced classes. While there is no
analytic solution to integrate out the Gaussian distribution in Eq. (5), it
can be approximated by Monte-Carlo integration using $M_{A}$ samples at
training:

$p_{i,j}=\dfrac{1}{M_{A}}\sum_{i=1}^{M_{A}}sigmoid(\mu_{i,j}+\sigma_{i,j}\epsilon),\quad\quad\epsilon\sim\mathcal{N}(0,1),$
(6)

whereas no sampling is needed at test ($\epsilon=0$), and the network learns
to predict $\mu$ (which gives the landmark prediction heatmap) and $\sigma$
(which gives a measure of aleatoric uncertainty, _i.e.,_ gives
$\hat{u}_{alea}$) for each input frame in one forward pass. Here for the
purpose of demonstration, we assume a simple Gaussian distribution for the
per-pixel labels; however, the U-LanD framework does not have a specific
assumption about the choice of the method for aleatoric uncertainty
estimation.

###  4.7 Statistical Calibration

BU-Net provides three criteria of prediction confidence, namely contextual
quality control, pixel-level epistemic uncertainty, and pixel-level aleatoric
uncertainty. At test time, the indices of the key frames are unknown, and the
above three confidence criteria are used to detect key frames vs. non-key
frames, as shown in  in Fig. 2. If a frame passes the three confidence
criteria, it will be added to a pool of key frame measurements (see  in Fig.
2), otherwise, it will be discarded as a non-key frame (see  in Fig. 2). The
first criterion (contextual quality control) is a spatial sanity check based
on the prediction heatmaps, explained in Section 4.4, which does not require a
calibration phase. After passing this first criterion, the predictions are
evaluated in terms of epistemic (Section 4.5) and aleatoric (Section 4.6)
uncertainty, for which a threshold of validity needs to be tuned in a
calibration phase. We set aside a part of unseen training data, called
validation-calibration set $\operatorname{\mathcal{X}}_{calib}$ with
$N^{^{\prime}}$ samples. After BU-Net’s training is done, in a calibration
phase, we run BU-Net on the key frames of $\mathcal{X}_{calib}$ and find the
$sum()$ of pixel-level uncertainty values for each prediction, _i.e.,_ a
vector of size $N^{^{\prime}}$ is obtained, for both the aleatoric and
epistemic maps. Thereafter, the mean ($\mu_{alea}$ and $\mu_{epi}$) and
standard deviation ($\sigma_{alea}$ and $\sigma_{epi}$) of uncertainty vectors
are calculated. In test time, we measure $Z_{score}={\lvert
sum(\hat{u})-\mu\rvert}/{\sigma}$ for both the aleatoric and epistemic
uncertainty maps ($\hat{u}_{epi}$ and $\hat{u}_{alea}$) of each frame, and
discard the frame as a non-key frame (OOD frame) if $Z_{score}>\xi$. Further,
a temporal sanity check is used, where each frame is only added to the pool of
key frames if a temporal window of $\lambda$ frames in its adjacency have also
passed the above confidence criteria.

Using the above confidence criteria, U-LanD automatically selects the key
frames for landmark measurement directly based on the quality of the predicted
landmarks by BU-Net. We use the pool of detected key frames to obtain the key
landmark measurements. The experiment details and the evaluations showing the
effectiveness of this approach are presented in the following Section.

a) Sample Non-key Frame |  | Epistemic | Aleatoric
---|---|---|---
|  |  |
b) Sample Key Frame |  | Epistemic | Aleatoric
|  |  |

(a)  (b)

(c)

(d)

(e)  (f)

(g)

(h)

Figure 3: Sample frames of a test video, along with their corresponding aleatoric and epistemic uncertainty maps. a) Sample non-key frame, automatically discarded due to high uncertainty (no landmark prediction). b) Sample key frame with predicted landmarks superimposed on the frame. The targeted object is LVOT in heart ultrasound videos; the landmarks are used to measure LVOT length (the line between the two dots). Image (b) shows the predicted landmarks (green), ground-truth landmarks (orange), and their overlap (yellow). The prediction error (mean absolute difference in length) for this case is 0.18 mm. Further, sample video results are presented in the supplementary materials. |  |
---|---|---
Figure 4: Object length measured using predicted landmarks versus ground-truth
length. Left: U-Net analyzing all the frames. Middle: Semi-automatic approach,
where video key frames are selected by an expert human. Right: U-LanD
(Proposed). The targeted object is LVOT in heart ultrasound videos, where LVOT
length is measured using the predicted landmarks. Table 1: Evaluation of the
video landmark measurement performance. Best results are in bold.

Method |  $R^{2}$ (%) |  $\Delta R^{2}$ (%) |  MAE (mm) |  STD (mm) |  MAX (mm) |  Reject Rate ($\%$)
---|---|---|---|---|---|---
I) U-Net; per-frame analysis of all frames [66, 67, 68] | 24 | n/a | 1.73 | 3.14 | 42.81 | 0
II) Semi-automatic U-Net; analyzing the key frames [41, 69, 15] | 41 | +71 | 1.50 | 2.14 | 25.39 | 2.52
U-LanD (CQC+AL+EP) - Proposed | 66 | +175 | 1.08 | 0.89 | 4.66 | 10.69

|
---|---
Figure 5: Object length measured using predicted landmarks versus ground-truth
length. Left: U-LanD using contextual quality control (CQC). Right: U-LanD
using CQC and predictive uncertainty; aleatoric (AL) and epistemic (EP). The
targeted object is LVOT in heart ultrasound videos, where LVOT length is
measured using the predicted landmarks.

##  5 Experiments

###  5.1 Dataset

The data used for the experiments are collected from Picture Archiving and
Communication System at Vancouver General Hospital (VGH), Canada, following
approval from the Medical Research Ethics Board in coordination with the
privacy office. We gathered echo videos of 4,493 patients, captured by a
variety of ultrasound devices, namely Philips iE33, Vivid i/7/9/95, Sonosite,
and Sequoia. The echo cines show the parasternal long axis (PLAX) view of the
heart. For all the echo videos, the ground-truth locations of $\tau=2$ points
regarding LVOT landmarks are only available for one frame throughout the
video. The ground-truth annotations are derived from the archived information
of the patients, originally provided by expert cardiologists. The videos are
split into train and test set with a ratio of 90% and 10%, respectively. Also,
10% of training data are set aside as the $\operatorname{\mathcal{X}}_{calib}$
to learn the statistics of BU-Net uncertainty signatures, as explained in
Section 4.7. There is no overlap between patients in the train, validation-
calibration, and test set.

For echo LVOT landmark detection in clinical settings, the expert cardiologist
examines the echo video, selects a key frame around mid-systole phase of the
heart, where the LVOT landmarks are clearly visible, and annotates two points
used to measure LVOT length. Sample echo frames and the corresponding visual
results and uncertainty maps are presented in Fig. 3. In U-LanD, the BU-Net
takes all the frames of the video and predicts landmarks for the recognized
key frames, _i.e.,_ the frames where the prediction confidence passes the
quality criteria explained in Section 4.7, and otherwise discards low
confidence predictions regarding them as non-key frames.

Due to clinical diagnostic needs, the expert cardiologists usually tend to
annotate the LVOT landmarks at the key frame that shows the maximum lengths of
the LVOT. In U-LanD, if the predicted LVOT landmarks from the key frames were
perfectly correct, we could just select the maximum length among the key
frames to be reported as our predicted LVOT length; however, we have a pool of
predicted key frame measurements, and selecting the single maximum value in
the pool might be prone to error, we therefore instead choose to select the
75% percentile of the predicted lengths to be reported as the LVOT length.
This approach helps all the high quality predicted landmarks in key frames
contribute to the reported measurement, and improve the accuracy of the
predicted LVOT length by further cancelling out the noisy observations.

Table 2: Ablation studies. Best results are in bold.

Method |  $R^{2}$ (%) |  $\Delta R^{2}$ (%) |  MAE (mm) |  STD (mm) |  MAX (mm) |  Reject Rate ($\%$)
---|---|---|---|---|---|---
U-LanD (CQC) | 59 | +146 | 1.19 | 1.09 | 8.57 | 2.94
U-LanD (CQC+AL) | 63 | +162 | 1.12 | 0.94 | 5.15 | 5.24
U-LanD (CQC+EP) | 63 | +162 | 1.11 | 0.94 | 6.78 | 9.85
U-LanD (CQC+AL+EP) | 66 | +175 | 1.08 | 0.89 | 4.66 | 10.69

###  5.2 Implementation Details

The landmark detector BU-Net is based on a U-Net architecture with three down-
sampling blocks and three reverse up-sampling blocks, where skip connections
concatenate feature maps from the encoder to the corresponding layer in the
decoder path. Each down-sampling block includes two convolutional layers with
kernel size $3\times 3$ and a maxpooling of size $2\times 2$. The decoder
mirrors the down-sampling blocks, except transpose convolution is used for
upsampling. The number of filters in the first convolutional layer is $32$
which is doubled after each down-sampling step. The covolutional layers are
followed by batch normalization ($\textrm{momentum}=0.8$), ReLU, and dropout
($p_{drop}=0.2$). The last layer includes two output maps (as Section 4.6),
namely predicted landmark heatmap ($1\times 1$ convolutional kernel followed
by sigmoid) and aleatoric uncertainty map ($1\times 1$ convolutional kernel
followed by softplus). The number of MC integral samples at training time used
for aleatoric uncertainty learning is $M_{A}=100$. MC dropout samples in test
are $M_{E}=30$. BU-Net is run once with dropout disabled to obtain aleatoric
uncertainty in one pass. The input images are resized to $224\times 224$
pixels. The radius of the circles placed over the location of ground-truth
landmark points in training masks is $\delta=7$, the OOD threshold is $\xi=1$,
and the size of the temporal window is $\lambda=5$, which are hyperparameters
tuned using a validation set. We use the Adam optimizer with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and a learning rate initially set to $1e-3$. On the fly
data augmentation is done in each iteration, including random shift, rotation,
zoom, and gamma transform.

###  5.3 Evaluations

We compare the predicted LVOT length by U-LanD against the ground-truth LVOT
length in the patient’s record based on the cardiologist’s examination of the
echo video. We calculate $R^{2}$ score, the square of the Pearson product-
moment correlation coefficient; $R^{2}\in[0\%,100\%]$, where 0% denotes no
correlation, and 100% denotes total correlation. We also report the percentage
of improvement in $R^{2}$ over the baseline method ($\Delta R$), mean absolute
error (MAE), standard deviation (STD) of absolute error, and worst-case
performance, _i.e.,_ maximum absolute error (MAX). Table 1 presents the
comparison with two state-of-the-art paradigms for video object detection.

Paradigm I) All the frames of the videos are analyzed for object measurement
[66, 67, 68]. To have a comparative evaluation of U-LanD against paradigm I,
the non-Bayesian U-Net, as a state-of-the-art base model, is used to predict
landmarks on all the frames. The network architecture and the method used to
calculate LVOT length are the same in the applied U-Net and U-LanD, except
there are no measures of prediction confidence in the baseline U-Net to be
used to discard uncertain (non-key frame) landmarks through time. The results
of this experiment can be found in the first line of Table 1, and the scatter
plots comparing predicted LVOT length to ground-truth are given in Fig. 4.

Paradigm II) Solving the key frame video object detection in two steps, II-A)
a temporal model (_e.g.,_ attention RNN, 3D/(2+1)D CNN, transformer) [41, 69,
15, 39, 17] is trained to detect the indices of the key frames, II-B) followed
by object detection at the recognized key frames. In order to compare U-LanD
framework against paradigm II, we consider a semi-automatic approach, where
the ground-truth indices of the key frames are suggested by the cardiologist,
followed by landmark detection by U-Net on these ground-truth key frames
(called semi-automatic U-Net); the semi-automatic U-Net could be an ideal case
of paradigm II, where there is no error in step II-A, by using the ground-
truth key frame index. The semi-automatic U-Net results can be found in the
second line of Table 1, and the scatter plots are presented in Fig. 4.

As can be seen in Table 1 and Fig 4, U-LanD significantly improves the state-
of-the-art non-Bayesian base model. U-LanD’s superiority could be attributed
to its ability to analyze intervals of the video specifically selected by the
Bayesian landmark detector (BU-Net) when the prediction confidence is high
enough for accurate landmark measurement, particularly discarding the noisy
frames in the challenging echo videos. Futhermore, the right-most column of
Table 1 and the orange text on Fig. 4 and Fig. 5 (Reject Rate) shows the
percentage of rejected videos by U-LanD, when the model has automatically
detected that the video quality is not good enough for accurate landmark
measurement. The rejected rate in semi-automatic U-Net denotes videos where
the U-Net has failed to predict two landmarks needed to calculate LVOT
distance on the ground-truth key frame index.

Ablation Studies: The ablation study is given in Fig. 5 and Table 2; U-LanD
(CQC): only using the contextual quality control (Section 4.4), U-LanD
(CQC+AL): CQC and aleatoric uncertainty (Section 4.6), U-LanD (CQC+EP): CQC
and epistemic uncertainty (Section 4.5), and U-LanD (CQC+AL+EP): proposed
method using all three criteria of prediction confidence. As can be seen, the
three criteria of quality control, namely CQC, AL, and EP have complementary
effects, significantly improving the results by accounting for different
aspects of prediction confidence.

Further Analysis: Compared to non-Bayesian base, U-LanD has $M_{E}$ runtime
overhead for epistemic uncertainty sampling; however, for time-sensitive
applications, the proposed U-LanD (CQC+AL) could be the suitable choice, where
the uncertainty is obtained in one forward pass (no need for sampling), while
the $R^{2}$ score of CQC+AL is decreased only by 3% compared to the U-LanD’s
full mode. Compared to 3D or attention-based models conventionally used for
video segmentation and object measurement, U-LanD is built upon a small-size
2D (per-frame) model, which enables its integration into memory-limited
embedded or mobile systems. U-LanD almost has no memory overhead compared to
the non-Bayesian counterpart (less than 1e-3%); only one extra feature map is
added in the last layer of the detector to predict aleatoric uncertainty.

##  6 Conclusions

In this paper we proposed U-LanD, a Bayesian framework for joint key frame and
landmark detection in videos with extremely sparse and noisy labels. U-LanD
leveraged the variations of uncertainty through time as an unsupervised signal
to detect landmarks on key frames of the video. We demonstrated U-LanD on a
challenging dataset of heart ultrasound series, where only one frame in each
training video is annotated, and yet the annotations are noisy clinical
labels. For experiments, we gathered a large-scale video dataset of echo
series from 4,493 patients, showing the U-LanD efficiency; improving the
$R^{2}$ score of state-of-the-art non-Bayesian counterpart with a noticeable
margin of 42% (+175%), with no extra-label cost, and almost no memory
overhead. U-LanD (fully automatic) can also outperform the semi-automatic
video landmark detection by 25% (+61%) in $R^{2}$ score. U-LanD can achieve
good results even with simple choices of backbone detector and OOD rejection
method. Future work could include investigating temporal uncertainty in multi-
object detection and tracking.

## References

  * [1] P. Chandran, D. Bradley, M. Gross, and T. Beeler, “Attention-driven cropping for very high resolution facial landmark detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5861–5870.
  * [2] L. Chen, H. Su, and Q. Ji, “Deep structured prediction for facial landmark detection,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 2450–2460.
  * [3] C. Ma, Z. Jiang, Y. Rao, J. Lu, and J. Zhou, “Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 5569–5578.
  * [4] O. T. Nartey, G. Yang, J. Wu, and S. K. Asare, “Semi-supervised learning for fine-grained classification with self-training,” _IEEE Access_ , vol. 8, pp. 2109–2121, 2019.
  * [5] K. Sun, W. Wu, T. Liu, S. Yang, Q. Wang, Q. Zhou, Z. Ye, and C. Qian, “FAB: A robust facial landmark detection framework for motion-blurred videos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 5462–5471.
  * [6] A. Kumar, T. K. Marks, W. Mou, Y. Wang, M. Jones, A. Cherian, T. Koike-Akino, X. Liu, and C. Feng, “Luvli face alignment: Estimating landmarks’ location, uncertainty, and visibility likelihood,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 8236–8246.
  * [7] W. Yu, X. Liang, K. Gong, C. Jiang, N. Xiao, and L. Lin, “Layout-graph reasoning for fashion landmark detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 2937–2945.
  * [8] F. Zhang, X. Zhu, H. Dai, M. Ye, and C. Zhu, “Distribution-aware coordinate representation for human pose estimation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 7093–7102.
  * [9] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 4724–4732.
  * [10] R. Bhalodia, L. Kavan, and R. T. Whitaker, “Self-supervised discovery of anatomical shape landmarks,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. Springer, 2020, pp. 627–638.
  * [11] J. M. Noothout, B. D. De Vos, J. M. Wolterink, E. M. Postma, P. A. Smeets, R. A. Takx, T. Leiner, M. A. Viergever, and I. Išgum, “Deep learning-based regression and classification for automatic landmark localization in medical images,” _IEEE Transactions on Medical Imaging_ , 2020.
  * [12] Q. Yao, Z. He, H. Han, and S. K. Zhou, “Miss the point: Targeted adversarial attack on multiple landmark detection,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. Springer, 2020, pp. 692–702.
  * [13] B. A. Griffin and J. J. Corso, “Bubblenets: Learning to select the guidance frame in video object segmentation by deep sorting frames,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 8914–8923.
  * [14] S. Kulhare, S. Sah, S. Pillai, and R. Ptucha, “Key frame extraction for salient activity recognition,” in _International Conference on Pattern Recognition (ICPR)_. IEEE, 2016, pp. 835–840.
  * [15] S. R. Mishra, T. K. Mishra, G. Sanyal, A. Sarkar, and S. C. Satapathy, “Real time human action recognition using triggered frame extraction and a typical CNN heuristic,” _Pattern Recognition Letters_ , vol. 135, pp. 329–336, 2020.
  * [16] Z. Wu, C. Xiong, C.-Y. Ma, R. Socher, and L. S. Davis, “Adaframe: Adaptive frame selection for fast video recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1278–1287.
  * [17] X. Yan, S. Z. Gilani, H. Qin, M. Feng, L. Zhang, and A. Mian, “Deep keyframe detection in human action videos,” _arXiv preprint arXiv:1804.10021_ , 2018\.
  * [18] S. N. Aakur and S. Sarkar, “A perceptual prediction framework for self supervised event segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 1197–1206.
  * [19] L.-C. Chen, R. G. Lopes, B. Cheng, M. D. Collins, E. D. Cubuk, B. Zoph, H. Adam, and J. Shlens, “Semi-supervised learning in video sequences for urban scene segmentation,” _arXiv preprint arXiv:2005.10266_ , 2020.
  * [20] Y. Zhu, K. Sapra, F. A. Reda, K. J. Shih, S. Newsam, A. Tao, and B. Catanzaro, “Improving semantic segmentation via video propagation and label relaxation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 8856–8865.
  * [21] X. Dong, Y. Yang, S.-E. Wei, X. Weng, Y. Sheikh, and S.-I. Yu, “Supervision by registration and triangulation for landmark detection,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)_ , 2020.
  * [22] X. Wang, A. Jabri, and A. A. Efros, “Learning correspondence from the cycle-consistency of time,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 2566–2576.
  * [23] C. Mitchell, P. S. Rahko, L. A. Blauwet, B. Canaday, J. A. Finstuen, M. C. Foster, K. Horton, K. O. Ogunyankin, R. A. Palma, and E. J. Velazquez, “Guidelines for performing a comprehensive transthoracic echocardiographic examination in adults: recommendations from the american society of echocardiography,” _Journal of the American Society of Echocardiography_ , vol. 32, no. 1, pp. 1–64, 2019.
  * [24] J. Lv, X. Shao, J. Xing, C. Cheng, and X. Zhou, “A deep regression architecture with two-stage re-initialization for high performance facial landmark detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 3317–3326.
  * [25] S. Zhu, C. Li, C. Change Loy, and X. Tang, “Face alignment by coarse-to-fine shape searching,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 4998–5006.
  * [26] A. Alansary, O. Oktay, Y. Li, L. Le Folgoc, B. Hou, G. Vaillant, K. Kamnitsas, A. Vlontzos, B. Glocker, B. Kainz _et al._ , “Evaluating reinforcement learning agents for anatomical landmark detection,” _Medical Image Analysis_ , vol. 53, pp. 156–164, 2019.
  * [27] A. Vlontzos, A. Alansary, K. Kamnitsas, D. Rueckert, and B. Kainz, “Multiple landmark detection using multi-agent reinforcement learning,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. Springer, 2019, pp. 262–270.
  * [28] Z. Zhao, Q. Wang, and X. Li, “Deep reinforcement learning based lane detection and localization,” _Neurocomputing_ , vol. 413, pp. 328–338, 2020.
  * [29] A. Gilbert, M. Holden, L. Eikvil, S. A. Aase, E. Samset, and K. McLeod, “Automated left ventricle dimension measurement in 2D cardiac ultrasound via an anatomically meaningful cnn approach,” _arXiv preprint arXiv:1911.02448_ , 2019.
  * [30] Z. Liu, X. Zhu, G. Hu, H. Guo, M. Tang, Z. Lei, N. M. Robertson, and J. Wang, “Semantic alignment: Finding semantically consistent ground-truth for facial landmark detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3467–3476.
  * [31] D. Merget, M. Rock, and G. Rigoll, “Robust facial landmark detection via a fully-convolutional local-global context network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 781–790.
  * [32] R. Valle, J. M. Buenaposada, A. Valdes, and L. Baumela, “A deeply-initialized coarse-to-fine ensemble of regression trees for face alignment,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 585–601.
  * [33] X. Wang, L. Bo, and L. Fuxin, “Adaptive wing loss for robust face alignment via heatmap regression,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 6971–6981.
  * [34] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 3431–3440.
  * [35] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. Springer, 2015, pp. 234–241.
  * [36] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 801–818.
  * [37] M. Tan, R. Pang, and Q. V. Le, “Efficientdet: Scalable and efficient object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 10 781–10 790.
  * [38] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3D convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 4489–4497.
  * [39] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 6450–6459.
  * [40] Y. Yu, Y. Xiong, W. Huang, and M. R. Scott, “Deformable siamese attention networks for visual object tracking,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 6728–6737.
  * [41] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, “Video action transformer network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 244–253.
  * [42] X. Li, S. Liu, S. De Mello, X. Wang, J. Kautz, and M.-H. Yang, “Joint-task self-supervised learning for temporal correspondence,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 318–328.
  * [43] J. Peng, C. Wang, F. Wan, Y. Wu, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, and Y. Fu, “Chained-tracker: Chaining paired attentive regression results for end-to-end joint multiple-object detection and tracking,” in _Proceedings of the European Conference on Computer Vision (ECCV)_. Springer, 2020, pp. 145–161.
  * [44] J. Chen, Z. Li, J. Luo, and C. Xu, “Learning a weakly-supervised video actor-action segmentation model with a wise selection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 9901–9911.
  * [45] K. Nishimura, J. Hayashida, C. Wang, R. Bise _et al._ , “Weakly-supervised cell tracking via backward-and-forward propagation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_. Springer, 2020, pp. 104–121.
  * [46] T. DeVries and G. W. Taylor, “Leveraging uncertainty estimates for predicting segmentation quality,” _arXiv preprint arXiv:1807.00502_ , 2018.
  * [47] A. Jungo, R. Meier, E. Ermis, E. Herrmann, and M. Reyes, “Uncertainty-driven sanity check: application to postoperative brain tumor cavity segmentation,” _arXiv preprint arXiv:1806.03106_ , 2018.
  * [48] Y. Kwon, J.-H. Won, B. J. Kim, and M. C. Paik, “Uncertainty quantification using Bayesian neural networks in classification: Application to biomedical image segmentation,” _Computational Statistics and Data Analysis_ , vol. 142, p. 106816, 2020.
  * [49] R. Robinson, V. V. Valindria, W. Bai, H. Suzuki, P. M. Matthews, C. Page, D. Rueckert, and B. Glocker, “Automatic quality control of cardiac mri segmentation in large-scale population imaging,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_. Springer, 2017, pp. 720–727.
  * [50] T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs, “Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in _International Conference on Information Processing in Medical Imaging (IPMI)_. Springer, 2017, pp. 146–157.
  * [51] F. Liu, Y. Xia, D. Yang, A. L. Yuille, and D. Xu, “An alarm system for segmentation algorithm based on shape model,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 10 652–10 661.
  * [52] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural network,” in _International Conference on Machine Learning (ICML)_ , 2015, pp. 1613–1622.
  * [53] R. Krishnan and O. Tickoo, “Improving model calibration with accuracy versus uncertainty optimization,” in _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [54] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 13 991–14 002.
  * [55] C. Louizos and M. Welling, “Multiplicative normalizing flows for variational bayesian neural networks,” in _International Conference on Machine Learning (ICML)_ , 2017, pp. 2218–2227.
  * [56] A. Atanov, A. Ashukha, D. Molchanov, K. Neklyudov, and D. Vetrov, “Uncertainty estimation via stochastic batch normalization,” _arXiv preprint arXiv:1802.04893_ , 2018.
  * [57] Y. Gal and Z. Ghahramani, “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,” in _International Conference on Machine Learning (ICML)_ , 2016, pp. 1050–1059.
  * [58] A. Kendall, V. Badrinarayanan, and R. Cipolla, “Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding,” in _British Machine Vision Conference (BMVC)_ , 2017.
  * [59] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6402–6413.
  * [60] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped DQN,” in _Advances in Neural Information Processing Systems_ , 2016, pp. 4026–4034.
  * [61] Y. Gal and Z. Ghahramani, “Bayesian convolutional neural networks with Bernoulli approximate variational inference,” _arXiv preprint arXiv:1506.02158_ , 2015.
  * [62] N. Moshkov, B. Mathe, A. Kertesz-Farkas, R. Hollandi, and P. Horvath, “Test-time augmentation for deep learning-based cell segmentation on microscopy images,” _Scientific Reports_ , vol. 10, no. 1, pp. 1–7, 2020\.
  * [63] G. Wang, W. Li, M. Aertsen, J. Deprest, S. Ourselin, and T. Vercauteren, “Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks,” _Neurocomputing_ , vol. 338, pp. 34–45, 2019.
  * [64] A. Kendall and Y. Gal, “What uncertainties do we need in Bayesian deep learning for computer vision?” in _Advances in Neural Information Processing Systems_ , 2017, pp. 5574–5584.
  * [65] Q. V. Le, A. J. Smola, and S. Canu, “Heteroscedastic Gaussian process regression,” in _International Conference on Machine Learning (ICML)_ , 2005, pp. 489–496.
  * [66] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2014, pp. 1725–1732.
  * [67] Y. Liu, C. Shen, C. Yu, and J. Wang, “Efficient semantic video segmentation with per-frame inference,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * [68] F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and A. Sorkine-Hornung, “Learning video object segmentation from static images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2663–2672.
  * [69] J. Li, X. Liu, M. Zhang, and D. Wang, “Spatio-temporal deformable 3D convnets with attention for action recognition,” _Pattern Recognition_ , vol. 98, p. 107037, 2020.


* [Br09] J. Brown, Twisted Yangians and finite W-algebras. Transform. Groups 14 (2009), no. 1, 87–114.
  * [CM93] D.H. Collingwood and W. McGovern, “Nilpotent orbits in semisimple Lie algebras”. Van Nostrand Reinhold, New york, 1993.
  * [DSKV16] A. De Sole,V. G. Kac, D. Valeri, Structure of classical (finite and affine) $\mathcal{W}$-algebras J. Eur. Math. Soc. (JEMS) 18 (2016), no. 9, 1873–1908.
  * [Dr87] V. Drinfeld, Quantum groups American Mathematical Society, Providence, RI, 1987, 798–820.
  * [Dr88] V. Drinfeld, A new realization of Yangians and quantized affine algebras, Soviet Math. Dokl. 36 (1988), 212–216.
  * [EK05] A.G. Elashvili, V.G. Kac Classification of good gradings of simple Lie algebras Amer. Math. Soc. Transl. Ser. 2, 213 Adv. Math. Sci., 56. American Mathematical Society, Providence, RI, 2005, 85–104.
  * [FRT90] L. Faddeev, N. Reshetikhin, L. Takhtadzhyan, Quantization of Lie groups and Lie algebras, Leningrad Math. J. 1 (1990), 193–225.
  * [Fr02] E. Frenkel, Langlands correspondence for loop groups. Cambridge Studies in Advanced Mathematics, 103. Cambridge University Press, Cambridge, 2007.
  * [GG02] W. L. Gan, V. Ginzburg, Quantization of Slodowy slices. Internat. Math. Res. Notices 5 (2002), 243–255.
  * [Gi09] V. Ginzburg, Harish-Chandra bimodules for quantized Slodowy slices. Represent. Theory 13 (2009), 236–271.
  * [GT19a] S. M. Goodwin & L. Topley, Modular finite $W$-algebras, Int. Math. Res. Not. IMRN (2019), no. 18, 5811–5853.
  * [GT19b] by same author, Minimal-dimensional representations of reduced enveloping algebras for$\mathfrak{gl}_{n}$. Compos. Math. 155 (2019), no. 8, 1594–1617.
  * [GT19c] by same author, Restricted shifted Yangians and restricted finite $W$-algebras. arXiv:1903.03079 (2019).
  * [Ja04] J.C. Jantzen, “Nilpotent orbits in representation theory”, in: B. Orsted (ed.), “Representation and Lie theory”, Progr. in Math., 228, 1–211, Birkhäuser, Boston 2004.
  * [Fr07] E. Frenkel, Langlands correspondence for loop groups. Cambridge Studies in Advanced Mathematics, 103. Cambridge University Press, Cambridge, 2007.
  * [GGRW04] I. Gelfand, S. Gelfand, V. Retakh, R. Wilson, Quasideterminants. arXiv:0208.146 (2004).
  * [KWWY14] J. Kamnitzer, B. Webster, A. Weekes, O. Yacobi, Yangians and quantizations of slices in the affine Grassmannian. Algebra Number Theory 8 (2014), no. 4, 857–893.
  * [LPV13] C. Laurent-Gengoux, A. Pichereau, P. Vanhaecke, “Poisson structures”. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], 347. Springer, Heidelberg, 2013.
  * [Lo10a] I. Losev, Quantized symplectic actions and $W$-algebras. J. Amer. Math. Soc. 23 (2010), no. 1, 35–59.
  * [Lo10b] by same author, Quantizations of nilpotent orbits vs 1-dimensional representations of $W$-algebras. arXiv:1004.1669, 2010.
  * [Lo15] by same author, Dimensions of irreducible modules over W-algebras and Goldie ranks. Invent. Math. 200 (2015), no. 3, 849–-923.
  * [Lo22] by same author, Deformations of symplectic singularities and Orbit method for semisimple Lie algebras. Selecta Math. (N.S.) 28 (2022), no. 2, Paper No. 30, 52 pp.
  * [LMM21] I. Losev, L. Mason-Brown, D. Matvieievskyi, Unipotent Ideals and Harish-Chandra Bimodules. arXiv:2108.03453v2 (2021).
  * [LWZ23] K. Lu, W. Wang, W. Zhang, A Drinfeld type presentation of twisted Yangians. arXiv:2308.12254 (2023).
  * [Mo07] A. Molev, “Yangians and classical Lie algebras”. Math. Surveys Monogr., 143 American Mathematical Society, Providence, RI, 2007, xviii+400 pp.
  * [Ol92] G. I. Olshanskii, Twisted Yangians and infinite-dimensional classical Lie algebras, in ‘Quantum Groups (P. P. Kulish, Ed.)’, Lecture Notes in Math. 1510, Springer, Berlin Heidelberg, 1992, pp 103–120.
  * [Pr02] A. Premet, Special transverse slices and their enveloping algebras. With an appendix by Serge Skryabin. Adv. Math. 170 (2002), no. 1, 1–55.
  * [Pr07a] by same author, Enveloping algebras of Slodowy slices and the Joseph ideal. J. Eur. Math. Soc. (JEMS) 9 (2007), no. 3, 487–543.
  * [Pr10] by same author, Commutative quotients of finite W-algebras. Adv. Math. 225 (2010), no. 1, 269–306.
  * [Pr14] by same author, Multiplicity-free primitive ideals associated with rigid nilpotent orbits. Transform. Groups 19 (2014), no. 2, 569–641.
  * [PT14] A. Premet, L. Topley, Derived subalgebras of centralisers and finite W-algebras. Compos. Math. 150 (2014), no. 9, 1485–1548.
  * [RS99] E. Ragoucy, P. Sorba. Yangian Realisations from Finite $\mathcal{W}$-Algebras. Comm. Math. Phys. 203 (1999), 551–572.
  * [Ra01] E. Ragoucy, Twisted Yangians and folded $W$-algebras. Internat. J. Modern Phys. A 16 (2001), no. 13, 2411–2433.
  * [Ser06] J.-P. Serre, Lie algebras and Lie groups. 1964 lectures given at Harvard University. Lecture Notes in Mathematics, 1500. Springer-Verlag, Berlin, 2006.
  * [Sl80] P. Slodowy, “Simple Singularities and Simple Algebraic Groups”. Lecture Notes in Mathematics 815, Springer-Verlag, Berlin Heidelberg, 1980.
  * [To23] L. Topley, One dimensional representations of finite W-algebras, Dirac reduction and the orbit method. Invent. Math. 234 (2023), no.3, 1039–1107.
  * [Ya10] O. Yakimova, On the derived algebra of a centraliser. Bull. Sci. Math., 134 (2010), 579–587.

Contact details:

Lukas Tappeiner<EMAIL_ADDRESS>
Lewis Topley<EMAIL_ADDRESS>
Department of Mathematical Sciences, University of Bath, Claverton Down, Bath
BA2 7AY, United Kingdom.