# Adversarially Trained Convolutional Neural Networks for Semantic
Segmentation of Ischaemic Stroke Lesion using Multisequence Magnetic Resonance
Imaging

Rachana Sathish1, Ronnie Rajan2, Anusha Vupputuri1, Nirmalya Ghosh1 and
Debdoot Sheet1 1R. Sathish, A. Vupputuri, N. Ghosh and D. Sheet are with
Department of Electrical Engineering, Indian Institute of Technology
Kharagpur, India - 721302<EMAIL_ADDRESS>Rajan is with School
of Medical Science and Technology, Indian Institute of Technology Kharagpur,
India - 721302

###### Abstract

Ischaemic stroke is a medical condition caused by occlusion of blood supply to
the brain tissue thus forming a lesion. A lesion is zoned into a core
associated with irreversible necrosis typically located at the center of the
lesion, while reversible hypoxic changes in the outer regions of the lesion
are termed as the penumbra. Early estimation of core and penumbra in ischaemic
stroke is crucial for timely intervention with thrombolytic therapy to reverse
the damage and restore normalcy. Multisequence magnetic resonance imaging
(MRI) is commonly employed for clinical diagnosis. However, a sequence singly
has not been found to be sufficiently able to differentiate between core and
penumbra, while a combination of sequences is required to determine the extent
of the damage. The challenge, however, is that with an increase in the number
of sequences, it cognitively taxes the clinician to discover symptomatic
biomarkers in these images. In this paper, we present a data-driven fully
automated method for estimation of core and penumbra in ischaemic lesions
using diffusion-weighted imaging (DWI) and perfusion-weighted imaging (PWI)
sequence maps of MRI. The method employs recent developments in convolutional
neural networks (CNN) for semantic segmentation in medical images. In the
absence of availability of a large amount of labeled data, the CNN is trained
using an adversarial approach employing cross-entropy as a segmentation loss
along with losses aggregated from three discriminators of which two employ
relativistic visual Turing test. This method is experimentally validated on
the ISLES-2015 dataset through three-fold cross-validation to obtain with an
average Dice score of 0.82 and 0.73 for segmentation of penumbra and core
respectively.

##  I Introduction

Cerebrovascular accident (CVA), more commonly known as stroke, is one of the
most common causes of death and disability in the world [1]. It is
characterized by a sudden focal neurological deficit due to cerebral
infarction caused by poor vascular supply. Ischaemic stroke is the most common
type of stroke [2], caused by occlusion of a blood vessel due to
atherosclerotic stenosis, or by an embolus of atherosclerosis in a large
artery, or may be of cardiac origin. This perfusion deficit causes
irreversible necrosis of a small area of cerebral tissue which becomes
completely devoid of blood supply. This area is called the core of the lesion
and is surrounded by an area of hypoperfusion, which develops a reversible
functional impairment due to temporary hypoxia. If the perfusion is not
restored, this surrounding area which is called the penumbra of the lesion
undergoes delayed apoptosis over the following days and weeks to form a
permanent structural lesion with irreversible loss of function [3]. Early
intervention to restore perfusion to this salvageable area can reverse the
impairment and prevent the extension of the lesion [4]. Therefore, one of the
most crucial investigations into stroke lesion detection is evaluating the
extent of penumbra as compared to the core of the lesion, which helps the
physician decide on interventions like thrombolytic therapy. Multisequence
magnetic resonance imaging (MRI) is classically employed, especially using
perfusion-weighted imaging (PWI) and diffusion-weighted imaging (DWI), where
PWI indicates the region of hypo-perfusion as in core and the penumbra, while
DWI indicates the region of restricted diffusion as in core.

Figure 1: Overview of the proposed method

Challenge: There is a wide range of variability in visual appearance across
these sequences, which along with growth in the number of sequences makes
symptomatic biomarker discovery for clinical use challenging.

Approach: In this context we propose a method for core and penumbra estimation
employing recent developments in deep learning (DL) employing adversarial
learning of a convolution neural network (CNN) for semantic segmentation as
illustrated in Fig. 1. The limited availability of annotated data in stroke
segmentation makes it difficult to train deep neural networks for automated
detection with good generalisability and hence an adversarial approach is
employed.

(a) Phase 1: Train with segmentation loss

(b) Phase 2: Train discriminator 1

(c) Phase 3: Train discriminator 2

(d) Phase 4: Train discriminator 3

(e) Phase 5: Train with adversarial loss

(f) Effect of including adversarial loss during training

Figure 2: Training of the proposed framework using adversarial losses from
three relativistic discriminators in addition to segmentation loss, for
boosting ability to detect file lesions.

Related work: Random Fields (RF) based techniques have been most commonly
employed for penumbra segmentation [5, 6, 7] on the ISLES-2015 dataset. Patch
based stacked sparse auto-encoder for feature learning followed by a support
vector machine (SVM) classifier have also been employed as a data-driven
approach [8]. A multi-scale 11-layer deep 3D CNN with a 3D conditional random
field (CRF) based post-processing was proposed by [9]. More recently, several
other approaches [10] have been proposed based on a modification of UNet [11]
for later versions of the ISLES challenge on stroke lesion segmentation.

Organization: The problem statement is defined in Sec. II with the proposed
solution detailed in Sec. III. Sec IV outlines the various experiments
conducted for validation of performance. Results and their discussion of the
results are presented in Sec. V and Sec. VI concludes the work.

##  II Problem Statement

We consider the task of lesion segmentation to be carried out on a per slice
basis. Given all the MRI sequences for a particular slice, it is arranged as a
$T\times M\times N$ sized tensor $\mathbf{I}$ with $T$ denoting the number of
MRI sequences and $M\times N$ being the spatial size of each slice. We model
the segmentation problem as that of a three class semantic segmentation where
each pixel is classified as belonging to either of the {un-annotated brain
tissue and background} denoted as class 0, or {penumbra} denoted as class 1 or
{core} denoted as class 2.

TABLE I: Performance comparison with baselines Baseline | Dice | Precision | Recall
---|---|---|---
Pen. | Core | Pen. | Core | Pen. | Core
BL1 | $0.44\pm 0.38$ | $0.10\pm 0.17$ | $0.47\pm 0.41$ | $0.14\pm 0.12$ | $0.42\pm 0.37$ | $0.15\pm 0.26$
BL2 | $0.77\pm 0.04$ | $0.33\pm 0.31$ | $0.75\pm 0.06$ | $0.39\pm 0.35$ | $0.80\pm 0.05$ | $0.30\pm 0.29$
BL3 | $0.76\pm 0.05$ | $0.47\pm 0.41$ | $0.78\pm 0.03$ | $0.75\pm 0.22$ | $0.75\pm 0.08$ | $0.50\pm 0.44$
Proposed | $0.82\pm 0.06$ | $0.73\pm 0.05$ | $0.82\pm 0.05$ | $0.80\pm 0.08$ | $0.83\pm 0.08$ | $0.68\pm 0.08$

##  III Method

In Phase 1 the segmentation CNN ($\mathtt{net_{seg}}(\cdot)$) in Fig. 2(a)
predicts $\mathbf{\hat{O}}=\mathtt{net_{seg}}(\mathbf{I})$ where
$\mathbf{\hat{O}}$ is a tensor of size $C\times M\times N$ with $C$ denoting
the number of tissue classes, and the objective is to minimize the cross
entropy loss $J_{Seg}(\cdot)$ between $\mathbf{\hat{O}}$ and $\mathbf{O}$,
where $\mathbf{O}$ is the ground truth. Subsequently in Phase 2 the first
relativistic Turing test discriminator in Fig. 2(b) learns to identify the
ground truth (GT) annotation for penumbra from the segmented map obtained from
($\mathtt{net_{seg}}(\cdot)$) by minimizing the binary cross entropy loss
$J_{D1}(\cdot)$. Similarly in Phase 3 the second discriminator in Fig. 2(c)
learns to identify GT annotation of core from the segmented map by minimizing
the binary cross entropy loss $J_{D2}(\cdot)$. In Phase 4 the discriminator
learns to predict which channel contains the penumbra when fed with a shuffled
channels in the input as in Fig. 2(d) thus minimizing the binary cross entropy
loss $J_{D3}(\cdot)$. Finally in Phase 5 the ($\mathtt{net_{seg}}(\cdot)$)
parameters are optimized to be minimize the adversarial loss
$J_{Adv}(\cdot)=-\alpha J_{D1}(\cdot)-\beta J_{D2}(\cdot)-\gamma
J_{D3}(\cdot)$ and thus learn to be able to produce segmentation which closely
resembles the GT. This impact of incorporating the adversarial losses in
learning is finely visible in Fig. 2(f) where finer details of core and
penumbra are evident in our segmentation approach.

Segmentation CNN: The segmentation CNN ($\mathtt{net_{seg}}(\cdot)$) used is
an encoder-decoder like architecture [12] with the encoder having layer
definitions similar to that of VGG11 [13]. Concatenation of features across
matched layers in the encoder and decoder is present in this architecture
along with the passing of max pooling indices for up-sampling in the decoder.
We additionally add batch normalization after each convolutional layer. The
VGG11 like encoder is initialized with ImageNet pre-trained model weights.

Discriminator networks: The three discriminators ($D1,D2,D3$) are a shallow
convolutional neural network with five convolutional layers each with $4\times
4$ kernels, interleaved with batch normalization layers and leaky ReLU non-
linearity. Sigmoid activation is added to the last layer. The first layer has
32 channels. The number of channels in the subsequent layers is multiplied by
a factor of 2.

##  IV Experiments

Dataset description: This method is experimentally validated using the
Ischaemic Stroke Lesion Segmentation Challenge (ISLES) - 2015
dataset111www.isles-challenge.org/ISLES2015/. We have used the SPES dataset
from the challenge which consists of data from 30 subjects with an average of
70 slices per patient. Seven sequence maps viz. T1c, T2, DWI, CBF, CBV, TTP
and Tmax of $94\times 110$ size on average is available for each patient. We
have trained our network using only the DWI, TTP and Tmax. Whitening transform
is performed on the slices using the mean and standard deviation of
corresponding sequences in the training set. Performance evaluation was
conducted using 3-fold cross-validation. In each fold, 20 subjects were used
for training, 5 for validation and 5 for testing.

Baselines: The following baselines are used for comparison of performance.
BL1: SegNet[14] trained using only segmentation loss. BL2: SegNet trained
using adversarial losses as employed in the proposed framework. BL3: SUMNet
trained using segmentation loss only. The performance of the proposed method
in comparison with the baselines is tabulated in Tab. I. The reported scores
are mean and standard deviation across the three folds on the held-out test
set. The performance of the proposed method is evaluated in comparison with
the baselines using Dice coefficient, precision, and recall.

(a) Tmax

(b) TTP

(c) DWI

(d) GT

(e) SegNet Core - BL1

(f) SegNet Pen. - BL1

(g) SegNet Core - BL2

(h) SegNet Pen. - BL2

(i) SUMNet Core - BL3

(j) SUMNet Pen. - BL3

(k) Proposed Core

(l) Proposed Pen.

Figure 3: Performance comparison for different baselines and our approach for
estimating the core and penumbra using multisequence MRI. (a-e) denotes the
different sequences used which constitutes the input to the network, (d)
represents the GT with black representing the unannotated class 0, gray
representing penumbra class 1 and white representing core as class 2. (e-f)
represent the results of BL1, (g-h) for BL2, (i-j) for BL3 and (k-l) for our
proposed approach. Red denotes under-segmentation, Green denotes over-
segmentation and White denotes proper segmentation.

Training setup: The segmentation network and the discriminators were trained
using Adam optimizer for $200$ epochs, with a learning rate of $0.001$ without
any augmentation of the dataset. In the adversarial loss, $\alpha=0.001$,
$\beta=0.001$ and $\gamma=0.001$.

##  V Results and Discussion

Qualitative results for the proposed method on a sample slice from the test
set is illustrated in Fig. 3 along with the sequences and the ground truth.
The models compared were trained on the same fold of the data. It can be
observed that the performance of segmentation improves significantly with the
proposed adversarial training framework. There is a notable reduction in over-
segmentation and under-segmentation.

Also, it can be seen from Tab. I that the proposed method exhibits the least
standard deviation in terms of dice coefficient across different folds of the
dataset. This shows the generalization capability of the proposed framework
despite being trained with a few annotated samples.

##  VI Conclusion

Early estimation of the extent of penumbra is one of the most crucial aspects
of stroke management. This delineation between core and penumbra helps the
physician decide on thrombolytic therapy that could reverse the damage to the
salvageable tissue. Traditional methods of acute stroke lesion estimation
utilize distinct image processing algorithms and handcrafted machine learning-
based feature extraction techniques, to separately segment core and penumbra
from DWI and PWI respectively. Our proposed method gives a unified framework
that uses both diffusion and perfusion maps as inputs for deep learning based
supervised learning of features to segment both core and penumbra with
comparable accuracy. The limited availability of annotated diffusion and
perfusion maps for the same patient has lead to over-fitting of networks in
training data. This is mitigated by the use of adversarial learning.

## References

  * [1] J. Adamson, A. Beswick, and S. Ebrahim, “Is stroke the most common cause of disability?” _J. Stroke Cerebrovascular Diseases_ , vol. 13, no. 4, pp. 171–177, 2004.
  * [2] C. Warlow, “Epidemiology of stroke,” _The Lancet_ , vol. 352, pp. S1–S4, 1998\.
  * [3] U. Dirnagl, C. Iadecola, and M. A. Moskowitz, “Pathobiology of ischaemic stroke: an integrated view,” _Trends in Neurosciences_ , vol. 22, no. 9, pp. 391–397, 1999.
  * [4] W. Hacke, G. Donnan, C. Fieschi, M. Kaste, J. Broderick, T. Brott, M. Frankel, J. Grotta, J. E. Haley, T. Kwiatkowski _et al._ , “Association of outcome with early stroke treatment: pooled analysis of atlantis, ecass, and ninds rt-pa stroke trials,” _The Lancet_ , vol. 363, no. 9411, pp. 768–774, 2004.
  * [5] O. Maier, B. H. Menze, J. von der Gablentz, L. Häni, M. P. Heinrich, M. Liebrand, S. Winzeck, A. Basit, P. Bentley, L. Chen _et al._ , “Isles 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri,” _Med. Image Anal._ , vol. 35, pp. 250–269, 2017\.
  * [6] H.-L. Halme, A. Korvenoja, and E. Salli, “Isles (siss) challenge 2015: segmentation of stroke lesions using spatial normalization, random forest classification and contextual clustering,” in _Int. Workshop on Brain Lesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries_. Springer, 2015, pp. 211–221.
  * [7] S. M. Reza, L. Pei, and K. Iftekharuddin, “Ischemic stroke lesion segmentation using local gradient and texture features,” in _Proc. Int. Conf. Med. Image Comput. Computer Assisted Interv._ , 2015, pp. 23–26.
  * [8] G. Praveen, A. Agrawal, P. Sundaram, and S. Sardesai, “Ischemic stroke lesion segmentation using stacked sparse autoencoder,” _Comp. Bio. Med._ , vol. 99, pp. 38–52, 2018.
  * [9] K. Kamnitsas, C. Ledig, V. F. Newcombe, J. P. Simpson, A. D. Kane, D. K. Menon, D. Rueckert, and B. Glocker, “Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation,” _Med. Image Anal._ , vol. 36, pp. 61–78, 2017.
  * [10] S. Winzeck, A. Hakim, R. McKinley, J. A. Pinto, V. Alves, C. Silva, M. Pisov, E. Krivov, M. Belyaev, M. Monteiro _et al._ , “Isles 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral mri,” _Frontiers in Neurology_ , vol. 9, 2018.
  * [11] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Int. Conf. Medical Image Comput. Comp. Assist. Interv._ Springer, 2015, pp. 234–241.
  * [12] S. Nandamuri, D. China, P. Mitra, and D. Sheet, “Sumnet: Fully convolutional model for fast segmentation of anatomical structures in ultrasound volumes,” _arXiv preprint arXiv:1901.06920_ , 2019.
  * [13] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _Int. Conf. Learn. Repr._ , 2015.
  * [14] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE Trans. Patt. Anal. Mach. Intell._ , vol. 39, no. 12, pp. 2481–2495, 2017.


#  UNRealNet: Learning Uncertainty-Aware Navigation Features from High-
Fidelity Scans of Real Environments

Samuel Triest1∗111work done as an intern at Field AI, David D. Fan2, Sebastian
Scherer1, and Ali-Akbar Agha-Mohammadi2 * Work done as an intern at Field AI.1
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA. {striest,
<EMAIL_ADDRESS>Field AI, Mission Viejo, CA, USA.

###### Abstract

Traversability estimation in rugged, unstructured environments remains a
challenging problem in field robotics. Often, the need for precise, accurate
traversability estimation is in direct opposition to the limited sensing and
compute capability present on affordable, small-scale mobile robots. To
address this issue, we present a novel method to learn [u]ncertainty-aware
[n]avigation features from high-fidelity scans of [real]-world environments
(UNRealNet). This network can be deployed on-robot to predict these high-
fidelity features using input from lower-quality sensors. UNRealNet predicts
dense, metric-space features directly from single-frame lidar scans, thus
reducing the effects of occlusion and odometry error. Our approach is label-
free, and is able to produce traversability estimates that are robot-agnostic.
Additionally, we can leverage UNRealNet’s predictive uncertainty to both
produce risk-aware traversability estimates, and refine our feature
predictions over time. We find that our method outperforms traditional local
mapping and inpainting baselines by up to $\mathbf{40\%}$, and demonstrate its
efficacy on multiple legged platforms.

##  I Introduction

More and more robots are being deployed in rough-terrain applications such as
construction and disaster response [1, 2, 3, 4, 5]. While these robots (which
are often legged) have the promise of performing work that is potentially
time-consuming and dangerous, these environments often present mobility
challenges due to clutter, dangerous terrain, occlusions, and change in the
environment over time. As such, there is need for fast, precise and
uncertainty-aware traversability analysis.

At a high level, traversability analysis is the process of constructing cost
functions that quantify how desirable it is to reside in a particular region.
This process is performed using on-board sensing (such as lidar or camera) and
on-board compute. Classical approaches to traversability estimation typically
rely on accurate perception to reconstruct the local environment, and apply
hand-crafted rules to determine the cost function sent to planning and control
modules. Unfortunately, this means that traversability estimation modules are
subject to limitations and errors in perception such as odometry drift and
occlusion, and often propagate these errors to planning and control, with
potentially dangerous consequences for the robot and its operators.

Figure 1: We propose UNRealNet (b), a network that produces high-quality,
robot-agnostic navigation features and traversability (c) directly from
pointclouds from an on-board sensor (a). Colormap for risk maps included for
clarity.

Learning presents a promising solution to this problem, as deep neural
networks are able to generate complex predictions that capture abstract
environmental priors (e.g. most rooms have flat floors, boxes have a
particular shape, etc.). However, generating precise ground-truth labels in
high quantities to train such a neural network is challenging. Prior work [6,
7] often leverages aggregation of future sensing (i.e. predict the map at time
$t+k$ from the sensing at time $t$), but such approaches are limited by the
quality of the SLAM algorithm, the accuracy of the sensor, and the mobility
limitations of the robot. The end result is that training labels may be noisy
and incomplete. While simulation may provide a means to address these issues
[8], simulated environments are often overly simplistic and models trained
purely in simulation may not successfully bridge the sim-to-real gap.

Additionally, uncertainty is a critical component of traversability
estimation. In practice, there will be environmental ambiguities (such as
occlusion and out-of-distribution objects) that cannot be resolved by on-board
sensing. It is thus critical that traversability estimation methods be able to
both recognize and address said uncertainty. Learning based-methods have shown
promise in uncertainty estimation [9, 10] for their ability to learn
environmental priors, producing uncertainty estimates that go beyond common
heuristics.

In this work, we leverage a high-precision laser scanner to generate very
accurate and dense reconstructions of several unstructured environments. This
allows us to generate high-quality traversability features without the need to
address common robotic perception limitations such as occlusion or drift in
state estimation. In order to deploy on-robot, we use a label-free learning
pipeline where we generate large amounts of simulated lidar data, and the
corresponding traversability features. We then train a neural network to
predict these features directly from raw pointclouds and demonstrate its
efficacy in producing dense, metric traversability estimates directly from
sparse lidar data.

##  II Related Work

Given its importance in mobile robot autonomy, traversability estimation has
been the focus of much research [11, 12, 13, 14]. Of particular importance to
this work is the approach described by Fankhauser et al. [15] and Chilian et
al. [16], which is a commonly-used baseline for legged traversability. This
method relies on computing several terrain features from a digital elevation
map (DEM), which are combined via Equation 1 into a cost function for
downstream planning. For each feature $f$, the user is required to specify two
parameters $\alpha_{f},f_{crit}$, which specify the importance of that
feature, and a critical value over which the robot should not traverse,
respectively.

$c^{i,j}=\sum_{f}(\alpha_{f}\frac{f^{i,j}}{f_{crit}})$ (1)

While the above methods are well-suited to estimating traversability on fully-
observed, accurately-measured terrain, they are brittle with respect to
sensing errors and limitations, which are omnipresent in practice. Errors in
odometry can lead to terrain surfaces and obstacles being mis-estimated.
Dynamic objects such as people and vehicles can cause false positives.
Occlusions and negative obstacles can result in no sensing data in areas
dangerous to the robot. While methods exist to combat these issues [17, 18,
19], they are often heuristic, and time-consuming to compute.

Recent research in learning for traversability focuses on learning environment
priors to improve traversability estimation in occluded or mis-estimated
regions of the environment. Ramakrishnan et al. [20] train a neural network to
predict occupancy maps from a ego-centric RGB-D images. They demonstrate this
network’s ability to leverage environment priors from indoor environments [21,
22] to improve navigation and exploration. Additional work by Georgakis et al.
[23, 10] extends this idea to semantic mappping and uncertainty-aware
exploration. Work done by Fan et al. [24, 25] estimate traversability using
geometric features of a terrain surface. In contrast to prior work, they
define a mapping from terrain features to distributions of cost, allowing
their navigation behaviors to reason about uncertainty. Stolzle et al. [26]
train a UNet to inpaint elevation maps, given partial elevation maps. They
demonstrate that their network is capable of predicting occluded elevation
values on real-world data. Gan et al. [7] train a neural network to predict
semantics and traversability by building a 3D reconstruction of the
environment using the robot’s trajectory and running traversability analysis
on it. Meng et al. [6] train a network to predict BEV-space semantics and
terrain information from stereo images. Similarly to Gan et al. [7], ground
truth is generated by aggregating pointclouds over time and hand-labeling the
resulting output. They then demonstrate this network’s ability to produce cost
maps for planning by hand-designing a cost function that uses the predicted
semantic and terrain information.

We differentiate ourselves from prior work by leveraging high-precision laser
scanners to generate high-quality ground truth environment models. This gives
us a more accurate and complete pointcloud of the environment to leverage for
learning. In fact, the quality of our environment representation is sufficient
for us to simulate realistic lidar scans for training. Additionally, we
differentiate ourselves from prior work in that we predict a number of
geometric features important to traversability with uncertainty estimates, as
well as a principled way of leveraging this uncertainty in traversability
estimation. This allows our method to be aware of uncertainty and robot-
agnostic, while remaining label-free.

##  III Methodology

Our goal is to train a neural network to predict dense map of traversability
features ($\mathcal{M}\in\mathbb{R}^{C\times W\times H}$, where $C$ is the
number of features, $W$ is the map width, and $H$ is the map height) from a
sparse, instantaneous pointcloud ($\mathcal{P}\in\mathbb{R}^{P\times 3}$,
where $P$ is the number of points). Doing so will result in a local map that
is quick to compute and is robust to occlusions as well as odometry drift. A
high-level summary of our method is presented in Figure 2.

Figure 2: An overview of our algorithm. We first create a high-quality map (1)
from a pointcloud from a laser scanner. We then generate synthetic pointclouds
(2) and their corresponding ground-truth map features (3). We train UNRealNet
to predict the high-quality, robot-agnostic map features from the noisy,
synthetic pointclouds (4, 5). We can then deploy this network on multiple
robots (6) by leveraging a robot-specific traversability function (7). Red
arrows denote network inputs, blue networks denote network outputs, and the
orange arrow denotes the training objective (Equation 3).

###  III-A Creating a High-Quality, Real-World Dataset

####  III-A1 Mapping

To begin, we leverage a high-quality laser scanner (a FARO Focus and Leica
BLK360) and their pointcloud registration software to generate a high-quality
pointcloud of six locations, over four unique environments. This approach is
preferred to relying on offline SLAM algorithms for the following reasons:

  1. 1.

With high precision and accuracy (up to 1 million points per scan, at 1.0mm
precision) [27], the precision of these laser scanners exceeds that of common
sensors in robotics such as lidar and camera.

  2. 2.

The high density of points in these scans makes it possible to generate high
quality data from synthetic viewpoints. This allows us to create training
pairs from places the robot did not (or could not) traverse.

  3. 3.

Near full coverage of the environment can be achieved by not being reliant on
the traversability capabilities of the robot using the sensor.

In order to generate high quality, metric-space traversability labels from
this pointcloud, we run a simple elevation mapping and freespace detection
algorithm. The resulting freespace was then manually refined. This step can
easily be replaced with a more sophisticated algorithm [28, 29] if needed.
Given this elevation map, we can then construct high-quality traversability
features. As is common in prior work [16, 15], we use the step, slope and
roughness of a local neighborhood of elevation cells (the same as Chilian et
al. [16]). We find that the high precision of the pointcloud allows for
relatively nuanced costing of subtle terrain features such as loose wires and
smaller changes in elevation (see Figure 3). The end result of this local
mapping step is that we now have a dense, accurate pointcloud
$\mathcal{P}_{gt}$ and corresponding map features $\mathcal{M}_{gt}$ which can
be leveraged for training.

Figure 3: An example that highlights the value of the high-resolution scanner.
(a) the pointcloud from the scanner. (b) the slope feature of the
corresponding local map. We are able to identify the edges of each wire. (c) A
photo of the environment and (d) the slope feature from the registered
pointcloud. The noise from the lidar and SLAM is too high to see the wires.

####  III-A2 Synthetic pointclouds

A key advantage of using a dense pointcloud of the environment is that it is
possible to generate high-quality synthetic data from many viewpoints that
were not directly experienced by the robot. Thus, in order to generate
training pairs $(\mathcal{P},\mathcal{M})$ from
$\mathcal{M}_{gt},\mathcal{P}_{gt}$ to train our network, we can sample a pose
in freespace $p$ and project $\mathcal{P}_{gt}$ into the frame defined by $p$
(equivalent to the CP method described by Langer et al. [30]). This gives us a
pointcloud $\mathcal{P}$ that resembles what the sensor would have observed
from pose $p$. In order to generate the corresponding set of map features, we
can transform and crop the global map based on $p$. Finally, we can apply a
noising pipeline (including range noise, robot self-hit masks, etc.) to make
$\mathcal{P}$ resemble the output of a robot sensor [31].

Figure 4: A visualization of a datapoint in our dataset, showing (a) the depth
image obtained from simulating lidar in the ground-truth pointcloud
(blue=close, red=far), (b) the depth image obtained by running the original
depth image through the noising pipeline, (c) the BEV projection of the
original pointcloud, (d) the BEV projection of noised pointcloud, and
corresponding crops from the ground-truth elevation map (e) and a sample
traversability map (f).

###  III-B Traversability Prediction with UNRealNet

UNRealNet is a PointPillars-based [32] network designed to predict
traversability features directly from the above synthetic pointclouds.
Similarly to PointPillars, we first define a metric region around the ego-pose
(consisting of a resolution, width and height). All points in this region are
passed through a PointPillars network (consisting of a small PointNet [33],
cell-wise max-pooling, and then a UNet [34], which produces a cell-wise,
factorized Gaussian distribution for several traversability features (Equation
2). This network is supervised by minimizing the negative log-likelihood of
the ground-truth map crop under the predicted map distribution, where
unobserved cells in the ground truth map crop are masked out via a mask
$M_{gt}$ (Equation 3). Note that the network receives a training signal for
cells observed in the label that are not observed in the input, thus requiring
the network to inpaint map features.

$f_{\theta}(\mathcal{P})=\mathcal{N}(\tilde{\mu},\tilde{\sigma}),\quad\tilde{\mu}\in\mathbb{R}^{C\times
W\times H},\quad\tilde{\sigma}\in\mathbb{R}^{C\times W\times H}$ (2)

$\begin{split}\mathcal{L}(\hat{\mathcal{M}},f_{\theta}(\mathcal{P}))&=\frac{M_{gt}}{\sum
M_{gt}}\sum^{W}_{i=0}\sum^{H}_{j=0}\sum^{C}_{f=0}\bigg{[}\\\
&-log(p(\hat{\mathcal{M}}^{f,i,j},\mathcal{N}(\tilde{\mu}^{f,i,j},\tilde{\sigma}^{f,i,j})))\bigg{]}\end{split}\vspace{-0.4cm}$
(3)

###  III-C Probabilistic Traversability Function

While it is possible to compute a single traversability function at train-
time, this approach has a significant downside in that traversability
estimates cannot be changed at deployment time. This limitation would require
re-training of the network for any additional robot platform, or even when
tuning the cost function to specific user needs or risk tolerances. As such,
it is desirable to be able to re-compute traversability using the predicted
features from the network.

It is common practice to use Equation 1 as a cost function for planning, where
the features in question are geometric terrain properties such as local slope
and roughness. However, given our uncertainty estimates, it is more intuitive
to treat traversability as the probability that all map features are below
their critical values (Equation 4).

$p(T^{i,j})=p(T^{i,j}<f_{crit},\forall f)$ (4)

Assuming that each map feature is an independent Gaussian (i.e.
$f_{i,j}=(\mu_{i,j},\sigma_{i,j})$) allows us to simplify Equation 4 into
Equation 5. This formulation only requires the evaluation and multiplication
of several Gaussian cdfs, which are easily computable at the speeds required
for multi-Hz traversability estimation over maps with tens of thousands of
cells. Additionally, this simplifies the burden on the robot operator through
the elimination of the $\alpha$ parameter.

$p(T^{i,j})=\prod_{f}cdf(\mathcal{N}(\mu^{f,i,j},\sigma^{f,i,j}),f_{crit})$
(5)

##  IV Experiments, Results and Analysis

In total, we collected high-quality laser scans of five environments at three
active construction sites. Each scan consists of a single floor (two floors
were collected at two sites). The above data generation process was run on
each environment, yielding a total of around 30,000 train samples and 12,000
test samples (roughly a $70-30$ test-train split). Additionally, a sixth
environment (at a different site) was collected and used solely for testing.
This site yielded an additional 7,000 test samples. UNRealNet was trained to
predict $7m\times 7m$ local maps at a resolution of $5cm$, though maps of
arbitrary size can be predicted using this pipeline. Our traversability
features consisted of the following:

  1. 1.

terrain: 1st percentile of heights in a cell.

  2. 2.

elevation: 99th percentile of heights in a cell, after removing points over a
certain height.

  3. 3.

step: difference between elevation and terrain.

  4. 4.

local slope: slope in a foothold-sized radius.

  5. 5.

local rough: height variance in a foothold-sized radius.

  6. 6.

slope: slope in a robot footprint-sized radius.

  7. 7.

rough: height variance in a robot footprint radius.

Slope, step and roughness were computed in the same way as Fankhauser et al.
[15]. In total, this resulted in a $7\times 140\times 140$ tensor of labels
per datapoint.

###  IV-A Network Performance

In order to evaluate UNRealNet we ask the following:

  1. 1.

Does UNRealNet outperform standard baselines?

  2. 2.

Does the probabilistic traversability function better capture ground-truth
traversability?

####  IV-A1 Does UNRealNet Outperform Standard Baselines?

We first examine UNRealNet’s ability to produce traversability features from
simulated lidar scans. In order to quantify this, we use the ground-truth map
features for datapoints in our test set. Our primary baseline is the mapping
algorithm used to generate features from the aggregated laser scans, but given
the simulated pointcloud used as input. This baseline is chosen because it a
perfectly observed pointcloud run through this baseline would generate zero
error. Thus it serves as a good measure of how well each method can mitigate
the effect of partial sensing.

Since the baseline local mapping method does not produce uncertainty
estimates, we report RMSE between the mean of the predicted distribution and
the ground-truth features for networks. We compare the following ablations:

  1. 1.

(baseline): The local mapping method used to generate ground-truth features,
applied to the synthetic pointcloud. Empty cells are filled with a value of
zero.

  2. 2.

(Telea): baseline, but leverages the inpainting method proposed by Telea et
al. [35] for unknown cells.

  3. 3.

(NS): baseline, but leverages the inpainting method based on Navier-Stokes
proposed by Bertalmio et al. [36] for unknown cells.

  4. 4.

(oracle): Equivalent to baseline, except the placeholder is determined by
computing the RMSE-minimizing value for each channel (using the ground-truth).

  5. 5.

(no aug): An ablation with the omission of the pointcloud augmentation
pipeline. This ablation is provided to quantify the importance of making the
simulated lidar reflect real sensing.

We then evaluate each method via the following metrics:

  1. 1.

(both) avg. RMSE for cells with ground-truth values.

  2. 2.

(observed) avg. RMSE for cells observed in the input pointcloud. This captures
each method’s ability to correct for occlusion-induced mapping errors.

  3. 3.

(inpaint) avg. RMSE for cells which are not observed in the pointcloud, but
have ground-truth values. This captures the ability of each method to
interpolate and extrapolate known information to unknown areas.

Results for synthetic scans on a spatially held-out region of each environment
are reported in Table I. We also report results for the sixth environment
(env6, in red to denote that it was not in the train set) to demonstrate
UNRealNet’s ability to generalize to novel environments. We also report
results from a robot traversal of env6 with the on-board lidar (env6, ouster).
To get proper correspondences with the ground-truth map, we first align
ground-truth cloud and the aggregated pointcloud via ground-plane matching and
ICP. Since there is some drift in the SLAM solution, we further refine each
individual scan by performing another ICP step for each individual scan. In
total, this yields around 800 scans.

Standard deviations are reported over three seeds for net and no aug. Note
that the augmentation pipeline introduces a small amount of variance in the
baselines, which are otherwise deterministic. Also note that the RMSE on
observed cells for all non-learned methods is nearly identical as they copy
the values of observed cells.

Env | Method | RMSE (observed) | RMSE (inpaint) | RMSE (both)
---|---|---|---|---
env1 | net (ours) | $\mathbf{0.1562\pm 0.0001}$ | $\mathbf{0.1987\pm 0.0009}$ | $\mathbf{0.1849\pm 0.0006}$
| no aug | $0.2117\pm 0.0101$ | $0.2618\pm 0.0121$ | $0.2453\pm 0.0113$
| oracle | $0.2817\pm 0.0003$ | $0.2736\pm 0.0003$ | $0.2779\pm 0.0003$
| telea | $0.2817\pm 0.0003$ | $2.0402\pm 0.0017$ | $1.6424\pm 0.0029$
| ns | $0.2817\pm 0.0003$ | $0.3502\pm 0.0003$ | $0.3298\pm 0.0003$
| baseline | $0.2817\pm 0.0003$ | $0.4097\pm 0.0007$ | $0.3740\pm 0.0005$
env2 | net (ours) | $\mathbf{0.2718\pm 0.0030}$ | $\mathbf{0.3608\pm 0.0036}$ | $\mathbf{0.3284\pm 0.0035}$
| no aug | $0.3313\pm 0.0141$ | $0.4208\pm 0.0142$ | $0.3877\pm 0.0137$
| oracle | $0.3731\pm 0.0016$ | $0.4210\pm 0.0007$ | $0.4080\pm 0.0011$
| telea | $0.3731\pm 0.0016$ | $2.0930\pm 0.0154$ | $1.6450\pm 0.0153$
| ns | $0.3731\pm 0.0016$ | $0.4638\pm 0.0015$ | $0.4327\pm 0.0017$
| baseline | $0.3731\pm 0.0016$ | $0.4659\pm 0.0007$ | $0.4350\pm 0.0012$
env3 | net (ours) | $\mathbf{0.1705\pm 0.0021}$ | $\mathbf{0.2370\pm 0.0017}$ | $\mathbf{0.2172\pm 0.0017}$
| no aug | $0.2319\pm 0.0127$ | $0.2902\pm 0.0114$ | $0.2724\pm 0.0111$
| oracle | $0.2669\pm 0.0005$ | $0.2937\pm 0.0004$ | $0.2883\pm 0.0004$
| telea | $0.2669\pm 0.0005$ | $2.3610\pm 0.0058$ | $1.9318\pm 0.0064$
| ns | $0.2669\pm 0.0005$ | $0.3769\pm 0.0003$ | $0.3474\pm 0.0003$
| baseline | $0.2669\pm 0.0005$ | $0.4370\pm 0.0002$ | $0.3940\pm 0.0002$
env4 | net (ours) | $\mathbf{0.1413\pm 0.0008}$ | $\mathbf{0.2252\pm 0.0014}$ | $\mathbf{0.2071\pm 0.0012}$
| no aug | $0.2033\pm 0.0099$ | $0.2845\pm 0.0077$ | $0.2663\pm 0.0078$
| oracle | $0.2629\pm 0.0001$ | $0.2739\pm 0.0003$ | $0.2728\pm 0.0002$
| telea | $0.2629\pm 0.0001$ | $2.8353\pm 0.0028$ | $2.4492\pm 0.0017$
| ns | $0.2629\pm 0.0001$ | $0.4132\pm 0.0005$ | $0.3830\pm 0.0005$
| baseline | $0.2629\pm 0.0001$ | $0.4165\pm 0.0009$ | $0.3868\pm 0.0007$
env5 | net (ours) | $\mathbf{0.1748\pm 0.0016}$ | $\mathbf{0.2792\pm 0.0019}$ | $\mathbf{0.2483\pm 0.0016}$
| no aug | $0.2313\pm 0.0104$ | $0.3260\pm 0.0135$ | $0.2969\pm 0.0114$
| oracle | $0.2439\pm 0.0005$ | $0.3345\pm 0.0005$ | $0.3084\pm 0.0002$
| telea | $0.2439\pm 0.0005$ | $2.4452\pm 0.0238$ | $1.9733\pm 0.0227$
| ns | $0.2439\pm 0.0005$ | $0.3635\pm 0.0008$ | $0.3297\pm 0.0008$
| baseline | $0.2439\pm 0.0005$ | $0.4263\pm 0.0005$ | $0.3761\pm 0.0008$
env6 | net (ours) | $\mathbf{0.1920\pm 0.0012}$ | $\mathbf{0.2708\pm 0.0016}$ | $\mathbf{0.2537\pm 0.0015}$
| no aug | $0.2466\pm 0.0087$ | $0.3156\pm 0.0086$ | $0.3001\pm 0.0079$
| oracle | $0.2985\pm 0.0004$ | $0.2998\pm 0.0003$ | $0.3023\pm 0.0003$
| telea | $0.2985\pm 0.0004$ | $2.8835\pm 0.0098$ | $2.4849\pm 0.0099$
| ns | $0.2985\pm 0.0004$ | $0.4364\pm 0.0005$ | $0.4092\pm 0.0005$
| baseline | $0.2985\pm 0.0004$ | $0.4598\pm 0.0008$ | $0.4260\pm 0.0008$
env6, | net (ours) | $\mathbf{0.2200\pm 0.0014}$ | $\mathbf{0.2666\pm 0.0006}$ | $\mathbf{0.2567\pm 0.0007}$
ouster | no aug | $0.2217\pm 0.0018$ | $0.2744\pm 0.0014$ | $0.2628\pm 0.0014$
| oracle | $0.3229\pm 0.0007$ | $0.2941\pm 0.0000$ | $0.3046\pm 0.0002$
| telea | $0.3229\pm 0.0007$ | $2.4206\pm 0.0213$ | $2.1006\pm 0.0209$
| ns | $0.3229\pm 0.0007$ | $0.4322\pm 0.0020$ | $0.4118\pm 0.0017$
| baseline | $0.3229\pm 0.0007$ | $0.4658\pm 0.0003$ | $0.4374\pm 0.0004$
TABLE I: Results for local mapping accuracy (env6 colored red to signify that
it is a held-out location)

Overall, we observe that UNRealNet outperforms all baselines, for all metrics,
on all evaluation datasets. Interestingly, we observe that this performance
gain is fairly similar for both observed and unobserved cells. This is likely
due to factors such as occlusion and sparse sensing resulting in mis-
estimation of terrain features (e.g. only observing the top part of an object
and mistaking it for the ground). Furthermore, we observe that both inpainting
baselines perform quite poorly, especially when compared to the oracular
baseline. This is likely due to the relative sparsity of the input data (we
only have returns for a relatively low fraction of cells, compared to
traditional CV datasets for inpainting). Furthermore, data is sparser towards
the edges of the map, meaning that extrapolation, in addition to
interpolation, is necessary to achieve low RMSE. Finally, we observe that
adding augmentations similar to the actual robot payload improves overall
performance.

####  IV-A2 Does the Probabilistic Traversability Function Better Capture
Ground-Truth?

In order to evaluate the efficacy of the probabilistic traversability
function, we compare the accuracy to a ground-truth traversability derived
from crops of the global map. For each data point, a random set of
traversability threshods was generated. These traversability thresholds were
used to generate a traversability map by checking if any ground-truth features
exceeded their corresponding threshold, per cell. We then evaluated the
proposed traversability function (Prob.) and the one used by Chilian and
Fankhauser [16, 15] (Det.), on the predicted traversability features from the
network. We report the MAE of the predicted traversability value to the ground
truth map. Results are presented in Table II for the test set, over a random
sampling of traversability thresholds.

Method | MAE
---|---
Det. [16, 18] | $0.3000\pm 0.0546$
Prob. (ours) | $\mathbf{0.2326\pm 0.0694}$
TABLE II: Probabilistic traversability experiment results. Figure 5:
Traversability predictions using the same network output on a sample in env6.
(a) A crop from the FARO-scanned environment. There is a pile of wires in the
middle. (b) The ground-truth elevation map. (c) The traversability map
generated from parameters for a Spot. (d) The traversability map generated
from parameters for an AlienGo. (e) A mosaic of traversability maps by varying
the local slope (y axis) and robot slope (x axis) thresholds.

Overall, we observe that the uncertainty-aware traversability function is
closer to the ground-truth traversability. This is likely because
incorporating uncertainty allows the traversability estimation to be less
sensitive to mispredictions of features (due to assigning high uncertainty to
said areas).

###  IV-B Hardware Experiments

We also ran a number of experiments on multiple legged platforms to
demonstrate the efficacy of our method in practice. Our traversability
pipeline was run on both a Boston Dynamics Spot and Unitree Aliengo, each with
a custom payload similar to Bouman et al. [37].

Figure 6: Case study for elevation prediction. (a) An FPV image of the scene.
(b) The input pointcloud (red=low, purple=high) (c) The learned, Kalman-
filtered elevation map from UNRealNet (purple=low, red=high) has inpainted the
floor behind the wall, as well as the construction material and pillars. (d)
Elevation map from STEP [24] is incomplete due to occlusions. Figure 7:
Additional qualitative results from a seventh environment. We are able to
capture detailed traversability estimates including small plugs (top), shallow
negative obstacles (middle) and complex geometry (bottom). Correspondences
between FPV and BEV are circled in blue. FPV viewpoints were taken at
different times than BEV to keep the object of interest in the field of view.

####  IV-B1 Robot-Agnosticism

Presented in Figure 5 are multiple traversability maps generated from the same
UNRealNet prediction from env 6. Our traversability function is able to take
into account robot-specific mobility limitations and produce a corresponding
cost. Of note is that UNRealNet is able to detect and evaluate traversability
of a small pile of wires, which becomes more costly the more stringent the
traversability thresholds become.

####  IV-B2 Kalman Filter

When deploying on the robot, we can fuse multiple predictions together to get
a higher-quality map (at the cost of becoming reliant on accurate odometry).
This is accomplished by leveraging the uncertainty in the neural network
predictions ($\mu,\sigma$, per feature, per cell) to run a one-dimensional
Kalman Filter (Equation 6, where $\hat{\mu},\hat{\sigma}$ are the running
estimates of the Kalman filter) [38]. Traversability $\hat{T}^{i,j}$ can then
be computed from the Kalman-filtered estimates $(\hat{\mu},\hat{\sigma})$. We
observe that the Kalman filter generally improves the estimation of terrain
features.

$\hat{\mu}^{\prime}=\frac{\hat{\mu}\sigma^{2}+\mu\hat{\sigma}^{2}}{\hat{\sigma}^{2}+\sigma^{2}},\quad\hat{\sigma}^{\prime}=\sqrt{\frac{\hat{\sigma}^{2}\sigma^{2}}{\hat{\sigma}^{2}+\sigma^{2}}}$
(6)

####  IV-B3 Inference Speed

We first report the speed of our method in comparison to STEP [24]. Both
methods were run with an AMD Ryzen 9 5900hx CPU and NVIDIA 3060 Laptop GPU. We
observe that although slower, the runtime of our method is comparable to STEP
(around $7hz$ compared to STEP’s $13hz$, and $10hz$ rate of pointclouds from
the lidar). The majority of the runtime in our method is consumed by
preprocessing the pointcloud for the network, and converting the network’s
output into a grid map. We argue that much of this runtime can be reduced by
re-implementing our Python inference node in C++, and leveraging tools such as
TensorRT [39]. Additionally, we note that $7hz$ inference is fast enough for
legged platforms with a top speed of under $2m/s$, though this may not be
sufficient for faster platforms.

####  IV-B4 Case Studies

One particular case study is presented in Figure 6. We compare the learned
elevation map to the elevation map produced by STEP. We can observe that
UNRealNet has learned to extrapolate using environmental priors, in that it
successfully inpaints the crate, pillars, and the floor in between.
Additionally, it correctly predicts the floor behind the wall. We also provide
additional qualitative results from a seventh environment (for which we didn’t
have a ground-truth pointcloud) in Figure 7 that highlight additional nuances
in our traversability estimation.

##  V Conclusion and Future Work

We presented a method for learning traversability information by generating
synthetic sensing from high-quality scans of real-world environments. We
demonstrate that this method results in improved local mapping performance and
demonstrate its efficacy for traversability estimation on hardware.

Future work includes generating better training labels, such as
correspondences between observed terrain features and proprioception [40, 41,
42] or demonstrations [43, 44, 45]. The work done by Meng et al. [6] provides
a good roadmap for how to incorporate semantics into a similar mapping
pipeline. Additionally, datasets such as Matterport 3d [21] and TartanAir [8]
can be leveraged to obtain joint geometric and semantic labels without any
additional human labeling (although fine-tuning on real data may be required).
Lastly, combining UNRealNet with object detection and tracking algorithms
would allow for increased performance in the presence of dynamic obstacles.

##  VI Acknowledgements

The authors thank the rest of the Field AI team for their help in collecting
data and testing on hardware.

## References

  * [1] C. D. Bellicoso, M. Bjelonic, L. Wellhausen _et al._ , “Advances in real-world applications for legged robots,” _Journal of Field Robotics_ , vol. 35, no. 8, pp. 1311–1326, 2018.
  * [2] K. Afsari, S. Halder, M. Ensafi _et al._ , “Fundamentals and prospects of four-legged robot application in construction progress monitoring,” _EPiC Series in Built Environment_ , vol. 2, pp. 274–283, 2021.
  * [3] C. Gehring, P. Fankhauser, L. Isler _et al._ , “Anymal in the field: Solving industrial inspection of an offshore hvdc platform with a quadrupedal robot,” in _Field and Service Robotics: Results of the 12th International Conference_. Springer, 2021, pp. 247–260.
  * [4] A. Agha, K. Otsu, B. Morrell _et al._ , “Nebula: Quest for robotic autonomy in challenging environments; team costar at the darpa subterranean challenge,” _arXiv preprint arXiv:2103.11470_ , 2021.
  * [5] S. Scherer, V. Agrawal, G. Best _et al._ , “Resilient and modular subterranean exploration with a team of roving and flying robots,” _Field Robotics Journal_ , pp. 678–734, May 2022.
  * [6] X. Meng, N. Hatch, A. Lambert _et al._ , “Terrainnet: Visual modeling of complex terrain for high-speed, off-road navigation,” _arXiv preprint arXiv:2303.15771_ , 2023.
  * [7] L. Gan, Y. Kim, J. W. Grizzle _et al._ , “Multitask learning for scalable and dense multilayer bayesian map inference,” _IEEE Transactions on Robotics_ , vol. 39, no. 1, pp. 699–717, 2022.
  * [8] W. Wang, D. Zhu, X. Wang _et al._ , “Tartanair: A dataset to push the limits of visual slam,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 4909–4916.
  * [9] A. G. Kendall, “Geometry and uncertainty in deep learning for computer vision,” Ph.D. dissertation, University of Cambridge, UK, 2019.
  * [10] G. Georgakis, B. Bucher, A. Arapin _et al._ , “Uncertainty-driven planner for exploration and navigation,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 11 295–11 302.
  * [11] S. R. Team, “Stanford racing team’s entry in the 2005 darpa grand challenge,” _Published on DARPA Grand Challenge_ , 2005.
  * [12] L. D. Jackel, E. Krotkov, M. Perschbacher _et al._ , “The darpa lagr program: Goals, challenges, methodology, and phase i results,” _Journal of Field robotics_ , vol. 23, no. 11-12, pp. 945–973, 2006.
  * [13] P. Papadakis, “Terrain traversability analysis methods for unmanned ground vehicles: A survey,” _Engineering Applications of Artificial Intelligence_ , vol. 26, no. 4, pp. 1373–1385, 2013.
  * [14] “Arl autonomy stack,” 2022, accessed 13-September-2022. [Online]. Available: https://www.arl.army.mil/business/collaborative-alliances/current-cras/sara-cra/sara-overview/
  * [15] P. Fankhauser, M. Bjelonic, C. D. Bellicoso _et al._ , “Robust rough-terrain locomotion with a quadrupedal robot,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 5761–5768.
  * [16] A. Chilian and H. Hirschmüller, “Stereo camera based navigation of mobile robots on rough terrain,” in _2009 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2009, pp. 4571–4576.
  * [17] J. Larson and M. Trivedi, “Lidar based off-road negative obstacle detection and analysis,” in _2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2011, pp. 192–197.
  * [18] P. Fankhauser and M. Hutter, “A universal grid map library: Implementation and use case for rough terrain navigation,” in _Robot Operating System (ROS)_. Springer, 2016, pp. 99–120.
  * [19] T. Miki, L. Wellhausen, R. Grandia _et al._ , “Elevation mapping for locomotion and navigation using gpu,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 2273–2280.
  * [20] S. K. Ramakrishnan, Z. Al-Halah, and K. Grauman, “Occupancy anticipation for efficient exploration and navigation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_. Springer, 2020, pp. 400–418.
  * [21] A. Chang, A. Dai, T. Funkhouser _et al._ , “Matterport3d: Learning from rgb-d data in indoor environments,” _arXiv preprint arXiv:1709.06158_ , 2017\.
  * [22] F. Xia, A. R. Zamir, Z. He _et al._ , “Gibson env: Real-world perception for embodied agents,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 9068–9079.
  * [23] G. Georgakis, B. Bucher, K. Schmeckpeper _et al._ , “Learning to map for active semantic goal navigation,” _arXiv preprint arXiv:2106.15648_ , 2021\.
  * [24] D. D. Fan, K. Otsu, Y. Kubo _et al._ , “Step: Stochastic traversability evaluation and planning for risk-aware off-road navigation,” _arXiv preprint arXiv:2103.02828_ , 2021.
  * [25] D. D. Fan, A.-A. Agha-Mohammadi, and E. A. Theodorou, “Learning risk-aware costmaps for traversability in challenging environments,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 1, pp. 279–286, 2021.
  * [26] M. Stölzle, T. Miki, L. Gerdes _et al._ , “Reconstructing occluded elevation information in terrain maps with self-supervised learning,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 1697–1704, 2022\.
  * [27] “Faro focus datasheet,” 2023, accessed 04-September-2023. [Online]. Available: https://media.faro.com/-/media/Project/FARO/FARO/FARO/Resources/2_TECH-SHEET/TechSheet_Focus_Laser_Scanner/TechSheet_Focus_Laser_Scanner_EN.pdf?rev=1291d7f5ef814fe99d471dde60321833
  * [28] W. Zhang, J. Qi, P. Wan _et al._ , “An easy-to-use airborne lidar data filtering method based on cloth simulation,” _Remote sensing_ , vol. 8, no. 6, p. 501, 2016.
  * [29] M. Velas, M. Spanel, M. Hradis, and A. Herout, “Cnn for very fast ground segmentation in velodyne lidar data,” in _2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC)_. IEEE, 2018, pp. 97–103.
  * [30] F. Langer, A. Milioto, A. Haag _et al._ , “Domain transfer for semantic segmentation of lidar data using deep neural networks,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 8263–8270.
  * [31] “Os1 mid-range high-resolution imaging lidar,” 2021, accessed 01-September-2023. [Online]. Available: https://data.ouster.io/downloads/datasheets/datasheet-revd-v2p0-os1.pdf
  * [32] A. H. Lang, S. Vora, H. Caesar _et al._ , “Pointpillars: Fast encoders for object detection from point clouds,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 12 697–12 705.
  * [33] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [34] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [35] A. Telea, “An image inpainting technique based on the fast marching method,” _Journal of graphics tools_ , vol. 9, no. 1, pp. 23–34, 2004.
  * [36] M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in _Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001_ , vol. 1. IEEE, 2001, pp. I–I.
  * [37] A. Bouman, M. F. Ginting, N. Alatur _et al._ , “Autonomous spot: Long-range autonomous exploration of extreme environments with legged locomotion,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 2518–2525.
  * [38] P. Fankhauser, M. Bloesch, and M. Hutter, “Probabilistic terrain mapping for mobile robots with uncertain localization,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3019–3026, 2018.
  * [39] “Nvidia tensorrt,” 2023, accessed 01-September-2023. [Online]. Available: https://developer.nvidia.com/tensorrt
  * [40] M. G. Castro, S. Triest, W. Wang _et al._ , “How does it feel? self-supervised costmap learning for off-road vehicle traversability,” _arXiv preprint arXiv:2209.10788_ , 2022.
  * [41] X. Cai, M. Everett, J. Fink, and J. P. How, “Risk-aware off-road navigation via a learned speed distribution map,” _arXiv preprint arXiv:2203.13429_ , 2022.
  * [42] J. Frey, M. Mattamala, N. Chebrolu _et al._ , “Fast traversability estimation for wild visual navigation,” _arXiv preprint arXiv:2305.08510_ , 2023.
  * [43] J. A. Bagnell, D. Bradley, D. Silver _et al._ , “Learning for autonomous navigation,” _IEEE Robotics & Automation Magazine_, vol. 17, no. 2, pp. 74–84, 2010.
  * [44] M. Wigness, J. G. Rogers, and L. E. Navarro-Serment, “Robot navigation from human demonstration: Learning control behaviors,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 1150–1157.
  * [45] S. Triest, M. G. Castro, P. Maheshwari _et al._ , “Learning risk-aware costmaps via inverse reinforcement learning for off-road navigation,” _arXiv preprint arXiv:2302.00134_ , 2023.