# Learning to Assist and Communicate with Novice Drone Pilots for Expert Level
Performance

Kal Backman, Dana Kulić and Hoam Chung Authors are with the Department of
Electrical and Computer Systems Engineering and the Department of Mechanical
and Aerospace Engineering, Monash University. [Kal.Backman1, Dana.Kulic,
<EMAIL_ADDRESS>
###### Abstract

Multi-task missions for unmanned aerial vehicles (UAVs) involving inspection
and landing tasks are challenging for novice pilots due to the difficulties
associated with depth perception and the control interface. We propose a
shared autonomy system, alongside supplementary information displays, to
assist pilots to successfully complete multi-task missions without any pilot
training. Our approach comprises of three modules: (1) a perception module
that encodes visual information onto a latent representation, (2) a policy
module that augments pilot’s actions, and (3) an information augmentation
module that provides additional information to the pilot. The policy module is
trained in simulation with simulated users and transferred to the real world
without modification in a user study ($\mathbf{n=29}$), alongside
supplementary information schemes including learnt red/green light feedback
cues and an augmented reality display. The pilot’s intent is unknown to the
policy module and is inferred from the pilot’s input and UAV’s states. The
assistant increased task success rate for the landing and inspection tasks
from [16.67% & 54.29%] respectively to [95.59% & 96.22%]. With the assistant,
inexperienced pilots achieved similar performance to experienced pilots.
Red/green light feedback cues reduced the required time by 19.53% and
trajectory length by 17.86% for the inspection task, where participants rated
it as their preferred condition due to the intuitive interface and providing
reassurance. This work demonstrates that simple user models can train shared
autonomy systems in simulation, and transfer to physical tasks to estimate
user intent and provide effective assistance and information to the pilot.

###### Index Terms:

Cognitive Human-Robot interaction, Deep Learning in Robotics and Automation,
Aerial Systems: Perception and Autonomy, Shared Autonomy

##  I Introduction

Unmanned aerial vehicles (UAVs) are renowned for their mobility, often
deployed in search and rescue [1, 2, 3] and inspection [4, 5, 6] related tasks
due to their ability to manoeuvre in full 3D space. However this
manoeuvrability comes at a cost of increased teleoperation complexity due to
difficulties associated with pilots’ relative depth perception of nearby
objects [7, 8] and control input mapping to relative UAV dynamic state
changes. Due to these challenges it is difficult for novice pilots to
successfully complete the aforementioned tasks.

Autonomous solutions have been proposed for such tasks [4, 5, 6], however
often require that the structure of the environment be known a priori, or
contain a set of fixed, known mission objectives. The main limitation of fully
autonomous solutions is their inability to dynamically adapt their objective
in response to external stimuli within the environment that is not predefined
by their developers due to the associated difficulty of replicating high-level
human decision making and general artificial intelligence [9, 10]. Therefore
teleoperation control schemes are preferred in real-life UAV operations over
fully autonomous solutions [11] to take advantage of high-level human decision
making, despite the requirement of expert pilots.

Assistance strategies include the use of shared autonomy, which combines the
control inputs of human pilots with that of artificial intelligence to
collaboratively complete a set of objectives. Three main challenges arise when
developing shared autonomy systems: inferring the intent of the user,
providing control outputs to complete the inferred objective and deciding how
and what information should be communicated back to the user. The first
challenge, inferring the intent of the user, is performed by observing the
user’s actions within the context of the observable environment. Although
inferring intent implicitly poses the risk of incorrect goal estimation
leading to a misalignment of objectives between the AI and user, users often
prefer implicit intent estimation methods due to their intuitiveness and
reduction in cognitive workload [8, 12].

For the second challenge, the automated assistant must deliver its control
outputs considering its uncertainty about the user’s intent. Acting too early
risks taking an incorrect action not aligned with the user, while waiting to
build sufficient confidence in the user’s intent before acting can lead to
delayed assistance and task failure. Further issues arise with how much
control should the assistant exert over the system. Providing insufficient
assistance can lead to task failure while excessive control deteriorates team
effectiveness in collaborative tasks [13].

For the third challenge, communication feedback promotes transparency of the
shared autonomy system, providing increased observability and predictability
of system behaviour [14]. However developing natural feedback communication
channels that do not hinder a user’s control input capabilities, prevent loss
of focus from context switching and are designed for environments with high
auditory noise is difficult.

Prior works on UAV systems focus on autonomous landing [15, 16, 17, 18] or
inspection [6, 19, 20, 21] tasks which rely on predefined mission objectives.
Of the limited shared autonomy works, none yet deal with multi-task missions
containing multiple ambiguous goals and instead focus on single goal
inspection [22] or landing [23] tasks, or are restricted to obstacle avoidance
for use in inspection tasks [11, 24]. Our prior work [25, 8] proposed a shared
autonomy solution capable of providing assistance under ambiguity of the
pilot’s intent in unknown environments, but were limited to landing tasks and
a restricted control scheme where the UAV’s yaw is locked. Our prior
experiments indicated that participants were able to significantly improve
task performance, however sometimes struggled to discern the intent of the
assistant or what was expected of them as a pilot, where lack of communication
feedback led to a sense of uncertainty. Therefore in this paper we propose a
system capable of UAV shared autonomy for multi-task missions, while
introducing learnt feedback communication and information augmentation to
further enhance user experience and task performance.

The proposed work is capable of delivering beneficial assistance to pilots
where the pilot’s objective, intent and environment are not known a priori,
allowing the system to be transferred to new operational environments. The
system is trained purely on simulated data, removing the need for human
supervision during training which is time costly and hazardous for the robot
and human overseer. Despite being trained on synthetic data with simulated
users, the model is directly transferrable to the physical domain without
modification, able to assist both novice and expert human pilots to achieve
equal performance. The approach simultaneously achieves task-relevant
assistance and communication feedback by jointly learning the two components
into a single model, assisting naïve pilots who are unaware of the inner
workings of the assistant and without pre-establishing operation conventions
or user preferences.

##  II Related work

_Autonomous Drone Inspection_ : A number of prior works propose algorithms for
autonomous drone inspection in specific industrial [20] and infrastructure [6,
21, 19] inspection tasks. These works generate trajectories or propose optimal
viewpoints for inspection related tasks however are limited to being either
validated in simulation, relying on a specific structured environment or
limited to a small immediately visible search area.

_Shared Autonomy Drone Inspection_ : Previous shared autonomy works for UAV
inspection related tasks are limited. [22] demonstrates a physical UAV for
pole-like object inspections by transforming pilot inputs into cylindrical
coordinates. [11] implements a semi-autonomous UAV for GPS denied search and
rescue missions using the pilot’s inputs to create waypoints for an autonomous
safe trajectory path planner to follow. [24] flies a simulated UAV through a
virtual forest-like environment using haptic feedback and the closest safe
action to the pilot’s input computed from a control barrier function. The
previous shared autonomy approaches are limited in handling ambiguity in the
pilot’s intent, where [22] focuses on single goal environments, while [11, 24]
are delegated to obstacle avoidance tasks irrespective of the pilot’s
objective. The prior works have yet to be validated in physical user studies,
where [22, 11] performs physical demonstrations while [24] hosts a simulated
user study.

_Autonomous Drone Landing_ : Similar to inspection related tasks, many prior
works related to UAV landing focus on fully autonomous solutions over shared
autonomy. Deep-reinforcement learning methods for UAV landing have been
proposed in [15, 16, 17, 18], where [15] uses a sequence of deep Q-networks to
search and land on physical targets. [16] uses PPO to autonomously detect and
land at undefined safe landing zones for physical emergency landing
situations. Both works [15, 16] are limited to a set of discrete action
outputs, limiting the manoeuvrability of the UAV.

Continuous action approaches have been demonstrated in [17, 18], where [17]
uses DDPG [26] to land a physical drone on a moving platform by outputting a
target velocity along the horizontal plane at a constant descent. Similarly,
[18] uses DDPG to land on a simulated moving platform by outputting a target
velocity along the horizontal plane and a simple heuristic model to control
for vertical descent. Both [17, 18] are require the position of the target
platform to be directly observable within their respective input state spaces
and are limited to horizontal movement to counter the instability of
continuous action space reinforcement learning.

Figure 1: System overview. (Top) Assistant network architecture. (Bottom left)
Information augmentation using Microsoft HoloLens. (Bottom right) Example of a
potential multi-task mission.

_Shared Autonomy Drone Landing_ : Shared autonomy UAV landing approaches
alleviate the need to explicitly define the landing zone due to the AI
assistant inferring the pilot’s goal from observations of the their actions.
[23] implements a shared autonomy system to assist users in safely landing in
the lunar lander game where the location of safe landing zones is only known
to the user. The system uses a model free DQN to provide assistance to
participants in a user study ($n=12$). However when applied to a physical
drone ($n=4$), the landing pad location is included within the network’s state
space and is limited to a discrete set of locations.

Our initial work [25] demonstrated a shared autonomy system to assist pilots
landing a simulated UAV on one of several platforms. The assistant comprised
of two modules; a perception module responsible for encoding visual data onto
a compressed latent representation and a policy module which was trained on
simulated users using the reinforcement algorithm TD3 [27] to augment the
pilot’s actions. The assistant was not aware of the pilot’s true goal nor the
structure of the environment, where the pilot’s goal must be inferred from
observations of their actions in the context of the observed environment. The
approach was validated on human pilots ($n=33$) in a simulated environment and
demonstrated in a physical environment.

Our subsequent work [8] further built on [25] and evaluating in a physical
user study. The assistant in [8] comprised of a perception and policy module
which were both trained purely on synthetic data. Compared to [25], the
perception module fused two downwards facing RGB-D cameras onto a consolidated
latent representation using a novel camera projection model that is only
attainable in simulation for greater field of view. The policy module
introduced an LSTM-cell to address the inconsistency concerns participants had
in [25], where policy convergence time and performance were significantly
improved by providing hidden information of the simulated user’s intent to the
critic only. The simulated user’s parametric model was expanded to take into
account pilots’ joystick thumb movements and task complexity was increased due
to additional safe landing criteria and a greater diversity in environment
structures. The assistant successfully transferred to reality and was
validated in a physical user study with human pilots ($n=28$) where it
increased landing success rate from 51.4% in the unassisted condition to 98.2%
in the assisted condition. Participants’ stated that their most preferred
aspect was the intuitive design, however the non-intrusive behaviour made it
difficult to discern the intent of the assistant or what was expected of them
as a pilot, where the lack of communication feedback led to a sense of
uncertainty.

As of yet there are no prior works to the authors’ knowledge that enable
shared autonomy control of UAVs for multi-task missions.

##  III Problem Statement and Model Overview

We consider an unknown environment in which the UAV is operating. The
environment contains several points of interest to the pilot (potential goals)
where a task is to be performed. The onboard assistant is unaware of the
global structure of the environment, capable of only sensing the local area
with onboard sensors. The assistant is unaware of the pilot’s goal, which must
be deciphered from the pilot’s joystick inputs in the context of the
observable environment. Together with a goal estimate, the assistant is
required to estimate and provide assistance with the type of task the pilot
needs to perform on the goal. To further enhance teamwork, the assistant can
not only provide task specific controls that influence the physical state of
the UAV, but can also communicate with the pilot by providing indication about
the potential success of the task or to prompt them to take a desired action.

Our approach is summarised in Fig. 1 and is comprised of three components: (i)
a perception module that is responsible for compressing high-dimensional
visual information onto a latent representation, (ii) a policy module that is
responsible for providing control and communicative outputs to assist the
pilot in successfully completing tasks, and (iii) an information augmentation
module that provides additional information to the pilot in discerning the
state of the UAV. The perception module and the assistant are trained purely
on synthetic data, learning to provide assistance with simulated users based
on a parametric model.

Multi-task missions comprise of landing and inspection tasks on points of
interest named _platforms_. These platforms are indistinguishable from one
another and have a defined orientation, which is marked by a small
checkerboard pattern at the top-left corner. To achieve success for an
inspection task, an image must be captured where all four corners of the goal
platform are visible to the downwards facing cameras, with the image correctly
aligned to the platform below a certain yaw threshold. A landing task is
considered successful given the UAV remains at rest atop of the goal platform
while the UAV is correctly aligned to the platform below a certain yaw
threshold.

##  IV Perception module

The purpose of the perception module is to allow the assistant to visually
perceive the environment so that potential goal targets can be identified. The
perception module aims to do this by encoding high-dimensional visual
information onto a low-dimensional latent vector for use in the policy
module’s input state-space. As the policy module accumulates experience by
interacting with the simulated user within the environment in real time, it is
costly to explore large state-spaces, therefore a reduction in the
dimensionality of the input state-space leads to accelerated policy
convergence [28].

Learning from the low-dimensional mean latent vector $\mu$ aims to reduce
policy convergence times by reducing the time required to the explore the
input state-space. However, given a data input $x_{k}$ which is derived from
sensor readings that capture a high-dimensional and complex system such as
cameras, artificially generating $x_{k}$ in simulation becomes computationally
expensive. From the data efficiency point of view, it is preferred that data
generated in simulation avoid computationally expensive high-frequency details
and capture only the raw essence of what they are intended to represent.
However, learning from simplistic simulated data sources is not indicative of
the complexity and diversity of real life data sources and can later cause the
learned policy to fail to bridge the sim-to-real gap. In order to maximise
both the computational efficiency of learning from simple data source
simulations and the robustness in transferring to reality with high-detailed
and diversified data sources, we aim to encode both data sources onto an
identical latent representation such that they can be used interchangeably.

Given a data source $\bar{x}_{k}$ which represents the basic low-detail
information of $x_{k}$, encoder $q_{k}$ maps both $\bar{x}_{k}$ and $x_{k}$
onto a normally distributed latent vector with means $\bar{\mu}$ & $\mu$ and
variances $\bar{\sigma}^{2}$ & $\sigma^{2}$ respectively. As both
$\bar{x}_{k}$ and $x_{k}$ represent the same observation but in varying
detail, both latent samples
$\bar{z}\sim\mathcal{N}(\bar{\mu},\bar{\sigma}^{2})$ &
$z\sim\mathcal{N}(\mu,\sigma^{2})$ are reconstructed using decoder $p_{l}$ to
generate a simplified representation of $y_{l}$ which is denoted as
$\bar{y}_{l}$. Both $\bar{x}_{k}$ and $x_{k}$ are reconstructed to
$\bar{y}_{l}$ and not $\bar{y}_{l}$ & $y_{l}$ respectively due to
$\bar{x}_{k}$ not containing the high-detail information required to
accurately reconstruct $y_{l}$, which would result in both latent
representations $\bar{\mu}$ & $\mu$ not being identical. To further ensure
consistency between latent representations $\bar{\mu}$ & $\mu$, the cosine
similarity loss is introduced as a regularizing term such that both latent
vectors are co-linear with each other. The introduction of the cosine
similarity loss transforms the CM-VAE architecture into a cross-modal
similarity variational auto-encoder (CM-SVAE).

Information regarding the implementation of the CM-SVAE architecture can be
found in Appendix A.

##  V Policy module

The purpose of the policy module is to assist pilots in successfully
completing multi-task missions from a set of two unique tasks, inspection and
landing. The policy module achieves this by outputting a target velocity and
yaw rate that is subsequently averaged with the pilot’s current joystick
velocity and yaw rate input. This joint signal is the command input to the
drone flight controller. The policy module can provide additional information
to pilots through the use of feedback cues by turning on red or green lights
attached to the UAV. We formulate our problem as a partially observable Markov
decision process (POMDP) where the set of all possible states is defined as
$\mathcal{S}$. The state transition processes $T$ is treated as a stochastic
process due to the difficulty in modeling UAV dynamics from unobservable
forces resulting from the ground effect:
$T:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$, where
$\mathcal{A}$ is the set of all actions that can be taken by the policy
module. The policy module observes the environment using the observation
function $\mathcal{O}$, where the pilot’s goal and intent are treated as
hidden states which must be inferred from successive observations:
$\mathcal{O}:\mathcal{S}\times\Omega\rightarrow[0,1]$ where $\Omega$ denotes
the set of observations. The reward function is defined as
$R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$, where
the aim of reinforcement learning is to find the closest approximation to the
optimal policy: $\pi:\mathcal{S}\times\mathcal{A}$ which maximises the
expected future reward with discount factor $\gamma\in[0,1]$.

For optimal policy approximation we build upon our prior works [25, 8] and
implement TD3 [27] but with additional novel contributions to promote shared
learning between the actor and critic. TD3 is chosen as a base algorithm due
to providing model-free continuous action space control whilst alleviating the
instability concerns of its predecessor DDPG [26]. TD3 is an off-policy,
policy gradient, actor-critic approach that uses the minimum value of the two
critics in estimating the Q-value for the next state action pair, the full
algorithm is described in Algorithm 1 in [27].

From our prior work [8], it was found that the greatest influence of the
policy network’s performance was providing the critic with the simulated
user’s hidden state, the goal landing platform location. This additional
information only attainable during training in simulation can supplement the
critic’s state space, transforming the POMDP problem into a fully observable
Markov decision process (MDP) for the critic only. As MDPs present a simpler
problem to POMDPs, the critic was able to learn the task requirements quicker
and at a greater proficiency. Using this principle we aim to further improve
TD3 [27] by instilling knowledge from the critic to the actor through shared
networks that both actor and critic can jointly optimise. We also introduce
supervised learning to the reinforcement learning problem for these shared
networks to promote desired actor behavioural characteristics. We call this
novel approach Shared-TD3.

###  V-A Simulated Environment

For modelling UAV dynamics we use the Unreal Engine plugin Airsim [29] as our
simulation environment. Compared to our previous works [25] and [8], the
simulated UAVs fly in an empty Airsim environment devoid of textures, lighting
and platforms, where all graphical rendering is offloaded to the OpenGL
renderer described in Appendix A for increased computational efficiency.
Platforms are generated in the OpenGL rendering environment by randomly
placing $N_{p}$ platforms with random orientations such that adjacent
platforms are not within a minimum threshold distance while the platforms
width, length and height are randomly sampled.

###  V-B Simulating Users

To train the policy module we use simulated users to generate potential
actions human pilots could take. Previously [8] we defined simulated users
based on a parametric model containing four parameters: $\alpha$, $\beta$,
$\Psi$ & $\Phi$. $\alpha\in[0,1]$ describes the simulated user’s conformance
to the assistant’s actions and models how likely a pilot will adopt the policy
of the assistant. $\beta\in[0,1]$ describes the simulated user’s depth
perception proficiency, the ability to improve one’s estimate of the current
goal platform’s position and orientation. Both $\alpha$ and $\beta$ influence
the simulated user’s current estimate of the goal position and orientation by:

$\hat{G}_{i+1}=\hat{G}_{i}+\alpha\frac{a_{a}-a_{u}}{K_{\alpha}}+\beta\frac{G-\hat{G}_{i}}{K_{\beta}},$
(1)

where $a_{a}$ & $a_{u}$ are the previous action taken by the assistant and
simulated user respectively, while $K_{\alpha}$ & $K_{\beta}$ are scaling
constants.

$\Psi$ and $\Phi$ aim to model pilots’ thumb movements on a joystick
controller where $\Psi\in[0,1]$ describes how aggressively a pilot accelerates
their thumbs on the joystick controller while $\Phi\in[0,1]$ determines how
far on the joystick they will push, controlling for maximum desired flight
speed. The direction in which the simulated user flies is determined by
setting waypoints using a state machine comprising of three states: approach,
descent and inspection state. The simulated user initially aligns the yaw to a
set orientation, after which it will set waypoints to travel to based on the
current estimate of the goal location $\hat{G}_{i}$ and which state it is
currently in.

To account for the introduction of feedback cues from the assistant turning on
red and green lights, simulated users contain a new state variable “patience”
$\in[0,1]$ and are able to enter a “red” or “green” state in response to the
assistant’s feedback. The default state of the simulated user is the “no-
light” state, to enter the red or green state $N$ consecutive timesteps where
only the respective light has been displayed is required. The simulated user
will remain in the red or green state given the respective light continues to
be displayed. The simulated user will return to the default no-light state
given $M$ timesteps without the respective light being displayed or the
simulated user’s patience has expired. The simulated user’s patience decreases
given a timestep where the red or green light is displayed in proportion to
the simulated user’s $\alpha$ parameter and increases when no light is
displayed in proportion to $\alpha$. Once a simulated user’s patience
decreases to zero they are considered frustrated and remain in the frustrated
state until their patience has returned to the maximum value of one.

Whilst in the red state simulated users will exert less control over the UAV
by having their output action $a_{u}$ scaled by $S_{r}\in[0.0,0.5]$ to become
smaller in magnitude, allowing the assistant to more easily take control of
the UAV. Whilst in the green state the simulated user will initiate actions
that are essential to completing the objective. If the current task is an
inspection task, entering the green state will cause the simulated user to
immediately request a picture to be taken. If the current task is to land,
then the simulated user will start to descend. By causing the simulated user
to enter the green state the assistant is able to quickly progress the current
task and receive the task completion reward in fewer state transitions. To
represent a broader range of potential human pilots, samples of simulated
users are assigned to either: ignore all feedback and will not enter the red /
green state, have a hard requirement to be in the green state to progress the
mission or not having a hard requirement for the green state but still able to
enter the red / green state.

When training the assistant, new populations of simulated users are sampled
after each epoch by independently generating each of the four parameters
$\alpha$, $\beta$, $\Psi$ & $\Phi$ from a uniform distribution. Two seeds for
random number generators which control deterministic and non-deterministic
characteristics described in [8] are sampled, including the simulated user’s
aforementioned response to light feedback. Examples of simulated users flying
the UAV in various generated environments can be seen in the supplementary
video.

###  V-C Shared-TD3

In our prior work [8], the actor-critic networks followed a multi-branch
configuration where the first network branch focused on extracting information
from the current state $s_{t}$ whilst the second branch extracted temporal
information using the prior states $s_{t-n}$. The two branch’s outputs were
concatenated and fed through the output layers to produce an action or Q-value
for the respective actor-critic networks. Our shared TD3 approach follows our
previous approach using multi-branch actor-critic networks. However, each
actor and critic shares an identical network weight temporal branch which is
referred to as the Temporal-Encoder. There are two main advantages of
introducing the Temporal-Encoder: reduction in computation time per training
iteration by reusing the Temporal-Encoder’s LSTM state for calculating both
the actor and critic’s output and a reduction in total training iterations
required for the actor to learn a suitable policy due to the critic
propagating additional gradients through the network.

To further condition the LSTM cell’s temporal embedding, an additional network
is introduced, the Temporal-Decoder which is trained in a supervised manner.
The objective of the Temporal-Decoder is to take the output of the Temporal-
Encoder and estimate key features that can only be attained from subsequent
state observations across time such as the time since the pilot has requested
an inspection image. The advantage of introducing supervised learning is the
ability to forcefully embed information required for task completion onto the
LSTM embedding without the need for the actor to explore and learn the
representation by itself, which is not guaranteed using reinforcement
learning.

####  V-C1 Shared-TD3 Algorithm

The training procedure for shared TD3 initially samples a mini-batch of $N$
state transitions $(s,a,r,s^{\prime})$ and an additional ground truth state
$y$ for the Temporal-Decoder to estimate. State $s$ comprises of three
components $s=[s_{T},s_{A},s_{C}]$ which corresponds to the Temporal-Encoder
input of $n$ time lags ($s_{T}$), actor current branch input state ($s_{A}$)
and the critic current branch input state ($s_{C}$). Network optimisation is
split into three steps: 1) Temporal-Decoder, 2) critic then 3) actor
optimisation. For Step 1, the temporal state $s_{T}$ is passed into the
Temporal-Encoder $q_{T}$, which is subsequently fed into the Temporal-Decoder
$p_{T}$ to produce an estimate of the current output values i.e.
$\hat{y}=p_{T}(q_{T}(s_{T}))$. The error between the estimate $\hat{y}$ and
$y$ is computed using loss function $f_{L}$ where each of the respective
elements in $y$ are summed then are averaged across mini-batch of size $N$:
$Loss=N^{-1}\Sigma(\Sigma f_{L}(y_{i},\hat{y}_{i}))$. Gradients are then
propagated through $p_{T}$ & $q_{T}$ with their respective weights updated.

For Step 2, critic optimisation, the next temporal LSTM state embedding
$x^{\prime}_{LSTM}$ is calculated using the target temporal encoder:
$x^{\prime}_{LSTM}=q^{\prime}_{T}(s^{\prime}_{T})$. The next temporal LSTM
state embedding is subsequently fed into the target actor network
$\pi^{\prime}$ alongside the actor’s next current state input $s^{\prime}_{A}$
from which clipped gaussian noise is added to produce the target actor’s
action for the next state:
$a^{\prime}=\pi^{\prime}(x^{\prime}_{LSTM},s^{\prime}_{A})+\epsilon$, where
$\epsilon\sim$clip$(\mathcal{N}(\mu,\sigma^{2}),-c,c)$. $a^{\prime}$ is then
concatenated with the critic’s next current state input $s^{\prime}_{C}$ and
fed into the target critic ${Q^{\prime}}_{i}$ along with $x^{\prime}_{LSTM}$
to produce two estimate Q-values from which the most pessimistic value
(maximum) is taken in computing the time discounted reward of the Bellman
equation:
${Q^{\prime}}_{v}=r+\gamma$Max$({Q^{\prime}}_{i}(x^{\prime}_{LSTM},[s^{\prime}_{C},a^{\prime}]))$.
The temporal LSTM embedding for the current state is then computed using the
source Temporal-Encoder which is subsequently fed into the source critic from
which the average critic Q-value error is calculated:
$Loss_{c}=Q_{i}(q_{T}(s_{T}),[s_{C},a])-{Q^{\prime}}_{v}$. Gradients are then
propagated through $Q_{i}$ and $q_{T}$ with their respective weights updated.

For Step 3, actor optimisation, the current temporal LSTM state embedding
$x_{LSTM}$ is initially calculated using the source Temporal-Encoder $q_{T}$.
$x_{LSTM}$ is then fed into the source actor along with the current actor
state $s_{A}$, whose output is concatenated with $s_{C}$ and fed into the
first source critic alongside the previously computed $x_{LSTM}$:
$Q_{v}=Q_{1}(x_{LSTM},[s_{C},\pi(x_{LSTM},s_{A})])$. $\pi$ and $q_{T}$ are
both optimised via policy gradient by minimising $Q_{v}$, where caution needs
to be taken to ensure that gradients are not propagated directly from the
critic to the Temporal-Encoder i.e. $Q_{1}\rightarrow q_{T}$, but must first
pass through the actor: $Q_{1}\rightarrow\pi\rightarrow q_{T}$ to avoid
$q_{T}$ directly being optimised by the critic during the actor optimisation
step. The actor optimisation step is performed every second iteration.

Information regarding training of the policy module including input state
information, reward structuring and hyperparameters can be found in Appendix
B.

##  VI Supplementary Information Schemes

To further enhance the shared autonomy assistant discussed in Section V,
supplementary information schemes are proposed to provide pilots with a
greater understanding of the current state of the UAV and the assistant. From
our prior work [8], participants stated that they experienced the greatest
difficulties with estimating the depth of the UAV, resulting in failed
landings primarily from position errors along the depth axis. The difficulty
to correctly infer the depth of the UAV motivates our approach to supply
supplementary information to aid pilots in perceiving the current spatial
state of the UAV. Further issues pertaining to a lack of information arose
from participants stating that their most disliked aspect of the assistant was
the inability to perceive its intent. The lack of communication of the
assistant left participants with a sense of uncertainty in what was expected
of them and their role during the mission. Therefore supplementary information
schemes for shared autonomy UAV missions should focus on two aspects: (i)
providing spatial information of the UAV relative to the environment and (ii)
providing information about the current intent of the assistant and what the
assistant expects of the pilot.

When developing an information display interface for UAV applications,
maintaining eye-contact with the UAV is important to prevent mental burden and
lapses of concentration from context switching between the information
interface and the UAV. Therefore proposed information displays should either
be contained onboard the UAV, or augment the pilot’s natural vision with
augmented reality displays. For providing information about the state/intent
of the assistant, onboard information displays are chosen using onboard LED
lights to display red or green lights. Red/green lights are chosen for an
intuitive, universally understood representation of being in a bad/good
condition which the assistant learns to display to promote specific behaviours
as discussed in Section V-B. To convey spatial information of the UAV relative
to the environment an augmented reality display is chosen using the HoloLens.
The augmented reality display implemented in the HoloLens projects a vertical
line from the centroid of the drone onto the floor to help pilots visualise
the depth of the UAV. A secondary horizontal line whose endpoint is coincident
to the vertical line’s endpoint on the floor is rendered to display the UAV’s
current orientation. An example of the HoloLens augmented reality display and
red/green light display can be seen in Fig. 2.

Figure 2: (Left) Example of the augmented reality display from the HoloLens.
(Right) Example of red and green light feedback.

##  VII User Study

To assess the performance of the proposed assistant and to determine what
preferences human pilots have in regards to what additional information is
provided to them, a user study was conducted. The user study was approved by
the Monash University Human Research Ethics Committee (MUHREC), project ID
35281.

The physical arena consisted of nine labeled platforms of size
0.5$\times$0.5$\times$0.12m, which were placed within a 7.2m diameter circular
truss. Participants completed a total of 3 conditions, the order of which was
determined by iterating over the potential permutations given their
participant ID. Each condition contained an identical set of 6 missions, where
each mission comprised of $0-2$ inspection tasks followed by a landing task.
Each condition consisted of of seven inspection tasks and six landing tasks,
for a total of 21 inspections and 18 landings for the entire study per
participant. During each mission participants were asked to stand on a marker
that was 11m away from the center of the arena and were told to remain
stationary throughout each mission. The physical layout of the user study
arena can be seen in Fig. 3.

The three conditions participants flew the drone under were Assisted,
Assisted-Lights and Assisted-HoloLens. For the Assisted condition,
participants flew the drone using the proposed model outlined in Section V but
where the assistant’s red and green light display action was hidden to the
pilot. For the Assisted-Lights condition, participants flew with an identical
assistant to that of the Assisted condition however the assistant’s red and
green light display action was shown to participants. For the Assisted-
HoloLens condition, participants flew with an identical assistant to that of
the Assisted condition where the assistant’s red and green light display
action remained hidden to the pilot but instead were asked to fly wearing the
HoloLens augmented reality display as disscussed in Section VI. For each
condition, participants were not told the specific details of the assistant
and how it operates but instead had to figure out themselves the optimal
strategy for working with it. Participants were not explicitly told the
meaning behind the red/green lights but were instead prompted to think about
what a red or green light could indicate.

Figure 3: Physical layout of user study environment. (Left) View of the arena
from participant’s perspective. (Top right) UAV used in user study resting
atop of landing platform. (Bottom right) Participants’ relative position to
the arena.

Participants flew a custom-built UAV using the Pixhawk Cube flight controller
running ArduPilot. An Odroid-N2+ was used as an onboard companion computer to
poll RGB-D images from two Intel RealSense D435i cameras situated on the left
and right of the UAV. All perception and policy module network inference was
performed using the onboard Odroid companion computer. Fourteen Bonita-10
Vicon motion capture cameras were used for recording UAV pose information
which were wirelessly sent to the Odroid, then subsequently forwarded onto the
flight controller to allow for pose stabilization in the GPS denied
environment. Participants were constrained to fly within a 6m safe-to-fly zone
by an automatic safety system. The safety system alerted participants by
strobing RGB LED light strips attached to the arms of the drone and would
forcibly take over and push the drone towards the center of the arena if the
pilot’s actions were deemed unsafe.

Participants were initially asked to fill out a demographics survey inquiring
about their previous drone, joystick and video game experience as well as if
they participated in our previous study [8]. Participants were then shown an
introductory video explaining the controls of the UAV and the general user
study procedure. Participants were then given the opportunity to practice
flying the UAV to understand the control scheme and dynamics of the drone.
After the conclusion of the practice flight, the ceiling lights were turned
off and eight LED flood lights situated around the base of the arena were
turned on to mimic the user study conditions in [8] so that participants could
not rely on well-defined shadows cast by the UAV as a substitute for depth
perception.

Participants were then initially assigned a condition out of the three
potential conditions based on their participant ID number. Participants
received a mission plan detailing the required tasks to perform for each of
the six missions within the condition. To aid the participant’s memory, the
experimenter briefly shone a laser light at the center of the next goal
platform after the completion of each task within the mission. The first
author initiated the take-off and brought the UAV to an altitude of 1.4m using
a separate master RC transmitter, after which the LED lights on the UAV
strobed to indicate the participant can start the mission.

After completing all six missions within the condition, participants were then
given a NASA Task Load Index (TLX) [30] survey as well as 5-point scale
multiple choice questions querying how they perceived the given condition.
Participants then repeated the process of completing the remaining two
conditions and their associated surveys, the order of which determined by
their participant ID number. After completing all three conditions,
participants were then given the final survey which consisted of several short
answer questions. The entire user study took on average 1.5 hours for each
participant to complete. The full list of survey questions can be seen in
Table. III, Table. IV & Table. V, alongside the user study results in Section
VIII-B.

An additional sub-user study was held where prior participants of the main
user study returned to complete an identical set of six missions but without
the help of the assistant or HoloLens. This sub-user study’s aim is to
reaffirm the findings of our prior work [8] that an assistant is necessary to
ensure task success and not to explore pilot’s preferences towards auxiliary
information.

##  VIII Results

###  VIII-A User Study Quantitative Results

A total of 29 participants completed the main user study for a total of 174
landings and 203 inspections for each of the three assisted conditions. For
the sub-user study a total of 10 participants completed a total of 60 landings
and 70 inspections in the unassisted condition. A summary of key performance
metrics can be viewed in Table. I.

TABLE I: User study performance metrics summary

|  Assisted |  Assisted +
Lights |  Assisted +
HoloLens |  Unassisted
---|---|---|---|---
Target approach
| 5.52s/m | 5.67s/m | 6.13s/m | 5.75s/m
*Time / init. dist. | ( 3.92 ) | ( 3.96 ) | ( 5.12 ) | ( 4.04 )
| 1.29 | 1.32 | 1.44 | 1.09
*Dist. traveled / init. dist. | ( 0.81 ) | ( 0.89 ) | ( 1.10 ) | ( 0.55 )
Landing task
| 95.98% | 96.55% | 94.25% | 16.67%
Success rate | ( 0.20 ) | ( 0.18 ) | ( 0.23 ) | ( 0.37 )
| 0.08m | 0.08m | 0.08m | 0.48m
Landing error | ( 0.10 ) | ( 0.09 ) | ( 0.11 ) | ( 0.26 )
| 5.93 deg | 4.95 deg | 5.52 deg | 35.95 deg
Yaw error | ( 5.87 ) | ( 4.71 ) | ( 6.79 ) | ( 39.91 )
| 13.83s/m | 14.34s/m | 15.73s/m | 24.53s/m
*Time / init. dist. | ( 7.03 ) | ( 10.82 ) | ( 9.15 ) | ( 18.42 )
| 2.18 | 2.33 | 2.53 | 3.18
*Dist. traveled / init. dist. | ( 1.22 ) | ( 1.72 ) | ( 1.62 ) | ( 2.33 )
Inspection task
| 96.55% | 96.06% | 96.06% | 54.29%
Success rate | ( 0.18 ) | ( 0.19 ) | ( 0.19 ) | ( 0.5 )
| 0.11m | 0.13m | 0.12m | 0.26m
Inspection error | ( 0.06 ) | ( 0.17 ) | ( 0.16 ) | ( 0.19 )
| 5.97 deg | 7.11 deg | 7.87 deg | 17.56 deg
Yaw error | ( 8.78 ) | ( 13.50 ) | ( 16.80 ) | ( 24.04 )
| 11.11s/m | 8.94s/m | 12.14s/m | 11.38s/m
*Time / init. dist. | ( 7.97 ) | ( 5.44 ) | ( 9.02 ) | ( 6.50 )
| 1.64 | 1.34 | 1.96 | 1.38
*Dist. traveled / init. dist. | ( 1.08 ) | ( 0.72 ) | ( 1.39 ) | ( 0.96 )

Bolded values represent a statically significant difference compared to the
Assisted condition under a Welch’s t-test of significance level $\alpha=0.01$.

*A learning bias was observed for the given metrics where participants were observed to require more time and generate less efficient trajectories within their first condition. The order for the Assisted, Assisted + Lights and Assisted + HoloLens was randomly assigned whilst the unassisted condition was always performed last.

The standard deviation for all metrics are shown below within brackets.

For the Time / init. dist. and Dist. traveled / init. dist. metrics, the
initial distance is calculated as the Euclidean distance from the starting
point of the given task (the finishing point of the prior task) to that of the
goal platform for the current task. Both time and distance traveled metrics
account for the time taken and the distance traveled for the current task and
not the entire mission.

####  VIII-A1 Task Performance

The success rate for both landing and inspection tasks remained consistent
through all three assisted conditions, averaging a 96% task success rate,
compared to the landing and inspection success rates in the unassisted
condition of [16.67% & 56.06%]. Compared to our previous work [8] where the
UAV’s yaw remained locked and did not require participants to align the
orientation of the UAV to that of the platform, the average unassisted success
rate for landing was 51.43%. The introduction of orientation control greatly
decreased unassisted landing success to 32.41% of the previous success rate,
with 50% of unassisted participants unable to achieve a single successful
landing. These results provide strong evidence that the assistant, trained
only in simulation, is able to effectively infer pilot intent and assist
diverse pilots to successfully complete challenging tasks.

Two tailed Welch’s t-tests under a 99% confidence interval were performed to
test for statistical significance amongst the three assisted conditions for
metrics outlined in Table. I. There was insufficient evidence to suggest that
either assisted condition had a statistically significant greater performance
for the success, position and yaw errors over the other assisted conditions
for both landing and inspection tasks. This is as expected due to the
assistant being identical in each condition where the assistant’s aim is to
ensure task success. However for the time and distance metrics, a
statistically significant difference was found between the Assisted and
Assisted + Lights conditions for the inspection task, indicating that the
presence of red / green light feedback cues led to an overall reduction in the
average time taken by 19.41% and distance travelled by 17.80% for the
inspection task. The remaining time and distance metrics were found to be
statistically insignificant.

####  VIII-A2 Learning Effect and Yaw Control

Compared to our prior work [8], the introduction of yaw control caused a
strong learning effect as participants progressed further into the study,
which was observed through progressively efficient trajectories and task
completion times. An example trajectory is shown in Fig. 4 where in the first
condition, participants often mistake the current heading of the UAV causing
them to fly in an unintended direction. When attempting the same mission for
the third condition they become accustomed to the UAV control scheme and
generate a more efficient trajectory.

Figure 4: Example trajectories of participants during the user study. Image A
compares trajectories generated in the first and last condition where the
participant initially generates an inefficient trajectory needing multiple
adjustments while in the last condition they are able to fly directly to each
of the goal platforms. Images B and C demonstrates participants getting
confused with the orientation of the drone, causing them to consistently fly
in unintended directions.

The observed learning effect is empirically supported through regression
analysis where a multi-variable regression model is formulated using the
model: $Y=A+AL+AH+CO$. The model’s output Y regresses six terms: the time
taken / initial task distance for the approach, inspection and landing task,
and the distance traveled / initial task distance for the approach, inspection
and landing task. The model uses four input independent variables where $A$,
$AL$ and $AH$ are binary variables indicating what condition the mission was
flown in: Assisted, Assisted + Lights and Assisted + HoloLens respectively.
The final independent variable $CO$ controls for the order in which the
condition was performed and can take a value of 0, 1 or 2 depending when a
given condition was performed during the study.

Significance testing was performed on the coefficient affiliated with the $CO$
independent variable to confirm if the condition order has an impact on the
aforementioned time and distance traveled metrics. The $CO$ variable was
considered statistically significant under a two-tailed t-test of significance
level $\alpha$ = 0.01 for all six metrics. For all six metrics it was
predicted that the further into the study a participant is, on average their
expected task completion time and expected task travel distance is shorter.
The full regression results can be seen in Table VI in Appendix C.

####  VIII-A3 Pilot Proficiency Influence

Regression analysis is performed to assess whether a participant’s prior
experience affects their assisted performance, preference to each of the
assisted conditions and how much assistance the assistant provides. The model
was formulated using: $Y=\beta+P+k$, where $\beta$ represents a participant’s
UAV piloting proficiency and is calculated using the linear regression model
outlined in [8] which uses participants’ responses from the demographics
survey seen in Table III. Independent variable $P$ denotes a prior participant
of our previous user study [8], while $k$ represents the intercept constant.

The model was initially fitted to the performance metrics in Table I.
Independent variable $\beta$ was found to be statistically significant for the
Time / init. dist and Dist. travelled / init. dist. for the approach, landing
and inspection tasks under a two-tailed t-test of significance level $\alpha$
= 0.01, where an increase in pilot’s proficiency was on average observed to
reduce the time required and distance travelled to approach or complete a
landing or inspection task. There was insufficient evidence to suggest that a
participant’s prior participation ($P$) affected the time and distance
travelled metrics and that $\beta$ or $P$ affects the success, position or yaw
error of the task.

These results are partially congruent with our prior work [8] showing task
success and position error to be invariant of piloting proficiency whilst
assisted. However the inverse result is found for the time and distance
travelled metrics, where a pilot’s proficiency in [8] was found to have no
statistically significant effect on how fast or how efficient a trajectory a
participant generated is whilst assisted. However in the current study,
$\beta$ was found to be statistically significant in estimating a pilot’s
average time and distance travelled to complete a task. This is primarily due
to the introduction of yaw control where novice pilots tended to get confused
with the orientation of the UAV in the approach stage before the assistant
could receive any cues about their intent. This ambiguity of inferring the
pilot’s intent is further explored in Section VIII-A4.

Regression analysis was then performed to estimate whether a pilot’s expertise
influenced their preferred assistance condition. There was insufficient
evidence to suggest that $\beta$ or $P$ impacted pilots’ preferences to either
of the experimental conditions under a 99% confidence interval. The regression
model was finally used to estimate the total assistance provided by the
assistant along the XYZ and yaw axis. It was found that both $\beta$ and $P$
were statistically significant in estimating the amount of assistance provided
for the linear XYZ assistance, where an increase in pilot proficiency or if
the participant had participated in the prior study, decreased the overall
assistance required to complete the task.

For the assistance received along the yaw axis, only pilot proficiency
($\beta$) was found to be statistically significant under a 99% confidence
interval, where an increase in pilot proficiency reduced the amount of
assistance required along the yaw axis. This is expected as prior participants
did not experience yaw control in the previous study. These results suggest
that the assistant appropriately adjusted the level of assistance based on the
pilot’s capability. The full regression results can be seen in Table II.

TABLE II: Pilot proficiency regression model

|  $\beta$ |  $P$
---|---|---
Target approach
Time / init. dist. | -2.07 | -0.42
Dist. traveled / init. dist. | -0.47 | -0.11
Landing task
Success rate | 0.08 | 0.03
Landing error | -0.03 | -0.02
Yaw error | -0.03 | -0.02
Time / init. dist. | -9.68 | -0.96
Dist. traveled / init. dist. | -0.82 | -0.25
Inspection task
Success rate | 0.03 | 0.04
Inspection error | -0.01 | -0.01
Yaw error | -0.06 | -0.04
Time / init. dist. | -8.26 | -0.86
Dist. traveled / init. dist. | -0.90 | -0.19
Condition preference
Assisted ranking | -0.42 | -0.01
Assisted + Lights ranking | 0.69 | 0.26
Assisted + HoloLens ranking | -0.27 | -0.25
Assistance received
XYZ assistance | -271.49 | -140.66
Yaw assistance | -549.95 | -59.99

Regression coefficients for the model $Y=\beta+P+k$.

Regression constant $k$ omitted for clarity.

Bolded values represent statistically significant values under a two-tailed
t-test of significance $\alpha=0.01$.

####  VIII-A4 Task Assistance

The degree of assistance provided by the assistant is dependent on the
confidence in the estimated intent of the pilot. During the approach, the
ability of the assistant to estimate the desired goal is low due to the
presence of multiple potential goals with little indication from the pilot of
targeted actions focusing on a single goal. As the task transitions from
approach to the goal objective (inspection or landing), the ability to
estimate the pilot’s intent increases, hence an increase in the assistance
provided is observed as seen in the left plot in Fig. 5.

The cumulative average assistance for both landing and inspection tasks are
equal, where the landing task is observed to have a greater bias towards
assistance along the XYZ direction due to the precise success requirements in
the XY position of the UAV for landing compared to inspection. While the
inspection task was observed to have a greater need towards assistance along
the yaw direction compared to the landing task as seen in the right plot in
Fig. 5. The higher observed yaw assistance is due to precise alignment with
the target platform maximises platform image coverage whilst minimising risk
of task failure from corners being omitted when an image is taken.

Figure 5: Assistance received for each task and transitions between tasks.
(Left-figure) Average assistance received (black) and interquartile range of
assistance received (blue region). (Right-figure) Cumulative average
assistance received for inspection and landing tasks split along the XYZ axis
(green) and yaw axis (blue).

###  VIII-B User Study Qualitative Results

####  VIII-B1 Task Load Perception

Summary results of participants’ perception of the task can be seen in Fig. 6.
From the TLX survey, participants perceived a large difference in the required
workload when comparing the assisted conditions to that of the unassisted
condition. The general perceived workload ranking for each of the conditions
from the least burdened condition to the most workload burdened condition is:
Assisted + Lights $<$ Assisted + HoloLens $<$ Assisted $<$ Unassisted.

The TLX survey responses were analysed with a Welch’s t-test under a 95%
confidence interval to determine if a statistical difference exists between
any of the study conditions for each of the workload metrics. A statistically
significant difference was found between all of the assisted conditions and
the unassisted condition for each of the metrics aside from physical demand. A
statistically significant difference was also found between the Assisted
condition and the Assisted + Lights condition for the mental demand and effort
workload metric, indicating that the inclusion of light feedback reduced
pilots’ mental workload and effort required to complete the task. For all
other metrics between each of the four conditions, there was insufficient
evidence to suggest that a statistically significant difference exists.

####  VIII-B2 Condition Perception

The full list of survey questions and responses can be seen in Table. III,
Table. IV and Table. V. From the condition perception survey as shown in
Table. IV, it was found that compared to the assisted condition, introducing
the ability for the assistant to provide feedback via flashing red and green
lights enhanced pilot’s confidence in the landing and inspection task.
Furthermore allowing the assistant to provide feedback created a greater
understanding between the pilot and the assistant, where pilots better
understood what the intent of the assistant was and what the assistant
expected of them.

In our prior work [8], participants’ most disliked aspect of the assistant was
the uncertainty in inferring the intent of the assistant and what the
assistant expected of them during the study. The inclusion of feedback cues
addresses the prior work’s limitations by alleviating the uncertainty in what
the assistant is trying to achieve and what the assistant wants the pilot to
do by signalling when a task is ready to be completed by displaying the green
light. This is supported by a statistically significant difference being
observed between the Assisted & Assisted + Lights condition for questions 1,
2, 3 & 4 and a statistically significant difference between the Assisted +
HoloLens & Assisted + Lights condition for question 3 and 4 in Table. IV under
a 95% confidence interval using a Welch’s t-test.

Figure 6: Results of NASA TLX survey.

####  VIII-B3 Participant Free-Form Response

From the free-form responses, when asked about the most difficult aspect of
the task, 53% of participants who responded mentioned difficulties with
tracking the orientation of the UAV and correctly mapping joystick commands to
move the UAV in the intended direction. It was observed that participants
often got confused with the orientation of the UAV causing them to fly in an
unintended direction where common solutions to this problem were to “explore
by picking a direction and see if it goes where expected” or to “rotate the
drone to face the same way I was”. Compared to our prior work [8] where the
yaw remained locked, the previous most difficult aspect of the user study as
stated by 68% of participants was difficulties in perceiving depth of the UAV,
while in the current study only 14% of participants mentioned difficulties
involving depth. Participants flying in an unintended direction due to
incorrect orientation estimation often led to flying out of bounds and
triggering the safety system. A total of 119 safety system triggers were
observed during the 3 assisted conditions compared to a single safety system
trigger in our prior study [8].

When asked about their most preferred aspect for each of the three conditions,
for the Assisted condition 37% of participants most liked how the assistant
automatically aligned the UAV to the platform as they “didn’t have to worry
about perfectly piloting the drone” and instead having to only approximately
reach the intended platform. For the Assisted + Lights condition 62% of
participants most liked the confirmation and certainty the assistant provided
by flashing the green light when a task was ready to be completed where
participants “found this condition to be both intuitive and learnable” with
the feedback it provided. For the Assisted + HoloLens condition 46% of
participants most liked the display of the UAV’s orientation as the “augmented
reality addition makes the orientation of the robot easiest to track,
therefore making the control easiest”, which addresses the most difficult
aspect of the user study from question 1 in Table. V. A further 36% of
participants mentioned they liked how the floor projection made it easy to
tell if the UAV was above a platform by observing the change in height of the
projected vertical line.

When participants were asked about their most disliked aspect of the task for
each of the three conditions, for the Assisted condition the most disliked
aspect generally pertained to the lack of certainty in the UAV’s position,
orientation, inspection image quality or what the assistant wanted the
participant to do. For the Assisted + Lights condition the majority of
participants gave no response. For the Assisted + HoloLens condition 29% of
participants left comments stating that the HoloLens was “uncomfortable”,
“heavy” or “gives you a headache” when wearing it. A further 36% of
participants stated that using the HoloLens degraded their visual abilities
due to “decreased contrast” from the darkened glass visor, the augmented
reality overlay obscuring real-life features and an obstruction of their
peripheral vision from the visor.

TABLE III: Participant Demographics

Did you participate in our previous physical drone landing user study? No Yes
62% 37% How many hours of experience have you with flying physical drones? 0-5
68% 5-10 13% 10-20 6% 20-50 3% 50+ 6% How often do you play video games? Never
24% Monthly 34% Weekly 20% Regularly 17% Daily 3% How many hours have you
spent using a joystick controller? 0-10 31% 10-25 10% 25-100 10% 100-200 17%
200+ 31%

TABLE IV: Condition perception survey

A A+L A+H 1: I felt confident in completing the landing task 2: I felt
confident in completing the inspection task 3: I understood what the assistant
wanted to do 4: I understood what the assistant wanted me to do 5: The
assistant didn’t do what I wanted it to do 6: The assistant and I worked well
as a team 7: I felt that I received a lot of help from the assistant
pt3.14pt3.55pt3.28pt3.07pt3.52pt3.07pt3.10pt3.62pt3.14pt2.45pt3.38pt2.83pt1.03pt0.97pt0.93pt2.97pt3.41pt3.07pt3.00pt3.31pt3.07

The minimum value of 0.0 corresponds to strongly disagree whilst the maximum
value of 4.0 corresponds to strongly agree. Coloured bars represents the
distribution of participant responses.

TABLE V: Final survey worded questions 1: What part of the task did you have
the most difficulties with?
---
2: What did you most like about each of the conditions?
3: What did you most dislike about each of the conditions?
4: Rank the conditions from most preferred (1) to least preferred (3)
5: What features or changes would you make to any of the conditions?
6: Describe any differences in flying strategy for each of the conditions
7: Any additional comments/observations you would like to add?

####  VIII-B4 Condition Ranking

Participants were asked to rank the three conditions from their most preferred
(1) to least preferred (3), where the most preferred condition was the
Assisted + Lights condition with an average ranking of 1.59, followed by the
HoloLens condition with an average ranking of 1.96 then the Assisted condition
at 2.44. Participants most preferred the Assisted + Lights condition due to
the intuitive design and easy to interpret green light feedback which
alleviated the uncertainties in task success. The Assisted + HoloLens
condition was the second most preferred condition due to providing additional
information that indicates the orientation of the UAV to aid the pilot in
moving in their intended direction. However the HoloLens was reported as
uncomfortable to wear and hindered the pilot’s natural vision.

###  VIII-C Model Training Results

####  VIII-C1 Perception Module

The aim of introducing an identical latent representation for both high
quality and simplistic images is the reduction in image rendering time during
exploration. Introducing the CM-SVAE architecture allowed for a total of 24
UAVs flying simultaneously at 2x simulation speed whereas in our prior work
[8], a total of 16 UAVs simultaneously flying at 1x simulation speed was
achievable on identical hardware. The CM-SVAE architecture accounted for a
reduction in exploration time by a factor of 3, a significant improvement.
When further accounting for the increase in image size from our prior work of
141w $\times$ 80h to 210w $\times$ 120h in our current work, the introduction
of the simplistic image representation allowed for an increased rendered pixel
output by a factor of 6.7, without any noticeable loss in network performance
and capable of transferring to real unseen environments.

Figure 7: Architecture ablation study results. Each metric is averaged over
the three training initialisations. For the top plots, a landing is considered
a success if the UAV lands on the intended platform with all four legs
contacting the surface with a relative yaw error of less than 20 degrees. For
the bottom plots, an inspection is considered a success if all four corners of
the intended platform are within the captured image with a relative yaw error
of less than 20 degrees. Left column: model success rates whilst training.
Middle column: model success rates during simulated validation. Right column:
model success rates during physical validation.

####  VIII-C2 Policy Module Training

To quantify the impact of network architecture design decisions from the
introduction of Shared-TD3, an ablation study was performed where three unique
models were trained. (i) Shared-TD3, the proposed model architecture outlined
in Section V-C1 where the Temporal-Encoder network is shared by the actor,
critic and the Temporal-Decoder. (ii) No temporal decoder, where the Temporal-
Encoder is shared by both the actor and critic but the Temporal-Decoder is
omitted from the model. (iii) Standard-TD3, which follows the base algorithm
outlined in [27] where both actor and critic have their own unique LSTM
temporal branch similar to our prior work [8]. Each of the three models were
trained over three random initialisations, where each initialisation consisted
of four-million training iterations. The training results of the ablation
study can be seen in the left column of Fig. 7.

The proposed model Shared-TD3 was found to have the best performance in
training with a final average success rate for the landing and inspection task
of [91.7% & 94.6%], compared to [70.8% & 65.3%] and [41.7% & 12.8%] for the no
temporal decoder and standard TD3 models respectively.

Introducing a shared Temporal-Encoder not only reduced the iterations required
to learn a suitable policy, where the Shared-TD3 and no temporal decoder model
was able to achieve the standard TD3 landing success rate within 33% and 58%
of the total training iterations respectively, but also reduced the time taken
to perform a single training optimisation iteration. Using a batch size of 64
and a LSTM lookback range of 50 iterations, which is equivalent to 10 seconds
worth of prior state-transitions, the standard TD3 model took an average
39.75ms to perform a single optimisation iteration. Compared to the no
temporal decoder model taking on average 27.95ms per iteration, 70% of the
required time for the standard TD3 algorithm. The Shared-TD3 model took on
average 36.55ms per iteration, 92% of the required time for the standard TD3
algorithm despite an additional network, the Temporal-Decoder, needing to be
trained. The reduction in time per optimisation iteration is due to reusing
previously computed LSTM encoding branch outputs during training, which is the
most computationally inefficient component of the network forward passes. All
tests were performed using an RTX-3090 GPU and AMD-5950 CPU.

####  VIII-C3 Policy Module Simulated Validation

The ablated models were subjected to a standardized validation sequence
consisting of 6 missions. The simulation environment and mission plan
reflected the conditions that were used in the user study outlined in Section
VII. For each model initialisation, the sequence of 6 missions were performed
a total of 101 times, where $\beta$ was swept from 0.0 to 1.0 in 0.01
increments while the remaining simulated user’s parameters $\alpha$, $\Psi$
and $\Phi$ were held constant at 0.5. To ensure that the simulated user’s
behaviour remained consistent throughout each $\beta$ value for each of the
model initialisations, a unique number to seed the simulated user’s random
number generators was assigned to each of the 6 missions. The simulated
validation results can be seen in the middle column of Fig. 7.

The highest performing network architecture was the proposed Shared-TD3 model
with an average landing and inspection success rate of [97.6% & 94.6%],
followed by the no temporal decoder model at [84.4% & 63.9%], then the
standard TD3 model at [76.4% & 46.3%]. The baseline simulated user without any
additional assistance scored an average landing and inspection success rate of
[48.8% & 22.6%].

The Shared-TD3 model demonstrated an equal ability in completing both the
landing and inspection task whilst the no temporal decoder model and standard
TD3 model demonstrated a disparity between the landing and inspection tasks,
where both models had poorer performance in the inspection task. Observing the
trajectories during the inspection task, the average iterations that the
assistant captures an image after the simulated user’s request for the no
temporal decoder and standard TD3 model is 1.0 (0.2s) and 0.4 (.08s)
respectively, compared to the Shared-TD3 model of 7.5 (1.5s). This indicates
that without the Temporal-Decoder to forcefully condition the Temporal-Encoder
to extract temporal information regarding the simulated user’s current image
request status, the models have difficulties to extract this information via
reinforcement learning and instead request an image whilst the simulated
user’s image request is directly observable within the current input. This
concept is further strengthened by observing the ratio of total image requests
of the assistant divided by the image requests of the simulated user for the
no temporal decoder and standard TD3 models, which have a value of 2.22 and
3.25 respectively compared to a value of 1.01 for the Shared-TD3 model. This
indicates that the two models without the Temporal-Decoder struggle to extract
the temporal information required to remember whether they have already taken
a picture for the given task.

####  VIII-C4 Policy Module Physical Validation

To assess the performance of the proposed Shared-TD3 model when transferring
from simulation to reality, an identical set of 6 missions performed in the
simulated validation were performed in a physical environment where the
simulated user piloted the physical UAV. The results of the physical
validation can be seen in the right column of Fig. 7.

The performance of the Shared-TD3 model was comparable to that of the
simulated validation experiment where only a slightly lower success rate for
the inspection task was observed for lower $\beta$ simulated users. The
Shared-TD3 model was successfully able to be transferred to reality despite
being trained only in simulation, on simplistic input images.

##  IX Outdoor Demonstration

To demonstrate that the proposed approach is feasible in non-laboratory
conditions without access to high-quality pose feedback from the Vicon motion
capture system, an outdoor flight experiment was held. An Intel RealSense
Tracking Camera T265 was fitted to the UAV for onboard pose estimation which
the flight controller used for flight stabilisation. The UAV was not fitted
with a GPS to emulate a GPS-denied environment. The software architecture
remained identical as outlined in Section VII, where all network inference
continued to be computed using only the onboard companion computer.

Two outdoor experiments were held where the first experiment tested the
assistant’s robustness in transferring to unfamiliar environments. The first
author piloted the UAV in the assisted condition where they initially
performed an inspection task followed by an intentional unsafe landing on top
of an obstacle to observe the assistant’s behaviour. The second experiment
aimed to measure the difference in performance to that of the indoor user
study. For the second experiment the first author stood 10m behind the center
platform and followed a standardised mission plan involving two inspection
tasks followed by a final landing task. The mission was repeated a total of 20
times in both the assisted and unassisted condition. The outdoor experiment
flight arena can be observed in Fig. 8.

Figure 8: (Top left) Pilot attempting to intentionally land in an unsafe
location. (Bottom left) Pilot performing an inspection during the second
experiment. (Right) UAV attached with the Intel RealSense Tracking camera T265
for onboard pose estimation.

For the first experiment, the assistant was able to successfully abort the
landing when the pilot intentionally attempted to land at an unsafe location
by providing an opposing action to prevent the UAV from descending. For the
second experiment, the average success rate for the landing and inspection
task was [80% & 85%] respectively compared to the unassisted condition which
scored a success rate of [45% & 62.5%]. Compared to the average assisted
indoor user study results of [95.59% & 96.49%], the lower success rate in the
outdoor demonstration can be attributable to the poorer quality of pose
estimation. During the experiment the UAV was observed to drift and be less
responsive to pilot inputs, which degraded the performance of the assistant.

##  X Discussion

###  X-A Participant Behaviour to Task and Study Conditions

Task performance across all three assisted conditions were similar to each
other, sharing equally high success rates for the landing and inspection tasks
due to the assistant being identical in each condition. Compared to our prior
work [8], a greater diversity of flight strategies were observed in how
participants approached the goal platform, primarily in how they accounted for
the rotation of the UAV. Four fundamental strategies were observed to account
for yaw when approaching a goal platform: (i) initially align the UAV with the
ego-centric view so that the relative coordinate frame of the UAV is identical
to that of the participant; (ii) initially align the UAV to face the target
platform such that the participant only has to push the joystick forward to
approach the platform; (iii) the participant would command the UAV to fly
forward whilst adjusting the yaw to move to the target platform similar to
driving a car and (iv) the participant would make no yaw adjustments and
attempt to provide relative target velocity inputs to approach the platform.
Although not all approach strategies were modelled in the simulated user
model, it did not degrade the performance of the assistant in achieving task
success.

Due to the diverse approach strategies, each mission was separated into
approach, landing and inspection tasks in order to assess the impact each of
the assisted conditions has on the landing and inspection tasks, without being
influenced by the participant’s chosen approach strategy. Participants in the
Assisted + Lights condition completed the inspection task more rapidly by
requesting an image to be taken earlier due to the green light feedback from
the assistant. Participants were not explicitly told the meaning behind the
light feedback provided by the assistant, but were quickly able to learn that
when the assistant displays the green light they are in a good position to
complete the task by either requesting an image or descending. However it was
noted that a specific mission with a short initial approach distance and the
UAV already being correctly aligned to the goal platform would cause the
assistant to require more time to display the green light as there was
insufficient information about the intent of the participant. This led to some
participants hovering above the platform, waiting for the assistant’s signal
before descending where they would have had otherwise attempted to land in an
alternative condition.

Despite the Assisted + HoloLens being ranked as the second most preferred
condition, participants’ free-form responses to the condition showed a varied
perception, where some stated they enjoyed it whilst others expressed a strong
dissatisfaction. The main sources of dissatisfaction were the weight and
physical discomfort experienced, especially from participants who required
prescription glasses. Other concerns from participants were the loss of
calibration accuracy between the HoloLens and Vicon motion capture system,
where some participants stated that the vertical floor projection line lost
its centered alignment to the UAV and would display at the legs of the UAV.
This issue would be further exacerbated in conditions without accurate motion
capture feedback and limits the potential deployment of the HoloLens in
practical UAV applications.

Practical deployment of the HoloLens requires relative localisation between
the UAV and HoloLens which is implemented in [31]. An additional concern of
practical implementation of the HoloLens in real-use scenarios is
participants’ change in flight strategy. Some participants solely focused on
the horizontal line projected onto the floor whilst flying, pretending they
were driving a remote-controlled car. This proposed flight strategy may be
optimal for controlled environments however could be dangerous in uncontrolled
environments as it ignores potential overhanging and airborne obstacles.

###  X-B Learning and Training Efficiencies

The success of the proposed approach can be attributable to three main
factors: (i) a focus on training efficiency, (ii) improvements made to the
base reinforcement algorithm TD3 and (iii) simulated user modeling in reacting
to the assistant. Two prominent sources of training efficiency improvements
implemented were the reduction of GPU workload when generating input images
for policy exploration and the reduction in policy optimisation computation
time by reusing previously calculated LSTM cell embeddings. GPU workload
reduction led to an increased exploration output by a factor of 3 compared to
our prior work [8], where increasing exploration further resulted in the
training procedure becoming CPU bound. Sharing previously calculated LSTM cell
embeddings during policy optimisation for the actor and critic led to
requiring 70% of the computation time to perform a single policy training
iteration. The aforementioned efficiency improvements led to an approximate
total training time of 3 days per model which allowed for frequent reward
structure modification to optimise the performance and piloting experience
when transferring the model to the physical UAV.

The second factor for model success were the improvements made to the
reinforcement learning algorithm TD3 by including a Temporal-Encoder network
to share an LSTM embedding between the actor, critic and the introduced
Temporal-Decoder. The Temporal-Decoder used supervised learning to forcefully
embed information that can only be attained through successive temporal
observations. Without the Temporal-Decoder it was empirically observed that
the actor would often take a picture whilst the simulated user’s image request
action was directly observable within the input state vector, regardless of
image quality due to the actor not being able to remember whether an image was
requested or not. This is further observed where models without the Temporal-
Encoder would demonstrate suboptimal or inconsistent behaviour in displaying
the red/green lights by either sporadically flashing them, continue to display
them after a task is complete or chose to ignore light feedback completely.
Sharing the Temporal-Encoder also resulted in fewer training iterations to
reach convergence. However the increased performance with temporal dependant
tasks may be due to increased amount of optimisation iterations being
performed on the Temporal-Encoder. For the standard implementation of TD3, the
Temporal-Encoder receives 1 actor optimisation per 2 training iterations
whilst for the improved version a total of 5 Temporal-Encoder optimisations
were performed every 2 training iterations.

Compared to our prior work [8] which successfully implemented the baseline TD3
reinforcement learning algorithm, the current study demonstrated that the
standard TD3 model gave unsatisfactory results when task complexity was
increased. In [8] the model was able to achieve a landing success rate of 80%
within the first 0.75 million training iterations and a final landing success
rate of greater than 90% after 2 million training iterations. However when
transferring the TD3 algorithm to multi-task missions the model was only able
to achieve a final landing success rate of 41.7% despite being trained for
twice as long for 4 million training iterations. This poorer performance can
be attributable to the increased complexity in state-transitions with respect
to the actor’s output due to the introduction of yaw control.

Initial development focused on replicating the prior work’s [8] task of only
landing the yaw locked UAV. The introduction of yaw control led to an
increased observation of network instability with actor/critic loss spikes and
catastrophic forgetting. This suboptimal behaviour is hypothesised to be due
to sparser rewards from additional success requirements, increased
dimensionality in actor’s output which has shown to cause instability in
policy-gradient approaches for UAV landing tasks [17, 18] and the increased
difficulty in critic evaluation of the actor’s action for a given state due to
the non-linearities introduced with rotation. However with the novel
contributions to the TD3 reinforcement learning algorithm, the model was able
to overcome the aforementioned difficulties and successfully complete multi-
task missions.

###  X-C Simulated User Model

The third factor of model success was the simulated user model and its ability
to react to the assistant’s actions and communication feedback. Developing
simulated user models for shared autonomy systems is challenging due to the
difficulties associated with realistically replicating the broad range of
policies pilots may implement. Implementing realistic reactions to the
assistant’s actions is further difficult due to the unpredictability of the
assistant throughout the training procedure. Parameterizing the simulated user
model to include a parameter associated with the conformance to the
assistant’s actions ($\alpha$) has shown success in our prior works [25, 8].
However teaching the assistant to provide meaningful communication feedback
without an explicit reward structure whilst being easily interpretable by
human pilots that are unfamiliar with the assistant is challenging. The
simulated user model reacted to the red and green lights generated by the
assistant by halting or progressing the current task, using commonly accepted
social conventions behind the meaning of red and green lights. Participants
were able to quickly understand these signals and perceived the feedback as
intuitive in the survey feedback. Having a hard requirement that samples of
simulated users need the green state to progress the task forced the assistant
to learn feedback cues to achieve task success. This hard requirement
accurately reflects samples of human participants where the majority of pilots
waited until the green light was displayed before progressing the task.

##  XI Conclusion

In this work we propose a shared autonomy approach that assists pilots in
successfully completing multi-task missions comprised of inspection and
landing tasks. We implement an AI assistant trained using our improved
implementation of the deep reinforcement learning algorithm TD3 to augment the
pilot’s actions, alongside supplementary information from red/green light
feedback cues and augmented reality using the HoloLens to assist pilots. The
assistant is comprised of three modules: a perception module responsible for
encoding visual information onto a compressed latent vector, a policy module
responsible for providing control outputs that are averaged with the pilot’s
actions to control the UAV and an information augmentation module responsible
for providing information about the state of the UAV and provide feedback to
the pilot. The assistant was trained concurrently in simulation against a
population of simulated users defined using a parametric model. The assistant
was able to further influence the behaviour of simulated users by providing
red/light feedback cues to entice certain actions.

A physical user study ($n=29$) was held to validate the proposed assistant and
to assess participants’ perception to the proposed supplementary information
schemes. Participants were tasked with completing 6 multi-task missions for
each condition by taking multiple pictures of the nine specified platforms
after which were tasked with safely landing on a specified platform. The
assistant raised task success rate from [16.67% & 56.06%] respectively in the
unassisted condition to [95.59% & 96.49%] in the assisted condition. The
red/green light feedback cues were found to reduce the time required by
participants to complete the inspection task by 19.41% and the distance
traveled by 17.80%, whilst being their most preferred condition due to being
perceived as intuitive to understand and provided the needed reassurance and
confidence to the pilot about the current successfulness of the task.

An outdoor experiment was held where the UAV flew in a GPS-denied environment.
The assistant attained a success rate for the landing and inspection task of
[80% & 85%] respectively compared to [45% & 62.5%] in the unassisted condition
and was able to successfully transfer to unseen conditions without the need
for motion capture feedback.

##  Appendix A Perception Module Implementation

Implementation of the CM-SVAE architecture follows our previous implementation
[8] of the CM-VAE architecture where two downwards facing RGB-D cameras
situated to the left $x_{L}$ = [$x_{LRGB}$, $x_{LD}$] and right $x_{R}$ =
[$x_{RRGB}$, $x_{RD}$] of the drone are used. Both $x_{L}$ and $x_{R}$
alongside their simplified versions $\bar{x}_{L}$ and $\bar{x}_{R}$ are passed
through a Siamese feature extraction layer $F_{DN}$ using DroNet [32]. The
resultant vectors are concatenated and encoded onto the latent embedding of
width 24 with encoder $q_{RGBD}$ i.e.
$[\mu,\sigma^{2}]=q_{RGBD}([F_{DN}(x_{L}),F_{DN}(x_{R})])$ and
$[\bar{\mu},\bar{\sigma}^{2}]=q_{RGBD}([F_{DN}(\bar{x}_{L}),F_{DN}(\bar{x}_{R})])$.
The CM-SVAE contains three output modalities: the simplified grayscale image
$\bar{y}_{G}$, the simplified depth map $\bar{y}_{D}$ and segmentation map
$\bar{y}_{S}$ which denotes areas which are safe and unsafe to land. To
reconstruct $\bar{y}_{G}$, $\bar{y}_{D}$ and $\bar{y}_{S}$, a sample is taken
from the latent embedding $z\sim\mathcal{N}(\mu,\sigma^{2})$ &
$\bar{z}\sim\mathcal{N}(\bar{\mu},\bar{\sigma}^{2})$, which is subsequently
fed through decoders $p_{G}$, $p_{D}$ and $p_{S}$:
$\hat{\bar{y}}_{G}=p_{G}([z,\bar{z}])$, $\hat{\bar{y}}_{D}=p_{D}([z,\bar{z}])$
and $\hat{\bar{y}}_{S}=p_{S}([z,\bar{z}])$.

In order to create a consolidated latent representation instead of two
separate latent representations for $x_{L}$ and $x_{R}$, we implement a camera
projection model that fuses the field of view of both cameras when
reconstructing output images $\bar{y}_{G}$, $\bar{y}_{D}$ and $\bar{y}_{S}$.
The camera projection model in [8] is modified to account for differences in
the position and orientation of the RGB-D cameras. We use input images of size
210w $\times$ 120h with 20 intermediate cameras ($N_{c}$ = 20), to create a
combined output image size of 210w $\times$ 140h which is subsequently down-
sampled to 96w $\times$ 64w.

Figure 9: Example images used to train the CM-SVAE architecture

High visual fidelity images $x_{L}$ & $x_{R}$ are rendered with the AirSim
[29] Unreal Engine plugin, while light-weight image rendering for
$\bar{x}_{L}$ & $\bar{x}_{R}$ uses OpenGL. To generate a scene, $N_{p}$
platforms are randomly placed with random orientations such that adjacent
platforms are not within a minimum distance threshold. For $x_{L}$ & $x_{R}$,
each generated scene has their lighting conditions altered and textures are
randomly sampled from a database of 500 materials for the platforms, walls and
floor. For $\bar{x}_{L}$ & $\bar{x}_{R}$ lighting effects are disabled where
the platforms, walls and floor are given a default colour which persists over
all generated scenes. For each generated scene, a total of 10 left and right
input RGB-D images are taken from both the Unreal Engine and OpenGL renderers
in identical randomly generated poses. The output images $\bar{y}_{G}$,
$\bar{y}_{D}$ and $\bar{y}_{S}$ are generated using only the OpenGL renderer.
As the images generated using AirSim are intended to be reflective of those
from real RGB-D cameras, a noise generating function $G_{n}$ is used to mimic
sensor noise present in real-life cameras. For greater domain randomisation
and robustness against sensor noise, $G_{n}$ is applied to input batches of
clean AirSim images which are dynamically noised during training by parallel
worker threads. Example generated training images can be seen in Fig. 9

Losses used to train the perception module can be classified into two
categories: (i) those that aim to embed information onto the latent vector and
(ii) those that aim to condition the latent vector. For (i) we used the mean
squared error between the true grayscale images and the estimated grayscale
images: $L_{G}=MSE(\bar{y}_{G},\hat{\bar{y}}_{G})$, the mean squared error
between the true depth map and estimated depth map:
$L_{D}=MSE(\bar{y}_{D},\hat{\bar{y}}_{D})$ and the binary cross entropy loss
between the true segmentation map and estimated segmentation map:
$L_{Seg}=BCE(\bar{y}_{S},\hat{\bar{y}}_{S})$. For (ii) we use the Kullback-
Leibler divergence to maintain the desired latent distribution of $\mu=0$ and
$\sigma^{2}=1$ ($L_{KL})$ and the cosine similarity loss to maintain an
identical latent embedding between the AirSim and OpenGL rendered images
$L_{Sim}=\frac{\mu\cdot\bar{\mu}}{\left\lVert\mu\right\rVert\left\lVert\bar{\mu}\right\rVert}$.
For each loss: $L_{G}$, $L_{D}$, $L_{Seg}$, $L_{KL}$ and $L_{Sim}$ we give a
weighting of 3.0, 1.0, 0.5, 4.0 and 1.0 respectively. An overview of the CM-
SVAE architecture can be seen in Fig. 10.

Figure 10: Overview of the CM-SVAE architecture

##  Appendix B Policy Module Training

An overview of the policy module training architecture can be seen in Fig. 11.
For the input states $s=[s_{T},s_{A},s_{C}]$ associated with the Temporal-
Encoder, actor and critic respectively, $s_{T}$ is comprised of three inputs:
the previous mean latent vector from encoder $q_{RGBD}$, the user’s previous
action and the assistant’s previous action. The user’s action includes their
target XYZ velocity, yaw rate and take-photo action if an image was requested
in the given state. The assistant’s action includes its target XYZ velocity,
yaw rate, take-photo action, red and green light display actions. $s_{A}$ is
comprised of two components: the current mean latent vector from encoder
$q_{RGBD}$ and the user’s current action. As $s_{C}$ is required in training
only, it can be supplemented with privileged information that can only be
attained during simulation which was found to have the greatest impact in
network performance in our prior work [8]. $s_{C}$ comprises of six
components: the UAV dynamics, user’s action, actor’s action, current goal
information, simulated user’s state and the mean latent vector from encoder
$q_{RGBD}$. The UAV dynamics includes the current position, yaw, linear and
angular velocities of the UAV. The current goal information includes the
relative position and yaw of the goal platform with respect to the current
UAV’s pose, the type of mission the simulated user is attempting to complete
(landing or inspection) and the physical dimensions of the goal platform. The
simulated user’s state information includes their patience, green and red
light state, as well as what the simulated user’s current objective is
(approach platform / landing / inspecting).

The reward structure used to compute the reward for a given state transition
can be categorised into three components: landing rewards, inspection rewards
and cooperation rewards where the total reward for a given state transition is
defined as:

$\displaystyle
R=R_{\mathrm{Landing}}+R_{\mathrm{Inspection}}+R_{\mathrm{Cooperation}}.$ (2)

$R_{\mathrm{Landing}}$ contains four terms: landing success $R_{\mathrm{LS}}$
and the landing velocities along the horizontal plane $R_{\mathrm{HVel}}$,
vertical direction $R_{\mathrm{VVel}}$ and yaw rate $R_{\mathrm{YVel}}$.
$R_{\mathrm{LS}}$ is a discrete reward which takes on a value of -1 given a
successful landing or a value of 1 given a failed landing. A successful
landing is defined as one where the UAV lands with all four legs on the
correct goal platform, where the intent of the user is to land and not inspect
the goal platform and that the difference between the yaw of the drone to that
of the platform’s orientation is within a threshold of $\pm 20$ degrees. For
$R_{\mathrm{HVel}}$, $R_{\mathrm{VVel}}$ and $R_{\mathrm{YVel}}$, the penalty
is based on the velocity term squared scaled by the height $H$ of the UAV
above the platform up to a threshold height $H_{T}$:

$R_{\mathrm{iVel}}=\begin{cases}\frac{(H_{T}-H)}{H_{T}}v_{i}^{2}&\text{if
$H<H_{T}$}\\\ 0&\text{else}\end{cases},$ (3)

where $v_{i}$ represents the specific velocity direction term. The full
equation for landing reward term is:
$R_{\mathrm{Landing}}=k_{0}R_{\mathrm{LS}}+k_{1}R_{\mathrm{HVel}}+k_{2}R_{\mathrm{VVel}}+k_{3}R_{\mathrm{YVel}}$
where the weight of the scaling coefficients $k_{0}$ – $k_{3}$ are given
values of 12.0, 30.0, 5.0 & 15.0 respectively.

Figure 11: Overview of assistant training architecture.

$R_{\mathrm{Inspection}}$ comprises of four terms: inspection pending time
penalty $R_{\mathrm{IT}}$, inspection success $R_{\mathrm{IS}}$, inspection
velocity $R_{\mathrm{IVel}}$ and inspection angular velocity
$R_{\mathrm{IA}}$. $R_{\mathrm{IT}}$ is included to promote the assistant in
taking images promptly after the pilot has requested one and is computed as:
$R_{\mathrm{IT}}=1.07^{T}$, where $T$ is the number of timesteps that have
elapsed since the pilot has requested an image to be taken. For
$R_{\mathrm{IS}}$, a success is considered if all four corners of the goal
platform are present within the combined image representation and that the
relative yaw error between the UAV and goal platform is within a threshold of
$\pm 20$ degrees. $R_{\mathrm{IS}}$ is calculated as:

$R_{\mathrm{IS}}=\begin{cases}\frac{-P}{WH}&\text{if success}\\\
1&\text{else}\end{cases},$ (4)

where $P$ is the total pixels within the combined image representation that
correspond to a platform while $W$ and $H$ are the width and height of the
combined image. $R_{\mathrm{IS}}$ is scaled by the proportion of platform
pixels within an image to ensure the assistant takes a close-up image in order
to capture as much detail as possible. $R_{\mathrm{IVel}}$ and
$R_{\mathrm{IA}}$ are included to ensure the assistant takes a photo whilst
the UAV is stationary to avoid motion blur and is calculated as
$R_{\mathrm{IVel}}=V^{2}$ and $R_{\mathrm{IA}}=\omega^{2}$, where $V$ and
$\omega$ are the corresponding linear and angular velocities. The full
equation for the inspection reward term is:
$R_{\mathrm{Inspection}}=k_{4}R_{\mathrm{IT}}+k_{5}R_{\mathrm{IS}}+k_{6}R_{\mathrm{IVel}}+k_{7}R_{\mathrm{IA}}$
where the weight of the scaling coefficients $k_{4}$ – $k_{7}$ are given
values of 0.4, 50.0, 150.0 & 150.0 respectively.

$R_{\mathrm{Cooperation}}$ comprises of four terms: XYZ action difference
between the pilot and assistant $R_{\mathrm{XYZ}}$, the yaw rate action
difference between the pilot and assistant $R_{\mathrm{Yaw}}$, the red light
action output of the assistant $R_{\mathrm{R}}$ and the green light action
output of the assistant $R_{\mathrm{G}}$. $R_{\mathrm{R}}$ and
$R_{\mathrm{G}}$ are given a value of 1 when the respective light is turned
on, and 0 otherwise. The full equation for the cooperation reward term is:
$R_{\mathrm{Cooperation}}=k_{8}R_{\mathrm{XYZ}}+k_{9}R_{\mathrm{Yaw}}+k_{10}R_{\mathrm{R}}+k_{11}R_{\mathrm{G}}$,
where the scaling coefficients $k_{8}$ – $k_{11}$ are given values of 0.45,
0.3, 0.07 & 0.07 respectively.

For training the Temporal-Decoder we utilize five different losses to
condition the Temporal-Encoder. 1) The mean squared error between the
Temporal-Decoder’s estimated linear velocity and the UAV’s actual linear
velocity: $L_{LV}=MSE(\hat{V},V)$. 2) The mean squared error between the
Temporal-Decoder’s estimated angular velocity and the UAV’s actual angular
velocity: $L_{AV}=MSE(\hat{\omega},\omega)$. 3) The mean squared error between
the Temporal-Decoder’s estimate of the time since the pilot has requested an
image yet an image has not been taken to that of the actual outstanding
capture time: $L_{OCT}=MSE(\hat{OCT},OCT)$. 4) The binary cross entropy loss
between the Temporal-Decoder’s estimate of whether the simulated user is
currently in the red state and the simulated user’s red state variable:
$L_{RS}=BCE(\hat{RS},RS)$. 5) The binary cross entropy loss between the
Temporal-Decoder’s estimate of whether the simulated user is currently in the
green state and the simulated user’s green state variable:
$L_{GS}=BCE(\hat{GS},GS)$. The total loss for the Temporal-Decoder in training
step (1) in Section V-C1 is defined as:
$L_{TD}=k_{12}L_{LV}+k_{13}L_{AV}+k_{14}L_{OCT}+k_{15}L_{RS}+k_{16}L_{GS}$
where $k_{12}-k_{16}$ are given values of 1.0, 1.0, 5.0, 1.5, 1.5
respectively.

During training, the exploration process is parallelised with 24 concurrent
simulated UAVs flying within a single Unreal Engine instance where the UAV
simulation plugin AirSim has its simulation clock speed sped up by a factor of
2. An iteration is performed every 200ms in simulated time (100ms in real-
life). For exploratory noise an Ornstein-Uhlenbeck process was used which was
successively decayed after each epoch.

The target XYZ velocity and yaw rate that the simulated UAV is to follow is
calculated by averaging the simulated users and assistant’s actions, after
which clipped normally distributed noise is added, with the variance of the
normally distributed noise is derived from the current distance from the
ground to emulate disturbances caused from the ground effect. After each
epoch, policy module optimisation iterations are performed in accordance to
the number of state transitions observed during the epoch using a batch size
of 64. An epoch is considered complete when all 24 UAVs have completed their
randomly generated mission list containing [1-3] tasks or if a total of
3-minutes of simulation time has elapsed. Training and model validation
results are described in Section VIII-C2.

##  Appendix C Learning effect regression result

TABLE VI: Learning effect regression model

|  $A$ |  $AL$ |  $AH$ |  $CO$
---|---|---|---|---
Target approach
Time / init. dist. | 6.31 | 6.50 | 6.98 | -0.82
Dist. traveled / init. dist. | 1.40 | 1.43 | 1.55 | -0.11
Landing task
Time / init. dist. | 15.60 | 16.17 | 17.62 | -1.83
Dist. traveled / init. dist. | 2.39 | 2.54 | 2.75 | -0.21
Inspection task
Time / init. dist. | 12.31 | 10.19 | 13.43 | -1.25
Dist. traveled / init. dist. | 1.77 | 1.49 | 2.11 | -0.14

Regression coefficients for the model $Y=A+AL+AH+CO$.

Bolded values represent statistically significant values under a two-tailed
t-test of significance $\alpha=0.01$.

## References

  * [1] D. C. Schedl, I. Kurmi, and O. Bimber, “Search and rescue with airborne optical sectioning,” _Nature Machine Intelligence_ , vol. 2, pp. 783–790, 2020.
  * [2] M. JN, G. CJ, N. BM, and H. AP, “Using an unmanned aircraft system (drone) to conduct a complex high altitude search and rescue operation: A case study,” _Wilderness & environmental medicine_, vol. 30, pp. 287–290, 2019.
  * [3] A. Albanese, V. Sciancalepore, and X. Costa-Pérez, “Sardo: An automated search-and-rescue drone-based solution for victims localization,” _IEEE Transactions on Mobile Computing_ , vol. 21, no. 9, pp. 3312–3325, 2022.
  * [4] J. Nikolic, M. Burri, J. Rehder, S. Leutenegger, C. Huerzeler, and R. Siegwart, “A uav system for inspection of industrial facilities,” in _2013 IEEE Aerospace Conference_ , 2013, pp. 1–8.
  * [5] I.-H. Kim, S. Yoon, J. H. Lee, S. Jung, S. Cho, and H.-J. Jung, “A comparative study of bridge inspection and condition assessment between manpower and a uas,” _Drones_ , vol. 6, no. 11, 2022.
  * [6] L. Morando, C. T. Recchiuto, J. Calla, P. Scuteri, and A. Sgorbissa, “Thermal and visual tracking of photovoltaic plants for autonomous uav inspection,” _Drones_ , vol. 6, no. 11, 2022.
  * [7] N. Smolyanskiy and M. Gonzalez-Franco, “Stereoscopic first person view system for drone navigation,” _Frontiers in Robotics and AI_ , vol. 4, p. 11, 2017\.
  * [8] K. Backman, D. Kulic, and H. Chung, “Reinforcement learning for shared autonomy drone landings,” _CoRR_ , vol. abs/2202.02927, 2022.
  * [9] C. Wong, E. Yang, X.-T. Yan, and D. Gu, “Autonomous robots for harsh environments: a holistic overview of current solutions and ongoing challenges,” _Systems Science & Control Engineering_, vol. 6, no. 1, pp. 213–219, 2018.
  * [10] P. Xia, F. Xu, T. Zhou, and J. Du, “Benchmarking human versus robot performance in emergency structural inspection,” _Journal of Construction Engineering and Management_ , vol. 148, no. 8, p. 04022070, 2022.
  * [11] F. Perez-Grau, R. Ragel, F. Caballero, A. Viguria, and A. Ollero, “Semi-autonomous teleoperation of uavs in search and rescue scenarios,” in _2017 International Conference on Unmanned Aircraft Systems (ICUAS)_ , 2017, pp. 1066–1074.
  * [12] S. Li and X. Zhang, “Implicit intention communication in human–robot interaction through visual behavior studies,” _IEEE Transactions on Human-Machine Systems_ , vol. 47, no. 4, pp. 437–448, 2017.
  * [13] S. Nikolaidis, D. Hsu, and S. Srinivasa, “Human-robot mutual adaptation in collaborative tasks: Models and experiments,” _The International Journal of Robotics Research_ , vol. 36, pp. 618 – 634, 2017.
  * [14] V. Alonso and P. de la Puente, “System transparency in shared autonomy: A mini review.” _Frontiers in neurorobotics_ , vol. 12, p. 83, 2018.
  * [15] R. Polvara, M. Patacchiola, M. Hanheide, and G. Neumann, “Sim-to-real quadrotor landing via sequential deep q-networks and domain randomization,” _Robotics_ , vol. 9, no. 1, 2020.
  * [16] L. Bartolomei, Y. Kompis, L. Teixeira, and M. Chli, “Autonomous emergency landing for multicopters using deep reinforcement learning,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 3392–3399.
  * [17] A. Rodriguez-Ramos, C. Sampedro, H. Bavle, P. D. L. Puente, and P. Campoy, “A deep reinforcement learning strategy for uav autonomous landing on a moving platform,” _Journal of Intelligent & Robotic Systems_, vol. 93, pp. 351–366, 2019.
  * [18] J. Xie, X. Peng, H. Wang, W. Niu, and X. Zheng, “Uav autonomous tracking and landing based on deep reinforcement learning strategy,” _Sensors_ , vol. 20, no. 19, 2020.
  * [19] Y. Feng, K. Tse, S. Chen, C.-Y. Wen, and B. Li, “Learning-based autonomous uav system for electrical and mechanical & device inspection,” _Sensors_ , vol. 21, no. 4, 2021.
  * [20] Y. Sun and O. Ma, “Automating aircraft scanning for inspection or 3d model creation with a uav and optimal path planning,” _Drones_ , vol. 6, no. 4, 2022.
  * [21] A. Neagu, H. Wang, H. Zhou, H. Liu, Z. Huang, and M. Feng, “Research on determining the inspection point of multirotor uav power tower,” _Mathematical Problems in Engineering_ , vol. 2021, p. 8894055, 2021.
  * [22] I. Sa, S. Hrabar, and P. Corke, “Inspection of pole-like structures using a visual-inertial aided vtol platform with shared autonomy,” _Sensors_ , vol. 15, pp. 22 003–22 048, 2015.
  * [23] S. Reddy, A. Dragan, and S. Levine, “Shared autonomy via deep reinforcement learning,” in _Proceedings of Robotics: Science and Systems_ , 2018.
  * [24] D. Zhang, R. Tron, and R. P. Khurshid, “Haptic feedback improves human-robot agreement and user satisfaction in shared-autonomy teleoperation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 3306–3312.
  * [25] K. Backman, D. Kulić, and H. Chung, “Learning to assist drone landings,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 3192–3199, 2021\.
  * [26] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _International Conference on Learning Representations_ , 2016\.
  * [27] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in _Proceedings of the 35th Int. Conference on Machine Learning_ , vol. 80, 2018, pp. 1587–1596.
  * [28] W. Curran, R. Pocius, and W. D. Smart, “Neural networks for incremental dimensionality reduced reinforcement learning,” in _2017 IEEE/RSJ IROS_ , 2017, pp. 1559–1565.
  * [29] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in _Field and Service Robotics_. Springer International Publishing, 2018, pp. 621–635.
  * [30] S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” _Human mental workload_ , vol. 52, pp. 139–183, 1988.
  * [31] K. Konstantoudakis, K. Christaki, D. Tsiakmakis, D. Sainidis, G. Albanis, A. Dimou, and P. Daras, “Drone control in ar: An intuitive system for single-handed gesture control, drone tracking, and contextualized camera feed visualization in augmented reality,” _Drones_ , vol. 6, no. 2, 2022.
  * [32] A. Loquercio, A. I. Maqueda, C. R. del-Blanco, and D. Scaramuzza, “Dronet: Learning to fly by driving,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 2, pp. 1088–1095, 2018.