figurec

2D

    two-dimensional
3D

    three-dimensional
AHRS

    attitude and heading reference system
AUV

    autonomous underwater vehicle
COMA

    counterfactual multi-agent policy gradients
CPP

    Chinese Postman Problem
DoF

    degree of freedom
DVL

    Doppler velocity log
FSM

    finite state machine
IMU

    inertial measurement unit
LBL

    Long Baseline
MCM

    mine countermeasures
MDP

    Markov decision process
MCTS

    Monte Carlo Tree Search
POMDP

    Partially Observable Markov Decision Process
PRM

    Probabilistic Roadmap
RL

    Reinforcement learning
ROI

    region of interest
ROS

    Robot Operating System
ROV

    remotely operated vehicle
RRT

    Rapidly-exploring Random Tree
SLAM

    Simultaneous Localisation and Mapping
SSE

    sum of squared errors
STOMP

    Stochastic Trajectory Optimization for Motion Planning
TRN

    Terrain-Relative Navigation
UAV

    unmanned aerial vehicle
USBL

    Ultra-Short Baseline
IPP

    informative path planning
FoV

    field of view
CDF

    cumulative distribution function
ML

    maximum likelihood
RMSE

    Root Mean Squared Error
MLL

    Mean Log Loss
GP

    Gaussian Process
KF

    Kalman Filter
IP

    Interior Point
BO

    Bayesian Optimization
SE

    squared exponential
UI

    uncertain input
MCL

    Monte Carlo Localisation
AMCL

    Adaptive Monte Carlo Localisation
SSIM

    Structural Similarity Index
MAE

    Mean Absolute Error
RMSE

    Root Mean Squared Error
AUSE

    Area Under the Sparsification Error curve

# Multi-UAV Adaptive Path Planning Using Deep Reinforcement Learning

Jonas Westheider, Julius Rückin, Marija Popović This work was funded by the
Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
Germany’s Excellence Strategy - EXC 2070 – 390732324. Authors are with the
Cluster of Excellence PhenoRob, Institute of Geodesy and Geoinformation,
University of Bonn. Corresponding<EMAIL_ADDRESS>
###### Abstract

Efficient aerial data collection is important in many remote sensing
applications. In large-scale monitoring scenarios, deploying a team of
unmanned aerial vehicles (UAVs) offers improved spatial coverage and
robustness against individual failures. However, a key challenge is
cooperative path planning for the UAVs to efficiently achieve a joint mission
goal. We propose a novel multi-agent informative path planning approach based
on deep reinforcement learning for adaptive terrain monitoring scenarios using
UAV teams. We introduce new network feature representations to effectively
learn path planning in a 3D workspace. By leveraging a counterfactual
baseline, our approach explicitly addresses credit assignment to learn
cooperative behaviour. Our experimental evaluation shows improved planning
performance, i.e. maps regions of interest more quickly, with respect to non-
counterfactual variants. Results on synthetic and real-world data show that
our approach has superior performance compared to state-of-the-art non-
learning-based methods, while being transferable to varying team sizes and
communication constraints.

##  I Introduction

Efficient aerial data collection is crucial for mapping and monitoring
phenomena on the Earth’s surface. Unmanned aerial vehicles (UAVs) provide a
flexible, labour-, and cost-efficient solution for remote sensing applications
such as precision agriculture [1, 2, 3], wildlife conservation [4], and search
and rescue [5, 6]. For monitoring large terrains, replacing a single unmanned
aerial vehicle (UAV) with a multi-UAV system can improve spatial coverage,
versatility, and robustness to individual failures at lower overall cost [7].
However, to fully unlock its potential, a key challenge is planning UAV paths
cooperatively in complex environments, given on-board constraints on runtime
efficiency and communication.

This paper addresses active data collection using a team of UAVs in terrain
monitoring scenarios. Our goal is to map an initially unknown, non-homogeneous
binary target variable of interest on a 2D terrain, e.g. crop infestations in
an agricultural scenario or to-be-rescued victims in a disaster scenario,
using image measurements taken by the UAVs. We tackle the problem of multi-
agent  informative path planning (IPP): we plan information-rich paths for the
UAVs to cooperatively gather sensor data subject to constraints on energy,
time, or distance. Our motivation is to allow the UAVs to adaptively monitor
the terrain in areas of interest where information value is high.

Traditional approaches for data collection are non-adaptive and rely on
static, predefined paths. In multi-UAV coverage path planning [8], the terrain
is equally partitioned and a sweep pattern is assigned to each UAV. The main
drawback of such methods is that they assume uniformly distributed target
variables of interest, e.g. anomalies, hotspots, victims, and do not allow for
targeted inspection of specific areas. To address this, IPP methods [9, 1, 10,
11, 12, 13] have been proposed enabling online decision-making based on
incoming information. However, the runtime of these strategies usually scales
exponentially with the planning horizon, since they rely on evaluating many
candidate paths online. In team scenarios, reasoning about the other UAVs’
behaviours causes the number of evaluations to further grow exponentially with
the team size, which leads to intractable runtime complexity.

Figure 1: Our RL-based approach applied in a multi-UAV surface temperature
mapping scenario. The UAVs take images of the terrain (white transparent) and
communicate them (grey dashed arrows). Based on locally available information,
each UAV decides from a set of actions (orange arrows) where to take the next
measurement. Inset image: resulting trajectories for $4$ deployed UAVs
(different colours). By planning paths cooperatively, our approach enables
adaptively mapping warm (red) areas of interest on the field.

RL has emerged as a popular approach for efficiently learning online decision-
making in robotics [11, 14, 15, 12, 13, 16]. Recent works apply Reinforcement
learning (RL) for single-agent IPP to enhance path quality and computation
time for adaptive data collection [14, 11, 17, 18]. However, learning
informative paths for multiple cooperating agents remains an open challenge.
First studies show promising results [12, 13], but are limited to 2D action
spaces and do not address the credit assignment problem [19], i.e. how much
each agent contributes to the overall team performance, which adversely
impacts cooperation capabilities.

The main contribution of this paper is a novel multi-agent deep RL-based IPP
approach for adaptive terrain monitoring scenarios using UAV teams. Bridging
the gap between recent advances in RL and robotic applications, we build upon
counterfactual multi-agent policy gradients (COMA) [20] to explicitly address
the credit assignment problem in cooperative IPP. Our approach supports
decentralised on-board decision-making and achieves cooperative 3D path
planning with variable team sizes. We show that (i) our designed network input
representations are effective for multi-agent IPP in 3D action spaces; (ii)
our COMA-based algorithm accounts for credit assignment resulting in improved
planning performance; and (iii) our RL-based approach improves planning
performance compared to non-learning-based methods across varying team sizes
and communication constraints without re-training. To back up these claims, we
demonstrate the performance of our approach using synthetic and real-world
data in a thermal hotspot mapping scenario. We will open-source our code at:
https://github.com/dmar-bonn/ipp-marl.

##  II Related Work

Our work brings together multi-robot IPP for autonomous data collection and
advances in multi-agent RL. This section overviews relevant literature for
both, distinguishing between single- and multi-robot settings.

Informative path planning has been widely studied for efficient data gathering
using autonomous robots [9, 1, 12, 21, 13, 2, 11, 17]. Non-adaptive coverage
planners [8] monitor a terrain exhaustively based on pre-defined paths. In
contrast, we focus on adaptive strategies replanning paths online based on
incoming sensor data. Hollinger et al. [9] study the benefit of adaptive
online replanning to maximise the information value of sensor measurements
given a limited mission budget for single-vehicle underwater inspection tasks.
Recent works [21, 22] propose optimisation-based IPP methods for single-UAV
monitoring. However, these methods require computationally costly evaluations
of candidate paths’ expected information value. Extending them naïvely to
multiple UAVs yields exponentially scaling complexity which restricts
applicability to online applications.

Deploying cooperative robotic teams is beneficial for monitoring tasks over
large terrains such as post-disaster assessment [5, 6] and precision
agriculture [1, 2, 3]. We focus on decentralised planning strategies, where
robots make locally informed decisions on-board to enable robustness to
individual failures and scalability with the team size. Best et al. [23]
introduce a decentralised variant of Monte Carlo tree search over a joint
probability distribution of action sequences to plan individual paths for
active perception. Similar to our multi-UAV setup, Albani et al. [1] and
Carbone et al. [2] propose methods for monitoring areas of interest in crop
fields. The former use biology-inspired swarm behavior for adaptive IPP.
Despite promising results, their method relies on manual problem-specific
parameter tuning.

An alternative line of work leverages learning-based methods for active data
collection. Following the paradigm of centralised training and decentralised
policy execution, Li et al. [24] learn inter-robot communication policies to
exchange local robot information. Tzes et al. [10] propose a more general
multi-robot framework relying on graph neural networks. Both approaches are
trained via supervised imitation learning and require an expert to learn from.
We learn the desired team behaviour using deep RL, which enables flexible,
foresighted planning in various applications.

Reinforcement learning (RL) is increasingly utilised in robotics and UAV
applications [14, 11, 17, 18, 25]. Pirinen et al. [25] introduce a strategy
for finding an unknown goal region using a UAV based on limited visual cues.
Recently, RL has also been applied to IPP to efficiently replan robotic paths
online. Chen et al. [14] develop a graph-based deep RL method for exploration,
selecting map frontiers that reduce map uncertainty and travel time. However,
their approach is limited to 2D workspaces, while we consider 3D planning.
Other works reward agent actions that lead to high information gain [18] and
map uncertainty reduction in target regions [17]. In a similar problem setup
to ours, Rückin et al. [11] propose an IPP method combining deep RL and
sampling-based planning for adaptive UAV terrain monitoring in 3D workspaces.
Naïvely extending single-agent algorithms to multi-agent setups incurs
exponentially growing complexity with the number of agents.

Multi-agent RL for IPP is a relatively unexplored research area. Existing
approaches for cooperative team applications do not account for the credit
assignment problem [19], i.e. do not discriminate the contribution of one
agent to the overall team performance. Most works address related applications
including navigation [15], target assignment [26], and coverage planning [27].
Similar to us, Bayerlein et al. [13] propose a multi-agent RL-based IPP
approach that maximises harvested data without inter-agent communication.
Viseras and Garcia [12] allow agents to exchange information via a
communication module. Both works are limited to constant UAV altitudes
ignoring potentially varying sensor noise with altitude. Recently, Luis et al.
[16] proposed a deep Q-learning algorithm that supports learning cooperation
by penalising redundant measurements based on the inter-UAV distance. Their
reward design is tailored to pure exploration instead of adaptively monitoring
areas of interest.

These works independently assign hand-engineered individual agent rewards.
Thus, they do not fully account for the cooperative nature of IPP and require
manual reward tuning. In contrast, we propose a new approach solely relying on
generally applicable global rewards for the whole UAV team. We adapt the
counterfactual multi-agent (COMA) RL algorithm [20] to active robotic data
collection in 3D workspaces. This way, we explicitly assign credit to
individual agents during training. Our experimental results emphasise the need
for explicit credit assignment to achieve cooperative behaviour and verify
that it improves IPP performance.

##  III Problem Statement

We consider a team of homogeneous UAVs monitoring a flat terrain. The goal is
to plan information-rich UAV paths on-the-fly as new measurements arrive to
accurately and efficiently map regions of interest on the terrain given a
finite mission budget. We briefly describe the general multi-agent IPP
problem, our mapping strategy, and how to quantify information value for our
RL approach.

###  III-A Multi-Agent Informative Path Planning

We address the problem of multi-agent informative path planning (IPP)
optimising an information-theoretic criterion
$\mathrm{I}:\Psi^{N}\to\mathbb{R}^{+}$ over all $N$ UAV paths
$\psi=\\{\psi_{1},\ldots,\psi_{N}\\}$, where $\Psi$ is the set of all possible
individual UAV paths:

$\psi^{*}=\operatorname*{argmax}_{\psi\in\Psi^{N}}\mathrm{I}(\psi),\,\text{s.t.
}\mathrm{C}(\psi_{i})\leq B\ \forall\ i\in\\{1,\ldots,N\\}\,.$ (1)

Each set of UAV paths $\psi$ is composed of individual paths
$\psi_{i}=(\bm{p}_{i}^{0},\ldots,\bm{p}_{i}^{B})\in\mathrm{\Psi}$ of length
$B$ with 3D measurement positions $\bm{p}_{i}^{t}\in\mathbb{R}^{3}$ in an
equi-distant grid $\mathrm{P}$ of resolution $r_{P}$ over multiple altitudes
above the terrain. The function $\mathrm{C}:\Psi\to\mathbb{R}^{+}$ maps a UAV
path $\psi_{i}$ to its associated execution cost; in our work, a maximum
number $B\in\mathbb{N}^{+}$ of measurements taken along a path $\psi_{i}$.

Figure 2: Inter-UAV communication. The UAVs (filled circles) exchange their
current measurements (square footprints) with each other when closer than a
limited communication range (black dotted circle). Green and dotted red lines
indicate in-range and out-of-range communications, respectively. The UAVs use
the received measurements to update their local map states (top to bottom
row).

###  III-B Terrain Mapping

The UAVs map the terrain by taking images using downward-facing cameras. The
image information is processed, e.g. by semantic segmentation, communicated,
and fused into a terrain map. A measurement $z_{i}^{t}$ taken by UAV $i$ at
time step $t$ is a likelihood estimate projected onto the flat terrain. We
consider binary per-pixel classification and sequentially fuse $z_{i}^{t}$
using probabilistic occupancy grid mapping [28]. Each UAV $i$ stores a local
posterior map belief $\mathcal{\bm{M}}_{i}$ discretised into $M$ grid cells
$\mathcal{M}_{i}^{j}$ of resolution $r_{M}$. The altitude of the current
measurement position $\bm{p}_{i}^{t}$ determines the mapped field of view and
the mapping resolution. We align $r_{M}$ with the mapping resolution at the
lowest altitude and upsample higher-altitude measurements to $r_{M}$ to fuse
heterogeneous mapping resolutions together in $\mathcal{M}_{i}$. Similar to
Popović et al. [21], we leverage an altitude-dependent sensor model specifying
$p(z_{i}^{t}\,|\,\mathcal{M}_{i}^{j},\bm{p}_{i}^{t})$ to update a UAV’s local
posterior map belief over each $\mathcal{M}_{i}^{j}$ at time step $t$.

Fig. 2 visualises the inter-UAV communication protocol. All UAVs $i$ send
their measurements $z_{i}^{t}$ to another UAV $k$ and receive measurements
$z_{k}^{t}$ to update the individual local map beliefs over $\mathcal{M}_{i}$.
UAVs $i$ and $k$ share measurements, if
$\lVert\bm{p}_{i}^{t}-\bm{p}_{k}^{t}\rVert_{2}\leq D$, where
$D\in\mathbb{R}^{+}$ is a radius approximating a range-limited communication
channel. The communication message contains the UAV identifier $i$, its
position $\bm{p}_{i}^{t}$, and its collected measurement $z_{i}^{t}$. The
receiving UAV $k$ utilises $z_{i}^{t}$ and $p_{i}^{t}$ to update its local map
belief over $\mathcal{M}^{k}$.

###  III-C Utility Definition for Adaptive Terrain Monitoring

Our goal is to plan future measurement positions
$\psi_{i}^{t+1}=(\bm{p}_{i}^{t+1},\ldots,\bm{p}_{i}^{B})$ for each UAV $i$ at
time step $t$ from which the next measurements
$\\{z_{i}^{t+1},\ldots,z_{i}^{B}\\}$ maximally reduce the uncertainty of the
current posterior belief over a global map $\mathcal{M}$. The global map
$\mathcal{M}$ contains measurements $z_{i}^{0:t}$ taken by the $N$ UAVs up to
the current time step $t$. We quantify the uncertainty reduction of a set of
measurements $z^{t+1}=\\{z_{1}^{t+1},\ldots,z_{N}^{t+1}\\}$ taken at the next
time step $t+1$ by computing the entropy reduction over $\mathcal{M}$ after
fusing $z^{t+1}$. Then, the information criterion $\mathrm{I}(\psi^{t+1})$ is
computed as the summed entropy reduction along paths $\psi^{t+1}$:

$\displaystyle\mathrm{I}(\psi^{t+1})=\sum_{m=0}^{B-t-1}$ $\displaystyle
H(\mathcal{M}\,|\,z^{0:t+m},\bm{p}^{0:t+m})-$ (2) $\displaystyle
H(\mathcal{M}\,|\,z^{0:t+m+1},\bm{p}^{0:t+m+1})\,.$

We consider adaptive mapping with one interesting target class, i.e. one class
that holds information value, and a complement class. Thus, the map entropy
$H(\mathcal{M}\,|\,z^{0:t},\bm{p}^{0:t})$ at time step $t$ is weighted by the
importances of the interesting and the uninteresting classes:

$\displaystyle
H(\mathcal{M}\,|\,z^{0:t},\bm{p}^{0:t})=\sum_{j=1}^{M}H(\mathcal{M}^{j}\,|\,z_{0:t},\bm{p}_{0:t})$
(3) $\displaystyle=$
$\displaystyle-\sum_{j=1}^{M}\mathrm{W}(\mathcal{M}^{j})\,p(\mathcal{M}^{j}\,|\,z^{0:t},\bm{p}^{0:t})\,\log\big{(}p(\mathcal{M}^{j}\,|\,z^{0:t},\bm{p}^{0:t}\big{)}$
$\displaystyle+\mathrm{W}(\overline{\mathcal{M}^{j}})\,p(\overline{\mathcal{M}^{j}}\,|\,z^{0:t},\bm{p}^{0:t})\,\log\big{(}p(\overline{\mathcal{M}^{j}}\,|\,z^{0:t},\bm{p}^{0:t})\big{)}\,,$

where
$p(\overline{\mathcal{M}^{j}}\,|\,z^{0:t},\bm{p}^{0:t})=1-p(\mathcal{M}^{j}\,|\,z^{0:t},\bm{p}^{0:t})$.
The importance weighting $W(\mathcal{M}^{j})$ is defined as:

$W(\mathcal{M}^{j})=\begin{cases}w_{1}&\text{if
}p(\mathcal{M}^{j}\,|\,z^{0:t},\bm{p}^{0:t})>0.5\,,\\\ w_{2}&\text{if
}p(\mathcal{M}^{j}\,|\,z^{0:t},\bm{p}^{0:t})<0.5\,,\\\
0.5&\text{else}\,,\end{cases}$ (4)

where $w_{1},w_{2}\geq 0$ and $w_{1}+w_{2}=1$. The importance weights $w_{1}$,
$w_{2}$ encourage the UAVs to plan paths $\psi^{t+1}$ targeting potentially
interesting regions according to the current posterior map belief
$p(\mathcal{M}^{j}\,|\,z^{0:t},\bm{p}^{0:t})$.

##  IV Our Approach

We present our novel multi-agent RL-based IPP approach for UAV teams. Our goal
is to plan UAV paths in a 3D workspace to achieve cooperative adaptive terrain
mapping. As shown in Fig. 3, we train agents offline based on global terrain
information using RL to learn UAV paths in a centralised way. A key aspect is
the integration of a counterfactual baseline, allowing us to estimate each
agent’s mapping contribution to the overall team performance and improve
cooperation. During a mission, we leverage the trained agent behaviour and
deploy a fully decentralised system to replan informative measurement
positions online.

###  IV-A RL for Sequential Decision-Making

We formulate the multi-UAV IPP task as a sequential decision-making problem
for a team of agents and train it using RL. The UAVs execute missions, where
at each time step $t$ each UAV $i$ simultaneously plans its next measurement
position $\bm{p}_{i}^{t+1}$ based on the current local on-board state
$\omega_{i}^{t}$. The local state $\omega_{i}^{t}$ includes the local map
belief $\mathcal{M}_{i}$ that sequentially fuses past (communicated)
measurements $z^{0:t}$ encoding the measurement history. This way, we account
for partial observability induced by communication constraints and noisy
sensor measurements. At each time step, all UAV actions, i.e. next measurement
positions, define the joint action $\bm{u}^{t}=(u_{1}^{t},\ldots,u_{N}^{t})\in
U^{N}$. The UAVs receive one global team reward $R^{t}:S\times U^{N}\times
S\rightarrow\mathbb{R}$ quantifying the joint information value of mapped
measurements $z^{t+1}=\\{z_{1}^{t},\ldots,z_{N}^{t}\\}$. The discounted return
$G^{t}=\sum_{k=0}^{B-t}\gamma^{k}R^{t+k}$, where $\gamma\in[0,1)$ is a
discount factor, measures the information value of the team’s paths from $t$
until the mission budget is spent. Although each UAV chooses actions in a
decentralised way, we evaluate the performance of the team as a whole to
enforce cooperative IPP behaviour.

Figure 3: Overview of our approach. At each time step during a mission, each
UAV takes a measurement and updates its local map state. The local map is
input to an actor network, which outputs a policy from which an action is
sampled. During training, a centralised critic network is additionally trained
using global map information and outputs Q-values for each action from the
current state, i.e. the expected future return.

State. The global environment state $s^{t}\in S$ is defined as
$s^{t}=\\{\mathcal{M}^{t},\bm{p}_{1:N}^{t},b\\}$. $\mathcal{M}^{t}$ captures
the current global map belief
$p(\mathcal{M}\,|\,z^{0:t}_{1:N},\bm{p}^{0:t}_{1:N})$, $\bm{p}_{1:N}^{t}$ are
the current agent positions, and $b\leq B$ is the remaining mission budget.
$s^{t+1}$ is the next state after the agents have moved to the next positions
$\bm{p}_{1:N}^{t+1}$ reached by executing actions $\bm{u}^{t}$.
$\mathcal{M}^{t+1}$ is updated based on measurements $z^{t+1}$ taken at
$\bm{p}_{1:N}^{t+1}$.

Actions. The agents move within a discrete position grid $\mathrm{P}$, bounded
by the environment borders and discrete minimum and maximum altitudes. Each
UAV $i$ selects an action $u_{i}$ from a discrete 3D action space $U$
containing movements $\\{up,north,east,south,west,down\\}$ with a fixed step
size. We prevent actions leading to movement outside of the environment or
UAVs having concurrent 2D terrain coordinates.

Reward. We explicitly design the reward function $R$ to reflect the
information criterion defined in Eq. 2 to adaptively and quickly reduce map
uncertainty in target areas:

$R^{t}(s^{t},\bm{u}^{t},s^{t+1})=\alpha\,\frac{H(\mathcal{M}^{t+1})-H(\mathcal{M}^{t})\,}{H(\mathcal{M}^{t})}+\beta\,.$
(5)

We reward the weighted map entropy reduction from the current to the next map
state $\mathcal{M}^{t+1}$ after mapping new measurements $z^{t+1}$. We
normalise the reward by the current map entropy to keep its magnitude
approximately constant during the mission and apply affine scaling factors
$\alpha$ and $\beta$ to improve training stability. Note that, for $\gamma=1$,
the return $G^{t}$ resembles the IPP criterion in Eq. 2 up to the
normalisation and scaling factors.

###  IV-B Algorithm

Our goal is to learn a policy enabling cooperative UAV team behaviour for the
adaptive monitoring task. To do this, we build upon the COMA RL algorithm of
Foerster et al. [20] to our IPP application for UAV teams. COMA is an actor-
critic algorithm using a centralised critic network to evaluate each agent’s
behaviour, described by the policy $\pi(\cdot\,|\,\omega_{i}^{t})$ and
parameterised by an actor network, and to optimise the policy accordingly. The
critic evaluates the current policy $\pi$ by estimating the agent’s $i$ state-
action value
$Q_{\pi}\big{(}s^{t},(u_{1}^{t},\ldots,u_{i}^{t},\ldots,u_{N}^{t})\big{)}$
given the other agents’ actions $\bm{u}_{-i}^{t}$. The critic network is
trained on-policy via TD($\lambda$) [29] to estimate the discounted return
$G_{t}$, introduced in Sec. IV-A, for taking the joint action $\bm{u}^{t}$
from the current state $s^{t}$ and following the current policy $\pi$
afterwards. The critic uses global information $s^{t}$ during training, while
the actor utilises only on-board information $\omega_{i}^{t}$ to predict the
next-best measurement position decentralised during both training and
deployment. We leverage the counterfactual baseline to assign credit to
individual agents according to their contribution to the team performance,
which fosters cooperative team behaviour. The advantage $A_{i}^{t}$ for agent
$i$ taking action $u_{i}^{t}$ in the team’s action $\bm{u}^{t}$ is:

$\small{A_{i}^{t}(s^{t},\bm{u}^{t})=Q_{\pi}(s^{t},\bm{u}^{t})-\sum_{{u^{\prime}}_{i}^{t}\in
U}\pi({u^{\prime}}_{i}^{t}\,|\,\omega_{i}^{t})\,Q_{\pi}\big{(}s^{t},(\bm{u}_{-i}^{t},{u^{\prime}}_{i}^{t})\big{)}\,.}$
(6)

The contribution of agent $i$ to the joint state-action value
$Q_{\pi}(s^{t},\bm{u}^{t})$ of the team’s action
$\bm{u}^{t}=(u_{1}^{t},\ldots,u_{i}^{t},\ldots,u_{N}^{t})$ by taking action
$u_{i}^{t}$ is estimated by marginalising over all possible individual actions
${u^{\prime}}_{i}^{t}\in U$ while keeping the other agents’ actions
$\bm{u}_{-i}^{t}$ fixed. For optimising $\pi(\cdot\,|\,\omega_{i}^{t})$, we
apply the policy gradient theorem using Eq. 6 and minimise:

$\mathcal{L}=-\log\pi(u_{i}^{t}|\omega_{i}^{t})\,A_{i}^{t}(s^{t},\bm{u}^{t})\,,$
(7)

using mini-batch stochastic gradient descent. For further details, we refer to
Foerster et al. [20].

###  IV-C Network Architecture & Feature Design

We propose new actor and critic representations to exploit COMA in 3D robotic
applications. As shown in Fig. 4, actor and critic are represented by neural
networks $f_{\theta^{\pi}}$ and $f_{\theta^{c}}$, parameterised by
$\theta^{\pi}$ and $\theta^{c}$. The actor network is conditioned on the agent
$i$ and its local state $\omega_{i}^{t}$. Feature inputs are the agent’s
identifier $i$, the remaining mission budget $b$, and the following spatial
inputs: (a) a position map centred around the agent’s position encoding the
map boundaries and the communicated other agents’ positions, where the values
represent the agent altitudes; (b) the local map state $\mathcal{M}_{i}$; (c)
the weighted entropy of the local map state
$H(\mathcal{M}_{i}\,|\,z^{0:t},\bm{p}^{0:t})$ in Eq. 3; (d) the weighted
entropy of the measurement $H(z_{i}^{t}\,|\,\bm{p}_{i}^{t})$; and (e) the map
cells currently spanned by all agents’ fields of view within the communication
range (’footprint map’).

Our critic network receives the same input (a)-(e) and, in addition, global
environment information accessible during training. Specifically, it further
receives: (f) a global position map encoding all agent positions
$\bm{p}_{1:N}^{t}$; (g) the global map state $\mathcal{M}$; (h) its weighted
entropy $H(\mathcal{M}\,|\,z^{0:t},\bm{p}^{0:t})$; (i) the map cells currently
spanned by all agents’ fields of view; and, to enable learning the
counterfactual baseline in Eq. 6, (j) the other agents’ actions. Inputs are
provided in the position grid resolution $r_{P}$. We downsample the map
resolution $r_{M}$ by $\frac{r_{P}}{r_{M}}$ to align both resolutions. The
scalar inputs $i$ and $b$ are expanded to constant-valued 2D feature maps.

To handle spatial information necessary for the terrain monitoring task, the
networks consist of convolutional encoders and multi-layer perceptron heads
predicting the policy $\pi$ and Q-values $Q_{\pi}$, respectively. Fig. 4-Top
illustrates the architecture of both networks. The actor’s logits
$f_{\theta^{\pi}}(x^{t}_{i})$ given a collection $x^{t}_{i}$ of feature maps
(a)-(e) above, are passed through a bounded softmax function
$\pi(u_{i}\,|\,x^{t}_{i})=(1-\epsilon)\,\mathrm{softmax}\big{(}f_{\theta^{\pi}}(x^{t}_{i})\big{)}+\frac{\epsilon}{|U|}$
to predict the stochastic policy. The hyperparameter $\epsilon\in[0,1]$
fosters exploration during training and is set to $0$ at deployment. The
critic outputs Q-values
$Q_{\pi}\big{(}s^{t},(u_{1}^{t},\ldots,u_{i}^{t},\ldots,u_{N}^{t})\big{)}$ for
each action $u_{i}^{t}\in U$ of agent $i$ after the last linear layer.

Figure 4: Architecture and inputs for our actor and critic. Both networks
consist of convolutional encoders and linear layers in the prediction head.
For the actor, the output is produced by a softmax layer (grey). Both actor
and critic receive local inputs (‘Actor input’). The critic also receives
global inputs for centralised training (‘Critic input’). All inputs (a)-(j)
are detailed in Sec. IV-C.

###  IV-D Network Training

We train an actor and a critic network offline and utilise the trained actor
online at deployment. We simulate UAV missions and learn from rewards received
after fusing measurements $z$ into the global map $\mathcal{M}$. Before each
mission, we generate a new terrain of resolution $r_{M}$. The terrains are
split into connected interesting and complementary uninteresting regions. The
split is randomly oriented and interesting regions cover between $30\%$ to
$60\%$ of the terrain to foster generalisation. We fix the initial UAV
positions and execute a mission until budget $B$ is spent. During training,
actions are sampled from the actor’s policy as described in Sec. IV-C, where
$\epsilon$ is linearly decreased from $0.5$ to $0.02$ over the first $10,000$
missions.

We alternate between generating $3,000$ environment interactions on-policy and
policy optimisation using Eq. 7. Both networks are optimised via stochastic
mini-batch gradient descent for $5$ epochs using the Adam optimiser with
learning rates of $1e^{-5}$ (actor) and $1e^{-4}$ (critic), and a batch size
of $600$. The critic network is trained to estimate the expected return
$G_{t}$ applying TD($\lambda$) with $\lambda=0.8$ and $\gamma=0.99$ using a
target critic network copying the critic network each $30,000$ environment
interactions.

##  V Experimental Results

We present experiments to show the capabilities of our multi-agent RL-based
IPP approach for adaptive terrain monitoring using UAV teams. Our experimental
results support our claims, which are: (i) our designed network
representations are effective for multi-agent IPP for UAVs in a 3D workspace;
(ii) accounting for the credit assignment problem via a counterfactual
baseline improves planning performance; and (iii) our approach outperforms
non-learning-based state-of-the-art approaches in terms of planning
performance across varying team sizes and communication constraints. Moreover,
we demonstrate our approach applied to a real-world surface temperature
monitoring scenario.

###  V-A Experimental Setup

For each experiment, we execute $50$ terrain monitoring missions with changing
regions of interest as described in Sec. IV-D. The terrains are of size $50$ m
$\times$ $50$ m with a map resolution of $r_{M}=10$ cm. We set the planning
resolution to $r_{P}=5$ m, bound altitudes between $5$ m and $15$ m, and use
camera field of views of $60^{\circ}$, so that adjacent measurements do not
overlap when taken from the lowest altitude. To account for increased sensor
noise at higher altitudes, we simulate
$p(z_{i}^{t}\,|\,\mathcal{M}_{i}^{j},\bm{p}_{i}^{t})$ to be
$\\{0.99,0.735,0.625\\}$ at $\\{5,10,15\\}$ m altitude. The UAV team consists
of $4$ agents and the communication radius is limited to $25$ m unless
reported otherwise. As metrics, we use the entropy of the map state
$H(\mathcal{M}\,|\,z^{0:t},\bm{p}^{0:t})$ to assess the map uncertainty and
the F1-score between the map state $\mathcal{M}$ and the ground truth map to
evaluate the correctness of $\mathcal{M}$. We report the mean and standard
deviation of these performance metrics in ground truth regions of interest
given $B=15$ measurements.

Figure 5: Feature input ablations. We systematically add (left) and remove
(right) components of our approach and show the map entropy reduction over the
mission time. The results show that the entropy map input has the largest
impact, while the footprint map only contributes to a better performance at
the end of a mission. Our full proposed network input (green) performs best.

###  V-B Ablation Study of Network Feature Design

In this section, we provide ablations to show that our proposed network input
representation is effective for multi-agent IPP for UAVs in a 3D workspace.
Our ablation studies systematically add and remove single input feature maps
to assess their effect on the overall planning performance.

Fig. 5 shows the mean map entropy reduction and standard deviation during the
mission with varying network input features. Steeper falling curves indicate
better planning performance. In Fig. 5-Left, we add single input feature maps
to base input features (a)-(d), as described in Sec. IV-C, which are necessary
to model the IPP problem. In Fig. 5-Right, we remove single input feature
maps.

As expected, the proposed full input (green) performs best, i.e. leads to the
lowest final map entropy. The entropy map feature (blue) is most beneficial
for planning performance, suggesting that it holds most relevant information
for learning informative paths. Interestingly, adding the footprint map (pink)
slightly harms planning performance, but leads to worse final performance when
removed from the input. This indicates that, when combined with the other
inputs, the footprint map has a larger contribution at the end of the mission.
Intuitively, this is because identifying the currently observed map cells,
which is essential for planning the next measurement positions, becomes harder
as more measurements are mapped. Adding the current measurement’s entropy
(red) decreases uncertainty faster during the first $\sim 12$ mission time
steps but harms final performance, presumably since this local feature causes
myopic planning bias. As part of the full input with enough global features,
however, this feature is significant for planning performance providing
additional map information in close proximity. Additionally, not masking
invalid actions (light green) and not centring the agents’ position map
(orange) impairs performance since this removes valuable spatial information
for planning. In sum, the results confirm that our network input features
(‘Full’) are most effective for multi-agent IPP for UAVs in a 3D workspace,
leading to superior planning performance of our proposed RL-based approach
compared to possible variants.

###  V-C Credit Assignment Mechanism Study

The experiments in this section show that accounting for the credit assignment
problem results in improved planning performance, which verifies the need for
explicit credit assignment mechanisms in cooperative RL-based multi-agent IPP.
We perform a systematic study comparing the UAV team’s IPP performance with
varying advantage functions in Eq. 6 and thus changing policy gradient updates
in Eq. 7. To this end, we compare our proposed approach (Sec. IV-B) against
three variants of itself, as described in the following.

Figure 6: Credit assignment study. We compare our COMA-approach (green) with
variants which do not explicitly tackle the credit assignment problem in terms
of map entropy (left) and F1-score (right). In later mission stages, our
approach allows for better cooperation for adaptive mapping when most of the
environment has already been explored.

Central-QV. Similar to Foerster et al. [20], we verify the effectiveness of
the counterfactual baseline in our adaptive IPP scenario by replacing the
baseline in Eq. 6 with a state value $V_{\pi}(s^{t})$. This way, we do not
estimate an individual agent’s contribution but consider the team’s joint
performance alone. We train two critic networks estimating $Q$ and $V$, both
as described in Sec. IV-D, and change the advantage function to
$A_{i}^{t}(s^{t},\bm{u}^{t})=Q_{\pi}(s^{t},\bm{u}^{t})-V_{\pi}(s^{t})$.

Actor-Independent. As described in Sec. IV-B, we utilise a centralised critic
exploiting global state information during training. However, to investigate
the effect of reasoning about the other team members’ actions, we now do not
account for them and exclude them from the critic network input. We adapt the
advantage function to depend solely on the agent’s own action:
$A_{i}^{t}(s^{t},u_{i}^{t})=Q_{\pi}(s^{t},u_{i}^{t})-\sum_{{u}_{i}^{{}^{\prime}t}\in
U}\pi(u_{i}^{{}^{\prime}t}\,|\,\omega_{i}^{t})Q_{\pi}(s^{t},u_{i}^{{}^{\prime}t})$.

Decentralised. In this variant, we remove all global information and consider
a purely decentralised critic based on local agent information
$\omega_{i}^{t}$ only. As for the actor-independent variant, the critic
ignores the other agents’ actions using the own agent’s state value as a
baseline. The advantage function reduces to:
$A_{i}^{t}(\omega_{i}^{t},u_{i}^{t})=Q_{\pi}(\omega_{i}^{t},u_{i}^{t})-\sum_{{u}_{i}^{{}^{\prime}t}\in
U}\pi(u_{i}^{{}^{\prime}t}\,|\,\omega_{i}^{t})Q_{\pi}(\omega_{i}^{t},u_{i}^{{}^{\prime}t})$.

Fig. 6 shows the planning performance using different advantage functions. We
focus on performance at later stages of the mission, when most of the
environment is explored and cooperation is crucial for adaptive mapping. The
counterfactual baseline utilised in our approach (green) performs best,
indicating its effectiveness for achieving cooperative behaviour. All variants
lack explicit credit assignment mechanisms, which adversely impacts planning
performance irrespective of using centralised or decentralised critics. This
confirms that our COMA-based algorithm improves planning performance by
addressing the credit assignment problem.

Figure 7: Comparison of planning methods for $4$ agents and a communication range of $25m$. By planning paths cooperatively, our approach reduces map uncertainty (left) and improves map accuracy (right) quickest, indicating its superior performance. TABLE I: Robustness to varying team sizes and communication constraints. We report the mean and standard deviation of entropy and F1-score over $50$ trials after $33\%$, $66\%$, and $100\%$ of the mission time. Best results in bold. Setting | Approach | 33% Entropy $\downarrow$ | 67% Entropy $\downarrow$ | 100% Entropy $\downarrow$ | 33% F1 $\uparrow$ | 67% F1 $\uparrow$ | 100% F1 $\uparrow$
---|---|---|---|---|---|---|---
2 agents | Ours | $\textbf{0.8826}\pm 0.0267$ | $\textbf{0.7343}\pm 0.0548$ | $\textbf{0.5802}\pm 0.0451$ | $\textbf{0.3608}\pm 0.0661$ | $\textbf{0.5035}\pm 0.0768$ | $\textbf{0.6268}\pm 0.0436$
| IG | $0.9155\pm 0.0154$ | $0.8449\pm 0.0410$ | $0.7432\pm 0.0801$ | $0.3275\pm 0.0386$ | $0.4353\pm 0.0539$ | $0.5340\pm 0.0857$
| Coverage | $0.9163\pm 0.0472$ | $0.7845\pm 0.0628$ | $0.6970\pm 0.0332$ | $0.1603\pm 0.0871$ | $0.3687\pm 0.0912$ | $0.4863\pm 0.0418$
4 agents | Ours | $\textbf{0.7360}\pm 0.0349$ | $\textbf{0.4137}\pm 0.0294$ | $\textbf{0.2842}\pm 0.0408$ | $\textbf{0.5765}\pm 0.0168$ | $\textbf{0.7350}\pm 0.0268$ | $\textbf{0.7858}\pm 0.0308$
| IG | $0.8302\pm 0.0298$ | $0.6805\pm 0.0554$ | $0.5176\pm 0.0700$ | $0.5396\pm 0.0435$ | $0.6728\pm 0.0375$ | $0.7599\pm 0.0289$
| Coverage | $0.8615\pm 0.0762$ | $0.6614\pm 0.0317$ | $0.6052\pm 0.0396$ | $0.1603\pm 0.0870$ | $0.3687\pm 0.0913$ | $0.4864\pm 0.0418$
8 agents | Ours | $\textbf{0.5621}\pm 0.0235$ | $\textbf{0.2952}\pm 0.0403$ | $\textbf{0.2209}\pm 0.0459$ | $\textbf{0.7115}\pm 0.0336$ | $0.7859\pm 0.0322$ | $0.8171\pm 0.0283$
| IG | $0.6887\pm 0.0234$ | $0.4886\pm 0.0460$ | $0.3077\pm 0.0446$ | $0.6957\pm 0.0260$ | $\textbf{0.7881}\pm 0.0194$ | $\textbf{0.8576}\pm 0.0151$
| Coverage | $0.8185\pm 0.0832$ | $0.6072\pm 0.0274$ | $0.5149\pm 0.0309$ | $0.2455\pm 0.1281$ | $0.5246\pm 0.0358$ | $0.5800\pm 0.0433$
Zero communication | Ours | $\textbf{0.7638}\pm 0.0408$ | $\textbf{0.5445}\pm 0.0674$ | $\textbf{0.3699}\pm 0.0486$ | $\textbf{0.5476}\pm 0.0256$ | $\textbf{0.6846}\pm 0.0327$ | $\textbf{0.7620}\pm 0.0282$
| IG | $0.8284\pm 0.0283$ | $0.6814\pm 0.0529$ | $0.5286\pm 0.0775$ | $0.5396\pm 0.0453$ | $0.6510\pm 0.0442$ | $0.7151\pm 0.0432$
Limited communication | Ours | $\textbf{0.7360}\pm 0.0349$ | $\textbf{0.4137}\pm 0.0294$ | $\textbf{0.2842}\pm 0.0408$ | $\textbf{0.5765}\pm 0.0168$ | $\textbf{0.7350}\pm 0.0268$ | $\textbf{0.7858}\pm 0.0308$
| IG | $0.8302\pm 0.0298$ | $0.6805\pm 0.0554$ | $0.5176\pm 0.0700$ | $0.5396\pm 0.0435$ | $0.6728\pm 0.0375$ | $0.7599\pm 0.0289$
Full communication | Ours | $\textbf{0.7428}\pm 0.0276$ | $\textbf{0.4623}\pm 0.0602$ | $\textbf{0.3674}\pm 0.0843$ | $\textbf{0.5818}\pm 0.0294$ | $\textbf{0.7077}\pm 0.0481$ | $0.7524\pm 0.0465$
| IG | $0.8294\pm 0.0290$ | $0.6809\pm 0.0501$ | $0.5266\pm 0.0593$ | $0.5389\pm 0.0449$ | $0.6721\pm 0.0432$ | $\textbf{0.7606}\pm 0.0306$

###  V-D Comparison against Non-Learning-based Approaches

The following experiments back up our claim that our RL-based approach
outperforms state-of-the-art non-learning-based multi-agent IPP approaches. We
compare our approach against three methods: (i) an adaptive information gain
approach for UAV swarms proposed by Carbone et al. [2], which selects a UAV’s
action greedily by maximising the estimated map entropy reduction without
explicit credit assignment (‘IG’); (ii) a non-adaptive coverage lawnmower-like
pattern with equidistant ($5$ m) measurement positions at the best-performing
altitude (‘Coverage’); (iii) non-adaptive random exploration sampling UAV
actions uniformly at random (‘Random’). All approaches share the same state
space, action space, and action masking strategy.

Fig. 7 shows the evaluation metrics over the mission time for the considered
approaches. The information gain-based strategy (blue) and our RL-based
approach (green) outperform the non-adaptive methods as they can actively
focus on regions of interest. The superior performance of our approach
confirms the benefits of addressing the credit assignment problem and the
applicability of our learning-based planning to varying terrains.

Next, we show the generalisability of our learning-based approach to different
team sizes and communication settings. We deploy our actor trained on the
$4$-agent setting with communication limited to $25$ m in scenarios with $2$,
$4$, and $8$ agents, and communication radii of $0$ m, $25$ m, and unlimited
communication without re-training. Table I shows the planning performance of
our approach compared to the information gain-based and the coverage method.
As expected, more agents lead to faster mapping, i.e. entropy decrease and
F1-score increase. Though changing communication radii leads to small
performance drops, our approach consistently outperforms both. This verifies
that our RL approach generalises to varying numbers of agents and
communication requirements without re-training, showcasing its broad
applicability without additional training costs.

###  V-E Temperature Mapping Scenario

We demonstrate the performance of our approach in a surface temperature
monitoring scenario using real-world data of a $40$ m $\times 40$ m crop field
near Jülich, Germany. The field data was collected with a DJI Matrice 600 UAV
carrying a Vue Pro R 640 thermal sensor and is shown in Fig. 8-Left. We
discretise the terrain into a $500\times 500$ grid map with resolution
$r_{M}=8$ cm. The planning grid resolution is $r_{P}=4$ m to guarantee the
same network input dimensions as used for training. Regions with surface
temperature $\geq 25^{\circ}$ C are considered as being interesting for
adaptive hotspot mapping. Fig. 8-Right shows the map entropy reduction over
$B=15$ measurements for $4$ agents and a communication radius of $25$ m, using
our approach compared to the methods introduced in Sec. V-D. Note that our
approach is trained solely in simulation on synthetic data as described in
Sec. IV-D. Our approach clearly outperforms all other methods showcasing its
applicability for real-world terrain monitoring missions.

Figure 8: Real-world evaluation. We deploy our approach on surface temperature
data (left) and compare the map entropy reduction of the considered planning
methods. Warmer regions are considered as interesting. Although our approach
is trained on synthetic data, it outperforms all other methods. Fig. 1 shows
the resulting UAV paths planned in this experiment.

##  VI Conclusions and Future Work

In this paper, we introduced a novel multi-agent deep RL-based IPP approach
for adaptive terrain monitoring using UAV teams. Our method features new
network representations and exploits a counterfactual baseline to address the
credit assignment problem for cooperative UAV path planning in a 3D workspace.
This allows us to successfully outperform state-of-the-art non-learning-based
approaches in terms of monitoring efficiency, while generalising to different
mission settings without re-training. Experiments using UAV-acquired thermal
data validate the real-world applicability of our approach. Future work will
investigate heterogeneous robot teams and dynamically growing maps in
environments of unknown bounds.

## References

  * Albani et al. [2018] D. Albani, T. Manoni, D. Nardi, and V. Trianni, “Dynamic UAV Swarm Deployment for Non-Uniform Coverage,” in _Proc. of the Intl. Conf. on Autonomous Agents and Multi-Agent Systems (AAMAS)_ , 2018.
  * Carbone et al. [2022] C. Carbone, D. Albani, F. Magistri, D. Ognibene, C. Stachniss, G. Kootstra, D. Nardi, and V. Trianni, “Monitoring and mapping of crop fields with UAV swarms based on information gain,” in _Distributed Autonomous Robotic Systems_. Springer, 2022, pp. 306–319.
  * Maes and Steppe [2019] W. H. Maes and K. Steppe, “Perspectives for Remote Sensing with Unmanned Aerial Vehicles in Precision Agriculture,” _Trends in Plant Science_ , vol. 24, pp. 152–164, 2019.
  * Bondi et al. [2018] E. Bondi, D. Dey, A. Kapoor, J. Piavis, S. Shah, F. Fang, B. Dilkina, R. Hannaford, A. Iyer, L. Joppa, and M. Tambe, “AirSim-W: A Simulation Environment for Wildlife Conservation with UAVs,” in _Proc. of the ACM SIGCAS Conf. on Computing and Sustainable Societies_ , 2018.
  * Goodrich et al. [2008] M. A. Goodrich, B. S. Morse, D. Gerhardt, J. L. Cooper, M. Quigley, J. A. Adams, and C. Humphrey, “Supporting wilderness search and rescue using a camera-equipped mini UAV,” _Journal of Field Robotics (JFR)_ , vol. 25, no. 1-2, pp. 89–110, 2008.
  * Hayat et al. [2017] S. Hayat, E. Yanmaz, T. X. Brown, and C. Bettstetter, “Multi-objective UAV path planning for search and rescue,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2017.
  * Yan et al. [2013] Z. Yan, N. Jouandeau, and A. A. Cherif, “A Survey and Analysis of Multi-Robot Coordination,” _Intl. Journal of Advanced Robotic Systems_ , vol. 10, no. 12, p. 399, 2013.
  * Bähnemann et al. [2017] R. Bähnemann, D. Schindler, M. Kamel, R. Siegwart, and J. Nieto, “A decentralized multi-agent unmanned aerial system to search, pick up, and relocate objects,” in _Proc. of the Intl. Symposium on Robotic Research (ISRR)_ , 2017.
  * Hollinger et al. [2013] G. A. Hollinger, B. Englot, F. S. Hover, U. Mitra, and G. S. Sukhatme, “Active planning for underwater inspection and the benefit of adaptivity,” _Intl. Journal of Robotics Research (IJRR)_ , vol. 32, pp. 3–18, 2013.
  * Tzes et al. [2022] M. Tzes, N. Bousias, E. Chatzipantazis, and G. J. Pappas, “Graph Neural Networks for Multi-Robot Active Information Acquisition,” _arXiv preprint arXiv:2209.12091_ , 2022.
  * Rückin et al. [2022] J. Rückin, L. Jin, and M. Popović, “Adaptive Informative Path Planning Using Deep Reinforcement Learning for UAV-based Active Sensing,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2022\.
  * Viseras and Garcia [2019] A. Viseras and R. Garcia, “DeepIG: Multi-robot information gathering with deep reinforcement learning,” _IEEE Robotics and Automation Letters (RA-L)_ , vol. 4, no. 3, pp. 3059–3066, 2019.
  * Bayerlein et al. [2021] H. Bayerlein, M. Theile, M. Caccamo, and D. Gesbert, “Multi-UAV path planning for wireless data harvesting with deep reinforcement learning,” _IEEE Open Journal of the Communications Society (OJ-COMS)_ , vol. 2, pp. 1171–1187, 2021.
  * Chen et al. [2020] F. Chen, J. D. Martin, Y. Huang, J. Wang, and B. Englot, “Autonomous Exploration Under Uncertainty via Deep Reinforcement Learning on Graphs,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2020.
  * Fan et al. [2020] T. Fan, P. Long, W. Liu, and J. Pan, “Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios,” _Intl. Journal of Robotics Research (IJRR)_ , vol. 39, no. 7, pp. 856–892, 2020.
  * [16] S. Y. Luis, M. P. Esteve, D. G. Reina, and S. T. Marın, “Deep Reinforcement Learning applied to multi-agent informative path planning in environmental missions.”
  * [17] Y. Cao, Y. Wang, A. Vashisth, H. Fan, and G. A. Sartoretti, “CAtNIPP: Context-Aware Attention-based Network for Informative Path Planning,” in _Proc. of the Conf. on Robot Learning (CoRL)_.
  * Lodel et al. [2022] M. Lodel, B. Brito, A. Serra-Gómez, L. Ferranti, R. Babuška, and J. Alonso-Mora, “Where to look next: Learning viewpoint recommendations for informative trajectory planning,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_. IEEE, 2022\.
  * Agogino and Tumer [2004] A. K. Agogino and K. Tumer, “Unifying temporal and structural credit assignment problems,” in _Proc. of the Intl. Conf. on Autonomous Agents and Multi-Agent Systems (AAMAS)_ , 2004.
  * Foerster et al. [2018] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson, “Counterfactual multi-agent policy gradients,” in _Proc. of the Conf. on Advancements of Artificial Intelligence (AAAI)_ , 2018.
  * Popović et al. [2020] M. Popović, T. Vidal-Calleja, G. Hitz, J. J. Chung, I. Sa, R. Siegwart, and J. Nieto, “An informative path planning framework for UAV-based terrain monitoring,” _Autonomous Robots_ , vol. 44, pp. 889–911, 2020.
  * Blanchard and Sapsis [2022] A. Blanchard and T. Sapsis, “Informative path planning for anomaly detection in environment exploration and monitoring,” _Ocean Engineering_ , vol. 243, p. 110242, 2022.
  * Best et al. [2019] G. Best, O. M. Cliff, T. Patten, R. R. Mettu, and R. Fitch, “Dec-MCTS: Decentralized planning for multi-robot active perception,” _Intl. Journal of Robotics Research (IJRR)_ , vol. 38, pp. 316–337, 2019\.
  * Li et al. [2020] Q. Li, F. Gama, A. Ribeiro, and A. Prorok, “Graph Neural Networks for Decentralized Multi-Robot Path Planning,” in _Proc. of the IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS)_ , 2020.
  * Pirinen et al. [2022] A. Pirinen, A. Samuelsson, J. Backsund, and K. Åström, “Aerial View Goal Localization with Reinforcement Learning,” _arXiv preprint arXiv:2209.03694_ , 2022.
  * Qie et al. [2019] H. Qie, D. Shi, T. Shen, X. Xu, Y. Li, and L. Wang, “Joint Optimization of Multi-UAV Target Assignment and Path Planning Based on Multi-Agent Reinforcement Learning,” _IEEE Access_ , vol. 7, pp. 146 264–146 272, 2019.
  * Puente-Castro et al. [2022] A. Puente-Castro, D. Rivero, A. Pazos, and E. Fernandez-Blanco, “UAV swarm path planning with reinforcement learning for field prospecting,” _Applied Intelligence_ , vol. 52, pp. 14 101–14 118, 2022.
  * Elfes [1989] A. Elfes, “Using occupancy grids for mobile robot perception and navigation,” _Computer_ , vol. 22, no. 6, pp. 46–57, 1989.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_. MIT press, 2018.

  *[UAV]: unmanned aerial vehicle
  *[UAVs]: unmanned aerial vehicle
  *[IPP]: informative path planning
  *[RL]: Reinforcement learning
  *[COMA]: counterfactual multi-agent policy gradients