# Defined the predictors of the lightning over India by using artificial
neural network

Pradip Kumar Gautam
School of earth ocean and atmosphere sceinces
Indian Institude of Technology
Bhubaneswar, Odisha, India
<EMAIL_ADDRESS>
&Deweshvar Singh
School of earth ocean and atmosphere sceinces
Indian Institude of Technology
Bhubaneswar, Odisha, India
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

Lightning casualties cause tremendous loss to life and property. However, very
lately lightning has been considered as one of the major natural calamities
which is now studied or monitored with proper instrumentation. In this study,
the lightning characteristics over India has been studies by using daily data
LRTS (low resolution time saris) and monthly data HRMC (high resolution
monthly climatology) of LIS/OTD during the period 2000-2013. LIS (lightning
imaging sensor) provides smaller global coverage than OTD (Optical Transient
Detector) and it is still thought to have detected 90 percent of the world’s
lightning on annual basis. We have used ANN time series method ( a neural
network ) to analyze the time series and defined which one will be best
predictor of lightning over India. I have taken the the time series of
lightning as output(dependent ) and input (independent ) as k-index, AOD,
Cape, Relative humidity( 500 to 1000 pressure level) and Vertical integral of
the divergence of cloud frozen of water flux and Vertical integral of
divergence the cloud liquid of water flux.These input variable are analyzed by
GPR, SVM, RT and LR show approximately linear relation.

_K_ eywords Artificial neural network (ANN) $\cdot$ aerosol optical depth
(AOD) $\cdot$ convective available potential energy(cape) $\cdot$ Gaussian
process regression (GPR) $\cdot$ support vector machine (SVM) $\cdot$
regression trees(RT) and linear regression(LR)

.

##  1 Introduction

Natural disasters have been causing huge loss to life and property. However,
lightning casualties have not been given due attention in the recent history.
There are no doubt random studies and research going on, but it has not yet
become one of the prime focus of the disaster response forces unlike cyclones,
floods, forest fire, landslides etc. It has been recorded that 5259 people
lost their lives due to lightning strikes in India during 1979 to 2011. Among
the fatalities, 89 percent are male and are from the state of Maharashtra,
West Bengal, Uttar Pradesh and others (Singh Singh, 2015). Lightning has been
recognized as one of the most powerful, spectacular and all-pervasive
atmospheric hazards that mankind has encountered throughout history (Cooray et
al., 2007). Statistically it has been assessed that 1.4 billion flash rates
(responsible for lightning) have been recorded annually (both inter-cloud and
cloud to ground) over the entire Earth which translates into 44+/-5 lightning
flashes every second around the globe (Christian et al., 2003). It has also
been established that globally the frequency of lightning is the highest in
the region between 30 N to 30 S. Thus, India after Africa can be regarded as
one of the most vulnerable regions to lightning disasters.

Figure 1: The annualised distribution of total lightning activity (flash/per
km/year) (Christian et al., 2003)

Figure 2: Seasonality of Lightning and Mean Surface Temperature over India
during 1998-2002 (Kandalgaonkar et al., 2005).

##  2 Data and Methodology

TRMM (Tropical Rainfall Measuring Mission) which have five different type of
the sensor that is used for different purposes. It is flown on the Tropical
Rainfall Measuring Mission (TRMM) satellite as part of the National
Aeronautics and Space Administration’s (NASA) Earth Observing System (EOS).
The LIS sensor was designed with a higher sensitivity and spatial accuracy
than the OTD sensor. The LIS (Lightning Imaging Sensor and OTD (Optical
Transient Detector) are two of them which are used for this study. The LIS/OTD
gridded Lightning datasets consist of flash rate climatology, raw flashes,
scaled flashes, flash rate, and flash rate time series data which have uniform
flash detection efficiency, 93±4 % in night and 73±11 % day. The independent
variables as k-index, AOD, Cape, Relative humidity( at 500 to 1000 pressure
level) and Vertical integral of the divergence of cloud frozen of water flux
and Vertical integral of divergence the cloud liquid of water flux datasets
are taken from the era-5.

Variable | Dimension | Unit | Resolution oC
---|---|---|---
AOD | dimensionless | unitless | 1x1
Lightning | 5845x144x72 | count/$km^{2}$/day | 1x1
cape | 5845x144x72 | j/kg | 1x1
k-index | 5845x144x72 | c | 1x1
RH | 5845x144x72 | g/kg | 1x1
Vertical velocity | 5845x144x72 | m/sec | 1x1

###  2.1 Artificial Neural Network

The basic element of a FFNN is the neuron, which is a logical-mathematical
model that seeks to simulate the behaviour and functions of a biological
neuron. Figure 3 shows the schematic structure of a neuron. Typically, a
neuron has more than one input. The elements in the input vector P= (P1,P2
,…., PR)are weighted by elements (w1, w2,………,wj) of the weight matrix W
respectively.

Figure 3: structure of the multi-input neuron.

###  2.2 Training Algorithm and Parameter

####  2.2.1 Training Parameter

This algorithm typically requires more memory but less time. Training
automatically stops when generalization stops improving, as indicated by an
increase in the mean square error of the validation samples.

Parameter | Value
---|---
Epoch between displays | 2
Maximum epochs | 10
Time | second
Maximum Validation Failure | 0
Performance Goal | 0
Minimum Gradient error | $3.831e^{-08}$
Initial | $10^{-3}$
Maximum $\mu$ | $10^{10}$
No. of neurons used for hidden layer | 10

####  2.2.2 Activation Function

In this study, the log-sigmoid activation function is adopted. It can be given
by the following expression:

$f(x)=1/(1+e^{-x})$ (1)

###  2.3 Nonlinear auto-regressive model with exogenous inputs (NARX)

The nonlinear auto-regressive network with exogenous inputs (NARX) is a
recurrent dynamic network, with feedback connections enclosing several layers
of the network. The NARX model is based on the linear ARX model, which is
commonly used in time- series modelling. Predict series y (t) given d past
values of y(t) and another series x (t).

$y(t)=f(x(t-1),(t-2),...,x(t-d),y(t-1),y(t-2),....y(t-d))$ (2)

###  2.4 Correlation Coefficient (r)

The correlation coefficient indicates the strength and direction of a linear
relationship between two random variables.

$r=\frac{\sum_{i=1}^{n}((x{i}-\bar{x})(y{i}-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x{i}-\bar{x})^{2}(y{i}-\bar{y})^{2}}}$
(3)

The value of r is such that -1 < r < +1. The +ve and –ve signs are used for
positive linear correlations and negative linear correlations, respectively. A
correlation greater than 0.8 is generally described as strong, whereas a
correlation less than 0.5 is generally described as weak.

###  2.5 Mean Squared Error

Mean Squared Error is the average squared difference between outputs and
targets. Lower values are better and Zero means no error.

$MSE=\frac{\sum_{i=1}^{n}(t{i}-o{i})^{2}}{p}$ (4)

where ti target value, oi output and p-pattern. ANN divides the time series
data in to 70 percentage in training and 15 percentage in testing and 15
percentage in validation. I have done with 10 neurons (nodes) and with 2
hidden layer. After that ,I have trained the data by Levenberg-Marquardt
method because it has much accuracy. The ANN gives the first result, how the
each predictor perform with lightning which is given into the figure 5 . The
performance is nothing but current status of the training process. This graph
is showing the current status of the training process. The X - axis indicates
the number of iterations (5 Epochs). The Y- axis represents the MSE occurred
for each iteration. The line graph plotted in blue colour represents the
training results. The graph with green colour represents the validation.
Results and the graph in red colour represent the test results. The
performance graph is computed for every iteration in the training process and
the graph in which all the three results of training, validation and testing
coincide at almost all points is chosen to be the best performance. At that
point of time the training should be stopped and no further iteration should
be preceded. It means that no further training is required and if done, may
predict the results wrongly.

###  2.6 Root Mean Square Error

The root mean square error (RMSE) is a frequently used measure of the
differences between values predicted by a model and the values actually
observed. It measures average error, weighted according to the square of the
error. It does not indicate the direction of the deviation.

$RMSE=\sqrt{\frac{\sum_{i=1}^{n}(t{i}-o{i})^{2}}{p}}$ (5)

##  3 Result

###  3.1 Defined the variable which affects the lightning by LR, RT, GPR and
SVM

The regression learner app of the MATLAB software 2018a trains regression
models to predict data. Using this app, we can explore your data, select
features, specify validation schemes, train models, and assess results. we can
perform automated training to search for the best regression model type,
including linear regression models, regression trees, Gaussian process
regression models, support vector machines, and ensembles of regression trees.
Perform supervised machine learning by supplying a known set of observations
of input data (predictors) and known response (lightning). Use the
observations to train a model that generates predicted responses for new input
data.

Figure 4: This is the scatter plot of Lightning (dependent) and meteorological
variables by LR, GPR, SVM, RT models. Only those variable are here, which are
vary with lightning.

We have taken number of the meteorological data to establish the, best
predictor of the lightning. Firstly, The GPR model is showing 0.76 maximum
value of R2 from other Model in Table. 1. That means the response variable
lightning AOD, RH, Cape, K-Index, VIDCLWF and VIDCFWF are better correlated to
the lightning. The GPR is not showing maximum R value but also it is
representing the minimum RMSE (root mean square error), and maximum training
time. Prediction speed is less but higher than support vector machine.
Secondly, the regression trees model is representing the maximum R value. In
last we have combined the all model and trained that one, we are getting that.
It is not showing the better result than GPR and regression trees model. In
figure 1 the blue point is the actual data and the yellow one is the predicted
data.

Models | $R^{2}$ | R | RMSE | Prediction speed | Training Time
---|---|---|---|---|---
LR | 0.62 | 0.7874 | 0.00073867 | 260000 obs/sec | 3.4078sec
SVM | 0.59 | 0.7681 | 0.00067807 | 5200 obs/sec | 19.967 sec
GPR | 0.76 | 0.8717 | 0.00070017 | 8300 obs/sec | 322sec
RT | 0.65 | 0.8717 | 0.00072151 | 360000 obs/sec | 1.4704 sec
All Models | 0.62 | 0.7873 | 0.00084561 | 71000 obs/sec | 11.768sec

In table 1, we can see that the RMSE ( Root mean Square Error) and
R(correlation coefficient) and R2(square of the correlation coefficient) of
the variable is given. R-square is the percentage of the response variable
variation that is explained by a linear model. R-square is always between 0 to
100 percentage. 0 percentage indicates that the model explains none of the
variability of the response data around its mean.100 percentage indicates that
the model explains all the variability of the response data around its mean.
the higher the R-squared, the better the model fits our data.

###  3.2 experiment-1

In given fig 5 we can see that RH (Relative Humidity), AOD ( aerosol optical
depth) k-index, Cape (convective available Potential energy) VIDCFWF (
vertical Integral of the divergence of the cloud liquid frozen water flux) and
VIDCLWF (vertical Integral of the divergence of the cloud liquid water flux)
is showing best performance with lightning at 9 Epoch at which point the
training , testing and validation are coincide.

Figure 5: This is the performance of the predictor AOD, VIDCLFWF, KI, Cape,
VIDCLWF, RH, with lightning during training.

###  3.3 Experiment-2

In figure 6, the x axis is the epoch (iteration) and y-axis is gradient
descend, cross-entropy and validation fail.The figure 6 is showing the
training state of the lightning and and independent variables in which we can
tell about the gradient descend, cross entropy and validation. The performance
(1447e-09) indicating how much minimised errors occur during the training,
Gradient (3831e-08) indicating how much variance occurs in the error rate,
Mu(cross entropy with 1e-10) is the threshold value for each iteration which
is updated for each iteration and the Validation Check indicates whether the
currently completed iteration (has minimized error compared to the previous,
iterations.

Figure 6: This is the performance of the predictor AOD, VIDCLFWF, KI, Cape,
VIDCLWF, RH, with lightning during training.

###  3.4 Experiment-3

Figure 7 is the error histogram with 20 bin of the Training, validation and
Test during training processes of the Model. In these histogram , There is a
vertical yellow line in each which is represent the zero error . x- axis is
the error and y- axis is Instances( occurrence of number of the error) in each
figure, each predictor’s have range of error of the training, validation and
test.

Figure 7: This is error histogram by which we can tell model are best or fail.

###  3.5 experiment-4

Figure 8: This figure is representing that model is well fitting or not.

This ( figure-8) is the regression plots to know the correlation during
training,validation and Test, of lightning with each predictor’s. Regression R
values measure the correlation between outputs and targets. An R value of 1
means a close relationship and MSE vale is also very less that’s why fitting
is well. Regression R values of 0 means a random relationship.

###  3.6 experiment 5

The Figure 9 describes how the prediction errors are related in time. For a
perfect prediction model, there should only be one nonzero value of the auto
correlation function, and it should occur at zero lag. (This is the mean
square error.) This would mean that the prediction errors were completely
uncorrelated with each other (white noise). If there was significant
correlation in the prediction errors, then it should be possible to improve
the prediction, perhaps by increasing the number of delays in the tapped delay
lines. In this case, the correlations, except for the one at zero lag, fall
approximately within the 95percent confidence limits around zero, so the model
seems to be adequate. If even more accurate results were required, you could
retrain the network by clicking Retrain in ntstool. This will change the
initial weights and biases of the network, and may produce an improved network
after retraining.

Figure 9: This is the auto correlation of error data which averaged as it
divided by the number of samples in error data.

###  3.7 experiment-6

The figure 10 is showing correlation input and error 1. The error 1 is
subtraction of the target and output.This input-error cross-correlation
function illustrates how the errors are correlated with the input sequence
x(t). For a perfect prediction model, all of the correlations should be zero.
If the input is correlated with the error, then it should be possible to
improve the prediction, perhaps by increasing the number of delays in the
tapped delay lines. In this case, all of the correlations fall within the
confidence bounds around zero.

Figure 10: This is the auto correlation of error data which averaged as it
divided by the number of samples in error data.

###  3.8 experiment-7

This figure 11 displays the inputs, targets and errors versus time. It also
indicates which time points were selected for training, testing and
validation. The vertical yellow line is lie in between plus (blue color) mark
and point (blue color) mark in figure 6, that is representing error during
training. Similarly green and red color is indicating for validation and
testing respectively.

Figure 11: This is the time series response for the output element

##  4 Conclusion

In this experiment we have taken number of meteorological variables to defined
the relative variable of lightning by using four models LR, SVM, GPR, RT and
combine of thses four model. As a result cape, k-index, RH, AOD, vertical
velocity, Vertical integral of the divergence of cloud frozen of water flux
and Vertical integral of divergence the cloud liquid of water flux are showing
good relation with lightning which are defined on the basis of the value of R
and R 2 o f different model. Finally artificial neural network are used to
defined much better relation based on MSE and regression values which are
given in below table.The regression approximately one and mean square error is
very less. According to above the study,the MSE is very less as well as
regression value is nearly approximately one. So the result is that these
variables named cape, k-index, RH, AOD, vertical velocity, Vertical integral
of the divergence of cloud frozen of water flux and Vertical integral of
divergence the cloud liquid of water flux, are good predictor of the lighting
over India.

Data Type | Target Value | MSE | Regression
---|---|---|---
Training | 3581 | $1.37698e^{-9}$ | 0.99645
Validation | 767 | $1.44171e^{-9}$ | 0.99634
Testing | 767 | $1.36990e^{-9}$ | 0.99652

According to above the study,the MSE is very less as well as regression value
is nearly approximately one. So the result is that these variable are good
predictor of the lighting over India.

##  5 References

[1] Litta A. J, Sumam Mary Idicula and C. Naveen Francis 2012. Artificial
Neural Network Model for the Prediction of Thunderstorms over Kolkata.
International Journal of ComputerApplications (0975 – 8887). Volume 50 –
No.11, July 2012.

[2] Omvir Singh*, and Jagdeep Singh, Lightning fatalities over India:
1979–2011. Meteorol. Appl. 22: 770–778 (2015).

[3] Hugh J. Christian 1 , Richard J. Blakeslee 1 , Dennis J. Boccippio 1 ,
William L. Boeck 2, Dennis E. Buechler 3, Kevin T. Driscoll 3 , Steven J.
Goodman 1 , 1John M. Hall 4 ,William J. Koshak 1 , Douglas M. Mach 3 , and
Michael F. Stewart 3 , 2003. Global frequency and distribution of lightning as
observed from space by the Optical Transient Detector, JOURNAL OF GEOPHYSICAL
RESEARCH, VOL. 108, NO. D1, 4005, doi:10.1029/2002JD002347, 2003.

[4] D.M. Lal, S.D. Pawar, 2008. Relationship between rainfall and lightning
over central Indian region in monsoon and premonsoon seasons, Atmospheric
Research 92 (2009) 402–410.

[5] P. Murugavel, S. D. Pawar and V. Gopalakrishan*, Climatology of lightning
over Indian region and its relationship with convective available potential
energy. Int. J. Climatol. 34: 3179–3187 (2014).

[6] Philippe Lopez, Research Department, A lightning parameterization for the
ECMWF model, U.Schumann and H. Huntrieser, The global lightning-induced
nitrogen oxides source, Atmos. Chem. Phys., 7, 3823–3907, 2007.

[7] D. M. Lal 1 , · Sachin D. Ghude 1 · M. Mahakur 1 · R. T. Waghmare 1 · S.
Tiwari 1 Manoj K. Srivastava 2 · G. S. Meena 1. D. M. Chate 1 , 2000.
Relationship between aerosol and lightning over Indo-Gangetic Plain (IGP),
India, Clim Dyn (2018) 50:3865–3884.

[8] Daniel J. Cecil* , Dennis E. Buechler a , 2014, Richard J. Blakeslee b ,
Gridded lightning climatology from TRMM-LIS and OTD, Atmospheric Research
135–136 (2014) 404–414.

[9] Chen Lv, Member, IEEE, Yang Xing, Junzhi Zhang, Xiaoxiang Na, Yutong Li,
Teng Liu,Dongpu Cao, Member, IEEE and Fei-Yue Wang, Fellow, IEEE, Levenberg-
Marquardt Backpropagation Training of Multilayer Neural Networks for State
Estimation of A Safety Critical Cyber-Physical System, 1551-3203 (c) 2017
IEEE.

[10] Devendraa Siingh a , * , P.S. Buchunde a , R.P. Singh b , Asha Nath a ,
Sarvan Kumar b , R.N. Ghodpage c , Lightning and convective rain study in
different parts of India, Atmospheric Research 137 (2014) 35–48.


Safe and Efficient Manoeuvring for Emergency Vehicles in Autonomous Traffic using Multi-Agent Proximal Policy Optimisation

Leandro Parada a,1 CONTACT Leandro Parada. Email<EMAIL_ADDRESS>1  These authors contributed equally., Eduardo Candela a,1, Luis Marques a,  Panagiotis Angeloudis a
a Centre for Transport Studies, Department of Civil and Environmental Engineering, Imperial College London, London SW7 2AZ, United Kingdom

Manoeuvring in the presence of emergency vehicles is still a major issue for vehicle autonomy systems. Most studies that address this topic are based on rule-based methods, which cannot cover all possible scenarios that can take place in autonomous traffic. Multi-Agent Proximal Policy Optimisation (MAPPO) has recently emerged as a powerful method for autonomous systems because it allows for training in thousands of different situations. In this study, we present an approach based on MAPPO to guarantee the safe and efficient manoeuvring of autonomous vehicles in the presence of an emergency vehicle. We introduce a risk metric that summarises the potential risk of collision in a single index. The proposed method generates cooperative policies allowing the emergency vehicle to go at $15 \%$ higher average speed while maintaining high safety distances. Moreover, we explore the trade-off between safety and traffic efficiency and assess the performance in a competitive scenario.

Deep Reinforcement Learning,
Multi-Agent Reinforcement Learning,
Autonomous Traffic,
Autonomous Vehicles,
Emergency Vehicles,
Cooperative overtaking

§ INTRODUCTION

Autonomous Vehicles (AVs) can potentially help to reduce the response time and the accidents involving Emergency Vehicles (EMVs). However, many challenges related to the control of autonomous vehicles in traffic still remain. One of these major challenges is the safe and efficient manoeuvring in the presence of emergency vehicles. The interaction between AVs and EMVs in traffic generates unusual and complex multi-agent scenarios that are difficult to train for current vehicle autonomy systems.

EMVs move at high speeds, constantly overtaking other vehicles and passing through busy intersections. Moreover, ambulances and other EMVs can move in contraflow through the opposite lane in some countries, such as the UK. Those mentioned above are why many accidents involve EMVs, several of which have fatal consequences.

In 2020, 180 people died in the US in collisions involving EMVs. Collisions with police vehicles accounted for 132 deaths, 31 were related to ambulances, and 17 to fire trucks. Of these deaths, 69% were due to multi-vehicle crashes <cit.>. These numbers have been steadily increasing during the past years, as roads are becoming increasingly congested. In the UK, police-related road accidents reached a 13-year high in 2019 <cit.>. EMV accidents are also costly, both for repairs and legal services. In London, ambulance crashes cost more than $\pounds 2$ million a year, considering only repair and insurance costs, according to a Freedom of Information Act request <cit.>.

Manoeuvring in the presence of emergency vehicles is still a major challenge for current vehicle autonomy systems (SAE Automation Level 2 <cit.>). Several incidents have been reported in the last 5 years related to autonomy systems in urban settings. For instance, in May 2022 an autonomous car blocked a Fire Truck that was responding to an emergency, which caused property damage and personal injuries <cit.>. Moreover, vehicles using the Tesla Autopilot system have been repetitively reported to crash against EMVs <cit.>, which have resulted in several injuries and one death. Therefore, there is a need to develop algorithms that can guarantee the safe manoeuvring around EMVs.

A few studies have proposed methods for efficient overtaking for EMVs in autonomous traffic systems. The majority of these studies focus on determining the vehicle-passing sequence that minimises the delay of EMVs <cit.>. A recent study <cit.>  proposed a solution method based on a Genetic Algorithm to find the optimal passing sequence of EMVs in an intersection where all vehicles are connected autonomous vehicles. Using a slightly different approach, a study proposed a game-theoretic control method for efficient overtaking manoeuvres for autonomous EMVs in non-autonomous traffic <cit.>. The method considers both safety and efficiency and includes a behavioural model of the human driver's willingness to cooperate. A conceptual model for routing and scheduling EMVs in the context of smart cities has also been presented <cit.>. These methods are based on heavy assumptions of the environment, such as a set of predefined routes, or rule-based vehicle movement <cit.>.

Multi-agent Deep Reinforcement Learning (MARL) has emerged as a promising approach for the development of multi-AV driving policies <cit.>. MARL policies are represented using Deep Neural Networks (DNN), which can model complex Autonomous Driving behaviour that would otherwise require hand-crafted features or complicated rule-based methods. In addition, MARL allows for model-free learning—the agents can learn based on their experience; therefore, no underlying model of the process is required. This feature is especially convenient for multi-agent problems, where the interaction between agents and the evolution of the environment can be challenging to model. MARL has been successfully applied to different applications in transportation <cit.>, although the applications in multi-AV environments are still scarce <cit.>.

The overtaking of EMVs in an autonomous traffic system is fundamentally a multi-agent problem. To guarantee a safe and efficient passage of EMVs, all vehicle movements (including the EMV) should be optimised in real-time. To the best of the authors' knowledge, only two studies <cit.> have approached the EMV overtaking problem as a multi-agent problem. However, these studies are based on the assumption that a fixed set of rules governs the interaction between agents. Rule-based methods cannot possibly cover all situations that can occur in real-world traffic. Moreover, they cannot represent coordination between agents and adapt to changing environments.

In this study, we propose a method based on MARL to design safe and efficient policies for manoeuvring in the presence of EMVs. Specifically, we use the Multi-Agent Proximal Policy Optimisation (MAPPO) algorithm to generate real-time actions for all vehicles. MAPPO has recently shown to be superior to other MARL algorithms in complex cooperative environments <cit.>. We first formulate the overtaking of EMVs in autonomous traffic as a stochastic game, where each agent can take independent decisions based on partial observation of the environment. In order to guarantee safe manoeuvring by both the EMV and the AVs, we introduce safety metrics and use them as an objective for each agent. We then develop a traffic environment consisting of several AVs and one EMV driving through heavily congested roads/highways. We compare the proposed method with two baseline microscopic traffic models, highlighting the benefit of using a multi-agent learning algorithm. Moreover, we assess the trade-off between safety and traffic efficiency over the reward function. We show that the proposed method is scalable and can guarantee the safe passage of EMVs under heavy traffic conditions and competitive scenarios.

The contributions of this paper can be summarised as follows:

* This paper is the first to propose an approach for strategic manoeuvring in the presence of EMVs based on the multi-agent learning paradigm. Previous approaches are based on rule-based vehicle behaviour.

* It provides a new risk modelling approach based on a single risk metric that allows to avoid collisions and maintain safe distances between all vehicles in traffic.

* It provides a analysis on the trade-off between traffic efficiency (represented as the average speed) and safety (represented by the unifying risk metric).

We recognise that deploying fully autonomous EMVs will pose additional technical, regulatory and even ethical challenges. Nevertheless, it is crucial to improve how current AV technologies handle scenarios involving EMVs. Therefore, we focus on developing risk-aware AV controllers at the strategic decision-making level <cit.>, while paving the way for autonomous EMV controllers, which will be required in the future.

The rest of the paper is organised as follows. Section 2 summarises the relevant literature on the control of EMVs and autonomous traffic. Section 3 contains the risk modelling approach. Section 4 contains the MARL formulation. Section 5 presents the main results of the different case studies. Finally, section 6 presents the conclusion along with insights into future work.

Graphic representation of the multi-agent autonomous traffic approach. Each AV as well as the EMV correspond to an actor network. During the training phase, all agents share a centralised critic. During execution, the critic is no longer needed.

§ RELATED WORK

In this section, we present the relevant literature on the control of EMVs and autonomous traffic systems. We classify the works into four categories: Intelligent Signal Control (ITS), Autonomous Traffic and, Risk-aware Autonomous Systems and Multi-Agent Reinforcement Learning (MARL).

§.§ Intelligent Signal Control in presence of EMVs

Extensive work has been done regarding Intelligent Traffic Signal (ITS) control in the presence of EMVs, i.e., determining the best green light sequence. A study <cit.> presented an ITS method to optimise the travel time of EMVs that considers a priority system based on the type of incident. Similarly, a recent work <cit.> used a priority-based system to estimate the number of signal interventions while considering the impact on traffic in the surrounding areas. Virtual Traffic Lights (VTL) have also been used for efficient crossing of EMVs in autonomous traffic intersections <cit.>. In VTL, an AV acts as a leader and manages the traffic by serving as a virtual traffic light for other vehicles. Another study <cit.> combined a message-passing approach with a priority-based traffic signal method that minimises the delay of data packages. A method for scheduling both green light sequence and duration was proposed by  <cit.>. Their method is based on sensing information, such as the distance to the intersection and the number of cars on the road. Two articles <cit.> have provided joint methods to optimise the traffic signals and the vehicle movements. The latter couples a Decentralised Reinforcement Learning method to control the traffic signals with a dynamic Dijkstra's algorithm to find the optimal routes <cit.>.

§.§ Autonomous Traffic

Several studies have been conducted on optimising autonomous driving in traffic, particularly in highway scenarios <cit.>. These studies have focused almost exclusively on single-agent control, where traffic is considered part of the environment. A study proposed a game-theoretic traffic model and evaluated two single-agent policies based on Stackelberg games and Decision Trees <cit.>. Deep Reinforcement Learning has also been applied to generate high-level policies for the ego vehicle  <cit.>. A similar study <cit.> presented a Reinforcement Learning-based method that considers seven different autonomous driving goals in order to guarantee safe and efficient overtaking manoeuvres. The Q-learning algorithm has also been used to develop efficient single-agent lane-changing policies <cit.>. A recent study proposed a multi-agent game-theoretic planning approach for competitive racing scenarios <cit.>. In this approach, the agent has to collaborate to avoid collisions, but the overall goal is to beat its competitors. The study used an algorithm that solves for the approximate Nash Equilibrium. It assumed that the information of other vehicles' intentions and objective functions were completely known to every other vehicle.

Other studies have focused on the control of autonomous traffic in intersections <cit.>. These studies use different methods to determine the optimal passing sequence of AVs in intersections without traffic signals. The methods include Linear Programming <cit.>, Genetic Algorithms <cit.>, Polling Systems <cit.> and Decentralized Optimal Control <cit.>.

§.§ Risk-aware Autonomous Systems

Only limited research has been conducted on risk-aware AV controllers. A recent study <cit.> used the soft actor-critic (SAC) algorithm and control theory-based Lyapunov functions to train an agent subject to safety constraints. The authors in <cit.> proposed Parallel Constrained Policy Optimization (PCPO), which consists of three neural networks to estimate the policy function, value function, and a risk function. They use constraints and lack explicit risk models that capture risk when constraints are not violated. In another study <cit.>, risk metrics were coupled with a sampling-based trajectory planner. Trajectory planners require a probabilistic model to sample from, which is complex to obtain and can be overcome with model-free RL approaches. An RL method introduced hard constraints for Autonomous Driving <cit.>, where cars are not allowed to take certain actions, e.g., change lane when there's a car on the other lane. Similarly, another study <cit.> introduced an explicit short-horizon safety check to avoid making catastrophic decisions. In this study, we use state-of-the-art risk metrics for AVs to avoid collisions and maintain safe distances.

The field of RL that pertains to learning policies where safety is a critical objective is called Safe RL <cit.>. <cit.> segment existing literature into two categories: modification of the optimisation criteria or the exploration process. In some approaches, risk functions are learned during training, such as in <cit.>. To the best of our knowledge, no work has been carried out on Safe Multi-agent RL for AVs with modification of the optimisation criterion using explicit risk functions.

§.§ Multi-agent Reinforcement Learning for Autonomous Traffic systems

A couple of studies have introduced Multi-Agent autonomous driving gym environments for the purpose of evaluating learning algorithms.  <cit.> presented a Multi-Agent Connected Autonomous Driving Gym environment (MACAD) that allows the training of different Deep RL algorithms on an intersection scenario. The environment contains different features, such as the nature of agents (homogeneous/non-homogeneous), the nature of tasks (cooperative/competitive), and the nature of observations (full/partial). Similarly, <cit.> introduced a multi-agent gym environment for AVs that considers different traffic situations, such as lane merging, intersections, overtaking and roundabouts. It incorporates different controllers and baselines, as well as several evaluation metrics.

The use of MARL has been explored for different tasks in autonomous traffic.
<cit.> approached the double-merging problem using a Policy Gradient method. Cooperative lane-changing manoeuvres in the presence of Human-Driven Vehicles (HDVs) have also been explored <cit.>. Similarly,  <cit.> proposed a MARL method for highway ramp-merging that considers the presence of HDVs was explored. <cit.> presented a combination of Graph Convolutional Neural Networks (GCN) with the Deep Q Network (DQN) algorithm for safe and cooperative lane-changing decisions on highways. Despite these varied applications, MARL has not been applied for the safe manoeuvring of autonomous traffic in the presence of EMVs.

Summary of literature on the passing of EMVs in traffic.

Author     Approach     Autonomous Traffic     Rule-based     Scenario

<cit.>     Genetic Algorithm         x     Junction

<cit.>     Priority rules             Junction

<cit.>     Distance-based algorithm     x         Junction

<cit.>     Game-theoretic         x     Highway

<cit.>     Priority rules     x         Junction

<cit.>     Priority ITS     x         Junction

<cit.>     Reservation-based             Junction

<cit.>     Priority rules             Junction

§ MODELLING RISK

The deployment of AVs on roads poses significant safety risks. Moreover, training and validating AV controllers requires exposure to complex hazardous scenarios involving humans. Hence the necessity for risk-aware AV controllers that can better predict and avoid dangerous situations.

We introduce a single risk index $r\in[0,1]$ that represents the likelihood of a collision happening between any two vehicles moving on a two-dimensional surface. This index is the product of a longitudinal risk index $r_{lon}\in[0,1]$ and a lateral risk index $r_{lat}\in[0,1]$, which are piecewise linear functions that depend on the definitions of safe longitudinal and lateral distances , respectively <cit.>. The minimum safe longitudinal distance is

\begin{eqnarray}
d^{lon}_{min} = \left[ v_{r}\rho + \frac{1}{2}\rho^{2}a_{max} + \frac{(v_{r}
+ \rho a_{max})^{2}}{2b_{min}} - \frac{v_{f}^{2}}{2b_{max}} \right]_{+},
\end{eqnarray}

where $[x]_{+}:=\text{max}\{x,0\}$; $c_{r}$ is a car with velocity $v_{r}$ that drives behind another car $c_{f}$ (in the same direction) with velocity $v_{f}$. For any braking of $c_{f}$ of at most $b_{max}$, $c_{r}$ has a response time of $\rho$ during which it accelerates by at most $a_{max}$, and immediately after the response starts braking by at least $b_{min}$ until necessary. Notice that the closer the two vehicles are to colliding, $c_{r}$ will have to brake harder to avoid crashing, until its brakes reach their maximum capability (which we denote $B$) and a collision is unavoidable. When replacing $b_{min}$ with $B$ in $d^{lon}_{min}$, a tighter bound is defined for the safe longitudinal distance as

\begin{eqnarray}
d^{lon}_{min,brake} = \left[ v_{r}\rho + \frac{1}{2}\rho^{2}a_{max} + \frac{(v_{r}
+ \rho a_{max})^{2}}{2B} - \frac{v_{f}^{2}}{2b_{max}}
\right]_{+}.
\end{eqnarray}

Let $d^{lon}$ be the current longitudinal distance between cars. We define the longitudinal risk index

\begin{eqnarray}
\begin{array}{@{}ll@{}}
0, & \text{if}\ d^{lon} \geq d^{lon}_{min} > 0 \\
1-\frac{d^{lon} - d^{lon}_{min,brake}}{d^{lon}_{min} - d^{lon}_{min,brake}}, & \text{if}\ d^{lon}_{min} \geq d^{lon} \geq d^{lon}_{min,brake} > 0 \\
1, & \text{otherwise} \\
\end{array}\right.\qquad.
\end{eqnarray}

Similarly, for the lateral analysis, the safe lateral distance is

\begin{eqnarray}
d^{lat}_{min} = \left[ \frac{v_{left} + v_{left,\rho}}{2}\rho
+ \frac{v^{2}_{left,\rho}}{2b^{lat}_{min}}
- \left(\frac{v_{right} + v_{right,\rho}}{2}\rho
- \frac{v^{2}_{right,\rho}}{2b^{lat}_{min}} \right)
\right]_{+},
\end{eqnarray}

where w.l.o.g. car $c_{left}$ is to the left of $c_{right}$, $v_{left}$ and $v_{right}$ are their respective lateral velocities, $\rho$ is the same response time as in the longitudinal case, during which the two cars will apply a maximum lateral acceleration of $a^{lat}_{max}$ toward each other, and after that both cars will apply lateral braking of at least $b^{lat}_{min}$ until necessary; $v_{left,\rho} = v_{left} + \rho a^{lat}_{max}$ and $v_{right,\rho} = v_{right} - \rho a^{lat}_{max}$. As with the safe longitudinal distance, when replacing $b^{lat}_{min}$ with $B^{lat}$ (the maximum capable lateral braking of the vehicles) in $d^{lat}_{min}$, a tighter bound is defined for the safe lateral distance as

\begin{eqnarray}
d^{lat}_{min,brake} = \left[ \frac{v_{left} + v_{left,\rho}}{2}\rho
+ \frac{v^{2}_{left,\rho}}{2B^{lat}}
- \left(\frac{v_{right} + v_{right,\rho}}{2}\rho
- \frac{v^{2}_{right,\rho}}{2B^{lat}} \right)
\right]_{+}.
\end{eqnarray}

Let $d^{lat}$ be the current lateral distance between cars. We define the lateral risk index

\begin{eqnarray}
\begin{array}{@{}ll@{}}
0, & \text{if}\ d^{lat} \geq d^{lat}_{min} > 0 \\
1-\frac{d^{lat} - d^{lat}_{min,brake}}{d^{lat}_{min} - d^{lat}_{min,brake}}, & \text{if}\ d^{lat}_{min} \geq d^{lat} \geq d^{lat}_{min,brake} > 0 \\
1, & \text{otherwise} \\
\end{array}\right.\qquad.
\end{eqnarray}

When multiplying the lateral and longitudinal risk indices, and adding risk propensity parameters $\beta$, $\gamma > 0$, the unified risk index is

\begin{eqnarray}
r = \left(r_{lon}\right)^{\beta}\left(r_{lat}\right)^{\gamma},
\end{eqnarray}

which clearly lies in the range $[0,1]$, and only takes a value $>0$ when both longitudinal and lateral risk indices are also $>0$ simultaneously. The larger the risk propensity parameters are, the smaller the risk index becomes. When the propensity parameters are $<1$, the risk propensity becomes a risk aversion, since the longitudinal and lateral risk indices are increased.

$r=0$ if and only if a collision is impossible to occur in the future given the current states of the vehicles (i.e. positions, velocities and accelerations) and following the braking and response time assumptions.

$r=0 \iff r_{lon}=0 \lor r_{lat}=0 \iff (d^{lon} \geq d^{lon}_{min} > 0) \lor (d^{lat} \geq d^{lat}_{min} > 0)$, meaning at least one safety distance is being held.

If $r=1$, avoiding a potential collision cannot be guaranteed under the assumptions of safe longitudinal and lateral distances, independent of the risk propensity parameters $\beta$ and $\gamma$.

$r=1 \iff r_{lon}=1 \land r_{lat}=1 \iff (\geq d^{lon} \geq d^{lon}_{min,brake}) \land (\geq d^{lat} \geq d^{lat}_{min,brake})$, meaning both the longitudinal and lateral safety distances under maximum braking are being violated. If the other vehicle brakes, a collision can not be guaranteed to be avoided under the model conditions.

§ LEARNING APPROACH

This section presents our proposed model for the safe and efficient passing of EMVs in autonomous traffic. A diagram can be found in Figure <ref>. Most studies have addressed this problem based on rules and heavy simplifications of vehicle-to-vehicle interactions. Rule-based methods cannot cover all possible situations in autonomous traffic; therefore, we adopt a Multi-Agent Reinforcement learning approach. MARL allows us to simulate and train agents on thousands of different scenarios and achieve effective coordination among agents. In our formulation, all AVs and the EMV are represented as learnable agents that can take independent actions based on partial observability of the environment. We adopt the following assumptions and simplifications:

The environment considers a set of high-level discrete actions.
In this study, we are interested in the multi-agent coordination policies, rather than the ability of individual vehicles to follow the road and stay on track. Thus, we use a discrete action space that considers high-level actions; accelerate, brake and change lane. This is a common assumption of multi-agent autonomous driving simulators <cit.>, where vehicle dynamics are considered part of the environment.

All agents have partial observability of the environment.
Fixed-range AV sensors and the stochastic nature of vehicle-to-vehicle interactions cause AVs to have a limited perception of their surrounding environment <cit.>. We adopt a fixed range of 70m, which is realistic according to current communication protocols <cit.>.

§.§ Multi-Agent Deep Reinforcement Learning

Diagram of the risk-aware learning methodology

The MARL problem statement can be formalised as a Markov game $G$, defined by the tuple $G = (N, \Omega, S, A, \mathcal{P}, \mathcal{R} , \gamma )$ where $N$ corresponds to the number of agents, $\Omega$ to the set of individual observations, $S$ to the set of states, $A$ to the joint action space, $P$ to the transition probabilities, $R$ to the reward functions and $\gamma$ to the discount factor.

We implement the Multi-Agent Policy Optimization algorithm (MAPPO), a state-of-the-art actor-critic MARL method that has shown to achieve superior performance in several coordination-heavy tasks <cit.>. Proximal Policy Optimisation is a gradient-based method that approximates the policy directly. It includes a clipping objective $L^{Clip}$ that controls the policy update during the training period. This forces the policy update to be conservative if it deviates too much from the current policy. The actor and critic update equations for MAPOO are presented in the Appendix.

In contrast to the single-agent version, in MAPPO, all agents share a centralised critic during the training phase, but only the actors are needed in execution. Thus, it belongs to the Centralised Training Decentralized Execution (CTDE) class of algorithms. MAPPO is an on-policy method. Therefore, it does not rely on previous experience. This can make it less sample efficient than off-policy methods such as MADDPG <cit.> and QMIX <cit.>, at least in theory. In practice, MAPPO has been found to be a powerful and robust method in several cooperative environments, even outperforming state-of-the-art off-policy algorithms <cit.>. The implemented MAPPO algorithm is presented in Algorithm <ref>.

MAPPO for autonomous traffic control

Initialize $\theta$, the parameters for the critic $V$ and the parameters $\phi$ for the actor $\pi$.
$ep=1$ to $M$
Retrieve initial state $s_{0}, \Omega_{0}$: Individual speeds, positions, lanes and perception of surrounding vehicles.
$t=1$ to $T$
$j$ in $agents$
Choose high-level action $a_{t}^{(j)} = \pi (\Omega_{t}^{(j)}; \theta)$
Calculate value $v^{(j)}=V(s_{t}^{(j)}, \phi)$
Execute high-level actions and calculate new pose based on vehicle dynamics.
Observe $r_{t}, s_{t+1}, \Omega_{t+1}$.
Store $[s_{t}, \Omega_{t}, a_{t}, r_{t}, s_{t+1}, \Omega_{t+1}]$.

Compute advantage estimate $\hat{A}$ via GAE using the stored transitions.
Compute reward-to-go $\hat{R}$ on the stored transitions.
Update $\theta$ using Adam optimiser on mini-batch
Update $\phi$ using Adam optimiser on mini-batch

A graphic representation of the MAPPO approach for autonomous traffic is presented in figure <ref>. In the first instance, we consider a single critic for all agents during the training phase. A separate actor network allows agents to take individual actions during execution, without the need for a centralised state. Therefore the implementation of the proposed method results in decentralised multi-agent policies. We also explore using a separate critic network for each agent in a competitive version of the environment.

We represent the EMV as an actor taking independent actions in the environment, although we acknowledge it can be an autonomous or a human-driven vehicle. It should be noted that given the challenging and critical nature of emergency driving, it is not expected to have reliable autonomous EMVs in the coming years. Nevertheless, improving the safety of AV systems in the presence of EMVs is a crucial first step in the reliable deployment of AVs.

§.§ Emergency Vehicle Multi-Agent Environment

We use a bespoke Autonomous Traffic gym environment consisting of several AVs and one EMV driving through a congested road. In contrast to other gym environments modelling Autonomous Traffic, this environment considers all vehicles to be trainable agents with partial observability of their surroundings. Figure <ref> shows a representation of the environment. The environment was designed to ensure the safe and efficient transit of EMVs. To ensure this, we incorporate the risk metrics mentioned in the previous subsection, and performance metrics related to the speed of each vehicle.
All AVs and the EMV can take independent actions and receive an individual reward on each step. The reward function was designed to include the speed and the risk metrics of the EMV in order to guarantee cooperation between agents for prioritising the EMV.

Representation of the multi-agent environment

We define the individual elements of the environment as follows:

* Agents $N$: There are two types of agents; AVs and the EMV. The AVs all have the same physical characteristics, range of speeds and perception range. The EMV has different physical characteristics and is allowed to go at higher speeds than the AVs.

* Observation $\Omega$: The local observation of the AVs consists of their own speed, position, lane and a limited perception of surrounding vehicles. Thus, AVs can recognise when an EMV is approaching at a certain distance but cannot see its global position.

* Action $A$: All agents have a discrete action space with seven possible high-level decisions—accelerate; brake; heavy acceleration; heavy brake; change lane to the left; change lane to the right; keep previous velocity (no acceleration/braking). If an agent tries to perform a lane change but cannot for physical reasons, the action does nothing, and agents maintain their lane and speed.

* Reward $\mathcal{R}$: The reward function  is defined by the following equation:

\begin{eqnarray}
\mathcal{R} = r + v^{i} + p_{col} + p_{lcm} + v^{EV} + p_{lcm}^{EV}
\end{eqnarray}

Where r is agent's risk of collision as defined by equation (7), $v^{i}$ is the current speed measured in $m/s$, $p_{col}$ is a collision penalty and $p_{lcm}$ is a penalty for lane changing maneuvers.  In addition, $v^{EV}$ is EMV's current speed and $p_{lcm}^{EV}$ corresponds to an additional penalty for lane-changing manoeuvres. These last two components of the reward function guarantee cooperation among agents in the presence of the EMV. We normalised the individual components of the reward function between $0$ and $1$, except for the collision penalty, which we set to a very high value ($-100$) because of their relatively low frequency.

§ ANALYSIS

The proposed MARL method was tested on a long track, where agents can go in all lanes in a single direction. We test two different scenarios, a two-lane road and a four-lane highway. However, the environment can be easily adapted to a multiple-lane setting with traffic going in both directions. The EMV has to make its way through the traffic by either overtaking or by AVs moving to another lane. First, we evaluate the performance of the proposed method, comparing it against two Car Following Models. Second, we consider different variations of the reward function and evaluate the trade-off between collision risk and traffic efficiency. Third, we assess the scalability of the proposed method by evaluating scenarios with a different number of agents. Finally, we present a fully competitive case, where AVs have their own independent reward and neural networks.

We used the EMV Multi-Agent Environment to train all policies in simulation. We performed a preliminary hyperparameter tunning experiment considering an instance with ten agents (1 EMV and 9 AVs). The combination of the best parameters for the MAPPO are presented in table <ref>. The parameters for the risk indices were selected using the studies on acceleration <cit.>, they are presented in table <ref>. The MARL results were averaged over four different random seeds. The response time was set to the frame rate of the simulations, which was $10 Hz$. All experiments were conducted using a 10-core, 3.70Ghz Intel Core i9-10900X CPU and an Nvidia GTX 3090 GPU. Furthermore, the parameters for the EMV Multi-agent Environment are presented in table <ref>. The minimum velocity is set to $7m/s$, which

Environment parameters.

Parameter       Values

AV Max. Speed      $20 m/s$

AV Min. Speed     $7 m/s$

AV Dimensions     $2m \times 4m \times 1.5m$

EMV Max. Speed      $30 m/s$

EMV Min. Speed     $7 m/s$

Perception radius     $20 m$

Road loop length     $400 m$

Risk indices parameters

Parameter       Values

Response time $\rho$     $0.1s$

$a_{max}$     $2.5 m/s^{2}$

$b_{max}$       $2.5 m/s^{2}$

$b_{min}$     $1.0 m/s^{2}$

$B$     $3.0 m/s^{2}$

$a^{lat}_{max}$     $1 m/s^{2}$

$b^{lat}_{min}$     $2.5 m/s^{2}$

$B^{lat}$     $4.0 m/s^{2}$

$\beta$     1

$\gamma$     1

MAPPO parameters.

Parameter       Values

N° episodes for training     $2000$

N° steps per episode      $400$

Learning Rate     $5 \times 10^{-4}$

Critic Learning Rate     $5 \times 10^{-4}$

PPO epochs     $15$

Actor Network      $MLP(64, 64, 7)$

Critic Network     $MLP(64, 64, 1)$

§.§ Comparison to Car Following baselines

We compare the proposed MARL approach against two baselines: the Gipps Car Following model <cit.> and an Adaptive Cruise Control (ACC) method based on Model Predictive Control (MPC).

§.§.§ Gipps Car Following Model

The Gipps model is a well-known car-following approach that corresponds to the safe-distance class of models. We chose this model because it considers the risk of potential collision between vehicles and is simple to calibrate <cit.>. In this model, the actions to change lanes are governed by a fixed set of rules. The rules consider the risk of collision, the presence of vehicles in the other lane and the potential gain in the average speed of the decision. The main disadvantage of this model is that it does not consider vehicle perception  <cit.>. In addition, minor variations on the leader vehicle can heavily affect the followers' reaction.

Gipps model determines the speed of each vehicle at time $t$ by applying two constraints on the acceleration: the desired speed must not be surpassed, and the critical safe distance should be maintained. The acceleration and deceleration can be expressed by the following equations:

\begin{eqnarray}
v^{acc}_{n}(t+T) = v_{n}(t) + 2.5\cdot a_{n}\cdot T (1-\frac{v_{n}(t)}{v^{d}_{n}}) \cdot \sqrt{0.025 + \frac{v_{n}(t)}{v^{d}_{n}}}
\end{eqnarray}

\begin{eqnarray}
v^{decc}_{n}(t+T) = -T \cdot d_{n} + \sqrt{T^{2} \cdot d^{2}_n + d_{n} \{ 2x_{n-1} (t) - x_{n} (t) - (S_{n-1}) - T \cdot v_{n} (t) + \frac{v_{n-1}(t)^{2}}{d'_{n-1}}}
\end{eqnarray}

where $n$ and $n-1$ are the follower and leader, respectively, $T$ is the reaction time, $v_{n}(t)$ and $v_{n-1}(t)$ are the speeds of the follower and leader respectively at time t. $v^{d}_{n}$ and $a_{n}$ are the follower's maximum desired speed and maximum acceleration, respectively. $d_{n}$ and $d'_{n-1}$ are the most severe braking capability and $x_{n}(t)$ and $x_{n-1}(t)$ are the longitudinal position of the follower and leader at time $t$. $S_{n-1}$ is the distance between vehicles at a stop.

The implemented version of the Gipps model uses safe following distances set according to the RSS model introduced in <cit.>. The reaction time was set to $T = 0.7 s$.

§.§.§ Adaptive Cruise Control based on MPC

This subsection describes the ACC method based on MPC. The method was adapted from <cit.> for a multi-agent setting and discrete action space. The model considers a kinematic point-mass model that can be described by the following equation:

\begin{eqnarray}
x(t+1) = Ax(t) + Bu(t)
\end{eqnarray}

where $x(t)=(S_{n-1, n}(t), \Delta V_{n-1,n}(t), V_{n}(t))^T$, $u(t)=a(t)$ and

\begin{equation}
\begin{bmatrix}
1 & \Delta T & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
\end{equation}

\begin{equation}
\begin{bmatrix}
-0.5\Delta T^{2} \\
-\Delta T^{2} \\
\Delta
\end{bmatrix}
\end{equation}

The objective of an ACC is to follow the leader vehicle (LV) at the desired distance $\tilde{S}_{n-1,n}$, considering the vehicle speed and the headway $t_{hw}$:

\begin{eqnarray}
\tilde{S}_{n-1,n} = V_n t_{hw}
\end{eqnarray}

The mathematical formulation of the MPC model is presented below:

\beginmini!
a∑_t=0^N-1 [(S_n-1,n(t)-S̃_n-1,n(t)/S_max)^2+(ΔV_n-1,n/ΔV_max)+(jerk(t)/jerk_max)^2]
x(t+1)=Ax(t) + Bu(t)
S_n-1,n> 0
V_n> 0
-3 m/s^2≤a_n ≤3m/s^2
\endmini!

where N is the prediction horizon, $S_{max}$, $\Delta V_{max}$ and $jerk_{max}$ are constants. Constraint (14b) corresponds to the point-mass model equations. Constraint (14c) ensures that the relative distance between the leader and the following vehicle is always greater than 0. Constraints (14d) and (14e) state that the speed of the following can't be negative, and its acceleration should be within a predefined range, respectively. We adopt the same parameter values used in <cit.>, that is, $t_{hw}=1.2s$, a $S_{max}=15m$, $\Delta V_{max} = 8 m/s$. In addition, we set the jerk term to be equal to zero because the MARL formulation does not consider it as part of the objective.

§.§.§ Results comparison

Table <ref> presents the comparison results between the proposed MARL method, the Gipps lane-changing model and the MPC method. We used a fixed number of AVs for this set of experiments and trained the algorithm for 2000 episodes. The results shown in the table correspond to averages over 10 test episodes. The proposed method consistently overperforms both the Gipps and MPC methods. It produces policies with higher average speeds for the EMV while maintaining safer distances between all vehicles. The collision risk determined by the risk index is also considerably lower for the MARL method.

Comparison to the Car Following baselines.

Scenario     Method     Reward      EMV Speed     AV Speed     Col. rate      Col. risk     Safety Dist. (m)

Gipps     $-452.06 \pm 279.99$     $19.76 \pm 0.36$     $\mathbf{19.57 \pm 0.34}$      $15.30 \pm 10.23$       $0.306 \pm 0.047$     $6.51 \pm 0.92$

MPC     $401 \pm 17.05$     $14.29 \pm 0.45$     $14.09 \pm 0.22$     $1.00 \pm 3.00$     $\mathbf{0.16 \pm 0.41}$     $16.71 \pm 4.01$

MAPPO     $\mathbf{653 \pm 67.11}$     $\mathbf{22.67 \pm 1.35}$       $19.05 \pm 0.42$     $\mathbf{0.0} \pm \mathbf{0.0} $      $0.175 \pm 0.013$     $\mathbf{17.73 \pm 3.60}$

Gipps     $229.23 \pm 110.23$     $19.58 \pm 0.46$     $19.47 \pm 0.25$      $5.12 \pm 2.00$       $0.360 \pm 0.06$     $17.06 \pm 1.15$

MPC     $523.44 \pm 96.08$     $16.29 \pm 0.36$     $14.94 \pm 0.24$     $0.0 \pm 0.0$     $0.11 \pm 0.020$     $19.26 \pm 3.65$

MAPPO     $\mathbf{877.28 \pm 53.45}$     $\mathbf{29.98 \pm 0.95}$       $\mathbf{19.87 \pm 0.35}$     $\mathbf{0.0} \pm \mathbf{0.0} $      $\mathbf{0.03 \pm 0.0025}$     $\mathbf{37.01 \pm 7.01}$

In contrast to the Car Following Models, the proposed method allows for the efficient transit of the EMV in heavy traffic conditions. Thus, the EMV can achieve higher speeds while keeping safe distances from other vehicles. Figure <ref> shows the distribution of average speeds for the EMV and the AVs for the three methods. The speed of the AVs is slightly lower than the maximum allowable speed $(20 m/s)$, because AVs can effectively collaborate and slow down to facilitate the passage of the EMV. On the other hand, the Gipps model does not prioritise the passage of the EMV because it maximises the speeds of all vehicles. The MPC method achieves nearly no collisions with a low overall risk of collision; however, the average speeds of the vehicles are also considerably lower.

Comparison of the policies obtained with the three different methods. (i) Average speeds for the EMV and AVs and (ii) Safety distances

The proposed method guarantees the safe transit of the EMV by incorporating the risk metric in the reward. No collisions occurred during the testing episodes, and the average collision risk is substantially lower than the Gipps method for both scenarios. The proposed method achieves a low collision risk thanks to preemptive manoeuvring actions from the AVs when the EMV is within their perception range. An example of these types of actions is shown in figure <ref>.

The MPC method achieves a low number of collisions and the lowest overall risk of collision. However, it is conservative in terms of speed, which causes it to perform worse in terms of total reward when compared to the MARL method.

Example of preemptive lane-changing maneuvers by the AVs.

§.§ Reward shaping: Risk vs Efficiency

We performed different experiments by varying the two main components of the reward function: risk of collision and traffic flow efficiency. The risk component is determined by equation (7), meanwhile the efficiency is determined by the average speed. If two vehicles are within the range of collision risk as defined by equation (3), then an increase in speed has a negative impact on the risk index. Conversely, if the speed decreases, the risk index improves. Thus, there is a trade-off between the speed and the risk of collision.

Table <ref> presents the results for different combinations of risk and efficiency within the reward function. We used a fixed number of agents (30) for these experiments and trained the MAPPO algorithm for 2000 episodes. The results shown in table <ref> correspond to averages over 10 test episodes. As expected, the speeds of both the EMV and the AVs are very high when the risk component is reduced. However, the collision rate and the collision risk are considerably worse compared to risk-aware policies (when the risk coefficient is greater than 0). In addition, the average safety distance between vehicles is lower when risk is not present in the reward function.

Results of different reward functions.

Risk      Efficiency      EV Speed (m/s)     AV Speed (m/s)     Col. rate      Col. risk     Safety Dist. (m)

$0.0$     $1.0$     $27.83 \pm 1.16$     $19.77 \pm 0.24$      $3.90 \pm 1.40$       $0.189 \pm 0.013$     $16.95 \pm 3.82$

$0.5$     $1.0$     $18.86 \pm 0.81$       $17.86 \pm 0.16$     $4.01 \pm 3.21$      $0.176 \pm 0.021$     $17.03 \pm 2.87$

$1.0$     $1.0$     $7.22 \pm 0.27$     $7.22 \pm 0.11$     $0.8 \pm 2.4$      $0.114 \pm 0.031$     $16.69 \pm 3.76$

$1.0$     $0.5$     $7.14 \pm 0.18$    $7.12 \pm 0.16$     $0.0 \pm 0.0$     $0.105 \pm 0.015$      $16.23 \pm 4.61$

$1.0$     $0.0$     $7.13 \pm 0.33$    $7.09 \pm 0.055$     $0.0 \pm 0.0$     $0.094 \pm 0.019$      $16.36 \pm 3.64$

Figure <ref> shows the effect of increasing the risk component in the reward function over the EMV average speed and the risk of collision. The speed of the EMV is reduced drastically as the risk component approaches the value of $1.0$. Moreover, when the risk component reaches the value $1.0$, the average speed is equal to the minimum velocity $(7 m/s)$. This is because the speed has a direct negative impact over the safe lateral and longitudinal safe distances, according to equations (1), (2), (4) and (5).

Influence of the risk component over the EMV speed and the risk of collision. The risk component is determined by equation (7).

§.§ Scalability

To test the scalability of the proposed method, we explored the effect of using a different number of agents over the training process and the performance of the trained policies. Additional agents do not alter the state dimension because agents have partial observability of the environment. Thus, a larger number of agents does not translate into higher neural network complexity. However, a larger number of agents produces higher environment complexity and, therefore, longer training time.

Figure <ref> shows the effect of different numbers of agents over the learning curves. The reward corresponds to the sum of rewards for all agents. For each case, four seeds were used, and the Figure shows the average and interval. Due to the higher number of vehicles on the road, the total average reward converges to considerably lower values when increasing the number of agents.

Learning curves for different number of agents.

Table <ref> shows the individual results for the trained multi-agent policies with a different number of agents. Results correspond to the average over 10 test episodes. All vehicles' initial positions and speeds were randomised for every test episode. When increasing the number of agents, AVs and the EMV must move at lower speeds, which causes the individual rewards to be lower. Moreover, vehicles will be closer to each other, which reduces the safety distance and thereby increases the risk of collision. The last column of the table contains the training time for the 2000 episodes, expressed as Frames Per Second (FPS). The total training time can be obtained by dividing the total number of steps by the FPS. For example, for the case with 20 agents, the total training time is $t = (2000 \cdot 400) /(31\cdot 3600) = 7.17 hrs$.

Results for different number of agents.

N° agents      Reward      EMV Speed     AV Speed     Average Dist. (m)      Col. risk     FPS

$20$     $563.29 \pm 30.00$     $26.89 \pm 0.49$     $18.68 \pm 0.15$      $17.03 \pm 3.87$       $0.18 \pm 0.02$     $31$

$30$     $171.92\pm 36.64$     $26.15 \pm 0.48$       $18.60 \pm 0.16$     $13.97 \pm 3.64$      $0.19 \pm 0.052$     $21$

$40$     $123.45 \pm 19.18$     $18.30 \pm 1.08$     $15.17 \pm 0.18$     $8.67 \pm 5.54$      $0.21 \pm 0.018$     $14$

$50$     $94.82 \pm 42.38$     $16.27 \pm 2.37$     $11.52 \pm 0.92$     $8.70 \pm 4.38$     $0.45 \pm 0.069$      $10$

$60$     $61.01 \pm 25.050$     $11.37 \pm 0.99$      $10.11 \pm 0.58$     $7.2 \pm 3.41$      $0.50 \pm 0.028$     $8$

§.§ Competitive scenario

Until now, all experiments were carried out assuming a fully cooperative game, where all agents share the same networks (actor and critics), episode buffer and the reward function. We explored the performance of the proposed method in a competitive scenario, i.e., where agents seek to maximise their own individual rewards. For this, we created a separate actor and critic network for each agent and allowed them to maximise their own speed and penalise their collisions. Having a separate network for each agent makes the training substantially less efficient, increasing the training time nearly threefold. For this set of experiments, we used the two-lane road scenario and trained the MAPPO algorithm for 2000 episodes.

In both the cooperative and competitive scenarios, the MAPPO algorithm can guarantee no collisions and high safety distances between all vehicles. In addition, the resulting policies allow the EMV to go at a faster average speed. However, in the competitive scenario, vehicles do not coordinate to let the EMV pass efficiently. Figure <ref> contrasts the difference between cooperative and competitive multi-agent policies. In the fully cooperative case, vehicles move aside to minimise the braking and the number of lane changes of the EMV. In contrast, in the competitive scenario, the AVs do not change lanes preemptively, and they just maximise their own velocity; therefore, the EMV has to constantly overtake other vehicles. This causes the EMV to achieve a considerably lower average speed, which can be seen in figure <ref> (i). On the other hand, figure <ref> (ii) shows the difference in average risk of collision for the EMV. Due to the constant overtaking manoeuvres done by the EMV, the risk of collision of the substantially greater in the competitive case. The MAPPO algorithm can still generate collision-free policies in this case but the overall traffic efficiency is substantially lower.

Cooperative vs. competitive multi-agent policies. In the fully cooperative scenario, the AVs move aside to let the EMV pass, while in the competitive scenario the EMV has to constantly perform overtaking manoeuvres.

Comparison between the cooperative and competitive scenarios (i) Average speeds (ii) EMV Risk of collision

§ CONCLUSION

AVs have the potential to improve road safety and efficiency, although there are still many challenges regarding their practical application in traffic systems. This study proposed a multi-agent approach for the safe and efficient manoeuvring of autonomous traffic in the presence of EMVs. We used the MAPPO algorithm combined with risk metrics to determine manoeuvring actions for all cars in congested road settings. We explored the capabilities of the proposed approach in four case studies. First, we compared it with a traditional rule-based Car Following Model. Second, we contrasted the two main components of the reward function; risk and traffic efficiency. Third, we investigated the scalability and performance with different numbers of agents. Finally, we presented a competitive scenario where vehicles seek to maximise their own independent reward.

Results of the four case studies show the potential and capabilities of the proposed method. It consistently outperforms the car following baselines, obtaining higher average speeds for the EMV while maintaining a lower overall risk of collision. Furthermore, we show how the reward function can be tuned to achieve higher safety or traffic efficiency. However, a minimum amount of risk component is needed to guarantee safe transit and no collisions. A higher number of agents translates into lower average speeds of the EMV and AVs; however, the algorithm can still guarantee safe manoeuvring. The proposed method is also applicable to competitive scenarios, where vehicles act selfishly, maximising their own speed. The resulting policies involve constant manoeuvring of the EMV, which increases its risk of collision when compared to a fully cooperative case.

There is some future work that can explore new settings and can overcome some of the limitations of this study. We considered just a single road environment, with vehicles going in one direction. Other scenarios can be evaluated, such as a highway merging, two-directional lanes, intersections and the possibility of the EMV moving in the contraflow. In addition, explicit communication between agents to improve overall traffic safety should be explored in the future. However, having a one-to-one communication network between CAVs is not scalable; thus, innovative, efficient methods are needed for cases with multiple agents.

§ ACKNOWLEDGEMENT(S)

This research was partially supported by the President's Scholarship Programme funded by Imperial College London, and the Chilean National Agency for Research and Development (ANID) through the "BECAS DOCTORADO EN EXTRANJERO" program, Grant No. 72210279.

, Huang

Ahmed, HU.
, Huang, Y.
Lu, P.

A Review of Car-Following Models and Modeling
Tools for Human and Autonomous-Ready Driving Behaviors in
Micro-Simulation A Review of Car-Following Models and
Modeling Tools for Human and Autonomous-Ready Driving Behaviors
in Micro-Simulation.
Smart Cities41314–335.

Bokare, PS.
Maurya, AK.

Acceleration-deceleration behaviour of various vehicle
types Acceleration-deceleration behaviour of various vehicle types.
Transportation research procedia254733–4749.

, Da Lio

Bosetti, P.
, Da Lio, M.
Saroldi, A.

On the human control of vehicles: an experimental study
of acceleration On the human control of vehicles: an experimental study of
European Transport Research Review62157–170.

, Schwarting
, Karaman

Buckman, N.
, Schwarting, W.
, Karaman, S.
Rus, D.

Semi-Cooperative Control for Autonomous
Emergency Vehicles Semi-Cooperative Control for Autonomous
Emergency Vehicles.
2021 IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS) 2021 IEEE/RSJ
International Conference on Intelligent Robots and Systems
(IROS) ( 7052–7059).
Prague, Czech RepublicIEEE.

Chen, D.
, Li, Z.
, Hajidavalloo, M.
, Chen, K.
, Wang, Y.
, Jiang, L.
Wang, Y.

Deep Multi-agent Reinforcement Learning for
Highway On-Ramp Merging in Mixed Traffic Deep Multi-agent
Reinforcement Learning for Highway On-Ramp Merging in Mixed
arXiv:2105.05701 [cs, eess].

arXiv: 2105.05701

Chowdhury, A.

Priority based and secured traffic management system for
emergency vehicle using IoT Priority based and secured traffic management
system for emergency vehicle using IoT.
2016 International Conference on Engineering &
MIS (ICEMIS) 2016 International Conference on Engineering &
MIS (ICEMIS) ( 1–6).
Agadir, MoroccoIEEE.

, Revilloud
, Gruyer

Claussmann, L.
, Revilloud, M.
, Gruyer, D.
Glaser, S.

A Review of Motion Planning for Highway
Autonomous Driving A Review of Motion Planning for Highway
Autonomous Driving.
IEEE Transactions on Intelligent Transportation
Conference Name: IEEE Transactions on Intelligent Transportation

Dai, P.
, Liu, K.
, Zhuge, Q.
, Sha, EHM.
, Lee, VCS.
Son, SH.

Quality-of-Experience-Oriented Autonomous
Intersection Control in Vehicular Networks
Quality-of-Experience-Oriented Autonomous Intersection Control in
Vehicular Networks.
IEEE Transactions on Intelligent Transportation

, Chen
, Ha
, Li

Dong, J.
, Chen, S.
, Ha, PYJ.
, Li, Y.
Labi, S.

A DRL-based Multiagent Cooperative Control
Framework for CAV Networks: a Graphic Convolution Q Network A
DRL-based Multiagent Cooperative Control Framework for CAV
Networks: a Graphic Convolution Q Network.
arXiv:2010.05437 [cs, eess].

arXiv: 2010.05437

Dresner, K.
Stone, P.

Human-Usable and Emergency Vehicle–Aware
Control Policies for Autonomous Intersection Management
Human-Usable and Emergency Vehicle–Aware Control Policies for
Autonomous Intersection Management.

, Ahamed
, Barua

Farazi, NP.
, Ahamed, T.
, Barua, L.
Zou, B.

Deep Reinforcement Learning and Transportation
Research: A Comprehensive Review Deep Reinforcement Learning
and Transportation Research: A Comprehensive Review.
Transportation Research Interdisciplinary

Ambulance accidents - a Freedom of Information request to
London Ambulance Service NHS Trust. Ambulance accidents - a
Freedom of Information request to London Ambulance Service NHS

Garcia, J.
Fernández, F.

Safe exploration of state and action spaces in
reinforcement learning Safe exploration of state and action spaces in
reinforcement learning.
Journal of Artificial Intelligence

Garcıa, J.
Fernández, F.

A comprehensive survey on safe reinforcement learning
A comprehensive survey on safe reinforcement learning.
Journal of Machine Learning

Gayle, D.

Number of road deaths involving police vehicles at
13-year high Number of road deaths involving police vehicles at 13-year
The Guardian.

Geibel, P.
Wysotzki, F.

Risk-sensitive reinforcement learning applied to control
under constraints Risk-sensitive reinforcement learning applied to control
under constraints.
Journal of Artificial Intelligence

Gipps, PG.

A model for the structure of lane-changing decisions A
model for the structure of lane-changing decisions.
Transportation Research Part B:

, Pulido
, Alberola
, Julian

González, CL.
, Pulido, JJ.
, Alberola, JM.
, Julian, V.
Niño, LF.

Autonomous Distributed Intersection Management for
Emergency Vehicles at Intersections Autonomous Distributed
Intersection Management for Emergency Vehicles at
F. De La Prieta, A. El Bolock, D. Durães, J. Carneiro, F. Lopes  V. Julian (), Highlights in Practical
Applications of Agents, Multi-Agent Systems, and Social Good.
The PAAMS Collection Highlights in Practical Applications of
Agents, Multi-Agent Systems, and Social Good. The PAAMS
Collection ( 261–269).
ChamSpringer International Publishing.

Haydari, A.
Yılmaz, Y.

Deep Reinforcement Learning for Intelligent
Transportation Systems: A Survey Deep Reinforcement Learning
for Intelligent Transportation Systems: A Survey.
IEEE Transactions on Intelligent Transportation
Conference Name: IEEE Transactions on Intelligent Transportation

Humagain, S.
Sinha, R.

Routing Autonomous Emergency Vehicles in Smart
Cities Using Real Time Systems Analogy: A Conceptual Model
Routing Autonomous Emergency Vehicles in Smart Cities Using
Real Time Systems Analogy: A Conceptual Model.
2019 IEEE 17th International Conference on
Industrial Informatics (INDIN) 2019 IEEE 17th International
Conference on Industrial Informatics (INDIN) ( 1097–1102).
Helsinki, FinlandIEEE.

, Almufareh

Humayun, M.
, Almufareh, MF.
Jhanjhi, NZ.

Autonomous Traffic System for Emergency
Vehicles Autonomous Traffic System for Emergency

Number: 4 Publisher: Multidisciplinary Digital Publishing

Ilgin Guler
, Menendez

Ilgin Guler

Ilgin Guler, S.
, Menendez, M.
Meier, L.

Using connected vehicle technology to improve the
efficiency of intersections Using connected vehicle technology to improve
the efficiency of intersections.
Transportation Research Part C: Emerging

Isidore, C.
Valdes-Dapena, P.

Tesla is under investigation because its cars keep hitting
emergency vehicles. Tesla is under investigation because its cars keep
hitting emergency vehicles.

, Imura
, Hayakawa
, Ohata

Kamal, MAS.
, Imura, Ji.
, Hayakawa, T.
, Ohata, A.
Aihara, K.

A Vehicle-Intersection Coordination Scheme for
Smooth Flows of Traffic Without Using Traffic Lights A
Vehicle-Intersection Coordination Scheme for Smooth Flows of
Traffic Without Using Traffic Lights.
IEEE Transactions on Intelligent Transportation

, Chowdhury
, Kamruzzaman

Karmakar, G.
, Chowdhury, A.
, Kamruzzaman, J.
Gondal, I.

A Smart Priority-Based Traffic Control
System for Emergency Vehicles A Smart Priority-Based Traffic
Control System for Emergency Vehicles.
IEEE Sensors Journal211415849–15858.
Conference Name: IEEE Sensors Journal

Kenney, JB.

Dedicated Short-Range Communications (DSRC)
Standards in the United States Dedicated Short-Range
Communications (DSRC) Standards in the United States.
Proceedings of the IEEE9971162–1182.
Conference Name: Proceedings of the IEEE

, Rhoulami

Lazar, H.
, Rhoulami, K.
Rahmani, MD.

A review analysis of optimal velocity models A review
analysis of optimal velocity models.

, Yemdji-Tchassi

Lecerf, U.
, Yemdji-Tchassi, C.
Michiardi, P.

Safer Autonomous Driving in a Stochastic,
Partially-Observable Environment by Hierarchical Contingency
Planning Safer Autonomous Driving in a Stochastic,
Partially-Observable Environment by Hierarchical Contingency
Planning   arXiv:2204.06509.

arXiv:2204.06509 [cs] type: article

Li, N.
, Oyler, D.
, Zhang, M.
, Yildiz, Y.
, Kolmanovsky, I.
Girard, A.

Game-Theoretic Modeling of Driver and Vehicle
Interactions for Verification and Validation of Autonomous Vehicle
Control Systems Game-Theoretic Modeling of Driver and Vehicle
Interactions for Verification and Validation of Autonomous Vehicle
Control Systems.
arXiv:1608.08589 [cs].

arXiv: 1608.08589

Lowe, R.
, Wu, Y.
, Tamar, A.
, Harb, J.
, Abbeel, P.
Mordatch, I.

Multi-Agent Actor-Critic for Mixed
Cooperative-Competitive Environments Multi-Agent Actor-Critic
for Mixed Cooperative-Competitive Environments.
arXiv:1706.02275 [cs].

Lu, Q.
Kim, KD.

A Genetic Algorithm Approach for Expedited
Crossing of Emergency Vehicles in Connected and Autonomous
Intersection Traffic A Genetic Algorithm Approach for Expedited
Crossing of Emergency Vehicles in Connected and Autonomous
Intersection Traffic.
Journal of Advanced Transportation20171–14.

Marshall, A.

An Autonomous Car Blocked a Fire Truck
Responding to an Emergency An Autonomous Car Blocked a Fire
Truck Responding to an Emergency.

Section: tags

Miculescu, D.
Karaman, S.

Polling-systems-based Autonomous Vehicle
Coordination in Traffic Intersections with No Traffic Signals
Polling-systems-based Autonomous Vehicle Coordination in Traffic
Intersections with No Traffic Signals.
arXiv:1607.07896 [cs, math].

arXiv: 1607.07896

, Tseng

Nageshrao, S.
, Tseng, HE.
Filev, D.

Autonomous Highway Driving using Deep
Reinforcement Learning Autonomous Highway Driving using Deep
Reinforcement Learning.
2019 IEEE International Conference on Systems,
Man and Cybernetics (SMC) 2019 IEEE International Conference on
Systems, Man and Cybernetics (SMC) ( 2326–2331).
ISSN: 2577-1655

Nellore, K.
Hancke, G.

Traffic Management for Emergency Vehicle
Priority Based on Visual Sensing Traffic Management for
Emergency Vehicle Priority Based on Visual Sensing.

Ngai, DCK.
Yung, NHC.

A Multiple-Goal Reinforcement Learning Method
for Complex Vehicle Overtaking Maneuvers A Multiple-Goal
Reinforcement Learning Method for Complex Vehicle Overtaking
IEEE Transactions on Intelligent Transportation
Conference Name: IEEE Transactions on Intelligent Transportation

Emergency Vehicles – Injury Facts. Emergency vehicles –
injury facts.

, Pek
, Dal Col
, Norén

Nyberg, T.
, Pek, C.
, Dal Col, L.
, Norén, C.
Tumova, J.

Risk-aware motion planning for autonomous vehicles with
safety specifications Risk-aware motion planning for autonomous vehicles
with safety specifications.
2021 IEEE Intelligent Vehicles Symposium (IV) 2021
ieee intelligent vehicles symposium (iv) ( 1016–1023).

Palanisamy, P.

Multi-Agent Connected Autonomous Driving using
Deep Reinforcement Learning Multi-Agent Connected Autonomous
Driving using Deep Reinforcement Learning.

, Di Franco
, Marinoni

Pannocchi, L.
, Di Franco, C.
, Marinoni, M.
Buttazzo, G.

Integrated Framework for Fast Prototyping and
Testing of Autonomous Systems Integrated Framework for Fast
Prototyping and Testing of Autonomous Systems.
Journal of Intelligent & Robotic

Rashid, T.
, Samvelyan, M.
, de Witt, CS.
, Farquhar, G.
, Foerster, J.
Whiteson, S.

QMIX: Monotonic Value Function Factorisation
for Deep Multi-Agent Reinforcement Learning QMIX: Monotonic
Value Function Factorisation for Deep Multi-Agent Reinforcement
arXiv:1803.11485 [cs, stat].

J3016C: Taxonomy and Definitions for Terms Related to
Driving Automation Systems for On-Road Motor Vehicles - SAE
International. J3016C: Taxonomy and Definitions for Terms
Related to Driving Automation Systems for On-Road Motor
Vehicles - SAE International.

, Shammah

Shalev-Shwartz, S.
, Shammah, S.
Shashua, A.

Safe, Multi-Agent, Reinforcement Learning for
Autonomous Driving Safe, Multi-Agent, Reinforcement Learning
for Autonomous Driving.
arXiv:1610.03295 [cs, stat].

arXiv: 1610.03295

, Shammah

Shalev-Shwartz, S.
, Shammah, S.
Shashua, A.

On a formal model of safe and scalable self-driving
cars On a formal model of safe and scalable self-driving cars.
arXiv preprint arXiv:1708.06374.

, Malhotra

Shelke, M.
, Malhotra, A.
Mahalle, PN.

Fuzzy priority based intelligent traffic congestion
control and emergency vehicle management using congestion-aware routing
algorithm Fuzzy priority based intelligent traffic congestion control and
emergency vehicle management using congestion-aware routing
Journal of Ambient Intelligence and Humanized

, Kang
, Park
, Park

So, JJ.
, Kang, J.
, Park, S.
, Park, I.
Lee, J.

Automated Emergency Vehicle Control Strategy
Based on Automated Driving Controls Automated Emergency Vehicle
Control Strategy Based on Automated Driving Controls.
Journal of Advanced Transportation20201–11.

, Zhong
, Dey

Su, H.
, Zhong, YD.
, Dey, B.
Chakraborty, A.

EMVLight: A Decentralized Reinforcement
Learning Framework for Efficient Passage of Emergency Vehicles
EMVLight: A Decentralized Reinforcement Learning Framework for
Efficient Passage of Emergency Vehicles.
arXiv:2109.05429 [cs, eess].

arXiv: 2109.05429

Viriyasitavat, W.
Tonguz, OK.

Priority Management of Emergency Vehicles at
Intersections Using Self-Organized Traffic Control Priority
Management of Emergency Vehicles at Intersections Using
Self-Organized Traffic Control.
2012 IEEE Vehicular Technology Conference (VTC
Fall) 2012 IEEE Vehicular Technology Conference (VTC Fall)  ( 1–4).
Quebec City, QC, CanadaIEEE.

M. Wang
, Wang
, Talbot
, Gerdes

M. Wang

Wang, M.
, Wang, Z.
, Talbot, J.
, Gerdes, JC.
Schwager, M.

Game-Theoretic Planning for Self-Driving Cars
in Multivehicle Competitive Scenarios Game-Theoretic Planning for
Self-Driving Cars in Multivehicle Competitive Scenarios.
IEEE Transactions on Robotics3741313–1325.
Conference Name: IEEE Transactions on Robotics

P. Wang
, Chan
de La Fortelle

P. Wang

Wang, P.
, Chan, CY.
de La Fortelle, A.

A Reinforcement Learning Based Approach for
Automated Lane Change Maneuvers A Reinforcement Learning
Based Approach for Automated Lane Change Maneuvers.
2018 IEEE Intelligent Vehicles Symposium (IV)
2018 IEEE Intelligent Vehicles Symposium (IV) (  1379–1384).

, Duan
, Li
, Xu

Wen, L.
, Duan, J.
, Li, SE.
, Xu, S.
Peng, H.

Safe reinforcement learning for autonomous vehicles
through parallel constrained policy optimization Safe reinforcement
learning for autonomous vehicles through parallel constrained policy
2020 IEEE 23rd International Conference on Intelligent
Transportation Systems (ITSC) 2020 ieee 23rd international conference on
intelligent transportation systems (itsc) ( 1–7).

Yu, C.
, Velu, A.
, Vinitsky, E.
, Wang, Y.
, Bayen, A.
Wu, Y.

The Surprising Effectiveness of PPO in
Cooperative, Multi-Agent Games The Surprising Effectiveness of
PPO in Cooperative, Multi-Agent Games.
arXiv:2103.01955 [cs].

L. Zhang
L. Zhang

Zhang, L.
, Zhang, R.
, Wu, T.
, Weng, R.
, Han, M.
Zhao, Y.

Safe reinforcement learning with stability guarantee for
motion planning of autonomous vehicles Safe reinforcement learning with
stability guarantee for motion planning of autonomous vehicles.
IEEE Transactions on Neural Networks and Learning

YJ. Zhang
, Malikopoulos

YJ. Zhang

Zhang, YJ.
, Malikopoulos, AA.
Cassandras, CG.

Optimal control and coordination of connected and
automated vehicles at urban traffic intersections Optimal control and
coordination of connected and automated vehicles at urban traffic
2016 American Control Conference (ACC) 2016
American Control Conference (ACC) ( 6227–6232).
Boston, MA, USAIEEE.

M. Zhou
M. Zhou

Zhou, M.
, Luo, J.
, Villella, J.
, Yang, Y.
, Rusu, D.
, Miao, J.
Wang, J.

SMARTS: Scalable Multi-Agent Reinforcement
Learning Training School for Autonomous Driving SMARTS:
Scalable Multi-Agent Reinforcement Learning Training School for
Autonomous Driving.
arXiv:2010.09776 [cs, eess].

W. Zhou
W. Zhou

Zhou, W.
, Chen, D.
, Yan, J.
, Li, Z.
, Yin, H.
Ge, W.

Multi-agent Reinforcement Learning for Cooperative
Lane Changing of Connected and Autonomous Vehicles in Mixed
Traffic Multi-agent Reinforcement Learning for Cooperative Lane
Changing of Connected and Autonomous Vehicles in Mixed
Autonomous Intelligent Systems215.

arXiv: 2111.06318

F. Zhu

F. Zhu

Zhu, F.
Ukkusuri, SV.

A linear programming formulation for autonomous
intersection control within a dynamic traffic assignment and connected
vehicle environment A linear programming formulation for autonomous
intersection control within a dynamic traffic assignment and connected
vehicle environment.
Transportation Research Part C: Emerging

M. Zhu
M. Zhu

Zhu, M.
, Wang, Y.
, Pu, Z.
, Hu, J.
, Wang, X.
Ke, R.

Safe, efficient, and comfortable velocity control based
on reinforcement learning for autonomous driving Safe, efficient, and
comfortable velocity control based on reinforcement learning for autonomous
Transportation Research Part C: Emerging

§ APPENDICES

In our formulation, the policy maps the agents' actions to a categorical distribution, which represents the different possible actions within the environment. During training, the actor network seeks to maximise the following equation <cit.>:

\begin{equation}
L(\theta) = [\frac{1}{Bn} \sum_{i=1}^{B} \sum_{k=1}^{n}  min(r_{\theta, i}^{(k)}A_{i}^{(k)}, clip(r_{\theta, i}^{(k)}, 1-\epsilon, 1+ \epsilon)A_{i}^{(k)}] + \sigma \frac{1}{Bn} \sum_{i=1}^{B} \sum_{k=1}^{n} S [\pi_{\theta}(\Omega_{i}^{(k)})],
\end{equation}

\begin{equation}
r_{\theta, i}^{(k)} = \frac{\pi_{\theta}(a_{i}^{(k)}| o_{i}^{(k)})}{\pi_{\theta_{old}}(a_{i}^{(k)}| o_{i}^{(k)})},
\end{equation}

where $A_{i}^{(k)}$is calculated using Generalized Advantage Estimation (GAE) with advantage normalization, $S$ is the policy entropy and $\sigma$ is the entropy coefficient. The critic minimises the following equation:

\begin{equation}
L(\phi) = \frac{1}{Bn} \sum_{i=1}^{B} \sum_{k=1}^{n}(max[V_{\phi}(s_{i}^{(k)}) - \hat{R_{i}})^{2}, (clip(V_{\phi}(s_{i}^{(k)}), V_{\phi_{old}}(s_{i}^{(k)})-\epsilon, V_{\phi_{old}}(s_{i}^{(k)})+\epsilon)- \hat{R_{i}})^{2}],
\end{equation}

where $\hat{R_{i}}$ is the discounted error to go, $B$ is the batch size and $n$ the number of agents.