# Building Safer Autonomous Agents by Leveraging Risky Driving Behavior
Knowledge

Ashish Rana  Thapar Institute of Eng. & Tech., India
<EMAIL_ADDRESS>Avleen Malhi  Bournemouth University, UK
<EMAIL_ADDRESS>
###### Abstract

Simulation environments are good for learning different driving tasks like
lane changing, parking or handling intersections etc. in an abstract manner.
However, these simulation environments often restrict themselves to operate
under conservative interaction behavior amongst different vehicles. But, as we
know, real driving tasks often involve very high risk scenarios where other
drivers often don’t behave in the expected sense. There can be many reasons
for this behavior like being tired or inexperienced. The simulation
environment doesn’t take this information into account while training the
navigation agent. Therefore, in this study we especially focus on
systematically creating these risk prone scenarios with heavy traffic and
unexpected random behavior for creating better model-free learning agents. We
generate multiple autonomous driving scenarios by creating new custom Markov
Decision Process (MDP) environment iterations in the highway-env simulation
package. The behavior policy is learnt by agents trained with the help from
deep reinforcement learning models. Our behavior policy is deliberated to
handle collisions and risky randomized driver behavior. We train model free
learning agents with supplement information of risk prone driving scenarios
and compare their performance with baseline agents. Finally, we casually
measure the impact of adding these perturbations in the training process to
precisely account for the performance improvement obtained from utilizing the
learnings from these scenarios.

###### Index Terms:

Autonomous Agents, Driving Simulations, Trajectory Prediction, Causality

##  I Introduction

The arrival of autonomous driving agents have had a great impact on the
automobile industry. And it will be responsible for shaping the future of this
industry as well. The current industry trend progression demonstrates that the
connected fleet of autonomous agents will be dominating our driving
infrastructure [1]. But, that vision is still quite far considering the safety
infrastructure, security and public policy reasons [2, 3, 4, 5]. Therefore,
our immediate focus should be on making our driving agents safe and
efficacious. Creating safer agents has been explored well for the past few
years [6, 7, 8, 9, 10, 11, 12]. As it is crucial to know the expected agent
behavior in different environments especially safety critical ones. In natural
driving environments risk prone scenarios don’t happen frequently which makes
the learning process harder from these scenarios [13, 14]. And it is also
unethical to create these real risky driving scenarios for experimentation
purposes. Therefore, generating and studying these scenarios systematically is
a daunting task.

Perception systems with adversarial approaches of noisy labelling do
demonstrate promising path [15, 16, 17]. But the underlying fundamental
problem remains focused around safe interactions with other vehicles which
themselves are operating independently. Simulation environments with
appropriate system dynamics design assumptions do overcome this expressed
safety issue. Along with that these environments allow us to systematically
study the risk prone scenarios with near realistic driving behaviors. For this
study we have used the highway-env simulation package which allows us to
simulate different driving tasks [18]. It also provides simple interfacing
capabilities to modify these environments and quickly create baseline
prototypes [19] on it. We formulate the dynamics of the simulation system as
Markov Decision Process (MDP) in our experimentation [20]. We model our value
function approximation agents with deep reinforcement learning for these
defined MDP systems [21, 22] for our study.

Figure 1: Treatment and Control agents evaluated randomly for safe and
perturbed test environments.

For our experiment design we create two distinct variants of each model
architecture for above stated simulation dynamics. One of these model variants
is trained with our increased dangerous driving behavior interventions for all
the driving related tasks present in highway-env package. Second model variant
is the control variable in our experiment which is used to define the reward
baseline for the trained agents on regular simulations. In our study we do a
methodological doping of these environments with randomized dangerous driving
scenarios to create more risk prone environments for driving. This is done
specifically in two ways in our study, first by increasing the traffic at
strategically important locations challenging the agent to make dangerous
overtakes. Second, we increase the randomization factor and the clogged lanes
make the environment more collision prone for our agent (ego-vehicle). This is
done to create more robust agents in a given environment for that particular
dynamic scenario. Which are essentially better at post impact trajectory
predictions of other interacting vehicles. Figure 1 explains our causal
analysis experimentation setup in sequential form.

We also attempt to understand our experimentation setup from a causal
standpoint. We hold complete control of the data generation process in our
experimentation. Which allows us to have a unique vantage point of conducting
an experiment study which is equivalent to a randomized control trial (RCT).
We train our agents with a good enough assumption of absence in unobservable
confounding variables. As we have strictly defined the state dynamics and
vehicle behavior governing models. Our customization capabilities in the
highway-env package allows us to keep every condition the same while training
except our collision scenario perturbations. Meaning that treatment and
control groups are same in all aspects except our treatment i.e. there is a
comparability and co-variate balance in our experimental setup [23, 24, 25,
26]. With this special relation establishment we can imply that association
found in our experiment setup is causation. As shown in Figure 2 our treatment
(T) is subjecting the agent learning process to risk prone interacting vehicle
dynamics in a given environment. After that our sample test experiment
population involves evaluating the two model variants against regular and
perturbed environments with excessive risk prone scenario doping. And finally
by using expectation equations derived from the above causal graph we estimate
the causal effect our novel learning changes for enhanced safety.

Figure 2: Observational (Left) and Interventional (Right) distributions
represented with causal graphs.

Our contributions by the means of this paper include providing benchmarking
environment simulations for collision robustness predictions. Along with that
we provide experimentation methodology that creates more robust agents that
provide better on-road safety. And we also causally measure the impact of our
risky driving behavior doping interventions for different driving
environments. For the remaining paper we first discuss related work
corresponding to utilizing risk prone behavior for creating safer autonomous
vehicles. Second, we formally define our problem and elaborate on causal
aspects of it. Third, we explain the experiment setup for creating these
robust agents and elaborate upon our results. Finally, we provide the
conclusion to our autonomous driving agent study.

##  II Previous Work

Deep Reinforcement Learning has been used extensively for traffic control
tasks [27, 28]. The simulated environment provided by CARLA [29] gives a
framework for systems that estimates several affordances from sensors in a
simulated environment [30]. Navigation tasks like merging traffic requiring
good exploration capabilities have also shown promising results in simulators
[31]. ChauffeurNet [32] elaborates on the idea of imitation learning for
training robust autonomous agents that leverages worst case scenarios in the
form of realistic perturbations. A clustering based collision case generation
study [33] systematically defines and generates the different types of
collisions for effectively identifying valuable cases for agent training.

The highway-env package specifically focuses on designing safe operational
policies for large-scale non-linear stochastic autonomous driving systems
[20]. This environment has been extensively studied and used for modelling
different variants of MDP, for example: finite MDP, constraint-MDP and
budgeted-MDP (BMDP) [34]. BMDP is a variant of MDP which makes sure that risk
notion implemented as cost signal stays below a certain adjustable threshold
[35]. The problem formalization for workings of vehicle kinematics, temporal
abstraction, partial observability and reward hypothesis has been studied
extensively as well [34]. Robust optimization planning has been studied in the
past for finite MDP systems with uncertain parameters [36, 37, 38] and it also
has shown promising results under conservative driving behavior. For BMDP,
efficacy and safety analysis has been extended into continuous kinematics
states and unknown human behavior from the existing known dynamics and finite
state space [35]. Model free learning networks that approximate value function
for these MDPs like Deep Q-Learning (DQN) and Dueling Deep Q-Learning (DQN)
networks have demonstrated promising results in continuous agent learning [39,
40].

Worst case scenario knowledge in traffic analysis has been leveraged for model
based algorithms by building regions of high confidence containing true
dynamics with high probability. Tree based planning algorithms were used to
achieve robust stabilisation and mini-max control with generic costs. These
studies also leveraged non-asymptotic linear regression and interval
prediction [34, 20] for safer trajectory predictions. Behavior guided action
studies which use proximity graphs and safety trajectory computes for working
with aggressive and conservative have shown promising results [41] as well.
This study used CMetric measure [42] for generating varying levels of
aggressiveness in traffic. Whereas in our case we have used more randomization
and traffic clogging at key areas for risk prone scenarios to measure more
granular observation results.

Causal modelling techniques have contributed a lot in terms of providing great
interpretative explanations in many domains [43, 44, 45, 46]. Also, Fischer’s
randomized control trials (RCTs) have served as the gold standard for causal
discovery from observational data [24]. Sewall’s path diagrams were the first
attempt of generating causal answers with mathematics [47]. Now, causal
diagrams and different adjustments on these diagrams do offer direct causal
relation information about any experimental variables under study [23, 24, 48,
49]. In our experimental study we use these existing mathematical tools to
draw direct causal conclusions of our learnings from our environment
interventions.

##  III Problem Formulation

Our goal is to design and build agents on collision prone MDPs for navigation
tasks across the different traffic scenarios. The MDP comprises a behavior
policy $\pi$(a $\mid$ s) that outputs action a for a given state s. With this
learnt policy our goal is to predict discrete safe and efficient action from
finite action set (left, right, break, accelerate, idle) for next time step
for given driving scenarios.

The simulation platform that we used is compatible with OpenAI gym package
[50]. The highway-env package provides the traffic flow which is governed by
Intelligent Driver Model (IDM) [51] for linear acceleration & MOBIL model [52]
for lane changing. MOBIL model primarily consists of safety criterion and
incentive criterion. First safety criterion checks whether after lane change
the given vehicle is having enough acceleration space and second criterion
determines the total advantage of lane change in terms of total acceleration
gain.

Given MDP $\pazocal{M}$ is defined as a set of ($\pazocal{S}$, $\pazocal{A}$,
$\pazocal{T}$, r) where action a $\in$ $\pazocal{A}$, state s $\in$
$\pazocal{S}$, reward function r $\in$ [0,1] $\pazocal{S}$ X $\pazocal{A}$ and
state transition probabilities T(s’ $\mid$ s, a) $\in$
$\pazocal{M}$($\pazocal{S}$) $\pazocal{S}$ X $\pazocal{A}$ . With Deep-RL
algorithms we search the the behavior policy $\pi$(s$\mid$a) that helps us in
navigating across the traffic environments to gather maximum discounted reward
$\pazocal{R}$. The state-action value function $\pazocal{Q}$$\pi$(s,a)  for
given (s,a) assists in estimating future rewards of given behavior policy
$\pi$. Therefore, the optimal state-action value function
$\pazocal{Q}$$*$(s,a) provides maximum value estimates for all s $\in$
$\pazocal{S}$ and is evaluated by solving Bellman Equation [53], stated below
for reference. From this the optimal policy $\pi$ is expressed as $\pi$(s) =
arg maxa$\in$$\pazocal{A}$$\pazocal{Q}$$*$(s,a). We used DQN with duelling
network architecture having value & advantage streams to approximate the
state-action value function  $\pazocal{Q}$(s,a)  which predicts best possible
action as learned from the policy $\pi$.

$Q\textsuperscript{*}(s,a)=\mathop{\mathbb{E}}[R(s,a)+\gamma\sum\limits_{s^{\prime}}P(s^{\prime}|s,a)\max\limits_{a^{\prime}}Q\textsuperscript{*}(s^{\prime},a^{\prime})]$
(1)

From our experimentation setup we intend to derive direct causal impact of our
interventions in traffic environment scenarios. And as we can refer back from
Figure 2, our treatment (T) is subjecting the learning agent to more risky
driving behavior. Our testing sample set involves random agent reward
calculation against perturbed and control environments which makes our
experiment equivalent to RCT. Meaning that there is no unobserved confounding
present in our experimentation i.e. backdoor criterion is satisfied. Also, in
RCTs distribution of all co-variates are same except the treatment. Co-variate
balance in observational data also implies that association is equal to
causation while calculating the potential outcomes, refer equation stated
below.

$\displaystyle P(X\mid T=1)\stackrel{{\scriptstyle d}}{{=}}P(X\mid T=0)$ (2)
$\displaystyle P(X\mid T=1)\stackrel{{\scriptstyle
d}}{{=}}P(X),T\perp\\!\\!\\!\perp X$ $\displaystyle P(X\mid
T=0)\stackrel{{\scriptstyle d}}{{=}}P(X),T\perp\\!\\!\\!\perp X$

Essentially meaning that we can use the associative difference quantity to
infer the effect of treatment on outcomes. Meaning that we can use the Average
Treatment Effect (ATE) approach for calculating the causal effect by simply
subtracting the averaged out values treatment and control potential outcomes.
In below stated equations Y(1) $\triangleq$ Y i $\mid$ do(T=1) & Y(0)
$\triangleq$ Y i $\mid$ do(T=0) and these equations hold true in case of RCTs
where causal difference can be calculated with associated difference.

$\displaystyle E[Y(1)-Y(0)]$ $\displaystyle=E[Y(1)]-E[Y(0)]$ (3)
$\displaystyle E[Y(1)]-E[Y(0)]$ $\displaystyle=E[Y\mid T=1]-E[Y\mid T=0]$

Figure 3: Possibly realistic collision examples simulated from the different
driving tasks in highway-env package environments.

Model free learning approaches generally don’t have explicit information about
the dynamics of the systems. Therefore, during the training these agents will
generalize their policies corresponding to the particular given scenarios
only. We introduce risky randomized behavior in these environment vehicles
with collision prone randomized behavior. Figure 3 shows different probable
real life collisions simulated in different highway-env package environments.
It increases generalizations on less common but highly critical scenarios
which can save users from hefty collisions. We critically analyze the
performance of our treated agents in comparison to control agents for
important tasks like performing roundabouts, handling intersections, u-turn &
two-way traffic overtakes and lane changing environments in our study.

Figure 4: Overlapping (Left) and Non-Overlapping (Right) projection analysis
for collision detection in highway-env package.

The collision between two vehicles is equivalent to intersection of two
polygons in the rendered environment output of highway-env. And we detect
these collisions between rectangular polygons with the separating axis theorem
for the given two convex polygons. Essentially, the idea is to find a line
that separates both polygons. If that line exists then polygons are separated
and collision hasn’t happened yet. Algorithmically for each edge of our base
rectangular polygon we find a perpendicular axis to current edges under
review. After that we project these edges onto that axis and in case these
projections don’t overlap it functionally means no collision as rectangular
polygons are not intersecting, refer Figure 4.

##  IV Experiment Setup

In our experimentation setup we calculate the ATE metric for namely lane
changing, two-way traffic, roundabout, intersection and u-turn tasks, refer
Figure 5. These five tasks are evaluated against increasing traffic from
default vehicle count/density to a 200% increase. For each of these traffic
scenarios we create our treatment environment with varying degrees of
acceleration & steering parameters which would not comply with MOBIL model
criteria of safety and incentives. This randomization behavior is governed by
equations stated below which lays the foundation of simulating collision prone
behavior. We create collision prone behavior by significantly changing the
equation max-min acceleration parameters in create_random() function defined
in the kinematics rules of the highway-env package. With our experimentation
setup we quantify the causal model performance improvements from introduction
of this risk factor knowledge in the agent learning process. And compare it
with the control baseline models across these five different navigation tasks
against a spectrum of increasing traffic density.

$\displaystyle
acc_{p}=acc\textsubscript{min}+\textit{rand[0,1]}*(acc\textsubscript{max}-acc\textsubscript{min})$
(4) $\displaystyle
str_{p}=str\textsubscript{min}+\textit{rand[0,1]}*(str\textsubscript{max}-str\textsubscript{min})$

Figure 5: Different driving related navigation environment tasks for collision
prone scenario analysis.

We rebuild the highway-env package environments with our custom changes by
altering the environment configurations, randomizing behavior and adding new
vehicles to strategically increase the traffic density in order to simulate
risky driver behavior. In the lane changing task for treatment & control model
training we incrementally increase the vehicle count from 50 to 150 vehicles
accompanied with an equivalent intermittent increase of vehicle density by
100% and having increased randomized risky behavior on episode duration length
of 20 seconds. We train unique agents for navigation on each different traffic
count environments in our experimentation setup corresponding to every vehicle
count increase. Also, we plot a comparative analysis performance graph of
control and treatment agent across these different environment iterations and
calculate ATE of our perturbations. Similarly, for the u-turn environment we
uniformly increase our vehicle count from 3 to 12 with incremental increase of
vehicle count of 3. Also, for two-way traffic we reduce original environment
length to 2/3rd of the original and incrementally increase the vehicle traffic
count from base 5 in direction and 2 in opposite direction vehicles to 15 in
direction and 6 in opposite direction vehicles. For collision prone treatment
in intersection task driving tasks we rewire the randomization behavior to a
more risky one with our acceleration and deceleration tuning. We also increase
the vehicle count from 10 to 30 incrementally with an interval gap of 5
vehicles and alongside we incrementally increase the spawning probability by
0.1 until it reaches its maximum value. Finally, for roundabout tasks we
incrementally increase the traffic from 5 to 15 vehicles with risk prone
randomization in our treatment environment for agent training and performance
comparison with control baseline. Each of these environment configurations
requires the environment to be rebuilt iteratively. Additionally, repeated
model training against each new treatment & control environment samples with
different initiating random seeds. For evaluating our model’s performance we
have kept the traffic as constant in our test population set against the
corresponding treatment and control environment on which agents were trained.
And we have only changed the risky behavior in treatment and control
environment sets to calculate the ATE for measuring causal performance
improvement.

Figure 6: Dueling Deep Q-Learning baseline model architecture used for agent
training.

We use the DQN reinforcement learning modelling technique with ADAM optimizer
for our experiment with a learning rate of 5e-4. Our discount factor used is
0.99 and environment observation vector data is fed in a batch size of 100.
Our agents are trained over 3072 episodes until they converge to average
reward from a given driving environment. We use the dueling network design
which utilizes advantage function A(s,a) which helps in estimating state-
action value function Q(s,a) for state-action pairs more precisely. This is
done by splitting the network into two streams, value and advantage ones which
share some base hidden layers. The shared network consists of 3 fully
connected layers of 256, 192 and 128 units respectively. Value and advantage
stream consists of 2 layers of 128 units each. The final output of these
streams is also fully connected to the network. The value stream has 1 output
of the calculated value function for a given state. And the advantage stream
has na outputs representing the number of discrete possible actions for a
given state. The output vectors from these two streams are combined to
calculate the state-action value function estimate with the help from below
stated equation, refer Figure 6 for model architecture. 111Baseline DQN agent
implementations referenced from: github.com/eleurent/rl-agents

$Q(s,a)=V(s)+A(s,a)$ (5)

##  V Results

Another randomization factor in our experimentation setup involves initial
random seed values.This is done for adding random behavior to tasks like
randomizing vehicle acceleration & deceleration, spawning vehicles, vehicle
location etc. Therefore, we test our trained treatment and control models
against the risk-prone and regular driving environments with different
randomization seed values to average any anomalous results. Hence, we measure
the ATE by evaluating our agents against several randomized seed values for
risk prone and regular driving environments. This calculation is simply
summarized by the equation below where summation of first two terms evaluate
the average reward calculated from treatment models in risk prone and regular
environment samples. The last two terms calculate the same for control agent
in both these environment samples again respectively and the test set sample
count is expressed as $\pazocal{N}$ T = $\pazocal{N}$ C = 100. The associated
difference of these quantities gives us the ATE for performance improvements
in our robust agents trained on perturbed environments as explained earlier in
the problem formulation section.

$ATE=1/N\textsubscript{T,
C}*[\sum_{i=1}^{\textsubscript{N\textsubscript{T}}}R^{T}\textsubscript{avg\textsuperscript{i}}+\sum_{i=1}^{\textsubscript{N\textsubscript{C}}}R^{T}\textsubscript{avg\textsuperscript{i}}-\sum_{i=1}^{\textsubscript{N\textsubscript{T}}}R^{C}\textsubscript{avg\textsuperscript{i}}+\sum_{i=1}^{\textsubscript{N\textsubscript{C}}}R^{C}\textsubscript{avg\textsuperscript{i}}]\
$ (6)

Figure 7: ATE plots for quantifying the performance improvements attained by
leveraging the knowledge learnt from collision prone driving scenarios.

ATE results from Figure 7 clearly demonstrates to us the advantage of teaching
agents these risk prone critical scenarios. 222For better readability purposes
we converted the calculated ATE values to their respective percentages in
Figure 7. Across all tasks we observe that as traffic density increases like
real-life scenarios in heavily populated cities the positive effect of knowing
perturbed scenarios is pronounced for our robust treatment agents. More
importantly there is also a declining trend of average reward as the traffic
increases for every driving task analyzed for highway-env. This depreciation
in agent performance both for control and treatment models can be attributed
to constantly decreasing safety distance amongst all vehicles causing more
than expected collisions and slow progression by our ego-vehicle across these
environments due to heavy traffic. Even with decreasing average reward trend
across our test set environment samples the performance of treatment models
has always exceeded the control models. Also, the relative improvements in ATE
values further increases as the traffic continues to increase demonstrating
strong robustness of treatment agents.

Currently our scope of work is limited to few but critically important driving
scenarios. Also, we have used only homogeneous agents in our analysis and
attempted to analyze critical knowledge leveraging components on single agent
only i.e. our ego-vehicle. Plus our randomization mechanism though uniform
doesn’t necessarily follow human-like behavior while generating risk prone
scenarios. But, our causal effect estimation approach that quantifies the
information learnt from perturbed scenarios does demonstrate promising
results. It holds a vast scope of practical applications for creating more
interpretable, metric-oriented & key performance indicator  (KPI) driven
autonomous agent systems.

##  VI Conclusion

Our experiments from this paper provide insights into the importance of using
deliberate interventions of collision prone behaviors while training agents
for stochastic processes like autonomous driving. By using the MDP formulation
of the discussed driving scenarios with collision simulation perturbations we
were able to generate more robust agents. Our treatment model experimentation
setup used episode data from traffic clogged lanes and risky randomized
behavior during training which finally resulted in positive ATE result values.
Which proved that agents trained with a wider range of collision prone
scenarios perform better than the existing vanilla simulation agents. Also, we
casually quantified the impact of our interventions for the discussed model
free learning DQN technique which assisted us in accurately estimating the
performance improvements. For every driving scenario environment our new
agents produced better results and were proved to be better collision
deterrents. Therefore, underscoring the importance of learning valuable
lessons from risk prone scenario simulations for creating safe autonomous
driving agents.

## References

  * [1] D. Elliott, W. Keen, and L. Miao, “Recent advances in connected and automated vehicles,” _journal of traffic and transportation engineering (English edition)_ , vol. 6, no. 2, pp. 109–131, 2019.
  * [2] S. A. Bagloee, M. Tavana, M. Asadi, and T. Oliver, “Autonomous vehicles: challenges, opportunities, and future implications for transportation policies,” _Journal of modern transportation_ , vol. 24, no. 4, pp. 284–303, 2016.
  * [3] J. Joy and M. Gerla, “Internet of vehicles and autonomous connected car-privacy and security issues,” in _2017 26th International Conference on Computer Communication and Networks (ICCCN)_. IEEE, 2017, pp. 1–9.
  * [4] J. M. Anderson, K. Nidhi, K. D. Stanley, P. Sorensen, C. Samaras, and O. A. Oluwatola, _Autonomous vehicle technology: A guide for policymakers_. Rand Corporation, 2014.
  * [5] T. Litman, “Autonomous vehicle implementation predictions: Implications for transport planning,” 2020.
  * [6] L. Li, W.-L. Huang, Y. Liu, N.-N. Zheng, and F.-Y. Wang, “Intelligence testing for autonomous vehicles: A new approach,” _IEEE Transactions on Intelligent Vehicles_ , vol. 1, no. 2, pp. 158–166, 2016.
  * [7] M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer, “Adaptive stress testing for autonomous vehicles,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 1–7.
  * [8] L. Cui, J. Hu, B. B. Park, and P. Bujanovic, “Development of a simulation platform for safety impact analysis considering vehicle dynamics, sensor errors, and communication latencies: Assessing cooperative adaptive cruise control under cyber attack,” _Transportation research part C: emerging technologies_ , vol. 97, pp. 1–22, 2018.
  * [9] P. Liu, Z. Xu, and X. Zhao, “Road tests of self-driving vehicles: affective and cognitive pathways in acceptance formation,” _Transportation research part A: policy and practice_ , vol. 124, pp. 354–369, 2019.
  * [10] P. Liu, R. Yang, and Z. Xu, “How safe is safe enough for self-driving vehicles?” _Risk analysis_ , vol. 39, no. 2, pp. 315–325, 2019.
  * [11] E. Thorn, S. C. Kimmel, M. Chaka, B. A. Hamilton _et al._ , “A framework for automated driving system testable cases and scenarios,” United States. Department of Transportation. National Highway Traffic Safety …, Tech. Rep., 2018.
  * [12] L. Li, X. Wang, K. Wang, Y. Lin, J. Xin, L. Chen, L. Xu, B. Tian, Y. Ai, J. Wang _et al._ , “Parallel testing of vehicle intelligence via virtual-real interaction,” _Science robotics_ , vol. 4, no. 28, p. eaaw4106, 2019.
  * [13] X. Yan, S. Feng, H. Sun, and H. X. Liu, “Distributionally consistent simulation of naturalistic driving environment for autonomous vehicle testing,” _arXiv preprint arXiv:2101.02828_ , 2021.
  * [14] S. Feng, X. Yan, H. Sun, Y. Feng, and H. X. Liu, “Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment,” _Nature communications_ , vol. 12, no. 1, pp. 1–14, 2021.
  * [15] W. Ding, B. Chen, M. Xu, and D. Zhao, “Learning to collide: An adaptive safety-critical scenarios generating method,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 2243–2250.
  * [16] K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song, “Robust physical-world attacks on deep learning visual classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1625–1634.
  * [17] C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie, and A. Yuille, “Adversarial examples for semantic segmentation and object detection,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1369–1378.
  * [18] E. Leurent, “An environment for autonomous driving decision-making,” https://github.com/eleurent/highway-env, 2018.
  * [19] ——, “rl-agents: Implementations of reinforcement learning algorithms,” https://github.com/eleurent/rl-agents, 2018.
  * [20] E. Leurent, Y. Blanco, D. Efimov, and O.-A. Maillard, “Approximate robust control of uncertain dynamical systems,” _arXiv preprint arXiv:1903.00220_ , 2019.
  * [21] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [22] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 30, no. 1, 2016.
  * [23] J. Pearl _et al._ , “Causal inference in statistics: An overview,” _Statistics surveys_ , vol. 3, pp. 96–146, 2009.
  * [24] J. Pearl and D. Mackenzie, _The book of why: the new science of cause and effect_. Basic books, 2018.
  * [25] L. G. Neuberg, “Causality: Models, reasoning, and inference,” 2003.
  * [26] P. W. Holland, “Statistics and causal inference,” _Journal of the American statistical Association_ , vol. 81, no. 396, pp. 945–960, 1986.
  * [27] F. Belletti, D. Haziza, G. Gomes, and A. M. Bayen, “Expert level control of ramp metering based on multi-task deep reinforcement learning,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 4, pp. 1198–1207, 2017.
  * [28] C. Wu, A. Kreidieh, E. Vinitsky, and A. M. Bayen, “Emergent behaviors in mixed-autonomy traffic,” in _Conference on Robot Learning_. PMLR, 2017, pp. 398–407.
  * [29] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Conference on robot learning_. PMLR, 2017, pp. 1–16.
  * [30] A. Sauer, N. Savinov, and A. Geiger, “Conditional affordance learning for driving in urban environments,” in _Conference on Robot Learning_. PMLR, 2018, pp. 237–252.
  * [31] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi-agent, reinforcement learning for autonomous driving,” _arXiv preprint arXiv:1610.03295_ , 2016.
  * [32] M. Bansal, A. Krizhevsky, and A. Ogale, “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst,” _arXiv preprint arXiv:1812.03079_ , 2018.
  * [33] H. Sun, S. Feng, X. Yan, and H. X. Liu, “Corner case generation and analysis for safety assessment of autonomous vehicles,” _arXiv preprint arXiv:2102.03483_ , 2021.
  * [34] E. Leurent, “Safe and efficient reinforcement learning for behavioural planning in autonomous driving,” Ph.D. dissertation, Université de Lille, 2020.
  * [35] N. Carrara, E. Leurent, R. Laroche, T. Urvoy, O.-A. Maillard, and O. Pietquin, “Budgeted reinforcement learning in continuous state space,” _arXiv preprint arXiv:1903.01004_ , 2019.
  * [36] D. Ernst, P. Geurts, and L. Wehenkel, “Tree-based batch mode reinforcement learning,” _Journal of Machine Learning Research_ , vol. 6, pp. 503–556, 2005.
  * [37] E. Leurent, D. Efimov, T. Raissi, and W. Perruquetti, “Interval prediction for continuous-time systems with parametric uncertainties,” in _2019 IEEE 58th Conference on Decision and Control (CDC)_. IEEE, 2019, pp. 7049–7054.
  * [38] E. Leurent and O.-A. Maillard, “Practical open-loop optimistic planning,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2019, pp. 69–85.
  * [39] S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, “Continuous deep q-learning with model-based acceleration,” in _International Conference on Machine Learning_. PMLR, 2016, pp. 2829–2838.
  * [40] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, “Dueling network architectures for deep reinforcement learning,” in _International conference on machine learning_. PMLR, 2016, pp. 1995–2003.
  * [41] A. Mavrogiannis, R. Chandra, and D. Manocha, “B-gap: Behavior-guided action prediction for autonomous navigation,” _arXiv preprint arXiv:2011.03748_ , 2020.
  * [42] R. Chandra, U. Bhattacharya, T. Mittal, A. Bera, and D. Manocha, “Cmetric: A driving behavior measure using centrality functions,” _arXiv preprint arXiv:2003.04424_ , 2020.
  * [43] J. Hicks _et al._ , _Causality in economics_. Australian National University Press, 1980.
  * [44] D. A. Smirnov and I. I. Mokhov, “From granger causality to long-term causality: Application to climatic data,” _Physical Review E_ , vol. 80, no. 1, p. 016208, 2009.
  * [45] C. W. Granger, “Some recent development in a concept of causality,” _Journal of econometrics_ , vol. 39, no. 1-2, pp. 199–211, 1988.
  * [46] J. Pearl, “Does obesity shorten life? or is it the soda? on non-manipulable causes,” _Journal of Causal Inference_ , vol. 6, no. 2, 2018.
  * [47] S. Wright, “The method of path coefficients,” _The annals of mathematical statistics_ , vol. 5, no. 3, pp. 161–215, 1934.
  * [48] T. S. Richardson and J. M. Robins, “Single world intervention graphs: a primer,” in _Second UAI workshop on causal structure learning, Bellevue, Washington_. Citeseer, 2013.
  * [49] E. Bareinboim, J. Correa, D. Ibeling, and T. Icard, “On pearl’s hierarchy and the foundations of causal inference,” _ACM Special Volume in Honor of Judea Pearl (provisional title)_ , 2020.
  * [50] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv preprint arXiv:1606.01540_ , 2016.
  * [51] M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” _Physical review E_ , vol. 62, no. 2, p. 1805, 2000.
  * [52] A. Kesting, M. Treiber, and D. Helbing, “General lane-changing model mobil for car-following models,” _Transportation Research Record_ , vol. 1999, no. 1, pp. 86–94, 2007.
  * [53] R. E. Bellman and S. E. Dreyfus, _Applied dynamic programming_. Princeton university press, 2015, vol. 2050.