[1,3]Deepthi Pathare

1]Department of Computer Science and Engineering, Chalmers University of
Technology, Göteborg, 41296, Sweden

2]Department of Mechanics and Maritime Sciences, Chalmers University of
Technology, Göteborg, 41296, Sweden

3]Safe and Efficient Driving, Volvo Group of Trucks Technology, Göteborg,
41715, Sweden

# Tactical Decision Making for Autonomous Trucks by Deep Reinforcement
Learning with Total Cost of Operation Based Reward

<EMAIL_ADDRESS>Leo Laine<EMAIL_ADDRESS>Morteza Haghir
Chehreghani<EMAIL_ADDRESS>[  [  [

###### Abstract

We develop a _deep reinforcement learning_ framework for tactical decision
making in an autonomous truck, specifically for Adaptive Cruise Control (ACC)
and lane change maneuvers in a highway scenario. Our results demonstrate that
it is beneficial to separate high-level decision-making processes and low-
level control actions between the reinforcement learning agent and the low-
level controllers based on physical models. In the following, we study
optimizing the performance with a realistic and multi-objective reward
function based on Total Cost of Operation (TCOP) of the truck using different
approaches; by adding weights to reward components, by normalizing the reward
components and by using curriculum learning techniques.

###### keywords:

Autonomous Trucks, Deep Reinforcement Learning, Curriculum Learning, Total
Cost of Operation

##  1 Introduction

The efficiency and safety of transport networks have a huge impact on the
socio-economic development of the globe. A significant part of this network is
freight transport, of which more than $70\%$ is carried out by trucks [1]. The
modeling of these complex networks, with a particular focus on traffic
scenarios, including trucks and their Total Cost of Operation (TCOP) is thus
of paramount importance to developing sustainable traffic solutions.

At a mesoscopic scale, a truck can significantly affect the surrounding
traffic [2], primarily due to its comparatively larger size and length [3]. It
also needs more cooperation from surrounding vehicles in order to perform
specific maneuvers, such as lane change in multiple-lane dense traffic
scenarios [4]. Further, if it is a Long Combination Vehicle (LCV), its
influence on the safety and evolution of the surrounding traffic is
substantial [5].

Modern vehicles, including trucks, are equipped with a number of features that
enhance their performance at different levels. For example, modern trucks
operate in connected networks, constantly exchanging data related to their
location and performance with their clients. They are also equipped with
Advanced Driver Assistance Systems (ADAS) to assist the driver with complex
driving tasks and to enhance safety [6, 7]. Adaptive Cruise Control (ACC) is
such a driver assistance function that provides longitudinal control and
maintains a safe distance with the vehicle ahead [8].

Artificial Intelligence (AI) and Machine Learning have revolutionized the
connectivity and autonomy of vehicles, including trucks. With the integration
of sensors, cameras, and sophisticated onboard systems, machine learning
algorithms can analyze vast amounts of real-time data, allowing trucks to make
intelligent decisions on the road. The continuous advancements in machine
learning continue to push the boundaries of connectivity and autonomy,
transforming the trucking industry towards a more efficient and intelligent
future.

A widely used machine learning framework in the context of autonomous systems
is Reinforcement Learning (RL). RL overcomes the challenges of traditional
search based methods such as A* search which lack the ability to generalize to
unknown situations in a non-deterministic environment and are computationally
expensive [9]. RL has been increasingly used for solving complex problems
related to autonomous driving, such as navigation, trajectory planning,
collision avoidance, and behavior planning [10, 11, 12]. The application of
this framework to autonomous trucks is a relatively new area of research. An
important contribution in this direction is the work [13], which implements an
RL framework for autonomous truck driving in SUMO. They study how a Bayesian
RL technique, based on an ensemble of neural networks with additional
randomized prior functions (RPF), can be used to estimate the uncertainty of
decisions in autonomous driving. On the other hand, more results exist when we
consider autonomous driving of passenger cars. The study in [14] develops a
cooperative adaptive cruise control (CACC) using RL for securing longitudinal
following of a front vehicle using vehicle-to-vehicle communication. The CACC
system has a coordination layer that is responsible for the selection of high-
level actions, e.g., lane changing or secure vehicle following. The action
layer, which is a policy-gradient RL agent, must achieve this action by
selecting the appropriate low-level actions that correspond to the vehicle’s
steering, brakes, and throttle. Another study in [15] proposes a Supervised
Actor-Critic (SAC) [16, 17] approach for optimal control of ACC. Their
framework contains an upper level controller based on SAC, which chooses
desired acceleration based on relative velocity and distance parameters of
leading and following vehicles. A low level controller receives the
acceleration signal and transfers it to the corresponding brake and/or
throttle control action. In [18], the authors develop a Deep Reinforcement
Learning (DRL) framework for ACC and compare the results with Model Predictive
Control. In all these studies, the RL agent performs step control of
acceleration or other driving maneuvers such as braking and steering. For this
reason, it can take a longer time for the agent to achieve maximum speed even
though there is no leading vehicle in front, making the process apparently
inefficient.

There is only limited prior work where the RL agent is trained to learn high
level actions such as choosing the safe gap with the leading vehicle and the
actual speed control is performed by a low level controller. The work in [19]
develops a similar system that aims to achieve simultaneous optimization of
traffic efficiency, driving safety, and driving comfort through dynamic
adaptation of the inter-vehicle gap. It suggests a dual RL agent approach -
the first RL agent is designed to find and adapt the optimal time-to-collision
(TTC) threshold based on rich traffic information, including both macroscopic
and microscopic traffic data obtained from the surrounding environment and the
second RL agent is designed to derive an optimal inter-vehicle gap that
maximizes traffic flow, driving safety, and comfort at the same time. In [20],
authors develop a Deep Deterministic Policy Gradient (DDPG) - Proportional
Integral Derivative (PID) controller for longitudinal control of vehicle
platooning. They use the DDPG algorithm to optimize the $K_{p}$, $K_{d}$ and
$K_{i}$ constants for the PID controller which controls the desired speed.
These works mainly focus on longitudinal control of the vehicles in a platoon,
and the effects of lane change maneuvers are not studied sufficiently.

In this study, we develop ACC, together with lane change maneuvers for an
autonomous truck in a highway scenario. We first develop a baseline
architecture for tactical decision making and control based on the previous
work [13, 21]. Here, the speed control and lane change decision making are
done by an RL agent. With the baseline architecture, we show how adding
relevant features to the observation space improves the performance of the
agent. Then, we train a refined architecture where we separate the high-level
decision making and low-level control actions between the RL agent and low-
level controllers based on physical models. Compared to the existing
architectures in the literature, our RL agent only needs to learn a policy to
choose high level actions such as the gap to be maintained. Interestingly, we
find this optimized action space improves the learning compared to speed
control actions regardless of the RL algorithm used. We evaluate the
performance with three different RL algorithms: Deep Q-Network (DQN) [22],
Advantage Actor-Critic (A2C) [23] and Proximal Policy Optimization (PPO) [24].

We design two distinct reward functions to train the RL agents. The first one
focuses on safety aspects, while the second, more complex one, is based on
Total Cost of Operation (TCOP). This TCOP-oriented reward function is
realistic, encompassing actual expenses like energy consumption and human
resources. TCOP serves as a comprehensive metric covering various operational
expenses in autonomous trucking. By introducing a TCOP-based reward function,
we aim to optimize driving strategies for sustainable and economically viable
operations, in line with our broader objective of fostering cost-effective and
environmentally sustainable transportation solutions. Furthermore, our refined
RL architecture, combined with this TCOP-centric reward function, establishes
a robust framework for future research endeavours, aimed at enhancing the
economic viability and operational efficiency of autonomous driving systems.

This work is an extension of our previous work [25] wherein we additionally,
(i) investigate training of RL agent with a complex TCOP based reward function
consisting of realistic costs and revenue values with and without
normalization (ii) extend our RL framework by incorporating curriculum
learning techniques and compare the results with the non-curriculum learning
approach. In Curriculum Reinforcement Learning (CRL) [26, 27], the RL agent
starts learning simpler tasks and is gradually exposed to complex tasks.
Effectiveness of curriculum learning has been investigated in a number of
problems related to autonomous driving, though different from our setting. The
study in [28] introduces multi-stage learning in complex driving environments.
Different stages include a diverse set of starting locations, varying weather
conditions, and dense traffic scenarios. The action space consists of
accelerator or brake values and steering angles. The reward function penalizes
collisions, following the wrong route, and exceeding speed limits, guiding the
agents towards safe and efficient driving behaviors. Despite not yet reaching
state-of-the-art performance, the study demonstrates consistent driving
behavior across different scenarios and robustness to weather changes. The
papers [29] and [30] utilize a curriculum learning approach for overtaking in
autonomous driving. [29] employs a two-stage successive learning progression,
where agents first learn to drive as fast as possible and then master the
skill of overtaking efficiently. On the other hand, [30] introduces a three-
stage curriculum learning process where agents first learn to race, then focus
on overtaking maneuvers, and finally refine their skills to race at high
speeds while avoiding collisions. Both of these works demonstrate that
curriculum learning boosts convergence and contributes to a better final
policy.

All our results are obtained using Simulation of Urban Mobility (SUMO) [31],
which is a widely used simulation platform for the conceptual development of
autonomous vehicles. The tailored RL environment based on SUMO that we
developed for heavy vehicle combination driving in highway traffic is provided
as open access, which offers a great open source framework for investigating
various RL methods in complex settings.111Source Code:
https://github.com/deepthi-pathare/Autonomous-truck-sumo-gym-env

##  2 Tactical Decision Making with Reinforcement Learning

###  2.1 Environment Setup

In this work, we consider a dynamic highway traffic environment with an
autonomous truck (ego vehicle) and passenger cars as shown in Fig. 1. The
maximum speed of the ego vehicle is set to be 25 m/s. Furthermore, 15
passenger cars with speed between 15 m/s and 35 m/s are simulated based on the
default Krauss car following model [32] and LC2013 lane change model [33] in
SUMO. The initial position and speed pattern of the surrounding vehicles are
set randomly which makes the surrounding traffic nondeterministic throughout
the simulation. Starting from the initial position (which will be 800m after
initialization steps in SUMO), the ego vehicle is expected to reach the target
set at a distance of 3000m. This means that the ego vehicle has to drive 2200m
on the highway in each episode. An episode will also terminate if a hazardous
situation such as a collision or driving outside the road happens or if a
maximum of 500 RL steps are executed. The observation or state of the
environment at each step contains information about position, speed, and state
of left/right indicators for ego vehicle and vehicles within the sensor range.
The sensor range used here is 200m.

Figure 1: Schematic diagram of the highway simulation environment. The truck
in green color is the ego vehicle.

###  2.2 Reinforcement Learning Framework

Reinforcement learning is a branch of machine learning where an agent acts in
an environment and tries to learn a policy, $\pi$, that maximizes a cumulative
reward function. The policy defines which action, $a$, to take, given an
observation or state, $s$. This action leads to a new state of the environment
$s^{\prime}$, and returns a reward, $r$.

####  2.2.1 MDP Formulation

A reinforcement learning problem can be modeled as a Markov Decision Process
(MDP), which is defined by the tuple (S, A, T, R, $\gamma$), where S is the
state space, A is the action space, P is the transition dynamics, R is the
reward model, and $\gamma$ is a discount factor. The tactical decision making
problem for autonomous truck in this study is formulated as MDP as follows:

  * •

State Space (S): The state includes observations from ego vehicle and the
surrounding vehicles. Following are the observations for the ego vehicle:

    1. 1.

Longitudinal speed ($v_{x0}$)

    2. 2.

Lane change state ($sign(v_{y0})$)

    3. 3.

Lane number

    4. 4.

State of left indicator

    5. 5.

State of right indicator

    6. 6.

Target(leading) vehicle distance ($d_{lead}$)

Following are the observations for each vehicle in the sensor range of the ego
vehicle:

    1. 1.

Relative longitudinal distance from ego vehicle ($d_{xi}$)

    2. 2.

Relative lateral distance from ego vehicle ($d_{yi}$)

    3. 3.

Relative longitudinal speed with ego vehicle ($v_{xi}-v_{x0}$)

    4. 4.

Lane change state ($sign(v_{yi})$)

    5. 5.

Lane number

    6. 6.

State of left indicator

    7. 7.

State of right indicator

  * •

Action Space (A): The action space is defined separately for the baseline and
new architectures described in section 2.3.

  * •

Transition Dynamics(P): The transition dynamics is defined by SUMO, and is not
known to the RL agent.

  * •

Reward Function(R): A basic reward function focused on safety aspects is
designed to motivate the agent to drive at maximum speed and reach the target
without any hazardous situations. This reward function is similar to that in
[13] except that a reward for reaching the target is added, whereas the
penalty for emergency braking is not considered for simplicity. The reward at
each time step is given by (1).

$\displaystyle
r(t)=\frac{v_{t}}{max\\_v}-I_{l}\>P_{l}-I_{c}\>P_{c}-I_{nc}\>P_{nc}-I_{o}\>P_{o}$
(1) $\displaystyle+I_{tar}\>\frac{R_{tar}}{T}$

Here $v_{t}$ is the velocity of the vehicle at time step $t$ and $max\\_v$ is
the maximum velocity of the vehicle. $I$ is an indicator function, which takes
a value of 1 when the corresponding condition is satisfied and $P$ and $R$ are
the corresponding penalty and reward values respectively. The possible
conditions are lane change ($l$), collision ($c$), near collision ($nc$),
driving outside the road ($o$) and reaching the target ($tar$). $T$ is the
total time it takes to reach the target. The parameter values used are given
in Table 1.

Table 1: Parameter values used in the basic reward function. Parameter |  Value
---|---
$P_{l}$ |  1
$P_{c}$ |  10
$P_{nc}$ |  10
$P_{o}$ |  10
$R_{tar}$ |  100

Furthermore, we have also designed a complex and multi-objective reward
function based on TCOP which is described in section 2.4.

  * •

Discount Factor($\gamma$): $\gamma$ is set to 0.99.

####  2.2.2 Reinforcement Learning Algorithms

We choose three reinforcement learning algorithms as listed below for the
decision making in baseline and new architectures. The implementations of
these algorithms from the stable-baselines3 library with the default
hyperparameters are used [34].

  1. 1.

Deep Q-Network (DQN)
DQN is a reinforcement learning algorithm based on the Q-learning algorithm,
which learns the optimal action-value function by iteratively updating
estimates of the Q-value for each state-action pair. In DQN, the optimal
action-value function in a given environment is approximated using a neural
network model. The corresponding objective function is given by,

$\displaystyle
L(\theta)=\mathbb{E}_{s,a,r,s^{\prime}}\Big{[}\Big{(}y-Q(s,a;\theta)\Big{)}^{2}\Big{]}$
(2)

where $\theta$ represents the weights of the Q-network, $s$ and $a$ are the
state and action at time $t$, $r$ is the immediate reward, $s^{\prime}$ is the
next state, and
$y=r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta^{-})$ is the target
value. The target value is the sum of the immediate reward and the discounted
maximum Q-value of the next state, where $\gamma$ is the discount factor and
$\theta^{-}$ represents the weights of a target network with delayed updates.
The objective function is to minimize the mean squared error between the
estimated Q-value and the target value, which is then optimized using
stochastic gradient descent.

Another improvement of DQN over the standard Q learning approaches is the use
of a replay buffer and target network. The agent uses replay buffer to store
transitions and samples from it randomly during training. This enables the
efficient use of past experiences. The target network is a copy of the
Q-network with delayed updates. Together with the replay buffer, it improves
the learning stability of the model and help prevent over fitting.

  2. 2.

Advantage Actor-Critic (A2C)
A2C is a reinforcement learning algorithm that combines the actor-critic
method with an estimate of the advantage function. In the actor-critic method,
the agent learns two neural networks: an actor network that outputs a
probability distribution over actions, and a critic network that estimates the
state-value function. The actor network is trained to maximize the expected
reward by adjusting the policy parameters, while the critic network is trained
to minimize the difference between the estimated value function and the true
value function. Advantage function calculates how better taking an action at a
state is compared to the average value of the state which is computed as
below:

$A(s_{t},a_{t})=r_{t+1}+\gamma V(s_{t+1})-V(s)$ (3)

Actor uses the computed advantage function value from critic as a feedback and
updates the policy parameters $\theta$ as,

$\Delta\theta=\alpha_{\theta}A(s_{t},a_{t})\nabla_{\theta}\left(\log\pi_{\theta}(s_{t},a_{t})\right)$
(4)

Critic updates its value function parameters $w$ as,

$\Delta w=\alpha_{w}A(s_{t},a_{t})\nabla_{w}\hat{v}_{w}\left(s_{t}\right)$ (5)

Here, $\alpha_{\theta}$ and $\alpha_{w}$ are the learning rates.

  3. 3.

Proximal Policy Optimization (PPO)
PPO belongs to the family of policy gradient algorithms and has exhibited
great potential in effectively addressing diverse RL problems [24, 35]. In
addition, PPO benefits from simple yet effective implementation and a broad
scope of applicability.

The PPO algorithm is designed to address the challenges such as the
instability that can arise when the policies are rapidly updated. PPO works by
optimizing a surrogate objective function that measures the difference between
the updated policy and the previous one. The surrogate objective function of
PPO is defined as the minimum of two terms: the ratio of the new policy to the
old policy multiplied by the advantage function, and the clipped ratio of the
new policy to the old policy multiplied by the advantage function. In
mathematical terms, this function is given as,

$\displaystyle L^{CLIP}(\theta)=$ (6)
$\displaystyle\hat{{E}}_{t}\Big{[}\min\Big{(}r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\Big{)}\Big{]}$

where $\theta$ represents the policy parameters,
${r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}}$
is the likelihood ratio between the current policy and the previous policy,
$\hat{A}_{t}$ is the estimated advantage of taking action $a_{t}$ in state
$s_{t}$ and $\epsilon$ is a hyperparameter that controls the degree of
clipping. The clipping term ensures that the policy update does not result in
significant policy changes, which can destabilize the learning process. The
PPO algorithm maximizes this objective function using stochastic gradient
ascent to update the policy parameters. The expectation is taken over a batch
of trajectories sampled from the environment.

###  2.3 System Architectures

####  2.3.1 Baseline architecture

The baseline architecture we consider in this work is inspired from [13] and
is illustrated in Fig. 2. Here the decision making is done solely by an RL
agent, every 1s. The action space for the agent is discrete. Each action
consists of a combination of longitudinal action and lateral action.
Longitudinal actions are speed changes of 0, 1, -1 or -4 m/s. Lateral actions
are to stay on lane, change to left lane, or change to right lane. In this
paper, we compare the performance of this baseline architecture with our new
architecture described below.

Figure 2: Overview of the baseline architecture

####  2.3.2 New architecture

Fig. 3 shows the new architecture diagram for the automated truck driving
framework. It has mainly three components; a high level decision making agent
based on RL, a low level controller for longitudinal control and a low level
controller for lateral control. The RL agent chooses a longitudinal action or
a lateral action based on the current state of the SUMO environment.

Figure 3: Overview of the new architecture

The action space is defined below.

  1. 1.

Set short time gap with leading vehicle (1s)

  2. 2.

Set medium time gap with leading vehicle (2s)

  3. 3.

Set long time gap with leading vehicle (3s)

  4. 4.

Increase desired speed by 1 m/s

  5. 5.

Decrease desired speed by 1 m/s

  6. 6.

Maintain current desired speed and time gap

  7. 7.

Change lane to left

  8. 8.

Change lane to right

The longitudinal action can be one of the following: set desired time gap
(short, medium, long), increment or decrement desired speed or maintain the
current time gap and desired speed. When RL agent chooses one of these
actions, the longitudinal controller will compute acceleration of ego vehicle
based on Intelligent Driver Model (IDM) [36] given by,

$\displaystyle\dot{v}_{\alpha}=\frac{\mathrm{d}v_{\alpha}}{\mathrm{d}t}=a\left(1-\left(\frac{v_{\alpha}}{v_{0}}\right)^{\delta}-\left(\frac{s^{*}\left(v_{\alpha},\Delta
v_{\alpha}\right)}{s_{\alpha}}\right)^{2}\right),$ (7) $\displaystyle
s^{*}\left(v_{\alpha},\Delta
v_{\alpha}\right)=s_{0}+v_{\alpha}T+\frac{v_{\alpha}\Delta
v_{\alpha}}{2\sqrt{ab}}$

where $\alpha$ is the ego vehicle and $\alpha-1$ is the leading vehicle. $v$
denotes the velocity and $l$ denotes the length of the vehicle.
$s_{\alpha}:=x_{\alpha-1}-x_{\alpha}-l_{\alpha-1}$ is the net distance and
$\Delta v_{\alpha}:=v_{\alpha}-v_{\alpha-1}$ is the velocity difference.
$v_{0}$ (desired velocity), $s_{0}$ (minimum spacing), $T$ (desired time gap),
$a$ (maximum acceleration), and $b$ (comfortable braking deceleration) are
model parameters.

Here IDM uses the latest desired speed and time gap set by the RL agent. It
computes a new acceleration for the truck and sets the resulting speed in SUMO
every 0.1s. This process continues for a total duration of 1s, after which the
RL agent chooses the next high level action. Note that in the new
architecture, the ACC mode is always activated for the truck by construction.

The lateral action is to change the truck’s course to left or right lane. When
the RL agent chooses a lateral action, the lateral controller initiates the
lane change. Lane change is performed using the default LC2013 lane change
model [33] in SUMO. The lane width is set to $3.2m$ and the lateral speed of
the truck is set to $0.8m/s$. Hence, in total, it takes $4s$ to complete a
lane change action, which corresponds to 40 SUMO time steps of $0.1s$
duration. Following this, the RL agent chooses the next high level action.

###  2.4 Total Cost of Operation

The TCOP of a truck encompasses various expenses incurred during its operation
such as energy cost, driver cost and insurance cost. We have designed a TCOP-
centric reward function to provide the agent with more realistic goals similar
to those of a human truck driver and to motivate the agent to learn safe and
cost-effective actions. The reward at each time step is given by (8). Here, we
consider realistic cost or revenue values (in euros) for each component to see
if the agent can learn a safe and cost effective driving strategy with this
reward function.

$\displaystyle r(t)=-C_{el}\>e_{t}-C_{dr}\>\Delta t-I_{l}\>P_{l}-I_{c}\>P_{c}$
(8) $\displaystyle-I_{nc}\>P_{nc}-I_{o}\>P_{o}+I_{tar}\>R_{tar}\>W_{tar}$

$C_{el}$ is the electricity cost, $e_{t}$ is the electricity consumed at time
step $t$, $C_{dr}$ is the driver cost and $\Delta t$ is the duration of a time
step. The electricity consumed during the time step $t$ ($e_{t}$) is
calculated as,

$\displaystyle e_{t}=f_{t}\>v_{t}\>\Delta{t},$ (9)

where $f_{t}$, force at time step $t$ is given by,

$\displaystyle
f_{t}=m\>a_{t}+\frac{1}{2}C_{d}\>A_{f}\>\rho_{\text{air}}\>v^{2}+m\>g\>C_{r}$
(10) $\displaystyle+m\>g\>\sin(\arctan(\frac{\text{slope}}{100}))$

Here $m$ is the mass of the vehicle, $C_{d}$ is the coefficient of air drag,
$A_{f}$ is the frontal area, $\rho_{\text{air}}$ is the air density, $C_{r}$
is the coefficient of rolling resistance, $g$ is the acceleration due to
gravity and $a$ is the acceleration of the vehicle at time step $t$. The slope
of the road is set to be 0 in this study. The parameter values used are given
in Table 2.

Table 2: Parameter values used in TCOP based reward function. Parameter |  Value
---|---
$P_{l}$ |  0.1
$P_{c}$ |  1000 euros
$P_{nc}$ |  1000 euros
$P_{o}$ |  1000 euros
$R_{tar}$ |  2.78 euros
$C_{el}$ |  0.5 euro per kwh
$C_{dr}$ |  50 euro per hour
$\Delta t$ |  1 s
$m$ |  40000 kg
$C_{d}$ |  0.36
$A_{f}$ |  10 $m^{2}$
$\rho_{\text{air}}$ |  1.225 $kg/m^{3}$
$g$ |  9.81 $m/s^{2}$
$C_{r}$ |  0.005

In the reward function (8), the penalties are defined as the average cost
incurred during hazardous situations. The average cost considered here is the
own risk payment which is the portion of an insurance claim made by the
vehicle owner or the insured for any loss and or damage that occurs when
submitting a claim. Similarly, the reward $R_{tar}$ denotes the revenue that
can be achieved by the truck when it completes the target. Revenue is computed
as the total expected cost with $20\%$ profit in an ideal scenario where the
truck drives with an average speed of $22m/s$ and zero acceleration. The total
length of travel is 2200 meters and hence the time to reach the target will be
$100s$. Based on (9) and parameter values from Table 2, the total energy
consumed can be computed as $1.85$ kwh. Then, the total energy cost will be
$1.85$ kwh $\times 0.5=0.925$ euros. The total driver cost for $100s$ will be
$1.39$ euros and the total cost becomes $0.925+1.39=2.315$ euros. Further,
adding the $20\%$ profit gives the net revenue $2.78$ euros, which is used in
the reward function along with a weight $W_{tar}$.

###  2.5 Curriculum Reinforcement Learning

Curriculum learning is a training strategy where the model is gradually
exposed to increasingly complex tasks or difficult examples over time. This
approach is based on the intuition that starting with simpler tasks allows the
model to build foundational knowledge and gradually progress to more
challenging tasks, which can help improve learning efficiency. We combine
curriculum learning with RL in the new system architecture and investigate its
effectiveness in training the RL agent with the complex reward function based
on TCOP. Here, the curriculum for training the RL agent is designed as
follows:

  * •

Curriculum-1: Learning longitudinal and lateral control without
collision/driving outside the road and by reducing driver cost

  * •

Curriculum-2: Learn to minimize energy cost

  * •

Curriculum-3: Learn to reach the destination successfully within maximum steps

In this CRL approach, we update the reward function in each curriculum whereas
the RL environment, state and action space remain the same. In this work, we
have redefined TCOP based reward function as shown in Equation 11 by removing
the penalty for lane change and the weight component $W_{tar}$ such that the
reward function only contains actual cost and revenue values during a truck’s
operation. Training an agent with realistic reward functions can lead to
policies that are more aligned with human expectations.

$\displaystyle r(t)=-C_{el}\>e_{t}-C_{dr}\>\Delta
t-I_{c}\>P_{c}-I_{nc}\>P_{nc}-I_{o}\>P_{o}+I_{tar}\>R_{tar}$ (11)

Here, the notations and values are the same as in Equation 8. This reward
function is divided into smaller components and added to the reward functions
$r_{1}(t),r_{2}(t),r_{3}(t)$ defined in Equation 12 for curriculum-1,
curriculum-2 and curriculum-3 respectively.

$\displaystyle r_{1}(t)$
$\displaystyle=-I_{c}\>P_{c}-I_{nc}\>P_{nc}-I_{o}\>P_{o}-C_{dr}\>\Delta t$
(12) $\displaystyle r_{2}(t)$ $\displaystyle=r_{1}(t)-C_{el}\>e_{t}$
$\displaystyle r_{3}(t)$ $\displaystyle=r_{2}(t)+I_{tar}\>R_{tar}$

Here, the negative reward components for drive cost and energy cost contradict
to each other as driver cost motivates the agent to drive faster while energy
cost motivates the agent to drive slower. This will lead to challenges in
learning an optimal policy as shown in the experiments section. Therefore, we
tuned the reward function for curriculum-1 and curriculum-2 to normalize these
two components with the distance travelled per time step $\Delta d$ as shown
in Equation 13.

$\displaystyle r_{1}(t)$
$\displaystyle=-I_{c}\>P_{c}-I_{nc}\>P_{nc}-I_{o}\>P_{o}-\frac{(C_{dr}\>\Delta
t)}{\Delta d}$ (13) $\displaystyle r_{2}(t)$
$\displaystyle=r_{1}(t)-\frac{(C_{el}\>e_{t})}{\Delta d}$

##  3 Experiments

This section presents the results from different experiments conducted on the
SUMO platform using the baseline and new architectures with different RL
algorithms. The experiments are conducted on a linux cluster that consists of
28 CPUs with the model Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz. The average
time elapsed for training the new architecture with DQN, A2C and PPO for
$1e^{6}$ timesteps are 12528s (3h 28min), 13662s (3h 47min), 14124s (3h 55min)
respectively.

###  3.1 Performance improvement based on selection of states

First, we show how the performance can be improved by adding relevant features
to the observation space. As mentioned in section II, the state space includes
position, speed, and lane change information of the ego vehicle and the
surrounding vehicles. In this experiment, we compare the performance of
baseline architecture by adding the distance to the leading vehicle as an
explicit feature in the above mentioned state space. The basic reward function
is used in both cases. The learning curves of average reward (over 5
realizations) using different RL algorithms are shown in Fig. 4. The results
from PPO and A2C show that the average reward has improved by adding distance
to leading vehicle as a feature in the state. Results from DQN show the
opposite trend where the average reward for state without leading vehicle
distance is higher. However, the average reward values are very low for DQN in
both state spaces. and the experiments in upcoming sections also show that DQN
is not able achieve good performance for this specific problem.

Table 3 shows the validation results for PPO algorithm with different
evaluation metrics (average of 5 validations with 100 episodes each). The
collision rate is considerably reduced and the average speed of the ego
vehicle is slightly increased. Note that this distance was already available
in the observation space among the properties of surrounding vehicles. Our
results show that explicitly adding this as a separate feature helps the agent
to learn better how to control the speed w.r.t. the leading vehicle and avoid
forward collisions.

(a) DQN

(b) A2C

(c) PPO

Figure 4: Comparison of episode rewards in baseline architecture with and without leading vehicle distance in state space. Table 3: Evaluation of baseline architecture using PPO with and without leading vehicle distance in state space. Evaluation Metric |  without leading distance |  with leading distance
---|---|---
Reached target successfully |  61% |  70.6%
Driven successfully, but not reached target within maximum steps |  0% |  0%
Terminated by collision or driving outside road |  39% |  29.4%
Average speed |  18.17 m/s |  19.43 m/s
Average distance travelled |  1503.93 m |  1667.86 m
Average executed steps |  88.43 |  89.65

###  3.2 Performance comparison of the two architectures

In this experiment, we compare the performance of our new architecture shown
in Fig. 3 with the baseline framework shown in Fig. 2 using different RL
algorithms. For both architectures, we use the observation space including
leading vehicle distance and the basic reward function. However, the new
architecture introduces a low level controller based on a physical model to
perform speed control actions. Fig. 5 shows the comparison of average episode
rewards in both architectures for different RL algorithms. It can be seen that
the new architecture outperforms the baseline regardless of the chosen RL
algorithm. Further, Fig. 6 compares performance of different RL agents in each
architecture. For both cases, DQN obtains lowest average rewards. PPO obtains
highest average reward in baseline architecture whereas both PPO and A2C shows
similar performance in new architecture.

As PPO gives consistent performance in both architectures, it is used for
further validation.The validation results in Table 4 clearly show that the new
architecture has improved the performance compared to the baseline. The
collision rate is reduced immensely from 29.4% to 1.6%. The slight decrease in
the average speed of the new framework is negligible when considering the
improvement in safety. The reduced collision rate also explains why there is
an increment in the average travelled distance and average executed steps.

(a) DQN

(b) A2C

(c) PPO

Figure 5: Comparison of average episodic reward in the baseline and new
architectures with different RL agents

(a) Baseline architecture

(b) New architecture

Figure 6: Performance comparison of different RL agents Table 4: Evaluation of baseline and new architectures using PPO. Evaluation Metric |  Baseline architecture |  New architecture
---|---|---
Reached target successfully |  70.6% |  97.8%
Driven successfully, but not reached the target within maximum steps |  0% |  0.6%
Terminated by collision or driving outside the road |  29.4% |  1.6%
Average speed |  19.43 m/s |  18.56 m/s
Average distance travelled |  1667.86 m |  2178.37 m
Average executed steps |  89.65 |  128.75

###  3.3 Performance with TCOP based reward

####  3.3.1 Comparison with different weight values

In this analysis, we apply the TCOP-based reward function in Equation 8 to the
new architecture in order to assess whether the agent can successfully acquire
a driving strategy that is both safe and cost-effective. The episodic reward
is evaluated with varying weights $W_{tar}$ assigned to the target completion
reward $R_{tar}$, as depicted in Fig. 7. As mentioned in Section II, in an
ideal scenario, the cost would amount to 2.314 euros, while the revenue
($R_{tar}$) would be 2.78 euros. Consequently, in this ideal case, the
episodic reward would equal $2.78-2.315=0.46$ when $W_{tar}$ is 1. This value
is relatively small, making it challenging for the agent to learn how to reach
the target, as illustrated in Fig. 7. Moreover, Table 5 demonstrates that the
agent exhibits slow driving behavior and struggles to reach the target.
However, it becomes evident that increasing the weight assists the agent in
learning to drive at higher speeds and successfully reach the target. Notably,
as the weight is increased, the total cost of operation—comprising energy cost
and driver cost—is minimized to 3.58 euros. This cost is lower than the cost
incurred when the ego vehicle was entirely controlled by SUMO, using the
default Krauss car following model and LC2013 lane change model in the same
traffic environment, which resulted in a cost of 3.85 euros.

Figure 7: Learning of TCOP-based rewards in the new architecture with different $W_{tar}$ values using PPO. Table 5: Evaluation of new architecture with TCOP based reward function, having different weights, $W_{tar}$. Evaluation Metric |  $W_{tar}=1$ |  $W_{tar}=10$ |  $W_{tar}=36$
---|---|---|---
Reached target successfully |  3.6% |  1% |  98.8%
Driven successfully, but not reached the target within maximum steps |  93.4% |  96.8% |  0.4%
Terminated by collision or driving outside road |  3% |  2.2% |  0.8%
Average speed |  2.39 m/s |  1.56 m/s |  18.24 m/s
Average distance travelled |  861.64 m |  580.65 m |  2187.62 m
Average executed steps |  477.02 |  486.11 |  126.46
Average energy cost |  0.26 euros |  0.16 euros |  1.82 euros
Average driver cost |  6.62 euros |  6.75 euros |  1.76 euros
Average tcop |  6.88 euros |  6.91 euros |  3.58 euros

####  3.3.2 Comparison with and without reward normalization in CRL and non-
CRL

In these experiments, instead of adding weights to the reward components, we
investigate whether the agent can learn an optimal policy with the modified
TCOP reward function as in Equation 11 that contains only the actual costs and
revenue values. We compare the performance with and without CRL approach using
PPO algorithm. PPO was chosen because it proved to give consistent performance
in earlier experiments. To improve exploration, we used an adaptive entropy
coefficient of 0.01 which gradually decreases to 0.001 whereas default values
in stable-baselines3 are used for all other hyperparameters.

Figure 8: Learning of TCOP based reward function without normalization using CRL and non-CRL approaches. Table 6: Evaluation of agent trained with TCOP based reward function without normalization using CRL and non-CRL approaches. Evaluation Metric |  Curriculum-1 |  Curriculum-2 |  Curriculum-3 |  Non-Curriculum
---|---|---|---|---
Reached target successfully |  44.80% |  10.35% |  7.15% |  2.50%
Terminated by collision or driving outside road |  13.45% |  8.80% |  9.60% |  6.20%
Driven successfully, but not reached the target within maximum steps |  41.75% |  80.85% |  83.25% |  91.30%
Average distance travelled |  1269.80 m |  837.52 m |  739.12 m |  603.66 m
Average executed steps |  265.91 |  431.43 |  438.64 |  463.48
Average speed |  11.38 m/s |  3.80 m/s |  2.99 m/s |  2.14 m/s
Average energy cost |  1.23 euros |  0.38 euros |  0.27 euros |  0.20 euros
Average driver cost |  3.7 euros |  6 euros |  6.1 euros |  6.44 euros
Average tcop |  4.93 euros |  6.38 euros |  6.37 euros |  6.64 euros
Figure 9: Learning of TCOP based reward function with normalization using CRL and non-CRL approaches. Table 7: Evaluation of agent trained with TCOP based reward function with normalization using CRL and non-CRL approaches. Evaluation Metric |  Curriculum-1 |  Curriculum-2 |  Curriculum-3 |  Non-Curriculum
---|---|---|---|---
Reached target successfully |  57.45% |  49.15% |  73.50% |  73.25%
Terminated by collision or driving outside road |  8.25% |  4.45% |  2.90% |  3.50%
Driven successfully, but not reached the target within maximum steps |  34.30% |  46.40% |  23.60% |  23.25%
Average distance travelled |  1569.99 m |  1382.20 m |  1794.74 m |  1770.03 m
Average executed steps |  252.24 |  294.00 |  210.22 |  206.94
Average speed |  12.27 m/s |  10.38 m/s |  14.18 m/s |  14.42 m/s
Average energy cost |  1.33 euros |  1.08 euros |  1.68 euros |  1.62 euros
Average driver cost |  3.50 euros |  4.08 euros |  2.92 euros |  2.88 euros
Average tcop |  4.83 euros |  5.17 euros |  4.60 euros |  4.50 euros

(a) CRL

(b) Non-CRL

Figure 10: Performance comparison with and without normalization in TCOP based rewards Table 8: Comparison of validation results with and without normalization in TCOP based rewards in CRL and non-CRL approaches. | CRL | Non-CRL
---|---|---
Evaluation Metric |  Without normalization |  With Normalization |  Without normalization |  With Normalization
Reached target successfully |  7.15% |  73.50% |  2.50% |  73.25%
Terminated by collision or driving outside road |  9.60% |  2.90% |  6.20% |  3.50%
Driven successfully, but not reached the target within maximum steps |  83.25% |  23.60% |  91.30% |  23.25%
Average distance travelled |  739.12 m |  1794.74 m |  603.66 m |  1770.03 m
Average executed steps |  438.64 |  210.22 |  463.48 |  206.94
Average speed |  2.99 m/s |  14.18 m/s |  2.14 m/s |  14.42 m/s
Average energy cost per meter |  0.00036 euros |  0.00094 euros |  0.00033 euros |  0.00092 euros
Average driver cost per meter |  0.0082 euros |  0.0016 euros |  0.0107 euros |  0.0016 euros
Average tcop per meter |  0.0086 euros |  0.0026 euros |  0.011 euros |  0.0025 euros

Figure 8 shows the episodic reward training curve for the RL agent trained
with curriculum learning and compares it to the training curve without using
curriculum learning. Here, the agent is trained with curriculum learning using
the reward functions without normalization given in Equation 12 where
curriculum-1 is trained for $7e^{5}$ time steps and curriculum-2 and
curriculum-3 are trained for $5e^{5}$ time steps each. Non-curriculum learning
results are obtained from RL agent trained with the reward function Equation
11 for $17e^{5}$ time steps. There are noticeable drops in the reward curve at
certain points during training in both curriculum and non-curriculum learning.
The validation results in Table 6 also show a poor success rate. Here, the
evaluation is performed by averaging the validation results from 5 trained
models for curriculum and non-curriculum approaches.

It can be seen that the average speed of the ego vehicle drops down largely
after training with curriculum-2 when the penalty for energy cost is
introduced, which also leads to a reduction in the success rate. This
motivates us to tune the reward function by normalizing driver cost and energy
cost by distance travelled per time step. Figure 9 and Table 7 show the
results of the agent trained with this normalized reward function in Equation
13 and corresponding non-curriculum learning. Here, the reduction in speed
from curriculum-1 to curriculum-2 is smaller and this reduction is recovered
when the reward for target completion is added to the next curriculum.
Consequently, the success rate is higher in last curriculum compared to the
previous case. Figure 10 compares the training curves with and without reward
normalization where we could observe that the average reward is comparatively
higher with normalization in both CRL and non-CRL approaches. The training
curve with normalization also shows a more stable and steady improvement in
rewards, even though the drop in the last curriculum persists. Overall, the
normalized reward function significantly improves the results for both CRL and
non-CRL approaches as shown explicitly in Table 8. We could observe
significant improvement in success rate, average speed and TCOP. However, the
results obtained for curriculum and non-curriculum approaches are very similar
which demonstrates that there is no particular advantage in using curriculum
learning in this RL setting.

##  4 Conclusion

We implemented an RL framework for tactical decision decision making of
autonomous trucks in a highway environment by integrating RL with low-level
controllers. Our results demonstrate that training the agent to focus on high
level decisions, such as maintaining a time gap with the leading vehicle,
while leaving the low level speed control to a physics-based controller,
accelerates learning and improves overall performance. Additionally, we
explored the effectiveness of training the agent with realistic rewards and
penalties using a multi-objective TCOP based reward function. We study this
setting with different approaches, by adding weights to reward components, by
normalizing the reward components and by using curriculum learning techniques.
We could observe that reshaping the reward function with weights or
normalization significantly improves the performance whereas CRL shows
comparable results with non-CRL approach.

## Acknowledgment

This work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. We
would like to thank Nikolce Murgovski, Erik Börve and Stefan Börjesson for
valuable discussions.

## Declarations

##### Confict of interest

There is no confict of interest.

##### Ethics approval

Not applicable.

##### Consent to participate

Not applicable.

##### Consent for publication

Not applicable.

##### Code availability

The source code is available at:
https://github.com/deepthi-pathare/Autonomous-truck-sumo-gym-env

## References

  *   * De Jong et al. [2004] De Jong, G., Gunn, H., Walker, W.: National and international freight transport models: an overview and ideas for future development. Transport Reviews 24 (2004)
  * Moreno et al. [2018] Moreno, G., Nicolazzi, L.C., Vieira, R.D.S., Martins, D.: Stability of long combination vehicles. International journal of heavy vehicle systems 25 (2018)
  * Yang et al. [2015] Yang, D., Qiu, X., Yu, D., Sun, R., Pu, Y.: A cellular automata model for car–truck heterogeneous traffic flow considering the car–truck following combination effect. Physica A: Statistical Mechanics and its Applications 424 (2015)
  * Nilsson et al. [2018] Nilsson, P., Laine, L., Sandin, J., Jacobson, B., Eriksson, O.: On actions of long combination vehicle drivers prior to lane changes in dense highway traffic – a driving simulator study. Transportation Research Part F: Traffic Psychology and Behaviour 55 (2018)
  * Grislis [2010] Grislis, A.: Longer combination vehicles and road safety. Transport 25 (2010)
  * Shaout et al. [2011] Shaout, A., Colella, D., Awad, S.: Advanced driver assistance systems-past, present and future. In: International Computer Engineering Conference (2011). IEEE
  * Jiménez et al. [2016] Jiménez, F., Naranjo, J.E., Anaya, J.J., García, F., Ponz, A., Armingol, J.M.: Advanced driver assistance system for road environments to improve safety and efficiency. Transportation research procedia 14 (2016)
  * Xiao and Gao [2010] Xiao, L., Gao, F.: A comprehensive review of the development of adaptive cruise control systems. Vehicle system dynamics (2010)
  * Sutton and Barto [2018] Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. (2018)
  * Sallab et al. [2017] Sallab, A.E., Abdou, M., Perot, E., Yogamani, S.: Deep reinforcement learning framework for autonomous driving. arXiv preprint arXiv:1704.02532 (2017)
  * Shalev-Shwartz et al. [2016] Shalev-Shwartz, S., Shammah, S., Shashua, A.: Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 (2016)
  * Kiran et al. [2021] Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A.A., Yogamani, S., Pérez, P.: Deep reinforcement learning for autonomous driving: A survey. Transactions on Intelligent Transportation Systems 23 (2021)
  * Hoel et al. [2020] Hoel, C.-J., Wolff, K., Laine, L.: Tactical decision-making in autonomous driving by reinforcement learning with uncertainty estimation. In: Intelligent Vehicles Symposium (2020). IEEE
  * Desjardins and Chaib-draa [2011] Desjardins, C., Chaib-draa, B.: Cooperative adaptive cruise control: A reinforcement learning approach. IEEE Transactions on Intelligent Transportation Systems 12 (2011)
  * Zhao et al. [2013] Zhao, D., Wang, B., Liu, D.: A supervised actor–critic approach for adaptive cruise control. Soft Computing 17 (2013)
  * Konda and Tsitsiklis [1999] Konda, V., Tsitsiklis, J.: Actor-critic algorithms. In: Solla, S., Leen, T., Müller, K. (eds.) Advances in Neural Information Processing Systems (1999)
  * Haarnoja et al. [2018] Haarnoja, T., Zhou, A., Abbeel, P., Levine, S.: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018)
  * Lin et al. [2021] Lin, Y., McPhee, J., Azad, N.L.: Comparison of deep reinforcement learning and model predictive control for adaptive cruise control. IEEE Transactions on Intelligent Vehicles 6 (2021)
  * Das and Won [2021] Das, L.C., Won, M.: Saint-acc: Safety-aware intelligent adaptive cruise control for autonomous vehicles using deep reinforcement learning. In: ICML (2021)
  * Yang et al. [2020] Yang, J., Liu, X., Liu, S., Chu, D., Lu, L., Wu, C.: Longitudinal tracking control of vehicle platooning using ddpg-based pid. In: 2020 4th CAA International Conference on Vehicular Control and Intelligence (CVCI) (2020)
  * Hoel [2020] Hoel, C.-J.E.: Source code for ‘Tactical decision-making in autonomous driving by reinforcement learning with uncertainty estimation’ (2020). https://github.com/carljohanhoel/BayesianRLForAutonomousDriving
  * Mnih et al. [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M.A., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)
  * Mnih et al. [2016] Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: Balcan, M.F., Weinberger, K.Q. (eds.) Proceedings of The 33rd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 48, pp. 1928–1937. PMLR, New York, New York, USA (2016). https://proceedings.mlr.press/v48/mniha16.html
  * Schulman et al. [2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal Policy Optimization Algorithms (2017)
  * Pathare et al. [2023] Pathare, D., Laine, L., Chehreghani, M.H.: Improved tactical decision making and control architecture for autonomous truck in sumo using reinforcement learning. In: 2023 IEEE International Conference on Big Data (BigData), pp. 5321–5329 (2023). https://doi.org/10.1109/BigData59044.2023.10386803
  * Bengio et al. [2009] Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48 (2009)
  * Narvekar et al. [2020] Narvekar, S., Peng, B., Leonetti, M., Sinapov, J., Taylor, M.E., Stone, P.: Curriculum learning for reinforcement learning domains: A framework and survey. The Journal of Machine Learning Research 21(1), 7382–7431 (2020)
  * Anzalone et al. [2022] Anzalone, L., Barra, P., Barra, S., Castiglione, A., Nappi, M.: An end-to-end curriculum learning approach for autonomous driving scenarios. IEEE Transactions on Intelligent Transportation Systems 23(10), 19817–19826 (2022) https://doi.org/10.1109/TITS.2022.3160673
  * Liu et al. [2023] Liu, J., Li, H., Yang, Z., Dang, S., Huang, Z.: Deep dense network-based curriculum reinforcement learning for high-speed overtaking. IEEE Intelligent Transportation Systems Magazine 15(1), 453–466 (2023) https://doi.org/10.1109/MITS.2022.3174410
  * Song et al. [2021] Song, Y., Lin, H., Kaufmann, E., Durr, P., Scaramuzza, D.: Autonomous overtaking in gran turismo sport using curriculum reinforcement learning, pp. 9403–9409 (2021). https://doi.org/10.1109/ICRA48506.2021.9561049
  * Lopez et al. [2018] Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flötteröd, Y.-P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P., Wießner, E.: Microscopic traffic simulation using sumo. In: International Conference on Intelligent Transportation Systems (2018)
  * Krauss et al. [1997] Krauss, S., Wagner, P., Gawron, C.: Metastable states in a microscopic model of traffic flow. Phys. Rev. E 55 (1997)
  * Erdmann [2014] Erdmann, J.: Lane-changing model in sumo. In: Proceedings of the SUMO2014 Modeling Mobility with Open Data. Reports of the DLR-Institute of Transportation SystemsProceedings, vol. 24. Deutsches Zentrum für Luft- und Raumfahrt e.V., ??? (2014)
  * Raffin et al. [2021] Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research 22 (2021)
  * Svensson et al. [2023] Svensson, H.G., Tyrchan, C., Engkvist, O., Chehreghani, M.H.: Utilizing Reinforcement Learning for de novo Drug Design (2023)
  * Treiber et al. [2000] Treiber, M., Hennecke, A., Helbing, D.: Congested traffic states in empirical observations and microscopic simulations. Physical review E 62(2), 1805 (2000)


# Graded components of local cohomology modules of $\mathfrak{C}$-monomial
ideals in characteristic zero

Tony J. Puthenpurakal  Department of Mathematics, Indian Institute of
Technology Bombay, Powai, Mumbai 400076, India<EMAIL_ADDRESS>and
Sudeshna Roy  School of Mathematics, Tata Institute of Fundamental Research,
Homi Bhabha Road, Mumbai 400005, India<EMAIL_ADDRESS>
###### Abstract.

Let $A$ be a commutative Noetherian ring of characteristic zero and
$R=A[X_{1},\ldots,X_{d}]$ be a polynomial ring over $A$ with the standard
$\mathbb{N}^{d}$-grading. Let $I\subseteq R$ be an ideal which can be
generated by elements of the form $aU$ where $a\in A$ (possibly nonunit) and
$U$ is a monomial in $X_{i}$’s. We call such an ideal as a
‘$\mathfrak{C}$-monomial ideal’. Local cohomology modules supported on
monomial ideals gain a great deal of interest due to their applications in the
context of toric varieties. It was observed that for
$\underline{u}\in\mathbb{Z}^{d}$, their $\underline{u}^{th}$ components depend
only on which coordinates of $\underline{u}$ are negative. In this article, we
show that this statement holds true in our general setting, even for certain
invariants of the components. We mainly focus on the Bass numbers, injective
dimensions, dimensions, associated primes, Bernstein-type dimensions, and
multiplicities of the components. Under the extra assumption that $A$ is
regular, we describe the finiteness of Bass numbers of each component and
bound its injective dimension by the dimension of its support. Finally, we
present a structure theorem for the components when $A$ is the ring of formal
power series in one variable over a characteristic zero field.

2010 Mathematics Subject Classification: Primary 13D45; Secondary 13N10

Key words: Local cohomology, multigraded local cohomology, Weyl algebra,
multigraded generalized Eulerian module, monomial ideal,
$\mathfrak{C}$-monomial ideal

##  1\. Introduction

Let $K$ be a field and $S=K[X_{1},\ldots,X_{d}]$ be a polynomial ring over
$K$. In recent years explicit information about the structure of local
cohomology modules $H^{j}_{J}(S)$ supported on monomial ideals $J$ has been
obtained based on the following two facts:

  1. (i)

they have a structure of $\mathbb{Z}^{d}$-graded modules which allows one to
study these modules using combinatorial approach, see [Mus], [Ter], [Yan2001].

  2. (ii)

they have a structure of ${\mathcal{D}}$-modules, see [Alv], [AlvZar],
[AlvGarZar].

In fact, there is a dictionary between the $\mathbb{Z}^{d}$-graded structure
and the ${\mathcal{D}}$-module structure of $H^{j}_{J}(S)$, see [JAM_LC,
III.2.9]. These modules are important for their connection to the cohomology
of coherent sheaves on a toric variety over a field (see [EisMusSti]) and
useful for constructing examples and non-examples. However, it is long known
that every local cohomology is associated to certain sheaf cohomology, see
[24Hrs, Theorem 12.41]. The goal of this article is to analyze the
$\mathbb{Z}^{d}$-graded structure of local cohomology modules in the following
framework:

######  1.1.

Setup. Let $A$ be any commutative Noetherian ring containing a field of
characteristic zero and $R=A[X_{1},\ldots,X_{d}]$ be a polynomial ring over
$A$. Consider an $\mathbb{N}^{d}$-grading on $R$ by setting $\deg z=0$ for all
$z\in A$ and $\deg X_{i}=e_{i}\in\mathbb{N}^{d}$ for $i=1,\ldots,d$. Let
$I\subseteq R$ be a $\mathfrak{C}$-monomial ideal, that is, $I$ can be
generated by elements of the form $aU$ where $a\in A$ (possibly nonunits) and
$U$ is a monomial in $X_{1},\ldots,X_{d}$.

It is well-known that the local cohomology modules are rarely finitely
generated. The prime way to tackle local cohomology modules supported on
monomial ideals is to restore finiteness by considering their components under
suitable grading. In our setting, although $H^{i}_{I}(R)$ has an induced
$\mathbb{Z}^{d}$-graded structure for each $i\geq 0$, its components are not
necessarily finitely generated, see Theorem LABEL:non-fg. So we assume that
the base ring has characteristic zero and use the techniques of D-modules,
which was pioneered by G. Lyubeznik in [Lyu93]. For this study, we introduce
the notion of $\mathbb{Z}^{d}$-graded generalized Eulerian
$A_{\underline{1}}(A)$-module (Definition LABEL:MGE). This is motivated from
the authors’ previous analysis in [Put2015], [Put], [PutRoy22]. Previously,
researchers realized that to understand the $\mathbb{Z}^{d}$-graded structure
of $H^{j}_{J}(S)$ one only needs to study their $\underline{a}$-th pieces for
$\underline{a}\in\\{-1,0\\}^{d}$. This observation is reflected in our results
regarding Bass numbers (Theorem LABEL:multi-bass), injective dimensions
(Theorem LABEL:multi-injdim-dim) and associated primes (Theorem LABEL:multi-
ass) of components of $H^{i}_{I}(R)$. Moreover, under the extra assumption
that $A$ is a regular ring, in Theorem LABEL:injdim-dim we prove that

  1. (i)

all Bass numbers of $H^{i}_{I}(R)_{\underline{u}}$ are finite for every
$\underline{u}\in\mathbb{Z}^{d}$.

  2. (ii)

$\operatorname{injdim}H^{i}_{I}(R)_{\underline{u}}\leq\dim\operatorname{supp}H^{i}_{I}(R)_{\underline{u}}$
for fix $\underline{u}\in\mathbb{Z}^{d}$.

We further show that if $I$ is a usual monomial ideal, then
$H^{i}_{I}(R)_{\underline{u}}$ is a free $A$-module. We call an ideal a _usual
monomial ideal_ if it is generated by monomials in $X_{1},\ldots,X_{d}$.

Let $K$ be a field of characteristic zero. For the case when $A=K[[Y]]$ we
establish a structure theorem (Theorem LABEL:struc) for the components of
$H^{i}_{I}(R)$ illustrating their torsion parts and torsion-free parts. It
appears that their behavior is almost identical to the mixed characteristic
case which was studied by the authors in [PutRoy22_pre].

We now assume that $A=K[[Y_{1},\ldots,Y_{m}]]$. When $m=0$, it follows from
[Yan2001, Theorem 2·11] that the multiplicity of the
$\underline{u}^{th}$-component of $H^{i}_{I}(R)$ depends only on which
coordinates of $\underline{u}$ are negative. This motivates us to investigate
the asymptotic behavior of the multiplicities of the components in the
generality. Let ${\mathcal{D}}$ be the ring of $A$-linear differential
operators on $R$ and let $L$ be a finitely generated $\mathbb{Z}^{d}$-graded
${\mathcal{D}}$-module. We define a $\mathbb{Z}^{d}$-graded Bernstein type
good filtration $\mathcal{F}$ on ${\mathcal{D}}$ (see LABEL:D-fil) and a
$\mathbb{Z}^{d}$-graded $\mathcal{F}$-compatible good filtration $\Gamma$ on
$L$ (see LABEL:comp_fil). We show that for each
$\underline{u}\in\mathbb{Z}^{d}$ there is an induced filtration
$\Gamma_{\underline{u},*}$ on $M_{\underline{u}}$ such that the associated
graded module
$\operatorname{gr}_{\Gamma_{\underline{u},*}}(M_{\underline{u}})$ is a
finitely generated
$\operatorname{gr}_{\mathcal{F}_{\underline{0},*}}({\mathcal{D}}_{\underline{0}})$-module.
We observe that
$\operatorname{gr}_{\mathcal{F}_{\underline{0},*}}({\mathcal{D}}_{\underline{0}})$
is a polynomial ring over $A$ in $(m+d)$-variables, see
(LABEL:polynomialRing). It allows us to define Bernstein type dimension and
multiplicity of $M_{\underline{u}}$ (see LABEL:dim-mul) as in [Bjo, Definition
6.3, Chapter 2]. Similar to other invariants, we prove that the Bernstein type
dimension and the multiplicity of $H^{i}_{I}(R)_{\underline{u}}$, for
$\underline{u}\in\mathbb{Z}^{d}$, depend only on which coordinates of
$\underline{u}$ are negative (Theorem LABEL:multi-Bdim-mul).

This article is structured as follows. In Section 2, we introduce and discuss
certain properties of multi-graded generalized Eulerian modules. In Section 3,
we establish their rigidity property. We connect this class to the class of
straight modules due to K. Yanagawa in Section 4. Section 5 is devoted to
study some finiteness properties and asymptotic behaviour of specific
invariants of the components of $H^{i}_{I}(R)$ under the hypotheses in 1.1.
Suppose that ${\mathcal{D}}$ denotes the ring of $A$-linear differential
operators on $R$. Notice there is a natural induced $\mathbb{Z}^{d}$-grading
on ${\mathcal{D}}$. In Section 6, we develop a Bernstein-type theory for the
components of finitely generated $\mathbb{Z}^{d}$-graded
${\mathcal{D}}$-modules when $A=K[[Y_{1},\ldots,Y_{m}]]$ is a power series
ring over a field $K$ of characteristic zero. In Section 7, we produce a
structure theorem for the components of $H^{i}_{I}(R)$ when $A=K[[Y]]$.
Finally, in Section 8, we give examples to show that both torsion parts and
torsion-free parts of the components can be non-finitely generated.

Throughout this article, we use $\mathcal{S}$ to denote
$[d]:=\\{1,\ldots,d\\}$.

##  2\. Multi-graded generalized Eulerian