# Fair Credit Scorer through Bayesian Approach

Zhuo Zhao
Department of Applied Mathematics
Johns Hopkins University
<EMAIL_ADDRESS>

###### Abstract

Machine learning currently plays an increasingly important role in people’s
lives in areas such as credit scoring, auto-driving, disease diagnosing, and
insurance quoting. However, in many of these areas, machine learning models
have performed unfair behaviors against some sub-populations, such as some
particular groups of race, sex, and age. These unfair behaviors can be on
account of the pre-existing bias in the training dataset due to historical and
social factors. In this paper, we focus on a real-world application of credit
scoring and construct a fair prediction model by introducing latent variables
to remove the correlation between protected attributes, such as sex and age,
with the observable feature inputs, including house and job. For detailed
implementation, we apply Bayesian approaches, including the Markov Chain Monte
Carlo simulation, to estimate our proposed fair model.

##  1 Introduction

Nowadays, Machine Learning methods are used to automate decisions in a variety
of areas, including determining credit scores Nanni and Lumini (2009),
classifying tumor components from MRI images Lundervold and Lundervold (2019),
detecting pedestrians on the road Dollar et al. (2011), and understanding
natural languages Goldberg and Levy (2014), etc. However, machine learning
methods are heavily dependent on data Mitchell and Mitchell (1997) and this
data-dependent nature makes the learned models sensitive to the latent bias
existing in the training datasets Mehrabi et al. (2021). Thus, the final
decisions made by the learned models are unfairly biased against certain sub-
populations, differentiated by some sensitive/protected attributes, such as
race, sex, or age, etc. For example, cameras sometimes fail to recognize
whether Asian blink their eyes Sharp (2009) and the beauty pageant judged by
AI would prefer light skin Guardian (2016). However, we would expect AI to
give the same decision independent from the protected attributes and thus we
concern about the fairness of machine learning methods Mehrabi et al. (2021).

In this paper, we focus on constructing fair machine learning models to
predict the credit score, with using the German Credit Risk dataset Hoffman
(2016) (Sec. 3). The goal is to predict the credit score based on some
observable variables, including housing and job information. However, this
personal financial information, such as income, housing, and saving, are
usually highly correlated to gender and age due to historical and social
reasons Rennison and Planty (2003). Therefore, it is necessary to learn an
effective model to filter the prediction bias against sex and age, caused by
the latent correlation between these observable variables and the protected
attributes. In detail, we analyze and compare from the fairness perspective
across the full model Montgomery et al. (2021), unaware model Dwork et al.
(2012), and fair model based on causals and counterfactuals Kusner et al.
(2017) (Sec. 4). Then, we apply the Markov Chain Monte Carlo (MCMC) simulation
Mooney (1997) and the Gibbs’ sampling Gelfand (2000) to solve the
corresponding parameters in these models and evaluate the performances (Sec. 5
and Sec. 6).

##  2 Related work

Fairness. Many recent works (Calders and Verwer (2010); Bolukbasi et al.
(2016); Dwork et al. (2012); Hardt et al. (2016); Joseph et al. (2016); Kusner
et al. (2017)) have been focusing on fairness in machine learning algorithms.
Bolukbasi et al. (2016) pointed out that there is a risk of amplifying the
bias introduced from the dataset, if using machine learning algorithms without
taking effects to handle the pre-existing bias. For example, in the word
embedding, learned over Google News with pre-existing gender stereotypes, the
gender-neutral words widely spread along a latent embedding direction
capturing gender difference, such as "receptionist" falling far along the
direction related to "female" Bolukbasi et al. (2016). Calders and Verwer
(2010) modifies the Naive Bayes classifier by adding independence restriction
toward sensitive attributes. Dwork et al. (2012) proposes a task-specific
metric to evaluate the similarity between individuals relative to the
classification task and optimizes over the proposed metric with the goal that
similar individuals are treated similarly in the classification task. Kusner
et al. (2017) focuses on causal inference and counterfactual, with introducing
the latent confounding variables, which are related to the observable
variables but independent from the protected attributes. Our work is based on
the Kusner et al. (2017) idea to construct a fair prediction model over the
German Credit Risk dataset Hoffman (2016).

##  3 Dataset

We consider the Kaggle German Credit Risk dataset Hoffman (2016) to analyze
and compare different types of unfair models and our method for constructing a
fair model using Bayesian approaches. In this dataset, each entry represents a
person who takes credit from a bank. The objective is to predict the credit
amount of a person based on his/her attributes. "Sex" and "age" are the
sensitive/protected attributes related to the bias during training and
prediction in the unfairness problem. Feature "job" is a binary variable
representing whether a person has a job or not. Feature "house" is a binary
variable that indicates whether or not a person owns a house. The "credit
amount" is our prediction target.

The dataset is composed of 1000 records. We randomly pick 800 records for
training and 200 records for testing. Figure 1 shows the detailed
distributions of all these features in the whole dataset. In Figure 2, we
illustrate the covariance between all the input features and the prediction
target. We can observe a high correlation from the sensitive / protected
attributes, i.e. "sex" and "age", to the "job" and "house". Thus, it is
necessary to consider the issue of fairness when constructing a prediction
model over "job" and "house".

Figure 1: Distribution of features in the German Credit Risk dataset Hoffman
(2016). "Age" and "sex" are the sensitive / protected attributes. "Job" and
"house" are the observable variables. "Credit amount" is the prediction
target. Figure 2: Illustration of the covariance matrix between all the input
features and the prediction target. Here, we observe a high correlation from
"age" and "sex" (the sensitive/protected attributes) to "job" and "house" (the
observable variables).

##  4 Methods

Full Model: The full model Montgomery et al. (2021) completely ignores
fairness issues and includes sensitive variables like sex and age in the
learning process. It is easy to understand that the full model is unfair
because the predictions depend on sex and age. Figure 3 presents the directed
acyclic graph (DAG) of the full model. In the full model, all the features are
assumed to be connected.

Unaware Model: The unaware model Dwork et al. (2012) does not use sensitive
variables in the learning and prediction process, but it is still unfair. Even
though the sensitive variables do not influence the target directly in the
learning and prediction processes, it still has an indirect impact on the
target through the non-sensitive variables. In our example, to predict a
person’s credit amount, sex may influence whether a person can get a job. The
job attribute still preserves the information of sex. Simply ignoring the sex
attribute will not fully eliminate its impact on the predictions. Figure 3
presents the DAG of an unaware model. The attributes under the grey circles
are unobserved. In the unaware model, sex and age are not directly connected
with the credit amount, but they are connected with job and house. It is still
unfair because the change of sex and age will change the status of job and
house, and thus influence the credit amount predictions.

Figure 3: Two types of unfair models. Left: full model, which builds
regression over all possible attributes without the consideration of fairness.
Right: unaware model, which excludes sensitive/protected attributes, i.e. sex
and age in our case.

Fair Model:  In order to build a fair model, we need to find a proxy variable
that is independent of sensitive variables but still preserves the information
in the credit amount prediction Kusner et al. (2017). We can introduce the
concept of latent confounding variable to resolve this issue. The confounding
variable is a variable that influences both the independent variable and
dependent variables. In our fair model, we assume that there is an unobserved
confounder $C$ that reflects how reliable a person is in paying back the loan.
The confounder should be independent of the sensitive variables to make the
model fair. Figure 4 shows the DAG of the fair model structure. In the
inference stage, we assume that job, house, and credit amount are confounded
by the unobserved reliability level $C$ and $C$ is independent of sex and age.
The reason is that sex and age can neither determine nor be related to how
reliable a person is in paying back loans. Meanwhile, reliability is co-
related to a person’s job performance, housing situation, and also credit
amount. Then, in the prediction stage, we only use the inferred $C$ as our
feature to predict the credit amount. In this way, the predicting process does
not contain any information about sex or age, and thus this procedure is an
effective, fair learning algorithm in our scenario.

Figure 4: DAG of the fair model. Here, we introduce the latent confounding
variable "unobserved reliability level", which is independent to "sex" and
"age" (the sensitive/protected attributes) but related to "job", "house", and
"credit amount". Left: during the inference stage, we estimate this latent
"reliability" feature with Bayesian approaches. Right: during the prediction
stage, we only use this inferred "reliability" feature to predict the "credit
amount".

##  5 Experiments

We can represent the DAG of the fair model in a probabilistic way. We sample
the two binary variables, job and house from two Bernoulli distributions and
sample the confounder from the normal distribution. In the meantime, we choose
the Poisson distribution as a prior for the credit amount. Our choices of
priors correspond to the nature of the data. The job and house features are
binary. And the credit amount is a positive attribute with a shape alike the
Poisson distribution. The probabilistic model can be written as:

$Job\sim
Bernoulli(logit(b_{j}+Sex\times\beta_{j,s}+Age\times\beta_{j,a}+C\times\beta_{j,c}))$
(1) $House\sim
Bernoulli(logit(b_{h}+Sex\times\beta_{h,s}+Age\times\beta_{h,a}+C\times\beta_{h,c}))$
(2) $Credit\sim
Poisson(Exp(Sex\times\beta_{c,s}+Age\times\beta_{c,a}+C\times\beta_{c,c}))$
(3) ${C\sim Normal(0,1)}\text{\quad where \quad}C\perp Sex,\text{\quad}C\perp
Age$ (4)

The parameters we need to find are in the set $\Theta=\\{\beta_{m,n},b_{m}\\}$
where $m=j,h$ and $n=s,a,c$. We assume that these parameters are sampled from
the normal distributions:

$\beta_{m,n}\sim N(0,1)$ (5) $b_{m}\sim N(0,1)$ (6)

We implement the Metropolis–Hastings algorithm to infer the probabilistic
model. M-H algorithm Hastings (1970) is a Markov Chain Monte Carlo (MCMC)
method for obtaining a sequence of random samples from a probability
distribution from which direct sampling is difficult. Algorithm 1 explains how
to infer the reliability level $C$.

Algorithm 1 Infer C by Metropolis–Hastings

for $i=1$ to $N$ do

Choose
$J(C_{i}^{*}|C_{i}^{(s)})=uniform(C_{i}^{(s)}-\delta,C_{i}^{(s)}+\delta)$;

Set an initial state $C_{i}^{0}$;

for $s=1$ to $5000$ do

Sample $C_{i}^{*}\sim J(C_{i}^{*}|C_{i}^{(s)})$;

Compute the acceptance ratio $r=\frac{p(C_{i}^{*}\mid y)}{p(C_{i}^{(s)}\mid
y)}=\frac{p(y\mid C_{i}^{*})p(C_{i}^{*})}{p(y\mid
C_{i}^{(s)})p(C_{i}^{(s)})}$;

sample $u\sim uniform(0,1)$;

if u < r then;

$C_{i}^{(s+1)}=C_{i}^{*}$;

else

$C_{i}^{(s+1)}=C_{i}^{(s)}$;

end if

end for

end for

Once we obtain the posteriors of the inferred reliability level $C$, we can
fit a new model using kernel $g(.)$ based on the $C$ in the prediction stage.
In our experiment, since there is a nonlinear relationship between credit
amount and "Reliability Level" in our inference stage setup (Poisson), we
decide to use random-forest as the kernel function $g(.)$ in our second stage
prediction.

$Credit\sim g(C)$ (7)

##  6 Results

In this section, we provide experimental results and a discussion of the MCMC
process performance. Specifically, in Sec. 6.1, we firstly present the MCMC
estimation result and the convergence analysis on the fair model’s latent
confounding variable $C$ and parameters. Then, we compare the prediction and
fairness performance across the three types of models in Sec. 6.2.

###  6.1 Fair model’s MCMC performance:

Figure 5: Auto-correlation plots of parameters in Eq. 1 and Eq. 2 throughout
the MCMC process. "alpha" refers to the constant offset term $b$ in the
equations. Figure 6: Posterior estimation (left column) and trace plot (right
column) of parameters in Eq. 1 and Eq. 2 throughout the MCMC process. "alpha"
refers to the constant offset term $b$ in the equations.

In Figure 5, we illustrate the auto-correlation plot of the model’s parameters
in Eq. 1 and Eq. 2. We observe a clear decrease in auto-correlation throughout
the MCMC process. Thus, this is an efficient MCMC process that leads to
convergence. Further, in Figure 6, we provide the posterior estimation and the
trace plot of the fair model parameters throughout the MCMC process. Though we
still observe some fluctuations till the end of the process, however, this is
reasonable and acceptable. The reason is that we are applying over a real-
world dataset, rather than a simulated dataset. Therefore, it is impossible to
make our assumed distributions perfectly capture the behavior of the real-
world dataset. Then, in Table 1, we provide the confidence interval over the
posterior estimation of the fair model’s parameters.

| std | 5% | median | 95% | ess_bulk | ess_tail
---|---|---|---|---|---|---
$b_{j}$ | 1.02 | -1.66 | 0.03 | 1.71 | 4643.63 | 3709.55
$\beta_{j,s}$ | 0.98 | -1.32 | 0.27 | 1.88 | 7128.36 | 3907.04
$\beta_{j,a}$ | 0.65 | -2.64 | -1.57 | -0.50 | 1502.60 | 2245.58
$\beta_{j,c}$ | 0.47 | 2.82 | 3.46 | 4.36 | 2058.64 | 2494.02
$b_{h}$ | 1.01 | -1.61 | 0.03 | 1.67 | 5113.35 | 3167.94
$\beta_{h,s}$ | 0.99 | -1.73 | -0.11 | 1.55 | 5506.02 | 3900.82
$\beta_{h,a}$ | 0.67 | -0.04 | 1.05 | 2.17 | 1625.86 | 2583.99
$\beta_{h,c}$ | 0.46 | 3.00 | 3.65 | 4.50 | 1896.31 | 2939.44
$\beta_{c,s}$ | 0.54 | -7.78 | -6.85 | -5.98 | 3255.68 | 3206.54
$\beta_{c,a}$ | 0.52 | -3.17 | -2.26 | -1.46 | 4326.49 | 3800.31
$\beta_{c,c}$ | 0.23 | -0.37 | 0.01 | 0.38 | 4455.72 | 3211.78
Table 1: The confidence intervals of the parameters estimated in Eq. 1 and Eq.
2 through the MCMC process.

###  6.2 Performance comparison across models:

In this section, we compare how three distinct models perform while making
predictions. In Table 2, we present the $R^{2}$ of three models in both
training and testing environments. The full model outperforms the unaware
model in both fitting and predicting by including sensitive information. It is
surprising to see that the fair model outperforms the other two unfair models
with $R^{2}=0.801$ in the training set and $R^{2}=0.768$ in the testing set.
It turns out that our fair model does not only resolve the fairness issue but
distills the information on the reliability level. The fair model is robust
enough to be used to make fair and accurate predictions.

$R^{2}$ | Full Model | Unaware Model | Fair Model Random Forest Kernel
---|---|---|---
Training | 0.597 | 0.466 | 0.801
Testing | 0.521 | 0.424 | 0.768
Table 2: The $R^{2}$ of three types of models defined in Sec. 4.

##  7 Conclusion

In this paper, we have presented a fair model focusing on predicting the
German credit score with considering the job and housing features.
Specifically, we introduce the latent confounding variable "reliability
level", which is independent of the protected attributes, i.e., "sex" and
"age", but related to other observable variables and the prediction goal. For
implementation, we apply the MCMC approach to solve for the latent confounding
variable and the parameters of the model. Compared with tradition models, our
model effectively eliminates the bias related to sex and age and thus achieves
a fair prediction of the credit amount. For the future work, we recommend
trying different types of assumptions on the distribution for the variables
over the German Credit Risk dataset and checking the effects on the choice of
distributions over the convergence of the MCMC process and the final
prediction.

## References

  * Bolukbasi et al. [2016] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in neural information processing systems_ , 29, 2016.
  * Calders and Verwer [2010] Toon Calders and Sicco Verwer.  Three naive bayes approaches for discrimination-free classification.  _Data mining and knowledge discovery_ , 21(2):277–292, 2010.
  * Dollar et al. [2011] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona.  Pedestrian detection: An evaluation of the state of the art.  _IEEE transactions on pattern analysis and machine intelligence_ , 34(4):743–761, 2011.
  * Dwork et al. [2012] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In _Proceedings of the 3rd innovations in theoretical computer science conference_ , pages 214–226, 2012.
  * Gelfand [2000] Alan E Gelfand.  Gibbs sampling.  _Journal of the American statistical Association_ , 95(452):1300–1304, 2000.
  * Goldberg and Levy [2014] Yoav Goldberg and Omer Levy.  word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method.  _arXiv preprint arXiv:1402.3722_ , 2014.
  * Guardian [2016] The Guardian.  A beauty contest was judged by ai and the robots didn’t like dark skin, Sep 2016.  URL https://www.theguardian.com/technology/2016/sep/08/artificial-intelligence-beauty-contest-doesnt-like-black-people.
  * Hardt et al. [2016] Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Hastings [1970] W Keith Hastings.  Monte carlo sampling methods using markov chains and their applications.  1970\.
  * Hoffman [2016] Donald Hoffman.  German credit risk, Dec 2016.  URL https://www.kaggle.com/datasets/uciml/german-credit.
  * Joseph et al. [2016] Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth.  Rawlsian fairness for machine learning.  _arXiv preprint arXiv:1610.09559_ , 1(2):19, 2016\.
  * Kusner et al. [2017] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva.  Counterfactual fairness.  _Advances in neural information processing systems_ , 30, 2017.
  * Lundervold and Lundervold [2019] Alexander Selvikvåg Lundervold and Arvid Lundervold.  An overview of deep learning in medical imaging focusing on mri.  _Zeitschrift für Medizinische Physik_ , 29(2):102–127, 2019.
  * Mehrabi et al. [2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54(6):1–35, 2021\.
  * Mitchell and Mitchell [1997] Tom M Mitchell and Tom M Mitchell.  _Machine learning_ , volume 1.  McGraw-hill New York, 1997.
  * Montgomery et al. [2021] Douglas C Montgomery, Elizabeth A Peck, and G Geoffrey Vining.  _Introduction to linear regression analysis_.  John Wiley & Sons, 2021.
  * Mooney [1997] Christopher Z Mooney.  _Monte carlo simulation_.  Number 116. Sage, 1997.
  * Nanni and Lumini [2009] Loris Nanni and Alessandra Lumini.  An experimental comparison of ensemble of classifiers for bankruptcy prediction and credit scoring.  _Expert systems with applications_ , 36(2):3028–3033, 2009.
  * Rennison and Planty [2003] Callie Rennison and Mike Planty.  Nonlethal intimate partner violence: Examining race, gender, and income patterns.  _Violence and victims_ , 18(4):433–443, 2003\.
  * Sharp [2009] Gwen Sharp.  Nikon camera says asians: People are always blinking - sociological images, 2009.  URL https://thesocietypages.org/socimages/2009/05/29/nikon-camera-says-asians-are-always-blinking/.


# Autonomous Platoon Control with Integrated Deep Reinforcement Learning and
Dynamic Programming

Tong Liu, Lei Lei, Kan Zheng, Kuan Zhang

###### Abstract

Autonomous vehicles in a platoon determine the control inputs based on the
system state information collected and shared by the Internet of Things (IoT)
devices. Deep Reinforcement Learning (DRL) is regarded as a potential method
for car-following control and has been mostly studied to support a single
following vehicle. However, it is more challenging to learn an efficient car-
following policy with convergence stability when there are multiple following
vehicles in a platoon, especially with unpredictable leading vehicle behavior.
In this context, we adopt an integrated DRL and Dynamic Programming (DP)
approach to learn autonomous platoon control policies, which embeds the Deep
Deterministic Policy Gradient (DDPG) algorithm into a finite-horizon value
iteration framework. Although the DP framework can improve the stability and
performance of DDPG, it has the limitations of lower sampling and training
efficiency. In this paper, we propose an algorithm, namely Finite-Horizon-DDPG
with Sweeping through reduced state space using Stationary approximation (FH-
DDPG-SS), which uses three key ideas to overcome the above limitations, i.e.,
transferring network weights backward in time, stationary policy approximation
for earlier time steps, and sweeping through reduced state space. In order to
verify the effectiveness of FH-DDPG-SS, simulation using real driving data is
performed, where the performance of FH-DDPG-SS is compared with those of the
benchmark algorithms. Finally, platoon safety and string stability for FH-
DDPG-SS are demonstrated.

###### Index Terms:

Platoon Control; Deep Reinforcement Learning; Dynamic Programming

##  I Introduction

As a key application scenario of Internet of Vehicles (IoV) [1], vehicle
platooning has progressed dramatically thanks to the development of
Cooperative Adaptive Cruise Control (CACC) and advanced Internet of Things
(IoT) technologies in various fields [2, 3, 4, 5, 6, 7]. Autonomous platoon
control aims to determine the control inputs for the following autonomous
vehicles so that all the vehicles move at the same speed while maintaining the
desired distances between each pair of preceding and following vehicles. The
autonomous vehicles are considered to have L$3$ to L$5$ driving automation as
defined by the Society of Automotive Engineers (SAE) classification standard
[8]. A well-designed platoon controller is able to increase road capacity,
reduce fuel consumption, as well as enhance driving safety and comfort [9,
10].

Platoon controllers have been proposed based on classical control theory, such
as linear controller, $\mathcal{H}_{\infty}$ controller, and Sliding Mode
Controller (SMC) [9, 11, 12]. Platoon control is essentially a Sequential
Stochastic Decision Problem (SSDP), where a sequence of decisions have to be
made over a specific time horizon for a dynamic system whose states evolve in
the face of uncertainty. To solve such an SSDP problem, a few existing works
relied on the Model Predictive Control (MPC) method [10, 13, 14, 15, 16, 17],
where the trajectories of the leading vehicles are predicted by a model.

Although the MPC controller provides some safety guarantees to the control
policy, the control performance is still restricted by the accuracy of the
model itself. As another promising method, Reinforcement Learning (RL) can
learn an optimal control policy directly from experience data by trial and
error without requiring the stochastic properties of the underlying SSDP model
[18, 19, 20]. Moreover, the more powerful Deep Reinforcement Learning (DRL)
methods can deal with the curse-of-dimensionality problem of RL by
approximating the value functions as well as policy functions using deep
neural networks [21, 22]. In recent years, research on DRL has made
significant progress and many popular DRL algorithms have been proposed,
including value-based methods such as Deep Q Network (DQN) [23] and Double DQN
[24]; and actor-critic methods such as Deep Deterministic Policy Gradient
(DDPG) [25], Asynchronous Advantage Actor-Critic (A3C) [26], and Trust Region
Policy Optimization (TRPO) [27]. The RL/DRL algorithms have been applied to
solve the platoon control problem in a few recent literature[28, 29, 30, 31,
32, 33, 34, 35, 36, 37].

To elaborate, most of the contributions have addressed the car-following
control problem of supporting a single following vehicle. An adaptive
Proportional-Integral (PI) controller is presented in [29] whose parameters
are tuned based on the state of the vehicle according to the control policy
learned by actor-critic with kernel machines. However, the PI controller needs
to predefine the candidate set of parameters before learning. In order to
avoid this problem, an adaptive controller with parameterized batch actor-
critic is proposed in [30]. A few works improve the RL/DRL-based control
policy by modelling/predicting the leading vehicle (leader)’s behavior[31,
32]. In [31], a predictive controller based on the classical DRL algorithm
DDPG [25] is presented as an alternative to the MPC controller, which uses
advanced information about future speed reference values and road grade
changes. The human driving data has been used in [33, 34] to help RL/DRL
achieve improved performance. In [35], the DDPG is applied for car-following
control problem taking into account the acceleration-related delay, where the
leader is assumed to drive at a constant speed. The proposed algorithm is used
for comparing the performance of DRL and MPC for Adaptive Cruise Control
(ACC)-based car-following control problems in [36]. It is shown that DDPG has
more advantages over MPC in the presence of uncertainties. In [37], a
deterministic promotion RL method is proposed to improve training efficiency,
where the direction of action exploration is evaluated by a normalization-
based function and the evaluated direction works as a model-free search guide
in return.

Meanwhile, DRL-based platoon control with multiple following vehicles has only
been studied in a few recent works [38, 39, 40]. Based on Predecessor-Leader
Following (PLF) topology, a CACC-based control algorithm using DDPG is
proposed in [38]. While DDPG is the most widely used algorithm in the existing
DRL-based car-following controllers [31, 34, 35, 36, 38], it is shown that
although DDPG performs well in the single following vehicle system, it is more
difficult to learn an efficient control policy with convergence stability in a
platoon system where there are multiple following vehicles and unpredictable
leading vehicle behavior [39, 40]. To address this problem, the DDPG-based
technique is invoked in [39] for determining the parameters of Optimal
Velocity Model (OVM) instead of directly determining the accelerations.
Meanwhile, [40] proposes a Hybrid Car-Following Strategy (HCFS) that selects
the best actions derived from the DDPG controller and the linear controller,
which is used to determine vehicle acceleration in the platoon. By combining
with the classical control solutions, the performances are improved in [39,
40]. However, the classical controllers also limit the performance of the
above solutions, especially in the complex driving environment with random
disturbance and non-linear system dynamics.

In this context, we adopt an integrated DRL and Dynamic Programming (DP)
approach to improve the convergence stability and performance of DDPG-based
platoon control policy without resorting to the help of the classical
controllers. Specifically, we propose an algorithm that builds upon the
Finite-Horizon DDPG (FH-DDPG) algorithm that was applied for the energy
management of microgrids [41]. FH-DDPG addresses the unstable training problem
of DDPG in a finite-horizon setting by using two key ideas: backward induction
and time-dependent actors/critics. The DDPG algorithm is embedded into a
finite-horizon value iteration framework, and a pair of actor and critic
networks are trained for each time step by backward induction. It has been
demonstrated in [41] that compared with DDPG, FH-DDPG is much more stable and
achieves better performance.

However, FH-DDPG also suffers from some limitations that can be considered as
the “side-effects” of its DP framework, i.e., low sampling efficiency and
training efficiency. Firstly, since FH-DDPG has to train $K$ actor and critic
networks for a finite-horizon problem with $K$ time steps, the sampling
efficiency of FH-DDPG is $1/K$ that of DDPG. Specifically, for $E$ episodes of
training experience, the actor and critic networks of DDPG are trained with
$EK$ data entries, while each pair of the $K$ actor and critic networks in FH-
DDPG is only trained with $E$ data entries at the corresponding time step.
Secondly, FH-DDPG has to sweep through the entire state space when training
the actor and critic networks at each time step. As a result, the exhaustive
sweeps approach considers a large portion of the inconsequential states,
resulting in many wasted training updates.

To address the above two limitations in FH-DDPG and improve the sampling and
training efficiency, we use three key ideas in our proposed DRL algorithm for
platoon control, namely FH-DDPG with Sweeping through reduced state space
using Stationary policy approximation (FH-DDPG-SS). The contributions of this
paper are itemized next and an approach summary of related works on DRL-based
platoon controller design is given in Table I to describe characteristics of
existing methods and highlight the contributions of our proposed algorithm.

TABLE I: Approach summary of related works on RL/DRL-based platoon controller design Scenario | Approach | References |  General description
---|---|---|---
Single following vehicle | RL/DRL | [29, 30, 32, 33, 37, 31, 34, 35, 36] |  DDPG is the most widely used DRL algorithm and performs well in the single following vehicle system. However, it is more difficult to learn an efficient control policy with convergence stability in a platoon system where there are multiple following vehicles and unpredictable leading vehicle behavior.
Multiple following vehicles | DDPG | [38]
DDPG-OVM | [39] |  By combining with the classical control solutions, the performance of DDPG is improved. However, the classical controllers also limit the performance of these solutions, especially in the complex driving environment with random disturbance and non-linear system dynamics
HCFS | [40]
FH-DDPG-SS | Proposed |  We adopt an integrated DRL and DP approach to improve the convergence stability and performance of DDPG-based platoon control policy without resorting to the help of the classical controllers. We also use three key ideas to overcome the limitations of the DP framework and improve the sampling efficiency and training efficiency.

  * •

To overcome the first limitation of FH-DDPG, i.e., the low sampling
efficiency, we propose two key ideas, namely transferring network weights
backward in time and stationary policy approximation for earlier time steps.
The first key idea is to transfer the trained actor and critic network weights
at time step $k+1$ to the initial network weights at time step $k$. The second
key idea is using FH-DDPG to train the actors and critics from time steps $K$
to $m+1$, and then train a single pair of actor and critic networks using DDPG
from time steps $1$ to $m$, where the initial target network weights are set
to the trained actor and critic network weights at time step $m+1$.

  * •

To address the second limitation of FH-DDPG, i.e., the wasteful updates due to
exhaustive sweeps, we propose the third key idea, namely sweeping through
reduced state space. Specifically, we train and test a “kick-off” policy to
obtain a more refined state space. By sweeping through the reduced state
space, the training efficiency and performance are enhanced as agents can
focus on learning the states that good policies frequently visit.

  * •

To implement the above three key ideas, the FH-DDPG-SS algorithm is proposed
to combine and integrate the three improvements for FH-DDPG.

The remainder of the paper is organized as follows. The system model is
introduced in Section II. Section III formulates the SSDP model for platoon
control. The proposed DRL algorithms to solve the SSDP model are presented in
Section IV. In Section V, the performance of FH-DDPG-SS is compared with those
of the benchmark algorithms by simulation. Moreover, platoon safety and string
stability are demonstrated. Section VI concludes the paper.

##  II System Model

We consider a platoon control problem with a number of $N>2$ vehicles, i.e.,
$\mathcal{V}=\\{0,1,\cdots,N-1\\}$, wherein the position, velocity and
acceleration of a following vehicle (follower)
$i\in\mathcal{V}\backslash\\{0\\}$ at time $t$ are denoted by $p_{i}(t)$,
$v_{i}(t)$, and $acc_{i}(t)$, respectively. Here $p_{i}(t)$ represents the
one-dimensional position of the center of the front bumper of vehicle $i$.
Each follower $i$ is manipulated by a distributed car-following policy of a
DRL controller with Vehicle-to-Everything (V2X) communications.

Each vehicle $i\in\mathcal{V}$ obeys the dynamics model described by a first-
order system.

$\dot{p}_{i}(t)=v_{i}(t),$ (1) $\dot{v}_{i}(t)=acc_{i}(t),$ (2)
$\dot{acc}_{i}(t)=-\frac{1}{\tau_{i}}acc_{i}(t)+\frac{1}{\tau_{i}}u_{i}(t),$
(3)

where $\tau_{i}$ is a time constant representing driveline dynamics and
$u_{i}(t)$ is the vehicle control input (commanded acceleration) at time $t$.
In order to ensure driving safety and comfort, the following constraints are
applied

$acc_{\mathrm{min}}\leq acc_{i}(t)\leq acc_{\mathrm{max}},$ (4)
$u_{\mathrm{min}}\leq u_{i}(t)\leq u_{\mathrm{max}},$ (5)

where $acc_{\mathrm{min}}$ and $acc_{\mathrm{max}}$ are the acceleration
limits, while $u_{\mathrm{min}}$ and $u_{\mathrm{max}}$ are the control input
limits.

The headway of follower $i$ at time $t$, i.e., bumper-to-bumper distance
between follower $i$ and its preceding vehicle (predecessor) $i-1$, is denoted
by $d_{i}(t)$ with

$d_{i}(t)=p_{i-1}(t)-p_{i}(t)-L_{i-1},$ (6)

where $L_{i-1}$ is the the body length of vehicle $i-1$.

We adopt Constant Time-Headway Policy (CTHP) in this paper, i.e., follower $i$
aims to maintain a desired headway $d_{r,i}(t)$, which satisfies

$d_{r,i}(t)=r_{i}+h_{i}v_{i}(t),$ (7)

where $r_{i}$ is a standstill distance for safety of follower $i$ and $h_{i}$
is a constant time gap of follower $i$ which represents the time that it takes
for follower $i$ to bridge the distance in between the vehicles $i$ and $i-1$
when continuing to drive with a constant velocity.

The control errors, i.e., gap-keeping error $e_{pi}(t)$ and velocity error
$e_{vi}(t)$ of follower $i$ are defined as

$e_{pi}(t)=d_{i}(t)-d_{r,i}(t),$ (8) $e_{vi}(t)=v_{i-1}(t)-v_{i}(t).$ (9)

##  III SSDP Model for Platoon Control

An SSDP can be formulated to determine the vehicle’s control action. The time
horizon is discretized into time intervals of length $T$ seconds (s), and a
time interval $[(k-1)T,kT)$ amounts to a time step $k$, $k=1,2,\cdots,K$,
where $K$ is the total number of time steps. In the rest of the paper, we will
use $x_{k}:=x((k-1)T)$ to represent any variable $x$ at time $(k-1)T$.

In the following, the state space, action space, system dynamics model, and
reward function of the SSDP model are presented, respectively.

###  III-A State Space

At each time step $k\in\\{1,2,\cdots,K\\}$, the controller of follower $i$
determines the vehicle control input $u_{i,k}$, based on the observations of
the system state. $v_{i,k}$ and $acc_{i,k}$ can be measured locally, while
$e_{pi,k}$ and $e_{vi,k}$ can be measured through a radar unit mounted at the
front of the vehicle. Thus, the state that the follower $i$ can obtain locally
is denoted by $x_{i,k}=[e_{pi,k},e_{vi,k},acc_{i,k}]^{\mathrm{T}}$.

Additionally, the follower $i$ can obtain the driving status $x_{j,k}$ and
control input $u_{j,k}$ of the other vehicles
$j\in\mathcal{V}\backslash\\{i\\}$ via V2X communications. In this paper, we
adopt the Predecessors Following (PF) information topology, in which
$acc_{i-1,k}$ and $u_{i-1,k}$ are transmitted to the follower
$i\in\mathcal{V}\backslash\\{0\\}$. It has been proved in our previous work
that the optimal policy of the SSDP model with the preceding vehicle’s
acceleration $acc_{i-1,k}$ and control input $u_{i-1,k}$ in the state performs
at least as well as the optimal policy for the SSDP model which does not
include this information [5]. Thus, the system state for the follower $i$ is
denoted as: $S_{i,k}=[x_{i,k},acc_{i-1,k},u_{i-1,k}]^{\mathrm{T}}$. The state
space is
$\mathcal{S}=\\{S_{i,k}|e_{pi,k},e_{vi,k}\in[-\infty,\infty],acc_{i,k},acc_{i-1,k}\in[acc_{\mathrm{min}},acc_{\mathrm{max}}],u_{i-1,k}\in[u_{\mathrm{min}},u_{\mathrm{max}}]\\}$.

###  III-B Action Space

The control input $u_{i,k}$ of the follower $i\in\mathcal{V}\backslash\\{0\\}$
is regarded as the action at the time step $k$111An action is normally denoted
as $a$ in the RL literature. In this paper, we adopt the convention in the
optimal control literature and denote the action as $u$ for consistency.. The
action space is
$\mathcal{A}=\\{u_{i,k}|u_{i,k}\in[u_{\mathrm{min}},u_{\mathrm{max}}]\\}$.

###  III-C System Dynamics Model

The system dynamics are derived in discrete time on the basis of forward Euler
discretization. Note that for the leader $0$, $e_{p0,k}=e_{v0,k}=0$, thus the
system dynamics model evolves in discrete time according to

$x_{0,k+1}=A_{0}x_{0,k}+B_{0}u_{0,k},$ (10)

where

$A_{0}=\begin{bmatrix}0&0&0\\\ 0&0&0\\\
0&0&1-\frac{T}{\tau_{0}}\end{bmatrix},B_{0}=\begin{bmatrix}0\\\ 0\\\
\frac{T}{\tau_{0}}\end{bmatrix}.$ (11)

For the follower $i\in\mathcal{V}\backslash\\{0\\}$ in the platoon, we have

$x_{i,k+1}=A_{i}x_{i,k}+B_{i}u_{i,k}+C_{i}acc_{i-1,k},$ (12)

where

$A_{i}=\begin{bmatrix}1&T&-h_{i}T\\\ 0&1&-T\\\
0&0&1-\frac{T}{\tau_{i}}\end{bmatrix},B_{i}=\begin{bmatrix}0\\\ 0\\\
\frac{T}{\tau_{i}}\end{bmatrix},C_{i}=\begin{bmatrix}0\\\ T\\\
0\end{bmatrix}.$ (13)

###  III-D Reward Function

Reward function can guide the optimization objectives and has an impact on the
convergence of the DRL algorithm. Our objective is to minimize gap-keeping
error $e_{pi,k}$ and velocity error $e_{vi,k}$ to achieve the platoon control
target while penalizing control input $u_{i,k}$ to reduce the fuel consumption
and the jerk to improve the driving comfort. Note that the jerk is the change
rate in acceleration, which is given by

$\displaystyle j_{i,k}$
$\displaystyle=\frac{acc_{i,k+1}-acc_{i,k}}{T}=-\frac{1}{\tau_{i}}acc_{i,k}+\frac{1}{\tau_{i}}u_{i,k},$
(14)

where the second equality is due to the forward Euler discretization of (3).

Although the quadratic cost function is normally adopted in optimal control
problems, it is found that it does not work well for DDPG algorithm as the
sudden large changes of reward values often decrease its training stability.
Therefore, an absolute-value cost function is adopted for DDPG in [35, 36, 40]
to improve its performance. However, we found that the absolute-value cost
function could hinder the further performance improvement when the control
errors are relatively small. Therefore, we design a Huber loss function [42]
as the reward function for each follower $i\in\mathcal{V}\backslash\\{0\\}$,
which is given by

$R(S_{i,k},u_{i,k})=\left\\{\begin{array}[]{ll}r_{\mathrm{abs}},&\mathrm{if}\
r_{\mathrm{abs}}<\varepsilon\\\ r_{\mathrm{qua}},&\mathrm{if}\
r_{\mathrm{abs}}\geq\varepsilon\\\ \end{array}\right.,$ (15)

where

$r_{\mathrm{abs}}=-\\{|\frac{e_{pi,k}}{\hat{e}_{p,\mathrm{max}}}|+a|\frac{e_{vi,k}}{\hat{e}_{v,\mathrm{max}}}|+b|\frac{u_{i,k}}{u_{\mathrm{max}}}|+c|\frac{j_{i,k}}{2acc_{\mathrm{max}}/T}|\\},$
$r_{\mathrm{qua}}=-\lambda\\{(e_{pi,k})^{2}+a(e_{vi,k})^{2}+b(u_{i,k})^{2}+c(j_{i,k}T)^{2}\\},$

where $\varepsilon$ is the reward threshold, $\hat{e}_{p,\mathrm{max}}$ and
$\hat{e}_{v,\mathrm{max}}$ are the nominal maximum control errors such that it
is larger than most possible control errors. $\lambda$ is the reward scale.
$a$, $b$ and $c$ are the positive weights and can be adjusted to determine the
relative importance of minimizing the gap-keeping error, the velocity error,
the control input and the jerk.

Thus, the expected cumulative reward $J_{\pi_{i}}$ of the follower $i$ over
the finite time horizon $K$ under a policy $\pi_{i}$ can be expressed as

$J_{\pi_{i}}=\mathrm{E}_{\pi_{i}}[\sum_{k=1}^{K}\gamma^{k-1}R(S_{i,k},u_{i,k})],$
(16)

where $\gamma$ is the reward discount factor.

The ultimate objective is to find the optimal policy $\pi_{i}^{*}$ that
maximizes the expected cumulative reward $J_{\pi_{i}}$, i.e.,

$\pi_{i}^{*}=\arg\max_{\pi_{i}}J_{\pi_{i}}.$ (17)

##  IV DRL Algorithms

To solve the above SSDP, we propose a DRL algorithm which improves on the FH-
DDPG algorithm[41]. In the following, we will first provide a brief
introduction to the FH-DDPG algorithm, and then elaborate on the proposed
improvements.

###  IV-A FH-DDPG

FH-DDPG is a combination of DRL and DP, where the DDPG algorithm is embedded
into a finite-horizon value iteration framework. It is designed to solve
finite-horizon SSDP and improve the stability of the DDPG algorithm.

DDPG is a well-known DRL algorithm widely applied to continuous control. It
trains both a pair of actor and critic networks, i.e.,
$\mu(S_{k}|\theta^{\mu})$ and $Q(S_{k},u_{k}|\theta^{Q})$, to derive the
optimal policy $\mu^{*}(s|\theta^{\mu})$ and the corresponding action-value
(Q-value) $Q^{*}(S_{k},u_{k}|\theta^{Q})$, respectively[25]. The action-value
is defined as the expected cumulative discounted reward from time step $k$:
$Q(S_{k},u_{k}|\theta^{Q})=\mathrm{E}_{\pi}[\sum_{k^{\prime}=k}^{\infty}\gamma^{k^{\prime}-k}R(S_{k^{\prime}},u_{k^{\prime}})]$.
Experience replay is adopted in DDPG to enable stable and robust learning.
When the replay buffer is full, the oldest sample will be discarded before a
new sample is stored in the buffer. A minibatch from the buffer is sampled at
each time step in order to update the actor and critic networks. DDPG creates
a copy of the actor and critic networks as target networks, i.e.,
$\mu^{\prime}(S_{k+1}|\theta^{\mu^{\prime}})$ and
$Q^{\prime}(S_{k+1},u_{k+1}|\theta^{Q^{\prime}})$, to calculate the target
values. The critic network is updated by minimizing the Root Mean Square Error
(RMSE) $L_{k}=R(S_{k},u_{k})+\gamma
Q^{\prime}(S_{k+1},\mu^{\prime}(S_{k+1}|\theta^{\mu^{\prime}})|\theta^{Q^{\prime}})-Q(S_{k},u_{k}|\theta^{Q})$
using the sampled gradient descent with respect to $\theta^{Q}$. The actor
network is updated by using the sampled deterministic policy gradient ascent
on $Q(S_{k},\mu(S_{k}|\theta^{\mu})|\theta^{Q})$ with respect to
$\theta^{\mu}$. In order to improve the stability of learning, the weights of
these target networks are updated by soft target update, where the target
networks are constrained to slowly track the learned networks.

DDPG is designed to solve the infinite-horizion SSDPs, where the actors and
critics are the same for every time step. On the other hand, the optimal
policies and the corresponding action-values are normally time-dependent in a
finite-horizon setting [43]. Therefore, there are $K$ actors and critics in
FH-DDPG for an SSDP with $K$ time steps. As shown in Fig.1, FH-DDPG starts by
having the myopic policy $\mu_{K}^{*}(S_{K})=\mu^{\mathrm{mo}}(S_{K})$ as the
optimal policy with the terminal reward $R_{K}$ for the final time step $K$.
And then, the finite horizon value iteration starts from time step $K-1$, and
uses backward induction to iteratively derive the optimal policy
$\mu^{*}_{k}(S_{k}|\theta^{\mu_{k}})$ and the action-value
$Q^{*}_{k}(S_{k},u_{k}|\theta^{Q_{k}})$ for each time step $k$, until it
reaches the first time step $k=1$. In each time step, an algorithm similar to
DDPG is adopted to solve a one-period MDP in which an episode only consists of
two time steps. However, different from DDPG, the target actor network
$\mu^{\prime}_{k}(S_{k+1}|\theta^{\mu^{\prime}_{k}})$ and critic network
$Q^{\prime}_{k}(S_{k+1},u_{k+1}|\theta^{Q^{\prime}_{k}})$ of the current time
step $k$ are fixed and set as the trained actor network
$\mu_{k+1}(S_{k+1}|\theta^{\mu_{k+1}})$ and critic network
$Q_{k+1}(S_{k+1},u_{k+1}|\theta^{Q_{k+1}})$ of the next time step $k+1$. This
greatly increases the stability and performance of the algorithm. The
pseudocode of the FH-DDPG algorithm is given in Appendix A. Note that in each
time step, the $\mathrm{DDPG-FT}$ function is used to train the respective
actor and critic networks, where $\mathrm{DDPG-FT}$ is the abbreviation for
DDPG with fixed targets.

Figure 1: FH-DDPG framework.

###  IV-B Improving sampling efficiency

For a finite-time horizon consisting of $K$ time steps, when both DDPG and FH-
DDPG are trained for $E$ episodes, a total of $EK$ data entries are generated
from the experience data as a data entry is generated per time step. Since
DDPG has a single pair of actor and critic networks, all the $EK$ data entries
are used to train this pair of networks. Meanwhile, there are $K$ pairs of
actor and critic networks in FH-DDPG. For each time step $k$, the
corresponding pair of actor and critic networks is trained only with the $E$
data entries of time step $k$. Therefore, the sampling efficiency of FH-DDPG
is only $1/K$ of that of DDPG. To improve sampling efficiency, two key ideas
are proposed in the following.

####  IV-B1 Transferring network weights backward in time

In FH-DDPG, the actor ${\mu}_{i,k}(S_{i,k}|\theta^{\mu_{i,k}})$ and the critic
${Q}_{i,k}(S_{i,k},u_{i,k}|\theta^{Q_{i,k}})$ at each time step $k$ are
trained with random initial parameters. Inspired by the parameter-transfer
approach in transfer learning[44], we transfer the trained actor and critic
network weights at time step $k+1$, i.e., $\theta^{\mu_{i,k+1}}$ and
$\theta^{Q_{i,k+1}}$ to the initial network weights at time step $k$, i.e.,
$\theta^{\mu_{i,k}}$ and $\theta^{Q_{i,k}}$ respectively. Thus, although
$\mu_{i,k}(S_{i,k}|\theta^{\mu_{i,k}})$ and
$Q_{i,k}(S_{i,k},u_{i,k}|\theta^{Q_{i,k}})$ are trained based on the $E$ data
entries of the $E$ episodes at time step $k$, the trainings are built upon the
initial weights $\theta^{\mu_{i,k+1}}$ and $\theta^{Q_{i,k+1}}$, which were in
turn trained based on the $E$ data entries of the $E$ episodes at time step
$k+1$ and built upon the initial weights $\theta^{\mu_{i,k+2}}$ and
$\theta^{Q_{i,k+2}}$. In this way, the actor and critic networks at time step
$k$ are actually trained from the experiences of the $E(K-k)$ data entries of
the $E$ episodes from time steps $k$ to $K$, instead of only the $E$ data
entries as in FH-DDPG. The proposed algorithm is given in Algorithm 1, namely
FH-DDPG with Network weights transferring Backward in time (FH-DDPG-NB).

Algorithm 1 FH-DDPG-NB algorithm

1: Randomly initialize actor and critic network weights as $\theta^{\mu 0}$
and $\theta^{Q0}$

2: Set $\mu_{i,K}^{*}(S_{i,K})=\mu^{\mathrm{mo}}(S_{i,K})$ for the final time
step $K$

3: for $k=K-1,\cdots,1$ do

4: if $k=K-1$ then

5: $\theta^{\mu_{i}}=\theta^{\mu 0}$ and $\theta^{Q_{i}}=\theta^{Q0}$

6: else

7: $\theta^{\mu_{i}}=\theta^{\mu_{i,k+1}}$ and
$\theta^{Q_{i}}=\theta^{Q_{i,k+1}}$

8: end if

9: $\theta^{\mu_{i,k}},\theta^{Q_{i,k}}\leftarrow\mathrm{DDPG-
FT}(\theta^{\mu_{i}},\theta^{Q_{i}},\theta^{\mu^{\prime}_{i}},\theta^{Q^{\prime}_{i}},k)$

10: Update the target network:
$\theta^{\mu^{\prime}_{i}}\leftarrow\theta^{\mu_{i,k}},\
\theta^{Q^{\prime}_{i}}\leftarrow\theta^{Q_{i,k}}$

11: end for

12: return $\\{\theta^{\mu_{i,k}},\theta^{Q_{i,k}}\\}_{k=1}^{K-1}$

####  IV-B2 Stationary policy approximation for earlier time steps

Although the action-values are generally time-dependent for the finite-horizon
control problems, there is a vanishing difference in action-values when the
horizon length is sufficiently large [45]. Taking the finite-horizon Linear
Quadratic Regulator (LQR) as an example, the action-value gradually converges
to the steady-state value as the time step $k$ decreases from the horizon $K$.
Moreover, for $k$ not close to horizon $K$, the LQR optimal policy is
approximately stationary [46, 47]. Even though the platoon control problem is
not an LQR problem in the strict sense, since both the system state $S_{i,k}$
and action $u_{i,k}$ have constraints and there are non-Gaussian random
disturbances, we can observe similar trends in the learned policy by FH-DDPG.
This allows us to improve sampling efficiency by first obtaining the time step
threshold $m$ such that the action-values and optimal policies are
approximately constant and stationary when $k\leq m$. And then, we adopt a
single pair of actor and critic networks from time steps $1$ to $m$.

To elaborate, we first obtain $m$ by solving the LQR problem for platoon
control and analyzing the corresponding results, ignoring the state/action
constraints and random disturbances. This enables us to determine the value of
$m$ in an efficient manner. Then FH-DDPG is trained from time steps $K$ to
$m+1$. Next, instead of training a separate pair of actor and critic networks
for each time step from $1$ to $m$, we train a single actor network
$\mu_{i}(S_{i,k}|\theta^{\mu_{i}})$ and critic network
$Q_{i}(S_{i,k},u_{i,k}|\theta^{Q_{i}})$ for all the time steps
$k\in\\{1,2,...,m\\}$. Specifically, the actor and critic networks are trained
using DDPG, where the initial values of the target networks are set to those
of the trained actor and critic networks at time step $m+1$, i.e.,
$\theta^{\mu_{i,m+1}}$ and $\theta^{Q_{i,m+1}}$. The well-trained initial
values for the target networks can significantly increase the stability and
performance of the DDPG algorithm. In this way, the actor and critic networks
are trained from the experiences of the $Em$ data entries of the $E$ episodes
from time steps $1$ to $m$. The proposed algorithm is given in Algorithm 2,
namely FH-DDPG with Stationary policy Approximation for earlier time steps
(FH-DDPG-SA). The function $\mathrm{FH-DDPG}$ is realized by the FH-DDPG
algorithm given in Appendix A. Note that FH-DDPG-SA can be combined with FH-
DDPG-NB, by adopting the latter algorithm instead of FH-DDPG to train the
actor and critic networks from time steps $K$ to $m+1$ in line 2 of the
pseudocode. This will result in the FH-DDPG-SA-NB algorithm.

Algorithm 2 FH-DDPG-SA-(NB) algorithm

1: Set the time horizon as $\\{m+1,\cdots,K\\}$

2:
$\\{\theta^{\mu_{i,k}},\theta^{Q_{i,k}}\\}_{k=m+1}^{K-1}\leftarrow\mathrm{FH-
DDPG(-NB)}$

3: Set the time horizon as $\\{1,\cdots,m\\}$

4: Set the initial target networks weights with
$\theta^{\mu^{\prime}_{i}}=\theta^{\mu_{i,m+1}}$ and
$\theta^{Q^{\prime}_{i}}=\theta^{Q_{i,m+1}}$

5: $\theta^{\mu_{i}},\theta^{Q_{i}}\leftarrow\mathrm{DDPG}$

###  IV-C Sweeping through reduced state space

FH-DDPG embeds DRL under the framework of DP, while the classical approach of
DP is to sweep through the entire state space at each time step $k$. This
exhaustive sweeps approach leads to many wasteful updates during training,
since many of the states are inconsequential as they are visited only under
poor policies or with very low probability. An alternative approach is
trajectory sampling which sweeps according to on-policy distribution[18].
Although trajectory sampling is more appealing, it is impossible to be adopted
by FH-DDPG due to the latter’s backward induction framework.

Inspired by trajectory sampling, we improve FH-DDPG by sweeping through a
reduced state space. Specifically, we first learn a relatively good “kick-off”
policy by exhaustive sweeps, and then obtain a reduced state space by testing
the “kick-off” policy, and finally continue to train the policy by sweeping
through the reduced state space to further improve the performance. This
approach can help agents to focus on learning the states that good policies
frequently visit, which improves training efficiency. Note that a good policy
in RL should achieve a large expected cumulative reward $J_{\pi_{i}}$ as given
in (18). For example in platoon control, the control errors are normally small
under good policies as the ends of the training episodes are approached, and
it is not necessary to sweep through large control error states.

For platoon control, although the theoretical bounds of gap-keeping error
$e_{pi,k}$ and velocity error $e_{vi,k}$ are infinity, it is impossible to
sweep through an infinite range when training. Therefore, we need to restrict
sweeping to a finite range at first. In practice, there are some empirical
limits to $e_{pi,k}$ and $e_{vi,k}$ for a reasonable platoon control policy.
Since we consider that FH-DDPG is trained from scratch, some relatively large
control error states could be visited during training due to the random
initial policy and exploration. Therefore, we first sweep through a relatively
large state space, i.e.,

$\displaystyle\mathcal{S}^{\mathrm{ls}}=$
$\displaystyle\\{[e_{pi,k},e_{vi,k},acc_{i,k}]^{\mathrm{T}}|$ $\displaystyle
e_{pi,k}\in[-e_{p,\mathrm{max}},e_{p,\mathrm{max}}],$ $\displaystyle
e_{vi,k}\in[-e_{v,\mathrm{max}},e_{v,\mathrm{max}}],$ $\displaystyle
acc_{i,k}\in[acc_{\mathrm{min}},acc_{\mathrm{max}}]\\},$ (18)

where $e_{p,max}$ and $e_{v,max}$ are the same for each time step and are
larger than most control errors during the training of FH-DDPG. Thus, we first
train FH-DDPG in the state space $\mathcal{S}^{\mathrm{ls}}$ to learn a “kick-
off” policy $\hat{\mu}_{i,k}(S_{i,k}|\hat{\theta}^{\mu_{i,k}})$ for the
follower $i$ at time step $k$, and then obtain the upper and lower bounds of a
more refined state space, i.e.,

$\displaystyle\mathcal{S}_{i,k}^{\mathrm{rs}}=$
$\displaystyle\\{[e_{pi,k},e_{vi,k},acc_{i,k}]^{\mathrm{T}}|$ $\displaystyle
e_{pi,k}\in[\overline{e}_{pi,k,\mathrm{min}},\overline{e}_{pi,k,\mathrm{max}}],$
$\displaystyle
e_{vi,k}\in[\overline{e}_{vi,k,\mathrm{min}},\overline{e}_{vi,k,\mathrm{max}}],$
$\displaystyle
acc_{i,k}\in[\overline{acc}_{i,k,\mathrm{min}},\overline{acc}_{i,k,\mathrm{max}}]\\},$
(19)

for the follower $i$ at time step $k$ by testing
$\hat{\mu}_{i,k}(S_{i,k}|\hat{\theta}^{\mu_{i,k}})$. Next, the actor network
$\hat{\mu}_{i,k}(S_{i,k}|\hat{\theta}^{\mu_{i,k}})$ and critic network
$\hat{Q}_{i,k}(S_{i,k},u_{i,k}|\hat{\theta}^{Q_{i,k}})$ are further trained by
FH-DDPG, which only sweeps through $\mathcal{S}_{i,k}^{\mathrm{rs}}$.

Combining the above three improvements for FH-DDPG, we propose a novel DRL
algorithm, namely FH-DDPG with Sweeping through reduced state space using
Stationary policy approximation (FH-DDPG-SS), which is given in Algorithm 3.
Note that the overall procedure of FH-DDPG-SS is the same as that described in
Section IV.C, except that in line 2 and line 12, the FH-DDPG-SA-NB and FH-
DDPG-SA algorithms are adopted instead of the FH-DDPG algorithm to incorporate
the improvements in Algorithm 1 and Algorithm 2. The reason why we use FH-
DDPG-SA instead of FH-DDPG-SA-NB in line 12 is that the initial actor and
critic networks weights for all the time steps are carried over from the
previous training, so we no longer need to transfer network weights backward
in time.

Algorithm 3 FH-DDPG-SS algorithm

1: Set $\mathcal{S}^{\mathrm{ls}}$ according to (IV-C)

2:
$\\{\hat{\theta}^{\mu_{i,k}},\hat{\theta}^{Q_{i,k}}\\}_{k=1}^{K-1}\leftarrow\mathrm{FH-
DDPG-SA-NB}(\mathcal{S}^{\mathrm{ls}})$

3: for $g=1,\dots,G$ do

4: Test $\\{\hat{\mu}_{i,k}(S_{i,k}|\hat{\theta}^{\mu_{i,k}})\\}_{k=1}^{K-1}$

5: Store
$\\{x_{i,k}^{(g)}\\}_{k=1}^{K-1}=\\{[e_{pi,k}^{(g)},e_{vi,k}^{(g)},acc_{i,k}^{(g)}]^{\mathrm{T}}\\}_{k=1}^{K-1}$
in $\\{\mathcal{B}_{i,k}^{\mathrm{t}}\\}_{k=1}^{K-1}$

6: end for

7: for $k=1,\dots,K-1$ do

8: Find the upper and lower bounds $\overline{e}_{pi,k,\mathrm{min}}$,
$\overline{e}_{pi,k,\mathrm{max}}$, $\overline{e}_{vi,k,\mathrm{min}}$,
$\overline{e}_{vi,k,\mathrm{max}}$, $\overline{acc}_{i,k,\mathrm{min}}$ and
$\overline{acc}_{i,k,\mathrm{max}}$ in $\mathcal{B}_{i,k}^{\mathrm{t}}$

9: Set $\mathcal{S}_{i,k}^{\mathrm{rs}}$ according to (IV-C)

10: end for

11: Set the initial actor and critic network weights as
$\\{\hat{\theta}^{\mu_{i,k}},\hat{\theta}^{Q_{i,k}}\\}_{k=1}^{K-1}$

12:
$\\{\theta^{\mu_{i,k}},\theta^{Q_{i,k}}\\}_{k=1}^{K-1}\leftarrow\mathrm{FH-
DDPG-SA}(\\{\mathcal{S}_{i,k}^{\mathrm{rs}}\\}_{k=1}^{K-1})$

##  V Experimental Results

In this section, we present the simulation results of the proposed FH-DDPG-SS
algorithm as well as benchmark DRL algorithms, i.e., DDPG, FH-DDPG, and HCFS
[40].

###  V-A Experimental Setup

All the algorithms are trained/tested using the open source data given in
[34]. Specifically, the data was extracted from the Next Generation Simulation
(NGSIM) dataset [48], which was retrieved from the eastbound I-80 in the San
Francisco Bay area in Emeryville, CA, on April $13$, $2005$, at a sampling
rate of $10$ Hz, with $45$ minutes of precise location data available in the
full dataset. Then, car-following events were extracted by applying a car-
following filter as described in [49], where the leading and following vehicle
pairs of each event stay in the same lane. In our experiment, we used $1000$
car-following events. Moreover, although there are data for both the leading
vehicle and following vehicle in the dataset, we only used the velocity data
of the leading vehicle to simulate the real-world environment with
uncertainty, so that the DRL algorithms can be trained and evaluated. $80\%$
of the data (i.e., $800$ car-following events) is used for training and $20\%$
(i.e., $200$ car-following events) is used for testing. The platoon control
environment and the DRL algorithms are implemented in Tensorflow 1.14 using
Python [50]. We compare the performance of the proposed FH-DDPG-SS algorithm
with the benchmark algorithms in terms of the average cumulative reward.
Moreover, the platoon safety and string stability performance for FH-DDPG-SS
are also demonstrated. To ensure a fair comparison, all the algorithms are
trained and tested in the same environment with the same reward function.

The technical constraints and operational parameters of the platoon control
environment are given in Table II. In general, the parameters of the platoon
environment and state/action space in Table I are determined using the values
reported in [35, 36] as a reference. The interval for each time step is set to
$T=0.1$ $\rm s$, and each episode is comprised of $100$ time steps (i.e.,
$K=100$) with a duration of $10$ $\rm s$. As the number of vehicles set in the
existing literature on platoon control ranges from $3$ to $8$[11, 13, 16, 17,
38, 39, 40], we set the number of vehicles to $N=5$. We initialize the states
for each of $4$ followers with $x_{i,0}=[1.5,-1,0]$, $\forall
i\in\\{1,2,3,4\\}$. We experiment with several sets of reward coefficients
$a,b$, and $c$ to balance the minimization of gap-keeping error $e_{pi,k}$,
velocity error $e_{vi,k}$, control input $u_{i,k}$, and jerk $j_{i,k}$. In
addition, the absolute-value cost function can improve the algorithm’s
convergence stability when control errors are large, and the square-value cost
function is conducive to further performance improvement when control errors
are small. Thus, we determine the reward threshold $\varepsilon$ with the best
performance for all algorithms by a simple grid search. The nominal maximum
control errors in the reward function (15) are set to $\hat{e}_{p,\rm max}=15$
$\rm m$ and $\hat{e}_{v,\rm max}=10$ $\rm m/s$ so that it is larger than most
possible control errors during training for all DRL algorithms. For the FH-
DDPG-SS algorithm, the time step threshold $m=11$. Moreover, the maximum gap-
keeping error $e_{p,\rm max}$ and maximum velocity error $e_{v,\rm max}$ in
(IV-C) are set to $2$ $\rm{m}$ and $1.5$ $\rm{m/s}$, respectively.
Additionally, to reduce large oscillations in $u_{i,k}$ and $acc_{i,k}$, we
set $j_{i,k}$ in FH-DDPG and FH-DDPG-SS in the testing phase within
$[-0.3,0.6]$ when $k>11$ by clipping $u_{i,k}$.

TABLE II: Technical constraints and operational parameters of the platoon control environment Notations | Description | Values
---|---|---
Platoon environment |  |
$T$ | Interval for each time step |  $0.1$ $\rm s$
$K$ | Total time steps in each episode | $100$
$m$ | Time step threshold | 11
$N$ | Number of vehicles | 5
$\tau_{i}$ | Driveline dynamics time constant |  $0.1$ $\rm s$
$h_{i}$ | Time gap |  $1$ s
State & action |  |
$e_{p,\rm max}$ | Maximum gap-keeping error |  $2$ $\rm{m}$
$e_{v,\rm max}$ | Maximum velocity error |  $1.5$ $\rm{m/s}$
$acc_{\rm min}$ | Minimum acceleration |  $-2.6$ $\rm{m/s^{2}}$
$acc_{\rm max}$ | Maximum acceleration |  $2.6$ $\rm{m/s^{2}}$
$u_{\rm min}$ | Minimum control input |  $-2.6$ $\rm{m/s^{2}}$
$u_{\rm max}$ | Maximum control input |  $2.6$ $\rm{m/s^{2}}$
Reward function |  |
$a$ | Reward coefficient | $0.1$
$b$ | Reward coefficient | $0.1$
$c$ | Reward coefficient | $0.2$
$\hat{e}_{p,\rm max}$ | Nominal maximum gap-keeping error |  $15$ $\rm{m}$
$\hat{e}_{v,\rm max}$ | Nominal maximum velocity error |  $10$ $\rm{m/s}$
$\varepsilon$ | Reward threshold | $-0.4483$
TABLE III: Hyper-Parameters of the DRL algorithms for training Parameter | Value
---|---
| DDPG | FH-DDPG | FH-DDPG-SS
Actor network size | $256,128$ | $400,300,100$ | $400,300,100$
Critic network size | $256,128$ | $400,300,100$ | $400,300,100$
Actor activation function | relu, relu, tanh | relu, relu, relu, tanh | relu, relu, relu, tanh
Critic activation function | relu, relu, linear | relu, relu, relu, linear | relu, relu, relu, linear
Actor learning rate $\alpha$ |  $1$e$-4$ |  $1$e$-4$ |  $1$e$-4$
Critic learning rate $\beta$ |  $1$e$-3$ |  $1$e$-3$ |  $1$e$-3$
Total training episodes $E$ | $5000$ | $5000$ | $3000,2000$
Batch size $N_{b}$ | $64$
Replay buffer size | $250000$ | $2500$ | $2500,2000$
Reward scale $\lambda$ |  $5$e$-3$
Reward discount factor $\gamma$ | 1
Soft target update $\eta$ | $0.001$ | / | FH-DDPG: /, DDPG: $0.001$
Noise type | Ornstein-Uhlenbeck Process with $\theta=0.15$ and $\sigma=0.5$
Final layer weights/biases initialization | Random uniform distribution $[-3\times 10^{-3},3\times 10^{-3}]$
Other layer weights biases initialization |  Random uniform distribution$[-\frac{1}{\sqrt{f}},\frac{1}{\sqrt{f}}]$ ($f$ is the fan-in of the layer)

The hyper-parameters for training are summarized in Table LABEL:table_2. The
values of all the hyper-parameters were selected by performing a grid search
as in [23], using the values reported in [25] as a reference. DDPG has two
hidden layers with $256$ and $128$ nodes, respectively; while FH-DDPG and FH-
DDPG-SS have three hidden layers with $400$, $300$, and $100$ nodes,
respectively. The sizes of input layers for all DRL algorithms are the same
and decided by the PF information topology. Moreover, an additional
$1$-dimensional action input is fed to the second hidden layer for each critic
network. The total number of training episodes $E$ for all DRL algorithms is
set to $5000$. For FH-DDPG-SS, we first train the algorithm for $3000$
episodes to learn the “kick-off” policy in the first phase, and then continue
to train $2000$ episodes within the reduced state space in the second phase.
The replay buffer sizes for DDPG and FH-DDPG are $250000$ and $2500$,
respectively. This is because the replay buffer for FH-DDPG only stores the
data entries for the corresponding time step. Since FH-DDPG-SS is trained in
two phases, the replay buffer sizes for the first and second phases are $2500$
and $2000$, respectively. Moreover, FH-DDPG-SS leverages the FH-DDPG-SA-(NB)
algorithm, which trains the $K-m-1$ actors and critics for time steps $K$ to
$m+1$ using FH-DDPG, and a single pair of actor and critic for time steps $1$
to $m$ using DDPG. The soft target update is implemented with a parameter of
$\eta=0.001$ for DDPG. As FH-DDPG uses a fixed target network, there is no
soft target update.

###  V-B Comparison of FH-DDPG-SS with the benchmark algorithms

####  V-B1 Performance for testing data

The individual performance of each follower $i\in\\{1,2,3,4\\}$ as well as the
sum performance of the $4$ followers are reported in Table IV for DDPG, FH-
DDPG, HCFS, and FH-DDPG-SS, respectively. The individual performance of each
follower and the sum performance of all followers are obtained by averaging
the returns of the corresponding followers and the sum returns of all
followers, respectively, over $200$ test episodes after training is completed.
Note that in RL terminology, a return is the cumulative rewards of one
episode. It can be observed that the individual performance of the preceding
vehicles are attenuated by following vehicles upstream the platoon for each
algorithm. Compared with the other algorithms, FH-DDPG-SS consistently shows
the best individual performance for each follower $i\in\\{1,2,3,4\\}$. We can
observe that the ranking in terms of the sum performance of all followers for
the different algorithms is FH-DDPG-SS$>$HCFS$>$FH-DDPG$>$DDPG, where FH-DDPG-
SS outperforms DDPG, FH-DDPG, and HCFS algorithms by $46.62\%$, $13.73\%$, and
$10.60\%$, respectively. Note that HCFS outperforms DDPG because two actions
are obtained from DDPG and the classical linear controller at each time step,
and the one that achieves the maximum reward is selected. On the other hand,
FH-DDPG performs better than DDPG as it applies backward induction and time-
dependent actors/critics. Moreover, FH-DDPG-SS further improves the
performance of FH-DDPG by implementing the three key ideas proposed in Section
IV.

In Table V, we present the maximum, minimum, and standard deviation of the sum
returns of the $4$ followers across the $200$ test episodes for DDPG, FH-DDPG,
HCFS, and FH-DDPG-SS, respectively. It can be observed that although the
objective of DRL algorithms is to maximize the expected return as given in
(16) and (17), FH-DDPG-SS achieves the best performance in terms of the
maximum, minimum, and standard deviation of the returns among all the DRL
algorithms.

TABLE IV: Performance after training with NGSIM dataset. Each episode has $100$ time steps in total. We present the individual performance of each follower as well as the sum performance of the 4 followers for DDPG, FH-DDPG, HCFS, and FH-DDPG-SS, respectively. Algorithm | Individual performance | Sum performance
---|---|---
Follower 1 | Follower 2 | Follower 3 | Follower 4
DDPG | -0.0680 | -0.0876 | -0.0899 | -0.2980 | -0.5435
FH-DDPG | -0.0736 | -0.0845 | -0.0856 | -0.0927 | -0.3364
HCFS | -0.0673 | -0.0740 | -0.0828 | -0.1005 | -0.3246
FH-DDPG-SS | -0.0600 | -0.0691 | -0.0776 | -0.0835 | -0.2902
TABLE V: The maximum, minimum, and standard deviation of the sum returns of the $4$ followers across the $200$ test episodes for DDPG, FH-DDPG, HCFS, and FH-DDPG-SS, respectively. Algorithm | Maximum | Minimum | Standard deviation
---|---|---|---
DDPG | -0.5018 | -0.5822 | 0.0137
FH-DDPG | -0.3248 | -0.3532 | 0.0071
HCFS | -0.3011 | -0.3458 | 0.0068
FH-DDPG-SS | -0.2726 | -0.3114 | 0.0063

####  V-B2 Convergence properties

(a) DDPG

(b) FH-DDPG

(c) FH-DDPG-SS

Figure 2: Performance during DRL algorithms training. The vertical axis
corresponds to the average returns over 10 test episodes.

The performance of DRL algorithms is evaluated periodically during training by
testing without exploration noise. Specifically, we run $10$ test episodes
after every $100$ training episodes, and average the returns over the $10$
test episodes as the performance for the latest $100$ training episodes. The
performance as a function of the number of training episodes for each follower
$i$ with DDPG, FH-DDPG, and FH-DDPG-SS is plotted in Fig. 2. The convergence
curve of HCFS is not plotted here since HCFS combines the trained DDPG
controller with the linear controller, and thus the convergence property of
HCFS is the same as that of DDPG. It can be observed from Fig. 2 that the
initial performances of FH-DDPG and FH-DDPG-SS are much better than that of
DDPG. This is due to the different ways how DDPG and FH-DDPG/FH-DDPG-SS work.
For DDPG, the first point in Fig. 2(a) is obtained by running $10$ test
episodes of the initial policy and calculating the average return. Since the
initial weights of the actor network are random, the initial policy has poor
performance. In FH-DDPG, there is a pair of actor/critic networks associated
with each time step, and the actors/critics are trained backward in time.
Therefore, when we train the actor/critic of the first time step, the
actors/critics of the rest of the time steps are already trained. Similarly,
the first point in Fig. 2(c) for FH-DDPG-SS is obtained by running $10$ test
episodes where only the policies (actors) for time steps $1$ to $m$ are not
trained, while the policies (actors) of the time steps $m+1$ to $K$ are
already trained.

Fig. 2(a) shows that although the performances of all the followers in DDPG
improve quickly in the first $200$ episodes, the performance curves exhibit
significantly larger oscillation during the following training episodes
compared to those of FH-DDPG and FH-DDPG-SS. This is especially true for
follower $4$, whose performance drops below $-20$ at around $1500$ episodes.
The above results demonstrate that the convergence of DDPG is relatively
unstable, especially for the followers upstream the platoon. As shown in Fig.
2(b), the convergence stability of FH-DDPG is significantly improved over
DDPG, as there are only small fluctuations in the performance curve. This is
due to the fact that FH-DDPG trains backward in time, where DDPG with a fixed
target is adopted in each time step to solve a one-period MDP. However, as FH-
DDPG is under the framework of dynamic programming, it sacrifices the sampling
and training efficiency to achieve the convergence stability. As shown in Fig.
2(b), although the performance of FH-DDPG quickly increases in the first $200$
episodes, there is hardly any further improvement during the remaining
training episodes. Fig. 2(c) shows that the convergence stability of FH-DDPG-
SS is similar to that of FH-DDPG and much better than that of DDPG. Moreover,
the performance of FH-DDPG-SS is consistently better than that of FH-DDPG for
each follower during the whole training episode. For example, focusing on
follower $1$, the performance of FH-DDPG is around $-0.0735$ at the beginning
of training and finally reached $-0.0715$ towards the end. On the other hand,
the performance of FH-DDPG-SS is around $-0.0665$ initially and finally
reached $-0.0600$ at the end of training. The superior performance of FH-DDPG-
SS before $3000$ episodes is due to the two proposed key ideas of transferring
network weights backward in time and stationary policy approximation for
earlier time steps, which improve the sampling efficiency. In other words,
from the same number of data entries generated by the $3000$ training
episodes, FH-DDPG-SS can make more efficient use of the data entries to learn
a better policy. Moreover, it can be observed that there is a sudden
performance improvement for all the vehicles of FH-DDPG-SS at around $3000$
episodes. This salient performance gain is due to the third key idea of
sweeping through the reduced state space. Since FH-DDPG-SS is trained in the
refined state space for the last $2000$ episodes, the training efficiency is
improved as agents focus on learning the states that good policies frequently
visit.

####  V-B3 Testing results of one episode

(a) DDPG

(b) FH-DDPG

(c) HCFS

(d) FH-DDPG-SS

Figure 3: Results of a specific test episode. The driving status $e_{pi,k}$,
$e_{vi,k}$, and $acc_{i,k}$ along with the control input $u_{i,k}$ and jerk
$j_{i,k}$ of each follower $i$ are represented as different curves,
respectively.

We focus our attention on a specific test episode having $100$ time steps, and
plot driving status $e_{pi,k}$, $e_{vi,k}$, $acc_{i,k}$ and control input
$u_{i,k}$ along with jerk $j_{i,k}$ of each follower $i$ for all the time
steps $k\in\\{1,2,\cdots,100\\}$. Fig. 3(a), Fig. 3(b), Fig. 3(c), and Fig.
3(d) show the results of a specific test episode for DDPG, FH-DDPG, HCFS, and
FH-DDPG-SS, respectively. It can be observed that the overall shapes of the
corresponding curves of all the algorithms look very similar except that the
performance curves for follower $4$ using DDPG have large oscillations. This
observation is aligned with the results in Table III, where follower $4$ has
significantly worse performance when using DDPG compared with using other
algorithms. Fig. 3 shows that in general for each follower
$i\in\\{1,2,3,4\\}$, $e_{pi,k}$ has an initial value of $1.5$ $\rm m$ and is
reduced over time to approximately $0$ $\rm m$; $e_{vi,k}$ has an initial
value of $-1$ $\rm{m/s}$ and is increased to approximately $0$ $\rm{m/s}$;
$u_{i,k}$ is relatively large at the beginning of the episode to increase
$acc_{i,k}$ as fast as possible, so that $e_{pi,k}$ and $e_{vi,k}$ can
promptly converge to approximately $0$. Correspondingly, $acc_{i,k}$ of each
follower $i$ has an initial value of $0$ $\rm m/s^{2}$ and is suddenly
increased to a relatively large value. Then both $u_{i,k}$ and $acc_{i,k}$ are
quickly reduced to a negative value, and finally are increased over time to
approximately $0$ $\rm{m/s^{2}}$. After the driving status and control input
converge to near $0$, the values fluctuate around $0$ with $u_{i,k}$ trying to
maximize the expected cumulative reward in (17) without knowing the future
control inputs $u_{i-1,k^{\prime}}$, $k<k^{\prime}<K$, of the predecessor
$i-1$. Additionally, $j_{i,k}$ of each follower $i$ starts with a large
positive value and is then reduced to a negative value. After converging to
near $0$ $\rm{m/s^{3}}$, the value of $j_{i,k}$ fluctuates around $0$
$\rm{m/s^{3}}$.

A closer examination of Fig. 3 reveals that the performance differences of the
algorithms are reflected in convergence speed to steady-state and the
oscillations of the driving status and control input. Focusing on $e_{pi,k}$,
it can be observed that there are still positive gap-keeping errors for
followers $2$, $3$, and $4$ in DDPG up to the end of the episode. Moreover,
$e_{pi,k}$ of follower $4$ in DDPG has the slowest convergence speed to $0$
$\rm m$ and the largest oscillations among all the algorithms. Meanwhile,
$e_{pi,k}$ in FH-DDPG is reduced to $0$ $\rm m$ for each follower, but there
are relatively large oscillations after convergence. $e_{pi,k}$ in HCFS has
smaller oscillations than that in FH-DDPG and also converges to $0$ $\rm m$,
but there are also large oscillations near the end of the episode for follower
$4$. $e_{pi,k}$ in FH-DDPG-SS has the fastest convergence speed to $0$ $\rm
m$, and then remains around $0$ $\rm m$ with small oscillations. Now focusing
on $e_{vi,k}$, the velocity error in DDPG has the slowest convergence speed to
$0$ $\rm m/s$ among all the algorithms. Moreover, $e_{vi,k}$ of follower $4$
in DDPG has the largest oscillations among all the algorithms. $e_{vi,k}$ in
FH-DDPG and HCFS has relatively large oscillations after convergence to $0$
$\rm m/s$. $e_{vi,k}$ in FH-DDPG-SS has the smallest oscillations around $0$
$\rm m/s$ after convergence to steady-state. Finally, compared with the other
algorithms, FH-DDPG-SS has the smallest jerk $j_{i,k}$ at the beginning of the
episode. Although $j_{i,k}$ in FH-DDPG-SS is not as small as those in DDPG and
HCFS in the later stage of the episode, it is smaller than that in FH-DDPG and
remains at a relatively small level, which can ensure the driving comfort.

###  V-C Platoon safety

In order to demonstrate that the platoon safety is ensured in the proposed FH-
DDPG-SS algorithm, Table VI summarizes the average, maximum, and minimum
returns as well as the standard deviation across the 200 test episodes for
each follower $i$ in FH-DDPG-SS. Additionally, $e_{pi,k}$ per time step $k$
for the worst test episode among the $200$ test episodes is plotted in Fig. 4.

It can be observed from Table VI that the standard deviation of each follower
$i$ is small ranging from $0.0015$ to $0.0017$. Moreover, the differences
between the maximum and minimum returns are small for all followers.
Specifically, the minimum return is worse than the maximum return by
$16.28\%$, $15.22\%$, $13.95\%$ and $12.96\%$ for the $4$ followers,
respectively.

TABLE VI: The average, maximum, and minimum return as well as the standard deviation across the 200 test episodes for each follower $i$ of FH-DDPG-SS Return | Follower 1 | Follower 2 | Follower 3 | Follower 4
---|---|---|---|---
Average | -0.0600 | -0.0691 | -0.0776 | -0.0835
Maximum | -0.0559 | -0.0644 | -0.0731 | -0.0787
Minimum | -0.0650 | -0.0742 | -0.0833 | -0.0889
Standard deviation | 0.0015 | 0.0016 | 0.0017 | 0.0017

To demonstrate that platoon safety is ensured even in the worst test episode,
it can be observed in Fig. 4 that the followers have an initial gap-keeping
error $e_{pi,0}$ of $1.5$ $\rm m$ and the gap-keeping error is reduced over
time to approximately $0$ $\rm m$. The most negative $e_{pi,k}$ is $-0.1014$
$\rm m$ at $k=35$, which will not result in vehicle collision since the
absolute value of the position error is much smaller than the desired headway.

Figure 4:  $e_{pi,k}$ of the worst test episode for FH-DDPG-SS.

###  V-D String stability

The string stability of a platoon indicates whether oscillations are amplified
upstream the traffic flow. The platoon is called string stable if sudden
changes in the velocity of a preceding vehicle are attenuated by following
vehicles upstream the platoon [51]. To show the string stability of the
proposed FH-DDPG-SS algorithm, we simulate the platoon where the leader
acceleration is set to $2$ $\rm{m/s^{2}}$ when $20<k\leq 30$, and $0$
$\rm{m/s^{2}}$ otherwise. The followers’ initial gap-keeping and velocity
errors are all set to $0$.

Figure 5: Results of a test episode for FH-DDPG-SS in a specific setting.

As shown in Fig.5, the amplitude of the oscillations in $e_{pi,k}$ and
$e_{vi,k}$ of each follower $i\in\\{2,3,4\\}$ are both smaller than those of
its respective predecessor $i-1$, demonstrating the string stability of the
platoon.

##  VI Conclusion

This paper has studied how to solve the platoon control problem using an
integrated DRL and DP method. Firstly, the SSDP model for platoon control has
been formulated with a Huber loss function for the reward. Then, the FH-DDPG-
SS algorithm has been proposed to improve sampling and training efficiency
over the baseline FH-DDPG algorithm with three key ideas. Finally, the
performance of FH-DDPG-SS has been compared with DDPG, FH-DDPG and HCFS based
on real driving data extracted from the NGSIM. The results have shown that FH-
DDPG-SS has learned better policies for platoon control, with significantly
improved performance and better convergence stability. Moreover, the platoon
safety and string stability for FH-DDPG-SS have been demonstrated.

The focus of this paper is a decentralized platoon control algorithm that
optimizes each vehicle’s local performance independently. Our future work
involves extending the proposed algorithm to deal with multi-agent SSDP, where
the objective is to optimize the global performance (i.e., sum of local
performances). In addition, we will take into account the actuator delay and
V2X communication delay by formulating a random delay SSDP, and develop DRL-
based platoon control algorithms with improved performance in the delayed
environment.

###  -A Pseudocode of FH-DDPG Algorithm

The pseudocode of FH-DDPG algorithm [41] is given below as Function
$\mathrm{FH-DDPG}$.

Function 1 $\mathrm{FH-DDPG}$

1: Randomly initialize actor and critic network weights as $\theta^{\mu 0}$
and $\theta^{Q0}$

2: Set $\mu_{i,K}^{*}(S_{i,K})=\mu^{\mathrm{mo}}(S_{i,K})$ for the final time
step $K$

3: for $k=K-1,\cdots,1$ do

4: $\theta^{\mu_{i}}=\theta^{\mu 0}$ and $\theta^{Q_{i}}=\theta^{Q0}$

5: $\theta^{\mu_{i,k}},\theta^{Q_{i,k}}\leftarrow\mathrm{DDPG-
FT}(\theta^{\mu_{i}},\theta^{Q_{i}},\theta^{\mu^{\prime}_{i}},\theta^{Q^{\prime}_{i}},k)$

6: Update the target network:
$\theta^{\mu^{\prime}_{i}}\leftarrow\theta^{\mu_{i,k}},\
\theta^{Q^{\prime}_{i}}\leftarrow\theta^{Q_{i,k}}$

7: end for

8: return $\\{\theta^{\mu_{i,k}},\theta^{Q_{i,k}}\\}_{k=1}^{K}$

Function 2 $\mathrm{DDPG-
FT}(\theta^{\mu_{i}},\theta^{Q_{i}},\theta^{\mu^{\prime}_{i}},\theta^{Q^{\prime}_{i}},k)$

1: Initialize replay buffer $R$

2: Initialize a random process $\mathcal{N}$ for action exploration

3: for episode $e=1,\dots,E$ do

4: Receive state $S_{i,k}^{(e)}$

5: Select action $u_{i,k}^{(e)}$ according to the current policy and
exploration noise

6: Execute action $u_{i,k}^{(e)}$ and observe reward $r_{i,k}^{(e)}$ and
observe new state $S_{i,k+1}^{(e)}$

7: Store transition
$(S_{i,k}^{(e)},u_{i,k}^{(e)},r_{i,k}^{(e)},S_{i,k+1}^{(e)})$ in $R$

8: Sample a random minibatch of $N_{b}$ transitions
$(S_{i,k}^{(n)},u_{i,k}^{(n)},r_{i,k}^{(n)},S_{i,k+1}^{(n)})$ from $R$

9: if $k=K-1$ then

10: Set $y_{i,k}^{(n)}=r_{i,k}^{(n)}+\gamma
r_{K}(S_{i,k+1}^{(n)},\mu^{\mathrm{mo}}(S_{i,k+1}^{(n)}))$

11: else

12: Set $y_{i,k}^{(n)}=r_{i,k}^{(n)}+\gamma
Q^{\prime}_{i}(S_{i,k+1}^{(n)},\mu^{\prime}_{i}(S_{i,k+1}^{(n)}|\theta^{\mu^{\prime}_{i}})|\theta^{Q^{\prime}_{i}})$

13: end if

14: Update critic by minimizing the loss:
$L=\frac{1}{N_{b}}\sum_{i}(y_{i,k}^{(n)}-Q_{i}(S_{i,k}^{(n)},u_{i,k}^{(n)}|\theta^{Q_{i}}))$
$\theta^{Q_{i}}\leftarrow\theta^{Q_{i}}+\beta\bigtriangledown_{\theta^{Q_{i}}}L$

15: Update the actor using the sampled policy gradient:
$\displaystyle\bigtriangledown_{\theta^{\mu_{i}}}J\approx$
$\displaystyle\frac{1}{N_{b}}(\sum_{i}\bigtriangledown_{u}Q_{i}(s,u|\theta^{Q_{i}})|_{s=S_{i,k}^{(n)},u=\mu(S_{i,k}^{(n)})}$
$\displaystyle\bigtriangledown_{\theta^{\mu_{i}}}\mu_{i}(s|\theta^{\mu_{i}})|_{S_{i,k}^{(n)}})$
$\theta^{\mu_{i}}\leftarrow\theta^{\mu_{i}}+\alpha\bigtriangledown_{\theta^{\mu_{i}}}J$

16: end for

17: return $\theta^{\mu_{i}}$, $\theta^{Q_{i}}$

## References

  * [1] R. Silva and R. Iqbal, “Ethical implications of social internet of vehicles systems,” _IEEE Internet of Things Journal_ , vol. 6, no. 1, pp. 517–531, 2019.
  * [2] Y. Chen, C. Lu, and W. Chu, “A cooperative driving strategy based on velocity prediction for connected vehicles with robust path-following control,” _IEEE Internet of Things Journal_ , vol. 7, no. 5, pp. 3822–3832, 2020.
  * [3] V. Lesch, M. Breitbach, M. Segata, C. Becker, S. Kounev, and C. Krupitzer, “An overview on approaches for coordination of platoons,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–17, 2021.
  * [4] Y. Ma, Z. Li, R. Malekian, S. Zheng, and M. A. Sotelo, “A novel multimode hybrid control method for cooperative driving of an automated vehicle platoon,” _IEEE Internet of Things Journal_ , vol. 8, no. 7, pp. 5822–5838, 2021.
  * [5] L. Lei, T. Liu, K. Zheng, and L. Hanzo, “Deep reinforcement learning aided platoon control relying on v2x information,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 6, pp. 5811–5826, 2022.
  * [6] C. Huang, S. Coskun, J. Wang, P. Mei, and Q. Shi, “Robust $h_{\infty}$ dynamic output-feedback control for cacc with rosss subject to rodas,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 1, pp. 137–147, 2022.
  * [7] X. Zhang and T. Wang, “Elastic and reliable bandwidth reservation based on distributed traffic monitoring and control,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 33, no. 12, pp. 4563–4580, 2022.
  * [8] “Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles,” _SAE International_ , p. J3016, 2016.
  * [9] S. E. Li, Y. Zheng, K. Li, Y. Wu, J. K. Hedrick, F. Gao, and H. Zhang, “Dynamical modeling and distributed control of connected and automated vehicles: Challenges and opportunities,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 9, no. 3, pp. 46–58, 2017.
  * [10] H. Guo, J. Liu, Q. Dai, H. Chen, Y. Wang, and W. Zhao, “A distributed adaptive triple-step nonlinear control for a connected automated vehicle platoon with dynamic uncertainty,” _IEEE Internet of Things Journal_ , vol. 7, no. 5, pp. 3861–3871, 2020.
  * [11] S. Öncü, J. Ploeg, N. van de Wouw, and H. Nijmeijer, “Cooperative adaptive cruise control: Network-aware analysis of string stability,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 15, no. 4, pp. 1527–1537, 2014.
  * [12] T. Yang and C. Lv, “A secure sensor fusion framework for connected and automated vehicles under sensor attacks,” _IEEE Internet of Things Journal_ , pp. 1–1, 2021.
  * [13] J. Lan and D. Zhao, “Min-max model predictive vehicle platooning with communication delay,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 11, pp. 12 570–12 584, 2020.
  * [14] Y. Lin and H. L. T. Nguyen, “Adaptive neuro-fuzzy predictor-based control for cooperative adaptive cruise control system,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 3, pp. 1054–1063, 2020.
  * [15] C. Massera Filho, M. H. Terra, and D. F. Wolf, “Safe optimization of highway traffic with robust model predictive control-based cooperative adaptive cruise control,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 18, no. 11, pp. 3193–3203, 2017.
  * [16] E. van Nunen, J. Reinders, E. Semsar-Kazerooni, and N. van de Wouw, “String stable model predictive cooperative adaptive cruise control for heterogeneous platoons,” _IEEE Transactions on Intelligent Vehicles_ , vol. 4, no. 2, pp. 186–196, 2019.
  * [17] Y. Zheng, S. E. Li, K. Li, F. Borrelli, and J. K. Hedrick, “Distributed model predictive control for heterogeneous vehicle platoons under unidirectional topologies,” _IEEE Transactions on Control Systems Technology_ , vol. 25, no. 3, pp. 899–910, 2017.
  * [18] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [19] B. O’Donoghue, I. Osband, R. Munos, and V. Mnih, “The uncertainty bellman equation and exploration,” in _International Conference on Machine Learning_ , 2018, pp. 3836–3845.
  * [20] N. Heess, G. Wayne, D. Silver, T. Lillicrap, T. Erez, and Y. Tassa, “Learning continuous control policies by stochastic value gradients,” in _Advances in Neural Information Processing Systems_ , 2015, pp. 2944–2952.
  * [21] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” _science_ , vol. 313, no. 5786, pp. 504–507, 2006\.
  * [22] L. Lei, Y. Tan, K. Zheng, S. Liu, K. Zhang, and X. Shen, “Deep reinforcement learning for autonomous internet of things: Model, applications and challenges,” _IEEE Communications Surveys Tutorials_ , vol. 22, no. 3, pp. 1722–1760, 2020.
  * [23] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _Nature Publishing Group_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [24] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016.
  * [25] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [26] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International conference on machine learning_ , 2016, pp. 1928–1937.
  * [27] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _International Conference on Machine Learning_ , 2015, pp. 1889–1897.
  * [28] C. Desjardins and B. Chaib-draa, “Cooperative adaptive cruise control: A reinforcement learning approach,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 12, no. 4, pp. 1248–1260, 2011.
  * [29] J. Wang, X. Xu, D. Liu, Z. Sun, and Q. Chen, “Self-learning cruise control using kernel-based least squares policy iteration,” _IEEE Transactions on Control Systems Technology_ , vol. 22, no. 3, pp. 1078–1087, 2014.
  * [30] Z. Huang, X. Xu, H. He, J. Tan, and Z. Sun, “Parameterized batch reinforcement learning for longitudinal control of autonomous land vehicles,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 49, no. 4, pp. 730–741, 2019.
  * [31] M. Buechel and A. Knoll, “Deep reinforcement learning for predictive longitudinal control of automated vehicles,” in _Proc. 21st Int. Conf. Intelligent Transportation Systems (ITSC)_ , 2018, pp. 2391–2397.
  * [32] Z. Li, T. Chu, I. V. Kolmanovsky, and X. Yin, “Training drift counteraction optimal control policies using reinforcement learning: An adaptive cruise control example,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 9, pp. 2903–2912, 2018.
  * [33] S. Wei, Y. Zou, T. Zhang, X. Zhang, and W. Wang, “Design and experimental validation of a cooperative adaptive cruise control system based on supervised reinforcement learning,” _Applied sciences_ , vol. 8, no. 7, p. 1014, 2018.
  * [34] M. Zhu, Y. Wang, Z. Pu, J. Hu, X. Wang, and R. Ke, “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving,” _Transportation Research Part C: Emerging Technologies_ , vol. 117, p. 102662, 2020.
  * [35] Y. Lin, J. McPhee, and N. L. Azad, “Longitudinal dynamic versus kinematic models for car-following control using deep reinforcement learning,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 1504–1510.
  * [36] Y. Lin, J. McPhee, and N. Azad, “Comparison of deep reinforcement learning and model predictive control for adaptive cruise control,” _IEEE Transactions on Intelligent Vehicles_ , vol. 6, no. 2, pp. 221–231, 2021.
  * [37] Y. Zhang, L. Guo, B. Gao, T. Qu, and H. Chen, “Deterministic promotion reinforcement learning applied to longitudinal velocity control for automated vehicles,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 1, pp. 338–348, 2020.
  * [38] G. Wang, J. Hu, Y. Huo, and Z. Zhang, “A novel vehicle platoon following controller based on deep deterministic policy gradient algorithms,” in _CICTP 2018: Intelligence, Connectivity, and Mobility_. American Society of Civil Engineers Reston, VA, 2018, pp. 76–86.
  * [39] T. Chu and U. Kalabić, “Model-based deep reinforcement learning for cacc in mixed-autonomy vehicle platoon,” in _2019 IEEE 58th Conference on Decision and Control (CDC)_. IEEE, 2019, pp. 4079–4084.
  * [40] R. Yan, R. Jiang, B. Jia, J. Huang, and D. Yang, “Hybrid car-following strategy based on deep deterministic policy gradient and cooperative adaptive cruise control,” _IEEE Transactions on Automation Science and Engineering_ , pp. 1–9, 2021.
  * [41] L. Lei, Y. Tan, G. Dahlenburg, W. Xiang, and K. Zheng, “Dynamic energy dispatch based on deep reinforcement learning in iot-driven smart isolated microgrids,” _IEEE Internet of Things Journal_ , p. 1, 2020.
  * [42] P. J. Huber, _Robust statistics_. John Wiley & Sons, 2004, vol. 523.
  * [43] M. L. Puterman, _Markov decision processes: discrete stochastic dynamic programming_. John Wiley & Sons, 2014\.
  * [44] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 22, no. 10, pp. 1345–1359, 2010.
  * [45] T. W. Vossen, F. You, and D. Zhang, “Finite-horizon approximate linear programs for capacity allocation over a rolling horizon,” _Production and Operations Management_ , 2022.
  * [46] S. P. Boyd, “Lecture 1 linear quadratic regulator: Discrete-time finite horizon,” https://stanford.edu/class/ee363/lectures/dlqr, 2009.
  * [47] M. Shah, R. Ali, and F. M. Malik, “Control of ball and beam with lqr control scheme using flatness based approach,” in _2018 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube)_ , 2018, pp. 1–5.
  * [48] “U.d. of transportation,” _NGSIM–next generation simulation_ , 2009.
  * [49] X. Wang, R. Jiang, L. Li, Y. Lin, X. Zheng, and F.-Y. Wang, “Capturing car-following behaviors by deep learning,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 3, pp. 910–920, 2018.
  * [50] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard _et al._ , “Tensorflow: A system for large-scale machine learning,” in _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, 2016, pp. 265–283.
  * [51] G. J. L. Naus, R. P. A. Vugts, J. Ploeg, M. J. G. van de Molengraft, and M. Steinbuch, “String-stable cacc design and experimental validation: A frequency-domain approach,” _IEEE Transactions on Vehicular Technology_ , vol. 59, no. 9, pp. 4268–4279, 2010.