# Co-design of Control and Scheduling in Networked Systems under Denial-of-
Service attacks

Jingyi Lu1 and Daniel E. Quevedo2 This work was supported by the German
Research Foundation (DFG) (project number 3050102303).1Jingyi Lu is with the
Department of Mechanical Engineering, University of Paderborn, Germany.
(Email:jingyi.lu@upb.de)2Daniel E. Quevedo is with the School of Electrical
Engineering & Robotics, Queensland University of Technology (QUT), Brisbane,
Australia<EMAIL_ADDRESS>
###### Abstract

We consider the joint design of control and scheduling under stochastic
Denial-of-Service (DoS) attacks in the context of networked control systems. A
sensor takes measurements of the system output and forwards its dynamic state
estimates to a remote controller over a packet-dropping link. The controller
determines the optimal control law for the process using the estimates it
receives. An attacker aims at degrading the control performance by increasing
the packet-dropout rate with a DoS attack towards the sensor-controller
channel. Assume both the controller and the attacker are rational in a game-
theoretic sense. We establish a partially observable stochastic game to derive
the optimal joint design of scheduling and control. Using dynamic programming
we prove that the control and scheduling policies can be designed separately
without sacrificing optimality, making the problem equivalent to a complete
information game. We employ Nash Q-learning to solve the problem and prove
that the solution is guaranteed to constitute an $\epsilon$-Nash equilibrium.
Numerical examples are provided to illustrate the tradeoffs between control
performance and communication cost.

##  I INTRODUCTION

Cyber-physical systems (CPSs) have successfully integrated computing devices
and networking infrastructure to remotely sense and control the physical
world. This enables the development of exciting applications in manufacturing,
transportation, and smart grid [1]. However, the wireless communication among
components of CPSs introduces vulnerability against malicious adversaries [2].
For example, data integrity and availability can be easily deteriorated by
replay attacks [3] and denial-of-service (DoS) attacks [4]. This weakness
brings a great threat to the safety and security of CPSs with real-time
operations. This raises the issue of how to design systematic prevention
mechanisms to enhance system security.

Game theory is often employed as a tool for resilient design and analysis in
the presence of adversaries [5]. Assuming the rationality of the adversary,
the game-theoretic framework can model the interactive decision-making process
and yield non-conservative strategies. Game theory has been widely applied to
the secure design of control and estimation. For example, Zhu and Başar [6]
formulated an infinite time horizon zero-sum game to derive a secure control
policy that is resilient against various types of cyberattacks. Li _et al._
[7] presented a power control strategy for remote state estimation in the
presence of DoS attacks through a stochastic game. The idea has been recently
extended to address linear quadratic Gaussian control problems in [8]. Miao
_et al._ [9] proposed a stochastic game with hybrid states for designing
optimal switching control policies to counteract data injection attacks.

Figure 1: Networked control system with DoS attacks: a local sensor takes
measurements of the process outputs $y_{k}$, generates a state estimate
$\hat{x}_{k}^{s}$ and transmits the estimate to the remote controller over a
stochastic channel according to the scheduling command $\nu_{k}$. After
receiving the packet, the TCP link sends an acknowledgment signal. The
controller determines the system input $u_{k}$ using the state estimate. An
attacker intends to degrade the control performance by jamming the
communication channel between the controller and the sensor and thereby
increasing the packet dropout probability.

In this work, we employ the game-theoretic framework to study a secure co-
design of control and scheduling to save communication costs and enhance CPS
security. As shown in Fig. 1, we consider a remote control problem where the
system state estimate is sent from the sensor to the controller across a TCP
link. The communication channel suffers from potential DoS attacks launched by
an adversary who intends to degrade the control performance and increase the
controller’s communication costs. Since the controller and the adversary have
opposite interests, we model this interactive decision-making process by
formulating a zero-sum stochastic game with the controller and the adversary
taken as two players.

To date, control-scheduling co-design has been extensively studied without
considering security issues and proved to be an efficient approach to reduce
communication cost in networked control systems [10, 11, 12, 13]. However, due
to the coupling effects of control and scheduling, co-design methods will
commonly lead to optimization problems of increased computational complexity.
Similar issues arise when the co-design is conducted in a game-theoretic
framework. Specifically, to derive the jointly optimal strategy for the
control and scheduling, both the system input and scheduling command have to
be taken as actors in the stochastic game. Since the controller output is
commonly a continuous-valued variable and the scheduling command is a discrete
variable, the resulting game constitutes a game with hybrid states. An
additional difficulty arises in the ”natural” situation where the cost
function is related to the system state $x_{k}$, which is unobserved to both
of the players, leading to a partially observable stochastic game.

To tackle the challenges in computation, in the present work we explore the
possibility of separating the control and scheduling. With the help of dynamic
programming, we establish that when the controller works as the scheduler, the
optimal feedback gain is independent of packet arrival distributions and
realizations and the optimal scheduling command is independent of the system
state estimates. Based on these properties, we can simplify the partially
observable stochastic game with hybrid states into a complete information
stochastic game with countably infinite discrete states. This allows us to use
Nash Q-learning, where we prove that it provides an $\epsilon$-Nash
equilibrium.

We emphasize that, in contrast to existing co-design works, including [8, 14,
15], where the scheduling variable is _a priori_ assumed to be independent of
the system state, in the present work we do not impose any assumptions on the
structure of the solution. Therefore, the current separate design procedure
does not sacrifice optimality.

The contribution of this work is three-fold: (1) we establish a partially
observable stochastic game with hybrid states to derive the jointly optimal
scheduling and control strategy under a DoS attack in the context of the
remote control; (2) we prove that the design of control and scheduling can be
separated without sacrificing optimality and show that the game can be reduced
to a complete information stochastic game with countably finite discrete
states; (3) we show that Nash Q-learning can efficiently solve such problems
and guarantee an $\epsilon$-equilibrium.

The remaining parts of this note are organized as follows: Section II
describes the model of the networked control system and the problem setup;
Section III presents the formulation of the partially observable stochastic
game and analyzes the optimality of the separate design, as well as proves the
$\epsilon$-equilibrium of the resulting solution; Section IV provides a
numerical example to verify the theoretical results and illustrate the
tradeoffs between the communication cost and the control performance. Finally,
Section V draws conclusions.

##  II PROBLEM FORMULATION

###  II-A System model

Consider a linear dynamic system

$\displaystyle x_{k+1}=$ $\displaystyle Ax_{k}+Bu_{k}+w_{k},$ (1)
$\displaystyle y_{k}=$ $\displaystyle Cx_{k}+v_{k},$ (2)

where $x_{k}\in\mathbb{R}^{n_{x}}$, $u_{k}\in\mathbb{R}^{n_{u}}$ and
$y_{k}\in\mathbb{R}^{n_{y}}$ denote the system state, input and output
respectively; $w_{k}\in\mathbb{R}^{n_{x}}$ and $v_{k}\in\mathbb{R}^{n_{y}}$
are i.i.d. Gaussian noises with zero mean, i.e., $\mathbb{E}(w_{k})=0$,
$\mathbb{E}(v_{k})=0$, and
$\mathbb{E}(w_{i}w_{j}^{\top})=\mathbb{E}(v_{i}v_{j}^{\top})=0$ if $i\neq j$.
Denote $\mathbb{E}(w_{k}w_{k}^{\top})=Q$ and
$\mathbb{E}(v_{k}v_{k}^{\top})=R$, with $Q>0$ and $R>0$. Assume the pair
$(A,B)$ is controllable and $(A,C)$ is detectable.

As depicted in Fig. 1, a local sensor takes measurements of the system output
$y_{k}$ and generates a state estimate which is denoted as $\hat{x}_{k}^{s}$.
This estimate is wrapped as a packet and forwarded to the remote controller
over a TCP link; $\nu_{k}\in\\{0,1\\}$ denotes the transmission command that
the sensor commits to. Assume the link has i.i.d. packet dropouts and the
probability of a successful transmission is $\lambda$. Define $\gamma_{k}^{c}$
as an indicator variable of successful transmission. We define

$\displaystyle\gamma_{k}^{c}\triangleq\left\\{\begin{array}[]{ll}0&\text{if a
packet dropout occurs,}\\\ 1&\text{if $\hat{x}_{k}^{s}$ is successfully
received}\end{array}\right.$ (5)

and note that

$\displaystyle\mathbb{P}(\gamma_{k}^{c}=0\mid\nu_{k}=0)=1,$ (6)
$\displaystyle\mathbb{P}(\gamma_{k}^{c}=0\mid\nu_{k}=1)=1-\lambda.$

The remote controller determines the input $u_{k}$ and the scheduling command
$\nu_{k}$ based on the received information. If the packet is successfully
received, then the TCP-link will send an acknowledgment back to the sensor,
from which the sensor can infer the value of $\gamma_{k}^{c}$.

###  II-B State estimation

Denote the information set at the sensor and at the controller at time $k$ as
$\mathcal{I}_{k}^{s}$ and $\mathcal{I}_{k}^{c}$, respectively. According to
the problem setup described above, we have

$\displaystyle\mathcal{I}_{k}^{s}=\big{\\{}y_{0},\dots,y_{k},\hat{x}_{0}^{s},\dots,\hat{x}_{k}^{s},\nu_{0},\dots,\nu_{k},\gamma_{0}^{c},\dots,\gamma_{k}^{c}\big{\\}},$
(7)
$\displaystyle\mathcal{I}_{k}^{c}=\big{\\{}\gamma_{0}^{c}\hat{x}_{0}^{s},\dots,\gamma_{k}\hat{x}_{k}^{s},\nu_{0},\dots,\nu_{k},\gamma_{0}^{c},\dots,\gamma_{k}^{c}\big{\\}},$

so that $\mathcal{I}_{k}^{c}\subset\mathcal{I}_{k}^{s}$. Assume the sensor
knows the control policy. Then, it can infer $u_{k}$ from
$\\{\gamma_{k}^{c}\\}$ for each $k$. Note that the separation principle of
estimation and control holds when both $\\{u_{0},\dots,u_{k}\\}$ and
$\\{y_{0},\dots,y_{k}\\}$ are available to the controller. Thus, despite of
the control strategy, the sensor can run a Kalman filter to obtain the optimal
estimate of the state. Specifically, we have

$\displaystyle\hat{x}_{k}^{s}=$ $\displaystyle
A\hat{x}_{k-1}^{s}+Bu_{k-1}+K_{k}\Big{(}y_{k}-C(A\hat{x}_{k-1}^{s}+Bu_{k-1})\Big{)}$
(8) $\displaystyle\tilde{x}_{k}=$
$\displaystyle(I_{n_{x}}-K_{k}C)A\tilde{x}_{k-1}+(I-KC)w_{k-1}-v_{k-1}$ (9)
$\displaystyle P_{k}^{s}=$
$\displaystyle(I_{n_{x}}-K_{k}C)\Big{(}AP_{k-1}^{s}A^{\top}+Q\Big{)},$ (10)

where $\tilde{x}_{k}=x_{k}-\hat{x}_{k}^{s}$ denotes the state estimation error
at the sensor. In (10) $P_{k}^{s}$ denotes the estimation error covariance
defined as
$P_{k}^{s}=\mathbb{E}\left(\tilde{x}_{k}\tilde{x}_{k}^{\top}\right)$ and

$\displaystyle
K_{k}\triangleq(AP_{k-1}^{s}A^{\top}+Q)C^{\top}\Big{(}C(AP_{k-1}^{s}A^{\top}+Q)C^{\top}+R\Big{)}^{-1}.$

Let $\bar{P}$ denote the steady-state error covariance [16]. Assume that the
Kalman filter has reached the steady state, i.e., $P_{k}^{s}=\bar{P}$.

Taking into account the packet dropouts, the optimal state estimate at the
controller, denoted as $\hat{x}_{k}$, follows the recursion

$\displaystyle\hat{x}_{k}=\left\\{\begin{array}[]{ll}\hat{x}_{k}^{s}&\nu_{k}\gamma_{k}^{c}=1,\\\
\bar{x}_{k}&\nu_{k}\gamma_{k}^{c}=0,\end{array}\right.$ (13)

where

$\bar{x}_{k}=A\hat{x}_{k-1}+Bu_{k-1}.$ (14)

Accordingly, the expected error covariance _at the controller_ , denoted as
$P_{k}$, obeys

$\displaystyle
P_{k}=\left\\{\begin{array}[]{ll}\bar{P}&\nu_{k}\gamma_{k}^{c}=1,\\\
f(P_{k-1})&\nu_{k}\gamma_{k}^{c}=0.\end{array}\right.$ (17)

where $f(X)=AXA^{\top}+Q$. Define $\tau_{k}$ as the holding time, which
indicates the time steps since the last successful transmission, i.e.

$\displaystyle\tau_{k}=\left\\{\begin{array}[]{ll}0&\nu_{k}\gamma_{k}^{c}=1\\\
\tau_{k-1}+1&\nu_{k}\gamma_{k}^{c}=0\end{array}.\right.$ (20)

Given (17) and (20), we can express $P_{k}$ in terms of $\tau_{k}$ and
$\bar{P}$ as $P_{k}=f^{\tau_{k}}(\bar{P})$. Our subsequent analysis uses the
following properties of the state estimate and the estimation error.

###### Lemma 1

The following facts are true:

  * (a)

$\mathbb{E}[(x_{k}-\bar{x}_{k})\bar{x}_{k}^{\top}\mid\bar{x}_{k}]=0$;

  * (b)

$\mathbb{E}[(\hat{x}^{s}_{k}-\bar{x}_{k})\bar{x}_{k}^{\top}\mid\bar{x}_{k}]=0$;

  * (c)

$\mathbb{E}\left((\hat{x}_{k}^{s}-\bar{x}_{k})(\hat{x}_{k}^{s}-\bar{x}_{k})^{\top}\mid\bar{x}_{k}\right)=f^{\tau_{k-1}+1}(\bar{P})-\bar{P}$.

###### Proof:

See Appendix A. ∎

###  II-C Model of the adversary

Consider an adversary that intends to degrade the control performance by
deteriorating the availability of the communication channel. In particular,
the adversary may launch a DoS attack, e.g., by increasing the signal noise
ratio, and decrease the successful transmission probability [7]. We model the
DoS attack as a binary process where the successful transmission probability
is lowered to $\lambda_{a}<\lambda$ when an attack is launched and remains
unchanged otherwise.111The conclusions presented in our work can be easily
extended to adversaries with multi-level attack capabilities.

Let $a_{k}$ be an indicator variable of the DoS attack, i.e.

$\displaystyle a_{k}=\left\\{\begin{array}[]{ll}1&\text{an attack is
launched},\\\ 0&\text{no attack is launched}.\end{array}\right.$ (23)

In view of Eq. (6), we have

$\displaystyle\mathbb{P}\Big{(}\gamma_{k}^{c}=0\mid(\nu_{k},a_{k})=(0,\cdot)\Big{)}=1,$
(24)
$\displaystyle\mathbb{P}\Big{(}\gamma_{k}^{c}=0\mid(\nu_{k},a_{k})=(1,0)\Big{)}=1-\lambda,$
$\displaystyle\mathbb{P}\Big{(}\gamma_{k}^{c}=0\mid(\nu_{k},a_{k})=(1,1)\Big{)}=1-\lambda_{a}.$

From Eq. (20) and Eq. (24), the evolution of the holding time $\tau_{k}$ in
the presence of an attacker can be modeled as a Markov decision process (MDP)
where $\tau_{k}$ is taken as the state, and the pair $(\nu_{k},a_{k})$ is
taken as the action, as illustrated in Fig. 2. State transition probabilities
are computed accordingly as

$\displaystyle\mathbb{P}\Big{(}\tau_{k}=i+1\mid(\tau_{k-1},\nu_{k},a_{k})=(i,0,j)\Big{)}=1,~{}j\in\\{0,1\\},$
(25)
$\displaystyle\mathbb{P}\Big{(}\tau_{k}=0\mid(\tau_{k-1},\nu_{k},a_{k})=(i,1,0)\Big{)}=\lambda,$
$\displaystyle\mathbb{P}\Big{(}\tau_{k}=0\mid(\tau_{k-1},\nu_{k},a_{k})=(i,1,1)\Big{)}=\lambda_{a}.$

In addition, assume that the attacker can overhear the acknowledgments sent
from the controller to the sensor and make use of them to efficiently schedule
the attacks. Denote the information available to the attacker at time $k$ as
$\mathcal{I}_{k}^{a}$. We then have

$\mathcal{I}_{k}^{a}=\\{\gamma_{0},\dots,\gamma_{k}\\}$ (26)

and, when comparing with (7), note that the attacker has significantly less
information than the controller-scheduler.

Figure 2: State transition probabilities: the circle denotes the value of the
state $\tau_{k-1}$; the square denotes the value of the action pair
$(\nu_{k},a_{k})$.

##  III Co-design of control and scheduling

In this section, we will formulate the interactive decision making process as
a partially observable stochastic game and show that, from the controller’s
perspective, the design of the control and scheduling can be conducted
separately without sacrificing optimality. This property simplifies the game
into a complete information stochastic game.

###  III-A Design of the partially observable stochastic game

Consider that the legitimate controller aims at optimizing the control
performance by using the information in $\mathcal{I}_{k}^{c}$ to adjust the
input $u_{k}$ and scheduling the transmission command $\nu_{k}$. In
particular, we evaluate the control performance via a discounted linear
combination of a quadratic term of the state $x_{k}$ and the input $u_{k}$, as
well as the controller’s transmission cost and the negative of the adversary’s
attacking cost:

$J=\frac{1}{K}\sum_{k=0}^{K}\eta^{k}\mathbb{E}\Big{(}x_{k}^{\top}Wx_{k}+u_{k}^{\top}Uu_{k}+c_{s}\nu_{k}-c_{a}a_{k}\Big{)}.$
(27)

Here $\eta\in(0,1)$ denotes the discount factor, $c_{s}$ denotes the cost of
transmission and $c_{a}$ denotes the cost of launching a jamming attack.

Contrarily, the adversary intends to maximize $J$ by smartly scheduling the
attack $a_{k}$ according to its observations $\mathcal{I}_{k}^{a}$.

Assume the two agents select their actions independently. The interaction can
then be formulated as a zero-sum stochastic game. Specifically, we denote the
stochastic game $\mathcal{G}$ as a tuple
$\langle\mathcal{I},\mathcal{S},\\{b^{0}\\},\mathcal{A}_{c},\mathcal{A}_{a},\mathcal{O}_{c},\mathcal{O}_{a},\mathcal{P},\mathcal{C}\rangle$,
where

  * •

$\mathcal{I}$ denotes the number of agents. Here we have two agents, i.e. the
controller and the attacker.

  * •

$\mathcal{S}$ denotes the set of states $s_{k}$. Here we define

$s_{k}=[x_{k}^{\top},\tau_{k-1}]^{\top}$

where $x_{k}$ is the system state vector taking values from the continuous
space $\mathbb{R}^{n_{x}}$ and the holding time $\tau_{k}$ is an integer
taking values from the set $\\{0,1,2,\dots\\}$.

  * •

$\\{b^{0}\\}$ denotes the initial value of the state. Without loss of
generality, assume $x_{0}$ is a zero-mean Gaussian noise and $\tau_{0}=0$.

  * •

$\mathcal{A}_{c}$ denotes the action space of the controller. In our problem,
the controller has two actors, i.e., the system input $u_{k}$ which is
continuous and the scheduling command $\nu_{k}\in\\{0,1\\}$.

  * •

$\mathcal{A}_{a}$ denotes the action space of the attacker, which is
$a_{k}\in\\{0,1\\}$.

  * •

$\mathcal{O}_{c}$ denotes the set of observations available to the controller.
Denote the observation available at time $k$ as $o_{k}^{c}$. Consider that the
information in $\mathcal{I}_{k-1}^{c}$ is available to the controller. From
Eq. (7) and (14), the controller knows $\bar{x}_{k}$, which is an unbiased
estimate of $x_{k}$. Beyond that, the value of $\tau_{k-1}$ can be inferred
from $\\{\gamma_{k-1}^{c}\\}$. After the transmission action $\nu_{k}$ is
determined, the controller has a stochastic access to $\hat{x}_{k}^{s}$.
Therefore, we define $o_{k}^{c}$ as follows.

$o_{k}^{c}=\\{\bar{x}_{k},\tau_{k-1},\nu_{k}\gamma_{k}^{c}\hat{x}_{k}^{s}\\}.$
(28)

  * •

$\mathcal{O}_{a}$ denotes the set of observations available to the attacker.
Since the value of $\tau_{k-1}$ can be inferred from the acknowledgments, we
have

$o_{k}^{a}=\\{\tau_{k-1}\\}.$ (29)

  * •

$\mathcal{P}$ is the transition probabilities. From Eq. (1), we may notice
that the transition of $x_{k}$ depends on $u_{k-1}$ and the distribution of
$w_{k-1}$. The transition probability of $x_{k}$ follows

$\mathbb{P}(x_{k}\mid x_{k-1},u_{k-1})\sim\mathcal{N}(Ax_{k-1}+Bu_{k-1},Q).$

In addition, the evolution of the holding time $\tau_{k}$ depends on the
actions $\nu_{k}$ and $a_{k}$ as given in Eq. (25) and Fig. 2.

  * •

$\mathcal{C}$ denotes the cost function. From Eq. (27), we have the immediate
cost at time $k$ given as

$\displaystyle
c_{k}=x_{k}^{\top}Wx_{k}+u_{k}^{\top}Uu_{k}+c_{s}\nu_{k}-c_{a}a_{k}.$

The controller aims at minimizing the cost while the attacker aims at
maximizing the cost.

In view of Eq. (28) and (29), neither of the two agents has complete
information of the state $s_{k}$. Therefore, the game is essentially a
partially observable stochastic game with a hybrid of discrete and continuous
states [9]. In the following section, we will use dynamic programming ideas to
show that the design of control and scheduling can be conducted separately
without sacrificing optimality. This will enable us to simplify the problem
into a complete information game with countably infinite discrete states.

###  III-B Separation of scheduling and control

In this section, we will study the separation of control and scheduling. We
will fist consider the case where the time horizon $K$ in (27) is finite, and
then extend the conclusions to infinite horizons, $K\rightarrow\infty$. For
the zero-sum stochastic game $\mathcal{G}$ with finite stages, the existence
of a stationary Nash equilibrium and the separability is summarized as
follows:

###### Theorem 1

For the stochastic game $\mathcal{G}$, the stationary Nash equilibrium exists
and the design of the control and scheduling can be conducted separately. In
particular, the optimal scheduling command $\nu_{k}$ solely depends on the
holding time $\tau_{k-1}$;

###### Proof:

See Appendix B. ∎

###### Remark 1

Theorem 1 established that in the present setting, the optimal scheduling
command $\nu_{k}$ is independent of the state estimate. This fundamental
property is often assumed without detailed justification in co-design works,
e.g., [15].

Our result guarantees that the separate design does not sacrifice optimality,
when the controller works as the scheduler. However, if the sensor schedules,
such as considered in [8, 14], then this separation property does not hold. In
fact, consider that $\nu_{k}$ is determined by the sensor, so that
$\hat{x}_{k}^{s}$ and $\bar{x}_{k}$ are available before the decision is made.
The expectations in Eq. (55) (in the appendix) should be computed conditioned
on both $\bar{x}_{k}$ and $\hat{x}_{k}^{s}$, in which case both $\bar{x}_{k}$
and $\hat{x}_{k}^{s}$ will appear in the coefficient terms of $\nu_{k}$.
Therefore, the optimal schedule $\nu_{k}$ will in general depend on
$\bar{x}_{k}$ and $\hat{x}_{k}^{s}$. Since $\nu_{k}$ contains information of
$\hat{x}_{k}^{s}$, the optimal estimate at the controller does not follow Eq.
(13), but becomes a nonlinear estimation problem, as analyzed in [17, 18], and
will further complicate the design of the feedback gain. $\square$.

###  III-C Equivalence to a complete information game

We will next show that the partially observable game $\mathcal{G}$ is
equivalent to a stochastic game with complete information.

###### Corollary 1

The optimal control policy is in the form of

$\displaystyle u_{k}^{\star}=-\eta(B^{\top}\eta
S_{k+1}B+U)^{-1}B^{\top}S_{k+1}A\hat{x}_{k},$ (30)

and the optimal scheduling policy is equivalent to the solution to the minimax
optimization problem

$\displaystyle\min_{\\{\nu_{k}\\}}\max_{\\{a_{k}\\}}$
$\displaystyle\sum_{k=0}^{K-1}\eta^{k}\Big{(}\text{tr}\big{(}(A^{\top}\eta
S_{k+1}A+W-S_{k})f^{\tau_{k}}(\bar{P})\big{)}$
$\displaystyle+c_{s}\nu_{k}-c_{a}a_{k}\Big{)},$ (31)

where $S_{K+1}=W$ and $S_{k}=A^{\top}\eta
S_{k+1}A+W-A^{\top}\eta^{2}S_{k+1}B(\eta
B^{\top}S_{k+1}B+U)^{-1}B^{\top}S_{k+1}A$ with $k=K,\dots,0$.

###### Proof:

See Appendix C. ∎

Corollary 1 simplifies the game into a complete information stochastic game,
where the state $s_{k}$ is reduced to $\tau_{k-1}$ and the immediate cost
becomes

$\displaystyle\hat{c}_{k}=\text{tr}\Big{(}(\eta
A^{\top}S_{k+1}A+W-S_{k})f^{\tau_{k}}(\bar{P})\Big{)}+c_{s}\nu_{k}-c_{a}a_{k}.$

Since both the controller and the attacker can observe the holding time
$\\{\tau_{k}\\}$, the game is of complete information and can be efficiently
solved with dynamic programming for this finite horizon scenario.

Next, by taking limits as $K\rightarrow\infty$, we can derive similar results
for the infinite horizon scenario.

###### Corollary 2

Suppose that

$\lambda_{a}>1-\frac{\eta}{\max_{i}|\sigma_{i}(A)|^{2}}.$ (32)

Then the infinite horizon stochastic game $\mathcal{G}$ has a Nash
equilibrium. The optimal control policy is given as

$\displaystyle u_{k}^{\star}=-\eta(B^{\top}\eta
S_{\infty}B+U)^{-1}B^{\top}S_{\infty}A\hat{x}_{k},$ (33)

where

$S_{\infty}=\eta A^{\top}S_{\infty}A+W\\\ -A^{\top}\eta^{2}S_{\infty}B(\eta
B^{\top}S_{\infty}B+U)^{-1}B^{\top}S_{\infty}A.$

The optimal scheduling variables $\\{\nu_{k}\\}$ and $\\{a_{k}\\}$ can be
derived by solving

$\displaystyle\min_{\\{\nu_{k}\\}}\max_{\\{a_{k}\\}}$
$\displaystyle\lim_{K\rightarrow\infty}\frac{1}{K}\sum_{k=0}^{K-1}\eta^{k}\Big{(}\text{tr}\Big{(}(A^{\top}\eta
S_{\infty}A+W-S_{\infty})f^{\tau_{k}}(\bar{P})\Big{)}$ (34)
$\displaystyle+c_{s}\nu_{k}-c_{a}a_{k}\Big{)}.$

We note that (32) gives a sufficient condition to ensure the boundedness of
(34), such that even if the attacker launches the DoS attack at all time
instances, the expected estimation error covariances are ensured to be
bounded.

Figure 3: The effects of truncation.

###  III-D Nash Q-learning and State truncation

The state $\tau_{k}$ takes values from a set with countably infinite elements.
Since Nash Q-learning can only be applied to MDPs with finite states, we
truncate the MDP in Fig. 2 by aggregating the states with $\tau_{k}\geq N-1$.
Specifically, we modify the transition probability for $\tau_{k}\geq N-1$ as

$\mathbb{P}\big{(}\tau_{k}=N-1\mid(\tau_{k-1},\nu_{k},a_{k})=(N-1,0,j)\big{)}=1$

with $j=0,1$. In this way, we keep $N$ states in the MDP and therefore
facilitate the application of Nash Q-learning. We learn the $Q$-function,
denoted as $Q(\tau,\nu_{k},a_{k})$, for $\tau\in[0,N-1]$ according to the Nash
Q-learning algorithm presented in [7]. Thereafter, the transmission and attack
policies for $\tau\in[0,N-2]$ are derived from (35):

$\displaystyle\pi_{N}^{c}(\tau)=\underset{\pi_{N}^{c}}{\arg\min}\underset{\pi_{N}^{a}}{\max}\underset{\nu_{k},a_{k}}{\sum}Q(\tau,\nu_{k},a_{k})\mathbb{P}(\nu_{k}|\pi_{N}^{c})\mathbb{P}(a_{k}|\pi_{N}^{a}),$
(35)
$\displaystyle\pi_{N}^{a}(\tau)=\underset{\pi_{N}^{a}}{\arg\max}\underset{\pi_{N}^{c}}{\min}\underset{\nu_{k},a_{k}}{\sum}Q(\tau,\nu_{k},a_{k})\mathbb{P}(\nu_{k}|\pi_{N}^{c})\mathbb{P}(a_{k}|\pi_{N}^{a}),$

where $\pi_{N}^{c}(\tau)$ and $\pi_{N}^{a}(\tau)$ give the probabilities of
$\mathbb{P}(\nu_{k})$ and $\mathbb{P}(a_{k})$ for each $\tau\in[0,N-2]$ and
$\nu_{k},a_{k}\in\\{0,1\\}$.

For $\tau\geq N-1$, we set $\mathbb{P}(\nu_{k}=1)=1$ and
$\mathbb{P}(a_{k}=1)=1$. Next, we will show that this truncated policy gives
an $\epsilon$-Nash equilibrium when $N$ is large.

###### Theorem 2

Denote the truncated policy in Eq. (35) for the stochastic game $\mathcal{G}$
as $(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})$. Denote the value of Problem
(34) as $J$. Then, the pair $(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})$ is an
$\epsilon$-Nash equilibrium of the game $\mathcal{G}$, i.e.

$\displaystyle
J(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})\leq\min_{\pi^{c}}J(\pi_{c},\pi_{N}^{a\star})+\epsilon_{N},$
(36) $\displaystyle
J(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})\geq\max_{\pi^{a}}J(\pi_{N}^{c^{\star}},\pi^{a})-\epsilon_{N},$
(37)

where $\lim_{N\rightarrow\infty}\epsilon_{N}=0$.

###### Proof:

See Appendix D. ∎

##  IV Simulation

Consider a linear system with

$\displaystyle A=\left[\begin{array}[]{cc}1.25&0.1\\\
0&1\end{array}\right],\quad B=\left[\begin{array}[]{c}1\\\
2\end{array}\right],\quad C=\left[\begin{array}[]{c}1\\\
1\end{array}\right]^{\top}.$ (44)

The noise covariances are set as $Q=I$ and $R=I$. When there is no attack, the
successful transmission rate is $\lambda=0.9$. At instances when an attack is
launched, the rate is reduced to $\lambda_{a}=0.6$.

Figure 4: Tradeoff between the control performance and the communication cost.
The averaged transmission frequency and attack frequency are defined as
$\big{(}\sum_{i=0}^{\text{Iter}}\nu_{k}\big{)}/\text{Iter}$ and
$\big{(}\sum_{i=0}^{\text{Iter}}a_{k}\big{)}/\text{Iter}$, respectively.

We consider an infinite time horizon co-design problem with weight matrices
$U=I$ and $W=I$, and a discount factor $\eta=0.999$. The transmission cost is
set to $c_{s}=300$ and the attack cost is $c_{a}=200$.

The optimal control law is computed from (33) as
$L=\left[\begin{array}[]{cc}-1.09&-0.1\end{array}\right]$. Based on that, we
employ the Nash Q-learning in [7] to solve the complete information stochastic
game in (31) with the truncation horizon $N$ increased from $3$ to $15$. The
corresponding optimal value of the game is plotted in Fig. 3, showing the
convergence of the value as $N$ is increased.

We next study the tradeoff between the transmission cost and the empirical
control performance, evaluated with

$\log\Big{(}\frac{1}{\text{Iter}}\sum_{i=0}^{\text{Iter}}x_{i}^{\top}Wx_{i}+u_{i}^{\top}Uu_{i}\Big{)},$

where Iter is the length of the simulation. To this end, we run Monte Carlo
simulations with the cost $c_{s}$ and $c_{a}$ screening from $0$ to $2000$
with an interval $100$. Iter is set to be $10^{6}$ for each group of
$(c_{s},c_{a})$. Fig. 4 illustrates that the control performance deteriorates
when the attack frequency increases and the transmission frequency decreases.

##  V Conclusions

In the present work, we have studied the design of control and scheduling in
the presence of stochastic DoS attacks. We have designed a zero-sum partially
observable stochastic to jointly optimize the control and scheduling
strategies. We have established that the optimal transmission scheduling
command is independent of the state estimate and that the design of control
and scheduling can be conducted separately without sacrificing optimality.
Based on this property, the partially observable game is simplified as a
complete information game. Moreover, we have proved that by applying a Nash
Q-learning to the stochastic game with appropriately truncated states, an
$\epsilon$-equilibrium can be derived. Future works may consider employing the
sensor as the scheduler, in which case more information can be exploited for
scheduling and enhanced security can be expected.

## APPENDIX

###  V-A Proof of Lemma 1

###### Proof:

Item (a) is the same as Lemma 4.1 in [19]. We will prove Item (b) and (c).
Assume that the latest successful transmission occurs at time $k_{0}$ where
$k_{0}\leq k$. From Eq. (13), we have

$\displaystyle\bar{x}_{k}=A^{k-k_{0}}\hat{x}_{k_{0}}^{s}+\sum_{i=k_{0}}^{k-1}A^{k-1-i}Bu_{i}.$
(45)

From Eq. (8), we have

$\displaystyle\hat{x}_{k}^{s}=$ $\displaystyle
A\hat{x}_{k-1}^{s}+Bu_{k-1}+\underbrace{KCA\tilde{x}_{k-1}+KCw_{k-1}+v_{k-1}}_{\triangleq
r_{k-1}},$ (46) $\displaystyle=$ $\displaystyle
A^{k-k_{0}}\hat{x}_{k_{0}}^{s}+\sum_{i=k_{0}}^{k-1}A^{k-1-i}Bu_{i}+\sum_{i=k_{0}}^{k-1}A^{k-1-i}r_{i}.$

Eq. (45) and (46) together give that

$\hat{x}_{k}^{s}-\bar{x}_{k}=\sum_{i=k_{0}}^{k-1}A^{k-1-i}r_{i}.$

Since $r_{i}$ is independent of $\hat{x}_{k_{0}}^{s}$ and $u_{i}$ for $i\geq
k_{0}$, $\hat{x}_{k}^{s}-\bar{x}_{k}$ is independent of $\bar{x}_{k}$.
Therefore, we have

$\mathbb{E}[(\hat{x}^{s}_{k}-\bar{x}_{k})\bar{x}_{k}^{\top}\mid\bar{x}_{k}]=0.$

Moreover, from Eq. (1), we have

$\displaystyle
x_{k}=A^{k-k_{0}}x_{k_{0}}+\sum_{i=k_{0}}^{k-1}A^{k-1-i}Bu_{i}+\sum_{i=k_{0}}^{k-1}A^{k-1-i}w_{i},$
(47)
$\displaystyle\tilde{x}_{k}=A^{k-k_{0}}\tilde{x}_{k_{0}}+\sum_{i=k_{0}}^{k-1}A^{k-1-i}(r_{i}-w_{i}).$

Since $\tilde{x}_{k_{0}}$, $w_{i}$ and $v_{i}$ are i.i.d and mutually
independent when $i\geq k_{0}$, we have
$\mathbb{E}(\tilde{x}_{k}\bar{x}_{k})=0$, which further gives that

$\displaystyle\mathbb{E}\left((x_{k}-\hat{x}_{k}^{s})(x_{k}-\hat{x}_{k}^{s})^{\top}\mid\bar{x}_{k}\right)=$
$\displaystyle\mathbb{E}\left((x_{k}-\hat{x}_{k}^{s})(x_{k}-\hat{x}_{k}^{s})^{\top}\right)$
$\displaystyle=$ $\displaystyle\bar{P},$

and

$\displaystyle\mathbb{E}\left((\hat{x}_{k}^{s}-\bar{x}_{k})(\hat{x}_{k}^{s}-\bar{x}_{k})^{\top}\mid\bar{x}_{k}\right)$
$\displaystyle=$
$\displaystyle\mathbb{E}\left((\hat{x}_{k}^{s}-x_{k}+x_{k}-\bar{x}_{k})(\hat{x}_{k}^{s}-x_{k}+x_{k}-\bar{x}_{k})^{\top}\mid\bar{x}_{k}\right)$
$\displaystyle=$
$\displaystyle\mathbb{E}((\hat{x}_{k}^{s}-x_{k})(\hat{x}_{k}^{s}-x_{k})^{\top}+(x_{k}-\bar{x}_{k})(x_{k}-\bar{x}_{k})^{\top}+$
$\displaystyle
2(\hat{x}_{k}^{s}-x_{k})\underbrace{(x_{k}-\bar{x}_{k})^{\top}}_{=x_{k}-\hat{x}_{k}^{s}+\hat{x}_{k}^{s}-\bar{x}_{k}}\mid\bar{x}_{k})$
$\displaystyle=$
$\displaystyle\mathbb{E}((x_{k}-\bar{x}_{k})(x_{k}-\bar{x}_{k})^{\top}-(\hat{x}_{k}^{s}-x_{k})(\hat{x}_{k}^{s}-x_{k})^{\top}\mid\bar{x}_{k})$
$\displaystyle=$ $\displaystyle f^{\tau_{k-1}+1}(\bar{P})-\bar{P}.$

∎

###  V-B Proof of Theorem 1

###### Proof:

Denote the strategy adopted by the controller at stage $k$ as
$\pi^{c}_{k}(o_{k}^{c})$ and the strategy adopted by the attacker at stage $k$
as $\pi_{k}^{a}(o_{k}^{a})$. From the controller’s perspective, the game can
be formulated in a recursive form as

$\displaystyle J_{K+1}^{c}=$
$\displaystyle\mathbb{E}(x_{K+1}^{\top}S_{K+1}x_{K+1}),$ (48)
$\displaystyle\vdots$ $\displaystyle J_{k}^{c}=$
$\displaystyle\inf_{\pi_{k}^{c}(o_{k}^{c})}\sup_{\pi_{k}^{a}(o_{k}^{a})}\mathbb{E}\big{(}c_{k}+\eta
J_{k+1}^{c}\big{)},$

with $k=K-1,\dots,0$. Similarly, from the adversary’s perspective, the game
can be expressed as

$\displaystyle J_{K+1}^{a}=$
$\displaystyle\mathbb{E}(x_{K+1}^{\top}S_{K+1}x_{K+1}),$ (49)
$\displaystyle\vdots$ $\displaystyle J_{k}^{a}=$
$\displaystyle\sup_{\pi_{k}^{a}(o_{k}^{a})}\inf_{\pi_{k}^{c}(o_{k}^{c})}\mathbb{E}\big{(}c_{k}+\eta
J_{k+1}^{a}\big{)},$

with $k=K-1,\dots,0$. Next, we will prove the existence of a stationary Nash
equilibrium by showing that $J_{k}^{c}=J_{k}^{a}$ for all $k\in[0,K]$.

We will start with $k=K$ and follow the principle of dynamic programming. From
Eq. (48), we have

$\displaystyle J_{K}^{c}=$
$\displaystyle\inf_{\pi_{k}^{c}(o_{k}^{c})}\sup_{\pi_{k}^{a}(o_{k}^{a})}\mathbb{E}\big{(}x_{K}^{\top}Wx_{K}+u_{K}^{\top}Uu_{K}+c_{s}\nu_{K}-c_{a}a_{K}$
$\displaystyle+\eta\mathbb{E}(x_{K+1}^{\top}S_{K+1}x_{K+1})\big{)}.$

If $\gamma_{K}^{c}=0$, from Eq. (13), we obtain

$\displaystyle J_{K,\gamma_{K}^{c}=0}^{c}=$
$\displaystyle\min_{u_{k}(\bar{x}_{k})}\mathbb{E}\big{(}x_{K}^{\top}(W+\eta
A^{\top}S_{K+1}A)x_{K}+u_{K}^{\top}(U+$ $\displaystyle\eta
B^{\top}S_{K+1}B)u_{K}+2\eta
u_{K}^{\top}B^{\top}S_{K+1}Ax_{K}\mid\bar{x}_{K}\big{)}+$
$\displaystyle\eta\text{tr}(S_{K+1}Q)+c_{s}\nu_{k}-c_{a}a_{k}.$

According to [20], the optimal input is given as

$\displaystyle u_{K}^{\star}=-\eta(\eta
B^{\top}S_{K+1}B+U)^{-1}B^{\top}S_{K+1}A\bar{x}_{K}.$ (50)

Substitution of (50) into $J_{K}^{c}\mid_{\gamma_{K}^{c}=0}$ provides

$\displaystyle J_{K,\gamma_{K}^{c}=0}^{c}=$
$\displaystyle\bar{x}_{K}^{\top}S_{K}\bar{x}_{K}+\text{tr}((W+\eta
A^{\top}S_{K+1}A)f^{\tau_{K-1}+1}(\bar{P}))$ (51)
$\displaystyle+\eta\text{tr}(S_{K+1}Q)+c_{s}\nu_{k}-c_{a}a_{k}.$

If $\gamma_{K}^{c}=1$, then $\hat{x}_{k}^{s}$ is available to the controller.
Similar to Eq. (50), the optimal control law follows

$\displaystyle u_{K}^{\star}=-\eta(\eta
B^{\top}S_{K+1}B+U)^{-1}B^{\top}S_{K+1}A\hat{x}_{K}^{s}$ (52)

The corresponding value is

$\displaystyle J_{K,\gamma_{K}^{c}=1}^{c}=$
$\displaystyle\hat{x}_{K}^{s\top}S_{K}\hat{x}_{K}^{s}+\text{tr}((W+\eta
A^{\top}S_{K+1}A)f^{0}(\bar{P}))$ (53)
$\displaystyle+\eta\text{tr}(S_{K+1}Q)+c_{s}\nu_{k}-c_{a}a_{k},$

and

$\displaystyle\mathbb{E}(J_{K,\gamma_{K}^{c}=1}^{c}\mid\bar{x}_{k})=$
$\displaystyle\mathbb{E}(\hat{x}_{K}^{s\top}S_{K}\hat{x}_{K}^{s}\mid\bar{x}_{k})+\text{tr}((W+\eta
A^{\top}S_{K+1}$ (54) $\displaystyle
A)f^{0}(\bar{P}))+\eta\text{tr}(S_{K+1}Q)+c_{s}\nu_{k}-c_{a}a_{k}.$

Eqs. (50) and (52) indicate that the optimal feedback gain is independent of
the network properties. In addition, $J_{K}^{c}$ can be expressed as

$\displaystyle J_{K}^{c}=$
$\displaystyle\min_{\nu_{k}(\bar{x}_{K},\tau_{K-1})}\max_{a_{K}(\tau_{K-1})}\mathbb{E}\big{(}(1-\gamma_{K}^{c})J_{K,\gamma_{K}^{c}=0}^{c}+\gamma_{K}^{c}J_{K,\gamma_{K}^{c}=1}^{c}\big{)}.$

From Lemma 1 (b) and (c), we have

$\displaystyle\mathbb{E}\big{(}(1-\gamma_{K}^{c})J_{K,\gamma_{K}^{c}=0}^{c}+\gamma_{K}^{c}J_{K,\gamma_{K}^{c}=1}^{c}~{}\big{|}~{}\bar{x}_{K},\tau_{K-1}\big{)}$
(55) $\displaystyle=$
$\displaystyle\bar{x}_{K}^{\top}S_{K}\bar{x}_{K}+\big{(}1-\nu_{K}(1-a_{K})\lambda-\nu_{K}a_{K}\lambda_{a}\big{)}\text{tr}((W+$
$\displaystyle\eta
A^{\top}S_{K+1}A)f^{\tau_{K-1}+1}(\bar{P}))+\big{(}\nu_{K}(1-a_{K})\lambda+$
$\displaystyle\nu_{K}a_{K}\lambda_{a}\big{)}\text{tr}(S_{K}f^{\tau_{K-1}+1}(\bar{P})+(W+\eta
A^{\top}S_{K+1}-S_{K})\bar{P})$
$\displaystyle+c_{s}\nu_{K}-c_{a}a_{K}+\eta\text{tr}(S_{K+1}Q),$

where the coefficients of $\nu_{K}$ are all functions of $\tau_{K-1}$, showing
that the optimal $\nu_{K}$ is independent of the state estimate $\bar{x}_{K}$.
In addition, we may notice that
$\mathbb{E}\big{(}(1-\gamma_{K}^{c})J_{K,\gamma_{K}^{c}=0}^{c}+\gamma_{K}^{c}J_{K,\gamma_{K}^{c}=1}^{c}~{}\big{|}~{}\bar{x}_{K},\tau_{K-1}\big{)}$
is a linear function of both $\nu_{K}$ and $a_{K}$. According to the minimax
theorem, we have

$\displaystyle\min_{\nu_{K}(\tau_{K-1})}\max_{a_{K}(\tau_{K-1})}\mathbb{E}\big{(}(1-\gamma_{K}^{c})J_{K,\gamma_{K}^{c}=0}^{c}+\gamma_{K}^{c}J_{K,\gamma_{K}^{c}=1}^{c}\big{)}$
$\displaystyle=$
$\displaystyle\max_{a_{K}(\tau_{K-1})}\min_{\nu_{K}(\tau_{K-1})}\mathbb{E}\big{(}(1-\gamma_{K}^{c})J_{K,\gamma_{K}^{c}=0}^{c}+\gamma_{K}^{c}J_{K,\gamma_{K}^{c}=1}^{c}\big{)},$

which is equivalent to $J_{K}^{c}=J_{K}^{a}$, verifying the existence of the
stationary Nash equilibrium.

In a similar way, we can prove that the conclusion holds for any
$k=0,\dots,K$. ∎

###  V-C Proof of Corollary 1

###### Proof:

Since both $\\{\nu_{k}\\}$ and $\\{a_{k}\\}$ solely depend on
$\\{\tau_{k}\\}$, the indicator variable $\gamma_{k}^{c}$ is independent of
the system state. From [15, Theorem 1] and [21], the original minimax
optimization problem is equivalent to

$\displaystyle\max_{\\{a_{k}\\}}\min_{\\{\nu_{k}\\}}\min_{\\{u_{k}\\}}\sum_{k=0}^{K}\eta^{k}c_{k},$
(56)

and the optimal solution to $\min_{\\{u_{k}\\}}\sum_{k=0}^{K}\eta^{k}c_{k}$ is
in the form of Eq. (30). Accordingly, the optimal cost is in the form of

$\displaystyle\text{tr}(S_{0}P_{0})+\sum_{k=0}^{K-1}\eta^{k+1}\text{tr}(S_{k+1}Q)+\sum_{k=0}^{K-1}\eta^{k}\Big{(}\text{tr}\Big{(}(\eta
A^{\top}S_{k+1}A$
$\displaystyle+W-S_{k})f^{\tau_{k}}(\bar{P})\Big{)}+c_{s}\nu_{k}-c_{a}a_{k}\Big{)}.$

Since the first two terms are irrelevant to $\\{\nu_{k}\\}$ and $\\{a_{k}\\}$,
the solution to (31) is the same as that to (56). ∎

###  V-D Proof of Theorem 2

###### Proof:

Define $J_{N}$ as the value of the stochastic game with truncated states, i.e.

$\displaystyle J_{N}=$
$\displaystyle\lim_{K\rightarrow\infty}\frac{1}{K}\sum_{k=0}^{K-1}\eta^{k}\Big{(}\text{tr}\Big{(}M_{\infty}f^{\min(\tau_{k},N)}(\bar{P})\Big{)}$
$\displaystyle+c_{s}\nu_{k}-c_{a}a_{k}\Big{)},$

where $M_{\infty}=A^{\top}\eta S_{\infty}A+W-S_{\infty}$. Since
$\pi_{N}^{c^{\star}}$ and $\pi_{N}^{a^{\star}}$ give the stationary Nash
equilibrium, we have

$\max_{\pi_{N}^{a}}J_{N}(\pi_{N}^{c^{\star}},\pi_{N}^{a})\leq
J_{N}(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})\leq\min_{\pi_{N}^{c}}J_{N}(\pi_{N}^{c},\pi_{N}^{a^{\star}}).$
(57)

Next, we will prove Eq. (36) and (37) by showing that

$\displaystyle\big{|}J_{N}(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})-J(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})\big{|}<\frac{1}{2}\epsilon_{N},$
(58)
$\displaystyle\big{|}\min_{\pi_{N}^{c}}J_{N}(\pi_{N}^{c},\pi_{N}^{a^{\star}})-\min_{\pi^{c}}J(\pi^{c},\pi_{N}^{a^{\star}})\big{|}<\frac{1}{2}\epsilon_{N},$
(59)
$\displaystyle\big{|}\max_{\pi_{N}^{a}}J_{N}(\pi_{N}^{c^{\star}},\pi_{N}^{a})-\max_{\pi^{a}}J(\pi_{N}^{c^{\star}},\pi^{a})\big{|}<\frac{1}{2}\epsilon_{N}.$
(60)

To this end, we define the occupation measure corresponding to the policy
$(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})$ as
$\omega_{N}^{\star}\big{(}\tau,(\nu,a)\big{)}$, which is interpreted as the
total expected discounted time spent in the state-action pairs
$\big{(}\tau,(\nu,a)\big{)}$ (Section 3.1 in [22]). Inspired by the Bellman
equation, the occupation measure satisfies that

$\displaystyle\sum_{0\leq\tau\leq
N-1}\sum_{\nu,a\in\\{0,1\\}}\omega_{N}^{\star}(\tau,(\nu,a))=1$ (61)
$\displaystyle\omega_{N}^{\star}(\tau,(\nu,a))\geq 0$
$\displaystyle\sum_{0\leq\tau\leq
N-1}\sum_{\nu,a\in\\{0,1\\}}\omega_{N}^{\star}(\tau,(\nu,a))(1\\{\tau^{\prime}=\tau\\}-\eta\mathcal{P}_{\tau\\{\nu,a\\}\tau^{\prime}})$
$\displaystyle=(1-\eta)1\\{\tau=\tau^{\prime}\\},~{}\forall
0\leq\tau^{\prime}\leq N-1.$

Here $\mathcal{P}_{\tau\\{\nu,a\\}\tau^{\prime}}$ is a short form for the
transition probability $\mathbb{P}\big{(}s_{k+1}=\tau^{\prime}\mid
s_{k}=\tau,\nu_{k}=\nu,a_{k}=a\big{)}$. Transmission and attack policies can
be derived from the occupation measures according to

$\displaystyle\mathbb{P}\big{(}(\nu_{k},a_{k})\mid\tau\big{)}=\frac{\omega_{N}^{\star}(\tau,(\nu,a))}{\sum_{\nu_{k},a_{k}\in\\{0,1\\}}\omega_{N}^{\star}(\tau,(\nu,a))}.$
(62)

$J_{N}(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})$ can be expressed in terms of
$\omega_{N}^{\star}$ as

$\displaystyle
J_{N}(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})=\sum_{0\leq\tau\leq
N-1}\sum_{\nu,a\in\\{0,1\\}}\omega_{N}^{\star}\big{(}\tau,(\nu,a)\big{)}\text{tr}f^{\tau}\big{(}\bar{P}\big{)}.$
(63)

Similarly, we define $\omega(\tau,(\nu,a))$ as an occupation measure
corresponding to the policy $(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})$ for
the non-truncated stochastic game. Then, we have

$\displaystyle
J(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})=\sum_{\tau\in[0,\infty)}\sum_{\nu,a\in\\{0,1\\}}\omega\big{(}\tau,(\nu,a)\big{)}\text{tr}f^{\tau}\big{(}\bar{P}\big{)}.$
(64)

Since the policies corresponding to
$\\{\omega_{N}^{\star}\big{(}\tau,(\nu,a)\big{)}\\}$ and
$\\{\omega\big{(}\tau,(\nu,a)\big{)}\\}$ are the same and the number of states
are different, from (62), we have

$\displaystyle\omega\big{(}\tau,(\nu,a)\big{)}$
$\displaystyle=\omega_{N}^{\star}\big{(}\tau,(\nu,a)\big{)},\quad\tau\in[0,N),$
$\displaystyle\sum_{i=N}^{\infty}\omega\big{(}i,(\nu,a)\big{)}$
$\displaystyle=\omega_{N}^{\star}\big{(}\tau,(\nu,a)\big{)},\quad\tau=N.$

According to Eq. (32), $J(\pi_{N}^{c^{\star}},\pi_{N}^{a^{\star}})$ is
bounded, indicating that

$\lim_{N\rightarrow\infty}\sum_{\tau\in[N,\infty)}\sum_{\nu,a\in\\{0,1\\}}\omega\big{(}\tau,(\nu,a)\big{)}\text{tr}f^{\tau}\big{(}\bar{P}\big{)}=0.$

Compare Eq. (63) and Eq. (64). We can prove that for any scalar
$\frac{1}{2}\epsilon_{N}>0$, there exists an integer $N$ such that Eq. (58)
holds.

Similarly, we fix the attack policy as $\pi_{N}^{a^{\star}}$. Define the
occupation measure corresponding to the policy $\pi_{N}^{c}$ as
$\omega_{N}^{c}(\tau,\nu)$ for the truncated game and the occupation measure
corresponding to the policy $\pi^{c}$ as $\omega^{c}(\tau,\nu)$ for the non-
truncated game. The value of
$\min_{\pi_{N}^{c}}J_{N}(\pi_{N}^{c},\pi_{N}^{a^{\star}})$ is equivalent to
the value of

$\displaystyle\tilde{J}_{N}=\min_{\omega_{N}^{c}(\tau,\nu)}\sum_{0\leq\tau\leq
N-1}\sum_{\nu\in\\{0,1\\}}\omega_{N}^{c}\big{(}\tau,\nu\big{)}\text{tr}f^{\tau}\big{(}\bar{P}\big{)},$
(65)

and the value of $\min_{\pi^{c}}J(\pi^{c},\pi_{N}^{a^{\star}})$ is equivalent
to the value of

$\displaystyle\tilde{J}=\min_{\omega^{c}(\tau,\nu)}\sum_{\tau\in[0,\infty)}\sum_{\nu\in\\{0,1\\}}\omega^{c}\big{(}\tau,\nu\big{)}\text{tr}f^{\tau}\big{(}\bar{P}\big{)}.$
(66)

Denote the optimal solution to the problem in (66) as
$\omega^{c^{\star}}(\tau,\nu)$, from which we can construct a group of
feasible solutions to the problem in (65) as

$\displaystyle\omega_{N}^{c}(\tau,\nu)=$
$\displaystyle\omega^{c^{\star}}(\tau,\nu),\quad\tau\in[0,N),$ (67)
$\displaystyle\omega_{N}^{c}(\tau,\nu)=$
$\displaystyle\sum_{i=N}^{\infty}\omega^{c^{\star}}(i,\nu),\quad\tau=N.$

Since the value of

$\sum_{0\leq\tau\leq
N-1}\sum_{\nu\in\\{0,1\\}}\omega_{N}^{c}\big{(}\tau,\nu\big{)}\text{tr}f^{\tau}\big{(}\bar{P}\big{)}$

corresponding to the feasible solution in (67) is smaller than $\tilde{J}$,
according to the optimality of $\tilde{J}_{N}$, we have
$\tilde{J}_{N}\leq\tilde{J}$. In the meantime, from the optimal policy derived
from (65), we can construct a feasible solution to (65) using (62). Denote the
corresponding value as $\tilde{J}_{f}$. The optimality of $\tilde{J}$ gives
that $\tilde{J}\leq\tilde{J}_{f}$. Similar to the proof of (58), we can prove
that $|\tilde{J}_{N}-\tilde{J}_{f}|\leq\frac{1}{2}\epsilon_{N}$, which further
establishes (59). The bound in (60) can be proved similarly. ∎

## References

  * [1] K.-D. Kim and P. R. Kumar, “Cyber–physical systems: A perspective at the centennial,” _Proceedings of the IEEE_ , vol. 100, no. Special Centennial Issue, pp. 1287–1308, 2012.
  * [2] M. S. Chong, H. Sandberg, and A. M. Teixeira, “A tutorial introduction to security and privacy for cyber-physical systems,” in _2019 18th European Control Conference (ECC)_. IEEE, 2019, pp. 968–978.
  * [3] Y. Mo and B. Sinopoli, “Secure control against replay attacks,” in _2009 47th annual Allerton conference on communication, control, and computing (Allerton)_. IEEE, 2009, pp. 911–918.
  * [4] H. Zhang, P. Cheng, L. Shi, and J. Chen, “Optimal dos attack scheduling in wireless networked control system,” _IEEE Transactions on Control Systems Technology_ , vol. 24, no. 3, pp. 843–852, 2015.
  * [5] M. H. Manshaei, Q. Zhu, T. Alpcan, T. Bacşar, and J.-P. Hubaux, “Game theory meets network security and privacy,” _ACM Computing Surveys (CSUR)_ , vol. 45, no. 3, pp. 1–39, 2013.
  * [6] Q. Zhu and T. Basar, “Game-theoretic methods for robustness, security, and resilience of cyberphysical control systems: games-in-games principle for optimal cross-layer resilient control systems,” _IEEE Control Systems Magazine_ , vol. 35, no. 1, pp. 46–65, 2015.
  * [7] Y. Li, D. E. Quevedo, S. Dey, and L. Shi, “Sinr-based dos attack on remote state estimation: A game-theoretic approach,” _IEEE Transactions on Control of Network Systems_ , vol. 4, no. 3, pp. 632–642, 2017.
  * [8] J. Zhang, J. Sun, and C. Zhang, “Stochastic game in linear quadratic gaussian control for wireless networked control systems under dos attacks,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , 2020.
  * [9] F. Miao, Q. Zhu, M. Pajic, and G. J. Pappas, “A hybrid stochastic game for secure control of cyber-physical systems,” _Automatica_ , vol. 93, pp. 55–63, 2018.
  * [10] D. Antunes, W. Heemels, and P. Tabuada, “Dynamic programming formulation of periodic event-triggered control: Performance guarantees and co-design,” in _2012 IEEE 51st IEEE conference on decision and control (CDC)_. IEEE, 2012, pp. 7212–7217.
  * [11] E. G. Peters, D. E. Quevedo, and M. Fu, “Controller and scheduler codesign for feedback control over ieee 802.15. 4 networks,” _IEEE Transactions on Control Systems Technology_ , vol. 24, no. 6, 2016.
  * [12] M. Kögel, D. Quevedo, and R. Findeisen, “Combined control and communication scheduling for constrained system using robust output feedback mpc,” in _2019 18th European Control Conference (ECC)_. IEEE, 2019, pp. 1778–1783.
  * [13] S. Wildhagen and F. Allgöwer, “Scheduling and control over networks using mpc with time-varying terminal ingredients,” in _2020 American Control Conference (ACC)_. IEEE, 2020, pp. 1913–1918.
  * [14] S. Knorn and S. Dey, “Optimal energy allocation for linear control with packet loss under energy harvesting constraints,” _Automatica_ , vol. 77, pp. 259–267, 2017.
  * [15] A. S. Leong, D. E. Quevedo, T. Tanaka, S. Dey, and A. Ahlén, “Event-based transmission scheduling and lqg control over a packet dropping link,” _IFAC-PapersOnLine_ , vol. 50, no. 1, pp. 8945–8950, 2017.
  * [16] B. Sinopoli, L. Schenato, M. Franceschetti, K. Poolla, M. I. Jordan, and S. S. Sastry, “Kalman filtering with intermittent observations,” _IEEE transactions on Automatic Control_ , vol. 49, no. 9, pp. 1453–1464, 2004.
  * [17] W. Li, Z. Wang, Q. Liu, and L. Guo, “An information aware event-triggered scheme for particle filter based remote state estimation,” _Automatica_ , vol. 103, pp. 151–158, 2019.
  * [18] E. Kung, J. Wang, J. Wu, D. Shi, and L. Shi, “On the nonexistence of event triggers that preserve gaussian state in presence of packet-drop,” _IEEE Transactions on Automatic Control_ , vol. 65, no. 10, pp. 4302–4307, 2019.
  * [19] L. Schenato, B. Sinopoli, M. Franceschetti, K. Poolla, and S. S. Sastry, “Foundations of control and estimation over lossy networks,” _Proceedings of the IEEE_ , vol. 95, no. 1, pp. 163–187, 2007.
  * [20] V. Gupta, B. Hassibi, and R. M. Murray, “Optimal lqg control across packet-dropping links,” _Systems & Control Letters_, vol. 56, no. 6, pp. 439–446, 2007.
  * [21] D. P. Bertsekas, “Dynamic programming and optimal control 3rd edition, volume ii,” _Belmont, MA: Athena Scientific_ , 2011.
  * [22] E. Altman, _Constrained Markov decision processes_. CRC Press, 1999, vol. 7.