# Quasi-local study of the circular orbits of spin test body in static
spherical symmetric spacetimes

Yong Song111e-mail address<EMAIL_ADDRESS>Kai Diao222e-mail address:
<EMAIL_ADDRESS>Leilei Tang333e-mail address<EMAIL_ADDRESS>Jiabao
Hu444e-mail address<EMAIL_ADDRESS>Xiaofeng Zhao555e-mail address:
<EMAIL_ADDRESS>Shunping Shi666e-mail address<EMAIL_ADDRESS>(corresponding author) College of Mathematics and Physics
Chengdu University of Technology, Chengdu, Sichuan 610059, China

###### Abstract

In this work, we propose a new perspective on the quasi-local studies of the
circular orbits in spacetime. Unlike the definition of the photon surface
given by studying the geometry of the surface, we give a quasi-local
definition of the pole-dipole particle surface in general static spherical
symmetric spacetime based on the condition of the circular orbits in the
effective potential method. From this definition, we study the circular orbits
of a single-pole particle surface in static spherical symmetric spacetime and
get the same result as the previous work. For pole-dipole case, we give an
argumentation of the equivalence of this definition and the effective
potential method.

##  I Introduction

In 2019, the Event Horizon Telescope (EHT) Collaborations published the first
images of a supermassive black hole at the center of the M87 galaxy
EventHorizonTelescope:2019dse . Later, in 2021, the EHT Collaborations
released the polarized images of the black hole EventHorizonTelescope:2021bee
; EventHorizonTelescope:2021srq . Very recently, the EHT announced the image
of the Galactic Center Supermassive Black Hole EventHorizonTelescope:2022xnr .
To analysis these images, it is important to study the circular orbits of the
system EventHorizonTelescope:2019pgp ; EventHorizonTelescope:2022xqj . The
photon sphere, where the location of the circular photon orbits, describes the
boundary of the black hole shadow which corresponding to the shaded part of
the image Synge:1966okc ; Virbhadra:1999nm . The innermost stable circular
orbit (ISCO) describes the inner edge of the accretion disk which
corresponding to the luminous part of the image Abramowicz:2011xu ;
Cardoso:2008bp .

In recent years, the quasi-local studies of photon sphere, photon surface and
their generalization definitions have attracted some attention. The first
quasi-local definition of the photon surface is given by Claudel, Virbhadra,
and Ellis which based on the umbilical hypersurface Claudel:2000yi . Based on
this definition, they studied the photon surfaces in general spherically
symmetric spacetimes. However, there are some problems in this definition
Cao:2019vlu : (i). The definition allows that spacetime, which in the absence
of gravity, exists a photon surface. (ii). From their definition, one can not
to get the boundary condition when solving the equation of the photon surface.
(iii). The umbilical condition, i.e., the shear tensor of a hypersurface is
vanishing, is too restrictive and makes their definition does not work in an
axisymmetric stationary spacetime. The problem (i) and (ii) has been solved by
Cao:2019vlu in general spherically symmetric spacetimes which based on the
codimension-2 surface of the spacetime. For the problem (iii), there are many
generalized studies. Such as Yoshino et al. generalize the photon surface to
be a loosely trapped surface Shiromizu:2017ego and (dynamically) transversely
trapping surface Yoshino:2017gqv ; Yoshino:2019dty ; Galtsov:2019bty ;
Kobialko et al. generalize the photon surface to be a fundamental photon
hypersurfaces and fundamental photon regions Kobialko:2020vqf ;
Kobialko:2021aqg .

Another important object is the spinning extended test body. How to study its
circular orbit quasi-locally is a question worth studying. Here, we only focus
on spinning extended test bodies up to the pole-dipole order. Unlike the point
particle or the single-pole particle, it does not satisfy the geodesic
equation but the Mathisson-Papapetrous-Dixon (MPD) equation Mathisson:1937zz ;
Papapetrou:1951pa ; Tulczyjew:1959 ; Taub:1964 ; Madore:1969 ; Dixon:1964 ;
Dixon:1965:1 ; Dixon:1970:1 ; Dixon:1970:2 ; Dixon:1973:1 ; Dixon:1974:1 ;
Dixon:1979 ; Obukhov:2015eqa ; Semerak:1999qc ; Hackmann:2014tga ;
Steinhoff:2009tk . A direct generalization of the quasi-local definition given
by Claudel:2000yi is hard to check its correctness, and we will discuss this
point in Appendix A. In this paper, inspired by the method of obtaining
circular orbits from the effective potential of the system, we give the quasi-
local definition of the single-pole and pole-dipole particle surface in static
spherically symmetric spacetime. This definition is a direct correspondence of
the condition of the circular orbits in the effective potential method.

This paper is organized as follows: In section II, we will give a breif review
of the equations of motion of spinning extended test bodies, i.e., MPD
equations. In section III, we will give the quasi-local definition of the
pole-dipole particle surface. In section IV, we will give the quasi-local
definition of the single-pole particle surface in static spherical symmetric
spacetime and check its correctness. In section V we will demonstrate the
equivalence of the definition and the effective potential method. Section VI
is devoted to the conclusion and discussion. In Appendix A, we will give a
disussion about the definition of the pole-dipole particle surface based on
the geometry definition of the photon surface.

Convention of this paper: We choose the system of geometrized unit, i.e., set
$G=c=1$. We use the symbol $(\mathcal{M},\nabla_{a},g_{ab})$ to denote a
manifold $\mathcal{M}$ with metric $g_{ab}$ and covariant derivative operator
$\nabla_{a}$, and ($g_{ab},\nabla_{a}$) satisfies the compatibility condition,
i.e., $\nabla_{a}g_{bc}=0$. The abstract index formalism has been used to
clarify some formulas or calculations. The curvature $R_{abcd}$ of the
spacetime is defined by
$R_{abcd}v^{d}=(\nabla_{a}\nabla_{b}-\nabla_{b}\nabla_{a})v_{c}\,$ for an
arbitrary tangent vector field $v^{a}$.

##  II Review the Mathisson-Papapetrous-Dixon (MPD) equations

In this section, we will give a brief review of the Mathisson-Papapetrous-
Dixon (MPD) equations. For more details, one can find in the references
Mathisson:1937zz ; Papapetrou:1951pa ; Dixon:1964 ; Dixon:1965:1 ;
Dixon:1970:1 ; Dixon:1970:2 ; Dixon:1973:1 ; Dixon:1974:1 ; Dixon:1979 .

The equations of motion of spinning extended test bodies up to the pole-dipole
order are given by the MPD equations which read

$\displaystyle\dot{p}^{a}=-\frac{1}{2}R^{a}{}_{bcd}u^{b}S^{cd}\;,$ (1)
$\displaystyle\dot{S}^{ab}=2p^{[a}u^{b]}\;.$ (2)

where $u^{a}=dx^{a}/ds$ is the 4-velocity of the body along its world line,
and the dot denotes the covariant derivative with respect to the proper time
$``s"$, i.e., $``\cdot"=D/ds=u^{a}\nabla_{a}$. The antisymmetric tensor
$S^{ab}$ is the spin tensor and $p^{a}$ is 4-momentum of the test body.

In order to close the system of eqs. (1) and (2), a supplementary condition
has to be imposed. In this work, we focus on the Tulczyjew spin-supplementary
condition Tulczyjew:1959 , i.e.,

$\displaystyle S^{ab}p_{b}=0\;.$ (3)

From eq.(3), it turns out that the canonical momentum and the spin of the body
provide two independent conserved quantities given by the relations
Semerak:1999qc ; Hackmann:2014tga

$\displaystyle p^{a}p_{a}=-M^{2}\;,$ (4)
$\displaystyle\frac{1}{2}S^{ab}S_{ab}=S^{2}\;,$ (5)

where $M$ is the ‘dynamical’, ‘total’ or ‘effective’ rest mass of the body and
$S$ is the spin length of the body. In addition to the conserved quantities
resulting from the Tulczyjew condition, there exist also the conserved
quantities associated to the spacetime symmetries given by the Killing vectors
$\xi^{\mu}$, which can be expressed as

$\displaystyle
p^{a}\xi_{a}-\frac{1}{2}S^{ab}\nabla_{b}\xi_{a}=p^{a}\xi_{a}-\frac{1}{2}S^{ab}\partial_{b}\xi_{a}=\mathrm{constant}\;.$
(6)

##  III Definition

In this section, we will give the quasi-local definition of the pole-dipole
particle surface in static spherical symmetric spacetime. The surface where
the circular orbits of a pole-dipole particle located can be defined as:

###### Definition 1

Let $(\mathcal{M},\nabla_{a},g_{ab})$ is a static spherical symmetric
spacetime. Let $\mathcal{S}$ be a timelike hypersurface of $\mathcal{M}$ and
be foliated by a family of non-evolved codimension-2 surfaces. Let $x^{a}(s)$
is the world line of a pole-dipole particle on $\mathcal{S}$. Let $v^{a}$ be a
unit normal vector to $\mathcal{S}$ and $p^{a}$ be the 4-momentum of the pole-
dipole particle. Then $\mathcal{S}$ is a pole-dipoe particle surface if it
satisfies

$\displaystyle p^{a}v^{b}\nabla_{b}p_{a}=0\;,$ (7)

and the codimension-2 surface can be called the pole-dipole particle sphere.

###### Definition 2

A pole-dipoe particle surface (sphere) is called stable if
$v^{c}\nabla_{c}(p^{a}v^{b}\nabla_{b}p_{a})\geq 0$, and unstabe if
$v^{c}\nabla_{c}(p^{a}v^{b}\nabla_{b}p_{a})<0$.

Below, we give some remarks:

(i). Here, we have assumed that $v^{a}$ is out pointing. Roughly speaking, the
out pointing requirement refers to a direction from the center of the system
to infinity.

(ii). From the condition $v^{c}\nabla_{c}(p^{a}v^{b}\nabla_{b}p_{a})=0$, one
can get the innermost stable circular orbits.

(iii). This definition is given by studying the condition of the circular
orbits in effective potential method. For a detail argument, one can find in
the section of the pole-dipole case.

(iv). The above definitions may have a widely application and they can be
easily generalized to other situations, such as the circular orbits in
electromagnetic systems.

##  IV Single-pole particle

###  IV.1 Definition

In this subsection, we will give the quasi-local definition of the circular
orbits of a single-pole particle in static spherical symmetric spacetime. The
equation of motion of the single-pole particle satisfies the geodesic equation
Steinhoff:2009tk . It is easy reduce the pole-dipole particle surface to
single-pole case as follows:

###### Definition 3

Let $(\mathcal{M},\nabla_{a},g_{ab})$ is a static spherical symmetric
spacetime. Let $\mathcal{S}$ be a timelike hypersurface of $\mathcal{M}$ and
be foliated by a family of non-evolved codimensional-2 spacelike surfaces. Let
$\gamma$ is a timelike or a null geodesic on $\mathcal{S}$. Let $v^{a}$ be a
unit normal vector to $\mathcal{S}$ and $k^{a}$ be the tangent vector of the
geodesic. Then $\mathcal{S}$ is a single-pole particle surface if it satisfies

$\displaystyle k^{a}v^{b}\nabla_{b}k_{a}=0\;,$ (8)

where $k^{a}$ is a timelike or a null vector, and the non-evolved
codimensional-2 spacelike surface can be called the single-pole particle
sphere.

###### Definition 4

A single-pole particle surface (sphere) is called stable if
$v^{c}\nabla_{c}(k^{a}v^{b}\nabla_{b}k_{a})\geq 0$, and unstabe if
$v^{c}\nabla_{c}(k^{a}v^{b}\nabla_{b}k_{a})<0$.

###  IV.2 static spherical symmetric spacetime

Let’s study the circular orbit of a single-pole particle in general static
spherical symmetric spacetimes by using this definition. In the
$\\{t,r,\theta,\phi\\}$ coordinates, the metric of the general static
spherical symmetric spacetimes can be written as

$\displaystyle ds^{2}=-F(r)dt^{2}+H(r)dr^{2}+r^{2}(d\theta^{2}+\sin^{2}\theta
d\phi^{2})$ (9)

where $F$ and $H$ are functions of radial coordinate $r$. Considering the
untraped region, we have $F(r)>0$ and $H(r)>0$. Because of the spherical
symmetry of the line element (9), we can choose the equatorial plane, i.e.,
$\theta=\pi/2$. And we can suppose the component of the tangent vector of the
geodesic to be $k^{a}=\\{k^{t},k^{r},0,k^{\phi}\\}$, where
$k^{t}\;,k^{r}\;,k^{\phi}\;$ are only functions of $r$. From the orthogonal
condition that $k^{a}v_{a}=0$, we have

$\displaystyle k^{r}\sqrt{H}=0\;,$ (10)

Then we have $k^{r}=0$. So, we can choose $k=\\{k^{t},0,0,k^{\phi}\\}$. Along
the geodesic, there are two conserved quatities, i.e.,

$\displaystyle e=k_{a}\bigg{(}\frac{\partial}{\partial
t}\bigg{)}^{a}=-Fk^{t}\;,$ (11) $\displaystyle
l=k_{a}\bigg{(}\frac{\partial}{\partial\phi}\bigg{)}^{a}=r^{2}k^{\phi}\;.$
(12)

Then, we get

$\displaystyle k^{t}=-\frac{e}{F}\;,\qquad k^{\phi}=\frac{l}{r^{2}}\;.$ (13)

From eq.(8), we have the following result

$\displaystyle-\frac{2l^{2}}{r^{3}}+\frac{F^{\prime}e^{2}}{F^{2}}=0\;.$ (14)

If the single-pole particle is a photon, the surface where the circular orbits
located is usually called the photon sphere. From the normalized condition of
the null geodesic, i.e., $k^{a}k_{a}=0$, we have

$\displaystyle-\frac{e^{2}}{F}+\frac{l^{2}}{r^{2}}=0\;,$ (15)

Then, combining eq.(14) and (15), we get the photon sphere satisfies

$\displaystyle\frac{e^{2}}{l^{2}}=\frac{F}{r^{2}}\;,\quad 2F=rF^{\prime}\;,$
(16)

which is consistent with the result in Cardoso:2008bp 777It should be noticed
that the signature of the metric in Cardoso:2008bp is $``(+,-,-,-)"$ and the
signature in this paper is $``(-,+,+,+)"$. So, the result between the two may
be differ by a minus sign.. From definition 3, we have

$\displaystyle
v^{c}\nabla_{c}(k^{a}v^{b}\nabla_{b}k_{a})=-\frac{1}{2}\frac{l^{2}}{r^{4}FH}\big{[}2F-r^{2}F^{\prime\prime}\big{]}\;,$
(17)

which is proportional to the second derivative of the potential of the null
geodesic in Cardoso:2008bp .

If the single-pole particle is a timelike point particle, from the normalized
condition of the timelike geodesic, i.e., $k^{a}k_{a}=-1$, we have

$\displaystyle-\frac{e^{2}}{F}+\frac{l^{2}}{r^{2}}=-1\;,$ (18)

Then, combining eq.(14) and (18), we get the circular orbits of a timelike
geodesic satisfy

$\displaystyle e_{o}^{2}=\frac{2F^{2}}{2F-rF^{\prime}}\;,\quad
l_{o}^{2}=\frac{r^{3}F^{\prime}}{2F-rF^{\prime}}\;,$ (19)

which is consistent with the result in Cardoso:2008bp . From definition 4, we
have

$\displaystyle
v^{c}\nabla_{c}(k^{a}v^{b}\nabla_{b}k_{a})=-\frac{1}{FH}\frac{-3FF^{\prime}/r+2F^{\prime
2}-FF^{\prime\prime}}{2F-rF^{\prime}}\;,$ (20)

which is proportional to the second derivative of the potential of the
timelike geodesic in Cardoso:2008bp .

##  V pole-dipole particle surface

In this section, we will illustrate the equivalence between the effective
potential method and the quasi-local definition to study the circular orbits
of a pole-dipole particle in the general static spherical symmetric
spacetimes.

Because of the spherical symmetry of the line element (9), we can choose the
equatorial plane, i.e., $\theta=\pi/2$. And we can suppose 4-momentum of the
pole-dipole particle to be $p^{a}=\\{p^{t},p^{r},0,p^{\phi}\\}$, where
$p^{t}\;,p^{r}\;,p^{\phi}\;$ are only functions of $r$. Along the world line,
there are two conserved quantities for the pole-dipole particle, i.e., the
energy $E$ and the angular momentum $L$. From eq.(6), the conserved quantities
can be expressed as

$\displaystyle-E=p_{t}+\frac{1}{2}F^{\prime}S^{tr}\;,$ (21) $\displaystyle
L=p_{\phi}+rS^{r\phi}\;.$ (22)

where a prime denotes the derivative with respect to radial coordinate $r$.
From the Tulczyjew spin supplementary condition (3), we have

$\displaystyle S^{t\phi}=-\frac{p_{r}}{p_{\phi}}S^{tr}\;,$ (23) $\displaystyle
S^{r\phi}=\frac{p_{t}}{p_{\phi}}S^{tr}\;.$ (24)

From eq.(4), we can get

$\displaystyle(p^{r})^{2}=\frac{1}{FH}p_{t}^{2}-\frac{1}{H}\bigg{(}\frac{p_{\phi}^{2}}{r^{2}}+\mathcal{M}^{2}\bigg{)}\;.$
(25)

Combining eq.(23), (24) and (25), from the spin conservation eq.(5), we have

$\displaystyle S^{tr}=\frac{sp_{\phi}}{\sqrt{FH}r}\;,$ (26)

where $s=S/\mathcal{M}$ is specific spin parameter. It should be noted that
$s$ can have both negative and positive values depending on the direction of
spin with respect to direction of $p_{\phi}$. From the conservation of energy
(21) and angular momentum (22), we have

$\displaystyle
p_{t}=-\frac{2rFHE+F^{\prime}\sqrt{FH}sL}{2rFH-F^{\prime}s^{2}}\;,$ (27)
$\displaystyle p_{\phi}=\frac{2r[FHL+\sqrt{FH}sE]}{2rFH-F^{\prime}s^{2}}\;.$
(28)

###  V.1 Effictive potential method

In this subsection, we will review the effective potential method to get the
circular orbits of the pole-dipole particles Toshmatov:2020wky .

Putting eq.(27) and (28) into eq.(25), we get the result that

$\displaystyle(p^{r})^{2}=A(E-V_{+})(E-V_{-})\;,$ (29)

where

$\displaystyle A=\frac{4F(r^{2}H-s^{2})}{(2FHr-F^{\prime}s^{2})^{2}}$ (30)

and

$\displaystyle
V_{\pm}=\frac{(2F-rF^{\prime})\sqrt{FH}sL}{2(FHr^{2}-Fs^{2})}\pm\frac{F^{\prime}s^{2}-2FHr}{FH(s^{2}-Hr^{2})}\sqrt{H(L^{2}+M^{2}r^{2})-M^{2}s^{2}}\;.$
(31)

which is consistent with the result in Toshmatov:2020wky . According to
eq.(29), the energy of the particle must satisfy the conditions

$\displaystyle E\geq V_{+}\;,\quad\mathrm{or}\quad E\leq V_{-}\;,$ (32)

in order to have $(p^{r})^{2}\geq 0$. Below, we focus on the case of the pole-
dipole particle with positive energy which coincides with the effective
potential to be $V_{\mathrm{eff}}=V_{+}$. The particle moves along a circular
orbit when two conditions are satisfied simultaneously which has been pointed
out in Jefremov:2015gza ; Toshmatov:2020wky :

(1). The particle has zero radial velocity, i.e.

$\displaystyle\frac{dr}{d\lambda}=0\;,\quad\Longrightarrow
V_{\mathrm{eff}}=E\;,$ (33)

(2). The particle has zero radial acceleration, i.e.

$\displaystyle\frac{d^{2}r}{d\lambda^{2}}=0,\quad\Longrightarrow\frac{dV_{\mathrm{eff}}}{dr}=0\;.$
(34)

and the stability condition is

$\frac{d^{2}V_{\mathrm{eff}}}{dr^{2}}=\left\\{\begin{array}[]{ll}<0&\textrm{unstable}\;,\\\
\\\ \geq 0&\textrm{stable}\;.\end{array}\right.$

When $d^{2}V_{\mathrm{eff}}/dr^{2}=0$, the circular orbits correspond to the
innermost stable circular orbit (ISCO).

###  V.2 Quasi-local study of the circular orbits

In this subsection, we will use the definition 1 and 2 to study the circular
orbits of a pole-dipole particle in static spherical symmetric spacetime.

In static spherical symmetric spacetime, the circular orbits of the pole-
dipole particles are not evolved, and they satisfy the condition that
$r_{o}=\mathrm{constant}$, where $r_{o}$ is the location of the circular
orbit. Then, the normal vector $v^{a}$ of the particle surface can be written
as

$\displaystyle v^{a}=\frac{1}{\sqrt{H}}\bigg{(}\frac{\partial}{\partial
r}\bigg{)}^{a}\;.$ (35)

From the orthogonal condition that $p^{a}v_{a}=0$, we have

$\displaystyle p^{r}\sqrt{H}=0\;,$ (36)

Then we get $p^{r}=0$. So, we can choose $p^{a}=\\{p^{t},0,0,p^{\phi}\\}$.
From eq.(4), we have the following result

$\displaystyle\frac{(4FHs^{2}-4FH^{2}r^{2})E^{2}+(8FH\sqrt{FH}sL-4H\sqrt{FH}F^{\prime}rsL)E+4F^{2}H^{2}L^{2}-HF^{\prime
2}s^{2}L^{2}}{(2FHr-s^{2}F^{\prime})^{2}}=-\mathcal{M}^{2}\;,$ (37)

Organizing the above result, we get

$\displaystyle B(E-V_{+})(E-V_{-})=0\;,$ (38)

where $V_{\pm}$ have been given by eq.(31), and

$\displaystyle B=\frac{4FH(s^{2}-r^{2}H)}{(2FHr-s^{2}F^{\prime})^{2}}\;.$ (39)

From eq.(7), we have

$\displaystyle
p^{a}v^{b}\nabla_{b}p_{a}=\frac{1}{2}v^{b}\nabla_{b}(p_{a}p^{a})=\frac{1}{2}v^{b}\partial_{b}(p_{a}p^{a})=\frac{1}{2}v^{r}\partial_{r}(p_{\mu}p^{\mu})\;.$
(40)

Considering the result in eq.(37) and (38), we can get

$\displaystyle\partial_{r}(p_{\mu}p^{\mu})=\partial_{r}(p_{\mu}p^{\mu}+\mathcal{M}^{2})=\partial_{r}[B(E-V_{+})(E-V_{-})]=-B(E-V_{-})\frac{\partial
V_{+}}{\partial r}=0\;,$ (41)

Then, we have

$\displaystyle\frac{\partial V_{+}}{\partial r}=\frac{\partial
V_{\mathrm{eff}}}{\partial r}=0\;.$ (42)

Further, it is not hard to find that the stability condition, i.e.,
$v^{c}\nabla_{c}(p^{a}v^{b}\nabla_{b}p_{a})\geq 0$, is equivalent to the
following condition

$\displaystyle\frac{\partial^{2}V_{+}}{\partial
r^{2}}=\frac{\partial^{2}V_{\mathrm{eff}}}{\partial r^{2}}\geq 0\;.$ (43)

Using the condition that $v^{c}\nabla_{c}(p^{a}v^{b}\nabla_{b}p_{a})=0$, one
can get the ISCO of the pole-dipole particle in static spherical symmetric
spacetime.

Here, we make a summary of this section: In this section, we illustrated the
equivalence between the effective potential method and the quasi-local
definition in the study of the circular orbits of a pole-dipole particle in
the general static spherical symmetric spacetimes.

(1). The condition $V_{\mathrm{eff}}=E$ is equivalent to

$\displaystyle p^{a}v_{a}=0\;.$ (44)

(2). The condition $dV_{\mathrm{eff}}/dr=0$ is equivalent to

$\displaystyle p^{a}v^{b}\nabla_{b}p_{a}=0\;.$ (45)

(3). The stability condition $d^{2}V_{\mathrm{eff}}/dr^{2}\geq 0$ is
equivalent to

$\displaystyle v^{c}\nabla_{c}(p^{a}v^{b}\nabla_{b}p_{a})\geq 0\;.$ (46)

##  VI Discussion and conclusion

In the Appendix A, we give a possible definition of the pole-dipole particle
surface based on the definition of the photon surface given by Claudel:2000yi
. If this generalized definition is right, it can be applied to quite general
spacetimes. But by this definition, because of the complicated calculation of
the result, it is hard to check its correctness even in Schwarzschild
spacetime. So, we need to find a simple way to get the quasi-local definition
of the pole-dipole particle surface.

In this paper, enlightened by the effective potential method to obtain the
circular orbits of the spacetime, we give a quasi-local definition of single-
pole and pole-dipole particle surface in general static spherical symmetric
spacetime. These definitions provide new insights into the quasi-local study
of circular orbits of the spacetime. We argued the equivalence of the
definition of the pole-dipole particle surface with the effective potential
method, at least in static spherically symmetric spacetime. Also, these
definitions can be easily generalized to other situations, such as the
circular orbits in electromagnetic systems.

Although this definition can be applied to the static spherically symmetric
spacetime, it is hard to generalize to dynamical spacetime and difficult to
deal with the result in real calculation. So, it is important to find a way to
simplify the calculation.

## Acknowledgement

This project is supported by the Open Research Fund of Computational Physics
Key Laboratory of Sichuan Province, Yibin University (Grant No. YBXYJSWL-
ZD-2020-005). This work is also supported by the Student’s Platform for
Innovation and Entrepreneurship Training Program (No. S202110616084).

##  Appendix A Quasi-local definition of the pole-diople particle surface
based on the concept of the photon surface

Based on the quasi-local definition of the photon surface given by
Claudel:2000yi , one may give the following possible definition for a pole-
dipole particle surface:

###### Definition 5

Let $\mathcal{S}$ be a timelike hypersurface of
$(\mathcal{M},\nabla_{a},g_{ab})$. Let $x^{a}(s)$ is the world line of a pole-
dipole particle on $\mathcal{S}$ and $u^{a}=dx^{a}/ds$ is the 4-velocity of
the body along its world line. Let $v^{a}$ be a unit normal vector to
$\mathcal{S}$ and $p^{a}$ be the 4-momentum of the pole-dipole particle. Then
$\mathcal{S}$ is a pole-dipoe particle surface if it satisfies

$\displaystyle-\frac{1}{2}R^{a}{}_{bcd}u^{b}S^{cd}v_{a}+p^{a}u^{b}\nabla_{b}v_{a}=0\;,$
(47)

where $S^{ab}$ is the spin tensor.

Below, we give some discussion about this definition:

(i). The condition (47) can be obtained by the following consideration: The
condition of the photon surface in Claudel:2000yi can be expressed as

$\displaystyle K_{ab}k^{a}k^{b}=0\;,$ (48)

where $K_{ab}$ is the second fundamental form of the photon surface and
$k^{a}$ is the tangent vector of a null geodesic. Consider the normal vector
of the particle surface is $v^{a}$, eq.(48) can be obtained by

$\displaystyle k^{a}\nabla_{a}(k^{b}v_{b})=0\Rightarrow K_{ab}k^{a}k^{b}=0\;,$
(49)

where we have used the geodesic equation of $k^{a}$. Then, for a pole-dipole
particle, following a similar consideration, one may get the following
condition for the pole-dipole particle surface,

$\displaystyle
u^{b}\nabla_{b}(p^{a}v_{a})=0\Rightarrow-\frac{1}{2}R^{a}{}_{bcd}u^{b}S^{cd}v_{a}+p^{a}u^{b}\nabla_{b}v_{a}=0\;,$
(50)

where we have used the MPD equations. Further, Combining the relation
Semerak:1999qc ; Obukhov:2010kn

$\displaystyle
u^{a}=\frac{m}{M^{2}}\bigg{(}p^{a}+\frac{2S_{ab}R_{bcde}p^{c}S^{de}}{4M^{2}+R_{abcd}S^{ab}S^{cd}}\bigg{)}\;,$
(51)

where $m:=-p^{a}u_{a}$ is a scalar parameter (the ‘kinematical’ or ‘monopole’
rest mass of a particle), one can get the equation of the pole-dipole particle
surface.

(ii). This definition is a natural generalization of the condition of a photon
surface. If this definition is correct, it holds not only for static
spherically symmetric spacetimes, but also for arbitrary spacetimes. However,
one can check that, although it can be utilized to obtain the closed equation
of the pole-dipole particle surface, it is difficult to check whether the
result is right or not even in Schwarzschild spacetime. So, in this paper, we
used a different approach to study the pole-dipole particle surface.

## References

  * (1) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 875, L1 (2019) doi:10.3847/2041-8213/ab0ec7 [arXiv:1906.11238 [astro-ph.GA]].
  * (2) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 910, no.1, L12 (2021) doi:10.3847/2041-8213/abe71d [arXiv:2105.01169 [astro-ph.HE]].
  * (3) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 910, no.1, L13 (2021) doi:10.3847/2041-8213/abe4de [arXiv:2105.01173 [astro-ph.HE]].
  * (4) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 930, no.2, L12 (2022) doi:10.3847/2041-8213/ac6674
  * (5) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 875, no.1, L5 (2019) doi:10.3847/2041-8213/ab0f43 [arXiv:1906.11242 [astro-ph.GA]].
  * (6) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 930, no.2, L17 (2022) doi:10.3847/2041-8213/ac6756
  * (7) J. L. Synge, Mon. Not. Roy. Astron. Soc. 131, no.3, 463-466 (1966) doi:10.1093/mnras/131.3.463
  * (8) K. S. Virbhadra and G. F. R. Ellis, Phys. Rev. D 62, 084003 (2000) doi:10.1103/PhysRevD.62.084003 [arXiv:astro-ph/9904193 [astro-ph]].
  * (9) M. A. Abramowicz and P. C. Fragile, Living Rev. Rel. 16, 1 (2013) doi:10.12942/lrr-2013-1 [arXiv:1104.5499 [astro-ph.HE]].
  * (10) V. Cardoso, A. S. Miranda, E. Berti, H. Witek and V. T. Zanchin, Phys. Rev. D 79, no.6, 064016 (2009) doi:10.1103/PhysRevD.79.064016 [arXiv:0812.1806 [hep-th]].
  * (11) C. M. Claudel, K. S. Virbhadra and G. F. R. Ellis, J. Math. Phys. 42, 818-838 (2001) doi:10.1063/1.1308507 [arXiv:gr-qc/0005050 [gr-qc]].
  * (12) L. M. Cao and Y. Song, Eur. Phys. J. C 81, no.8, 714 (2021) doi:10.1140/epjc/s10052-021-09502-0 [arXiv:1910.13758 [gr-qc]].
  * (13) T. Shiromizu, Y. Tomikawa, K. Izumi and H. Yoshino, PTEP 2017, no.3, 033E01 (2017) doi:10.1093/ptep/ptx022 [arXiv:1701.00564 [gr-qc]].
  * (14) H. Yoshino, K. Izumi, T. Shiromizu and Y. Tomikawa, PTEP 2017, no.6, 063E01 (2017) doi:10.1093/ptep/ptx072 [arXiv:1704.04637 [gr-qc]].
  * (15) H. Yoshino, K. Izumi, T. Shiromizu and Y. Tomikawa, PTEP 2020, no.2, 023E02 (2020) doi:10.1093/ptep/ptz161 [arXiv:1909.08420 [gr-qc]].
  * (16) D. V. Gal’tsov and K. V. Kobialko, Phys. Rev. D 99, no.8, 084043 (2019) doi:10.1103/PhysRevD.99.084043 [arXiv:1901.02785 [gr-qc]].
  * (17) K. V. Kobialko and D. V. Gal’tsov, Eur. Phys. J. C 80, no.6, 527 (2020) doi:10.1140/epjc/s10052-020-8070-z [arXiv:2002.04280 [gr-qc]].
  * (18) K. Kobialko, I. Bogush and D. Gal’tsov, Phys. Rev. D 104, no.4, 044009 (2021) doi:10.1103/PhysRevD.104.044009 [arXiv:2104.02167 [gr-qc]].
  * (19) M. Mathisson, Acta Phys. Polon. 6, 163-2900 (1937)
  * (20) A. Papapetrou, Proc. Roy. Soc. Lond. A 209, 248-258 (1951) doi:10.1098/rspa.1951.0200
  * (21) W. Tulczyjew. Acta Phys. Pol., 18:393, 1959.
  * (22) A. H. Taub. J. Math. Phys. (N.Y.), 5:112, 1964.
  * (23) W. G. Dixon. Nuovo Cimento, 34:317, 1964.
  * (24) W. G. Dixon. PhD. thesis, University of Cambridge, 1965.
  * (25) J. Madore. Ann. Inst. Henri Poincaré, A11:221, 1969.
  * (26) W. G. Dixon. Proc. R. Soc. A, 314:499, 1970a.
  * (27) W. G. Dixon. Proc. R. Soc. A, 319:509, 1970b.
  * (28) W. G. Dixon. Gen. Rel. Grav., 4:199, 1973.
  * (29) W. G. Dixon. Phil. Trans. Roy. Soc. Lond. A, 277:59, 1974.
  * (30) W. G. Dixon. Isolated gravitating systems in General Relativity, Proceedings of the International School of Physics “Enrico Fermi,” Course LXVII, edited by J. Ehlers, North Holland, Amsterdam, page 156, 1979.
  * (31) Y. N. Obukhov and D. Puetzfeld, Fund. Theor. Phys. 179, 67-119 (2015) doi:10.1007/978-3-319-18335-0_2 [arXiv:1505.01680 [gr-qc]].
  * (32) O. Semerak, Mon. Not. Roy. Astron. Soc. 308, 863-875 (1999) doi:10.1046/j.1365-8711.1999.02754.x
  * (33) E. Hackmann, C. Lämmerzahl, Y. N. Obukhov, D. Puetzfeld and I. Schaffer, Phys. Rev. D 90, no.6, 064035 (2014) doi:10.1103/PhysRevD.90.064035 [arXiv:1408.1773 [gr-qc]].
  * (34) J. Steinhoff and D. Puetzfeld, Phys. Rev. D 81, 044019 (2010) doi:10.1103/PhysRevD.81.044019 [arXiv:0909.3756 [gr-qc]].
  * (35) B. Toshmatov, O. Rahimov, B. Ahmedov and D. Malafarina, Eur. Phys. J. C 80, no.7, 675 (2020) doi:10.1140/epjc/s10052-020-8254-6 [arXiv:2003.09227 [gr-qc]].
  * (36) P. I. Jefremov, O. Y. Tsupko and G. S. Bisnovatyi-Kogan, Phys. Rev. D 91, no.12, 124030 (2015) doi:10.1103/PhysRevD.91.124030 [arXiv:1503.07060 [gr-qc]].
  * (37) Y. N. Obukhov and D. Puetzfeld, Phys. Rev. D 83, 044024 (2011) doi:10.1103/PhysRevD.83.044024 [arXiv:1010.1451 [gr-qc]].


# A Relevance Model for Threat-Centric Ranking of Cybersecurity
Vulnerabilities ††thanks: This work was unfunded and performed as part of
McCoy’s Ph.D. dissertation work at Old Dominion University.

Corren McCoy

G2 Ops
Virginia Beach, VA 23452
<EMAIL_ADDRESS>
Ross Gore

Office of Enterprise Research and Innovation
Old Dominion University
Suffolk, VA 23435
<EMAIL_ADDRESS>
Michael L. Nelson, Michele C. Weigle

Dept. of Computer Science
Old Dominion University
Norfolk, VA
{mln<EMAIL_ADDRESS>

###### Abstract

The relentless process of tracking and remediating vulnerabilities is a top
concern for cybersecurity professionals. The key challenge is trying to
identify a remediation scheme specific to in-house, organizational objectives.
Without a strategy, the result is a patchwork of fixes applied to a tide of
vulnerabilities, any one of which could be the point of failure in an
otherwise formidable defense. Given that few vulnerabilities are a focus of
real-world attacks, a practical remediation strategy is to identify
vulnerabilities likely to be exploited and focus efforts towards remediating
those vulnerabilities first. The goal of this research is to demonstrate that
aggregating and synthesizing readily accessible, public data sources to
provide personalized, automated recommendations for organizations to
prioritize their vulnerability management strategy will offer significant
improvements over using the Common Vulnerability Scoring System (CVSS). We
provide a framework for vulnerability management specifically focused on
mitigating threats using adversary criteria derived from MITRE ATT&CK. We test
our approach by identifying vulnerabilities in software associated with six
universities and four government facilities. Ranking policy performance is
measured using the Normalized Discounted Cumulative Gain (nDCG). Our results
show an average 71.5% - 91.3% improvement towards the identification of
vulnerabilities likely to be targeted and exploited by cyber threat actors.
The return on investment (ROI) of patching using our policies results in a
savings of 23.3% - 25.5% in annualized costs. Our results demonstrate the
efficacy of creating knowledge graphs to link large data sets to facilitate
semantic queries and create data-driven, flexible ranking policies.

_K_ eywords Cyber attack $\cdot$ Cyber threat intelligence $\cdot$ Data
integration $\cdot$ Data analysis $\cdot$ Open data $\cdot$ Ranking $\cdot$
Vulnerability Management

##  1 Introduction

The relentless process of tracking and prioritizing vulnerabilities for
patching is a top concern for cybersecurity professionals [1]. Ideally every
organization would apply the security updates for their operating systems and
critical applications as soon as possible after updates are released. However,
since patches from top vendors are delivered in monthly blocks on Patch
Tuesday, system administrators often find it difficult to select which patches
to apply and identify which ones are not applicable [2, 3]. Patch Tuesday is
the term used to refer to second Tuesday of each month when Microsoft, Adobe,
Oracle, and others regularly release software patches for their software
products [4]. Vulnerability prioritization is further hampered when companies
delay the automatic installation of security updates in case the patch proves
more troublesome than expected [5, 6].

Successful vulnerability management must balance two opposing goals: (1)
coverage (fix everything that matters), and (2) efficiency (delay or
deprioritize what does not matter) [7]. In industry, the most prevalent
vulnerability management strategy identifies the base CVSS scores for all
identified vulnerabilities and patches them in descending score order (10
highest to 0 lowest) [8, 9, 10]. Unfortunately, research has shown that CVSS
scores are not strongly linked to the emergence of new cyber exploits and
system administrators can be overwhelmed by the volume of vulnerabilities with
nearly indistinguishable high scores [11]. While a CVSS score is indicative of
vulnerability severity, it does not predict the exploit potential of the
underlying software flaw or the operational impact to the organization.

The goal of this research is to demonstrate that aggregating and synthesizing
readily accessible, public data sources, can provide an automated patch
priority ranking by understanding what vulnerabilities and adversaries are
relevant to an organization. Furthermore we show that this proposed strategy
offers significant improvements over using CVSS base scores.

Our proposed relevance-based ranking model enables businesses to adopt a
proactive strategy for vulnerability management [12]. Such an approach
delivers the most efficient use of people, tools, time, and dollars to address
cyber threats that pose the greatest operational risk. Just as search engines
provide a better ranking of results based on personalization, so will the
ranking of vulnerabilities. Within this context, we seek to define an approach
to cybersecurity vulnerability mitigation that improves upon rankings that
employ strategies based on the global CVSS metrics associated with known
software vulnerabilities published in the National Vulnerability Database
(NVD) [13].

In Section 2, we describe the public data sources we aggregate, the methods we
use to synthesize them, and our framework for ranking software vulnerabilities
with respect to different organizations for patching. In Section 3 we evaluate
our approach and present the results. Section 4 discusses how our
contributions are positioned in the landscape of software vulnerability
management research and identifies several limitations to our work.
Ultimately, we conclude our study in Section 5.

##  2 Data and Methods

Our goal is to remediate vulnerabilities in the most efficient way possible.
This requires leveraging, associating, and analyzing different sources of
cyber threat intelligence. The relationships among them need to be understood
and they must be organized into a structure for analysis that supports
generating prioritized recommendations for effective vulnerability management.

These data sources are used to model software vulnerabilities with respect to
the skill level of a cyber adversary and their motivation to target a specific
industry domain (e.g., national defense, higher education, finance, health
care). The relationships among these data sources and the software
vulnerability lifecycle are summarized in Figure 1. Specifically, these data
sets are:

  1. 1.

The Common Weakness Enumeration (CWE) captures data related to the discovery
of a software weakness.

  2. 2.

Data from the Common Vulnerabilities and Exposures (CVE), and Common
Vulnerability Scoring System (CVSS) prioritize a vulnerability’s severity.

  3. 3.

The Exploit Database (ExploitDB), Department of Homeland Security’s
Cybersecurity and Infrastructure Security Agency’s Known Exploited
Vulnerabilities (KEV) catalog, and Exploit Prediction Scoring System (EPSS)
assess the likelihood of a software vulnerability being exploited in the wild.

  4. 4.

The Common Attack Pattern Enumeration and Classification (CAPEC), and MITRE
ATT&CK knowledge base provides data on how to remediate and mitigate published
exploits.

  5. 5.

The National Vulnerability Database (NVD) catalogs and reports vendor provided
patches to vulnerabilities in commercial or open source software.

Figure 1: Software vulnerability lifecycle phases and their relationships to
our public data sources.

Next, we describe these data sources in more detail and the specific
attributes from them that we leverage. Then we highlight how we synthesize
them together in a knowledge base to connect data about how an adversary
capability can exploit a vulnerability to execute a cyber attack on an
organization.

###  2.1 Data

####  2.1.1 Software Weaknesses Dataset

The Software Weaknesses Dataset consists of data from the CWE. The CWE
provides a common language for describing security weaknesses in software
architecture, design, or code. It is an encyclopedia of hundreds of types of
software weaknesses including: buffer overflow, directory traversal, OS
injection, race condition, cross-site scripting, hard-coded password, and
insecure random numbers. Each weakness has a technical impact of which only
eight technical impacts lead to failure. These are the weaknesses we focus on.
The technical impacts of software weaknesses that lead to failure are: (1)
read data, (2) modify data, (3-4) deny service: unreliable execution and
resource consumption, (5) execute unauthorized code or commands, (6) gain
privileges / assume identity, (7) bypass protection mechanism, and (8) hide
activities.

####  2.1.2 Vulnerability Dataset

The Vulnerability Dataset is based on linking entries in the CVE with scoring
information from the CVSS. The CVE is the authoritative source of publicly
known vulnerabilities. The CVSS is an international standard for measuring the
severity of a vulnerability. The CVSS base score is composed of metrics that
reflect the intrinsic characteristics of the vulnerability. Each CVE entry
includes a unique identifier (CVE number), a short free-text description, and
a list of references to additional details of the vulnerability (in the form
of URLs). We include this information in our dataset and link it with the CVSS
base scores for the vulnerability.

####  2.1.3 Vendor Product Dataset

The Vendor Product Dataset is based on the CPE. Each entry (i.e. CPE-ID)
defines a specific hardware device, operating system, or application software.
We excluded entries marked as deprecated and restricted the CPE-IDs of
interest to those written in US English. Our Vendor Product Dataset contains
more than 15,000 CPE entries representing more than 3,000 products from
approximately 200 vendors.

####  2.1.4 Attack Pattern Dataset

Our Attack Patterns Dataset includes 545 unique instances of CAPEC
identifiers. CAPEC is a comprehensive dictionary and classification taxonomy
of known attacks that can be used by analysts, developers, testers, and
educators to advance community understanding and enhance defense sponsored by
the United States Department of Homeland Security. A CAPEC identifier can be
linked to the MITRE ATT&CK Enterprise tactics, techniques, and sub-techniques.
ATT&CK provides a common taxonomy for both offense and defense, and has become
a standard across many cyber security disciplines to convey threat
intelligence, perform testing through red teaming or adversary emulation, and
improve network and system defenses against intrusions.

####  2.1.5 Exploit Dataset

Our Exploit dataset is based on a one-to-many mapping between an identified
exploit kit (ExploitDB) to the vulnerabilities that are the target of that
exploit (CVE). ExploitDB is updated daily and provided by MITRE. It is
augmented with the data from CISA’s Known Exploited Vulnerabilities (KEV) and
the EPSS. The CISA KEV provides real-time updates via email alerts when a
newly, identified CVE-ID is exploited. The EPSS model is based on observations
of exploitation attempts against vulnerabilities, analysis of ancillary
information about each of those vulnerabilities, then uses historical events
to make predictions about future ones. The EPSS score associated with a CVE-ID
represents the probability [0-1] of exploitation in the wild in the next 30
days (following score publication) and the percentile of the current score
compared to all scored vulnerabilities with the same or lower EPSS score.

####  2.1.6 Adversary Tactics and Techniques Dataset

The combination of MITRE ATT&CK and CAPEC datasets form our adversary Tactics
and Techniques Dataset. The MITRE ATT&CK matrices are focused on network
defense and describe the operational phases in an adversary’s lifecycle. The
matrices also detail the specific tactics, techniques, and procedures (TTPs)
that Advanced Persistent Threat (APT) groups use to execute their objectives
while targeting, compromising, and operating inside a network. Attack patterns
enumerated by CAPEC are employed by adversaries through specific techniques
described by MITRE ATT&CK. Our dataset is formed by linking the CAPEC attack
patterns and the related MITRE ATT&CK techniques together. This enables
contextual understanding of the attack patterns within an adversary’s
operational lifecycle.

Figure 2: The entities of the knowledge graph. Figure 3: Graph schema
representing the type of entities via color in the knowledge graph and the
relationship between them.

###  2.2 Synthesizing Data Sources Into A Knowledge Graph

The datasets described in Subsection 2.1 can be combined to form a knowledge
graph. The purpose of the graph is to support queries to effectively rank
vulnerabilities for mitigation. This organizational structure is needed. A
wealth of information about what vulnerabilities are targeted, who exploits
those vulnerabilities, and how they do so currently exists. However, this
information is not organized into a structure that comprehensively defines the
relationships among the datasets. Our knowledge graph addresses this
deficiency. Its overall schema is described in Figure 3 and Table
LABEL:tab:schema-rel-color.

Table 1: Legend for node labels and relationships in knowledge graph schema. Label |  Relationship |  Label
---|---|---
NVD CVE |  REFERENCE EXPLOIT |  ExploitDB
NVD CVE |  EXPLOITS KNOWN |  CISA Exploit Catalog
NVD CVE |  WEAKENED BY |  CWE
CWE |  KNOWN ATTACK |  CAPEC
CAPEC |  EMPLOYS |  Attack Enterprise Techniques
Attack Groups |  ACHIEVES GOAL |  Attack Enterprise Techniques
Attack Groups |  ORIGINATES |  Countries
Attack Groups |  TARGETS |  Countries
Attack Groups |  FOCUS ON |  DHS Sectors
Attack Enterprise Tactics |  ACHIEVED THROUGH |  Attack Enterprise Techniques
NVD CVE |  AFFECTS |  CPE
DHS Sectors |  AFFILIATED WITH |  Organizations
Organizations |  OPERATES IN |  Countries
Organizations |  INSTALLS |  Software
Software |  HAS VERSION |  CPE
NVD CVE |  INFORMS |  NVD References

###  2.3 Leveraging the Knowledge Graph to Link Vulnerabilities to Sector
Specific Threat Actors

Our knowledge graph enables us to link vulnerabilities to APTs targeting
sectors within the United States and outside of the United States. Here we
describe what data processing is required to populate the knowledge graph, and
how once populated it links the data together.

####  2.3.1 Defining a Standard Set of Sectors

The critical infrastructure (CI) sectors denoted by the Department of Homeland
Security (DHS) reflect assets, systems, and networks that are vital enough to
the United States that incapacitating or destroying them would have a
debilitating effect on national security, economics, public health or public
safety [14]. Sectors can also be divided into subsectors [15]. The CI sectors
and subsectors are used to provide an affiliation for both threat actors and
the organizations they target in the knowledge graph.

####  2.3.2 Defining Standard Locations

Our knowledge graph requires a standard nomenclature to determine the country
or region of origin for cyber threat actors and country of residence for
organizations they target for attack. To meet this requirement we leverage the
U.S. State Department list of independent states. In this list, the term
“independent state” refers to people politically organized into a sovereign
state with a definite territory recognized as independent by the U.S. [16].

####  2.3.3 Assigning Attributes to Adversary Groups

APTs are an extension of nation-states’ military forces because of the
potential damages and chaos caused by successful critical infrastructure cyber
attacks. MITRE keeps track of the APTs. Currently it lists 129 threat groups
[17] in their _Enterprise Framework_ that can be associated with known
techniques. Using their defined threat profiles, we identify adversaries or
threat groups who employ the same tactics and techniques.

####  2.3.4 Where Attacks Originate

For each APT group description provided by MITRE, we use natural language
processing to extract keywords to determine the country or independent state
from which the group operates. For example, a _North Korean state-sponsored
threat group_ would be assigned to North Korea with our mapping. We also mined
the descriptions to determine year of origin (e.g., 2008) to ascertain the
potential longevity of each group. If a year was not explicitly stated in the
description, we used the creation date of the MITRE description (e.g., has
been active since at least 2009).

####  2.3.5 Who Attacks Each Sector

Next, we identify adversarial groups relevant to organizations based on who
they target for attacks by mapping APTs and their country to DHS critical
infrastructure sectors. To accomplish this we extract the subject of the term
‘targets’, ‘targeted’, or ‘targeting’ in the group description from MITRE. Our
knowledge graph includes those where the United States is a targeted country.
This choice reflects that our focus is on attacks aimed at the United States.
The attribution of APTs to sectors is shown in Table LABEL:tab:attack-group-
sectors. Some groups target more than one sector.

Table 2: DHS sectors ranked by the number of attack groups targeting those sectors based on mentions in MITRE ATT&CK. Sector Name |  Groups Targeting
---|---
Government Facilities |  50
Information Technology |  33
Financial Services |  19
Healthcare and Public Health |  17
Defense Industrial Base |  14
Energy |  14
Critical Manufacturing |  10
Communications |  9
Transportation Systems |  7
Chemical |  2
Water and Wastewater Systems |  1
Nuclear Reactors, Materials, and Waste |  1
Emergency Services |  0
Dams |  0
Commercial Facilities |  0
Food and Agriculture |  0

###  2.4 Relevance-Based Ranking Model

Recall that our goal is to define an approach to cybersecurity vulnerability
mitigation that improves upon rankings that employ strategies based on the
global CVSS metrics associated with known software vulnerabilities published
in the NVD. The outcome is a relevance-based ranking model that can be
employed before an adversary takes advantage of a particular vulnerability.
The model requires the following components:

  * •

Profiles that describe the organization under evaluation in terms of the DHS
sector and country in which they operate.

  * •

Collection and normalization of a complete software inventory for each
organization.

  * •

Threat-centric ranking policy definitions based on attack groups of interest
and their skill level.

  * •

Scoring method for each ranking policy.

###  2.5 Creating Organizational Profiles

A vulnerability ranking policy needs to consider the installed software for
the organization under evaluation. We identify and define a representative set
of organizations in the Government and Education Facilities to serve as
organizational benchmarks for evaluating our vulnerability management
approach.

###  2.6 Software Used in the Education Subsector

CollegeSimply[18] provides a list of Virginia colleges and sources public
domain college data from the U.S. Department of Education National Center for
Education Statistics. Using the list, we chose six universities of varying
sizes and funding sources (public and private). The public universities are
the University of Virginia (UVA), Virginia Tech (VT), Old Dominion University
(ODU), and William & Mary University (WM), and the private universities are
Regent University (REGENT) and Washington and Lee University (WLU). For each
university, we located a published list of supported academic software on the
university’s web site. Then we assign CPE-IDs to each piece of software. The
full academic software listing is provided in [12]. A summary of the number of
vulnerabilities found in the academic software associated with each university
is shown in Table LABEL:tab:academicSoftware-cpe. We assign “Size” (small (S),
medium (M), large (L), extra large (XL)) based on the number of software
products that were publicly listed. However, this does not reflect the size of
the university or the number of software products that the university actually
uses.

Table 3: Academic software associated with vendor product CPE-ID. |  WM |  ODU |  VT |  RU |  UVA |  WLU
---|---|---|---|---|---|---
CPEs |  24 |  47 |  12 |  23 |  30 |  13
Soft. |  33 |  69 |  22 |  31 |  49 |  23
Size |  Medium |  Extra Large |  Small |  Medium |  Large |  Small

###  2.7 Software Used by Government Facilities

Government facilities do not routinely publish the software they use. However,
the National Information Assurance Acquisition Policy, NSTISSP No. 11 [19],
requires government agencies to purchase only commercial security products
that have met specified third-party assurance requirements and have been
tested by an accredited national laboratory[19].111The list of certified
products is available at https://www.commoncriteriaportal.org/products/. In
accordance with NSTISSP, the _Common Criteria_ is an internationally
recognized set of guidelines (ISO 15408) [20] that define a common framework
for evaluating security features and capabilities of Information Technology
security products against functional and assurance requirements.

We reduced Common Criteria to the set of products certified for use in the
U.S. Then we searched for CPE-IDs across all categories based on the vendor
and product name. The software list shown in Table
LABEL:tab:governmentSoftware-cpe, consisting of applications and operating
systems, was generated by randomly selecting software from the Common Criteria
with assigned CPE-IDs in groups of 14, 24, 30, and 47 to approximately match
the cardinality of the small (S), medium (M), large (L), and extra-large (XL)
university software lists.

Table 4: Government facility software associated with vendor product CPE-ID. |  GOV-S |  GOV-M |  GOV-L |  GOV-XL
---|---|---|---|---
Software Assigned |  14 |  24 |  30 |  47
Common Criteria |  57 |  57 |  57 |  57

###  2.8 Ranking Policy Definitions

Deciding which vulnerabilities to remediate is a daunting task. In a perfect
world, all vulnerabilities would be remediated as they were discovered, but
unfortunately that does not happen in reality. An exploit observed in the wild
is the most relevant proxy for the probability that an exposed vulnerability
can be used to compromise an organization’s network. To that end the
predictive ranking policies we evaluate identify candidate vulnerabilities
that fit the pattern of known attack groups. Formally, this is the
intersection of vulnerabilities in the software used by an organization and
vulnerabilities being actively targeted by threat actors.

The criteria for our ranking policies using the attacker characteristics and
targets is discussed in Subsection 2.2. Each policy leverages data points in
the knowledge graph to provide a scoring methodology that considers:

  * •

Which threat actors use the same technique to initiate an attack?

  * •

Given an industry, which threat actors target it?

  * •

Given a type of attack, which vulnerabilities does it exploit?

  * •

At present day, what is the probability of exploit?

  * •

Given an organization, which vulnerabilities are present in the installed
software?

We created four different ranking policies that answer these questions. Each
policy prioritizes different information based on organizational information
preferences with regard to specific threats. The policies also include
knowledge on whether an exploit for the CVE-ID has been observed.

  * •

Policy One: CVSS Base Score Ranking: Vulnerabilities are remediated based on
the assigned CVSS Base Score ranking from most severe (“critical”) to least
severe (“low”).

  * •

Policy Two: APT Threat Ranking: Vulnerabilities are remediated based on the
likelihood of present day exploit and the existence of a technique employed by
an attack group that targets the industry in the country where the
organization operates.

  * •

Policy Three: Generalized Threat Ranking: Vulnerabilities are remediated based
on the likelihood of exploit by a low-skilled or highly-skilled adversary that
have high impact on the organization.

  * •

Policy Four: Ideal Ranking: The ideal ranking employs the same criteria as the
APT and Generalized threat rankings, Policy Two and Three, but has the
foreknowledge that a vulnerability has already been exploited using
information from the ExploitDB and CISA KEV databases.

###  2.9 Ranking Policy Implementations

For each CVE-ID, we examine 16 features using the cyber intelligence data
sources. The features inform each policy and create a set of relevance scores
for ranking CVE-IDs as they are published. The features are: (1) CVE-ID, (2)
CVSS Base Score Metrics, (3) Publication date, (4) Modification date, (5)
CAPEC-ID, (6) CAPEC skill level, (7) ATT&CK technique name, (8) MITRE ATT&CK
group id, (9) MITRE ATT&CK group country of operation, (10) Risk Appetite,
(11) EPSS Probability of Exploit, (12) CISA Known Exploit Catalog, (13)
Exploit-DB, (14) Organization identifier, (15) Critical infrastructure sector,
and (16) Organization’s country of residence. The source code used in our
implementation of the ranking policies is available in [21].

Based on the policy definitions, the CVSS V3.1 Base Score is the only feature
needed to implement Policy One. The features needed to implement Policy Two
and its ideal version in Policy Four are listed in Table
LABEL:tab:relevanceAlgo1.

Table 5: Policy Two scoring features using MITRE ATT&CK data feed to characterize the threat to the organization. Feature |  Specific Threat Relevance Rank Value |  Ideal Rank Value
---|---|---
CVSS Base Metric (Attack Vector) |  Network |  Network
DHS Sector |  Govt. Facilities, Education |  Govt. Facilities, Education
Org Country |  United States |  United States
Attack Group Country |  China, Russia, Iran |  China, Russia, Iran
Risk Appetite |  [0, 100] |  [0, 100]
EPSS Probability |  0.876 |
CISA KEV or Exploit-DB |  |
Entry Exists |  |  True
Software Affected |  True |  True
Scoring Range |  [1-6] |  [1-6]

For Policies Two, Three, and Four, we use a binary weighting [0,1] for each
feature to determine its existence as applicable to a specific CVE-ID. The sum
of the categorical values is presented as the relevance score we used to rank
the associated CVE-IDs using the logic shown in the algorithm provided in
[12]. The minimum assigned relevance score is set to 1 using this algorithm to
avoid a long tail of non-relevant CVE-IDs and ensure only relevant CVE-IDs
associated with the organization’s installed software are candidates for
ranking.

When determining what to patch, one must consider both the setup and business
disruption costs. Then one has to weigh them against the potential
exploitation cost, and decide when and how often to patch an enterprise system
or application. The total costs of a vulnerability are the sum of its direct
costs (level of effort employed by human resources) and indirect costs
(productivity losses, interruption of production processes after patching).
Previous research has established that these costs can be measured in non-
monetary units based on the severity of the vulnerability where Low=0.25,
Medium=1, High=1.5, and Critical=3 units [22]. We evaluate the economic cost
of remediating vulnerabilities using these established units.

##  3 Evaluation and Results

###  3.1 Candidate Generation

In our study, we use 55,939 CVE-IDs published between 2019-2021 as the corpus
from which to identify a much smaller subset of candidate vulnerabilities for
ranking. We use the CVE modification date to simulate examining the
vulnerabilities as they were published. A total of 3,079 unique CVE-IDs were
found to be applicable across all of the Government Facilities (Table
LABEL:tab:total-vulns-govt) and Education subsector (Table LABEL:tab:total-
vulns-education) software lists. The data and source code used in our
evaluation is available in [21].

For the Government Facilities shown in Table LABEL:tab:total-vulns-govt, we
observed low, annual vulnerability counts for three of the four proxy
organizations at less than 2% of all CVE-IDs analyzed. Even the largest
government organization, GOV-XL, which was designed to mirror the breadth of
software (i.e., 47 products) of its counterpart ODU in the Education
subsector, experienced less than 4% of all CVE-IDs analyzed. The low number of
vulnerabilities in the sector may be attributed to the selection process for
software products assigned to Government Facilities in this study which were
selected exclusively from the certified product list approved by the _Common
Criteria_ [19]. This outcome may provide an indication that the rigor imposed
upon these products in terms of security requirements and on-going evaluation
may potentially reduce their exposure to published vulnerabilities.

Table 6: Total vulnerabilities by year for government facilities sector. Year | GOV-S | GOV-M | GOV-L | GOV-XL
---|---|---|---|---
2019 | 8 | 41 | 51 | 102
2020 | 11 | 34 | 55 | 144
2021 | 16 | 84 | 140 | 285
Total Vulns | 35 | 159 | 246 | 531
Pct of All Vulns | 0.25% | 1.15% | 1.77% | 3.85%
Table 7: Weekly vulnerability traffic by year for government facilities sector. Year | Org | Avg Vuln Per Week | Min Vuln Per Week | Max VulnPer Week | Weeks Per Year
---|---|---|---|---|---
2019 | GOV-XL | 4 | 1 | 20 | 32
2019 | GOV-L | 3 | 1 | 9 | 24
2019 | GOV-M | 2 | 1 | 4 | 23
2019 | GOV-S | 2 | 1 | 2 | 5
2020 | GOV-XL | 4 | 1 | 13 | 40
2020 | GOV-L | 3 | 1 | 10 | 23
2020 | GOV-M | 3 | 1 | 10 | 16
2020 | GOV-S | 2 | 1 | 3 | 6
2021 | GOV-XL | 7 | 1 | 25 | 43
2021 | GOV-L | 4 | 1 | 14 | 40
2021 | GOV-M | 3 | 1 | 11 | 29
2021 | GOV-S | 2 | 1 | 3 | 10

For the Education subsector shown in Table LABEL:tab:total-vulns-education, we
also observed low vulnerability counts of less than 2% for organizations with
small amounts of reported software, such as VT and WLU. Conversely, we note
that universities who reported more software in use such as ODU, REGENT, and
WM need to evaluate hundreds of vulnerabilities as candidates for remediation
during the course of any given year.

Table 8: Total vulnerabilities by year for education subsector. Year | VT | WLU | REGENT | WM | UVA | ODU
---|---|---|---|---|---|---
2019 | 14 | 3 | 1396 | 1370 | 188 | 1457
2020 | 6 | 57 | 565 | 556 | 279 | 751
2021 | 15 | 144 | 1721 | 1704 | 639 | 2026
Total Vulns | 35 | 204 | 3682 | 3630 | 1106 | 4234
Pct of All Vulns | 0.25% | 1.45% | 26.56% | 26.19% | 7.98% | 30.54%
Table 9: Weekly vulnerability traffic by year for education subsector. Year | Org | Avg Vuln Per Week | Min Vuln Per Week | Max Vuln Per Week | WeeksPer Year
---|---|---|---|---|---
2019 | VT | 2 | 1 | 4 | 7
2019 | WLU | 1 | 1 | 1 | 3
2019 | REGENT | 35 | 1 | 442 | 40
2019 | WM | 43 | 1 | 441 | 32
2019 | UVA | 11 | 1 | 44 | 18
2019 | ODU | 35 | 1 | 444 | 42
2020 | VT | 1 | 1 | 1 | 6
2020 | WLU | 5 | 1 | 20 | 12
2020 | REGENT | 15 | 1 | 57 | 40
2020 | WM | 15 | 1 | 58 | 39
2020 | UVA | 9 | 1 | 34 | 33
2020 | ODU | 18 | 1 | 59 | 43
2021 | VT | 3 | 1 | 4 | 7
2021 | WLU | 7 | 1 | 23 | 21
2021 | REGENT | 36 | 1 | 264 | 48
2021 | WM | 36 | 1 | 258 | 48
2021 | UVA | 15 | 1 | 120 | 43
2021 | ODU | 41 | 1 | 315 | 50

Figures 4 shows the accumulated vulnerabilities by month and year for each
organization in our study. It is important to note the unpredictable manner in
which newly published and modified CVE-IDs can present themselves for analysis
and remediation to an organization.

Figure 4: Vulnerabilities by month and year for CVE-IDs between 2019 and 2021
for government facilities (left) and education (right) sectors.

###  3.2 Normalized Discounted Cumulative Gain

Within the field of cybersecurity there is no consensus approach for
measuring, testing, and comparing the accuracy of a ranking model. Therefore,
this research, similar to others in discussed in Section 4, builds upon
measurements derived from the field of Information Retrieval (IR). Evaluation
measures for IR assess how well the search results from a recommender satisfy
a given query. Specifically, recommender systems use the Normalized Discounted
Cumulative Gain (nDCG) [23] score to evaluate the ranking of items (e.g.,
individual vulnerabilities) in a collection (e.g., NVD).

The nDCG varies from 0.0 to 1.0, with 1.0 representing the ideal ranking
order. The nDCG is commonly used to evaluate Search Engine Result Pages
(SERPs) where the position of an entry indicates its search result relevancy.
Higher ranked pages are more likely to gain the attention of the consumer. We
apply the same approach towards creating a ranking list for patching
vulnerabilities. Order is important to ensure higher ranked CVE-IDs are
considered first. The main difficulty encountered when using nDCG is the
availability of an ideal ordering of results when feedback (e.g., human
judgment) is not available. We address this shortcoming faced by SERPS with
Policy Four, introduced in Section 2.9 as a data-driven proxy of an ideal
ordering of vulnerabilities.

In order to compare the results of rankings between each relevance policy and
the ideal ranking (Policy Four), we calculated the nDCG of each CVE-ID for
every organizational interaction with our ranking system. The nDCG values were
averaged for each weekly collection of CVE-IDs to obtain a measure of the
average performance of our ranking algorithms. The application of nDCG in this
study is interpreted as follows:

  1. 1.

“G” is for gain – it corresponds to the magnitude of each vulnerability’s
relevance.

  2. 2.

“C” is for cumulative – it refers to the cumulative gain, or summed total, of
every vulnerability’s relevance score.

  3. 3.

“D” is for discounted – it divides each vulnerability’s scored relevance by
the scored relevance of the associated ideal policy to reflect our goal of
having the most relevant vulnerabilities ranked towards the top of our
mitigation lists.

  4. 4.

“N” is for normalized – it divides Discounted Cumulative Gain (DCG) scores by
ideal DCG scores calculated for a ground truth data set, as represented by the
relevance scores and ranking resulting from our ideal policy (i.e. Policy
Four), which used foreknowledge of exploited vulnerabilities contained within
historical ExploitDB and CISA KEV intrusion detection reports.

Once the relevance value is computed for each CVE-ID, we rank each entry based
on the relevance value and compute the nDCG using the formulas shown below.

$DCG_{k}=\sum_{i=1}^{k}\frac{2^{rel_{i}}-1}{\log_{2}{\left(i+1\right)}}$ (1)

The cumulative gain at K is the sum of gains of the first K items recommended.
$iDCG_{k}$ is the maximum possible (ideal) $DCG$ for a given set of queries,
vulnerabilities, and relevance scores.

$nDCG_{k}=\frac{DCG_{k}}{iDCG_{k}}$ (2)

The chart in Figure 5 illustrates the average values of nDCG for each position
K based on weekly vulnerability collections. Recall, K reflects the number of
CVE-IDs to remediate. The number of observations ranges from 383 when K=1 to
16 when K=100. The x-axis reports the rank (from 1 to 100), while the y-axis
displays the respective value of nDCG@K. Figure 5 shows that the CVSS Base
Score performs moderately well at the ends of the spectrum when K=1 and K=100.
However, the performance decreases when 5 $\leq$ K $\leq$ 50\. Policy Two (APT
Threat) is not impacted by the number of weekly CVE-IDs; it performs at a
consistent level regardless of the number of CVE-IDs encountered.

Figure 5: Average value of nDCG at different rank levels (K) for CVSS Base Score versus APT Threat policy for the ODU, REGENT, and WM organizations. Table 10: Average performance of Policy One (CVSS Base Score) versus Policy Two (APT Threat) where China is the source region of interest (nDCG@20). Year | CVSS Base Score | APT Threat China | Avg. Difference in nDCG | Known Exploits
---|---|---|---|---
ODU
2019 | 0.601 | 0.996 | 0.394 | 4
2020 | 0.557 | 0.998 | 0.441 | 2
2021 | 0.571 | 0.986 | 0.415 | 12
REGENT
2019 | 0.592 | 0.999 | 0.407 | 2
2020 | 0.557 | 0.998 | 0.441 | 1
2021 | 0.585 | 0.985 | 0.399 | 12
WM
2019 | 0.598 | 0.998 | 0.400 | 3
2020 | 0.565 | 0.998 | 0.433 | 1
2021 | 0.585 | 0.985 | 0.399 | 12

In our results, the nDCG@20 measures for Policy One, the CVSS Base Score
policy, were in the range of [0.343, 1], as shown in Figure 6. Lower values
for nDCG@20 were observed with Policy One when the number of vulnerabilities
collected exceeded the minimum threshold (i.e. 20) by more than 1000% (e.g.,
200+). Higher nDCG@20 values were observed when the number of vulnerabilities
was closer to the threshold (e.g., 20 to 30). Policy Two, the APT threat-
centric policy, was minimally impacted by the number of vulnerabilities and
ranged from [0.878, 1].

Figure 6: nDCG@20 for Policy One (CVSS Base Score) versus Policy Two (APT
Threat) for the ODU, REGENT, and WM organizations.

Table LABEL:tab:ranking-eval-capec shows similar results for Policy Three, our
general threat-centric policy. The average difference in nDCG@20 of 0.35
indicates Policy Three performs 91.3% better than Policy One, the CVSS Base
Score policy, as an indicator of vulnerabilities that might be targeted by a
highly-skilled cyber threat actor.

Table 11: Average performance of Policy One (CVSS Base Score) versus Policy Three (General Threat) with a highly skilled adversary (nDCG@20). Year | CVSS Base Score | General Threat Highly Skilled | Avg. Difference in nDCG | Known Exploits
---|---|---|---|---
ODU
2019 | 0.543 | 0.988 | 0.444 | 4
2020 | 0.548 | 0.998 | 0.450 | 2
2021 | 0.474 | 0.986 | 0.511 | 12
REGENT
2019 | 0.528 | 0.995 | 0.467 | 2
2020 | 0.512 | 0.999 | 0.487 | 1
2021 | 0.500 | 0.984 | 0.484 | 12
WM
2019 | 0.538 | 0.992 | 0.454 | 3
2020 | 0.520 | 0.999 | 0.478 | 1
2021 | 0.499 | 0.984 | 0.485 | 12
Figure 7: nDCG@20 for the Policy One (CVSS Base Score) versus Policy Three
(General Threat) for the ODU, REGENT, and WM organizations.

Using all of the weekly observations (n=163) across organizations, we perform
a paired t-test [24] to compare the mean of the nDCG for Policy One (CVSS Base
Score) against Policy Two (APT Threat). Results of the paired t-test indicate
that there is a significant large difference between Policy One (CVSS Base
Score) [Mean = 0.58, STDEV = 0.1] and Policy Two (APT Threat) [Mean = 0.992,
STDEV = 0.02] and the p-value equals 0. The Policy Two (APT Threat)
population’s nDCG@20 average is greater than the Policy One (CVSS Base Score)
population’s average, and the difference is large enough to be statistically
significant.

We perform a similar test to compare the mean of the nDCG for Policy One (CVSS
Base Score) against Policy Three (General Threat). Results of the paired
t-test indicate that there is a significant large difference between Policy
One (CVSS Base Score) [Mean = 0.512, STDEV = 0.139] and Policy Three (General
Threat) [Mean = 0.99, STDEV = 0.022] and the p-value equals 0. The Policy
Three (General Threat) population’s average nDCG@20 greater than the Policy
One (CVSS Base Score) population’s average, and the difference is large enough
to be statistically significant.

These results show that CVSS base score metrics do not contain a data element
or scoring component that allows for enumeration of a specific threat. The
paired t-test indicates that the difference in the recommended ranking
positions of CVE-IDs between policies is statistically significant (p-value
equals 0). Therefore, any relevance ranking based solely on the CVSS base
score will fall short of the organization’s specified goals. These results
also provide another indication that the severity of a vulnerability, as
measured by its CVSS base score, may not be the optimal ranking approach for
every organization.

###  3.3 Cost of Patch Prioritization

Past research has shown that organizations cannot fix all of their known
vulnerabilities. Instead, they are able to fix 5% - 20% [8] of known
vulnerabilities per month. Here, we examine the annualized cost of remediating
the top 20 vulnerabilities produced by our different ranking policies: Policy
One (CVSS Base Score), Policy Two (APT threat-centric), and Policy Three
(General Threat Centric). Recall, we use the non-monetary units, defined by
Fhurwith et al., associated with patching. The results of this analysis are
shown in Table LABEL:tab:cost-eval-china. In all cases, there is decreased
average cost of 23.3% when Policy Two (APT threat-centric) is used for
prioritizing CVE-IDs for remediation. Specifically, Policy Two realizes (APT
threat-centric) decreases of 498 units for ODU, 390.5 units for REGENT, and
455.75 units for WM over the three year evaluation period when compared to
Policy One (CVSS Base Score).

Table 12: Difference in the cost of patching the top 20 CVE-IDs for Policy One (CVSS Base Score) versus Policy Two (APT Threat) where China is the source region of interest. Year | CVSS Base Score Cost | APT Threat China Cost | Average Savings
---|---|---|---
ODU
2019 | 631.50 | 449.25 | 182.25
2020 | 531.00 | 439.00 | 92.00
2021 | 994.50 | 770.00 | 224.50
REGENT
2019 | 604.75 | 422.25 | 182.50
2020 | 375.50 | 308.50 | 67.00
2021 | 960.00 | 752.00 | 208.00
WM
2019 | 603.75 | 424.75 | 179.00
2020 | 374.00 | 308.50 | 65.50
2021 | 960.00 | 748.75 | 211.25

Table LABEL:tab:cost-eval-capec shows increased savings in patch costs using
Policy Three (General Threat). The cost of patching remains the same across
all organizations using the CVSS Base Score. However, for each organization
Table LABEL:tab:cost-eval-china shows additional savings over using Policy Two
(APT Threat). Specifically, decreases of 548.25 units for ODU, 500.75 units
for REGENT, and 499.75 for WM represent an average 25.6% improvement over the
CVSS Base Score approach. Recall, Policy Two (APT Threat) only provided a
23.3% improvement.

Table 13: Difference in the cost of patching the top 20 CVE-IDs for Policy One (CVSS Base Score) versus Policy Three (General Threat) from a highly skilled adversary. Year | CVSS Base Score Cost | General Threat Cost | Average Savings
---|---|---|---
ODU
2019 | 631.50 | 438.50 | 193.00
2020 | 531.00 | 424.50 | 106.50
2021 | 994.50 | 745.75 | 248.75
REGENT
2019 | 604.75 | 412.00 | 192.75
2020 | 375.50 | 294.50 | 81.00
2021 | 960.00 | 733.00 | 227.00
WM
2019 | 603.75 | 412.50 | 191.25
2020 | 374.00 | 296.50 | 77.50
2021 | 960.00 | 729.00 | 231.00

Using all of the weekly observations (n=163) across organizations, we perform
a paired t-test [24] to compare the mean of the patch costs for Policy One
(CVSS Base Score) against Policy Two (APT Threat). Results of the paired
t-test indicate that there is a significant large difference between Policy
One (CVSS Base Score) [Mean = 37.025, STDEV = 10.291] and Poicy Two (APT
Threat) [Mean = 28.362, STDEV = 5.475] and the p-value equals 7.45e-27. The
population of Policy Two’s (APT Threat) average patch cost is considered to be
less than Policy One’s (CVSS Base Score), and the difference is large enough
to be statistically significant.

Similarly, we perform a paired t-test [24] to compare the mean of the patch
costs for Policy One (CVSS Base Score) against Policy Three (General Threat).
Results of the paired t-test indicated that there is a significant large
difference between Policy One (CVSS Base Score) [Mean = 37.025, STDEV =
10.291] and Policy Three (General Threat) [Mean = 27.523, STDEV = 4.905] and
the p-value equals 9.989e-30. The population of Policy Three’s (General
Threat) average patch cost is considered to be less than Policy One’s (CVSS
Base Score), and the difference is large enough to be statistically
significant.

###  3.4 Predicting Exploits

Only a small subset (2%-7%) of published vulnerabilities are exploited in the
wild [8]. Given that such a small number of CVE-IDs are actually exploited, it
is advantageous for organizations to leverage as much insight as possible to
identify potential threats. Here, we demonstrate how Policy Two (APT Threat)
can be used to prioritize a vulnerability with a known exploit. The ODU
organization identified 39 CVE-IDs to mitigate during the week of November 23,
2021.

In this case study, the top 20 are ranked in accordance with Policy Two (APT
Threat) as shown in Table LABEL:tab:ODU-TOP20-Example. We note that three CVE-
IDs in this group, CVE-2021-38000, CVE-2021-30632, and CVE-2021-30633, have
known exploits. The CISA Known Exploits entry for CVE-2021-38000, which
impacts Google Chrome, is shown in Figure 8. The entries in Table
LABEL:tab:ODU-TOP20-Example show that all three CVE-IDs are identified as
relevant using Policy Two. However, CVE-2021-38000 is ranked at position 29
using Policy One based on its CVSS base score of 6.1 (medium severity). This
highlights that when using Policy One (CVSS Base Score), CVE-2021-38000 falls
outside of the top 20 range for remediation by IT administrators at ODU. In
contrast, Policy Two (APT Threat) elevates this CVE-ID to position number 3
because of its high relevance score.

Table 14: Application of ranking policies by ODU for vulnerabilities published during the week of 23-November-2021. Known exploits are bolded and highlighted in grey. CVE-ID | CVSS Base Score | Relevance Score | Policy 1 Rank | Policy 2 Rank
---|---|---|---|---
CVE-2021-37966 | 4.3 | 6 | 34 | 1
CVE-2021-37999 | 6.1 | 6 | 28 | 2
CVE-2021-38000 | 6.1 | 6 | 29 | 3
CVE-2021-30542 | 8.8 | 2 | 5 | 4
CVE-2021-30543 | 8.8 | 2 | 6 | 5
CVE-2021-30626 | 8.8 | 2 | 7 | 6
CVE-2021-30627 | 8.8 | 2 | 8 | 7
CVE-2021-30628 | 8.8 | 2 | 9 | 8
CVE-2021-30629 | 8.8 | 2 | 10 | 9
CVE-2021-30630 | 4.3 | 2 | 31 | 10
CVE-2021-30632 | 8.8 | 2 | 11 | 11
CVE-2021-30633 | 9.6 | 2 | 2 | 12
CVE-2021-34423 | 9.8 | 2 | 1 | 13
CVE-2021-34424 | 7.5 | 2 | 26 | 14
CVE-2021-37956 | 8.8 | 2 | 12 | 15
CVE-2021-37957 | 8.8 | 2 | 13 | 16
CVE-2021-37958 | 5.4 | 2 | 30 | 17
CVE-2021-37959 | 8.8 | 2 | 14 | 18
CVE-2021-37961 | 8.8 | 2 | 15 | 19
CVE-2021-37962 | 8.8 | 2 | 16 | 20
Figure 8: CISA Known Exploits catalog entry for CVE-2021-38000 (Reproduced
from [25]).

##  4 Discussion

There is a myriad of existing research that falls within the scope of our
work. Here, we discuss related research, identify the limitations of our work
and highlight the contributions it provides.

###  4.1 Related Research

Multiple researchers have created ontologies to represent the cyber security
domain by aggregating multiple sources of information [26, 27, 28, 29, 30,
31]. This work provides the foundation for building automated tools which
reduce the scope, complexity, and volume of security data that must be managed
by security professionals which we leverage in our approach. However, our
research differs from these efforts in that we extract more information and
sources to achieve completeness in our knowledge graph. In addition,
categorization is a necessary pre-cursor to our ranking policies for
vulnerability management. Multiple research efforts have shown that
identifying and categorizing additional metadata about vulnerabilities,
exploits, attacks, and targets can be beneficial [22, 32, 33, 34]. More
recently, the application of text-mining to extra additional data about these
entities has led to models which predict the severity of a vulnerability using
only text-based data [35, 36, 37, 38, 39].

Even with an organized understanding of the cyber threat domain, understanding
how to minimize the cost of managing and protecting information assets is a
challenge. A core component of this challenge is adopting a vulnerability
management process that can detect and remediate known vulnerabilities [11]. A
common approach is to remediate all vulnerabilities above a certain severity
score. However, this approach has been found to be sub-optimal [40] and in
some cases, no better than randomly choosing vulnerabilities to remediate
[11]. Furthermore, in many cases it is infeasible to patch all the CVEs with
the highest CVSS base scores due to the time and resources required for
remediation actions. This is because 13.5% of the NVD vulnerabilities are
scored between 9 and 10 [41].

This has led to extensive work in evaluating if the CVSS score can be a good
predictor for vulnerability exploitation [42], and whether it can be improved
by additional information [43, 44, 45, 46]. Machine learning approaches have
also been explored [47, 35] as well as exploit prediction models that leverage
data from online sources generated by the white-hat community (i.e., ethical
hackers) [37]. Vulnerability exploitation can also be modeled as a transition
between system states [48, 49, 50, 51, 52, 53, 54]. However, these graphs
often tend to be unwieldy as network size grows, making the identification of
realistic paths to compromise difficult to achieve [55]. Customized and target
specific ranking approaches also exist [41, 56, 11, 57]. However, these
approaches assume the existence of site-specific threat intelligence
information.

###  4.2 Contributions of our approach

Prior research has demonstrated the ability to examine adversary capabilities,
vulnerability management, and exploit prediction at a particular point in time
or with isolated threat scenarios. However, little research has been done to
create an end-to-end prioritization approach which encompasses the entire
vulnerability management life cycle. We address this gap by:

  * •

Extracting dozens of essential features about the vulnerability, including its
potential for harm, the degree to which it is exploitable, and how frequently
the vulnerability is targeted by adversaries.

  * •

Leveraging the ability of property graphs to offer a flexible schema where one
can add attributes to extend the data model, create hierarchies with different
levels of granularity, and combine multiple dimensions to better manage big
data.

  * •

Performing an assessment of current and predicted future attacker activity
based on known tactics and techniques.

  * •

Correlating threat and exploit intelligence from publicly available
authoritative sources.

  * •

Devising an approach to convert raw data about threat indicators into
contextual risk scores.

  * •

Identifying how important the affected asset is to an organization in any
industry.

  * •

Inferring indirect facts and hidden relationships which can further inform our
results.

Parsing real-time, open source cyber threat intelligence data cannot be
accomplished by a human analyst. Therefore, we automate its correlation and
analysis using a knowledge graph. We can also leverage the Application
Programming Interfaces (APIs) and data feeds maintained by NIST to provide
awareness of the changing threat landscape while allowing for dynamic and
continuous assessment of the underlying network architecture. Our research
provides benefits to organizations seeking to create high-level strategies to
examine cybersecurity posture in a manner that is predictive not just
reactive.

###  4.3 Known Limitations

It is important to note that our work is not without limitations. To apply our
approach, organizations must have a methodology to accurately construct a
software inventory that can be correlated with an entry in the Common Platform
Enumeration (CPE) database. Vulnerabilities cannot be allocated without a CPE-
ID and low fidelity inventory reporting may result in residual cyber risk. The
relevance ranking policies we have identified can only be effectively applied
to a known software architecture. Furthermore, it is important to note that
the attack group list in MITRE ATT&CK is not all encompassing. A Google search
will identify emerging APT groups that are not included in the MITRE’s
enterprise matrices. In addition, the proof of concept code entries collected
via ExploitDB does not include a time component which indicates when the POC
entry was made. As a result, it is not possible to discretely link the CVE-
ID’s publication or modification date with the subsequent appearance of an
intrusion report. The inclusion of a timestamp would have allowed us to
evaluate the predictive portion of our policies based on a timeline of events.
Our approach is naive with regard to exploitation and does not consider the
publication date for exploit code maturity using ExploitDB. The ExploitDB to
CVE mapping webpage is also not well covered in the Internet Archive.

Time lapse dynamics related to data sources exist, too. The EPSS probability
scores and percentiles are dynamic and should be collected near the time of
the CVE publication date. In order to maintain consistency in our data set,
all cyber intelligence data was collected and frozen for analysis as of
31-December-2021. Future work can utilize the API provided by the EPSS team to
dynamically collect the scores and percentiles in real time. This is a
candidate for future work. Finally, the optimum ordering approach that we
prescribe may not ease patch hesitancy or prevent a culture of “wait and see”
with regard to patching vulnerabilities. Our policies also cannot control the
quality of vendor patch distributions on Patch Tuesday that in some cases lead
to recalls later in the month. These scenarios are outside the scope of this
research. Our ranking policies can, however, reduce the amount of unnecessary
work spent patching CVE-IDs that are neither applicable nor associated with a
known cyber threat actor.

##  5 Conclusions

The process of tracking and remediating vulnerabilities is relentless. The key
challenge is trying to identify a remediation scheme specific to in-house,
organizational objectives. Without a strategy, the result is a patchwork of
fixes applied to a tide of vulnerabilities, any one of which could be the
point of failure in an otherwise formidable defense. The goal of this research
is to demonstrate that aggregating and synthesizing readily accessible, public
data sources to provide personalized, automated recommendations for
organizations to prioritize their vulnerability management strategy will offer
significant improvements over the current state of the art solutions. Our
results showed an average 71.5% - 91.3% improvement towards the identification
of vulnerabilities likely to be targeted and exploited by cyber threat actors.
The ROI of patching using our policies results in a savings in the range of
23.3% - 25.5% in annualized costs. A paired t-test demonstrates these findings
are statistically significant and offer an improvement over the industry
standard approach to vulnerability management.

Overall, the relevance ranking strategy described in this study emphasizes the
capability of threat-centric scenarios for ranking and prioritizing
vulnerabilities with due consideration of the threat environment. We have
shown that a network defender, who typically has to address thousands of
exposed vulnerabilities, can spend fewer resources to patch more
vulnerabilities that are much more likely to be exploited and of interest to a
specific set of cyber threat actors. The automated data aggregation within our
knowledge graph allows one to submit queries to identify new vulnerabilities
that affect the most important software and servers. This ability to
differentiate among vulnerabilities and how they might be targeted by an
adversary enhances the state of the art in vulnerability management.

## Acknowledgments

This work was unfunded and performed as part of McCoy’s Ph.D. dissertation
work at Old Dominion University.

## References

  * [1] Oleg Sheyner, Joshua Haines, Somesh Jha, Richard Lippmann, and Jeannette M Wing.  Automated generation and analysis of attack graphs.  In Proceedings of the 2002 IEEE Symposium on Security and Privacy, pages 273–284. IEEE, 2002.  doi: 10.1109/SECPRI.2002.1004377.
  * [2] Christopher C. Krebs.  Emergency directive 20-03 (ED 20-03), mitigate windows DNS server vulnerability from July 2020 Patch Tuesday, July 2020.  https://cyber.dhs.gov/ed/20-03/.
  * [3] Sarah Coble.  CISA issues emergency vulnerability warning.  https://www.infosecurity-magazine.com/news/cisa-issues-emergency/, July 2020\.  Accessed 2020-07-19.
  * [4] Wikipedia.  Patch Tuesday — Wikipedia, the free encyclopedia.  http://en.wikipedia.org/w/index.php?title=Patch%20Tuesday&oldid=992184086, 2020\.  Accessed 2020-12-05.
  * [5] Netta Schmeidler.  Microsoft Patch Tuesday: All or nothing patches.  https://blog.morphisec.com/microsoft-patch-tuesday-all-or-nothing-patching, October 2016.  Accessed 2022-10-25.
  * [6] Lindsey O’Donnell.  Microsoft pulls bad windows update after Patch Tuesday headaches.  https://threatpost.com/microsoft-windows-update-patch-tuesday/163981/, February 2021.  Accessed 2021-03-13.
  * [7] Park Foreman.  Vulnerability Management.  Taylor and Francis Group, 2010.  ISBN: 978-1-4398-0150-5.
  * [8] Forum of Incident Response and Security Teams (FIRST).  Exploit Prediction Scoring System v2022.01.01.  https://www.first.org/epss/, 2022.  Accessed 2022-10-06.
  * [9] Forum of Incident Response and Security Teams (FIRST).  Probability, percentiles, and binning - how to understand and interpret EPSS scores.  https://www.first.org/epss/articles/prob_percentile_bins, 2022.  Accessed 2022-10-06.
  * [10] Peter Mell, Karen Scarfone, and Sasha Romanosky.  A complete guide to the Common Vulnerability Scoring System version 2.0.  In FIRST-forum of incident response and security teams, volume 1, pages 1–23, 2007.
  * [11] Luca Allodi and Fabio Massacci.  Comparing vulnerability severity and exploits using case-control studies.  ACM Transactions on Information and System Security (TISSEC), 17(1):1–20, 2014.
  * [12] Corren G McCoy.  A Relevance Model for Threat-Centric Ranking of Cybersecurity Vulnerabilities.  PhD thesis, Old Dominion University, 2022.
  * [13] NIST.  National Vulnerability Database.  https://nvd.nist.gov/vuln-metrics/cvss.  Accessed 2020-12-05.
  * [14] Cybersecurity and Infrastructure Security Agency.  Critical infrastructure sectors.  https://www.cisa.gov/critical-infrastructure-sectors, October 2020\.  Accessed 2022-10-08.
  * [15] US Election Assistance Commission.  CI Scoop: What are sectors and sub-sectors?  https://www.eac.gov/ci-scoop-what-are-sectors-and-sub-sectors, May 2017.  Accessed 2022-10-08.
  * [16] U.S. Department of State.  Independent states in the world.  https://www.state.gov/independent-states-in-the-world/, September 2022.  Accessed 2022-10-08.
  * [17] MITRE ATT&CK Enterprise, Version 10.1.  https://web.archive.org/web/20220215084226/https://attack.mitre.org/, 2022\.  Accessed 2022-02-17.
  * [18] CollegeSimply.  Virginia colleges ranked by largest enrollment.  https://www.collegesimply.com/colleges/rank/colleges/largest-enrollment/state/virginia/, 2021\.  Accessed 2021-12-18.
  * [19] National Security Telecommunications and Information Systems Security Policy (NSTISSP) No. 11.  https://www.hsdl.org/?view&did=487791, July 2003.  Accessed 2022-10-18.
  * [20] International Organization for Standardization.  Information technology — security techniques — evaluation criteria for it security — part 1: Introduction and general model.  https://www.iso.org/standard/50341.html, August 2022.
  * [21] Corren McCoy.  oduwsdl/CyberThreatRelevanceRank: CyberThreatRelevanceRank, January 2024\.
  * [22] Christian Fruhwirth and Tomi Mannisto.  Improving CVSS-based vulnerability prioritization and response with context information.  In Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement, pages 535–544. IEEE, 2009\.  doi: 10.1109/ESEM.2009.5314230.
  * [23] Kalervo Järvelin and Jaana Kekäläinen.  Cumulated gain-based evaluation of ir techniques.  ACM Transactions on Information Systems (TOIS), 20(4):422–446, 2002\.
  * [24] Wikipedia.  Student’s T-Test.  https://en.wikipedia.org/wiki/Student%27s_t-test, 2022.  Accessed 2022-11-17.
  * [25] Cybersecurity and Infrastructure Security Agency.  Known exploited vulnerabilities catalog.  https://www.cisa.gov/known-exploited-vulnerabilities-catalog.  Accessed 2022-10-05.
  * [26] Erik Hemberg, Jonathan Kelly, Michal Shlapentokh-Rothman, Bryn Reinstadler, Katherine Xu, Nick Rutar, and Una-May O’Reilly.  Linking threat tactics, techniques, and patterns with defensive weaknesses, vulnerabilities and affected platform configurations for cyber hunting.  arXiv preprint arXiv:2010.00533, 2020.
  * [27] Robert A Bridges, Corinne L Jones, Michael D Iannacone, Kelly M Testa, and John R Goodall.  Automatic labeling for entity extraction in cyber security.  arXiv preprint arXiv:1308.4941, 2013.
  * [28] Corinne L Jones, Robert A Bridges, Kelly MT Huffer, and John R Goodall.  Towards a relation extraction framework for cyber-security concepts.  In Proceedings of the 10th Annual Cyber and Information Security Research Conference, pages 1–4, 2015.  doi: 10.1145/2746266.2746277.
  * [29] Michael Iannacone, Shawn Bohn, Grant Nakamura, John Gerth, Kelly Huffer, Robert Bridges, Erik Ferragut, and John Goodall.  Developing an ontology for cyber security knowledge graphs.  In Proceedings of the 10th Annual Cyber and Information Security Research Conference, pages 1–4, 2015.
  * [30] Christian Bizer, Tom Heath, and Tim Berners-Lee.  Linked data: The story so far.  In Semantic Services, Interoperability and Web Applications: Emerging Concepts, pages 205–227. IGI global, 2011.
  * [31] Ju An Wang and Minzhe Guo.  OVM: An ontology for vulnerability management.  In Proceedings of the 5th Annual Workshop on Cyber Security and Information Intelligence Research: Cyber Security and Information Intelligence Challenges and Strategies, pages 1–4, 2009.  doi: 10.1145/1558607.1558646.
  * [32] Stefan Frei, Martin May, Ulrich Fiedler, and Bernhard Plattner.  Large-scale vulnerability analysis.  In Proceedings of the 2006 SIGCOMM workshop on Large-scale attack defense, pages 131–138, 2006.  doi: 0.1145/1162666.1162671.
  * [33] Christopher J Alberts and Audrey J Dorofee.  Managing information security risks: The OCTAVE approach.  Addison-Wesley Professional, 2003.
  * [34] Emmanuel Tsukerman.  Cybersecurity Threat Modeling with OCTAVE.  https://www.pluralsight.com/guides/cybersecurity-threat-modeling-with-octave, September 2020.  Accessed 2021-05-20.
  * [35] Jay Jacobs, Sasha Romanosky, Idris Adjerid, and Wade Baker.  Improving vulnerability remediation through better exploit prediction.  Journal of Cybersecurity, 6(1):tyaa015, 2020.  doi: 10.1093/cybsec/tyaa015.
  * [36] Tianqi Chen and Carlos Guestrin.  XGBoost: A scalable tree boosting system.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 785–794, 2016.  doi: 10.1145/2939672.2939785.
  * [37] Mohammed Almukaynizi, Eric Nunes, Krishna Dharaiya, Manoj Senguttuvan, Jana Shakarian, and Paulo Shakarian.  Proactive identification of exploits in the wild through vulnerability mentions online.  In Proceedings of the 2017 International Conference on Cyber Conflict (CyCon US), pages 82–88. IEEE, 2017.
  * [38] Atefeh Khazaei, Mohammad Ghasemzadeh, and Vali Derhami.  An automatic method for CVSS score prediction using vulnerabilities description.  Journal of Intelligent & Fuzzy Systems, 30(1):89–96, 2016.
  * [39] Wikipedia.  Open Source Vulnerability Database (OSVDB).  https://en.wikipedia.org/wiki/Open_Source_Vulnerability_Database.  Accessed 2020-05-01.
  * [40] Debabrata Dey, Atanu Lahiri, and Guoying Zhang.  Optimal policies for security patch management.  INFORMS Journal on Computing, 27(3):462–477, 2015.  doi: 10.1287/ijoc.2014.0638.
  * [41] Kenneth Alperin, Allan Wollaber, Dennis Ross, Pierre Trepagnier, and Leslie Leonard.  Risk prioritization by leveraging latent vulnerability features in a contested environment.  In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, pages 49–57, 2019.  doi: 10.1145/3338501.3357365.
  * [42] Luca Allodi and Fabio Massacci.  A preliminary analysis of vulnerability scores for attacks in wild: The EKITS and SYM datasets.  In Proceedings of the 2012 ACM Workshop on Building analysis datasets and gathering experience returns for security, pages 17–24, 2012.
  * [43] Yulia Tatarinova and Olga Sinelnikova.  Extended vulnerability feature extraction based on public resources.  Theoretical and applied cybersecurity, 1(1), 2019.
  * [44] Greg R Notess.  The Wayback Machine: The web’s archive.  Online, 26(2):59–61, 2002.
  * [45] Internet Archive: Wayback Machine.  https://archive.org/web/.  Accessed 2021-05-01.
  * [46] Sameera Horawalavithana, Abhishek Bhattacharjee, Renhao Liu, Nazim Choudhury, Lawrence O. Hall, and Adriana Iamnitchi.  Mentions of security vulnerabilities on Reddit, Twitter and GitHub.  In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, pages 200–207, 2019.
  * [47] Jay Jacobs, Sasha Romanosky, Benjamin Edwards, Michael Roytman, and Idris Adjerid.  Exploit prediction scoring system (EPSS).  arXiv preprint arXiv:1908.04856, 2019.
  * [48] Anoop Singhal and Xinming Ou.  Security risk analysis of enterprise networks using probabilistic attack graphs.  In Network Security Metrics, pages 53–73. Springer, 2017.  doi: 10.1007/978-3-319-66505-4_3.
  * [49] Xinming Ou, Wayne F Boyer, and Miles A McQueen.  A scalable approach to attack graph generation.  In Proceedings of the 13th ACM Conference on Computer and Communications Security, pages 336–345, 2006.
  * [50] Xinming Ou, Sudhakar Govindavajhala, and Andrew W Appel.  MulVal: A logic-based network security analyzer.  In Proceedings of the USENIX Security Symposium, volume 8, pages 113–128. Baltimore, MD, 2005.
  * [51] John Homer, Xinming Ou, and David Schmidt.  A sound and practical approach to quantifying security risk in enterprise networks.  Kansas State University Technical Report, (2009-3), 2009.  http://people.cs.ksu.edu/~xou/publications/tr_homer_0809.pdf.
  * [52] Laurent Gallon and Jean-Jacques Bascou.  CVSS attack graphs.  In 2011 Seventh International Conference on Signal Image Technology & Internet-Based Systems, pages 24–31. IEEE, 2011.  doi: 10.1109/SITIS.2011.24.
  * [53] Steven Noel, Eric Harley, Kam Him Tam, and Greg Gyor.  Big-data architecture for cyber attack graphs.  MITRE case number 14-3549, 2014.  https://csis.gmu.edu/noel/pubs/2015_IEEE_HST.pdf.
  * [54] Steven Noel, Eric Harley, Kam Him Tam, and Greg Gyor.  Big-data architecture for cyber attack graphs representing security relationships in NoSQL graph databases.  In Proceedings of the IEEE Symposium on Technologies for Homeland Security (HST), April 2015.
  * [55] Jorge Lucangeli Obes, Carlos Sarraute, and Gerardo Richarte.  Attack planning in the real world.  arXiv preprint arXiv:1306.4044, 2013.
  * [56] Luca Allodi and Sandro Etalle.  Towards realistic threat modeling: Attack commodification, irrelevant vulnerabilities, and unrealistic assumptions.  In Proceedings of the 2017 Workshop on Automated Decision Making for Active Cyber Defense, pages 23–26, 2017.  doi: 10.1145/3140368.3140372.
  * [57] Luca Allodi, Fabio Massacci, and Julian Williams.  The work-averse cyberattacker model: Theory and evidence from two million attack signatures.  Risk Analysis, 42(8):1623–1642, 2022.