fortiss GmbH
White Paper, 2020 Security Engineering for ISO 21434
A Closer Look into the ISO 21434 and Beyond Yuri Gil Dantas, Vivek Nigam* and
Harald Ruess The ISO 21434 is a new standard that has been proposed to address
the future challenges of automotive cybersecurity. This white paper takes a
closer look at the ISO 21434 helping engineers to understand the ISO 21434
parts, the key activities to be carried out and the main artefacts that shall
be produced. As any certification, obtaining the ISO 21434 certification can
be daunting at first sight. Engineers have to deploy processes that include
several security risk assessment methods to produce security arguments and
evidence supporting item security claims. In this white paper, we propose a
security engineering approach that can ease this process by relying on
Rigorous Security Assessments and Incremental Assessment Maintenance methods
supported by automation. We demonstrate by example that the proposed approach
can greatly increase the quality of the produced artefacts, the efficiency to
produce them, as well as enable continuous security assessment. Finally, we
point out some key research directions that we are investigating to fully
realize the proposed approach.

###### Contents

  1. Introduction
  2. 1 ISO 21434 at a Glance
  3. 2 Running Example
  4. 3 Risk Assessment
  5. 4 Incremental Approach
  6. 5 Approach by Example
  7. 6 Related Work
  8. 7 Next Research Directions
  9. 8 Conclusions

## Introduction

New vehicle business models are in the process of being forged based on the
adoption of Information and Communication Technologies (ICT), such as V2X
communication and artificial intelligence. According to Juniper Research [9],
there are currently tens of millions of connected vehicles, and this number
will reach 775 million vehicles by 2023. Connected vehicles already enable
important services, such as on-line streaming and Advanced Driver Assistance
Systems (ADAS). In the near future, they will enable services such as vehicles
with very high level of autonomy.

The adoption of ICT, however, has also greatly increased vehicle cybersecurity
concerns. Until some years ago, attackers would necessarily have to be
physically close to carry out attacks against vehicles. This is no longer the
case with connected cars. As already demonstrated by the now infamous Jeep
attack [7], without appropriate countermeasures, attackers can remotely take
control of pretty much any vehicle function, such as the vehicle’s braking
system. The year of 2019 has seen an emergence of cyber-attacks [8]: 51
incidents have been reported only in the first quarter of 2019. This number is
an increase of more than 300% to 2018. Moreover, while in the past attacks
were carried out in its majority by white-hats111Security experts that carry
out attacks to find vulnerabilities with the intent of improving security.,
now the majority of the attacks reported are carried out by black-
hats222Cyber-criminals that carry out attacks for normally financial gains or
notoriety. with a rise of 72% from 2018. There is much reason to believe that
this trend will continue and even accelerate. Indeed, as reported by the
Hornetsecurity Security Lab [10], the automotive industry is the third most
targeted by attackers, behind only the energy and logistics sectors.

The cost of handling attacks, _e.g._ , car re-calls, updates, mitigate damage
to reputation, will also increase as more connected vehicles enter the roads.
Current estimations predict that the cost of an attack could reach USD 23
billion by 2023 [11]. Therefore, it is increasingly important that
vulnerabilities are found and mitigated before production and that new
vulnerabilities discovered after production are dealt within a short period of
time.

To address these concerns, the new ISO 21434 [3] has been developed replacing
the SAE J3061 [4]. It is expected that the ISO 21434 will follow the success
of the ISO 26262 [2] for vehicle safety, but now for security. One main
objective of this white paper is to explain the ISO 21434 providing to
cybersecurity engineers a comprehensive overview of the main activities that
shall be carried out and the key artefacts that shall be produced in order to
acquire ISO 21434 certification.

This paper also goes beyond the ISO 21434. We describe an incremental approach
for security for enabling the continuous security assessment of vehicles and
therefore, enabling the continuous certification of vehicles. Our approach is
based on three key elements:

  * •

Rigorous Security Assessments: Vulnerabilities are introduced or not correctly
mitigated because of the lack of rigor in the security argument used to
support a claim. For example, arguments often contain implicit assumptions
which leads to implementation errors. For the construction of such arguments,
we advocate rigorous security arguments written by using logic-based methods
supported by domain-specific languages. Logic provides precise semantics to,
for example, reasoning principles used to construct safety and security
arguments and more importantly the means to check their correctness and
completeness.

  * •

Incremental Assessment Maintenance: As new vulnerabilities and attacks are
constantly being discovered, security updates will be a norm for connected
vehicles. The question that follows is how to demonstrate that after an update
a vehicle still complies with the ISO 21434 certification, _e.g._ , its
security is supported by a valid security argument and appropriate evidence.
Re-doing the whole certification process from scratch whenever there is an
update is not practical. A better approach is to modify the existing security
assessments and certification artefacts incrementally by modifying only the
pieces that have been affected by an update.

  * •

Automation: Increasing the automation of vehicle’s security assessments is a
necessary requirement for efficient continuous vehicle security assessment and
certification. Moreover, it also less prone to human-error. Fortunately, there
has been in the past year a great increase in the maturity of automated
reasoning and verification tools. For example, existing logic programming
tools can automatically generate correct safety and security arguments
[17][18], collect [16] and maintain evidence generated from workflows using
automated tools [14].

#### Benefits.

The benefits of our approach with respect to existing approaches are two-fold.

Firstly, our rigorous security assessments with precise semantics enables the
automated checking of when an argument is incomplete, _e.g._ , some threats
are not appropriately mitigated, or plain wrong. Moreover, since our methods
are based on logic, engineers can query our specifications like the following
ones:

  * •

Have all the identified threats been adequately mitigated? If not, which ones
remain?

  * •

Which architecture options with additional countermeasures can I use to
mitigate the threats that have not yet been mitigated?

  * •

What are the impacts of a particular design to other issues such as safety and
performance? Will the new countermeasure affect some safety assumption, _e.g._
, increased communication latency?

  * •

What is the security argument used to support the security of an item?

Secondly, the use of incremental methods and automation greatly increases the
efficiency of re-certification in terms of time and cost. Efficiency is key
for enabling continuous certification. Otherwise, the overhead costs of re-
certification is very high without adequate automation. This has been
observed, for example, in the avionics domain, where the change of a single
line of code can imply costs of in the order of millions of USD.333Stated by
the DARPA project coordinator https://www.youtube.com/watch?v=bXuBm_awZvg

The target audience for the white paper is all security engineers who are
interested in the ISO 21434 itself and in an approach that enables the
continuous certification for automotive vehicles. We expect that, by reading
this white paper, security engineers will be able to obtain a clearer picture
of the ISO 21434, especially of the activities performed during the risk
assessment. We also expect that security engineers will both learn that parts
of the ISO 21434 may be automated by suitable techniques (_e.g._ , our
machinery) as well as get a perspective on how to build continuous
certification for automotive vehicles using our incremental approach.

We start in Section 1 by providing an overview of the ISO 21434, and then in
Section 3 focusing more on the parts related to risk assessment and concept
phase. We describe in Section 4 our approach for the continuous vehicle
assessment and certification. Section 5 demonstrates our approach through an
illustrative example described in Section 2. After a discussion of related
work in Section 6, we conclude the white paper in Section 8.

Figure 1: Technical parts of the ISO 21434.

##  1 ISO 21434 at a Glance

As with the ISO 26262, the ISO 21434 is structured into multiple parts (a.k.a.
clauses) that tackle different aspects of cybersecurity. The technical parts
of the ISO 21434 are depicted in Figure 1. These parts are preceded by parts
defining the scope (Part 1), the normative references (Part 2), the terms and
abbreviations (Part 3) and general considerations (Part 4).

We provide next an overview of all the technical parts depicted in Figure 1,
while entering in more details about the Parts 8 (Risk Assessment Methods),
and 9 (Concept Phase) in Section 3.

In a nutshell, the Parts 5 and 6 contain the requirements for organizational
cybersecurity. Parts 7 and 8 describe activities and methods that can be used
during the lifecycle of a product to ensure the security of items. Part 9
details the requirements during concept phase. Parts 10 and 11 detail the
requirements during production, while Parts 12-14 detail the requirements
during the post-production. Finally, Part 15 details the requirements for
supplier management.

#### Part 5.

This part details the requirements for building and maintaining a
cybersecurity culture and governance. This is accomplished by setting
cybersecurity policies, rules and processes for overall cybersecurity
management and for project dependent cybersecurity management. The policies
and rules for the overall cybersecurity management shall (1) define the
organization-specific rules and processes for cybersecurity, _e.g._ , the
executive management’s commitment to manage security risks; (2) assign
responsabilities for security activities; (3) support the implementation of
cybersecurity; (4) institute and maintain cybersecurity culture; (5) perform
organizational cybersecurity audits; (6) manage the sharing of cybersecurity
information; (7) institute and maintain management systems that support
cybersecurity activities; (8) and provide evidence that the tools used do not
adversely affect cybersecurity.

#### Part 6.

This part describes the requirements for the management of cybersecurity
activities for the development of a specific project. The key objectives of
these requirements are (1) assign the responsibilities for cybersecurity
activities; (2) plan the cybersecurity activities; (3) create a cybersecurity
case that provide the high-level argument for the achieved degree of
cybersecurity.

#### Part 7.

This part details cybersecurity activities that can be performed at any stage
of the lifecycle. The main objective of these requirements are (1) to collect
relevant cybersecurity information; (2) to triage collected cybersecurity
information; (3) to assess cybersecurity events; (4) to identify and analyze
cybersecurity vulnerabilities; (5) to manage identified cybersecurity
vulnerabilities.

#### Part 8.

This part details methods that organizations can use to determine the risks to
stakeholders of cybersecurity threats. The main objective of the methods are
to (1) identify assets; (2) identify threat scenarios; (3) rate the impact;
(4) identify or update the attack paths for threat scenarios; (5) assess the
ease with which the identified attack paths can be exploited; (6) determine
the risk value of a threat scenario; and (7) select the appropriate risk
treatment.

#### Part 9.

This part details the requirements for the concept phase. Its main objectives
are (1) define the item, the operational environment and its interaction with
other items; (2) specify cybersecurity goals and cybersecurity claims; and (3)
specify cybersecurity requirements and allocate them to the item or to the
operational environment.

#### Part 10.

This part details the requirements for the product development. The main
objectives of this part are to (1) specify refined cybersecurity requirements
and architectural design; (2) verify that the refined cybersecurity
requirements and architectural design comply with cybersecurity requirements
from a higher level; (3) identify vulnerabilities in the design and manage
them accordingly; (4) provide evidence that the component complies with the
cyberscurity specification and does not contain undesired functionality
regarding cybersecurity.

#### Part 11.

This part details the activities for cybersecurity validation that are
performed after the integration of components. Its main objectives are (1) to
validate the cybersecurity goals and claims; (2) to confirm that the item
satisfies the cybersecurity goals; (3) to confirm that the residual risk is
acceptable.

#### Part 12.

This part details the requirements during the fabrication, assembly and
configuration of an item or component. Its objectives are (1) to apply the
cybersecurity requirement for post-development to the item or component; and
(2) to prevent the introduction of vulnerabilities during production.

#### Part 13.

This part describes the requirements for cybersecurity incident response and
updates. Its objectives are (1) to handle cybersecurity events that become a
cybersecurity incident; and (2) to preserve cybersecurity during and after
updates to items or components post-production.

#### Part 14.

This part details the requirements on decommissioning of an item or component.
The main objective is to ensure that the item or component is decommissioned
in a secure manner.

#### Part 15.

This part details the requirements for distributed cybersecurity activities.
The main objective of these requirements is to define the interactions,
dependencies, and responsibilities between customers and suppliers for
cybersecurity activities.

Section 4 describes our approach and how our approach can help to achieve the
objectives of Parts 8, 9 and 10 in an automated fashion. ISO 21434 has several
appendixes that illustrate how the ISO 21434 could be used in practice.
Section 5 illustrates our approach using the running example in the ISO 21434,
Annex G. This running example is described in Section 2.

Figure 2: Preliminary architecture of the headlamp system (Placeholder. Pic
needs to be beautified)

##  2 Running Example

We consider a headlamp system as the running example for this white paper. The
example is taken from ISO-21434, Annex G [3].

A headlamp system is responsible for turning on/off the headlamp of a vehicle
following the demand of the driver. It has two specific features, namely high-
beam light and low-beam light. If the headlamp is in high-beam mode, the
headlamp system switches the headlamp automatically to the low-beam mode when
an oncoming vehicle is detected. It returns automatically the headlamp to the
high-beam mode if the oncoming vehicle is no longer detected [3]. The headlamp
system is a critical system as harm, _e.g._ , accidents, may occur if the lamp
is unexpectedly turned off over night.

A preliminary architecture of the headlamp system is depicted in Figure 2. The
Body Control ECU sends messages to the Power Switch Actuator through the CAN
bus. These messages are requests to turn on/off the headlamp. A Camera ECU is
connected to the Power Switch Actuator so that oncoming vehicles can be
detected. This is needed to automatically switch the headlamp from high-beam
mode to low-beam mode and vice versa. The CAN bus is connected with a Gateway
ECU. This Gateway ECU controls access to the CAN bus for other ECUs located
outside the headlamp system such as the Navigation ECU. The Navigation ECU has
two interfaces, namely Cellular and Bluetooth, that may be used to send
requests to turn on/off the lamp. The Gateway ECU is also connected to a OBD-
II connector through another CAN bus. OBD-II is an on-board computer that
monitors data about the vehicle such as speed. Both the Bluetooth and Cellular
interfaces and the ODB-II connector are located outside the item
boundary444The description of the item boundary can include interfaces with
other item and components internal and/or external to the vehicle [3]. They
might be entry points for attacking the headlamp system, as discussed in the
following sections.

##  3 Risk Assessment

We enter into the details of the more technical parts of the ISO 21434, namely
the Parts 8, 9, and 10. Parts 9 and 10 may use the risk assessment methodology
described in Part 8. This methodology can be partially carried out with
automated support, thus enabling the automated production of several artefacts
required for the security assessment of an item.

Figure 3 depicts the activities described in Part 8 of the ISO 21434 for risk
assessment, including the artefacts that they are supposed to produce and
suggestions of methods that can be used to produce them.

In the following, we will describe these activities and illustrate them with
the running example from Section 2.

Figure 3: Activities, Artefacts and Methods mentioned in the Part 8. The full
arrow from an activity to an artefact specifies that this artefact is produced
at the end of the activity. A full arrow from an artefacts to an activity
specifies that this artefact is used as input to the corresponding activity.
Finally, the dashed arrows from an artefact to an activity specifies that the
artefact can used as input, but not required, to the corresponding activity.

#### Item Definition.

This is an early-on activity that defines the item555An item is a system or
combination of systems to implement a function. whose risk assessment is going
to be evaluated. During this activity, a preliminary architecture of the item
is described and the assumptions about its operational environment.

###### Example 3.1

Taking into account the running example from Section 2, an item can be the
entire headlamp system.

#### Asset Identification.

From the artefacts produced by the item definition activity, the item’s damage
scenarios666A damage scenario is an adverse consequence or undesirable result
due to the compromise of a cybersecurity property of an asset are identified.
Moreover, one should also enumerate the identified assets with cybersecurity
properties that would lead to a damage scenario. Three methods are suggested
by the ISO 21434 for carrying out this activity:

  * •

Impact rating: This method rates the impact of cyberattacks to assets.

  * •

Deriving Assets from Threat Scenarios:777A threat scenario is a statement of
potential negative actions that lead to a damage scenario. Threat scenarios
(produced in the threat scenario activity) can help identify key assets.

  * •

Pre-defined Catalogues: Existing catalogues provide a good source for
identifying assets.

###### Example 3.2

An asset from the headlamp is the CAN bus that transmits messages such as
requests to turn on/off the lamp. Cybersecurity properties relevant for the
CAN bus include integrity and availability. Integrity-wise, the CAN bus must
ensure accuracy and completeness of the message transmitted. Availability-
wise, the CAN bus must be available at all times whenever, _e.g._ , the Body
Control ECU requests to turn on/off the lamp. Damage scenarios for the CAN bus
include unexpected behavior like:

  1. 1.

Unexpected loss of head light: the headlamp turns off during night driving
resulting from the loss of integrity of CAN signal

  2. 2.

Unable to switch on head light: the Power Switch Actuator does not receive a
request from the Body Control ECU to turn on the lamp resulting from the loss
of availability of the CAN bus.

#### Threat Scenario Identification.

The threat scenarios for each damage scenario shall be identified. Notice that
a damage scenario may be associated with multiple threat scenarios. A threat
scenario may include the targeted asset, the compromised cybersecurity
property and the actions that need to be carried out by an attacker to
accomplish a damage scenario.

While there is a number of methodologies to carry out this activity [33], the
ISO 21434 proposes the use of the two following methods or their combination:

  * •

misuse-case elicitation: Threats scenario can often be identified by using
items in an possible, but unintended way.

  * •

taxonomy mnemonic-based approaches, such as STRIDE: Using threat categories
enables a systematic identification of threat scenarios. (See [33] for more
about this type of technique.)

For this activity, we believe that there should be an additional artefact
specifying what type of attacker is being considered. We find this important
as attacker intentions, (_e.g._ , is he a script-kid, cyber-criminal or a
agent supported by a government?, and capabilities, _e.g._ , which types of
channels is he able to hack?, lead to different types of threat scenarios.

###### Example 3.3

Consider the damage scenarios from Example 3.2. Threat scenarios for Damage
Scenarios 1 and 2 can be, respectively:

  1. 1.

An attacker spoofing a CAN signal leads to loss of integrity of the CAN
message of ‘Lamp request’.

  2. 2.

An attacker flooding the CAN bus buffer leads to loss of availability of the
CAN bus.

For the Threat Scenario 2, e.g., an attacker could flood the CAN bus with high
priority messages so that the lamp switch off requests may be processed with
significant delay.

This artefact describing the attacker would enable the use of formal
approaches to enumerate in an automated fashion threat scenario identification
by using precise mathematical models [18, 19, 28, 34]. We demonstrate this in
Section 5 illustrating how automation is a cornerstone for incremental
security engineering.

#### Impact Rating.

This activity takes as input the damage scenarios derived from the activity
Asset Identification. Each damage scenario is assessed according to four
impact categories for the stakeholders: safety (S), financial (F), operational
(O) and privacy (P). Other categories may be considered. Moreover, one of the
following impact rating is associated with the damage scenario: severe, major,
moderate, or negligible. Finally, safety related impacts shall be derived from
the ISO 26262 [2]. The resulting artefact is are the damage scenarios
assessments.

###### Example 3.4

The impact rating for the damage scenarios from Example 3.2 is described in
Table 1.

Regarding the first scenario, an accident might happen if the lamp switches
off over night driving. Hence, the safety impact for this scenario is major.
The operational impact is also major, because if the lamp is not working
properly the driver needs to take the vehicle to a mechanic for repair. The
damage impact for both financial and privacy categories are negligible for
this scenario.

For the second scenario, we consider that the driver is not able to switch on
the lamp over morning driving. The safety impact is then moderate, as the
impact would not be as severe as driving at night. The operational impact is
major, following the same reason explained for the first scenario. The damage
impact on financial and privacy are both negligible for this scenario.

Damage scenario | Safety | Financial | Operational | Privacy
---|---|---|---|---
Unexpected loss of head light | Major | Negligible | Major | Negligible
Unable to switch on head light | Moderate | Negligible | Major | Negligible
Table 1: Impact rating for the damage scenarios from Example 3.2.

While no concrete method is proposed by the ISO 21434 for assessing damage
scenarios, we point out that there are methods in the literature that can
help, in particular, assess damage scenarios related to safety. For example
[23] proposes methods for extracting in an automated fashion security relevant
information from safety analysis, _e.g._ , Fault Tree Analysis (FTA), Failure
Modes and Effects Analysis (FMEA), and safety cases.

#### Attack Path Analysis.

From the preliminary architecture, assumptions about the item/environment, and
the damage scenario, one produces the possible attack paths and their
association to threat scenarios. To do so, the ISO 21434 proposes the use,
possibly in combination, of both top-down, _e.g._ , Attack Trees, and bottom-
up, _e.g._ , vulnerability analysis. The attack paths shall include the
following information: vulnerabilities or weaknesses that could be exploited,
and how these could be exploited in an attack realizing a threat scenario.

###### Example 3.5

Both Bluetooth and Cellular interfaces may be accessible from the outside,
possibly by an attacker. Hence, we consider such interfaces as the entry point
vulnerabilities for the headlamp system.

Consider the threat scenarios from Example 3.3. A threat path for Threat
Scenario 1 can be:

  1. (a)

an attacker compromises the Navigation ECU from the Cellular interface to send
malicious signals to turn off the lamp during night driving,

  2. (b)

compromised Navigation ECU transmits the received malicious signals, and

  3. (c)

Gateway ECU forwards the malicious signals to Power Switch Actuator through
the CAN bus.

Note that this very same attack could be also carried out from the Bluetooth
interface. Similarly, a threat path for Threat Scenario 2 can be:

  1. (a)

an attacker compromises the Navigation ECU from the Bluetooth interface to
flood the CAN bus with high priority messages,

  2. (b)

compromised Navigation ECU transmits the received malicious messages, and

  3. (c)

Gateway ECU forwards the malicious messages to the CAN bus.

#### Attack Feasibility Rating.

Each attack path is rated according to the following categories: high, medium,
low or very low. The following three methods are suggested for rating attack
paths:

  * •

Attack Potential: The rating of an attack path is obtained by evaluating the
attack level taking into account core factors required, including elapsed
time, specialist expertise and item knowledge required, window of opportunity,
and equipment.

  * •

CVSS2: The attack path rating is obtained from common vulnerability scoring
systems, _e.g._ , available at https://www.first.org/cvss/.

  * •

Attack vector: The rating is determined by analysing the predominant attack
vector of the attack path.

###### Example 3.6

This example uses the attack potential approach. Table 2 shows the attack
feasibility rating for the attack path for Threat Scenario 1 from Example 3.5.
Note that the chosen parameters in Table 2 are samples only and may not
reflect reality.

Expertise is the skill level required to exploit an attack path. We consider
an expert is required to compromise the Navigation ECU though the Cellular
interface (attack step (a)). We consider that the total time for an expert to
exploit attack step (a) is 14 days. Equipment refers to hardware or software
required to exploit an attack path. We consider that the Cellular interface is
readily available and easily accessible by the attacker though, e.g., wireless
network. Therefore, the equipment required for exploiting attack step (a) is
standard. The knowledge required by the attack is public, as information
regarding the Cellular interface may be available on the Internet. Opportunity
is the amount of access required by the attack to exploit an attack path. We
consider that the opportunity is unlimited for step (a), because the Cellular
interface may be available without any time limitation to the attacker.

The attack potential approach considers the most expensive argument from each
attack step to compute the overall feasibility for the attacker’s path. For
example, if you need an expert for one attack step and a proficient for
another attacker step the overall value w.r.t. expertise is expert. The value
for each attack’s step is high (see Table 2). Hence, the overall attack
feasibility for the attack path for Threat Scenario 1 is high. We consider
that the overall attack feasibility for the attack path for Threat Scenario 2
is medium.

attack steps | expertise | time (#days) | equipment | knowledge | opportunity | level
---|---|---|---|---|---|---
(a) | expert | 14 | standard | public information | unlimited | high
(b) | expert | 7 | standard | restricted information | unlimited | high
(c) | proficient | 1 | standard | restricted information | unlimited | high
Table 2: Attack feasibility for the attack path for Threat Scenario 1 (Example
3.5).

#### Risk Determination.

The risk of threat scenarios are determined from the impact associated to its
corresponding damage scenario and the likelihood of its associated attack
paths. The risk are integers from 1 (lowest risk) to 5 (highest risk). A risk
matrix with the likelihood of attack paths and impact of the damage scenario
is a suggested method to carry out this activity.

###### Example 3.7

We can determine the risk value for both threat scenarios from Example 3.3
based on the risk matrix. To this end, consider both the impact rating from
Example 3.4 and the attack feasibility rating from Example 3.6.

We consider the safety impact rating only from Example 3.4. The risk value for
Threat Scenario 1 is 4, as the impact on safety is major and the likelihood of
its attack path is high. The risk value for Threat Scenario 2 is 2, as the
impact on safety is moderate and the likelihood of its attack path is medium.

#### Risk Treatment Decision.

From all the artefacts produced by the other activities, excepting the attack
feasibility rating which is optional, a risk treatment decision is produced
for each threat scenario. This artefact shall contain treatment options, such
as avoiding risk by removing risk sources, reducing risk, by, _e.g._ ,
inserting countermeasures, sharing or transferring the risk to, _e.g._ ,
suppliers or through insurance, or accepting or retaining the risk.

##  4 Incremental Approach

While the ISO 21434 is a step forward for the cybersecurity of connected
vehicles, it does not enter into the details of how to _efficiently
operationalize cybersecurity_. Given the complexity of connected vehicles,
carrying out all activities and producing all artefacts while still satisfying
the production timelines is not feasible without automation. Moreover,
cybesecurity is a continuous task as new vulnerabilities and attacks are
discovered (after vehicle production) requiring new countermeasures and
analysis.

We propose an incremental approach for ISO 21434 that enables continuous and
efficient cybersecurity based on three main pillars:

  * •

Rigorous Security Assessments: We propose security rigorous assessment
specification based on precise logical models with precise semantics. In
constrast with existing assessment languages, such as Goal Structure Notation
[6], that include textual elements, rigorous security assessments can be
automatically checked for missing assumptions or flaws in the argumentation.

Following our previous work [18, 17, 23], we propose domain specific languages
enabling the specification of system architectures, _i.e._ , components and
channels, including physical components, _e.g._ , CAN bus or ECUs, and safety
and security elements, _e.g._ , hazards and threats, as well as safety and
security architectural patterns, _e.g._ , safety and security monitors,
watchdogs, and firewalls.

From this language, we specify safety and security reasoning principles
written as logic programming rules. They specify, for example, under which
conditions a particular security pattern used in a given architecture can
guarantee the security against some given threats.

  * •

Incremental Assessment Maintenance: Whenever there is a change in the system,
_e.g._ , a new function is added or removed, or a change in the environment,
_e.g._ , a new threat is identified, the security assessment (argument and
evidence) shall be re-assessed.

A better approach than re-assessing the whole security assessment is to re-
assess the parts of the security assessment that are affected by these
incremental changes.

Automated incremental techniques have been developed for logic programming
[22, 26]. These work specifies algorithms for maintaining incrementally
(distributed) logic specifications used in databases and for network routing.
It seems possible to use and adapt these technique for the maintenance of
rigorous security assessments.

  * •

Automation: For increased process efficiency and reduced errors, increased
automated support is key.

Fortunately, the past years have witnessed the emergence of several automated
methods for the reasoning about system architectures by, _e.g._ , design
exploration, and the generation of evidence by using (formal) tools, such as
static analysers and model-checkers.

Figure 4: Incremental Security Approach for ISO 21434.

Figure 4 depicts how the three pillars above are used for the incremental
security engineering of ISO 21434.

Assume a given item definition $I$ for which a rigorous security assessment
has been carried out. Moreover, assume that there is a new increment,
$\Delta$, to the item, _e.g._ , new feature, or to the assumptions about the
environment, _e.g._ , new threat, that may invalidate the existing security
assessment for $I$.

The increment $\Delta$ is fed to our logic specification based on a Domain-
Specific Language on which architectural patterns and reasoning rules are
specified. By using design exploration techniques, our machinery automatically
generates a collection of item increments all of which are associated with a
security assessment.

A trade-off analysis among the generated item increments taking into account
other aspect, such as safety, performance, costs or quality. This trade-off is
supported by automated techniques enabling the selection of a best suited item
increment.

Finally, the certification artefacts, _e.g._ , security assessments, are
generated for the new item.

#### Benefits.

Our approach has two key benefits to existing approaches based on model-based
engineering, _e.g._ , GSN models, or textual models, _e.g._ , Excel sheets.

The first benefit is derived from the use of logic-based specifications for
the generation of security assessment. We, therefore, can rely on concepts
like soundness and completeness using precise semantics, instead of the
textual semantics which may be ambiguous in GSN models for example. This
enables the automatic checking of whether the argument used in security
assessments is correct or whether there are flaws in the reasoning or missing
assumptions. Moreover, using logic-based tools, it is even possible to
automatically enumerate which evidence is still required for completing a
security argument.

The second benefit is provided by the use of incremental methods and
automation. This enables a less human error-prone and efficient certification
process. Without these two elements, the overhead costs for certification is
very high as already observed in the avionics domain where the change of a
single line of code can imply costs of in the order of millions of USD.

A key impact of the two benefits above is that our proposed incremental
approach enables continuous continuous certification of connected vehicles.
Whenever there is a new incremental change to the item or to the environment
assumptions, new certification artefacts should be produced for certification.
The use of rigorous security assessment and automation, our incremental
approach enables the production of these artefacts in a highly automated
fashion. Moreover, certification authorities have to check whether the
artefacts are compliant to the ISO 21434. Since our methods are based on
rigorous security assessments, certification authorities can use existing
tools to automatically check whether the security arguments are correct.

##  5 Approach by Example

The goal of this section is to describe (1) how our approach can be used to
automate parts of the ISO 21434 risk assessment and (2) how our approach deals
with incremental changes in the system architecture, thus supporting
continuous security assessment and certification. To this end, we apply our
machinery to the headlamp system explained in Section 2.

###  5.1 Automating ISO 21434

Our machinery takes as input the preliminary system architecture as well as
the damage scenarios associated to both assets from the system architecture
and cyber-security properties. We also expect as input the impact rating for
each given damage scenario. These artefacts cover three activities from the
risk assessment, namely item definition, asset identification, and impact
rating.888Since these three activities are specific to the product being
developed, they cannot be automated in general.

Figure 5: Headlamp system with a Firewall

Consider the Example 3 from Section 3.2. The asset is the CAN bus transmitting
messages from the Gateway ECU to the item identified (_i.e._ , the headlamp
system). We consider the Damage Scenario 1 from Example 3, namely unexpected
loss of head light.

###### Example 5.1

In our DSL, one can specify both the identified asset and damage scenarios as
follows:

    asset(can2).
    dmgScenario("hl turns off
    over night driving",can2,int,
    [maj,neg,maj,neg]).

The argument can2 denotes a Controller Area Network (CAN) used to communicate
between ECUs. The fact asset(can2) denotes that can2 is an asset. The fact
dmgScenario denotes a damage scenario (headlamp turns off over night driving)
associated with can2 and cyber-security property int (short for integrity).
Following the impact rating given in Table 1, we specify the impact rating for
this scenario as [maj,neg,maj,neg] (maj and neg are short for major and
negligible, respectively).

Note that we omit how the system architecture is specified in our DSL. We
refer the interested reader to [17] for detailed description on how to specify
elements of the system architecture such as channels and components.

The next activities are the threat scenario identification and attack path
analysis. We can automate these activities. Our reasoning principles specify a
potential threat for each asset associated to a damage scenario. A potential
threat becomes an actual threat if the asset may be reached by an attacker.
Our machinery expects as input the public elements of the system architecture,
_i.e._ , the elements that may be accessible by external users. We consider
the Bluetooth and Cellular interfaces and the OBD-II connector as public
elements. We have specified reachability rules to automatically check whether
from a public element (_e.g._ , Bluetooth) an attacker may access an asset
(_e.g._ , CAN bus) through a path P. Our machinery identified three possible
paths to access the CAN bus (located between Gateway ECU and headlamp system)
from the public elements.

Next in the risk assessment are the attack feasibility rating and risk
determination. As shown in Table 2, a level (_e.g._ , high) is assigned to
each step of an attacker’s path. Our machinery takes as input the assigned
level to each attack step of a given attack path. Our reasoning principles
then automatically calculates the overall level of an attack path following
the attack potential approach. Given the overall level of an attack path, we
specify reasoning principles rules to automatically calculate the risk
determination for each threat following the risk matrix method.

###### Example 5.2

Our machinery outputs the following w.r.t. the attack feasibility rating and
risk determination.

    {attFS([can2,[can2,gw,can1,bt],
    int,maj],high)}
    {riskDT([can2,[can2,gw,can1,bt],
    int,maj],4)}

For both facts, attFS and riskDT, the first argument is the threat ID. The
threat ID is composed of the identified asset (can2), the attacker’s path
(direction from right to left), the cyber-security property (integrity), and
the impact rating (major). For the sake of example, we are considering the
safety impact rating only. The last argument of attFS and riskDT denotes,
respectively, the overall feasibility rating (high), and the risk value for
the identified threat (4).

The last activity in the risk assessment is risk treatment decision. Our
machinery enables the automated recommendation of security patterns to
mitigate identified threats. To this end, we specify reasoning principles for
(1) _Mitigation_ : which threats can be mitigated by a given deployed security
pattern and which threats cannot be mitigated. (2) _Security Pattern
Recommendation:_ which security patterns, _e.g._ , Firewall or Security
Monitor, can be used and where exactly they should be deployed to mitigate
threats that have not yet been mitigated.

(a) Security monitor associated to the Body Control ECU

(b) Security monitor associated to the CAN bus

Figure 6: Headlamp system with a Firewall and a Security Monitor

We run our machinery on the headlamp system. Our machinery recommended to use
a firewall to mitigate the identified threat. The architecture of the headlamp
system with a firewall is depicted in Figure 5. A firewall is deployed between
the Gateway ECU and the headlamp system. The goal is to mitigate malicious
incoming flows from public elements such as Bluetooth, Cellular or OBD-II
connector.

In summary, our machinery can automate the following parts of the ISO 21434
risk assessment:

  * •

identification of threats given damage scenarios (threat scenario
identification)

  * •

identification of the attacker’s path to exploit identified threats (attack
path analysis)

  * •

calculation of the overall level of an attacker’s path (attack feasibility
rating) given the assigned level to each attack step of an attack path

  * •

calculation of the risk value for each identified threat (risk determination)

  * •

recommendation of security patterns to mitigate identified threats (risk
treatment decision)

###  5.2 Incremental Security Approach

To illustrate our incremental approach depicted in Figure 4, assume that there
is an incremental change in the headlamp system as follows.

#### Increment.

Consider as increment the ability to perform software updates on the Body
Control ECU. We assume that at the some point during the life-cycle of the
headlamp system a software update on the Body Control ECU is performed. This
software update may be performed by, _e.g._ , a wired connection through the
OBD-II connector, or through-the-air by using the Bluetooth or Cellular
interfaces.

The considered increment may lead to new threats to the headlamp system as new
flows are generated between public elements (_e.g._ , OBD-II connector) and
the headlamp system. A potential threat is an attacker injecting malicious
code into the Body Control ECU (asset) in an unauthorized fashion. This
potential threat may lead to loss of integrity of the Body Control ECU due to
unauthorized access.

Following our approach illustrated in Figure 4, our machinery takes the inputs
described in Section 5.1 and a potential threat on the Body Control ECU. Our
machinery then automatically checks whether the given potential threat is an
actual threat by checking possible attacker’s paths. Three attacker’s paths
are identified, initiated from the OBD-II connector, Bluetooth and Cellular
interface, respectively.

#### Automated Design Exploration.

Next, our recommendation machinery automatically suggests solutions for
mitigating the identified threat. Two solutions are depicted in Figure 6. Our
recommendation machinery suggested the deployment of a security monitor to
mitigate the identified threat. The goal is to mitigate malicious access from
public elements by enforcing access control policies, such as “only authorized
users from public elements may write data into the Body Control ECU”.

Figure 6(a) illustrates a security monitor associated to the Body Control ECU,
whereas Figure 6(b) illustrates a security monitor associated to the CAN bus.
Both solutions can mitigate the identified threat. The main difference is that
the former may be deployed by means of software instrumentation, and the
latter may be deployed as a physical proxy between the Gateway ECU and the
Body Control ECU.

#### Trade-off Analysis.

A trade-off analysis should be carried out to help engineers to choose the
best suited solution for the system. For example, the security monitor
deployed as a proxy might be more expensive (performance-wise) than the one
deployed by means of software instrumentation. The reason is that the deployed
proxy shall intercept each incoming message from the Gateway ECU regardless
the destination of the message. The security monitor associated to the Body
Control ECU shall only intercept messages destined to the Body Control ECU
itself. Hence, based on the performance impact of such solutions, one may
choose the security monitor illustrated in Figure 6(a) over the one
illustrated in Figure 6(b).

#### Automated Security Argument Generation.

Once a design solution is chosen, our machinery can construct the security
argument based on the (previously and newly) identified threats, and the
security pattern that have been proposed. That is, our machinery can construct
security arguments in the form of a GSN model. Figure 7 depicts a (piece of
the) GSN model derived from our machinery for the headlamp system. It includes
the identified threat as well as which security pattern can used to mitigate
which threat.

Figure 7: Security argument in the form of a GSN model derived by our
machinery for the headlamp system

##  6 Related Work

Recent white papers [12][32] also give an overview on ISO 21434. They both
discuss the set of guidelines proposed for securing automotive vehicles. The
white paper [12] focuses on the activities performed in the risk assessment.
It proposes an approach to automotive cybersecurity engineering. This approach
suggests the use of offense and defense mechanisms for helping engineers to
implement the guidelines of ISO 21434. The white paper [32] proposes a layered
approach for securing automotive vehicles. The advantage of this approach is
to reduce the probability of an attack’s success by providing multi-layered
response to attacks for protection, detection, and response. Our white paper
proposes an incremental approach to enable the continuous certification for
automotive vehicles.

ThreatGet is a model-based engineering tool for security analysis [30][5].
ThreatGet can perform security analysis on system architectures following the
ISO 21434 risk assessment. ThreatGet automatically identifies threats given a
path between a source and a target element. ThreatGet lists possible
countermeasure that can be selected by users to mitigate the identified
threats. The main advantages of our machinery over ThreatGet include: Our
machinery can automate more parts of the ISO 21434 risk assessment such as
both attack feasibility rating and risk determination. Our recommendation
machinery explicit shows where countermeasures shall be deployed in the system
architecture to mitigate identified threats. Our machinery enables the
construction of security arguments in the form of a GSN model. To the best of
our knowledge, ThreatGet does not support the generation of such security
arguments. Finally, our approach considers an incremental approach to risk
assessment, thus enabling continuous certification.

Goal-oriented approaches, such as GSN [6] and KAOS [20], have been used for
modeling safety cases. Similar to these approaches, several papers [13] have
proposed using Attack Trees [31] for modeling risk assessments. Extensions
include quantitative models for evaluating how defenses can be used to
mitigate attacks [13, 15]. A key difference of our approach is that we focus
on automated analysis. Whereas in the previous work, Attack Defense trees
shall be manually constructed, our approach can carry out such analysis in an
automated fashion.

A number of works [29, 21, 24] have proposed using general models encompassing
both safety and security concerns. For example, GSN extensions with security
features, so that in a single framework, one can express both security and
safety [24]. Moreover, our previous work [23, 27] proposed mechanisms for
extracting security relevant information from safety analysis, such as Fault
Tree Analysis or Failure Mode and Effects Analysis.

While this white paper is inspired by this previous work, we focus on
incremental methods for safety and security co-analysis. As illustrated by our
running example and in our previous work [18], our approach takes into account
both safety and security during trade-offs to determine the best design
increments.

##  7 Next Research Directions

Our vision is to build an incremental safety and security co-analysis with
patterns. To this end, we aim at developing an incremental process for system
safety and security assurance cases using automated methods that incorporate
both safety and security reasoning principles.

We have made promising steps towards our vision. We have proposed safety
reasoning principles with patterns in [17]. Our security reasoning principles
with patterns have been partially presented in this white paper and in [18].

Some more research is needed to fully realize our vision. The next steps
toward our vision include:

  1. 1.

Make Assumptions Explicit: A safety case often contains implicit assumptions
(_e.g._ , decisions based on own experience), _e.g._ , assumptions on channel
independence or lack of side-channel attacks. Often no evidence is collected
validating such implicit assumptions which may lead to implementation errors.
Our intention is to make such assumptions more explicit, _e.g._ , what are the
assumptions needed for choosing a particular safety or security pattern? We
will augment our reasoning principles rules with explicit assumptions to
enable a more precise analysis w.r.t., _e.g._ , pattern selection.

  2. 2.

Improve our trade-off analysis: Safety and security co-analysis may lead to
interrelations. There can be conflicts, synergies or no conflicts between
safety and security. For example, implementing a firewall needed for security
reasons might lead to new faults (safety) as the firewall might incorrectly
blocks messages from the system. Upon finding out the trade-offs, safety and
security engineers will possibly decide what actions to take to solve
conflicts or synergies. Engineers shall take into account aspects, _e.g._ ,
performance or safety, to reach a consensus w.r.t. solutions for the system.
We will investigate directions on how to integrate such aspects into our
trade-off analysis in a (semi-) automated fashion. As a result, we can provide
a more holistic trade-off analysis with multiple aspects that can assist
engineers to decide the best design solution for the system.

  3. 3.

Tooling. Currently, we are using the logic programming engine DLV [25] for
automating safety and security reasoning principles such as pattern
recommendation. As mentioned earlier, performance is one of the aspects we
plan to integrate into our trade-off analysis. To this end, we need to specify
reasoning principles with timing so that we can analyze possible trade-offs
regarding the runtime overhead caused by solutions. DLV might not be suitable
for specifying such reasoning principles with timing. Moreover, we also plan
to specify reasoning principles on the probability of faults occurrence so
that we can enable a more precise analysis w.r.t. pattern selection. We will
investigate what tools can be used to help us to specify such reasoning
principles. For example, we could combine DLV with SMT-solvers so that we can
specify reasoning principles with fault probabilities.

  4. 4.

Integration into Model-Based Engineering Approaches. One can imagine our
machinery (including DSL, patterns, and reasoning principles) as a back-end
for model-based engineering approaches. That is, our machinery can be used as
a process inside a model-based engineering tool to provide rigorous safety and
security assessment to the given system architectures. Therefore, we plan to
integrate our machinery into the model-based engineering tool AutoFOCUS3 [1].
This integration leads to a number of benefits include (a) assisting engineers
to operationalize the process of providing rigorous assessment to system
architecture, and (b) improving the usability of our machinery by providing a
user-friendly graphical interface tool for engineers.

##  8 Conclusions

This white-paper describes the ISO 21434 standard for automotive security
focusing on the activities and artefacts that shall be produced by engineers.
Moreover, we go beyond the ISO 21434 describing our incremental security
engineering approach. We illustrate by example how our approach can support
engineers in constructing arguments for item security claims. Finally, we
point to several research directions that we are currently pursuing in order
to realize the proposed incremental approach in an operational setting.

We strongly believe that the ISO 21434 is a step forward towards increasing
the vehicle security as it addresses some of the key challenges upcoming with
the increase adoption of ICT in vehicles. However, without adequate automated
methods, it may fall short given the tight deadlines in placing vehicles in
the market. This pressure may cause engineers to cut corners by overlooking
(implicit) assumptions or not taking into account the trade-offs between
safety and security thus leading to insecure vehicles.

Therefore, the next years will require a co-joint effort between research
institutes, industry and certification bodies to develop the framework needed,
_i.e._ , methods and processes, to efficiently derive appropriate security
arguments supported by comprehensive evidence.

## References

  * [1] AF3 – AutoFOCUS 3.  More information at https://af3.fortiss.org/.
  * [2] ISO 26262, Road vehicles — Functional safety — Part 6: Product development: software level.  Available from https://www.iso.org/standard/43464.html.
  * [3] ISO/SAE AWI 21434, Road Vehicles – Cybersecurity engineering.  Under development.
  * [4] SAE - Society of Automotive Engineers, SAE J3061 - Cybersecurity Guidebook for Cyber-Physical Automotive Systems.  Available from https://www.sae.org/standards/content/j3061_201601/.
  * [5] Threatget - threat analysis and risk management.  Available at https://www.threatget.com/.
  * [6] GSN Community Standard Version 1.  2011\.  Available at http://www.goalstructuringnotation.info/documents/GSN_Standard.pdf.
  * [7] Hackers remotely kill a Jeep on the highway—with me in it, 2015.  Available at https://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/.
  * [8] Q1 2019 sees a rapid growth of automotive cyber incidents, 2019.  Available at https://upstream.auto/blog/q1-2019-sees-a-rapid-growth-of-automotive-cyber-incidents/.
  * [9] The changing face of automotive cyber attacks, 2020.  Available at https://www.trustonic.com/opinion/the-changing-face-of-automotive-cyber-attacks/.
  * [10] Cyber attacks on automotive sector picking up speed, 2020.  Available at https://www.hornetsecurity.com/en/security-information/cyber-attacks-on-automotive-sector-picking-up-speed/.
  * [11] Upstream security’s 2020 global automotive cybersecurity report, 2020.  https://upstream.auto/upstream-security-global-automotive-cybersecurity-report-2020/.
  * [12] Dr. Hasan I. Akram.  Bridging the Cybersecurity Gap in Automotive: The Upcoming ISO/SAE 21434 and its Implications for Automotive Cybersecurity Engineering.  Matrickz GmbH, White Paper, 2020.
  * [13] Alessandra Bagnato, Barbara Kordy, Per Håkon Meland, and Patrick Schweitzer.  Attribute decoration of attack-defense trees.  IJSSE, 3(2):1–35, 2012.
  * [14] Tewodros A. Beyene and Harald Ruess.  Evidential and continuous integration of software verification tools.  In Formal Methods - 22nd International Symposium, FM 2018, Held as Part of the Federated Logic Conference, FloC 2018, Oxford, UK, July 15-17, 2018, Proceedings, pages 679–685, 2018.
  * [15] Stefano Bistarelli, Fabio Fioravanti, and Pamela Peretti.  Defense tree for economic evaluations of security investment.  In ARES 06, pages 416–423, 2006.
  * [16] Simon Cruanes, Grégoire Hamon, Sam Owre, and Natarajan Shankar.  Tool Integration with the Evidential Tool Bus.  In Verification, Model Checking, and Abstract Interpretation, 14th International Conference, VMCAI. Proceedings, 2013.
  * [17] Yuri Gil Dantas, Antoaneta Kondeva, and Vivek Nigam.  Less manual work for safety engineers: Towards an automated safety reasoning with safety patterns.  In International Conference on Logic Programming (ICLP), 2020.
  * [18] Yuri Gil Dantas, Antoaneta Kondeva, and Vivek Nigam.  Towards automating safety and security co-analysis with patterns (position paper).  In Safecomp, 2020.
  * [19] Yuri Gil Dantas, Vivek Nigam, and Carolyn Talcott.  A formal security assessment framework for cooperative adaptive cruise control.  In IEEE Vehicular Networking Conference (VNC), 2020.
  * [20] Anne Dardenne, Axel van Lamsweerde, and Stephen Fickas.  Goal-directed requirements acquisition.  Sci. Comput. Program., 20(1-2):3–50, 1993.
  * [21] Edward Griffor, editor.  Handbook of System Safety and Security.  2016\.
  * [22] Ashish Gupta, Inderpal Singh Mumick, and V. S. Subrahmanian.  Maintaining views incrementally.  In Peter Buneman and Sushil Jajodia, editors, Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, Washington, DC, USA, May 26-28, 1993, pages 157–166. ACM Press, 1993.
  * [23] Antoaneta Kondeva, Carmen Carlan, Harald Ruess, and Vivek Nigam.  On computer-aided techniques for supporting safety and security co-engineering.  In The 9th IEEE International Workshop on Software Certification WoSoCer, 2019.
  * [24] Per Håkon Meland, Elda Paja, Erlend Andreas Gjære, Stéphane Paul, Fabiano Dalpiaz, and Paolo Giorgini.  Threat analysis in goal-oriented security requirements modelling.  Int. J. Secur. Softw. Eng., 5(2):1–19, 2014.
  * [25] Wolfgang Faber Thomas Eiter Georg Gottlob Simona Perri Francesco Scarcello Nicola Leone, Gerald Pfeifer.  The dlv system for knowledge representation and reasoning.  ACM Trans. Comput. Logic, 7:499–562, 2006.
  * [26] Vivek Nigam, Limin Jia, Boon Thau Loo, and Andre Scedrov.  Maintaining distributed logic programs incrementally.  Computer Languages, Systems & Structures, 38(2):158–180, 2012\.
  * [27] Vivek Nigam, Alexander Pretschner, and Harald Ruess.  Model-based safety and security engineering.  White Paper, 2018.
  * [28] Vivek Nigam and Carolyn L. Talcott.  Formal security verification of industry 4.0 applications.  In 24th IEEE International Conference on Emerging Technologies and Factory Automation, ETFA 2019, Zaragoza, Spain, September 10-13, 2019, pages 1043–1050, 2019.
  * [29] Nicola Nostro, Andrea Bondavalli, and Nuno Silva.  Adding security concerns to safety critical certification.  In Symposium on Software Reliability Engineering Workshops, 2014\.
  * [30] Magdy El Sadany, Christoph Schmittner, and Wolfgang Kastner.  Assuring compliance with protection profiles with threatget.  In Alexander B. Romanovsky, Elena Troubitsyna, Ilir Gashi, Erwin Schoitsch, and Friedemann Bitsch, editors, Computer Safety, Reliability, and Security - SAFECOMP 2019 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku, Finland, September 10, 2019, Proceedings, volume 11699 of Lecture Notes in Computer Science, pages 62–73. Springer, 2019.
  * [31] B. Schneier.  Attack trees: Modeling security threats.  Dr. Dobb’s Journal of Software Tools, 24:21–29, 1999.
  * [32] Vit Sembera.  ISO/SAE 21434: Setting the Standard for Connected Cars’ Cybersecurity.  Trend Micro Research, White Paper, 2020.
  * [33] Adam Shostack.  Threat Modeling: Designing for Security.  Wiley.
  * [34] Abraão Aires Urquiza, Musab A. AlTurki, Max I. Kanovich, Tajana Ban Kirigin, Vivek Nigam, Andre Scedrov, and Carolyn L. Talcott.  Resource-bounded intruders in denial of service attacks.  In 32nd IEEE Computer Security Foundations Symposium, CSF 2019, Hoboken, NJ, USA, June 25-28, 2019, pages 382–396, 2019.