# Design, Monitoring, and Testing of Microservices Systems:
The Practitioners’ Perspective

Muhammad Waseem<EMAIL_ADDRESS>School of Computer Science, Wuhan
University, 430072 Wuhan, China  Peng Liang<EMAIL_ADDRESS>Mojtaba Shahin
<EMAIL_ADDRESS>Department of Software Systems and Cybersecurity,
Faculty of Information Technology, Monash University, 3800 Melbourne,
Australia  Amleto Di Salle<EMAIL_ADDRESS>Department of Information
Engineering, Computer Science and Mathematics, University of L’Aquila, I-67100
L’Aquila, Italy  Gastón Márquez<EMAIL_ADDRESS>Department of
Electronics and Informatics, Federico Santa María Technical University,
4030000 Concepción, Chile

###### Abstract

Context: Microservices Architecture (MSA) has received significant attention
in the software industry. However, little empirical evidence exists on design,
monitoring, and testing of microservices systems.

Objective: This research aims to gain a deep understanding of how
microservices systems are designed, monitored, and tested in the industry.

Method: A mixed-methods study was conducted with 106 survey responses and 6
interviews from microservices practitioners.

Results: The main findings are: (1) a combination of domain-driven design and
business capability is the most used strategy to decompose an application into
microservices, (2) over half of the participants used architecture evaluation
and architecture implementation when designing microservices systems, (3) API
gateway and Backend for frontend patterns are the most used MSA patterns, (4)
resource usage and load balancing as monitoring metrics, log management and
exception tracking as monitoring practices are widely used, (5) unit and end-
to-end testing are the most used testing strategies, and (6) the complexity of
microservices systems poses challenges for their design, monitoring, and
testing, for which there are no dedicated solutions.

Conclusions: Our findings reveal that more research is needed to (1) deal with
microservices complexity at the design level, (2) handle security in
microservices systems, and (3) address the monitoring and testing challenges
through dedicated solutions.

###### keywords:

Microservices Architecture , Design , Monitoring, Testing, Industrial Survey

††journal: Journal of Systems and Software

iint iiint TXFiint TXF iiint

##  1 Introduction

Microservices Architecture (MSA)111All the abbreviations used in this paper
are provided in Table LABEL:tab:Abbreviations. inspired by Service-Oriented
Architecture (SOA) has gained immense popularity in the past few years [1].
The International Data Corporation (IDC) predicted that by the end of 2021,
80% of cloud software will be developed using the MSA style [2]. With the MSA
style, an application is designed as a set of business-driven microservices
that can be developed, deployed, and tested independently [3]. MSA follows a
“share-as-little-as-possible” architecture approach and communicates with each
other through an API layer [4, 5]. In contrast, SOA adopts a “share-as-much-
as-possible” architecture approach and typically uses the Enterprise Service
Bus for communication purpose [4, 5]. Another key difference between MSA and
SOA is that MSA advocates one data storage per microservice, while SOA uses
one data storage for the whole system [4]. Moreover, the systems that adopt
the MSA style (i.e., microservices systems) have a better fault-tolerance than
SOA-based systems [6]. The advantages of the MSA style are enormous, and the
key drivers of adopting MSA are faster delivery, improved scalability, and
greater autonomy compared to other architectural styles (e.g., SOA) [7].

There is a growing body of literature investigating different aspects of MSA
[8, 9, 10, 11, 12, 13, 14]. One notable research area is how to migrate a
monolithic system to MSA (e.g., [2, 15, 16, 17]). Others propose solutions for
securing microservices systems (e.g., [1, 18]). Researchers have recently
shown an increasing interest in understanding how microservices systems are
developed and migrated in the industry (e.g., [19, 20, 21]). Knoche and
Hasselbringa conducted a survey study to investigate the drivers, barriers,
and goals of adopting MSA [22]. Ghofrani and Lübke reported the services
boundaries identification techniques (e.g., Domain-Driven Design (DDD)) and
notations used to design and develop microservices systems [23]. The survey
study conducted by Viggiato et al. reports the popular programming languages,
technologies, advantages, and challenges in microservices systems [24].
Haselbock et al. surveyed MSA experts to identify essential design areas
(e.g., infrastructure design) and design challenges of microservices systems
[25].

In another study, Baškarada et al. empirically explored the practitioners’
perspective on the advantages of MSA over monolithic architectures and the
challenges faced by MSA practitioners when developing microservices systems
[26]. Zhang et al. examined the characteristics, benefits, and costs of MSA in
practice [27]. Zhou et al. investigated the fault analysis and debugging of
microservices through an industrial survey [28]. In the same line, Wang et al.
conducted a mixed-method study (i.e., interview and survey) with practitioners
to collect the best practices, challenges, and solutions for the successful
development of microservices systems [29]. Taibi et al. conducted a mixed-
method study (i.e., interview and survey) to investigate the motivations
(e.g., maintainability, scalability), issues (e.g., monolith decoupling,
database migration, and data splitting), and benefits (e.g., maintainability
improvement, scalability improvement) of microservices architecture [16]. They
also developed a migration process framework which is composed of three parts
(i.e., system structure analysis, definition of the system architecture, and
prioritization of feature development) [16]. The mixed-method study by Di
Francesco et al. focused on the activities (e.g., domain decomposition,
services identification) and challenges (e.g., high coupling among parts of
the legacy systems, identification of the boundary of each service) that
practitioners face when migrating towards MSA [17].

Despite these efforts, it is little empirically known how the software
industry designs, monitors, and tests microservices systems. These three
aspects play a significant role in succeeding with microservices systems [6,
30]. The design of a microservices system (i.e., MSA design) provides a basis
for communicating architectural decisions among stakeholders, exposing system
structure, realizing use cases, and addressing both functional and quality
requirements [31]. However, to maintain the benefits of microservices systems
(e.g., flexibility and separation of concerns), it is essential to reduce
complexity at the design level. Monitoring and testing help to achieve a
better understanding of the system and improve quality throughout the entire
development process. However, microservices systems are increasingly complex
and need more intensive monitoring and testing than other types of systems
(e.g., SOA, monolith) [6]. This stems from the unique characteristics of
microservices systems: greater complexity, polyglot programming languages,
heterogeneous and independent microservices, communication behavior at
runtime, deployment infrastructure model (e.g., application server vs.
container), frequent changes, and several others [32, 33]. Hence, software
development organizations need to develop new monitoring and testing
strategies or evolve traditional ones to match the unique characteristics of
microservices systems.

This study aims to bridge the gap mentioned above by providing a comprehensive
understanding of how microservices systems are designed, monitored, and tested
in practice. To this end, we conducted a mixed-methods study, including a
relatively large-scale survey with 106 responses and 6 interviews from
microservices practitioners. Overall, our results suggest that a combination
of DDD and business capability is the most effective strategy to decompose an
application into microservices and addressing security and availability is a
significant concern for MSA practitioners. We found that “API gateway” and
“Backend for frontend” patterns are the most used MSA patterns for providing
the entry point to clients’ requests of microservices systems. We observed
that MSA practitioners consider “clearly defining the boundaries of
microservices” as the most critical challenge when designing microservices
systems. Concerning monitoring microservices systems, most of the respondents
collect metrics that support monitoring quality aspects, such as performance
(e.g., resource usage), availability (e.g., ratio of the system available time
to the total working time), scalability (e.g., load balancing), and
reliability (e.g., service level agreement). About testing microservices
systems, we found that End-to-End (E2E) testing and contract related (e.g.,
consumer-driven) testing are gaining space in the industry. Our findings also
indicate that the complexity of microservices systems poses challenges for
their design, monitoring, and testing.

This study makes the following three major contributions:

  1. 1.

It brings quantitative and qualitative evidence of how practitioners design,
monitor, and test microservices systems.

  2. 2.

It identifies the challenges that practitioners face and the solutions
employed by them when designing, monitoring, and testing microservices
systems.

  3. 3.

It provides an in-depth discussion on the findings from the microservices
practitioners’ perspective and implications for researchers and practitioners.

The rest of this paper is structured as follows: Section 2 presents the
background of the survey. Section 3 describes the research method in detail.
Section 4 report the survey results. Section 5 and Section 6 discuss the
findings and threats to the validity of the survey respectively. Section 7
summarizes related work. Section 8 concludes this survey with future research
directions.

##  2 Background

Microservices is an architecture style where an application is built as a set
of independent services. The main characteristics of the MSA style are (i)
small, i.e., a service accomplishes a single business capability; and (ii)
autonomous, i.e., a single deployable unit contains all needed data to be
independently executed [30]. In contrast, a monolithic system implements
features as a single unit to be deployed as a whole [34]. In recent years,
microservices have become popular, and the software industry has moved from a
monolithic architecture to MSA. A substantial body of literature (e.g., [7,
35, 36, 37, 38, 39]) has concluded that MSA is significantly different from
monolithic architecture and SOA in terms of design, implementation, test, and
deployment. MSA contains a suite of fine-grained services working together as
a single application that can be executed independently, while in a monolithic
architecture, all functionalities are encapsulated into one single application
that cannot be executed independently [35]. MSA is more fault-tolerant, highly
scalable, and loosely coupled than monolithic architecture [35]. MSA and SOA
are also different in several aspects. For instance, MSA-based systems can use
shared or single database dedicated to a specific microservice, whereas SOA-
based systems can only use a shared database [39]. In MSA, each service can
run its own processes and communicate via lightweight mechanisms (e.g., HTTP,
REST), whereas in SOA, each service shares a common communication mechanism
(e.g., Enterprise Service Bus) [5].

This section presents different aspects related to design, monitoring, and
testing of microservices systems that are investigated in this study.

###  2.1 Design of microservices systems

According to Bass et al. [40] “the software architecture of a system is the
set of structures needed to reason about the system, which comprises software
elements, relations among them, and properties of both”. The design elements
of microservices systems are (micro)services, views (i.e., solutions),
processes, tools, culture, and organization structure [41]. This survey
investigates several aspects of the MSA design, such as application
decomposition and implementation strategies, architecting activities,
description methods, design patterns, Quality Attributes (QAs), skills
required to design and implement MSA, and MSA design challenges and their
solutions. We briefly describe some of these aspects below.

Architecting activities: The architecture of a software-intensive system is
the result of various architecting activities that are performed by related
stakeholders. Hofmeister et al. defined a general model of architecture
design, which is composed of a set of architecting activities, including
architectural analysis, synthesis, and evaluation [42]. Architecting
activities are widely explored in the context of monolithic architecture, but
the activities are rarely investigated in the context of MSA. We identified
only one Systematic Mapping Study (SMS), i.e., [10], that reports architecting
activities in the context of MSA. In this study, we investigated the level of
agreement and disagreement of the practitioners about the general architecting
activities employed in the context of MSA. According to the literature (e.g.,
[42, 43, 44]), the general architecting activities we considered for MSA
include Architectural Analysis (AA), Architectural Synthesis (AS),
Architectural Evaluation (AE), Architectural Implementation (AI), and
Architectural Maintenance and Evolution (AME).

MSA description methods: Architecture description methods consist of a set of
practices that are used to express the architecture of a system-of-interest
[45]. To describe the architecture of software systems, architects may use
various notations [46], such as boxes and lines, Unified Modeling Language
(UML), Architectural Description Languages (ADLs), and Domain-Specific
Languages (DSLs). In this study, we investigated the description methods and
diagrams used by microservices practitioners to describe and represent the
architecture of microservices systems.

MSA design patterns: Patterns are reusable solutions to occurring problems in
specific contexts [47]. Since the emergence of MSA, dozens of patterns have
been documented for implementing different aspects (e.g., data management,
deployment) of microservices systems [11, 48, 49, 50, 51]. We investigated how
often microservices practitioners use various patterns to address the problems
related to design of MSA-based systems, such as application decomposition,
data management, deployment, services communication, server-side service
discovery, and security in the context of microservices systems.

Quality attributes: A QA is a “measurable or testable property of a system
that is used to indicate how well the system satisfies the needs of its
stakeholders” [40]. Architecture design of software systems is the first phase
of software development, in which QAs can be satisfied. Some empirical studies
(e.g., [10, 11, 13, 14]) on MSA have taken QAs into consideration, but the
results reported are not from the practitioners’ perspective. The ISO/IEC
25010 standard regulates a comprehensive set of QAs of software systems [52],
but these QAs are considered with different priorities in the design of
microservices systems [8]. Thus, we decided to investigate the importance of
those QAs that are reported in the existing studies (e.g., [49, 53, 14]) in
the context of MSA.

Besides the aspects described above, we also investigated several other
aspects, such as application decomposition strategies (e.g., DDD), development
processes (e.g., DevOps, Scrum, XP), architectural components for
microservices systems, and skills required to design MSA. Also, we identified
the MSA design challenges, solutions, as well as the impact of the MSA
challenges on monitoring and testing of microservices systems.

###  2.2 Monitoring of microservices systems

A microservices system is primarily based on isolated microservices that can
be developed, deployed, and operated independently [54]. Microservices
interact with each other at runtime to exchange information. The runtime
architecture of microservices systems is continuously evolved. Thus,
information on the system architecture and service interaction and behavior
needs to be generated and updated automatically [55]. The dynamic nature of
microservices systems needs monitoring infrastructure to diagnose and report
errors, fault, failure, and performance issues. In this study, we investigated
the monitoring metrics, monitoring practices, tools, challenges, and solutions
for monitoring microservices systems.

Monitoring metrics are numerical representations of data measured over
intervals of time that generally come from resource usage and system behavior
[56, 33]. Monitoring metrics can be used to predict the system’s future
behavior. We identified 12 monitoring metrics from grey literature, e.g., [33,
57, 58] (see Figure 15) that are used for infrastructure-level monitoring and
microservices monitoring. Specifically, metrics such as resource usage (e.g.,
CPU, memory), threads, the status of dependencies, and database connections
are used for infrastructure-level monitoring. In contrast, language-specific
metrics (e.g., garbage collector behaviour, database latency), availability,
Service-Level Agreement (SLA), endpoint success, and errors and exceptions are
used to monitor microservices [57].

Monitoring practices concern with how to monitor microservices systems. We
identified six monitoring practices from grey literature (e.g., [6, 33, 59]),
i.e., log management, exception tracking, health check API, log deployment,
audit logging, and distributed tracking. Each of these practices consists of
several activities. For instance, log management is related to generating,
collecting, centralizing, parsing, transmitting, archiving, and disposing log
data (e.g., errors, failure) produced by applications servers, network
devices, and other components [59]. Similarly, exception tracking is a
monitoring practice with which exceptions are identified, understood, and
resolved with monitoring tools. Health check API is a service implemented
within microservices systems to get the operational status and dependencies of
microservices. It also returns the performance information (e.g., connection
time) of the microservices [48].

Monitoring tools can be classified into two categories: libraries and
platforms. Monitoring libraries are used during the development of
microservices and permit collecting the application data. In contrast,
monitoring platforms allow gathering and analyzing data from different
sources, such as the hosts, infrastructure, and microservices [33]. We
identified 13 candidate monitoring tools (see Table
LABEL:tab:MonitoringofMSA), which are listed in our survey question, from the
literature (e.g., [6, 8, 33, 59]).

###  2.3 Testing of microservices systems

Software testing is intended to identify problems and to fix them for
improving the quality of software systems [60]. However, the testing of
microservices systems requires extra effort due to, for example, the polyglot
code base in multiple repositories, feature branches, and database per service
[61]. Some other aspects that bring difficulties in microservices testing are
inter-process communication, inter-service dependencies, cloud infrastructure,
and third-party services components [9, 62]. We identified several studies
(e.g., [6, 61, 63, 64, 65, 66, 67]) in which microservices testing was
discussed. However, we did not find any study that presents the empirical
evidence on testing of microservices systems from the practitioners’
perspective. Our previous work also indicates that there is a lack of knowing
the industrial reality concerning testing for MSA-based applications [9].
Therefore, this industrial survey explores different aspects of testing
microservices systems, such as testing strategies and skills required to test
microservices systems.

Testing strategies: A testing process for microservices systems includes a set
of testing strategies [66]. These strategies contain functional (e.g., unit
testing) and non-functional testing strategies (e.g., performance testing)
[67]. The aim of using multiple testing strategies in the testing process is
to make sure a developed application successfully operates across various
environments and platforms. We identified 12 testing strategies (see Table
LABEL:tab:TestingofMSA), which are listed in our survey question, from the
literature (e.g., [9, 6, 61, 63, 64, 65, 66, 67]).

Testing tools: Microservices practitioners may use various testing tools to
detect bugs and evaluate applications’ behavior, depending on the chosen
testing strategy, for instance, tools to perform unit, performance, load, and
contract testing. We identified 20 candidate testing tools (see Table
LABEL:tab:TestingofMSA), which are listed in our survey question, from the
literature (e.g., [9, 6, 61, 63, 64, 65, 66, 67]).

##  3 Research method

Our research aims to provide a comprehensive insight into how microservices
systems are designed, monitored, and tested in practice. Given this objective
and inspired by the guidelines for selecting empirical methods for software
engineering research [68], we decided to use a mixed-methods study, which
collected data from a survey and interviews. We first conducted a survey to
reach out to a wide range of practitioners to gain a broad overview of how
microservices systems are designed, tested, and monitored. We then carried out
in-depth interviews to understand the reasons behind the key findings of the
survey. We also followed the guideline proposed by Kitchenham and Pfleeger
[69] and used the survey template recommended by evidence-based software
engineering [70].

###  3.1 Research questions

We formulated the Research Questions (RQs) in Table
LABEL:tab:ResearchQuestions based on the study aim presented in Section 1. The
findings from this set of RQs are expected to contribute to the growing body
of evidential knowledge about the state of the practice of design, monitoring,
and testing of microservices systems.

Table 1: Research questions and their rationale Research Questions |  Rationale
---|---
RQ1: How microservices systems are designed in the industry? |  The answer to this RQ helps to understand various aspects of designing microservices systems from the practitioners’ perspective. This RQ aims to investigate the team structure, application decomposition strategies, MSA implementation strategies, architecting activities, description methods, design components, QAs, MSA design patterns, skills required to design and implement MSA, design challenges and their solutions, as well as the reasons for the key findings related to the design of microservices systems.
RQ2: How microservices systems are monitored in the industry? |  Monitoring microservices systems is challenging because a large number of independent services are communicating with other independent services in many ways. This RQ aims to investigate microservices monitoring metrics, practices, tools, monitoring challenges and their solutions, as well as the reasons for the key findings related to the monitoring of microservices systems.
RQ3: How microservices systems are tested in the industry? |  Microservices systems are challenging to test because of several factors, such as polyglot code base, feature branches, database per service. This RQ aims to investigate microservices testing strategies, tools, skills, testing challenges and their solutions, as well as the reasons for the key findings related to the testing of microservices systems.

###  3.2 Survey

This section briefly discusses the design of the survey.

####  3.2.1 Survey type

We conducted this survey as a descriptive survey with a cross-sectional
design. Cross-sectional design surveys are appropriate for gathering
information at a set point of time [69]. Surveys can be administered in many
ways, including online questionnaires, phone surveys, or face-to-face
interviews [71]. We decided to conduct the survey study through an online
questionnaire because (1) this type of survey is time and cost-effective, (2)
our potential participants were from different countries, and (3) online
questionnaires are convenient for collecting the responses from a large number
of participants [71].

####  3.2.2 Sample and population

We did not limit our target population to a specific region or country (e.g.,
China). We invited the potential practitioners working in the software
industry, and whose organizations had adopted or planned to adopt MSA. We
asked several questions about the background of the participants (see Table
LABEL:tab:Demographics), such as their main responsibilities, number of years
working in the IT industry, number of years working with microservices
systems, and professional training related to MSA. We used the following
approaches to reach the target population.

  1. 1.

Personal contacts and authors of industrial track research papers and web
blogs: We sent an invitation email to our personal contacts who work on
microservices in the IT industry and the authors of the industrial track
research papers (e.g., conference proceedings) and web blogs related to
microservices. We also requested the respondents to disseminate our survey to
their colleagues who are eligible to answer the questions or share the
contacts of relevant practitioners with us.

  2. 2.

Microservices practitioners on GitHub: We executed a search string (i.e.,
“microservice” OR “micro service” OR “micro-service”) in GitHub. We sorted the
search results as “best match” to our search string and collected 9,000
publicly available email addresses of MSA-based project contributors. Then we
sent an invitation email with a survey link to all practitioners whose email
addresses were collected.

  3. 3.

Social and professional platforms: The authors are active members of several
microservices groups on LinkedIn, Facebook, and Google, where microservices
practitioners from different countries share their issues, experiences, and
knowledge. We posted our survey to all the groups in Table
LABEL:tab:platforms.

Table 2: Microservices practitioners social and professional platforms used to post our survey # |  Group Name |  URL | Platform
---|---|---|---
1 |  Microservice architecture |  https://tinyurl.com/42pazdhj | LinkedIn
2 |  Microservices meet DevOps in a Docker Container and Kubernetes world |  https://tinyurl.com/akwt44jj
3 |  Cloud, Kubernetes, Docker, microservices, Gitops discussions |  https://tinyurl.com/pfatv6s4
4 |  .Net people |  https://tinyurl.com/3an82ktz
5 |  Software architects and enterprise architects Group |  https://tinyurl.com/2r7y4cnw
6 |  Microservice architecture |  https://tinyurl.com/fh2evwwc | Facebook
7 |  Microservice Java Spring |  https://tinyurl.com/35c4xsf2
8 |  Developing microservices |  https://tinyurl.com/8nbs9p2f
9 |  Spring boot and microservices developers |  https://tinyurl.com/yuc658tv
10 |  Microservices developer |  https://tinyurl.com/3tzbmz85
11 |  Microservices community |<EMAIL_ADDRESS>| Google groups

####  3.2.3 Obtaining a valid sample

To identify valid answers from the collected responses, we applied the
inclusion and exclusion criteria presented in Table LABEL:tab:inlu_exclu. We
used the criterion I1 to include the participants’ responses and evaluated
this criterion by the answers of survey questions DQ2 (we asked the
participants their major responsibilities in the companies), DQ3 (we asked the
participants their usage of MSA for developing applications), and DQ5 (we
asked the participants their experience (in years) working with microservices
systems). On the other hand, we used the criterion E1 to exclude those
responses that were inconsistent, randomly filled, or meaningless. We also
used the criterion E2 to exclude the entire response from a respondent if more
than 5 SQs have inconsistent, randomly filled, or meaningless answers. This is
because practitioners had started our survey but he/she might not have
sufficient knowledge or intention to answer all the survey questions.
Inconsistent responses mean that the answers to two or more survey questions
are conflicting. For example, a participant chose “we use MSA style for
building all applications in our organization/team” to survey question “DQ3:
Regarding the usage of MSA for developing applications.”, and he or she also
answered “I have no idea” to survey question “SQ12: How do you decompose an
application into microservices?”. We also found several answers that were
randomly selected. For example, a respondent answered the usage of all the MSA
design patterns as “Very often”, answered all the quality attributes as “Very
important”, and rated the severity of all the challenges as “Catastrophic”.
Moreover, we considered logically senseless answers to some or all the open-
ended survey questions as meaningless. For example, a participant wrote the
answer “It is easy to get” to survey question “SQ26: What solutions (e.g.,
practices, tools, team structures) can be employed to address or mitigate the
challenges related to architecting for MSA-based systems?”.

Table 3: Inclusion and exclusion criteria for selecting valid responses Inclusion |  I1: The respondent must have responsibilities associated with MSA-based system development and operation.
---|---
Exclusion |  E1: An inconsistent, randomly filled, or meaningless response is excluded. E2: If more than 5 survey questions (SQs) have inconsistent, randomly filled, or meaningless answers, the whole response is excluded.

####  3.2.4 Sampling method

The non-probabilistic sampling method was used to obtain a valid sample for
this survey. We used the non-probabilistic sampling because the target
population was specific to microservices practitioners, which were hard to
identify through a random selection. As a type of non-probabilistic sampling,
we also used convenience and snowball sampling. Convenience sampling “refers
to acquiring responses from available people who are willing to participate”
[69]. The reason for using the convenience sampling method for this survey is
that it provides easy access to the target population through personal
contacts and social and professional platforms (e.g., GitHub, LinkedIn,
Facebook). To maximize the chances of getting more responses, we also used the
snowball sampling technique, which asks the participants in a survey to invite
other people who have relevant expertise and agree to participate in the
survey [69].

####  3.2.5 Survey instrument preparation

In this section, we describe the survey instrument and questionnaire format.
Through reviewing the grey literature (e.g., [54, 33, 66, 61]), peer-reviewed
literature (e.g., [10, 11, 21, 12, 13]), and conducting SMSs (i.e., [8, 9]) on
microservices systems, we developed a questionnaire consisting of 39 survey
questions (see Appendix B). The survey questions are organized into the
following four groups.

  * 1.

Demographic information: We used ten demographics questions (DQ1-DQ10, see
Table LABEL:tab:Demographics) to collect comprehensive demographic information
about practitioners (e.g., country, responsibilities, experience) and their
respective companies (e.g., size, work domain, development methods). We asked
for demographic information to determine whether the respondents represent
microservices practitioners for results generalization purpose.

  * 2.

Design of microservices systems: We formulated 17 survey questions with the
purpose of exploring the practitioners’ perspective on the design of
microservices systems (SQ11-SQ27, see Table LABEL:tab:DesignofMSA). The
questions are about individuals or teams who are responsible for designing the
architecture of microservices systems (SQ11), application decomposition
strategies (SQ12), MSA design implementation strategies (SQ13), MSA
architecting activities (SQ14-SQ18), MSA description methods (SQ19, SQ20), MSA
design components (SQ21), quality attributes (SQ22), and MSA design patterns
(SQ23). Moreover, we asked three questions to explore the MSA design
challenges (SQ24), the impact of MSA design challenges on monitoring and
testing of microservices systems (SQ25), and solutions for the MSA design
challenges (SQ26). Finally, the skills required to design and implement MSA
are investigated through SQ27.

  * 3.

Monitoring of microservices systems: The third part of our survey consists of
six questions that aim to explore practitioner’s perspectives on monitoring
microservices systems (SQ28-SQ33, see Table LABEL:tab:MonitoringofMSA). We
explored the monitoring metrics (SQ28), monitoring practices (SQ29),
monitoring tools (SQ30), monitoring challenges (SQ31), the severity of
monitoring challenges (SQ32), and the solutions for monitoring challenges
(SQ33).

  * 4.

Testing of microservices systems: The survey questions (SQ34-SQ39, see Table
LABEL:tab:TestingofMSA) are used to solicit the participants’ perspectives on
the testing strategies (SQ34), tools (SQ35), and challenges (SQ36), the
severity of testing challenges (SQ37), skills required to test microservices
systems (SQ38), and the solutions for testing challenges (SQ39).

####  3.2.6 Questionnaire format

The survey starts from the Welcome page to explain the purpose and design of
the survey and provides our contact information (see Appendix B). Our survey
questionnaire has both closed-ended (e.g., SQ11, SQ12, SQ13) and open-ended
questions (e.g., SQ26, SQ33, SQ39). The closed-ended questions of our survey
are composed of multiple-choice, rating scale, Likert scale, matrix, and drop-
down questions. The open-ended questions in our survey allow the participants
to provide their solutions for design, monitoring, and testing challenges. We
used Google Forms for the implementation of the survey questionnaire.

####  3.2.7 Pilot survey

We performed a pilot survey to ensure that (i) the length of the survey is
appropriate, (ii) the terms used in the survey are clear and understandable,
and (iii) the answers to survey questions are meaningful. We recruited 10
practitioners for the pilot survey through our personal contacts and sending
emails to publicly available email addresses of microservices practitioners on
GitHub. We got only a few suggestions to refine some adopted terms, and we
incorporated them in the final version of the survey instrument.

###  3.3 Interviews

To get deeper insights into the survey results, we conducted follow-up
interviews with microservices practitioners. We present the detailed procedure
of the interview process below.

####  3.3.1 Participant selection

We recruited microservices practitioners from IT companies in five countries:
Pakistan, Chile, Netherlands, Sweden, and Colombia. Interviewees were
recruited by sending an email to some of the survey’s original respondents and
personal contacts in each country. Since the interviews were expected to last
about 40 to 55 minutes in order to ensure a sufficient time allocation to each
question, we decided to focus on gaining a deep insight into the key findings
obtained from the survey. We informed the participants that this interview is
entirely voluntary with no compensation. We finally recruited 6 interviewees
and asked them to provide their demographics information (see Table
LABEL:tab:IQDMinfo), and the information about the interviewees is presented
in Table LABEL:tb:Interviewees.

Table 4: Interviewees and their information # | Responsibilities | Domain | Experience in | Country
---|---|---|---|---
Microservices | IT Industry
P1 | Design and development | Telecom | 6 years | 17 years | Pakistan
P2 | Design and development | E-commerce | 5 years | 15 years | Sweden
P3 | Design, development, and maintenance | Banking | 4 years | 4 years | Netherlands
P4 | Design and development | Telecom | 3 years | 9 years | Pakistan
P5 | Development | Telecom | 2 years | 5 Years | Colombia
P6 | Design and development | Telecom | 1 year | 20 Years | Chile

The roles of the interviewees are mainly related to the design and development
of microservices systems. However, they also actively participated in other
microservices system development stages, such as testing, monitoring, and
deployment. Their average experience in the IT industry is 11.7 years (minimum
4 years and maximum 20 years). The interviewees’ average experience on
microservices is 3.5 years (minimum 1 year and maximum 6 years). We refer to
the interviewees as P1 to P6.

####  3.3.2 Interviewing process

The first, fourth, and fifth authors of the study conducted the interviews.
The average time for all interviews is 53 minutes. The interviews were
conducted in Urdu, Italian, and Spanish, and later were translated into
English. The interviews are semi-structured and divided into four sections.

  * 1.

Demographic information: We asked five questions regarding interviewees’
demographics, such as experience in the IT industry and microservices,
responsibilities, and organization domains.

  * 2.

Design: We asked 13 open-ended questions to get a deeper understanding of
seven key findings concerning microservices system design. The questions
include: (i) confirmation of results about the combination of DDD and business
capability, (ii) reasons for using a combination of DDD and business
capability strategy, (iii) decomposition strategies other than DDD and
business capability, iv) reasons for creating the only high-level design for
microservices systems, (v) possible risks if practitioners do not create a
design or detailed design for microservices systems, (vi) reasons for the
frequent use of Architecture Evaluation (AE) activity, (vii) reasons for
paying more attention to architecting activities close to the solution space
instead of the problem space, (viii) reasons for not using ADLs and DSLs, (ix)
reasons for using informal and semi-formal languages, (x) reasons for
considering security, availability, performance, and scalability as most
important, (xi) reasons for using API gateway and Backend for frontend,
Database per service, and Access token patterns as most often, (xii) reasons
for the most prominent challenges of designing microservices systems, and
(xiii) solutions for the most prominent design challenges.

  * 3.

Monitoring: We asked 5 open-ended questions to get a deeper understanding of
four key findings from the survey regarding microservices system monitoring.
The questions include: (i) reasons for frequent use of the monitoring metrics,
like resource usage, load balancing, and availability, (ii) reasons for
frequent use of the monitoring practices, like log management, exception
tracking, and health check APIs, (iii) opinions about using Jira for
monitoring purpose, (iv) reasons for the most prominent challenges of
monitoring microservices systems, and (v) solutions for the most prominent
monitoring challenges.

  * 4.

Testing: The interviewees were asked four questions to elaborate on the key
findings obtained from the survey regarding microservices system testing. The
questions include: (i) reasons for frequent use of the testing strategies,
like unit testing, integration testing, and E2E testing, (ii) reasons for
frequent use of the testing tools, like Junit, JMeter, and Mocha to test
microservices systems, (iii) reasons for the most prominent challenges of
testing microservices systems, and (v) solutions for the most prominent
testing challenges.

###  3.4 Data analysis

####  3.4.1 Survey data analysis

We used two techniques for analyzing the obtained data from survey
participants. First, a descriptive statistic technique was used to analyze the
closed-ended questions’ responses (e.g., SQ11, SQ12, SQ13). Second, we used
open coding and constant comparison techniques from Grounded Theory (GT) [72,
73] to analyze the qualitative response to the open-ended questions (e.g.,
SQ26, SQ33, SQ39). The relationship between survey questions, Interview
Questions (IQs), data analysis methods, and RQs are shown in Table
LABEL:tab:Questions_AnalysisMethods.

####  3.4.2 Interview data analysis

The main goal of this step is to identify the reasons obtained from the
interviews for the key findings of the survey results. The interviews were
translated and transcribed by the first, fourth, and fifth authors from audio
recordings to text transcripts. Before applying the open coding and constant
comparison techniques, the second and third authors read the transcribed
interviews and highlighted the irrelevant information. We dropped several
sentences that are not related to “design, monitoring, and testing of
microservices systems”. After removing the extraneous information from the
transcribed interviews, the first author read and coded the interview
transcripts’ contents to get the answers to the interview questions. The
second and third authors of the study verified the coding results and provided
suggestions to improve the quality of coding results. We used the MAXQDA
tool222https://www.maxqda.com/ for analyzing and coding the qualitative data.

Table 5: Relationship between questions, data analysis methods, and research questions RQs/Section | Survey and interview questions | Data Analysis Method
---|---|---
Demographics | DQ1-DQ10 | Descriptive Statistics
RQ1 | SQ11-SQ25 | Descriptive Statistics
RQ2 | SQ28-SQ32 | Descriptive Statistics
RQ3 | SQ34-SQ38 | Descriptive Statistics
RQ1 | SQ27 | Open Coding and Constant Comparison
RQ2 | SQ33 | Open Coding and Constant Comparison
RQ3 | SQ39 | Open Coding and Constant Comparison
Demographics | IQ1.1-IQ1.5 | Descriptive Statistics
RQ1 | IQ2.1-IQ2.7 | Open Coding and Constant Comparison
RQ2 | IQ3.1-IQ3.4 | Open Coding and Constant Comparison
RQ3 | IQ4.1-IQ4.3 | Open Coding and Constant Comparison

####  3.4.3 Result comparison

To better understand practitioners’ perspective on design, monitoring, and
testing of microservices systems, we divided the responses about Likert scale
survey questions (i.e., SQ14-SQ18, SQ19, SQ22, SQ23, SQ29, SQ32, SQ34, SQ37)
into three major demographic groups (i.e., Experienced-based comparison,
Organization size-based comparison, MSA style-based comparison).

  * 1.

Participants who have $\leq$ 2 years of experience with microservices systems
(63 responses)

  * 2.

Participants who have > 2 years of experience with microservices systems (43
responses)

  * 3.

Participants whose organization or team does not use the MSA style at all (15
responses)

  * 4.

Participants whose organization or team uses the MSA style for building some
or all applications (91 responses)

  * 5.

Participants who are working in a relatively small and medium organization
($\leq$ 100 employees) (51 responses)

  * 6.

Participants who are working in a relatively large organization (> 100
employees) (55 responses)

We computed the statistically significant difference (i.e., P-value) for the
three groups. We summarized the survey responses to Likert scale survey
question statements, SQ#, mean values, P-values, and effect size in Table 10,
Table 11, Table 12, Table 16, Table 17, Table 21, and Table 22. The detail
about the gaps between the different demographic groups is reported in Section
4.2.6, Section 4.3.5, and Section 4.4.5. In this survey, we used five points
Likert scale for obtaining levels of Agreement (e.g., strongly agree, agree),
Frequency (e.g., very often, often), Importance (e.g., very important,
important), and Severity (e.g., catastrophic, major) from the practitioners
about several aspects of microservices systems (see Likert scale survey
questions in Table LABEL:tab:DesignofMSA, Table LABEL:tab:MonitoringofMSA, and
Table LABEL:tab:TestingofMSA). We analyzed each Likert scale level to
corresponding survey question option to identify the statistically significant
differences between the two chosen groups (e.g., experience $\leq$ 2 years vs.
experience > 2 years) at 95% confidence level. We first use the independent
sample T-test by assuming equal variances to test the statistically
significance difference between two group means (e.g., experience $\leq$ 2
years vs. experience > 2 years), and then we applied Bonferonni correction for
the multiple comparisons on the means of survey question statements (e.g., see
Table 10). We also calculated the effect size [74] to quantify the difference
between two groups. We used different colors to indicate participants from
which groups have more responses about options to survey questions. The
responses to survey questions are sorted according to the number of responses.
The dark grey color shows that the first group is more likely to agree with
survey question options, and the light grey color shows that the second group
is more likely to agree with survey question options. We use the following
symbols to show the statistically significant difference of the chosen groups.

  * 1.

symbol indicates a significant difference between participant groups who have
experience $\leq$ 2 years vs. experience > 2 years.

  * 2.

symbol indicates a significant difference between participant groups who use
the MSA style vs. No MSA style.

  * 3.

symbol indicates a significant difference between participant groups who are
from an organization that has Employees size $\leq$ 100 vs. Employees size >
100.

##  4 Results

In this section, we provide the our mixed-methods study results according to
the collected responses and answer the RQs defined in Section 3.1.

###  4.1 Demographics

Countries: We received 106 responses to the survey (indicated by R1 to R106),
interviewed 6 microservices practitioners (indicated by P1 to P6) from 6
organizations in 6 countries (see Table LABEL:tb:Interviewees). Respondents
came from 29 countries of 5 continents (see Figure 1) working in diverse teams
and roles to develop microservices systems. The majority of them are from
Pakistan (47 out of 106, 63.2%), China (13 out of 106, 12.2%), and Germany (7
out of 106, 6.6%), and a close look of the results shows that the respondents
from other countries tend to answer similarly to the respondents from the top
three countries (i.e., Pakistan, China, Germany).

Figure 1: The number of respondents from each country

Responsibilities: Figure 2 shows that the majority of participants were
application developer (69 out of 106, 65.1%), architect (45 out of 106,
42.4%), and DevOps engineer (21 out of 106, 19.8%). Note that one participant
may have multiple major responsibilities in the company (DQ2), and
consequently the sum of the percentages exceeds 100%.

$0$$5$$10$$15$$20$$25$$30$$35$$40$$45$$50$$55$$60$$65$$70$Application
developerArchitectDevOps engineerDatabase developerSystem
analystConsultantSoftware quality engineerC-level executiveDatabase
administratorBusiness analystOperations
staffOthers$69$$45$$21$$21$$17$$11$$11$$9$$6$$4$$4$$4$Num of participants
Figure 2: Survey participants’ responsibilities in their companies

MSA adoption across organization: As shown in Figure 3, 59.5% (63 out of 106)
of the respondents said that they only used MSA to develop specific
applications in their respective organizations. 26.4% (28 out of 106) of the
respondents answered that they used MSA to develop all the applications in
their organizations. Overall, almost 86% of the respondents are from
organizations that have adopted MSA to develop applications. Note that
although 14.1% (15 out of 106) of the respondents answered that their
organizations did not use MSA to develop applications, these respondents still
have experience and expertise with MSA-based systems, which is one of the
criteria to answer this survey (see the Welcome page of the survey in Appendix
B).

Figure 3: Status of MSA adoption across organization

Experience: We asked the participants about their experiences in the IT
industry (DQ4) and in the development of microservices systems (DQ5). Figure 4
(Left) shows that the majority of respondents (41 out of 106, 38.7%) have
worked in the IT industry for 3 to 5 years. On the other hand, Figure 4
(Right) shows that over half of the respondents (63 out of 106, 59.4%) had
little experience (from 0 to 2 years) in the development of microservices
systems. We also received three responses in which the participants mentioned
that they had more than 10 years of experience working on microservices.
However, considering that the term “microservices” was first coined in 2011
[75], which is around 9 years ago. We accept these results because (1)
responses to all other survey questions from these three participants were
valid, (2) the overall working experience of these participants in the IT
industry was more than ten years and they were working as “C-level
executives”, and (3) we also found the blogs related to microservices (e.g.,
[76]) in which practitioners claim microservices have a very long history, not
as short as many believe. For example, the similar term “micro web services”
was presented in 2005 [77].

Figure 4: Experience of the respondents in the IT industry (Left) and the
development of microservices systems (Right)

Professional training: Figure 5 shows that 74.5% (79 out of 106) of the
respondents did not receive any professional training to develop microservices
systems. Only 25.5% (27 out of 106) of the respondents answered that they had
such training. Note that we explicitly mentioned in DQ6 that professional
training does not mean higher education (e.g., MSA course at universities).

Figure 5: Participants professional training related to MSA

Organization size and the number of people working on microservices: The size
of organizations varies from small (1-20 employees) to large (more than 1000
employees) (see Figure 6 (Left)). We found that 29.2% (31 out of 106) of the
respondents were from medium scale organizations (21-100 employees) and 21.7%
(23 out of 106) of the respondents were from large organizations. Similarly,
Figure 6 (Right) shows the number of persons working on one microservice.
43.4% (46 out of 106) of the respondents stated that they were working in a
group of 2 to 3 people, and 27.4% (29 out of 106) answered that they were
working in a group of 4 to 5 people.

Figure 6: An overview on organization size (Left) and number of people working
on one microservice (Right)

Application domains and development methods: Figure 7 (Left) shows the domains
of the participants’ organizations where the survey participants worked.
“E-commerce” (35 out of 106, 33.1%), “Professional services” (24 out of 106,
22.6%), and “Financial software” (23 out of 106, 21.6%) are the most popular
domains. We also found that Scrum (63 out of 106, 59.4%) and DevOps (53 out of
106, 50.0%) are the two widely used development methods in the participants’
organizations to develop microservices systems (see Figure 7 (Right)). Other
popular development methods are iterative development and Rapid Application
Development (RAD), which were mentioned by 25 and 14 participants,
respectively. We also received 12 responses in which the participants
indicated that they did not employ any development method. Note that one
organization may adopt one or more development methods in their MSA projects.

$0$$5$$10$$15$$20$$25$$30$$35$E-commerceProfessional
servicesFinancialHealthcareInternetEducationInsuranceTransportationManufacturingTelecommunicationEmbedded
systemsReal estateOthers$35$$24$$23$$21$$19$$17$$11$$11$$9$$7$$6$$4$$4$Num of
participants

$0$$10$$20$$30$$40$$50$$60$$70$ScrumDevOpsIterativeRADXPWaterfallFDDSpiralLeanJADOthersNot
defined$63$$53$$16$$14$$9$$8$$7$$7$$6$$4$$3$$12$Num of participants

Figure 7: An overview on the domains of the organizations (Left) and the
software development methods employed (Right)

Demographics Finding 1. The majority of participants did not receive any
training to develop microservices systems. Finding 2. The participants
indicated that a team of 2-5 people often works on (develop, test, and deploy)
each microservice. Finding 3. Scrum and DevOps are the two most widely used
development methods in MSA projects.

###  4.2 Design of microservices systems (RQ1)

To realize how microservices systems are designed in the industry (RQ1), we
asked the respondents to answer 17 survey questions (SQ11 to SQ27) (see Table
LABEL:tab:DesignofMSA in Appendix B).

####  4.2.1 Team structure, application decomposition, and implementation
strategies

We asked the survey respondents about whether a dedicated individual or team
is responsible for creating and managing the architecture of microservices
systems (SQ11) in their organizations. 36.7% (39 out of 106) of the
respondents said that their organizations do not have a dedicated person or
team. We further investigated our results and found that most of these
respondents are from small companies or small teams that possibly cannot
afford a software architect. In contrast, 31.1% (33 out of 106) answered that
their organizations have such a team or person. Moreover, 32.1% (34 out of
106) of the respondents shared that they have a dedicated team or person
designing MSA, who has other responsibilities as well.

We were also interested in understanding how an application is decomposed into
microservices in the industry (SQ12) (see Figure 8). The number of valid
responses to this survey question is 104 because two responses are
meaningless. 42.3% (44 out of 104) of the respondents answered that they used
a combination of business capability and DDD to decompose an application into
microservices. 29.8% (31 out of 104) of the respondents decomposed an
application into microservices by applying business capability. 27.9% (29 out
of 104) of the respondents used the DDD strategy for this purpose. For
example, one respondent answered that “we decompose an application into
microservices by using the Banking Industry Architecture Network
(BIAN)333https://bian.org/ domain model, i.e., the DDD strategy” Architect
(R14).

On the other hand, all of the interviewees (IQ2.1.1 in Table
LABEL:tab:IQMsaDesign) confirmed that the results of the survey reflect the
actual use of application decomposition strategies (e.g., a combination of DDD
and business capability) for microservices systems in the industry. Five out
of six interviewees also confirmed that they are not familiar with any other
approach that can be used for application decomposition. Four interviewees
(i.e., P2, P3, P4, and P6) mentioned several reasons for using a combination
of DDD and business capability, such as (i) a single strategy cannot meet the
application decomposition goal in a good way, (ii) it supports the creative
and iterative collaboration process between system stakeholders (e.g.,
technical expert, domain expert, and customers) to achieve the decomposition
goals, (iii) it is suitable when implementing microservices in an agile or
DevOps process, (iv) it provides the controlled flexibility for future changes
as most of the changes remain around the particular domain, (v) it reduces the
misunderstanding because requirements are gathered from a specific domain
perspective, and (vi) it provides better team communication because all of the
stakeholders use a shared set of the terminologies. Following is one
representative quotation about the application decomposition strategies for
microservices systems.

“Combination of DDD and business capability, helps to build microservices with
clear responsibilities and thus increase the maintainability and evolvability
of microservices systems” Architect and Application developer (P1).

Figure 8: Strategies of application decomposition into microservices

Figure 9 shows the results of SQ13, in which the practitioners were asked to
select the strategies to design and develop microservices systems. One
response to SQ13 is invalid as the participant answered that “I have no idea”,
therefore, the number of valid responses is 105. From Figure 9, it shows that
46.7% (49 out of 105) of the participants only modeled the high-level design
(the major components) and 30.5% (32 out of 105) of the participants modeled
the detail design before writing the code. In the context of this finding, our
interview demonstrates that practitioners have stated several reasons for
using high-level design for microservices systems. For instance, four
interviewees (i.e., P2, P4, P5, and P6) mentioned that high-level design is
enough to (i) visualize the identified microservices and their major
functionalities, (ii) show communication and orchestration style between
microservices, (iii) indicate scalability points for each microservice, (vi)
identify vulnerable and insecure points in microservices systems, and vii)
present the boundaries of microservices. Another interviewee (i.e., P3)
mentioned that the reasons for only using high-level design are (i) time
pressure, which does not allow teams to spend time on low-level design, and
(ii) having an experienced development team who does not need low-level
design. Following is one representative quotation about the use of only high-
level design for microservices systems.

“We always make the high-level design for the whole system (end to end)
because it is enough to visualize the identified microservices and their
fundamental functionalities, communication or orchestration style between
microservices, and the boundaries of microservices. Architect and Application
developer (P1).

Moreover, a small percentage (6 out of 105, 5.7%) of the practitioners applied
the Model-Driven Development (MDD) approach for generating the code of
microservices systems. Surprisingly, 17.1% (18 out of 105) of the participants
reported that they wrote the code for microservices systems without having
either a high-level or detailed design.

Figure 9: Strategies from design to implementation of microservices systems

We summarize some of the prominent risks from interviewees’ answers that many
organizations may face the situation that there is no design or detailed
design for microservices systems. The potential risks are (i) teams cannot
correctly estimate the amount of work (e.g., person-hour) and time required to
complete projects, (ii) implementing poor security standards and solutions,
(iii) having no clear or shared vision for teams, (iv) flawed integration plan
for microservices, (v) understandability issues for novice developers, (vi)
system migration and evolution issues, and (vii) poor communication among the
system stakeholders. Two interviewees (i.e., P1, P2) explicitly mentioned that

“Without a design of microservices systems, a system could become a nightmare
for developers. Several kinds of risks can affect the microservices system as
a whole, including poor work and time management, no clear and common vision
about security standards, integration of microservices, and understandability
of novice developers” Architect and Application developer (P3).

Key Findings of RQ1 Finding 4. Most of the practitioners used a combination of
Domain-Driven Design (DDD) and business capabilities to break a monolith
application into microservices. Finding 5. Most of the practitioners only
created a high-level design for microservices systems before implementing
them.

####  4.2.2 MSA architecting activities and design description

Five questions (SQ14-SQ18) were asked to understand to what extent the
participants agree or disagree to use architecting activities (i.e.,
Architectural Analysis (AA), Architectural Synthesis (AS), Architectural
Evaluation (AE), Architectural Implementation (AI), Architectural Maintenance
and Evolution (AME)) in MSA projects. Figure 10 shows that 53.7% (54+3 out of
106) of the participants either “agree” or “strongly agree” with the use of AE
(i.e., evaluating candidate architecture solutions when designing
microservices systems to ensure design decisions are right). 52.8% (49+7 out
of 106) of the participants “agree” or “strongly agree” with the use of AI
(i.e., refining coarse-grained architecture design of microservices systems
into fine-grained detailed design with which developers can write code for
microservices systems).

Concerning to frequent use of AE activity, four interviewees (i.e., P1, P3,
P5) mentioned that they use AE activity to (i) evaluate the functional and
quality concerns (e.g., scalability, availability, performance, deployability,
and security), ii) assess the architectural decisions along with their
strengths and weaknesses, iii) get feedback about missing or additional
architecturally significant requirements, and (iv) evaluate the business goals
in the context of the designed architecture and current development processes.
Three interviewees (i.e., P2, P4, P6) stated that they are unfamiliar with
these architecting activities and their organizations have dedicated
architects or teams to design microservices systems who are responsible for
these activities. In the following, we provide one interviewee’s response.

“We use AE to determine whether the microservices system’s architecture is
necessary, beneficial, and useful? Whether the current architecture or design
requires continuous deployment? Whether the system requires greater
availability and fault tolerance? Whether the system requires scalability?
Whether the architecture of the system has or adds complexity to the existing
system?” Application developer (P5).

Our results also indicate that the survey respondents pay more attention to
architecting activities close to the solution space (i.e., AE, AI, AME)
instead of the problem space (i.e., AA, AS). About this finding, four out of 6
interviewees (i.e., P1, P2, P3, P5) stated that they mainly focus on the
solution space because their primary goal is to meet the customer requirements
by providing them with working software. For instance, one participant
explained his opinion in the following comment with an example.

“It is because, in the case of the solution space, scenarios are more
comfortable to evaluate. For example, if we have a Zookeeper cluster located
on two data centers. Losing two nodes within the data centers is an easy
scenario to simulate when we have different architectures in the solution
space. If we only focus on problem space and design an architecture resilient
to the loss of one data center, it is more complicated. In the solution space,
we have a set of proven reference architectures, and we use those and enrich
them” Architect and Application developer (P3).

It is interesting to note that a significant number of respondents took a
“neutral” position about several MSA architecting activities, for example, AS
(50 out of 106, 47.2%), Architectural Analysis (43 out of 106, 40.6%), and AE
(34 out of 106, 32.1%). A considerable number of participants “disagree” or
“strongly disagree” on the usage of MSA architecting activities. For example,
21.7% (19+4 out of 106) of the participants answered that they do not use AME,
and 20.8% (20+2 out of 106) of them responded that they do not use AI.

$0\%$ $10\%$ $20\%$ $30\%$ $40\%$ $50\%$ $60\%$ $70\%$ $80\%$ $90\%$ $100\%$
ASAEAAAIAME$3.8$ $2.8$ $3.8$ $1.9$ $3.8$ $8.5$ $11.3$ $10.4$ $18.9$ $17.9$
$47.2$ $32.1$ $40.6$ $26.4$ $28.3$ $39.6$ $50.9$ $38.7$ $46.2$ $34.9$ $0.9$
$2.8$ $6.6$ $6.6$ $15.1$ Strongly AgreeAgreeNeutralDisagreeStrongly disagree
Figure 10: MSA architecting activities used in industry (AA-Architectural
Analysis, AS-Architectural Synthesis, AE-Architectural Evaluation, AI-
Architectural Implementation, AME-Architectural Maintenance and Evolution

Key Findings of RQ1 Finding 6. Concerning MSA architecting activities, over
half of the participants agreed to use Architecture Evaluation and
Implementation when designing microservices systems.

We also investigated about the MSA description methods in this survey. The
results show that practitioners prefer to use informal Boxes and Lines for
representing MSA. Figure 11 shows that almost 51% (29+25 out of 106) of the
participants answered that they “very often” or “often” used Boxes and Lines
for describing MSA. UML is reported as the second most used MSA description
method, as 41.5% (31+13 out of 106) of the participants answered that they
used UML frequently (often or very often) for describing MSA. Regarding the
frequent use of informal and semi-formal description methods, we summarize the
responses of all the interviewees: (i) interactively visualize or describe
several views of the system for stakeholders, (ii) provide visual
representation and flexibility to changes, (iii) are low cost and easy to
learn, (iv) are quick to implement and maintain the architecture documents,
(v) are not limited to formal notations, (vi) effectively represent the
communication between different parts of the system, vii) are easier to
interpret, and (viii) have interactive tools support. One interview
participant stated that

“We are not feeling the need for other approaches besides boxes and lines
(informal approaches). Informal approaches are simpler to implement and
effective to describe microservices architecture” Architect and Application
developer (P3).

A vast majority of the participants (89 out of 106, 84.7%) responded that they
never (41 out of 106, 38.7%), sometimes (25 out of 106, 23.6%), or rarely (23
out of 106, 21.6%) used ADLs for describing MSA. DSLs are also not much
popular in the industry, as only 29.2% (8+23 out of 106) of the participants
reported that they used (often or very often) DSLs for this purpose. According
to interview participants, there are several reasons behind the limited use of
ADLs and DSLs, including (i) more effort is required to learn ADLs and DSLs
within a tight project delivery schedule, (ii) ADLs and DSLs cannot achieve
all the architecting goals (e.g., visualization of architectural views), (iii)
ADLs and DSLs lack of generalizability (most ADLs or DSLs are limited for a
specific purpose), (iv) ADLs and DSLs have insufficient tooling support, (v)
ADLs and DSLs cannot describe the complexity of microservices systems
effectively, and (vi) ADLs and DSLs are not delivering benefits to working
software. In the following, we provide one example comment from an interviewee
about the use of ADLS and DSLs.

“I described architecture for hundreds of systems during my more than fifteen
years of work experience from small to very large-scale companies (both
monolithic and microservices). However, we did not use ADLs and DSLs either
for monolithic or microservices systems because of their low business values
and limited visualization to architectural views for the key system
stakeholders” Architect and Application developer (P1).

Overall, we identified that the use of formal approaches (e.g., ADLs and DSL)
for describing MSA is not widespread. In contrast, practitioners prefer to use
informal (e.g., Boxes and Lines) or semi-formal (e.g., UML) approaches for
describing MSA.

$0\%$ $10\%$ $20\%$ $30\%$ $40\%$ $50\%$ $60\%$ $70\%$ $80\%$ $90\%$ $100\%$
ADLDSLUMLBoxes and Lines$38.7$ $32.1$ $12.3$ $16$ $23.6$ $18.9$ $27.4$ $21.7$
$14.2$ $21.7$ $29.2$ $23.6$ $21.7$ $19.8$ $18.9$ $11.3$ $1.9$ $7.5$ $12.3$
$27.4$ Very OftenOftenSometimesRarelyNever Figure 11: Methods used for
describing MSA in industry (UML-Unified Modeling Language, DSL-Domain Specific
Languages, ADL-Architectural Description Language)

About the diagrams used to represent MSA, Figure 12 shows the frequency of
various types of diagrams used by the respondents to represent the design and
architecture of microservices systems (SQ20). Flowchart (56 out of 106, 52.8%)
followed by Use case diagram (49 out of 106, 46.2%), Data flow diagram (46 out
106, 43.3%), and Activity diagram (46 out of 106, 43.3%) are the widely used
diagrams to represent MSA in the industry. Other popular diagrams are Sequence
diagram (44 out of 106, 41.5%), Class diagram (30 out of 106, 28.3%), and
Deployment diagram (27 out of 106, 25.4%). Interestingly, except Microservices
circuits444https://jolielang.gitbook.io/docs/architectural-
composition/aggregation, all the diagrams above have been used for
representing the scenario, process, logical, implementation, and physical view
of monolithic architecture.

$0$$5$$10$$15$$20$$25$$30$$35$$40$$45$$50$$55$FlowchartUse case
diagramActivity diagramData flow diagramSequence diagramClass diagramComponent
diagramDeployment diagramBPMNFunctional flow block diagramCommunication
diagramPackage diagramState machine diagramInteraction overview diagramObject
diagramTiming diagramComposite structure diagramMicroservice
circuits$56$$49$$46$$46$$44$$30$$27$$27$$23$$21$$13$$12$$11$$5$$5$$5$$3$$1$Num
of participants Figure 12: Diagrams for representing the design and
architecture of microservices systems

Key Findings of RQ1 Finding 7. The use of formal approaches (e.g., ADLs, DSLs)
is not widespread in the industry to describe MSA. In contrast, practitioners
prefer to use informal (e.g., Boxes and Lines) and semi-formal (e.g., UML)
approaches for describing MSA. Finding 8. Responses of the participants about
the diagrams (e.g., flowchart, data flow, activity, sequence) for representing
the design and architecture of microservices systems indicate that
practitioners prefer to model the process view of MSA.

Another important aspect that we explored through this survey is MSA design
components that can be used to describe MSA. We provided a list of potential
architectural elements (e.g., domain, service, service interface, process, and
storage) to the participants [78], and asked them to select which of these
elements could be considered as MSA design components. Figure 13 shows that
45.2% (48 out of 106) of the respondents selected services (e.g., service
gateway, service client) as MSA design components whereas 35.2% (37 out of
106) of the participants said that they considered all the architectural
elements in the list as MSA design components.

$20$$25$$30$$35$$40$$45$$50$ServicesDomainService interfaceProcessStorageAll
the above$48$$27$$26$$25$$23$$37$Num of participants Figure 13: Design
components of microservices systems

####  4.2.3 Quality attributes and design patterns

We selected 15 QAs that are popular in the MSA and SOA context [49, 53, 14].
For each QA, the participants were asked to rank their importance for
microservices systems using a Likert-scale question rated as “very important”,
“important”, “somewhat important”, “not important”, and “not sure”. Table
LABEL:tab:MSAQAs shows that 55.7% (59 out of 106) of the respondents ranked
Security as “very important”, followed by Availability (58 out of 106, 54.7%),
Performance (56 out of 106, 52.8%), and Scalability (51 out of 106, 48.1%).
About these results, we asked the interview participants with IQ2.5.1 “what
could be the reasons that practitioners consider security, availability,
performance, and scalability the most important quality attributes for
microservices systems”. All the interviewees mentioned that they consider
Security an important QA because (i) microservices systems are distributed in
nature, ii) microservices systems are complex and have security
vulnerabilities that an intruder can exploit. Two of the interviewees (P1, P4)
also stated that “Security is equally crucial for microservices or any other
type of distributed applications”. Following is one representative quotation
about the importance of Security in the context of microservices systems.

“We have to keep up with the various security patches. However, in my
understanding, providing security to microservices systems is relatively easy
than monolithic applications because normally, we need to secure API gateway
from intruders. Also, the isolated nature of microservices helps to limit the
access of intruders to only vulnerable services” Architect and Application
developer (P1).

About availability, three interviewees (i.e., P1, P2, P3) stated that end-
users usually do not compromise on the non-availability of the systems. One
interviewee (i.e., P1) mentioned that end-users could compromise on the system
performance to some extent in the peak time of system usage. However, it is
not acceptable for the end-users that the system goes down for some time.
Moreover, DDD plays a critical role in the availability of microservices
systems because applications are decomposed into microservices according to
subdomains of the system. If any problem occurs, it remains limited to
specific microservice of subdomain and does not propagate in the whole system.
We also investigated the reasons for considering Performance as an important
QA during the design of microservices systems, and four interviewees (i.e.,
P1, P2, P3, P5) mentioned that (i) performance should not be degraded due to
frequent communication calls and (ii) effective load balancing techniques can
be introduced (e.g., creating multiple instances of one microservice and using
a load balancer to improve the performance). Regarding Scalability, three
interviewees (i.e., P2, P4, P6) stated that it is because scalability is one
of the key features of microservices systems that provides a simplified way to
scale a particular microservice instance by having a minimal effect on the
availability of the system to end-users. Following is one representative
quotation about the importance of Availability in the context of microservices
systems.

“To achieve reliability for microservices systems, practitioners want to make
their systems available 24/7. It is also essential when the system has several
competitors or millions of users” Architect and Application developer (P5).

On the other hand, only 26.4% (28 out of 106) and 23.5% (25 out of 106) of the
respondents ranked Resilience and Portability as “very important” QAs,
respectively. We also found several QAs that were ranked as “somewhat
important” by the participants. Such QAs are Reusability (25 out of 106,
23.6%), Compatibility (24 out of 106, 22.6%), and Monitorability (23 out of
106, 21.7%). Overall, the results show that QAs are an essential architectural
concern to design microservices systems.

Table 6: Importance of QAs (in %) for designing microservices systems (VI-Very Important, I-Important, SI-Somewhat Important, NI-Not Important, NS-Not Sure) ID | Quality attributes | VI | I | SI | NI | NS
---|---|---|---|---|---|---
QA1 | Security | 55.7 | 27.4 | 9.4 | 3.8 | 3.8
QA2 | Availability | 54.7 | 29.2 | 14.2 | 1.9 | 0.0
QA3 | Performance | 52.8 | 33.0 | 9.4 | 2.8 | 1.9
QA4 | Scalability | 48.1 | 38.7 | 6.6 | 3.8 | 2.8
QA5 | Reliability | 40.6 | 38.7 | 12.3 | 2.8 | 5.7
QA6 | Maintainability | 35.8 | 42.5 | 20.8 | 0.0 | 0.9
QA7 | Usability | 35.8 | 40.6 | 14.2 | 2.8 | 6.6
QA8 | Compatibility | 34.0 | 34.0 | 22.6 | 1.9 | 7.5
QA9 | Testability | 33.0 | 41.5 | 19.8 | 0.9 | 4.7
QA10 | Monitorability | 32.1 | 38.7 | 21.7 | 1.9 | 5.7
QA11 | Functional suitability | 30.2 | 42.5 | 15.1 | 4.7 | 7.5
QA12 | Reusability | 30.2 | 35.8 | 23.6 | 5.7 | 4.7
QA13 | Resilience | 26.4 | 30.2 | 22.6 | 5.7 | 15.1
QA14 | Portability | 23.6 | 35.8 | 26.4 | 7.5 | 6.6
QA15 | Interoperability | 21.7 | 42.5 | 17.0 | 5.7 | 13.2

Key Findings of RQ1 Finding 9. Over 80% of the participants indicated that
security, availability, performance, and scalability are very important QAs
for the design of microservices systems.

The participants were also asked to indicate how often they use patterns when
designing microservices systems. The majority of MSA design patterns are
decently (i.e., very often, often, sometimes) used in practice, and only some
of them are not used or used less often (e.g., rarely, never) to design
microservices systems. The results show that the following four MSA design
patterns are used most often: API gateway (26.4% very often, 42.5% often,
11.3% sometimes), Backend for frontend (17.9% very often, 35.8% often, 15.1%
sometimes), Access token (17.9% very often, 34.9% often, 16.0% sometimes), and
Database per service (12.3% very often, 36.8% often, 21.7% sometimes). These
four patterns are used to manage external APIs, application security, and
databases for microservices systems. Similarly, we identified some other MSA
design patterns that are frequently used for data management, deployment, and
communication of microservices. The examples of such patterns are API
composition (14.2% very often, 33.0% often, 35.8% sometimes), Service instance
per container (16.6% very often, 30.3% often, 24.5% sometimes), and Messaging
(14.2% very often, 32.1% often, 24.5% sometimes). Overall, the combined usage
(e.g., very often, often, Sometimes) reported by the participants for each MSA
design pattern is more than 50.0%.

We also asked IQ2.6.1 to the interviewees about the leading MSA design
patterns used in industry. Four out of six interviewees (i.e., P1, P2, P4, P5)
provided the following reasons for the frequent use of API gateway and Backend
for frontend patterns: (i) API gateway and Backend for frontend are the key
patterns for request routing, API composition, and protocol translation for
microservices systems, (ii) these patterns have naturally bound with
microservices architecture, (iii) API gateway offers a single entry-point for
a specific group of microservices, (iv) Backend for frontend is a variant of
the API gateway pattern that can be used to separate the frontend from the
backend of the applications, and (v) these two patterns enhance the loose
coupling property of MSA. About the frequent use of security pattern Access
token, four interviewees (P1, P2, P4, P6) mentioned several reasons, such as
(i) token-based authentication allowing an application to access an API, (ii)
one-time login, iii) no session is persevered on the server side (stateless),
iv) servers do not need to track and keep the information of users, (v) the
security token is short-lived and can be revoked easily, (vi) the security
token helps to save the server memory consumption, and (vii) if the service
goes down for some reason, the security token is still valid because it is
also used on the client side. Regarding the frequent use of Database per
service pattern, six interviewees (P1, P2, P3, P4, P5, P6) provided four
reasons that the Database per service pattern (i) helps to design and develop
the loosely coupled applications, (ii) provides support for selecting
databases according to the requirement of each microservice, (iii) helps to
control the changes or failure by limiting them to a specific microservice or
database, and (iv) provides freedom of using specific databases according to
the team’s skills. In the following, we provide one representative quotation
about the use of API gateway pattern for microservices systems.

“In my understanding, the reasons behind using these patterns are rooted in
microservices systems. For instance, instead of the API gateway pattern, we
did not use any other communication pattern, and it is almost considered a
ditto standard for communication in the industry” Application developer (P4).

On the other hand, some MSA design patterns, such as Sagas (14.2% rarely,
34.0% never), Event sourcing (21.7% rarely, 23.6% never), and Self-
registration (20.8% rarely, 22.6% never), are rarely or never used. More
details about the use of MSA design patterns can be found in Table
LABEL:tab:MSApatterns.

Table 7: MSA design patterns (in %) used in the industry (VO-Very Often, O-Often, S-Sometimes, R-Rarely, N-Never) ID | MSA patterns | VO | O | S | R | N
---|---|---|---|---|---|---
MSP1 | API gateway | 26.4 | 42.5 | 11.3 | 10.4 | 9.4
MSP2 | Backend for frontend | 17.9 | 35.8 | 15.1 | 17.9 | 13.2
MSP3 | Access token | 17.9 | 34.9 | 16.0 | 17.0 | 14.2
MSP4 | Service instance per container | 16.0 | 30.2 | 24.5 | 15.1 | 14.2
MSP5 | Service registry | 16.0 | 29.2 | 23.6 | 12.3 | 18.9
MSP6 | API composition | 14.2 | 33.0 | 35.8 | 10.4 | 6.6
MSP7 | Messaging | 14.2 | 32.1 | 24.5 | 17.9 | 11.3
MSP8 | Remote procedure invocation | 13.2 | 32.1 | 17.0 | 17.9 | 19.8
MSP9 | Database per service | 12.3 | 36.8 | 21.7 | 17.9 | 11.3
MSP10 | Circuit breaker | 11.3 | 30.2 | 18.9 | 19.8 | 19.8
MSP11 | Client-side discovery | 10.4 | 34.9 | 19.8 | 19.8 | 15.1
MSP12 | Server-side discovery | 10.4 | 24.5 | 28.3 | 17.9 |  18.9
MSP13 | Serverless deployment | 9.4 | 25.5 | 18.9 | 17.9 | 28.3
MSP14 | Service instance per host | 8.5 | 29.2 | 24.5 | 18.9 | 18.9
MSP15 | Self-registration | 8.5 | 25.5 | 22.6 | 20.8 | 22.6
MSP16 | Polling publisher | 8.5 | 17.9 | 29.2 | 16.0 | 28.3
MSP17 | Service instance per VM | 7.5 | 30.2 | 18.9 | 17.0 | 26.4
MSP18 | Application events | 7.5 | 24.5 | 33.0 | 18.9 | 16.0
MSP19 | Sagas | 7.5 | 24.5 | 19.8 | 14.2 | 34.0
MSP20 | Service deployment platform | 6.6 | 32.1 | 23.6 | 17.0 | 20.8
MSP21 | Multiple service instance per host | 6.6 | 31.1 | 22.6 | 24.5 | 15.1
MSP22 | Shared database | 6.6 | 31.1 | 27.4 | 19.8 | 15.1
MSP23 | Event sourcing | 5.7 | 20.8 | 28.3 | 21.7 | 23.6
MSP24 | CQRS | 5.7 | 17.0 | 32.1 | 20.8 | 24.5
MSP25 | Transaction log tailing | 4.7 | 30.2 | 19.8 | 24.5 | 20.8
MSP26 | Domain-specific protocol | 3.8 | 33.0 | 18.9 | 23.6 | 20.8

The participants also mentioned several other patterns via the “Other” field
in SQ23. Such patterns are Model-view-controller (5 responses), Two-phase
commit555https://tinyurl.com/yyr3sl7q (3 responses), Facade (1 response),
Singleton (1 response), Repository (1 response), API
key666https://tinyurl.com/y2sm2jqz (1 response), Rate
limit777https://tinyurl.com/yxroch9s (1 response), Service
decorator888https://tinyurl.com/y5rphfsx (1 response), Session memory (1
response), and Tolerant reader (1 response).

Key Findings of RQ1 Finding 10. API gateway and Backend for frontend as
communication patterns, Database per service as data management pattern, and
Access token as security pattern are reported as the most often used MSA
design patterns in practice.

####  4.2.4 Skills

We asked the participants about the skills (including hard and soft skills)
required to design and implement MSA (SQ27). We identified these skills from
the literature, e.g., [79, 80, 81]. 72.6% (77 out of 106) of the participants
believed that practitioners should have the skills to break down a complex
coding task into smaller tasks, and 56.6% (60 out of 106) of the participants
mentioned that practitioners should have skills to implement the functionality
of a microservice by using suitable MSA design patterns. On the other hand,
only 30.1% (32 out of 106) of the participants answered that practitioners
should know container technologies (see Figure 14 for more details).

$30$$35$$40$$45$$50$$55$$60$$65$$70$$75$$80$Breaking down a complex coding
taskSkills to implement MSA design patternsArchitectural knowledgeKnowledge of
DevOpsKnowledge of securing microservicesWorking in a dynamic team
structureKnowledge of container technologies$77$$60$$57$$44$$42$$35$$32$
Figure 14: Skills required to design and implement MSA

####  4.2.5 Challenges and solutions

Challenges: The participants were asked about architectural challenges that
they may encounter during the design of microservices systems (SQ25). We
provided a list of challenges identified in a recent literature review [8].
The results show that DC1: clearly defining the boundaries of microservices is
the most common challenge in MSA design (74 out of 106, 69.8%). The other two
top-ranked challenges in MSA design are DC2: addressing security concerns (49
out of 106, 46.2%) and DC3: managing microservices complexity at the design
level (37 out of 106, 34.9%).

Concerning these design challenges, we asked IQ2.7.1 to the interviewees about
their reasons. All the interviewees agreed that these are valid challenges.
Regarding the reasons for the challenge DC1, four out of six interviewees
(i.e., P1, P2, P5, P6) mentioned that DC1 occurs due to (i) failure in
deciding about the responsibilities or functionalities of each microservice,
(ii) failure to locate the overlapping and shared functionalities between
microservices, (iii) lack of business knowledge when creating a new
application, and (iv) no iterative approaches (e.g., sprint meetings) being
employed to refine the design of microservices systems. About the reasons for
the challenge DC2, three out of six interviewees (i.e., P1, P3, P4) provided
the following reasons: (i) insecure communication between microservices, (ii)
vulnerable security standards and tools, (iii) insecure cloud infrastructure
for deploying and scaling microservices, and (iv) poor design and introducing
insecure code for microservices systems. About the reasons for the challenge
DC3, four out of six interviewees (i.e., P1, P2, P3, P6) provided the
following reasons: (i) hard to identify end-to-end microservices and decide
about their communication patterns, (ii) defining microservices for every
system feature or activity (too fine-grained), (iii) poorly defining the
boundaries of microservices (i.e., DC1), and (iv) too many interactions and
dependencies between microservices. Following is one representative quotation
about the challenge DC3 during the design of microservices systems.

“We don’t have any silver bullet to reduce the complexity. We need to balance
the complexity of the system while defining the boundaries of microservices.
Complexity cannot be mitigated entirely, but we need to bring it to the level
where it does not severely affect the system’s quality (performance,
availability, security)” Architect and Application developer (P3).

On the other hand, for each design challenge, the participants were further
asked to indicate if these challenges impact monitoring and testing of
microservices systems. The fourth and fifth columns of Table 8 show how many
respondents believed that certain challenge has an impact on monitoring and
testing of microservices systems. More than half of the respondents felt that
the challenge of DC1: clearly defining the boundaries of microservices impacts
both monitoring and testing of microservices systems. Similarly, 61.9% (63 out
of 106) and 59.4% (58 out of 106) of the respondents agreed that DC2:
Addressing security concerns can affect testing and monitoring of
microservices systems, respectively. More details can be seen in Table 8.

Table 8: Challenges faced during the design of microservices systems and their
impact on monitoring and testing of microservices systems (in %)

|  |  | Impact of Challenges on
---|---|---|---
ID | MSA Design Challenge | Response | Monitoring | Testing
DC1 | Clearly defining the boundaries of microservices | 69.8 | 59.4 | 54.7
DC2 | Addressing security concerns | 46.2 | 59.4 | 61.3
DC3 | Managing microservices complexity at the design level | 34.9 | 57.5 | 52.8
DC4 | Recovering MSA from existing code | 34.9 | 57.5 | 50.9
DC5 | Reaching scalability | 26.4 | 60.4 | 51.9
DC6 | Finding appropriate modelling abstractions for microservices | 23.6 | 49.1 | 48.1
DC7 | Separating functional and operational concerns | 23.6 | 67.0 | 49.1
DC8 | Addressing microservices communication at the design level | 22.6 | 62.3 | 50.0
DC9 | Addressing data management at the design level | 21.7 | 60.4 | 50.9
DC10 | Aligning team structure with the architecture of MSA-based system | 19.8 | 50.9 | 60.4

Solutions for general challenges – from survey results: Through an open-ended
survey question (SQ26), we asked the participants what solutions they used to
address or mitigate the challenges in MSA design. We received 36 answers, and
after evaluating each answer, only 23 responses were found as valid. We
excluded those solutions that were incomplete, inconsistent, or do not
adequately address the MSA design challenges. We classified the valid answers
into two categories (see Table LABEL:tab:DesignSol) as presented below:

  * 1.

Strategy, patterns, and DSLs: Most of the participants recommended the DDD
strategy (10 responses) to address the challenge of DC1: clearly defining the
boundaries of microservices. This strategy can help in determining the
boundaries of microservices through concepts like bounded context [25]. To
give an example, one participant explicitly mentioned that we built a micro
services-oriented platform (i.e.,
Dolittle999https://dolittle.io/contributing/guidelines/domain_driven_design/)
that helps in the identification of MSA boundaries by using the DDD strategy
C-level executive (R5). However, one respondent questioned the usefulness of
DDD in determining the boundaries of microservices and said that DDD is not
always the right option for identification of MSA boundaries as it does not
always address problems of scalability and architectural issues for
microservices systems. For instance, some boundaries that defined through DDD
may not fit with the business model or may not even help to achieve the
scalability Application developer (R45). Vertical decomposition pattern was
deemed to achieve appropriate granularity, scalability, and reliability in
microservices systems [82, 83]. One participant stated that we use vertical
decomposition pattern for identification of microservices boundaries System
analyst (R33). One other participant mentioned that Circuit breaker pattern
was considered useful to control the flow of traffic and APIs calls between
the services, as well as to protect the system from both transient failures
and buggy code Architect (R19). We also received a few answers in which the
participants recommended patterns and DSL as solutions to mitigate the
challenges related to the design of MSA, such as microservice API
patterns101010https://microservice-api-patterns.org, structural design
pattern111111https://csrc.nist.gov/publications/detail/sp/800-204/draft (e.g.,
re-deployable Jolie Circuit breaker), and DSL Jolie121212https://jolie-
lang.org.

  * 2.

Adoption of software development methods: We received one response in which a
participant suggested that developers who have good knowledge of microservices
should be involved during the architecting process of the MSA in order to
effectively deal with challenges that may arise on design level Consultant
(R102). Furthermore, some participants also proposed to use DevOps (3
responses), Scrum (2 responses), Feature-driven development (1 response), and
Team sprints (1 response) to address the challenges (DC1 and DC10 in Table 8)
related to the design of microservices systems. However, none of the
participants provided detailed descriptions about how software development
methods (e.g., DevOps) can address the challenges of MSA design (e.g.,
security concerns).

Table 9: Proposed solutions to address the MSA design challenges Type |  Proposed Solution |  | Challenge ID
---
Addressed
Count
Strategy, patterns, and DSL |  DDD | DC1 | 15
Vertical decomposition pattern | DC1, DC5
Microservice API patterns (e.g., public API, community API ) | DC2, DC5
Structural design pattern | DC3
Circuit breaker, Jolie (DSL) | DC8
Software development methods |  DevOps, Scrum, FDD, and Team sprints | DC1, D10 | 8

Solutions for top three challenges - from interview results: We summarized the
solutions from the interview results for the leading three challenges, i.e.,
DC1: clearly defining the boundaries of microservices, DC2: addressing
security concerns, and DC3: managing microservices complexity at the design
level.

  * 1.

Solutions for clearly defining the boundaries of microservices: The
interviewees proposed several measures to address this challenge: (i) gain a
deeper understanding of business requirements, application domains,
technologies, business processes, and development processes, (ii) introduce
and implement the concept of miniservices [84] for defining the boundaries of
microservices, (iii) identify possible contexts or subdomains of the
application by using certain strategies (e.g., DDD), (iv) discover the common
and isolated functionality of each subdomain, (v) find the suitable
granularity level for microservices, and (vi) ensure the visibility of defined
boundaries of microservices.

  * 2.

Solutions for addressing security concerns: Microservices systems can be
secured on the network, load balancing, firewall, and application level by
simply buying the updated security solutions from cloud providers. For
example, AWS provides their own way of securing microservices (e.g., API
facade pattern), Kubernetes offers Role-Based Access Control component to
limit the access of specific microservice to authorized users. It is suggested
that end-users constantly update their firewalls and other security standards
to address several security challenges. One of the practitioners (i.e., P2)
also recommended employing DevSecOps131313https://www.devsecops.org/ and
microservices together as a solution to address security challenges, because
both technologies have a natural alliance to protect the applications.

  * 3.

Solutions for managing microservices complexity at the design level: There is
no silver bullet for addressing the complexity of microservices. However, it
can be reduced by taking these measures: (i) do not design or develop
microservices for every system’s feature, (ii) define a manageable set of
responsibilities for each microservice, (iii) understand the operating
environment of microservices (e.g., deployment and monitoring infrastructure),
and (iv) clearly define the boundaries of microservices.

Key Findings of RQ1 Finding 11. Clearly defining the boundaries of
microservices and addressing security concerns are the most reported
challenges during the design of microservices systems. Moreover, it has been
identified that MSA design challenges have a significant impact on monitoring
and testing of microservices systems. Finding 12. Practitioners indicated that
they could better understand and define the boundaries of microservices by
using Domain-Driven Design (DDD) strategy.

####  4.2.6 Differences between the subpopulations of the study for MSA design

We analyzed the answers to Likert scale survey questions in Table 10, Table
11, and Table 12. To better understand practitioners’ perspectives on the
design of microservices systems, we divided the survey respondents into three
groups (i.e., Experience $\leq$ 2 years vs. Experience > 2 years, MSA style
vs. No MSA style, Employees size $\leq$ 100 vs. Employees size > 100). We
calculated the P-value to indicate statistically significant difference for
each survey question statement and the Effect size to indicate the magnitude
of difference between two groups. The dark grey color in table cells shows
that the first group (e.g., Experience $\leq$ 2 years) is more likely to agree
with the survey question statements. In contrast, the light grey color shows
that the second group (e.g., Experience > 2 years) is more likely to agree
with the survey question statements. In the following, we report the
statistical comparison for MSA architecting activities and description
approaches, quality attributes, and MSA design patterns.

MSA architecting activities and description approaches (see Table 10):
Experience $\leq$ 2 years vs. Experience > 2 years: There are two survey
question statements (i.e., Architectural maintenance and evolution and Domain
specific languages) with statistically significant differences between
Experience $\leq$ 2 years and Experience > 2 years groups. The Experience > 2
years group is more likely to agree on using Architectural maintenance and
evolution activity in MSA projects than the Experience $\leq$ 2 years group.
On the other hand, the Experience $\leq$ 2 years group is more likely to use
Domain Specific Languages for describing MSA than the Experience > 2 years
group. MSA style vs. No MSA style: There are three survey question statements
(i.e., Architectural maintenance and evolution, Unified modeling language,
Domain specific languages) with statistically significant differences between
MSA style and No MSA style groups. The MSA style group is more likely to agree
on using Architectural maintenance and evolution activity in MSA projects than
the No MSA style group. Moreover, the MSA style group is also more likely to
use Unified modeling language and Domain specific languages for describing the
architecture of microservices systems than the MSA style group. Employees size
$\leq$ 100 vs. Employees size > 100: We did not find any survey question
statement with statistically significant difference between Employees size
$\leq$ 100 and Employees size > 100 groups.

Table 10: Statistically significant differences on the survey results about
MSA architecting activities and description approaches

|  | Likert Distro. | Exper. $\leq$ 2 vs. > 2 years | MSA style vs. No MSA style | Empl. $\leq$ 100 vs. > 100
---|---|---|---|---|---
Survey Question Statement | SQ# | Mean value | P-value | Effect size | P-value | Effect size | P-value | Effect size
MSA architecting activities (Strongly agree (5), Agree (4), Neutral (3),
Disagree (2), Strongly disagree (1))
Architectural analysis | SQ14 | 3.36 | 0.55 | 0.29 | 0.12 | -0.19 | 0.93 | -0.13
Architectural synthesis | SQ15 | 3.26 | 0.63 | 0.31 | 0.23 | -0.03 | 0.96 | 0.01
Architectural evaluation | SQ16 | 3.39 | 0.59 | -0.05 | 0.23 | 0.13 | 0.96 | -0.23
Architectural implementation | SQ17 | 3.38 | 0.52 | -0.16 | 0.13 | -0.32 | 0.95 | 0.09
Architectural maintenance and evolution | SQ18 | 3.40 |  0.03 | -0.11 |  0.04 | 0.21 | 0.89 | -0.17
Approaches used for describing the architecture of microservices systems (Very
often (5), Often(4), Sometimes (3), Rarely (2), Never (1))
Boxes and lines | SQ19 | 3.35 | 0.33 | -1.05 | 0.12 | 1.10 | 0.91 | -0.91
Unified modeling language | 3.12 | 0.15 | 0.13 |  0.01 | 0.42 | 0.90 | 0.27
Domain specific languages | 2.57 |  0.04 | 0.70 |  0.00 | 0.09 | 0.91 | 0.50
Architectural description languages | 2.22 | 0.46 | 0.58 | 0.06 | -0.59 | 0.94 | 0.45

Quality attributes: Experience $\leq$ 2 years vs. Experience > 2 years (see
Table 11): There are three survey question statements (i.e.,  Usability,
Resilience, Interoperability) with statistically significant differences
between Experience $\leq$ 2 years and Experience > 2 years groups. The
Experience > 2 years group is more likely to agree on importance of Usability
and Resilience for designing MSA than the Experience $\leq$ 2 years group. On
the other hand, the Experience $\leq$ 2 years group is more likely to agree on
importance of Interoperability for designing MSA than the Experience > 2 years
group. MSA style vs. No MSA style: There are seven survey question statements
(i.e., Security, Availability, Scalability, Reliability, Maintainability,
Compatibility, Resilience) with statistically significant differences between
MSA style and No MSA style groups. The MSA style group is more likely to agree
on importance of Security, Availability, Scalability, Reliability,
Maintainability, and Compatibility for designing MSA than the No MSA style
group. On the other hand, the No MSA style group is more likely to agree on
importance of Resilience for designing MSA than the MSA style group. Employees
size $\leq$ 100 vs. Employees size > 100: We identified two survey question
statements (i.e., Functional suitability, Resilience) with statistically
significant differences between Employees size $\leq$ 100 and Employees size >
100 groups. The Employees size $\leq$ 100 group is more likely to agree on the
importance of the Functional suitability and Resilience quality attributes for
designing MSA than the Employees size > 100 group. On the other hand, we did
not find any survey question statement about the importance of quality
attributes with statistically significant difference for the Employees size >
100 group.

Table 11: Statistically significant differences on the survey results about
quality attributes in MSA design

Likert Distro. Exper. $\leq$ 2 vs. > 2 years MSA style vs. No MSA style Empl.
$\leq$ 100 vs. > 100 Survey Question Statement SQ# Mean value P-value Effect
size P-value Effect size P-value Effect size Quality attribute when designing
MSA-based systems (Very important (5), Important(4), Somewhat important (3),
Important (2), Not sure (1)) Security SQ22 3.87 0.50 -0.17 0.14 0.10 0.80 0.00
Availability 3.87 0.14 -0.40 0.15 0.50 0.87 -0.12 Performance 3.80 0.41 -0.36
0.14 0.59 0.54 -0.23 Scalability 3.69 0.38 -0.21 0.09 0.25 0.93 -0.07
Reliability 3.52 0.29 -0.45 0.05 0.50 0.92 -0.36 Maintainability 3.51 0.87
-0.03 0.03 0.18 0.78 0.00 Usability 3.43 0.03 -0.05 0.15 0.09 0.78 -0.01
Compatibility 3.42 0.75 0.36 0.01 0.36 0.88 -0.05 Testability 3.42 0.90 -0.20
0.05 0.53 0.97 0.06 Monitorability 3.40 0.32 -0.25 0.06 0.02 0.62 -0.11
Functional suitability 3.36 0.34 0.15 0.12 -0.05 0.04 0.14 Reusability 3.41
0.19 -0.09 0.07 0.36 0.72 -0.07 Resilience 3.20 0.04 -0.04 0.00 -0.07 0.03
0.17 Portability 3.28 0.18 0.14 0.09 0.47 0.56 0.16 Interoperability 3.13 0.03
0.35 0.05 0.32 0.59 0.04

MSA design patterns: Experience $\leq$ 2 years vs. Experience > 2 years (see
Table 12): There are four survey question statements (i.e., Service registry,
Serverless deployment, Polling publisher, Shared database) with statistically
significant differences between Experience $\leq$ 2 years and Experience > 2
years groups. The Experience > 2 years group is more likely to use Service
registry, Serverless deployment, Polling publisher, and Shared database
patterns for designing MSA than the Experience $\leq$ 2 years group. On the
other hand, we did not find any survey question statement about the use of MSA
design patterns with statistically significant differences for Experience
$\leq$ 2 years group. MSA style vs. No MSA style: There are sixteen survey
question statements (e.g., Backend for frontend, Access token, Service
registry) with statistically significant differences between MSA style and No
MSA style groups. We identified that the No MSA style group is more likely to
use patterns (e.g., Backend for frontend, Access token, Service registry) for
designing MSA than the MSA style group. On the other hand, we did not find any
survey question statement about using MSA design patterns with statistically
significant differences for the MSA style group. Employees size $\leq$ 100 vs.
Employees size > 100: We identified three survey question statements (i.e.,
Remote procedure invocation, Polling publisher, Multiple service instance per
host) with statistically significant differences between Employees size $\leq$
100 and Employees size > 100 groups. The Employees size > 100 group is more
likely to use Polling publisher and Multiple service instance per host
patterns for designing MSA than the Employees size $\leq$ 100 group. Moreover,
the Employees size $\leq$ 100 group is more likely to use Remote procedure
invocation pattern for designing MSA than the Employees size > 100.

Table 12: Statistically significant differences on the survey results about
MSA design patterns

Likert Distro. Exper. $\leq$ 2 vs. > 2 years MSA style vs. No MSA style Empl.
$\leq$ 100 vs. > 100 Survey Question Statement SQ# Mean value P-value Effect
size P-value Effect size P-value Effect size MSA design patterns when
designing MSA-based systems (Very important (5), Important (4), Somewhat
important (3), Important (2), Not sure (1)) API gateway SQ23 3.67 0.48 -0.37
0.07 0.26 0.94 -0.53 Backend for frontend 3.29 0.23 0.02 0.00 -0.54 0.90 0.05
Access token 3.26 0.31 0.27 0.00 -0.22 0.90 -0.41 Service instance per VM 2.75
0.21 0.03 0.06 -0.60 0.95 -0.12 Service registry 3.13 0.01 -0.15 0.00 -0.37
0.95 -0.60 API composition 3.42 0.45 -0.11 0.06 -0.03 0.93 -0.37 Messaging
3.21 0.32 -0.20 0.06 -0.19 0.90 -0.07 Remote procedure invocation 3.03 0.19
0.12 0.00 -0.31 0.02 0.06 Database per service 3.26 0.40 0.06 0.01 -0.09 0.96
-0.13 Circuit breaker 2.97 0.17 -0.60 0.06 -0.40 0.95 -0.82 Client-side
discovery 3.07 0.29 0.12 0.01 -0.12 0.91 -0.21 Server-side discovery 2.92 0.21
-0.42 0.00 -0.21 0.94 -0.20 Serverless deployment 2.70 0.01 -0.16 0.00 -0.52
0.93 0.21 Service instance per host 2.94 0.24 0.01 0.05 0.09 0.89 -0.20 Self-
registration 2.79 0.14 0.28 0.00 -0.53 0.94 0.02 Polling publisher 2.64 0.02
-0.29 0.24 -0.68 0.02 -0.38 Application events 2.92 0.30 0.10 0.01 -0.13 0.91
-0.21 Sagas 2.60 0.29 0.07 0.22 -0.81 0.96 -0.22 Service deployment platform
2.88 0.40 0.22 0.01 -0.15 0.95 0.06 Multiple service instance per host 2.90
0.22 -0.06 0.32 -0.17 0.03 -0.43 Shared database 2.93 0.03 -0.24 0.01 -0.13
0.95 0.12 Event sourcing 2.65 0.26 0.18 0.01 -0.32 0.89 0.12 Command query
responsibility segregation 2.60 0.25 -0.02 0.20 -0.48 0.92 -0.38 Transaction
log tailing 2.75 0.26 0.28 0.01 -0.55 0.95 -0.16 Service instance per
container 3.22 0.20 -0.36 0.00 -0.03 0.95 -0.15 Domain-specific protocol 2.80
0.33 0.10 0.01 -0.49 0.92 -0.45

###  4.3 Monitoring of microservices systems (RQ2)

####  4.3.1 Monitoring metrics

We asked about thirteen monitoring metrics (SQ28) identified from the grey
literature, e.g., [57, 33, 58]. According to participants’ responses, the most
popular monitoring metric is resource usage (e.g., CPU and memory usage) (71
out of 106, 66.9%), followed by load balancing (57 out of 106, 53.7%) and
availability (56 out of 106, 52.8%).

To understand the reasons behind using monitoring metrics, we asked IQ3.1.1
(see Table LABEL:tab:IQmonitoring). All the interviewees were familiar with
the monitoring metrics. The interviewees have acknowledged that they mainly
use resource usage, load balancing, and availability as the monitoring
metrics. They argued that monitoring metrics are used to (i) provide a better
customer experience (e.g., avoiding failure and improving performance), (ii)
estimate the budget (e.g., payment according to the use of cloud
infrastructure, such as Microsoft Azure, Amazon AWS), (iii) assess application
behavior on high traffic time (e.g., Christmas), (iv) monitor the overall
health of host infrastructure (e.g., memory/CPU utilization), (v) count the
number of hosts and pods, (vi) detect problems with multi-threading, (vii)
identify degraded states or failure points before the system entirely fails,
(viii) improve the performance of microservices, and (ix) monitor the
availability state (percentage) of a system for end-users. Following is one
representative quotation about the monitoring metrics for microservices
systems.

“The use of monitoring metrics allows us to get first-hand information about
each microservice’s behaviour. Monitoring metrics can help to understand,
visualize, and isolate failures points. We can also use monitoring metrics to
check the efficiency of microservices systems” Application developer (P5).

The other popular monitoring metrics reported by the participants are database
connections (54 out of 106, 50.9%), threads (47 out of 106, 44.3%), errors and
exceptions (46 out of 106, 43.3%), and endpoints success (44 out of 106,
41.5%) (see Figure 15 for more details).

$20$$25$$30$$35$$40$$45$$50$$55$$60$$65$$70$Resource usageLoad
balancingAvailabilityDatabase connectionsThreadsErrors and exceptionsEndpoint
successDatabase latencyMicroservice’s open fileSLAStatus of
dependenciesLanguage specific
metrics$71$$57$$56$$54$$47$$46$$44$$35$$33$$33$$31$$23$ Figure 15: Metrics
used for monitoring microservices systems

####  4.3.2 Monitoring practices

We asked the participants which practices they use to monitor microservices
systems (SQ29) about six monitoring practices that have been identified from
grey literature (e.g., [59, 33, 6]), (see Figure 16). The participants
mentioned the usage of the following three monitoring practices most often:
log management (33.0% very often, 35.8% often, 18.9% sometimes), exception
tracking (25.5% very often, 34.9% often, 25.5% sometimes), and health check
API (23.6% very often, 34.0% often, 28.3% sometimes).

To understand the reasons behind using the top three monitoring practices, we
asked IQ3.2.1. We summarized the collected reasons for using the top three
monitoring practices according to the responses from five out of six
interviewees (i.e., P1, P2, P3, P4, P5): monitoring practices (i) provide the
ability to track user activities for preventing, detecting, or minimizing the
impact of a data breach, (ii) help to continue analyzing the stability of
applications, (iii) help to perform the root cause analysis for specific
issues, (iv) help to track the causes of errors and analyze security issues,
(v) are used to track and notify the exceptions for developers, and (vi) show
the operational status of the service and its interconnection with other
services. Following is one representative quotation about the reason of using
the top three monitoring practices.

“In my opinion, these three practices complement each other to efficiently
manage the quality of each microservice in different operational environments.
Besides that, the Health Check API shows the operational status of the service
and its interconnection with other microservices.” Application developer (P5).

On the other hand, distributed tracking (17.0% rarely, 11.3% never) and audit
logging (14.2% rarely, 8.5% never) are less used (see Figure 16 for more
details).

$0\%$ $10\%$ $20\%$ $30\%$ $40\%$ $50\%$ $60\%$ $70\%$ $80\%$ $90\%$ $100\%$
Distributed trackingAudit loggingLog deploymentHealth check APIException
trackingLog management$11.3$ $8.5$ $6.6$ $3.8$ $2.8$ $3.8$ $17$ $14.2$ $7.5$
$10.4$ $11.3$ $8.5$ $31.1$ $26.4$ $34$ $28.3$ $25.5$ $18.9$ $30.2$ $34.9$
$31.1$ $34$ $34.9$ $35.8$ $10.4$ $16$ $20.8$ $23.6$ $25.5$ $33$ Very
oftenOftenSometimesRarelyNever Figure 16: Practices used for monitoring
microservices systems

####  4.3.3 Tools

A vast majority of the participants indicated Jira (42 out of 106, 39.6%) and
Datadog Kafka Dashboard (41 out of 106, 38.6%) as their monitoring tools,
which are followed by Spring Boot Actuator (24 out of 106, 22.6%), Grafana (19
out of 106, 17.9%), and Zipkin (18 out of 106, 16.9%) (see Figure 17 for
details). These tools are mainly used to monitor network performance (e.g.,
speed, bandwidth), resource usage (e.g., CPU, memory), and errors and
exceptions for microservices systems. We also received 17 short answers in the
free text field of survey question SQ30, where the participants provided 11
monitoring tools that they used to monitor microservices systems. These tools
are Apache skywalking141414https://skywalking.apache.org/ (3 responses), Azure
portal151515https://azure.microsoft.com/en-us/features/azure-portal/ (3
responses), Stackdriver161616https://cloud.google.com/stackdriver/ (2
responses), Dynatrace171717https://www.dynatrace.com/ (2 responses),
Jaeger181818https://www.jaegertracing.io/ (1 response), DaoCloud microservice
platform191919http://blog.daocloud.io/dmp/ (1 response),
Heapster202020https://github.com/kubernetes-retired/heapster (1 response),
Sensu212121https://sensu.io/ (1 response), Red Hat Jboss
fuse222222https://www.redhat.com/en/technologies/jboss-middleware/fuse (1
response), KintoHub232323https://www.kintohub.com/ (1 response), and self-
developed RPC tracking tool (1 response).

Besides that, we also asked IQ3.3.1 about the use of Jira as a monitoring
tools. Out of the six interviewees, three interviewees stated that Jira could
be integrated with Integrated Development Environment (IDE) to monitor
microservices systems. Two other interviewees considered issue tracking as
part of monitoring of microservices systems and other types of systems, and
they said that Jira as an issue tracking tool can be considered as a
monitoring tool. One interviewee was not sure about the use of Jira for
monitoring purposes, and he mentioned that “we are using Dynatrace for
monitoring microservices systems in our company”.

$0$$5$$10$$15$$20$$25$$30$$35$$40$$45$JiraDatadog Kafka DashboardSpring Boot
ActuatorGrafanaZipkinApache MesosPrometheusCactiiPerfNagiosPactApache
skywalkingAzure portalStackdriverDynatraceJaegerDaoCloud
microserviceHeapsterSensuRed Hat Jboss fuseKintoHubOmniaSelf-developed RPC
tracking$42$$41$$24$$19$$18$$17$$15$$9$$7$$6$$3$$3$$3$$2$$2$$1$$1$$1$$1$$1$$1$$1$$1$
Figure 17: Tools used for monitoring microservices systems

Key Findings of RQ2 Finding 13. Resource usage, load balancing, and
availability are the most frequently used monitoring metrics for microservices
systems. Finding 14. Log management, exception tracking, and health check API
are the most frequently employed monitoring practices for microservices
systems.

####  4.3.4 Challenges and solutions

Challenges: The respondents were asked to indicate what monitoring challenges
they faced (i.e., SQ31). The monitoring challenges listed as answers of SQ31
were collected from the academic and grey literature [8, 85, 86]. The third
column of Table LABEL:tab:monitoringChallenges shows the percentage of the
practitioners’ responses for each challenge. The three main challenges that
the participants faced during the monitoring of microservices systems are
collection of monitoring metrics data and logs from containers (58 out of 106,
54.7%), distributed tracing (48 out of 106, 45.2%), and having many components
to monitor (complexity) (44 out of 106, 41.5%). Other highly ranked challenges
are performance monitoring (43 out of 106, 40.5%), analyzing the collected
data (37 out of 106, 34.9%), and failure zone detection (36 out of 106,
33.9%).

Table 13: Challenges faced during the monitoring of microservices systems (in %) ID | Monitoring Challenges | Responses
---|---|---
MC1 | Collection of monitoring metrics data and logs from containers | 54.7
MC2 | Distributed tracing | 45.3
MC3 | Having many components to monitor (complexity) | 41.5
MC4 | Performance monitoring | 40.6
MC5 | Analysing the collected data | 34.9
MC6 | Failure zone detection | 34.0
MC7 | Availability of the monitoring tools | 21.7
MC8 | Monitoring of application running inside containers | 21.7
MC9 | Maintaining monitoring infrastructures | 17.0

We asked IQ3.4.1 about the reasons for the top three monitoring challenges.
Four out of six interviewees (i.e., P1, P2, P3, P5) provided the following
reasons about the monitoring challenges: (i) communication between hundreds of
microservices, (ii) runtime monitoring of microservices systems often does not
have a standardized infrastructure, (iii) different languages, databases, and
frameworks for developing microservices, and (iv) logs and dataflows in
different formats. Following is one representative quotation about the reason
for the monitoring challenges.

“Indeed, these are significant challenges. However, I think this is due to the
complex nature of microservices systems” Architect and Application developer
(P2).

Table 14: The severity of the challenges on the monitoring of microservices systems (in %) | Severity Scale
---|---
ID | Catastrophic | Major | Moderate | Minor | Insignificant
MC1 | 3.8 | 17.9 | 48.1 | 23.6 | 6.6
MC2 | 3.8 | 25.5 | 38.7 | 24.5 | 7.5
MC3 | 5.7 | 34.0 | 33.0 | 17.0 | 10.4
MC4 | 6.6 | 35.8 | 32.1 | 20.8 | 4.7
MC5 | 6.6 | 38.7 | 38.7 | 12.3 | 3.8
MC6 | 9.4 | 31.1 | 36.8 | 16.0 | 6.6
MC7 | 3.8 | 35.8 | 35.8 | 18.9 | 5.7
MC8 | 5.7 | 19.8 | 43.4 | 24.5 | 6.6
MC9 | 7.5 | 24.5 | 44.3 | 17.0 | 6.6

Solutions for monitoring challenges - from interview and survey results: We
received 24 distinct answers as the solutions for addressing monitoring
challenges (SQ33). After checking the answers, we found only 11 valid
solutions. We excluded those solutions that were incomplete, inconsistent, or
did not adequately address the challenges of monitoring microservices systems.
The participants recommended ten tools and one in-house developed framework to
address the challenges related to monitoring of microservices systems.

  * 1.

Monitoring tools and a self-developed framework: We received five responses in
which the participants named the tools and provided the purposes of using
tools. An respondent said that we use Prometheus and Spring Actuator for
collecting, aggregates, and visualize the monitoring data Operational staff
(R6). Similarly, another participant shared that our team uses ELK
(Elasticsearch, Logstash, and Kibana) Stack to address the challenges related
to a different kind of monitoring, for instance, application performance and
log monitoring DevOps engineer (R11). In another example, a participant
pointed out that we use health check API tools to detect the health of
microservices (e.g., failure zone detection) DevOps engineer (R91). Some
participants only recommended a few tools without explaining the reasons
behind using them in monitoring microservices systems. For example,
Grafana242424https://grafana.com/ (1 response), Zipkin252525https://zipkin.io/
(2 responses), and Raygun APM262626https://raygun.com/platform/apm (2
responses). However, we identified that these tools could be used to address
several challenges. For instance, MC1, MC2, MC4, and MC5 (see Table
LABEL:tab:MonitoringSol). On the other hand, only one participant referred to
their in-house developed framework to address the challenges of MC1:
collection of monitoring metrics data, MC6: failure zone detection, and MC8:
monitoring of application running inside containers. However, this participant
did not provide any reference for his developed framework. We also asked
IQ3.4.2 about the solutions of monitoring challenges. Three out of six
interviewees (i.e., P1, P3, P4) only suggested the names of several tools that
they were using in their organizations and these tools were successfully used
to address most of the monitoring challenges. The tool names are (i)
Dynatrace, (ii) Elastic Stack, (iii) Amazon CloudWatch, (iv) Jaeger, (v)
Graphana Dashboard, and (vi) Prometheus.

Table 15: Proposed solutions to address the monitoring challenges Type |  Proposed Solution |  | Challenge ID
---
Addressed
Count
Monitoring tools and frameworks |  ELK (Elasticsearch, Logstash, and Kibana) Stack | MC1, MC4, MC5 | 11
Prometheus and Spring Boot Actuator | MC1
Grafana | MC1
Zipkin | MC2
Raygun APM | MC4
Health check API tools | MC6
In-house framework | MC6

Key Findings of RQ2 Finding 15. The most prominent challenges of monitoring
microservices systems are collection of monitoring metrics data and logs from
containers, and distributed tracing.

####  4.3.5 Differences between the subpopulations of the study for monitoring
of microservices systems

We analyzed the answers to Likert scale survey questions related to monitoring
microservices in Table 16 and Table 17. In the following, we report the
statistical comparison for monitoring practices and challenges of
microservices systems.

Monitoring practices (see Table 16): Experience $\leq$ 2 years vs. Experience
> 2 years: There are two survey question statements (i.e., Health check API,
Log deployment and changes) with statistically significant differences between
Experience $\leq$ 2 years and Experience > 2 years groups. The Experience > 2
years group is more likely to use Health check API and Log deployment and
changes for monitoring microservices systems than the Experience $\leq$ 2
years group. On the other hand, we did not find any survey question statement
about using monitoring practices with statistically significant differences
for Experience $\leq$ 2 years group. MSA style vs. No MSA style: There are two
survey question statements (i.e., Audit logging, Distributed tracking) with
statistically significant differences between MSA style and No MSA style
groups. The No MSA style group is more likely to use Audit logging and
Distributed tracking for monitoring microservices systems than the MSA style
group. However, we did not find any survey question statement about using
monitoring practices with statistically significant differences for the MSA
style group. Employees size $\leq$ 100 vs. Employees size > 100: We identified
one survey question statement (i.e., Distributed tracking) with statistically
significant difference between Employees size $\leq$ 100 and Employees size >
100 groups. The Employees size > 100 group is more likely to use Distributed
tracking for monitoring microservices systems than the Employees size $\leq$
100 group. Moreover, we did not find any survey question statement about using
monitoring practices with statistically significant differences for the
Employees size $\leq$ 100 group.

Table 16: Statistically significant differences on the survey results about
monitoring practices

|  | Likert Distro. | Exper. $\leq$ 2 vs. > 2 years | MSA style vs. No MSA style | Empl. $\leq$ 100 vs. > 100
---|---|---|---|---|---
Survey Question Statement | SQ# | Mean value | P-value | Effect size | P-value | Effect size | P-value | Effect size
Practices to monitor microservices systems (Very important (5), Important (4),
Somewhat important (3), Important (2), Not sure (1))
Log management | SQ29 | 3.89 | 0.51 | -0.42 | 0.10 | 0.93 | 0.94 | -0.54
Audit logging | 3.42 | 0.43 | -0.42 |  0.02 | -0.09 | 0.92 | -0.54
Distributed tracking | 3.13 | 0.53 | -0.47 |  0.02 | -0.18 |  0.02 | -0.38
Exception tracking | 3.73 | 0.60 | -0.31 | 0.07 | 0.41 | 0.93 | -0.01
Health check API | 3.64 |  0.02 | -0.40 | 0.05 | 0.24 | 0.89 | 0.00
Log deployment and changes | 3.54 |  0.03 | -0.17 | 0.05 | 0.17 | 0.94 | 0.37

Monitoring challenges (see Table 17): Experience $\leq$ 2 years vs. Experience
> 2 years: There are two survey question statements (i.e., MC3, MC8) with
statistically significant differences between Experience $\leq$ 2 years and
Experience > 2 years groups. The Experience $\leq$ 2 years group is more
likely to agree that MC3 (i.e., having many components to monitor) challenge
has a severe impact on monitoring of microservices systems than the Experience
> 2 years group. On the other hand, the Experience > 2 years group is more
likely to agree that MC8 (i.e., monitoring of application running inside
containers) challenge has a critical impact on monitoring of microservices
systems than Experience $\leq$ 2 years group. MSA style vs. No MSA style:
There are two survey question statements (i.e., MC4, MC6) with statistically
significant differences between MSA style and No MSA style groups. The MSA
style group is more likely to agree that MC4 (i.e., performance monitoring)
and MC6 (i.e., failure zone detection) challenges have severe impact on
monitoring of microservices systems than the No MSA style group. However, we
did not find any survey question statement about monitoring challenges with
statistically significant differences for the No MSA style group. Employees
size $\leq$ 100 vs. Employees size > 100: We identified one survey question
statement (i.e., MC5) with statistically significant difference between
Employees size $\leq$ 100 and Employees size > 100 groups. The Employees size
> 100 group is more likely to agree that MC5 (i.e., analyzing the collected
data) challenge has a severe impact on monitoring of microservices systems
than the Employees size $\leq$ 100 group. Moreover, we did not find any survey
question statement about monitoring challenges with statistically significant
difference for the Employees size $\leq$ 100 group.

Table 17: Statistically significant differences on the survey results about
monitoring challenges

|  | Likert Distro. | Exper. $\leq$ 2 vs. > 2 years | MSA style vs. No MSA style | Empl. $\leq$ 100 vs. > 100
---|---|---|---|---|---
Survey Question Statement | SQ# | Mean value | P-value | Effect size | P-value | Effect size | P-value | Effect size
Severity of each challenge related to the monitoring of MSA-based systems
(Catastrophic (5), Major (4), Moderate (3), Minor (2), Insignificant(1))
MC1 | SQ32 | 3.15 | 0.16 | -0.03 | 0.09 | -0.02 | 0.94 | -0.13
MC2 | 2.94 | 0.44 | -0.21 | 0.06 | 0.03 | 0.94 | 0.23
MC3 | 3.09 |  0.04 | 0.21 | 0.08 | 0.99 | 0.94 | -0.25
MC4 | 3.20 | 0.14 | 0.04 |  0.03 | -0.43 | 0.94 | -0.20
MC5 | 3.33 | 0.19 | -0.09 | 0.14 | -0.06 |  0.04 | -0.01
MC6 | 3.24 | 0.12 | -0.12 |  0.04 | -0.08 | 0.97 | 0.10
MC7 | 2.89 | 0.19 | -0.04 | 0.12 | 0.26 | 0.95 | -0.16
MC8 | 2.94 |  0.02 | -0.06 | 0.07 | -0.04 | 0.94 | 0.04
MC9 | 3.09 | 0.15 | 0.08 | 0.08 | 0.41 | 0.94 | 0.04

###  4.4 Testing of microservices systems (RQ3)

####  4.4.1 Testing strategies

We asked SQ34 to collect information relevant to the use of testing strategies
for microservices systems [9, 87, 61, 66]. The top three most frequently used
testing strategies are: Unit testing (34.0% very often, 29.2% often, and 10.4%
sometimes), E2E testing (26.4% very often, and 29.2% often, 10.4% sometimes),
and Integration testing (23.6% very often, 35.8% often, and 6.6% sometimes).

We asked IQ4.1.1 (see Table LABEL:tab:IQMsatesting) about the reasons for
using the top three testing strategies. All of the interviewees were familiar
with these testing strategies. They also acknowledged that they frequently use
these testing strategies to test microservices systems. According to the
interviewees, these are fundamental testing strategies that can be used for
all type of systems (e.g., MSA-based systems, SOA-based systems). In the
following, we summarized the collected reasons. Concerning the reasons of
using Unit testing, all the interviewees mentioned that they use this strategy
because Unit testing (i) allows isolated testing of the design components of
microservices systems (e.g., services, domains, service interfaces), (ii)
makes sure each microservice working correctly and delivering the required
functionalities, (iii) helps to identify the problems with individual
microservice in the early stage of system development. About the reasons of
using integration testing, four out of six interviewees (i.e., P2, P3, P5, P6)
stated that this strategy integrates microservices together to verify that (i)
microservices are working collaboratively to achieve the business goals, (ii)
communication paths between microservices are working correctly, and (iii)
third-party APIs and microservices are working together as expected. About the
reasons of using E2E testing, three out of six interviewees (i.e., P2, P5, P6)
stated that they typically perform E2E testing for testing the whole
microservices systems, and the purposes of using E2E testing are to (i) ensure
the correctness of the whole microservices systems and (ii) understand how the
customers will use the developed systems. Following is one representative
quotation about the reason of using testing strategies for microservices
systems.

“The reasons for using these testing strategies are the same for microservices
and monolithic systems. To the best of my knowledge. There is no specific
testing technique that is designed or used in the industry specifically to
test microservices systems” Architect and Application developer (P1).

On the other hand, the following three testing strategies are used sparingly:
A/B testing (26.4% rarely, 27.4% never), Consumer-driven contract testing
(29.2% rarely, 16.0% never), and Service component testing (33.0% rarely, 5.7%
never) (see Figure 18 for details).

$0\%$ $10\%$ $20\%$ $30\%$ $40\%$ $50\%$ $60\%$ $70\%$ $80\%$ $90\%$ $100\%$
Consumer-driven contractService componentA/B testingIntegration
contractScalabilityComponentUser interfaceIntegrationE2EUnit$16$ $5.7$ $27.4$
$10.4$ $10.4$ $11.3$ $7.5$ $3.8$ $5.7$ $5.7$ $29.2$ $33$ $26.4$ $24.5$ $25.5$
$28.3$ $31.1$ $30.2$ $28.3$ $20.8$ $19.8$ $18.9$ $13.2$ $17.9$ $17$ $8.5$
$14.2$ $6.6$ $10.4$ $10.4$ $28.3$ $29.2$ $19.8$ $33$ $32.1$ $32.1$ $26.4$
$35.8$ $29.2$ $29.2$ $6.6$ $13.2$ $13.2$ $14.2$ $15.1$ $19.8$ $20.8$ $23.6$
$26.4$ $34$ Very OftenOftenSometimesRarelyNever Figure 18: Testing strategies
used to test microservices systems

####  4.4.2 Tools