Conceptualization of this study, Methodology, Investigation, Data curation,
Writing - Original draft preparation

Methodology, Investigation, Data curation, Software, Writing - Original draft
preparation

[1] Conceptualization of this study, Methodology, Writing - Original draft
preparation

Conceptualization of this study, Methodology

Methodology, Writing

Conceptualization of this study, Methodology

Investigation, Data curation

[cor1]Corresponding author.

# Technical Debt Management in OSS Projects: An Empirical Study on GitHub

Zengyang Li<EMAIL_ADDRESS>School of Computer Science & Hubei
Provincial Key Laboratory of Artificial Intelligence and Smart Learning,
Central China Normal University, Wuhan, China
Yilin Peng<EMAIL_ADDRESS>Peng Liang<EMAIL_ADDRESS>School of
Computer Science, Wuhan University, Wuhan, China  Apostolos Ampatzoglou
<EMAIL_ADDRESS>Department of Applied Informatics, University of
Macedonia, Thessaloniki, Greece  Ran Mo<EMAIL_ADDRESS>Hui Liu
<EMAIL_ADDRESS>School of Artificial Intelligence and Automation, Huazhong
University of Science and Technology, Wuhan, China  Xiaoxiao Qi
<EMAIL_ADDRESS>
###### Abstract

Technical debt (TD) refers to delayed tasks and immature artifacts that may
bring short-term benefits but incur extra costs of change during maintenance
and evolution in the long term. TD has been extensively studied in the past
decade, and numerous open source software (OSS) projects were used to explore
specific aspects of TD and validate various approaches for TD management
(TDM). However, there still lacks a comprehensive understanding on the
practice of TDM in OSS development, which penetrates the OSS community’s
perception of the TD concept and how TD is managed in OSS development. To this
end, we conducted an empirical study on the whole GitHub to explore the
adoption and execution of TDM based on issues in OSS projects. We collected
35,278 issues labeled as TD (TD issues) distributed over 3,598 repositories in
total from the issue tracking system of GitHub between 2009 and 2020. The
findings are that: (1) the OSS community is embracing the TD concept; (2) the
analysis of TD instances shows that TD may affect both internal and external
quality of software systems; (3) only one TD issue was identified in 31.1% of
the repositories and all TD issues were identified by only one developer in
69.0% of the repositories; (4) TDM was ignored in 27.3% of the repositories
after TD issues were identified; and (5) among the repositories with TD
labels, 32.9% have abandoned TDM while only 8.2% adopt TDM as a consistent
practice. These findings provide valuable insights for practitioners in TDM
and promising research directions for further investigation.

###### keywords:

Technical debt, technical debt management, issue, open source software,
empirical study, GitHub

##  1 Introduction

Technical debt (TD) refers to delayed tasks and immature artifacts that
constitute a “debt” since they may bring short-term benefits but incur extra
costs of change during maintenance and evolution in the long term (Cunningham,
1992; Avgeriou et al., 2016). TD can be classified into several types from the
perspective of different stages of the software development lifecycle, such as
architecture TD and design TD (Tom et al., 2013; Li et al., 2015). Due to its
significant impact on software quality, TD has been a hot topic attracting
great attention from both academia and industry in the last decade (Li et al.,
2015; Alves et al., 2016; Tufano et al., 2017; Kruchten et al., 2019;
Ciolkowski et al., 2021; de Toledo et al., 2021; Ernst et al., 2021; Vidoni et
al., 2022) , and many open source software (OSS) projects were used to explore
various aspects of TD (Tsoukalas et al., 2020; Li et al., 2020b; Xiao et al.,
2021; Tan et al., 2021; Yu et al., 2022).

However, there still lacks a comprehensive understanding on TD management
(TDM) in OSS development, which penetrates the OSS community’s perception of
the TD concept and how TD is managed in OSS development. On GitHub, a
proportion of issues that are explicitly tagged as TD, which are called as TD
issues in this paper. Every TD issue is intentionally tagged with a TD label
by a practitioner, which means that the practitioner actively performs TD
identification, a core activity of TDM (Li et al., 2015). Since TD issues are
identified by the practitioners themselves and no biases of researchers are
introduced, TD issues can directly reflect the practitioner’s perception and
management strategy of TD. TD issues are essentially a type of self-admitted
technical debt (SATD) (Dai and Kruchten, 2017; Xavier et al., 2020).
Therefore, TD issues can be used as an important data source to
comprehensively investigate TDM in OSS development.

The goal of this work is to explore TDM in OSS development based on TD issues.
Specifically, we aim to investigate the trend of the adoption of TDM, the
practitioners’ perception on the TD concept, and how practitioners manage TD
in OSS development. To this end, we conducted a large-scale empirical study on
the TD issues of all OSS projects hosted on GitHub.

The main contributions of this work are summarized as follows: (1) To our
knowledge, this work is the first study that took all repositories hosted on
GitHub as the search space to understand the state of TDM in practice in a
large-scale OSS ecosystem. (2) We performed a large-scale empirical study on
TDM based on 35,278 issues with TD labels (i.e., TD issues) distributed over
3,598 repositories. (3) We explored TD issues from the perspectives of the
involved people and management process. (4) We found that the practitioners’
awareness of TDM is increasing over the past decade given the relatively high
growth rates of new TD issues and new repositories adopting TDM. (5) We found
that only 298 (8.2%) repositories adopt TDM as a consistent practice, while
1,187 (32.9%) repositories have abandoned TDM. (6) We found that the power-law
characteristic exists in the distributions of the proportion of repositories
with TD labels over the number of TD issues and of the proportion of resolved
TD issues over their open time.

The remaining of this paper is organized as follows. Section 2 discusses the
related work; Section 3 describes the study design; Section 4 presents the
study results; Section 5 interprets the study results with the implications;
Section 6 highlights the threats to the validity of the results; and Section 7
concludes this work with future research directions.

##  2 Related Work

###  2.1 Technical Debt in OSS

Many studies investigated TD in OSS from diverse perspectives. Digkas _et al_.
looked into the evolution of TD in 66 Java projects from the Apache ecosystem
(Digkas et al., 2017). They studied three aspects, i.e., the evolution of
normalized TD, the most common types of TD, and the most costly TD. Tan et al.
conducted a case study also on the Apache ecosystem to investigate the
evolution of TD remediation in Python (Tan et al., 2021). Li _et al_.
performed an empirical study on 59 Apache OSS projects to investigate the
characteristics of the interest of defect TD (Li et al., 2020b). Tsoukalas _et
al_. carried out an empirical study on more than 100 open-source repositories
from multiple OSS platforms for TD forecasting (Tsoukalas et al., 2020).
Lenarduzzi _et al_. performed a case study of 33 Apache projects to explore
the spread of TD and how quickly it can be fixed (Lenarduzzi et al., 2019).
Digkas _et al_. conducted a case study of 27 Apache projects to investigate
the stability of the introduction of TD, and the correlation between the
introduction of TD and the workload of the development team (Digkas et al.,
2020).

The aforementioned studies took specific OSS projects as research objects to
explore certain phenomena with respect to TD. In contrast, our work takes all
repositories hosted on GitHub as our search space, and much more repositories
with TD issues were included in our dataset.

###  2.2 Technical Debt Management

Literature reviews on technical debt management. A number of literature
reviews have been performed to study TDM as a whole or some specific aspects
of TDM. Li _et al_. conducted a systematic mapping study to examine the
current state of TDM, including categories of TD, activities of TDM,
approaches and tools for TDM, and challenges to TDM (Li et al., 2015). Becker
et al. performed a systematic literature review on trade-off decisions across
time in TDM (Becker et al., 2018). Lenarduzzi _et al_. published a systematic
literature review to understand the priority of refactoring TD over the
lifespan of software, compared with developing features and fixing bugs
(Lenarduzzi et al., 2021). In the works mentioned above, the state of TDM was
studied based on literature, while in our work we investigated the state of
TDM based on practices in OSS repositories.

Management of self-admitted technical debt. In recent years, intensive efforts
have been invested to the research of a special kind of TD – self-admitted
technical debt (SATD). Potdar and Shihab coined the concept of SATD, which is
sub-optimal solutions intentionally introduced (e.g., temporary fixes) and
explicitly documented using code comments (Potdar and Shihab, 2014; Sierra et
al., 2019). Most studies on SATD investigated TD admitted in the comments of
source code. Huang _et al_. proposed a model to predict whether a comment in
software contains TD (Huang et al., 2018). Ren _et al_. presented a CNN-based
approach to detect SATD in code comments (Ren et al., 2019). Yu _et al_.
proposed Jitterbug to automatically identify SATD (Yu et al., 2022). Bavota
_et al_. analyzed the diffusion, evolution, participants, and impacted quality
of SATD based on commits and comments of projects (Bavota and Russo, 2016),
and their study facilitates the evaluation of the impact of TD in TDM. Codabux
_et al_. explored the types and characteristics of TD in software systems
written in R language by manually analyzing the comments in the peer-review
documentation of R-language OSS packages (Codabux et al., 2021). Zampetti et
al. surveyed industrial developers and OSS developers to compare the practices
on SATD in industry and OSS development (Zampetti et al., 2021). They found
that both industrial and OSS developers share similar perceptions and actions
regarding TD (Zampetti et al., 2021). Besides, industry developers were more
reluctant to acknowledge TD than OSS developers. Unlike the aforementioned
works that investigate SATD and its management based on code comments, our
work studies TD and TDM based on issues.

Technical debt management based on issues. There are also a few studies that
used issues in issue tracking systems as a data source to investigate TD and
TDM. Bellomo _et al_. manually examined 1,264 issues from four software
projects and identified 109 issues as TD items, which were used as the dataset
to explore characteristics of TD (Bellomo et al., 2016). Dai and Kruchten
manually checked all issues of a commercial software project and identified
331 issues containing TD, and then used natural language processing and
machine learning to automatically detect TD issues of the same project (Dai
and Kruchten, 2017). Li et al. regarded manually-identified issues containing
TD in issue tracking systems as SATD, and further investigated the
identification and remediation of such TD issues (Li et al., 2020a). In
contrast, Xavier _et al_. considered issues with TD labels (i.e., TD issues)
as a kind of SATD, and looked into why TD issues are introduced and paid
(Xavier et al., 2020). Similar to Xavier _et al_. (Xavier et al., 2020), in
this work we only investigated issues that were tagged with TD labels (TD
issues) in issue tracking systems.

##  3 Study Design

###  3.1 Objective and Research Questions

The objective of this study, described using the Goal-Question-Metric (GQM)
approach (Basili, 1992), is: to _analyze_ software issues tagged as TD _for
the purpose of_ exploration _with respect to_ the current state of TDM in
practice _from the point of view of_ software practitioners _in the context
of_ OSS development. Based on the aforementioned goal, we formulated five
research questions (RQs), which are described as follows:

  * RQ1:

How popular do software repositories adopt TDM?
Rationale: With this RQ, we investigated the popularity of the adoption of TDM
in OSS development, which can reflect the degree of acceptance of the TD
concept and practitioners’ awareness of TDM in the context OSS development.

  * RQ2:

What is the understanding of practitioners on the TD concept?
Rationale: Practitioners understand the TD concept in different ways, which
may influence the strategies and activities of TDM. The answer to this RQ is
helpful to understand and improve TDM strategies.

  * RQ3:

How are TD issues identified?
Rationale: TD identification is the first step of TDM. With this RQ, we
investigated when and by whom TD issues are identified. The answer to this RQ
enables us to evaluate practitioners’ awareness of TDM and existing strategies
of TDM.

  * RQ4:

How are TD issues resolved?
Rationale: With this RQ, we investigated how TD issues are handled. In
particular, the open time, characteristics, and proportion of reopens of
resolved TD issues can provide us with multiple perspectives to understand the
resolution process of TD issues.

  * RQ5:

What is the continuity of adopting TDM for the repositories with TD labels?
Rationale: This RQ investigates whether TDM has been continuously employed for
the repositories with TD labels, and to further explore the different
characteristics between repositories with different levels of continuity of
adopting TDM.

###  3.2 Data Collection

####  3.2.1 Data items to be collected

First, to answer the five RQs formulated in Section 3.1, we needed to collect
all TD issues on the whole GitHub. Such TD issues are included in dataset DS1,
which is the main dataset of this work. Second, to answer RQ2, besides DS1, we
also needed to collect additional data items regarding the quality attributes
(QAs) affected by and specific TD types of TD issues. Since manual analysis of
the affected QAs and TD types of TD issues requires considerable effort, we
analyzed a representative subset of DS1, which is denoted as DS2. Besides, in
order to get a deeper understanding of the features of TD issues, we needed to
collect data on TD issues and non-TD issues of repositories that each includes
a specific amount of TD issues. Such data constitute another dataset, which is
denoted as DS3. The data items to be collected are listed in Table 1, which
also provides the information on the dataset(s) containing each data item.
Table 2 shows the dataset(s) and data items used by each RQ. In this study, we
collected TD issues that were reported between Jan. 1st, 2009 and Dec. 31st,
2020. Extra data items on the whole GitHub were also collected (Section
3.2.5). The data collection processes for the three datasets are shown in
Figure 1, in which the data collection process for the extra data is not
included.

Table 1: Data items to be collected for each issue.

# |  Name |  Description |  Dataset
---|---|---|---
D1 |  IssueID |  The global identity number of the issue on GitHub. |  DS1, DS2, DS3
D2 |  Reporter |  The GitHub user identifier of the person who reported the issue. |  DS1, DS3
D3 |  RepTime |  The time when the issue was reported. |  DS1, DS3
D4 |  ResTime |  The time when the issue was resolved. |  DS1, DS3
D5 |  Labels |  All labels assigned to the issue. |  DS1, DS3
D6 |  IsTD |  Whether the issue was tagged with a TD label. |  DS3
D7 |  LabeledEvent |  The labeled event of a given issue. Each labeled event contains information such as the participant who tagged this issue, the string of label, the time of creating labeled event. |  DS1
D8 |  Repo |  The repository to which the issue belongs. |  DS1, DS3
D9 |  QA |  The quality attribute affected by the issue. |  DS2
D10 |  TDType |  The TD type of a TD issue. |  DS2
D11 |  IsResolved |  Whether the issue is resolved. |  DS1, DS3
D12 |  CmntEvent |  The comment event of a given issue. Each comment event contains information such as the participant who added this comment, the comment content, the time of creating this event. |  DS3
D13 |  RepoRelation |  The relationships between issue reporter and repository. |  DS1
D14 |  IssueTitle |  The text of the issue title. |  DS1, DS3
D15 |  IssueDesc |  The text of the issue description. |  DS1, DS3
D16 |  ReopenedEvent |  The reopened event of a given issue. Each reopen event contains information such as the participant who reopened issue to be resolved, the time of creating reopened event. |  DS3

Table 2: The dataset(s) and data items used by each RQ.

RQ |  Dataset[items]
---|---
RQ1 |  DS1[D1, D3, D8]
RQ2 |  DS1[D5]; DS2[D1, D9, D10]
RQ3 |  DS1[D1, D2, D3, D6, D7, D8, D14, D13, D15]; DS3[D6, D14, D15]
RQ4 |  DS1[D1, D3, D4, D8, D11]; DS3[D1, D3, D4, D6, D12, D14, D15, D16]
RQ5 |  DS1[D1, D2, D3, D4, D6, D8, D11, D12]

Figure 1: Procedure of data collection.

####  3.2.2 Data collection for DS1

The data collection procedure for dataset DS1 is composed of the following 5
steps:

Step1: Collect TD labels. In GitHub, developers can use a TD label (e.g.,
tech-debt) to explicitly tag an issue as TD. To collect a complete list of
candidate TD labels, we first came up with a set of equivalents (e.g., case
conversion and word abbreviation) to "technical debt". Then, we used "debt" as
a keyword to search on the issue tracking system of GitHub, and manually
checked the returned issues to collect more candidate TD labels. We could not
find more new TD labels after we read around the top 300 returned issues and
stopped the reading process after we read the top 500 returned issues.
Finally, we added labels of TD types according to the classification of TD in
(Li et al., 2015). The final set of candidate TD labels to search TD issues
are provided in Table 3, where the case of the TD labels is ignored.

Step 2: Download requested data. In this step, we downloaded TD issues and
related data (e.g., labels) according to the candidate TD labels collected,
using a dedicated tool that we developed.

Step 3: Clean auto-generated data. ImDone, an automatic issues generating
tool, can automatically generate issues according to code comments (imDone
team, 2022). When the text of the comment contains "TODO", imDone
automatically generates an issue and tags it as “debt”. Such auto-generated
issues with label “debt” are invalid: the ImDone team forked many GitHub
repositories to test ImDone, producing a large number of issues with label
“debt” in a short time, which damaged the true and real state and life cycle
of some issues. In addition, since an issue was automatically tagged with a TD
label by imDone, it cannot guarantee that the code comment author considered
the “TODO” as TD. Hence, such auto-generated issues with label “debt” by
imDone should be filtered out to ensure the authenticity and accuracy of the
datasets.

Step 4: Clean duplicated TD issues. Considering that some issues were
submitted multiple times on GitHub, duplicated issues should be cleaned to
ensure the uniqueness of data. Duplicated issues are defined as follows: if
the Repo, Reporter, RepTime, IssueTitle, IssueDesc of issues are exactly the
same, these issues are considered duplicated. Only one issue will retain in
the dataset.

Step 5: Clean issues with TD labels without the meaning of TD. Some issues
tagged with TD labels do not have practical meaning related to TD, but have
repository-specific meaning. For example, label "TD" does not refer to
technical debt in repository w3c/wot. Therefore, such issues should be
filtered out.

Table 3: TD labels to be chosen. TD label
---
technical debt | td | code debt
technicaldebt | kind/tech-debt | test debt
technical-debt | refactoring / tech debt | design debt
techdebt | infrastructure debt | build debt
tech-debt | requirements debt | doc debt
tech debt | architectural debt | defect debt
debt | versioning debt | ux/ui debt

####  3.2.3 Data collection for DS2

In order to explore the understanding of practitioners on the concept of TD,
we built dataset DS2 in the following two steps:

Step1: Sample TD issues. According to Israel’s theory of determining sample
size (Israel, 1992), we randomly sampled the resolved TD issues in dataset
DS1, setting the margin of error as 5% and the confidence level as 99%. The
reason for choosing resolved TD issues is that the status of such TD issues is
stable.

Step2: Analyze the sample manually. In this step, we manually tagged the TD
type of and the QA affected by each TD issue in DS2 to answer RQ2. The
definitions of TD types can be found in (Li et al., 2015), and the QAs are
adopted from the product QAs defined in the ISO 25010 standard (ISO/IEC,
2011). In the tagging process, if an issue involves multiple TD types or QAs,
we chose the major one. The tagging process is described as follows: At first,
two researchers made a pilot tagging on 50 TD issues to reach a consensus on
the understanding of the TD types and QAs. All disagreements were recorded,
and a third researcher joined in to discuss until the three researchers
reached a consensus. After this step, the researchers reached an agreement on
the scope and boundary of TD types and QAs. Later, the two researchers
independently tagged a round of 50 TD issues and the Cohen’s Kappa coefficient
indicating the agreement between the two researchers was calculated (Cohen,
1960). If the coefficient was less than 0.8, the issues with inconsistent
tagging results were revisited among the three researchers until agreement was
reached. The next round of tagging was performed until the Cohen’s Kappa
coefficient is greater than 0.8. After that, the remaining issues were divided
into two parts and each was tagged by one of the two researchers
independently.

####  3.2.4 Data collection for DS3

In order to understand the characteristics of TD issues, we additionally added
a dataset DS3, containing resolved TD issues and resolved non-TD issues. The
data collection process for DS3 consists of two steps:

Step1: Select TD issues from specific repositories. Specifically, this step is
to select resolved TD issues of repositories with not less than 100 resolved
TD issues from DS1.

Step2: Select non-TD issues from specific repositories. To be specific, this
step is to download and store all the resolved non-TD issues of the
repositories mentioned in Step1 from GitHub.

The reason for selecting such repositories is that we believed that the
participants of such repositories are more likely to have mature knowledge
about TD, and the status of a resolved issue is more stable than an unresolved
one.

####  3.2.5 Extra data

In order to understand the overall issues on GitHub and compare them with TD
issues, we collected the number of newly reported issues on GitHub each year,
the number of newly reported resolved issues on GitHub each year, the number
of repositories newly adopting TDM on GitHub each year, the number of issues
of each repository, the number of resolved issues of each repository. To
understand the maintenance status of each repository, the creating time of the
last pull request for each repository was collected as well. Since the extra
data can be retrieved from GitHub by simply using the GitHub APIs, the
collection process of extra data is not shown in Figure 1.

###  3.3 Data Analysis

####  3.3.1 Popularity of adopting TDM (RQ1)

To answer RQ1, we analyze dataset DS1 from three aspects: (1) trend of the
number of repositories newly adopting TDM, (2) trend of the number of newly
reported TD issues, and (3) distribution of ratio of repositories over the
number of TD issues. The time when the first TD issue is reported for a
repository is considered as the time of the repository adopting TDM. A
repository containing one or more TD issues is considered as it adopting TDM.
Every TD issue is intentionally tagged with a TD label by a practitioner,
which means that the TD identification activity of TDM is actually performed
by the practitioner of the repository. Thus, we consider that the practitioner
actively adopts TDM for the repository even it contains only one single TD
issue.

Besides descriptive statistics, the compound annual growth rate (CAGR) was
used to measure the growth trend of the number of repositories newly adopting
TDM and newly reported TD issues in whole DS1. CAGR is defined as:

$CAGR=((n_{2}/n_{1})^{1/(y_{2}-y_{1})}-1)\times{100\%}$ (1)

where $n_{1}$ and $n_{2}$ are the numbers in the years $y_{1}$ and $y_{2}$,
respectively.

####  3.3.2 Understanding on the TD concept (RQ2)

QAs affected by TD are a central concern in TD research (Li et al., 2015), and
issue labels tagged to TD issues can reflect the characteristics of TD issues
to some extent. Thus, we investigate the understanding of practitioners on the
TD concept from two perspectives: (1) QAs affected by TD, and (2) co-occurring
issue labels with TD labels.

QAs affected by TD. As described in Section 3.2.3, we manually analyzed a
sample dataset (i.e., DS2) of all TD issues to explore their TD types and the
QAs affected. We then analyzed how each QA is affected by TD. From the QA
perspective, we analyzed the TD types affecting each QA, and the extent of the
impact.

Co-occurring labels with TD labels. To understand characteristics of TD issues
in terms of co-occurring labels, descriptive statistics were used to analyze
the number of occurrences of TD labels and the number of labels accompanying
TD labels. After obtaining all the issue labels, we further screened all the
issue labels to pick out labels with clear meaning and general purposes. The
screening process includes three steps: a) keep clear and general issues
(i.e., not specific to a certain project/repository and not describing the
status of the issue), b) filter out labels that occur for less than 100 times,
and c) combine labels with the same meaning.

####  3.3.3 Identification of TD issues (RQ3)

On GitHub, we can trace the timeline of the labeled events (i.e., data item
LabeledEvent) pertaining to each issue to further analyze the characteristics
of TD issues. A LabeledEvent contains information, such as the person who
tagged a label, the label name, the time of tagging the label to the issue. In
the following, RQ3 is analyzed from three perspectives.

(1) When TD issues are identified. We considered the time when a TD label is
assigned to the issue as the time when TD is identified for an issue. We
counted the difference between the reported time and the identification time
for each TD issue. Further, TD identification time can happen at two possible
points of time: a) the time when an issue is reported and b) a specific time
after an issue is opened to resolve.

(2) Who identifies TD issues. To understand the participants’ awareness of
TDM, we analyzed the participants from several viewpoints. First, we examined
the overall distribution and characteristics of participants. Second, we
focused on issue reporters who are a key type of participants to identify TD
issues. GitHub defines a set of relationships between issue reporter and
repository, including: COLLABORATOR, CONTRIBUTOR, FIRST_TIMER,
FIRST_TIME_CONTRIBUTOR, MANNEQUIN, MEMBER, OWNER, and NONE. Please refer to
Table 4 for their detailed definitions. We calculated the distribution of the
reporters who identified TD issues.

(3) Phase of adopting TDM in the development lifecycle. For each repository,
let $Period_{TD}$ denote the period between the time when the first TD issue
is identified and the time when the first issue is reported, let
$Period_{Full}$ denote the period between the time when the last pull request
is handled and the time when the first issue is reported, and $Phase_{TD}$ is
defined as

$Phase_{TD}=Period_{TD}/Period_{Full}.$ (2)

Then, $Phase_{TD}\geq 0$, and a smaller $Phase_{TD}$ means earlier awareness
of TDM for the repository.

Table 4: Relationships between issue reporter and repository.

Relationship | Description
---|---
COLLABORATOR | The issue reporter has been invited to collaborate on the repository.
CONTRIBUTOR | The issue reporter has previously committed to the repository.
FIRST_TIMER | The issue reporter has not previously committed to GitHub.
FIRST_TIME_CONTRIBUTOR | The issue reporter has not previously committed to the repository.
MANNEQUIN | The issue reporter is a placeholder for an unclaimed user.
MEMBER | The issue reporter is a member of the organization that owns the repository.
OWNER | The issue reporter is the owner of the repository.
NONE | The issue reporter has no association with the repository.

####  3.3.4 Resolution of TD issues (RQ4)

To answer RQ4, we made analysis in four aspects:

(1) Proportion of resolved TD issues. For each repository in dataset DS1, we
calculated the proportion of resolved TD issues over the total TD issues based
on data item IsResolved (i.e., D11). We then calculated the distribution of
the number of repositories over the proportion of resolved TD issues. In
addition, we also calculated the proportion of TD issues resolved and the
proportion of resolved issues on the whole GitHub from 2009 to 2020.

(2) Open time of resolved TD issues. At first, we calculated the open time of
each resolved TD issue as the difference between ResTime and RepTime of the
issue. In the subsequent data analysis, we calculated the distribution of the
proportion of resolved TD issues to all resolved TD issues in DS1 over the
open time, and observed the characteristics of resolved TD issues in terms of
open time; we also compared the differences in the distribution of resolved TD
issues and resolved non-TD issues in DS3.

(3) Characteristics of resolved TD issues.

We explored the TD issue resolution process based on the characteristics of
resolved issues. Only resolved issues were used as the data source because
their status is much more stable than unresolved issues.

On GitHub, each issue allows participants to discuss the issue by posting
comments on it, in addition to the title and descriptive text provided by the
reporter. When a participant posts a comment on the issue, a comment event
(i.e., data item CmntEvent) for the issue is generated.

Then, we investigated whether the resolved TD issue and the resolved non-TD
issue differ significantly in terms of the issue content and the corresponding
comment events, including four characteristics: the size of the title and
description (IssueSize), the number of all CmntEvents (CmntNo), the size of
the content of all CmntEvents (CmntSize), the number of the participants who
added all CmntEvents (ParticipantNo). We ran Mann-Whitney U Tests (Field,
2013) on the four issue characteristics of resolved TD issues and resolved
non-TD issues in DS3, and corrected the p-value of Mann-Whitney U Tests by the
Bonferroni correction in account for the multiple comparisons problem (Dunn,
1961).

(4) Reopen of resolved TD issues. On GitHub, if an issue is resolved, the
status of the issue is updated to "closed". However, an issue can be changed
to "opened" again, which means that the issue may need further discussion and
fixes. We considered that the probability for a resolved issue to reopen
reflects the quality of the previous fix of the issue. Thus, we calculated the
probabilities of resolved TD issues and non-TD issues based on the reopened
events (i.e., data item ReopenedEvent) of all issues in DS3.

####  3.3.5 Continuity of TDM for the repositories with TD labels (RQ5)

To better understand the continuity of TDM of repositories with TD labels, we
defined three levels of TDM continuity (namely Abandoned, Unclear, and
Consistent) for such repositories. First, for convenient description, we
define several notations. $Period_{PT}$ denotes the period between the time
when the last pull request is handled and the time when the last TD issue is
reported. $Period_{TT}$ denotes the period between the time when the last TD
issue is reported and the time when the first TD issue is reported.
$Period_{Y}$ is a constant, and defined as: suppose that the sequence of time
differences between the tagging time of each pair of adjacent TD issues of
each repository is known, and $Period_{Y}$ is the 99th percentile of this
sequence of time differences. $Count_{R}$ denotes the number of TD issues of
the repository. $Count_{Z}$ is a constant, which denotes the 75th percentile
of the number of TD issues of the repositories in DS1. Then, the conditions
for the three TDM continuity levels for a repository are defined as follows.
1) Abandoned: the repository has abandoned TDM, i.e.,
$Period_{PT}>Period_{Y}$.
2) Unclear: the repository does not explicitly show TDM is adopted
continuously, i.e., ($Period_{PT}\leq Period_{Y}$) $\land$ (($Period_{TT}\leq
Period_{Y}$)$\vee$($Count_{R}<Count_{Z}$)) is true.
3) Consistent: the repository keeps adopting TDM, i.e.,
($Period_{PT}\leq{Period_{Y}}$)$\land$($Period_{TT}>Period_{Y}$)$\land$($Count_{R}\geq{Count_{Z}}$)
is true.

Subsequently, we analyzed the levels of all repositories in DS1, and further
analyzed the differences between repositories at level Abandoned and level
Consistent. We focused on the differences between the two levels of
repositories in terms of total issues and TD issues. Seven characteristics for
each repository were extracted: the number of total issues (RepoIssueNo), the
proportion of total issues resolved (RepoResolvedIssuePro), the number of TD
issue reporters (RepoTDRepoterNo), the number of TD issues (RepoTDIssueNo),
the proportion of TD issues resolved (RepoResolvedTDIssuePro), the average
open time of TD issues (RepoOpentimeAvg), and the average length of the
comments of TD issues (RepoCmntSizeAvg).

Furthermore, we used Mann-Whitney U Tests (Field, 2013) on the characteristics
to compare if there is a significant difference between the repositories of
level Abandoned and those of level Consistent. In addition, given the problem
of multiple comparisons, we performed the Bonferroni correction to correct the
p-value of Mann-Whitney U tests (Dunn, 1961).

##  4 Results

###  4.1 Overview of the Obtained Datasets

Following the data collection procedure described in Section 3.2, we obtained
DS1, DS2, and DS3. 48,915 issues were retrieved by the searches with candidate
TD labels, and finally DS1 includes 35,278 TD issues collected from 3,598
repositories after data cleaning steps. Noticeably, among the 35,278 TD
issues, there are 914 issues whose LabeledEvents cannot be retrieved from
GitHub, but their TD labels can be retrieved. Considering the relatively large
number of such issues, we kept them in the final dataset (DS1) of TD issues.
This may influence the results related to the LabeledEvents of TD issues.
Specifically, when analyzing TD issues related to the LabeledEvents, we used
the left 34,364 TD issues excluding these 914 TD issues. DS2 contains 652
issues randomly extracted from DS1 according to the sampling requirements
stated in Section 3.2.3. DS3 contains 5,202 resolved TD issues and 205,927
resolved non-TD issues of the 24 repositories with at least 100 resolved TD
issues. The list of these repositories is available online (Li et al., 2022b).
All the three datasets (i.e., DS1, DS2, and DS3) are also available online Li
et al. (2022a).

###  4.2 RQ1: Popularity of adopting TDM

We studied the popularity of adopting TDM in OSS development practice in the
following three aspects.

####  4.2.1 Trend of the number of repositories newly adopting TDM

Figure 2 shows that the number of repositories newly adopting TDM had been
lastingly increasing over the years from 2 in 2009 to 880 in 2020, and the
CAGR is 73.9%. In the same period, the number of newly created repositories
had also been continuously growing from 80,318 in 2009 to 23,807,596 in 2020,
and its CAGR is 67.8%, which is little lower than the CAGR of the number of
repositories newly adopting TDM.

Figure 2: Distribution of the number of repositories newly adopting the TDM
practice over years (RQ1).

####  4.2.2 Trend of the number of newly reported TD issues

Figure 3 shows that the number of newly reported TD issues had been
continuously increasing over the years from 3 in 2009 to 9,840 in 2020, and
the CAGR is 108.7%. Meanwhile, the number of newly reported issues had been
continuously growing from 92,552 in 2009 to 58,335,696 in 2020, and its CGAR
is 79.7%, which is much lower than the CGAR of the number of newly reported TD
issues. In addition, the CGAR of the proportion of newly reported TD issues
over the total newly reported issues was also calculated, and the CGAR is
16.2%. However, the proportion of newly reported non-TD issues over the total
newly reported issues decreases with a CGAR close to 0.0% (i.e., -0.00124%)
during the 12 years.

Figure 3: Distribution of the number of newly reported TD issues over years
(RQ1).

####  4.2.3 Distribution of the ratio of repositories over the number of TD
issues

Among the total 3,598 repositories with TD labels, there are 1,084 (30.1%),
563 (15.6%), and 2,937 (81.6%) repositories that each has only one, two, and
less than ten TD issues, respectively. Figure 4 depicts the distribution of
the ratio of repositories over the number of TD issues. This distribution
obeys a power law (Barabási and Albert, 1999), i.e., $y=bx^{-a}$, where $y$
denotes the ratio of repositories, $x$ denotes the number of TD issues, $a$
and $b$ are constants. The fitted results are that $a=1.275$ and $b=0.170$,
i.e., $y=0.170x^{-1.275}$ (see Figure 4).

Figure 4: Distribution of ratio of repositories with TD labels over the number
of TD issues (RQ1).

Summary (RQ1): The number of repositories adopting TDM and the number of newly
reported TD issues had been continuously and rapidly increasing, and their
CAGRs are 73.9% and 108.7%, respectively. The distribution of the ratio of
repositories over the number of TD issues obeys a power law.

###  4.3 RQ2: Understanding on the TD Concept

####  4.3.1 QAs affected by TD

During our manual analysis, we found that the existing widely-used TD
classification proposed in (Li et al., 2015) cannot fully cover all the TD
issues. Therefore, except for the ten TD types proposed in (Li et al., 2015),
we added “Deployment TD” as a new TD type. Deployment TD refers to flaws that
slow down software deployment or make software deployment unnecessarily
complicated. For example, from issue
#151111https://github.com/konveyor/pelorus/issues/151 of project
konveyor/pelorus, we can see that some warnings have affected the deployment
of Kubernetes, and thus this issue contains deployment TD.

Table 5 shows the results of our manual analysis. For each QA, we calculated
the number and percentage of which TD types it was affected by. In addition,
for each column, we fill the cells in green according to the value of the
number, in order to show more clearly the relationship between affected QAs
and TD types. The darker the color of a cell, the larger the number in the
cell. As shown in this table, the top three QAs most affected by TD issues are
maintainability, reliability, and functional suitability, and the top three TD
types most likely contained in TD issues are design TD, architectural TD, and
test TD.

In the following, we present the results on how each QA is affected by TD. (1)
Maintainability is mainly affected by design TD, architectural TD, and test
TD, accounting for 30.9%, 27.3%, and 23.8%, respectively. (2) Reliability is
mainly affected by five types of TD, of which defect TD has the greatest
impact on this QA, accounting for 34.0%. (3) For functional suitability,
design TD is the most influential TD type, followed by architectural TD. (4)
For usability, design TD, architectural TD, and requirements TD are the top
three TD types that account for more than 10.0%. Among the three types of TD,
design TD with 58.8% ranks in the first place. (5) Performance efficiency is
most affected by design TD and architectural TD, accounting for 51.5% and
33.3%, respectively. (6) Portability is mainly affected by architectural TD,
accounting for more than a half. Portability requires a developer to consider
the relationship between the software system and the computing platforms
during design. Thus, architectural TD has a great impact on portability. (7)
Security is mainly affected by design TD, architectural TD, and test TD,
accounting for 23.1%, 46.2%, and 15.4%, respectively. (8) Compatibility is
mainly affected by infrastructure TD with 36.4%.

Table 5: How each QA is affected by TD (RQ2).

| Maintainability | Reliability | Functional suitability | Usability | Performance efficiency | Portability | Security | Compatibility | All QAs
---|---|---|---|---|---|---|---|---|---
Design TD | 122, 30.9% | 20, 21.3% | 21, 41.2% | 20, 58.8% | 17, 51.5% | 3, 14.3% | 3, 23.1% | 2, 18.2% | 208, 31.9%
Architectural TD | 108, 27.3% | 16, 17.0% | 13, 25.5% | 4, 11.8% | 11, 33.3% | 12, 57.1% | 6, 46.2% | 2, 18.2% | 172, 26.4%
Test TD | 94, 23.8% | 12, 12.8% | 3, 5.9% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 2, 15.4% | 1, 9.1% | 112, 17.2%
Defect TD | 0, 0.0% | 32, 34.0% | 5, 9.8% | 1, 2.9% | 0, 0.0% | 1, 4.8% | 1, 7.7% | 1, 9.1% | 41, 6.3%
Code TD | 20, 5.1% | 13, 13.8% | 3, 5.9% | 2, 5.9% | 1, 3.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 39, 6.0%
Infrastructure TD | 23, 5.8% | 0, 0.0% | 1, 2.0% | 0, 0.0% | 1, 3.0% | 4, 19.0% | 1, 7.7% | 4, 36.4% | 34, 5.2%
Documentation TD | 15, 3.8% | 0, 0.0% | 1, 2.0% | 3, 8.8% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 19, 2.9%
Build TD | 9, 2.3% | 1, 1.1% | 0, 0.0% | 0, 0.0% | 3, 9.1% | 0, 0.0% | 0, 0.0% | 1, 9.1% | 14, 2.1%
Requirements TD | 0, 0.0% | 0, 0.0% | 4, 7.8% | 4, 11.8% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 8, 1.2%
Versioning TD | 3, 0.8% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 1, 4.8% | 0, 0.0% | 0, 0.0% | 4, 0.6%
Deployment TD | 1, 0.3% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 0, 0.0% | 1, 0.2%
Total | 395, 100.0% | 94, 100.0% | 51, 100.0% | 34, 100.0% | 33, 100.0% | 21, 100.0% | 13, 100.0% | 11, 100.0% | 652, 100.0%

####  4.3.2 Co-occurring labels with TD labels

Table 6 lists the TD labels used in the repositories on GitHub and the number
of their occurrences. Fifteen TD labels are used, technical debt was used most
frequently by 11,787 issues, and doc debt was used most scarcely by only 12
issues. There are 3,974 co-occurring labels and most of them occurred just for
a few times. After filtering, the final co-occurring labels and their numbers
of the occurrences are shown in Table 7. As we can see, the most frequent co-
occurring label is enhancement/improvement, followed by bug. Co-occurring
labels may indicate specific characteristics of TD issues. For instance, some
co-occurring labels are associated with the purposes (e.g., enhancement) and
expectations (e.g., nicetohave) of the TD issues. Some co-occurring labels
(e.g., testing) indicate the TD types contained by the issues, while some
others (e.g., security) indicate the QAs affected by TD.

Table 6: TD labels and their occurrences (RQ2).

TD label | # | TD label | #
---|---|---|---
technical debt | 11,787 | technicaldebt | 137
tech debt | 9,341 | code debt | 129
debt | 4,294 | ux/ui debt | 112
tech-debt | 3,535 | design debt | 37
techdebt | 2,855 | refactoring / tech debt | 37
technical-debt | 2,542 | td | 14
test debt | 241 | doc debt | 12
kind/tech-debt | 205 |  |

Table 7: Labels co-occurring with TD labels (RQ2).

Label | # | Label | #
---|---|---|---
enhancement, improvement | 2,152 | task | 178
bug | 1,499 | epic | 174
help wanted | 860 | code | 173
testing, tests, test | 831 | should | 162
good first issue | 827 | backlog | 158
refactoring, refactor | 368 | security | 157
feature, feature request | 348 | performance | 157
documentation | 341 | area/integration tests | 148
stale, status/stale | 318 | server | 145
api | 251 | devops | 142
nicetohave | 233 | area/unit tests | 130
must | 192 | infrastructure | 120
question | 191 | debug | 108
wontfix | 190 |  |

Summary (RQ2): Maintainability, reliability, and functional suitability are
the top three QAs affected most by TD issues. Labels enhancement/improvement
and bug are more likely associated with TD labels.

###  4.4 RQ3: Identification of TD issues

We looked into the identification of TD issues from the following three
perspectives: when TD issues are identified, who identifies TD issues, and
phase of adopting TDM during the development lifecycle. As mentioned in
Section 4.1, when we needed to analyze the labeled events of the TD issues in
DS1, we excluded the 914 TD issues whose labeled events of TD labels were
missing. Therefore, we analyzed the left 34,364 TD issues to answer RQ3 from
the first two perspectives, and analyzed the full DS1 to answer RQ3 from the
third perspective.

####  4.4.1 When TD issues are identified

In dataset DS1, there are 34,480 TD labeled events that record the assignment
of TD labels to 34,364 TD issues. Among the 34,364 TD issues, 16,967 (49.4%)
were tagged with TD labels when they were reported, and the other 17,397
(50.6%) were tagged with TD labels during the issue resolution process. For
the 17,397 TD issues, the maximum duration between the issue reporting time
and the TD tagging time is 2,784 days, the minimum is 1 day, and the average
is 63 days. In addition, the 70th percentile is 15 days, and the median is 1
day.

####  4.4.2 Who identifies TD issues

In dataset DS1, all TD issues were identified by 4,526 participants, 1,408
(31.1%) of which identified only one TD issue, and 23 (0.5%) of which
identified more than 100 TD issues. On average, each participant identified
7.6 TD issues. In addition, there were 2,438 (69.0%) repositories in which all
TD issues were identified by only one participant. In more than 3,306 (91.9%)
of all repositories had no more than 4 participants who identified TD issues.

Among the 34,364 TD issues, 27,690 (80.5%) were identified as TD issues by
their reporters, and 6,674 (19.5%) were identified as TD by other participants
other than the issue reporters. Through analyzing the 27,690 TD issues, we
found that the proportion of CONTRIBUTOR and MEMBER among the reporters of the
TD issues is relatively high. Table 8 shows the number and proportion of each
relationship. Among them, the numbers of FIRST_TIMER, FIRST_TIME_CONTRIBUTOR,
and MANNEQUIN are 0, and thus these three relationships do not appear in Table
8.

Table 8: The number and proportion of each relationship between issue
reporters and repositories of TD issues (RQ3).

Relationship | #Reporter | %Reporter
---|---|---
COLLABORATOR | 3,946 | 14.3%
CONTRIBUTOR | 9,236 | 33.4%
MEMBER | 9,052 | 32.7%
OWNER | 4,934 | 17.8%
NONE | 522 | 1.9%
Total | 27,690 | 100.0%

####  4.4.3 Phase of adopting TDM during the development lifecycle

Figure 5: Distribution of percentage of repositories with TD labels over the
intervals of $Phase_{TD}$ (RQ3).

As shown in Figure 5, the number of repositories with TD labels tends to
decrease as $Phase_{TD}$ increases. Notably, 35.1% of the repositories have a
$Phase_{TD}$ value no more than 0.1, and more than half of the repositories
have a $Phase_{TD}$ value less than or equal to 0.3. This means that the
participants start investing effort to TDM in the early phase of the
development lifecycle.

Summary (RQ3): Around a half (49.4%) of TD issues were identified when they
were reported, and the others were identified during the resolution process.
31.1% of the participants identified only one TD issue, and 69.0% of the
repositories were with all TD issues identified by only one participant. More
than a half of the repositories adopt TDM in the early phase of the
development lifecycle.

###  4.5 RQ4: Resolution of TD Issues

####  4.5.1 Proportion of resolved TD issues

The distribution of the number of repositories over the proportion of resolved
TD issues is shown in Figure 6. The proportion of resolved TD issues is
divided into multiple intervals, e.g., [0.0, 0.0], (0.0, 0.1], and [1.0, 1.0].
The number and proportion of repositories in each interval are shown in Figure
6. All TD issues were resolved in 1,137 out of 3,598 (31.6%) repositories
(corresponding to interval [1.0, 1.0]), and each repository contains 4.5 TD
issues on average. None of the TD issues was resolved in 982 out of 3,598
(27.3%) repositories (corresponding to interval [0.0, 0.0]), and each
repository contains 2.1 TD issues on average. In each of the rest 1,479 out of
3,598 (41.1%) repositories, the proportion of resolved TD issues falls into
(0.0, 1.0), i.e., part of the TD issues were resolved.

The median of the proportion of resolved TD issues for all 3,598 repositories
in DS1 is 0.582. The total number of all resolved TD issues between 2009 and
2020 is 22,841, the total number of all reported TD issues is 35,278, and thus
the proportion of resolved TD issues is 64.7%. In contrast, the total number
of resolved issues and the total number of issues by the end of 2020 are
49,694,427 and 72,813,679, respectively, and hence the proportion of resolved
issues is 68.2%, which is slightly greater than the proportion of resolved TD
issues. Repository microsoft/vscode has 1,610 TD issues, and is the only
repository that has more than 1,000 TD issues. The proportion of resolved TD
issues in this repository is 92.7%, which is a very high proportion,
especially considering that some unresolved TD issues reported recently would
be resolved afterwards.

Figure 6: Distribution of the number of repositories over the proportion of
resolved TD issues (RQ4).

####  4.5.2 Open time of TD issues

Out of all the 22,841 resolved TD issues, 2,761 (12.1%), 1,012 (4.4%), and
7,601 (33.3%) were resolved in one, two, and less than ten days, respectively.
The median is 25 days. As shown in Figure 7, the distribution of the
proportion of resolved TD issues over open time (day) follows a power law
(Barabási and Albert, 1999), _i.e._ , $y=bx^{-a}$, where $y$ denotes the ratio
of resolved TD issues, $x$ denotes the open time of the resolved TD issue, $a$
and $b$ are constants. The fitting results are $a=1.228$ and $b=0.369$, _i.e._
, $y=0.369x^{-1.228}$ (see Figure 7).

Figure 7: Distribution of the proportion of resolved TD issues over open time
(RQ4). Figure 8: Distribution of the open time of resolved TD issues (RQ4).

In addition, in order to study the difference on the open time between the
resolved TD issues and non-TD issues, we calculated their distributions, which
show that: (1) There are discrete values of the distribution of open time for
the TD issues and non-TD issues, and the distributions show a right skewed
pattern. The maximum open time for the TD issues is 2,403 days, which is
extremely long. (2) The inter-quartile range shows that the open time of TD
issues is longer than that of non-TD issues. The average open time of resolved
TD issues is 84.5% longer than that of resolved non-TD issues, which is 155
days for the former and 84 days for the latter. (3) A vast majority of TD
issues are resolved in a long period of time compared to non-TD issues. Figure
8 provides a violin plot based on the distributions of resolved TD issues and
non-TD issues. The vertical coordinate of this plot is logarithmized (into
natural logarithm) in order to ensure that the readers’ attention can be
focused on the core data (i.e., the inter-quartile range).

####  4.5.3 Characteristics of resolved TD issues

The results of Mann-Whitney U tests on the characteristics of resolved TD
issues and non-TD issues are shown in Table 9, in which $Avg_{TD}$ and
$Avg_{non-TD}$ denote the average values of the characteristic in the
corresponding row of resolved TD issues and non-TD issues, respectively.

It can be found that for IssueSize and ParticipantNo, their p-values are less
than 0.05. Considering the average values of these two characteristics, it
means that issue characteristics IssueSize and ParticipantNo of resolved TD
issues are significantly smaller than those of resolved non-TD issues,
respectively. On the contrary, the p-values of CmntSize and CmntNo are larger
than 0.05, which means that there is no significant difference between
resolved TD issues and non-TD issues with respect to characteristics CmntSize
and CmntNo.

####  4.5.4 Reopen of TD issues

There are 24 repositories with at least 100 resolved TD issues. Out of the
total 5,202 resolved TD issues in the 24 repositories, 326 (6.3%) resolved TD
issues had been reopened before. In addition, for 205,927 resolved non-TD
issues, 9,677 (4.7%) resolved non-TD issues had been reopened before. From the
percentages mentioned above, it can be concluded that TD issues are more
likely to be fixed for multiple times.

Table 9: Results of comparison on issue characteristics (RQ4).

Feature | $Avg_{AR}$ | $Avg_{AR}$ | p-value
---|---|---|---
IssueSize | 723.22 | 914.66 | 0.000
CmntNo | 3.87 | 3.87 | 0.056
CmntSize | 991.99 | 1,037.50 | 0.492
ParticipantNo | 2.13 | 2.27 | 0.006

Summary (RQ4): All TD issues were resolved in 31.6% of the repositories while
no TD issues were resolved at all in 27.3% of the repositories. The
distribution of the proportion of resolved TD issues over open time obeys a
power law. The mean open time of resolved TD issues is 84.5% longer than that
of resolved non-TD issues. TD issues are more likely to be reopened than non-
TD issues.

###  4.6 RQ5: Continuity of TDM

####  4.6.1 Repositories with different TDM continuity levels

After calculation, the value of $Period_{Y}$ is around 368 and the value of
$Count_{Z}$ is 8. Following the definitions of the three continuity levels,
1,187 (32.9%) repositories with TD labels are identified as Abandoned, while
only 298 (8.2%) repositories with TD labels are identified as Consistent. The
number of repositories of level Abandoned is around 4 times as large as the
number of those of level Consistent. In addition, the remaining repositories
are identified as Unclear. Since too few TD issues are managed or the period
for TDM is too short, it is not clear whether these repositories are
consistently using or have abandoned TDM.

####  4.6.2 Characteristics of repositories with different TDM continuity
levels

We compared the characteristics of the repositories of level Consistent and
repositories of level Abandoned, and the specific results of Mann-Whitney U
tests on the 7 characteristics of the repositories of level Consistent or
Abandoned are shown in Table 10, in which $Avg_{CR}$ and $Avg_{AR}$ denote the
average values of the characteristic in the corresponding row of the
repositories of level Consistent and repositories of level Abandoned,
respectively. We found that the repositories of these two levels differ
significantly within the 95% confidence interval on 6 characteristics:
RepoIssueNo, RepoResolvedIssuePro, RepoTDRepoterNo, RepoTDIssueNo,
RepoOpentimeAvg, and RepoCmntSizeAvg. Specifically, the average of these 6
characteristics are higher for repositories of level Consistent than that for
repositories of level Abandoned.

Table 10: Results of comparison on repository characteristics (RQ5).

Characteristic | $Avg_{AR}$ | $Avg_{CR}$ | p-value
---|---|---|---
RepoIssueNo | 242.206 | 1765.034 | <0.001
RepoResolvedIssuePro | 0.734 | 0.836 | <0.001
RepoTDRepoterNo | 1.832 | 8.028 | <0.001
RepoTDIssueNo | 7.234 | 53.652 | <0.001
RepoResolvedTDIssuePro | 0.603 | 0.634 | 0.872
RepoOpentimeAvg | 113.941 | 165.543 | <0.001
RepoCmntSizeAvg | 346.598 | 521.687 | <0.001

Summary (RQ5): Surprisingly, only 8.2% (i.e., 298) of the repositories with TD
labels adopt TDM consistently, while 32.9% (i.e., 1,187) of the repositories
with TD labels have abandoned TDM. The repositories with consistent TDM have
significantly more issues, reporters of TD issues, TD issues, and open time as
well as larger size of comments and a higher proportion of resolved issues
than the repositories with abandoned TDM.

##  5 Discussion

###  5.1 Interpretation of Study Results

RQ1: First, the lasting and fast increase of the number of repositories with
TD labels and TD issues indicates that the TD concept has been increasingly
popular over the last decade. In other words, the OSS community on GitHub has
been raising the awareness of TDM in software development. Second, the power-
law characteristic of the distribution of ratio of repositories with TD labels
over the number of TD issues indicates that only a small portion of
repositories have a large number of TD issues and a large portion of
repositories have only a few TD issues.

RQ2: The results of our manual analysis on sampled TD instances indicate that
1) both internal QAs (e.g., maintainability) and external QAs (e.g.,
usability) are affected by TD, 2) internal QAs (maintainability in particular)
are affected the most by TD. In addition, co-occurring labels with TD labels
are extremely diverse and can carry different kinds of messages. They may help
to identify the affected QAs, TD types, purposes, and so forth.

RQ3: First, around a half of all the TD issues were identified when they were
reported, which indicates that the reporters of such issues may have good
awareness of TDM and that such issues might be relatively easy to be
identified as TD. Second, in 91.9% of the repositories, all TD issues were
identified only by no more than four participants, which means that only a
really small number of participants put TDM into practice in most repositories
adopting TDM. Third, the fact that 80.5% of the TD issues were identified by
issue reporters indicates that issue reporters tend to pay more attention to
the characteristics of issues. More than a half of the repositories with TD
labels adopt TDM in the early phase of the development lifecycle, which
indicates an early awareness of TDM in most repositories with TD labels.

RQ4: First, TD issues did not receive enough attention considering that the
median of the proportion of resolved TD issues for all repositories is 0.582
and no TD issues were resolved in more than a fourth of all repositories.
Second, the power-law characteristic of the distribution of the proportion of
resolved TD issues over open time indicates that most resolved TD issues
experienced short open time and a small portion of resolved TD issues had long
open time. Third, TD issues take much (i.e., 84.5%) longer to be resolved than
non-TD issues. One potential reason is that most TD issues are not bugs and
invisible to users, and thus they are usually assigned with a relatively low
priority. Fourth, on average, a non-TD issue attracts more participants to
join discussion than a TD issue. In other words, a non-TD issue receives more
attention than a TD issue. Finally, on average, a TD issue is more likely to
be reopened than a non-TD issue, which means that TD issues are more difficult
to be cleanly fixed than non-TD issues.

RQ5: Over the 3,598 repositories with TD labels, around a third (32.9%) of the
repositories have abandoned TDM, which is partially due to the situation that
only one participant is involved in TDM for 69.0% of all the repositories. If
this only participant left the repository, the practice of TDM there is likely
to be dropped. This is further confirmed by that the number of TD issue
reporters for the repositories keeping consistent TDM is significantly larger
than that for the repositories having abandoned TDM.

###  5.2 Implications

Based on the study results and their analysis, we obtained a number of
implications for practitioners and researchers, respectively.

####  5.2.1 Implications for practitioners

(1) The OSS community are embracing the TD concept, which in turn reflects the
practical value of TDM. The TD concept is well adopted in some large
repositories, such as microsoft/vscode, in the sense that hundreds and even
more of TD issues were lastingly managed in such repositories.

(2) Although the TD concept has been adopted in some OSS repositories, TDM is
possible to be ignored in the sense that no TD issues were resolved in 27.3%
of the repositories, and TDM can even be abandoned in the sense that 32.9% of
the repositories are identified as having abandoned TDM. In the context of OSS
development, to benefit from TDM, we recommend to pre-define TD labels for a
project (or repository) and to explicitly make clear and concrete TDM policies
that the participants should obey. In OSS development, if there are no TDM
policies exerted on a project, the awareness of TDM is likely to be brought
away from the project by the participants who are the only ones care about TDM
when they quit the project.

####  5.2.2 Implications for researchers

(1) As the TD practice has been increasingly popular in the OSS community,
which resonates with the intensive research on TD in the last decade.

(2) The reasons for why only one TD issue was identified in 31.1% of all the
repositories and only one participant identified all TD issues in 69.0% of all
the repositories, need to be further investigated. This can help us to
understand why the TD concept is not adopted by other participants in such
repositories, to evaluate the value of the TD concept in such repositories,
and to explore the conditions required by well adoption of the TDM practices
in such repositories.

(3) There is in need for a deep investigation on the cost/benefit of adopting
TDM in OSS development, in order to clarify the reasons why around a third
(32.9%) of the repositories have abandoned TDM and only 8.2% of the
repositories keep TDM as a consistent practice.

(4) We highlight the gap between the academic research and OSS development
practice on TDM. TDM has been a hot research topic in software engineering for
more than a decade (Li et al., 2015; Ciolkowski et al., 2021), a large number
of research papers on TDM have been published. Although the awareness of TDM
has been increasing in the sense that the growth of the number of repositories
newly adopting TDM is higher than that of newly created repositories, there
are only 298 repositories keep TDM as a consistent practice on the whole
GitHub. This demonstrates a big gap between academic research on TDM and
practical use of TDM in OSS development.

##  6 Threats to Validity

There are several threats to the validity of the study results. We discuss
these threats according to the guidelines in (Runeson and Höst, 2009).
Internal validity is not discussed, since we did not study causal
relationships.

###  6.1 Construct validity

Construct validity is concerned with whether the values of the variables
(listed in Table 1) we obtained are in line with the real values that we
expected. First, considering that many projects hosted on GitHub do not record
their issues on GitHub, our collected dataset is not complete. Second, it is
possible that the keywords used to search TD issues do not cover all labels
for tagging TD issues. However, to reduce this risk, we chose the string
"technical debt" as well as its variable forms (e.g., word abbreviations), and
selected the appropriate TD types according to the study of Li _et al_. (Li et
al., 2015). Admittedly, issues with the list of TD labels may not cover all
issues containing TD. However, in order to investigate the practice of TDM
performed by practitioners in the OSS development, we do not want to judge by
ourselves whether an issue contains TD or not, but let the practitioner speak
out. Therefore, we chose the labels that are directly related to TD as TD
labels. Third, the 914 TD issues without labeled events may influence the
completeness of the data. Dataset DS1 excluding 914 TD issues without labeled
events was analyzed to partially answer RQ3. Considering that the 914 TD
issues did not come from a small number of repositories and were not reported
in a narrow time slot, the threat caused should be limited. Finally, the
results of our manual analysis of the affected QAs and TD types of TD issues
may be biased due to the differences of experience and knowledge between the
researchers. This threat was greatly mitigated by the pilot tagging process,
discussion of the disagreement of individual tagging results with a third
researcher.

###  6.2 External validity

External validity is concerned with the generalizability of the study results.
We only investigated issues reported on repositories hosted on GitHub,
therefore, the findings obtained in this paper are merely valid for GitHub,
and may not be generalized to other OSS platforms or ecosystems. However,
GitHub is the largest OSS platform, the repositories included in this study
come from different application domains and adopt a wide range programming
languages, and thus our datasets used in this study should be sufficiently
representative. Hence, the threats to external validity should be alleviated.

###  6.3 Reliability

Reliability is concerned with whether the study yields the same results when
it is replicated by other researchers. In all the data analysis tasks, only
descriptive statistics were used, and all datasets are provided on GitHub.
Hence, the threats to reliability should be minimal.

##  7 Conclusions and Future Work

In this work, we conducted an empirical study on all issues with TD labels on
the whole GitHub to investigate the execution of TDM in OSS development. The
following findings were obtained: (1) the awareness of TDM in the OSS
community has been arising; (2) maintainability, reliability, and functional
suitability are the top three QAs affected most by TD issues, and design TD,
architectural TD, and test TD are the top three TD types occurred most
frequently in TD issues; (3) only one TD issue was identified in 31% of the
repositories, all TD issues were identified by only one developer in 69% of
the repositories, 81% of TD issues were identified by the issue reporters, 49%
of TD issues were identified when the issues were reported and the other 51%
of TD issues were identified during the resolution process of the issues; (4)
TDM was ignored in 27.3% of the repositories after TD issues were identified;
and (5) only 8.2% of the repositories keep TDM as a consistent practice, while
32.9% of the repositories have abandoned TDM.

Based on the findings in this work, we plan (1) to conduct a broad survey on
TDM in the closed source software industry to understand the state of the
adoption and execution of TDM, and (2) to investigate the reasons for ignoring
and abandoning TDM in the repositories that have adopted TDM.

## Acknowledgment

This work is supported by the Natural Science Foundation of Hubei Province of
China under Grant No. 2021CFB577, and the National Natural Science Foundation
of China under Grant No. 62176099 and No. 62172311.

## References

  * Alves et al. (2016) Alves, N.S., Mendes, T.S., de Mendonça, M.G., Spínola, R.O., Shull, F., Seaman, C., 2016\.  Identification and management of technical debt: A systematic mapping study.  Information and Software Technology 70, 100–121.
  * Avgeriou et al. (2016) Avgeriou, P., Kruchten, P., Ozkaya, I., Seaman, C., 2016\.  Managing technical debt in software engineering (dagstuhl seminar 16162).  Dagstuhl Reports 6, 110–138.
  * Barabási and Albert (1999) Barabási, A.L., Albert, R., 1999\.  Emergence of scaling in random networks.  Science 286, 509–512.
  * Basili (1992) Basili, V.R., 1992.  Software modeling and measurement: The goal/question/metric paradigm.  URL: http://drum.lib.umd.edu/bitstream/1903/7538/1/Goal_Question_Metric.pdf.
  * Bavota and Russo (2016) Bavota, G., Russo, B., 2016\.  A large-scale empirical study on self-admitted technical debt, in: Proceedings of the 13th International Conference on Mining Software Repositories (MSR’16), ACM. pp. 315–326.
  * Becker et al. (2018) Becker, C., Chitchyan, R., Betz, S., McCord, C., 2018\.  Trade-off decisions across time in technical debt management: A systematic literature review, in: Proceedings of the 2nd International Conference on Technical Debt (TechDebt’18), IEEE. pp. 85–94.
  * Bellomo et al. (2016) Bellomo, S., Nord, R.L., Ozkaya, I., Popeck, M., 2016\.  Got technical debt? surfacing elusive technical debt in issue trackers, in: Proceedings of the 13th International Conference on Mining Software Repositories (MSR’16), IEEE. pp. 327–338.
  * Ciolkowski et al. (2021) Ciolkowski, M., Lenarduzzi, V., Martini, A., 2021.  10 years of technical debt research and practice: Past, present, and future.  IEEE Software 38, 24–29.
  * Codabux et al. (2021) Codabux, Z., Vidoni, M., Fard, F.H., 2021.  Technical debt in the peer-review documentation of R packages: a ropensci case study, in: Proceedings of the 18th IEEE/ACM International Conference on Mining Software Repositories (MSR’21), IEEE. pp. 195–206.
  * Cohen (1960) Cohen, J., 1960.  A coefficient of agreement for nominal scales.  Educational and Psychological Measurement 20, 37–46.
  * Cunningham (1992) Cunningham, W., 1992.  The WyCash portfolio management system.  ACM SIGPLAN OOPS Messenger 4, 29–30.
  * Dai and Kruchten (2017) Dai, K., Kruchten, P., 2017\.  Detecting technical debt through issue trackers, in: Proceedings of the 5th International Workshop on Quantitative Approaches to Software Quality (QuASoQ’17), IEEE. pp. 59–65.
  * Digkas et al. (2020) Digkas, G., Ampatzoglou, A., Chatzigeorgiou, A., Avgeriou, P., 2020\.  On the temporality of introducing code technical debt, in: Proceedings of the 13th International Conference on the Quality of Information and Communications Technology (QUATIC’20), Springer. pp. 68–82.
  * Digkas et al. (2017) Digkas, G., Lungu, M., Chatzigeorgiou, A., Avgeriou, P., 2017\.  The evolution of technical debt in the Apache ecosystem, in: Proceedings of the 11th European Conference on Software Architecture (ECSA’17), Springer. pp. 51–66.
  * Dunn (1961) Dunn, O.J., 1961.  Multiple comparisons among means.  Journal of the American Statistical Association 56, 52–64.
  * Ernst et al. (2021) Ernst, N., Kazman, R., Delange, J., 2021.  Technical Debt in Practice: How to Find It and Fix It.  First ed., MIT Press, USA.
  * Field (2013) Field, A., 2013.  Discovering Statistics using IBM SPSS Statistics.  Fourth ed., Sage Publications Ltd., Singapore.
  * Huang et al. (2018) Huang, Q., Shihab, E., Xia, X., Lo, D., Li, S., 2018.  Identifying self-admitted technical debt in open source projects using text mining.  Empirical Software Engineering 23, 418–451.
  * ISO/IEC (2011) ISO/IEC, 2011.  ISO/IEC 25010: 2011 systems and software engineering — systems and software quality requirements and evaluation (SQuaRE) — system and software quality models.
  * Israel (1992) Israel, G.D., 1992.  Determining sample size.  Fact Sheet PEOD-6 .
  * Kruchten et al. (2019) Kruchten, P., Nord, R., Ozkaya, I., 2019.  Managing Technical Debt: Reducing Friction in Software Development.  First ed., Addison-Wesley Professional, USA.
  * Lenarduzzi et al. (2021) Lenarduzzi, V., Besker, T., Taibi, D., Martini, A., Fontana, F.A., 2021.  A systematic literature review on technical debt prioritization: Strategies, processes, factors, and tools.  Journal of Systems and Software 171, 110827.
  * Lenarduzzi et al. (2019) Lenarduzzi, V., Saarimaki, N., Taibi, D., 2019.  On the diffuseness of code technical debt in java projects of the Apache ecosystem, in: Proceedings of the 2nd IEEE/ACM International Conference on Technical Debt (TechDebt’19), IEEE. pp. 98–107.
  * Li et al. (2020a) Li, Y., Soliman, M., Avgeriou, P., 2020a.  Identification and remediation of self-admitted technical debt in issue trackers, in: Proceedings the 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA’20), IEEE. pp. 495–503.
  * Li et al. (2015) Li, Z., Avgeriou, P., Liang, P., 2015.  A systematic mapping study on technical debt and its management.  Journal of Systems and Software 101, 193–220.
  * Li et al. (2022a) Li, Z., Peng, Y., Liang, P., Ampatzoglou, A., Mo, R., Liu, H., Qi, X., 2022a.  Datasets for "technical debt management in oss projects: An empirical study on github".  URL: https://github.com/ypyixiuxiu/TDMdatasets.
  * Li et al. (2022b) Li, Z., Peng, Y., Liang, P., Ampatzoglou, A., Mo, R., Liu, H., Qi, X., 2022b.  Repositories with not less than 100 resolved TD issues.  URL: https://github.com/ypyixiuxiu/TDMdatasets/blob/main/DS3/DS3repo.csv.
  * Li et al. (2020b) Li, Z., Yu, Q., Liang, P., Mo, R., Yang, C., 2020b.  Interest of defect technical debt: An exploratory study on Apache projects, in: Proceedings of 36th IEEE International Conference on Software Maintenance and Evolution (ICSME’20), IEEE. pp. 629–639.
  * Potdar and Shihab (2014) Potdar, A., Shihab, E., 2014\.  An exploratory study on self-admitted technical debt, in: Proceedings the 30th IEEE International Conference on Software Maintenance and Evolution (ICSME’14), IEEE. pp. 91–100.
  * Ren et al. (2019) Ren, X., Xing, Z., Xia, X., Lo, D., Wang, X., Grundy, J., 2019.  Neural network-based detection of self-admitted technical debt: From performance to explainability.  ACM Transactions on Software Engineering and Methodology 28, 1–45.
  * Runeson and Höst (2009) Runeson, P., Höst, M., 2009\.  Guidelines for conducting and reporting case study research in software engineering.  Empirical Software Engineering 14, 131–164.
  * Sierra et al. (2019) Sierra, G., Shihab, E., Kamei, Y., 2019.  A survey of self-admitted technical debt.  Journal of Systems and Software 152, 70–82.
  * Tan et al. (2021) Tan, J., Feitosa, D., Avgeriou, P., Lungu, M., 2021\.  Evolution of technical debt remediation in python: A case study on the Apache software ecosystem.  Journal of Software: Evolution and Process 33, e2319.
  * imDone team (2022) imDone team, 2022.  imdone.  URL: https://github.com/imdone/imdone-atom.
  * de Toledo et al. (2021) de Toledo, S.S., Martini, A., Sjøberg, D.I., 2021.  Identifying architectural technical debt, principal, and interest in microservices: A multiple-case study.  Journal of Systems and Software 177, 110968.
  * Tom et al. (2013) Tom, E., Aurum, A., Vidgen, R., 2013.  An exploration of technical debt.  Journal of Systems and Software 86, 1498–1516.
  * Tsoukalas et al. (2020) Tsoukalas, D., Kehagias, D., Siavvas, M., Chatzigeorgiou, A., 2020\.  Technical debt forecasting: An empirical study on open-source repositories.  Journal of Systems and Software 170, 110777.
  * Tufano et al. (2017) Tufano, M., Palomba, F., Bavota, G., Oliveto, R., Penta, M.D., De Lucia, A., Poshyvanyk, D., 2017.  When and why your code starts to smell bad (and whether the smells go away).  IEEE Transactions on Software Engineering 43, 1063–1088.
  * Vidoni et al. (2022) Vidoni, M., Codabux, Z., Fard, F.H., 2022.  Infinite technical debt?  Journal of Systems and Software 190, 111336.
  * Xavier et al. (2020) Xavier, L., Ferreira, F., Brito, R., Valente, M.T., 2020\.  Beyond the code: Mining self-admitted technical debt in issue tracker systems, in: Proceedings of the 17th International Conference on Mining Software Repositories (MSR’20), ACM. pp. 137–146.
  * Xiao et al. (2021) Xiao, L., Cai, Y., Kazman, R., Mo, R., Feng, Q., 2021\.  Detecting the locations and predicting the costs of compound architectural debts.  IEEE Transactions on Software Engineering , 1–30.
  * Yu et al. (2022) Yu, Z., Fahid, F.M., Tu, H., Menzies, T., 2022.  Identifying self-admitted technical debts with jitterbug: A two-step approach.  IEEE Transactions on Software Engineering 48, 1676–1691.
  * Zampetti et al. (2021) Zampetti, F., Fucci, G., Serebrenik, A., Di Penta, M., 2021\.  Self-admitted technical debt practices: a comparison between industry and open-source.  Empirical Software Engineering 26, 1–32.