11institutetext:  Center for Informatics Research in Science & Scholarship
School of Information Sciences
University of Illinois at Urbana-Champaign, USA
11email<EMAIL_ADDRESS>
#  : Modeling and Visualizing OpenRefine Histories as YesWorkflow Diagrams

Nikolaus Nova Parulian 11 Lan Li 11 Bertram Ludäscher 11

###### Abstract

OpenRefine is a popular open-source data cleaning tool. It allows users to
export a previously executed data cleaning workflow in a JSON format for
possible reuse on other datasets. We have developed , a novel tool that maps a
JSON-formatted OpenRefine operation history to a YesWorkflow (YW) model, which
then can be visualized and queried using the YW tool. The latter was
originally developed to allow researchers a simple way to annotate their
program scripts in order to reveal the workflow steps and dataflow
dependencies implicit in those scripts. With the user can automatically
generate YW models from OpenRefine operation histories, thus providing a
“workflow view” on a previously executed sequence of data cleaning operations.

The tool can generate different types of YesWorkflow models, e.g., a _linear
model_ which mirrors the sequential execution order of operations in
OpenRefine, and a _parallel model_ which reveals independent workflow
branches, based on a simple analysis of dependencies between steps: if two
operations are independent of each other (e.g., when the columns they read and
write do not overlap) then these can be viewed as parallel steps in the data
cleaning workflow. The resulting YW models can be understood as a form of
_prospective provenance_ , i.e., knowledge artifacts that can be queried and
visualized (i) to help authors document their own data cleaning workflows,
thereby increasing transparency, and (ii) to help other users, who might want
to reuse such workflows, to understand them better.

###### Keywords:

Data Cleaning OpenRefine Provenance Workflows

##  1 Introduction: OpenRefine Histories as Provenance

OpenRefine is an open-source data cleaning tool that allows users to execute
data transformations in a browser-based, spreadsheet-like GUI [10]. OpenRefine
also records the data cleaning operations used in a project in an _operation
history_ (Fig. 1). This history can be understood as a form of _provenance_
because it reveals what operations were executed on the dataset.

Provenance information has many uses and applications [2], e.g., to increase
transparency (and thus also improve the reproducibility) of workflows, to aid
quality estimation, and more generally to contribute to the trustworthiness of
workflow products. We can distinguish different types of provenance
information:

  * •

Prospective provenance describes the general _workflow_ (or “recipe”) that has
been (or is being) used for cleaning the data, without reference to details
that may vary with each workflow run.

  * •

Retrospective provenance, in contrast, describes specific details of workflow
runs that _have been_ executed and that are _not_ common across all runs.

Figure 1: OpenRefine operation history (left); checklist (center); and JSON
objects for export and reuse (right). After clicking Extract near the upper
left, a user can select the operations to be exported with a checklist. The
selected operations appear in the JSON panel on the right. Unlike other
operations, single cell edits are not user-selectable and thus not included in
the JSON-formatted history to be exported: while other operations apply to all
cells in a column uniformly (and hence are reusable), a single cell edit is by
design a “one-off” update that is not reusable in another data cleaning
recipe.

The separation between both forms of provenance is not necessarily strict.
While OpenRefine histories can be understood as carrying workflow information
(i.e., _prospective_ provenance), there are also elements of _retrospective_
provenance buried in various OpenRefine artifacts, e.g., in history and
internal project files. Using both kinds of provenance together in a _hybrid_
provenance model increases the expressive power and utility of both forms of
provenance [5, 11, 12, 13].

As part of its operation history, OpenRefine records names of data
transformations, the linear sequence in which operations were executed, and a
description of each execution step. This information can also be understood
quite naturally as prospective provenance. On the other hand, the number of
rows or cells affected by an operation constitutes retrospective provenance.

Fig. 1 depicts pieces of provenance information from the operation history
checklist. The operation described in the highlighted red box explains the
first step from the transformation list: it executed a column-split operation
on column date, resulting in 16,959 cell changes. Additional provenance
captured by OpenRefine includes: which columns were used to produce which
other columns, and the specific function parameters and arguments111Similar
details are captured by noWorkflow [7] for Python function calls. used during
runtime. The latter can be viewed as both prospective and retrospective
provenance: In Fig. 1 on the right, the column-rename operation was applied on
column date 2. The operation identified by the blue arrow shows more details
of parameters of this execution including the oldColumnName and newColumnName
of this transformation.

The extracted OpenRefine history is a powerful feature that allows users to
reuse some or all steps of a data cleaning workflow on a different dataset as
long as the new dataset has the same (or at least a compatible) data schema.
For single cell edits, however, i.e., “one-off” operations that have been
applied to a specific cell only (rather than the whole column), OpenRefine
does not capture provenance in the same way as for reusable operations (green
boxes in Fig. 1). This export limitation highlights the intended use of the
extracted operation history, i.e., as generic “recipes” that can be applied
uniformly across rows or columns. A single cell operation only fixes one cell
value in the current dataset and thus is not part of the exported JSON recipe.

### From Data Cleaning Histories to Workflows

In the approach, we first analyze OpenRefine recipes extracted from histories.
We can distinguish different kinds of transformations, i.e., _intra-column_
and _inter-column_ operations. The former read and write a single column $C$,
independent of other columns; the latter may read or write multiple columns
[9]. For example, a to-lowercase transformation applied to column $A$ only
depends on the values in $A$ and is thus independent of other intra-column
operations, e.g., to-uppercase or trim-whitespace performed on a different
column $B$. On the other hand, operations such as split-column or merge-
columns affect multiple columns and are not independent of operations that
read or write those columns.

Figure 2: The tool in action: A user cleans dataset _D_ 1 with OpenRefine,
obtaining a clean(er) dataset _D_ 2. She then extracts from the operation
history a JSON recipe, which she feeds into . This generates a YW model that
can be queried and visualized (via Graphviz). The original user (a data
curator or researcher) and other users can make use of YW models and
OpenRefine recipes when developing new data cleaning workflows.

For a user, it is not easy to understand the effects of and dependencies
between operations when presented in their native JSON form. Tracing
provenance information such as column and operation dependencies quickly
becomes challenging, especially if the interactive data cleaning workflow is
long-running (hours or days) and requires many “user breaks”. Motivated by
these challenges, we try to address these research questions:

  * •

Can we extract the provenance information provided by an OpenRefine history
and generate a corresponding workflow graph?

  * •

Can we identify and expose independent subworkflows, e.g., by visualizing them
as parallel branches in the data cleaning workflow?

We answer both questions in the affirmative with our tool: the tool allows
users to automatically generate YesWorkflow models from OpenRefine histories.
These workflow models can then be processed by a YW tool, e.g., to query or
visualize the resulting data cleaning workflows [5, 6]. Thus, the tool (Fig.
2) helps uncover provenance information by visualizing the operations workflow
with its dataflow dependencies and effects on the data schema.

##  2 OpenRefine Transformations and the Prototype

The tool is open source and freely available from Github [4] and
PyPI222https://pypi.org/project/or2ywtool. There are currently three different
workflow options that dictate which type of model is being generated: _linear_
(sequential), _parallel_ , and _collapsed_.

### The Linear Model

Figure 3: Sequential (Linear) YesWorkflow model of the JSON-recipe depicted in
Fig. 1. The _combined view_ (left) depicts the YW model consisting of data
transformation _steps_ (green boxes), _data_ nodes (yellow), and _parameter_
nodes (white). By omitting certain details, the _process view_ (center) and
_data view_ (right) emphasize steps and data, respectively. For the sequential
model the data granularity is “table”, i.e., each snapshot
$\mathsf{table}_{n+1}$ depends on (1) the preceding snapshot
$\mathsf{table}_{n}$, (2) a transformation operation, and (3) the
transformation parameters.

The OpenRefine operation history is linear by nature, as it reflects the
user’s sequential execution of interactive data cleaning steps. In the linear
workflow model, we simply view this sequence of operations as a dataflow
pipeline in which transformation steps $S_{i}$ and “data snapshots” $D_{j}$
alternate. For example, the exported recipe from Fig. 1 gives rise to a linear
8-step workflow:

$(D_{0})\to[S_{1}]\to(D_{1})\to\cdots\to[S_{8}]\to(D_{8})$

The actual workflow output for the linear model is shown in Fig. 3, using a
small sample of the _menus dataset_ [8] that was crowd-sourced by volunteers
for the New York Public Library. The _combined view_ on the left in Fig. 3
depicts the alternation of data snapshots $\mathsf{table}_{i}$ with OpenRefine
steps such as column-split, column-rename, etc. The _process view_ (in the
center) suppresses data elements, thus highlighting the eight data
transformation steps instead. Finally, the _data view_ on the right of the
figure highlights the simple linear sequence of table snapshots. In this
variant of a data view, edges are labeled with the operations applied to the
data tables.

Note that this simple linear model is coarse-grained in the sense that data
elements of workflows in this model represent different states (“snapshots”)
of the data table, as it undergoes a sequence of data transformations. As a
consequence, this model does not take into account that different steps may in
fact be independent from each other when considering a finer-grained data
model, e.g., based on columns rather than the whole table.

Figure 4: Parallel YesWorkflow model of the JSON-recipe depicted in Fig. 1.
The _combined view_ (top) depicts a detailed YW model. The process view (lower
left) emphasizes workflow steps, while the data view (lower right) emphasizes
data derivations. In all three YW diagrams, the independent, parallel nature
of execution paths and data derivations is visible. For the parallel model the
data granularity is “column”, thus allowing a finer-grained dependency
analysis relative to the sequential model in Fig. 3 (which uses “table” data
elements instead).

### The Parallel Model

In order to achieve a finer-grained dependency resolution between steps, the
parallel model considers individual columns $C_{i}$ as the basic data
elements. In this way, when analyzing a recipe, the tool can analyze
operations at the column level (rather than at the table level) and thereby
reveal data independence in a data cleaning workflow through parallel
branches. To this end, employs a simple rule: if the data transformation
functions $f_{i}$ and $f_{i+1}$ of two consecutive steps $S_{i}$ and $S_{i+1}$
are independent of each other, i.e., if the output columns of $S_{i}$ do not
overlap with the input columns of $S_{i+1}$ and vice versa, then the
associated functions commute, i.e.,

$f_{i}\circ f_{i+1}=f_{i+1}\circ f_{i}~{}.$

When detects such a situation, it creates two new branches in the parallel
workflow model, one for $S_{i}$ and one for $S_{i+1}$.

Fig. 4 depicts the parallel YW model generated by for the example from Fig. 1:
the combined view (top of the figure) contains alternating sequences of the
basic data elements (here: columns) and the transformation steps operating on
those columns. Note that this view contains three independent
subworkflows.333In this particular example, the layout algorithm of Graphviz
seems to “entangle” the three independent subworkflows – but a closer
inspection reveals that there are indeed three independent subworkflows. The
largest of these exhibits a _split-and-merge_ pattern in Fig. 4 consisting of
three branches. This pattern can be seen even more clearly in the reduced
views, i.e., in the process view (lower left) and in the data view (lower
right).

In addition to the “process story” which is evident from the process view (and
the combined view), the data view tells the “data story” quite nicely:

There are three separate and independent data derivations, originating with
the dish_count, event, and date columns, respectively. The data derivation
subgraph starting from date has three parallel branches. Inspection of the
combined view (or the process view) reveals that a split-column operation is
generating those parallel branches, one each for day, month, and year. These
three columns are then merged into a single new column repaired_date which
represents the final output of the date subworkflow.

Internally, the tool employs different data structures and node types to
represent the various workflow patterns. These include _parallel nodes_ to
represent independent (disconnected) subworkflows, _split nodes_ to represent
the creation of new columns from an existing one, and _merge nodes_ to
represent the combination of dataflow branches into a shared branch.

### The Collapsed Model

This is an option that can be used to collapse large groups of similar
operations. Experience in a limited data cleaning study has demonstrated that
for some use cases involving clustering and mass-edit operations, the
operation histories can contain very long sequences of variants of essentially
the same data cleaning operation. In the resulting workflow model, these long
sequences (sometimes several hundred steps long) of highly repetitive steps
can be more succinctly represented by collapsing highly similar operations
into a new node type, representing a summary of the collapsed sequence of
steps. When abstracting or collapsing a subworkflow into a single node, a
separate file is produced, representing the structure of the subworkflow, so
that the detailed process information can be checked if needed.

##  3 Conclusions

We have presented , a novel open-source tool that can automatically generate
YesWorkflow annotations (and thus YW models) from OpenRefine data cleaning
recipes, extracted from operation histories. When used in conjunction with
YesWorkflow and Graphviz these workflow graphs can be queried and/or
visualized. Here we have focused on the generation of two different kinds of
YW models: a coarse-grained _linear model_ , and a more fine-grained, column-
oriented _parallel model_. The various workflow views generated by in
conjunction with YesWorkflow allow users to document their data cleaning
workflows in a more transparent manner, thereby also facilitating reuse of
data cleaning recipes.

Numerous lines of further research and development exist. For example, in the
current work we have mainly focused on generating useful workflow and
prospective provenance visualizations for OpenRefine recipes. In addition, we
can expand what information we extract from OpenRefine, e.g., retrospective
provenance contained in OpenRefine project files [3]. This would allow us to
populate a hybrid provenance model, combining both prospective and
retrospective provenance information. The resulting integrated model allows
users to answer more complex and powerful provenance queries over their
OpenRefine histories and update logs.

Another line of research is to investigate the use of alternative dataflow and
data cleaning models, e.g., the formal models developed in [1] and [9].

## References

  * [1] Delpeuch, A.: A Complete Language for Faceted Dataflow Programs. Electronic Proceedings in Theoretical Computer Science 323, 1–14 (Sep 2020). https://doi.org/10.4204/eptcs.323.1
  * [2] Herschel, M., Diestelkämper, R., Lahmar, H.B.: A survey on provenance: What for? What form? What from? The VLDB Journal 26(6), 881–906 (2017)
  * [3] Li, L., Ludäscher, B., Zhang, Q.: Towards more transparent, reproducible, and reusable data cleaning with OpenRefine. iConference 2019 Proceedings (poster paper) (2019), hdl.handle.net/2142/103330
  * [4] Li, L., Parulian, N., Ludäscher, B.: or2yw: Generating YesWorkflow models from OpenRefine histories. github.com/idaks/OR2YWTool (2021)
  * [5] McPhillips, T., Bowers, S., Belhajjame, K., Ludäscher, B.: Retrospective Provenance without a Runtime Provenance Recorder. In: Theory and Practice of Provenance (TaPP) (2015), dl.acm.org/doi/abs/10.5555/2814579.2814580
  * [6] McPhillips, T., Song, T., Kolisnik, T., Aulenbach, S., Belhajjame, K., Bocinsky, R.K., Cao, Y., Cheney, J., Chirigati, F., Dey, S., Freire, J., Jones, C., Hanken, J., Kintigh, K.W., Kohler, T.A., Koop, D., Macklin, J.A., Missier, P., Schildhauer, M., Schwalm, C., Wei, Y., Bieda, M., Ludäscher, B.: YesWorkflow: A User-Oriented, Language-Independent Tool for Recovering Workflow Information from Scripts. International Journal of Digital Curation 10(1), 298–313 (May 2015). https://doi.org/10.2218/ijdc.v10i1.370
  * [7] Murta, L., Braganholo, V., Chirigati, F., Koop, D., Freire, J.: noWorkflow: capturing and analyzing provenance of scripts. In: International Provenance and Annotation Workshop. pp. 71–83. Springer (2014)
  * [8] New York Public Library: What’s on the menu? menus.nypl.org/data (2020-10-01)
  * [9] Núñez-Corrales, S., Li, L., Ludäscher, B.: A First-Principles Algebraic Approach to Data Transformations in Data Cleaning: Understanding Provenance from the Ground Up. In: 12th Intl. Workshop pn Theory and Practice of Provenance (TaPP) (2020), www.usenix.org/biblio-4498
  * [10] OpenRefine: A free, open source, power tool for working with messy data. github.com/OpenRefine, 2021
  * [11] Pimentel, J.F., Dey, S.C., McPhillips, T.M., Belhajjame, K., Koop, D., Murta, L., Braganholo, V., Ludäscher, B.: Yin & Yang: Demonstrating Complementary Provenance from noWorkflow & YesWorkflow. In: IPAW. Lecture Notes in Computer Science, vol. 9672, pp. 161–165. Springer (2016), doi.org/10.1007/978-3-319-40593-3_13
  * [12] Zhang, Q., Cao, Y., Wang, Q., Vu, D., Thavasimani, P., McPhillips, T., Missier, P., Slaughter, P., Jones, C., Jones, M., Ludäscher, B.: Revealing the Detailed Lineage of Script Outputs Using Hybrid Provenance. Intl. Journal of Digital Curation (IJDC) 12(2), 390–408 (2017), doi.org/10.2218/ijdc.v12i2.585
  * [13] Zhang, Q., Morris, P.J., McPhillips, T., Hanken, J., Lowery, D., Ludäscher, B., Macklin, J., Morris, R., Wieczorek, J.: Using YesWorkflow hybrid queries to reveal data lineage from data curation activities. Biodiversity Information Science and Standards 1, e20380 (2017). https://doi.org/10.3897/tdwgproceedings.1.20380


#  Gamma: Revisiting Template-based Automated Program Repair via Mask
Prediction

Anonymous Author(s)  Quanjun Zhang
Bowen Yu  State Key Laboratory for Novel
Software Technology
Nanjing University, China
<EMAIL_ADDRESS>State Key Laboratory for Novel
Software Technology
Nanjing University, China
<EMAIL_ADDRESS>Chunrong Fang1
Weisong Sun 1Chunrong Fang and Zhenyu Chen are the corresponding authors.
State Key Laboratory for Novel
Software Technology
Nanjing University, China
<EMAIL_ADDRESS>State Key Laboratory for Novel
Software Technology
Nanjing University, China
<EMAIL_ADDRESS>Tongke Zhang
Zhenyu Chen1  State Key Laboratory for Novel
Software Technology
Nanjing University, China
<EMAIL_ADDRESS>State Key Laboratory for Novel
Software Technology
Nanjing University, China
<EMAIL_ADDRESS>
###### Abstract

Automated program repair (APR) aims to fix software bugs without human
intervention and plays a crucial role in software development and maintenance.
Template-based APR has been widely investigated and shown promising results.
However, it is challenging for template-based APR to select the appropriate
donor code, which is an important repair ingredient for generating candidate
patches. Inappropriate donor code may cause plausible but incorrect patch
generation even with correct fix patterns, limiting the repair performance.

In this paper, we aim to revisit template-based APR, and propose Gamma, to
directly leverage large pre-trained language models for donor code generation.
Our main insight is that instead of retrieving donor code in the local buggy
file, we can directly predict the correct code tokens based on the context
code snippets and repair patterns by a cloze task. Specifically, (1) Gamma
revises a variety of fix templates from state-of-the-art template-based APR
techniques (i.e., TBar) and transforms them into mask patterns. (2) Gamma
adopts a pre-trained language model to predict the correct code for masked
code as a fill-in-the-blank task. Although our idea is general and can be
built on various existing pre-trained language models, we have implemented
Gamma as a practical APR tool based on the recent UniXcoder model. The
experimental results demonstrate that Gamma correctly repairs 82 bugs on
Defects4J-v1.2, which achieves 20.59% (14 bugs) and 26.15% (17 bugs)
improvement over the previous state-of-the-art template-based approach TBar
and learning-based one Recoder. Furthermore, Gamma repairs 45 bugs and 22 bugs
from the additional Defects4J-v2.0 and QuixBugs, indicating the
generalizability of Gamma in addressing the dataset overfitting issue. We also
prove that adopting other pre-trained language models can provide substantial
advancement, e.g., CodeBERT-based and ChatGPT-based Gamma is able to fix 80
and 67 bugs on Defects4J-v1.2, indicating the scalability of Gamma. Overall,
our study highlights the promising future of adopting pre-trained models to
generate correct patches on top of fix patterns in practice.

###### Index Terms:

Automated Program Repair, Fix Pattern, Pre-trained Model, LLM4SE

##  I Introduction

The complexity and size of modern software systems are continuously enlarging,
leading to the soaring number of software bugs [1, 2]. Software bugs have
detrimental effects on software development, as they give users an annoying
experience, and sometimes can cause huge financial losses to developers [3]. A
considerable amount of time and budget is spent on identifying and fixing such
software bugs manually [4]. To facilitate the process of manual debugging,
automated program repair (APR), which aims to generate correct patches for
identified buggy code snippets automatically, is getting growing attention
from both academia and industry [5, 6], such as Meta [7], Google [8] and
Microsoft [9, 10].

In the literature, a variety of APR techniques have been proposed to generate
patches, such as heuristic-based [11, 12], constraint-based [13, 14],
template-based [15, 16]. Among these traditional APR techniques, template-
based APR, which employs repair patterns hand-crafted by human experts to
transform buggy code snippets into correct ones, has been widely investigated
and recognized as state-of-the-art [17, 18, 19]. Candidate patches are usually
generated by leveraging two kinds of repair ingredients (i.e., fix patterns
and donor code) that are found in existing code bases. The repair pattern
represents common code change actions (e.g., insertion of an If statement) and
donor code represents code fragments (e.g., identifier tokens such as method
names) to concretize patches guided by abstract patterns. A mass of studies
has been dedicated to template extraction schemes, such as manually extracted
templates and automatically mining templates [20, 21, 22, 23]. For example,
state-of-the-art template-based APR tool TBar [17] focuses on the local buggy
file and leverages the context of buggy code to prune away irrelevant donor
code. Previous works [24, 25] have shown a considerable number of bugs cannot
be fixed because the relevant donor code is not available in the local file.
Therefore, TBar may fail to generate correct patches with inappropriate donor
code although the fix pattern matches with correct code change actions.

In this paper, we propose a novel template-based APR tool called Gamma by
combining the advances of fix patterns and pre-trained language models. The
key insight is that considering pre-trained models can acquire adequate
general knowledge about programming language from all possible open-source
projects in the wild, we can directly employ such models to retrieve relevant
donor code from the fix pattern and surrounding code context. In particular,
we first collect and summarize a super-set of fix patterns drawn from previous
template-based work (e.g., TBar). We then transform these fix templates into
hole-filling-based patterns, which replace donor code with several masked
tokens to be filled. Finally, we perform a mask prediction task on the hole-
filling-based fix patterns with the help of pre-trained models in a fill-in-
the-blank manner, i.e., predicting the correct donor code for the masked
tokens. Although Gamma is conceptually generalizable to various pre-trained
models, we have implemented Gamma on top of one recent pre-trained language
model, UniXcoder [26]. Unlike current template-based APR tools which usually
retrieve fix ingredients in the local buggy file, Gamma directly utilizes
generic knowledge pre-trained with millions of code snippets from open-source
projects, allowing it to provide a variety of donor code to fix different
bugs.

We conduct extensive experiments to compare Gamma with state-of-the-art APR
approaches (including both traditional and learning-based ones) on the widely-
adopted Defects4J-v1.2 benchmark. The experimental results demonstrate that
Gamma is able to outperform all existing APR approaches, improving the number
of correctly-fixed bugs to 82 with a precision of 81.19%, and 14 unique bugs
that no prior work can fix, which is a new frontier in the APR field. Besides,
Gamma fixes 45 and 22 bugs on the additional Defects4J-v2.0 and QuixBugs, 27
and 5 more than state-of-the-art learning-based technique Recoder,
demonstrating that Gamma can address the important dataset overfitting issue
well. Moreover, we implement Gamma with CodeBERT [27] and ChatGPT [28], and
find 80 and 67 bugs are fixed correctly on Defects4J-v1.2. The results
demonstrate that Gamma with other pre-trained models can further provide
substantial advancement, highlighting the generalizability of Gamma.

To sum up, the contributions of this paper are as follows:

  * •

New Dimension. We bridge the gap between the advances in recent pre-trained
models and template-based APR. Different from existing template-based APR
retrieving donor code from local buggy files and existing learning-based APR
generating a patched code snippet from scratch, our work demonstrates that we
can leverage pre-trained models to generate correct code tokens in a given fix
pattern. More importantly, our work reveals the potential for leveraging pre-
trained models to resolve the important fix ingredient problem in template-
based APR.

  * •

Novel APR tool. We propose Gamma, which leverages the large pre-trained
language model to generate correct code with the help of fix patterns without
any additional historical bug-fixing pairs for training. We define a set of
fix patterns in a fill-in-the-blank format and leverage the original pre-
training objective of pre-trained models to predict actual masked-out tokens.
Considering the fill-in-the-blank task can leverage various pre-trained
language models, Gamma is general in concept and can be implemented with
different pre-trained models in practice.

  * •

Extensive study. We conduct an empirical study to investigate the
effectiveness of Gamma compared to state-of-the-art traditional and learning-
based APR techniques. The results on the widely-adopted Defects4J-v1.2 show
that Gamma is able to fix 82 bugs and 14 of them cannot be fixed by existing
APR tools, creating a new higher baseline of repair performance. More
importantly, Gamma fixes 45 and 22 bugs on the newly-developed Defects4J-v2.0
and QuixBugs, demonstrating that Gamma can avoid the important dataset
overfitting issue of existing APR techniques. Moreover, we adopt different
pre-trained models (e.g., ChatGPT) to further investigate the generalization
ability of Gamma.

  * •

Available artifacts. To support the open science community, we release the
relevant materials (including source code, experimental results, and correct
patches) in our experiment for replication and future research [29].

##  II Background and Motivation

###  II-A Automated Program Repair

As a promising technique to shift the heavy manual debugging to efficient
automated patch generation, APR has developed rapidly and received much
attention from a broad of research communities, such as software engineering,
software security, and artificial intelligence [6, 30]. The workflow of APR
usually involves three phases: (1) fault localization, i.e., the off-the-shelf
fault localization techniques are utilized to identify a ranked list of
suspicious code elements, with whose help APR can focus on a small code
region, thus reducing the workload [31]; (2) patch generation, i.e., candidate
patches are generated by applying a set of transformation rules to the
suspicious code snippets [32]; and (3) patch validation, i.e., the available
test suites are utilized as the program specifications to check the
correctness of candidate patches [33]. The candidate patches that pass all
available test suites are considered plausible ones. The plausible patches
that are semantically equivalent to the developer patch by manual inspection
are considered correct ones; otherwise overfitting ones [34, 35].

In the literature, as the core component of APR research, a mass of research
efforts are devoted to generating patches from different aspects, including
traditional and learning-based ones. In particular, traditional APR techniques
can be classified as heuristic-based [11, 12], constraint-based [13, 14],
template-based [15, 16]. Among them, template-based APR is proven to achieve
the best performance, which consists of two fix ingredients, i.e., fix
patterns and donor code. Fix patterns are hand-crafted by human experts to
denote the common code changes, and donor code is retrieved in buggy files to
denote the actual correct code tokens. Gamma aims to revise the important
donor code by employing pre-trained language models in a fill-in-the-blank
manner.

Compared with traditional APR techniques, learning-based techniques handle the
program repair problem as a neural machine translation (NMT) task, which
translates a code sequence from a source language (i.e., buggy code snippets)
into a target language (i.e., correct code snippets). Existing NMT repair
models are typically built on the top of the encoder-decoder architecture
[36]. The encoder extracts the hidden status of buggy code snippets with the
necessary context, and the decoder takes the encoder’s hidden status and
generates the correct code snippets [37, 22, 38]. Thanks to the powerful
ability of DL to learn hidden and intricate relationships from massive code
corpora, learning-based APR techniques have achieved remarkable performance in
the last couple of years. Although learning-based APR techniques have
demonstrated their promising future, they are still limited by the quality and
quantity of historical bug-fixing pairs for training [18]. We view Gamma as a
novel learning-based APR technique that attempts to boost traditional APR
techniques by utilizing deep learning technology. However, different from most
existing learning-based APR that treats patch generation as an end-to-end NMT
task with a limited number of bug-fixing pairs as training data, Gamma
integrates pre-trained language models into template-based APR and only
predicts masked code tokens with a zero-shot learning scenario.

###  II-B Pre-trained Model

Recently, pre-trained language models (e.g., UniXcoder [26] and ChatGPT [28])
have significantly boosted performance across a wide range of code-related
tasks [39, 40]. These models are pre-trained by self-supervised training on
large-scale unlabeled corpora and then fine-tuned by supervised training on
limited corpora to enhance performance on multiple downstream tasks. During
the pre-training process, a masked language modeling objective is usually
employed to derive generic language representations from the massive unlabeled
training data [41], i.e., a small percentage of tokens are replaced by masked
tokens, and the training objective is to predict the original values of the
masked tokens.

Existing pre-trained models usually adopt the encoder-decoder architectures,
where the former encodes an input sequence as a fixed-length vector
representation, and the latter generates an output sequence based on the input
representation. These models can be generally categorized into three
architectures: encoder-only, decoder-only, and encoder-decoder ones [42].
Encoder-only models (e.g., CodeBERT [41]) usually pre-train a bidirectional
transformer in which each token can attend to each other. Decoder-only models
(e.g., GPT [43]) are pre-trained using unidirectional language modeling that
only allows tokens to attend to the previous tokens and themselves to predict
the next token. Encoder-decoder models (e.g., UniXcoder [26]) often make use
of denoising pre-training objectives that corrupt the source input and require
the decoder to recover them.

In this work, we select UniXcoder to retrieve the doner code via a mask
prediction task. UnixCoder is pre-trained using the MLM objective which can be
used to directly generate masked code tokens from the appropriate fix pattern
and surrounding code context. Besides, CodeBERT and ChatGPT are employed to
investigate the generalization ability of Gamma.

###  II-C Fix Template

Fix templates are widely employed in the APR community [17]. A fix template is
a pre-defined code transformation rule that represents a common code change in
the bug-fixing process. The insight behind fix templates is that many software
bugs are similar in nature [22]. Therefore, with fix templates summarized from
previous bugs, it is possible to automatically fix some other flawed code
[24].

In the literature, there are several strategies to access fix templates: (1)
manual template mining [44, 45], i.e., through performing analysis on existing
bugs as well as their relevant patches, experienced developers can identify
similar code changes, and turn them into fix templates. (2) machine learning
[20, 15], i.e., learning approaches are used so that fix templates can be
generated automatically. (3) static analysis [16, 46], i.e., fix templates are
generated from assorted types of warnings raised by static analysis tools.

The process of applying a fix template to patch generation typically involves
two steps. First, the APR tool selects an appropriate fix template based on
the abstract syntax tree (AST) representation of the buggy code. Fix templates
are chosen according to the types of nodes in AST, and bugs are fixed by
mutating the target nodes. Second, the APR tool generates a repaired version
of the buggy code by searching and applying the relevant donor code to the fix
pattern.

⬇

1@@ -786,7 +786,7 @@ void replace() {

2 } else {

3- int indexOfDot = namespace.indexOf(’.’);

4+ int indexOfDot = namespace.lastIndexOf(’.’);

5 if (indexOfDot == -1) {

6 compiler.getNodeForCodeInsertion(minimumModule)

Listing 1: The Defects4J bug Closure-92

###  II-D Motivation Example

To better illustrate the limitation of existing template-based APR, we further
present a motivating example in this section. As shown in Listing 1, we use a
real-world bug Closure-92 from the widely-used benchmark Defects4J-v1.2 as an
example. Closure-92 denotes the 92nd buggy version of the Google Closure
Compiler project in Defects4J-v1.2. This bug is fixed by Gamma successfully,
while TBar fails to generate a correct patch. To fix this bug, the method name
“indexOf” is replaced by “lastIndexOf”. The fix template used here is to
mutate method names. TBar applies the selected fix patterns to the source code
in a naive way. In this case, TBar searches all the methods that appear in the
local file where the bug is localized, and replaces the buggy method with all
the other methods with the same return type one by one. As a result, TBar is
not able to generate method names that do not exist in the original file, like
the “lastIndexOf” in this example, limiting its repair performance. Different
from TBar, we replace the method name “indexOf” with a mask token (i.e.
$<$mask$>$) instead, and query the pre-trained model UniXcoder to fill the
mask with the fix pattern and corresponding context.

Based on the example, we can observe that, although the correct fix pattern is
selected, as a state-of-the-art traditional APR, TBar still fails to generate
a correct patch with inappropriate donor code (i.e., “lastIndexOf”). The
effectiveness of template-based APR is largely dependent on donor code, which
refers to the code tokens (e.g., variable name) that can be combined with the
fix template in order to produce a complete patch. Donor code can be accessed
in different scopes of the buggy program (e.g. a method, a file, or a
package), but for some bugs, the correct donor code cannot be found even if
the whole program is searched. For example, previous work demonstrates that
half of the bugs from the Defects4J benchmark cannot be fixed because the
relevant donor code is not available in the search space [24]. Thus, these
bugs cannot be correctly fixed by template-based APR tools like TBar[17],
which only identifies the donor code within the local file. With a larger
search space (e.g., searching donor code from other projects), there might be
more chances to fix these bugs. However, such a strategy leads to the search
space explosion problem and an unaffordable search time, reducing repair
efficiency. In this paper, we utilize pre-trained language models to retrieve
relevant donor code. These models have learned programming language knowledge
from a great number of programs in the wild, making it possible to repair bugs
that require donor code from outside the buggy program.

##  III Approach

Figure 1: The overall workflow of Gamma

In order to assess the effectiveness of fix ingredients, we build Gamma, a
template-based APR tool that combines the recurrently-used fix patterns and
pre-trained language models. Fig. 1 presents the workflow of Gamma. Given a
buggy program and a set of test suites that make the program fail, a list of
suspicious code elements is returned by fault localization approaches (i.e.,
fault localization phase). On top of the existing fix pattern corpus, Gamma
then selects appropriate fix patterns to the suspicious elements (i.e.,
pattern selection phase) and queries pre-trained models to retrieve donor code
via a mask prediction mask (i.e., patch generation phase). Gamma finally
employs the available test suites as the oracle to check the generated patches
and returns plausible patches for manual inspection (i.e., patch validation
phase). Considering that fault localization is usually developed as an
independent field and existing APR techniques employ off-the-shelf fault
localization tools in the repair pipeline, we do not discuss the fault
localization below. We describe the role and operation of other phases as well
as all necessary implementation details.

###  III-A Mask Template Definition

In the literature, a variety of fix patterns are designed based on manual
summarization or automatic mining. On top of the state-of-the-art template-
based approach TBar [17], we manually inspect all fix patterns and transform
them into hole-filling-based patterns. We show the related templates as well
as how they are applied to buggy code, or how statements with mask tokens are
generated based on these templates.

T1: Check Cast Expressions. Adding an instanceof check around a statement when
it contains an unchecked cast expression.

⬇

\+  if(exp instanceof T){

var=(T) exp ...

\+  }

T2: Mutate Conditional Expressions: Mutating an expression that returns a
boolean value by removing part of the expression, replacing it with masks or
adding new masks.

⬇

Removing expression:

\-  condExp1 op condExp2

\+  condExp1

Updating expression:

\-  condExp1 op condexp2

\+  condExp1 op <mask>

Adding expression:

\-  condExp1

\+  condExp1 <mask>

where condExp denotes conditional expressions and Op denotes the logical
operator (i.e., $\|$ or $\&\&$).

T3: Mutate Data Types: Using one or more masks to replace data types in the
variable declaration or cast expression nodes.

⬇

\-  T var = ...

\+  <mask> var = ...

\-  ... (T) exp ...

\+  ... (<mask>) exp ...

where both T denotes a data type and exp denotes the being casted expression
(e.g., variable).

T4: Mutate Literal Expressions: Replacing literal expressions, including
number literals, string literals, boolean literals, etc. with masks.

⬇

\-  ...literal...

\+  ...<mask>...

T5: Mutate Method Invocations. Mutating method invocation expressions by
changing either method names or arguments.

⬇

Method name replacement:

\-  method(...)

\+  <mask>(...)

Argument insertion:

\-  method(arg)

\+  method(arg,<mask>)

Argument removal:

\-  method(arg1,arg2)

\+  method(arg)

Argument replacement:

\-  method(arg)

\+  method(<mask>)

T6: Check Null Pointer: Adding a null check to a statement that contains an
expression that is probably null.

⬇

Null point skip:

\+  if (exp != null){

...exp...

\+  }

Return insertion:

\+  if (exp == null){

\+  return <mask>;

\+  }

...exp...

Continue:

\+  if (exp == null){

\+  continue;

\+  }

...exp...

Exception throw:

\+  if (exp == null){

\+  throw new IllegalArgumentException();

\+  }

...exp...

Re-assignment:

\+  if (exp == null){

\+  exp=<mask>

\+  }

...exp...

T7: Mutate Operators: Replacing an operator in a statement with masks or
changing the priority of operations.

⬇

Changing the priority:

\-  (exp1 op1 exp2) op2 exp3

\+  exp1 op1 (exp2 op2 exp3)

Replacing operator:

\-  exp1 op exp2

\+  exp1 <mask> exp2

T8: Check Array Range: Checking the range of index before accessing an element
in an array.

⬇

\+  if (index<array.length) {

...array[index]...

\+  }

T9: Mutate Return Statements: Replacing the expression (e.g., literals,
variables, and conditional expressions) that is returned in a method with
masks.

⬇

\-  return exp;

\+  return <mask>;

T10: Insert Statements: Inserting return statements, try catch statements, if
statements, method invocations, or simply some masks to the existing
statement.

⬇

Return statement:

\+  return <mask>;

statement;

Try-catch statement:

\+  try{

statement;

\+  } catch(Exception e){}

If statement:

\+  if (<mask>) {

statement;

\+  }

Simple statement:

\+  <mask>;

statement;

T11: Remove Statements: Directly deleting one or more buggy statements from
the original code.

⬇

\-  statement;

T12: Replace Variables: Replacing a variable in a buggy statement with masks.

⬇

\-  ...var...

\+  ...<mask>...

T13: Move statements: Moving a statement from its original position to a new
one.

⬇

\-  statement;

...

\+  statement;

###  III-B Template Selection

On top of the defined mask fix templates, Gamma determines which template
should be applied to the input buggy statements. Similar to TBar [17], Gamma
employs an AST-based matching approach with a depth-first strategy. We first
generate an AST for the input, and then all the nodes in the AST are
traversed. If the AST contains a type of node that is required in a specific
template, we apply the template to the bug. We give an example of the
_Closure-10_ bug from the Defects4J-v1.2 in Listing 2 and how we generate its
patch in Fig. 1. In the AST of the input line, we find a _Method Invocation_
node in the AST of the input line, so we choose the template _T5: Mutate
Method Invocation_. For the template _Method name replacement_ that alters the
name of a method, after locating the method name _allResultsMatch_ , we
replace it with a mask token, which is going to be predicted in the next
phase. It is worth noting that Gamma is built on top of UniXCoder, which is
able to predict a sequence of code tokens based on a masked token. Thus, we do
not need to consider how many tokens should be masked during patch generation
and only use one mask token in the selected repair template, which is
different from other pre-trained models, such as CodeBERT used in AlphaRepair
[18] (discussed in Section V-C). There might be multiple repair templates that
are suitable for a piece of buggy code at the same time. In this case, we stop
the selection of repair templates as soon as the first correct patch is
generated.

⬇

1@@ -1414,7 +1414,7 @@ static boolean mayBeString(Node n) {

2static boolean mayBeString(Node n, boolean recurse) {

3 if (recurse) {

4- return allResultsMatch(n, MAY_BE_STRING_PREDICATE);

5+ return anyResultsMatch(n, MAY_BE_STRING_PREDICATE);

6 } else {

7 return mayBeStringHelper(n);

8 }

Listing 2: The Defects4J bug Closure-10

###  III-C Patch Generation with Mask Prediction

After selecting an appropriate fix template for a buggy code, we use UniXcoder
[26] to generate the correct code tokens via a fill-in-the-blank format. To
this end, we leverage the original pre-training training objective of masked
language modeling in UniXcoder. UniXcoder is an advanced pre-trained model for
programming languages that support code understanding and generation tasks. It
contains a pre-training objective of Masked Language Modeling (MLM), which is
designed to predict some tokens that have been masked out. We leverage this
pre-training task to complete the clozes generated in the previous step with
any fine-tuning, so that candidate patches for the buggy programs can be
produced.

The prediction for the mask is largely dependent on the tokens surrounding the
mask. If the correct token appears in the input for the model, it is more
likely that the token will be chosen as one of the possible results. The
precision of mask prediction is quite limited when only a single masked buggy
line is given without any context of the code, where there may be some useful
information for bug fixing. To get more context of the buggy line in each bug,
we extract the method that contains the line and use the whole method with the
masked buggy line as the input for UniXcoder. Considering that some tokens in
the buggy line have been replaced with masks but these tokens may also contain
essential information for mask prediction, in the first line of our input, we
add the original buggy line in the form of a comment (i.e., add a “//” in
front of the line). The commented buggy line followed by the method the bug is
in together forms the final input. For every input, N candidate patches are
generated by the UniXcoder model. N is the beam size and is an adjustable
parameter of Gamma. Relatively large beam size increases the possibility of
generating correct patches.

###  III-D Patch Validation

After a candidate patch for a given bug is generated by Gamma, we apply the
corresponding changes to the buggy program. Following the practice in the APR
community [47, 18], we first recompile the patched program and filter out any
patches that fail to compile. We then execute the patched program against the
available test suite to identify the plausible patches that successfully pass
all the test suites. For those plausible patches, we examine them manually to
ensure the programs are fixed correctly, i.e., whether the patches are
semantically equivalent to developer patches.

##  IV Experimental Setup

###  IV-A Research Questions

In this paper, we study the following research questions:

RQ1: What is the performance of Gamma compared to state-of-the-art APR
approaches?

RQ2: What is the generalizability of Gamma in repairing additional real-world
bugs?

RQ3: What is the scalability of Gamma when employing other advanced pre-
trained models?

###  IV-B Benchmarks

To evaluate the repair performance, we use the standard benchmark of
Defects4J-v1.2 [48] in the APR community. Defects4J-v1.2 is a collection of
real-world bugs from open-source projects and is widely adopted by existing
traditional [17, 49] and learning-based APR approaches [39, 38, 47, 50]. In
particular, Defects4J-v1.2 contains 395 known and reproducible bugs, each of
which contains a buggy version and a fixed version, as well as a corresponding
test suite that triggers that bug for patch validation. Evaluation on
Defects4J-v1.2 can reflect the performance of Gamma in a real-world debugging
scenario and provides sufficient comparison results against most of the
existing APR techniques.

Besides, we choose Defects4J-v2.0 and QuixBugs as other bug benchmarks for
evaluation, so as to investigate the generalizability of Gamma. Defects4J-v2.0
provides 420 additional real-world bugs from 17 Java projects, which is
adopted by some recent APR studies [50, 18]. QuixBugs [51] is a multi-lingual
parallel bug-fixing dataset in Python and Java used in [18, 39]. QuixBugs
contains 40 small classic algorithms with a bug on a single line, along with
the bug-triggering test suite.

###  IV-C Baselines

To enable sufficient evaluations, we compare Gamma against both traditional
and learning-based APR approaches. We choose seven recent learning-based APR
tools, i.e., AlphaRepair [18], Recoder [50], CURE [47], CoCoNuT [52], CIRCLE
[39], DLFix [38], and SequenceR [53]. We also choose two state-of-the-art
template-based APR tools TBar [17] and PraPR [54] as representatives of
traditional APR. In total, we evaluate Gamma against nine advanced APR tools
from different categories. Although the fault localization configuration is a
significant part of APR, we do not take it into consideration in our
experiment because of potential deviations that fault localization may bring
about. Following recent APR studies [39, 38, 47, 50], we apply perfect fault
localization in the way of inputting the exact buggy lines into different APR
techniques to standardize the impact of fault localization on repair
performance, discussed in Section VI.

###  IV-D Evaluation Metrics

We use two common metrics to evaluate the performance of all involved APR
approaches [5, 6], i.e., plausible patch and correct patch. The first one
fixes the buggy functionality without harming other correct functionality
(i.e., passing all available test suites), and the second one is semantically
or syntactically equivalent to the developer patch (i.e., generalizing the
potential test suite). We manually inspect each plausible patch to identify
whether it is a correct patch by following the standard practice in APR
research.

###  IV-E Implementation Details

At the stage of fix template selection, we apply Eclipse JDT to parse the
input line into AST, and then the AST is traversed to examine if it contains
any node that is required by a fix template. There are several templates that
can fit all the input buggy lines. For example, the template _T10: Insert
Statements_ only requires adding statements around the buggy line and does not
mutate any nodes in the AST. Such templates are directly applied to all the
inputs without checking the AST.

In the mask prediction phase, we choose the UniXcoder model “unixcoder-base”.
This is a model pre-trained on natural language-programming language (NL-PL)
pairs and is reported in the original UniXcoder paper [26]. We use the
encoder-decoder mode of the model to give predictions for each mask and
generate candidate patches. We set the beam size as 250 due to the limitation
of our device, which is smaller than 1000 used in CURE [47] and CoCoNuT [52].
Following previous learning-based APR approaches [50, 18], we set a 5-hour
running-time limit for fixing one bug to perform a fair comparison. w

All experiments are conducted on one Ubuntu 18.04.3 server with two Tesla
V100-SXM2 GPUs.

##  V Evaluation and Results

###  V-A Comparison with State-of-the-arts

TABLE I: Comparison with state-of-the-art APR techniques. Following the common practice in the APR community [39, 47, 52], we reuse the released results from the most recent work [18] instead of directly running the APR tools. Due to the APR community’s subsequent validation of publicly released correct patches, the results of some APR tools may be different from the reported results in their published papers. (†) PraPR is evaluated with the results of Ochiai fault localization [31].  Project | SequenceR | CoCoNuT | CURE | DLFix | Recoder | AlphaRepair | CIRCLE | PraPR† | Tbar | Gamma
---|---|---|---|---|---|---|---|---|---|---
Chart | 3 | 7 | 10 | 5 | 10 | 9 | 7 | 7 | 11 | 11
Closure | 3 | 9 | 14 | 11 | 21 | 23 | 17 | 12 | 16 | 24
Lang | 2 | 7 | 9 | 8 | 11 | 13 | 10 | 6 | 13 | 16
Math | 6 | 16 | 19 | 13 | 18 | 21 | 27 | 10 | 22 | 25
Mockito | 0 | 4 | 4 | 1 | 2 | 5 | 1 | 3 | 3 | 3
Time | 0 | 1 | 1 | 2 | 3 | 3 | 2 | 3 | 3 | 3
Total | 14/19 | 44/85 | 57/104 | 40/68 | 65/112 | 74/109 | 64/182 | 41/146 | 68/95 | 82/101

Experimental Design. In this section, we aim to evaluate the performance of
Gamma. We employ the 395 real-world bugs presented in the Defects4J-v1.2
dataset and compare Gamma with state-of-the-art APR techniques, including
traditional and learning-based ones. We report the performance of all compared
techniques under perfect fault localization (i.e., the ground-truth buggy
statement is known to the techniques).

Results. Table I presents the number of bugs that different APR techniques
successfully fix on the Defects4J-v1.2 dataset. Overall, we find that Gamma
substantially outperforms the compared APR techniques including both
traditional and learning-based APR techniques. Gamma is able to generate
correct patches for 82 real-world bugs, 20.59% (14 bugs), 26.15% (17 bugs) and
14.8% (8 bugs) more than TBar, Recoder and AlphaRepair. In particular, Gamma
fixes 11, 24, 16, 25, 3, and 3 bugs for Chart, Closure, Lang, Math, Mockito,
and Time projects, respectively, four of which are best-performing (bold in
Table I). More importantly, we find that Gamma achieves a correct rate of
81.19% (82/101) for plausible patches, 9.61% (68/95), 23.15% (65/112) and
13.30% (74/109) higher than TBar, Recoder and AlphaRepair, indicating that
Gamma is able to alleviate the long-standing patch overfitting problem in the
community of APR.

Overlap Analysis. To investigate what extent Gamma complements existing APR
techniques, we further calculate the number of overlapping bugs fixed by
different techniques. One best-performing traditional technique (i.e., TBar)
and three best-performing learning-based techniques (i.e., AlphaRepair, CURE,
and Recoder) are selected. As shown in Fig. 2, Gamma fixes 14 unique bugs that
other APR approaches fail to fix, which is 11, 3, 8, and 10 more than TBar,
AlphaRepair, CURE, and Recoder, respectively. More importantly, as a template-
based APR technique, there are 22 correctly-fixed bugs unique to Gamma
compared with TBar, highlighting the benefits of mask prediction performed by
UniXcoder. Overall, the results demonstrate that Gamma is complementary to
these best-performing APR techniques, to increase the number of correctly-
fixed bugs in the Defects4J-v1.2 benchmark.

Figure 2: The overlaps of the bugs fixed by different approaches

⬇

1@@ -409,7 +409,7 @@ public static double factorialLog(final int n) {

2public static int gcd(int u, int v) {

3- if (u * v == 0) {

4+ if ((u == 0) || (v == 0)) {

5 return (Math.abs(u) + Math.abs(v));

6 }

Listing 3: The Defects4J bug Math-94

Case Study. We have demonstrated the superior performance of Gamma over a
state-of-the-art template-based tool TBar, which is most related to our work.
To further investigate the effectiveness of Gamma, we provide some examples of
bugs that Gamma is able to fix but TBar fails to. Listing 3 presents the bug
Math-94 from the Defects4J-v1.2. Math-94 can be fixed with the fix template of
mutating conditional expressions. The conditional expression “u * v == 0”
within an if statement is replaced by “(u == 0) $||$ (v == 0)”. TBar deals
with this template in the way of replacing the suspicious expression with
other compatible ones collected from the same local file, while Gamma directly
replaces the expression with a mask token so that they can later be predicted
by the mask prediction task from the pre-trained model, making it possible to
generate new expressions that can correctly fix the bugs.

⬇

1@@ -742,7 +742,7 @@ static boolean isSimpleNumber(String s) {

2 return false;

3 }

4 }

5- return len > 0;

6+ return len > 0 && s.charAt(0) != ’0’;

7 }

8 static double getSimpleNumber(String s) {

Listing 4: The Defects4J bug Closure-52

Listing 4 presents a similar example of the bug Closure-52 from the
Defects4J-v1.2. Closure-52 denotes the 52nd buggy version of the Google
Closure Compiler project in Defects4J. To fix this bug, we need to insert a
new sub-conditional expression into the original expression. TBar fails to
generate a correct patch with improper operation and variables while Gamma is
able to directly predict the masked expression with corresponding code
context.

###  V-B Generalizability of Gamma

TABLE II: Comparison on additional datasets Project | AlphaRepair | Recoder | CURE | CoCoNuT | CIRCLE | TBar | Gamma
---|---|---|---|---|---|---|---
Defects4J 2.0 | 36 | 11 | - | - | - | 8 | 45
QuixBugs | 28 | 17 | 26 | 13 | 19 | - | 22

Experimental Design. We have demonstrated that Gamma achieves impressive
performance to repair real-world bugs from the widely-adopted Defects4J-v1.2
benchmark on top of fix patterns. Durieux et al. [55] demonstrate that there
exists a common benchmark overfitting phenomenon in APR evaluation, i.e., APR
tools usually perform significantly better on Defects4J-v1.2 than on other
benchmarks. In this section, according to prior work [18, 50], to evaluate the
generalizability of Gamma, we continue to conduct some extended experiments on
additional projects for further evaluation.

Results. Table II presents the comparison results of Gamma against baselines
on Defects4J-v2.0 and QuixBugs. In Defects4J-v2.0, following some recent work
[56, 18], we only focus on those bugs whose patches are confined to a single
location. Overall, Gamma generates 45 correct patches in the given 257 buggy
programs, outperforming both traditional and learning-based approaches. We
find that the performance achieved on the Defects4J-v2.0 dataset is commonly
less than that achieved on the Defects4J-v1.2 dataset. For example,
AlphaRepair fixes 18.73% (74/395) of bugs from Defects4J-v1.2 while only fixes
14.01% (36/257) of bugs from Defects4J-v2.0. Based on our analysis of the two
datasets, the possible reason is that Defects4J-v2.0 contains a harder set of
projects for APR with a different variety of fixes compared to Defects4J-v1.2.
Despite that, Gamma is able to generate 9, 34, and 37 more correct patches,
which is the highest number among all approaches. We also find that as a
template-based approach, TBar is able to generate a high amount of correct
patches (68) for Defects4J-v1.2, while it only generates a limited number of
correct patches (8) for Defects4J-v2.0. The possible reason may be that most
fix patterns are designed to target Defects4J-v1.2, which may not generalize
to other unseen projects, such as Defects4J-v2.0; Besides, learning-based
approaches also suffer from moving to a harder evaluation dataset since the
code transformation patterns are learned from training datasets which might
not be present in Defects4J-v2.0. In contrast, Gamma is able to address the
generalizability issue without training on specific bug datasets, which makes
it less prone to suffer from generalizability issues of traditional template-
based or learning-based tools.

Apart from Defects4J-v2.0, we also try to validate our approach on QuixBugs,
which extracts bugs from Quixey Challenge and translates them into both Java
and Python languages. Since our fix templates are designed for Java, we only
focus on Java programs in QuixBugs following previous work [17]. Table II
shows that among 40 bugs in QuixBugs, 22 are correctly fixed by Gamma,
highlighting the competitive performance of Gamma against state-of-the-art
approaches. It is worth noting that most of the templates are summarized from
Defects4J-v1.2, which may mean that some templates cannot be applied to any
bugs except those from Defects4J-v1.2. Thus, Gamma may be limited by the lack
of more efficient fix templates when coming to other new bug datasets. For
example, although various types of templates along with sub-templates are
defined, some of the templates cannot be used to fix at least one bug from
QuixBugs. As a result, we expect to explore more general fix templates in the
future to further improve the performance of template-based APR.

###  V-C Scalability of Gamma

Experimental Design. To further investigate whether the performance of Gamma
is affected by different pre-trained models, we apply two other advanced
models to perform the mask prediction task: CodeBERT and ChatGPT. CodeBERT
[27] is a pre-trained model for programming and natural languages, and mask
prediction is one of its pre-training tasks. ChatGPT is a state-of-the-art
language model that has shown impressive ability in conversations with human
beings. We also use Defects4J-v1.2 as a benchmark but replace UniXcoder with
these two models in the process of filling masks to find out to what extent
pre-trained models influence the effectiveness of template-based program
repair.

Similar to UniXcoder, CodeBERT can also generate predictions for a mask token
“$<$mask$>$” in the given code snippet. The difference between them is that
UniXcoder can predict several continuous tokens for a single mask while
CodeBERT can only give one token for a mask. However, there is usually more
than one token under a mask, so when using CodeBERT, we have to use different
numbers of successive masks to mask the initial code and then predict them
sequentially. We do not know the exact number of masks we should use (i.e.,
the number of tokens in the fixed code) as there could be a great many
possibilities in the patch. So while masking, we naively try all the mask
numbers from 1 to 20, which is a range suitable to most cases. In every
iteration, a mask is predicted and a joint score for each prediction is
calculated. Those predictions with the highest scores will be chosen to
replace the mask, and the next mask will be predicted according to the
previous predictions. We set the beam size as 250, the same as that used in
UniXcoder, so in each iteration, CodeBERT will give 250 most possible
predictions for the mask. Taking the process of fixing the Defects4J
Closure-10 bug as an example (shown in Listing 2), the patch of the bug
involves a change in the method name. To fix this bug, the method name
_allResultsMatch_ is replaced with masks, and then CodeBERT is asked to give
250 predictions for the first mask. Among the predictions, the token “any” has
a relatively high score, so it replaces the first mask and CodeBERT will
continue to predict the next mask until all masks are filled.

Different from UniXcoder, ChatGPT is fine-tuned from GPT-3.5 and close-
sourced. We can access ChatGPT with ChatGPT’s API of gpt-3.5-turbo-0301, which
is the latest version available. We interact with ChatGPT through natural
language conversations, i.e., sending requests to ChatGPT or receiving
responses from ChatGPT. To fill the masks with ChatGPT, we first give it some
prompts, instructing it to return back predictions for the mask. Following the
prompts we then add the masked buggy line along with its context to form the
complete query for ChatGPT. In our experiment, the input for ChatGPT starts
with a prompt “Next token prediction task, the first line is a comment to help
prediction, just return 250 possible predictions for $<$mask$>$ with highest
probability: ”, and then the bug context we give is the same as the input for
UniXcoder, which consists of a commented buggy line and the whole method where
the buggy line belongs. Besides, due to the benefits of the designed prompt,
we do not set the number of masked tokens in the buggy code, which is the same
as UniXcoder.

Figure 3: The overlaps of the bugs fixed by different pre-trained language
models

Results. Fig. 3 presents the repair results of different pre-trained language
models. Overall, the combination of the three models is able to fix 93 bugs
from Defects4J-v1.2, demonstrating these models can be used together by Gamma
to further increase the number of correct patches that can be generated. In
particular, we find when using CodeBERT to perform the mask prediction task,
80 bugs in total are fixed by Gamma correctly, only two less than the bugs
that Gamma with UniXcoder fixes. However, it takes much more time for CodeBERT
to generate correct patches, as the number of masks that should be used in
fixing a bug is unpredictable, and we have to run the mask prediction program
on the same bug and the same fix template for a lot of times, each with a
different mask number. In contrast, UniXcoder circularly predicts the next
token for a mask until an EOF token is generated, so only one mask is required
to fix the bug. We also find that Gamma with ChatGPT only fixes 67 bugs
correctly, performing not well as Gamma with UniXcoder and CodeBERT in mask
prediction. The possible reason lies in that UniXcoder is pre-trained with a
masked language modeling objective, where some training text is artificially
masked and the training objective is to predict the real text. However,
ChatGPT is designed for natural language conversations and it is unclear how
ChatGPT is pre-trained due to it being close-sourced. Thus, it is natural to
employ UniXcoder to recover masked code tokens for buggy code snippets in our
approach. Future researchers should further explore how to betters utilize
ChatGPT (e.g., designing other prompts) for mask prediction.

##  VI Threats to Validity

The first threat to validity comes from the manual inspection of correct
patches. To alleviate the influence of potential bias, following previous
works [39, 47], three authors manually verify all plausible patches (i.e.
patches that successfully pass the test) based on ground truth patches (i.e.,
developer patches). A plausible patch is considered to be correct if all three
authors identify it as equivalent to a ground truth patch semantically. To
facilitate the replication and verification of our experiments, we also
release the relevant materials (including all source code and correct patches)
publicly [29].

The second threat to validity is the fault localization setting. We evaluate
Gamma and baselines on perfect fault localization (i.e., ground-truth buggy
location is known) due to two reasons. First, off-the-shelf fault localization
approaches affect the performance of APR techniques significantly, introducing
a bias in comparison results [57]. The perfect fault localization mitigates
the influence of differences in different fault localization approaches on the
repair results and enables a fair assessment of repair performance
independently of the fault localization approach used. Second, the most recent
APR techniques [39, 58, 47, 52, 59] are only evaluated with perfect fault
localization, which makes our work also use perfect localization to ensure
direct comparison. However, this comparison setting may bring bias in repair
performance since the perfect fault localization results are usually
unavailable in real practice. Despite that, we believe that perfect fault
localization has little impact on our results, as perfect localization can
show the pure performance of different APR approaches. In the future, we
attempt to report the repair performance of Gamma and baselines with both the
automated fault localization (e.g., Ochiai [31]) and perfect fault
localization.

The third threat to validity comes from the potential of data leakage of pre-
trained models. In our experiment, we implement Gamma on top of UniXCoder and
evaluate Gamma on the widely-adopted benchmark Defects4J-v1.2. Considering
that UniXCoder is pre-trained with millions of code snippets, there may exist
some bugs in the evaluation benchmark of Defects4J-v1.2 that appear in the
pre-training dataset of UniXcoder. We perform a manual inspection to check
whether the fixed bugs by Gamma are leaked into the pre-training dataset. In
particular, we query the pre-training datasets including 2.3M functions paired
with comments and 4.1M unimodal code from CodeSearchNet. The manual inspection
is performed by two authors independently and confirmed by a third author. We
find there are three bugs leaked into the pre-training set, i.e., Closure-73,
Closure-126, and Time-19. For the three bugs, we manually perturb the buggy
code (e.g., changing variable names, adding dead code) and find GAMMA is still
able to generate the correct patches for all three bugs. We also find that if
we exclude the three overlapping bugs, GAMMA still outperforms state-of-the-
art APR techniques (79 vs. 68 for TBar, 79 vs. 74 for AlphaRepair). Thus, we
are confident that the data leakage is not a key point to our conclusion.

##  VII Related Work

###  VII-A Automated Program Repair

Existing APR techniques can be divided into four categories, i.e., heuristic-
based [11, 12], constraint-based [13, 14, 60], template-based [15, 16, 17] and
learning-based repair techniques [38, 50, 52]. Our work is related to
template-based and learning-based APR, discussed as follows.

Template-based APR, which generates patches with the help of fix patterns,
represents state-of-the-art among traditional APR techniques. TBar [17]
systematically collects and summarizes fix patterns from previous literature
and investigates the effectiveness of patch generation based on these
templates. Some other techniques explore fix patterns in various ways. For
example, PAR [44] manually extracts fix patterns from 60,000 human-written
patches. FixMiner [15] mines fix patterns with an iterative clustering
strategy. AVATAR [16] leverages fix patterns from static bug detection tools
to generate patches. Different from most existing template-based APR
techniques that focus on mining fix patterns, Gamma is the first work that
aims to address the donor code issue by integrating pre-trained models with a
fill-in-the-blank task.

With large available open-source code corpora, learning-based APR, which
applies machine learning to the bug-fixing objective, is getting growing
attention. For example, DLFix [38] uses a tree-based recurrent neural network
(RNN) model to learn from bug fixes and surrounding contexts in the form of an
abstract syntax tree. CoCoNuT [52] introduces a novel context-aware neural
machine translation (NMT) architecture that separately represents the buggy
code and context. CURE [47] attempts to break the limit of existing NMT-based
techniques by pre-training a programming language model on a large codebase,
introducing a new code-aware search strategy, and using subword tokenization
to narrow the search space. Recoder [50] is a syntax-guided edit decoder with
placeholder generation, which provides a novel provider/decider architecture
to guarantee that patches with correct syntax are generated. Different from
existing learning-based APR techniques generating patches from scratch with
bug-fixing data training, Gamma aims to directly predict correct code tokens
with the help of fix patterns in a zero-shot scenario.

Recently, there exists an increasing number of APR techniques on top of pre-
trained models. For example, Yuan et al. [39] propose CIRCLE, a T5-based
program repair framework equipped with continual learning ability across
multiple languages. Xia et al. [18] propose AlphaRepair, a cloze-style APR
approach based on CodeBERT without fine-tuning on historical bug-fixing data.
In our work, we include CIRCLE and AlphaRepair as baselines in the experiment.
Sobania et al. [61] investigate the performance of ChatGPT on the QuixBugs
benchmark. Mashhadi et al. [62] investigate the performance of fine-tuning
CodeBERT to fix software bugs from ManySStuBs4J. Jiang et al. [58] explore the
performance of pre-trained models with and without fine-tuning for the program
repair domain. Xia et al. [59] further present an extensive evaluation of
recent pre-trained models for fixing real-world projects and find state-of-
the-art pre-trained models are able to fix a considerable number of bugs. For
example, CodeX, the most effective one, fixes 99 bugs in Defects4J-v1.2 with a
total combination of three repair settings. We exclude CodeX as a baseline in
our experiment due to the uncertainty of training data in such black-box large
pre-trained models.

###  VII-B Pre-trained Language Models and Applications

In this section, we introduce some typical pre-trained language models and
then discuss the applications of pre-trained language models to some code-
related tasks, e.g., code search.

####  VII-B1 Pre-trained Language Models

Pre-trained language models have shown promising results on NLP tasks. BERT
[63] is a model to condition on left and right contexts in all layers so as to
pre-train deep bidirectional representations from unlabeled text. GPT-3 [43]
is an autoregressive language model having 175 billion parameters,
significantly outnumbering the parameters in previous language models. ChatGPT
[28] is the currently most popular language model fine-tuned from GPT-3 and is
receiving attention from both scientific and industrial fields. The most
remarkable feature of ChatGPT is that it can generate human-like responses and
communicate with human beings like what a real human can do.

Inspired by the success of pre-trained models in NLP, many researchers apply
the pre-trained model to code-related tasks. Feng et al. [27] propose a
bimodal pre-trained model (CodeBERT) for both programming language and natural
language. CodeBERT is developed on Transformer-based neural architecture and
pre-trained with the task of masked language modeling, which is to predict
tokens, and replaced token detection. Guo et al. [26] present UniXcoder, a
unified cross-modal pre-trained model for programming language. UniXcoder
utilizes mask attention matrices with prefix adapters to control the behavior
of the model and leverages cross-modal contents such as AST and code comment
to enhance code representation. Different from these studies designing novel
pre-train models from scratch, we attempt to boost template-based APR on top
of these pre-trained models.

####  VII-B2 Applications of Pre-trained Models

In addition to the above-mentioned typical pre-trained models, researchers
have also applied such pre-trained models to some code-related domains (e.g.,
code completion, and program repair). Mastropaolo et al. [64] present an
empirical study to investigate the usage of pre-trained models for four code-
related tasks, including program repair, mutants injection, assertion
generation, and code summarization. A similar strategy combining mutation
patterns and pre-trained models is adopted in mutation testing. For example,
Degiovanni et al. [65] introduce $\mu$BERT, a CodeBERT-based mutation testing
tool by masking a token from the expression and replacing the masked token
with the predicted one from CodeBERT. Richter et al. [66] propose a contextual
mutation operator by employing CodeBERT to produce a context-dependent
distribution over feasible token replacements. Recently, Zhang et al. [67]
conduct an extensive empirical study to investigate the performance of pre-
trained models in repairing security vulnerabilities and propose a enhanced
approach with bug fixing transfer learning. Although there exist some SE tasks
(e.g., mutation testing and program repair) benefitting from pre-trained
models, in this work, we perform the first work to employ pre-trained models
to directly predict the correct code with the help of fix patterns.

##  VIII Conclusion

In this work, we present Gamma, an innovative template-based APR tool that
assimilates the advances of fix templates and pre-trained models. Gamma first
defines a set of mask fix templates by masking buggy code tokens with
corresponding code context. Gamma then uses the off-the-shelf pre-trained
models to directly recover the correct code with a mask prediction task. More
importantly, Gamma can be built on various pre-trained models under a zero-
shot learning setting and we implement it as a practical APR tool using the
recent UniXcoder model. The experimental results on the popular Defects4J-v1.2
dataset have shown promising performance, e.g., 82 bugs are fixed by Gamma,
outperforming all state-of-the-art APR techniques. We also demonstrate that
Gamma is able to address the dataset overfitting well, e.g., 45 and 22 bugs
are fixed in Defects4J-v2.0 and Quixbugs. We further demonstrate that Gamma is
generalizable to different pre-trained language models, such as CodeBERT and
ChatGPT.

## Acknowledgment

The authors would like to thank the anonymous reviewers for their insightful
comments. This work is supported partially by the National Natural Science
Foundation of China (61932012, 62141215).

## References

  * [1] M. Monperrus, “Automatic software repair: A bibliography,” _ACM Computing Surveys (CSUR)_ , vol. 51, no. 1, pp. 1–24, 2018.
  * [2] D. Hao, L. Zhang, L. Zang, Y. Wang, X. Wu, and T. Xie, “To be optimal or not in test-case prioritization,” _IEEE Transactions on Software Engineering_ , vol. 42, no. 5, pp. 490–505, 2015.
  * [3] L. Gazzola, D. Micucci, and L. Mariani, “Automatic software repair: A survey,” _IEEE Transactions on Software Engineering (TSE)_ , vol. 45, no. 1, pp. 34–67, 2019.
  * [4] S. Benton, X. Li, Y. Lou, and L. Zhang, “On the effectiveness of unified debugging: An extensive study on 16 program repair systems,” in _2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE’20)_. IEEE, 2020, pp. 907–918.
  * [5] X. Gao, Y. Noller, and A. Roychoudhury, “Program repair,” _arXiv preprint arXiv:2211.12787_ , 2022.
  * [6] Q. Zhang, C. Fang, Y. Ma, W. Sun, and Z. Chen, “A survey of learning-based automated program repair,” _arXiv preprint arXiv:2301.03270_ , 2023.
  * [7] A. Marginean, J. Bader, S. Chandra, M. Harman, Y. Jia, K. Mao, A. Mols, and A. Scott, “Sapfix: Automated end-to-end repair at scale,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP’19)_. IEEE, 2019, pp. 269–278.
  * [8] A. Mesbah, A. Rice, E. Johnston, N. Glorioso, and E. Aftandilian, “Deepdelta: Learning to repair compilation errors,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESE/FSE’19)_ , 2019, pp. 925–936.
  * [9] D. Drain, C. Wu, A. Svyatkovskiy, and N. Sundaresan, “Generating bug-fixes using pretrained transformers,” in _Proceedings of the 5th ACM SIGPLAN International Symposium on Machine Programming (MAPS’21)_ , 2021, pp. 1–8.
  * [10] M. Allamanis, H. Jackson-Flux, and M. Brockschmidt, “Self-supervised bug detection and repair,” _Advances in Neural Information Processing Systems (NeurIPS’21)_ , vol. 34, pp. 27 865–27 876, 2021.
  * [11] M. Martinez and M. Monperrus, “Astor: A program repair library for java,” in _Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA’16)_ , 2016, pp. 441–444.
  * [12] Y. Yuan and W. Banzhaf, “Arja: Automated repair of java programs via multi-objective genetic programming,” _IEEE Transactions on Software Engineering (TSE)_ , vol. 46, no. 10, pp. 1040–1067, 2018.
  * [13] T. Durieux and M. Monperrus, “Dynamoth: Dynamic code synthesis for automatic program repair,” in _Proceedings of the 11th International Workshop on Automation of Software Test (AST’16)_ , 2016, pp. 85–91.
  * [14] J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus, “Nopol: Automatic repair of conditional statement bugs in java programs,” _IEEE Transactions on Software Engineering (TSE)_ , vol. 43, no. 1, pp. 34–55, 2016.
  * [15] A. Koyuncu, K. Liu, T. F. Bissyandé, D. Kim, J. Klein, M. Monperrus, and Y. Le Traon, “Fixminer: Mining relevant fix patterns for automated program repair,” _Empirical Software Engineering (ESE)_ , vol. 25, no. 3, pp. 1980–2024, 2020.
  * [16] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “Avatar: Fixing semantic bugs with fix patterns of static analysis violations,” in _Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER’19)_ , 2019, pp. 1–12.
  * [17] K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyandé, “Tbar: Revisiting template-based automated program repair,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’19)_ , 2019, pp. 31–42.
  * [18] C. S. Xia and L. Zhang, “Less training, more repairing please: Revisiting automated program repair via zero-shot learning,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’22)_ , 2022, pp. 959–971.
  * [19] C. S. Xia, Y. Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in _Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery_ , 2023.
  * [20] K. Liu, D. Kim, T. F. Bissyandé, S. Yoo, and Y. Le Traon, “Mining fix patterns for findbugs violations,” _IEEE Transactions on Software Engineering_ , vol. 47, no. 1, pp. 165–188, 2018.
  * [21] M. Wen, J. Chen, R. Wu, D. Hao, and S.-C. Cheung, “Context-aware patch generation for better automated program repair,” in _Proceedings of the 40th IEEE/ACM International Conference on Software Engineering (ICSE’18)_ , 2018, pp. 1–11.
  * [22] J. Jiang, Y. Xiong, H. Zhang, Q. Gao, and X. Chen, “Shaping program repair space with existing patches and similar code,” in _Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’18)_ , 2018, pp. 298–309.
  * [23] J. Hua, M. Zhang, K. Wang, and S. Khurshid, “Towards practical program repair with on-demand candidate generation,” in _Proceedings of the 40th international conference on software engineering (ICSE’18)_ , 2018, pp. 12–23.
  * [24] D. Yang, K. Liu, D. Kim, A. Koyuncu, K. Kim, H. Tian, Y. Lei, X. Mao, J. Klein, and T. F. Bissyandé, “Where were the repair ingredients for defects4j bugs? exploring the impact of repair ingredient retrieval on the performance of 24 program repair systems,” _Empirical Software Engineering_ , vol. 26, pp. 1–33, 2021.
  * [25] K. Liu, J. Zhang, L. Li, A. Koyuncu, D. Kim, C. Ge, Z. Liu, J. Klein, and T. F. Bissyandé, “Reliable fix patterns inferred from static checkers for automated program repair,” _ACM Transactions on Software Engineering and Methodology_ , 2023.
  * [26] D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin, “Unixcoder: Unified cross-modal pre-training for code representation,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL’22)_ , 2022, pp. 7212–7225.
  * [27] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang _et al._ , “Codebert: A pre-trained model for programming and natural languages,” in _Findings of the Association for Computational Linguistics (EMNLP’20)_ , 2020, pp. 1536–1547.
  * [28] OpenAI, “Chatgpt: Optimizing language models for dialogue,” https://openai.com/blog/chatgpt, 2023, Lasted accessed 2023-05-01.
  * [29] Q. Zhang and T. Zhang, “The gamma project website,” https://github.com/iSEngLab/GAMMA, 2023, Lasted accessed 2023-05-01.
  * [30] Y. Yang, X. Xia, D. Lo, and J. Grundy, “A survey on deep learning for software engineering,” _ACM Computing Surveys (CSUR)_ , vol. 54, no. 10s, pp. 1–73, 2022.
  * [31] W. E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,” _IEEE Transactions on Software Engineering (TSE)_ , vol. 42, no. 8, pp. 707–740, Aug. 2016.
  * [32] K. Liu, S. Wang, A. Koyuncu, K. Kim, T. F. Bissyandé, D. Kim, P. Wu, J. Klein, X. Mao, and Y. L. Traon, “On the efficiency of test suite based program repair: A systematic assessment of 16 automated repair systems for java programs,” in _Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE’20)_ , 2020, pp. 615–627.
  * [33] Q. Zhang, C. Fang, W. Sun, Y. Liu, T. He, X. Hao, and Z. Chen, “Boosting automated patch correctness prediction via pre-trained language model,” _arXiv preprint arXiv:2301.12453_ , 2023.
  * [34] S. Benton, Y. Xie, L. Lu, M. Zhang, X. Li, and L. Zhang, “Towards boosting patch execution on-the-fly,” in _Proceedings of the 44th International Conference on Software Engineering (ICSE’22)_ , 2022, pp. 2165–2176.
  * [35] Q. Zhang, X. Zhai, S. Xu, W. Huang, J. Zhang, and Y. Fan, “Interactive patch filtering via test generation,” in _2022 9th International Conference on Dependable Systems and Their Applications (DSA’22)_. IEEE, 2022, pp. 42–53.
  * [36] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems (NeurIPS’17)_ , 2017, pp. 5998–6008.
  * [37] F. Long and M. Rinard, “Automatic patch generation by learning correct code,” in _Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL’16)_ , 2016, pp. 298–312.
  * [38] Y. Li, S. Wang, and T. N. Nguyen, “Dlfix: Context-based code transformation learning for automated program repair,” in _Proceedings of the 42nd ACM/IEEE International Conference on Software Engineering (ICSE’20)_ , 2020, pp. 602–614.
  * [39] W. Yuan, Q. Zhang, T. He, C. Fang, N. Q. V. Hung, X. Hao, and H. Yin, “Circle: Continual repair across programming languages,” in _Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’22)_ , 2022, pp. 678–690.
  * [40] Z. Li, S. Lu, D. Guo, N. Duan, S. Jannu, G. Jenks, D. Majumder, J. Green, A. Svyatkovskiy, S. Fu _et al._ , “Automating code review activities by large-scale pre-training,” in _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESE/FSE’22)_ , 2022, pp. 1035–1047.
  * [41] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [42] C. Niu, C. Li, V. Ng, J. Ge, L. Huang, and B. Luo, “Spt-code: Sequence-to-sequence pre-training for learning the representation of source code,” in _Proceedings of the 44th International Conference on Software Engineering (ICSE’22)_ , 2022, pp. 2006–2018.
  * [43] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” in _Proceedings of the Advances in Neural Information Processing Systems_ , vol. 33, 2020, pp. 1877–1901.
  * [44] D. Kim, J. Nam, J. Song, and S. Kim, “Automatic patch generation learned from human-written patches,” in _2013 35th International Conference on Software Engineering (ICSE’13)_. IEEE, 2013, pp. 802–811.
  * [45] K. Pan, S. Kim, and E. J. Whitehead, “Toward an understanding of bug fix patterns,” _Empirical Software Engineering_ , vol. 14, pp. 286–315, 2009\.
  * [46] X. Liu and H. Zhong, “Mining stackoverflow for program repair,” in _2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER’18)_. IEEE, 2018, pp. 118–129.
  * [47] N. Jiang, T. Lutellier, and L. Tan, “Cure: Code-aware neural machine translation for automatic program repair,” in _Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE’21)_ , 2021, pp. 1161–1173.
  * [48] R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in _Proceedings of the 23rd International Symposium on Software Testing and Analysis (ISSTA’14)_ , 2014, pp. 437–440.
  * [49] Q. Zhang, Y. Zhao, W. Sun, C. Fang, Z. Wang, and L. Zhang, “Program repair: Automated vs. manual,” _arXiv preprint arXiv:2203.05166_ , 2022.
  * [50] Q. Zhu, Z. Sun, Y.-a. Xiao, W. Zhang, K. Yuan, Y. Xiong, and L. Zhang, “A syntax-guided edit decoder for neural program repair,” in _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’21)_ , 2021, pp. 341–353.
  * [51] D. Lin, J. Koppel, A. Chen, and A. Solar-Lezama, “Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge,” in _Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH Companion’17)_ , 2017, pp. 55–56.
  * [52] T. Lutellier, H. V. Pham, L. Pang, Y. Li, M. Wei, and L. Tan, “Coconut: Combining context-aware neural translation models using ensemble for program repair,” in _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’20)_ , 2020, pp. 101–114.
  * [53] Z. Chen, S. Kommrusch, M. Tufano, L.-N. Pouchet, D. Poshyvanyk, and M. Monperrus, “Sequencer: Sequence-to-sequence learning for end-to-end program repair,” _IEEE Transactions on Software Engineering (TSE)_ , vol. 47, no. 9, pp. 1943–1959, 2019.
  * [54] A. Ghanbari, S. Benton, and L. Zhang, “Practical program repair via bytecode mutation,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’19)_ , 2019, pp. 19–30.
  * [55] T. Durieux, F. Madeiral, M. Martinez, and R. Abreu, “Empirical review of java program repair tools: A large-scale experiment on 2,141 bugs and 23,551 repair attempts,” in _Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’19)_ , 2019, pp. 302–313.
  * [56] H. Ye, J. Gu, M. Martinez, T. Durieux, and M. Monperrus, “Automated classification of overfitting patches with statically extracted code features,” _IEEE Transactions on Software Engineering (TSE)_ , vol. 48, no. 8, pp. 2920–2938, 2022.
  * [57] K. Liu, A. Koyuncu, T. F. Bissyandé, D. Kim, J. Klein, and Y. Le Traon, “You cannot fix what you cannot find! an investigation of fault localization bias in benchmarking automated program repair systems,” in _Proceedings of the 12th IEEE conference on software testing, validation and verification (ICST’19)_ , 2019, pp. 102–113.
  * [58] N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of code language models on automated program repair,” in _Proceedings of the 45th International Conference on Software Engineering (ICSE’23)_ , 2023, pp. 1430–1442.
  * [59] C. S. Xia, Y. Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in _Proceedings of the 45th International Conference on Software Engineering (ICSE’23)_ , 2023, pp. 1482–1494.
  * [60] S. Mechtaev, J. Yi, and A. Roychoudhury, “Angelix: Scalable multiline program patch synthesis via symbolic analysis,” in _Proceedings of the 38th international conference on software engineering (ICSE’16)_ , 2016, pp. 691–701.
  * [61] D. Sobania, M. Briesch, C. Hanna, and J. Petke, “An analysis of the automatic bug fixing performance of chatgpt,” in _Proceedings of the Fouth International Workshop on Automated Program Repair (APR’23)_ , 2023, pp. 1–8.
  * [62] E. Mashhadi and H. Hemmati, “Applying codebert for automated program repair of java simple bugs,” in _Proceedings Companion of the 18th IEEE/ACM International Conference on Mining Software Repositories (MSR’21)_ , 2021, pp. 505–509.
  * [63] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’19)_. Association for Computational Linguistics, 2019, pp. 4171–4186.
  * [64] A. Mastropaolo, N. Cooper, D. N. Palacio, S. Scalabrino, D. Poshyvanyk, R. Oliveto, and G. Bavota, “Using transfer learning for code-related tasks,” _IEEE Transactions on Software Engineering (TSE)_ , 2022.
  * [65] R. Degiovanni and M. Papadakis, “$\mu$bert: Mutation testing using pre-trained language models,” in _2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW’22)_. IEEE, 2022, pp. 160–169.
  * [66] C. Richter and H. Wehrheim, “Learning realistic mutations: Bug creation for neural bug detectors,” in _2022 IEEE Conference on Software Testing, Verification and Validation (ICST’22)_. IEEE, 2022, pp. 162–173.
  * [67] Q. Zhang, C. Fang, B. Yu, W. Sun, T. Zhang, and Z. Chen, “Pre-trained model-based automated software vulnerability repair: How far are we?” _IEEE Transactions on Dependable and Secure Computing_ , 2023.