# Privacy-Preserving Distributed Market Mechanism for Active Distribution
Networks

Matthias Franke1, Ognjen Stanojev1, Lesia Mitridati2, Gabriela Hug1  2 Center
for Electric Power and Energy, Technical University of Denmark, Kgs. Lyngby,
Denmark  1EEH - Power Systems Laboratory, ETH Zürich, Zürich, Switzerland
{mfranke, ognjens<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Amidst the worldwide efforts to decarbonize power networks, Local Electricity
Markets (LEMs) in distribution networks are gaining importance due to the
increased adoption of renewable energy sources and prosumers. Considering that
LEMs involve data exchange among independent entities, privacy and
cybersecurity are some of the main practical challenges in LEM design. This
paper proposes a secure market protocol using innovations from distributed
optimization and Secure MultiParty Computation (SMPC). The considered LEM is
formulated as an uncertainty-aware joint market for energy and reserves with
affine balancing policies. To achieve scalability and enable the use of SMPC,
market clearing is solved using the Consensus ADMM algorithm. Subsequently,
the data exchange among participants via ADMM iterations is protected using
the Shamir secret-sharing scheme to ensure privacy. The market protocol is
further reinforced by a secure and verifiable settlement process that uses
SMPC and ElGamal commitments to verify market quantities and by a secure
recovery scheme for missing network measurements. Finally, the feasibility and
performance of the proposed LEM are evaluated on a 15-bus test network.

###### Index Terms:

Consensus ADMM, Local Electricity Markets, Cyber Security, Secure Multiparty
Computation

##  I Introduction

With the goal to mitigate climate change, an increasing effort is made to
decarbonize power networks. This effort primarily involves the increased
adoption of Distributed Energy Resources (DERs) in distribution grids and
empowering households to become prosumers that can contribute to
decarbonization [1]. Despite these advancements, current electricity markets
still operate in a hierarchical, top-down manner, without adapting their
market structures to the emergence of DERs [2]. Furthermore, technical
challenges arise in the operation of distribution networks due to high bi-
directional power flows and voltage fluctuations [3, 4]. These challenges
suggest the development of Local Electricity Markets (LEMs) in which small-
scale producers and consumers can transact electricity in a decentralized
fashion while respecting the physical limits of the distribution network.
Nevertheless, such a collaborative principle involves data exchange among
independent entities, therefore rising privacy and cybersecurity concerns. We
propose an uncertainty-aware LEM that is operated in a secure and distributed
manner using innovations from optimization and theoretical cryptography.

There has been a wide range of research into innovating LEMs that promise to
safely and efficiently operate future networks and markets with DERs [1]. The
three main approaches for such LEMs are [2]: (i) pure Peer-to-Peer markets,
whose fully decentralized approach offers customers high flexibility at the
cost of lacking convergence and safety guarantees, (ii) community markets with
an operator efficiently managing trading both locally and with the upper-level
grid, however, resulting in computational scaling issues, and (iii) hybrid
methods that use a bilevel approach to leverage the advantages of the former
two. Of key interest for this paper are the community markets, which are
proven to successfully operate LEMs with DERs in the existing literature [5,
6]. However, these markets have encountered challenges due to the use of
oversimplified network models [5], the absence of DER uncertainty management
[2], and the lack of effective coordination with the upper-level grid [6].
Additionally, most works in this field focus on theoretical explorations of
such markets, and there is thus a lack of research into how to provide cyber
security and resilience to measurement failure for the actual implementation
of LEMs.

Efforts to overcome these gaps have been made, although they have not been
consolidated yet. For instance, improved models of distribution networks
together with techniques to capture the stochastic nature of DERs are
available [7], such as the LinDistFlow formulation [8, 9] that achieves high
accuracy through the utilization of linear and convex constraints. This
formulation overcomes shortcomings of the DC power flow formulations for
distribution networks [8] and is further highly suitable for adding chance
constraints to capture the uncertainty of DERs within a market model [6]. Of
particular interest to this paper is the usage of chance constraints to
concurrently optimize day-ahead energy and reserve needs [5].

To overcome computational complexity and scalability issues, the usage of
distributed optimization for the solution of LEM formulations is suitable and
further allows the integration of the relevant cryptographic techniques. The
work in [10] decomposes the collaborative optimization problem using
Lagrangian relaxation and solves the master problem via Secure Multi-Party
Computation (SMPC) with secret sharing protocols. However, it mainly serves as
a proof of concept, with intentionally simple network and market models, and
the authors themselves highlight the Lagrangian Relaxation’s inability to
provide convergence guarantees. Such guarantees can be provided using the
Consensus version of the Alternating Direction Method of Multipliers (ADMM)
[11], which has already been used with the LinDistFlow grid model [8, 12].

The main cryptographic technique employed within this study is SMPC, which
belongs to a family of cryptographic protocols that can execute arbitrary
multiparty computations with information-theoretic security in a threshold
model [13]. This characteristic renders SMPC well-suited for use in LEM
design. Notably, it sidesteps the challenges that previous privacy-preserving
optimization methods encountered, such as poor computational performance,
degraded convergence guarantees, and weaker security notions like differential
privacy [14, 15]. Furthermore, the applicability of SMPC extends to
reinforcing the security of LEMs protecting against network sensor failures
due to potential interference of adversaries [16].

This paper proposes a fully privacy-preserving LEM framework based on
distributed optimization and SMPC with secret sharing protocols. Building upon
prior research [6], we develop an uncertainty-aware joint market for energy
and reserves using the chance-constrained LinDistFlow formulation. The model
is further extended by introducing batteries as local energy storage [17] and
by capturing the tariff-switching behavior of the substation [18]. To overcome
the computational limitations encountered in [10], we propose a distributed
and privacy-preserving market clearing mechanism using an SMPC-secured
Consensus ADMM algorithm, as described in Sec. II-D and Sec. III-D. The secure
version of the LEM requires mechanisms to preserve the security of the system
between SMPC sessions, for which we use standard techniques called commitment
schemes and Zero-Knowledge Proofs that can be applied without impinging on the
overall privacy preservation [19]. In particular, we leverage SMPC and ElGamal
commitments [20] to provide a double verification scheme and a measurement
recovery scheme (Sec. III-E) to provide a secure and verifiable settlement
process (Sec. III-F). This setup allows the proposed market mechanism to
achieve similar solutions as a central insecure solver but with added privacy
preservation, as shown by the results in Sec. IV.

##  II Local Electricity Market Design

###  II-A Preliminaries

This paper studies a LEM in an active distribution grid. The distribution
network is represented as an undirected and connected tree graph
$\mathcal{G}(\mathcal{N},\mathcal{E})$, where $\mathcal{N}=\\{0,1,\dots,N\\}$
is the set of nodes in the graph and
$\mathcal{E}\subset\mathcal{N}\times\mathcal{N}$ is the set of $N$ edges. The
substation node is indexed by 0 and represents the interface between the
distribution network and the upper-level grid. Furthermore, we use
$\mathcal{N}^{+}=\mathcal{N}\setminus\\{0\\}$ to denote the set of non-
substation nodes. For a node $n\in\mathcal{N}$, its ancestor node is denoted
by $A_{n}$, while the set of its children nodes is denoted by
$\mathcal{C}_{n}$. The notation related to the physical quantities of the
LinDistFlow network model is introduced in the following.

For each node $n\in\mathcal{N}^{+}$, let $u_{n}$ denote the squared voltage
magnitude at the node, and ${u}_{n}^{\mathrm{max}}$ and
${u}_{n}^{\mathrm{min}}$ the corresponding maximum and minimum voltage limits.
The substation node is assumed to have a fixed predefined voltage magnitude
$u_{0}$. The active and reactive line flows to a node $n\in\mathcal{N}^{+}$
from its ancestor node $A_{n}$ are denoted by $f_{n}^{P}$ and $f_{n}^{Q}$,
respectively. Furthermore, each such line is characterized by a resistance
$r_{n}$, a reactance $x_{n}$, and a maximum apparent power limit ${S}_{n}$.

The participants in the considered LEM include: (i) consumers at all non-
substation nodes $i\in\mathcal{N}^{+}$, who each have known and inflexible
active $d_{i}^{P}$ and reactive $d_{i}^{Q}$ demand profiles, (ii) prosumers
$r\in\mathcal{R}\subseteq\mathcal{N}^{+}$ who own DERs with a forecasted
active power production $h_{r}^{f}$, (iii) batteries
$m\in\mathcal{M}\subseteq\mathcal{N}^{+}$, characterized by their State of
Charge (SoC) ${B}_{m}$, and the corresponding maximum ${B}_{m}^{\mathrm{max}}$
and minimum ${B}_{m}^{\mathrm{min}}$ SOC limits, and (iv) the substation node,
which is centrally operated to trade energy and procure reserves with the
wholesale markets. Batteries and prosumers constitute flexible generators
$v\in\mathcal{V}=\mathcal{R}\cup\mathcal{M}$ with adjustable active power
generation $g_{v}^{P}$ between maximum $P_{v}^{\mathrm{max}}$ and minimum
$P_{v}^{\mathrm{min}}$ limits. The cost of active power adjustment is
characterized by a quadratic $c^{q}_{v}$ and a linear $c_{v}^{l}$ cost
coefficient. The substation node is the “infinite bus”, with its generation
modeled as the difference between the active $l^{P}$ and reactive $l^{Q}$
inflow and active $s^{P}$ and reactive $s^{Q}$ outflow to capture a tariff-
switching scheme without the need for binary variables [18], i.e., the system
operator charges inflow to the LEM by the forecasted wholesale price plus a
flat usage tariff $\Phi^{+}$, and pays for outflow at the wholesale price
minus the tariff $\Phi^{-}$.

###  II-B Uncertainty Modeling

The DER generation of a prosumer $r\in\mathcal{R}$ introduces uncertainty in
the LEM and is thus modeled by

${h}_{r}=h_{r}^{f}+{\omega}_{r},$ (1)

where ${h}_{r}$ is the stochastic DER generation and ${\omega}_{r}$ is the
random forecast error. The forecast error is modeled as an independent
Gaussian random variable with zero mean $\mathbb{E}[{\omega}_{r}]=0$ and
variance $\mathrm{Var}[{\omega}_{r}]=\sigma_{r}$, known only to the prosumer
$r\in\mathcal{R}$ itself. The total DER forecast error in the system can thus
be calculated as ${\Delta}=\sum_{r\in\mathcal{R}}{\omega}_{r}$, resulting in a
zero-mean multivariate distribution with
$\Sigma=\mathrm{Var}[{\Delta}]=\mathrm{diag}(\\{\sigma_{r}\\}_{r\in\mathcal{R}})$.

The realized total forecast error creates a power imbalance in the system that
requires a global response from the flexible assets. To this end, we equip
each flexible generator $v\in\mathcal{V}$ with a linear reserve policy [5],
characterized by a participation factor $\alpha_{v}$, such that its total
generation is given by

$\tilde{g}_{v}^{P}=g_{v}^{P}-\alpha_{v}{\Delta}.$ (2)

Given that $\sum_{v\in\mathcal{V}}\alpha_{v}=1$, the flexible generators
completely balance the observed overall active power deviation.

###  II-C Market Clearing Formulation

In this section, we present the chance-constrained market formulation. The
variable set includes the previously defined network and adjustable generation
quantities and is defined by

$\displaystyle\Xi^{\mathbb{P}}=$
$\displaystyle\,\\{u_{i,t},f^{P}_{i,t},f^{Q}_{i,t}\\}_{i\in\mathcal{N}^{+},t\in\mathcal{T}}\cup\\{g_{v,t}^{P},\alpha_{v,t}\\}_{v\in\mathcal{V},t\in\mathcal{T}}$
$\displaystyle\cup\\{l^{P}_{t},s^{P}_{t},l^{Q}_{t},s^{Q}_{t}\\}_{t\in\mathcal{T}}\cup\\{B_{m,t}\\}_{m\in\mathcal{M},t\in\mathcal{T}},$
(3)

where $\mathcal{T}$ is the set of considered time steps. The aim of the
proposed local market is to minimize the expected cost of generation and
energy procurement from the upper-level grid while at the same time respecting
the network and generation constraints for all time steps $t\in\mathcal{T}$:

$\min_{\Xi^{\mathbb{P}}}\,\mathbb{E}\Big{[}\sum_{t\in\mathcal{T}}\sum_{v\in\mathcal{V}}\Big{(}c_{v}^{q}(\tilde{g}_{v,t}^{P})^{2}+c_{r}^{l}\tilde{g}_{v,t}^{P}\Big{)}+{l}^{P}_{t}\Phi^{+}_{t}-{s}_{t}^{P}\Phi^{-}_{t}\Big{]}$
(4a) $\displaystyle\text{s.t.}\quad
l_{t}^{P}-s_{t}^{P}=\sum_{j\in\mathcal{C}_{0}}f_{j,t}^{P}$ (4b) $\displaystyle
l_{t}^{Q}-s_{t}^{Q}=\sum_{j\in\mathcal{C}_{0}}f_{j,t}^{Q}$ (4c) $\displaystyle
l^{P}_{t}\geq 0,s^{P}_{t}\geq 0,l^{Q}_{t}\geq 0,s^{Q}_{t}\geq 0$ (4d)
$\displaystyle
f_{n,t}^{P}+(\tilde{g}_{n,t}^{P}-d_{n,t}^{P}+h_{n,t})=\sum_{j\in\mathcal{C}_{n}}f_{j,t}^{P},$
$\displaystyle\forall n\in\mathcal{N}^{+}$ (4e) $\displaystyle
f_{n,t}^{Q}-d_{n,t}^{Q}=\sum_{j\in\mathcal{C}_{n}}f_{j,t}^{Q},$
$\displaystyle\forall n\in\mathcal{N}^{+}$ (4f) $\displaystyle
u_{n,t}=u_{A_{n},t}-2(r_{n}f_{n,t}^{P}+x_{n}f_{n,t}^{Q}),$
$\displaystyle\forall n\in\mathcal{N}^{+}$ (4g) $\displaystyle
B_{m,t}=B_{m,t-1}-\tilde{g}_{m,t}^{P},$ $\displaystyle\forall m\in\mathcal{M}$
(4h) $\displaystyle\mathbb{P}[{u}_{n,t}\leq{u}_{n}^{\mathrm{max}}]\geq
1-\epsilon_{u},$ $\displaystyle\forall n\in\mathcal{N}^{+}$ (4i)
$\displaystyle\mathbb{P}[{u}_{n}^{\mathrm{min}}\leq{u}_{n,t}]\geq
1-\epsilon_{u},$ $\displaystyle\forall n\in\mathcal{N}^{+}$ (4j)
$\displaystyle\mathbb{P}[a^{1}_{n}{f}_{n,t}^{P}+a^{2}_{n}{f}_{n,t}^{Q}+a^{3}_{n}S_{n}\leq
0]\geq 1-\epsilon_{f},\quad$ $\displaystyle\forall n\in\mathcal{N}^{+}$ (4k)
$\displaystyle\mathbb{P}[\tilde{g}_{v,t}^{P}\leq P_{v}^{\mathrm{max}}]\geq
1-\epsilon_{g},$ $\displaystyle\forall v\in\mathcal{V}$ (4l)
$\displaystyle\mathbb{P}[P_{v}^{\mathrm{min}}\leq\tilde{g}_{v,t}^{P}]\geq
1-\epsilon_{g},$ $\displaystyle\forall v\in\mathcal{V}$ (4m)
$\displaystyle\mathbb{P}[B_{m,t}\leq B_{m}^{\mathrm{max}}]\geq
1-\epsilon_{b},$ $\displaystyle\forall m\in\mathcal{M}$ (4n)
$\displaystyle\mathbb{P}[B_{m}^{\mathrm{min}}\leq B_{m,t}]\geq
1-\epsilon_{b},$ $\displaystyle\forall m\in\mathcal{M}$ (4o)
$\displaystyle\sum_{v\in\mathcal{V}}\alpha_{v,t}=1,\quad 0\leq\alpha_{v,t}\leq
1,$ $\displaystyle\forall v\in\mathcal{V}.$ (4p)

Constraints related to inflow/outflow at the substation node are given in
(4b)-(4d). The LinDistFlow network model is established in (4e)-(4g), and the
battery SoC model111For the sake of simplicity, but without loss of
generality, we assume perfect charging and discharging efficiencies. in (4h).
The chance constraints on bus voltages are enforced in (4i)-(4j), power
generation limits in (4l)-(4m), battery SoC in (4n)-(4o), and the dodecagon
linear approximations (defined by coefficients
$a^{1}_{n},a^{2}_{n},a^{3}_{n},\forall n\in\mathcal{N}^{+}$) of the line flow
constraints in (4k). The LEM operator specifies an error term for each type of
CC, representing the maximum acceptable percentage of a constraint violation,
namely, $\epsilon_{g}$ for generation limits, $\epsilon_{b}$ for battery
limits, $\epsilon_{u}$ for voltage limits and $\epsilon_{f}$ for flow limits.
Finally, the bounds on reserve coefficients are given in (4p).

###  II-D Distributed Solution Method

To obtain a tractable form of the optimization problem in (4), each individual
linear chance constraint can be reformulated as a second-order cone
constraint. This process is omitted for brevity, and we refer the reader to
[6] for more details. Instead, we here focus on decomposing the centralized
optimization problem into a distributed form using the scaled formulation of
Consensus ADMM [11], as proposed in [12].

Considering that the objective function in (4a) is separable, let us introduce
a cost function $f_{n}(X_{n})$ related to each node $n\in\mathcal{N}$, where
$X_{n}$ denotes the vector of all variables related to node $n$. A subset of
these variables, collected in
$X_{\mathbb{C}_{n}}=(u_{n},f_{n}^{P},f_{n}^{Q},\alpha_{n})$ and called
coupling variables, appear in the constraints of other nodes. The coupling
variables have a global copy $Z_{\mathbb{C}_{n}}$ in the Consensus ADMM
algorithm that ensures their system-wide convergence. The full algorithm at
iteration $k$ is given by

$\displaystyle X_{\mathbb{C}_{n}}^{k+1}$
$\displaystyle=\mathrm{arg\,min}\,{f}_{n}(X_{n})+\frac{\rho_{0}}{2}||X_{\mathbb{C}_{n}}^{k}-Z_{\mathbb{C}_{n}}^{k}+U_{\mathbb{C}_{n}}^{k}||_{2}^{2},$
(5a) $\displaystyle Z^{k+1}_{\mathbb{C}_{n}}$
$\displaystyle=\mathbb{AVG}(X^{k+1}),$ (5b) $\displaystyle
U_{\mathbb{C}_{n}}^{k+1}$
$\displaystyle=U_{\mathbb{C}_{n}}^{k}+X_{\mathbb{C}_{n}}^{k+1}-Z_{\mathbb{C}_{n}}^{k+1},$
(5c)

where $\rho_{0}$ is a positive scaling factor and $U_{\mathbb{C}_{n}}$ are the
Lagrange multipliers. In (5b), an arithmetic mean of all relevant local values
is computed to find the global values of the coupled variables. The employed
convergence metrics are the $\ell_{2}$-norms of the primal and dual residuals
of the local problems and the total power surplus in the system relative to
the total demand.

###  II-E Financial Settlement

Solving the market problem (4) yields nodal $\lambda_{n}$ and flexibility
$\pi$ prices, calculated as dual variables of constraints (4e) and (4p),
respectively. Using these prices, the financial payoff for a market
participant $n$ from the daily market operation is

$\mathcal{B}_{n}=\sum_{t\in\mathcal{T}}\Big{[}\lambda_{n,t}^{P}(g_{n,t}^{P}+h_{n,t}^{f}-d_{n,t}^{P})+\pi_{t}(\alpha_{n,t}-\frac{\sigma_{n,t}}{\sum_{r\in\mathcal{R}}\sigma_{r,t}})\Big{]}$
(6)

As can be seen above, the participants are charged at the flexibility price
$\pi$ for their share of the overall uncertainty, captured by the standard
deviation of their forecast error, as well as a fixed tariff for the power
flow via the substation [18]. Their profit stems from the provided flexibility
and the injected DER controllable and forecasted active power. The final
balance $\mathcal{F}_{n}$ of a party $n$, however, also includes the results
from the two-price imbalance regulation $\mathcal{I}_{n}$ performed by the DSO
[21] on any remaining deviations:

$\mathcal{F}_{n}=\mathcal{B}_{n}+\mathcal{I}_{n}.$ (7)

##  III Secure Local Market Protocol

###  III-A Secure Multiparty Computation Preliminaries

The technique used to ensure the security of the market protocol is Secure
Multiparty Computation [13], which allows a group of parties to execute secure
computations without relying on a trusted third party. The communication links
between the parties are assumed to be encrypted and synchronous. Security in
SMPC is defined by input privacy and protocol correctness. Input privacy
requires the input values and calculation results to be kept hidden from other
participants unless intentionally revealed, whereas protocol correctness
requires that the computations yield the same results as with a trusted third
party. The goal is thus to have security no worse than an ideal scheme with a
trusted party, i.e., SMPC is allowed to be vulnerable to attacks that also
work against the ideal scheme.

The considered adversaries are assumed to be static and honest-but-curious.
The static adversary corrupts participants before the start of the protocol
but does not corrupt further during execution. Secondly, the honest-but-
curious (or passive) adversary seeks only to violate the input privacy
property and does otherwise not deviate from the established SMPC protocol
[13]. The choice to work with an honest-but-curious adversary is due to the
nature of ADMM. While the global stage (iteration) can be implemented with an
SMPC secure against misbehaving or active adversaries, these techniques cannot
protect against parties misbehaving in the local optimization stage. In
particular, an adversary could simply return random values at each iteration
and thereby prevent the ADMM from converging without violating any of the SMPC
guarantees. Overcoming this issue would necessitate delving into verifiable
computing aspects, which are out of the scope.

###  III-B Shamir Secret-sharing Scheme

The SMPC protocol used in this paper is based on the Shamir secret-sharing
scheme (SSS) [10]. It provides threshold security based on the number of
participants $N$ and a defined threshold $\Theta<N$, where a single, central
adversary can corrupt up to $\Theta$ participants without compromising the
protocol’s security. From a high-level perspective, the scheme involves
transforming a secret $s$ into a secure shared domain, yielding $[s]$, then
performing secure calculations on the shared values, and finally, recovering
relevant results $r$, by a reconstruction procedure $[r]\mapsto r$. It is
worth noting that due to the inner workings of Shamir secret sharing [10],
calculations in an SMPC scheme based on it can only use addition and
multiplication operations. As such, the secure market protocol that follows
uses a variety of transformations and simplifications to reduce computational
complexity as much as possible.

###  III-C Secure Market Protocol Overview

Figure 1: Overview of the complete (secure) market protocol.

The overview of the proposed market protocol is presented in Fig. 1, with the
secure blocks given in green and a dashed line separating the time of market
clearing and the time of operation. Each day, market participants initialize
the secure market clearing protocol by providing their desired internal
parameters such as battery limits, cost coefficients, generation limits, etc.
The secure market clearing is then performed, as will be explained in Sec.
III-D, which involves the distributed market formulation from Sec. II-D with
the updating of global variables and testing for convergence, both implemented
securely with SMPC. Upon completion, the parties then store the outcomes of
the market for later usage. After the simulated or real market operation, the
parties then execute a secure measurement recovery routine (Sec III-E), which
ensures the protocol has all the relevant information it needs to proceed.
Then, the secure settlement process (Sec. III-F) uses the stored market
outcomes and recovered measurements to securely compute the financial balances
of all parties. The protocol then ends for the day by having the parties
update their internal parameters based on the financial and operational
outcomes.

###  III-D Secure Market Clearing

The main usage of SMPC is to secure the distributed optimization process by
replacing the central node that performs the coordination calculations (5b).
Figure 2 shows how the ADMM update loop from (5) can be secured via SMPC,
where white boxes are local operations and boxes shaded green are SMPC
operations. Parties hereby securely share the new value of their local
variables $X^{k+1}_{\mathbb{C}_{n}}$, then calculate the shared value of the
new global variables $[Z]_{\mathbb{C}}^{k+1}$, and finally, output their true
values to the relevant parties for use in local optimization
$Z^{k+1}_{\mathbb{C}_{n}}$. To improve performance, the division operation in
the algebraic mean calculation is done locally, meaning the SMPC instance only
involves the addition of scaled variables.

The evaluation of convergence criteria is also done within the SMPC instance,
which can become computationally expensive due to the non-linear complexity of
secure inequality evaluations and Euclidean norm calculations. To address this
issue, a stricter version of the residual criteria is used – the infinity
norm, instead of the commonly employed $\ell_{2}$-norm.

Figure 2: Illustration of an ADMM iteration with SMPC.

###  III-E Measurement Recovery

A major concern in digital markets, including the considered LEM, is the
“Oracle Problem”, where a cyber attack on the underlying measurement systems
cause the ground truth and the market’s understanding to diverge, causing
strain on consumers and network operators [22]. This problem is exacerbated in
distribution grids where ensuring complete network observability would require
expensive deployment of PMUs or RTUs [23, 24]. The market protocol is thus
backstopped with a deterministic and secure measurement recovery procedure to
ensure the financial settlement process can be executed even if a subset of
nodes fails to report any measurements.

The two main failure modes are nodes not reporting measurements and nodes
reporting false measurements [16], with the latter being difficult to
counteract if done at scale. This paper focuses on the former, with the use
case being participants purposefully disconnecting their deployed sensor
during operation. We assume the use of modified smart meters that can measure
the active and reactive nodal injections and the active and reactive incoming
line flow at each node. The measurements along the lines are not required
since LinDistFlow assumes lossless lines. Under the assumption of altered
smart meters and an always reporting substation node, the following recovery
procedure generates feasible values for all outstanding net injection
measurements, even if they are not guaranteed to report the actual value.

Non-reporting nodes in the network are grouped into disjoint segments, called
islands, via the UnionFind algorithm [25]. For each such island $\mathcal{Y}$,
SMPC is then used to calculate its total active power inflow
$\mathrm{P}_{\mathcal{Y}}^{\mathrm{in}}$ and outflow
$\mathrm{P}_{\mathcal{Y}}^{\mathrm{out}}$ using the measurements of honest
ancestors and descendants. This determines the overall active power net
injection of the island, which is then split equally among the members of a
given island to ensure burden sharing, as follows

$\mathrm{net}_{y}^{P}=\frac{\mathrm{P}_{\mathcal{Y}}^{\mathrm{out}}-\mathrm{P}_{\mathcal{Y}}^{\mathrm{in}}}{|\mathcal{Y}|},\quad\quad\quad\forall
y\in\mathcal{Y},$ (8)

where $|\mathcal{Y}|$ denotes the cardinality of set $\mathcal{Y}$. Note that
the resulting value may not correspond to the withheld value for each
$y\in\mathcal{Y}$. Any such deviations are balanced via the balancing
mechanism at the time of operation and factor into the settlement process.

###  III-F Secure Settlement Process

The calculation of the financial payoff $\mathcal{B}_{n}$ for a party $n$
after the market clearing is performed employing the same SMPC protocol used
for ADMM coordination. Since the prices used for calculating $\mathcal{B}_{n}$
are dual variables of constraints, they can also be calculated globally as
Lagrange multipliers via SMPC, as explained in [26]. For the final financial
balances, the settlement process then factors in the results of the DSO
imbalance market (see Section II-E) and the measurement recovery process.

The secure market protocol introduces a break in time between market clearing
and final settlement, requiring each node to re-input their own balance
$\mathcal{B}_{n}$ into the SMPC instance computing the final settlement. A
Double Verification Scheme (DVS) is therefore proposed to allow secure
verification of these input balances by an honest majority of parties,
including actors not participating in the protocol, such as the market
operator. The DVS combines storing and securely comparing secret-shared
balances with the use of cryptographic commitments to verify the self-claimed
balances and is inspired by publicly verifiable secret-sharing schemes [27].
Specifically, parties re-share their post-market clearing financial balances,
which are securely compared using SMPC to the shared values the other parties
stored after market clearing. In the second phase, parties use the perfectly
binding ElGamal commitment scheme to “commit” to their balances after market
clearing. If needed, they can then “open” the balances in the second phase of
DVS to prove that they used the correct value without it actually being
revealed [20]. Thus, DVS Phase 1 provides threshold security via SMPC and DVS
Phase 2 provides information theoretic security via ElGamal commitments.

DVS thus protects against manipulations by corrupted parties while also
providing cryptographic proof for the calculated final financial balances. The
secure settlement can then use SMPC to calculate imbalances between the
scheduled and actual network values that have both been input securely to find
$\mathcal{I}_{n}$ for each party and then output the final financial balances
of all parties.

##  IV Results

###  IV-A Case Study Setup

The proposed secure market clearing protocol is evaluated on a 15-bus test
network depicted in Fig. 3. This synthetic case study is constructed using
simulated data from various real-world datasets, as described in the
following. The network model incorporates nodal demand and solar power
generation profiles from an English dataset [28], wind power generation
profiles from an Australian dataset [29], and battery sizing based on the
Tesla Powerwall [30]. Note that in this specific case study, all prosumers
have both a DER and a battery, with their flexible generation stemming
entirely from the battery. The power prices are obtained from various sources,
including suggested substation tariff [2], wholesale day-ahead prices [31],
and regulation market prices [32] from the Danish TSO. Finally, the cost
parameters and standard deviations for the prosumers’ generation are based on
the work in [6], and the battery parameters are derived from [33].

Figure 3: A synthetic 15-bus test system, with PV generation (in yellow),
batteries (in blue), and wind generation (in white).

The Python package MPyC [34] was utilized for implementing the SMPC protocol
and additionally provided the elliptic curve cryptography necessary for the
ElGamal commitment scheme. For the local optimization, the CVXPY optimization
library was used with ECOS solver for second-order cone programs and the OCSP
solver for quadratic programs. CVXPY also natively supports ADMM. The
implementation is done in Python, and the code was executed on a laptop with a
6-core processor and 16 GB of RAM.

The results section centers around a comparative analysis of the performance
of the following four local market solvers:

  * •

C-1: An insecure central solver solving the market problem in (4), with all
chance constraints reformulated as second-order cone constraints [6].

  * •

N-3: An insecure distributed solver solving the reformulated chance-
constrained market problem (4).

  * •

S-2: A secure distributed solver solving the reformulated chance-constrained
market problem (4) with deterministic voltage magnitude and line flow
constraints. Security is imposed using the protocols described in Sec. III.

  * •

S-3: A secure distributed solver solving the complete reformulated chance-
constrained market problem (4), making it the secure version of N-3.

###  IV-B Solver Comparison

The level of security offered by these solvers depends on the extent to which
they disclose the private data of the parties during operation. The central
solver (C-1), for instance, discloses all parties’ information solely to a
central computation node. If this central point is compromised, it could
potentially threaten input privacy and the correctness of the entire market
protocol. On the other hand, in the distributed solvers utilizing ADMM, nodes
share certain data only with their immediate neighbors (N-3). Furthermore,
when secured with SMPC, widespread data leakage can only occur if a majority
of parties are compromised (S-2 and S-3).

Table I compares the four solvers in terms of the time and global iterations
required for the algorithm to converge, as well as the resulting relative
accuracy. The latter refers to the total active power residual in the LEM
relative to the fixed total demand and gauges the feasibility of the solution.
All considered solvers converge with similar levels of relative accuracy, but
the use of both the decomposition and SMPC extend the time required for the
convergence. Notably, N-3 and S-3 require similarly many ADMM iterations,
indicating that the calculations in SMPC work adequately. However, comparing
S-2 and S-3 shows that full chance constraining doubled the ADMM iterations
and tripled the time to convergence. This points to a trade-off between the
model complexity and the computation performance. Overall, the results
indicate that the secure market protocol via solvers S-2 and S-3 is capable of
clearing a complex market with results that are slightly worse but comparable
to the centralized solver.

TABLE I: Comparison of the four featured solvers. Solver | C-1 | N-3 | S-2 | S-3
---|---|---|---|---
Time [min] | 0.49 | 66.76 | 38.06 | 119.8
Global iterations | 1 | 595 | 270 | 593
Rel. accuracy [%] | $0$ | $1.8$ | $-1.96$ | $4.21$

###  IV-C Convergence of Secure Market Protocols

As a result of the use of ADMM, all distributed solvers in the paper are
guaranteed to eventually converge [26]. However, as the convergence of the
residuals of the voltage magnitudes for the S-2 solver in Fig. 4 highlights,
there are some notable dynamics at play. The convergence of ADMM typically
exhibits a damped oscillatory behavior [11], as is also evident in the given
figure, persisting until approximately iteration 250. Around this iteration,
the residuals start to briefly increase again due to the inter-dependency of
voltage magnitudes and power flows in the LinDistFlow model. Their constant
trade-off (as well as the ripple effect across the nodes) implies that the
solver can locally diverge from the expected trends but will eventually
recover as the ADMM iterations progress. Interestingly, the residual
oscillations are of higher magnitudes at nodes closer to the substation. The
observed convergence pattern highlights the possibility for improvement, e.g.,
through the implementation of adaptive penalty terms or over-relaxation
techniques [26], which are, however, beyond the scope of this paper.

Figure 4: Voltage magnitude residuals across all nodes when using S-2 solver.

###  IV-D Market Clearing Outcomes

Figure 5: Global energy balance over a day for the S-2 solver.

Figure 5 showcases the different energy balance components over the entire
network. It can be observed that the day-ahead energy dispatch of the network
is driven by prosumers who use batteries to shift the DER production into the
high-demand morning and evening hours. Similar behavior has been seen in
previous research [35]. Despite the slight changes in formulations and
solution methods, all solvers had very similar energy schedules and prices as
the presented S-2 solver, indicating that the secure market protocol is able
to clear the market successfully.

The allocation of flexibility participation factors described in (2) by the
different solvers is observed to fall into one of the following two
arrangements, which is then maintained for the entire day. Specifically, all
fully chance-constrained solvers assigned the substation the entire
flexibility provision, while the S-2 solver, which does not factor in voltage
and line flow constraints, splits the flexibility provision equally amongst
all nodes with flexible generation. This resulted in the flexibility price
being constant throughout the day.

###  IV-E Financial Settlement Results

The above-discussed scheduling outcomes are reflected in the post-market
clearing financial balances, with Table II comparing the balances of 4
different LEM participants between solvers N-3, S-2, and S-3. Due to the large
variability found in flexibility prices across solvers, the table focuses on
the day-ahead energy results. As explained in Sec. III-F, in a distributed
setting, the market prices can either originate from the dual variables in the
local optimization problems that the parties then share with the LEM (Duals)
or from global Lagrange multipliers calculated in SMPC (Sec-Bal). To evaluate
the impact of using SMPC on these critical calculations, both secure solvers
are considered with both price origins and compared to the insecure,
distributed solver N-3. The nodes selected for evaluation are the substation
node 0, the PV-prosumer node 3, the wind-prosumer node 7, and the pure load at
node 15. Across solvers and price origins, there are only minor differences
arising from the usage of SMPC, highlighting both SMPC’s viability in the LEM
and the overall scheme’s success at secure market operation.

TABLE II: Final Day-Ahead Energy Balances in CHF at several nodes for Solvers
N-3, S-2, and S-3.

Solver |  N-3 (Duals) |  S-2 (Duals) |  S-2 (Sec-Bal) |  S-3 (Duals) |  S-3 (Sec-Bal)
---|---|---|---|---|---
Node 0 | $-8.76$ | $-7.81$ | $-7.83$ | $-8.41$ | $-8.40$
Node 3 | $0.47$ | $0.43$ | $0.41$ | $0.47$ | $0.47$
Node 7 | $3.71$ | $3.84$ | $3.84$ | $3.71$ | $3.71$
Node 15 | $-0.88$ | $-0.92$ | $-0.92$ | $-0.88$ | $-0.88$

##  V Conclusion

In this paper, we presented a Local Electricity Market framework in a
distribution network with uncertain distributed energy resources. To preserve
the input privacy of the market participants, we solved the market problem
using an ADMM-based distributed optimization with data exchanges protected by
leveraging secure multi-party computation protocols. Amongst the considered
secure solvers, S-2 stands out as the recommended choice. It avoids the single
point of failure of C-1, has security guarantees via SMPC that N-2 lacks, and
achieves comparable results in fewer iterations and minutes than S-3. Note
that since the secure market protocol is highly customizable, e.g., in its
convergence thresholds, chance constraint bounds, etc., the choice of the
preferred solver may vary. Additionally, the protocol may benefit from an
increased parallelization or the use of better hardware, which could make S-3
become more suitable.

##  VI Acknowledgement

This research was supported by NCCR Automation, a National Centre of
Competence in Research, funded by the Swiss National Science Foundation (grant
number 51NF40_180545).

## References

  * [1] F. Charbonnier, T. Morstyn, and M. D. McCulloch, “Coordination of resources at the edge of the electricity grid: Systematic review and taxonomy,” _Applied Energy_ , vol. 318, p. 119188, 2022.
  * [2] T. Sousa, T. Soares, P. Pinson, F. Moret, T. Baroche, and E. Sorin, “Peer-to-peer and community-based markets: A comprehensive review,” _Renewable and Sustainable Energy Rev._ , vol. 104, pp. 367–378, 2019.
  * [3] J. Kim and Y. Dvorkin, “A P2P-dominant distribution system architecture,” _IEEE Trans. on Power Systems_ , vol. 35, pp. 2716–2725, 7 2020.
  * [4] O. Stanojev, Y. Guo, P. Aristidou, and G. Hug, “Multiple ancillary services provision by distributed energy resources in active distribution networks,” _arXiv preprint, arXiv:2202.09403v2_ , 2022.
  * [5] A. Ratha, J. Kazempour, A. Virag, and P. Pinson, “Exploring market properties of policy-based reserve procurement for power systems,” in _58th Conference on Decision and Control (CDC)_ , 2019, pp. 7498–7505.
  * [6] R. Mieth and Y. Dvorkin, “Distribution electricity pricing under uncertainty,” _IEEE Transactions on Power Systems_ , vol. 35, no. 3, pp. 2325–2338, 2020.
  * [7] M. Zugno, T. Jonsson, and P. Pinson, “Trading wind energy on the basis of probabilistic forecasts both of wind generation and of market quantities,” _Wind Energy_ , vol. 16, pp. 909–926, 2013.
  * [8] D. K. Molzahn, F. Dörfler, H. Sandberg, S. H. Low, S. Chakrabarti, R. Baldick, and J. Lavaei, “A survey of distributed optimization and control algorithms for electric power systems,” _IEEE Transactions on Smart Grid_ , vol. 8, no. 6, pp. 2941–2962, 2017.
  * [9] B. Zhang _et al._ , “An optimal and distributed method for voltage regulation in power distribution systems,” _IEEE Transactions on Power Systems_ , vol. 30, pp. 1714–1726, 7 2015.
  * [10] N. Tian, Q. Guo, H. Sun, and X. Zhou, “Fully privacy-preserving distributed optimization in power systems based on secret sharing,” _iEnergy_ , vol. 1, no. 3, pp. 351–362, 2022.
  * [11] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” _Foundations and Trends in Machine Learning_ , vol. 3, pp. 1–122, 2010.
  * [12] B. Oh, D. H. Lee, W. C. Jeong, and D. Lee, “Distributed optimal power flow for distribution system using second order cone programming and consensus alternating direction method of multipliers,” _Journal of Electrical Engineering and Technology_ , vol. 17, pp. 999–1008, 3 2022.
  * [13] U. Maurer, “Secure multi-party computation made simple,” _Discrete Applied Mathematics_ , vol. 154, pp. 370–381, 2 2006.
  * [14] T. W. Mak, F. Fioretto, L. Shi, and P. V. Hentenryck, “Privacy-preserving power system obfuscation: A bilevel optimization approach,” _IEEE Transactions on Power Systems_ , vol. 35, pp. 1627–1637, 2020.
  * [15] V. Dvorkin, F. Fioretto, P. V. Hentenryck, P. Pinson, and J. Kazempour, “Differentially private optimal power flow for distribution grids,” _IEEE Transactions on Power Systems_ , vol. 36, no. 3, pp. 2186–2196, 4 2020.
  * [16] R. Deng _et al._ , “False data injection on state estimation in power systems-attacks, impacts, and defense: A survey,” _IEEE Transactions on Industrial Informatics_ , vol. 13, pp. 411–423, 4 2017.
  * [17] D. Pozo, “Linear battery models for power systems analysis,” _Electric Power Systems Research_ , vol. 212, p. 108565, 4 2022.
  * [18] C. Gerwin, R. Mieth, and Y. Dvorkin, “Compensation mechanisms for double auctions in peer-to-peer local energy markets,” _Current Sustainable/Renewable Energy Reports_ , vol. 7, pp. 165–175, 12 2020.
  * [19] Z. Sui and H. D. Meer, “Bap: A batch and auditable privacy preservation scheme for demand response in smart grids,” _IEEE Transactions on Industrial Informatics_ , vol. 16, pp. 842–853, 2020.
  * [20] L. Kamm, “Homological classification of commitment schemes,” Master’s thesis, University of Tartu, Institute of Computer Science, 2007.
  * [21] T. Jonsson, P. Pinson, H. A. Nielsen, and H. Madsen, “Exponential smoothing approaches for prediction in real-time electricity markets,” _Energies_ , vol. 7, pp. 3710–3732, 2014.
  * [22] G. Caldarelli, “Understanding the blockchain oracle problem: A call for action,” _Information (Switzerland)_ , vol. 11, pp. 1–19, 11 2020.
  * [23] K. Dehghanpour, Z. Wang, J. Wang, Y. Yuan, and F. Bu, “A survey on state estimation techniques and challenges in smart distribution systems,” _IEEE Transactions on Smart Grid_ , vol. 10, pp. 2312–2322, 3 2019.
  * [24] Y. Wang, Q. Chen, T. Hong, and C. Kang, “Review of smart meter data analytics: Applications, methodologies, and challenges,” _IEEE Transactions on Smart Grid_ , vol. 10, pp. 3125–3148, 2019.
  * [25] Z. Galil and G. F. Italiano, “Data structures and algorithms for disjoint set union problems,” _ACM Computing Surveys_ , vol. 23, no. 3, p. 319–344, 9 1991.
  * [26] K. Zhang, S. Hanif, and T. Hamacher, “Decentralized voltage support in a competitive energy market,” in _2019 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT)_, 2019, pp. 1–5.
  * [27] B. Schoenmakers, “A simple publicly verifiable secret sharing scheme and its application to electronic voting,” _Lecture Notes in Computer Science_ , vol. 1666, pp. 148–164, 1999.
  * [28] Northern Powergrid (Northeast) plc, “Dataset (TC5): Enhanced profiling of domestic customers with solar photovoltaics,” 8 2014.
  * [29] J. Dowell, “Australian electricity market operator (AEMO): 5 minute wind power data,” 2015.
  * [30] T. Moore, “How much does the tesla powerwall solar battery cost?” Forbes Home, 3 2023.
  * [31] Energinet, “Elspot prices,” 11 2022.
  * [32] ——, “Realtime market,” 11 2022.
  * [33] S. Castano, L. Gauchia, E. Voncila, and J. Sanz, “Dynamical modeling procedure of a li-ion battery pack suitable for real-time applications,” _Energy Conversion and Management_ , vol. 92, pp. 396–405, 3 2015.
  * [34] B. Schoenmakers, “MPyC–python package for secure multiparty computation,” in _Workshop on the Theory and Practice of MPC_ , 2018.
  * [35] Z. Zhang, R. Li, and F. Li, “A novel peer-to-peer local electricity market for joint trading of energy and uncertainty,” _IEEE Transactions on Smart Grid_ , vol. 11, pp. 1205–1215, 3 2020.


# Revisiting Grammatical Error Correction Evaluation and Beyond

Peiyuan Gong1 Xuebo Liu2 Heyan Huang1 Min Zhang2
1School of Computer Science and Technology, Beijing Institute of Technology,
Beijing, China
<EMAIL_ADDRESS>
2Institute of Computing and Intelligence, Harbin Institute of Technology,
Shenzhen, China
<EMAIL_ADDRESS>Corresponding author

###### Abstract

Pretraining-based (PT-based) automatic evaluation metrics (e.g., BERTScore and
BARTScore) have been widely used in several sentence generation tasks (e.g.,
machine translation and text summarization) due to their better correlation
with human judgments over traditional overlap-based methods. Although PT-based
methods have become the de facto standard for training grammatical error
correction (GEC) systems, GEC evaluation still does not benefit from
pretrained knowledge. This paper takes the first step towards understanding
and improving GEC evaluation with pretraining. We first find that arbitrarily
applying PT-based metrics to GEC evaluation brings unsatisfactory correlation
results because of the excessive attention to inessential systems outputs
(e.g., unchanged parts). To alleviate the limitation, we propose a novel GEC
evaluation metric to achieve the best of both worlds, namely PT-M2, which only
uses PT-based metrics to score those corrected parts. Experimental results on
the CoNLL14 evaluation task show that PT-M2 significantly outperforms existing
methods, achieving a new state-of-the-art result of 0.949 Pearson correlation.
Further analysis reveals that PT-M2 is robust to evaluate competitive GEC
systems. Source code and scripts are freely available at
https://github.com/pygongnlp/PT-M2.

##  1 Introduction

Grammatical error correction (GEC) is the task that takes a sentence with
grammatical errors as input, and outputs a corrected sentence. Due to the
important role of GEC in the field of second language learning and intelligent
writing, GEC has attracted wide attention from the community (Chollampatt and
Ng, 2018a; Lewis et al., 2020; Omelianchuk et al., 2020). As a typical natural
language generation task (NLG), the common practice for GEC evaluation is to
calculate the similarity between system outputs and their corresponding
references (Dahlmeier and Ng, 2012; Napoles et al., 2015; Bryant et al.,
2017).

Task | Training | Evaluation
---|---|---
Machine Translation | $\bullet$ | $\bullet$
Text Summarization | $\bullet$ | $\bullet$
Grammatical Error Correction | $\bullet$ | $\circ$

Table 1: The use of PT-based models for model training and evaluation in
various NLG tasks. $\bullet$ means used and $\circ$ means unused. Unlike other
tasks, GEC has not used PT-based methods for evaluation.

With the rapid development of pretraining (PT) (Devlin et al., 2019; Liu et
al., 2019; Lewis et al., 2020), several NLG tasks, such as machine
translation, text summarization and GEC, have been utilizing PT-based models
to improve the model training process (Kaneko et al., 2020; Omelianchuk et
al., 2020; Tarnavskyi et al., 2022), as shown in Table 1. Furthermore, since
PT-based models can learn rich syntactic and semantic knowledge from a large
amount of unlabeled data, they are able to calculate the similarity between
two sentences more accurately. Therefore, many mainstream NLG tasks try to
develop new evaluation metrics based on PT-based models, and the new metrics
have been sufficiently validated in terms of the high consistency with human
judgments (Zhang et al., 2020; Yuan et al., 2021). However, existing GEC
systems still use traditional metrics for evaluation.

In this paper, to find the reason why GEC systems do not use PT-based metrics
for evaluation, we revisit existing GEC evaluation, comparing the traditional
overlap-based evaluation metrics with recently popular PT-based metrics.
Surprisingly, our preliminary experiment on the CoNLL14 evaluation task shows
that arbitrarily applying PT-based metrics to GEC evaluation results in a
relatively worse correlation to human judgments than the traditional metrics.
Further analysis reveals that the scoring strategies of PT-based metrics are
questionable for GEC evaluation. GEC is a local substitution task that only
partially changes from the source sentences, but PT-based metrics have to
compute the score of the whole sentence despite most words staying the same
after correction. The scores from the unchanged words bias the final sentence
score, leading to an unreliable evaluation for GEC.

To alleviate the above limitation, in this paper we propose PT-M2, a novel PT-
based GEC metric that combines the advantages of both the PT-based metrics and
traditional metrics. Unlike the PT-based metrics scoring a whole sentence,
PT-M2 only uses PT-based metrics to score the changed words that can be
extracted by the M2 metric (Dahlmeier and Ng, 2012). Experimental results on
the CoNLL14 evaluation task show that PT-M2 achieves the highest correlation
compared to traditional metrics based on two kinds of system ranking methods,
either calculating score at the corpus- or sentence-level. We also show that
PT-M2 does not heavily rely on the scale of PT-based models, even equipped
with a small-scale model can obtain satisfactory results.

Our main contributions are listed as follows:

  * •

We revisit GEC evaluation and find that the arbitrary use of PT-based metrics
results in poor correlation with human judgments. We analyze the corresponding
reasons in terms of scoring strategies for different metrics.

  * •

We propose a novel PT-based GEC metric PT-M2, which only uses PT-based metrics
to score the correction words. PT-M2 achieves a new state-of-the-art of 0.949
Pearson correlation on the CoNLL14 evaluation task.

  * •

We find that PT-M2 still performs well in evaluating high-performing GEC
systems, which is helpful for promoting GEC research and upgrading the GEC
system in the future.

Latest GEC Work | M2 | ERRANT | GLEU
---|---|---|---
Omelianchuk et al. (2020) | $\bullet$ | $\bullet$ | $\circ$
Sun and Wang (2022) | $\bullet$ | $\bullet$ | $\circ$
Lai et al. (2022) | $\bullet$ | $\bullet$ | $\circ$
Tarnavskyi et al. (2022) | $\bullet$ | $\bullet$ | $\circ$
Stahlberg and Kumar (2020) | $\bullet$ | $\bullet$ | $\bullet$
Kaneko et al. (2020) | $\bullet$ | $\bullet$ | $\bullet$
Katsumata and Komachi (2020) | $\bullet$ | $\bullet$ | $\bullet$
Parnow et al. (2021) | $\bullet$ | $\bullet$ | $\bullet$

Table 2:  GEC metrics which are used in the latest GEC works. All of the works
evaluate with traditional GEC metrics, despite using PT-based models for
training.

##  2 Related Work

###  2.1 Overlap-based GEC Metrics

As a growing number of high-performance GEC models are proposed (Liu et al.,
2021; Li et al., 2022; Zhang et al., 2022), it is important to provide
interpretable and reliable evaluation metrics to measure their quality.
Several overlap-based GEC metrics have been provided to evaluate GEC models.
As shown in Table 2, despite employing PT-based models for model training,
recently published works still use overlap-based GEC metrics (e.g., M2
Dahlmeier and Ng (2012), GLEU Napoles et al. (2015), and ERRANT Bryant et al.
(2017)) for GEC evaluation.

Dale and Kilgarriff (2011) first align source sentences and hypothesis
sentences using the Levenshtein algorithm for GEC evaluation. Dahlmeier and Ng
(2012) convert each source-hypothesis pair to an edit sequence dynamically,
extracting the corresponding edits and using the F1 measure to represent the
score of each system. Similar to Dahlmeier and Ng (2012), Felice and Briscoe
(2015) evaluate corrections at the token level, leveraging a globally optimal
alignment algorithm on source-hypothesis pairs and source-reference pairs
respectively. Napoles et al. (2015) propose a variant of BLEU that rewards
n-grams appearing in hypothesis sentences and reference sentences but not in
source sentences, penalizing n-grams in source sentences and hypothesis
sentences but not in reference sentences and averaging scores over different
references. Bryant et al. (2017) employ a linguistically-enhanced Damerau-
Levenshtein algorithm to align sentence pairs, merging parts of the alignment
and automatically classifying each edit with pre-defined rules. Choshen et al.
(2020) propose a multilingual variant of ERRANT, extracting and classifying
edits with universal dependencies. Gotou et al. (2020) focus on difficulty for
the model to correct different types of errors and use several GEC models to
compute the difficulty weight for each edit. However, a natural weakness is
that traditional overlap-based metrics cannot capture the similarity between
semantically similar words, limiting their effectiveness.

###  2.2 PT-based NLG Metrics

PT-based methods are not only used in the training stage of NLG tasks but also
in designing evaluation metrics to assess the performance of NLG models, which
can overcome the above core weakness of overlap-based metrics. Several
recently proposed PT-based metrics have dominated a large number of NLG
evaluation tasks. Lo (2019) use cross-lingual PT-based models to encode source
sentences and hypothesis sentences and compute the cosine distances with a
semantic parser. Zhao et al. (2020) compute scores for source-hypothesis pairs
with cross-lingual PT-based models, re-aligning the vector space and using a
language model to score fluency. Belouadi and Eger (2022) develop a fully
unsupervised evaluation metric that leverages pseudo-parallel pairs obtained
from fully unsupervised evaluation metrics and use pseudo reference sentences
from unsupervised translation systems. Song et al. (2021) introduce a metric
combined by BERTScore, Word Mover’s Distance and sentence semantic similarity.
Gekhman et al. (2020) extract the entities from source sentences and
hypothesis sentences via a multilingual knowledge base respectively and
measure the recall of two entitie sets. Yoshimura et al. (2020) train a BERT-
based regression model to optimize multiple sub-metrics on the GEC evaluation
dataset. Islam and Magnani (2021) leverage GPT2 Radford et al. (2018) to
measure the grammaticality for the GEC outputs.

Zhang et al. (2020) compute the similarity score for each token in hypothesis
sentences with each token in reference sentences and use greedy matching to
maximize each similarity score. Zhao et al. (2019) combine contextualized
embeddings with Word Mover’s Distance to soft align tokens from hypothesis
sentences to reference sentences. Yuan et al. (2021) introduce that a high-
quality hypothesis will be easily generated based on source sentence or
reference sentence or vice-versa, and use the generation probability to
evaluate system outputs. Sellam et al. (2020) generate a large number of
synthetic hypothesis-reference sentence pairs and pretrain BERT on several
supervision signals with a parameterized regression layer to help the model
generalization. Rei et al. (2020) propose two evaluation frameworks that
predict the quality score and minimize the distance between the “better”
hypothesis sentence and the corresponding reference sentence respectively.
Zhan et al. (2021b) evaluate different PT-based NLG metrics for machine
translation tasks. Benefiting from PT-based metrics, the correlation with
human judgments has significantly improved on NLG evaluation. In this work, we
would like to investigate the effectiveness of PT-based metrics on GEC
evaluation.

System | Sentence | Rank | BERTScore
---|---|---|---
SRC | They play the important role in our life which can not be substituted . | - | -
REF | They play an important role in our life which can not be substituted . | - | -
AMU | They play an (0.99) important role in our life which can not be replaced (0.75) . | 1 | 0.94
UFC | They play the (0.63) important role in our life which can not be substituted (0.99) . | 2 | 0.95
Table 3: Example from the CoNLL14 evaluation task. “Red Bold” denotes the
right correction whereas “Blue Non-bolded” denotes the wrong one. Although
BERTScore scores higher for the correct edit of the AMU system, the final
overall score is worse than that of the UFC system.

##  3 Revisiting Existing Metrics

This section revisits existing overlap-based GEC metrics and PT-based metrics
on the GEC evaluation task, computing the metric correlation with human
judgments, analyzing experimental results and exploring the differences among
the metrics.

###  3.1 Experimental Setup

#### Dataset Settings

We conduct experiments on the CoNLL14 evaluation task Grundkiewicz et al.
(2015). There are 1,312 source sentences, and each source sentence corresponds
to two standard reference sentences. Twelve teams have provided their system
outputs and the source sentences are used as the thirteenth system. Eight
annotators judge the quality of hypothesis sentences generated by
corresponding GEC systems. Each hypothesis sentence is scored from 1 to 5,
which means from worst to best. Two system ranking lists are respectively
generated by Expected -Wins (EW) and Trueskill (TS) algorithms.

#### Experiment Settings

The method to estimate the performance of GEC evaluation metrics is measuring
the degree of correlation with human judgments. Following Grundkiewicz et al.
(2015), we measure the correlation between metrics and human judgments based
on the system-level ranking, computing Pearson $\gamma$ and Spearman $\rho$
respectively as the final correlations. We compute the score for each system
in two settings: corpus-level and sentence-level. Given the metric M, source
sentences $\mathbf{S}$, hypothesis sentences $\mathbf{H}$ and reference
sentences $\mathbf{R}$, the first setting computes the system score based on
the whole corpus $\text{M}(\mathbf{S},\mathbf{H},\mathbf{R})$, and the last
one uses the average of the sentence-level scores
$\sum_{i}^{I}\text{M}(\mathbf{S}_{i},\mathbf{H}_{i},\mathbf{R}_{i})/I$.

#### Evaluation Metrics

We compare the following overlap-based GEC metrics and PT-based metrics:

  * •

GLEU rewards hypothesis n-grams that match reference sentences but not source
sentences and penalizes hypothesis n-grams that match source sentences but not
reference sentences Napoles et al. (2015).

  * •

M2 aligns source sentences and hypothesis sentences with Levenshtein
algorithm, dynamically choosing the alignment that maximally matches the gold
edits and extracting the system edits. Dahlmeier and Ng (2012). F0.5 is used
as the system score.

  * •

SentM2 is a variant of M2, using the average of F0.5 scores computed at the
sentence-level as the system score.

  * •

ERRANT aligns sentence pairs with a linguistic-enhanced Damerau-Levenshtein
algorithm and uses two kinds of rules to merge alignment, extract and classify
edits Bryant et al. (2017). F0.5 is used as the system score.

  * •

SentERRANT is a variant of ERRANT, using the average of F0.5 scores computed
at the sentence-level as the system score.

  * •

BERTScore is a PT-based metric, which computes the token similarity between
the sentence pairs and uses greedy matching to maximize the matching
similarity score Zhang et al. (2020).111We use the reference sentences from
the CoNLL14 evaluation task as the corpus to compute the IDF weight.

  * •

BARTScore is a PT-based metric, which converts the evaluation task to a
sequence generation task and uses the generation probability to estimate the
quality of system outputs Yuan et al. (2021).222We use the reference sentence
as the input and generate the corresponding hypothesis sentence.

Figure 1: Overview of our approach PT-M2, $\mathbf{S}$, $\mathbf{H}$, and
$\mathbf{R}$ denote the source sentence, the hypothesis sentence, and the
reference sentence, respectively. There are three core modules in our
approach: 1) Extract two edit sets from corresponding sentence pairs; 2)
Compute the score for each edit in the edit set with PT-based metrics; 3)
Apply edit scores as corresponding edit weights on the overlap-based GEC
metrics M2. “Red Bold” denotes the right corrections and their corresponding
scores whereas “Blue Non-bolded” denotes the wrong corrections and their
corresponding scores.

Metric | EW | TS
---|---|---
$\gamma$ | $\rho$ | $\gamma$ | $\rho$
GLEU | 0.701 | 0.467 | 0.750 | 0.555
ERRANT | 0.642 | 0.659 | 0.688 | 0.698
SentERRANT | 0.870 | 0.742 | 0.846 | 0.747
M2 | 0.623 | 0.687 | 0.672 | 0.720
SentM2 | 0.871 | 0.731 | 0.864 | 0.758
BARTScore | 0.172 | 0.253 | 0.173 | 0.269
BERTScore | 0.262 | 0.074 | 0.166 | -0.022

Table 4: Correlations of metrics with human judgments on the CoNLL14
evaluation task. PT-based metrics such as BERTScore and BARTScore perform
relatively worse than the overlap-based GEC metrics.

###  3.2 Preliminary Results

#### PT-based Metrics Fail

The results of the existing metrics are shown in Table 4. The advantages are
different between GLEU and M2 as GLEU is better than M2 on Pearson and worse
on Spearman Chollampatt and Ng (2018b). We introduce ERRANT to compute its
correlation with human judgments since ERRANT can extract and classify edits
automatically, which has been used as the standard metric in GEC Bryant et al.
(2017). Without using the gold edits annotated by humans, ERRANT can still get
a relatively high correlation. We also find that computing the system score at
the sentence-level for M2 and ERRANT can get a higher correlation than
computing at the corpus-level, which is the same as in Napoles et al. (2016).

Besides, we observe that even though the PT-based metrics such as BERTScore
and BARTScore have dominated in the evaluation of multiple NLG tasks Scialom
and Hill (2021), they correlate much lower than overlap-based GEC metrics on
the GEC evaluation task, whether for Pearson or Spearman, and are even
negatively correlated with human judgments. So why are PT-based metrics not
suitable to evaluate GEC systems? What are the differences between the PT-
based metrics and overlap-based GEC metrics?

#### Discussion

This part aims to answer the above questions. As shown in Table 3, to figure
out why PT-based metrics are not suitable for GEC evaluation, we analyze a
representative from the CoNLL14 evaluation task. For the two edits provided by
the AMU system, even though the change of the correct edit score is larger
than that of the wrong correction edit, the AMU system score provided by
BERTScore is lower than that of the UFC system, which leads to the result
being different with human annotation. A possible reason is that BERTScore
computes scores for all the tokens in the sentence, a large percentage of the
score for the unchanged part affects the trend of the overall score. When an
edit is applied, not only the corresponding edit score is changed, but the
scores of surrounding tokens are also affected, causing BERTScore to fail to
make a correct assessment. The excessive attention to unchanged words
potentially biases the final score of PT-based metrics.

##  4 PT-M2: The Best of Both Worlds

###  4.1 Motivation

Based on the above findings, to inject pretrained knowledge into GEC
evaluation metrics, we propose PT-M2, which takes advantage of both PT-based
metrics and overlap-based GEC metrics. PT-M2 computes edit scores with PT-
based metrics. Without directly using PT-based metrics to score hypothesis-
reference sentence pairs, we use them at the edit-level to compute a score for
each edit. As shown in Figure 1, we first extract the system edit set and gold
edit set from the source-hypothesis sentence pair and source-reference
sentence pair respectively. We then compute the score of each edit with PT-
based metrics as the scorer. Lastly we apply edit scores as the edit weights
on the M2.

###  4.2 Implementation

#### Edit Extraction

Given a source sentence $S$, a hypothesis sentence $H$ and a reference
sentence $R$, the first step we have to do is to extract the system edit set
$E$ and gold edit set $G$ from the source-hypothesis sentence pair $(S,H)$ and
source-reference sentence pair $(S,R)$ respectively Dahlmeier and Ng (2012);
Bryant et al. (2017). As shown in Figure 1, each edit is composed of edit
operation, start index, end index and correct tokens. We use the edit
extraction module from M2 to extract both two edit sets.333Due to the fact
that the CoNLL14 evaluation task has provided the gold edit set, we only need
to extract the system edit set by M2. The intersection of the system edit set
and gold edit set represents all of the correct edits provided by
corresponding system.

Type | Metric | PT Model | EW (Corpus) | TS (Corpus) | EW (Sentence) | TS (Sentence)
---|---|---|---|---|---|---
$\gamma$ | $\rho$ | $\gamma$ | $\rho$ | $\gamma$ | $\rho$ | $\gamma$ | $\rho$
Overlap | GLEU | - | 0.701 | 0.467 | 0.750 | 0.555 | 0.784 | 0.720 | 0.828 | 0.775
ERRANT | - | 0.642 | 0.659 | 0.688 | 0.698 | 0.870 | 0.742 | 0.846 | 0.747
M2 | - | 0.623 | 0.687 | 0.672 | 0.720 | 0.871 | 0.731 | 0.864 | 0.758
PT | BARTScore | BART | - | - | - | - | 0.172 | 0.253 | 0.173 | 0.269
BERTScore | BERT | - | - | - | - | 0.262 | 0.074 | 0.166 | -0.022
Ours | PT-ERRANT | BART | 0.681 | 0.797 | 0.727 | 0.841 | 0.905 | 0.786 | 0.897 | 0.824
PT-ERRANT | BERT | 0.705 | 0.780 | 0.745 | 0.797 | 0.941 | 0.879 | 0.917 | 0.846
2-11 | PT-M2 | BART | 0.667 | 0.775 | 0.715 | 0.813 | 0.898 | 0.813 | 0.901 | 0.841
| PT-M2 | BERT | 0.693 | 0.758 | 0.737 | 0.769 | 0.949 | 0.907 | 0.938 | 0.874

Table 5: Correlations of metrics with human judgments on the CoNLL14
evaluation task. Our proposed PT-M2 and PT-ERRANT methods correlate better
with human judgments on both the corpus- and sentence-level. We highlight the
highest score in bold and the second-highest score with underlines.

#### Edit Score

Without treating each edit equally, we propose a novel method to score each
edit, employing PT-based metrics (e.g., BERTScore and BARTScore) as the edit
scorer (PTScore). As shown in Figure 1, we first build an edit set $U$, which
is the union of the system edit set $E$ and gold edit set $G$. Then, we
compute the score of each edit $u$ in the edit set $U$ and obtain an edit
score set $W$, in which $w_{u}$ is the score of the edit $u$. To achieve this
goal, the first step we applied is to use each edit in the edit set $U$ to
generate a partially correct version $S^{\prime}$ of the source sentence $S$.
Then we use PTScore to compute the scores of the sentence pair $(S,R)$ and
$(S^{\prime},R)$ respectively, the more similarity between the sentence pair
which input to PTScore, the higher the score that PTScore gives. The score
differences are used to measure whether the edit is beneficial or not:

$w=\text{PTScore}(S^{\prime},R)-\text{PTScore}(S,R)$ (1)

$w$ > 0 means the edit is helpful for correction, which shows it is a correct
edit, otherwise is a wrong correction edit. We use the absolute value
$\left|w\right|$ as the edit score. The larger the $\left|w\right|$ is, the
edit has a greater impact on the sentence, whether beneficial or harmful.

#### Final Score

As shown in Figure 1, given the system edit set $E$, the gold edit set $G$,
and the edit score set $W$, we treat each edit score as the corresponding edit
weight and apply edit weight on each edit to compute precision, recall and
F0.5 measure respectively:

$P=\frac{\sum_{c\in E\cap G}w_{c}}{\sum_{e\in E}w_{e}}$ (2)
$R=\frac{\sum_{c\in E\cap G}w_{c}}{\sum_{g\in G}w_{g}}$ (3)

$F_{\beta}=(1+{\beta}^{2})\cdot\frac{P\cdot R}{({\beta}^{2}\cdot P)+R}$ (4)

A $\beta$ value of 0.5 is used in PT-M2 follows vanilla M2. To compute the
system score, we compute F0.5 based on the whole corpus to represent the
corpus-level PT-M2 and use the average of F0.5 score for each sentence to
represent the sentence-level PT-M2 Napoles et al. (2016). Besides, since our
approach is transparent to the type of edits, it can be directly applied to
other overlap-based GEC metrics, such as ERRANT.

##  5 Experiment

PT-M2 can combine the advantages of both PT-based metrics and overlap-based
GEC metrics. In PT-M2, PT-based metrics only compute edit scores that can
reduce the impact of unchanged part on the final score, paying more attention
to the changes in the source sentences and editing scores are applied as
corresponding edit weights on the M2.

To demonstrate the effectiveness of our approach, we use PT-M2 to evaluate GEC
system outputs and compute the correlation with human judgments on the CoNLL14
evaluation task Grundkiewicz et al. (2015). We choose Pearson $\gamma$ and
Spearman $\rho$ to measure the correlations. We use PT-M2 to compute both the
system score at corpus- and sentence-level Napoles et al. (2016). We adopt two
well-known unsupervised PT-based metrics BERTScore and BARTScore to score
edits, which can be used in multiple domains, tasks and languages Scialom and
Hill (2021). The other metrics have been introduced in Section 3.1.

Rank | Metric | Human
---|---|---
BERTScore | M2 | SentM2 | PT-M2 | EW | TS
1 | INPUT ($\Uparrow$9) | CAMB ($\Uparrow$1) | CUUI ($\Uparrow$3) | AMU ($\checkmark$0) | AMU | AMU
2 | UFC ($\Uparrow$4) | CUUI ($\Uparrow$2) | AMU ($\Downarrow$1) | CUUI ($\Uparrow$2) | RAC | CAMB
3 | IITB ($\Uparrow$6) | AMU ($\Downarrow$2) | CAMB ($\checkmark$0) | CAMB ($\checkmark$0) | CAMB | RAC
4 | RAC ($\Downarrow$1) | POST ($\Uparrow$1) | POST ($\Uparrow$1) | RAC ($\Downarrow$1) | CUUI | CUUI
5 | SJTU ($\Uparrow$5) | NTHU ($\Uparrow$7) | NTHU ($\Uparrow$7) | POST ($\checkmark$0) | POST | POST
6 | PKU ($\checkmark$0) | RAC ($\Downarrow$3) | RAC ($\Downarrow$3) | PKU ($\checkmark$0) | UFC | PKU
7 | AMU ($\Downarrow$6) | UMC ($\checkmark$0) | PKU ($\checkmark$0) | UFC ($\Downarrow$1) | PKU | UMC
8 | POST ($\Downarrow$3) | PKU ($\Downarrow$1) | UMC ($\checkmark$0) | SJTU ($\Uparrow$2) | UMC | UFC
9 | CUUI ($\Downarrow$5) | SJTU ($\Uparrow$1) | SJTU ($\Uparrow$1) | IITB ($\checkmark$0) | IITB | IITB
10 | UMC ($\Downarrow$2) | UFC ($\Downarrow$2) | UFC ($\Downarrow$2) | NTHU ($\Uparrow$2) | SJTU | INPUT
11 | IPN ($\Uparrow$2) | IPN ($\Uparrow$2) | IITB ($\Downarrow$2) | INPUT ($\checkmark$0) | INPUT | SJTU
12 | CAMB ($\Downarrow$9) | IITB ($\Downarrow$3) | INPUT ($\Downarrow$1) | UMC ($\Downarrow$4) | NTHU | NTHU
13 | NTHU ($\Downarrow$1) | INPUT ($\Downarrow$2) | IPN ($\checkmark$0) | IPN ($\checkmark$0) | IPN | IPN
$\Delta$ | 53 | 27 | 21 | 12 | - | -

Table 6: System rankings by different metrics. $\Uparrow$/$\Downarrow$ denotes
that the rank given by the evaluation metric is higher/lower than human
judgments, and $\checkmark$ denotes that the given rank is equal to human
ranking. The lowest rank difference in each rank is highlighted in bold. PT-M2
successfully ranks the best system that the other metrics fail. Besides, it
also shows the lowest rank difference ($\Delta$).

###  5.1 Main Results

Table 5 reports the correlations of overlap-based GEC metrics, PT-based
metrics, and our PT-based GEC metric PT-M2. PT-M2 aligns better with human
judgments compared to traditional overlap-based GEC metrics, either computed
at corpus- or sentence-level Napoles et al. (2016). Besides, for the two human
rankings proposed by EW and TS, PT-M2 makes a significant improvement on both
Pearson and Spearman correlations. Although BERTScore and BARTScore are
computed in different ways Sai et al. (2022), we find that our approach uses
either of them as the edit scorer can get a similarly high correlation.

We also test a variant of our approach, namely PT-ERRANT, with the same
operations as in PT-M2. Compared to ERRANT, PT-ERRANT gets higher correlation
with human judgments. We find that comparing PT-M2 and PT-ERRANT, PT-M2
correlates better at the sentence-level while PT-ERRANT aligns better at the
corpus-level. We also test different values of $\beta$ when computing Fβ as
the final score. The results show that PT-M2 correlates more stable and is
consistently better than M2. It is worth mentioning that the sentence-level
PT-M2 with BERTScore as the scorer gets the highest correlation among all of
the metrics we have experimented with. Therefore, we treat it as the main
version of PT-M2 and use the sentence-level PT-M2 in the subsequent
experiments.

Models | Size | EW | TS
---|---|---|---
$\gamma$ | $\rho$ | $\gamma$ | $\rho$
BART | Small | 0.883 | 0.743 | 0.869 | 0.740
Base | 0.898 | 0.813 | 0.901 | 0.841
Large | 0.899 | 0.813 | 0.906 | 0.841
BERT | Small | 0.953 | 0.923 | 0.941 | 0.890
Base | 0.949 | 0.907 | 0.938 | 0.874
Large | 0.945 | 0.918 | 0.931 | 0.879

Table 7: Comparison between different PT-based models. There are no
significant differences between models of different sizes. Figure 2: Pearson
scores of Top-$K$ system based on the EW ranking list. PT-M2 is highly
correlated with human judgments especially when all the systems are
competitive (i.e., $K\leq 6$).

###  5.2 Analysis

#### Effect of PT-based models

To demonstrate the generality and generalizability of our approach, we use
different sizes of PT-based metrics as the edit scorer to verify if the
correlation of our approach is stable. As shown in Table 7, we experiment with
PT-M2 at the sentence-level, employing BERTScore and BARTScore in three
different sizes, from small to large respectively. Even if we use the
distilled version of BERTScore Sanh et al. (2019), PT-M2 also aligns much
better with human judgments than M2 and gets the similar correlation with the
base and even the large size of models. So we can use the smaller size of PT-
based metrics to accelerate the computation of GEC evaluation, with a trivial
performance drop.

#### System Ranking

Table 5 presents the ranking results. Compared to other metrics, PT-M2 shows
the lowest rank differences with human judgments. PT-M2 successfully ranks the
best system (AMU) while the other metrics fail. Compared to BERTScore, PT-M2
successfully ranks the INPUT system. Meanwhile, PT-M2 gets a relatively better
ranking result for the NTHU system than M2 and SentM2. This confirms the
effectiveness of PT-M2.

#### Effect of Top-$K$ Systems

To demonstrate the effectiveness of our approach, we evaluate high-performing
systems with our approach to observe the change of correlations among
different metrics Zhan et al. (2021a). Figure 2 compares the Pearson
correlation of the top-$K$ systems. When $K$ decreases, the correlations of M2
and SentM2 are lower than before correspondingly. Meanwhile, we find that the
correlation of our approach PT-M2 computed at the sentence-level drops much
slower than M2 and SentM2. Further, when $K$ is lower than 6, the Pearson
correlations of M2 and SentM2 drop sharply while PT-M2 does not act the same
way under the circumstances. More specifically, as the system count drops from
13 to 4, the Pearson correlations of M2 and SentM2 down three times or even
more. In contrast, the decline of PT-M2 correlation is not obvious, which
demonstrates the effectiveness of applying edit scores computed by PT-based
metrics as corresponding edit weights on the overlap-based GEC metrics.

System | Sentence | Rank | M2 | PT-M2
---|---|---|---|---
SRC | It is also entire incorrect to fault social media alone for the lack of interpersonal skill . | - | - |
REF | It is also entirely incorrect to fault social media alone for the lack of interpersonal skills . | - | - |
NTHU | It is also entirely incorrect to fault social media alone for lack of interpersonal skills . | 1 | 0.71 | 0.88
PKU | It is also entire incorrect to fault social media alone for the lack of interpersonal skills . | 2 | 0.83 | 0.44

Table 8: Example from the CoNLL14 evaluation task. “Red Bold” denotes the
right correction whereas “Blue Non-bolded” denotes the wrong one. The ranking
of PT-M2 is more in line with human judgments. Figure 3: Effectiveness of the
proposed edit score. Reversing the edit score harms the correlation.

#### Effect of Edit Score

We carry out a set of comparative experiments to demonstrate the necessity of
computing edit weights. In our approach PT-M2, we use PT-based metrics such as
BERTScore and BARTScore to compute edit scores. The more important the edit,
the higher score we provide. In the comparative experiments, we use the
inverse of each edit score computed by PT-based metrics as the corresponding
edit weight, $w=1/|\text{PTScore}(S^{\prime},R)-\text{PTScore}(S,R)|$, to
demonstrate the importance of edit weights on overlap-based GEC metrics. For
convenience, we use PT-M2 (inverse) as the above comparative approach.

Figure 3 the comparison between the Pearson and Spearman correlations of the
three metrics. After inversing edit weights computed by PT-based metrics, both
the correlations measured by Pearson and Spearman are lower than SentM2 for a
large. Based on the above results, we demonstrate the effectiveness and
advantages of employing PT-based metrics to directly compute edit weights.

#### A Case Study

As shown in Table 8, to explain why our approach PT-M2 correlates better with
human judgments than M2, we present an example from the CoNLL14 evaluation
task Grundkiewicz et al. (2015). The human ranking shows that the NTHU system
score is higher than that of the PKU system. M2 provides the wrong ranking
result while PT-M2 aligns the same as human judgments. In this example, PT-M2
measures the gold edit only provided by the NTHU system as the highest score,
which means it is much more important to predict this edit. Meanwhile, PT-M2
sets the wrong correction edit supplied by the NTHU system with a lower score,
which influences the source sentence a little. PT-M2 successfully scores
higher for the NTHU system.

##  6 Conclusion and Future Work

In this work, we revisit GEC metrics and explore the feasibility of employing
PT-based metrics to evaluate GEC systems. Compared to overlap-based GEC
metrics, PT-based metrics correlate worse with human judgments. The reason we
find is that PT-based metrics compute score for each token in ungrammatical
sentences, however, the scores of the unchanged part account for a large
proportion of the final score, leading to an inaccurate evaluation result. To
leverage pretrained knowledge in GEC evaluation, we propose a novel PT-based
GEC metric PT-M2, employing PT-based metrics to score edits extracted from
sentence pairs and applying edit scores as corresponding edit weights on the
M2. Experiments demonstrate that PT-M2 gets the highest correlation on the
CoNLL14 evaluation task, achieving a new state-of-the-art result of 0.949
Pearson correlation. Besides, further analysis shows that PT-M2 is competent
to evaluate high-performing GEC systems.

In the future, we would like to test the effectiveness of PT-M2 in the GEC
evaluation tasks of other languages (e.g., Chinese). It is also worthwhile to
explore the benefits of PT-M2 as a signal for reinforcement learning to train
better GEC systems.

## Limitations

Our PT-based GEC metric PT-M2 has got the highest correlation with human
judgments on GEC evaluation tasks. A reason that might limit the widespread
use of PT-based metrics is that the calculation speed is slower than that of
the traditional overlap-based GEC metrics. As PT-M2 uses PT-based metrics to
score each edit, it takes nearly 2 minutes to calculate a system score based
on a single NVIDIA GTX 3080TI card, which is slower than the calculation speed
of the traditional M2 score that costs nearly 10 seconds. Therefore, there is
still room for improvement in calculation speed, and we will continue
developing parallel computing to speed it up in the future.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of
China (Grant No. 62206076) and Shenzhen College Stability Support Plan (Grant
No. GXWD20220811173340003 and GXWD20220817123150002). The authors would like
to thank the anonymous reviewers for their insightful suggestions.

## References

  * Belouadi and Eger (2022) Jonas Belouadi and Steffen Eger. 2022.  Uscore: An effective approach to fully unsupervised evaluation metrics for machine translation.  _arXiv preprint arXiv:2202.10062_.
  * Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017.  Automatic annotation and evaluation of error types for grammatical error correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  * Chollampatt and Ng (2018a) Shamil Chollampatt and Hwee Tou Ng. 2018a.  A multilayer convolutional encoder-decoder neural network for grammatical error correction.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 5755–5762. AAAI Press.
  * Chollampatt and Ng (2018b) Shamil Chollampatt and Hwee Tou Ng. 2018b.  A reassessment of reference-based grammatical error correction metrics.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2730–2741, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Choshen et al. (2020) Leshem Choshen, Dmitry Nikolaev, Yevgeni Berzak, and Omri Abend. 2020.  Classifying syntactic errors in learner language.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 97–107, Online. Association for Computational Linguistics.
  * Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012.  Better evaluation for grammatical error correction.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 568–572, Montréal, Canada. Association for Computational Linguistics.
  * Dale and Kilgarriff (2011) Robert Dale and Adam Kilgarriff. 2011.  Helping our own: The HOO 2011 pilot shared task.  In _Proceedings of the 13th European Workshop on Natural Language Generation_ , pages 242–249, Nancy, France. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Felice and Briscoe (2015) Mariano Felice and Ted Briscoe. 2015.  Towards a standard evaluation method for grammatical error detection and correction.  In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 578–587, Denver, Colorado. Association for Computational Linguistics.
  * Gekhman et al. (2020) Zorik Gekhman, Roee Aharoni, Genady Beryozkin, Markus Freitag, and Wolfgang Macherey. 2020.  Kobe: Knowledge-based machine translation evaluation.  _arXiv preprint arXiv:2009.11027_.
  * Gotou et al. (2020) Takumi Gotou, Ryo Nagata, Masato Mita, and Kazuaki Hanawa. 2020.  Taking the correction difficulty into account in grammatical error correction evaluation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2085–2095, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Grundkiewicz et al. (2015) Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Edward Gillian. 2015.  Human evaluation of grammatical error correction systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 461–470, Lisbon, Portugal. Association for Computational Linguistics.
  * Islam and Magnani (2021) Md Asadul Islam and Enrico Magnani. 2021.  Is this the end of the gold standard? a straightforward reference-less grammatical error correction metric.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3009–3015.
  * Kaneko et al. (2020) Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020.  Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4248–4254, Online. Association for Computational Linguistics.
  * Katsumata and Komachi (2020) Satoru Katsumata and Mamoru Komachi. 2020.  Stronger baselines for grammatical error correction using pretrained encoder-decoder model.  _arXiv preprint arXiv:2005.11849_.
  * Lai et al. (2022) Shaopeng Lai, Qingyu Zhou, Jiali Zeng, Zhongli Li, Chao Li, Yunbo Cao, and Jinsong Su. 2022.  Type-driven multi-turn corrections for grammatical error correction.  _arXiv preprint arXiv:2203.09136_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2022) Bei Li, Quan Du, Tao Zhou, Yi Jing, Shuhan Zhou, Xin Zeng, Tong Xiao, JingBo Zhu, Xuebo Liu, and Min Zhang. 2022.  ODE transformer: An ordinary differential equation-inspired model for sequence generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8335–8351, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2021) Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, and Zhaopeng Tu. 2021.  Understanding and improving encoder layer fusion in sequence-to-sequence learning.  In _International Conference on Learning Representations_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lo (2019) Chi-kiu Lo. 2019.  Yisi-a unified semantic mt quality evaluation and estimation metric for languages with different levels of available resources.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 507–513.
  * Napoles et al. (2015) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015.  Ground truth for grammatical error correction metrics.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 588–593, Beijing, China. Association for Computational Linguistics.
  * Napoles et al. (2016) Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2016.  There’s no comparison: Reference-less evaluation metrics in grammatical error correction.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2109–2115, Austin, Texas. Association for Computational Linguistics.
  * Omelianchuk et al. (2020) Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020.  GECToR – grammatical error correction: Tag, not rewrite.  In _Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 163–170, Seattle, WA, USA → Online. Association for Computational Linguistics.
  * Parnow et al. (2021) Kevin Parnow, Zuchao Li, and Hai Zhao. 2021.  Grammatical error correction as GAN-like sequence labeling.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3284–3290, Online. Association for Computational Linguistics.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Sai et al. (2022) Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra. 2022.  A survey of evaluation metrics used for nlg systems.  _ACM Computing Surveys (CSUR)_ , 55(2):1–39.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Scialom and Hill (2021) Thomas Scialom and Felix Hill. 2021.  Beametrics: A benchmark for language generation evaluation evaluation.  _arXiv preprint arXiv:2110.09147_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Song et al. (2021) Yurun Song, Junchen Zhao, and Lucia Specia. 2021.  Sentsim: Crosslingual semantic evaluation of machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3143–3156.
  * Stahlberg and Kumar (2020) Felix Stahlberg and Shankar Kumar. 2020.  Seq2Edits: Sequence transduction using span-level edit operations.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5147–5159, Online. Association for Computational Linguistics.
  * Sun and Wang (2022) Xin Sun and Houfeng Wang. 2022.  Adjusting the precision-recall trade-off with align-and-predict decoding for grammatical error correction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 686–693.
  * Tarnavskyi et al. (2022) Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022.  Ensembling and knowledge distilling of large sequence taggers for grammatical error correction.  _arXiv preprint arXiv:2203.13064_.
  * Yoshimura et al. (2020) Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Kajiwara, and Mamoru Komachi. 2020.  SOME: Reference-less sub-metrics optimized for manual evaluations of grammatical error correction.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6516–6522, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  _Advances in Neural Information Processing Systems_ , 34.
  * Zhan et al. (2021a) Runzhe Zhan, Xuebo Liu, Derek F. Wong, and Lidia S. Chao. 2021a.  Difficulty-aware machine translation evaluation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 26–32, Online. Association for Computational Linguistics.
  * Zhan et al. (2021b) Runzhe Zhan, Xuebo Liu, Derek F. Wong, and Lidia S. Chao. 2021b.  Variance-aware machine translation test sets.  In _Thirty-fifth Conference on Neural Information Processing Systems, Datasets and Benchmarks Track_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zhang et al. (2022) Zheng Zhang, Liang Ding, Dazhao Cheng, Xuebo Liu, Min Zhang, and Dacheng Tao. 2022\.  Bliss: Robust sequence-to-sequence learning via self-supervised input representation.  _ArXiv_ , abs/2204.07837.
  * Zhao et al. (2020) Wei Zhao, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020.  On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation.  _arXiv preprint arXiv:2005.01196_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Appendix

Data: source sentences $\mathbf{S}$, hypothesis sentences $\mathbf{H}$,
reference sentences $\mathbf{R}$, gold edit sets $\mathbf{GOLD}$

Result: F-score F0.5

1 Function  _ComputeScore(_$S$ , $R$, $E$_)_:

2 initialize an empty dict $W$ ;

3 for  _$i$ $\leftarrow$ $1$ to Len(_$E$_)_ do

$e$ $\leftarrow$ $E_{i}$ ; // the $i$th edit in the edit set $E$

$S^{\prime}$ $\leftarrow$ Correct(_$S$ , $e$_) ; // apply edit $e$ to correct
the source sentence $S$

$W_{e}$ $\leftarrow$ |PTScore(_$S^{\prime}$ , $R$_) \- PTScore(_$S$ , $R$_)| ;
// compute the difference of PTScore between ($S^{\prime}$, $R$) and ($S$,
$R$)

4

5 end for

6 return $W$;

7

8

$F_{0.5}$ $\leftarrow$ $0$ ; // initialize $F_{0.5}$

9 for  _$i$ $\leftarrow$ $1$ to Len(_$\mathbf{S}$_)_ do

$f_{max}$ $\leftarrow$ $-1$ ; // initialize $f_{max}$

10 $S$, $H$, $R$, $GOLD$, $\leftarrow$ $\mathbf{S}_{i}$, $\mathbf{H}_{i}$,
$\mathbf{R}_{i}$, $\mathbf{GOLD}_{i}$;

11 for  _$j\leftarrow 1$ to Len(_$R$_)_ do

$G$ $\leftarrow$ $GOLD_{j}$ ; // the jth gold edit set of $S$

$E$ $\leftarrow$ ExtractEdit(_$S$ , $H$, $G$_); // extract system edit

$C$ $\leftarrow$ $E\cap G$ ; // choose the correct edit that system provides

$W$ $\leftarrow$ ComputeScore(_$S$ , $R$, $E\cup G$_); // compute the score of
each edit

$p$ $\leftarrow$ $\sum_{c\in C}W_{c}/\sum_{e\in E}W_{e}$ ; // compute the
precision score

$r$ $\leftarrow$ $\sum_{c\in C}W_{c}/\sum_{g\in G}W_{g}$ ; // compute the
recall score

$f$ $\leftarrow$ $(1+0.5^{2})\cdot p\cdot r/(0.5^{2}\cdot p+r)$ ; // compute
the f score

12 $f_{max}$ $\leftarrow$ Max(_$f_{max}$ , $f$_)

13 end for

14 $F_{0.5}$ $\leftarrow$ $F_{0.5}+f_{max}$

15 end for

$F_{0.5}$ $\leftarrow$ $F_{0.5}$ $/$ Len(_$\mathbf{S}$_)

Algorithm 1 Calculation Procedure of PT-M2

###  A.1 Algorithm

Algorithm 1 illustrates the calculation procedure of our PT-based GEC metric
PT-M2, showing the whole process of how to compute the sentence-level PT-M2.

###  A.2 Command-line Interface

We introduce how to compute PT-M2 and its variant PT-ERRANT in different
settings with our code: python evaluate.py –base [m2|sentm2|errant|senterrant]
–scorer [self|bertscore|bartscore] –source <src_file> –hypothesis <hyp_file>
–reference <ref_file> –output <out_file>