# A Comparative Study of Quality Evaluation Methods for Text Summarization

Huyen Nguyen, Haihua Chen, Lavanya Pobbathi, Junhua Ding*
University of North Texas, Denton, TX
<EMAIL_ADDRESS><EMAIL_ADDRESS>
lavanyapobbathi<EMAIL_ADDRESS>
Correspondence<EMAIL_ADDRESS>
###### Abstract

Evaluating text summarization has been a challenging task in natural language
processing (NLP). Automatic metrics which heavily rely on reference summaries
are not suitable in many situations, while human evaluation is time-consuming
and labor-intensive. To bridge this gap, this paper proposes a novel method
based on large language models (LLMs) for evaluating text summarization. We
also conducts a comparative study on eight automatic metrics, human
evaluation, and our proposed LLM-based method. Seven different types of state-
of-the-art (SOTA) summarization models were evaluated. We perform extensive
experiments and analysis on datasets with patent documents. Our results show
that LLMs evaluation aligns closely with human evaluation, while widely-used
automatic metrics such as ROUGE-2, BERTScore, and SummaC do not and also lack
consistency. Based on the empirical comparison, we propose a LLM-powered
framework for automatically evaluating and improving text summarization, which
is beneficial and could attract wide attention among the community.

{̇˙

A Comparative Study of Quality Evaluation Methods for Text Summarization

Huyen Nguyen, Haihua Chen, Lavanya Pobbathi, Junhua Ding* University of North
Texas, Denton, TX<EMAIL_ADDRESS><EMAIL_ADDRESS>lavanyapobbathi<EMAIL_ADDRESS>Correspondence:
<EMAIL_ADDRESS>

##  1 Introduction

Text summarization is the process of producing a concise and coherent summary
while preserving key information and meaning of the source text Allahyari et
al. (2017). This technique is widely used in various fields; for example, it
is commonly used to summarize scientific, medical, and legal documents, as it
enables users to quickly grasp key points of lengthy texts and efficiently
access relevant information.

There are two major approaches to automatic text summarization: extractive and
abstractive. Extractive summarization involves selecting important sentences
or phrases from the original document. Extractive summarization is considered
to be faster, simpler and more accurate because it retains authentic sentences
of the source documents. However, it is less fluent and less coherent than
abstractive summarization El-Kassas et al. (2021). On the other hand, the
abstractive summary generates the summary with sentences that are different
from those in the original text while not changing the central facts and
ideas.

With the remarkable achievements of pretrained language models (PLMs) and
natural language generation, recent research has shifted gears from extractive
to abstractive summarization. Nevertheless, the abstractive summarization
still remains a challenging task since models suffer from hallucinations Cao
et al. (2018); Maynez et al. (2020) and the generated summaries do not align
with human expectations He et al. (2020). A preliminary study of text
summarization techniques indicates that nearly 30% summaries generated by
existing SOTA neural abstractive summarization are unfaithful to original
documents Cao et al. (2018).

Summarization evaluation can be divided into reference-based and reference-
free. Reference-based metrics are based on the matching between the generated
summary and the reference summary, whereas the reference-free is based on the
source document to evaluate the generated summary. Evaluation of summaries can
be done either automatically or manually. The automatic method is fast,
inexpensive, and can handle large volumes of data without human intervention.
They can be categorized into three groups: text overlapping (e.g., ROUGE Chin-
Yew (2004); Ganesan (2018) and BLEU Papineni et al. (2002)), vector-space
distance (e.g., BERTScore Zhang et al. (2019); Kieuvongngam et al. (2020),
MoverScore Zhao et al. (2019)), and NLP task-based to measure the consistency
between the generated summary and the reference (e.g., SummaC Laban et al.
(2022), QuestEval Scialom et al. (2021)).

Summarization evaluation is still a challenging task as there are no ideal
evaluation methods. Some studies demonstrate that automatic evaluation metrics
such as BLEU, ROUGE, and BERTScore are not suitable for the automatic
evaluation of summaries Sun et al. (2022); Sulem et al. (2018); Reiter (2018);
Schluter (2017). Most of these automatic metrics, especially text overlapping
and vector-space similarity measurements, are originally reference-based;
therefore, they may not be appropriate to use as reference-free due to the
incompatible length and information compression.

On the other hand, the human evaluation is considered more reliable and
trustworthy. It is still the preferred choice for evaluating summaries Deutsch
et al. (2021). The process starts by sampling a small set of about 30-100
generated summaries. The recruited evaluators are asked to score the generated
summary on a Likert scale such as 1-5, or 1-7 on one or more evaluation
dimensions described above. Despite the advantages of human evaluation,
conducting this evaluation method is time-consuming, and labor-intensive, so
it is infeasible to use in model development.

Studies that access the evaluation methods reach inconsistent conclusions. For
example, Graham (2015) claim that text-overlap metrics achieve the strongest
correlation with human assessment. Meanwhile, recent studies demonstrate that
metrics such as BLEU, ROUGE, and BERTScore are not suitable for the automatic
evaluation of summaries Sun et al. (2022); Sulem et al. (2018); Schluter
(2017); Reiter (2018). Therefore, there is a need to revisit existing
summarization evaluation methods.

In this work, we conduct a comparative study to reevaluate existing automatic
metrics to evaluate abstractive summarization. Besides, we assess the ability
of LLMs to perform as an evaluation agent. We further propose a framework to
iteratively improve the quality of LLMs-generated summaries. Our contributions
can be summarized as follows:

  * •

We conduct a comprehensive evaluation of the latest off-the-shelf PLMs and
LLMs for the patent document summarization, using both automatic and human
evaluation methods.

  * •

We re-evaluate existing automatic metrics that are widely-used for evaluating
text summarization.

  * •

We propose a framework based on LLMs for automatically evaluating and
improving summarization.

##  2 Related Work

###  2.1 Text Summarization

Most current abstractive models rely on neural networks based sequence-to-
sequence learning Bahdanau et al. (2015); Vaswani et al. (2017). Seq2seq
summarization can be summarized into two main types of frameworks, RNN
encoder-decoder Bahdanau et al. (2015) and Transformer encoder-decoder Vaswani
et al. (2017).

Nallapati et al. (2016) introduced one of the first RNN-based summarization
models, utilizing a bidirectional RNN encoder enriched with POS tags and TFIDF
feature embeddings, and a unidirectional RNN decoder with an attention
mechanism. However, these models often faced issues with out-of-vocabulary
(OOV) words and word repetition. To address these problems, See et al. (2017)
proposed a pointer-generator network, which combines the base seq2seq model
with a pointer network that decides whether to generate a word from the
vocabulary or copy it from the input sequence. Additionally, a coverage
mechanism was implemented to track and prevent repetition See et al. (2017);
Nallapati et al. (2016).

Abstractive summarization using Transformer encoder-decoder framework has
rapidly advanced in recent years. Transformers with self-attention layers
allow parallelization learning, solving the vanishing or explosion gradient of
standard RNNs. It achieves SOTA performance in machine translation Vaswani et
al. (2017). Given this success, this approach is promising in abstractive
summarization. Currently, encoder-decoder Transformer models like BART Lewis
et al. (2020) and PEGASUS Zhang et al. (2020) have achieved SOTA summarization
results on short text. However, BART’s and PEGARUS’s maximum input length
limit at 1024 tokens, making it unsuitable for summarizing long text.

The major limitation of transformer models is the complexity of quadratic
self-attention that grows rapidly with sequence length Zaheer et al. (2020).
This has significantly impeded their effectiveness in summarizing long
documents. The simplest approach is truncating the document from the head or
tail to produce a short valid input. However, Meng et al. (2021) proves that
Transformers with this naive method is even worse than many unsupervised
algorithms, such as TextRank, LSA, etc., for long text summarization. Models
like Longformer Beltagy et al. (2020) and BigBird Zaheer et al. (2020)
incorporate sparse attention mechanisms in the encoder to reduce the
computational cost of standard self-attention operation Vaswani et al. (2017);
therefore, it can handle longer contexts. Longformer which is a Transformer
model that uses the BART architecture can take up to 16k input tokens while
Bird can support a sequence length of 4k tokens. The proposed attention
replaces the full self-attention in standard Transformers Vaswani et al.
(2017) with the attention pattern mechanism, including windowed, dilated, and
global attention. The model achieved SOTA performance on arXiv dataset, a long
text summarization dataset, surpassing BARD-base, Pegasus, and BigBird Beltagy
et al. (2020). BigBird reduces quadratic complexity with a sparse attention
mechanism combining random, windowed, and global attentions, allowing it to
handle longer sequences efficiently without significantly increasing
computational resources.

The model pretraining is continued from Pegasus Zhang et al. (2020) that is
specified for abstractive summarization. The model performs better than base
Transformers, Pegasus, BIGBIRD-RoBERTa, etc. on three long-text summarization
datasets, including BigPatent, arXiv, and Pubmed Zaheer et al. (2020).

###  2.2 Summarization Evaluation

Summarization systems can be evaluated with or without a reference summary.
Reference-based evaluation uses reference summaries to identify what content
from the input document is important and then evaluates a generated summary
based on how similar it is to the reference. On the other hand, reference-free
evaluation directly or indirectly defines a model to capture important
information in the document and uses that to evaluate the content of the
candidate summary. The recent success of LLMs has raised a lot of attention
about how to evaluate the generated summaries since the reference summaries
are too generic or unavailable. Therefore, in this study, we focus on
assessing the reference-free evaluation methods.

Evaluation of summaries can be done either automatically or manually. The
automatic method is fast, inexpensive, and can handle large volumes of data
without human intervention. However, this method may not be able to measure
the exact aspects of summaries that humans are interested in evaluating such
as clarity, accuracy, coverage, etc. On the other hand, manual evaluation is
slower, more expensive, and infeasible to use in model development, but it is
considered more reliable and trustworthy. A detailed discussion of the
automatic and human evaluation method is presented in Appendix A. We also
present an overview of summarization quality evaluation based on existing
studies in Figure 1 in Appendix A.

##  3 Methodology

###  3.1 Summarization Models

Summarization models studied in this research are SOTA PLMs, including the T5
family, XLNet, BART, BigBird, Pegasus, and GPT-3.5. These models have
demonstrated significant potential across various applications and are
categorized as follows: (1) domain-specific models (HUPD_T5_small and
HUPD_T5_base), (2) general-domain models (XLNet, BART, and Pegasus), (3)
models for long input sequences (LongT5 and BigBird), and (4) large language
models (LLMs) like GPT-3.5. This selection ensures a comprehensive evaluation
covering diverse types of SOTA summarization models.

Text-To-Text Transfer Transformer (T5 family), developed by Google, is an
encoder-decoder Transformer designed for a variety of NLP tasksRaffel et al.
(2020) thanks to its capability to convert any language task into an essential
text-to-text task. We implement the two off-the-shelf T5-based models:

HUPD-T5 Suzgun et al. (2022) Harvard University’s Policy Department (HUPD) has
tailored the T5 model for legal document summarization. We employ two versions
of this model: hupd-t5-base and hudp-t5-small. The two model versions are
finetuned for the abstractive summarization task on the Harvard USPTO Patent
Dataset (HUPD) Suzgun et al. (2024), a large-scale corpus of utility patent
applications filed to the United States Patent and Trademark Office (USPTO)
between January 2004 and December 2018. Therefore, these models are ideal for
the legal document summarization task that we focus on.

long-t5-tglobal-base-16384 + BookSum (so-called LongT5) Zhang et al. (2023):
The model is based on T5 with an expanded context window of 16384 tokens,
enabling it to comprehend and summarize long text efficientlyPeter Szemraj
(2022). The model is trained on a large corpus of book summaries, providing it
with a strong capability to digest and generate summaries of long texts
111https://huggingface.co/pszemraj/long-t5-tglobal-base-16384-book-summary.

XLNet Yang et al. (2020): an autoregressive transformer model, has 110 million
parameters and is trained on an assortment of datasets such as BooksCorpus,
Wikipedia, and Giga5Yang et al. (2020). Its distinguishing feature is the
permutation-based training, allowing the model to capture bidirectional
contexts and understand the intricate dependencies and relationships within
the text.

BART Lewis et al. (2020): is an encoder-decoder transformer model with 140
million parameters. The model was pre-trained on a wide variety of data
sources such as BookCorpus, Wikipedia, news articles, and stories Lewis et al.
(2020). The model is popular for text summarization and question-answering
tasks.

BigBird Zaheer et al. (2020): is a transformer model that pioneers the use of
a sparse attention mechanism with 110 million parameters. It is trained on
diverse data sources such as Wikipedia, BookCorpus, and news articles. The
introduction of the sparse attention mechanism enables BigBird to efficiently
manage very long sequences, thus, reducing the computational complexity that
is generally associated with processing long-range dependencies in text Zaheer
et al. (2020). It is suitable for processing lengthy texts due to its sparse
attention mechanism. However, this design might capture less context compared
to models with complete attention mechanisms, potentially affecting the
overall quality of the model’s outputs.

Pegasus Zhang et al. (2020): is an encoder-decoder transformer model with the
pretraining objective specifically optimized for summarization. It is trained
on diverse datasets like C4, HugeNews, PubMed, and arXiv. We use pegasus-x-
large-booksum-1, an off-the-shelf model with 568 million parameters for the
summarization experiments.

GPT-3.5 Liu et al. (2023):  Recent GPT models have demonstrated unprecedented
capabilities in understanding context, semantics, and syntactic structures,
enabling them to generate summaries that are concise, coherent, and human-
like. Experiments on multiple news datasets, Pu et al. (2023) even found that
humans significantly prefer summaries generated by zero-shot GPT-3.5 and GPT-4
to those written by humans or generated by small finetuned PLMs such as BART
Lewis et al. (2020), T5 Raffel et al. (2020), etc. In this study, we use
GPT-3.5-turbo-16k for summarization. It features an extended context window
handling up to 16k tokens, enabling it to maintain context over longer
conversations for more accurate and coherent responses.

###  3.2 Dataset

The dataset for this study consists of a corpus of 1630 patent documents
collected through web scraping of Google Patents
222https://patents.google.com. Human evaluation requires a good understanding
of the documents. To simplify the evaluation process, we have focused on
collecting patents related to communication and streaming technologies.

Although a patent document includes long description of the invention details
and many flow charts, the most important content in a patent document for the
summarization includes its abstract and the claims of the invention. The
abstract provides an overview of the invention, while the claims detail the
invention’s specifics that a summarization must capture. While other datasets,
such as BIGPATENTSharma et al. (2019), consider the abstract as an abstractive
summary of the patent document, the abstract itself does not cover the scope
of the claim and the novelty of the patent. To generate more useful and self-
contained patent summaries, we use the abstract and the claims from each
patent document as inputs for the summarization models.

####  3.2.1 Data Sample for Evaluation

Since human evaluation is only conducted in a small-scale, we randomly sample
a subset of 30 patent documents to generate summaries for evaluation with
humans. Figure 3 in Appendix C shows the text-length distribution of the
entire dataset and the evaluation sample. Besides, due to the high cost of
human evaluation, we only evaluate summaries from five summarization models,
including HUPD_T5_base, XLNet, BART, LongT5, GPT-3.5, and Llama-3. We select
representative models from each group based on their performance on
traditional automatic metrics.

###  3.3 Evaluation Methods

Many metrics have been proposed for text summarization evaluation; however,
not all of them are suitable for evaluating summarization in long text. We
select eight widely-used automatic evaluation metrics for this study. In
addition, we also conduct human evaluation and introduce a LLM-based
evaluation approach.

####  3.3.1 Automatic Metrics

ROUGE-1: measures the unigram overlap between the candidate summary (generated
summary) and the reference summary (ground truth summary). It calculates the
proportion of overlapping unigrams (individual words) between the candidate
and reference summaries. It considers only individual words and does not
capture word order or context Lin (2004).

ROUGE-2: measures the bigram overlap between the candidate summary and the
reference summary. It calculates the proportion of overlapping bigrams
(consecutive pairs of words) between the candidate and reference summaries. It
captures some level of word order and context by considering pairs of words
together Lin (2004).

ROUGE-L: measures the longest common subsequence between the candidate summary
and the reference summary. It finds the longest sequence of words that appear
in the same order in both the candidate and reference summaries. ROUGE-L
accounts for word order and captures the informativeness of the candidate
summary by considering sequences of words rather than just individual words
Lin (2004).

Bilingual Evaluation Understudy (BLEU) is a widely-used metric for evaluating
the quality of machine-generated text, especially in machine translation. The
BLEU score quantifies the similarity between the machine-generated text and
one or more reference texts. It considers the matching n-grams between the
generated text and the reference text. BLEU combines the scores for different
n-grams (usually 1 to 4) into a single score by calculating the geometric mean
of the modified precision counts Papineni et al. (2002). The final BLEU score
ranges from 0 to 1; 1 indicates a perfect match with the reference, while 0
indicates no overlap in n-grams.

BERTScore is an evaluation metric for language generation based on pretrained
BERT contextual embeddings Devlin et al. (2018). The metric computes a
similarity score for each token pair between the generated text and the
reference text using contextual embeddings. BERTScore has been shown to
correlate well with human judgment of text quality Zhang et al. (2019). It has
been used to evaluate a variety of text generation tasks, including machine
translation, summarization, and question-answering. The scores, such as
Precision, Recall, and F1, range from 0 to 1, with higher scores indicating
better performance.

SummaC measures the consistency between a summary and its source text. The
score is calculated by comparing the generated summary to the source text by
identifying any inconsistencies between them. SummaC effectively utilizes NLI
models for inconsistency detection by segmenting documents into sentence units
and aggregating scores between pairs of sentences Laban et al. (2022). The
authors introduce two versions: $SummaC_{ZS}$, and $SummaC_{Conv}$ in which
$SummaC_{ZS}$ use an out-of-the-box NLI model while $SummaC_{Conv}$ involves
finetuning using a convolutional neural network. $SummaC_{Conv}$ achieves
better performance on the proposed benchmark for the summary consistency
detection task than $SummaC_{ZS}$ and other models such as QuestEval Scialom
et al. (2021), FactCC-CLS, etc. Therefore, we utilize $SummaC_{Conv}$ to
evaluate the faithfulness of summarization models.

Flesch Reading Ease (FRE) score Flesch (1979) assesses the readability of an
English text by examining the sentence length and word length. It is
calculated as $FRE=206.835-(1.015*ASL)-(84.6*ASW)$, where $ASL$ is the average
sentence length and $ASW$ is the average number of syllables per word. The
score typically ranges from 0 to 100. Higher scores indicate that the text is
easier to read, while lower scores indicate that the text is more difficult to
read.

Dale-Chall Readability (DCR) score is another readability metric used to
assess the readability of English text. It considers a set of familiar words
and examines the sentence length to estimate the text’s difficulty level. The
DCR score is calculated: $DCR=(0.1579*PDW*100)+(0.0496*ASL)$, where $PDW$ is
the percentage of difficult words in the text, and $ASL$ is the average
sentence length. DCR provides an estimated percentage value representing the
difficulty level. Lower DCR scores indicate higher difficulty. DCR and FRE
have drawbacks. They primarily considers sentence length and the presence of
difficult words but does not account for factors like content, coherence, or
text structure, which also influence readability.

####  3.3.2 Human Evaluation

Existing automatic evaluation metrics do not consistently align with human
expectations, and there is no available human evaluation data for legal-text
summarization. Therefore, we conducted a manual evaluation study with
participants who are master’s students in computer and engineering fields.
Given the technical nature of the texts, participants needed a solid
understanding of such documents. To ensure quality, we designed test questions
mixed with other questions to filter out unreliable responses from
participants who failed the tests.

Based on our overview of existing summarization evaluation (Figure 1), we
choose to evaluate the following quality dimensions: (1) Clarity: whether the
summary is reader-friendly and expresses ideas clearly; (2) Accuracy: whether
the summary contains the same information as the source document; (3)
Coverage: how well the summary covers the important information from the
source document; and (4) Overall quality: how good the summary overall at
representing the source document; a good summary is a shorter piece of text
that has the essence of the original – tries to convey the same information as
the source document. The participants are asked to rate the given summary on
the above-mentioned quality dimensions on a Likert scale from 1 to 5,
corresponding to Poor to Excellent. Our study uses APPEN platform
333https://client.appen.com/sessions/new to design and conduct the evaluation.
Figure 2 shows the interface of the data collection form with APPEN. We also
release the response form of the study for the purpose of reproduction.
444Response form in APPEN

####  3.3.3 LLM-Based Evaluation

LLMs have demonstrated the exceptional ability to understand and follow
instructions. They potentially can serve as an evaluation agent Wu et al.
(2023); Chiang and Lee (2023). In this study, we explore the ability of LLMs
to assess the quality of model-generated summaries. To ensure a fair
comparison between LLMs and humans, we use the same instructions given to
humans as prompts to guide LLMs on this task. Similarly, we ask LLMs to
evaluate the summary quality on the four quality dimensions mentioned above on
a similar Likert scale 1-5.

###  3.4 Summarization Improvement Based on LLM’s Feedback

Motivated by the way humans refine written text, self-refining Madaan et al.
(2024) and self-reflection Shinn et al. (2024) have been proposed to enhance
the initial text generated by LLMs. In the context of summarization, the
initial draft iteratively refined will improve the quality. We adopt this
approach for summarization improvement. Particularly, we iteratively
incorporate the LLM’s evaluation of the summary generated in the previous
round into prompt to guide LLMs in generating a better version in the next
round. We provide the prompt template that we use for this experiment in
Appendix E.

##  4 Results and Discussion

###  4.1 Evaluation on Summarization Performance

##### Using existing automatic methods

To evaluate the performance of summarization, we use the most widely-used
automatic evaluation metrics, including BLEU, ROUGE, BERTScore, $SummaC$, FRE,
and DCR. While BLEU, ROUGE, BERTScore, and $SummaC$ are used to assess the
content quality of generated summaries, FRE and DCR are metrics to evaluate
the readability level. Table 1 shows performance of summarization models on
these metrics on the entire dataset (left) and on evaluation sample (right).
On the content-based evaluation metrics, the results show that XLNet, BART,
and GPT-3.5 are the best models. However, their generated summaries are as not
as readable as T5-generated summaries. BLEU metric yields low scores
($<<0.01$) for all summarization models, indicating a need to reconsider using
this metric for evaluation. XLNet performs best on ROUGE and BERTScore,
followed by GPT-3.5, and then BART. BigBird performs the worst among the eight
models. The results conducted on the entire dataset are consistent with the
results on the evaluation sample data that we sampled for human and LLM-based
evaluation.

Table 1: Performances of summarization models on automatic evaluation metrics.
Results are reported for the entire dataset and the evaluation sample of 30
source documents.

| Entire Dataset | Evaluation Sample
---|---|---
| HTS | HTB | X | B | BB | P | LT5 | GPT | Ll | HTS | HTB | X | B | BB | P | LT5 | GPT | Ll
BLEU | 0.0001 | 0.0000 | 0.0028 | 0.0700 | 0.0004 | 0.0000 | 0.0000 | 0.0046 | - | - | 0.0(0.0) | 0.0(0.0) | 0.0(0.0) | - | - | 0.0(0.0) | 0.002(0.01) | 0.00(0.00)
Rouge-1 | 0.2851 | 0.2261 | 0.5298 | 0.3665 | 0.1142 | 0.3202 | 0.2727 | 0.4239 | - | - | 0.233(0.05) | 0.523(0.07) | 0.366(0.06) | - | - | 0.297(0.05) | 0.431(0.07) | 0.38(0.06
Rouge-2 | 0.1682 | 0.1187 | 0.3993 | 0.2274 | 0.0308 | 0.0766 | 0.0471 | 0.2170 | - | - | 0.12(0.03) | 0.381(0.08) | 0.221(0.05) | - | - | 0.052(0.02) | 0.208(0.07) | 0.18(0.06)
Rouge-L | 0.2840 | 0.2249 | 0.5298 | 0.3656 | 0.1081 | 0.2906 | 0.2473 | 0.4053 | - | - | 0.232(0.05) | 0.523(0.07) | 0.366(0.06) | - | - | 0.268(0.05) | 0.412(0.06) | 0.36(0.06)
BERTScore | 0.6599 | 0.5985 | 0.7500 | 0.6741 | 0.4882 | 0.5943 | 0.5600 | 0.6700 | - | - | 0.605(0.05) | 0.738(0.06) | 0.677(0.04) | - | - | 0.571(0.04) | 0.67(0.04) | 0.65(0.03)
$SummaC$ | 0.8608 | 0.9189 | 0.9232 | 0.9268 | 0.3095 | 0.5191 | 0.4310 | 0.7797 | - | - | 0.914(0.09) | 0.925(0.04) | 0.936(0.07) | - | - | 0.474(0.14) | 0.76(0.16) | 0.63(0.20)
FRE | 36.2081 | 41.5381 | 23.3092 | 35.6559 | 23.0606 | 37.2800 | 38.5907 | 31.2010 | - | - | 40.743(14.49) | 23.422(11.97) | 35.064(11.07) | - | - | 38.428(12.8) | 31.116(10.66) | 30.89(9.23)
DCR | 9.0983 | 10.6610 | 9.8782 | 10.6644 | 6.9065 | 10.2207 | 10.1642 | 9.5479 | - | - | 10.719(1.85) | 10.188(0.92) | 10.941(0.96) | - | - | 10.053(0.98) | 9.717(0.91) | 10.16(0.77)

  * •

Note: $SummaC$ is the $SummaC_{conv}$, and all reported Rouge scores are the
Rouge F1. Green and Yellow indicate the best and the second-best model
performance on a certain metric, respectively. HTS, HTB, X and B denote
HUPD_T5_small, HUPD_T5_base, XLNet and BART BB, P, LT5, GPT and Ll denote
BigBird, Pegasus, LongT5, GPT-3.5 and Llama-3.

##### Human and LLM-based evaluation

The human and LLM-based evaluation is conducted regarding the four dimensions,
including clarity, accuracy, coverage, and overall quality. Table 2 present
results of human and GPT-4 evaluation for the five selected summarization
models. Overall, GPT-3.5 produces the best-quality summaries with the highest
scores on all evaluation dimensions, followed by XLNet and BART. XLNet and
BART have comparable accuracy and coverage quality scores. T5-base and LongT5
are the worst regarding the clarity. We further examine the summaries
producted by these low-performance models to propose improvement methods. Our
analysis results are presented in Appendix F. Surprisingly, the GPT-4
evaluation is highly consistent with humans in assessing the performance of
each model, demonstrating the potential of this automatic method to replace
expensive human evaluation.

Table 2: Human and LLM (GPT-4) summarization evaluation

| Human evaluation | LLM (GPT-4) evaluation
---|---|---
| HTB | X | B | LT5 | GPT | Ll | HTB | X | B | LT5 | GPT | Ll
Clarity | 2.183(0.48) | 2.6(0.55) | 3.083(0.59) | 2.267(0.68) | 4.55(0.27) | - | 3.32(0.76) | 3.2(0.66) | 3.8(0.48) | 2.167(0.53) | 4.167(0.38) | 4.300( 0.47)
Accuracy | 2.017(0.36) | 2.883(0.55) | 2.783(0.73) | 2.083(0.51) | 4.35(0.44) | - | 2.2(0.55) | 2.9(0.61) | 2.867(0.43) | 1.567(0.5) | 3.9(0.48) | 3.667(0.55)
Coverage | 1.8(0.45) | 2.517(0.56) | 2.517(0.56) | 2.133(0.61) | 4.517(0.33) | - | 1.433(0.5) | 2.033(0.49) | 2.0(0.26) | 1.4(0.5) | 3.567(0.57) | 3.333(0.55)
Overall quality | 1.7(0.43) | 2.467(0.47) | 2.6(0.62) | 2.0(0.66) | 4.45(0.3) | - | 2.0(0.64) | 2.467(0.63) | 2.7(0.54) | 1.467(0.51) | 3.967(0.41) | 3.733(0.43)

  * •

Note: Reported results are averaged over all evaluation samples with standard
deviations in brackets. Human evaluation scores are reported on the scale of
1-5,
respectively. HTB, X, B, LT5, GPT and Ll denote HUPD_T5_base, XLNet, BART,
LongT5, GPT-4 and Llama-3.

Table 3: Results of meta-analysis among automatic and human evaluation
(Kendall Tau-b)

| accuracy | overall | coverage | clarity | R-1 | R-2 | R-L | BERTscore | FRE | DCR | SummaC
---|---|---|---|---|---|---|---|---|---|---|---
accuracy | 1.0*** | 0.8** | 0.95** | 0.8** | 0.8 | 0.4 | 0.8 | 0.4 | -0.8 | -0.4 | 0
overall | 0.8** | 1.0*** | 0.95*** | 1.0** | 0.6 | 0.2 | 0.6 | 0.2 | -0.6 | -0.2 | 0.2
coverage | 0.95** | 0.95*** | 1.0*** | 0.95** | 0.738 | 0.316 | 0.738 | 0.316 | -0.738 | -0.316 | 0.105
clarity | 0.8** | 1.0** | 0.95** | 1.0*** | 0.6 | 0.2 | 0.6 | 0.2 | -0.6 | -0.2 | 0.2
R-1 | 0.8 | 0.6 | 0.738 | 0.6 | 1.0*** | 0.6* | 1.0*** | 0.6* | -1.0** | -0.2 | 0.2
R-2 | 0.4 | 0.2 | 0.316 | 0.2 | 0.6* | 1.0*** | 0.6* | 1.0** | -0.6* | 0.2 | 0.6
R-L | 0.8 | 0.6 | 0.738 | 0.6 | 1.0*** | 0.6* | 1.0*** | 0.6* | -1.0** | -0.2 | 0.2
BERTscore | 0.4 | 0.2 | 0.316 | 0.2 | 0.6* | 1.0** | 0.6* | 1.0*** | -0.6* | 0.2 | 0.6
FRE | -0.8 | -0.6 | -0.738 | -0.6 | -1.0** | -0.6* | -1.0** | -0.6* | 1.0*** | 0.2 | -0.2
DCR | -0.4 | -0.2 | -0.316 | -0.2 | -0.2 | 0.2 | -0.2 | 0.2 | 0.2 | 1.0*** | 0.6
SummaC | 0 | 0.2 | 0.105 | 0.2 | 0.2 | 0.6 | 0.2 | 0.6 | -0.2 | 0.6 | 1.0***

  * •

Note: *, **, *** for $p$-value $<$ 0.05, 0.01, 0.001, respectively. R-1, R-2,
and R-L denote Rouge-1, Rouge-2, and Rouge-L.

###  4.2 Meta-analysis

Meta-analysis explores the pairwise statistic correlation between evaluation
methods. Automatic evaluation metrics should strongly correlate with human
judgments. Since the automatic evaluation metrics we utilize have not been
specifically assessed for legal document summarization. Therefore, we would
like to re-evaluate these automatic evaluation metrics.

####  4.2.1 Automatic Metrics vs. Human Evaluation

Table 3 presents results of meta-analysis between the conventional automatic
metrics and human evaluation. Results indicate that ROUGE-1 and ROUGE-L
correlate with human evaluation on four dimensions (0.6-0.8 on Kendall Tau-b).
However, the significance test shows that the correlation is not statistically
significant. The scores between BERTscore and ROUGE-2 and human evaluation
show a low level of correlation (0.2-0.4). Surprisingly, SummaC, a factual
consistency measurement, displays very weak or non significant correlation
with human evaluation, and many other metrics such as ROUGE-1, ROUGE-L. Both
readability metrics have negative correlation with most metrics, including
human evaluation, ROUGE scores, and BERTscore. This implies that texts that
are easier to read tend to score lower on the content-based metrics, possibly
suggesting a trade-off betweeen complexity and ease-of-reading.

####  4.2.2 LLMs vs. Human Evaluation

Table 4 presents the results of a meta-analysis comparing evaluations by LLMs
and humans. The correlation scores for the three quality dimensions (accuracy,
coverage, and overall) between humans and LLMs (GPT-4 and Llama-3-8B) show an
extremely high positive correlation (0.8-0.9). Significance tests indicate
that these correlations are statistically significant. For the clarity
dimension, LLMs and human evaluation still exhibit a high positive correlation
(0.67-0.8); however, the statistical tests reveal that the correlations are
not statistically significant, suggesting that the observed relationship could
be due to random variation. The correlation scores are consistent for both
GPT-4 and Llama-3-8B. The results indicate (1) LLMs are capable of performing
summarization evaluation as effectively as humans, and (2) open-sourced LLMs
such as Llama-3-8B can produce reliable evaluations similar to GPT-4. In
specific domains like legal documents, where the cost of human evaluation is
prohibitively high, LLMs, including open-sourced models, could be utilized to
assess summarization.

In a previous analysis, we only used one type of correlation coefficient for
our meta-analysis (Spearman’s or Kendall’s Tau-b). Therefore, we further
compare the correlation results between GPT-4 and human evaluation using three
tests: Pearson’s $\rho$, Spearman’s, and Kendall’s Tau-b correlation
coefficients. The results indicate the consistency of these three correlation
coefficients. Kendall’s Tau-b produces lower scores than the two other tests.
See Table 6, Appendix D for more details.

Table 4: Results of meta-analysis between LLM (GPT-4) vs human evaluation and
LLM (Llama-3-8B) vs human evaluation (Spearman’s)

| LLM (GPT-4) vs. human evaluation | LLM (Llama-3-8B) vs. human evaluation
---|---|---
| llm_a | llm_o | llm_co | llm_cl | h_a | h_o | h_co | h_cl | llm_a | llm_o | llm_co | llm_cl | h_a | h_o | h_co | h_cl
llm_accuracy | 1.0*** | 0.9** | 1.0* | 0.821* | 0.9* | 0.8* | 0.872 | 0.8* | 1.0*** | 1.0** | 0.9** | 0.894 | 0.8* | 0.9* | 0.872* | 0.9*
llm_overall | 0.9** | 1.0*** | 0.9** | 0.975* | 0.8** | 0.9* | 0.872* | 0.9* | 1.0** | 1.0*** | 0.9* | 0.894 | 0.8* | 0.9 | 0.872 | 0.9*
llm_coverage | 1.0* | 0.9** | 1.0*** | 0.821 | 0.9*** | 0.8*** | 0.872** | 0.8** | 0.9** | 0.9* | 1.0*** | 0.783 | 0.9*** | 1.0** | 0.975** | 1.0**
llm_clarity | 0.821* | 0.975* | 0.821 | 1.0*** | 0.667 | 0.821 | 0.763 | 0.821 | 0.894 | 0.894 | 0.783 | 1.0*** | 0.783 | 0.783 | 0.803 | 0.783
human_accuracy | 0.9* | 0.8** | 0.9*** | 0.667 | 1.0*** | 0.9** | 0.975** | 0.9** | 0.8* | 0.8* | 0.9*** | 0.783 | 1.0*** | 0.9** | 0.975** | 0.9**
human_overall | 0.8* | 0.9* | 0.8*** | 0.821 | 0.9** | 1.0*** | 0.975*** | 1.0** | 0.9* | 0.9 | 1.0** | 0.783 | 0.9** | 1.0*** | 0.975*** | 1.0**
human_coverage | 0.872 | 0.872* | 0.872** | 0.763 | 0.975** | 0.975*** | 1.0*** | 0.975** | 0.872* | 0.872 | 0.975** | 0.803 | 0.975** | 0.975*** | 1.0*** | 0.975**
human_clarity | 0.8* | 0.9* | 0.8** | 0.821 | 0.9** | 1.0** | 0.975** | 1.0*** | 0.9* | 0.9* | 1.0** | 0.783 | 0.9** | 1.0** | 0.975** | 1.0***

  * •

Note: *, **, *** for $p$-value $<$ 0.05, 0.01, 0.001, respectively. llm_a,
llm_o, llm_co, and llm_cl denote llm_accuracy, llm_overall, llm_coverage, and
llm_clarity.
h_a, h_o, h_co and h_cl denote human_accuracy, human_overall, human_coverage
and human_clarity.

###  4.3 Summarization Quality Improvement

Table 5 presents the results of our method for improving summarization It
shows the impact of LLM’s verbal feedback on summarization performance
improvement. It demonstrates that by integrating evaluation feedback from LLMs
into the prompt, the quality of the summaries significantly improves in terms
of clarity (from 4.167 to 4.5) and coverage (from 3.567 to 3.833), indicating
a substantial enhancement. However, we have also noticed that this method
slightly reduces accuracy. In the future, we aim to enhance this quality
dimension further.

Table 5: Results of improving summarization (GPT-3.5) with LLM’s verbal
feedback.

| w/o LLM’s feedback | w/ LLM’s feedback
---|---|---
Clarity | 4.167(0.38) | 4.5(0.51) $\uparrow$$\uparrow$$\uparrow$
Accuracy | 3.9(0.48) | 3.833(0.38) $\downarrow$
Coverage | 3.567(0.57) | 3.833(0.70) $\uparrow$$\uparrow$
Overall quality | 3.967(0.41) | 3.933(0.41) $\downarrow$

  * •

Note: The arrows in the table indicate the direction of change: double arrows
for
significant improvements or declines, and single arrows for moderate changes.

##  5 Conclusion and Future Work

In this work, we compare various methods for evaluating abstractive
summarization of legal documents, including existing automatic metrics and
human evaluation. We also explore the potential use of LLMs for evaluation
purposes. We conduct different meta-analyses to compare the evaluations among
these methods. Our findings reveal that widely-used automatic evaluation
metrics such as ROUGE-2, BERTScore, and SummaC exhibit very weak or non-
significant correlation with human evaluation. Additionally, readability
metrics show a negative correlation with most other metrics, including human
evaluation. In contrast, our results suggest that LLMs can effectively perform
summarization evaluation. Open-sourced LLMs like Llama-3-8B demonstrates
reliable evaluations similar to GPT-4. Futher, we attempt to improve the
summarization quality through iterative improvement by leveraging the verbal
evaluation feedback from LLMs. The results indicate that the quality of the
summaries significantly improves in terms of clarity and coverage, suggesting
the potential of this approach. In the future, we aim to further enhance other
quality dimensions such as accuracy.

##  6 Limitations

In this study, our focus is solely on evaluating the summarization of legal
documents due to the high cost of human evaluation. Therefore, our findings
may not be applicable to other domains where the document structure and
vocabulary may differ. Additionally, the size of our human evaluation sample
is limited, which may not accurately reflect the overall performance or
reliability of the findings. This limitation could lead to the statistical
insignificance of some meta-analyses we dicussed before and potentially cannot
capture all possible variation within the dataset.

## Acknowledgments

We would like to thank Sai Kolapudi for his valuable help with data collection
and preprocessing.

## References

  * Allahyari et al. (2017) Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut. 2017.  Text summarization techniques: A brief survey.  _International Journal of Advanced Computer Science and Applications (ijacsa)_ , 8(10).
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  In _3rd International Conference on Learning Representations, ICLR 2015_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018.  Faithful to the original: fact-aware neural abstractive summarization.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence_ , pages 4784–4791.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-Yi Lee. 2023.  Can large language models be an alternative to human evaluations?  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15607–15631.
  * Chin-Yew (2004) Lin Chin-Yew. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Proceedings of the Workshop on Text Summarization Branches Out, 2004_.
  * Deutsch et al. (2021) Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021.  Towards question-answering as an automatic metric for evaluating the content quality of a summary.  _Transactions of the Association for Computational Linguistics_ , 9:774–789.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * El-Kassas et al. (2021) Wafaa S El-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021.  Automatic text summarization: A comprehensive survey.  _Expert systems with applications_ , 165:113679.
  * Flesch (1979) Rudolf Flesch. 1979.  _How to write plain English: A book for lawyers and consumers_ , volume 76026225.  Harper & Row New York.
  * Ganesan (2018) Kavita Ganesan. 2018.  Rouge 2.0: Updated and improved measures for evaluation of summarization tasks.  _arXiv preprint arXiv:1803.01937_.
  * Graham (2015) Yvette Graham. 2015.  Re-evaluating automatic summarization with bleu and 192 shades of rouge.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 128–137.
  * He et al. (2020) Junxian He, Wojciech Kryściński, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2020.  Ctrlsum: Towards generic controllable text summarization.  _arXiv preprint arXiv:2012.04281_.
  * Kieuvongngam et al. (2020) Virapat Kieuvongngam, Bowen Tan, and Yiming Niu. 2020.  Automatic text summarization of covid-19 medical research articles using bert and gpt-2.  _arXiv preprint arXiv:2006.01997_.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A Hearst. 2022.  Summac: Re-visiting nli-based models for inconsistency detection in summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out: Proceedings of the ACL-04 Workshop_ , pages 74–81.
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the middle: How language models use long contexts.  _Preprint_ , arXiv:2307.03172.
  * Madaan et al. (2024) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024.  Self-refine: Iterative refinement with self-feedback.  _Advances in Neural Information Processing Systems_ , 36.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919.
  * Meng et al. (2021) Rui Meng, Khushboo Thaker, Lei Zhang, Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. 2021.  Bringing structure into summaries: a faceted summarization dataset for long scientific documents.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 1080–1089.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu̇lçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Peter Szemraj (2022) Peter Szemraj. 2022.  long-t5-tglobal-base-16384-book-summary (revision 4b12bce).
  * Pu et al. (2023) Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023.  Summarization is (almost) dead.  _arXiv preprint arXiv:2309.09558_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Preprint_ , arXiv:1910.10683.
  * Reiter (2018) Ehud Reiter. 2018.  A structured review of the validity of bleu.  _Computational Linguistics_ , 44(3):393–401.
  * Schluter (2017) Natalie Schluter. 2017.  The limits of automatic summarisation according to rouge.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 41–45. Association for Computational Linguistics.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021.  Questeval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6594–6604. Association for Computational Linguistics.
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083.
  * Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019.  Bigpatent: A large-scale dataset for abstractive and coherent summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2204–2213.
  * Shinn et al. (2024) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024.  Reflexion: Language agents with verbal reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36.
  * Sulem et al. (2018) Elior Sulem, Omri Abend, and Ari Rappoport. 2018.  Bleu is not suitable for the evaluation of text simplification.  In _2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018_ , pages 738–744. Association for Computational Linguistics.
  * Sun et al. (2022) Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuan-Jing Huang. 2022.  Bertscore is unfair: On social bias in language model-based metrics for text generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3726–3739.
  * Suzgun et al. (2024) Mirac Suzgun, Luke Melas-Kyriazi, Suproteem Sarkar, Scott D Kominers, and Stuart Shieber. 2024.  The harvard uspto patent dataset: A large-scale, well-structured, and multi-purpose corpus of patent applications.  _Advances in Neural Information Processing Systems_ , 36.
  * Suzgun et al. (2022) Mirac Suzgun, Luke Melas-Kyriazi, Suproteem K. Sarkar, Scott Duke Kominers, and Stuart M. Shieber. 2022.  The harvard uspto patent dataset: A large-scale, well-structured, and multi-purpose corpus of patent applications.  _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , 1:3552–3565.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wu et al. (2023) Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. 2023.  Large language models are diverse role-players for summarization evaluation.  _arXiv preprint arXiv:2303.15078_.
  * Yang et al. (2020) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2020.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Preprint_ , arXiv:1906.08237.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in neural information processing systems_ , 33:17283–17297.
  * Zhang et al. (2023) Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023.  Summit: Iterative text summarization via chatgpt.  _Preprint_ , arXiv:2305.14835.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020.  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.  _Preprint_ , arXiv:1912.08777.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhao et al. (2019) Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Appendix: Existing studies on text summarization evaluation

Figure 1: An Overview of Summarization Evaluation Methods

###  A.1 Automatic methods

Many studies have been done to determine which automatic metrics are
sufficient for evaluation. They include ROUGE Chin-Yew (2004); Ganesan (2018),
BLEU Papineni et al. (2002), BERTScore Zhang et al. (2019); Kieuvongngam et
al. (2020), MoverScore Zhao et al. (2019), SummaC Laban et al. (2022),
QuestEval Scialom et al. (2021), etc. They can be categorized into three
groups: text overlapping (including ROUGE and BLEU), vector-space distance
(BERTScore, MoverScore), and NLP task-based to measure the consistency between
the generated summary and the reference (SummaC, QuestEval).

For the simplicity of the overlapping-based evaluation methods like ROUGE and
BLEU, they are the most favorable to use. The use of text-overlapping metrics
to assess the quality of summarization is still a topic of debate. Graham
(2015) uses summary coverage computations and human coverage scores to assert
that text overlap-based metrics are suitable for evaluation. However, some
studies demonstrate that evaluation tools such as BLEU, ROUGE, and BERTScore
are not suitable for the automatic evaluation of summaries Sun et al. (2022);
Sulem et al. (2018); Reiter (2018); Schluter (2017). Similarly, Schluter
(2017) shows ROUGE’s limitations, specifically, its inability to attain a
perfect evaluation score. Therefore, most studies combine different metrics,
such as text-overlap-based (ROUGE, BLEU), vector-space-based (BERTscore,
MoverScore), and QA-based (QuestEval, SummaQA) to evaluate summarization
performance.

On the other hand, some reference-free metrics have recently been proposed
since summary references are not always available and high-quality, for
example, QuestEval Scialom et al. (2021), QAEval Deutsch et al. (2021).
Compared to reference-free methods, reference-based metrics have more
advantages. Additional studies also assert that reference-based metrics such
as BERTScore correlate more closely with human judgments Zhang et al. (2019).

###  A.2 Manual methods

Human evaluation is commonly used because automatic metrics are imperfect, and
humans can perform tasks that automated methods cannot do as reliably. The
method involves asking human judges to score summaries based on the given
reference (called reference-based) or directly assess the generated summary
according to specific criteria (called reference-free or direct assessment).

##### Reference-based

The manual reference-based approach is found to be simpler to conduct than the
manual reference-free approach, which requires a well-designed scoring scheme
and rubrics, and participants must have a good understanding of the source
document.

As mentioned earlier, we aim to explore the reference-free human evaluation
methods in this study.

##### Reference-free

Most human evaluation methods for summarization are reference-free. Current
works consider multiple dimensions to evaluate summary quality, including
readability (ease of reading), fluency (grammaticality), consistency (factual
support from the input document), faithfulness (completeness of information
from the input document), relevance (selection of important content), and
content quality (inclusion of salient information). Due to the high cost,
evaluations often focus on fluency, coherence, consistency, and relevance. The
process starts by sampling a small set of about 50-100 generated summaries.
The recruited evaluators are asked to score the generated summary on a Likert
scale such as 1-5, or 1-7 on one or more evaluation dimensions described
above. The results are then compared with automatic metrics using several
correlation analysis tests such as Pearson’s, Spearman’s, and Kendall’s Rank
correlation coefficient.

Manual evaluation has been an important tool for measuring the quality of
generated summaries. By focusing on specific dimensions and using a carefully
selected group of evaluators, manual evaluation can provide valuable insights
into the strengths and weaknesses of a summarization system.

##  Appendix B Appendix: Description of the APPEN platform used for evaluation

Figure 2: An example of the evaluation data collection form designed on APPEN
platform.

The APPEN platform was utilized for the manual evaluation of generated
summaries in our study. APPEN provides a user-friendly interface that allows
researchers to design and conduct surveys or evaluations efficiently. For this
study, we created a form where participants could assess the quality of
summaries based on predefined dimensions: Clarity, Accuracy, Coverage, and
Overall Quality. Participants rated each summary on a Likert scale from 1
(Poor) to 5 (Excellent). The platform’s robust functionality ensured accurate
data collection and facilitated the identification of unreliable responses
through carefully designed test questions. This systematic approach helped in
obtaining reliable and consistent evaluations, which are crucial for assessing
the performance of summarization models.

Figure 2 illustrates the interface of the data collection form designed on the
APPEN platform used in our manual evaluation study. Participants are presented
with a summary and are asked to rate it across multiple dimensions: Clarity,
Accuracy, Coverage, and Overall Quality. Each dimension is evaluated on a
Likert scale from 1 to 5, ranging from Poor to Excellent. This interface
ensures a structured and consistent approach to gathering evaluations from
participants, facilitating the analysis of summarization quality against
predefined criteria.

##  Appendix C Appendix: More detailed analysis of the datasets

Figure 3 contains two visualizations that shows the word count distribution
for patent documents related to communication and streaming technologies,
collected through web scraping of Google Patents. The first visualization
presents the word count distribution for the entire dataset, consisting of
1,630 patent documents. The histogram displays the frequency of documents
against their respective word counts, with the majority falling between 1,000
and 2,500 words, peaking around 1,500 words. This distribution indicates a
right-skewed pattern with a long tail extending towards higher word counts,
highlighting the typical length of patent documents used in the study, which
focuses on abstracts and claims to generate comprehensive and informative
summaries.

Figure 3: Length distributions of the entire dataset (top) and the evaluation
sample (bottom).

The second visualization shows the word count distribution for a randomly
selected subset of 30 patent documents used for human evaluation. This subset
was chosen from the larger dataset to ensure a diverse range of document
lengths for evaluating the performance of various summarization models. The
histogram displays the frequency of documents against their respective word
counts, showing a relatively even distribution across different ranges, with
most documents falling between 750 and 2,500 words and peaking around 1,750
words.

##  Appendix D Appendix: Choosing a meta-analysis correlation coefficients

Pearson’s $\rho$, Spearman’s, and Kendall’s Tau-b are widely-used correlation
coefficients. We further conduct an experiment with these three coefficients
to measure the strength of the relationship between GPT-4 and human evaluation
scores. Table 6 present the result of this experiment. The results indicate
the consistency of these three correlation coefficients. However, compared to
two others, Kendall’s Tau-b produces lower scores. Therefore, when the
correlation is not very strong, it may be harder to interpret the correlation
results of Kendall’s Tau-b. In such cases, Pearson’s $\rho$ and Spearman’s
coefficients may be more appropriate.

Table 6: Results of meta-analysis among LLM (GPT-4) and human evaluation using
three different tests.

| human_accuracy | human_coverage | human_clarity | human_overall
---|---|---|---|---
| $r$ | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$ | p | $\rho$ | $\tau$ | $r$ | $\rho$ | $\tau$
llm_accuracy | 0.939* | 0.9* | 0.8* | 0.866 | 0.872 | 0.738 | 0.896* | 0.8* | 0.6* | 0.889* | 0.8* | 0.6*
llm_coverage | 0.996*** | 0.9*** | 0.8*** | 0.989** | 0.872** | 0.738** | 0.981** | 0.8** | 0.6** | 0.992*** | 0.8*** | 0.6***
llm_clarity | 0.777 | 0.667 | 0.527 | 0.694 | 0.763 | 0.667 | 0.795 | 0.821 | 0.738 | 0.732 | 0.821 | 0.738
llm_overall | 0.965** | 0.8** | 0.6** | 0.924* | 0.872* | 0.738* | 0.958* | 0.9* | 0.8* | 0.942* | 0.9* | 0.8*

  * •

Note: *, **, *** for $p$-value $<$ 0.05, 0.01, 0.001. $r$, $\rho$, and $\tau$
denote Pearson’s $\rho$, Spearman’s, and Kendall’s Tau-b correlation
coefficients, respectively.

##  Appendix E Appendix: Improving Summarization Quality

Figure 4: Prompt designed for iteratively improving summarization quality

##  Appendix F Appendix: Summarization models’ error analysis

We conducted an error analysis to identify the specific areas where the
summarization models underperformed. The purpose was to better understand the
types and root sources of the errors and to provide guidelines for model
selection and improve the model. We identified three major types of errors
that affect the quality of summarization: low abstractiveness, incompleteness,
and hallucinations. The low abstractiveness issue happens the most in the
summaries generated by HUPD_T5_base, XLNet, and BART.

Table 7 presents some selected examples of error types for illustration. The
examples of HUPD_T5_base and XLNet generated summaries show that the models
mostly copied long phrases or even entire sentences from the abstract and
claims instead of generating new summary sentences. This likely led to the
high performance of these models on ROUGE metrics which are based on n-gram
overlaps. HUPD_T5_base often copies and then combines long phrases to create a
summary. However, its summaries do not have complete meaning, which makes them
hard to understand. This is why the clarity score given to this model is the
lowest, according to both humans and LLM evaluators (Table 1). Furthermore, we
found LongT5 suffered the most from the hallucination issue. The following
example illustrates a significant hallucination by LongT5. In the source
document, “RIBS” refers to “radio interface based synchronization”. However,
the model failed to recognize this and instead attempted to replace the term
with “bribs”, which was nonsensical in the context of the sentence.

Table 7: Examples of error types in generated summaries.

Error type |  Source document |  Generated summary |  Model
---|---|---|---
Low abstractiveness |  Abstract: The invention relates to a method, comprising tracking the gaze of an eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on the screen of a near-to-eye display (NED), wherein the movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly; observing a gesture of one hand of the user by a second camera, and executing a further operation if at least one gesture of the hand is detected. The invention further relates to an apparatus and a computer program product configured to perform the method. Claims: … 3\. The method of claim 1, wherein the detected gesture is a movement of a fist, shaking of a palm or movement of a palm from or to the second camera … |  the method of claim 1 comprises tracking the gaze of an eye by a first camera. a second camera observes a gesture of one hand of the user. the detected gesture may be the movement of a fist, shaking of a palm or movement of a palm. |  HUPD_T5_base
Low abstractiveness |  Abstract: Achieving phase synchronization in single frequency networks may be useful in various communication systems. For example, the use of synchronous Ethernet, timing over packet, and radio interface based synchronization (RIBS) may be applied in mobile and/or data communication networks. Certain embodiments improve phase accuracy using a hybrid synchronization method beyond standard RIBS and clustering carriers using the same transmission waveforms across carriers. Certain embodiments also select a transmission point to maximize the signal quality of the composite waveforms in a cluster. Claims: 9.10. … |  Achieving phase synchronization in single frequency networks may be useful in various communication systems. Certain embodiments also select a master transmission point to maximize the signal quality of the composite waveforms in a cluster. The apparatus according to claim 9, wherein the one or more quality metrics comprise an indication of at least one of signal power, signal to interference and noise ratio, path loss, and channel rank. The apparatus according to claim 9, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to at least: receive, by the transmission point, phase synchronization data… |  XLNet
Incompleteness and potential hallucinations |  Abstract: Achieving phase synchronization in single frequency networks may be useful in various communication systems. For example, the use of synchronous Ethernet, timing over packet, and radio interface based synchronization (RIBS) may be applied in mobile and/or data communication networks. Certain embodiments improve phase accuracy using a hybrid synchronization method beyond standard RIBS and clustering carriers using the same transmission waveforms across carriers. Certain embodiments also select a master transmission point to maximize the signal quality of the composite waveforms in a cluster. Claims: … |  use of synchronous Ethernet, timing over packet, and radio interface based synchronization (RIBS) may be applied. certain embodiments improve phase accuracy using a hybrid synchronization method. |  HUPD_T5_base
Hallucination |  Abstract: Achieving phase synchronization in single frequency networks may be useful in various communication systems. For example, the use of synchronous Ethernet, timing over packet, and radio interface based synchronization (RIBS) may be applied in mobile and/or data communication networks. Certain embodiments improve phase accuracy using a hybrid synchronization method beyond standard RIBS and clustering carriers using the same transmission waveforms across carriers. Certain embodiments also select a master transmission point to maximize the signal quality of the composite waveforms in a cluster. Claims: 1\. An apparatus, comprising: at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to at least… |  An apparatus for achieving phase resolving in single-frequency networks. The device includes a memory and a program code that causes the device to receive signals from multiple different types of sources, each of which has a quality metrics associated with it. A cluster is formed wherein information regarding the selected point is given and an indication about the other points is given. In further embodiments, the device selects a particular point to achieve better signal quality than standard bribs. |  LongT5

  * •

Note: Phrases in yellow are copied from Abstract, while phrases in orange are
excerpted from Claims.