#  AWESOME: GPU Memory-constrained Long Document Summarization
using Memory Mechanism and Global Salient Content

Shuyang Cao  Lu Wang
Computer Science and Engineering
University of Michigan
Ann Arbor, MI
{caoshuy<EMAIL_ADDRESS>
###### Abstract

Long document summarization systems are critical for domains with lengthy and
jargon-laden text, yet they present significant challenges to researchers and
developers with limited computing resources. Existing solutions mainly focus
on efficient attentions or divide-and-conquer strategies. The former reduces
theoretical time complexity, but is still memory-heavy. The latter methods
sacrifice global context, leading to uninformative and incoherent summaries.
This work aims to leverage the memory-efficient nature of divide-and-conquer
methods while preserving global context. Concretely, our framework AWESOME
uses two novel mechanisms: (1) External memory mechanisms track previously
encoded document segments and their corresponding summaries, to enhance global
document understanding and summary coherence. (2) Global salient content is
further identified beforehand to augment each document segment to support its
summarization. Extensive experiments on diverse genres of text, including
government reports, transcripts, scientific papers, and novels, show that
AWESOME produces summaries with improved informativeness, faithfulness, and
coherence than competitive baselines on longer documents, while having a
similar or smaller GPU memory footprint.

##  1 Introduction

Large pre-trained transformer models have demonstrated impressive performance
across popular abstractive summarization benchmarks Lewis et al. (2020);
Raffel et al. (2020). Yet, transformer’s quadratic memory complexity presents
challenges for summarizing long documents with more than hundreds of words,
such as scientific papers and investigation reports Cohan et al. (2018); Huang
et al. (2021), making it infeasible for researchers and developers with
limited hardware resources (e.g., GPUs with insufficient memories) to
contribute to this important research field.

The NLP community has made several algorithmic innovations to address the long
document challenge. Prior work divides a document into smaller chunks and
summarizes each separately Gidiotis and Tsoumakas (2020), reduces the
complexity of attention calculations Beltagy et al. (2020), and removes
unimportant content before running an abstractor Pilault et al. (2020). In
terms of memory efficiency, divide-and-conquer methods obtain the most
significant advantage Moro and Ragazzi (2022). However, useful information
outside of a document segment and their corresponding summaries become
inaccessible, leading to uninformative and incoherent summary generation.
Unsurprisingly, state-of-the-art performance is obtained by models that can
maintain global context, e.g., by combining global attentions with local
attentions in transformer-based summarization models Zaheer et al. (2021);
Phang et al. (2022). Yet, they still require a large GPU memory footprint in
practice.111These approaches require a GPU memory of ¿40GB to process
documents with over 8K tokens, while the most cost-effective GPUs only have
24GB of memory Li (2022). Though large language models, such as GPT-4 OpenAI
(2023), are trained to handle up to 32K tokens, the privacy and security of
data transmitted and shared through the API remain concerning, particularly in
sectors dealing with sensitive information, e.g., clinical notes. Local model
development can bolster privacy and security; however, limited computational
resources in these scenarios necessitate the exploration of efficient modeling
techniques.

Therefore, this work aims to address the problem of efficient long document
summarization using constrained resources, especially, we focus on constrained
GPU memory.222We leave the efficiency study of summarization systems with
regard to energy and model scales to future work. We propose AWESOME333Our
code will be made available at https://shuyangcao.github.io/projects/awesome/,
which is built on the memory-efficient divide-and-conquer approach, and
Augmented With Estimated Salient cOntent and MEmory mechanism. In essence,
AWESOME maintains global context of both the source document and the summary
generated so far with a limited memory usage, to enhance summary
informativeness, faithfulness, and coherence.

First, external memory mechanism is used on the encoder side of AWESOME to
store information as it reads in document segments in sequence. This maintains
relevant context for improved document understanding and salient content
detection, thus promoting summary informativeness and faithfulness. Another
memory is applied on the decoder side to improve generation coherence by
tracking the partial summaries generated for prior document segments.
Importantly, to ensure the GPU memory efficiency of AWESOME, we limit the
number of vectors stored in the external memories and curb gradients from
propagating to other document and summary segments.

Second, AWESOME incorporates global salient content selected by an efficiently
trained extractor through (1) direct text concatenation, or (2) inserting
their key-value matrices into attention calculation. This lets the summarizer
be aware of important topics at a global level, to enhance salience estimation
and summary informativeness.

We experiment with five popular long-input benchmarks of different genres:
investigation reports in GovReport Huang et al. (2021), meeting transcripts in
QMSum Zhong et al. (2021), TV transcripts in SummScreen Chen et al. (2022),
scientific papers in arXiv Cohan et al. (2018), and fictions in BookSum
Kryscinski et al. (2022). First, on all the five datasets, all AWESOME
variants uniformly outperform Se3 Moro and Ragazzi (2022), the divide-and-
conquer baseline, on summary informativeness as evaluated by ROUGE Lin (2004)
and on coherence as measured by DiscoScore Zhao et al. (2022) and a metric
based on entity graphs Guinaudeau and Strube (2013)—both metrics are highly
correlated with human judgment, according to Zhao et al. (2022). Second,
AWESOME with memory mechanisms also improves summary faithfulness over Se3 on
GovReport, according to SummaC Laban et al. (2022), an entailment-based
faithfulness metric. Lastly, compared with more memory-intensive models that
also maintain global context, such as Phang et al. (2022) and Liu et al.
(2022), AWESOME achieves higher automatic scores for informativeness,
coherence, and faithfulness on GovReport Huang et al. (2021). On BookSum which
comprises the lengthiest documents and summaries among the five datasets,
AWESOME also achieves higher levels of informativeness and coherence compared
to the most recent comparisons.

##  2 Related Work

###  2.1 Efficient Long Document Summarization

Approach | In$\rightarrow$Out | Enc | Enc$\leftarrow$Dec | Dec
---|---|---|---|---
Efficient Attention | $x\rightarrow y$ | $\blacksquare$ | $\blacksquare$ | $\CIRCLE$
Extract-Abstract | $x_{e}\rightarrow y$ | $\square$ | $\bigstar$ | $\CIRCLE$
Dynamic Weight | $x\rightarrow y$ | $\square$ | $\blacksquare+\bigstar$ | $\CIRCLE$
Divide-Conquer | $x_{i}\rightarrow y_{i}$ | $\square$ | $\square$ | $\Circle$
Table 1:  Existing approaches to long document summarization (§2.1).
In$\rightarrow$Out: Longer inputs ($|x|>|x_{e}|>|x_{i}|$) or outputs
($|y|>|y_{i}|$) produce more nodes in the computation graph, thus the higher
memory consumption. Enc: Encoder accessing partial documents ($\square$) hurts
document understanding, compared to reading the full text ($\blacksquare$).
Enc$\leftarrow$Dec: Decoder reading the full document ($\blacksquare$) or pre-
identified salient content ($\bigstar$) enhances summary informativeness,
compared to a segment ($\square$). Dec: Decoder accessing previously generated
summary content ($\CIRCLE$) is crucial for generation coherence than reading a
current summary segment only ($\Circle$).

We categorize existing efficient long document summarization models into four
major types, as summarized in Table 1. The model input can be an original
document, extracted important segments of the document, or a document segment,
which are denoted as $x$, $x_{e}$, or $x_{i}$ (for the $i$-th segment), and
typically, $|x|>|x_{e}|>|x_{i}|$. The output can be the full summary $y$ or a
summary segment $y_{i}$ (for $x_{i}$), where $|y|>|y_{i}|$. Importantly,
longer inputs and outputs expand larger computation graph, leading to higher
GPU memory usage. Moreover, we analyze both the document context and the
summary context used by each approach when generating summaries. Specifically,
we check (1) full vs. partial documents that are consumed to obtain the
encoder representations (Enc); (2) full vs. partial encoder representations
that are attended by the decoder (Enc$\leftarrow$Dec); and (3) full vs.
partial output that is accessed by the decoder (Dec).

Efficient attentions are designed to reduce the quadratic complexity of the
original transformer architecture Vaswani et al. (2017) and maintain full
encoding context by combining global attentions with local attentions built on
sliding windows Beltagy et al. (2020); Zaheer et al. (2021), text blocks Phang
et al. (2022); Tay et al. (2020), or clusters of similar tokens Kitaev et al.
(2020); Roy et al. (2021). Besides the aforementioned attention variants
designed for self-attentions, recent work has reduced the memory usage of
decoder cross attentions by distributing encoder outputs to different
attention heads Huang et al. (2021) or selecting attendable encoder outputs
via KNN search Bertsch et al. (2023). Despite the reduced complexity,
efficient attention-based systems effectively require reading the full
document $x$ to generate a summary $y$ during model training and thus still
need huge GPU memory that scales with the input length.

Extract-then-abstract systems circumvent the long sequence challenge by first
identifying the salient segments, $x_{e}$ (e.g., sentences), using an
extractor, and then running an abstractor over $x_{e}$ to produce the final
summary Pilault et al. (2020); Liu and Lapata (2019); Zhao et al. (2020).
However, the extracted segments may contain incomplete and out-of-context
information that leads to incomprehensible and unfaithful summaries.

To mitigate the error propagation issue of a two-stage approach, recent
studies bridge the extractor and abstractor via dynamic weights over document
segments. Rather than feeding the extracted segments directly to the
abstractor, at each summary decoding step, DYLE Mao et al. (2022) first
predicts an output token distribution for each segment separately, and then
aggregates over all the extracted segments as weighted by their extraction
salience. PageSum Liu et al. (2022) further alleviates context loss by
averaging decoder output representations conditioned on all document segments.
Though their abstractor processes each document segment $x_{i}$ separately,
jointly training the extractor and the abstractor still requires loading the
full document $x$ into the GPU memory.

Divide-and-conquer systems split a document into multiple non-overlapping
segments and summarize each segment separately, as done in Gidiotis and
Tsoumakas (2020) and Se3 Moro and Ragazzi (2022). SummN Zhang et al. (2022)
uses an additional summarization stage to further condense the segmented
summaries. As each document segment $x_{i}$ is summarized separately, the
divide-and-conquer approach’s fixed GPU memory footprint is independent from
the document length. This fits well with our goal of long document
summarization with limited memory. However, without access to other parts of
the document and their summaries, the summarizer struggles for content
salience estimation in each isolated segment, and generates incoherent outputs
when piecing together summaries. Though Wu et al. (2021) concatenate
previously generated summaries as part of the input, a complicated strategy is
required for training sample construction.

AWESOME is built on the memory-efficient divide-and-conquer approach, and
improves summary informativeness, coherence, and faithfulness by using newly
designed external memories for accumulating salient information from other
document segments and their generated summaries. We further augment AWESOME
with global salient content to provide important topics at the document level,
when summarizing each segment.

###  2.2 Memory and Content Augmentation

Different memory mechanisms have been studied on transformers for performing
long-range text understanding tasks. For instance, Transformer-XL Dai et al.
(2019) caches intermediate representations produced in the last document
segment and attends over these representations. Compressive Transformer Rae et
al. (2020) further increases the context range by compressing the oldest
cached representations. To simulate memory reading and writing, Recurrent
Memory Transformer Bulatov et al. (2022) includes extra memory vectors in each
text segment and passes their corresponding output vectors to the next
segment. Instead of using a memory with a fixed size, Memorizing Transformer
Wu et al. (2022a) stores all prior representations as key-value pairs, and
performs an approximate kNN lookup to retrieve representations to augment the
current segment. However, existing work on memory mechanisms focuses on
language modeling, while incorporating memory mechanisms into the decoding
process for generation tasks is nontrivial as it requires updating both
decoding states (e.g., beams) and memory states. Our work is the first to
leverage both memory mechanisms and content augmentation to incorporate global
context for the purpose of memory-efficient long document summarization.

##  3 External Memory and Global Salient Content Augmentation

Figure 1:  Illustration of AWESOME. Encoder and decoder memories can be
accessed any time and updated after reading each document segment and
generating the corresponding summary. They accumulate global context that
improves summary informativeness and coherence (§3.1). When encoding each
segment, global salient content from other segments (lines with
$\blacklozenge$-shaped ends, from both past and future) are provided to
further assist salience estimation (§3.2).

The architecture of AWESOME (Figure 1) is based on Se3 Moro and Ragazzi
(2022), where a document is summarized segment by segment in order, with the
final summary obtained by concatenating the resultant summaries. Document
sentences are split into segments with up to 768 tokens each, while reference
summary sentences are assigned to their most overlapping segment to create the
oracle summary, as detailed in Appendix A. Following Longformer Beltagy et al.
(2020), we initialize the model parameters from BART Lewis et al. (2020),
which is sufficient for separately processing each segment. AWESOME preserves
the global context and builds communications across segments with minimal GPU
memory increase, by (1) employing external memories in both the encoder and
the decoder to gather relevant information (§3.1), and (2) augmenting the
encoder with salient content from other segments (§3.2).

###  3.1 External Memory Mechanisms

We design two external memory mechanisms to efficiently install the
information flow from prior segments to the current segment. Specifically,
each memory module maintains a matrix $M\in\mathbb{R}^{m\times d}$, where
$m=1024$ is the memory size and $d=1024$ is the hidden state dimension of
BART. $M$ is updated after encoding each document segment and then passed to
the next segment. We denote the memory matrix after the $t$-th segment as
$M^{t}$. We equip each layer of the encoder and decoder with one such external
memory, with layer index omitted in the formulas for simplicity. Below we
describe two mechanisms to update $M^{t}$ and incorporate it in both the
encoding and decoding processes.

#### Compressive Memory.

For each document segment, compression-based memory caches its input vectors
to be fed into self-attention calculation. Since storing the input vectors as-
is requires the memory usage $m$ to scale linearly with the context length, we
dedicate half of $M^{t}$ to store the compressed memory, with a compression
ratio of $r$. With $H^{t}_{inp}$ denoting the matrix that contains input
vectors to the transformer self-attention, the memory compression and update
processes are:

$\displaystyle M^{t-1}_{c},M^{t-1}_{u}$
$\displaystyle=M^{t-1}[:\frac{m}{2}],M^{t-1}[\frac{m}{2}:]$ (1) $\displaystyle
M^{\prime}_{u}$
$\displaystyle=\mathrm{concat}(M^{t-1}_{u},\mathrm{SG}(H^{t}_{inp}))$ (2)
$\displaystyle M^{\prime}_{c}$
$\displaystyle=\mathrm{compress}(M^{\prime}_{u}[:-\frac{m}{2}])$ (3)
$\displaystyle M^{t}_{u}$ $\displaystyle=M^{\prime}_{u}[-\frac{m}{2}:]$ (4)
$\displaystyle M^{t}_{c}$
$\displaystyle=\mathrm{concat}(M^{t-1}_{c},M^{\prime}_{c})[-\frac{m}{2}:]$ (5)
$\displaystyle M^{t}$ $\displaystyle=\mathrm{concat}(M^{t}_{c},M^{t}_{u})$ (6)

where $\mathrm{SG}(\cdot)$ denotes stopping the gradient backpropagation to
lower GPU usage, which is a novel design by this work, and
$\mathrm{compress}(\cdot)$ performs convolutions with their stride and kernel
size set to the compression ratio $r$. $r$ is set to $5$ after tuning on the
development sets.

Next, to leverage the memory from the previous segment in summarizing the
current segment, $M^{t-1}$ is concatenated with the inputs to the self-
attentions to obtain the key-value matrices:

$\displaystyle H^{t}_{mem}$
$\displaystyle=\mathrm{concat}(M^{t-1},H^{t}_{inp})$ (7) $\displaystyle
H^{t}_{self}$
$\displaystyle=\mathrm{Attn}(\underbrace{H^{t}_{inp}}_{query},\underbrace{H^{t}_{mem}}_{key},\underbrace{H^{t}_{mem}}_{value})$
(8)

where $H^{t}_{self}$ is the output of the self-attention.

Our compression-based memory is adopted from Compressive Transformer Rae et
al. (2020), a decoder-only model for language modeling. We are the first to
apply it to both the encoder and the decoder of a Transformer model and on
long document summarization tasks.

It should be noted that compressive memory favors recency, particularly the
previous segment and its summary, so that older history can be lost during
compression despite their relevance to the current segment.

#### Attentive Memory.

To mitigate the recency bias by compressive memory, we further investigate an
attention-based memory updating mechanism, to selectively include content in
$M^{t}$. First, the memory is additionally accompanied by an extra cross-
attention in each of the encoder and decoder layers, specialized in retrieving
relevant information from $M^{t}$. Following prior study Lei et al. (2020)
that uses memories in video captioning, we update $M^{t}$ with a gate matrix
$G^{t}$ to control the amount of content to be updated:

$M^{t}=G^{t}\odot U^{t}+(1-G^{t})\odot M^{t-1}$ (9)

where $\odot$ denotes the element-wise product and $U^{t}$ is the matrix
containing vectors to update the memory. $U^{t}$ and $G^{t}$ are obtained as
follows:

$\displaystyle U^{t}$ $\displaystyle=\tanh(W_{u1}M^{t-1}+W_{u2}S^{t})$ (10)
$\displaystyle G^{t}$ $\displaystyle=\sigma(W_{g1}M^{t-1}+W_{g2}S^{t})$ (11)
$\displaystyle S^{t}$
$\displaystyle=\mathrm{Attn}(\underbrace{M^{t-1}}_{query},\underbrace{\mathrm{SG}(H^{t}_{self})}_{key},\underbrace{\mathrm{SG}(H^{t}_{self})}_{value})$
(12)

where $W_{*}$ are learnable matrices, $S^{t}$ synthesizes the current segment
via an attention calculation, and $\mathrm{SG}(\cdot)$ indicates stopping the
gradient backpropagation. In each encoder and decoder layer, an extra cross-
attention is inserted after the self-attention, where $M^{t-1}$ is attended
and incorporated into the current segment’s summarization process.

Unlike our approach, the memory in Lei et al. (2020) does not employ gradient
stopping. This omission eliminates the memory efficiency gained from the
divide-and-conquer strategy, leading to comparable high memory usage as the
efficient attention strategy. While their memory is suitable for generating
short image captions, our design with gradient stopping is crucial for
efficient long document summarization.

###  3.2 Global Salient Content Augmentation

The memory mechanisms only grant access to prior content in the documents, yet
subsequent context can also help with salience estimation, e.g., elaborating
the pros and cons of a proposed solution makes it necessary to introduce the
problem and the solution. Moreover, memories store content implicitly, so it
is unclear whether relevant information can be stored and retrieved
effectively. Therefore, we inform the system of a document’s important
sentences, which are pre-identified by a separately-trained
extractor.444Salient sentences account for $8\%$ of the sentences on
GovReport. Training an extractor takes $16.6$GB GPU memory.

The details of extractor training can be found in Appendix D. After extracting
important sentences in a document, we study two methods of injecting them into
the summarizer.

#### Text Concatenation.

For each segment, we include the extracted sentences in the following way to
prioritize long-term context. We start with the “outermost” extracted
sentences, i.e., the earliest sentence in the past segments and the last
sentence in the future segments, and repeat this process until the input has
reached the maximum length accepted by the positional encoding of the model
($1024$ for BART).555Other inclusion strategies can be explored in future
work. To differentiate the content in the current segment from the added
sentences, we prefix the current segment and the added sentences from
before/after the current segment with “Current chunk:”, “Previous important
sentences:”, and “Next important sentences:”, respectively. Text concatenation
is easy to implement and most compatible with the source modality, but the
memory usage increase is quadratic to the length of the augmented content.

#### Key-value Vectors.

To circumvent the quadratic memory increase, we join the key-value
representations of tokens in important sentences in the encoder self-
attentions, and directly inject them into the summarizer encoder. The memory
increase is only linear to the augmented content’s length.

Concretely, the summarizer encoder first encodes all document segments and
obtains the representations (i.e., encoder outputs) of tokens belonging to the
extracted important sentences. During training, the token representations of
these sentences are concatenated with the key-value matrices in the encoder
self-attentions while the query matrix remains in its original form. Up to
$1024$ tokens are concatenated via the same inclusion method for text
concatenation, to prioritize the outermost sentences. A similar idea has been
used by Memorizing Transformer Wu et al. (2022a) to include retrieved text
representations from past segments for long-form language modeling. Our method
differs in two aspects. First, we extract representations from future segments
for accurately identifying salient content. Second, we apply a learnable
projection to the augmented representations prior to key-value concatenation.
This process is crucial in improving compatibility with the original key-value
matrices.

##  4 Experimental Setups

Model |  R-1 $\uparrow$ |  R-2 $\uparrow$ |  R-L $\uparrow$ |  Ent Prec $\uparrow$ |  SummaC $\uparrow$ |  Disco $\downarrow$ |  Ent Graph $\uparrow$ |  GPU Mem $\downarrow$
---|---|---|---|---|---|---|---|---
Se3 | 46.56 | 23.22 | 44.36 | 98.24 | 14.71 | 7.37 | 1.41 | 11.1
BlockAttn | 57.46 | 26.78 | 54.82 | 97.45 | 20.43 | 5.91 | 2.05 | 25.6
Longformer | 57.40 | 26.92 | 54.70 | 97.52 | 20.39 | 5.68 | 2.05 | 25.3
LongT5 | 52.85 | 23.69 | 49.71 | 95.57 | 10.41 | 5.07 | 1.47 | 25.4
Unlimiformer | 56.35 | 25.94 | 53.83 | 92.19 | 6.05 | 5.36 | 1.96 | 27.0
Extract-Abstract | 56.89 | 24.76 | 54.26 | 92.82 | 22.07 | 4.03 | 2.09 | 16.6
PageSum | 56.80 | 23.26 | 54.11 | 89.56 | 6.82 | 3.04 | 1.88 | 24.9
Our Models
AWESOME using External Memory Only
Compressive |  56.16† |  24.73† |  53.41† | 93.55 | 17.07 |  3.83† |  1.89† | 19.9
Attentive (Attn) |  58.54∗ |  27.71∗ |  55.83∗ | 98.41 |  19.79† |  3.59† |  2.00† | 22.1
AWESOME using Global Salient Content Only
Text-concat (Txt) |  56.65† |  27.68∗ |  54.11† | 97.93 | 12.23 |  5.05† |  2.09† | 16.6
Key-value Vectors |  55.02† |  26.39† |  52.41† | 98.22 | 11.52 |  4.75† |  1.75† | 16.6
AWESOME (Attn + Txt) |  58.96∗ |  28.08∗ |  56.21∗ | 98.14 |  20.57† |  3.69† |  2.00† | 24.0
Table 2:  Results on GovReport. The best and second best results per metric
are bolded and underlined. Results by AWESOME variants that are better than
all comparisons and Se3 are shaded with green and blue, respectively. AWESOME
with attentive memory only and its full version that additionally uses salient
content through text concatenation obtain the highest ROUGE scores (in green)
and are comparable or better on faithfulness (Ent Prec & SummaC) and coherence
(Disco & Ent Graph) than base model Se3. $\ast$: our model is better than all
comparisons with approximation randomization test ($p<0.0005$); ${\dagger}$:
our model is better than Se3 ($p<0.0005$).

#### Datasets.

We conduct experiments on five long document summarization datasets with
diverse genres. GovReport Huang et al. (2021) contains long reports and their
summaries written by government research agencies. QMSum Zhong et al. (2021)
is a query-focused long meeting transcript summarization dataset, with
summary-worthy content spread over the documents. We prepend the query to all
segments. We further use another transcript summarization dataset, SummScreen
Chen et al. (2022), which contains the transcripts of TV series. The TMS
subset, with more samples and longer summaries, is selected. Moreover, we
experiment with the scientific papers and their abstracts from arXiv Cohan et
al. (2018). Finally, we test our models on summarizing full novels in BookSum
Kryscinski et al. (2022). For all datasets, we use the official train/dev/test
splits if their original data files are released. The average input lengths of
these datasets range from 6K to 143K, while the average summary lengths range
from 70 to 1294, as reported in Appendix C.1.

#### Experiment Setups and Comparisons.

Our main experiments are conducted with a GPU memory constraint of 27GB. For
each model, we truncate the input such that its maximum GPU memory usage
during training does not exceed the constraint when gradient checkpointing
Chen et al. (2016) is disabled. The constraint is specifically chosen such
that the baselines perform reasonably.666The implementations by some baseline
models require the input length to be a multiplier of $1024$. Setting a
constraint of 24GB, a more common number, would lead to further truncation and
significant performance drop. Although our model theoretically maintains a
consistent GPU memory consumption during training regardless of the number of
input tokens processed, we have chosen to restrict the maximum number of input
tokens in a training sample to $51200$ for reasonable training time. Appendix
C.2 provides information on the maximum number of input tokens that can
conform to the constraint for other models.

For baselines, in addition to the divide-and-conquer Se3 model Moro and
Ragazzi (2022), we compare with state-of-the-art or popular long document
summarization systems including Transformer models with block-wise attentions
Phang et al. (2022) (BlockAttn) and sliding-window attentions Beltagy et al.
(2020) (Longformer), where a global token can attend to and be attended by all
tokens, while other tokens can only attend to tokens in the same block or
window. Furthermore, we experiment with LongT5 Guo et al. (2022), a sliding-
window attention model pre-trained on long sequences, and Unlimiformer Bertsch
et al. (2023) that extends BART by selecting input tokens to be attended to
via KNN searching. For the extract-then-abstract approach (Extract-Abstract),
we use the same extractor as in the global salient content augmentation of our
model, and the abstractor takes as input oracle extracted sentences during
training. We also include PageSum Liu et al. (2022), which synthesizes the
output representations given by different document segments with dynamic
weights. All models are initialized from BART-large and are fine-tuned for an
identical number of steps on each dataset, employing the same learning rate,
except for LongT5 that is pre-trained on long-form data (Appendix D).

#### Evaluation Metrics.

We evaluate summary informativeness using ROUGE Lin (2004), including its
variants based on unigram and bigram overlaps, and the longest common
subsequences.

Coherence. We use DiscoScore Zhao et al. (2022) (Disco), a reference-based
metric that evaluates discourse coherence by comparing focus (e.g., nouns)
frequency and semantics between the system summary and the reference. We also
report a graph-based reference-free coherence metric Guinaudeau and Strube
(2013) (Ent Graph). Intuitively, Ent Graph measures the connectivity of
summary sentences linked by entities, reflecting the coherence of topic
transitions.

Faithfulness. We follow prior work on text generation Iv et al. (2022) and
show the precision of the entities (Ent Prec) in the summary with respect to
the document. In addition, a model-based faithfulness metric, SummaC Laban et
al. (2022), is used. SummaC measures the faithfulness of a summary sentence,
by finding the highest entailment score between the summary sentence and any
sentence in the document. The average of sentence-level faithfulness scores is
taken as the summary faithfulness score. Though there exist QA-based
faithfulness metrics Scialom et al. (2021); Honovich et al. (2021); Fabbri et
al. (2022), they rely on pre-trained QA models that are designed for short
passages and require expensive question generation processes, and are thus not
used here. Official codes for the metrics are used, whenever available.

Finally, we show the maximum size of allocated GPU memory by each model during
training. For models that depend on an external extractor (e.g., the extract-
then-abstract approach), the GPU memory usage during the extractor training is
shown if it surpasses the one used for the abstractor training.

##  5 Results

We report results by all AWESOME variants and comparison models on GovReport
in Table 2. Compared with Se3, AWESOME variants consistently achieve better
performance on both ROUGE and coherence scores, indicating the importance of
maintaining global context for accurate salience estimation of local content
and enforcing coherent transitions across segment-level summaries. This can
also be demonstrated by the sample outputs in Table 3. Summaries generated by
Se3 tend to be shorter, as Se3 fails to plan at a global level. On
faithfulness, AWESOME with attentive memory has the best entity precision
among all models and also improves SummaC over Se3, while only augmenting
AWESOME with global salient content hurts faithfulness. Inspecting the model
outputs, we find that using attentive memory improves understanding concepts
of long-term dependencies, e.g., connecting a strategy with its related
information that appears earlier in the report.

Se3: VA is required to publish information on appointment wait times at each
VA medical facility for primary care, specialty care, and hospital care and
medical services, which it does through two public websites. VA has taken a
number of actions to address deficiencies GAO found in wait-time measurement
and implementation of its scheduling policy. For wait-time measurement, these
actions included changes to the wait-time measurement definitions, provision
and documentation of scheduler training, and improved oversight through
audits, all of which have been in a state of flux for the past 6 years. On
July 12, 2019, VA provided GAO additional updates on efforts to implement
GAO’s related recommendations.
---
AWESOME: GAO recommended that VA either clarify its scheduling policy to
better define the desired date, or identify clearer wait-time measures that
are not subject to interpretation and prone to scheduler error. VA concurred
with the recommendation, which GAO has identified as among those
recommendations that warrant priority attention. VA has taken a number of
actions to address GAO’s recommendations regarding deficiencies GAO found in
wait-time measurement and implementation of its scheduling policy. For wait-
time measurement, these actions included changes to the wait-time measurement
definitions, provision and documentation of scheduler training, and improved
oversight through audits, all of which have been in a state of flux for the
past 6 years. On July 12, 2019, VA provided GAO additional updates on efforts
to implement GAO’s related recommendations.
Table 3:  Summary snippets generated by Se3 and AWESOME. AWESOME’s summary is
more coherent, with natural transitions surrounding “GAO’s recommendation”,
while Se3 abruptly introduces the topic.

Of the two types of external memory mechanisms, attentive memory outperforms
compression-based memory on all metrics, which highlights the advantage of
adaptively updating the stored context. Meanwhile, directly concatenating
salient content with the input yields higher ROUGE scores than injecting key-
value vectors into the attention calculation, though the latter is less
memory-intensive. We believe natural language-based augmentation better
interleaves with the document segment, echoing the findings by prior work on
using retrieval for question answering Wu et al. (2022b).

Importantly, under a strict GPU memory constraint, AWESOME with external
memory mechanisms and global salient content augmentation achieves the best
ROUGE scores among all models, while obtaining competitive results on other
measures. Although models with efficient attentions and PageSum can perform
remarkably when given higher-capacity GPUs as in the original work, they
generate much less informative summaries when truncation is required to comply
with the memory constraint, emphasizing the importance of studying memory-
efficient long document summarization models.

Model |  R-1 $\uparrow$ |  R-2 $\uparrow$ |  R-L $\uparrow$ |  Disco $\downarrow$ |  GPU $\downarrow$
---|---|---|---|---|---
Se3 | 29.28 | 10.51 | 25.93 | 0.77 | 8.1
BlockAttn | 30.76 | 8.26 | 26.49 | 0.50 | 22.8
LongT5 | 31.88 | 10.07 | 27.82 | 0.44 | 25.4
Unlimiformer | 30.57 | 8.82 | 26.89 | 0.49 | 26.9
Extract-Abstract | 17.63 | 5.65 | 16.02 | 4.02 | 16.5
PageSum | 29.55 | 7.38 | 26.11 | 0.31 | 21.5
AWESOME |  |  |  |  |
Attn Only |  32.84† | 11.11 |  29.04∗ | 0.75 | 19.7
Attn + Txt |  30.83† | 9.40 | 26.78 | 0.75 | 21.1
Table 4:  Results on meeting transcripts in QMSum. Equipped with attentive memory only, AWESOME achieves the best ROUGE scores. Though better than some baselines, adding extracted salient content does not further boost the performance, due to the low performance of the extractor on dialog data.  Model |  R-1 $\uparrow$ |  R-2 $\uparrow$ |  R-L $\uparrow$ |  Ent G $\uparrow$ |  GPU $\downarrow$
---|---|---|---|---|---
Se3 | 38.09 | 11.30 | 36.56 | 0.50 | 11.3
BlockAttn | 32.01 | 8.99 | 30.90 | 1.61 | 25.7
Longformer | 42.78 | 13.21 | 41.34 | 0.97 | 25.3
LongT5 | 42.03 | 12.67 | 40.76 | 1.03 | 25.4
Unlimiformer | 35.17 | 11.98 | 34.28 | 1.33 | 27.0
Extract-Abstract | 19.95 | 5.58 | 19.70 | 0.06 | 16.9
AWESOME |  |  |  |  |
Attn Only |  45.24† |  12.85† |  43.51† |  0.86† | 22.3
Attn + Txt | 39.21 | 10.51 | 37.85 | 0.54 | 23.9
Table 5:  Results on TV transcripts in SummScreen. We report Ent Graph instead of DiscoScore, as DiscoScore encounters errors when identifying focus. AWESOME with the attentive memory obtains the best R1 and RL scores, while the low accuracy of the extracted salient content leads to performance drop of the summarizer.  Model |  R-1 $\uparrow$ |  R-2 $\uparrow$ |  R-L $\uparrow$ |  Disco $\downarrow$ |  GPU $\downarrow$
---|---|---|---|---|---
Se3 | 40.74 | 17.96 | 36.87 | 1.33 | 12.8
BlockAttn | 49.12 | 21.69 | 44.40 | 1.77 | 25.7
Longformer | 48.59 | 21.45 | 43.99 | 2.17 | 25.2
LongT5 | 48.25 | 20.74 | 43.41 | 0.97 | 25.5
Unlimiformer | 47.78 | 20.58 | 43.22 | 1.22 | 26.8
Extract-Abstract | 42.37 | 16.43 | 38.62 | 1.03 | 19.5
PageSum | 46.01 | 18.77 | 41.55 | 0.88 | 26.2
AWESOME |  |  |  |  |
Attn Only |  42.59† |  18.79† |  38.55† | 1.26 | 23.7
Attn + Txt |  44.75† |  18.97† |  40.62† | 1.25 | 25.7
Table 6:  Results on arXiv papers. AWESOME variants again outperform Se3. For $80\%$ of the arXiv documents, efficient attention models and PageSum can fully train on their first halves, covering $90\%$ of the salient content that appear in the references Huang et al. (2021), thus the better ROUGE scores than models encoding smaller segments.  Model |  R-1 $\uparrow$ |  R-2 $\uparrow$ |  R-L $\uparrow$ |  Disco $\downarrow$ |  GPU $\downarrow$
---|---|---|---|---|---
Se3 | 40.78 | 10.16 | 39.77 | 010.46 | 11.5
BlockAttn | 23.45 | 3.09 | 22.09 | 190.27 | 25.7
Longformer | 20.20 | 2.45 | 18.55 | 204.48 | 25.3
LongT5 | 33.15 | 6.74 | 32.62 | 24.24 | 25.5
Unlimiformer | 38.09 | 9.55 | 37.41 | 47.72 | 27.0
AWESOME |  |  |  |  |
Attn Only | 41.11 | 10.63 | 40.20 | 010.36 | 24.0
Table 7:  Results on novels in BookSum. AWESOME with attentive memory achieves
the best performance on all metrics. Methods requiring external extractors are
not included due to the computational cost of building extractive oracles for
long novels.

On QMSum (Table 4), AWESOME with attention-based memory outperforms all
comparisons on ROUGE scores. While our models’ summaries are more coherent
than the summaries by Se3, as measured by DiscoScore, the differences among
all models are less pronounced compared to the ones on GovReport. This is
because QMSum contains shorter summaries than GovReport (69 vs. 553), thus
involving fewer topic transitions. We also find that the extractor performs
poorly on QMSum, leading to degraded results after augmenting our model with
the extracted salient content. Specifically, the F1 score of the extractor on
the test set is only $1.29$, as opposed to $27.85$ on GovReport. Compared to
our model, the extract-then-abstract model is more sensitive to the extraction
results and is thus more prone to its errors and produce summaries of the
lowest quality on QMSum.

This trend is similarly observed on SummScreen (Table 5), where the extract-
then-abstract method performs poorly and adding extracted content leads to
performance drop of AWESOME due to the low performance of the extractor.
Meanwhile, AWESOME with the attentive memory is able to obtain the best
ROUGE-1 and ROUGE-L scores.

In Table 6, we report results on arXiv, where our models consistently
outperform Se3. Models that use efficient attentions obtain the higher ROUGE
scores, because truncating arXiv documents has little effect on summary
generation—arXiv articles have the most uneven distributions of salient
content, where only about $10\%$ of new salient bigrams are located in the
second halves of the documents Huang et al. (2021). With truncation,
BlockAttn, Longformer, and LongT5 still reads in the first halves of $80\%$ of
arXiv documents.

Finally, experiments on BookSum show that the divide-and-conquer method
produces better summaries for long novels, while our method can further boost
its performance. However, the improvement is less significant compared to the
other datasets because novel plots are usually sequential and the content is
less redundant than the other types of documents we experiment with, making
the memory mechanism less necessary.

#### Human Evaluation.

We ask three human annotators who are fluent English speakers and have
extensive NLP data annotation experience to examine the outputs by BlockAttn,
Se3, and AWESOME using attentive memory and text concatenation for content
augmentation on GovReport. $15$ documents are randomly selected,777Human
evaluation at a greater scale would be too time-consuming, given the repeated
need for referring to original documents with tens of thousands of words. each
summarized by all three systems, with an average of $19.5$ sentences in the
output. Outputs from different systems are randomly shuffled and displayed.

For each summary sentence, the annotators give binary labels on where the
sentence is coherent—it uses natural transitions to logically connect with the
previous content, and does not contradict any prior statement. The annotators
also compare each summary sentence with the document and check if it is
faithful, i.e., it can be verified and entailed from the document. We further
ask the annotators to rank the summaries generated by the three systems based
on their informativeness—how well the summary captures the salient content of
the document.

Model | Coh. (%)$\uparrow$ | Faith. (%)$\uparrow$ | Inf.$\downarrow$
---|---|---|---
Se3 | 81.2 | 97.0 | 2.69
BlockAttn | 88.3 | 92.3 | 1.57
AWESOME | 86.7 | 98.7 | 1.38
Table 8:  Percentages of system summary sentences that are coherent (Coh.) and
faithful (Faith.) on GovReport, as rated by human. The average rankings of the
informativeness (Inf.) of system outputs are also reported. Krippendorff’s
$\alpha$: 0.43 (Coh.), 0.46 (Faith.), 0.61 (Inf.).

As seen in Table 8, AWESOME’s summaries are rated by human judges to be more
coherent, faithful, and informative than Se3, again evincing the importance of
incorporating global context. Though BlockAttn yields the most coherent
summaries, it produces more faithfulness errors and is less informative, due
to document truncation under a constrained memory.

#### Effects of Encoder Memory and Decoder Memory.

Model |  R-2 $\uparrow$ |  R-L $\uparrow$ |  SC $\uparrow$ |  Disco $\downarrow$ |  GPU $\downarrow$
---|---|---|---|---|---
AWESOME (Attn) | 27.71 | 55.83 | 19.79 | 3.59 | 22.1
w/o Dec Mem | 27.74 | 55.41 | 14.75 | 3.96 | 18.6
w/o Dec & Enc Mem | 23.22 | 44.36 | 14.71 | 7.37 | 11.1
Table 9:  Effects of encoder and decoder memories on AWESOME on GovReport. SC:
SummaC. Both types of memories contribute to the summary coherence, while the
decoder memory is more important for faithfulness and the encoder memory
advances the summary informativeness more significantly.

We conduct ablation studies on the usage of the encoder and decoder memories
used by AWESOME. As shown in Table 9, taking out the external memory from the
decoder significantly affects summary faithfulness and coherence, while the
informativeness measures remain comparable. This indicates that the
information from past summary segments tracked by the decoder memory is
crucial for producing coherent transitions. Furthermore, the decoder memory
mechanism may also store information relevant to the key topics or entities,
which promotes the understanding of their mentions in the current summary
segment and boosts faithfulness. The encoder memory allows comparing content
in the current segment versus its past context, which is crucial for salience
estimation. Therefore, removing encoder memory results in a more significant
drop in ROUGE scores.

##  6 Conclusion

We present AWESOME for summarizing long documents in a memory-constrained
setting. AWESOME is based on the memory-efficient divide-and-conquer strategy
and uses two mechanisms to gather global context and improve summary quality.
First, external memories on the encoder and decoder are employed to track
previously read document content and the corresponding summaries, to boost
summary informativeness, faithfulness, and coherence. Second, summary
informativeness is further promoted by informing the encoder with global
salient content predicted by an extractor via text or representation
concatenation. Experiments on GovReport, QMSum, SummScreen, arXiv, and BookSum
show that AWESOME generates summaries with better informativeness,
faithfulness, and coherence than a baseline divide-and-conquer system. Under
the same memory constraint, AWESOME also outperforms competitive models that
leverage efficient attentions or dynamic extraction to preserve global
context, highlighting its effectiveness in supplying global context.

##  7 Limitations

AWESOME’s external memory mechanism only operates from the past segments to
the future segments. We augment global salient content to cover context from
the future segments, yet more advanced solutions can be explored in future
work (e.g., making the external memory bidirectional on the encoder).

While being memory-efficient, AWESOME runs recurrently due to the external
memory mechanisms and thus requires a longer running time. We report the
running time of selected models in Appendix B.2. Compared with PageSum, which
doesn’t use divided-and-conquer, our model is twice as fast. Although the
BlockAttn is faster than our model, it suffers from higher GPU memory
requirement and worse performance. In future work, we will keep improving
model speed.

##  8 Ethical Consideration

We anticipate that one of the major use cases of AWESOME is to allow ordinary
users who have computing devices with limited memory to quickly understand
government policies and other types of long documents. However, we recognize
that the system generated summaries might not comprehensively cover the
salient content that is essential for correctly understanding the policies,
causing risks ranging from capital loss to legal liability. Moreover, as
observed in our human evaluation, system summaries might contain statements
that cannot be verified through the document, which further adds to the risks
of real-world deployment. We suggest developers who intend to use our model
for real-world application carefully study the outputs by our model before the
actual deployment.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Bertsch et al. (2023) Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley. 2023.  Unlimiformer: Long-range transformers with unlimited length input.
  * Bulatov et al. (2022) Aydar Bulatov, Yuri Kuratov, and Mikhail Burtsev. 2022.  Recurrent memory transformer.  In _Advances in Neural Information Processing Systems_.
  * Chen et al. (2022) Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022.  SummScreen: A dataset for abstractive screenplay summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8602–8615, Dublin, Ireland. Association for Computational Linguistics.
  * Chen et al. (2016) Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 615–621, New Orleans, Louisiana. Association for Computational Linguistics.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Fabbri et al. (2022) Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022.  QAFactEval: Improved QA-based factual consistency evaluation for summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2587–2601, Seattle, United States. Association for Computational Linguistics.
  * Gidiotis and Tsoumakas (2020) Alexios Gidiotis and Grigorios Tsoumakas. 2020.  A divide-and-conquer approach to the summarization of long documents.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:3029–3040.
  * Guinaudeau and Strube (2013) Camille Guinaudeau and Michael Strube. 2013.  Graph-based local coherence modeling.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 93–103, Sofia, Bulgaria. Association for Computational Linguistics.
  * Guo et al. (2022) Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022.  LongT5: Efficient text-to-text transformer for long sequences.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 724–736, Seattle, United States. Association for Computational Linguistics.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $q^{2}$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7856–7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Huang et al. (2021) Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021.  Efficient attentions for long document summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1419–1436, Online. Association for Computational Linguistics.
  * Iv et al. (2022) Robert Iv, Alexandre Passos, Sameer Singh, and Ming-Wei Chang. 2022.  FRUIT: Faithfully reflecting updated information in text.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3670–3686, Seattle, United States. Association for Computational Linguistics.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_.
  * Kryscinski et al. (2022) Wojciech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022.  BOOKSUM: A collection of datasets for long-form narrative summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6536–6558, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022.  SummaC: Re-visiting NLI-based models for inconsistency detection in summarization.  _Transactions of the Association for Computational Linguistics_ , 10:163–177.
  * Lei et al. (2020) Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara Berg, and Mohit Bansal. 2020.  MART: Memory-augmented recurrent transformer for coherent video paragraph captioning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2603–2614, Online. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li (2022) Chuan Li. 2022.  Best gpu for deep learning in 2022 (so far).
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5070–5081, Florence, Italy. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2022) Yixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb, Chenguang Zhu, Ahmed H. Awadallah, and Dragomir Radev. 2022.  Leveraging locality in abstractive text summarization.
  * Mao et al. (2022) Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Awadallah, and Dragomir Radev. 2022.  DYLE: Dynamic latent extraction for abstractive long-input summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1687–1698, Dublin, Ireland. Association for Computational Linguistics.
  * Moro and Ragazzi (2022) Gianluca Moro and Luca Ragazzi. 2022.  Semantic self-segmentation for abstractive summarization of long documents in low-resource regimes.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(10):11085–11093.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Phang et al. (2022) Jason Phang, Yao Zhao, and Peter J Liu. 2022.  Investigating efficiently extending transformers for long input summarization.  _arXiv preprint arXiv:2208.04347_.
  * Pilault et al. (2020) Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. 2020.  On extractive and abstractive neural document summarization with transformer language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9308–9319, Online. Association for Computational Linguistics.
  * Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020.  Compressive transformers for long-range sequence modelling.  In _International Conference on Learning Representations_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Roy et al. (2021) Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021.  Efficient content-based sparse attention with routing transformers.  _Transactions of the Association for Computational Linguistics_ , 9:53–68.
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021.  QuestEval: Summarization asks for fact-based evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tay et al. (2020) Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020.  Sparse sinkhorn attention.  In _International Conference on Machine Learning_ , pages 9438–9447. PMLR.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wu et al. (2021) Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021.  Recursively summarizing books with human feedback.  _arXiv preprint arXiv:2109.10862_.
  * Wu et al. (2022a) Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. 2022a.  Memorizing transformers.  In _International Conference on Learning Representations_.
  * Wu et al. (2022b) Yuxiang Wu, Yu Zhao, Baotian Hu, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2022b.  An efficient memory-augmented transformer for knowledge-intensive nlp tasks.
  * Zaheer et al. (2021) Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2021.  Big bird: Transformers for longer sequences.
  * Zhang et al. (2022) Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. 2022.  Summn: A multi-stage summarization framework for long input dialogues and documents.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1592–1604, Dublin, Ireland. Association for Computational Linguistics.
  * Zhao et al. (2022) Wei Zhao, Michael Strube, and Steffen Eger. 2022.  Discoscore: Evaluating text generation with bert and discourse coherence.
  * Zhao et al. (2020) Yao Zhao, Mohammad Saleh, and Peter J Liu. 2020.  Seal: Segment-wise extractive-abstractive long-form text summarization.  _arXiv preprint arXiv:2006.10213_.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021.  QMSum: A new benchmark for query-based multi-domain meeting summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5905–5921, Online. Association for Computational Linguistics.

##  Appendix A Divide-and-Conquer Architecture

We choose Se3 Moro and Ragazzi (2022) as our base divide-and-conquer
architecture because it can be applied to any document-summary pair. In order
to create divide-and-conquer training data for summarization, for each
document-summary pair, the document is first divided into segments (§A.1) and
each summary sentence is then assigned to a document segment as part of the
generation target (§A.2).

###  A.1 Document Segmentation

The length of each document segment is between $512$ and $768$ tokens. During
segmentation, the algorithm loops through all document sentences, as shown in
Algorithm 1. A document sentence will be added to the current segment if the
segment contains less than $512$ tokens. The current segment will be finalized
if the current segment contains more than $768$ tokens or the current sentence
is more semantically similar to the next pseudo segment than the current
segment, where the next pseudo segment is created by including future
sentences until reaching $512$ tokens. To measure the similarity between the
current sentence and a segment, we use the average cosine similarity between
the representation of the current sentence and representations of the
sentences in the segment. Sentence representations are obtained using Sentence
Transformer Reimers and Gurevych (2019) with the all-roberta-large-v1 model.

1

Data: Input document $doc$; Segment min, max length $l_{min}$, $l_{max}$

2 $segs\leftarrow[]$;

3 $currSeg\leftarrow[]$;

4

5foreach  _$sent$ in $doc$_ do

6 if  _$\text{len}(currSeg) <l_{min}$_ then

7 $currSeg\leftarrow currSeg+[sent]$;

8

9 end if

10 else if  _$\text{len}(currSeg) >l_{max}$_ then

11 $segs\leftarrow segs+[currSeg]$;

12 $currSeg\leftarrow[sent]$;

13

14 end if

15 else

16 $nextSeg\leftarrow\text{pseudoSegment}$;

17 if  _$\text{sim}(nextSeg,sent) >\text{sim}(currSeg,sent)$_ then

18 $segs\leftarrow segs+[currSeg]$;

19 $currSeg\leftarrow[sent]$;

20

21 end if

22 else

23 $currSeg\leftarrow currSeg+[sent]$

24 end if

25

26 end if

27

28 end foreach

29$segs\leftarrow segs+[currSeg]$;

30 return _$segs$_

Algorithm 1 Document Segmentation

###  A.2 Target Assignment

For each sentence in the reference summary, we calculate its ROUGE scores with
the document segments. The sentence will then be assigned to the document
segment with which yields the highest ROUGE-1 and ROUGE-2 scores.

##  Appendix B Additional Results

###  B.1 Sensitivity to Extractor Performance

| GovReport | QMSum
---|---|---
Model | R-2 | R-L | R-2 | R-L
Extract-Abstract
Prediction | 24.76 | 54.26 | 5.65 | 16.02
Oracle | 45.08 | 71.34 | 22.20 | 41.43
AWESOME (Attn + Txt)
Prediction | 28.08 | 56.01 | 9.40 | 26.78
Oracle | 29.73 | 58.84 | 12.14 | 30.08
Table 10:  Sensitivity to extractor performance inspected by feeding models
with oracle salient sentences. While AWESOME is less prone to extraction
errors, it benefits less from the improvement of the extractor, compared to
the extract-then-abstract model.

During testing, instead of providing the model with important sentences
predicted by the extractor, we feed in oracle important sentences that are
used for training the extractor and the summarizer, to discover the
summarizer’s upper-bound performance with a perfect extractor, and
characterize its sensitivity to the predicted extraction. As shown in Table
10, AWESOME is less sensitive to the performance of the extractor, as
indicated by the smaller differences in ROUGE scores between the prediction
setup and the oracle setup. While this allows AWESOME to maintain a decent
performance even when the extractor is faulty, improving the extractor
performance boosts the summary quality less significantly. The low sensitivity
might be due to the model being agnostic to the important sentences in the
current segment, as we only inform the model of the important sentences from
other segments. A possible tweak is to also highlight the importance sentences
in the current segment, which will be explored in future work.

###  B.2 Running Time

Figure 2:  Running time (batch per second) of each model. A higher number of
batches processed per second indicates a faster running speed. All models use
a batch size of 1 and the input is truncated to 16384 tokens. Under this
setup, BlockAttn and PageSum use over 48GB of GPU memory, which requires
enabling gradient checkpointing, while Se3 and AWESOME can still fit into 24GB
of GPU memory.

To compare the running time, we set the truncation threshold to $16384$ tokens
for all models and separately train them on GovReport for one epoch with a
batch size of 1. No other computation-heavy program is running at the same
time. Running time for training each model is reported in Figure 2. Se3 and
AWESOME take longer time to complete training on one batch than BlockAttn,
though they maintain a GPU memory usage of less than 24GB. By comparison,
BlockAttn runs fastest but requires more than 48GB of GPU memory.

##  Appendix C Dataset Details

###  C.1 Statistics

Statistics of datasets are reported in Table 11. For GovReport888https://gov-
report-data.github.io/, QMSum999https://github.com/Yale-LILY/QMSum, and
SummScreen (TMS subset)101010https://github.com/mingdachen/SummScreen, we use
the data released by the original papers. For arXiv, we use the version
provided by Huggingface
Datasets.111111https://huggingface.co/datasets/scientific_papers As the
original data files for BookSum are not released due to summary copyright, we
use the version reproduced by Unlimiformer Bertsch et al. (2023).

| # Samples | # Word
---|---|---
Dataset | Train | Dev | Test | Doc | Summ
GovReport | 17,516 | 974 | 973 | 9,409 | 553
QMSum | 1,257 | 272 | 279 | 9,070 | 70
SummScreen | 18,915 | 1,795 | 1,793 | 6,421 | 381
arXiv | 203,037 | 6,436 | 6,440 | 6,030 | 273
BookSum | 314 | 45 | 46 | 143,301 | 1,294
Table 11: Statistics of datasets used in our experiments.

###  C.2 Input Truncation

| Dataset
---|---
Model | Gov | arXiv | QMSum | SumScrn | Book
Se3 | 50x | 50x | 50x | 50x | 50x
Ext-Abs † | 1x ($\infty$) | 1x ($\infty$) | 1x ($\infty$) | 1x ($\infty$) | -
BlockAttn | 6x | 6x | 8x | 6x | 6x
Longformer | 8x | 8x | - | 8x | 8x
LongT5 | 6x | 6x | 6x | 6x | 6x
Unlimiformer | 2x | 2x | 2x | 2x | 2x
PageSum | 3x | 5x | 2x | - | -
AWESOME | 50x | 50x | 50x | 50x | 50x
Table 12:  Truncation thresholds (multiply by $1024$) used by each model on
different datasets to comply with the memory constraint during training.
${\dagger}$: For the extract-then-abstract model, the abstractor has a maximum
input length of $1024$, while the extractor can consume all sentences in the
document.

To fit models into our memory constraint, we truncate the model inputs. The
truncation thresholds used by each model on different datasets are shown in
Table 12. Note that Se3 and AWESOME can process documents longer than $51200$
under the memory constraint during training, but we limit the input length to
$51200$ to achieve a reasonable training time.

##  Appendix D Implementation Details

#### Extractor.

The extractor first uses a RoBERTa Liu et al. (2019) to encode each sentence
and takes the average of the final layer’s outputs as the sentence
representation. It then applies a self-attention on top of all sentence
representations. The resulting representations are converted to extraction
scores after applying a multi-layer perception with one hidden layer. The
extractor is trained with oracle extractive labels that are constructed by
greedily searching for document sentences that maximize the sum of ROUGE-1 and
ROUGE-2 scores, compared against the reference summary. We do not compute
ROUGE-L as in DYLE Mao et al. (2022), because finding the longest common
subsequence is computationally expensive and does not yield performance gain.

#### Training Parameters.

We train all models with a maximum learning rate of $5\text{\times}{10}^{-5}$,
except that LongT5 is trained with a maximum learning rate of
$1\text{\times}{10}^{-4}$. We use a running batch size of $1$ and apply
gradient accumulation to achieve an effective batch size of $8$. The numbers
of training epochs are $3$, $9$, $6$, $2$, $10$ on GovReport, QMSum,
SummScreen, arXiv, and BookSum, with warmup steps of $300$, $100$, $300$,
$1000$, and $40$. Due to the computational cost of training long document
summarization, each model is trained for a single run.

#### Model Size.

AWESOME is based on BART-large121212https://huggingface.co/facebook/bart-large
and has 708 millions of parameters.

#### Computing Infrastructure.

All experiments are conducted on RTX A6000 GPUs.

#### Evaluation Metrics.

For ROUGE Lin (2004), we use the Python implementation by
Google.131313https://pypi.org/project/rouge-score/ The official code for
DiscoScore Zhao et al. (2022) is
used141414https://github.com/AIPHES/DiscoScore, which also provides an
implementation of the Ent Graph metric Guinaudeau and Strube (2013). We
implement the entity precision measure ourselves and run the official code for
SummaC Laban et al. (2022).151515https://github.com/tingofurro/summac All
metrics used are open-source and can be distributed for research purposes.

##  Appendix E Human Evaluation Details

The three annotators in our human evaluation are all US college students and
they have taken undergraduate-level or graduate-level natural language
processing courses. Before starting the annotation, the goal of the annotation
is explained and the instruction is presented to the annotators. The
annotators are fairly compensated ($12/hr). Figure 3 shows the detailed
instruction.

Figure 3: Human evaluation instruction.