# How Important is Domain Specificity in Language Models and Instruction
Finetuning for Biomedical Relation Extraction?

###### Abstract

Cutting edge techniques developed in the general NLP domain are often
subsequently applied to the high-value, data-rich biomedical domain. The past
few years have seen generative language models (LMs), instruction finetuning,
and few-shot learning become foci of NLP research. As such, generative LMs
pretrained on biomedical corpora have proliferated and biomedical instruction
finetuning has been attempted as well, all with the hope that domain
specificity improves performance on downstream tasks. Given the nontrivial
effort in training such models, we investigate what, if any, benefits they
have in the key biomedical NLP task of relation extraction. Specifically, we
address two questions: (1) Do LMs trained on biomedical corpora outperform
those trained on general domain corpora? (2) Do models instruction finetuned
on biomedical datasets outperform those finetuned on assorted datasets or
those simply pretrained? We tackle these questions using existing LMs, testing
across four datasets. In a surprising result, general-domain models typically
outperformed biomedical-domain models. However, biomedical instruction
finetuning improved performance to a similar degree as general instruction
finetuning, despite having orders of magnitude fewer instructions. Our
findings suggest it may be more fruitful to focus research effort on larger-
scale biomedical instruction finetuning of general LMs over building domain-
specific biomedical LMs.

Keywords: relation extraction, language models, biomedical information
extraction, instruction finetuning

How Important is Domain Specificity in Language Models and Instruction
Finetuning for Biomedical Relation Extraction?

Aviv Brokman, Ramakanth Kavuluru
---
University of Kentucky, Lexington, KY USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Biomedical entities and relations among them are at the forefront of
biomedical knowledge discovery. Novel relational information among biomedical
entities is often conveyed through text in scientific literature or clinical
notes. For instance, protein-protein interactions (to understand disease
etiology and progression), gene-disease associations (to identify potential
drug targets), drug-disease treatment relations (to spot off-label usage or
assess potential for repositioning), and drug-gene interactions (to design
targeted therapies) are often discussed in literature. The extracted relations
are often used to build knowledge graphs to facilitate knowledge discovery and
knowledge-based search systems (Bakal et al., 2018; Lever et al., 2018; Zhang
et al., 2021). Adverse event relations (linking medications and adverse
effects) are often reported in clinical text and their extraction facilitates
more effective post-market pharmacovigilance (Liu et al., 2019; Botsis et al.,
2012). As such, biomedical relation extraction (RE) is a frequently pursued
task in the BioNLP community, featuring in many shared task series (e.g.,
BioCreative, i2b2, n2c2). Our current contribution is about high-level
questions regarding the value of biomedical domain specific language models
(LMs) and instruction finetuning for biomedical RE. In the rest of this
section, we overview the recent RE relevant methods landscape and frame our
contributions in that context.

The last few years have seen a proliferation of transformer-based generative
(encoder-decoder and decoder-only) LMs. In part, these have become popular
because tasks such as summarization, paraphrasing, and code generation are
fundamentally generative. But generative models have advantages over encoder-
only LMs even in some tasks that were previously addressed with the latter.
Encoder models were largely designed for classification problems, where a
classifier is trained on top of the final layer embedding of the [CLS] token.
They have been used in myriad other ways — notably in named entity recognition
(NER) and RE where embeddings of spans are fed to classifiers (Eberts and
Ulges, 2020, 2021; Zhong and Chen, 2021). These methods finetune weights from
scratch and therefore require ample data. But few- and zero-shot learning have
become focal areas of NLP research; their maturation promises improved
scalability of NLP, as manual creation of large datasets is time consuming.
Few-/zero-shot learning in encoder models is often achieved by aligning the
NLP task objective function with the pretraining objective function. Prompting
with templates containing [MASK] tokens to be predicted has been the
few-/zero-shot method for encoder-only models (Schick and Schütze, 2021; Wang
et al., 2022). Using such a strategy largely rests on whether the desired NLP
task is a classification task or can be formulated as one. In summation,
encoder-only LMs are well-suited to full finetuning (as opposed to the few-
shot setting) for a variety of non-generative tasks. The additional layering
and customization required on top of encoder models for RE can be obviated in
generative models, by contrast, given their relatively flexible approach of
formulating RE with straightforward natural language templates. This
particularly comes in handy in few- and zero-shot learning.

In the last two years, instruction finetuning (IFT) in the general-domain has
become a hot area of NLP research for generative LMs (Ouyang et al., 2022; Wei
et al., 2022; Sanh et al., 2022; Ouyang et al., 2022; Wei et al., 2021;
Touvron et al., 2023). In its basic form, IFT is conducted by converting
examples of a wide variety of labeled datasets into pairs of natural language
instructions (along with input text) and responses (containing the correct
output), and then training a generative LM on them. This typically improves
few-shot performance by aligning the model towards the goals that human users
have. Then, for a new task, the corresponding instructions are generally more
in-distribution than they would have been sans IFT.

Most advances in NLP are originally developed for the general domain, and only
later adapted to the biomedical domain. Recently, this has manifested in the
development of biomedical versions of popular general-domain LMs. Intuitively,
it is sensible to do so because the distribution of text in biomedical NLP
tasks is more similar to biomedical text than to general-domain text, and
because biomedical-specific tokenizers can more parsimoniously represent
biomedical entities. Likewise, inspired by the striking success of IFT, Parmar
et al. (2022) instruction finetuned BART-Base (Lewis et al., 2019) on a
biomedical meta-dataset they assembled (called BoX) to create In-BoXBART.

Biomedical generative LMs are quickly being trained, and we can safely assume
that larger biomedical models will be trained in the near-future. Given the
high labor, monetary, and effort costs of creating biomedical-specific LMs, it
is prudent to ask if they are worthwhile. To some degree, this has already
been tested in biomedical versions of T5 (Raffel et al., 2020; Phan et al.,
2021) and BART (Lewis et al., 2019; Yuan et al., 2022) in the context of full-
finetuning, yielding underwhelming improvements in scores. However, the tasks
tested largely require extremely short, simple generations. Plausibly, these
models would differentiate themselves from their general-domain counterparts
on more complex tasks, especially in the few-shot context. We investigate
whether this is the case in the high-value task of RE111Here we are explicitly
dealing with end-to-end RE, where entities must be extracted by the RE system
and are not provided a priori.. Specifically we ask:

  * •

Do generative models pretrained on PubMed-based scientific text outperform
those trained on general-domain text?

  * •

How do biomedical and general-domain instruction finetuned models perform
compared to their base models?

  * •

Are the answers to these questions different in the fully finetuned and few-
shot settings?

We test these questions across four biomedical RE datasets using a variety of
open biomedical and general domain LMs. (We make our code available for
review222Code available at
https://anonymous.4open.science/r/domain_specificity-BCB8 and intend to make
it public if accepted.)

## 2\. Related Work

### 2.1. Generative Relation Extraction

RE has traditionally been conducted via a pipeline in two steps — NER followed
by relation classification (RC) between pairs of entities from NER. However,
this pipeline approach was joined by end-to-end approaches that employ a joint
loss function for both NER and RC phases beginning with Miwa and Sasaki
(2014). Among them, Zeng et al. (2018) used a copy mechanism to tackle
generative RE without an intermediate NER step; several extensions followed
suit (Nayak and Ng, 2020; Hou et al., 2021; Nayak and Ng, 2020; Zeng et al.,
2020; Giorgi et al., 2022). One advantage of their strategy is that generation
can naturally handle discontinuous entities (e.g., _hand pain_ in the phrase
_hand and arm pain_). Discontinuous entities pose such a challenge to NER that
there is an entire line of research dedicated to solving it (Wang et al.,
2021; Dirkson et al., 2021). But these copy-based models require training
neural network components from scratch, and are therefore not well-suited to
few-shot learning.

Huguet Cabot and Navigli (2021) use BART to directly generate relation triples
using special tokens to demarcate roles in the triple; this still requires
training randomly initialized special tokens. Luo et al. (2022) finetune
BioGPT on natural language templates, aligning the current NLP task’s output
form most closely with the pretraining strategy. At test time, relations are
extracted from generated text using regular expressions. We adopt this general
strategy of Luo et al. (2022) in the present study. Wadhwa et al. (2023)
finetune Flan-T5 (Wei et al., 2021) using templates that convert relations
into a target sequence of structured data, rather than natural language. We
note that in preliminary experiments, our performance was much lower with
templates of this form.

### 2.2. Biomedical Language Models

Early transformer-based biomedical LMs had encoder-only architectures, usually
based on BERT (Devlin et al., 2019), and were trained on some combination of
PubMed abstracts, PubMed Central full-text articles (PMC), and MIMIC III
(Johnson et al., 2016) clinical notes. BioBERT (Lee et al., 2019) was
initialized using weights from BERT and then continually pretrained on PubMed
and PMC. Alsentzer et al. (2019) continued pretraining BioBERT on the MIMIC-
III corpus, yielding ClinicalBERT. Likewise, Gururangan et al. (2020)
continually pretrained RoBERTa (Liu et al., 2020) on additional full-text
biomedical articles from S2ORC (Lo et al., 2020), obtaining BioMed-RoBERTa.
Models that trained from scratch on biomedical text generally perform better
than their continually trained counterparts. The BERT-based BlueBERT (Peng et
al., 2019) and PubMedBERT (Gu et al., 2020), and the RoBERTa-based Bio-LM
(Lewis et al., 2020) are a few such models, with PubMedBERT being a popular
base model for many downstream applications. Shin et al. (2020) trained
versions of BioMegatron, both from scratch and through continued pre-training
from Megatron (Shoeybi et al., 2020). BioELECTRA (Kanakarajan et al., 2021), a
biomedical version of the high-performing ELECTRA (Clark et al., 2020), was
trained from scratch on PubMed and PMC.

As generative models (with encoder-decoder and decoder-only architectures)
have risen in popularity, PubMed-based generative models have been trained
from scratch as well. Biomedical versions of T5 (Raffel et al., 2020) —
SciFive, (Phan et al., 2021) and GPT — BioGPT (Luo et al., 2022) and BioMedLM
(Bolton et al., 2022) have all been trained from scratch on PubMed/PMC
corpora. BioBART (Yuan et al., 2022) is a continually pretrained BART (Lewis
et al., 2019) model with PubMed text. SciFive and BioBART showed negligible
improvement on most biomedical tasks, and modest improvements in a small
number of tasks, over their general-domain counterparts. However, we believe
the generative capabilities of SciFive and BioBART are under-explored by Phan
et al. (2021) and Yuan et al. (2022) because either the tasks tackled require
modest generation, (e.g., multiple choice answer) or are evaluated by ROUGE
scores, which only provide rough estimates of generation quality compared to
human evaluations. Luo et al. (2022) tested BioGPT on three end-to-end RE
datasets, where it substantially outperformed the comparably-sized GPT-2
Medium (Radford et al., 2019). We are unable to find any performance scores
for BioGPT-Large on end-to-end RE. BioMedLM (Bolton et al., 2022) scored much
higher than the comparably-sized GPT-Neo 2.7B model on three QA tasks.

In summary, there is evidence that on very short generation tasks, in a fully
finetuned setup, biomedical encoder-decoder LMs outperform general-domain
counterparts. But there is a dearth of information about the efficacy of
decoder-only model performance in more complex generation tasks, and even more
so for encoder-decoder models. Few-shot performance patterns of performance
are especially unknown.

### 2.3. Instruction Finetuning

Wei et al. (2022) explore the effect of IFT on zero-shot performance by
instruction finetuning on 60 tasks, constructing 10 templates per task,
finding consistent, strong improvements in performance. Sanh et al. (2022)
conduct similar experiments on a different set of datasets, yielding the T0
model. Chung et al. (2022) build on these previous attempts, ramping up the
scale of finetuning up to 1600 tasks and consequently release the Flan-T5
series of instruction finetuned models. InstructGPT (Ouyang et al., 2022) is
trained using reinforcement learning with human feedback (RLHF) (Christiano et
al., 2017; Stiennon et al., 2020) on instructions to align GPT-3 (Brown et
al., 2020) to human preferences. Wang et al. (2023) create synthetic
instruction data for IFT using GPT-3, attaining competitive performance with
the more expensively trained InstructGPT. In-BoXBART (Parmar et al., 2022),
the only attempt at biomedical instruction finetuning (to our knowledge), is
an IFT version of BART-Base trained on 32 biomedical tasks, with one template
per task, making its scale of IFT considerably smaller than that of other IFT
models.

## 3\. Task Setup

### 3.1. Training

Figure 1:  Conversion of training example of the DCE dataset into natural
language sequences.

RE instances generally consist of a source text and a set of semi-structured
relations that are asserted in it. To finetune generative LMs on a relation
extraction dataset, we convert its training examples into sequences of natural
language before finetuning the models in the usual manner. In essence, the
natural language sequences contain the source text along with an instruction
to extract relations followed by the relations expressed in natural language
form. Figure 1 illustrates our approach for the drug combination extraction
(DCE) task of extracting combination drug therapies from scientific text
(Tiktinsky et al., 2022) as we formalize below.

Let $\left(x,y\right)$ be a training example, where $x$ is the input text and
$y$ denotes the relations present present in it. To finetune LMs, we construct
a single prompting template $T=(T_{s},T_{t})$ for each dataset. A template
consists of a source transformation function $T_{s}$ and a target
transformation function $T_{t}$, which return source sequence $x^{T}$,
containing the source text $x$ along with an instruction to guide generation,
and target sequence $y^{T}$, conveying the relations $y$ in the form of
natural language:

$\displaystyle T_{s}\left(x\right)$ $\displaystyle=x^{T}$ $\displaystyle
T_{t}\left(y\right)$ $\displaystyle=y^{T}$

For encoder-decoder models, $x^{T}$ and $y^{T}$ respectively are supplied to
the encoder and decoder modules. For decoder models, the sequences $x^{T}$ and
$y^{T}$ are joined into one sequence with a function $J$, which then serves as
a single input to the model. We take $J$ to be

$J\left(x^{T},y^{T}\right)=x^{T}\|\text{\textbackslash n\textbackslash
n}\|y^{T}$

where $\|$ represents concatenation. Our formulation of $J$ separates the
source and target sequences with two line breaks, though other formulations
can be used.

### 3.2. Evaluation

For validation and test examples, we only have source sequences, and our aim
is to generate the target sequences. For encoder-decoder models, $x^{T}$ again
serves as input to the encoder. For decoder models, the input sequence is

$J\left(x^{T},\cdot\right)=x^{T}\|\text{\textbackslash n\textbackslash n}.$

The generated output sequence of the model $\hat{y}^{T}$ must be transformed
into a structured format so that predicted relations can be compared to
annotated relations to calculate performance metrics. We design an information
extractor for each template using regular expressions.

## 4\. Experimental Setup

We conduct two experiments to investigate the effect of domain-specificity.
First, we compare biomedical- and general-domain models. Second, we compare
biomedical instruction fine-tuned, general-domain instruction fine-tuned, and
base model performance. All experiments are repeated in fully finetuned and
few-shot settings. We will include appendices with templates for each dataset
as well as modeling choices (learning rate, batch size, …) in the final
camera-ready version (if accepted), given conference submission instructions
restrict their inclusion during initial review.

### 4.1. Experiments

#### 4.1.1. Biomedical vs General-Domain Pretraining

In our first experiment, we compare the efficacy of biomedical LMs and their
closest general-domain language equivalent in terms of architecture and size.
We evaluate performance of the biomedical decoder LMs BioGPT (350M) (Luo et
al., 2022), BioGPT-Large (1.5B) (Luo et al., 2022), and BioMedLM (2.7B)
(Bolton et al., 2022) and compare to their approximate general domain
equivalents, GPT2-Medium (350M), GPT2-XL (1.5B) (Radford et al., 2019), and
GPT-Neo (2.7B) (Black et al., 2021). We also evaluate performance of the
biomedical encoder-decoder LMs SciFive-Base (220M), SciFive-Large (770M) (both
PubMed version) (Phan et al., 2021), BioBART-Base (140M), and BioBART-Large
(400M) (Yuan et al., 2022) and compare to the general domain equivalents
T5-Base (220M), T5-Large (770M) (Raffel et al., 2020), BART-Base (140M), and
BART-Large (400M) (Lewis et al., 2019).

#### 4.1.2. Instruction Finetuning vs Base Models

In our second experiment, we compare the efficacy of instruction finetuned
models and their base models. No biomedical models have been instruction
finetuned in our literature review. General-domain models have been
instruction finetuned with general-domain datasets and separately with
biomedical datasets. For general-domain instructions, we compare the Flan-T5
(Wei et al., 2021) series of models to the base T5 models (Raffel et al.,
2020) they were trained from. For biomedical domain instructions, we compare
In-BoXBART (Parmar et al., 2022) to the base BART (Lewis et al., 2019), from
which it was trained.

#### 4.1.3. Few-Shot Setting

We repeat the experiments in Sections 4.1.1 and 4.1.2, but in the few-shot
setting. We emphasize that we finetune the models, rather than perform in-
context learning. We experiment with 16-shot and 64-shot learning. Few-shot
performance values are averaged over five runs with different random seeds for
selecting training examples. Barring a single exception, fully finetuned
decoder-only model performances were quite inferior (F1 $<$ 30) compared to
encoder-decoder models; so we did not perform few-shot experiments on decoder-
only models.

### 4.2. Datasets

We use four public biomedical RE datasets for our experiments. For datasets
without an established train/validation split, we select 20% of the training
dataset to serve as a validation set.

#### 4.2.1. CDR

CDR (Li et al., 2016) is a BioCreative V dataset consisting of PubMed
abstracts for which all mentions of chemicals and diseases are annotated, as
well as all relations between them describing chemical-induced diseases.
Relations all belong to a single class and are annotated at the entity level.
1500 abstracts are split into train, validation, and test sets of equal size.

#### 4.2.2. ChemProt

ChemProt (Krallinger et al., 2017) is a BioCreative VI dataset consisting of
PubMed abstracts for which all mentions of chemical compounds/drugs and
genes/proteins are annotated, as well as all intra-sentence relations between
the two entity types belonging to a set of relation types of interest.
Relations belong to the classes agonist, antagonist, upregulate, downregulate,
and substrate, and are annotated at the mention level. Since relations are
annotated at the mention level, an example may contain multiple annotated
relations that are conceptually identical but have different mentions. The
dataset is split into 1,020 training, 612 validation, and 800 test examples.

#### 4.2.3. DCE

The Drug Combinations Extraction (DCE) (Tiktinsky et al., 2022) dataset
consists of sentences selected from PubMed abstracts containing multiple
drugs, annotated for drugs given in combination. Sets of drugs are annotated
as consisting of a drug combination with a positive effect, a drug combination
with an effect not clarified by the text, or not a drug combination. Though
some studies combine the latter two relation types, (Tiktinsky et al., 2022;
Jiang and Kavuluru, 2023) we model them as-is. The dataset is split into 1362
training and 272 test instances.

#### 4.2.4. DDI

The DDI (Herrero-Zazo et al., 2013) dataset consists of texts from DrugBank
and Medline annotated for drugs/chemicals and relations holding for pairs of
co-occurring drugs. Mentions of drugs are classified as generic drug name,
brand-name drug, drug group, or non-drug active substance. Relations holding
for drug pairs are pharmacokinetic mechanism, combination effect, advice
regarding interaction, or interaction in the absence of supporting
information. Relations are annotated at the mention level. The dataset
consists of 784 DrugBank documents and 233 Medline abstracts, split up into
714 train and 303 test examples.

### 4.3. Evaluation

We calculate precision, recall, and F1 for all tasks. Evaluation is done in a
strict setting: relations are only considered correct if the entities are all
correct and relation type, if applicable, is correct. A correct entity
consists of an exact string match. For CDR, the predicted entity must match
one of the mentions corresponding to the MeSH ID of the gold entity.

## 5\. Results

Figure 2:  Comparison of fully finetuned LMs pretrained on general-domain and
biomedical text. Figure 3:  Comparison of few-shot finetuned LMs pretrained on
general-domain and biomedical text. Figure 4:  Comparison of fully finetuned
base and instruction-finetuned LMs across all datasets. Figure 5:  Few-shot
comparison of base and instruction finetuned LMs on high-performing datasets.

All results are displayed in Table 1 (page 8), primarily grouped into model
series (e.g., T5) with an in-group ordering by model size. Before moving into
the main results, we note two high level trends among the datasets and model
series: (1) Performance was higher in general for CDR and DCE than in ChemProt
and DDI, perhaps owing to the higher number of relation types in the latter
datasets. (Henceforth, we refer to CDR and DCE as high-performing and ChemProt
and DDI as low-performing as we explain other results). (2) Encoder-decoder
models performed much better than decoder-only models, with the exception of
BioGPT for CDR. In fact, decoder-only models performed so poorly that we
largely limit our further discussion to encoder-decoder models. Potentially,
this is because decoder models may be more amendable to generative outputs
such as summaries and not as suitable for RE.

|  | CDR | DCE | ChemProt | DDI
---|---|---|---|---|---
Model | # Par. | 16 | 64 | Full | 16 | 64 | Full | Full | Full
GPT-2-Medium | 350M | - | - | 16.3 | - | - | 0.2 | 14.7 | 14.2
BioGPTb | 350M | - | - | 44.1 | - | - | 0.7 | 9.7 | 17.1
GPT-2-XL | 1.5B | - | - | 16.1 | - | - | 0 | 14.9 | 24.3
BioGPT-Largeb | 1.5B | - | - | 15.6 | - | - | 1.2 | 11.1 | 21.4
GPT-Neo-2.7B | 2.7B | - | - | 10.0 | - | - | 1.5 | 11.7 | 24.0
BioMedLMb | 2.7B | - | - | 7.9 | - | - | 11.9 | 17.1 | 19.1
BART-Base | 110M | 17.6 | 27.4 | 36.1 | 8.9 | 30.6 | 38.5 | 11.3 | 12.4
BioBART-Baseb | 110M | 29.0 | 33.1 | 38.0 | 8.9 | 24.1 | 41.8 | 9.6 | 9.5
In-BoXBART$\dagger$ | 110M | 21.4 | 28.9 | 36.4 | 10.4 | 26.7 | 45.1 | 10.5 | 13.4
BART-Large | 400M | 26.4 | 32.5 | 40.1 | 9.6 | 30.4 | 41.7 | 15.7 | 17.0
BioBART-Largeb | 400M | 30.7 | 33.0 | 40.1 | 8.7 | 29.2 | 43.3 | 13.2 | 13.7
T5-Small | 60M | 18.3 | 20.3 | 29.4 | 22.5 | 18.5 | 32.3 | 0.3 | 2.9
Flan-T5-Small* | 60M | 18.9 | 22.2 | 37.1 | 10.7 | 21.2 | 33.6 | 1.3 | 2.9
T5-Base | 220M | 30.3 | 29.8 | 42.8 | 30.0 | 39.3 | 50.3 | 15.6 | 15.4
SciFive-Baseb | 220M | 28.8 | 10.1 | 41.5 | 0 | 21.6 | 47.6 | 12.4 | 11.9
Flan-T5-Base* | 220M | 30.5 | 33.1 | 45.0 | 30.0 | 33.8 | 44.1 | 16.2 | 15.2
T5-Large | 770M | 33.9 | 37.7 | 50.8 | 25.7 | 42.4 | 47.6 | 27.3 | 27.5
SciFive-Largeb | 770M | 22.7 | 5.9 | 46.3 | 3.1 | 3.7 | 46.1 | 22.3 | 20.6
Flan-T5-Large* | 770M | 35.9 | 38.4 | 50.4 | 27.4 | 45.1 | 48.9 | 26.9 | 32.0
Table 1:  Comparison of model F1-scores across a range of model types and
sizes for four biomedical RE datasets. Few-shot values are averages across
five different random seeds for selecting training examples. Models marked
with “b” are biomedically pretrained. Models marked with * are instruction
finetuned with general-domain instructions, and those marked by $\dagger$ are
instruction finetuned with biomedical instructions.

### 5.1. Biomedical vs General-Domain Pretraining

Overall, general-domain pretraining was superior to biomedical-domain
pretraining in full and few-shot finetuning experiments (Figure 2 and Figure
3). In full finetuning, this pattern held for T5 models across all datasets
and for BART models in low-performing datasets. Notable exceptions are the
BART models in high-performing datasets, where biomedical models outperformed,
or performed similarly to, corresponding general-domain models, and BioGPT
which substantially outperformed its GPT2-Medium counterpart. However, BioGPT
performed poorly in all other circumstances. In few-shot finetuning,
biomedical T5 models performed much worse than general-domain models (Figure
3). Biomedical pretraining had inconsistent effects in BART models, improving
performance for CDR while generally lowering performance for DCE.

### 5.2. Effect of Instruction Finetuning

In full finetuning, IFT versions of models generally performed similarly or
slightly better than their base versions (Figure 4). The small benefits of IFT
mostly manifested in high-performing datasets. In few-shot finetuning, both
general-domain instructions and biomedical instructions consistently improved
performance and to a similar degree (Figure 5), even though there were orders
of magnitude fewer biomedical instruction datasets.

## 6\. Discussion

We were surprised that decoder-only models performed so poorly and so much
worse than encoder-decoder models. Luo et al. (2022) found BioGPT to attain
state-of-the-art (SOTA) performance on CDR, DDI, and KD-DTI (Hou et al.,
2022). While we differed from Luo et al. (2022) in some preprocessing of the
dataset (e.g., we did not remove examples containing no relations, how many
epochs we trained for, and how we determined if predicted entities matched
gold entities), our performance on DDI was drastically lower than theirs, and
our results on two additional datasets, DCE and ChemProt are far below SOTA.
An unusual finding is that only for CDR, BioGPT performance is better than
BioGPT-Large and BioMedLM (2.7B models). Preliminary experiments with
Llama2-7b (Touvron et al., 2023) were similarly fruitless and hence we do not
report any scores given disappointing initial experiments. As encoder-decoder
models performed so much better, future practical generative RE efforts may be
more fruitful with encoder-decoder architectures; and if additional models are
to be pretrained on biomedical corpora (PubMed), encoder-decoder models may be
a safer bet.

However, training additional generative LMs (even encoder-decoder types) on
PubMed generally yielded inferior models, at least as far as RE is concerned.
This result held in both full and few-shot finetuning contexts. We found this
pattern counterintuitive, since most of the datasets we experimented with
consist of annotated texts from PubMed — one might expect that models trained
on PubMed itself would be particularly well suited for these tasks, as they
fall squarely inside of the pretraining corpus distribution. We hypothesize
that (1) the sheer quantity of tokens available for training on general-domain
text outweighs the benefits of domain-specificity, and/or (2) that biomedical-
domain models may learn inferior linguistic representations as biomedical
models are trained on a comparatively narrow subset of the distribution of the
English language, and RE requires linguistic sophistication. Thus, even for
encoder-decoder models, training biomedical LMs from scratch may not be
judicious.

As for the IFT experiments, our results do not suggest that domain-specific
training is altogether pointless. In the full finetuning case, IFT models
achieved roughly the same performance as their base models, though they
converged in far fewer epochs during training. In the few-shot setting, we
found that small-scale biomedical IFT produced performance gains comparable to
large-scale general-domain IFT. In-BoXBART was finetuned on 32 biomedical
tasks with one template per task. By contrast, Flan-T5 models were finetuned
on 1,836 tasks with multiple templates per task, and Wei et al. (2021) found
that performance continued to climb until ~300 tasks. This suggests that IFT
with additional biomedical datasets could yield strong benefits. While there
are far fewer annotated biomedical datasets than general-domain datasets, the
biomedical metadataset BigBIO (Fries et al., 2022), containing over 100
datasets currently, would be a valuable resource for biomedical IFT.
Alternatively, synthetic biomedical instruction examples could also be
generated à la Wang et al. (2023). Some nuance is required when interpreting
the few-shot performance of In-BoxBART on the CDR dataset. Among the tasks
that In-BoxBART was instruction finetuned on is the NER task for CDR. It is
possible that some of In-BoxBART’s capabilities on RE for CDR come from this
sort of leakage. In addition to our previous suggestions for further research,
our conclusions are limited to RE, a particularly high-value application of
LMs. Other biomedical tasks should be evaluated as well.

To conclude, the fast evolving landscape of public LMs and their use in many
fields naturally leads to domain specific models and IFT. In this paper, we
took the first step toward rigorous assessment of the need for domain
specificity in these models for biomedical RE. It would be interesting to see
if our findings apply to other tasks (e.g., biomedical summarization) and
domains (e.g., legal) as well.

## 7\. Bibliographical References

##

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Bakal et al. (2018) Gokhan Bakal, Preetham Talari, Elijah V Kakani, and Ramakanth Kavuluru. 2018.  Exploiting semantic patterns over biomedical knowledge graphs for predicting treatment and causative relations.  _Journal of biomedical informatics_ , 82:189–199.
  * Black et al. (2021) Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow.
  * Bolton et al. (2022) Elliot Bolton, David Hall, Michihiro Yasunaga, Tony Lee, Chris Manning, and Percy Liang. 2022.  BioMedLM.  https://crfm.stanford.edu/2022/12/15/biomedlm.html.  Accessed: 2023-08-18.
  * Botsis et al. (2012) Taxiarchis Botsis, Thomas Buttolph, Michael D Nguyen, Scott Winiecki, Emily Jane Woo, and Robert Ball. 2012.  Vaccine adverse event text mining system for extracting features from vaccine safety reports.  _Journal of the American Medical Informatics Association_ , 19(6):1011–1018.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dirkson et al. (2021) Anne Dirkson, Suzan Verberne, and Wessel Kraaij. 2021.  FuzzyBIO: A proposal for fuzzy representation of discontinuous entities.  In _Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis_ , pages 77–82, online. Association for Computational Linguistics.
  * Eberts and Ulges (2020) Markus Eberts and Adrian Ulges. 2020.  Span-based joint entity and relation extraction with transformer pre-training.  _Proceedings of the 24th European Conference on Artificial Intelligence (ECAI 2020)_ , page 2006–2013.
  * Eberts and Ulges (2021) Markus Eberts and Adrian Ulges. 2021.  An end-to-end model for entity-level relation extraction using multi-instance learning.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3650–3660, Online. Association for Computational Linguistics.
  * Fries et al. (2022) Jason Alan Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Myungsun Kang, Ruisi Su, Wojciech Kusa, Samuel Cahyawijaya, et al. 2022.  Bigbio: A framework for data-centric biomedical natural language processing.  _arXiv preprint arXiv:2206.15076_.
  * Giorgi et al. (2022) John Giorgi, Gary Bader, and Bo Wang. 2022.  A sequence-to-sequence approach for document-level relation extraction.  In _Proceedings of the 21st Workshop on Biomedical Language Processing_ , pages 10–25, Dublin, Ireland. Association for Computational Linguistics.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Herrero-Zazo et al. (2013) María Herrero-Zazo, Isabel Segura-Bedmar, Paloma Martínez, and Thierry Declerck. 2013.  The ddi corpus: An annotated corpus with pharmacological substances and drug–drug interactions.  _Journal of biomedical informatics_ , 46(5):914–920.
  * Hou et al. (2021) Yutai Hou, Yingce Xia, Lijun Wu, Shufang Xie, Yang Fan, Jinhua Zhu, Wanxiang Che, Tao Qin, and Tie-Yan Liu. 2021.  Discovering drug-target interaction knowledge from biomedical literature.
  * Hou et al. (2022) Yutai Hou, Yingce Xia, Lijun Wu, Shufang Xie, Yang Fan, Jinhua Zhu, Tao Qin, and Tie-Yan Liu. 2022.  Discovering drug–target interaction knowledge from biomedical literature.  _Bioinformatics_ , 38(22):5100–5107.
  * Huguet Cabot and Navigli (2021) Pere-Lluís Huguet Cabot and Roberto Navigli. 2021.  REBEL: Relation extraction by end-to-end language generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2370–2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Jiang and Kavuluru (2023) Yuhang Jiang and Ramakanth Kavuluru. 2023.  End-to-end $n$-ary relation extraction for combination drug therapies.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9.
  * Kanakarajan et al. (2021) Kamal raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. 2021.  BioELECTRA:pretrained biomedical text encoder using discriminators.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 143–154, Online. Association for Computational Linguistics.
  * Krallinger et al. (2017) Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Martın Pérez Pérez, Jesús Santamaría, Gael Pérez Rodríguez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio López, Umesh Nandal, et al. 2017.  Overview of the biocreative vi chemical-protein interaction track.  In _Proceedings of the sixth BioCreative challenge evaluation workshop_ , volume 1, pages 141–146.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lever et al. (2018) Jake Lever, Sitanshu Gakkhar, Michael Gottlieb, Tahereh Rashnavadi, Santina Lin, Celia Siu, Maia Smith, Martin R Jones, Martin Krzywinski, and Steven JM Jones. 2018.  A collaborative filtering-based approach to biomedical knowledge discovery.  _Bioinformatics_ , 34(4):652–659.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _CoRR_ , abs/1910.13461.
  * Lewis et al. (2020) Patrick Lewis, Myle Ott, Jingfei Du, and Veselin Stoyanov. 2020.  Pretrained language models for biomedical and clinical tasks: Understanding and extending the state-of-the-art.  In _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ , pages 146–157, Online. Association for Computational Linguistics.
  * Li et al. (2016) Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016.  Biocreative V CDR task corpus: a resource for chemical disease relation extraction.  _Database J. Biol. Databases Curation_ , 2016.
  * Liu et al. (2019) Feifan Liu, Abhyuday Jagannatha, and Hong Yu. 2019.  Towards drug safety surveillance and pharmacovigilance: current progress in detecting medication and adverse drug events from electronic health records.  _Drug safety_ , 42(1):95–97.
  * Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Ro{bert}a: A robustly optimized {bert} pretraining approach.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.  S2ORC: The semantic scholar open research corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4969–4983, Online. Association for Computational Linguistics.
  * Luo et al. (2022) Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. 2022.  BioGPT: generative pre-trained transformer for biomedical text generation and mining.  _Briefings in Bioinformatics_ , 23(6).  Bbac409.
  * Miwa and Sasaki (2014) Makoto Miwa and Yutaka Sasaki. 2014.  Modeling joint entity and relation extraction with table representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1858–1869.
  * Nayak and Ng (2020) Tapas Nayak and Hwee Tou Ng. 2020.  Effective modeling of encoder-decoder architecture for joint entity and relation extraction.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8528–8535. AAAI Press.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Parmar et al. (2022) Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. 2022.  In-BoXBART: Get instructions into biomedical multi-task learning.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 112–128, Seattle, United States. Association for Computational Linguistics.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets.  In _Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019)_ , pages 58–65.
  * Phan et al. (2021) Long N. Phan, James T. Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. 2021.  Scifive: a text-to-text transformer model for biomedical literature.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Shin et al. (2020) Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, and Raghav Mani. 2020.  BioMegatron: Larger biomedical domain language model.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4700–4706, Online. Association for Computational Linguistics.
  * Shoeybi et al. (2020) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020.  Megatron-lm: Training multi-billion parameter language models using model parallelism.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 3008–3021. Curran Associates, Inc.
  * Tiktinsky et al. (2022) Aryeh Tiktinsky, Vijay Viswanathan, Danna Niezni, Dana Meron Azagury, Yosi Shamay, Hillel Taub-Tabib, Tom Hope, and Yoav Goldberg. 2022.  A dataset for n-ary relation extraction of drug combinations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3190–3203, Seattle, United States. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wadhwa et al. (2023) Somin Wadhwa, Silvio Amir, and Byron C. Wallace. 2023.  Revisiting relation extraction in the era of large language models.
  * Wang et al. (2022) Han Wang, Canwen Xu, and Julian McAuley. 2022.  Automatic multi-label prompting: Simple and interpretable few-shot classification.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5483–5492, Seattle, United States. Association for Computational Linguistics.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  Self-instruct: Aligning language models with self-generated instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2021) Yucheng Wang, Bowen Yu, Hongsong Zhu, Tingwen Liu, Nan Yu, and Limin Sun. 2021.  Discontinuous named entity recognition as maximal clique discovery.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 764–774, Online. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Yuan et al. (2022) Hongyi Yuan, Zheng Yuan, Ruyi Gan, Jiaxing Zhang, Yutao Xie, and Sheng Yu. 2022\.  BioBART: Pretraining and evaluation of a biomedical generative language model.  In _Proceedings of the 21st Workshop on Biomedical Language Processing_ , pages 97–109, Dublin, Ireland. Association for Computational Linguistics.
  * Zeng et al. (2020) Daojian Zeng, Ranran Haoran Zhang, and Qianying Liu. 2020.  Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning.
  * Zeng et al. (2018) Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018.  Extracting relational facts by an end-to-end neural model with copy mechanism.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 506–514, Melbourne, Australia. Association for Computational Linguistics.
  * Zhang et al. (2021) Rui Zhang, Dimitar Hristovski, Dalton Schutte, Andrej Kastrin, Marcelo Fiszman, and Halil Kilicoglu. 2021.  Drug repurposing for covid-19 via knowledge graph completion.  _Journal of biomedical informatics_ , 115:103696.
  * Zhong and Chen (2021) Zexuan Zhong and Danqi Chen. 2021.  A frustratingly easy approach for entity and relation extraction.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 50–61, Online. Association for Computational Linguistics.


# Generating Educational Materials with Different Levels of Readability using
LLMs

Chieh-Yang Huang<EMAIL_ADDRESS>MetaMetrics Inc.DurhamNorth CarolinaUSA ,
Jing Wei<EMAIL_ADDRESS>MetaMetrics Inc.DurhamNorth CarolinaUSA and  Ting-
Hao Kenneth Huang<EMAIL_ADDRESS>Pennsylvania State UniversityUniversity
ParkPennsylvaniaUSA

(2018)

###### Abstract.

This study introduces the leveled-text generation task, aiming to rewrite
educational materials to specific readability levels while preserving meaning.
We assess the capability of GPT-3.5, LLaMA-2 70B, and Mixtral 8x7B, to
generate content at various readability levels through zero-shot and few-shot
prompting. Evaluating 100 processed educational materials reveals that few-
shot prompting significantly improves performance in readability manipulation
and information preservation. LLaMA-2 70B performs better in achieving the
desired difficulty range, while GPT-3.5 maintains original meaning. However,
manual inspection highlights concerns such as misinformation introduction and
inconsistent edit distribution. These findings emphasize the need for further
research to ensure the quality of generated educational content.

Educational Material Generation, Text Readability, Text Generation, Large
Language Model

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; May 11, 2024; Honolulu, Hawii††booktitle: Woodstock ’18: ACM Symposium
on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††isbn:
978-1-4503-XXXX-X/18/06††copyright: none Figure 1. Given (A) a source text,
(B) a readability level for the source text, and (C) an intended readability
level, the leveled-text generation aims at rewriting the source text in the
intended readability level while preserving the meaning. Note that in this
paper, we use a widely adopted readability level in the education context, the
Lexile Scale. In the provided example, the sentences are longer and the words
are less common in the resulting text.

##  1\. Introduction

Prior research has shown that students learn more effectively from reading
materials that match their level of readability, optimally balancing
improvement and cognitive load (MetaMetrics, 2022; Yang et al., 2021; Ehsan
Namaziandost and Ahmadi, 2019). Consequently, curating educational content to
meet the diverse reading abilities of students is a crucial step toward better
learning outcomes. Platforms such as Newsela (new, [n. d.]), Simple English
Wikipedia (sim, [n. d.]), and CommonLIT (com, [n. d.]), among others, have
developed texts at different levels of complexity for educational purposes.

The process of rewriting texts to different levels typically involves
iterative editing to ensure that the revised texts meet the desired difficulty
criteria. This readability assessment is based on various linguistic features,
with sentence length and word frequency identified as key factors in previous
studies (Fitzgerald et al., 2015). Although this process appears
straightforward, accurately adjusting these elements to achieve the target
reading difficulty is challenging. This task becomes even more complex for
young learners, where factors such as decodability (Menton and Hiebert, 1999),
information load (Landauer and Dumais, 1997), and other elements play a
significant role (Fitzgerald et al., 2015, 2016).

This paper introduces the leveled-text generation task. As shown in Figure 1,
given (A) a source text, (B) the readability level of the source text, and (C)
the desired readability level, the goal is to rewrite the given text so that
the resulting text (i) is within the desired readability level and (ii) also
preserves the original meaning.

As our first attempt at solving the leveled text generation task, we assess
the capability of three LLMs, including GPT-3.5 (OpenAI, 2022), LLaMA-2 70B
(Touvron et al., 2023), and Mixtral 8x7B (Jiang et al., 2024), to generate
content at various levels of readability through zero-shot and few-shot
prompting techniques. Our evaluation involved 100 educational materials
processed by GPT-3.5, LLaMA-2 70B, and Mixtral 8x7B. The findings indicate
that providing a few examples will significantly improve performance in both
manipulating the content readability and preserving the same information. This
might also suggest that the concept of the leveled-text generation and Lexile
scale does not inherently exist, as showing examples could help the model
understand the task and realize what the texts should look like within the
desired difficulty level. When comparing different LLMs, we found that LLaMA-2
70B was the most effective in achieving the desired difficulty range, while
GPT-3.5 excelled in maintaining the meaning of the original text.

We also conducted a manual quality inspection of 10 articles considering
potential concerns if the generated texts were used as learning materials. Our
analysis revealed a few issues: (i) the potential for introducing
misinformation, particularly in the form of altered quotations and factual
inaccuracies, and (ii) the inconsistent distribution of edits within the text,
leading to an uneven readability level throughout the article. These findings
highlight the need for further research and development to address these
concerns and ensure the quality of the generated educational content. This
preliminary survey helped us identify five critical points for future advances
in leveled text generation tasks.

##  2\. Related Work

This task is related to (i) text readability manipulation and (ii) educational
content generation.

#### Text Readability Manipulation.

Most of the tasks related to manipulating text readability focus on text
simplification. S. Bautista et al. (2009) simplified texts through (i) rule-
based conversion to sentence structures and (ii) replacing difficult words
with easier synonyms. Bingel and Søgaard (2016) introduced a structured
approach to text simplification using conditional random fields on dependency
graphs to predict compressions and paraphrases. Swain et al. (2019) developed
an efficient text simplification technique using the WordNet model in NLTK.
Alkaldi and Inkpen (2023) trained their own readability classifier and
designed a reinforcement learning framework to train a text simplification
model based on a GRU sequence-to-sequence model with attention. Alkaldi and
Inkpen (2023)’s work, however, is currently only trained in sentence-level
simplification. Recent advancements in LLMs have shown promising results. Feng
et al. (2023) investigated zero-shot and few-shot learning capabilities of
LLMs, demonstrating superior performance. Maddela et al. (2021) introduced a
hybrid approach combining linguistically-motivated rules with a neural
paraphrasing model. Other studies focused on improving transparency and
explainability. Cristina Garbacea et al. (2020) proposed a structured
pipeline, analyzing text complexity prediction and complex component
identification.

While these studies have made significant contributions, further research is
needed on generating texts at specific readability levels (including both
increasing and decreasing text readability) while preserving meaning. Our work
on the leveled-text generation task benchmarks the LLMs performance and
explores the possibility of this new topic.

#### Educational Content Generation.

Recent studies have explored the potential of large language models (LLMs) in
generating educational content. Leiker et al. (2023) investigated the use of
LLMs for creating adult learning content at scale; MacNeil et al. (2022)
focused on automatically generating computer science learning materials; Gao
et al. (2023) specifically investigated the application of LLMs to spoken
language learning; Jury et al. (2024) evaluated LLM-generated worked examples
in an introductory programming course; and Xiao et al. (2023) applied LLMs to
generate reading comprehension exercises. Despite the adoption of LLMs in
various fields, this particular work focuses on leveraging their potential for
language learning by rewriting texts to different readability levels.

##  3\. Benchmark the Leveled-Text Generation Task

To establish a first benchmark for the leveled-text generation task, we
compiled a parallel dataset comprising 30K pairs of leveled texts and
experimented with it using three LLMs, namely GPT-3.5, LLaMA-2 70B, and
Mixtral 8x7B. We evaluate the performance based on two aspects: (i)
manipulating readability and (ii) content preservation.

Note that we chose the Lexile scale as a tool to assess readability, given its
wide adoption in educational settings to measure text complexity. The Lexile
Framework evaluates a student’s reading skills and the complexity of reading
materials on the same scale (MetaMetrics, 2022). This makes it easier for
students to choose books that are just right for their reading level, helping
them to enhance their language skills more effectively. Given its wide
adoption in K-12 educational settings to measure text complexity, we decided
that the Lexile measure was the best fit for our readability analysis.

###  3.1. Task Definition

As shown in Figure 1, given (A) a source text, (B) a readability level of the
source text, and (C) an intended readability level of the target text, the
leveled-text generation task should rewrite the source text to meet
characteristics within the intended readability level, such as the vocabulary
used, sentence structure, sentence length, and so on.

The Lexile Framework suggests that readability measures should rely on
different features for different groups of students. For early-level learners
(usually lower than grade 4 or a Lexile level of 750L), readability relies on
many specific features, such as decodability (Menton and Hiebert, 1999),
information load (Landauer and Dumais, 1997), phonetic features, word
structure, sentence complexity, and so on (Fitzgerald et al., 2015, 2016). For
upper-level learners, the readability measurement mostly relies on word
frequency and sentence length (Fitzgerald et al., 2015). In this task, we
would like to see if LLMs can learn the characteristics of the target Lexile
level and rewrite the source text accordingly.

###  3.2. Evaluation Metrics.

We evaluate two aspects: (i) whether the model could correctly rewrite the
texts to the intended Lexile score and (ii) whether the model could still
preserve the same information.

To evaluate whether the resulting texts are aligned with the intended Lexile
score, we first used the Lexile Analyzer to measure the resulting texts and
obtained the resulting Lexile score. Several metrics were then calculated
based on the intended and the resulting Lexile score:

  1. (1)

Mean Absolute Error (MAE), representing the absolute deviation between the
intended and the resulting Lexile scores;

  2. (2)

Match Rate, indicating the proportion of instances where the resulting Lexile
score was within a $\pm$50 range of the intended score; and

  3. (3)

Directional Accuracy, reflecting the proportion of instances where the
resulting Lexile score moved in the intended direction (toward easier or more
difficult levels).

To measure information preservation, we used BERTScores (Zhang* et al.,
2020)111We used the microsoft/deberta-xlarge-mnli model to obtain the
embedding., semantic similarity (Reimers and Gurevych, 2019)222We used
sentence transformer with the sentence-transformers/all-mpnet-base-v2 model,
and normalized edit distance to assess content preservation between the source
texts and the resulting texts.

###  3.3. Dataset

We started with a leveled-text corpus, a collection of articles from various
language-learning books organized into 1,690 sets. Each set contains articles
that share the same title, meaning they cover the same topic but are written
at different readability levels to suit various reading abilities. These
articles range from two to six versions per set, with an average of 825 words
per article. The collection can be represented as follows:

(1) $\text{Leveled Text
Corpus}=\\{\\{A^{1}_{1},A^{1}_{2}\\},\\{A^{2}_{1},A^{2}_{2},...A^{2}_{6}\\},...,\\{A^{n}_{1},...,A^{n}_{m}\\}\\}$

In this representation, $A^{i}_{j}$ refers to a specific language learning
article, where $i$ indicates the set index and $j$ is the article index within
that set. Articles in the same group ($i$) discuss the same topic but at
different complexity levels. We assigned a readability score to each article
by running the Lexile analyzer (MetaMetrics, 2022).

The dataset was then split into three parts according to the set index: 90%
for training (1521 sets), 5% for validation (84 sets), and 5% for testing (85
sets). To form a parallel dataset for the leveled-text generation task, we
permuted all pairs of articles within each set. This new parallel dataset
consists of the following fields:

  1. (1)

Source text: The original article that needs to be rewritten to match a
different readability level.

  2. (2)

Source Lexile score: The readability score of the source text, used to guide
the adjustment process in terms of simplifying or complicating the text.

  3. (3)

Target text: The rewritten article, adjusted to the desired readability level.

  4. (4)

Target Lexile score: The intended readability score for the rewritten article,
serving as a goal for the leveled-text generation task.

After preprocessing, the train, validation, and test sets comprised 29,990,
1680, and 1700 pairs of leveled texts, respectively. Note that although our
test set comprised 1700 instances, we only selected 100 samples for this
preliminary analysis.

###  3.4. Leveled-Text Generation with LLMs

As the first attempt to solve the leveled-text generation task, we
experimented with prompting popular LLMs, namely GPT-3.5, LLaMA-2 70B, and
Mixtral 8x7B. The purpose of this method is to explore the capabilities of
LLMs in generating leveled texts and to establish a baseline for future
improvements.

We tried prompting techniques with both zero-shot learning and few-shot
learning. For zero-shot learning, we defined the Lexile score and then
provided the source text, source Lexile score, and target Lexile score to the
model (see Appendix A for the actual prompt used.)

However, the definition of Lexile score can still be vague for LLMs. To
address this, we also tried few-shot learning, where actual examples from the
training set were presented to teach LLMs what the text within a particular
readability level should look like.

The few-shot learning examples for each sample were chosen based on the
corresponding Lexile scores of the source and target texts. We identified
training samples whose source and target Lexile scores were both within a
50-point range of the target sample’s corresponding scores:

(2) A training sample is qualified if:
$\displaystyle\begin{Bmatrix}|Lexile_{train-source}-Lexile_{sample-
source}|\leq 50\\\ |Lexile_{train-target}-Lexile_{sample-target}|\leq
50\end{Bmatrix}$

From the qualifying training samples, we selected the shortest $n$ samples to
minimize the context size of the prompt. In the few-shot prompt, we included
the source text, source Lexile score, target text, and target Lexile score as
examples for in-context learning (see Appendix A for the actual prompt used).

Method | Model | #Shot | Support | Lexile Score | BERTScore |  Semantic Similarity$\uparrow$ |  Normalized Edit Distance
---|---|---|---|---|---|---|---
MAE$\downarrow$ | Match$\uparrow$ | Direction$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ | F1$\uparrow$
Zero-shot | GPT-3.5 | 0 | 100 | 257.6 | 15.00% | 80.00% | 79.87% | 76.54% | 78.03% | 0.893 | 0.941
LLaMA-2 70B | 0 | 100 | 206.5 | 15.15% | 71.72% | 75.49% | 71.82% | 73.56% | 0.894 | 0.947
Mixtral 8x7B | 0 | 100 | 256.0 | 11.00% | 79.00% | 74.69% | 73.74% | 74.18% | 0.894 | 0.951
Few-shot | GPT-3.5 | 3 | 100 | 205.3 | 15.00% | 75.00% | 82.85% | 80.18% | 81.45% | 0.937 | 0.934
LLaMA-2 70B | 1 | 99 | 172.9 | 22.22% | 86.87% | 72.96% | 71.25% | 73.01% | 0.887 | 0.949
Mixtral 8x7B | 3 | 100 | 210.9 | 12.00% | 83.00% | 72.89% | 69.88% | 71.27% | 0.935 | 0.929
Table 1. Performance comparison for different models. In Lexile score
measurements, MAE refers to the mean absolute error between the resulted
Lexile score and the intented Lexile score; Match measures whether the
resulted Lexile scores fall within the range of the intented Lexile score
$\pm$ 50; and Direction measures whether the resulted Lexile score moves
toward the intended direction (i.e., easier or harder).

###  3.5. Benchmark Results

Table 1 shows the benchmark results. Note that for few-shot learning, we tried
1-shot, 3-shot, and 5-shot samples, but only the best-performing one was
reported.

Our findings suggest that providing a few examples significantly improves
performance, as few-shot learning outperforms zero-shot learning. This
phenomenon also indicates that the concept of “leveled-text generation” with
respect to the Lexile scale does not inherently exist in these LLMs. To fully
utilize LLMs, it is necessary to teach them what Lexile means and what the
texts would possibly look like at each level.

When comparing different LLMs, we found that LLaMA-2 70B performs best in
adjusting readability, whereas GPT-3.5 demonstrates superior performance in
preserving content and meaning. However, a major concern is that although
LLaMA-2 70B achieves the best Lexile Score measure (lowest MAE = 172.9;
highest match rate = 22.22%), its normalized edit distance is also the
highest, meaning that it produces the least edit to the content (see Section
4: Bias in Current Models for the discussion of the desired edit behavior).
Another concern when using LLaMA-2 70B is its smaller context size (4K
tokens), which can cause the task to fail if the prompt exceeds the context
size limitation.

In Figure 2 and Figure 3, we present scatter plots to illustrate the
distribution of intended Lexile versus resulting Lexile scores and intended
Lexile shift versus resulting Lexile shift. These figures provide valuable
insights into the behavior of the language models. The red-shaded area
represents the region where resulting scores fall within $\pm$50 points of the
intended scores. In both Figure 2 and Figure 3, we can clearly observe that a
higher proportion of the data points are located above the red-shaded area,
indicating that the generated texts are generally more complex than the
intended level. Although this bias has been identified, more research is
necessary to understand its underlying causes and develop potential solutions
to mitigate its effects.

(a) Zero-shot GPT-3.5

(b) Zero-shot LLaMA-2 70B

(c) Zero-shot Mixtral 8x7B

(d) Few-shot GPT-3.5

(e) Few-shot LLaMA-2 70B

(f) Few-shot Mixtral 8x7B

Figure 2. Scatter plots comparing intended and resulting Lexile scores for
text generated by GPT-3.5, LLaMA-2 70B, and Mixtral 8x7B models in zero-shot
and few-shot settings. The red-shaded area represents the region where
resulting scores are within $\pm$50 points of the intended scores. A higher
proportion of data points fall above the red area, indicating that the
resulting Lexile scores tend to skew higher than the intended scores,
suggesting a tendency for the models to generate slightly more complex text
than the target difficulty level, regardless of the specific model or
prompting approach used.

(a) Zero-shot GPT-3.5

(b) Zero-shot LLaMA-2 70B

(c) Zero-shot Mixtral 8x7B

(d) Few-shot GPT-3.5

(e) Few-shot LLaMA-2 70B

(f) Few-shot Mixtral 8x7B

Figure 3. Scatter plots comparing intended and resulting Lexile shifts for
text generated by GPT-3.5, LLaMA-2 70B, and Mixtral 8x7B models in zero-shot
and few-shot settings. The Lexile shift is calculated as the difference
between the intended or resulting Lexile score and the source Lexile score.
Data points falling within the first and third quadrants indicate the correct
direction of change in text complexity. However, the overall distribution of
points still exhibits a skew towards higher difficulty levels, suggesting that
the models tend to generate text that is slightly more complex than the
intended shift, regardless of the specific model or prompting approach
employed.

##  4\. Detailed Inspection and Discussion

We conducted a manual investigation of 10 samples selected from the test set.
We reviewed the generated texts and discussed potential issues in using these
outputs as educational materials.

#### Problems with Text Shortening and Lengthening

Changing content readability may involve shortening or lengthening the given
text. LLMs are good at shortening texts with advanced prompting techniques
such as Chain-of-Density (Adams et al., 2023). Expanding texts, however,
requires the introduction of new information. While this is less problematic
for narrative genres, it poses significant challenges for factual content like
science and news, where maintaining accuracy and minimizing misinformation are
crucial. In our review of 30 generated articles, we noticed several
modifications to quotations, which is not ideal. Recent studies suggest that
Retrieval-Augmented Generation (RAG) can mitigate hallucinations (Shuster et
al., 2021), but implementing an RAG system that integrates up-to-date,
external information remains a challenge.

#### Limitations in Leveled-Text Generation

Leveled-text generation, particularly for scientific materials, might not be
feasible for all Lexile levels, especially for younger learners. Paraphrasing
sentences or selecting frequently appearing words have their limits in
changing text difficulty. For example, explaining “photosynthesis” to kids
might require a new explanation that involves age-appropriate analogies or
visuals. Thus, the leveled-text generation task, which aims to rewrite texts,
may not adequately address such needs. To significantly change the original
content to achieve the desired readability, a more sophisticated LLM might be
able to complete the task, but it is outside the scope of the current leveled-
text generation task.

#### Incorporating Educational Lessons

In educational contexts, materials usually come with learning objectives, such
as grammar, vocabulary, knowledge, etc. Integration of these educational
elements into rewritten texts remains an unresolved challenge. Moreover,
determining the appropriate learning objectives for students at different
levels is crucial yet challenging. We believe human involvement is essential,
but more research is needed to explore how such involvement can be done.

#### Important Information Should Remain Unchanged

There is a need among content creators to retain specific pieces of
information, such as key terms (e.g., “Photosynthesis”), essential sentences
(e.g., a quote), or particular sections deemed more important than others.
Current LLMs may address this requirement through prompt engineering. However,
developing an intuitive interface that allows users to (i) highlight areas of
text that should remain unchanged and (ii) verify whether the generated texts
meet these criteria is essential.

#### Limitations and Bias in Current Models

We identified biases in the three LLMs. First, we found a tendency for the
models to produce shorter texts than the originals (Original: 825 words, Few-
Shot: 350-500 words), regardless of whether the intention was to simplify or
complexify the texts. This bias may have been influenced by the use of shorter
few-shot samples. However, a similar pattern emerged in zero-shot scenarios
(Original: 825 words, Zero-Shot: 500-600 words), suggesting a possible bias in
off-the-shelf LLMs.

Second, the distribution of edits within articles often appeared uneven, with
some paragraphs remaining unchanged while others underwent significant
revisions (this appears more frequently in the texts generated by LLaMA-2
70B). This inconsistent editing pattern is unsuitable for educational
materials, even if the articles achieve the desired readability. More research
is needed to identify the causes and develop solutions.

##  5\. Conclusion

Our investigation into leveled-text generation using LLMs underscores the
potential and challenges of automating educational content creation. While
LLaMA-2 70B shows promise in adjusting text complexity, GPT-3.5 is better at
maintaining content meaning. Future research should address the nuanced
demands of educational content, including accurate representation of
information, integration of learning objectives, and retention of key
information, to enhance leveled-text generation process.

## References

  * (1)
  * com ([n. d.]) [n. d.].  CommonLit.  https://www.commonlit.org/en/texts/celebrating-chinese-new-year/paired-texts CommonLIT identified text pairings based on similar themes, topic, or writing style..
  * new ([n. d.]) [n. d.].  Newsela.  https://newsela.com/
  * sim ([n. d.]) [n. d.].  Simple English Wikipedia.  https://simple.wikipedia.org/wiki/Main_Page The Simple English Wikipedia is a Simple English language version of Wikipedia, an online encyclopedia, written in a language that is easy to understand but is still natural and grammatical..
  * Adams et al. (2023) Griffin Adams, Alexander Fabbri, Faisal Ladhak, Eric Lehman, and Noémie Elhadad. 2023.  From sparse to dense: GPT-4 summarization with chain of density prompting.  _arXiv preprint arXiv:2309.04269_ (2023).
  * Alkaldi and Inkpen (2023) Wejdan Alkaldi and Diana Inkpen. 2023.  Text Simplification to Specific Readability Levels.  _Mathematics_ 11, 9 (apr 26 2023), 2063.
  * Bingel and Søgaard (2016) Joachim Bingel and Anders Søgaard. 2016.  Text Simplification as Tree Labeling. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics.
  * Cristina Garbacea et al. (2020) Cristina Garbacea, Mengtian Guo, Samuel Carton, and Q. Mei. 2020.  An Empirical Study on Explainable Prediction of Text Complexity: Preliminaries for Text Simplification.  _arXiv.org_ (2020).
  * Ehsan Namaziandost and Ahmadi (2019) Fariba Rahimi Esfahani Ehsan Namaziandost and Sheida Ahmadi. 2019.  Varying levels of difficulty in L2 reading materials in the EFL classroom: Impact on comprehension and motivation.  _Cogent Education_ 6, 1 (2019), 1615740.  https://doi.org/10.1080/2331186X.2019.1615740 arXiv:https://doi.org/10.1080/2331186X.2019.1615740
  * Feng et al. (2023) Yutao Feng, Jipeng Qiang, Yun Li, Yunhao Yuan, and Yi Zhu. 2023.  Sentence Simplification via Large Language Models.  (2023).
  * Fitzgerald et al. (2016) Jill Fitzgerald, Jeff Elmore, Elfrieda H Hiebert, Heather H Koons, Kimberly Bowen, Eleanor E Sanford-Moore, and A Jackson Stenner. 2016.  Examining text complexity in the early grades.  _Phi Delta Kappan_ 97, 8 (2016), 60–65.
  * Fitzgerald et al. (2015) Jill Fitzgerald, Jeff Elmore, Heather Koons, Elfrieda H Hiebert, Kimberly Bowen, Eleanor E Sanford-Moore, and A Jackson Stenner. 2015.  Important text characteristics for early-grades text complexity.  _Journal of Educational Psychology_ 107, 1 (2015), 4.
  * Gao et al. (2023) Yingming Gao, Baorian Nuchged, Ya Li, and Linkai Peng. 2023.  An Investigation of Applying Large Language Models to Spoken Language Learning.  _Applied Sciences_ 14, 1 (dec 26 2023), 224.
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024\.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ (2024).
  * Jury et al. (2024) Breanna Jury, Angela Lorusso, Juho Leinonen, Paul Denny, and Andrew Luxton-Reilly. 2024.  Evaluating LLM-generated Worked Examples in an Introductory Programming Course. In _Proceedings of the 26th Australasian Computing Education Conference_. ACM.
  * Landauer and Dumais (1997) Thomas K Landauer and Susan T Dumais. 1997.  A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.  _Psychological review_ 104, 2 (1997), 211.
  * Leiker et al. (2023) Daniel Leiker, Sara Finnigan, Ashley Ricker Gyllen, and Mutlu Cukurova. 2023.  Prototyping the use of Large Language Models (LLMs) for adult learning content creation at scale.  (2023).
  * MacNeil et al. (2022) Stephen MacNeil, Andrew Tran, Juho Leinonen, Paul Denny, Joanne Kim, Arto Hellas, Seth Bernstein, and Sami Sarsa. 2022.  Automatically Generating CS Learning Materials with Large Language Models. In _Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2_. ACM.
  * Maddela et al. (2021) Mounica Maddela, Fernando Alva-Manchego, and Wei Xu. 2021.  Controllable Text Simplification with Explicit Paraphrasing. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics.
  * Menton and Hiebert (1999) Shailaja Menton and Elfrieda H Hiebert. 1999.  Literature Anthologies: The Task for First-Grade Readers.  (1999).
  * MetaMetrics (2022) MetaMetrics. 2022.  Lexile Framework for Reading Development and Validity Evidence.  https://hubsupport.lexile.com/Images/Lexile%20Framework%20for%20Reading%20Validity%20Evidence_2022.pdf
  * OpenAI (2022) OpenAI. 2022.  ChatGPT: A large language model.  https://openai.com/blog/chatgpt/ Knowledge cutoff: September 2021.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.  https://arxiv.org/abs/1908.10084
  * S. Bautista et al. (2009) S. Bautista, Pablo Gervás, and R. I. Madrid. 2009.  Feasibility Analysis for SemiAutomatic Conversion of Text to Improve Readability.  _Information and Communication Technologies and Accessibility_ (2009).
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval Augmentation Reduces Hallucination in Conversation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominican Republic, 3784–3803.  https://doi.org/10.18653/v1/2021.findings-emnlp.320
  * Swain et al. (2019) Debabrata Swain, Mrunmayee Tambe, Preeti Ballal, Vishal Dolase, Kajol Agrawal, and Yogesh Rajmane. 2019.  _Lexical Text Simplification Using WordNet_.  Springer Singapore, 114–122.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Xiao et al. (2023) Changrong Xiao, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Lei Xia. 2023.  Evaluating Reading Comprehension Exercises Generated by LLMs: A Showcase of ChatGPT in Education Applications. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_. Association for Computational Linguistics.
  * Yang et al. (2021) Ya-Han Yang, Hsi-Chin Chu, and Wen-Ta Tseng. 2021.  Text difficulty in extensive reading: Reading comprehension and reading motivation.  (2021).
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  BERTScore: Evaluating Text Generation with BERT. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=SkeHuCVFDr

##  Appendix A Prompt

Here, we provide the prompt we used for the experiments.

The zero-shot prompt we used is as follow. We quickly introduce the Lexile
measurement and then provide (i) {SOURCE-TEXT}, (ii) {SOURCE-LEXILE}, and
(iii) {TARGET-LEXILE} as the information.

  * A Lexile measure is defined as “the numeric representation of an individual’s reading ability or a text’s readability (or difficulty),” where lower scores reflect easier readability and higher scores indicate harder readability.

In this task, we are trying to rewrite a given text into the target Lexile
level and keep the original meaning and information. Given the original draft
(Lexile = {SOURCE-LEXILE}):

[TEXT START]

{SOURCE-TEXT}

[TEXT END]

Rewrite the above text and {TASK} to the difficulty level of Lexile = {TARGET-
LEXILE}.

The few-shot prompt we used is as follow. There are three sections in the
prompt, introduction, example, and task (separated by “START-INTRO” tags). The
section tag will not appear in the final prompt. For few-shot learning with
more than 1 shot, only the example section will be repeated several times.

  * ((START-INTRO))
A Lexile measure is defined as ”the numeric representation of an individual’s
reading ability or a text’s readability (or difficulty),” where lower scores
reflect easier readability and higher scores indicate harder readability.

In this task, we are trying to rewrite a given text into the target Lexile
level and keep the original meaning and information.
((END-INTRO))

((START-EXAMPLE))
Here is an example.
Lexile = {SOURCE-LEXILE}

[TEXT START]

{SOURCE-TEXT}

[TEXT END]

Rewritten Text of Lexile = {TARGET-LEXILE}

[TEXT START]

{TARGET-TEXT}

[TEXT END]

((END-EXAMPLE))

((START-TASK))
Now, given the original text (Lexile = {SOURCE-LEXILE}):

[TEXT START]

{SOURCE-TEXT}

[TEXT END]

Rewrite the above text and {TASK} to the difficulty level of Lexile = {TARGET-
LEXILE}. Do not include [TEXT START] and [TEXT END] in your response. Thanks.
((END-TASK))