# Token-level Adaptation of LoRA Matrices for Downstream Task Generalization

Joshua Belofsky
<EMAIL_ADDRESS>

(March 2023)

###### Abstract

This paper introduces a method for adapting LoRA matrices to arbitrary
downstream tasks in smaller-sized language models. The results demonstrate
that variable LoRA adapters outperform fixed adapters across mathematical
(GSM8K), scientific (ai2_arc, Challenge), reading comprehension (SQuAD), and
coding (CodeAlpaca-20k) tasks. Token-level adaptation of individual LoRA
matrices outperform LoRA adapters which are fine-tuned for specific tasks.
Further evaluations show a degradation of performance with less frequent
adaptation of the LoRA adapter. Finally, the performance of a 7B base Llama-2
model equipped with a variable LoRA adapter is compared to a 175B model
(GPT-3.5).

##  1 Introduction

Large language models (LLMs) are capable of solving common knowledge tasks
through a massive pre-training process. This pre-trained knowledge transfers
well to downstream tasks so long as both the neural network and training
dataset are sufficiently large enough to allow the said network to learn all
of the various tasks on the same single set of weights. Beyond algorithmic
improvments (e.g. attention mechanisms), sheer scale has enabled large neural
networks to represent many diverse tasks well. This contrasts the catastrophic
forgetfullness of older and smaller neural networks which could only
accurately learn a select number of similar tasks.

This paper aims to implement a method for smaller sized language models (7B
parameters) to generalize to different downstream tasks in a manner competitve
to that of the larger sized models. Low rank adapters are a type of parameter-
efficient weight matrices introduced by BLANK et al. which aims to offer
similar fine-funed perfomance at much less compute. Using LoRA adapters, fine-
tuned language models can compete

Healthcare is currently benefiting greatly from advances in Natural Language
Processing (NLP) to enhance the quality of service provided to clinicians,
patients, and biomedical researchers. The NLP field has made significant
progress in recent years, thanks to the availability of massive textual
databases and the use of deep learning techniques that allow for more
efficient exploitation of this data. Traditionally, the approach involved
training a generic masked language model (MLM) and then adapting it to a
specific domain or task, such as BERT models [devlin-etal-2019-bert].

However, the latest approaches aim to develop Large Language Models (LLMs)
that can directly process a wide range of NLP tasks and domains. They can then
handle tasks such as classification or entity extraction, as well as more
complex generative tasks like machine translation or question-answering. While
there is clear enthusiasm for LLMs among both scientists and the general
public, the evaluation of these models, also known as foundation models, is
still in its infancy. The initial evaluations demonstrate the usefulness of
these models in performing various NLP tasks, including classification and
generation tasks on general domains [liu2023gpteval, bang2023multitask].
However, in the medical field, these models have been evaluated to a lesser
extent, often on a limited number of tasks [rehana2023evaluation,
chen2023evaluation, lamichhane2023evaluation, singhal2022large,
chowdhery2022palm]. This is mainly due to the scarcity of tasks and data,
particularly sensitive data that is difficult to obtain, compared to other
fields.

To evaluate how well LLMs encode medical knowledge and to demonstrate their
capabilities in specific domains, a wide range of tasks that closely resemble
real-world applications and require appropriate medical knowledge and expert
reasoning were considered. Unlike other studies [fries2022bigbio, Medical2021]
that have compared performances of these models using automatic metrics (BLUE
[10.3115/1073083.1073135], ROUGE [lin-2004-rouge] or BertScore
[Zhang2020BERTScore]) or only accuracy on a small set of tasks, we decide to
showcase their relevance in various evaluation contexts by using more commonly
used metrics (Accuracy and F1) which are allowing a fair direct comparison
with BERT based models. In overall, we curate a collection comprising 13 real-
world medical tasks, including classification (CLS), question-answering (QA),
relation extraction (RE), natural language inference (NLI) and named-entity
recognition (NER). The main contributions of the paper are:

  1. 1.

Evaluation of four state-of-the-art instruction-tuned models (ChatGPT, Flan-T5
UL2, Tk-Instruct, and Alpaca) on a broad range of medical tasks beyond those
typically addressed by generative models.

  2. 2.

Assessment of the ability of the studied language models to perform zero-shot
and few-shot inference and comparison of their performance on the tasks with
that of a finetuned PubMedBERT model.

  3. 3.

Introduction of a novel method that enables performing the NER task on all
types of LLMs.

###  1.1 Large Language Models (LLMs)

While classical language models like BERT are efficient on various NLP tasks
and trained on large amounts of unannotated textual data, they still require a
substantial amount of annotated data to perform well on targeted tasks such as
NER, NLI, and RE. These models also have difficulty generalizing their
knowledge to other languages or domains once adapted to a particular task and
context [peng2021domain, amalvy2022bert]. Collecting such data for any
scenario is then expensive, as it requires highly qualified annotators and
raises privacy concerns.

Recently, LLMs have brought additional performance improvements, especially in
generative tasks. These models are composed of billion of parameters and
trained on gigantic amounts of data, from various natures, domains and
languages [gao2020pile, JMLR:v21:20-074, OrtizSuarezSagotRomary2019]. Previous
studies have demonstrated in particular that this gigantic number of
parameters associated with this massive data allowed the fine modeling of the
language, making it possible to achieve this level of performance
[zhang2022opt, black-etal-2022-gpt, hoffmann2022training, smith2022using].

New approaches using these generative LLMs capabilities have aimed to align
them with instructions [NEURIPS2022_b1efde53] (see Section 1.2), giving them
greater abilities to handle multiple NLP tasks in multiple languages in zero-
or few-shot learning [bang2023multitask].

###  1.2 Autoregressive Decoding

LLMs are trained to sequentially predict next-token probabilites from all
preceding tokens as input. This autoregressive decoding process is formalized
as follows:

$\displaystyle\begin{split}P(\mathbf{x})&=P(x_{1})\cdot P(x_{2}|x_{1})\cdot
P(x_{3}|x_{1}x_{2})\cdot\ldots\\\ &\quad\cdot P(x_{t}|x_{1}x_{2}\ldots
x_{t-1})\end{split}$ (1)
$\displaystyle=\prod_{t=1}^{n}P(x_{i}|x_{1}x_{2}\ldots x_{t-1})$

where $\mathbf{x}=(x_{1},x_{2},\ldots,x_{n})$ is a sequence of tokens, and
$P(x_{i}|x_{1}x_{2}\ldots x_{t-1})$ is the probability of the $i^{th}$ token
given all preceding tokens. The model is trained to minimize the negative log-
likelihood of the target sequence $\mathbf{x}$:

$\mathcal{L}=-\sum_{t=1}^{T}\log P(x_{t}|x_{<t};\theta)$ (2)

where $x_{t}$ is the $t^{th}$ token in the sequence, $x_{<t}$ is the sequence
of tokens preceding $x_{t}$, and $\theta$ are the model parameters. After pre-
training is complete, the model’s paramters $\theta$ remain a static
representation of all complex relationships the model has learned from the
training data between language and concepts. $\theta$ is then used to generate
a sequence of tokens from arbitrary prompts by sampling from the predicted
probability distribution at each step.

###  1.3 Parameter Adaptation

Mechanistically, LLMs can be viewed as large token decision trees whose
paramters determine which token branch to go down in order to forumulate a
final answer. Becuase the tree of potential tokens is so large, the
probability of any given LLM choosing the precise sequence of tokens which
perfectly answers the inputted prompt is never perfect. This is due to the
fact that the LLM’s paramters are not perfectly aligned with the task at hand
(it is forced to learn many tasks at once during pre-training). Fine-tuning
the model on domain-specific data has shown to improve performance on
downstream tasks precisely becuase the model is allowed to ’forget’ some of
the tasks it learned during pre-training and focus on the task at hand. Thus,
it seems like a natural extension to allow the model to adapt its paramters to
the task at hand.

Figure 1: Overview of 5-shot instruction inference.

###  1.4 Few-shot Learning with prompts

During inference, a few examples of the task are given to the model as
conditioning, without updating its weights. These examples usually comprise an
instruction, context, and desired completion (e.g., a premise, hypothesis, and
corresponding label for the NLI task). The few-shot technique involves
presenting the model with $k$ examples of context and completion, followed by
a final example of context, for which the model should provide the completion.
The value of $k$ typically ranges from 3 to 100, which depends on the number
of examples that can fit within the model’s context window (for instance,
Flan-UL2 has a context window of 2,048 tokens). See Appendix A.1.2 for more
details.

##  2 Experimental Protocol

In this section, we describe the models utilized and the datasets used to
benchmark the various models.

###  2.1 Studied Models

Our evaluation involves four distinct generic LLMs (ChatGPT, Flan-UL2, Tk-
Instruct and Alpaca) and a specific biomedical model (PubMedBERT) for
comparison purposes.

##### Flan-T5 UL2

abbreviated to Flan-UL2, is an encoder-decoder model based on UL2 20B
parameters model [tay2023ul2] and was finetuned using the Flan instruction
tuning tasks collection [chung2022scaling].

##### Tk-Instruct

is based on the T5 encoder-decoder model [JMLR:v21:20-074] and has been fine-
tuned on the 1600+ NLP tasks from the Super-NaturalInstructions dataset [wang-
etal-2022-super]. In our study, we chose the 3B parameter setting, since our
preliminary comparison with Flan-T5-XL
[https://doi.org/10.48550/arxiv.2210.11416] using the 3B parameter setting
showed that Tk-Instruct performed better on question answering tasks, which
are considered the most discriminative.

##### ChatGPT

is based on GPT 3.5 Turbo, finetuned on a set of private instructions and
continuously improved by using reinforcement learning from human feedback
(RLHF) techniques. Its weights are private and the model can only be accessed
via a payable API. This highlights some privacy concerns about its usage in
medical applications and cannot guarantee that the evaluated data has never
been seen before.

##### Stanford Alpaca

is based on LLaMA 7B parameters [touvron2023llama] and relies on a dataset of
52K instructions automatically generated using OpenAI’s text-davinci-003 model
in the style of self-instruct [wang2022selfinstruct]. Due to its based model
and data, it is only meant for academic research purposes and non-commercial
use.

##### PubMedBERT

is a biomedical-specific BERT-based model with 110M parameters
[10.1145/3458754]. It was trained entirely from scratch on the 3.1 billion
words of the PubMed corpus. We chose it as our baseline for comparison with
the zero- and few-shot performance of generative models.

###  2.2 Downstream evaluation tasks

We conducted an evaluation of the models’ capabilities by covering the 13
different tasks presented in Table 1. The tasks were chosen to allow for a
broad assessment in both clinical and biomedical domains, including tasks
oriented for generative and classical model evaluation.

Task | Dataset | Eval | Metric | Reference
---|---|---|---|---
CLS | HoC | Test | F1 |  DBLP:journals/bioinformatics/BakerSGAHSK16
LitCovid | Test | F1 |  chen2021overview
PubHealth | Test | Accuracy |  kotonya2020explainable
N2C2 2006 Smokers | Test | Accuracy |  uzuner2008identifying
QA | BioASQ 7b | Test | Accuracy |  tsatsaronis2015overview
MedMCQA | Dev | Accuracy |  pmlr-v174-pal22a
SciQ | Test | Accuracy |  welbl-etal-2017-crowdsourcing
Evidence Inference 2.0 | Test | Accuracy |  deyoung-etal-2020-evidence
RE | GAD | Test | Accuracy |  Bravo2015
NLI | SciTail | Test | Accuracy |  scitail
MedNLI | Test | Accuracy |  https://doi.org/10.13026/c2rs98
NER | BC5CDR | Test | F1 |  DBLP:journals/biodb/LiSJSWLDMWL16
NCBI-disease | Test | F1 |  Dogan2014NCBIDC
Table 1: List of evaluation tasks and their metrics.

###  2.3 Evaluation of generative outputs

The evaluation of generative models outputs is a difficult task due to their
free text form, which does not necessarily fit into a predefined range of
classes. On the contrary, we are dealing with a noisy output that may contain
correct answers. To tackle this issue, we manually built parsing scripts for
each task and model, according to their output style, in order to capture most
of the answers and compute metrics comparable with our baseline (PubMedBERT).

Task | Dataset | ChatGPT | Flan-UL2 | Tk-Instruct | Alpaca | PubMedBERT
---|---|---|---|---|---|---
zero-shot | 5-shot | zero-shot | 5-shot | zero-shot | 5-shot | zero-shot | 5-shot
CLS | HoC | 62.24 | 38.34 | 56.36 | 54.86 | 50.77 | 25.48 | 1.21 | 38.78 | 82.75
LitCovid | 67.20 | 72.77 | 51.48 | 46.95 | 36.42 | 57.49 | 1.58 | 64.09 | 90.60
PubHealth | 63.20 | 66.29 | 72.46 | 50.53 | 53.70 | 66.04 | 52.80 | 55.64 | 75.39
N2C2 2006 Smokers | 68.26 | 34.61 | 22.12 | 42.31 | 16.35 | 37.50 | 10.57 | 31.73 | 60.58
QA | BioASQ 7b | 89.24 | 92.03 | 90.97 | 91.64 | 88.09 | 86.36 | 79.05 | 79.82 | 73.39
MedMCQA | 48.91 | 56.37 | 41.05 | 43.34 | 33.85 | 33.18 | 24.91 | 29.50 | 38.15
SciQ | 90.10 | 93.50 | 87.00 | 88.40 | 55.30 | 47.00 | 24.90 | 36.80 | 74.20
Evidence Inference 2.0 | 59.98 | 63.83 | 66.45 | 65.06 | 41.33 | 38.79 | 32.49 | 94.18 | 65.47
RE | GAD | 47.75 | 52.25 | 49.81 | 53.37 | 48.88 | 57.87 | 51.12 | 57.68 | 79.78
NLI | SciTail | 73.57 | 65.62 | 93.51 | 92.66 | 57.53 | 71.31 | 39.60 | 40.26 | 93.51
MedNLI | NaN | NaN | 77.00 | 79.18 | 33.19 | 34.81 | 33.47 | 34.45 | 83.76
NER | BC5CDR | 92.12 | 93.12 | 68.26 | 83.32 | 84.54 | 83.23 | 82.11 | 84.07 | 97.65
NCBI-disease | 90.97 | 92.27 | 90.75 | 87.65 | 87.91 | 87.50 | 11.58 | 92.27 | 98.72
Table 2: 0- and 5-shot versus finetuning evaluation on clinical and biomedical
tasks. Bold values are the highest scores obtained for the task and in
underlined the seconds ones. Not allowed experiments are replaced by NaN.

###  2.4 Instruction format

As shown in previous studies [wei2022chain, jung-etal-2022-maieutic, mishra-
etal-2022-reframing], using a prompt designed for each task and model has been
proven to be effective. Therefore, we decided to construct the input
instruction prompt by concatenating three elements: (1) an instruction that
describes the overall task, the nature of the data, and what we expect from
the model, (2) the input argument, which provides the core information for the
task, and (3) the output space constraints, which guide the model during
output generation. Finally, the output serves as a reference during few-shot
strategy evaluation (see Appendix 6).

###  2.5 Few-shot examples using semantic retriever

In order to maximize few-shot performance compared to randomly sampled
examples, we introduced an additional retrieval module based on Sentence-
Transformers [reimers-2019-sentence-bert]. Its goal is to find the $k$ most
semantically similar examples from the training set. To achieve this, we first
fill a vector space using sentence representations of each individual
instruction prompt from the training set, obtained using a frozen PubMedBERT
[10.1145/3458754] model. Then, we calculate the cosine distance between the
query of the current test instance and all the elements present in the vector
space to fetch the $k$ closest examples. In our case, we set the $k$ value to
5.

###  2.6 Recursive Chain-of-Thought

We performed NER using two inference methods. The first one is based on the
method introduced by ye2023comprehensive and can only be applied using
ChatGPT. It consists of giving the model a sequence of words separated by
double vertical bars for word separation and single vertical bars for the
separation between words and labels, as shown in Table A.1.1. For the second
method, we introduce a method called Recursive Chain-of-Thought (RCoT). It is
very close to human reasoning and works for all the generative models we have
tried. It is derived from the Chain-of-Thought (CoT) concept [wei2022chain]
and the work of wang-etal-2022-super. It involves iterating over the sequence
of tokens and giving the current state of the prediction as input to the
model, asking for the generation of the label of the $N^{th}$ token, as
presented in Table A.1.2. This method guarantees an entity for each token of
the sequence and prevents forgotten tokens during generation. However, the
only drawback we have been able to identify with this method is its very high
computation cost due to its $\mathcal{O}^{N}$ complexity, with $N$ being the
number of tokens in the sequence, compared to the method used for ChatGPT,
which performs at $\mathcal{O}^{1}$ complexity.

##  3 Results and Discussions

Table 2 reports performance obtained by the studied LLMs in zero- and few-shot
scenario, as well as PubMedBERT finetuned on each task. Results are reported
by taking the best run out of four.

##### Zero-shot scenario

Compared to PubMedBERT, the zero-shot scenario results show a clear deficit
for the generative models on all the tasks except for QA, in which LLMs obtain
better performance. ChatGPT and Flan-T5 UL2 are particularly more performant
than Tk-Instruct and Alpaca on average, except for GAD dataset (RE task) for
which Alpaca obtains the best performance. We can also observe extremely poor
performance from Alpaca in zero-shot scenario on the two CLS tasks (HoC and
LitCovid). These scores are owing to the model only producing hallucinations
and the evading growth suppressors label for the whole test set of HoC. This
behavior does not appear to take place in few-shot scenario, where the model
seem to understand what we are expecting from it.

##### Few-shot capabilities

Unlike the zero-shot scenario, the few-shot inference (5-shots in our
experiments) shows impressive behavior. The biggest absolute gains are
obtained using Alpaca, which seems to perform much better in few-shot
scenarios on all tasks. We suspect this behavior to be correlated with
Alpaca’s training data, which does not contain many similar instructions for
the tasks we are trying to tackle, allowing it to better understand what we
are asking when confronted with similar examples. ChatGPT also benefits from
the additional knowledge to further improve the already good results,
especially on QA tasks. Flan-T5 UL2 appears to be less affected by the
additional context overall, except for the BC5CDR and N2C2 2006 Smokers tasks.

##  4 Conclusion

In this study, we have demonstrated that generic LLMs are capable of capturing
medical knowledge and performing exceptionally well in zero- and few-shot
scenarios, despite having no prior exposure to the tasks. Although open-source
models such as Flan-T5 UL2 are gradually approaching their closed-source
counterparts, like ChatGPT, their performance still lags behind. We suggest
that developing domain-specific models, finetuned on a diverse set of tasks
and specialized instruction prompts, could help bridge the gap with more
robust and performant proprietary models. We also note that domain-specific
BERT models remain a viable option, but require a significant amount of data
for finetuning on targeted languages and tasks. However, BERT-based models
offer much lower computational costs compared to LLMs, which could be a
significant obstacle to developing models in the healthcare domain.

###  4.1 Limitations

From all the experiments we have conducted, we discovered that LLMs trained
from instructions are frequently sensitive to the particular words used for
their input and impact whether or not it’s capable to produce the correct
outputs. This is perhaps unsurprising, given that LLMs are known to be very
sensitive to the prompt they are provided with in both zero and few-shot
settings [jiang-etal-2020-know, schick-schutze-2021-exploiting]. However, it
often requires adapting the prompt to the models and tasks or event to map the
classes to more effective ones. This behavior may be caused by the lack of
variety in the instruction collections used for training them.

One of the main limitations is related to our inability to guarantee that the
ChatGPT model has never seen the evaluation data during its training. This can
strongly bias the results obtained. Similarly, Flan-T5 UL2 and Tk-Instruct has
been trained on a wide range of tasks in which similar or identical data can
be seen if overlap has not been identified. So, we cannot ensure that training
data of some tasks has never been seen before.

###  4.2 Related work

We first introduce the concept of Large Language Models (LLMs) and their
limitations (Section 1.1). Next, we present the concept of instruction-tuning
(Section 1.2). Finally, we describe our few-shot learning strategy with
prompts (Section 1.4).

##  Appendix A Instructions examples

The following sections are giving example of prompts used for training and
inference for organized by tasks.

###  A.1 Named-Entities Recognition

####  A.1.1 Method 1

Prompts

Instruction: Do named-entity recognition task for the given text using the
categories in candidate list, output using the format as
“Word1|Category||Word2|Category||Word3|Category”

Candidate list: O, B-Disease or I-Disease

Text: Identification|Category || of|Category || APC2|Category || ,|Category ||
a|Category || homologue|Category || of|Category || the|Category ||
adenomatous|Category || polyposis|Category || coli|Category || tumour|Category
|| suppressor|Category || .|Category

Output:

Instruction: You are a healthcare named-entity recognition expert system and
we are giving you a sequence of words that you have to labelized using the
following output format ’Word1|Label||Word2|Label||Word3|Label’

Labels: O, B-Disease or I-Disease

Unfilled sequence:
Identification|Label||of|Label||APC2|Label||,|Label||a|Label||homologue|Label||of|Label
||the|Label
||adenomatous|Label||polyposis|Label||coli|Label||tumour|Label||suppressor|Label||.|Label

Constraints: The answer must be one and only one of the given labels.

Output:

Instruction: As a healthcare named-entity recognition expert, your job is to
label a sequence of words provided to you using the following format:
’Word1|Label||Word2|Label||Word3|Label’. Your goal is to identify all the
named entities in the given text. The available labels for this task are: O,
B-Disease or I-Disease

Input:
Identification|Label||of|Label||APC2|Label||,|Label||a|Label||homologue|Label||of|Label||the|Label
||adenomatous|Label
||polyposis|Label||coli|Label||tumour|Label||suppressor|Label||.|Label

Output:

Table 3: Sample of three instructions used for the named-entities recognition
task with ChatGPT.

####  A.1.2 Method 2 - Recursive Chain-Of-Thought (RCoT)

Prompt

Instruction: You are a highly intelligent and accurate healthcare domain
Named-entity recognition (NER) system. You are tasked to do Named-entity
recognition (NER) for ’disease’ and ’none’ only, please generate the
appropriate label.

Constraints: You can choose only one label from: none or disease.

Examples: //

Example 1 :  Mutations|none|| at|none|| the|none|| ataxia|disease||
-|disease|| telangiectasia|disease|| locus|none|| and|none|| clinical|none||
phenotypes|none|| of|none|| A|disease|| -|disease|| T|disease||
patients|none|| .|none

Example 2 :  Splicing|none|| defects|none|| in|none|| the|none||
ataxia|disease|| -|disease|| telangiectasia|disease|| gene|none|| ,|none||
ATM|none|| :|none|| underlying|none|| mutations|none|| and|none||
consequences|none|| .|none

Example 3 :  Somatic|none|| mutations|none|| in|none|| the|none|| BRCA1|none||
gene|none|| in|none|| sporadic|disease|| ovarian|disease|| tumours|disease||
.|none

Example 4 :  Malignant|disease|| neoplasms|disease|| in|none|| the|none||
families|none|| of|none|| patients|none|| with|none|| ataxia|disease||
-|disease|| telangiectasia|disease|| .|none

Example 5 :  Founder|none|| mutations|none|| in|none|| the|none|| BRCA1|none||
gene|none|| in|none|| Polish|none|| families|none|| with|none||
breast|disease|| -|disease|| ovarian|disease|| cancer|disease|| .|none

Considering the sentence : Clustering of missense mutations in the ataxia -
telangiectasia gene in a sporadic T - cell leukaemia .

And considering your precedents predictions :  Clustering|none|| of|none||
missense|none|| mutations|none|| in|none|| the|none|| ataxia|disease||
-|disease|| telangiectasia|disease|| gene|none|| in|none|| a|none||
sporadic|disease|| T|disease|| -|disease|| cell|disease|| leukaemia|Label

Input : The label of « leukaemia » at the position 17 of the sentence is ?

Output:

Table 4: Example of a 5-shot Recursive Chain-Of-Thought (RCoT) instruction
used for the named-entities recognition task of NCBI Disease dataset.

##  Appendix B Multiple-choice question answering

###  B.1 Method 1 - One-shot

Prompt

Instruction: You are given a science question (easy level) and four answer
options (associated with “A”, “B”, “C”, “D”). Your task is to find the correct
answer based on scientific facts, knowledge and reasoning. Don’t generate
anything other than one of the following characters: ’A B C D’.

Input: Heavy forces on periodontal ligament causes: (A) Hyalinization (B)
Osteoclastic activity around tooth (C) Osteoblastic activity around tooth (D)
Crest bone resorption

Constraints: The answer must be one or more of the following letters:
’A’,’B’,’C’,’D’. You must generate one and only one letter for each question.
All questions have an answer. No justification is required.

Output:

Table 5: Example of a 0-shot instruction used for the Multiple-Choice Question
Answering (MCQA) task of MedMCQA dataset.

###  B.2 Method 2 - Few-shot

In some cases, we mapped the original classes to more effective one’s for each
of the tasks, based on tries and errors (e.g: "entailment" has been map to
"entails" for ChatGPT and Flan-T5 UL2 based on noticeable performances gains).

Prompt

Instruction: You are given a science question (easy level) and four answer
options (associated with “A”, “B”, “C”, “D”). Your task is to find the correct
answer based on scientific facts, knowledge and reasoning. Don’t generate
anything other than one of the following characters: ’A B C D’.

Constraints: The answer must be one or more of the following letters:
’A’,’B’,’C’,’D’. You must generate one and only one letter for each question.
All questions have an answer. No justification is required.

Examples:

Example 1: Hyalinisation of the periodontal Ligament, due to excessive
orthodontic forces results in (A) Frontal resorption (B) Undermining
resorption (C) Cementum remaining intact (D) Dentine remaining intact

Output: B

Example 2: The earliest response of pulpitis is: (A) Cyst formation (B)
Calcification (C) Hyalinization (D) Formation of dental granuloma

Output: C

Example 3: Among the secondary changes in tooth the most useful one for age
determination is: (A) Attrition (B) Secondary dentine deposition (C) Root
resorption (D) Root transparency

Output: D

Example 4: Feature of aging periodontium is (A) Lacunae in bone and cementum
(B) Increased cell size (C) Increased cell number (D) Scalloping of cementum &
alveolar bone surface

Output: D

Example 5: Bacteria found in gingivitis are localized in (A) Connective tissue
fibres (B) Gingival sulcus (C) Alveolar bone (D) Periodontal ligament

Output: B

Input: Heavy forces on periodontal ligament causes: (A) Hyalinization (B)
Osteoclastic activity around tooth (C) Osteoblastic activity around tooth (D)
Crest bone resorption

Output:

Table 6: Example of a 5-shot instruction used for the Multiple-Choice Question
Answering (MCQA) task of MedMCQA dataset.

##  Appendix C Relation Extraction

###  C.1 Method 1 - One-shot

Prompt

Instruction: Your goal is to do relation extraction and identifying if a gene-
disease relation exist (positive) or not (negative).

Input : These results suggest that the C1772T polymorphism in @GENE$ is not
involved in progression or metastasis of @DISEASE$

Constraints: You have to output one label among « negative » or « positive ».
Justification and explanations are prohibited.

Output:

Table 7: Example of a 0-shot instruction used for the Relation Extraction (RE)
task of GAD dataset.

###  C.2 Method 2 - Few-shot

Prompt

Instruction: Your goal is to do relation extraction and identifying if a gene-
disease relation exist (positive) or not (negative).

Constraints: You have to output one label among « negative » or « positive ».
Justification and explanations are prohibited.

Examples:

Example 1: These findings suggest that the Gly460Trp polymorphism of @GENE$ is
not associated with @DISEASE$.

Output: Positive

Example 2: Our results suggest that deletion polymorphism of the @GENE$ gene
is not associated with the pathogenesis of @DISEASE$ in Taiwanese.

Output: Positive

Example 3: The results suggest that the 5A/6A polymorphism of @GENE$ gene may
not be linked with appearance and/or progression of @DISEASE$.

Output: Positive

Example 4: Our study implies that the G/C polymorphism of the @GENE$ gene may
not be directly involved in the development and=or progression of @DISEASE$.

Output: Positive

Example 5: Our study implies that the G/C polymorphism of the @GENE$ gene may
not be directly involved in the development and=or @DISEASE$ of breast cancer.

Output: Negative

Input: These results suggest that the C1772T polymorphism in @GENE$ is not
involved in progression or metastasis of @DISEASE$.

Output:

Table 8: Example of a 5-shot instruction used for the Relation Extraction (RE)
task of GAD dataset.

##  Appendix D Natural Language Inference

###  D.1 Method 1 - One-shot

Prompt

Instruction: Your goal is to do solve a natural language inference task by
identifying if the hypothesis is either « entails » or « neutral » to the
premise.

Input premise: The liver is divided into the right lobe and left lobes.

Input hypothesis: The gallbladder is near the right lobe of the liver.

Constraints: You have to output one label among « entails » or « neutral ».
Justification and explanations are prohibited.

Output:

Table 9: Example of a 0-shot instruction used for the Natural Language
Inference (NLI) task of SciTail dataset.

###  D.2 Method 2 - Few-shot

Prompt

Instruction: Your goal is to do solve a natural language inference task by
identifying if the hypothesis is either « entails » or « neutral » to the
premise.

Constraints: You have to output one label among « entails » or « neutral ».
Justification and explanations are prohibited.

Examples:

Example 1:

Premise: Located primarily on the right side of the abdominal cavity, just
above the duodenum, the liver aids in the digestion of fats by secreting bile
into the duodenum.

Hypothesis: Most digestion is completed in the duodenum.

Output: neutral

Example 2:

Premise: The brain is divided into the right and left hemisphere and each
hemisphere is divided into 4 lobes called the frontal, temporal, occipital and
parietal lobes.

Hypothesis: Each hemisphere of the cerebrum divided into 4 lobes.

Output: entails

Example 3:

Premise: The small intestine, where most digestion takes place, is a
convoluted tube in the abdomen that begins at the pylorus of the stomach and
ends at the opening to the large intestine.

Hypothesis: Most of the digestion reactions occur in the small intestine.

Output: entails

Example 4:

Premise: The small intestine is the long, thin segment of bowel that begins at
the stomach and ends at the large intestine or colon.

Hypothesis: The small intestine begins in the stomach.

Output: entails

Example 5:

Premise: The small intestine begins at the stomach and ends at the colon
(large intestine).

Hypothesis: The small intestine begins in the stomach.

Output: entails

Premise: The liver is divided into the right lobe and left lobes.

Hypothesis: The gallbladder is near the right lobe of the liver.

Output:

Table 10: Example of a 5-shot instruction used for the Natural Language
Inference (NLI) task of SciTail dataset.

##  Appendix E Classification

###  E.1 Method 1 - One-shot

Prompt

Instruction: Your goal is to do solve a classification task by identifying if
one or more of the following hallmarks of cancer are present in the document:
« evading growth suppressors », « tumor promoting inflammation », « enabling
replicative immortality », « cellular energetics », « resisting cell death »,
« activating invasion and metastasis », « genomic instability and mutation »,
« none », « inducing angiogenesis », « sustaining proliferative signaling » or
« avoiding immune destruction ».

Input: Cytotoxicity was shown in manganese-treated groups ( 100 , 200 , 400 ,
and 800microM of MnCl(2) ) , and cell viability was decreased to 58.8% of the
control group at 2days after treatment with 800microM of MnCl(2) .

Constraints: You have to output one or more label(s) among « evading growth
suppressors », « tumor promoting inflammation », « enabling replicative
immortality », « cellular energetics », « resisting cell death », « activating
invasion and metastasis », « genomic instability and mutation », « none », «
inducing angiogenesis », « sustaining proliferative signaling » or « avoiding
immune destruction ». Justification and explanations are prohibited.

Output:

Table 11: Example of a 0-shot instruction used for the classification (CLS)
task of HoC dataset.

###  E.2 Method 2 - Few-shot

Prompt

Instruction: Your goal is to do solve a classification task by identifying if
one or more of the following hallmarks of cancer are present in the document:
« evading growth suppressors », « tumor promoting inflammation », « enabling
replicative immortality », « cellular energetics », « resisting cell death »,
« activating invasion and metastasis », « genomic instability and mutation »,
« none », « inducing angiogenesis », « sustaining proliferative signaling » or
« avoiding immune destruction ».

Constraints: You have to output one or more label(s) among « evading growth
suppressors », « tumor promoting inflammation », « enabling replicative
immortality », « cellular energetics », « resisting cell death », « activating
invasion and metastasis », « genomic instability and mutation », « none », «
inducing angiogenesis », « sustaining proliferative signaling » or « avoiding
immune destruction ». Justification and explanations are prohibited.

Examples:

Example 1: However , significant cytotoxicity was only observed in PCB 52
concentrations larger than 0.1 microg ml(-1) , while there was no significant
inhibition in PCB 77-treated cells at concentrations selected .

Output: none

Example 2: In MeT-5A cells , both CNTs caused a dose-dependent induction of
DNA damage ( % DNA in comet tail ) in the 48-h treatment and SWCNTs
additionally in the 24-h treatment , with a statistically significant increase
at 40
u03bcg/cm(2) of SWCNTs and ( after 48 h ) 80
u03bcg/cm(2) of both CNTs .

Output: none

Example 3: Copper-induced DNA strand breakage was first observed after 24 h of
exposure , and was recorded again at 96 h , at a copper concentration of 20
microg l(-1) .

Output: genomic instability and mutation

Example 4: Drug concentrations of 12.5 to 300 0̆3bcM caused a pronounced
reduction in cell survival rates five days after treatment , whereas
concentrations higher than 25 0̆3bcM were effective in reducing the survival
rates to However , the maximum apoptosis frequency was 20.4% for 25 0̆3bcM
cisplatin in cells analyzed at 72 h , indicating that apoptosis is not the
only kind of cell death induced by cisplatin .

Output: none

Example 5: In contrast , in MCF 7 cells , molecular iodine ( 100 microM )
inhibited growth from 100% to 83% but delta-iodolactone ( 1 , 5 and 10 microM
) dose-dependently decreased growth rate from 100% to 82% and 62% ,
respectively .

Output: none

Input: Cytotoxicity was shown in manganese-treated groups ( 100 , 200 , 400 ,
and 800microM of MnCl(2) ) , and cell viability was decreased to 58.8% of the
control group at 2days after treatment with 800microM of MnCl(2) .

Output:

Table 12: Example of a few-shot instruction used for the classification (CLS)
task of HoC dataset.

##  Appendix F Semantic Textual Similarity

###  F.1 Method 1 - One-shot

Prompt

Instruction: Give me a similarity score beetween 0 et 5 and only the
similarity score.

Input: The original sentence is : "- Eviter le contact de l’embout avec l’œil
ou les paupières." can you tell me if the sentence is similar to : "Evitez le
contact de l’embout du flacon avec l’œil ou les paupières.".

Output:

Table 13: Example of a 0-shot instruction used for the Semantic Textual
Similarity (STS) task of DEFT-2020 task 1 dataset.