# Recognising Biomedical Names: Challenges and Solutions

Xiang Dai

(Date: 2021)

## Statement of Originality

This is to certify that to the best of my knowledge, the content of this
thesis is my own work. This thesis has not been submitted for any degree or
other purposes.

I certify that the intellectual content of this thesis is the product of my
own work and that all the assistance received in preparing this thesis and
sources have been acknowledged.

Xiang Dai

2021-Feb-27

## Abstract

The growth rate in the amount of biomedical documents—such as scholarly
articles, clinical notes and health forum discussions—is staggering. Unlocking
information trapped in these documents can enable researchers and
practitioners to operate confidently in the information world. Biomedical
Information Extraction (IE) system aims to automatically extract structured
information—such as biomedical concepts, attributes, events, and their
relations—from unstructured text. Within an IE system, the first step is
called Biomedical Named Entity Recognition (NER), the task of recognising
biomedical names.

NER has been heavily studied in the generic domain, recognising person,
organisation, and location names in newspaper articles. However, the
effectiveness of existing Biomedical NER model is still not satisfactory. In
contrast to entity mentions in the generic domain which are usually short
spans of text, biomedical names—surface forms that represent biomedical
concepts, such as genes, proteins, symptoms, diseases, and drugs—pose unique
challenges. For example, it is even common for an ordinary person to confuse
_‘severe acute respiratory syndrome coronavirus 2’_ (virus name), _‘severe
acute respiratory syndrome’_ (disease name), and _‘coronavirus disease 2019’_
(disease name). The variety of language used for different communicative
purposes makes biomedical NER even more challenging. Various groups of people
use totally different languages to describe the same biomedical concept. For
example, researchers tend to use standard names in biomedical vocabularies to
make the description more comprehensible and less confused; hospital doctors,
who write notes under time pressure, use abbreviations for efficient
communication with their colleagues; and, ordinary people use linguistically
noisy layman language to share their experiences.

State-of-the-art NER models, based on sequence tagging technique, are good at
recognising short entity mentions in the generic domain, especially when they
are enhanced by pre-trained language representation models. However, there are
several open challenges of applying these models to recognise biomedical
names:

  * •

Biomedical names may contain complex inner structure (discontinuity and
overlapping) which cannot be recognised using standard sequence tagging
technique;

  * •

The training of NER models usually requires large amount of labelled data,
which are difficult to obtain in the biomedical domain; and,

  * •

Commonly used language representation models are pre-trained on generic data,
such as the Wikipedia and books, a domain shift therefore exists between these
models and target biomedical data.

To deal with these challenges, we explore several research directions and make
the following contributions: (1) we propose a transition-based NER model which
can recognise discontinuous mentions. Through experiments on three datasets
from the biomedical domain, we show that our model can effectively recognise
discontinuous entity mentions without sacrificing the accuracy on continuous
mentions. Analysis also suggests that our model is good at recognising long
mentions, resulting in higher recall than other baselines; (2) We develop a
cost-effective approach that nominates the suitable pre-training data, via
measuring the similarity between different pre-training data options and
target task data. Through experiments on 56 source-target data pairs, we show
that simple similarity measures are good predictors of the usefulness of pre-
trained language representation models on downstream NER datasets; and, (3) We
design several data augmentation methods which do not rely on any external
trained models, for NER. Experimental results show that the proposed
augmentation methods can improve performance over strong baselines, where
large scale pre-trained language representation models are used.

Our contributions have obvious practical implications, especially when new
biomedical applications are needed. Our proposed data augmentation methods can
help the NER model achieve decent performance, requiring only a small amount
of labelled data. Our investigation regarding selecting pre-training data can
improve the model by incorporating language representation models, which are
pre-trained using in-domain data. Finally, our proposed transition-based NER
model can further improve the performance by recognising discontinuous
mentions without sacrificing the accuracy on continuous mentions.

## Acknowledgements

I enjoyed my journey of doing a PhD, and I am immensely thankful for the many
people I met during this journey. Foremost thanks must go to my PhD
supervisors Sarvnaz Karimi, Ben Hachey and Cecile Paris. Sarvnaz, thank you
for your encouragement, without which I would never have started working on
NLP. Also thank you for your detailed guidance, which shapes my research. Ben,
thank you for your completely honest criticisms and feedback, which always
help me revisit my work from a practical perspective. Cecile, thank you for
sensible advice and patience. I learn a lot from people like you who want to
make things to be perfect. I also want to thank my supervisor Joachim
Gudmundsson, who helped me a lot with university’s administration and funding.

I am very thankful to Dietrich Klakow from Saarland University. Because of the
COVID-$19$ pandemic, I was stranded in Germany after I finished my internship
at Bosch Center for Artificial Intelligence. Dietrich hosted me in his group,
providing me a shelter where I could finish my thesis. Thanks also to Heike
Adel, Matthew Honnibal, and Vera Demberg for their help.

I also want to thank my colleagues at Data61. In particular, I am grateful to
Aditya Joshi, Chang Xu, Maciej Rybinski, Vincent Nguyen, Stephen Wan, Sunghwan
Mac Kim, Wenyi Tay, and Sonit Singh for regular reading group meetings, and
inspiring discussions. Thanks also to Lukas Lange and Michael A. Hedderich
from Saarland University for feedback on drafts of this thesis, and exchange
of ideas.

Finally, thank you to my family for being encouraging and patient. Special
thanks to my dad, who explains me a lot about medications and medical
procedures.

## Authorship Attribution Statement

  * •

The Chapter 1 of this thesis relates to (Dai et al., 2017).

Xiang Dai, Sarvnaz Karimi, and Cecile Paris. 2017. Medication and adverse
event extraction from noisy text. In Proceedings of the Australasian Language
Technology Association Workshop, pages 79–87, Brisbane, Australia.

  * •

The Section 2.4 of this thesis relates to (Dai, 2018).

Xiang Dai. 2018. Recognizing complex entity mentions: A review and future
directions. Proceedings of ACL 2018, Student Research Workshop, pages 37–44,
Melbourne, Australia.

  * •

Chapter 3 of this thesis relates to (Dai and Adel, 2020).

Xiang Dai, Heike Adel. 2020. An Analysis of Simple Data Augmentation for Named
Entity Recognition. In Proceedings of the 28th International Conference on
Computational Linguistics, Online.

  * •

Chapter 4 of this thesis relates to (Dai et al., 2019, 2020a).

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2019. Using
similarity measures to select pretraining data for NER. In Proceedings of the
2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and
Short Papers), pages 1460–1470, Minneapolis, Minnesota.

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020. Cost-effective
Selection of Pretraining Data: A Case Study of Pretraining BERT on Social
Media. In Findings of the 2020 Conference on Empirical Methods in Natural
Language Processing, Online.

  * •

Chapter 5 of this thesis relates to (Dai et al., 2020b).

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020. An effective
transition-based model for discontinuous NER. In Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics, pages
5860–5870, Online.

I contribute to every aspect of the above mentioned publications, including
designing the method, conducting the experiment, and writing the paper.

###### Contents

  1. Authorship Attribution Statement
  2. 1 Introduction
    1. 1.1 Recognising Adverse Drug Events from Social Media—A Motivating Application
      1. 1.1.1 A unified architecture for sequence taggers
    2. 1.2 Key Open Challenges
      1. 1.2.1 Biomedical names are complex
      2. 1.2.2 Labelling data is difficult
      3. 1.2.3 Unlabelled biomedical data are limited
    3. 1.3 About the Thesis
      1. 1.3.1 Publications
      2. 1.3.2 Definition and clarification of used terms
    4. 1.4 Summary
  3. 2 Literature Review
    1. 2.1 A Brief History of NER
      1. 2.1.1 Dictionary based approach
      2. 2.1.2 Rule based approach
      3. 2.1.3 Statistical machine learning based approach
    2. 2.2 NLP for Low Resource Scenarios
      1. 2.2.1 Distant supervision
      2. 2.2.2 Active learning
      3. 2.2.3 Data augmentation
      4. 2.2.4 Summary
    3. 2.3 Transfer Learning
      1. 2.3.1 Cross-task transfer
      2. 2.3.2 Cross-domain transfer
      3. 2.3.3 Summary
    4. 2.4 Complex Entity Recognition
      1. 2.4.1 Definitions of complex entity mentions
      2. 2.4.2 Token-level Approach
      3. 2.4.3 Span-based Approach
      4. 2.4.4 Sentence-level Approach
      5. 2.4.5 Summary
  4. 3 Data Augmentation for NER
    1. 3.1 Overview
    2. 3.2 Proposed Data Augmentation Methods
    3. 3.3 Evaluation
      1. 3.3.1 Datasets
      2. 3.3.2 Backbone model
      3. 3.3.3 Experimental results
    4. 3.4 Analysis
      1. 3.4.1 The impact of hyperparameters
      2. 3.4.2 A closer look at errors
    5. 3.5 Summary
  5. 4 Cost-effective Selection of Pre-training Data
    1. 4.1 Overview
    2. 4.2 What Human Intuition Indicates
    3. 4.3 Similarity Measures
      1. 4.3.1 Target vocabulary covered
      2. 4.3.2 Jaccard similarity of vocabularies
      3. 4.3.3 Language model perplexity
      4. 4.3.4 Jensen-Shannon divergence
    4. 4.4 Datasets
    5. 4.5 Experimental Results
      1. 4.5.1 Predictiveness of similarity measures
      2. 4.5.2 Comparison to publicly available pre-trained models
    6. 4.6 Summary
  6. 5 Transition-based Model for Discontinuous NER
    1. 5.1 Overview
    2. 5.2 Datasets
    3. 5.3 Proposed Model
      1. 5.3.1 Representation of the parser state
      2. 5.3.2 Capturing discontinuous dependencies
      3. 5.3.3 Selecting an action
    4. 5.4 Experimental Results
      1. 5.4.1 Baseline models
      2. 5.4.2 Experimental setup
      3. 5.4.3 Results
    5. 5.5 Analysis
      1. 5.5.1 Impact of mention and interval length
      2. 5.5.2 Impact of overlapping structure
      3. 5.5.3 Example predictions
      4. 5.5.4 Ablation studies
    6. 5.6 Summary
  7. 6 Conclusions

###### List of Figures

  1. 1.1 An example input sentence and the entity mentions which are supposed to be recognised by the NER model.
  2. 1.2 State-of-the-art NER model is based on sequence tagging technique that assigns a tag to each token. Token positions of mentions can be extracted from the output tag sequence.
  3. 1.3 A unified architecture for sequence tagging models, consisting of a mapping function and a classifier.
  4. 1.4 An LSTM computes the output state by taking the entire past (left context) of the input sequence into consideration.
  5. 1.5 An forward model computes the output state by taking the entire past (left context) of the input sequence into consideration, whereas the backward model considers the entire right context. Both output hidden states are concatenated to form the final contextual string embedding and capture the information of the token itself as well as its surrounding tokens.
  6. 1.6 The semi-supervised learning approach used in Flair: pre-training two—forward and backward—character level language models, and using the pre-trained model as part of the mapping function in the downstream supervised tasks. For the sake of brevity, we show only the forward model and the backward model is omitted. The whitespace character is represented using ” in this figure.
  7. 1.7 We focus on Biomedical NER, and explore the following three research directions: recognising discontinuous entity mentions; pre-training domain-specific language representation models; and enhancing the effectiveness of NER models using data augmentation.
  8. 2.1 Example sequence of tags generated by a rule and two domain-specific dictionaries. For example, ‘Cymbalta’ is assigned the tag ‘I-Drug’ because it appears in the drug dictionary.
  9. 2.2 The active learning process usually have multiple rounds, each of which consists of five steps: (1) applying model on unlabelled data; (2) querying on unlabelled data; (3) presenting informative instances; (4) annotating instances; and (5) updating the model.
  10. 2.3 The extractive question answering model tends to use the question type (e.g., Who) and select the spans whose nature agrees with the question type (e.g., ‘Bill Clinton’, ‘George H. W. Bush’, and ‘Ross Perot’), without the necessity to understand the question.
  11. 2.4 Neural architectures for the settings of cross-domain, cross-task, cross-lingual transfer proposed in (Yang et al., 2017).
  12. (a) Cross-domain transfer when the label sets are mappable.
  13. (b) Cross-domain transfer when the label sets are disparate, and cross-task transfer.
  14. (c) Cross-lingual transfer. The $\oplus$ also represents a neural network that takes both character-level and word-level feature vectors as input and creates the final token feature vector.
  15. 2.5 The Skip-gram model aims to learn word representations that can be used to predict the surrounding words.
  16. 2.6 The mask language modelling pre-training task aims to learn contextual word representations that can be used to predict what the masked token is.
  17. 2.7 The replace token detection task aims to train the discriminator to predict whether the token is the original token or a fictional token.
  18. 2.8 Examples involving nested, overlapping and discontinuous entity mentions. In (a), ‘HIV-1 enhancer’ and ‘HIV-1’ are nested entity mentions. In (b), ‘intense pelvic pain’ and ‘back pain’ overlap, and ‘intense pelvic pain’ is a discontinuous mention.
  19. 2.9 In a linear-chain CRF model, the output for each token depends on the representation of that token in context and the output for the previous token.
  20. 2.10  An encoding example of two adverse drug event mentions: ‘intense pelvic pain’ and ‘back pain’.
  21. 2.11  An example of mention separators encoding two nested entity mentions: ‘IL2’ and ‘IL2 regulatory region’. Muis and Lu (2017) design three mention separators: S, also denoted as [, indicating a mention is starting at the next token; E (]), indicating a mention is ending at the previous token; and C (-), indicating a mention is continuing to the next token. X means none of the three separators applies. The standard sequence tagger, which takes as input a sequence of $N$ tokens and outputs a sequence of $N$-1 mention separators, can be used to recognise nested NER.
  22. 2.12  An example of a sentence with three entity mentions: ‘Bill Clinton’ and ‘Hilary Clinton’ are Person mentions, and ‘Canada’ is a Location mention. P and L refer to the entity categories: Person and Location, respectively.
  23. 2.13  An example sub-hypergraph with two nested entity mentions: ‘HIV-1’ (Virus) and ‘HIV-1 enhancer’ (DNA). Here, one mention corresponds to a path consisting of (AETI+X) nodes. Specifically, the path (A4E4T${}_{4}^{1}$I${}_{4}^{1}$X) corresponds to the mention ‘HIV-1’, and the path (A4E4T${}_{4}^{2}$I${}_{4}^{2}$I${}_{5}^{2}$X) corresponds to the mention ‘HIV-1 enhancer’.
  24. 2.14  An example sub-hypergraph with two entity mentions: ‘muscle pain’ and ‘muscle fatigue’. Muis and Lu (2016) extend the hypergraph representation proposed by Lu and Roth (2015) to capture discontinuous mentions through two new node types: B${}_{k}^{i}$ representing the $k$-th token is part of the $i$-th component of an entity mention, and O${}_{k}^{i}$ representing the $k$-th token appears in between $(i-1)$-th and $i$-th components of an entity mention. In this example, the path (A3E3TB${}_{3}^{0}$B${}_{4}^{1}$X) corresponds to the mention ‘muscle pain’ and the path (A3E3TB${}_{3}^{0}$O${}_{4}^{1}$O${}_{5}^{1}$B${}_{6}^{1}$X) corresponds to the discontinuous mention ‘muscle fatigue’.
  25. 3.1 High level overview of the BERT-CRF model.
  26. 3.2 Impact of the number of augmented instances per original training instance on the effectiveness of data augmentation. SR: synonym replacement. MR: mention replacement.
  27. (a) i2b2-2010, SR
  28. (b) NCBI-Disease, SR
  29. (c) i2b2-2010, MR
  30. (d) NCBI-Disease, MR
  31. 3.3 The impact of the ratio a token or a mention is replaced on the effectiveness of data augmentation. SR: synonym replacement. MR: mention replacement.
  32. (a) i2b2-2010, SR
  33. (b) NCBI-Disease, SR
  34. (c) i2b2-2010, MR
  35. (d) NCBI-Disease, MR
  36. 4.1 Survey questions regarding selection of pre-training data.
  37. 4.2 Likert scale ratings from NLP and ML practitioners ($N=30$) for the statement ‘Unsupervised pre-training on S would be useful for supervised named entity recognition learning on T.’ Target data T is described as ‘Online forum posts about medications,’ source data S1 as ‘Research papers about biology and health,’ and source data S2 as ‘Online reviews about restaurants, hotels, barbers, mechanics, etc.’
  38. 4.3 Correlation between different similarity measures and the effectiveness of domain-specific pre-trained models.
  39. 5.1 Examples involving discontinuous mentions, taken from the ShARe/CLEF 13 (Pradhan et al., 2013) and CADEC (Karimi et al., 2015a) datasets, respectively. The first example contains a discontinuous mention ‘left atrium dilated’, the second example contains two mentions that overlap: ‘muscle pain’ and ‘muscle fatigue’ (discontinuous).
  40. 5.2 An example sequence of transitions. Given the states of stack and buffer (blue highlighted), as well as the previous actions, predict the next action (i.e., LEFT-REDUCE) which is then applied to change the states of stack and buffer.
  41. 5.3 The impact of mention length and interval length on recall. Mentions with interval length of zero are continuous mentions. Numbers in parentheses are the number of gold mentions.
  42. (a) CADEC
  43. (b) CADEC
  44. (c) ShARe/CLEF 2013
  45. (d) ShARe/CLEF 2013
  46. (e) ShARe/CLEF 2014
  47. (f) ShARe/CLEF 2014

###### List of Tables

  1. 1.1 The sequence of tokens is treated as a sequence of characters.
  2. 1.2 Discrepancy between the pre-trained model used in Flair and the target datasets: ShARe/CLEF 2013 (clinical notes) and CoNLL 2003 (news stories).
  3. 2.1 The variants of word ‘ocular’ and the corresponding rules to generate them. The indentation reflects the hierarchical structure of these variants according to the history of how they are generated.
  4. 2.2 Decline in effectiveness of a model trained on NCBI-Disease (Doğan et al., 2014), when evaluated on other datasets: i2b2-2010 (Uzuner et al., 2011), and n2c2-2019 (n2c2, 2019). The mention-level $F_{1}$ score is reported.
  5. 2.3 Evaluation results, as reported by Lange et al. (2019), of automatically annotated labels against manual annotations.
  6. 2.4 Requirement of different types of resources by each approach.
  7. 2.5 A summary of techniques to represent and score span, given a sequence of token representations $\bm{h}_{1},\cdots,\bm{h}_{n}$. $\bm{h}(i,j)$, being a fixed-length vector representation of the span, with its dimension being a hyper-parameter. $\text{score}(i,j)$ is the (normalised) score for the span from $i$ to $j$ inclusive, where $1\leq i\leq j\leq n$. $\text{score}(i,j)$ is usually a $c$-dimension vector, where $c$ is the number of entity categories, including a special category for non-entity.
  8. 3.1 An original training instance and different types of augmented instances. We highlight changes using italics.
  9. 3.2 The descriptive statistics of the two English datasets from the biomedical domain: i2b2-2010 (Uzuner et al., 2011) and NCBI-Disease (Doğan et al., 2014).
  10. 3.3 Evaluation results in terms of span-level $F_{1}$ score. Small set contains 50 training instances; Medium contains 150 instances; Large contains 500 instances; Full uses the complete training set. Results that are better than the baseline model without using data augmentation are highlighted in bold. underline: the result is significantly better than the baseline model without data augmentation (paired student’s t-test, p: $0.05$)
  11. 3.4 The comparison of different types of errors—FPs (false positives) and FNs (false negatives)—made by the baseline model without using data augmentation and models using Synonym Replacement (SR) and Mention Replacement (MR) data augmentation methods. $\cap$ indicates the intersection of two sets, and $\neg$ indicates the negative set. For example, the ‘FPs’ column corresponding to the ‘Baseline $\cap$ SR $\cap$ $\neg$ MR’ row shows the number of false positives predicted by both the baseline model and the model using SR data augmentation, but not by the one using MR data augmentation.
  12. 4.1 Descriptive statistics of the source datasets.
  13. 4.2 List of the target NER datasets and their specifications.
  14. 4.3 Similarity values measured between source and target datasets. TVC: Target Vocabulary Covered. JSC: Jaccarrd similarity of Vocabularies. PPL: language model perplexity. JSD: Jensen-Shannon Divergence based on term distributions.
  15. 4.4 Similarity values measured between source and target datasets (continued). TVC: Target Vocabulary Covered. JSC: Jaccarrd similarity of Vocabularies. PPL: language model perplexity. JSD: Jensen-Shannon Divergence based on term distributions.
  16. 4.5 Pre-train hyper-parameters, which follow the practice of training ELECTRA-SMALL in (Clark et al., 2020).
  17. 4.6 The effectiveness of domain-specific pre-trained models on downstream NER tasks. We report the mention level $F_{1}$ scores.
  18. 4.7 Comparison between our best performing domain-specific models and the publicly available generic domain model.
  19. 5.1 The descriptive statistics of the datasets. ADE: adverse drug events; Disc.M: discontinuous mentions; Disc.M L.: discontinuous mention length, where intervals are not counted. Numbers in parentheses are the percentage of each category. Note that due to sentence segmentation issue, there are 13 and 64 mentions crossing multiple sentences in ShARe/CLEF 2013 and ShARe/CLEF 2014, respectively. We remove these mentions, as we frame the task as a sentence-level NER problem.
  20. 5.2 Evaluation results in terms of precision (P), recall (R) and $F_{1}$ score (F).
  21. 5.3 Evaluation results on sentences that contain at least one discontinuous mention.
  22. 5.4 Evaluation results on discontinuous mentions only.
  23. 5.5 Evaluation results on different categories of discontinuous mentions. ‘#’ columns show the number of gold discontinuous mentions in development set of each category.
  24. 5.6 Example sentences involving discontinuous entity mentions and predictions using different methods. These examples are taken from CADEC. Gold discontinuous mentions are highlighted in bold. We cross out the incorrect predictions (false positives) for easy understanding.
  25. 5.7 Ablation study to estimate the contribution of attention and ELMo components.

## Notations

$a$ |  A scalar
---|---
$\bm{a}$ |  A vector
$\bm{A}$ |  A matrix
$\mathbb{A}$ |  A set
$P(\textnormal{a})$ |  A probability distribution over a discrete variable
$f(x;\bm{\theta})$ |  A function of x parameterised by $\bm{\theta}$
$\bm{1}_{\mathrm{condition}}$ |  is 1 if the condition is true, 0 otherwise
$\oplus$ |  Concatenating two vectors
$\\{t_{i}\\}_{i=1}^{N}$ |  A sequence of $N$ elements, such as tokens or vectors

## Abbreviations

cf. |  confer/conferatur (compare)
---|---
e.g. |  exemplum gratia/conferatur (example)
et al. |  et alia (and others)
etc. |  et cetera (and so on)
i.e. |  id est (that is)
ADE |  Adverse Drug Event
---|---
ADR |  Adverse Drug Reaction
BERT |  Bidirectional Encoder Representations from Transformers
BIO |  Beginning-Inside-Outside
EHR |  Electronic Health Record
ELECTRA |  Efficiently Learning an Encoder that Classifies Token Replacements Accurately
ELMo |  Embeddings from Language Models
IE |  Information Extraction
LSTM |  Long Short-Term Memory
LM |  Language Model
NER |  Named Entity Recognition
NLP |  Natural Language Processing
UMLS |  Unified Medical Language System

##  Chapter 1 Introduction

The growth rate in the amount of biomedical documents—such as scholarly
articles, clinical notes and health forum discussions—is staggering. Unlocking
information trapped in these documents can enable researchers and
practitioners to operate confidently in the information world. Biomedical
Information Extraction (IE) system aims to automatically extract structured
information—such as biomedical concepts, attributes, events, and their
relations—from unstructured text. Within an IE system, the first step is
called Biomedical Named Entity Recognition (NER), the task of recognising
biomedical names.

In this chapter, we first use a real world application—recognising adverse
drug events from social media—as an example, to illustrate how a NER model can
be used to extract useful information (Section 1.1). Next, we describe a
unified architecture for the most popular sequence tagging based NER models,
dividing sequence taggers into two components: (1) a mapping function that
maps each token to a feature vector, and, (2) a classifier that predicts a
sequence of tags given the input sequence of feature vectors (Section 1.1.1).
Then, in Section 1.2, we identify three open challenges of applying state-of-
the-art sequence taggers, enhanced by pre-trained language representation
models, to recognise biomedical names: (1) complex structures—overlapping and
discontinuity—occur often in biomedical names; (2) the training of sequence
taggers requires large training sets which are usually difficult to obtain in
the biomedical domain; and, (3) there is a discrepancy between publicly
available language representation models pre-trained on generic data and
target biomedical data. To deal with these challenges, we explore different
research directions and make the following contributions: we propose a
transition-based model for discontinuous NER; we develop a cost-effective
approach that nominates the suitable pre-training data; and we design several
data augmentation methods for NER (Section 1.3).

###  1.1. Recognising Adverse Drug Events from Social Media—A Motivating
Application

An _Adverse Drug Reaction_ (ADR) is an injury occurring after a drug is used
at the recommended dosage, for recommended symptoms (Karimi et al., 2015b).
Detecting ADRs as early as possible can potentially have a major impact,
because ADRs are among the leading causes of death in many countries, and ADR-
related costs have exceeded the cost of medications (WHO, 2020). Bonn (1998);
Hadi et al. (2017); and Khalil and Huang (2020) estimate that ADRs account for
more than $100,000$ deaths per year in the United States, and $197,000$ deaths
annually in Europe. The situation in developing countries may be more severe.
For example, Mouton et al. (2015) estimate that in South Africa ADRs
contribute to the death of $2.9\%$ of medical admissions, and $16\%$ of deaths
are ADR-related.

Different from controlled clinical trials which are mainly conducted before
drugs are licensed for use, pharmacovigilance—the practice of monitoring the
ADRs of pharmaceutical products—focuses on identifying previously unreported
adverse reactions after the drugs are marketed. Establishing causality—whether
the given adverse reaction is caused by the drug—is often done by domain
experts. Causality assessment needs to investigate the statistical association
in laboratory parameters and exclude other causes, such as alcohol, disease-
related causes, other drugs and so on (Anderson and Borlak, 2011).
Surveillance systems, both passive and active, play an important role in
collecting potential _Adverse Drug Events_ (ADEs). Note that, when causality
between an adverse reaction and a drug is not known, it is referred to as an
adverse drug event.

_Passive surveillance_ of ADEs relies on spontaneous reporting systems which
allow health professionals and patients to voluntarily report observed or
suspected ADEs to regulatory agencies. For example, the MedWatch system has
been built by the Food and Drug Administration since the early $1990$s
(Piazza-Hepp and Kennedy, 1995). However, under-reporting is severe. Studies
estimate that more than $90\%$ of ADEs are not reported to these systems due
to various obstacles, such as lack of suspicions, lack of information about
reporting utility, lack of time, and difficulties in filling out forms
(Vallano et al., 2005; Hazell and Shakir, 2006).

_Active surveillance_ , in contrast, aims to discover ADEs automatically from
multiple sources, including Electronic Health Records (EHRs), medical
literature, search engine logs, and even social media. Since such information
is often trapped in free text representation, IE systems can be used to
extract information of interest. NER is usually employed at the very beginning
of the IE system. The sentence, represented as a sequence of tokens, is taken
as input of the NER model, and entity mentions, each of which is represented
as a set of token positions, are outputted. In addition, one entity category,
such as drug, disease, symptom, ADE and so on, is assigned to each entity
mention.

###### A simple example

In this section, we describe a simple example of a post from a patient forum
and explain how the NER model recognises biomedical names.

Given a sequence of tokens:

> After two days of being on Cymbalta , I noticed an increase in
> flatulance111The spelling error is from the original post. and the worst
> smelling gas I’ve ever smelled .

the NER model is supposed to recognise three entity mentions: ‘Cymbalta’, as a
drug mention, ‘increase in flatulance’ and ‘smelling gas’, as ADEs (Figure
1.1).

Figure 1.1. An example input sentence and the entity mentions which are
supposed to be recognised by the NER model.

###### Sequence tagging based NER model

The state-of-the-art NER model is based on sequence tagging technique that
assigns a tag to each token. The tag is usually composed of a position
indicator and an entity category. The position indicator is used to represent
the token’s role in a mention. For example, in the BIO schema (Sang and
Meulder, 2003), B stands for the Beginning of a mention, I for the Inside of a
mention, and O for Outside a mention. Figure 1.2 is an example of input
sequence of tokens and the corresponding output sequence of tags. Taking the
token ‘smelling’ as an example, its tag ‘B-ADE’ indicates that the token is
the beginning token of an ADE mention.

Figure 1.2. State-of-the-art NER model is based on sequence tagging technique
that assigns a tag to each token. Token positions of mentions can be extracted
from the output tag sequence.

Once the sequence of tags is outputted, token positions of mentions can be
extracted from the tag sequence via finding all sub tag sequences starting
with ‘B-$\star$’ tag, and including all succeeding ‘I-$\star$’ tags. Put
another way, for each token whose tag starts with ‘B’, there is a mention
starting at this token position, and ending before the next token position
where the corresponding tag is ‘O’ or starts with ‘B’. Note that it is
possible for the sequence tagger to predict an invalid sequence of tags, for
example, a tag ‘B-ADE’ followed by a tag ‘I-Drug’. Therefore, post-processing
steps, such as changing the tag’s position indicator ‘I’ to ‘B’ if its entity
category is different from the preceding category, are usually employed before
the tag sequence is decoded into mentions.

####  1.1.1. A unified architecture for sequence taggers

In general, sequence taggers can be divided into two components (Figure 1.3):

Mapping function:

it converts the input sequence of tokens into a sequence of features vectors,
each of which represents the corresponding token-in-the-context; and,

Classifier:

it predicts a sequence of tags given the input sequence of feature vectors.

Figure 1.3. A unified architecture for sequence tagging models, consisting of
a mapping function and a classifier.

The key to supervised machine learning based techniques is optimising the
model so that they can fit the labelled training data. In other words, for
each training instance consisting of the input sequence of tokens and the
output sequence of tags, the tagger aims to predict a sequence of tags as
close as possible to the grounding truth. The main advantage of recent deep
learning based techniques over conventional feature based machine learning
techniques is that the former optimises mapping function and the classifier
jointly, whereas the mapping function in the latter is usually handcrafted and
fixed during the model training stage.

Current state-of-the-art approaches (Huang et al., 2015; Lample et al., 2016;
Ma and Hovy, 2016; Yang et al., 2018) for sequence tagging use the
Bidirectional Long Short-Term Memory (BiLSTM) as the mapping function, and a
subsequent linear-chain Conditional Random Field (CRF) as the classifier.
Flair (Akbik et al., 2018) is a variant of BiLSTM-CRF sequence tagger, which
achieves the state-of-the-art performance in multiple sequence tagging
datasets, including the CoNLL 2003 English and German NER datasets. In this
section, we detail its components in a top-down manner.

###### Linear-chain CRF

Given a sequence of feature vectors: $\\{\bm{h}_{i}\\}_{i=1}^{n}$, each of
which representing a token-in-the-context, the simplest classifier can take
each feature vector as input and makes the prediction independently. That is,
for each feature vector at position $i$,

$\bm{o}_{i}=\mathrm{softmax}(\bm{W}\bm{h}_{i}+\bm{b}).$ (1.1)

This classifier ignores the relationship between neighbouring tags. For
example, if the tag at a position is ‘B-ADE’ (beginning token of an ADE), it
is impossible for the succeeding tag to be ‘I-Drug’ (inside token of a drug
name), because the tag ‘I-Drug’ should always follow a ‘B-Drug’ (beginning
token of a drug name) or another ‘I-Drug’.

CRF is a classifier that predicts the output sequence jointly, taking the
dependency between neighbouring outputs into consideration. That is, it aims
to predict a sequence of tags $\bm{\hat{O}}=\\{\bm{o}_{i}\\}_{i=1}^{n}$ which
has the maximum probability over all possible tag sequences:

$\bm{\hat{O}}=\operatorname*{arg\,max}_{\bm{O}}\,P\,(\bm{O}\mid\bm{H}),$ (1.2)

where

$P\,(\bm{O}\mid\bm{H})\propto\prod_{i=1}^{n}\psi(\bm{o}_{i-1},\bm{o}_{i},\bm{h}_{i})$
(1.3)

and

$\psi(\bm{o}_{i},\bm{o}_{j},\bm{h})=\exp\,(\bm{W}\bm{h}+\bm{A}_{\bm{o}_{i},\bm{o}_{j}}).$
(1.4)

In Equation 1.4, $\bm{A}_{i,j}$ is the compatibility score of a transition
from the tag i to tag j.

###### Token level BiLSTM layer

The LSTM variant of recurrent neural networks (Hochreiter and Schmidhuber,
1997; Graves et al., 2013) is widely used by recent work to create the
contextual representation, due to its ability to flexibly encode long-term
dependencies via a memory cell. In the LSTM architecture (Figure 1.4), the
output state at each position ($\bm{h}_{i})$ is computed by taking the input
at the current position ($\bm{x}_{i})$ as well as the hidden state and cell
state from the previous position ($\bm{h}_{i-1}$ and $\bm{c}_{i-1}$,
respectively) into consideration:

$\bm{h}_{i}=f(\bm{t}_{i},\bm{h}_{i-1},\bm{c}_{i-1};\bm{\theta}).$ (1.5)

Figure 1.4. An LSTM computes the output state by taking the entire past (left
context) of the input sequence into consideration.

By computing recursively the hidden state, the entire past history—left
context—of each position is incorporated. The term _bidirectional_ indicates
that there are two models—forward and backward models—used to capture both
left and right contexts. The backward model works in the same way but in the
reversed direction:

$\bm{h}_{i}=f(\bm{t}_{i},\bm{h}_{i+1},\bm{c}_{i+1};\bm{\theta}).$ (1.6)

In the following, we use the superscript $f$ to define states relating to the
forward model and $b$ to the backward model. For example, $\bm{h}_{i}^{f}$
indicates the contextual representation of the $i$-th token from the output in
Equation 1.5 and $\bm{h}_{i}^{b}$ the contextual representation of the $i$-th
token obtained from output in Equation 1.6. A convention of employing BiLSTM
is that the final contextual representation at each position $\bm{h}_{i}$—the
contextual representation of the $i$-th token—is usually extracted by
concatenating the hidden states for each position from both forward and
backward models:

$\bm{h}_{i}=\begin{bmatrix}\bm{h}_{i}^{f}\oplus\bm{h}_{i}^{b}\end{bmatrix}.$
(1.7)

###### Contextual string embeddings

Flair introduces a novel type of token embeddings based on character level
encoder. The input token is first treated as a sequence of characters. Table
1.1 is an example sequence of tokens and the corresponding character level
start indices.

| I | noticed | an | increase | in | flatulance | and | …
---|---|---|---|---|---|---|---|---
Token level indices | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7
Character level start indices | 0 | 2 | 10 | 13 | 22 | 25 | 37 | 41
Table 1.1. The sequence of tokens is treated as a sequence of characters.

Then the sequence of characters is taken as input of two—forward and
backward—pre-trained character level BiLSTM models. The final contextual
string embeddings for each token can be extracted by concatenating outputs
from these two BiLSTM models. Specifically, for the forward model, the output
hidden state after the last character in the token is used, and the output
hidden state before the token’s first character from the backward model is
used. Taking the token ‘increase’ in Table 1.1 as an example, the output state
of the $12$-th character from the backward model and the output state of the
$21$-st character from the forward model are concatenated as the contextual
string embedding (Figure 1.5).

Figure 1.5. An forward model computes the output state by taking the entire
past (left context) of the input sequence into consideration, whereas the
backward model considers the entire right context. Both output hidden states
are concatenated to form the final contextual string embedding and capture the
information of the token itself as well as its surrounding tokens.

Finally, the stacking embeddings, a concatenation of contextual string
embedding and pre-computed GloVe embedding (Pennington et al., 2014), are used
as the final token embedding and taken as input to the previous described
token-level BiLSTM layer.

###### Pre-trained language representation models

In contrast to supervised machine learning that optimises model using labelled
data only, semi-supervised learning aims to make use of both labelled data and
unlabelled data. Pre-training language representation models on unlabelled
data and then adapting pre-trained model to the downstream supervised task is
one type of semi-supervised learning. It has demonstrated its effectiveness in
NLP during the past decade (Mikolov et al., 2013b; Dai and Le, 2015; Howard
and Ruder, 2018; Peters et al., 2018; Devlin et al., 2019). In this section,
we briefly describe the design in Flair and more options are discussed in
Section 2.3.1.

During the pre-training stage, Akbik et al. (2018) train two separate
models—forward and backward—on the 1-billion word corpus (Chelba et al.,
2013). The pre-training task is a standard character level language modelling
task that predicts the next character given a sequence of characters. Taking
the backward model illustrated in Figure 1.5 as an example, the output state
of the $12$-nd character is taken as input to a classifier to predict the next
character, whose ground truth in this example is the character ‘n’. Once the
pre-training finishes, the pre-trained models are frozen and used as part of
the mapping function for downstream supervised task (Figure 1.6).

Figure 1.6. The semi-supervised learning approach used in Flair: pre-training
two—forward and backward—character level language models, and using the pre-
trained model as part of the mapping function in the downstream supervised
tasks. For the sake of brevity, we show only the forward model and the
backward model is omitted. The whitespace character is represented using ” in
this figure.

###  1.2. Key Open Challenges

Although Flair, enhanced by pre-trained language representation models, has
achieved state-of-the-art performance in multiple NER datasets in the generic
domain, we find there are several open challenges of applying Flair to
recognise biomedical names.

####  1.2.1. Biomedical names are complex

Different from entity mentions in the generic domain, which are usually short
spans of text, biomedical names may contain more complex inner structure.
Considering the following sequence of tokens:

> have much muscle pain and fatigue .

it contains two biomedical names, ‘muscle pain’ and ‘muscle fatigue’, that
share the token ‘muscle’. In this case, we call them _overlapping_ biomedical
names. In addition, ‘muscle fatigue’ is a _discontinuous_ mention, consisting
of two components that are separated from each other.

The main motivation for recognising these biomedical names with complex inner
structure is that they usually represent compositional concepts that differ
from concepts represented by individual components. Specifically, each of
these two names in the example sentence—‘muscle pain’ and ‘muscle
fatigue’—describes a disorder which has its own CUI (Concept Unique
Identifier) in UMLS (Unified Medical Language System), whereas ‘muscle’,
‘pain’, and ‘fatigue’ also have their own CUIs. In downstream applications,
such as pharmacovigilance, extracting these compositional concepts, such as
symptoms or ADEs, is often more useful than extracting individual components
which may refer to body locations or general feelings.

####  1.2.2. Labelling data is difficult

A well known limitation of deep neural models is that training these models
usually requires large amount of labelled data (LeCun et al., 2015). In other
words, the advantages of deep learning may diminish when working with small
training sets. For example, Shen et al. (2018) observe that a deep neural
model outperforms the best shallow model by absolute $F_{1}$ score of $2.2$,
when a large NER training set—OntoNotes 5.0, containing more than $1$ million
tokens —is available. In contrast, this advantage becomes only $0.4$, when
training on a comparatively small training set (CoNLL 2003, containing around
$0.2$ million tokens).

Labelling large amount of generic NER data is time-consuming, because the
annotation needs to be done at the token level. Labelling large amount of
biomedical NER data is even more difficult due to the following reasons:

  * •

The previously described complex structure increases the difficulty of
annotation. Standard NER annotation is usually done at the token level:
annotators need to scrutinise every token to decide whether it is part of
_one_ entity mention. However, due to the complex structure—overlapping and
discontinuity—in biomedical names, one token may belong to multiple biomedical
names, and tokens that are far away from each other may form one biomedical
name. Exhaustive enumeration of possible names, including discontinuous and
overlapping ones, is exponential to sentence length.

  * •

Domain knowledge is required to annotate biomedical NER datasets. Different
from the task of annotating generic entity mentions, such as person names or
locations, with which ordinary people are familiar, recognising biomedical
names, such as biological substances or disorders, requires the annotators to
have at least basic domain knowledge.

Worse still, the same entity category may have subtle meanings in different
biomedical applications. This may even require annotators to have expert level
knowledge in a specific application. For example, _family history extraction_
is a task that focuses on the detection of family history related disorders.
Therefore, it also pays attention to some behaviour patterns which may be
caused by genetic factors, and these behaviour patterns are usually overlooked
by popular disorder recognition tasks (Rybinski et al., 2021). In other words,
a behaviour pattern is usually not defined as a disorder of interest in most
of biomedical applications, but needs to be labelled as disorder in family
history extraction application, once the behaviour—such as a pattern of
alcohol use—may put people health at risk, and it may be influenced by genetic
factors.

  * •

Some annotation tasks in the biomedical domain may cause negative impacts on
annotators. For example, annotators may feel uncomfortable after continuing
annotating online posts about adverse drug events for a long time. These posts
are written by patients, containing complains about their sufferings after
drug usage. Proper protective arrangements need to be made to protect the
annotators, and they usually lead to longer annotation task duration.

  * •

The last, but not the least, reason relates to the cost-benefit analysis
widely used in project management activities. That is, a project for building
biomedical applications usually starts from defining target performance
specifications, and then estimates the cost of achieving the target
performance. Labelling training data is often the most expensive part of the
project, and, unfortunately, we do not have practical methods to estimate how
much training data is required to achieve the target performance (Johnson et
al., 2018). So a more practical strategy is that domain experts usually first
annotate a small set of training data, on which NLP practitioners need to
build pilot models. After the persuasive results are obtained using limited
amount of training data, domain experts and project managers are more likely
to commit more resources to create more labelled training data.

####  1.2.3. Unlabelled biomedical data are limited

The main strengths of Flair come from the use of stacking embeddings, that
consist of two types of embeddings: pre-trained GloVe embedding (Pennington et
al., 2014) and contextual string embeddings based on pre-trained language
models. Akbik et al. (2018) show that the use of pre-trained GloVe embedding
increases average $F_{1}$ score by $1.1$, and the use of contextual string
embeddings brings even larger improvements, around $4.5$ absolute $F_{1}$
score on the English NER dataset. However, both of these two types of
embeddings are pre-trained on generic data. For example, GloVe embeddings are
pre-trained on the English Wikipedia and Gigaword dataset (archive of news
stories). These generic data usually have very different characteristics from
the biomedical data.

|  CoNLL 2003 |  ShARe/CLEF 2013
---|---|---
GloVe (Vocabulary coverage) | 87.6 % | 37.2 %
18,415 / 21,089 | 5,282 / 14,172
String embeddings (Perplexity) | 7.746 | 29.839
Table 1.2. Discrepancy between the pre-trained model used in Flair and the
target datasets: ShARe/CLEF 2013 (clinical notes) and CoNLL 2003 (news
stories).

Pre-trained models used in Flair usually have sub-optimal performances on
biomedical datasets, such as ShARe/CLEF 2013, which is sourced from clinical
notes.

We measure this discrepancy between the pre-trained model and the target data
using two measures: vocabulary coverage and perplexity. Vocabulary coverage
indicates the ratio of target data’s vocabulary existing in the pre-trained
model. For example, there is only 37.2 % of ShARe/CLEF 2013’s vocabulary
covered by GloVe (Table 1.2). Perplexity is a way of evaluating the language
model, which is used to generate contextual string embeddings in Flair. The
pre-trained character level language models achieve higher perplexity—a
measurement of how well a language model predicts a test sentence—on
ShARe/CLEF 2013\. Note that high perplexity indicates the language model is
bad at predicting the test sentence, assigning low probability. The result
suggests that there is a higher discrepancy between pre-trained models and the
ShARe/CLEF 2013 than CoNLL 2003, which is sourced from news stories.

Unfortunately, the access to unlabelled data in the biomedical domain can be
restricted due to privacy and regulatory reasons. Documents with privacy
sensitive contents, such as electronic health records, are usually available
only after applying anonymisation operations. For example, the Health
Insurance Portability and Accountability Act (HIPAA) of the United States
defines that $18$ types of Protected Health Information (PHI), such as patient
names, ages, phone numbers etc., need to be removed from the documents before
they can be shared with third parties. Selecting proper pre-training data
which are large enough and also similar to target task data is a non-trivial
problem.

###  1.3. About the Thesis

To deal with these open challenges, we explore the corresponding research
directions, aiming to improve the Biomedical NER. Figure 1.7 is a high-level
overview of concepts we cover in this thesis. Although we focus on Biomedical
NER in this thesis, some of these contributions, including proposed
discontinuous NER models and new discoveries regarding the selection of pre-
training data, can be potentially applied to other NLP tasks in other domains.
Also it is worthy noting that we consider English text only.

Figure 1.7. We focus on Biomedical NER, and explore the following three
research directions: recognising discontinuous entity mentions; pre-training
domain-specific language representation models; and enhancing the
effectiveness of NER models using data augmentation.

####  1.3.1. Publications

The work in the thesis primarily relates to the following peer-reviewed
articles (sorted by publication date):

  1. (1)

Xiang Dai, Sarvnaz Karimi, and Cecile Paris. 2017. Medication and adverse
event extraction from noisy text. In Proceedings of the Australasian Language
Technology Association Workshop, pages 79–87, Brisbane, Australia. (Chapter 1)

  2. (2)

Xiang Dai. 2018. Recognizing complex entity mentions: A review and future
directions. Proceedings of ACL 2018, Student Research Workshop, pages 37–44,
Melbourne, Australia. (Section 2.4)

  3. (3)

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2019. Using
similarity measures to select pretraining data for NER. In Proceedings of the
2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and
Short Papers), pages 1460–1470, Minneapolis, Minnesota. (Chapter 4)

  4. (4)

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020. An effective
transition-based model for discontinuous NER. In Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics, pages
5860–5870, Online. (Chapter 5)

  5. (5)

Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2020. Cost-effective
Selection of Pretraining Data: A Case Study of Pretraining BERT on Social
Media. In Findings of the 2020 Conference on Empirical Methods in Natural
Language Processing, Online. (Chapter 4)

  6. (6)

Xiang Dai, Heike Adel. 2020. An Analysis of Simple Data Augmentation for Named
Entity Recognition. In Proceedings of the 28th International Conference on
Computational Linguistics, Online. (Chapter 3)

The following articles are related, but will not be extensively discussed in
this thesis:

  1. (7)

Nicky Ringland, Xiang Dai, Ben Hachey, Sarvnaz Karimi, Cecile Paris, and James
R. Curran. 2019. NNE: A dataset for nested named entity recognition in English
newswire. In Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics, pages 5176–5181, Florence, Italy.

  2. (8)

Aditya Joshi, Xiang Dai, Sarvnaz Karimi, Ross Sparks, Cecile Paris, and C
Raina MacIntyre. 2018. Shot or not: Comparison of NLP approaches for
vaccination behaviour detection. In Proceedings of the 2018 EMNLP Workshop
SMM4H: The 3rd Social Media Mining for Health Applications Workshop and Shared
Task, pages 43–47, Brussels, Belgium.

  3. (9)

Lukas Lange, Xiang Dai, Heike Adel, Jannik Strötgen. 2020. NLNDE at CANTEMIST:
Neural Sequence Labeling and Parsing Approaches for Clinical Concept
Extraction. In Iberian Languages Evaluation Forum (IberLEF 2020), Online.

####  1.3.2. Definition and clarification of used terms

The usage of technical terminologies in the literature is usually confusing
and inconsistent, especially when researchers from different communities use
the same term to refer to different concepts, or when some conventions are
only shared by a small group of people. For the sake of brevity, we define and
clarify some frequently used terms in this thesis.

Token:

An individual occurrence of a linguistic unit in text. We use a token to refer
to an individual word unless specified otherwise. If a word is further split
into several pieces, we use sub-tokens to refer to these pieces.

Span:

A consecutive sequence of tokens, or an individual token.

Biomedical concept:

Conceptual objects, events, and procedures in the biomedical ontologies. We
use entity and biomedical concept interchangeably.

Biomedical name:

An instance where a biomedical concept is referenced to in text. We use
mention and biomedical name interchangeably. Following (McDonald et al.,
2005), we denote the mention by the set of token positions that belong to the
mention. Therefore, a mention may consist of several spans and mentions may
overlap.

Embedding:

A mapping function that converts a token into a dense vector. It can also be
considered as a look-up dictionary, where the token is the key and the vector
is the value.

Encoder:

A mapping function that converts a sequence of tokens into a sequence of dense
vectors:

$[\bm{h}_{1},\bm{h}_{2},\cdots,\bm{h}_{N}]=\textbf{Text
encoder}\left([t_{1},t_{2},\cdots,t_{N}]\right),$

where $N$ is the sequence length.

Usually, the encoder is a trainable neural network, and the output vectors are
contextualised in the sense that they reflect both the corresponding token and
its contexts.

Attention:

A mechanism that is widely used in sequence models to allow the current state
‘attending’ to context states to obtain a context vector. The resulting
context vector is usually be used together with the current state for the
downstream layers. Given the current state $\bm{h}_{i}$ and a sequence of
context states $\bm{H}=\\{\bm{h}_{i}\\}_{i=1}^{N}$, we can calculate the
context vector $\bm{c}_{i}$ as the average of context states weighted with
attention scores:

$\bm{c}_{i}=\sum_{j=1}^{N}a_{ij}\bm{h}_{j},$ (1.8)

where $a_{ij}$ is the $j$-th element in the attention vector $a_{i}$:

$a_{i}=\mathrm{softmax}(f(\bm{h}_{i},\bm{H})).$ (1.9)

We use the function $f$ proposed by Luong et al. (2015), unless specified
otherwise:

$f(\bm{h}_{i},\bm{H})=\bm{h}_{i}^{\top}\cdot\bm{W}\cdot\bm{H},$ (1.10)

where $\bm{W}$ is a trainable weight matrix.

Language representation model:

Similar to text encoder, a language representation model can generate a
contextual vector representation for each input token. The subtle difference
between a text encoder and a language representation model is that the former
focuses more on generating task-oriented representations, and the latter
emphasises the general semantic and syntactic representations. For example, a
text encoder in an NER model may assign similar vectors to words belonging to
the same entity category, even though their semantic meaning are dissimilar.

Domain-specific vs. Generic domain:

The term ‘domain’ is loosely used in both machine learning and NLP communities
(Ramponi and Plank, 2020). In this thesis, we do not attempt to define what
constitutes a domain but assume the domain exists in a covert way.

We use domain-specific model to indicate that the model focuses on a specific
domain. For example, we call the language representation model trained on
biomedical corpora a domain-specific model. In contrast, we use generic model
to refer to a model which is supposed to capture any kind of knowledge. For
example, we call the language representation model trained on Common Crawl
corpus a generic domain model.

Supervised vs. Unsupervised:

We use supervised learning to indicate that human annotators are required to
label the training set. In contrast, if training does not require labels
annotated by human annotators, then we call it unsupervised learning, or self-
supervised learning. For example, we call pre-training language models
unsupervised learning. The task is to predict the next token, given a sequence
of tokens, and this task does not require human annotated labels.

###  1.4. Summary

In this chapter, we first illustrate how an NER model can be used to extract
useful information, improving applications in the biomedical domain. Next, we
describe a state-of-the-art NER model, Flair, which is based on sequence
tagging techniques. Then, we identify three open challenges of applying
current techniques to recognise biomedical names.

In the following chapters, we organise our related work and content chapters
into three groups, each of which focuses on solving one particular challenge:
Section 2.2 and Chapter 3 focusing on the lack of labelled data problem;
Section 2.3 and Chapter 4 on selecting suitable pre-training data given the
downstream task; and Section 2.4 and Chapter 5 on recognising biomedical names
containing complex inner structure.

##  Chapter 2 Literature Review

Named Entity Recognition (NER) techniques have developed gradually from
dictionary based and rule based to machine learning based approach during the
last several decades. Motivated by previously discussed challenges of training
supervised models—e.g., labelling biomedical NER dataset can often be
expensive and time-consuming—we provide an overview of promising approaches to
overcome the lack of training data problem, with a special focus on data
augmentation (Section 2.2.3) as well as transfer learning (Section 2.3).
Additionally, complex structures—overlapping and discontinuity—are common in
biomedical names. We review the existing methods for complex entity
recognition, and group these methods into token-level, span-level and
sentence-level approaches (Section 2.4).

###  2.1. A Brief History of NER

Information Extraction (IE) is an important Natural Language Processing (NLP)
task that aims to automatically extract structured information from
unstructured text. It has been widely used in many applications. For example,
a successful email system can identify messages that contain event
information, extract the attributes of the event (i.e., time, location, and
participants), and insert the extracted event to the calendar (Laclavík et
al., 2012). In the biomedical domain, IE has been widely used to extract
biomedical concepts, attributes, events, and their relations from scholarly
articles, clinical notes, and social media data (Sarawagi, 2008; Wang et al.,
2018b).

One of the common practices in IE is to separate processing into several
stages, among which NER is typically employed as the first step (Hobbs, 2002).
On the one hand, NER requires a deeper analysis than key word searches,
because the semantics of entity mentions are influenced by their contexts. For
example, ‘Washington’ may refer to a person, a city, a state, or an
organisation, depending on the contexts. On the other hand, NER does not seek
to fully understand every aspect of the text, such as the writer’s
communicative intent (Bender and Koller, 2020). Therefore, it focuses only on
relevant words and ignores the rest. Because of its location at a midpoint on
this spectrum, NER is a fundamental task, and it has received considerable
attention in the last several decades.

Grishman and Sundheim (1996) use the term _Named Entity_ , referring to
possible persons, organisations, and locations mentioned in text, and they aim
to recognise structured information of company activities and defence related
activities from newspaper articles. Florian et al. (2004) extend the task to
recognise mentions of textual references to conceptual objects, which can be
either named (e.g. ‘George Washington’), nominal (e.g. ‘The president’) or
pronominal (e.g, ‘He’). The entity categories studied in the generic domain
are mainly person, organisation, and location.

Biomedical NER, focusing on identifying and classifying biomedical names,
whose surface forms can represent biomedical concepts, has its unique
characteristics comparing to NER in the generic domain. Early stage efforts,
e.g., GENIA project (Collier et al., 1999; Kim et al., 2003) and BioCreAtIvE
(Critical Assessment for Information Extraction in Biology) challenges
(Hirschman et al., 2005), focus on automatically extracting genome information
from biochemical papers written by domain specialists. i2b2 (Informatics for
Integrating Biology & the Bedside) and n2c2 (National NLP Clinical Challenges)
projects (Kohane et al., 2006; Brownstein et al., 2010) start to bring
Electronic Health Records (EHRs) to researchers’ attention, by releasing
publicly available de-identified clinical notes. Additionally, the value of
informal sources, such as user generated text on the web and search engine
logs, have also been recognised by researchers. They start to use these data
for mining health related information, such as predicting epidemic events
(Joshi et al., 2019), and monitoring adverse drug events (Sarker et al.,
2015).

In this section, we provide a brief overview of the development of NER
techniques. Instead of exhaustively surveying different approaches and
discussing design variants, we describe representative work and focus on
identifying what are the strengths and limitations of different approaches.
For more detailed surveys of NER techniques in both generic and biomedical
domains, we refer the reader to (Nadeau and Sekine, 2007; Campos et al., 2012;
Yang et al., 2018; Yadav and Bethard, 2019; Li et al., 2020).

####  2.1.1. Dictionary based approach

Mikheev et al. (1999) build a minimal NER system equipped with dictionaries,
also known as gazetteers or name lists. They collect person names,
organisation names and location names from the MUC-7 training data, as well as
several external resources, including the CIA World Fact Book, financial web
sites, etc. Despite its simplicity, evaluation results on MUC-7 test set show
that pure list lookup—finding occurrences of exact matches with items from
dictionaries—performs reasonably well for locations (precision of $0.90$ and
recall of $0.86$), but not for the organisation and person categories (recall
of lower than $0.50$, precision of around $0.80$).

One serious limitation of this approach is that it cannot recognise unseen
entity mentions, i.e., entities not in the dictionaries. In addition,
maintaining large dictionaries requires great efforts. For example, there are
around 1.5 million unique family names, just in the United States. The
dictionary of company names, if at all available, would be much larger and out
of date quickly, because new companies emerge all the time.

Naming variation is another issue that needs to be overcome. For example, the
organisation dictionary might contain ‘University of Sydney’, but this
organisation may also be referred to as ‘Sydney Uni’. In the biomedical
domain, this problem is even more severe. For example, the drug
‘Acetylcysteine’, usually used for cough and other lung conditions, is also
known as ‘Acetyl Cysteine’, ‘Cysteine Hydrochloride’, ‘Cystine’, ‘N-acetyl
cysteine’, ‘N-acetylcysteine’, ‘N-acetyl-L-cysteine’, ‘N-Acétyl-L-Cystéine’,
etc.

Finally, ambiguity may be caused by the overlapping between dictionaries
belonging to different entity categories. For example, ‘J. P. Morgan’ could
belong to both the person name dictionary and the organisation name
dictionary. Ambiguity can also be caused by the usage of abbreviations and
acronyms. For example, ‘CRF’ may refer to ‘Conditional Random Field’ in the
context of natural language processing. However, the possible number of
meanings of the term ‘CRF’ in the context of biomedical is much larger,
including ‘Cardiorespiratory fitness’—relating to heart health, ‘Clinical risk
factors’, ‘Controlled Rate Freezer’—a medical equipment, ‘Chronic renal
failure’—a type of kidney disease, etc. Note that clinical notes are usually
written by practitioners under time pressure. So abbreviations and acronyms
are used frequently. All of these limitations make the dictionary based
approach more difficult to be widely employed.

####  2.1.2. Rule based approach

To overcome the previously mentioned limitations of dictionary based
approaches, efforts were made to handcraft a set of rules to alleviate the
reliance on the completeness of dictionaries. Rules can be created to expand
the dictionaries to identify previously unseen mentions. For example, MetaMap
(Aronson, 2001; Aronson and Lang, 2010) makes use of external knowledge
sources of biomedical terms—the SPECIALIST lexicon, and it employs complex
rules to identify all possible mention variants of an entity, including
acronyms, abbreviations, synonyms, or derivational variants. Table 2.1 is an
example that illustrates how expansion rules are used to generate variants
given a word. Expansion rules include ‘i’ (inflection), ‘p’ (spelling
variant), ‘a’ (acronym/abbreviation), ‘e’ (expansion of acronym/abbreviation),
‘s’ (synonym) and ‘d’ (derivational variant). For example, the expansion rule
of variant ‘ophthalmia’—‘ssd’—indicates that it is a derivational variant of a
synonym (‘ophthalmic’) of a synonym (‘eye’) of ‘ocular’.

Origin | Variant | PoS | Expansion Rule
---|---|---|---
ocular |  | adj | –
| eye |  |  | noun | s
|  | eyes |  | noun | si
|  | optic |  | adj | ss
|  | ophthalmic |  | adj | ss
|  |  | ophthalmia | noun | ssd
| oculus |  |  | noun | d
|  | oculi |  | noun | di
Table 2.1. The variants of word ‘ocular’ and the corresponding rules to
generate them. The indentation reflects the hierarchical structure of these
variants according to the history of how they are generated.

Rules can also be triggered by characteristic attributes of known entity
mentions, including their spellings and the contexts in which they appear. For
example, a spelling rule can be a simple look up for the string, such as _any
string containing ‘Mr.’ is a person_ ; or a spelling pattern, such as _any all
capitalised string is an organisation (e.g., ‘IBM’)_. A contextual rule gets
clues from surrounding words and their syntactic relationships, such as _any
proper name modified by an appositive whose head is ‘president’ is a person
(e.g., ‘Maury Cooper’ in the context of ‘… says Maury Cooper, a vice president
at …’)_.

One advantage of a rule based approach is that rules can be derived using
unlabelled text only, which are much easier to obtain. For example, Collins
and Singer (1999) build a named entity classifier using $90,000$ unlabelled
examples. They start from $7$ seed rules (_‘New York’, ‘California’ and ‘U.S.’
are locations_ ; _any name containing ‘Mr.’ is a person_ ; _any name
containing ‘Incorporated’ is an organisation_ ; _‘I.B.M.’ and ‘Microsoft’ are
organisations_), which is the only supervision in their approach. The
classifier, automatically inducing new spelling rules and contextual rules,
finally achieves over $91\%$ accuracy when evaluated on a test set of $1,000$
manually labelled instances.

However, applying these rules is challenging, when the number of rules become
large. That is, it is difficult to prioritise one particular rule over others,
especially when some of these rules may conflict with each other. For example,
the following three rules may be used to represent the same example: _any all
capitalised string is an organisation (e.g., ‘IBM’)_ ; _any string which is
all capitalised or full periods, and contains at least one period is a
location (e.g., ‘N.Y.’)_ and _any string has an appositive modifier whose head
is a singular noun (‘player’) is a person (e.g., ‘L.J., the greatest
basketball player’)_. Iterating over all possible applicable rules and
arranging them in order of importance, even if at all possible, will cause
heavy computations.

Another difficulty of applying these rule based systems is that they usually
rely on other NLP tools, such as a syntactic parser. For example, Zhang and
Elhadad (2013) use a noun phrase chunker to first identify candidate entity
mentions; and context rules used by Collins and Singer (1999) involve finding
the head word of the appositive modifier for the entity mention. Building
syntactic analysis tools itself is a challenge task, especially for
syntactically noisy text, such as clinical notes and social media data.

####  2.1.3. Statistical machine learning based approach

Statistical machine learning approaches replace ‘hard’ rules with ‘soft’
features and estimate the importance (weights) of features using labelled
training data. Tokens are typically represented by vectors, each of which can
consist of boolean, numeric and nominal values, representing each token-in-
the-context. For example, a boolean value can be used to indicate whether the
token is capitalised, and a nominal attribute can be used to represent the
stem of the token. The feature creating function, mapping from a token to a
sparse vector, is called a feature template. It controls the length of the
token vector and the meaning of each element in the vector. Once the feature
template is fixed, feature vectors—created via the same mapping function—can
be taken as input of any supervised classifier, including Decision Tree,
Maximum Entropy Models, Support Vector Machines, Hidden Markov Models and
Conditional Random Fields.

Despite the successful applications of machine learning based NER, its main
shortcoming is the requirement of sophisticated feature templates. These
features should be informative and generalisable for unseen data. This is
challenging because such high quality feature engineering requires expert
domain knowledge and is usually tailored to specific entity categories or text
types. Learning from these features may also suffer from the sparsity problem.
For example, if a stem appears only one time in the training data, it is
impossible to estimate its importance—the weight associated with the
feature—from such a rare observation.

To alleviate the burden of manually building feature templates, deep learning
models enable automated feature extraction. Distributed representations are
usually employed to solve the sparsity issue. In other words, the mapping
function is a neural network. It takes a token as input, and it outputs a
dense vector instead of a sparse vector. Feature vectors—created via the
neural network—can be combined with almost any previous mentioned classifier,
except for those which are better at sparse input vectors, such as Decision
Tree.

###  2.2. NLP for Low Resource Scenarios

Although supervised neural models have achieved state-of-the-art performance
on numerous benchmark NER datasets in the generic domain, due to the
availability of large amount of labelled training data (_high resource_), they
does not cover all applications. On one hand, the trained model usually does
not generalise well across different types of text, let alone to recognise
mentions belonging to new entity categories (Table 2.2). On the other hand,
re-annotation for a new task, domain or language requires considerable effort.

In this section, we describe several approaches—except transfer learning which
will be detailed in Section 2.3—to deal with the lack of labelled training
data problem (_low resource_). We describe data augmentation approaches in
details, because our methods described in Chapter 3 are built on top of such
related work.

Evaluation Dataset | $\mathbf{F_{1}}$ |  Task Description
---|---|---
NCBI-Disease | 87.5 |  Recognise disease names in biomedical publications
i2b2-2010 | 43.6 |  Recognise disease names (labelled as _problem_) in clinical notes
n2c2-2019 | 60.2 |  Recognise genetic disease names (labelled as _observation_) in clinical notes
Table 2.2. Decline in effectiveness of a model trained on NCBI-Disease (Doğan
et al., 2014), when evaluated on other datasets: i2b2-2010 (Uzuner et al.,
2011), and n2c2-2019 (n2c2, 2019). The mention-level $F_{1}$ score is
reported.

####  2.2.1. Distant supervision

Instead of manually labelling data, one research direction—often called
_distant_ or _weak_ supervision (Hoffmann et al., 2011)—aims to automatically
create training data by exploring existing knowledge base (e.g., Wikipedia,
MeSH111Medical Subject Headings: https://www.nlm.nih.gov/mesh/meshhome.html.
Accessed date: 3rd September 2024, CTD222Comparative Toxicogenomics Database:
http://ctdbase.org/downloads/. Accessed date: 3rd September 2024) or heuristic
rules.

Nothman et al. (2008) transform Wikipedia into named entity annotations by (1)
classifying Wiki articles into common entity categories; (2) finding all
possible inter-article links; and (3) assigning the entity category of the
target page to the anchor text. Because the authors of Wikipedia are dictated
to link only the first mention of an entity in each article, Nothman et al.
use several heuristic rules to infer additional links from shorter referential
forms. For example, the first or last word of a person name found later in the
article may also refer to the same person. In addition, heuristic rules are
employed to adjust link boundaries. For example, linked text may contain the
possessive ’s at the end of a name, and it should be removed from the entity
name.

Safranchik et al. (2020) describe a framework, which takes unlabelled data and
a set of rules as input, for creating labelled training data. Rules, which are
implemented as functions, can take unlabelled data as input and output
heuristic information about tags. For example, a simple rule can be ‘tagging
any token that appear in a dictionary of known entity category as I-$\star$,
and all other tokens as ABS, indicating that the rule abstains from assigning
a tag’. Taking the sentence in Figure 1.2 as an example, this rule—combined
with a drug dictionary and an adverse drug event dictionary—may create the
sequence of tags in Figure 2.1, if there are some tokens appearing in these
dictionaries.

Figure 2.1. Example sequence of tags generated by a rule and two domain-
specific dictionaries. For example, ‘Cymbalta’ is assigned the tag ‘I-Drug’
because it appears in the drug dictionary.

Note that it is possible that different rules output conflicting tags, if one
token appears in multiple dictionaries. Also, it is possible that the
identified span is incomplete. For example, ‘increase in flatulance’ should be
identified as an adverse drug event, but because ‘increase in’ may not appear
in the adverse drug event dictionary, these two tokens are labelled as ‘ABS’
(the rule abstains from assigning a tag). To reconcile the conflicting and
incomplete information, Safranchik et al. (2020) introduce a set of _linking
rules_ that decide whether adjacent tokens should be grouped into one span,
and which tag is used for the span. For example, ‘increase in flatulance’ can
be grouped into a span and share the tag assigned to ‘flatulance’. These
linking rules are usually implemented based on automatic phrase mining, or
with the help of language models that predicts which words may co-occur.

Although these described automatic labelling methods provide a cheap way to
obtain a large amount of labelled training data, the obtained labelled data
are usually _noisy_. Automatically annotated labels usually contain more
errors than the manual annotations (Table 2.3), which are in contrast called
_clean data_. Liang et al. (2020) point out that there is a trade-off between
recall and precision using automatic labelling. That is, setting strict rules
can generate high precision labels, but may not generalise well and thus have
low recall. In contrast, relaxed rules can increase the coverage of
annotation, leading to high recall and low precision.

Evaluation Dataset | Precision | Recall | $\bm{F}_{1}$
---|---|---|---
CoNLL Dutch (Sang, 2002) | 32.4 | 21.1 | 25.5
CoNLL Spanish (Sang, 2002) | 51.0 | 24.7 | 33.3
CoNLL English (Sang and Meulder, 2003) | 39.9 | 30.1 | 34.3
CoNLL German (Sang and Meulder, 2003) | 23.2 |  09.2 | 13.2
Estonian (Tkachenko et al., 2013) | 59.7 | 49.3 | 54.0
Table 2.3. Evaluation results, as reported by Lange et al. (2019), of
automatically annotated labels against manual annotations.

###### Learning in the presence of noisy labels

Training a supervised model on noisy labels can sometimes result in negative
results. Fang and Cohn (2016); Hedderich and Klakow (2018) show that training
on the combination of noisy training data and a small amount of clean training
data performs worse than training on clean data only. Therefore, efforts are
made to solve the noisy labelled data problem (Han et al., 2018; Liang et al.,
2020).

One popular approach of training with noisy labels is to model the true label
as a latent variable and learn a noisy model that relate the true and noisy
labels (Hedderich and Klakow, 2018; Lange et al., 2019). We use $P(y|x)$ to
represent the probability distribution of a small set of clean instances
$(x,y)\in\mathbb{C}$, and use $P(\tilde{y}|x)$ to represent the distribution
of a large set of noisy instances $(x,\tilde{y})\in\mathbb{N}$. Then, the
noisy distribution can be calculated using:

$P(\tilde{y}=j|x)=\sum_{i=1}^{k}P(\tilde{y}=j|y=i)P(y=i|x).$ (2.1)

To estimate the relationship between true and noisy labels, i.e.,
$P(\tilde{y}=j|y=i)$, Hedderich and Klakow (2018) first apply the same auto-
labelling operations on clean data $\mathbb{C}$ to obtain pairs of clean $y$
and corresponding noisy label $\tilde{y}$. Then, a simple noisy layer is used
to model the relationship between true and noisy labels using these label
pairs:

$P(\tilde{y}=j\mid
y=i)=\frac{\exp\left(b_{ij}\right)}{\sum_{l=1}^{k}\exp\left(b_{il}\right)},$
(2.2)

where

$b_{ij}=\log\left(\frac{\sum_{t=1}^{|C|}\bm{1}_{\left\\{y_{t}=i\right\\}}\bm{1}_{\left\\{\tilde{y}_{t}=j\right\\}}}{\sum_{t=1}^{|C|}\bm{1}_{\left\\{y_{t}=i\right\\}}}\right).$
(2.3)

Lange et al. (2019) further extend this method by taking the input features
into consideration. That is, they first cluster contextual token vectors, and
then build different distributions for each cluster, i.e.,
$P(\tilde{y}=j|y=i;x)$. Experimental results show that this method improves
the $F_{1}$ score up to 36% over methods without noise handling when evaluate
on low-resource NER settings.

####  2.2.2. Active learning

Active learning is a promising approach for efficient annotation, based on the
hypothesis that the learning algorithm can perform better with less training
if it is allowed to choose the data from which it learns (Settles, 2009). It
can be used when expert annotators are available during the development cycle,
but the number of instances they can annotate under budget is far less than
the usual number of labelled instances needed to train a supervised model, to
reach satisfactory performance. Instead of asking annotators to annotate a set
of randomly sampled (_passive_) instances, _active_ learning uses algorithms
to choose a small set of _informative_ instances to annotate.

A series of events in active learning is shown in Figure 2.2. They are
repeated until the annotation budget has run out or the model performance has
reached the satisfactory level. At the beginning, a model that may be trained
on a small number of labelled instances or transferred from other tasks is
available to make predictions on unlabelled data. The active learner chooses a
small number of instances, which are considered most informative, and presents
them to the expert annotators. After receiving human annotations, the model
parameters can be either retrained from scratch using all available labelled
data, or incrementally updated by training only on the newest batch of
labelled data (Shen et al., 2018).

Figure 2.2. The active learning process usually have multiple rounds, each of
which consists of five steps: (1) applying model on unlabelled data; (2)
querying on unlabelled data; (3) presenting informative instances; (4)
annotating instances; and (5) updating the model.

Although many variants exist during each step of the active learning cycle
(Settles, 2009), the key component in active learning is assessing how
_informative_ each unlabelled instance is. In the following, we describe two
widely used approaches with sequence models, including _uncertainty sampling_
and _query-by-committee_ , and refer the reader to (Settles and Craven, 2008;
Settles, 2009; Olsson, 2009) for more options.

###### Uncertainty sampling

Active learner employing uncertainty-based functions chooses the instance
whose label is most uncertain given the existing model.

Culotta and McCallum (2005) choose the instance for which the existing model
has the least confidence in its best prediction:

$\phi(\mathbf{x})=1-P(\mathbf{y}^{\star}|\mathbf{x};\bm{\theta}),$ (2.4)

where $\mathbf{y}^{\star}$ is the most likely label sequence, and
$\bm{\theta}$ represents the existing model.

Scheffer et al. (2001) choose the instance with the smallest margin between
its two best predicted label sequences:

$\phi(\mathbf{x})=-(P(\mathbf{y}_{1}^{\star}|\mathbf{x};\bm{\theta})-P(\mathbf{y}_{2}^{\star}|\mathbf{x};\bm{\theta})),$
(2.5)

where $\mathbf{y}_{1}^{\star}$ and $\mathbf{y}_{2}^{\star}$ are the first and
second most likely labelling, respectively.

###### Query-By-Committee

In contrast to uncertainty sampling where only one model is used, methods
belonging to query-by-committee category use multiple models, known as a
committee of models. The active learner chooses the instance over which a
committee of models are in most disagreement. Note that the committee needs to
be comprised of diverse models.

Settles and Craven (2008) use the bagging technique to train different models.
Several subsets are first randomly sampled with replacement from the original
labelled training set. The same base model is then trained on each subset to
create a committee of diverse models. Similarly, Shen et al. (2018) draw a
committee of models via applying independently sampled dropout masks—thus
different subsets of the neural network—to the same CNN-LSTM model.

To measure disagreement among a set of $C$ models, Argamon-Engelson and Dagan
(1999) introduce a measure called vote entropy:

$\phi(\mathbf{x})=-\frac{1}{T}\sum_{t=1}^{T}\sum_{j=1}^{J}\frac{F(\mathbf{y}_{t},j)}{C}\log\frac{F(\mathbf{y}_{t},j)}{C},$
(2.6)

where $\mathbf{y}_{t}$ be a list of $C$ labels predicted by all the committee
models at sequence position $t$, and $F(\mathbf{y}_{t},j)$ is the frequency of
label $j$ in $\mathbf{y}_{t}$. Shen et al. (2018) first find the most popular
choice $\mathbf{y}_{t}^{\star}$ in $\mathbf{y}_{t}$, and then measure the
disagreement by calculating the ratio of models which disagree with
$\mathbf{y}_{t}^{\star}$:

$\phi(\mathbf{x})=-\frac{1}{T}\sum_{t=1}^{T}(1-\frac{|{c:\mathbf{y}_{t}^{(c)}\neq\mathbf{y}_{t}^{\star}}|}{C}),$
(2.7)

where $|\cdot|$ denotes cardinality of a set.

Instead of measure disagreement on the token-level, Settles and Craven (2008)
describe two sequence-level measures, which consider the label sequence as a
whole. Given the posterior probability of a label sequence based on a
particular model, $P(\mathbf{\hat{y}}|\mathbf{x};\bm{\theta}^{(c)})$, they
first calculate the probability given the committee of models via:

$P(\mathbf{\hat{y}}|\mathbf{x};C)=\frac{1}{C}\sum_{c=1}^{C}P(\mathbf{\hat{y}}|\mathbf{x};\bm{\theta}^{(c)}).$
(2.8)

Then a set of predicted label sequences, $\mathcal{N}^{C}$, is obtained by
taking the union of the $N$-best predictions from all models. Finally, the
disagreement can be measured by calculating sequence Kullback-Leibler:

$\phi(\mathbf{x})=\frac{1}{C}\sum_{c=1}^{C}\sum_{\mathbf{\hat{y}}\in\mathcal{N}^{C}}P(\mathbf{\hat{y}}|\mathbf{x};\bm{\theta}^{(c)})\log\frac{P(\mathbf{\hat{y}}|\mathbf{x};\bm{\theta}^{(c)})}{P(\mathbf{\hat{y}}|\mathbf{x};C)},$
(2.9)

or sequence entropy:

$\phi(\mathbf{x})=-\sum_{\mathbf{\hat{y}}\in\mathcal{N}^{C}}P(\mathbf{\hat{y}}|\mathbf{x};C)\log
P(\mathbf{\hat{y}}|\mathbf{x};C).$ (2.10)

####  2.2.3. Data augmentation

Data augmentation, expanding the training set by transforming training
instances without changing their labels, is heavily studied in the field of
computer vision (Shorten and Khoshgoftaar, 2019). Simple augmentations, such
as cropping, resizing, rotating and flipping, have become standard practices
in vision tasks. However, data augmentation is still under exploration in NLP.
In this section, we survey data augmentations for sentence level NLP tasks,
such as text classification, natural language inference and machine
translation, and group them into four categories based on how they generate
augmented instances: (1) word replacement, (2) mention replacement, (3) word
position swapping, and (4) using generative models.

##### Word replacement

Various word replacement approaches have been explored to generate augmented
instances for text classification tasks. Zhang et al. (2015b) generate
augmented instances by replacing words in the original instance with their
synonyms, which are retrieved from an English thesaurus—WordNet (Miller et
al., 1990). They first extract all replaceable words from the original
instance, and randomly choose $n$—determined by a geometric distribution—of
them to be replaced. Then a random synonym given a word is chosen to replace
the original word. Similarly, Wei and Zou (2019) randomly choose $n$ words
that are not stop words and replace each of them with one of its synonyms
chosen at random. Wei and Zou show that, when the number of original training
instances is small (i.e., $500$), randomly choosing and replacing $10\%$ of
words from the sentence can increase the classification accuracy by $2\%$ on
average. However, when replacing too many words, for example more than $20\%$
of words in the sentence, performance gain diminishes.

Kobayashi (2018) proposes context-aware augmentation that replace words with
other words which are predicted by a language model at the word positions.
Specifically, the author pre-trains a BiLSTM language model on WikiText-103
(Merity et al., 2016) – a subset of English Wikipedia articles. Then, given
the surrounding words, denoted as $S$, at each word position $i$, replacement
is sampled from an annealed distribution, $P(\cdot|S)^{1/\tau}$, using the
language model. The parameter $\tau$ is used to control the strength of the
language model. That is, when $\tau$ becomes infinity, the words are sampled
from a uniform distribution. When it becomes zero, the augmentation word is
always the one with the highest probability. One problem of context-aware
augmentation is that the predicted word may not be compatible with the
original label. For example, in a sentiment analysis dataset, the original
instance ‘the actors are fantastic’ is labelled as _positive_. Given the word
position of ‘fantastic’, the language model often assigns high probabilities
to words such as ‘bad’ or ‘terrible’. To solve this problem, Kobayashi
concatenates the embedded label $y$ with surrounding words and use it as input
to the BiLSTM language model. In other words, when training the model,
Kobayashi calculate a label-conditional language model: $P(\cdot|y,S)$ instead
of $P(\cdot|S)$. Evaluation results on several classification datasets show
that context-aware augmentation slightly outperforms synonym-based
augmentation, by accuracy of $0.5\%$ on average.

For machine translation, word replacement has also been used to generate
augmented parallel sentence pairs. Wang et al. (2018a) replace words in both
the source and the target sentence by other words uniformly sampled from the
source and the target vocabularies. Fadaee et al. (2017) search for contexts
where a common word can be replaced by a low-frequency word, relying on
recurrent language models. Similarly, Gao et al. (2019) use a monolingual
language model to obtain the replacement for a randomly chosen word. Instead
of predicting a single replacement word, they propose to replace the word by a
soft word, which is a probabilistic distribution over the vocabulary,
represented using a weighted sum of the corresponding word vectors.
Experimental results show that, on both low-resource and high-resource machine
translation datasets, the soft data augmentation can achieve more than $1.0$
BLEU score improvement over the baseline without using data augmentation.

##### Mention replacement

Instead of creating augmented instances by replacing individual words,
replacement can be employed at the mention level, usually with the help of an
external knowledge base and heuristic rules.

After observing that question answering models tend to astray by selecting a
text span that shares the answer’s type but has the wrong underlying entity
(Figure 2.3), Raiman and Miller (2017) design an augmentation strategy to make
the model more robust to surface form variation. It includes three steps:

  1. (1)

Extract nominal groups in the training set using a part of speech tagger.

  2. (2)

Perform string matching with entities in Wikidata.

  3. (3)

Randomly replace matched entities in the training set with other entities of
the same category in Wikidata.

Figure 2.3. The extractive question answering model tends to use the question
type (e.g., Who) and select the spans whose nature agrees with the question
type (e.g., ‘Bill Clinton’, ‘George H. W. Bush’, and ‘Ross Perot’), without
the necessity to understand the question.

Specifically, they extract $47,598$ entities in SQuAD that fall under $6,380$
Wikidata _instance of_ types. During each training epoch, $T$—a
hyperparameter, tuned from a range $[0,10^{5}]$—augmented instances are
generated and used in combination with the original training set. Experimental
results on SQuAD show that the proposed data augmentation improves the
performance by $F_{1}$ of $1.0$.

In order to remove gender bias from coreference resolution systems, Zhao et
al. (2018) propose to generate an augmented set where all male entities are
replaced by female entities, and vice versa, and train the model on both
original and augmented sets. They use a rule based approach consisting of two
steps. First, named entities are anonymised. For example, ‘John went to his
house’ would be anonymised to ‘E1 went to his house’. Then a dictionary of
gendered terms and their realisation as the opposite gender is used to change
all matching tokens. For example, ‘she’ is changed to ‘he’, ‘Mr.’ is changed
to ‘Mrs.’. Finally, the augmented instance ‘E1 went to her house’ is generated
and added to the training set. Evaluation results on a benchmark dataset
focused on gender bias show that this data augmentation can effectively remove
the gender bias without significantly affecting the model performance on other
coreference benchmark datasets.

##### Word position swapping

Wei and Zou (2019) randomly choose two words in the sentence and swap their
positions to augment text classification training sets. They use only one
parameter to control the number of words changed based on the sentence length.
Experimental results show that random swap can yield high performance gains
when less than $20\%$ of words in the sentence are swapped, but decline when
more than $30\%$ of words are swapped.

Min et al. (2020) explore syntactic transformations (e.g., subject/object
inversion, passivisation) to augment the training data for Natural Language
Inference (NLI) to mitigate over-fitting. This transformation does not attempt
to ensure the naturalness of the generated examples, neither the correctness
of labels. For example, in the subject/object inversion transformation, the
sentence ‘The carriage made a lot of noise’ is transformed into ‘A lot of
noise made the carriage’, and the gold label of the augmented instance is set
to _neutral_ if the original label is _entailment_. Experimental results show
that the proposed augmentation does not harm overall performance on the MNLI
test set, but it can help the model achieve better generalisation, evaluated
on HANS.

##### Generative models

Instead of creating an augmented instance by manipulating one or several
tokens in the original instance, some approaches aim to create a new instance
via generative models.

Yu et al. (2018) train a question answering model with data augmented by back-
translation from a neural machine translation model. Specifically, they use
two translation models, one model from English to French and another model
from French to English. They feed the document from an original instance into
the English-to-French model to obtain $k$ French translations via the decoder
using beam search. Then each of the French translation is passed through a
French-to-English model with beam decoder, and can thus obtain $k^{2}$
paraphrased instances in total. Experimental results on SQuAD show that the
proposed data augmentation can improve the performance by $F_{1}$ of $1.1$,
when the training data is made three times as large by adding augmented
instances.

Similarly, Xia et al. (2019b) convert data from a high-resource language to a
low-resource language, using a bilingual dictionary and an unsupervised
machine translation model in order to expand the machine translation training
set for the low-resource language. Results show that, under extreme low
resource settings, the proposed data augmentation can improve translation
quality measured by BLEU compared to supervised back-translation baselines.

####  2.2.4. Summary

In this section, we reviewed three promising approaches—distant supervision,
active learning and data augmentation—that aim to achieve high accuracy with
as little annotating efforts as possible. Different approaches make use of
different types of resources (Table 2.4), and therefore can be suitable for
different scenarios. For example, active learning requires expert-in-the-loop,
and distant supervision makes use of domain-specific knowledge base or domain
knowledge for designing heuristic rules. They are good options once these
resources are available. In contrast, data augmentation is the most flexible
approach, since some augmentation methods can be applied without the
requirement of any domain-specific resources, e.g., word replacement.
Encouraged by its adaptability and existing data augmentation methods for
sentence-level NLP tasks, we investigate easy to use data augmentation methods
for NER, which will be detailed in Chapter 3.

| Labelled data | Unlabelled data | Knowledge base | Domain expert
---|---|---|---|---
Active learning |  | $\checkmark$ |  | $\checkmark$
Data augmentation | $\checkmark$ |  |  |
Distant supervision |  | $\checkmark$ | $\checkmark$ | $\checkmark$
Table 2.4. Requirement of different types of resources by each approach.

We note that there are other approaches to overcome the low resource problem,
such as unsupervised learning (Collins and Singer, 1999; Etzioni et al., 2005;
Zhang and Elhadad, 2013), as well as transfer learning, which we describe in
the following section. These approaches are not mutually exclusive, therefore
we can combine them. For example, transfer learning—pre-training language
representation models on unlabelled data, and then fine-tuning on target
labelled data—has become a standard practice in NLP. Methods belonging to
other approaches can be combined with transfer learning, such as using off-
the-shelf pre-trained models as the baseline model.

###  2.3. Transfer Learning

The standard supervised learning requires sufficient labelled data to train a
decent performing model, given a particular task, domain and language. In
other words, each model is trained individually for a combination of task,
domain and language. In contrast, transfer learning explores the relatedness
between tasks, domains and languages. The knowledge gained in solving a
_source_ task in a _source_ domain and a _source_ language is applied to solve
the _target_ task in the _target_ domain and _target_ language (Ruder, 2019).

Yang et al. (2017) develop a transfer learning approach for sequence tagging
and design different neural architectures for cross-domain, cross-task, and
cross-lingual transfer settings. In the cross-domain transfer, the authors
share all parameters of the model—BiLSTM-CRF—and perform a label mapping on
top of the classifier (Figure 2.4(a)). Note that, cross-domain transfer
typically has mappable label sets that labels in different domains can be
mapped to each other. For the unmappable setting, Yang et al. consider it the
same as cross-task transfer, and each task learns a separate classifier
(Figure 2.4(b)). The cross-lingual transfer is achieved by exploiting the
morphologies shared by different languages. For example, the morphological
similarity between ‘Canada’ in English and ‘Canadá’ in Spanish can be
exploited for NER. The transfer learning architecture shares only the
character level mapping function, which takes a sequence of characters as
input, building a token feature vector (Figure 2.4(c)).

(a) Cross-domain transfer when the label sets are mappable.

(b) Cross-domain transfer when the label sets are disparate, and cross-task
transfer.

(c) Cross-lingual transfer. The $\oplus$ also represents a neural network that
takes both character-level and word-level feature vectors as input and creates
the final token feature vector.

Figure 2.4. Neural architectures for the settings of cross-domain, cross-task,
cross-lingual transfer proposed in (Yang et al., 2017).

Although improvements have been reported by using cross-domain, cross-task,
and cross-lingual transfers, a big challenge in these approaches is finding
related source task, domain, and language. In other words, the knowledge
learned in solving a source task in a source domain and language can be
transferred, only if the knowledge is indeed shared between the source and
target. In this thesis, we focus on English datasets; therefore, we do not
discuss cross-lingual in details; we refer readers to (Rahimi et al., 2019;
Ruder et al., 2019b; Conneau et al., 2020) for more discussions.

####  2.3.1. Cross-task transfer

Patra and Moniz (2019) point out that it is easier and faster for an annotator
to answer a yes/no question than to recognise all entity mentions. That is,
the cognitive load of selecting whether an entity mention is present or not in
the sentence is less than that of highlighting and annotating mentions with
their entity categories. Therefore, they propose to use a model which is
trained on a sentence level multi-label classification task—whether an entity
mention is present or not—and transfer it to the entity recognition task.
Evaluation results on CoNLL 2003 show that the proposed method works
surprisingly well, achieving $F_{1}$ score of 81.1.

Ruder et al. (2019a) propose a meta-architecture for multi-task learning. They
use part-of-speech tagging—a fundamental syntactic task—as the auxiliary task,
and observe that chunking, NER, and semantic role labelling tasks can benefit
from the auxiliary task, outperforming the single task learning baseline.
Similar ideas have also been explored by Collobert et al. (2011); Søgaard and
Goldberg (2016). For example, Søgaard and Goldberg (2016) use _low level_ NLP
task, such as part-of-speech tagging, to improve the _higher level_ tasks,
such as chunking and CCG Supertagging. They design specialised multi level
LSTM networks that have part-of-speech supervision at the innermost layer, and
other tasks the outermost layer.

In contrast to transferring from a source task that requires labelled data,
transfer learning techniques can make the most of limited labelled data by
incorporating language representation models pre-trained on a large amount of
unlabelled data (Mikolov et al., 2013a; Pennington et al., 2014; Peters et
al., 2018; Devlin et al., 2019). Many pre-training tasks have demonstrated
their effectiveness for different downstream tasks. In this section, we
briefly review three pre-training tasks and refer readers to (Wang et al.,
2019) for more options.

###### Skip-gram model

The Skip-gram model, introduced by (Mikolov et al., 2013a), is an efficient
method for learning static word representations from unlabelled text. The
training objective of the Skip-gram model is to build word representations
that can be used to predict the surrounding words in a sentence (Figure 2.5).
Given a sequence of tokens $\\{t_{i}\\}_{i=1}^{N}$, the Skip-gram model aims
to maximise the objective:

$\frac{1}{N}\sum_{i=1}^{N}\sum_{-c\leq j\leq c,j\neq 0}\log p\left(t_{i+j}\mid
t_{i}\right),$

where $c$, a hyper-parameter, is the size of the context window.

Figure 2.5. The Skip-gram model aims to learn word representations that can be
used to predict the surrounding words.

Skip-gram model has been shown to effectively capture syntactic and semantic
information of words (Mikolov et al., 2013b). However, its main disadvantage
is that it always assigns the same vector to the word, no matter what the
context of the word is. The word representations learned using Skip-gram model
are therefore called _static_ word representations.

###### Masked language modelling

In contrast to _static_ word representations, where one word is always
assigned the same vector, _contextual_ word representations can assign
different vectors to the same word, depending on its context.

Dai and Le (2015) explored the idea of pre-training recurrent language model
and transferring it to the downstream supervised models. They use unlabelled
data from Amazon reviews to pre-train the language model and find that it can
improve classification accuracy on the Rotten Tomatoes dataset. Peters et al.
(2017) extend the single direction language model to bidirectional. Based on
these efforts, Devlin et al. (2019) propose the masked language modelling pre-
training task to better capture contexts from both sides. Different to Peters
et al. (2018) who build two language models—left-to-right and right-to-
left—which are trained separately, the masked language modelling is a fill-in-
the-blank task. That is, a small set of tokens are masked, and the model needs
to use the context tokens to try to predict what the masked tokens should be
(Figure 2.6).

Figure 2.6. The mask language modelling pre-training task aims to learn
contextual word representations that can be used to predict what the masked
token is.

###### Replace token detection

The _replace token detection_ task, proposed by Clark et al. (2020), is a
sample efficient variant of masked language modelling (Figure 2.7). Instead of
replacing some tokens as the special [MASK] token, Clark et al. (2020) employ
a small generator network to generate plausible alternatives. Then the
discriminator network predicts whether each token in the input is replaced by
a generator sample or not.

Another difference between the masked language modelling and replace token
detection pre-training tasks is that the former is performed on only masked
tokens, whereas the latter is defined over all input tokens. Therefore, a
replace token detection pre-training task requires less compute, measured
using floating point operations. Clark et al. (2020) show that it performs
comparably to masked language modelling pre-training task while using less
than 1/4 of their compute and outperforms masked language modelling when using
the same amount of compute. Because pre-training a language representation
model using replace token detection task can be done within several days using
a single GPU, we thus use it for our investigation in Chapter 4.

Figure 2.7. The replace token detection task aims to train the discriminator
to predict whether the token is the original token or a fictional token.

####  2.3.2. Cross-domain transfer

Han and Eisenstein (2019) propose _domain-adaptive fine-tuning_ , in which the
language representation models are adapted by masked language modelling on
text from the target domain. They evaluate this approach on sequence labelling
in two challenge domains: Early Modern English and Twitter. Results show that
domain-adaptive fine-tuning yields substantial improvements over strong BERT
baselines, with particularly strong results on out-of-vocabulary words.
Similarly, Gururangan et al. (2020) investigate whether it is helpful to
tailor a pre-trained model to the domain of a target task. They show that
domain-adaptive pre-training—continue pre-training on a large corpus of
unlabelled domain-specific text—leads to performance gains. Moreover, task-
adaptive pre-training—continue pre-training on the unlabelled set for a given
task—improves performance even after domain-adaptive pre-training. Gururangan
et al. (2020) consider domain vocabularies containing the top 10K most
frequent uni-grams and use the vocabulary overlap as the measure of domain
similarity. They find that pre-training data used in RoBERTa—over 160GB of
uncompressed text, consisting of Wikipedia, books, stories, news articles, and
web content extracted from URLs shared on Reddit—have a very low vocabulary
overlap with datasets sampled from biomedical scholarly articles (27.3%) and
computer science scholarly articles (19.2%).

Moore and Lewis (2010) propose a cross-entropy difference selection method to
select in-domain training data to build auxiliary language models for use in
tasks such as machine translation and speech recognition. Given the target
data set $\mathcal{T}$ and a generic source $\mathcal{S}$, they aim to select
a subset of the available source data as language model training data. Let
$H_{T}(s)$ be the per-word cross-entropy, according to a language model
trained on $\mathcal{T}$, of a sentence $s$ drawn from $\mathcal{S}$. Let
$H_{S}(s)$ be the per-word cross-entropy of $s$ according to a language model
trained on a random sample of $\mathcal{S}$. Moore and Lewis (2010) score the
sentences from $\mathcal{S}$ according to $H_{T}(s)-H_{S}(s)$, and all
sentences whose score is less than a threshold are selected as in-domain
training data. A similar idea was explored by Klakow (2000), who estimates a
language model from the entire $\mathcal{S}$, and scores the subset of
$\mathcal{S}$ by the change in the log likelihood of $\mathcal{T}$ according
to another language model, where that subset is removed from training data.
Those subsets whose removal would decrease the log likelihood of $\mathcal{T}$
more than a threshold are selected.

Plank and van Noord (2011) evaluate measures of domain similarity and their
impact on dependency parsing accuracy. Given a target article to parse and a
collection of annotated articles, they want to select the most similar
articles to train the parser which is then evaluated on the target article.
Both probabilistically motivated similarity functions—such as Jensen-Shannon
divergence, and skew divergence—and geometrically motivated distance
functions—such as cosine, euclidean, and variational distance functions—are
evaluated on different features. Plank and van Noord (2011) find that
comparing article topic distributions estimated by Latent Dirichlet Allocation
(LDA) (Blei et al., 2003) using variational distance function or Jensen-
Shannon divergence can effectively find the most similar source, and using
these automatic measures can outperform using human annotated genre labels. In
addition to above mentioned similarity measures, Asch and Daelemans (2010)
explore to use Rényi entropy and Bhattacharyya coefficient to estimate the
impact of domain similarity on cross-domain transfer.

####  2.3.3. Summary

Inspired by these efforts that use domain similarity to nominate suitable data
for labelling or training statistical language models. We explore whether
these similarity measures can also be used to nominate in-domain data for pre-
training large scale neural language representation models.

Our work is also inspired by several lines of work that aim to link the known
to the unknown, studying its impact. Ramponi and Plank (2020) study the
implications of variations of language on model performance. They argue that
treating text as just input data to machine learning is problematic, and it is
important to study how covert and overt factors, such as genre, social-
demographic aspect, stylistic and data sampling strategy, impact results, and
take these factors into consideration in modelling and evaluation. Johnson et
al. (2018) predict a system’s accuracy using larger training data from its
performance on much smaller pilot data. In Chapter 4, we aim to link the
similarity between source pre-training data and target task data to the
effectiveness of pre-trained models. In other words, we aim to design a cost-
effective approach that predicts the usefulness of pre-trained models for
target datasets based on the similarity between the source pre-training data
and the target task data.

###  2.4. Complex Entity Recognition

Commonly, the NER problem is framed as: given a sequence of tokens, output a
list of spans, each of which is an entity mention in text. Recall that a span
is a consecutive sequence of tokens, or an individual token. The mention can
therefore be represented using the starting and ending indices of the span:
$I_{s}$, $I_{e}$. Additionally, each mention is assigned to an entity
category. This perspective imposes two constraints:

  1. (1)

An entity mention consists of a continuous sequence of tokens, where all the
tokens indexed between $I_{s}$ and $I_{e}$ are part of the entity mention;
and,

  2. (2)

These linear spans do not overlap with each other. In other words, no token
can belong to more than one entity mention.

Most of the existing NER datasets in the generic domain, for example CoNLL
2003 (Sang and Meulder, 2003), or OntoNotes 5.0 (Weischedel et al., 2011), are
annotated satisfying these two constraints. Therefore, conventional sequence
taggers achieve state-of-the-art effectiveness in these datasets (Lample et
al., 2016; Ma and Hovy, 2016; Akbik et al., 2018; Baevski et al., 2019).

However, in practice, there are domains, such as the biomedical domain, in
which there can be entity mentions nested, overlapping, and discontinuous (see
examples in Figure 2.8). These _complex entity mentions_ cannot be directly
recognised by conventional sequence taggers because they break the above
mentioned constraints based on which sequence tagging techniques are built.

In this section, we first describe these complex entity mentions in details
(Section 2.4.1). We then review the existing methods which are proposed to
recognise complex entity mentions and categorise them into token-level
(Section 2.4.2), span-level (Section 2.4.3), and sentence-level (Section
2.4.4) approaches. Finally, we identify the research gap, that our proposed
method (described in Chapter 5) is going to fill.

Figure 2.8. Examples involving nested, overlapping and discontinuous entity
mentions. In (a), ‘HIV-1 enhancer’ and ‘HIV-1’ are nested entity mentions. In
(b), ‘intense pelvic pain’ and ‘back pain’ overlap, and ‘intense pelvic pain’
is a discontinuous mention.

####  2.4.1. Definitions of complex entity mentions

###### Nested entity mentions

One entity mention is completely contained by the other. We call both of the
mentions involved nested entity mentions. Figure 2.8 a) is an example taken
from the GENIA corpus (Kim et al., 2003). Here, ‘HIV-1 enhancer’ is a DNA
mention, and it contains another mention ‘HIV-1’, which is a virus.

###### Multi-type entity mentions

An extreme case of nested entity mentions is one in which a span corresponds
to multiple mentions. For example, in the EPPI corpus (Alex et al., 2007),
proteins can also be annotated as drug/compound, indicating that the protein
is used as a drug to affect the function of a cell. Such a mention should be
classified as both protein and drug/compound. In this case, we consider this
mention as two mentions of different categories, and these two mentions
contain each other.

###### Overlapping entity mentions

Two entity mentions overlap, but neither is completely contained by the other.
Figure 2.8 b) is an example taken from the CADEC corpus (Karimi et al.,
2015a), which is annotated for adverse drug events (ADE) and relevant
concepts. In this example, two ADEs: ‘intense pelvic pain’ and ‘back pain’,
share a common token ‘pain’, and neither is contained by the other.

###### Discontinuous entity mentions

The mention consists of a discontinuous sequence of tokens. In other words,
the mention contains at least one interval. In Figure 2.8 b), ‘intense pelvic
pain’ is a discontinuous entity mention since it is interrupted by ‘and back’.

Recognising complex entity mentions is important because these mentions can
hold very useful information (Ringland et al., 2019). First, the nested and
overlapping structures themselves are already good indicators of the
relationship between different entities involved. For example, an Organisation
mention ‘University of Sydney’ contains a Location mention ‘Sydney’. This
structure implies the location of the organisation, and recognition of these
mentions can potentially speed up the construction of a knowledge base
(Ringland et al., 2019). Second, recognising complex entity mentions can
simplify the design of downstream tasks. For example, separating overlapping
mentions rather than identifying them as a single mention is important for a
downstream entity linking task, where the assumption is that the input mention
refers to one entity, and the task can thus be regarded as one-to-one mapping
(Shen et al., 2014). Third, recognising complex entity mentions can improve
the performance of other NLP tasks. For example, entity mentions often have
fixed representations in different languages. Therefore, recognising entity
mentions, especially those discontinuous entity mentions, can improve the
performance of a machine translation system (Klementiev and Roth, 2006). Last
but not least, we notice that similar complex structures also exist in other
NLP tasks, such as multi-word expressions recognition (Baldwin and Kim, 2010;
Rohanian et al., 2019) and constituency parsing (Coavoux et al., 2019; Coavoux
and Cohen, 2019). We believe the ideas proposed for recognising complex entity
mentions should also apply to similar complex structures in other tasks.

####  2.4.2. Token-level Approach

The main component of sequence tagging techniques is a structural prediction
model, which takes a sequence of contextual token representations as input and
outputs a tag for each token. Figure 2.9 is an illustration of such a model.
That is, in a linear-chain CRF model, the tag of one token depends on both the
token representation and the tag of the previous token. These local decisions
are then chained together to perform joint inference, and the tag sequence
predicted by the tagger is finally decoded into entity mentions using explicit
rules. We categorise the methods based on conventional sequence tagging as
token-level approach.

Figure 2.9. In a linear-chain CRF model, the output for each token depends on
the representation of that token in context and the output for the previous
token.

In vanilla sequence tagging models, the intermediate outputs for each token
are usually BIO schema tags. However, since the BIO tags cannot effectively
represent complex entity mentions, the first natural direction is to expand
the BIO tag set so that different kinds of complex entity mentions can be
captured. Metke-Jimenez and Karimi (2016) introduce a BIO variant schema to
represent discontinuous and overlapping entity mentions. That is, in addition
to the BIO prefixes, four new position indicators, BD, ID, BH, and IH are
proposed to denote Beginning of Discontinuous body, Inside of Discontinuous
body, Beginning of Head, and Inside of Head. Here, the token sequences which
are shared by multiple mentions are called head, and the remaining parts of
the mention are called body. Figure 2.10 is an encoding example using this
schema. ‘pain’ is the beginning of the head that is shared by two mentions,
and therefore tagged as BH. ‘intense pelvic’ is the body of a discontinuous
mention, while ‘back’ is the beginning of a continuous mention. Here, we keep
only the position indicator and remove the entity category ‘ADE’, since this
schema can only represent overlapping mentions of the same entity category.
Note that, even in this simple example, it is still impossible to represent
several mentions unambiguously. For example, this encoding can also be decoded
as having three mentions: ‘intense pelvic pain’, ‘back pain’ and ‘pain’.

Figure 2.10.  An encoding example of two adverse drug event mentions: ‘intense
pelvic pain’ and ‘back pain’.

Tag variants are also proposed to deal with complex structures with
specialised constraints. To deal with nested NER (one mention is completely
contained by the other mention), Alex et al. (2007) propose a _joined
labelling_ variant that each token is assigned a tag by concatenating the tags
of all levels of nesting. For example, the token ‘HIV-1’ in Figure 2.8 is
assigned a tag ‘B-DNA+B-Virus’, indicating that the token is the beginning
token within a DNA mention and the beginning token within a Virus mention.
Then the tagger is trained on the data containing the joined labels. During
the inference stage, the joined labels are decoded into their original BIO
format for each entity category. Rohanian et al. (2019) introduce BIOG tags
for discontinuous structure without overlapping involved. The new G position
indicator is used for tokens in between the components. Muis and Lu (2017)
propose to assign tags to the gaps between tokens, while still regarding the
problem as a sequence labelling problem. In other words, they model the
mention boundaries instead of the role of tokens in forming mentions (Figure
2.11).

Figure 2.11.  An example of mention separators encoding two nested entity
mentions: ‘IL2’ and ‘IL2 regulatory region’. Muis and Lu (2017) design three
mention separators: S, also denoted as [, indicating a mention is starting at
the next token; E (]), indicating a mention is ending at the previous token;
and C (-), indicating a mention is continuing to the next token. X means none
of the three separators applies. The standard sequence tagger, which takes as
input a sequence of $N$ tokens and outputs a sequence of $N$-1 mention
separators, can be used to recognise nested NER.

Instead of elaborating schema to encode entity mentions with complex
structures, another direction based on sequence tagging techniques is to
employ multiple sequence tagging models or layers, that are arranged in a
series. Alex et al. (2007) employ several sequence tagging models, each of
which is used to recognise entity mentions belonging to a group of several
entity categories without nested structures. Similarly, Ju et al. (2018) stack
several BiLSTM-CRF layers together, each of which is used to recognise entity
mentions belonging to a particular nesting layer. Note that, although these
two methods achieve decent results in nested NER benchmarks, they have some
difficulties in dealing with special nested structures. The cascade approach
proposed by Alex et al. (2007) cannot deal with nested mentions of the same
entity category. For example, one DNA mention might contain another DNA
mention. The layered approach proposed by Ju et al. (2018) cannot deal with
multi-type entity mentions. For example, one mention might be annotated as
both Protein and Drug/Compound.

####  2.4.3. Span-based Approach

The vanilla idea of span-based approach enumerates all possible spans – up to
a certain length in a sentence – as potential entity mentions. It then
determines whether a span is a valid entity and what is its entity category.
These candidate spans do not need to exclude each other, so the predicted
entity mentions can also overlap with each other. This advantage makes a span-
based approach a strong option for nested NER, and it has been extensively
investigated.

Given vector representations of each token $\bm{h}_{i}$ in the sentence
$\bm{H}=\bm{h}_{1},\cdots,\bm{h}_{i},\cdots,\bm{h}_{n}$ and a candidate span
$(i,j)$, the key decision of span-based approaches is how to build the span
representation and score the span for each entity category. Once all candidate
spans are represented as fix-length vectors and scored for each entity
category, they are ranked by the scores and the top-ranked spans are outputted
as the final predictions.

A summary of representative techniques to build span representations and to
score span for each entity category is shown in Table 2.5. Building span
representations via directly using boundary token representations as well as
tokens within the span is the simplest solution. Sohrab and Miwa (2018)
represent the span by concatenating the boundary token representations and the
average of all token representations within the span
($\frac{1}{j-i+1}\sum_{k=i}^{j}\bm{h}_{k}$). The span representation is then
passed to a softmax output layer to classify the span into a specific entity
category or non-entity. Similarly, Luan et al. (2018, 2019) construct span
representation by concatenating the boundary token representations
($\mathbf{h}_{i}$ and $\mathbf{h}_{j}$), an attention-based soft _headword_ ,
and embedded span width features, and then use a feed-forward neural network
to produce per-class scores for span.

Model | Representing and scoring spans
---|---
(Xu et al., 2017) | $\begin{aligned} \bm{h}(i,j)&=\begin{bmatrix}f(1,i-1)\\\ f(1,j)\\\ f(i,j)\\\ f(i,n)\\\ f(j+1,n)\end{bmatrix}\\\ &\text{where }f(i,j)=\left\\{\begin{array}[]{l}\mathbf{h}_{i}\qquad\qquad\qquad\qquad\text{if }i=j\\\ \alpha\cdot f(i,j-1)+\mathbf{h}_{j}\;\;\text{ otherwise}\end{array}\right.\\\ \text{score}(i,j)&=\,\textsc{SoftMax}(\bm{W}\cdot\bm{h}(i,j))\end{aligned}$
(Luan et al., 2018, 2019) | $\begin{aligned} \bm{h}(i,j)&=\begin{bmatrix}\bm{h}_{i}\\\ \bm{h}_{j}\\\ \textsc{Self Attention}(\bm{H})\\\ \textsc{Span width feature}\end{bmatrix}\\\ \text{score}(i,j)&=\bm{W}\cdot\bm{h}(i,j)\end{aligned}$
(Sohrab and Miwa, 2018) | $\begin{aligned} \bm{h}(i,j)&=\begin{bmatrix}\bm{h}_{i}\\\ \frac{1}{j-i+1}\sum_{k=i}^{j}\bm{h}_{k}\\\ \bm{h}_{j}\end{bmatrix}\\\ \text{score}(i,j)&=\textsc{SoftMax}\Big{(}\bm{W}\cdot\bm{h}(i,j)\Big{)}\end{aligned}$
(Xia et al., 2019a) | $\begin{aligned} \bm{e}&=\textsc{BiLSTM}(\bm{h}_{i}\cdots\bm{h}_{j})\\\ \bm{a}&=\textsc{SoftMax}\big{(}\bm{H}\bm{W}\bm{e}^{T}\big{)}\\\ \bm{C}&=\bm{a}\star\bm{H}\\\ \bm{m}&=\textsc{BiLSTM}(\bm{C})\\\ \bm{h}(i,j)&=\begin{bmatrix}\bm{m}\\\ \bm{e}\end{bmatrix}\\\ \text{score}(i,j)&=\textsc{SoftMax}\Bigg{(}\bm{W}_{2}\cdot\bigg{(}\sigma\Big{(}\bm{W}_{1}\cdot\bm{h}(i,j)+\bm{b}_{1}\Big{)}\bigg{)}+\bm{b}_{2}\Bigg{)}\end{aligned}$
(Yu et al., 2020) | $\begin{aligned} \bm{h}_{s}&=\textsc{FFNN}_{s}(\bm{h}_{i})\\\ \bm{h}_{e}&=\textsc{FFNN}_{e}(\bm{h}_{j})\\\ \text{score}(i,j)&={\bm{h}_{s}}^{\top}\bm{W}_{1}\bm{h}_{e}+\bm{W}_{2}\cdot\big{(}\bm{h}_{s}\oplus\bm{h}_{e}\big{)}+\bm{b}\end{aligned}$
Table 2.5.  A summary of techniques to represent and score span, given a
sequence of token representations $\bm{h}_{1},\cdots,\bm{h}_{n}$.
$\bm{h}(i,j)$, being a fixed-length vector representation of the span, with
its dimension being a hyper-parameter. $\text{score}(i,j)$ is the (normalised)
score for the span from $i$ to $j$ inclusive, where $1\leq i\leq j\leq n$.
$\text{score}(i,j)$ is usually a $c$-dimension vector, where $c$ is the number
of entity categories, including a special category for non-entity.

Xu et al. (2017) employ Fixed-sized Ordinarily Forgetting Encoding (FOFE) to
encode the span and its contexts into a fixed-size vector and then use a feed-
forward neural network to predict its entity category. FOFE mimics bag-of-
words but incorporates a forgetting factor ($\alpha$ in Table 2.5) to capture
positional information (Zhang et al., 2015a). Xu et al. create both word-level
and character-level features for each span and its left and right contexts:
FOFE code of the span ($f(i,j)$); FOFE code for left context including the
span ($f(1,j)$), FOFE code for left context excluding the span ($f(1,i-1)$);
FOFE code for right context including the span ($f(i,n)$); FOFE code for right
context excluding the span ($f(j+1,n)$).

Yu et al. (2020) argue the contexts of the start and end of the span are
different. They apply two separate feed-forward neural networks to create
different boundary representations ($\bm{h}_{s}$ and $\bm{h}_{e}$, in Table
2.5) for the start and end of the span. Then they use a biaffine model (Dozat
and Manning, 2016) to score the span. Xia et al. (2019a) run an additional
BiLSTM on top of the token representations and use an attention mechanism to
let tokens within the span attend to contexts to get the span representation.
Finally, a two-layer feed-forward neural network is used to score the span.

One shortcoming of span-based approaches is that all candidate spans are
scored independently. The exhaustive enumeration of possible spans creates a
large number of negative instances. That is, the majority of candidate spans
belong to a non-entity category. Also, interactions among mentions are not
explored, because all span representations are built in parallel on top of the
same underlying token representations.

We describe efforts on overcoming this problem from different perspectives:

###### Solving class imbalance problem

Given a sequence of $n$ tokens, if we enumerate all possible spans in the
sentence, there are in total $\frac{n\times(n+1)}{2}$ candidates spans. These
candidate spans belong to one of three categories:  (1) exact match with a
gold entity mention;  (2) partial overlap with a gold mention; and,  (3)
disjoint with any mention.  The latter two (negative instances) significantly
outnumber the first exact match ones (positive instances). This class
imbalance problem may result in low predictive accuracy.

To solve this problem, Xu et al. (2017) and Xia et al. (2019a) use a down-
sampling strategy. That is, they fix the total number of candidate spans in
each training batch, including all positive spans and sampled negative spans.

Sun et al. (2019) remove those negative spans that highly overlap with spans
corresponding to gold mentions. The negative span $b$ is used for training,
only if

$\max\bigg{(}\bigg{[}\textnormal{IoU}(b,g)\,\textnormal{for}\,g\,\textnormal{in}\,G\bigg{]}\bigg{)}\leq\Gamma,$
(2.11)