Crude Oil-related Events Extraction and Processing: A Transfer Learning
Approach

Meisin Leea (mei.lee@monash.edu), Lay-Ki Soona (soon.layki@monash.edu), Eu-
Gene Siewa<EMAIL_ADDRESS>
a Monash University (Malaysia Campus), Jalan Lagoon Selatan, Bandar Sunway,
47500, Selangor, Malaysia

Corresponding Author:
Meisin Lee
Tel: +603-5514 6000
Email<EMAIL_ADDRESS>
###### Abstract

One of the challenges in event extraction via traditional supervised learning
paradigm is the need for a sizeable annotated dataset to achieve satisfactory
model performance. It is even more challenging when it comes to event
extraction in the finance and economics domain, a domain with considerably
fewer resources. This paper presents a complete framework for extracting and
processing crude oil-related events found in CrudeOilNews corpus Lee et al.
[2022], addressing the issue of annotation scarcity and class imbalance by
leveraging on the effectiveness of transfer learning. Apart from event
extraction, we place special emphasis on event properties (Polarity, Modality,
and Intensity) classification to determine the factual certainty of each
event. We build baseline models first by supervised learning and then exploit
Transfer Learning methods to boost event extraction model performance despite
the limited amount of annotated data and severe class imbalance. This is done
via methods within the transfer learning framework such as Domain Adaptive
Pre-training, Multi-task Learning and Sequential Transfer Learning. Based on
experiment results, we are able to improve all event extraction sub-task
models both in F1 and MCC111Matthews correlation coefficient-score as compared
to baseline models trained via the standard supervised learning. Accurate and
holistic event extraction from crude oil news is very useful for downstream
tasks such as understanding event chains and learning event-event relations,
which can be used for other downstream tasks such as commodity price
prediction, summarisation, etc. to support a wide range of business decision
making.

###### keywords:

event extraction , transfer learning , commodity news , information extraction
, low resource domain

††journal: Expert Systems with Applications

##  1 Introduction

Event extraction is an important task in Information Extraction. It is the
process of gathering knowledge about incidents found in texts, automatically
identifying information about what happened, when it happened, and other
details. Event extraction has long been a challenging task, addressed mostly
with supervised methods222Apart from supervised methods, there is a smaller
number of work proposed the use some form of Weak Supervision, Distant
Supervision, etc (see Xiang & Wang [2019] for a survey of existing event
extraction methods). that require massive amounts of annotated data. However,
annotated data is hard and expensive to obtain. This is evident that even
canonical datasets such as ACE2005 and TAC KBP are moderate in size. This
challenge is even more apparent in specialised domains such as finance and
economics, where only experts can provide reliable labels Konyushkova et al.
[2017]. The challenge of annotation scarcity and class imbalance is
acknowledged in Chen [2021], to which the authors proposed to use transfer
learning by using a number of corpora within the BioMedical domain to increase
the coverage of event detection (event trigger detection) on their target
corpus of the same domain.

In this work, we investigate the task of event extraction in CrudeOilNews
corpus, a dataset released by Lee et al. [2022] where commodity news articles
are annotated for the task of event extraction. More information about the
CrudeOilNews corpus is laid out in Section 1.1. Events found in commodity news
articles are mainly Geo-political, Macro-economics, supply and demand in
nature. Transfer Learning has been proven to be effective for a wide range of
applications, especially for low-resourced domains Meftah et al. [2021].
Inspired by this, we explore the usage of transfer learning to produce event
extraction and event property classification models with the best possible
accuracy despite of the limited training size. Transfer learning is a set of
methods that leverages resources from other domains or resources intended from
other tasks to train a model with better generalization properties. Resources
from other domains are known as source domain, while resources intended for a
different task is known as source task. Transfer learning aims at performing a
task on a target dataset using features learned from a source dataset Pan &
Yang [2010]. In this work, we experimented with various approaches within the
Transfer Learning paradigm333The definition of Transfer Learning is aligned to
the taxonomy of Transfer Learning described in Ruder [2019]., namely Domain
Adaptive Pre-training, Sequential Transfer Learning and Multi-task Learning.
Definition of the various types of Transfer Learning are laid out in Section
1.3.

###  1.1 Dataset

The dataset used here is the CrudeOilNews corpus introduced in Lee et al.
[2022]. The dataset size is moderate where it contains, in total, 425
documents, approx. 11k events, about approx. 23k arguments, and each event is
classified according to Polarity, Modality, and Intensity properties. There
are 21 entity types444for simplicity and convenience, values and temporal
expressions are also considered as entity mentions, 18 event types. Broadly,
the events can be grouped into the following main categories:

  * 1.

geo-political: Geo-political tension, civil unrest, embargo/sanctions, trade
tensions, and other forms of geo-political crisis.

  * 2.

macro-economic: US employment data, economic/gross domestic product (GDP)
growth, economic outlook, growth forecast, supply and demand

  * 3.

Supply-Demand related: oversupply

  * 4.

price movement: price increase, price decrease, price forecast.

As for event arguments, there are 21 argument types; each event type is
associated with a set of argument roles. For more details, such as entity
mention examples and description of event types, refer to A.

###  1.2 Definitions

Before we dive into the technical details of the proposed solution, this
section is dedicated to laying out the terminologies and task descriptions.

##### Terminologies:

  1. 1.

An entity mention is an explicit mention of an entity in a text that has an
entity type.

  2. 2.

An event trigger is the main word(s) that most clearly expresses the
occurrence of an event, usually a word or a multiworld phrase. It can come in
the form of verb, noun, adjective, or adverb).

  3. 3.

An event argument is an argument filler that plays a certain role in an event.

  4. 4.

Polarity, which is also known as negation in Morante & Blanco [2012], denotes
whether an event actually happened or was negated, not to be confused with the
polarity in sentiment analysis. Its value can be POSITIVE or NEGATIVE.

  5. 5.

Modality, also known as hedge in Farkas et al. [2010], denotes whether an
event actually happened or will happen in the future, or whether it is a
generic event. Its value can be ASSERTED or OTHER.

  6. 6.

Intensity denotes if an event further intensified and lessen. Its value can be
INTENSIFIED, NEUTRAL, and EASED.

According to Lee et al. [2022], the definition of Event Polarity and Modality
in CrudeOilNews corpus are aligned with ACE2005, while Event Intensity is a
newly defined property specially crafted for events found in commodity news.

Figure 1: This is an example taken from CrudeOilNews corpus; event trigger
words are underlined.

We propose a solution for sentence-level Event Extraction, which is made up of
a few sub-tasks. These tasks555We follow the same naming convention in Nguyen
& Nguyen [2019] are described below:

##### Tasks

  1. 1.

Entity Mention Detection (EMD): a task to detect entity mentions (named or
nominal) and assign each token an entity type or NONE for tokens that are not
an entity mention. See Table 1 top section for EMD task on the example shown
in Figure 1.

  2. 2.

Event Extraction :

    1. (a)

Event Detection (ED): similar to EMD, it is a task to detect event trigger
word(s) and assign it to an event type or NONE for tokens that are not an
event trigger. See Table 1 bottom section for ED task on the example in Figure
1.

    2. (b)

Argument Role Prediction (ARP): a task aims to assign an argument role label
or NONE to a candidate entity mention. If the candidate entity is not linked
to the event. See Table 2 for ARP task on the example shown in Figure 1.

  3. 3.

Event Properties Classification: Classifying each event’s in terms of their
Polarity, Modality, and Intensity classes. See Table 3 for the classification
of each event against Polarity, Modality and Intensity.

Table 1: Entity Mention Detection (EMD) and Event Detection (ED) for the example sentence shown in Figure 1. Task |  Text |  Entity Type
---|---|---
EMD |  World |  Location
oil |  COMMODITY
prices |  FINANCIAL ATTRIBUTE
months-long |  DATE
Saudi Arabia |  COUNTRY
production |  FINANCIAL ATTRIBUTE
supply |  FINANCIAL ATTRIBUTE
Task |  Trigger Word(s) |  Event Type
ED |  fall |  MOVEMENT_DOWN_LOSS
rout |  SLOW-WEAK
cuts |  CAUSE MOVEMENT_DOWN_LOSS
surplus |  OVERSUPPLY
Table 2: Argument Role Prediction (ARP) for the example sentence shown in Figure 1. Event |  Text |  Argument Role
---|---|---
fall |  oil |  ITEM
|  prices |  ATTRIBUTE
rout |  months-long |  DURATION
cuts |  Saudi Arabia |  SUPPLIER
|  production |  ATTRIBUTE
surplus |  supply |  ATTRIBUTE
Table 3: Event properties (Polarity, Modality and Intensity) for each of the events from the example sentence in Figure 1. Event | Polarity | Modality | Intensity
---|---|---|---
prices fall | POSITIVE | OTHER | NEUTRAL
rout | POSITIVE | ASSERTED | INTENSIFIED
production cuts | NEGATIVE | OTHER | NEUTRAL
supply surplus | POSITIVE | OTHER | EASED

Based on the example in Figure 1 and information extracted via regular event
extraction, which is made up subtasks (1) EMD and ED (Table 1), and (2) APR
(Table 2), we argue that the extracted information is incomplete without
without taking into consideration its events’ properties shown in Table 3,

###  1.3 Transfer Learning

Figure 2: A taxonomy of Transfer Learning for NLP (image taken from Ruder
[2019])

Here we provide the formal definition of Transfer Learning based on Weiss et
al. [2016], Pan & Yang [2010], Alyafeai et al. [2020]. Given a source-domain
task tuple ($D_{s},T_{s}$) and different target domain-task
pair($D_{t},T_{t}$), we define transfer learning as the process of using the
source domain and task in the learning process of the target domain task.

The definition of transfer learning used here is aligned to Ruder [2019], and
the taxonomy diagram used in Ruder [2019] is shown in Figure 2. According to
this taxonomy, the main branches of Transfer Learning are:

  1. 1.

Transductive transfer Learning: It is the setting where source and target
tasks are the same; we don’t have labeled data at all or we have very few
labeled data in the target domain, but sufficient labeled data in the source
domain.

  2. 2.

Inductive transfer learning: It is the setting where the source task and the
target task are different; labeled data is only available in the target
domain.

Based on the above definition and given that we intend to fully utilize the
labels in our dataset (target domain) to train (or fine-tune) the model, we
will only focus on Inductive transfer learning and its sub-types, namely:

  1. 1.

Sequential Transfer Learning (STL): the process of learning multiple tasks
($T_{1},T_{2},....,T_{n}$). At each step $t$, we learn a specific task
$T_{t}$. There are two types of STL, each illustrated by an example below:

    1. (a)

Cross-domain STL: A model is first trained on task $T$ using a source dataset
$D_{s}$ and is then transferred to train on the target dataset $D_{t}$ on the
same task ($T_{s}$ = $T_{t}$). In Gui et al. [2018b], authors used cross-
domain STL to transfer a POS-tagging model trained using News, a resource-rich
domain, to train on the same task using Tweets, a lower resource domain;

    2. (b)

Cross-task STL: A model is first trained on task $T_{s}$ and is then
transferred to train on a different but related task $T_{t}$ in the same
domain. This is seen in Meftah & Semmar [2018], where the authors first train
a Named Entity Recognition (NER) model and then transfer the model to train on
POS-tagging task in the same dataset Meftah & Semmar [2018].

  2. 2.

Multi-task Learning (MTL): the process of learning multiple tasks
($T_{1},T2,...,T_{n}$) at the same time. All tasks are learned in a parallel
fashion. For example, both chunking and POS-tagging are trained concurrently
in Søgaard & Goldberg [2016], Ruder et al. [2017], as chunking has been shown
to benefit from being jointly trained with low-level tasks such as POS
tagging.

Negative Transfer There are cases when transfer learning can lead to a drop in
performance instead of improving it. Negative transfer refers to scenarios
where the transfer of knowledge from the source to the target does not lead to
improvements, but rather causes a drop in the overall performance of the
target task that might be lower than that with a solely supervised training on
in-target data Torrey & Shavlik [2010]. There can be various reasons or
various situation that resulted in negative transfer, such as:

  1. 1.

in MTL or STL task-centered transfer learning when the source task is not
sufficiently related to the target task or if the transfer method could not
leverage the relationship between the source and target tasks very well
Rosenstein et al. [2005];

  2. 2.

in domain adaptation when the source domain is dissimilar or less related to
target domain Meftah et al. [2021], Blitzer et al. [2007], Ruder [2019], Gui
et al. [2018a]

In this work, we look into the training of event extraction and event
classification on crude oil-related events on CrudeOilNews corpus. The
contributions of this work is summarized as follows:

  1. 1.

Proposed an end-to-end Event Extraction solution placing equal emphasis on
accurate event properties classification apart from event extraction;

  2. 2.

Utilize Transfer Learning to improve final model performance through improving
embeddings or model representations to overcome issues of labeled data
scarcity and class imbalance.

The rest of the article is organized as follows: In Section 2, we lay out
related work according to each sub-task; in Section 3, we present our proposed
framework for a complete event extraction for CrudeOilNews corpus. We dive
deeper into the respective sub-tasks in subsequent sections: Section 4 covers
event extraction while Section 5 focuses on event properties classification.
Lastly, we present the conclusion in Section 6.

##  2 Related Work

This section is structured such that related work will be discussed based on
the specific area of interest, giving a targeted analysis of each area or sub-
task within the overall event extraction task.

###  2.1 Event Extraction

The question of how to improve the extraction accuracy from a somewhat limited
set of labeled gold data has become an important one. Recently many have
started exploring transfer learning to improve event extraction through
various types of Transfer Learning described below:

##### Multi-task Learning (MTL)

Multi-task Learning is also known as joint learning or joint training in most
event extraction literature. Here we list the past work according to the
different combinations of sub-tasks:

  1. 1.

Partial Multi-task learning: jointly training ED + APR. This approach is very
common in event extraction literature and is found in Lee et al. [2021], Liu
et al. [2018], Li et al. [2013], Nguyen et al. [2016], Sha et al. [2018].
Although the approach of jointly training ED and APR is the same, all of them
used different deep learning architecture. Lee et al. [2021] used Graphical
Convolution Network (GCN) + Pruned Dependency Parse Tree, Liu et al. [2018]
used GCN with Attention Mechanism, Nguyen et al. [2016] uses Recurrent Neural
Network (RNN), Sha et al. [2018] uses Dependency-Bridge RNN and Tensor-Based
Argument Interaction.

  2. 2.

Full Multi-task learning: Joint modeling of all three sub-tasks : EMD, ED and
ARP extraction. This approach was in reported in Li et al. [2014], Yang &
Mitchell [2016], Judea & Strube [2016], Nguyen & Nguyen [2019], Zhang et al.
[2019]. The authors in Yang & Mitchell [2016] consider structural dependencies
among sub-tasks by adopting a two-stage reranking procedure, first selecting
the k-best output of event triggers and entity mentions, then performing joint
inference via reranking. Nguyen & Nguyen [2019] build a multi-task model that
exploits mutual benefits among the three tasks by sharing common encoding
layers given an input sentence. In this setting, output structures of entity
mentions, event triggers, and argument semantic roles are decoded separately.
Zhang et al. [2019], on the other hand, used a neural transition-based
framework to predict complex joint structures incrementally in a state-
transition process.

  3. 3.

Hierarchical Multi-task learning: training sub-tasks according to a
hierarhical fashion. The idea is to utilize a set of low-level tasks learned
at the bottom layers of the model to create a set of shared semantic
representations that will progressively have a more complex representation
from the more complex tasks at the higher-level. The authors in Sanh et al.
[2019] showed that these low and higher level tasks benefit trained in a
hierarchical fashion benefit each other. The authors trained Named Entity
Recognition (NER), EMD, Entity Coreference Resolution, and Relation Extraction
via a hierarchical fashion. Similarly, authors in Wadden et al. [2019] also
aims to train the same set of sub-tasks (but treating entity co-reference as
an auxiliary task) using span representation from BERT (and Graph
propagation).

Note: Multi-task solutions involving auxiliary tasks other than event
extraction, for example, the combination of ED + Entity Relation Extraction is
excluded here.

##### Sequential Transfer Learning (STL)

According to Ruder [2019], STL is a type of transfer learning. In this
approach, We train a model on a task or a dataset, and then ‘transfer’ the
model to another task or dataset. This means that, as opposed to MTL, STL
models are not optimized jointly, but each task is learned sequentially. Chen
[2021] is an example of cross-domain STL; the authors used multiple source
datasets to help achieve a wider coverage of events in the target dataset
using adversarial network-based transfer learning. The authors capitalized on
four other corpora with varying degrees of relevance to their target dataset
(all within the BioMedical domain) to extract and transfer common features
from the related source corpora effectively to boost the performance of event
trigger detection in the target dataset.

##### Other forms of Transfer Learning

In Huang et al. [2018], the authors used zero-shot transfer learning to allow
their event extraction model to generalize to new unseen event types (events
without annotation). They model event extraction as a generic grounding
problem and designed a transferable architecture of structural and
compositional neural network, that leverages existing event schemas and human
annotations for a small set of seen types, and transfers the knowledge from
the existing types to the extraction of unseen types. In Lyu et al. [2021], on
the other hand, the authors formulate zero-shot event extraction as a set of
Textual Entailment (TE) and / or Question Answering (QA) queries, exploiting
pretrained (TE/QA) models for direct transfer (transfer learning) to do the
new target task of event extraction.

###  2.2 Event Properties Classification

Even though the ACE2005 dataset is annotated with not just event details but
also properties such as Polarity, Tense, Genericity, and Modality, previous
work within the ACE2005 and TAC-KBP stream focused almost exclusively on event
detection and event extraction and under-utilizing the annotation on event
properties. Even in survey papers such as Xiang & Wang [2019] and Hogenboom et
al. [2016], the focus is solely on event extraction, omitting event properties
classification and other event-related task. Instead, event properties related
tasks are established separately from event extraction through several shared
tasks that are not necessarily event extraction related. These tasks come in
various variations and different focuses, they are:

  1. 1.

Event Realis classification Mitamura et al. [2015] in TAC KBP dataset. There
are three types of Ralis values: ACTUAL, GENERIC, and OTHER. This is
equivalent to our Modality property;

  2. 2.

CoNLL-2010 shared task: Hedge detection and scope resolution. The task is to
detect hedges and their scope in natural language text. A detailed task
description is found in Farkas et al. [2010]. This is equivalent to our
Modality property;

  3. 3.

SEM 2012 Shared Task: Negation detection and scope resolution. The task is to
detect negation and resolve its scope and focus. A detailed description of the
task is found in Morante & Blanco [2012]. This is equivalent to our Polarity
property;

  4. 4.

Modal sense classification Marasović & Frank [2016]. This is similar to
Uncertainty hedge / Modality cue word detection;

  5. 5.

Event Factuality Prediction (EFP) Saurí & Pustejovsky [2009]. This is a
combination of Negation and Speculation detection but instead of
classification, EFP is a regression task to predict a score between [+3, -3].

Even though none of the corpora above are related to Economic / Finance
domain, their tasks are similar to our event property classification.
Therefore they are potential resources for consideration for using them as
resources as source datasets in cross-domain STL.

##  3 Proposed Framework

We present a framework for end-to-end event processing that includes both
event extraction and as well as event properties classification. First, we
build baseline models via supervised learning for each sub-task using only the
CrudeOilNews dataset. Then we explore various transfer learning techniques and
leverage on available resources (source task or source dataset) to train
target models with better performance. We demonstrate, through experiments,
how much improvement (if any) these new models show compared to the baseline
models.

By referring to Figure 3, the proposed framework is described in detail in the
following sections:

  1. 1.

Domain Adaptive Pre-training on BERT to produce ComBERT for contextualized
word embedding in Section 3.1

  2. 2.

Data pre-processing in Section 3.2;

  3. 3.

Event Extraction (EE) and its sub-task: (1) Entity Mention (EMD) and Event
Detection (ED) and (2) Argument Role Prediction (ARP) in Section 4. Here, we
explore the usage of cross-domain sequential transfer learning and inductive
transfer learning.

  4. 4.

Event properties classification in Section 5. We explore cross-task sequential
learning.

Figure 3: The framework is made up of different components, each corresponding
to the sub-tasks in EE. Event extraction (EMD, ED and ARP) is covered in
Section 4 while event properties classification (Polarity, Modality and
Intensity) is covered in Section 5.

###  3.1 ComBERT: Domain Adaptive Pre-training

Figure 4: Domain Adaptive Pre-training: Using BERT as baseline, we have
further pre-trained BERT on a commodity news corpus, adapting the model to the
finance and economic news domain.

As for tokens, the model takes an input sentence as a sequence of word tokens
encoded through ComBERT, as introduced in Lee et al. [2021]. As shown in
Figure 4, ComBERT is produced from BERT-based domain adaptive pre-training on
a large collection of commodity news on the task of masked language modeling.
It was trained on a collection of commodity news extracted from
www.investing.com, the same source as the CrudeOilNews corpus. The idea is to
adapt BERT, which was trained on English Wikipedia and Brown corpus Devlin et
al. [2019] to the finance and economics domains via Domain Adaptive Pre-
training. In Lee et al. [2021], it is shown that ComBERT produced better event
extraction results than BERT and other generic language models. This is
consistent with language models produced through domain adaptive pre-training
on in-domain corpus such as SciBERT Beltagy et al. [2019] and BioBERT Lee et
al. [2020]. All event extraction tasks described later use ComBERT embeddings
and models are trained via fine-tuning from ComBERT.

###  3.2 Data Preprocessing

The annotation files made public in Lee et al. [2022] were first converted
from Brat Annotation standoff format (.ann files) along with their
corresponding news articles (.txt files) to json format. Each sentence in the
dataset was parsed using Stanford CoreNLP toolkit Manning et al. [2014],
including sentence splitting, tokenization, POS-tagging (lexical information),
NER-tagging, and dependency parsing to generate dependency parse trees
(syntactic information). For all the sub-tasks, we adopt the “multi-channel”
channel strategy where apart from using word embeddings from ComBERT, we
include, as input, additional information on each token, ie. their POS tag,
NER tag and dependency parse tree tags.

##### Input

Let W = $w_{1},w_{2},.....w_{n}$ be a sentence of length n where $w_{i}$ is
the i-th token:

  1. 1.

The word embedding vector of $w_{i}$: this is the feature representation from
a word embedding of ComBERT. It is made up of WordPiece tokenization Wu et al.
[2016] with [CLS] and [SEP]666[CLS], [SEP], [MASK] are special tokens of BERT.
For experiments involving RoBERTa, Byte-Pair Encoding (BPE) tokenization and
its special tokens are used. are placed at the start and end of the sentence.

  2. 2.

The Part-of-Speech-tagging (POS-tagging) label embedding vector of $w_{i}$:
This is generated by looking up the POS-tagging label embedding.

  3. 3.

Dependency tags: The same ”Multi-channel” approach is used where the tokens’
Universal Dependency dependency tags embedding are used. Examples of tags are
nmod, nsubj, dobj, neg, etc. This is based on the fact that entity mentions do
not influence the Polarity and Modality of an event. On the other hand,
modifier words and modal auxiliary such as not, which has the dependency tag
neg, are key to determining the Modality / Polarity / Intensity of an event.
Dependency tags provide additional syntactic information, which is helpful in
the classification task.

###  3.3 Train-Test split

Due to the limited size of CrudeOilNews corpus, we run the experiments by
using 5-fold cross-validation. Out of the 5-folds, one fold is for testing
while the remaining four folds are for training (80% for training and 20% for
testing).

###  3.4 Measurement for dataset with class imbalance

##### F1-Score

F1-score reported here is macro-average F1-score averaged across $k$
experiments. We compute the F1 score for each fold (iteration); then, we
compute the average F1 score from these individual F1 scores.

$F1_{avg}=\frac{1}{k}\Sigma_{i=1}^{k}F1_{i}$ (1)

##### MCC

In Xie et al. [2013], apart from the familiar F1-measurement the authors used
an additional evaluation metric known as the Matthew Correlation Coefficient
(MCC) to avoid bias due to the skewness of data. It takes into account true
and false positives and negatives and is generally regarded as a balanced
measure, which can be used even if the classes are of very different sizes.
MCC is a single summary value that incorporates all four cells of a $2x2$
confusion matrix777For more information on Matthews Correlation Coeffecient
(MCC), visit.

The equation for Binary Classification:

$MCC=\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$ (2)

and for Multi-class Classificatioin:

$MCC=\frac{c\times s-\Sigma_{k}^{K}p_{k}\times
t_{k}}{\sqrt{(s^{2}-\Sigma_{k}^{K}p^{2}_{k})\times(s^{2}-\Sigma_{k}^{K}t^{2}_{k})}}$
(3)

with the following intermediate variables:

  * 1.

$t_{k}=\Sigma^{K}_{i}C_{ik}$ is the number of times class $k$ truly occurred,

  * 2.

$p_{k}=\Sigma^{K}_{i}C_{ki}$ is the number of times class $k$ was predicted,

  * 3.

$c=\Sigma^{K}_{k}C_{kk}$ is the total number of samples correctly predicted,

  * 4.

$s=\Sigma^{K}_{i}\Sigma^{K}_{j}C_{ij}$ is the total number of samples,

  * 5.

$TP$ is True Positive, $FP$ is False Positive, $TN$ is True Negative and $FN$
is False Negative.

##  4 Event Extraction

In this section, we first build baseline models for all event extraction
subtasks in Section 4.1. Then we investigate the extent of how transfer
learning improves the performance of Event Extraction models from our baseline
models. First, we look at cross-domain STL in Section 4.2, and then Inductive
Transfer Learning can be used among the sub-tasks in Section 4.3.

###  4.1 Baseline Model

First, we start with our baseline model, where we train each sub-task
individually. The model architecture of each sub-task is described below:

####  4.1.1 Model Architecture

##### Entity Mention Detection (EMD) & Event Trigger Detection (ED)

We formalize both Entity Mention Detection Model (EMD) and Event Trigger
Detection (ED) tasks as multi-class token classification where a token is
classified as being at the beginning, inside, or outside an entity
mention/event trigger (BIO notation). Similar to the approach used in Nguyen
et al. [2016], we employ BIO annotation schema to assign entity type labels to
each token in the sentences. For the model architecture, we use Huggingface’s
BERTForTokenClassification with ComBERT embedding to fine-tune on this task.
Both EMD and ED are trained separately on minimizing the cross-entropy loss
function.

Pipeline versus Golden Entity Mention: A large portion of prior work on Event
Extraction has taken a simplified approach that only focused on ED and ARP,
ignoring the training of EMD Li et al. [2013], Chen et al. [2015], Nguyen et
al. [2016], Lee et al. [2021], Liu et al. [2018]. Experiments for ED and ERP
are ran based on golden annotation for entity mentions as input. Here in this
work, we aim to have a more realistic approach where we start with the EMD
sub-task and train other sub-tasks based on the entity mentions predicted by
the EMD model. We explore various task setups, i.e. various combinations of
single task, multi-task joint training, and sequential transfer learning to
minimize the issue of error propagation inherent in the pipeline approach.
Details of the various task setups are covered in Section 4.3.

##### Argument Role Prediction (ARP)

We use the solution proposed by Lee et al. [2021], where each sentence’s
syntactic information (the dependency parse tree) is used to assist with the
classification of event argument roles even though they are identical in terms
of entity type, which also includes the abundant numerical values that play
different event roles. Specifically, the proposed solution uses the pruned
dependency parse tree along with Graphical Convolution Network (GCN) as the
architecture.

The architecture for the task of Argument Role Prediction (ARP) is similar to
that of Lee et al. [2021], which uses Graph Convolutional Network (GCN) with
pruned contextual parse tree. Here we explain what pruned contextual parse
tree is and how to encode this pruned tree to be used as input in the ARP
training. As part of data pre-processing, we use Stanford CoreNLP to generate
a dependency parse tree for each sentence. We pruned the dependency tree to
obtain a sub-tree between each candidate trigger with each candidate entity
within a sentence. To further provide more contextual information, the
dependency parse tree is pruned based on $k$ distance away. Figure 5 shows the
comparison between a pruned tree of the shortest path of the candidate
trigger-entity pair and a pruned tree of the same pair but pruned to include
additional words $k$ distance away. This task is trained on minimizing the
cross-entropy loss.

Figure 5: Left: pruned dependency parse with the shortest path between
candidate trigger-entity. Right: pruned dependency parse tree with $k=1$
distance with additional contextual information for the classification of
argument roles.

This task is trained to minimize cross-entropy loss. For technical details of
the implementation of Graph Convolutional Network (GCN) over pruned contextual
parse tree, see Lee et al. [2021].

####  4.1.2 Experiments

The result of the baseline solution is shown in Table 4 (see first row), with
results for individual sub-tasks under their respective header.

###  4.2 Cross-domain Sequential Transfer Learning

Inspired by the works of Meftah & Semmar [2018] of using cross-domain STL in
transferring model trained on POS-tagging task from Newswire domain (source
domain) to Twitter text (target domain), here we investigate if we can utilize
available source datasets on event extraction to improve performance of the
same task in our target dataset further, CrudeOilNews. There are two event
extraction datasets annotated according to the ACE/ERE standards: (1)
benchmark event extraction dataset ACE2005 in the generic domain, and (2)
SENTiVENT Jacobs & Hoste [2021] for company-specific events in the finance and
economics domain. However, unlike Chen [2021], the source datasets that they
used are basically from the same domain, i.e., BioMedical Domain. In our case,
we do not have any other event extraction corpus from the same domain as
CrudeOilNews. The only two candidate corpus identified above are rather
different from our target dataset; analysis for each one is listed down below:

##### ACE2005

is a general domain corpus; out of its 33 sub-event types, almost none overlap
with the events defined in CrudeOilNews corpus. Even though 2 of the events
Conflict - Attack, Conflict - Demonstrate) may seem the same as Civil-unrest,
however upon closer scrutiny, the types of conflict here are rather different:
ACE2005 ones are at a personal level, such as a person attacking another
person, while in CrudeOilNews the conflicts are geo-political, such as social
unrest, large-scale demonstration.

##### SENTiVENT

is a corpus made up of business news and its event types, mainly company-
related or company-level events. Among the event types, there is a
‘placeholder’ event type called ‘Macroeconomic’, a broad category that
captures all non-company specific events such as market trends, market-share,
competition, regulation issues, etc. This ‘Macroeconomic’ event type is the
only event type that overlaps with CrudeOilNews corpus, unfortunately, while
they lump non-company events into one category, CrudeOilNews corpus focuses on
Macro-economic and Geo-political events in a finer-detail. Furthermore,
SENTiVENT corpus is annotated with discontinuous, multiword triggers, e.g.,
“upgraded … to buy”, “cut back … expenses”, “EPS decline”). This is distinctly
different from the way triggers are annotated in ACE2005 and CrudeOilNews
where triggers are single-word or continuous multiwords. The baseline model
developed for event detection in CrudeOilNews cannot be readily applied to
SENTiVENT without any modification.

Based on the fact that there is minimal overlap of event types between the
candidate corpus above and our target dataset, we conclude that we are not
able to utilize these two candidate corpora for cross-domain STL. This
observation is supported by the results in Chen [2021], where two out of four
of the source dataset has a very low proportion of trigger overlap that
produced worse performance in the target dataset. This is the result of
Negative Transfer.

###  4.3 Inductive Transfer Learning

As shown in the section above, we are not able to utilize any available source
datasets to boost task model performance via cross-domain STL. Instead, here
we explore Inductive Transfer Learning, where we experimented with multi-task
learning (MTL) and sequential transfer learning (STL) and the combination of
the two among event extraction sub-tasks within our own dataset to obtain the
best possible model performance. In the experiment section, we investigate
Single Task (baseline) vs Multi-task Learning (MTL) vs Sequential transfer
Learning (STL) vs a combination of MTL and STL.

As observed by authors in Sanh et al. [2019], hierarchical multi-task transfer
learning in a neural network typically allows the different tasks involved to
benefit from each other via sharing the learned representations. STL consists
of two stages: a pre-training phase in which general representations are
learned on a source task or domain, followed by an adaptation phase during
which the learned knowledge is applied to a target task or domain. Figure 6
shows the ‘model transfer’ from the source task \- EMD + ED (top box) to the
target task \- ARP (top box).

Figure 6: The combination of MTL and STL: First, the model is jointly trained
on Entity Mention (EMD) and Trigger Extraction (ED) tasks via MTL and then the
model is transferred sequentially to train on the task of Entity Arguments
Extraction (ARP).

###  4.4 Experiments and Analysis

We carried out five types of experiments with different combinations of task
setups to determine the best transfer learning configuration with maximum
benefits in terms of sharing learned representations of source tasks and
target tasks. These five different task setups are:

  1. 1.

Single Task Learning (Baseline): this is also known as the pipeline approach
where the sub-tasks are trained independently one after another, each model is
trained from scratch (no transfer learning): EMD, ED, ARP.

  2. 2.

Full Multi-task training: For the experiment, we use the approach in Zhang et
al. [2019], where all sub-tasks are trained jointly using neural transition-
based framework, predicting joint output structure as a single task.

  3. 3.

Full Sequential Training: in this approach, we start training for the EMD task
and upon completion, transfer the model to train on ED and lastly on ARP: EMD
$\rightarrow$ ED $\rightarrow$ ARP.

  4. 4.

Combination of MTL and STL:

    1. (a)

Combination #1: EMD $\rightarrow$ ED + ARP. the approach of jointly training
ED + ARP is very common in event extraction literature. This setup is used in
Lee et al. [2021], and many others see Section 2.1. The difference between Lee
et al. [2021] and this work is that in Lee et al. [2021], they jointly trained
ED + ARP together using golden entity mentions, while in this work we use the
more realistic setting where the input to the ED + ARP task is based on
entities predicted from the earlier EMD + ED model (EMD $\rightarrow$ ED +
ARP).

$joint\\_loss=loss\\_EMD+\beta(loss\\_ED)$ (4)

. In the experiment, we use $\beta$=2 to give a higher weightage to the loss
of the ED task.

    2. (b)

Combination #2: EMD + ED $\rightarrow$ ARP as shown in Figure 6. The resulting
model from training EMD + ED via MTL (in the upper box of Figure 6) is then
transferred to train for ARP (lower box in the figure). EMD + ED acts as the
source task $T_{s}$ in the context of cross-task STL to benefit the ARP task,
the target task $T_{t}$. The joint loss for EMD + ED is as follows:

$joint\\_loss=loss\\_EMD+loss\\_ED$ (5)

The results of these experiments are shown in Table 4. The detailed breakdown
by entity mention type and event type are reported in Table 10 while results
breakdown by event argument roles are reported in Table 11, both in C.

Table 4: Experiments with various different task-setups for Event Extraction to investigate Single Task Learning vs Multi-task Learning vs Sequential transfer Learning. All setups use the list of entities extracted from the EMD task and not based on Golden annotation. Task setup | EMD | ED | ARP
---|---|---|---
P |  R |  F1 |  P |  R |  F1 |  F1 |  MCC
1\. Individual sub-task training (Baseline) |  0.903 |  0.912 |  0.907 |  0.915 |  0.899 |  0.907 |  0.802 |  0.694
Task setup | EMD + ED + ARP
P |  R |  F1 |  P |  R |  F1 |  F1 |  MCC
2\. Full Multi-task Training Zhang et al. [2019] |  0.879 |  0.891 |  0.880 |  0.901 |  0.905 |  0.904 |  0.854 |  0.710
Task setup | EMD $\rightarrow$ ED $\rightarrow$ ARP
P |  R |  F1 |  P |  R |  F1 |  F1 |  MCC
3\. Full Sequential Task Training |  0.879 |  0.891 |  0.880 |  0.901 |  0.905 |  0.904 |  0.854 |  0.710
Task setup | EMD $\rightarrow$ (ED + ARP)
P |  R |  F1 |  P |  R |  F1 |  F1 |  MCC
4\. Combination #1: EMD $\rightarrow$ ED+ARP Lee et al. [2021]88footnotemark: 8 |  0.903 |  0.912 |  0.907 |  0.905 |  0.890 |  0.902 |  0.833 |  0.723
Task setup | (EMD + ED) $\rightarrow$ ARP
P |  R |  F1 |  P |  R |  F1 |  F1 |  MCC
5\. Combination #2: EMD + ED $\rightarrow$ ARP |  0.926 |  0.937 |  0.931 |  xx |  xx |  xx |  0.888 |  0.797
77footnotetext: The results presented here is not a like-for-like comparison
to the results presented in Lee et al. [2021]. This is because in the paper
golden entity mention was used as input to the joint-training of ED+ARP, while
in this work, the output for EMD is used as input to joint-training of ED+ARP.

####  4.4.1 Analysis

As expected, the worst-performing setup is the individual tasks-pipeline
approach, where it not only suffers from error propagation, but each model is
trained from scratch for each sub-task (without any interaction between them).
Both full MTL and full STL achieved slightly better results. Between these
two, the full multi-task training took a few more iterations and took longer
to train because the approach is more complex.

The best performing models are those that utilize a combination of MTL and STL
task setups. Jointly training ED and ARP together (part of the combination #1)
is a common approach in the event extraction literature Liu et al. [2018], Lee
et al. [2021]. However, in our case, we find that jointly training EMD + ED
(combination #2) brings better performance. This is because CrudeOilNews does
not exhibit strong interdependence between event type and argument roles as it
is in ACE2005. This can be explained using example sentences found in the
respective datasets in Table 5.

The best performing task setup is Combination #2: EMD + ED $\rightarrow$ ARP.
The training of EMD and ED can be done jointly via MTL without much impact on
both task. This is because we noticed that entity mentions and trigger words,
by definition, are mutually exclusive, e.g. an entity such as crude oil is
never an event trigger, vice versa an event trigger such as glut, though a
noun, is never an entity mention. Treating EMD+ED as the source task is useful
for the target task. This is related to the fact that the lower embedding and
semantic information learned from joint training EMD + ED has a good level of
knowledge about entities and triggers, the resulting model has a presentation
that is useful for the ARP target task.

Table 5: Analysis of events in ACE2005 and in CrudeOilNews respectively. Both datasets exhibit different level of interdependence between event trigger words and its event arguments. The difference in the level of interdependence influenced the selection of the best MTL and STL combination for each of the dataset. Dataset |  Analysis
---|---
ACE2005: (1) In Baghdad, a cameraman died when an American tank fired on Palestine Hotel (2) He has fired his air defence chief. |  The first occurrence of “fired” is an event trigger of type ATTACK while the second “fired” takes END-POSITION as its event type. The authors in Li et al. [2013] argues that event arguments play a key role in helping classifying the correct event. For example, the presence of “tank” plays the role of WEAPON helps determine the right event type. Likewise, in the second sentence, “defence chief” plays the role of POSITION can help the model classify the second “fired” as END-POSITION. Hence jointly training both ED and ARP helps improve the accuracy of both ED and APR.
CrudeOilNews: U.S. crude stockpiles soared by 1.350 million barrels in December from a mere 200 million barrels to 438.9 million barrels, due to this oversupply crude oil prices plunged more than 50% on Tuesday. |  Events are more straight forward, ie. trigger words are tied to just one type of event, therefore there is no need to utilize arguments to help differentiate the event type.

##  5 Event Properties Classification

###  5.1 Baseline Model

One of the challenges we face when training a model for event properties
classification is to classify each event in sentences that contain multiple
events accurately. According to Lee et al. [2022], on average, the sentences
in the CrudeOilNews corpus has about 2 to 3 events. Accurate classification of
event properties requires identifying cue words at the event scope level and
not at the sentence level, i.e., classify the entire sentence. In comparison,
both Negation and Uncertainty detection tasks in SEM2012 and CoNLL2011 are
done at the sentence level. Therefore to accurately classify event properties,
rather than using the whole sentence that may consist of several events, we
have to narrow down the scope to use only the event scope, which is just a
portion of the sentences.

####  5.1.1 Model Architecture

To obtain this ‘scope’, we experimented with the following inputs:

  1. 1.

use the fixed window of words around event trigger word(s)
($x_{i-r}...x_{i-1}\oplus x_{i}\oplus x_{i+1}...x_{i+r}$) where $\oplus$ is
the concatenation operation, $r$ represents the length from trigger word
$x_{i}$. The sequential word representation is fed into an MLP to generate a
vector and then through a softmax activation function.

  2. 2.

Use dependency parse sub-tree of event trigger in Lee et al. [2021]

  3. 3.

Use SelfAttentiveSpanExtractor Gardner et al. [2018] (part of the AllenNLP
library) to weightedly combine the representations of multiple tokens and
create a single vector for the original event span. The span vectors are fed
into a two-layer feed-forward network with a softmax activation function.

###  5.2 Experiments

##### Train-Test Split

The main challenge in Event Property classification with the CrudeOilNews
corpus is class imbalance. To address this, we modified the k-fold cross-
validation from random sampling to stratified sampling. This is done to ensure
that both the training and testing set in each cross-validation maintain the
same class distribution (label ratio) of the original dataset as shown in
Figure 10.

##### Results

All three sub-tasks (Polarity, Modality, and Intensity classification) are
standalone and independent tasks where the outcome of one does not influence
the outcome of others. Therefore all three classification models were trained
independently of each other. Event Modality and Polarity classification is a
binary classification task, the labels for Modality are: ASSERTED and OTHER,
and for Polarity are: POSITIVE and NEGATIVE. Both classification tasks are
trained on Binary Cross-entropy Loss. As for Event Intensity, it is a multi-
class classification task, the labels are: NEUTRAL, EASED, and INTENSIFIED. It
is trained on multi-class Cross-entropy Loss. We run experiments to determine
the most suitable text span for the classification of Event Property
Classification by the text processing methods listed in Table 6.

Table 6: Experiment results of different Input Text Span. Text Span Generation Methods | Polarity | Modality | Intensity
---|---|---|---
F1 | MCC | F1 | MCC | F1 | MCC
$3$-grams fixed window centered around event trigger | 0.685 | 0.285 | 0.699 | 0.305 | 0.701 | 0.320
Dependency parse tree Lee et al. [2021] | 0.759 | 0.305 | 0.723 | 0.298 | 0.699 | 0.298
SelfAttentiveSpanExtractor Jiang & de Marneffe [2021] | 0.892 | 0.385 | 0.851 | 0.462 | 0.751 | 0.665

##### Analysis

Based on the results in Table 6, we conclude that the best text span is the
ones generated by SelfAttentiveSpanExtractor, then followed by using a
dependency parse tree. Dependency parse tree utilizes the syntactic structures
of the input sentence and work well for identifying modifiers and negations
such as WILL and NOT that are linked to the event trigger’s sub-parse tree.
However, it does not work for cases where event trigger is not a verb that
forms its sub-tree. We illustrate this with an example sentence: “The Trump
administration will not consider reimposing sanctions on the OPEC member
nation.” and a portion of the dependency tree in Figure 7.

Figure 7: An example of a pruned dependency parse tree (enclosed in dotted
lines) that did not generate a good text input for modality and polarity
classification. Based on the event trigger word - sanctions, this pruned
dependency parse tree does not contain the modality and polarity cue words:
will and not.

The worst performing text span is using neighboring words centering around
event trigger because it does not capture words that are far away from the
trigger (i.e.: words are located outside of the $3$-grams window). Based on
the example above, the word cue word ‘will’ is not extracted as part of the
text span for sanctions because it is located outside of the fixed window
centered around the event trigger. In the section that follows, cross-domain
sequential transfer learning is done using selfAttentiveSpanExtractor
approach, the input text span that produces the best event property
classification results.

The F1-score for both Polarity and Modality classification is high, while
their respective MCC scores are much lower. This situation is caused by class
imbalance in these two classifications, as shown in Figure 10. Error analysis
on these two tasks showed the errors are primarily False Positives. The models
tend to classify everything to the majority class (POSITIVE for Polarity
classification and ASSERTED for Modality classification), which result in low
precision. Given that MCC score takes into account all four values in the
confusion matrix, a low MCC score shows that the models are not good at
classifying the minority classes.

The most challenging among the three tasks is Event Intensity classification.
Some of the cue words for determining the event intensity (NEUTRAL, EASED,
INTENSIFIED) are themselves trigger words. For example, Oversupply could rise
next year when Iraq starts to export more oil. The correct interpretation
should be: The event oversupply might be further INTENSIFIED (cue word: rise,
but this word is also a trigger word for the event - MOVEMENT-UP-GAIN.

###  5.3 Cross-Domain Sequential Transfer Learning

Here we investigate the usage of other available corpora in all domains for
the purpose of cross-domain STL. The idea is to use resources from different
source domains to train a model on a task before fine-tuning the model to
adapt to a new domain on the same task. We carry out this STL by first
training the corpora from the source domain on Event Polarity / Modality
classification and then transfer the model to fine-tune on the same task on
the target domain, i.e., CrudeOilNews corpus. Figure 8 shows a graphical
depiction of the idea of STL. In the top section, a labeled dataset in the
source domain is used to train a model on event polarity or modality
classification. The model is then trained using the dataset from the target
domain and fine-tuned on the same task, as shown in the bottom section.

Figure 8: Sequential Transfer Learning (STL): The model is first trained on
labeled dataset in the source domain (see list of corpora listed in Table 7)
before being transferred to train on the Target Domain (CrudeOilNews). STL is
done for Polarity and Modality Classification. On the other hand, intensity
classification is trained from scratch with just the CrudeOilNews corpus due
to the lack of resources.

Before going into the implementation details of cross-domain STL, it is
important we align the task definitions as they may appear as different names:

  1. 1.

Event Polarity Classification999Not to be confused with Sentiment
Polarity(Positive / Negative sentiment) can be aligned to Negation Detection,
such as in SEM 2012 Shared Task: Negation Detection and Scope Resolution.

  2. 2.

Event Modality Classification can be aligned to Hedge / Uncertainty /
Speculation Detection such as in CoNLL 2010 shared task: Hedge detection and
scope resolution.

In the subsections that follow, we lay out the available resources for both
negation detection and uncertainty detection. Resources for Event Factuality
Prediction (EFP) is excluded here because EFP combines both negation and
speculation detection to determine the ‘factuality’ of an event. Instead, we
look to corpora that have Negation and Uncertainty annotated individually to
match how event polarity and modality are annotated in CrudeOilNews corpus.

##### Corpora for Negation Detection

  1. 1.

In SEM2012 Shared Task Morante & Blanco [2012], two corpora in the general
text were released for negation scope and focus detection; they are the Conan
Doyle stories and the Penn TreeBank corpus;

  2. 2.

In the survey paper Jiménez-Zafra et al. [2020], the authors listed out all
English and Spanish corpora annotated with negation (negation cue and its
respective scope). According to the list, majority of the available corpora
are in the following domains:

    1. (a)

Bio-related text domain: BioInfer, Genia Event, BioScope, and DrugDDI;

    2. (b)

Consumer reviews: product review Corpus, SFU Review, Movie review;

    3. (c)

General Domain: Prop Bank and SFU Opinion & Comments (SOCC)

##### Corpora for Uncertainty Detection

  1. 1.

The CoNLL2010 share task Farkas et al. [2010] is made up of a collection of
corpora include Biology-related publications and general domain factual text
from Wikipedia;

  2. 2.

Financial domain, Theil et al. [2018] introduced the 10-k financial
disclosures corpus for the task of classifying financial statements whether
they are certain or uncertain.

  3. 3.

Consumer reviews : SFU Reviews corpus Konstantinova et al. [2012] contains
both uncertainty and negation cue words and scope annotated.

Table 7: The list of open source corpora with Negation and Uncertainty Annotation. Dataset |  Domain | Negation | Uncertainty
---|---|---|---
Cue |  Scope |  Cue |  Scope
1\. ConanDoyle(neg) Morante & Blanco [2012] |  Fiction |  ✓ |  ✓ |  |
2\. SFU OCC(neg) Kolhatkar et al. [2020] |  Opinion News & Comments |  ✓ |  ✓ |  |
3\. 10kFinStatement(unc) Theil et al. [2018] |  Corporate Financial Disclosure |  |  | only class labels
4\. Wikipedia(unc) Farkas et al. [2010] |  General |  |  |  ✓ |
5\. Reviews(neg & unc) Konstantinova et al. [2012] |  Product Reviews |  ✓ |  ✓ |  ✓ |  ✓
6\. SENTiVENT Jacobs & Hoste [2021] |  Economic news | only class labels | only class labels

####  5.3.1 Domain Similarity

In transfer learning, it is observed that the more related the tasks, the
easier it is for transfer or cross-utilize the knowledge Sanh et al. [2019].
The same holds true for data where the more related the source domain to the
target domain, the easier it is for effective transfer learning Meftah et al.
[2021], Gururangan et al. [2020]. Based on this, we excluded Bio-medical-
related corpora from our experiments because the Bio-medical domain has its
specific vocabulary deemed different from the Finance and Economics domain and
is an ill-fit with ComBERT embeddings101010see Section 3.1 for more details
about ComBERT. Five corpora was selected and are listed in Table 7; the
details of each of these corpora are found in A.

Figure 9: Vocabulary overlap (%) between source datasets and CrudeOilNews
target dataset. Vocabularies of each domain are made up of the top 10K most
frequent words (excluding stopwords) in each corpus.

We evaluate domain similarity between the source datasets and CrudeOilNews by
obtaining the percentage of vocabulary overlap of the two as shown in Figure
9. On a continuum of the proximity between source datasets and target dataset,
the source datasets can be ranked as SENTiVENT $\rightarrow$ 10kFinStatement
$\rightarrow$ Wikipedia-CONLL2010 $\rightarrow$ ACE2005 $\rightarrow$ SOCC
$\rightarrow$ Reviews $\rightarrow$ ConanDoyle.

####  5.3.2 Task Modification

The shared task of CoNLL2010 (for Uncertainty Detection) and SEM2010 (for
Negation Detection) consist of two sub-tasks: (1) it involves first detecting
the cue words at the sentence level and then (2) resolving the scope based on
the cue words detected. Event property classification, on the other hand, is
slightly different where the main aim is to detect the event first and then
determine the event properties based on the event and its scope. Due to the
difference in the original source tasks and how the source datasets listed in
Table 7 are annotated, we modify the original tasks and adapted them to our
task of event property classification by making the following modifications:

  1. 1.

Simplify the shared task to just one task. The original Negation / Uncertainy
detection involves two sub-tasks : (1) cue word detection and (2) scope
resolution. This is simplified into a single sequence classification task.

  2. 2.

Align class labels:

    1. (a)

Event Polarity: For sentences that contain Negation cue words, we assign the
label NEGATIVE for the whole sentence. For sentences without, we assign the
label POSITIVE instead.

    2. (b)

Event Modality: Similar to polarity classification, sentences with Uncertainty
cue words or has the labeled ‘uncertain’ are labeled with OTHER. For sentences
without, we assign the label ASSERTED.

###  5.4 Experiments

First, we train Event Polarity / Modality using the corpora listed in Table 7;
these corpora are known as the “source domain” $D_{s}$. Then we transfer the
model to train on the same task on the “target corpus” $D_{t}$, the
CrudeOilNews corpus. In our experiments, we use the best model trained on the
source dataset for STL on CrudeOilNews. In other words, we picked the model at
the epoch with the highest performance on the source validation set.

##### Results

Table 8: The results of Polarity and Modality classification of CrudeOilNews
using various source datasets (identified in Table 7) as source domain $D_{s}$
in Sequential transfer learning. Event Polarity Classification
---
Source $D_{s}$ |  Target $D_{t}$ | F1 | MCC
1a. - | CrudeOilNews | 0.892 | 0.385
1b. ConanDoyle | CrudeOilNews |  0.699($\downarrow$) |  0.305($\downarrow$)
1c. OCC | CrudeOilNews |  0.893($\uparrow$) |  0.498($\uparrow$)
1d. Reviews | CrudeOilNews |  0.713($\downarrow$) |  0.412($\uparrow$)
1e. SENTiVENT | CrudeOilNews |  0.895($\uparrow$) |  0.678($\uparrow$)
Event Modality Classification
Source $D_{s}$ |  Target $D_{t}$ | F1 | MCC
2a. - | CrudeOilNews | 0.851 | 0.462
2b. 10kFinStatement | CrudeOilNews |  0.879$\uparrow$ |  0.695$\uparrow$
2c. Wikipedia | CrudeOilNews |  0.841($\downarrow$) |  0.501($\uparrow$)
2d. Reviews | CrudeOilNews |  0.723($\downarrow$) |  0.395($\downarrow$)
2e. SENTiVENT | CrudeOilNews |  0.883($\uparrow$) |  0.715($\uparrow$)

Event Intensity classification is excluded from cross-domain STL because to
the best of our knowledge, there is no available labeled dataset annotated for
event intensity classification.

##### Analysis

We analyze the results shown in Table 8 by the list of event properties below:

  1. 1.

Event Polarity: there is some form of improvement when the model is trained on
a source domain first before fine-tuning the model on the target domain on the
same task. The best “source domain” corpus for Event Polarity is SENTiVENT,
while models trained on ConanDoyle and Reviews performed worst than the
baseline model. The main reason is that these corpora are somewhat dissimilar
to CrudeOilNews corpus that resulted in Negative Transfer. Performance
deterioration is especially apparent in ConanDoyle because it is a corpus made
up of dialogues or conversations and has negation cues mainly in a
conversational form such as don’t, doesn’t, didn’t, isn’t, can’t, wasn’t that
are not found in the target corpus.

  2. 2.

Event Modality: Due to the similarity between the CrudeOilNews and the two
finance-related corpora : SENTiVENT and 10KFinStatement, the resulting cross-
domain STL models are able to provide a significant boost to model
classification performance. Similar to the ConanDoyle corpus, the Reviews
contains conversational-like sentences that have minimal overlap with
CrudeOilNews in terms of uncertainty cue words that resulted in model
performance worse than the baseline.

It is worth highlighting that by comparing between F1-score and MCC-score,
MCC-score has a more significant jump in improvement. As highlighted in
Section 5.2, results of baseline models show a lower MCC-score because of
class imbalance; the models tend to classify everything to the majority class
leading to a high number of False Positives. Upon training the model with
source datasets that do not have a serious class imbalance issue, the final
models have higher MCC-scores. Based on error analysis, it is shown that the
final models have higher True Negatives (higher prediction on minority class)
and thus lead to better MCC-score.

It can be concluded from the results above that the more similar the source
domain is to the target domain, the easier we can use cross-domain STL to
improve the final classifier’s performance. It is also observed in Ruder
[2019] that the more distant two domains are, the harder it is to adapt from
one to the other. Apart from improving the models’ F1 score, it is also
observed that the models’ MCC improves as well. An improved MCC score means
classification performance improves across all classes, including the minority
class, therefore addressing the issue of class imbalance.

##  6 Conclusion & Future Work

Table 9: Final results |  Type | Accuracy | Precision | Recall | F1 | MCC
---|---|---|---|---|---|---
EE |  1\. Entity Mention Detection (EMD) + Event Detection (ED) | 0.977 | 0.933 | 0.949 | 0.941 | -
2\. Argument Role Prediction (ARP) | 0.914 | 0.913 | 0.914 | 0.913 | 0.840
Prop. |  1\. Event Polarity | 0.821 | 0.697 | 0.917 | 0.792 | 0.717
2\. Event Modality | 0.852 | 0.803 | 0.842 | 0.822 | 0.771
3\. Event Intensity | 0.702 | 0.765 | 0.664 | 0.751 | 0.665

It is known that training models via the traditional approach of supervised
learning requires a substantial amount of annotated data. This challenge
becomes even more apparent for a lower-resource domain such as Finance and
Economics. To produce accurate event extraction models from the CrudeOilNews
corpus, we leveraged the effectiveness of transfer learning to build event
extraction and event classification models with the best possible performance
despite limited resources.

In the transfer learning setting, we are able to utilize source task $T_{s}$
or source domain $D_{s}$ to improve model embeddings and representations in
order to produce better model performance in target task $T_{t}$ or target
domain $D_{t}$. Based on the experiment results, we come to the following
conclusions:

  1. 1.

Domain Adaptive Pre-Training: Further pre-training BERT on in-domain data to
produce ComBERT produced better contextualized embedding that improved model
performance;

  2. 2.

Combination of Multi-Task Learning and Sequential Transfer Learning: After
experimenting with various permutations of MTL and STL sub-tasks
configurations, the best configuration for event extraction in CrudeOilNews is
EMD + ED $\rightarrow$ ARP, where EMD + ED are trained via MTL before
sequentially transferring the resulting model to train on ARP.

  3. 3.

Cross-domain Sequential Transfer Learning: Source datasets annotated for the
same tasks can be used to boost the performance of event property
classification in CrudeOilNews corpus despite the severe class imbalance.
However, it is worth noting that selecting a source domain with high source-
target similarity is vital to have the best model performance and to avoid the
undesirable result of negative transfer.

The final model performance for each sub-task is tabulated and shown again in
Table 9 for easier reference. Even though we have managed to use transfer
learning to achieve better model performance, one of the major weaknesses of
the framework is that the current event ontologies are exhaustive and will not
cater to new events. As part of future work, we are interested in
investigating ways to increase event coverage for crude oil news. One
promising direction is to use zero-shot learning like in Huang et al. [2018].
Another promising direction is to investigate domain adaptation of event
extraction from crude oil news to a similar domain, such as in gold-related
news or even FoRex (Foreign Exchange) news.

##  Appendix A Target Dataset: CrudeOilNews Corpus

  * 1.

Tasks: Event Extraction, Event Modality, Polarity and Intensity
Classification.

  * 2.

Domain: Commodity news surrounding Crude Oil.

  * 3.

Size: 425 documents, 7,059 sentences, 10,578 events, and 22,267 event
arguments.

  * 4.

Event Types: Movement-down-loss, movement-up-gain, movement-flat, cause-
movement-down-loss, caused-movement-up-gain, position-high, position-low,
slow-weak, grow-strong, prohibiting, oversupply, shortage, civil-unrest,
embargo, geo-political tension, crisis and negative-sentiment.

Here is a list of characteristics exhibited by this dataset and each pose a
unique challenge to the overall event extraction task:

  1. 1.

Limited amount of labeled data;

  2. 2.

Class imbalance / topic bias: serious class imbalance in event Properties
distribution as shown in Figure 10 where the majority class outnumbers the
minority classes by a large margin;

  3. 3.

Homogenous entity types but playing different argument roles; One or two
examples here will be good.

  4. 4.

Number intensity: Numbers (e.g., price, difference, percentage of change) and
dates (including date of the opening price, dates of closing price) are
abundant in commodity news.

Figure 10: Event Polarity, Modality and Intensity Distribution

###  A.1 Event Properties Examples

  1. 1.

Event Polarity

     * (a)

POSITIVE: OPEC members agreed to cut oil supplies.

     * (b)

NEGATIVE: OPEC countries refused to cut oil suppliers.

  2. 2.

Event Modality:

     * (a)

ASSERTED: Saudi Arabia continues to cut its production.

     * (b)

OTHER: Analysts were anticipating oil inventories to fall by 800,000.

  3. 3.

Event Intensity:

     * (a)

NEUTRAL: Oil rise third day in a row to all time high.

     * (b)

EASED: Libya ’s civil strife has been eased by potential peace talks.

     * (c)

INTENSIFIED: …could hit Iraq ’s output and deepen a supply shortfall.

##  Appendix B Source Datasets

  1. 1.

ACE2005

     * (a)

Tasks: Event Extraction, Event Modality and Polarity classification.

     * (b)

Domain: General, corpus made up of conversations, broadcast news, newsgroups,
weblogs.

     * (c)

Size: 587 documents, 5,055 events, 18,927 sentences, 6,040 event arguments,
34,474 entities.

     * (d)

Event Types: Life, Movement, Transaction, Business, Conflict, Contact,
Personnel, Justice

  2. 2.

SENTiVENT Jacobs & Hoste [2021]

     * (a)

Tasks: Event Extraction, Event Modality and Polarity Classification.

     * (b)

Domain: Business / Finance News

     * (c)

Size: 288 documents, 6,203 events, 6,883 sentences, 13,675 arguments.

     * (d)

Event Types: CSR / Brand, Deal, Dividend, Employment, Expense, Facility,
Financial Report, Financing, Investing, Legal, Macroeconomics, Merger /
Acquisition, Product / Service, Profit / Loss, Rating, Revenue, Sales Volume,
Security Value

  3. 3.

ConanDoyle Morante & Blanco [2012]

     * (a)

Tasks: Negation detection

     * (b)

Domain: Fiction / stories

     * (c)

Size: 3640 sentences from The Hound of the Baskervilles story, out of which
850 contain negations, and 783 sentences from The Adventure of Wisteria Lodge
story, out of which 145 contain negations.

     * (d)

Description: a corpus of Conan Doyle stories annotated with negation cues and
their scopes, as well as the event or property that is negated. Negation cues
are made up of the following types: lexical, syntactic, and morphological.

  4. 4.

SOCC(neg) Kolhatkar et al. [2020]

     * (a)

Tasks: Negation detection

     * (b)

Domain: opinion, review and comments

     * (c)

Size: 10399 opinion articles (editorials, columns and op-eds); 663,173
comments from 303,665 comment threads, from the main Canadian daily newspaper
(from January 2012 to December 2016)

     * (d)

Description: The corpus is organized into three subcorpora: the articles
corpus, the comments corpus, and the comment-threads corpus. Only the articles
sub-corpora is used for this work.

  5. 5.

10kFinStatement(unc) Theil et al. [2018]

     * (a)

Tasks: Uncertainty detection

     * (b)

Domain: Finance - 10-k reports

     * (c)

Size: 1000 sentences from 10-Ks111111A 10-K is a comprehensive report filed
annually by public companies about their financial performance.

     * (d)

Description: each labeled with certain and uncertain but does not have
uncertainty cue words nor scope annotated.

  6. 6.

Wikipedia-CoNLL2010(unc)

     * (a)

Tasks: dataset is provided as part of CoNLL2010 shared task - Learning to
Detect Hedges and their Scope in Natural Language Text.

     * (b)

Domain: General - Wikipedia articles

     * (c)

Size: 2,186 paragraphs collected from Wikipedia archives were also offered as
Task1 training data (11,111 sentences containing 2,484 uncertain ones). The
evaluation dataset contained 2,346 Wikipedia paragraphs with 9,634 sentences,
out of which 2234 were uncertain.

     * (d)

Description:

  7. 7.

Reviews(neg & unc) Konstantinova et al. [2012]

     * (a)

Tasks: Negation and Uncertainty detection

     * (b)

Domain: Reviews of movie, book and consumer product, taken from
www.Epinions.com.

     * (c)

Size: 400 documents

##  Appendix C Additional results

Detail breakdown of Event Extraction results by their respective classes are
presented here. Entity Mention Detection (EMD) + Event Detection (ED) results
are presented in Table 10 while Argument Role Prediction (ARP) results are
listed in Table 11.

Table 10: Entity Mention + Event Trigger Extraction |  Type | P | R | F1
---|---|---|---|---
Entity Mention |  1\. COMMODITY | 0.98 | 0.98 | 0.98
2\. COUNTRY | 0.97 | 0.98 | 0.98
3\. DATE | 0.94 | 0.96 | 0.95
4\. DURATION | 0.87 | 0.90 | 0.89
5\. ECONOMIC_ITEM | 0.89 | 0.95 | 0.92
6\. FINANCIAL_ATTRIBUTE | 0.98 | 0.99 | 0.99
7\. FORECAST_TARGET | 0.92 | 0.95 | 0.94
8\. GROUP | 0.56 | 0.50 | 0.53
9\. LOCATION | 0.90 | 0.91 | 0.90
10\. MONEY | 0.92 | 0.94 | 0.93
11\. NATIONALITY | 0.96 | 0.92 | 0.94
12\. NUMBER | 0.97 | 0.88 | 0.92
13\. ORGANIZATION | 0.94 | 0.96 | 0.95
14\. OTHER_ACTIVITY | 0.33 | 0.50 | 0.40
15\. PERCENT | 0.96 | 0.95 | 0.95
16\. PERSON | 0.98 | 0.97 | 0.98
17\. PHENOMENON | 0.0 | 0.0 | 0.0
18\. PRICE_UNIT | 0.96 | 0.99 | 0.97
19\. PRODUCTION_UNIT | 0.90 | 0.95 | 0.92
20\. QUANTITY | 0.80 | 0.92 | 0.86
21\. STATE_OR_PROVINCE | 0.77 | 0.85 | 0.80
Event Trigger |  1\. CAUSE_MOVEMENT_DOWN_LOSS | 0.92 | 0.93 | 0.92
2\. CAUSE_MOVEMENT_UP_GAIN | 0.87 | 0.89 | 0.88
3\. CIVIL_UNREST | 1.00 | 0.92 | 0.96
4\. CRISIS | 1.00 | 1.00 | 1.00
5\. EMBARGO | 0.97 | 1.00 | 0.98
6\. GEOPOLITICAL_TENSION | 0.75 | 0.88 | 0.81
7\. GROW_STRONG | 0.79 | 0.84 | 0.81
8\. MOVEMENT_DOWN_LOSS | xx | xx | xx
9\. MOVEMENT_FLAT | 1.00 | 0.86 | 0.92
10\. MOVEMENT_UP_GAIN | xx | xx | xx
11\. NEGATIVE_SENTIMENT | 0.93 | 0.94 | 0.93
12\. OVERSUPPLY | 0.80 | 0.83 | 0.82
13\. POSITION_HIGH | 0.91 | 1.00 | 0.96
14\. POSITION_LOW | 0.91 | 0.97 | 0.94
15\. PROHIBITING | 0.83 | 0.83 | 0.83
16\. SHORTAGE | 0.91 | 1.00 | 0.95
17\. SLOW_WEAK | 0.93 | 0.73 | 0.82
18\. TRADE_TENSIONS | 0.75 | 0.88 | 0.81
Table 11: Argument Role Prediction Type | P | R | F1
---|---|---|---
1\. NONE | 0.94 | 0.95 | 0.95
2\. ATTRIBUTE | 0.92 | 0.93 | 0.93
3\. ITEM | 0.88 | 0.84 | 0.86
4\. FINAL_VALUE | 0.85 | 0.78 | 0.82
5\. INITIAL_VALUE | 0.50 | 0.30 | 0.37
6\. DIFFERENCE | 0.89 | 0.91 | 0.90
7\. REFERENCE_POINT | 0.79 | 0.77 | 0.78
8\. INITIAL_REFERENCE_POINT | 0.43 | 0.53 | 0.47
9\. CONTRACT_DATE | 1.00 | 0.94 | 0.97
10\. DURATION | 0.83 | 0.87 | 0.85
11\. TYPE | 0.79 | 0.74 | 0.76
12\. IMPOSER | 0.84 | 1.00 | 0.91
13\. IMPOSEE | 0.84 | 0.84 | 0.84
14\. PLACE | 0.81 | 0.77 | 0.79
15\. SUPPLIER_CONSUMER | 0.84 | 0.82 | 0.83
16\. IMPACTED_COUNTRIES | 0.93 | 0.87 | 0.90
17\. PARTICIPATING_COUNTRIES | 0.86 | 0.92 | 0.89
18\. FORECASTER | 0.80 | 0.63 | 0.71
19\. FORECAST | 0.76 | 0.59 | 0.67
20\. SITUATION | 0.67 | 0.50 | 0.57

## References

  * Alyafeai et al. [2020] Alyafeai, Z., AlShaibani, M. S., & Ahmad, I. (2020).  A survey on transfer learning in natural language processing.  arXiv preprint arXiv:2007.04239, .
  * Beltagy et al. [2019] Beltagy, I., Lo, K., & Cohan, A. (2019).  Scibert: A pretrained language model for scientific text.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3615–3620).
  * Blitzer et al. [2007] Blitzer, J., Dredze, M., & Pereira, F. (2007).  Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  In Proceedings of the 45th annual meeting of the association of computational linguistics (pp. 440–447).
  * Chen [2021] Chen, Y. (2021).  A transfer learning model with multi-source domains for biomedical event trigger extraction.  BMC Genemics 22, .
  * Chen et al. [2015] Chen, Y., Xu, L., Liu, K., Zeng, D., & Zhao, J. (2015).  Event extraction via dynamic multi-pooling convolutional neural networks.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 167–176).
  * Devlin et al. [2019] Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171–4186).  Minneapolis, Minnesota: Association for Computational Linguistics.  URL: https://www.aclweb.org/anthology/N19-1423. doi:10.18653/v1/N19-1423.
  * Farkas et al. [2010] Farkas, R., Vincze, V., Móra, G., Csirik, J., & Szarvas, G. (2010).  The conll-2010 shared task: learning to detect hedges and their scope in natural language text.  In Proceedings of the fourteenth conference on computational natural language learning–Shared task (pp. 1–12).
  * Gardner et al. [2018] Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018).  Allennlp: A deep semantic natural language processing platform.  In Proceedings of Workshop for NLP Open Source Software (NLP-OSS) (pp. 1–6).
  * Gui et al. [2018a] Gui, L., Xu, R., Lu, Q., Du, J., & Zhou, Y. (2018a).  Negative transfer detection in transductive transfer learning.  International Journal of Machine Learning and Cybernetics, 9, 185–197.
  * Gui et al. [2018b] Gui, T., Zhang, Q., Gong, J., Peng, M., Liang, D., Ding, K., & Huang, X.-J. (2018b).  Transferring from formal newswire domain with hypernet for twitter pos tagging.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 2540–2549).
  * Gururangan et al. [2020] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020).  Don’t stop pretraining: Adapt language models to domains and tasks.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8342–8360).
  * Hogenboom et al. [2016] Hogenboom, F., Frasincar, F., Kaymak, U., De Jong, F., & Caron, E. (2016).  A survey of event extraction methods from text for decision support systems.  Decision Support Systems, 85, 12–22.
  * Huang et al. [2018] Huang, L., Ji, H., Cho, K., Dagan, I., Riedel, S., & Voss, C. (2018).  Zero-shot transfer learning for event extraction.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  * Jacobs & Hoste [2021] Jacobs, G., & Hoste, V. (2021).  Sentivent: enabling supervised information extraction of company-specific events in economic and financial news.  Language Resources and Evaluation, (pp. 1–33).
  * Jiang & de Marneffe [2021] Jiang, N., & de Marneffe, M.-C. (2021).  He thinks he knows better than the doctors: Bert for event factuality fails on pragmatics.  Transactions of the Association for Computational Linguistics, 9, 1081–1097.
  * Jiménez-Zafra et al. [2020] Jiménez-Zafra, S. M., Morante, R., Teresa Martín-Valdivia, M., & Ureña-López, L. A. (2020).  Corpora annotated with negation: An overview.  Computational Linguistics, 46, 1–52.
  * Judea & Strube [2016] Judea, A., & Strube, M. (2016).  Incremental global event extraction.  In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.
  * Kolhatkar et al. [2020] Kolhatkar, V., Wu, H., Cavasso, L., Francis, E., Shukla, K., & Taboada, M. (2020).  The sfu opinion and comments corpus: A corpus for the analysis of online news comments.  Corpus Pragmatics, 4, 155–190.
  * Konstantinova et al. [2012] Konstantinova, N., De Sousa, S. C., Díaz, N. P. C., López, M. J. M., Taboada, M., & Mitkov, R. (2012).  A review corpus annotated for negation, speculation and their scope.  In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12) (pp. 3190–3195).
  * Konyushkova et al. [2017] Konyushkova, K., Sznitman, R., & Fua, P. (2017).  Learning active learning from data.  Advances in neural information processing systems, 30.
  * Lee et al. [2020] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020).  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36, 1234–1240.
  * Lee et al. [2021] Lee, M., Soon, L.-K., & Siew, E.-G. (2021).  Effective use of graph convolution network and contextual sub-tree for commodity news event extraction.  In Proceedings of the Third Workshop on Economics and Natural Language Processing (pp. 69–81).
  * Lee et al. [2022] Lee, M., Soon, L.-K., Siew, E.-G., & Sugianto, L. F. (2022).  Crudeoilnews: An annotated crude oil news corpus for event extraction.  arXiv preprint arXiv:2204.03871, .
  * Li et al. [2014] Li, Q., Ji, H., Hong, Y., & Li, S. (2014).  Constructing information networks using one single model.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  * Li et al. [2013] Li, Q., Ji, H., & Huang, L. (2013).  Joint event extraction via structured prediction with global features.  In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 73–82).
  * Liu et al. [2018] Liu, X., Luo, Z., & Huang, H. (2018).  Jointly multiple events extraction via attention-based graph information aggregation.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 1247–1256).  Brussels, Belgium: Association for Computational Linguistics.  URL: https://www.aclweb.org/anthology/D18-1156. doi:10.18653/v1/D18-1156.
  * Lyu et al. [2021] Lyu, Q., Zhang, H., Sulem, E., & Roth, D. (2021).  Zero-shot event extraction via transfer learning: Challenges and insights.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (pp. 322–332).
  * Manning et al. [2014] Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S., & McClosky, D. (2014).  The stanford corenlp natural language processing toolkit.  In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations (pp. 55–60).
  * Marasović & Frank [2016] Marasović, A., & Frank, A. (2016).  Multilingual modal sense classification using a convolutional neural network.  In Proceedings of the 1st Workshop on Representation Learning for NLP (pp. 111–120).
  * Meftah & Semmar [2018] Meftah, S., & Semmar, N. (2018).  A neural network model for part-of-speech tagging of social media texts.  In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  * Meftah et al. [2021] Meftah, S., Semmar, N., Tamaazousti, Y., Essafi, H., & Sadat, F. (2021).  On the hidden negative transfer in sequential transfer learning for domain adaptation from news to tweets.  In Proceedings of the Second Workshop on Domain Adaptation for NLP (pp. 140–145).
  * Mitamura et al. [2015] Mitamura, T., Yamakawa, Y., Holm, S., Song, Z., Bies, A., Kulick, S., & Strassel, S. (2015).  Event nugget annotation: Processes and issues.  In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation (pp. 66–76).
  * Morante & Blanco [2012] Morante, R., & Blanco, E. (2012).  * sem 2012 shared task: Resolving the scope and focus of negation.  In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012) (pp. 265–274).
  * Nguyen et al. [2016] Nguyen, T. H., Cho, K., & Grishman, R. (2016).  Joint event extraction via recurrent neural networks.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 300–309).
  * Nguyen & Nguyen [2019] Nguyen, T. M., & Nguyen, T. H. (2019).  One for all: Neural joint modeling of entities and events.  In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 6851–6858).  volume 33.
  * Pan & Yang [2010] Pan, S. J., & Yang, Q. (2010).  A survey on transfer learning.  IEEE Transactions on Knowledge and Data Engineering, 22, 1345–1359.
  * Rosenstein et al. [2005] Rosenstein, M. T., Marx, Z., Kaelbling, L. P., & Dietterich, T. G. (2005).  To transfer or not to transfer.  In In NIPS’05 Workshop, Inductive Transfer: 10 Years Later.
  * Ruder [2019] Ruder, S. (2019).  Neural transfer learning for natural language processing.  Ph.D. thesis NUI Galway.
  * Ruder et al. [2017] Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. (2017).  Sluice networks: Learning what to share between loosely related tasks.  arXiv preprint arXiv:1705.08142, 2.
  * Sanh et al. [2019] Sanh, V., Wolf, T., & Ruder, S. (2019).  A hierarchical multi-task approach for learning embeddings from semantic tasks.  In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 6949–6956).  volume 33.
  * Saurí & Pustejovsky [2009] Saurí, R., & Pustejovsky, J. (2009).  Factbank: a corpus annotated with event factuality.  Language resources and evaluation, 43, 227–268.
  * Sha et al. [2018] Sha, L., Qian, F., Chang, B., & Sui, Z. (2018).  Jointly extracting event triggers and arguments by dependency-bridge rnn and tensor-based argument interaction.  Proceedings of the AAAI Conference on Artificial Intelligence, 32.
  * Søgaard & Goldberg [2016] Søgaard, A., & Goldberg, Y. (2016).  Deep multi-task learning with low level tasks supervised at lower layers.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 231–235).
  * Theil et al. [2018] Theil, C. K., Štajner, S., & Stuckenschmidt, H. (2018).  Word embeddings-based uncertainty detection in financial disclosures.  In Proceedings of the First Workshop on Economics and Natural Language Processing (pp. 32–37).
  * Torrey & Shavlik [2010] Torrey, L., & Shavlik, J. (2010).  Transfer learning.  In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques (pp. 242–264).  IGI global.
  * Wadden et al. [2019] Wadden, D., Wennberg, U., Luan, Y., & Hajishirzi, H. (2019).  Entity, relation, and event extraction with contextualized span representations.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  * Weiss et al. [2016] Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016).  A survey of transfer learning.  Journal of Big data, 3, 1–40.
  * Wu et al. [2016] Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., & Dean, J. (2016).  Google’s neural machine translation system: Bridging the gap between human and machine translation.  CoRR, abs/1609.08144. URL: http://arxiv.org/abs/1609.08144. arXiv:1609.08144.
  * Xiang & Wang [2019] Xiang, W., & Wang, B. (2019).  A survey of event extraction from text.  IEEE Access, 7, 173111--173137. doi:10.1109/ACCESS.2019.2956831.
  * Xie et al. [2013] Xie, B., Passonneau, R., Wu, L., & Creamer, G. G. (2013).  Semantic frames to predict stock price movement.  In Proceedings of the 51st annual meeting of the association for computational linguistics (pp. 873--883).
  * Yang & Mitchell [2016] Yang, B., & Mitchell, T. M. (2016).  Joint extraction of events and entities within a document context.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 289--299).  San Diego, California: Association for Computational Linguistics.  URL: https://aclanthology.org/N16-1033. doi:10.18653/v1/N16-1033.
  * Zhang et al. [2019] Zhang, J., Qin, Y., Zhang, Y., Liu, M., & Ji, D. (2019).  Extracting entities and events as a single task using a transition-based neural model.  In IJCAI (pp. 5422--5428).