# Multi-Scale Contrastive Co-Training for Event Temporal Relation Extraction

Hao-Ren Yao1, Luke Breitfeller2, Aakanksha Naik3, Chunxiao Zhou1, Carolyn
Rose2

###### Abstract

Extracting temporal relationships between pairs of events in texts is a
crucial yet challenging problem for natural language understanding. Depending
on the distance between the events, models must learn to differently balance
information from local and global contexts surrounding the event pair for
temporal relation prediction. Learning how to fuse this information has proved
challenging for transformer-based language models. Therefore, we present
MulCo: Multi-Scale Contrastive Co-Training, a technique for the better fusion
of local and global contextualized features. Our model uses a BERT-based
language model to encode local context and a Graph Neural Network (GNN) to
represent global document-level syntactic and temporal characteristics. Unlike
previous state-of-the-art methods, which use simple concatenation on multi-
view features or select optimal sentences using sophisticated reinforcement
learning approaches, our model co-trains GNN and BERT modules using a multi-
scale contrastive learning objective. The GNN and BERT modules learn a
synergistic parameterization by contrasting GNN multi-layer multi-hop
subgraphs (i.e., global context embeddings) and BERT outputs (i.e., local
context embeddings) through end-to-end back-propagation. We empirically
demonstrate that MulCo provides improved ability to fuse local and global
contexts encoded using BERT and GNN compared to the current state-of-the-art.
Our experimental results show that MulCo achieves new state-of-the-art results
on several temporal relation extraction datasets.

## Introduction

Extracting event temporal relations from the mixture of local and global event
pairs is challenging. Typically, event pairs are considered ”local” if their
event mentions are within 1-2 sentences in the original text, and considered
”global” if they are more than two sentences apart. This local vs global
distinction carries substantial weight in this task, as we find that local and
global event pairs are ordered based on different kinds of evidence. However,
local and global event pairs often co-mingle in realistic texts. In past work,
they have frequently been treated as though they can be ordered using the same
evidence, which forces complex conditions to be learned, where simpler ones
might be equally or more effective if a separation could be maintained. It is
known that BERT-based language models are successful at short-distance context
modeling for local event pairs (Ballesteros et al. 2020), and Graph Neural
Networks (GNNs) are suitable for capturing the temporal reasoning used to
order long-range global event pairs (Liu et al. 2021). The question is thus
how to fuse BERT and GNN together in such a way as to maintain their distinct
advantages for the cases where they are each most needed. In this paper, we
present a novel fusion approach and demonstrate that it enables a model to
more effectively order event pairs within a document including both local and
global even pairs, with a statistically significant improvement over the
previous state-of-the-art.

The most intuitive to fuse BERT and GNN outputs is simply to concatenate them.
Using this simple approach, TIMERS (Mathur et al. 2021) obtained the previous
SOTA performance in event temporal relation extraction. However, their
performance gains are mainly through BERT fine-tuning efforts, where such
improvement fails when removing the BERT part. Other fusion methods have been
proposed to fuse BERT and GNN and achieve promising results in many NLP tasks
(Xiao et al. 2021; Lu, Du, and Nie 2020; Lin et al. 2021; He, Wang, and Zhang
2020). Yet, in our experiments, we observe that BERT converges faster than a
GNN, especially on local evnet pairs. Once BERT has converged, the GNN halts
improving and thus remains in a convergence state. In this case, BERT can
easily dominate fusion training, which overfits in its over-reliance on BERT
and thus learns to disregard the potential contribution from the GNN. These
findings suggest that either GNN underperforms in learning to leverage long-
range evidence, which results in poor convergence, or joint training
neutralizes the GNN advantages in global contexts by overfitting to the BERT
local context encoding. We call such a phenomenon BERT overfitting.

Another limitation, a.k.a memory constraints, restricts BERT from fine-tuning
with the GNN training signal, as it would require full batch training on a
document-level graph. It now raises the necessity of developing an effective
fusion strategy to adopt both BERT and GNN modeling benefits. We surmise that
the ideal BERT and GNN fusion should be simple, efficient, and, most
importantly, robust to BERT overfitting. Therefore, we present MulCo: Multi-
Scale Contrastive Co-Training, a multi-modality training method to fuse BERT
and GNN that is robust to BERT overfitting. Particularly, MulCo overcomes such
difficulty in three ways: (i) more powerful local and global context encoding,
(ii) modality isolation via training signal and parameter separation, and
(iii) multi-scale contrastive learning. Theoretically, our contrastive
learning co-trains BERT and GNN without parameter sharing, which greatly
reduces interference.

We represent the whole document with one syntactic and one temporal graph,
representing sentence-word dependencies and time expressions as explicit cues
for the events. We propose event position-aware BERT (BERT encoder) and
subgraph-aware Relational Graph Attention Network (R-GAT encoder)111We use
BERT and R-GAT encoders for short. (Busbridge et al. 2019) to provide more
powerful learning capacity on the local and global context. To further
increase the learning capacity of R-GAT, we augment each node by their induced
k-hop syntactic and temporal subgraphs. Nodes in syntactic and temporal k-hop
subgraphs are cross-aligned with each other through Subgraph-Aware Cross Graph
Node Attention. We separate each encoder into projection and prediction heads
to split training signals from co-training and temporal relation extraction
and isolate BERT and R-GAT model parameters.

To co-train these two models, given an event pair, we contrast their global
context via BERT projection head to source and target event local context
through projection heads from all R-GAT layers in syntactic and temporal
graphs. Although model parameters in BERT and R-GAT are detached, their
components are mutually adjusted via end-to-end back-propagation through
multi-scale contrastive learning. Such contrastive learning on model feature
space has shown promising co-training efforts in visual representation
learning (Yuan et al. 2021). As such a training scheme, we can leverage simple
concatenation on BERT and R-GAT prediction heads to optimize their modeling
semantics via temporal relation prediction task. Thus, each modality preserves
its modeling capacity and coordinates its modeling semantics with one another
without interference. It ensures the co-training efficacy in the local and
global context provided via BERT and R-GAT. Experimental results on several
benchmark datasets demonstrate our newly achieved state-of-the-art
performance. We summarize our contributions as:

  * •

We present MulCo, a novel contrastive co-training method that obtains better
fusion on BERT and GNN.

  * •

We theoretically show contrasting two models achieves multi-modality training
without parameter sharing.

  * •

We empirically demonstrate MulCo provides robust co-training that prevents
BERT overfitting.

  * •

We achieve new SOTA performance on event temporal relation extraction
benchmarks.

Figure 1: Overview of our proposed MulCo framework.

## Related Work

### Event temporal relation extraction

Despite the success of recently proposed BERT-based temporal ordering models
(Ning, Subramanian, and Roth 2019; Han et al. 2019; Wang et al. 2020;
Ballesteros et al. 2020), they have mainly been tested on datasets largely
featuring local event pairs (e.g., TimeBank-Dense (Cassidy et al. 2014) and
MATRES (Ning, Wu, and Roth 2018)). When tested on datasets containing global
event pairs (e.g., TDDiscourse (Naik, Breitfeller, and Rose 2019)), they often
do not achieve comparable performance. Conversely, GNN-based models (Liu et
al. 2021) successfully handle global event pairs in isolation, but show
performance degradation on datasets containing both local and global event
pairs (e.g., TDDAuto, a subset of TDDiscourse). Man et al. (2022) try to
address this issue with an optimal sentence selection method that uses policy-
gradient reinforcement learning, and achieves current SOTA performance.
However, this method alternates between reinforcement learning and language
model fine-tuning, which is time-consuming, and requires additional effort to
design appropriate rewards. Instead, we focus on developing an effective way
to fuse BERT-based and GNN-based models, which can achieve strong performance
on both local and global pairs.

### Fusing Language Models and Graph Networks

Recent efforts have achieved performance gains on various NLP tasks by fusing
BERT and GNN models, typically via one of the following strategies: (i)
Augment GNN with BERT, which initializes GNN nodes with BERT pre-trained
embeddings (He, Wang, and Zhang 2020; Xiao et al. 2021), (ii) Augment BERT
with GNN, which incorporates GNN node features as input to BERT (Lu, Du, and
Nie 2020), and (iii) Joint training, which concatenates or interpolates BERT
embeddings and GNN node features (Mathur et al. 2021; Lin et al. 2021). But
all strategies suffer from the issue of BERT overfitting to the task earlier,
which hinders both models from leveraging their distinct advantages.

## Methodology

MulCo: Multi-Scale Contrastive Co-Training, our proposed method, consists of
three stages as shown in Figure 1. First, we construct BERT input and
syntactic and temporal graphs from an input document. Then, multi-scale
contrastive learning is used to co-train BERT and R-GAT encoders via their
projection heads. Finally, prediction heads from BERT and R-GAT are
concatenated for event temporal relation extraction. Thus, encoder outputs are
divided into projection and prediction heads, used for contrastive learning
and temporal relation prediction respectively. We argue that isolating co-
training and downstream prediction prevents BERT overfitting by reducing the
signal received from BERT fine-tuning, which often leads to faster
convergence/overfitting.

### Problem Definition

Let document $\mathcal{D}$ consist of $m$ sentences $D=[S_{1},\cdots,S_{m}]$,
where each sentence contains a sequence of words from $\mathcal{W}$ and $R$ be
the set of temporal relations. Document $D$ has several event pairs, with each
event pair $(e^{1}_{i},e^{2}_{i})$ having a relation label $r_{i}\in R$. The
event temporal relation extraction task is to predict $r_{i}$ for a given
event pair $(e^{1}_{i},e^{2}_{i})\in D$.

### BERT Encoder for Local Context

Many BERT-based temporal models simply encode all sentences between the events
in a pair, while ignoring their relative positions that may reflect key
temporal semantics. To capture this, we use an _event position-aware_
architecture (Ballesteros et al. 2020) and compute separate embeddings for
context sentences occurring _before_ , _after_ , and _between_ the events of
interest $(e^{1}_{i}$, and $e^{2}_{i})$. We employ a BERT-based pre-trained
language model to learn to encode local context for temporal relation
prediction.

#### BERT encoder input

For an event pair $(e^{1}_{i},e^{2}_{i})$ contained in sentences $S_{p}$ and
$S_{q}$, we extract a set of sentences to represent local context
$D^{bert}_{i}$ as follows:

$\displaystyle D^{bert}_{i}=\begin{cases}\\{S_{p-1},\cdots,S_{q+1}\\}&\text{if
}p<q\\\ \\{S_{q-1},\cdots,S_{p+1}\\}&\text{otherwise}\end{cases}$ (1)

$D^{bert}_{i}$ is divided into five contiguous subsequences to reflect
positions: (i) $D^{bert(1)}_{i}$, tokens before the first event, (ii)
$D^{bert(2)}_{i}$, tokens from first event span, (iii) $D^{bert(3)}_{i}$,
tokens between the two events, (iv) $D^{bert(4)}_{i}$, tokens from second
event span, and (v) $D^{bert(5)}_{i}$, tokens after the second event.

#### BERT Projection Head

To provide the BERT encoder with a channel to exchange semantic information
with R-GAT during co-training, we define a projection head $f_{\alpha}$. Input
to this projection head is a context encoding $h^{bert}_{i}$ for event pair
$(e^{1}_{i},e^{2}_{i})$ obtained by running BERT over $D^{bert}_{i}$:

$z^{bert}_{i}=f_{\alpha}(h^{bert}_{i})\in\mathcal{R}^{d_{p}}$ (2)

where $f_{\alpha}:\mathbb{R}^{d_{bert}}\mapsto\mathbb{R}^{d_{p}}$ denotes the
projection function that maps a BERT context encoding $h^{bert}_{i}$ into a
$d_{p}$-dimensional embedding space, which is co-trained with R-GAT.

#### BERT Prediction Head

The BERT encoder is additionally finetuned on temporal relation prediction via
a separate channel by running BERT over the tokens present in each
subsequence, followed by mean-pooling and concatenation to produce an event
position-aware local context encoding:

$\displaystyle c^{bert}_{i}$
$\displaystyle=\bigparallel^{5}_{r\prime=1}D^{bert(r\prime)}_{i}$ (3)
$\displaystyle o^{bert}_{i}$
$\displaystyle=[c^{bert}_{i}\mathbin{\|}h^{bert}_{i}]$ (4)

where $c^{bert}_{i}$ is event position-aware context, and $o^{bert}_{i}$ is
the prediction head used for temporal relation extraction.

### R-GAT Encoder for Global Context

TIMERS uses Relational Graph Convolutional Networks (R-GCN) to model global
context. But we observe that when trained jointly with BERT, R-GCN is unable
to contribute much due to lower learning capacity. Consequently, we propose to
use R-GAT, which has increased learning capacity due to relation attention. We
follow the same node feature initialization as in Mathur et al. (2021); Liu et
al. (2021), which frozen BERT embeddings.

#### Syntactic and Temporal Graphs

Following TIMERS, we represent document syntactic structure via a syntactic
graph
$\mathcal{G}_{sg}=(\mathcal{V}_{sg},{\mathcal{E}_{sg}},{\mathcal{R}_{sg}})$,
which contains four types (${\mathcal{R}_{sg}}$) of edges ${\mathcal{E}_{sg}}$
between document, sentence, and word nodes $\in\mathcal{V}_{sg}$: (i)
document-sentence, (ii) sentence-sentence, (iii) sentence-word, and (iv) word-
word. Temporal graphs
$\mathcal{G}_{tg}=(\mathcal{V}_{tg},{\mathcal{E}_{tg}},{\mathcal{R}_{tg}})$
are obtained using the Semantic Temporal Automated Grammar Extractor (STAGE)
proposed in Breitfeller, Naik, and Rose (2021), which automatically extracts
time expressions linked to events using a set of time cues, unlike TIMERS
which relies on human annotation. Time expressions are then transformed into
time nodes linked to event words $\mathcal{V}_{tg}$. As in TIMERS, edges
${\mathcal{E}_{tg}}$ between time and event nodes are assigned to be either
before, after, simultaneous, or none (for cases where the semantic temporal
information is incomplete). Figure 2 gives an example of a temporal graph from
STAGE.

Figure 2: An example of a STAGE-extracted temporal graph. Every time
expression is represented as a time node with edge type A before B or A
happens at time B.

#### Node Representation learning via R-GAT

Let graph $G=(\mathcal{V},\mathcal{E},\mathcal{R})$ be either syntactic or
temporal graphs with nodes222Note, we re-use notation $u_{i}$ to denote $i$-th
node $u$ in the graph. $u_{i}\in\mathcal{V}$ and labeled edges
$(u_{i},r,u_{j})\in\mathcal{E}$ where $r\in\mathcal{R}$ is the relation type.
Each node is initialized as a $d_{\text{in}}$-dimensional feature vector, and
we assign a $|\mathcal{R}|$-dimensional one-hot vector $a_{ij,r}$ for relation
$r$ between $u_{i},u_{j}$. We define the graph convolution at layer $l$ for
node $u_{i}$ with R-GAT as:

${\bm{h}}^{(l)}_{i}=\sigma({W}^{l}{\bm{h}}^{(l-1)}_{i}+\sum_{r\in\mathcal{R}}\sum_{j\in{\mathcal{N}_{i}}}\alpha^{(l)}_{ij,r}W^{l}_{ij,r}{\bm{h}}^{(l-1)}_{j})$
(5)

where $\sigma$ denotes ReLU activation function,
$h^{l}_{i}\in\mathbb{R}^{d_{h}}$ is the $d_{h}$-dimensional node embedding of
$u_{i}$, $\mathcal{N}_{i}$ is the neighborhood of node $u_{i}$, and
$W^{l}\in\mathbb{R}^{d_{\text{in}}\times d_{h}}$ is a trainable parameter.
$W^{l}_{ij,r}\in\mathbb{R}^{d_{h}}$ is the relation embedding of relation $r$
between $u_{i},u_{j}$ such that $W^{l}_{ij,r}=a_{ij,r}b^{(l)}$ where
$b^{(l)}\in\mathbb{R}^{|\mathcal{R}|\times d_{h}}$ is a trainable parameter.
The attention weight $\alpha^{(l)}_{ij,r}$ between node $u_{i}$ and node
$u_{j}$ under relation type $r$ is computed as:

$\alpha^{(l)}_{ij,r}=\frac{\exp(\tilde{\sigma}(W^{(l)}_{a}({\bm{h}}^{(l)}_{i}\mathbin{\|}{\bm{h}}^{(l)}_{j})+W^{l}_{ij,r}))}{\displaystyle\sum_{p\in\mathcal{R}}\sum_{q\in{\mathcal{N}_{i}}}\exp(\tilde{\sigma}(W^{(l)}_{a}({\bm{h}}^{(l)}_{i}\mathbin{\|}{\bm{h}}^{(l)}_{q})+W^{l}_{iq,p}))}$
(6)

where $\tilde{\sigma}$ denotes the LeakyReLU activation function,
$h^{l}_{j}\in\mathbb{R}^{d_{h}}$ is the node embedding of $u_{j}$, and
$W^{(l)}_{a}\in\mathbb{R}^{2d_{\text{in}}\times d_{h}}$ is a trainable
parameter. $\alpha^{(l)}_{ij,r}$ can be considered as cross-relation attention
to capture larger influence relation to node $u_{i}$.

#### Subgraph Augmentation

To provide more relevant structural information for co-training with BERT, we
augment the $l$-th layer node embedding $h^{l}_{i}$ of node $u_{i}$ with a
k-hop subgraph:

$\mathcal{G}^{(l)}_{i}=\\{h^{(l)}_{i}\\}\cup\\{h^{(l)}_{j}\mid
j\in\mathcal{N}^{k}_{i}\\}$ (7)

where $\mathcal{G}^{(l)}_{i}$ is the $l$-th layer $u_{i}$-induced subgraph and
$\mathcal{N}^{k}_{i}$ denotes a k-hop subgraph induced by node $u_{i}$. At
this point, we represent node $u_{i}$ as a set of $l$-th layer node embeddings
of $u_{i}$ itself and all other nodes $u_{j}\in\mathcal{N}^{k}_{i}$.

#### R-GAT Projection Head

We now define the projection head $f_{\beta}$ for each layer of R-GAT to
provide a channel for semantic information exchange with BERT. For event pair
$(e^{1}_{i},e^{2}_{i})$, we obtain their $l$-th layer node representations and
node-induced syntactic and temporal subgraphs of $e^{1}_{i},e^{2}_{i}$ via
equation 5 and 7. Then, we separately apply mean pooling over all nodes in
syntactic and temporal subgraphs to get single vector representations as
$g^{sg(l)}_{i},g^{tg(l)}_{i}\in\mathcal{R}^{d_{h}}$. The projection heads of
event pair $(e^{1}_{i},e^{2}_{i})$ are:

$\displaystyle z^{sg(l)}_{i}$
$\displaystyle=f_{\beta}(g^{sg(l)}_{i})\in\mathcal{R}^{d_{p}}$ (8)
$\displaystyle z^{tg(l)}_{i}$
$\displaystyle=f_{\beta}(g^{tg(l)}_{i})\in\mathcal{R}^{d_{p}}$ (9)

where $z^{sg(l)}_{i},z^{tg(l)}_{i}$ are the projection heads for $l$-th layer
subgraph representations from syntactic and temporal graphs for event pair
$(e^{1}_{i},e^{2}_{i})$ and
$f_{\beta}:\mathbb{R}^{d_{h}}\mapsto\mathbb{R}^{d_{p}}$ denotes a projection
function that maps input to $d_{p}$-dimensional embedding space, which is co-
trained with BERT.

#### R-GAT Prediction Head

Syntactic and temporal graphs express distinct types of discourse-level
semantic information about events. Therefore, for a given event node $u$, we
add cross-graph node attention to model this semantic information. We first
concatenate node representations from each R-GAT layer:

$h_{u}=\bigparallel_{l=1}^{L}h^{(l)}_{u}\in\mathcal{R}^{L\times d_{h}}$ (10)

where $h^{(l)}_{u}$ is the $l$-th layer $d_{h}$-dimension node representation.
Then, we cross-align nodes in syntactic and temporal k-hop subgraphs via
scaled dot-product attention (Vaswani et al. 2017). For event pair
$(e^{1}_{i},e^{2}_{i})$, we obtain their multi-layer syntactic and temporal
node representations $(h^{sg(1)}_{i},h^{sg(2)}_{i})$ and
$(h^{tg(1)}_{i},h^{tg(2)}_{i})$. Then, we derive their node-induced subgraphs
via Equation 7 as $(\mathcal{G}^{sg(1)}_{i},\mathcal{G}^{sg(2)}_{i})$ and
$(\mathcal{G}^{sg(1)}_{i},\mathcal{G}^{sg(2)}_{i})$. Let $H^{1}_{i}$ denote
all nodes in $\\{h^{sg(1)}_{i},h^{tg(1)}_{i}\\}$ and $H^{2}_{i}$ denote all
nodes in $\\{h^{sg(2)}_{i},h^{tg(2)}_{i}\\}$, we compute their attention
outputs as:

$\displaystyle\text{head}^{1}_{i}$
$\displaystyle=softmax(\frac{H^{1}_{i}W_{Q}(H^{1}_{i}W_{K})}{\sqrt{d_{a}}})H^{1}_{i}W_{V}$
(11) $\displaystyle\text{head}^{2}_{i}$
$\displaystyle=softmax(\frac{H^{2}_{i}W_{Q}(H^{2}_{i}W_{K})}{\sqrt{d_{a}}})H^{2}_{i}W_{V}$
(12)

where $W_{Q},W_{K},W_{V}\in\mathcal{R}^{Ld_{h}\times d_{a}}$ denote attention
weights and $\text{head}^{1}_{i},\text{head}^{2}_{i}$ are attention outputs
for event pair $(e^{1}_{i},e^{2}_{i})$. This attention mechanism encourages
the model to additionally learn interactions between syntactic and temporal
contexts. The prediction heads for event temporal relation extraction are then
defined as a mean pooling operation:

$\displaystyle o_{e^{1}_{i}}$
$\displaystyle=\frac{1}{|\text{head}^{1}_{i}|}\sum\nolimits^{|\text{head}^{1}_{i}|}_{n=1}{\text{head}^{1}_{i}}_{n}$
(13) $\displaystyle o_{e^{2}_{i}}$
$\displaystyle=\frac{1}{|\text{head}^{2}_{i}|}\sum\nolimits^{|\text{head}^{2}_{i}|}_{n=1}{\text{head}^{2}_{i}}_{n}$
(14)

### Multi-Scale Contrastive Co-Training

As discussed in Lu, Du, and Nie (2020), using hierarchical semantic features
from R-GAT, BERT can learn local context more effectively. However, doing
whole batch graph training alongside BERT is very memory-intensive. To tackle
this, we propose a multi-scale contrastive loss to co-train BERT and R-GAT,
which does not require model parameter sharing. We use normalized temperature-
scaled cross entropy loss (NT-Xent) (Chen et al. 2020) $\ell_{cl}$ for
positive pair $(p_{1},p_{2})$ from a batch (of size $N$) of projection heads
as contrastive loss:

$\ell_{cl}(p_{1},p_{2})=-log\frac{e^{\mathrm{sim}(p_{1},\hat{p_{2}})/\tau}}{\sum^{N}_{q=1}\mathbbm{1}_{[p_{q}\neq
p_{1}]}e^{\mathrm{sim}(p_{1},\hat{p_{q}})/\tau}}$ (15)

where $\hat{\cdot}$ stands for the stop gradient operator (Chen et al. 2020)
that sets the input variable to a constant, $\mathrm{sim}(\cdot)$ denotes
cosine similarity, $\mathbbm{1}_{[p_{q}\neq p_{1}]}\in\\{0,1\\}$ is an
indicator to evaluate if $p_{q}\neq p_{1}$, and $\tau$ is a temperature
parameter. Note that we do not perform any data augmentation. For each
positive pair in the training batch, we compute R-GAT and BERT projection
heads that are then used for co-training. Finally, we propagate information
between BERT and R-GAT by computing a multi-scale contrastive loss
$\mathcal{L}_{{CL}_{i}}$ between BERT projection head and R-GAT projection
heads from each layer for an event pair $(e^{1}_{i},e^{2}_{i})$:

$\begin{split}\mathcal{L}_{{CL}_{i}}=\frac{1}{L}\sum^{L}_{l=1}&\ell_{cl}(z^{bert}_{i},z^{sg(l)}_{i})+\ell_{cl}(z^{sg(l)}_{i},z^{bert}_{i})\\\
+&\ell_{cl}(z^{bert}_{i},z^{tg(l)}_{i})+\ell_{cl}(z^{tg(l)}_{i},z^{bert}_{i})\end{split}$
(16)

where $z^{bert}_{i}$ denotes BERT projection head and
$z^{sg(l)}_{i},z^{tg(l)}_{i}$ denote $l$-th layer projection heads from
syntactic and temporal R-GATs.

### Training on Event Temporal Relation Extraction

Given an event pair $(e^{1}_{i},e^{2}_{i})$ with relation label $r_{i}$, we
concatenate all prediction heads from BERT, syntactic and temporal R-GATs:

$o^{out}_{i}=[o^{bert}_{i}\mathbin{\|}o_{e^{1}_{i}}\mathbin{\|}o_{e^{2}_{i}}]$
(17)

where $o^{bert}_{i}$ denotes BERT prediction head and
$o_{e^{1}_{i}},o_{e^{2}_{i}}$ denote prediction heads from syntactic and
temporal R-GATs for event pair $(e^{1}_{i},e^{2}_{i})$. Then, we compute the
classification loss $\ell_{{CLS}_{i}}$ for classifier $f_{CLF}$ as cross-
entropy loss ($\ell_{ce}(\cdot)$):

$\ell_{{CLS}_{i}}=\ell_{ce}(f_{CLF}(o^{out}_{i}),r_{i})$ (18)

Finally, given a training batch of $N$ event pairs, we train the whole model
by jointly optimizing multi-scale contrastive loss and classification loss as:

$\mathcal{L}=\frac{1}{N}\sum^{N}_{i=1}\mathcal{L}_{{CL}_{i}}+\ell_{{CLS}_{i}}$
(19)

### MulCo Interpretation

The MulCo framework can be interpreted as two models (BERT and R-GAT), with
each model being optimized using the other’s output as the objective function.
In other words, it encourages both models to maximize their output similarity
in a shared embedding space. To simplify our interpretation, let
$F_{\theta},G_{\pi}$ denote BERT and R-GAT with built-in projection heads that
receive event $p$ as input. We can interpret contrastive co-training (Equation
15) as maximizing the probability that events from a positive pair are nearest
neighbors (Ko et al. 2022). Given an event $p$, Equation 15 tries to
simultaneously maximize the cosine similarity between
$F_{\theta}(p),G_{\pi}(p)$ and minimize it between
$F_{\theta}(p),G_{\pi}(\lnot p)$, where $\lnot p$ denotes other examples.
Following Chen et al. (2020), we now cast the original optimization task into
two subtasks:

$\displaystyle\min_{\theta,\pi}\quad\ell(F_{\theta}(p),\hat{G}_{\pi}(p))$
$\displaystyle\approx\min_{\theta}\quad\mathrm{sim}(F_{\theta}(p),G(p))$ (20)
$\displaystyle\min_{\pi,\theta}\quad\ell(G_{\pi}(p),\hat{F}_{\theta}(p))$
$\displaystyle\approx\min_{\pi}\quad\mathrm{sim}(G_{\pi}(p),F(p))$ (21)

where $\mathrm{sim}(\cdot)$ denotes cosine similarity, and the stop gradient
operator $\hat{\cdot}$ transforms the associated variable into a constant.
When computing the joint loss from Equation 16, we are actually solving the
individual optimization problem of maximizing cosine similarity for
$F_{\theta}$ given the embedding from $G$ as target objective and vice versa.
When optimizing BERT with R-GAT as the target objective, BERT is encouraged to
produce similar representations to R-GAT, hence, getting regularized by global
context, and vice versa. Thus, MulCo can jointly train BERT and R-GAT without
parameter sharing and mutually adjust their modeling semantics.

## Experiments

### Datasets

Dataset | Train | Val | Test | Labels
---|---|---|---|---
TimeBank-Dense | 4,032 | 629 | 1,427 | 6
MATRES | 231 | 25 | 20 | 4
TDDMan | 4,000 | 650 | 1,500 | 5
TDDAuto | 32,609 | 4,435 | 4,258 | 5
Table 1: Dataset statistics on Train/Val/Test splits on temporal relations and
the number of relation types in all datasets.

We evaluate MulCo on TDDiscourse (Naik, Breitfeller, and Rose 2019), MATRES
(Ning, Wu, and Roth 2018), and TimeBank-Dense (Cassidy et al. 2014), which are
widely used temporal ordering benchmarks. TDDiscourse, consisting of two
subsets TDDMan and TDDAuto, contains event pairs that are more than one
sentence apart. TDDMan contains global event pairs, and TDDAuto includes both
local and global event pairs. MATRES and TimeBank-Dense (TB-Dense) largely
contain local event pairs, where temporal relations are either located in the
same or adjacent sentence. Table 1 presents dataset statistics.

### Experimental and Implementation Settings

We follow the same data splits (train/val/test) and evaluation metrics
(Precision (P), Recall (R), and F1 score (F1)) as TIMERS. We choose RoBERTa
(Liu et al. 2019) as our BERT-based local context encoder and R-GAT (Busbridge
et al. 2019) as our GNN-based global context encoder. We tune hyper-parameters
on the validation set (details in supplementary material). R-GAT hidden
dimension is set to 256 with a dropout rate of 0.1, and learning rate is set
to 1e-5 across all experiments. We use the Adam (Kingma and Ba 2014) optimizer
and train for 1,000 epochs, returning the best model based on validation
accuracy.

### Baselines

We compare MulCo against several baselines: Majority a majority voting
classifier, CAEVO (Chambers et al. 2014) and SP+ILP (Ning, Feng, and Roth
2017). We also compare with neural network baselines: BiLSTM (Cheng and Miyao
2017), BiLSTM+MAP (Han, Ning, and Peng 2019), and DeepSSVM (Han et al. 2019).
Following Mathur et al. (2021); Liu et al. (2021); Man et al. (2022), we
include BERT (Jacob Devlin and Toutanova 2019) and RoBERTa (Liu et al. 2019)
as large-scale pre-trained language model baselines. From prior SOTA methods,
we include UCGraph+BERT (Liu et al. 2021), TIMERS (Mathur et al. 2021), and
the current best-performing approach SCS-EERE (Man et al. 2022). Finally, we
compare our fusion strategy with Node-BERT (He, Wang, and Zhang 2020),
Concatenation (Mathur et al. 2021), VGCN-BERT (Lu, Du, and Nie 2020), BERT4GCN
(Xiao et al. 2021), and Interpolation (Lin et al. 2021).

### Evaluation Results

#### Overall Performance

| TDDMan | TDDAuto | MATRES | TB-Dense
---|---|---|---|---
Model | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Majority | 37.8 | 36.3 | 37.1 | 34.2 | 32.3 | 33.2 | 50.7 | 50.7 | 50.7 | 40.5 | 40.5 | 40.5
CAEVO | 32.3 | 10.7 | 16.1 | 61.1 | 32.6 | 42.5 | - | - | - | 49.9 | 46.6 | 48.2
SP | 22.7 | 22.7 | 22.7 | 43.2 | 43.2 | 43.2 | 66.0 | 72.3 | 69.0 | 37.7 | 37.8 | 37.7
SP+ILP | 23.9 | 23.8 | 23.8 | 46.4 | 45.9 | 46.1 | 71.3 | 82.1 | 76.3 | 58.4 | 58.4 | 58.4
BiLSTM | 24.9 | 23.8 | 24.3 | 55.7 | 48.3 | 51.8 | 59.5 | 59.5 | 59.5 | 63.9 | 38.9 | 48.4
BiLSTM+MAP | 41.0 | 41.1 | 41.1 | 64.2 | 51.4 | 57.1 | - | - | 75.5 | - | - | 64.5
DeepSSVM | 41.2 | 40.8 | 41.0 | 65.6 | 53.3 | 58.8 | - | - | - | - | - | 63.2
BERT | 36.5 | 37.1 | 37.5 | 62.0 | 61.7 | 62.3 | 65.6 | 78.1 | 77.2 | 59.7 | 60.7 | 62.2
RoBERTa | 35.7 | 36.5 | 37.1 | 60.6 | 62.7 | 61.6 | 77.3 | 79.0 | 78.9 | 58.1 | 57.6 | 61.9
UCGraph+BERT | 44.5 | 42.3 | 43.4 | 66.1 | 56.9 | 61.2 | - | - | - | 62.4 | 56.1 | 59.1
TIMERS | 43.7 | 46.7 | 45.5 | 64.3 |  72.7 | 71.1 | 81.1 | 84.6 | 82.3 | 48.1 | 65.2 | 67.8
SCS-EERE | - | - |  51.1 | - | - |  76.7 | 78.8 | 88.5 | 83.4 | - | - | -
MulCo | 56.2* | 54.0* | 55.1* | 79.2* | 75.0* | 77.0 | 88.2* |  88.2 | 88.2* | 84.9* | 84.9* | 84.9*
Table 2: Performance comparison on all benchmark datasets. The best performing
model is bold in scores, and the second best performance model has an
underline under each score. * indicates statistical significance tested at
$p$-value 0.05.

Table 2 shows that our model, MulCo, establishes new SOTA performance on all
datasets. On MATRES, the recall score for SCS-EERE is 0.3 higher than ours,
but the difference is not statistically significant, and we outperform SCS-
EERE on overall F1 score. MulCo obtains the best performance on both local and
global event pair datasets, e.g., TimeBank-Dense and TDDMan, and remains
robustly competitive on the most challenging TDDAuto dataset, which contains a
mix of local and global pairs. This suggests that our co-training scheme helps
in successfully combining the individual modeling advantages of BERT and GNN
models on local and global event pairs.

(a) TDDMan

(b) TDDAuto

(c) MATRES

(d) TB-Dense

Figure 3: Heatmap comparison of F1 scores achieved with different choices of
BERT and GNN on all datasets.

#### Fusion Performance

To establish the effectiveness of MulCo, we compare its performance with other
BERT and GNN fusion strategies. For these experiments, we fix the local
encoder to RoBERTa and global encoder to R-GAT, which we find to be the best-
performing setup. Table 3 lists F1 scores for all selected fusion strategies,
showing that MulCo outperforms all of them, especially on TDDAuto. Some
strategies, e.g., sharing parameters between GNN and BERT, hit a memory limit
unlike MulCo. These results also how the BERT input length restriction (512
tokens) hampers its ability to model global context. Surprisingly, the
interpolation fusion method leads to very negative results in our setting. We
surmise that interpolating softmax output from BERT and GNN neglects the
training signal.

Figure 4: Comparison of MulCo-Global (R-GAT) and MulCo-Local (BERT) with other
strong baselines.

#### Performance under various BERT and GNN encoders

The heatmaps in Figure 3 show that MulCo performs best on all datasets when
selecting RoBERTa and R-GAT as local and global context encoders. On TDDMan,
we see that choosing the right GNN is more important than selecting between
BERT and Roberta. By contrast, selecting a proper BERT-based language model is
more important than choosing a GNN for TimeBank-Dense. We attribute this
competitive advantage to increased learning capacities. The relation type
attention in R-GAT provides more capacity to model global context
dependencies. Furthermore, sentence dynamic masking in RoBERTa provides more
variance allowing the language model to capture different aspects of the local
context. This stronger learning capacity for local and global contexts boosts
performance when co-training RoBERTa and R-GAT.

Fusion |  | TDD
---
Man
| TDD
---
Auto
MATRES |  | TB-
---
Dense
Node-BERT | Out of Memory
$\text{Node-BERT}_{\textit{fixed}}$ | 50.8 | 53.0 | 77.6 | 70.5
VGCN-BERT | Out of Memory
BERT4GCN | Out of Memory
Interpolation | 14.8 | 11.3 | 35.3 | 3.0
Concatenation | 53.6 | 66.1 | 87.5 | 78.1
MulCo | 55.1 | 77.0 | 88.2 | 84.9
Table 3: Performance comparison (F1 score) of different BERT and GNN fusion
strategies on all datasets.

#### Standalone BERT and R-GAT Performance

To validate the effectiveness of our proposed event position-aware BERT
architecture (MulCo-Local) and subgraph-aware R-GAT (MulCo-Global), we train
and evaluate them separately without co-training. Figure 4 indicates both
MulCo-Local and MulCo-Global alone outperform strong baselines like TIMERS and
UCG-Graph on the TimeBank-Dense and TDDMan datasets. In fact, our BERT and
R-GAT architectures obtain the best performance (after MulCo) on TimeBank-
Dense and TDDMan, respectively. But the inverse does not hold: R-GAT is not as
strong on TimeBank-Dense and BERT is not strong on TDDMan. We believe BERT
fails to capture long-distance context due to input length limitations, while
GNNs don’t model local event pairs well, motivating the need for fusion using
MulCo.

### Ablation Study and Analysis

Dataset | w/o Subgraph | w/ Subgraph
---|---|---
TDDMan | 52.5 | 55.1
TDDAuto | 72.4 | 77.0
MATRES | 86.6 | 88.2
TB-Dense | 80.4 | 84.9
Table 4: Performance comparison (F1 score) of MulCo variants with and without
subgraph augmentation.

We conduct a thorough analysis to characterize MulCo training behavior and
discuss performance gain based on experiments with individual model
components. Note that all experiments employ the same hyper-parameters, BERT,
and GNN achitectures as the best-performing model in Table 2.

| TDDMan
---|---
Head | P | R | F1
$\text{BERT}_{\text{w/o MulCo}}$ | 45.4 | 43.6 | 44.5
$\text{BERT}_{\text{w/ MulCo}}$ | 47.5 | 45.6 | 46.5
| TB-Dense
$\text{R-GAT}_{\text{w/o MulCo}}$ | 70.5 | 70.5 | 70.5
$\text{R-GAT}_{\text{w/ MulCo}}$ | 72.8 | 72.8 | 72.8
Table 5: Performance comparison on standalone R-GAT and BERT with / without
MulCo training.

#### Subgraph augmentation improves R-GAT performance

In Table 4, we demonstrate the effectiveness of adding subgraphs by showing
that training with subgraph augmentation achieves higher scores than single
node training. We hypothesize that subgraphs provide more learning capacity
for R-GAT, which can derive extra training signals from long-distance graph
structures. Fusion via MulCo allows BERT to also learn hierarchical semantics
from the R-GAT.

#### MulCo improves co-training on BERT and GNN without parameter sharing

MulCo does not share BERT model parameters with GNN during multi-modality co-
training. To observe possible performance gains over standalone BERT and GNN,
we perform temporal relation extraction on TDDMan solely with R-GAT and
TimeBank-Dense solely with BERT. We co-train BERT and R-GAT via MulCo and
compare it to performance without co-training. Table 5 shows improvement
achieved by MulCo training. It also demonstrates the effectiveness of our
proposed multi-scale contrastive co-training objective.

#### MulCo prevents BERT overfitting during co-training

MulCo

Concatenation

Interpolation

Figure 5: Comparison of training loss convergence.

To better understand the training dynamics of MulCo, we assign two separate
classifiers for BERT and R-GAT prediction heads and measure their cross-
entropy loss. We train for 1,500 epochs on MATRES, which contains a small
amount of mainly local event pairs where BERT can easily gain a competitive
advantage during training. Figure 5 clearly shows that all fusion strategies
overfit to BERT early on during training, since BERT achieves faster
convergence than R-GAT. However, MulCo overturns during the later stage of co-
training, with R-GAT now receiving more training signals and converging faster
to BERT. In other fusion types, e.g., interpolation and concatenation, BERT
still retains its competitive benefits with a higher convergence rate, with
R-GAT never rising beyond sub-optimal convergence.

## Conclusion

Learning to leverage local and global context surrounding event pairs is a
crucial step towards robust temporal relation extraction, yet, due to BERT
overfitting, current fusion efforts cannot maintain their distinct advantages.
MulCo’s multi-scale contrastive co-training model fuses BERT and GNNs. It
models semantics in local and global contexts, finds correlation via multi-
scale contrastive learning, and protects the training process from
interference and BERT overfitting. We demonstrate that with this effective
training scheme, our model achieves the new state-of-the-art performance on
the event temporal relation extraction task.

## References

  * Ballesteros et al. (2020) Ballesteros, M.; Anubhai, R.; Wang, S.; Pourdamghani, N.; Vyas, Y.; Ma, J.; Bhatia, P.; McKeown, K.; and Al-Onaizan, Y. 2020.  Severing the Edge Between Before and After: Neural Architectures for Temporal Ordering of Events.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 5412–5417. Online: Association for Computational Linguistics.
  * Breitfeller, Naik, and Rose (2021) Breitfeller, L.; Naik, A.; and Rose, C. 2021.  Stage: tool for automated extraction of semantic time cues to enrich neural temporal ordering models.  _arXiv preprint arXiv:2105.07314_.
  * Busbridge et al. (2019) Busbridge, D.; Sherburn, D.; Cavallo, P.; and Hammerla, N. Y. 2019.  Relational graph attention networks.  _arXiv preprint arXiv:1904.05811_.
  * Cassidy et al. (2014) Cassidy, T.; McDowell, B.; Chambers, N.; and Bethard, S. 2014.  An Annotation Framework for Dense Event Ordering.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 501–506.
  * Chambers et al. (2014) Chambers, N.; Cassidy, T.; McDowell, B.; and Bethard, S. 2014.  Dense event ordering with a multi-pass architecture.  _Transactions of the Association for Computational Linguistics_ , 2: 273–284.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607. PMLR.
  * Cheng and Miyao (2017) Cheng, F.; and Miyao, Y. 2017.  Classifying temporal relations by bidirectional LSTM over dependency paths.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 1–6.
  * Han et al. (2019) Han, R.; Hsu, I.-H.; Yang, M.; Galstyan, A.; Weischedel, R.; and Peng, N. 2019.  Deep Structured Neural Network for Event Temporal Relation Extraction.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , 666–106.
  * Han, Ning, and Peng (2019) Han, R.; Ning, Q.; and Peng, N. 2019.  Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 434–444.
  * He, Wang, and Zhang (2020) He, Q.; Wang, H.; and Zhang, Y. 2020.  Enhancing generalization in natural language inference by syntax.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 4973–4978.
  * Jacob Devlin and Toutanova (2019) Jacob Devlin, K. L., Ming-Wei Chang; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of NAACL-HLT_ , 4171–4186.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Ko et al. (2022) Ko, C.-Y.; Mohapatra, J.; Liu, S.; Chen, P.-Y.; Daniel, L.; and Weng, L. 2022.  Revisiting Contrastive Learning through the Lens of Neighborhood Component Analysis: an Integrated Framework.  In _International Conference on Machine Learning_ , 11387–11412. PMLR.
  * Lin et al. (2021) Lin, Y.; Meng, Y.; Sun, X.; Han, Q.; Kuang, K.; Li, J.; and Wu, F. 2021.  BertGCN: Transductive Text Classification by Combining GNN and BERT.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , 1456–1462.
  * Liu et al. (2021) Liu, J.; Xu, J.; Chen, Y.; and Zhang, Y. 2021.  Discourse-Level Event Temporal Ordering with Uncertainty-Guided Graph Completion.  In _IJCAI_ , 3871–3877.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lu, Du, and Nie (2020) Lu, Z.; Du, P.; and Nie, J.-Y. 2020.  VGCN-BERT: augmenting BERT with graph embedding for text classification.  In _European Conference on Information Retrieval_ , 369–382. Springer.
  * Man et al. (2022) Man, H.; Ngo, N. T.; Van, L. N.; and Nguyen, T. H. 2022.  Selecting optimal context sentences for event-event relation extraction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 11058–11066.
  * Mathur et al. (2021) Mathur, P.; Jain, R.; Dernoncourt, F.; Morariu, V.; Tran, Q. H.; and Manocha, D. 2021.  Timers: document-level temporal relation extraction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , 524–533.
  * Naik, Breitfeller, and Rose (2019) Naik, A.; Breitfeller, L.; and Rose, C. 2019.  TDDiscourse: A dataset for discourse-level temporal ordering of events.  In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_ , 239–249.
  * Ning, Feng, and Roth (2017) Ning, Q.; Feng, Z.; and Roth, D. 2017.  A Structured Learning Approach to Temporal Relation Extraction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 1027–1037.
  * Ning, Subramanian, and Roth (2019) Ning, Q.; Subramanian, S.; and Roth, D. 2019.  An Improved Neural Baseline for Temporal Relation Extraction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 6203–6209.
  * Ning, Wu, and Roth (2018) Ning, Q.; Wu, H.; and Roth, D. 2018.  A Multi-Axis Annotation Scheme for Event Temporal Relations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 1318–1328.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2020) Wang, H.; Chen, M.; Zhang, H.; and Roth, D. 2020.  Joint Constrained Learning for Event-Event Relation Extraction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 696–706.
  * Xiao et al. (2021) Xiao, Z.; Wu, J.; Chen, Q.; and Deng, C. 2021.  BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 9193–9200.
  * Yuan et al. (2021) Yuan, X.; Lin, Z.; Kuen, J.; Zhang, J.; Wang, Y.; Maire, M.; Kale, A.; and Faieta, B. 2021.  Multimodal contrastive training for visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6995–7004.