# Adaptive Pseudo-Siamese Policy Network for Temporal Knowledge Prediction

Pengpeng Shao1,2, Tong Liu1, Feihu Che1,2, Dawei Zhang1, Jianhua Tao1,2,3
1National Laboratory of Pattern Recognition, Institute of Automation, Chinese
Academy of Sciences, Beijing, China;
2School of Artificial Intelligence, University of Chinese Academy of Sciences,
Beijing, China;
3CAS Center for Excellence in Brain Science and Intelligence Technology,
Beijing, China

###### Abstract

Temporal knowledge prediction is a crucial task for the event early warning
that has gained increasing attention in recent years, which aims to predict
the future facts by using relevant historical facts on the temporal knowledge
graphs. There are two main difficulties in this prediction task. First, from
the historical facts point of view, how to model the evolutionary patterns of
the facts to predict the query accurately. Second, from the query perspective,
how to handle the two cases where the query contains seen and unseen entities
in a unified framework. Driven by the two problems, we propose a novel
adaptive pseudo-siamese policy network for temporal knowledge prediction based
on reinforcement learning. Specifically, we design the policy network in our
model as a pseudo-siamese policy network that consists of two sub-policy
networks. In sub-policy network I, the agent searches for the answer for the
query along the entity-relation paths to capture the static evolutionary
patterns. And in sub-policy network II, the agent searches for the answer for
the query along the relation-time paths to deal with unseen entities.
Moreover, we develop a temporal relation encoder to capture the temporal
evolutionary patterns. Finally, we design a gating mechanism to adaptively
integrate the results of the two sub-policy networks to help the agent focus
on the destination answer. To assess our model performance, we conduct link
prediction on four benchmark datasets, the experimental results demonstrate
that our method obtains considerable performance compared with existing
methods.

###### keywords:

Temporal Knowledge Graphs , Prediction , Reinforcement Learning

##  1 Introduction

Knowledge graphs (KGs), which store plenty of static triple facts in form of
(subject, relation, object), have been widely used in many natural language
processing applications, such as question answering [1], recommendation
systems [2], retrieval systems [3]. However, with the rise of triple facts,
many facts present dynamic attributes, and they only hold in a specific period
or a certain point in time, thus temporal knowledge graphs (TKGs) and
corresponding temporal knowledge reasoning tasks receive increasingly
extensive attention and research in recent years.

TKGs reasoning has two forms - interpolation and extrapolation. On a TKGs with
timestamps varying from $[\emph{t}_{0},\emph{t}_{T}]$, interpolation TKGs
reasoning aims to infer the answer for the query with time
$\emph{t}\in[\emph{t}_{0},\emph{t}_{T}]$. While extrapolation setting only
employs historical knowledge to predict future facts for time t
($\emph{t}>\emph{t}_{T}$), thus we call the extrapolation setting temporal
knowledge prediction here. Furthermore, the temporal knowledge prediction also
is a crucial task on event early warning. In this work, we focus on dealing
with the problems in temporal knowledge prediction.

Recently, many temporal knowledge prediction methods [4, 5, 6] based on
embedding are presented, and only a few works [7, 8] employ the models based
on reinforcement learning (RL) to predict the temporal knowledge. Although
embedding-based approaches are convenient to model the knowledge and obtain
considerable performance, they fail to take the symbolic compositionality of
KG relations into consideration, which limits their application in more
complex reasoning tasks. While RL-based approaches can allow the agent to find
the answer to the query by traversing the path in the interaction with the
complex environment, thus they possess powerful adaptability. However, there
exist two main difficulties for the agent in the process of traversing. First,
how to model the evolutionary patterns of the historical facts to predict the
future facts accurately. Second, the queries are usually uncertain, and it
usually contains seen entity or unseen entity, thus the second difficulty is
how to handle the two cases where the query contains seen and unseen entities
in a unified framework. Many related works have been proposed to handle these
two problems. In the embedding-based approaches, to model the temporal pattern
of the facts, Know-Evolve [9] and its extension DyRep [5] model the occurrence
of the facts as temporal point process, Re-NET [6] models the occurrence of a
fact as a probability distribution conditioned on temporal sequences of past
knowledge graphs. To copy with the second obstacle, XERTE [10] develops a
temporal relational attention mechanism and predicts over the relevant
substructure of TKGs by propagating attention, TITer [8] uses the inductive
mean representation of the trained entities with the same co-occurrence query
relation to represent unseen entities, thus obtains better entity embedding
distribution and answer score distribution. However, the unseen entity is an
entity that does not exist in historical facts and has no historical
neighbours. Thus, for the query containing the unseen entity, XERTE [10] can
not propagate attention to entities of previous facts, TITer [8] fails to
select candidate temporal actions to search for the answer.

In this work, we propose a novel RL-based temporal knowledge prediction model
to address the two problems. Specifically, we design the policy network of the
proposed model as a pseudo-siamese policy network that consists of two sub-
policy networks. The sub-policy network I is designed to capture the static
evolutionary patterns, and the agent I can search for the answer for the query
along the entity-relation paths instead of the time information. In sub-policy
network II, to deal with unseen entities, we add semantic edges in the TKGs,
the agent II searches for the answer for the query along the relation-time
paths instead of the entity information. Furthermore, we develop a temporal
relation encoder to capture the temporal evolutionary patterns. Finally, in
order to make the two sub-policy networks compensate for each other, we employ
a gating mechanism to adaptively integrate the results of two sub-policy
networks to help the agent focus on the destination entity. The extensive
experimental results show that our method obtains considerable performance
compared with existing approaches and demonstrate the effectiveness and
superiority of our method. Our contributions are as follows:

  * 1.

We develop a novel RL-based temporal knowledge prediction approach to deal
with the two cases where the query contains seen and unseen entities in a
unified framework.

  * 2.

We advocate the importance of more comprehensive modelling of the evolutionary
patterns of the facts, thus develop one sub-policy network to capture static
evolutionary patterns, and design another sub-policy network and temporal
relation encoder to model temporal evolutionary patterns. Finally, we employ a
gating mechanism to adaptively integrate the results of two sub-policy
networks.

  * 3.

We propose a new type of edge to build the relationship between the query
containing the unseen entity and historical facts and design a novel method to
handle the special type of query.

  * 4.

Experimental studies on four TKGs datasets demonstrate that our method
achieves state-of-the-art performance.

##  2 Related Work

###  2.1 Static KGs Reasoning

The static KGs reasoning methods can be roughly grouped into three categories:
embedding-based approaches, RL-based approaches, and logic rule-based
approaches. Therein, embedding-based approaches are the most popular one by
virtue of their high efficiency and outstanding effectiveness, which aim to
project the entities and relations in KGs into vector space and represent them
as low-dimensional embeddings. This type of method is also broadly classified
into three paradigms: (i) Translational distance-based models [11, 12, 13].
(ii) Tensor factorization-based models [14, 15, 16, 17]. (iii) Neural network-
based models [18, 19, 20].

Although embedding-based approaches present simplicity and convenience in
modelling knowledge and obtain considerable performance, they are less
sensitive to the reasoning distance and ignore the logical rules between
relations and paths, which limits their application in more complex reasoning
tasks and lacks interpretability. While RL-based approaches can allow the
agent to find the answer to the query by traversing the path on KGs, which
enables it to learn reasoning rules from relation paths. DeepPath [21] is the
first multi-hop reasoning work based on RL, it targets at searching for
generic representative paths between pairs of entities. MINERVA [22] utilizes
the history path to facilitate the agent searching answer entities of a
particular KG query in an end-to-end fashion. Based on MINERVA, M-Walk [23]
and Multi-HopKG [24] adopt Monte Carlo tree search and pre-trained embedding
model to overcome the problem of sparse rewards, respectively. However, all
these methods can not model the evolutionary patterns of the facts in TKGs.

###  2.2 Temporal KGs Reasoning

According to the relationship between query time and maximum training time,
TKGs reasoning can broadly be classified into two forms - interpolation and
extrapolation. Interpolation reasoning [25, 26, 27, 28, 29] aims to infer new
facts at history timestamps by employing historical information and future
information. Corresponding extrapolation reasoning aims to predict future
facts only based on historical information, so we call this reasoning task
temporal knowledge prediction here. In view of its great practical value,
especially in the event early warning, many temporal knowledge prediction
works have been presented in recent years. Know-Evolve [9] and its extension
DyRep [5] employ the temporal point process to model the occurrence of the
temporal facts in TKGs. While Re-NET [6] models the occurrence of the temporal
fact as a probability distribution conditioned on temporal sequences of past
knowledge graphs. To model repetitive facts, CyGNet [4] designs a copy mode to
learn from the known facts that appeared in history. XERTE [10] develops a
temporal relational attention mechanism and predicts the future facts by
propagating attention over the relevant substructure of TKGs. Especially,
CluSTeR [7] and TITer [8] are two temporal knowledge prediction approaches
based on RL. CluSTeR [7] first employs the agent to induce relevant clues from
historical facts, then adopts an embedding model to deduce answers from the
obtained clues. While TITer [8] forces the agent to travel on historical
knowledge graph snapshots and find the answer for the query directly.
Furthermore, to deal with the unseen entities in the query, TITer presents an
embedding method to obtain a more reasonable initial embedding for unseen
entities, thus improving the inductive inference ability of the model.
However, TITer [8] only uses the inductive mean representation to obtain
better entity embedding distribution and answer score distribution, and fails
to select candidate temporal edges to search for the answer except for the
self-loop edges for the query with an unseen entity.

##  3 The Proposed Model

###  3.1 Notations and Problem Formulation

The temporal facts in TKGs is defined by quadruples $(s,r,o,t)$ $\subseteq$
$\mathcal{E}$ $\times$ $\mathcal{R}$ $\times$ $\mathcal{E}$ $\times$
$\mathcal{T}$, where $\mathcal{E}$, $\mathcal{R}$, and $\mathcal{T}$ denote a
finite set of entities, relations, and timestamps, respectively. And TKGs also
can be represented as graph snapshots over time, namely $\mathcal{G}$ =
$\\{\mathcal{G}_{t_{1}},\mathcal{G}_{t_{2}},\dots,\mathcal{G}_{t_{T}}\\}$ =
$\\{(s,r,o,t_{l})\\}_{t_{l}=1}^{|\mathcal{T}|}$, where $s$, $o$ $\in$
$\mathcal{E}$, $r$ $\in$ $\mathcal{R}$, and each $\mathcal{G}_{t_{l}}$ is a
static multi-relational KGs at time $t_{l}$. Thus TKGs is a collection of
event fact at different timestamps, where the timestamps are arranged in
ascending order.

The task of temporal knowledge prediction is to predict the missing entity for
the query $(s_{q},r_{q},?,t_{q})$ or $(?,r_{q},o_{q},t_{q})$ according to the
historical facts $\mathcal{G}_{0:t_{q}-1}$. The inverse facts are added into
the training dataset, and predicting the subject entity for
$(?,r_{q},o_{q},t_{q})$ can be transformed into predicting the object entity
for $(o_{q},r_{q}^{-1},?,t_{q})$ without loss of generality. Note that, not
all historical facts are relevant to the query in $\mathcal{G}_{0:t_{q}-1}$,
thus the historical facts that related to the query subject $s_{q}$ are
represented as $E_{s}$ =
$\\{(s_{q},r,o,t_{l})\in\mathcal{G}_{t_{l}}|t_{l}<t_{q}\\}$, which have same
subject entity with query. In addition, we denote the historical facts which
related to the query relation $r_{q}$ as $E_{r}$ =
$\\{(s,r_{q},o,t_{l})\in\mathcal{G}_{t_{l}}|t_{l}<t_{q}\\}$, that is, the
facts in $E_{r}$ have same relation with query. Due to the relation of an
entity reflects the roles of the entity, so they have similar semantic
meanings. To be more specific, our task is to employ these historical facts
that related to the query to answer the given query.

###  3.2 Reinforcement Learning System

The temporal knowledge prediction task is formulated as a finite horizon
sequential decision-making problem, and the reinforcement learning system are
described as a deterministic Partially Observed Markov Decision Process
(POMDP). The system consists of agent and environment. The agent starts from
the query entity $s_{q}$, follows a relation path in $\mathcal{G}$ according
to its policy, and stops at the entity which is regarded as the right answer
to the query. The environment is described in detail as follows.

States. Each state can be represented as
$s_{k}=(e_{k},t_{k},s_{q},r_{q},o_{q},t_{q})\in\mathcal{S}$, where
$\mathcal{S}$ is the state space, $e_{k}$ is the entity visited at step $k$,
$t_{k}$ is the timestamp of the action taken at the previous step, and
$(s_{q},r_{q},t_{q})$ can be seen as the global context shared by all the
states for the given query, and $o_{q}$ is the answer. Due to the agent starts
from the query node $s_{q}$ and the default initial action is self-loop
action, so the initial state is $s_{0}=(s_{q},t_{q},s_{q},r_{q},o_{q},t_{q})$.

Figure 1: Illustration of the temporal edges and semantic edges

Observations. In the process of searching for answers, the agent can not
observe all the states of the environment but the current location and the
query information, and the answer $o_{q}$ remians invisible. Therefore, the
observation function is defined as:
$\mathcal{O}((e_{k},t_{k},s_{q},r_{q},o_{q},t_{q}))=(e_{k},t_{k},s_{q},r_{q},t_{q})$.

Actions. Given the current state $(e_{k-1},t_{k-1},s_{q},r_{q},o_{q},t_{q})$,
the set of the possible actions $\mathcal{A}_{k}\in\mathcal{A}$ at step $k$
consists of the outgoing edges of $e_{k-1}$, where $\mathcal{A}$ is the action
space. Here, we describe three types of outgoing edges of entity $e_{k-1}$ at
step $k$ as follows: (i) Self-loop Edges. The self-loop edges (i.e.
$(e_{k-1},r_{self},e_{k-1},t_{k-1})$) not only allow the agent to stay in a
place but also allow it to stop adaptively when it searches for a certain
number of steps or argues that it has found the final answer. (ii) Temporal
Edges. If there exist related historical facts $(e_{k},r,o,t_{l})$ for the
current entity $e_{k}$ at time $t_{k}$, we build the temporal edge between
$e_{k}^{t_{k}}$ and $o^{t_{l}}$ through relation $r$. For example, as shown in
the left of Figure 1, in the initial state, when the query contains the seen
entity $s_{q}$ and we can find the related historical facts
$(s_{q},r_{1},o_{1},t_{0})$, we build the temporal edges
$(s_{q}^{t_{q}},r_{1},o_{1},t_{0})$ . (iii) Semantic Edges. When the query
contains the unseen entity $s_{q}$, and there are no exist the related
historical facts $(s_{q},r,o,t_{l})$ for the entity $s_{q}$. Thus, in this
case, the agent can only stay in a place through the self-loop edges.
Fortunately, in the initial state, there exist the historical facts
$(s_{v},r_{q},o,t_{l})$ that related to the query relation $r_{q}$, as is
$(s_{v},r_{q},o_{8},t_{1})$ shown in the right of Figure 1, thus we build the
semantic edges $(s_{q},r_{q},o_{8},t_{1})$. Note that, the semantic edges only
exist in the first step to bridge the gap between the query and historical
facts. To summarize, when the query contains seen entity, the agent can walks
along the above three types of edges in the first step, then walks along
temporal edges or self-loop edges to find answers. When the query contains
unseen entity, the agent only can walks along the semantic edges in the first
step to arrive at seen entity, and then walks along temporal edges or self-
loop edges to find answers. Therefore,
$\mathcal{A}_{0}=\\{(r_{self},s_{q},t_{q})\cup\\{(r^{\prime},e^{\prime},t^{\prime})|(s_{q},r^{\prime},e^{\prime},t^{\prime})\in\mathcal{G}_{0:t_{q}-1},t^{\prime}<t_{q}\\}\cup(r_{q},e^{\prime},t^{\prime})|(s_{u},r_{q},e^{\prime},t^{\prime})\in\mathcal{G}_{0:t_{q}-1},(s_{q},r^{\prime},e^{\prime},t^{\prime})\notin\mathcal{G}_{0:t_{q}-1},t^{\prime}<t_{q}\\}$.
And
$\mathcal{A}_{k}=\\{(r_{self},e_{k},t_{k})\\}\cup\\{(r^{\prime},e^{\prime},t^{\prime})|(e_{k},r^{\prime},e^{\prime},t^{\prime})\in\mathcal{G}_{0:t_{k}-1},t^{\prime}<t_{k}\\}$,
$k>0$. Moreover, given the action $(r,e,t)$, here, we denote $(r,e)$ and
$(r,t)$ as static action and temporal action, respectively.

Transition. Given the current location $s_{k-1}$, once the action $a_{k}$ is
determined, the current location $s_{k-1}$ is transferred to the next location
$s_{k}$. Here, the transition function $\delta$ :
$\mathcal{S}\times\mathcal{A}\to\mathcal{S}$ defined by
$\delta(s_{k-1},a_{k})=s_{k}=(e_{k},t_{k},s_{q},r_{q},o_{q},t_{q})$. Therein,
the query information and answer are unchanged, the action
$a_{k}=(r_{k},e_{k},t_{k})$.

Rewards. After $K$-hop navigation, the agent reaches its final state
$(e_{K},t_{K},s_{q},r_{q},o_{q},t_{q})$. The agent receives a terminal reward
of 1 if $e_{K}=o_{q}$, and 0 otherwise. Formally, the reward mechanism is
defined as : $R(s_{K})=\mathbb{I}(e_{K}==o_{q})$.

###  3.3 Policy Network

The policy network $\pi_{\theta}$ is parameterized by using the action path
history and global context (query information), and it models the action of
the agent in a continuous space, thus we need to calculate the similarity
between the continuous value outputted by the policy network and the candidate
actions to determine the probability of selecting each candidate action. Here,
to deal with the case of the seen and unseen entities simultaneously in a
unified framework and model the evolutionary patterns of the facts, we design
two similar sub-policy networks to encode the path and score the candidate
actions, respectively, then employ a gating mechanism to obtain a final score
of each candidate action. The overview of our model is presented in Figure 2.

Figure 2: An overview of our model. Given the query $(s_{q},r_{q},?,t_{q})$,
the agent starts from query subject $s_{q}$, then samples an outgoing edge
(action) to walk and reaches the next entity according the policy network
$\pi_{\theta}$. For example, after $k$-1 hop navigation, the agent arrive at
$e_{k-1}$ (red node), and we can also obtain a historical path consisting of
$k$-1 historical actions. Then we use two sub-policy networks to encode the
two variant path of this historical path and obtain the scores of each
candidate action, respectively. Finally, we employ gating mechanism to obtain
the final scores of each candidate action adaptively and sample the next
action.

####  3.3.1 Policy Network I

The search history consisting of $k$ actions is
$h_{k}=\big{(}(s_{q},t_{q}),(r_{self},s_{q},t_{q}),(r_{1},e_{1},t_{1}),\\\
...,(r_{k},e_{k},t_{k})\big{)}$. To deal with the general case that the query
contains seen entity, we use the policy network to encode the action path
history $h_{k}$ as usual. However, to model the static and temporal
evolutionary patterns, here, we use one policy network (policy network I) to
encode the static action path history
$h_{k}^{s}=\big{(}(s_{q},t_{q}),(r_{self},s_{q}),...,(r_{k},e_{k})\big{)}$,
and employ another policy network (policy network II) to encode the temporal
action path
$h_{k}^{t}=\big{(}(s_{q},t_{q}),(r_{self},t_{q}),(r_{1},t_{1}),...,(r_{k},t_{k})\big{)}$.
With this design, we can model both static and temporal evolutionary patterns
in the general case that the query contains seen entity. The static action
path $h_{k}^{s}$ is encoded by policy network I as follows,

$\displaystyle\begin{aligned} &h_{0}^{s}=\emph{\rm
LSTM}(0,[r_{self},s_{q}])\\\ &h_{1}^{s}=\emph{\rm
LSTM}(h_{0},[r_{1},e_{1}])\\\
&~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\cdots\\\ &h_{k}^{s}=\emph{\rm
LSTM}(h_{k-1},[r_{k},e_{k}])\end{aligned}$ (1)

We then calculate probability of selecting each candidate action in policy
network I. First, we denote $N$ candidate actions at step $k$ as $\emph{\rm
A}_{k}=\\{(r_{i},e_{i},t_{i})\\}_{i=1}^{N}$. Similarly, the static candidate
actions is $\emph{\rm A}_{k}^{s}=\\{(r_{i},e_{i})\\}_{i=1}^{N}$, and the
temporal candidate actions is represented as $\emph{\rm
A}_{k}^{t}=\\{(r_{i},t_{i})\\}_{i=1}^{N}$. Thus, the probability distribution
$\phi^{s}_{\theta}(a_{k}|s_{k-1})$ of the candidate actions in the state
$s_{k-1}$ are calculated as,

$\displaystyle\begin{aligned} \phi_{\theta}^{s}(a_{k}|s_{k-1})=\emph{\rm
A}_{k}^{s}\emph{\rm W}_{2}^{s}\emph{\rm ReLU}(\emph{\rm
W}_{1}^{s}[h_{k-1}^{s},s_{q},r_{q}])\end{aligned}$ (2)

where $\emph{\rm W}_{1}^{s}$ and $\emph{\rm W}_{2}^{s}$ are the training
parameters in MLP.

####  3.3.2 Policy Network II

As mentioned above that the policy network II is used to encode the temporal
action path history
$h_{k}^{t}=\big{(}(s_{q},t_{q}),(r_{self},t_{q}),(r_{1},t_{1}),...,(r_{k},t_{k})\big{)}$
to model the temporal evolutionary patterns in the case that the query
contains seen entity. Here, it is worth noting that when the query contains
the unseen entity, we bridge the gap between the query and historical facts
through the semantic edges defined by query relation, thus we need to design a
policy network to encode the temporal action path history
$h_{k}^{t}=\big{(}(s_{q},t_{q}),(r_{self},t_{q}),(r_{1},t_{1}),...,(r_{k},t_{k})\big{)}$
instead of entity information. Therefore, encoding the temporal action path
$h_{k}^{t}$ not only can model the evolutionary patterns in the general case
but also deal with the special case that the query contains the unseen entity.

Here, As the time difference between the facts can better capture temporal
evolutionary patterns and express the dynamics of the facts, thus we first
employ temporal relation encoder to encode $h_{k}^{t}$ to obtain the temporal
relation path$\big{(}(s_{q},t_{q}),r_{self}^{t_{q}},r_{1}^{t_{1}},...,\\\
r_{k}^{t_{k}}\big{)}$, then use policy network II to encode the obtained
temporal relation path. Formally, the temporal action $r_{k-1}^{t}$ is encoded
by temporal relation encoder as follows,

$\displaystyle\begin{aligned} r_{k}^{t}=(\sigma(\emph{\rm
W}\Delta_{t})+\emph{b})*r_{k}\end{aligned}$ (3)

Where $w$ and $b$ are the training parameters, $\sigma$ is activation
function. ${\Delta}_{t}$ is the timestamps difference between temporal facts,
and $\Delta_{t}=t_{q}-t_{k}$. Likewise, we learn the temporal relation
$\emph{\rm A}_{k}^{tr}=\\{r_{i}^{t_{i}}\\}_{i=1}^{N}$ of the candidate
temporal actions $\emph{\rm A}_{k}^{t}$ for scoring the actions in a unified
form.

We then use policy network II to encode the temporal relation path,

$\displaystyle\begin{aligned} &h_{0}^{t}=\emph{\rm
LSTM}(0,r_{self}^{t_{q}})\\\ &h_{1}^{t}=\emph{\rm
LSTM}(h_{0},r_{1}^{t_{1}})\\\
&~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\cdots\\\ &h_{k}^{t}=\emph{\rm
LSTM}(h_{k-1},r_{k}^{t_{k}})\end{aligned}$ (4)

Afterward, the probability distribution $\phi_{\theta}^{t}(a_{k}|s_{k-1})$ of
the candidate actions in the state $s_{k-1}$ are calculated as,

$\displaystyle\begin{aligned} \phi_{\theta}^{t}(a_{k}|s_{k-1})=\emph{\rm
A}_{k}^{tr}\emph{\rm W}_{2}^{t}\emph{\rm ReLU}(\emph{\rm
W}_{1}^{t}[h_{k-1}^{t},r_{q}])\end{aligned}$ (5)

where $\emph{\rm W}_{1}^{t}$ and $\emph{\rm W}_{2}^{t}$ are the training
parameters in MLP.

####  3.3.3 Action scorer with gating mechanism

The future query events are usually uncertain, and it usually contains seen
entity or unseen entity. To deal with these two cases in a unified RL
framework and capture the evolutionary patterns of the facts more fully in the
general case that the query contains seen entity to predict the query
accurately, we design such a novel model with two sub-policy networks.
Therefore, given two probability distributions of the candidate actions
outputted by two sub-policy networks, we design a gating mechanism to
adaptively integrate them to help the agent focus on the destination entity
and obtain the final probability distribution $\pi_{\theta}(a_{k}|s_{k-1})$ of
the candidate actions,

$\displaystyle\begin{aligned} \pi_{\theta}(a_{k}|s_{k-1})=\emph{\rm
softmax}\big{(}(1-g_{t})\ast\phi_{\theta}^{s}(a_{k}|s_{k-1})+g_{t}\ast\phi_{\theta}^{t}(a_{k}|s_{k-1})\big{)}\end{aligned}$
(6) $\displaystyle\begin{aligned} g_{t}=\emph{\rm sigmoid}(\emph{\rm
W}_{g}[h_{k-1}^{t},r_{k}^{t_{k}},r_{q}])\end{aligned}$ (7)

where the gate function is parameterized by the temporal history, the
candidate temporal actions, and the query relation. $\emph{\rm W}_{g}$ is the
learnable parameter.

###  3.4 Optimization

We set the length of the search path to $K$, and the $K$-hop action path
generated by policy network $\pi_{\theta}$ is
$\\{a_{1},a_{2},\cdots,a_{K}\\}$. Thus, the policy network is optimized by
maximizing the expected reward over all queries in the training set
$\mathcal{D}_{train}$,

$\displaystyle\begin{aligned}
J(\theta)&=\mathbb{E}_{(s_{q},r_{q},o_{q},t_{q})\sim\mathcal{D}_{train}}[\mathbb{E}_{a_{1},\cdots,a_{K}\sim\pi_{\theta}}[R(s_{K}|s_{q},r_{q},t_{q})]]\end{aligned}$
(8)

We then employ the policy gradient method to optimize the policy network.
Specifically, Equation 8 is optimized by the REINFORCE algorithm, which will
iterate through all quadruple in $\mathcal{D}_{train}$ and update $\theta$
with the following stochastic gradient,

$\displaystyle\begin{aligned}
\nabla_{\theta}J(\theta)\thickapprox\nabla_{\theta}\sum_{k}R(s_{K}|s_{q},r_{q},t_{q})\emph{\rm
log}\pi_{\theta}(a_{k}|s_{k-1})\end{aligned}$ (9)

##  4 Experiment Result

Datasets | $N_{\emph{entity}}$ | $N_{\emph{relation}}$ | $N_{\emph{timestamps}}$ | _Time granularity_ | $N_{\emph{train}}$ | $N_{\emph{valid}}$ | $N_{\emph{test}}$
---|---|---|---|---|---|---|---
ICEWS2014 |  7,128 |  230 |  365 |  24 hours |  63685 |  13823 |  13222
ICEWS2018 |  23033 |  256 |  304 |  24 hours |  373018 |  45995 |  49545
WIKI |  12554 |  24 |  232 |  1 year |  539286 |  67538 |  63110
YAGO |  10623 |  10 |  189 |  1 year |  161540 |  19523 |  20026
Table 1: Dataset statistics. Datasets |  $N_{\emph{entity}}^{\emph{unseen}}$ |  $N_{\emph{quad}}^{\emph{unseen~{}entity}}$ |  $N_{\emph{quad}}^{\emph{unseen~{}object}}$ |  $N_{\emph{quad}}^{\emph{unseen~{}subject}}$ |  $N_{\emph{quad}}^{\emph{unseen~{}subject\&object}}$
---|---|---|---|---|---
ICEWS2014 |  496 |  862 (6.52%) |  438 |  497 |  73
ICEWS2018 |  1140 |  1948 (3.93%) |  975 |  1050 |  77
WIKI |  2968 |  27079 (42.91%) |  11086 |  22967 |  6974
YAGO |  540 |  1609 (8.03%) |  1102 |  873 |  366
Table 2: Number of unseen entities and quadruples containing unseen entities
in the test set.

###  4.1 Datasets

We assess our model performance on four public TKGs datasets : ICEWS14 [30],
ICEWS18 [30], WIKI [31], YAGO [32]. Therein, ICEWS14 and ICEWS18 are the two
subsets of the Integrated Crisis Early Warning System (ICEWS) dataset, they
contain the events in ICEWS that occurred in 2014 and 2018, respectively. WIKI
primarily extracts the temporal events from the Wikipedia dataset, and the
temporal facts in YAGO mainly come from Wikipedias, WordNet, and GeoNames. The
dataset is divided by timestamps in the form :
$\emph{train~{}time}<\emph{valid~{}time}<\emph{test~{}time}$, and more data
information is summarized in Table 1 and Table 2. Note that, in Table 2,
$N_{\emph{entity}}^{\emph{unseen}}$ denotes the number of new entities in the
test set. $N_{\emph{quad}}^{\emph{unseen~{}entity}}$ represents the number of
quadruples that the entities are unseen.
$N_{\emph{quad}}^{\emph{unseen~{}object}}$ is the number of quadruples that
object entities are unseen. $N_{\emph{quad}}^{\emph{unseen~{}subject}}$
denotes the number of quadruples that subject entities are unseen.
$N_{\emph{quad}}^{\emph{unseen~{}subject\&object}}$ represents the number of
quadruples that both subject entities and object entities are unseen.

###  4.2 Evaluation metrics

We conduct the temporal knowledge prediction task for evaluating our model. To
be specific, we predict two type of queries in the test set :
$q_{o}=(s_{q},r_{q},?,t_{q})$ and $q_{s}=(?,r_{q},o_{q},t_{q})$. We add
inverse facts into the training dataset, and transform predicting the subject
entity for $(?,r_{q},o_{q},t_{q})$ into predicting the object entity for
$(o_{q},r_{q}^{-1},?,t_{q})$ without loss of generality. Given the ground-
truth $o_{q}$ and $s_{q}$, we use Hits$@$1/3/10 and MRR [33] to assess our
model performance. MRR is the average of the reciprocal of the mean rank (MR)
assigned to the true triple overall candidate triples, which is defined as
follows :

$\displaystyle\begin{aligned} \emph{MRR}=\frac{1}{2\ast|test|}\sum_{q\in
test}\\!(\frac{1}{\emph{rank}(o_{q}|q_{o})}+\frac{1}{\emph{rank}(s_{q}|q_{s})})\end{aligned}$
(10)

Hits@$n$ measures the percentage of test set rankings where a true triple is
ranked within the top $n$ candidate triples, defined as:

$\displaystyle\begin{aligned} \emph{Hits@n}=\frac{1}{2\ast|test|}\sum_{q\in
test}(\mathbb{I}\\{\emph{rank}(o_{q}|q_{o})\\}\leq
n+\mathbb{I}\\{\emph{rank}(s_{q}|q_{s})\\}\leq n)\end{aligned}$ (11)

Furthermore, in this work, we employ the time-aware filtering [10] scheme
instead of the static filtering setting to filter out the quadruples that are
genuine at the timestamp, thus obtain more reasonable results.

###  4.3 Baselines

To assess the proposed model performance comprehensively, we compare it
against static KGs reasoning models including TransE [11], DistMult [15],
ComplEx [16], MINERVA [22], interpolation TKGs reaoning models including
TTransE [25], TA-DistMult [27], DE-SimplE [28], TNTComplEx [29], and several
extrapolation TKGs reasoning models including RE-NET [6], CyGNet [4], TANGO
[34], XERTE [10], and TITer [8].

In addition, to evaluate the importance of different components of our model,
we propose several variants of our model by adjusting the use of its
components. (1) policy network I: we only use policy network I to obtain the
final candidate action score. (2) policy network II: we only use policy
network II. (3) w/o TRE: ‘w/o’ refers to ‘without’, TRE denotes temporal
relation encoder. (4) w/o GM: GM represents the gating mechanism, and we use
the mean results of policy network I and policy network II as the final
candidate action score. (5) w/o Semantic Edges: It means removing the proposed
semantic edges in TKGs.

###  4.4 Implementation Details

In our experiments, we conduct the proposed model in Pytorch [35] framework.
The model is optimized by Adam [36] algorithm with the batch size of 512 and
the learning rate of 0.001. We set the entity embedding dimension to 100, the
relation embedding dimension to 100, the timestamp embedding dimension to 100,
and the discount factor of REINFORCE algorithm to 0.95. In addition, at each
hop, for all the candidate actions of the given state, we need to sample $N$
latest candidate actions to score. Here, $N$ is 100 on ICEWS14 and ICEWS18
dataset, 90 on WIKI, 100 on YAGO, and the length of the search path $K$ is 3.
In temporal relation encoder, the activation function $\sigma$ is $tanh$. Note
that, we reproduce the results of MINERVA with $N=50$ on all datasets, the
results of some baseline models are taken from the reported results of TANGO*
[37], XERTE [10], and TITer [8].

Method | ICEWS2014 | ICEWS2018 | WIKI | YAGO
---|---|---|---|---
MRR | Hit@1 | Hit@3 | Hit@10 | MRR | Hit@1 | Hit@3 | Hit@10 | MRR | Hit@1 | Hit@3 | Hit@10 | MRR | Hit@1 | Hit@3 | Hit@10
TransE | 0.224 | 0.133 | 0.256 | 0.412 | 0.124 | 0.058 | 0.128 | 0.251 | - | - | - | - | - | - | - | -
DistMult | 0.276 | 0.181 | 0.311 | 0.469 | 0.101 | 0.045 | 0.103 | 0.212 | 0.496 | 0.461 | 0.528 | 0.541 | 0.548 | 0.473 | 0.598 | 0.685
ComplEx | 0.308 | 0.215 | 0.344 | 0.495 | 0.210 | 0.118 | 0.234 | 0.398 | - | - | - | - | - | - | - | -
MINERVA* | 0.322 | 0.254 | 0.363 | 0.475 | 0.210 | 0.153 | 0.275 | 0.330 | 0.572 | 0.540 | 0.593 | 0.604 | 0.759 | 0.727 | 0.768 | 0.783
T-TransE | 0.134 | 0.031 | 0.173 | 0.345 | 0.083 | 0.019 | 0.085 | 0.218 | 0.292 | 0.216 | 0.344 | 0.423 | 0.311 | 0.181 | 0.409 | 0.512
TA-DistMult | 0.264 | 0.170 | 0.302 | 0.454 | 0.167 | 0.086 | 0.184 | 0.335 | 0.445 | 0.399 | 0.487 | 0.517 | 0.549 | 0.481 | 0.596 | 0.667
De-simplE | 0.326 | 0.244 | 0.356 | 0.491 | 0.193 | 0.115 | 0.218 | 0.348 | 0.454 | 0.426 | 0.477 | 0.495 | 0.549 | 0.516 | 0.573 | 0.601
TNTComplEx | 0.321 | 0.233 | 0.360 | 0.491 | 0.212 | 0.132 | 0.240 | 0.369 | 0.450 | 0.400 | 0.493 | 0.520 | 0.579 | 0.529 | 0.613 | 0.666
RE-NET | 0.382 | 0.286 | 0.413 | 0.545 | 0.288 | 0.190 | 0.324 | 0.475 | 0.496 | 0.468 | 0.511 | 0.534 | 0.580 | 0.530 | 0.610 | 0.662
CyGNet | 0.327 | 0.236 | 0.363 | 0.506 | 0.249 | 0.159 | 0.282 | 0.426 | 0.338 | 0.290 | 0.361 | 0.418 | 0.520 | 0.453 | 0.561 | 0.637
TANGO-Tucker | - | - | - | - | 0.286 | 0.193 | 0.321 | 0.470 | 0.504 | 0.485 | 0.514 | 0.535 | 0.578 | 0.530 | 0.607 | 0.658
TANGO-DistMult | - | - | - | - | 0.267 | 0.179 | 0.300 | 0.440 | 0.511 | 0.496 | 0.521 | 0.533 | 0.627 | 0.591 | 0.603 | 0.679
XERTE | 0.407 | 0.327 | 0.456 | 0.573 | 0.293 | 0.210 | 0.335 | 0.464 | 0.711 | 0.680 | 0.761 | 0.790 | 0.841 | 0.800 | 0.880 | 0.897
TITer | 0.417 | 0.327 | 0.464 | 0.584 | 0.299 | 0.220 | 0.334 | 0.448 | 0.755 | 0.729 | 0.774 | 0.790 | 0.874 | 0.848 | 0.899 | 0.902
Ours | 0.429 | 0.348 | 0.490 | 0.608 | 0.313 | 0.229 | 0.349 | 0.476 | 0.777 | 0.756 | 0.795 | 0.804 | 0.894 | 0.865 | 0.924 | 0.926
Table 3: The results of link prediction on ICEWS14, ICEWS18, WIKI, and YAGO
datasets. Compared metrics are filtered MRR and Hits@1/3/10. Best results are
in bold.

###  4.5 Results and Analysis

####  4.5.1 Comparative Study

Table 3 presents the temporal knowledge prediction results of the proposed
model and baseline models on four TKG datasets, and we have the following
observations and analyses from the results. First, intuitively, our model
outperforms all baseline models on four datasets, which justifies that our
method is feasible and achieves considerable results. Second, especially,
compared with the related baseline model TITer, our model achieves $2.2\%$ and
$2.7\%$ improvement in MRR and Hits@1 on the WIKI dataset, respectively, and
obtains different degrees of improvement on the other three datasets. Third,
the static reasoning model TransE is superior to its extension version
T-TransE on ICEWS2014 dataset, the main reason causes this phenomenon is that
TransE and T-TransE are designed for static and interpolation TKGs reasoning,
respectively, and they all can not model the unseen future timestamps in
query, thus the training time information may affect the prediction
performance of T-TransE.

Fourth, the performance of MINERVA is much higher than other static models,
all interpolation TKGs models, and part of extrapolation TKGs models on WIKI
and YAGO dataset, this is mainly because most entities in WIKI and YAGO have a
small number of neighbouring entities, which allows reinforcement learning-
based neighbour search algorithms to find answer entities quickly and
accurately. This also is one of the reasons why our method achieves
considerable performance on these two datasets.

####  4.5.2 Ablation Study

To verify the effectiveness of different components of the proposed model, we
present the results of five variants on ICEWS14 in Table 4. Policy Network I
and II exert different functions in our model, and each of them can calculate
the probability distribution of candidate actions and predict the answer for
the query. From the results, we can obtain that our model outperforms
component models Policy Network I and II, which shows that our model can
integrate these two component models adaptively by using the gating mechanism
and achieve remarkable results. In addition, we also observe that if we sum
the outputs of the two sub-policy networks directly instead of using the
gating mechanism, which leads to a drop of $3.5\%$ on MRR, indicating that the
gating mechanism can help our model to adaptively focus on the destination
answer of the query. The temporal relation encoder is a crucial component in
our model, to assess it, we remove the temporal relation encoder module in
policy network II and adopt the same encoding method as policy network I, the
model performance is dropped by $3.9\%$ on MRR compared with the proposed
model. Moreover, we propose to add semantic edges into TKGs to help the agent
find the actions in the case that the query contains an unseen entity, thus we
remove the semantic edges here to evaluate their importance. From the results,
we can obtain that this variant model performance is dropped by $1.1\%$ on MRR
compared with the proposed model. This also demonstrates the effectiveness of
the temporal relation encoder in temporal knowledge representation.

Model variant |  MRR |  Hits@1 |  Hits@3 |  Hits@10
---|---|---|---|---
Policy Network I |  0.391 |  0.290 |  0.441 |  0.590
Policy Network II |  0.415 |  0.328 |  0.460 |  0.578
w/o TRE |  0.390 |  0.288 |  0.441 |  0.593
w/o GM |  0.394 |  0.295 |  0.444 |  0.588
w/o Semantic Edges |  0.418 |  0.336 |  0.475 |  0.592
Ours |  0.429 |  0.348 |  0.490 |  0.608
Table 4: Ablation study on ICEWS14 dataset.

####  4.5.3 Ablation Study II

Our motivation for designing such a novel model is to better handle the two
cases where the query contains seen and unseen entities in a unified
framework. Specifically, when the query contains seen entity, the policy
network I and II can capture the evolutionary patterns of the facts
cooperatively. And when the query contains the unseen entity, the policy
network II is able to exert its particular function. Therefore, in this
section, we justify our design from the two perspectives.

Figure 3: Temporal knowledge prediction results on the subset of ICEWS14 that
contain seen entities. PN I and PN II represent Policy Network I and Policy
Network II, respectively.

We first verify our design from the first perspective that the policy network
I and II can capture the evolutionary patterns of the facts cooperatively when
the query contains seen entity, thus we conduct the link prediction on the
subset of ICEWS14 that contain seen entities, and the results are presented in
Figure 3. From the results, we can obtain that our model is superior to the
baseline models. Most importantly, the proposed model outperforms the
component models policy network I and II.

Figure 4: Temporal knowledge prediction results on the subset of ICEWS14 that
contain unseen entities. PN I and PN II represent Policy Network I and Policy
Network II, respectively.

To justify our design from the second view that the policy network II can
exert its particular function in the case where the query contains the unseen
entity, we also conduct the link prediction on the subset of ICEWS14 that
contain unseen entities, and the results are shown in Figure 4. Intuitively,
we can obtain that policy network II has explicit advantages in handling the
case that the query contains an unseen entity from the results, and our model
can absorb the advantages of policy network II to achieve considerable
performance. The above results demonstrate the effectiveness of our design and
provide a shred of evidence that our model can predict events in complex
scenarios.

Query  |  Path  |  Answer
---|---|---
(_Ed Royce_ , _Make pessimistic_ _comment_ , ?, _2014-11-23_)† |  (_Ed Royce_ , _Criticize or denounce_ , _Nuri al-Maliki_ , _2014-2-6_) $\Rightarrow$ (_Nuri al-Maliki_ , _Make statement_ , _Iran_ , _2014-1-25_) $\Rightarrow$ (_Iran_ , _self-loop_ , _Iran_ , _2014-1-25)_ |  _Iran_
(_Cabinet USA_ , _Consult_ , ?, _2014-11-11_)† |  (_Cabinet USA_ , _Deny responsibility_ -1, _Sergey_ , _2014-11-9_) $\Rightarrow$ (_Sergey_ , _self-loop_ , _Sergey_ ,_2014-11-9_) $\Rightarrow$ (_Sergey_ , _self-loop_ , _Sergey_ , _2014-11-9)_ |  _Sergey_
(_Mehdi Hasan_ , _Make an appeal_ _or request_ , ?, _2014-11-12_)‡ |  (_Shivraj_ , _Make an appeal or request_ , _India Citizen_ , _2014-6-22_) $\Rightarrow$ (_India_ _Citizen_ , _self-loop_ , _India Citizen_ , _2014-4-7_) $\Rightarrow$ (_India Citizen_ , _self-loop_ , _India Citizen_ , _2014-6-22_) |  _India Citizen_
(_India Chief_ , _Threaten_ , ?, _2014-11-19_)‡ |  (_Australia Citizen_ , _Threaten_ , _India Citizen_ , _2014-7-30_) $\Rightarrow$ (_India Citizen_ , _self-loop_ , _India Citizen_ , _2014-8-20_) $\Rightarrow$ (_India Citizen_ , _self-loop_ , _India Citizen_ , _2014-6-30)_ |  _India Citizen_
Table 5: Case study for two kinds of queries on ICEWS2014 dataset.
$(\cdot)^{\dagger}$ represents the query with seen entity, and
$(\cdot)^{\ddagger}$ indicates the query with unseen entity.

####  4.5.4 Case Study

Table 5 provides specific reasoning paths of four queries in the test set of
ICEWS2014, and the first two of them are the query with a seen entity, and the
last two of them are the query with an unseen entity. For the query with seen
entity (_Ed Royce_ , _Make pessimistic comment_ , ?, _2014-11-23_), the agent
arrives at the entity Nuri al-Maliki according the temporal edge (_Ed Royce_ ,
_Criticize or denounce_ , _Nuri al-Maliki_ , _2014-2-6_) from the start entity
Ed Royce in the first step. After three steps, the agent reaches the answer
entity Iran. While for the query with an unseen entity (_Mehdi Hasan_ , _Make
an appeal_ _or request_ , ?, _2014-11-12_), the entity Mehdi Hasan is unseen
in the historical facts, the agent can not find temporal edges to traverse
except for the self-loop edge. In this case, the agent first reaches the seen
entity India Citizen thorough the semantic edge (_Shivraj_ , _Make an appeal
or request_ , _India Citizen_ , _2014-6-22_), then arrives at the answer
thorough the temporal edges or self-loop edges in the following two steps.
From the above results, we can obtain that the semantic edge bridge the gap
between the query with an unseen entity and the historical facts, and it
exerts an important function in our method.

##  5 Conclusion

This work proposes a novel reinforcement learning model for temporal knowledge
prediction task on TKGs. The model is presented mainly to deal with two
problems, one is to model the evolutionary patterns of the facts, and the
other is to deal with uncertain queries that the query contains seen or unseen
entities. Thus we design a novel adaptive pseudo-siamese policy network to
deal with the two problems in a unified framework, policy networks I is used
to capture the static evolutionary patterns of the facts, and policy networks
II with the designed temporal relation encoder is employed to capture the
temporal evolutionary patterns of the facts and deal the special case that the
query contains the unseen entity. Finally, we develop a gating mechanism to
integrate the results of the two sub-policy networks adaptively to help the
agent focus on the destination answer. Experimental results on four public
datasets explicitly demonstrate the effectiveness and superiority of the
proposed model on the temporal knowledge prediction task.

## References

## References

  * [1] L. Dong, F. Wei, M. Zhou, K. Xu, Question answering over Freebase with multi-column convolutional neural networks, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 2015, pp. 260–269.
  * [2] Y. Koren, R. Bell, C. Volinsky, Matrix factorization techniques for recommender systems, in: Proceedings of Computer, 2009, pp. 30–37.
  * [3] C. Xiong, J. P. Callan, Query expansion with freebase, in: Proceedings of the International Conference on The Theory of Information Retrieval, 2015, pp. 111––120.
  * [4] C. Zhu, M. Chen, C. Fan, G. Cheng, Y. Zhang, Learning from history: Modeling temporal knowledge graphs with sequential copy-generation networks, in: Proceedings of AAAI Conference on Artificial Intelligence, 2021, pp. 4732–4740.
  * [5] R. S. Trivedi, M. Farajtabar, P. Biswal, H. Zha, Dyrep: Learning representations over dynamic graphs, in: Proceedings of International Conference on Learning Representations, 2019.
  * [6] W. Jin, M. Qu, X. Jin, X. Ren, Recurrent event network: Autoregressive structure inference over temporal knowledge graphs, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020.
  * [7] Z. Li, X. Jin, S. Guan, W. Li, J. Guo, Y. Wang, X. Cheng, Search from history and reason for future: Two-stage reasoning on temporal knowledge graphs, in: Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 2021, pp. 4732–4743.
  * [8] H. Sun, J. Zhong, Y. Ma, Z. Han, K. He, Timetraveler: Reinforcement learning for temporal knowledge graph forecasting, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021, pp. 8306–8319.
  * [9] R. Trivedi, H. Dai, Y. Wang, L. Song, Know-evolve: Deep temporal reasoning for dynamic knowledge graphs, in: Proceedings of the International Conference on Machine Learning, 2017, pp. 3462––3471.
  * [10] Z. Han, P. Chen, Y. Ma, V. Tresp, Explainable subgraph reasoning for forecasting on temporal knowledge graphs, in: Proceedings of International Conference on Learning Representations, 2021.
  * [11] A. Bordes, N. Usunier, A. Garcia-Durán, J. Weston, O. Yakhnenko, Translating embeddings for modeling multi-relational data, in: Proceedings of the International Conference on Neural Information Processing Systems, 2013, pp. 2787––2795.
  * [12] G. Ji, S. He, L. Xu, K. Liu, J. Zhao, Knowledge graph embedding via dynamic mapping matrix, in: Proceedings of Annual Meeting of the Association for Computational Linguistics, 2015, pp. 687––696.
  * [13] Z. Wang, J. Zhang, J. Feng, Z. Chen, Knowledge graph embedding by translating on hyperplanes, in: Proceedings of AAAI Conference on Artificial Intelligence, 2014, pp. 1112––1119.
  * [14] M. Nickel, V. Tresp, H.-P. Kriegel, A three-way model for collective learning on multi-relational data, in: Proceedings of the International Conference on Machine Learning, 2011, pp. 809––816.
  * [15] B. Yang, S. W.-t. Yih, X. He, J. Gao, L. Deng, Embedding entities and relations for learning and inference in knowledge bases, in: Proceedings of the International Conference on Learning Representations, 2015.
  * [16] T. Trouillon, J. Welbl, S. Riedel, E. Gaussier, G. Bouchard, Complex embeddings for simple link prediction, in: Proceedings of the International Conference on Machine Learning, 2016, pp. 2071––2080.
  * [17] I. Balažević, C. Allen, T. M. Hospedales, Tucker: Tensor factorization for knowledge graph completion, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, 2019, pp. 5185–5194.
  * [18] T. Dettmers, M. Pasquale, S. Pontus, S. Riedel, Convolutional 2d knowledge graph embeddings, in: Proceedings of AAAI Conference on Artificial Intelligence, 2018, pp. 1811–1818.
  * [19] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. v. d. Berg, I. Titov, M. Welling, Modeling relational data with graph convolutional networks, in: Proceedings of The Semantic Web, 2017, pp. 593–607.
  * [20] S. Vashishth, S. Sanyal, V. Nitin, P. Talukdar, Composition-based multi-relational graph convolutional networks, in: Proceedings of International Conference on Learning Representations, 2020.
  * [21] W. Xiong, T. Hoang, W. Y. Wang, Deeppath: A reinforcement learning method for knowledge graph reasoning, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2017, pp. 564–573.
  * [22] R. Das, S. Dhuliawala, M. Zaheer, L. Vilnis, I. Durugkar, A. Krishnamurthy, A. Smola, A. McCallum, Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning, in: Proceedings of International Conference on Learning Representations, 2018.
  * [23] Y. Shen, J. Chen, P.-S. Huang, Y. Guo, J. Gao, M-walk: Learning to walk over graphs using monte carlo tree search, in: Proceedings of the International Conference on Neural Information Processing Systems, 2018, pp. 6787––6798.
  * [24] X. V. Lin, R. Socher, C. Xiong, Multi-hop knowledge graph reasoning with reward shaping, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 1––9.
  * [25] T. Jiang, T. Liu, T. Ge, L. Sha, S. Li, B. Chang, Z. Sui, Encoding temporal information for time-aware link prediction, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016, pp. 2350–2354.
  * [26] S. S. Dasgupta, S. N. Ray, P. Talukdar, Hyte: Hyperplane-based temporally aware knowledge graph embedding, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2001–2011.
  * [27] A. García-Durán, S. Dumancic, M. Niepert, Learning sequence encoders for temporal knowledge graph completion, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018, pp. 4816–4821.
  * [28] R. Goel, S. M. Kazemi, M. Brubaker, P. Poupart, Diachronic embedding for temporal knowledge graph completion, in: Proceedings of AAAI Conference on Artificial Intelligence, 2020.
  * [29] T. Lacroix, G. Obozinski, N. Usunier, Tensor decompositions for temporal knowledge base completion, in: Proceedings of International Conference on Learning Representations, 2020, pp. 253––256.
  * [30] E. Boschee, J. Lautenschlager, S. O’Brien, S. Shellman, J. Starz, M. Ward, Icews coded event data, in: Proceedings of Harvard Dataverse, 2015.
  * [31] J. Leblay, M. W. Chekol, Deriving validity time in knowledge graph, in: Proceedings of the The Web Conference 2018, 2018, pp. 1771––1776.
  * [32] F. Mahdisoltani, J. Biega, F. M. Suchanek, Yago3: A knowledge base from multilingual wikipedias, in: Proceedings of CIDR, 2015.
  * [33] M. Nickel, K. Murphy, V. Tresp, E. Gabrilovich, A review of relational machine learning for knowledge graphs, in: Proceedings of the IEEE, 2016.
  * [34] Z. Ding, Z. Han, Y. Ma, V. Tresp, Temporal knowledge graph forecasting with neural ODE, in: Proceedings of CoRR, 2021.
  * [35] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Devito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch, in: Proceedings of the International Conference on Neural Information Processing Systems, 2017.
  * [36] J. C. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, in: Proceedings of Machine Learning Research, 2011, pp. 2121––2159.
  * [37] Z. Han, Z. Ding, Y. Ma, Y. Gu, V. Tresp, Learning neural ordinary equations for forecasting future links on temporal knowledge graphs, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 8352–8364.


# Sequence Mining and Pattern Analysis in Drilling Reports with Deep Natural
Language Processing

Júlio Hoffimann, Youli Mao, Avinash Wesley, and Aimee Taylor J. Hoffimann is
with the Department of Energy Resources Engineering, Stanford University, Palo
Alto, CA, 94305 USA e-mail<EMAIL_ADDRESS>Mao, A. Wesley and A.
Taylor are with Halliburton, Houston, TX, 77032 USA e-mail: e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Drilling activities in the oil and gas industry have been reported over
decades for thousands of wells on a daily basis, yet the analysis of this text
at large-scale for information retrieval, sequence mining, and pattern
analysis is very challenging. Drilling reports contain interpretations written
by drillers from noting measurements in downhole sensors and surface
equipment, and can be used for operation optimization and accident mitigation.
In this initial work, a methodology is proposed for automatic classification
of sentences written in drilling reports into three relevant labels (EVENT,
SYMPTOM and ACTION) for hundreds of wells in an actual field. Some of the main
challenges in the text corpus were overcome, which include the high frequency
of technical symbols, mistyping/abbreviation of technical terms, and the
presence of incomplete sentences in the drilling reports. We obtain state-of-
the-art classification accuracy within this technical language and illustrate
advanced queries enabled by the tool.

###### Index Terms:

natural language processing (NLP), noise-contrastive estimation (NCE),
convolutional neural networks (CNN), long short-term memory networks (LSTM)

##  I Introduction

Drilling activities in the oil and gas industry are a shared concern among
energy companies, government agencies, and the general public because they can
affect both company profitability and the natural environment. Having these
activities fully reported and classified is challenging to a limited human
workforce; ideally, this task would be assigned to an intelligent reporting
system during the drilling operation in real time.

The energy industry is in its initial steps toward the ultimate goal of smart
reporting. In their paper, “Augmenting Operations Monitoring by Mining
Unstructured Drilling Reports,” Sidahmed et al. [1] apply traditional natural
language processing (NLP) techniques to drilling reports collected daily from
three wells and show that, although unstructured, the text can provide new
insights about drilling events (e.g., stuck pipe) that are not easily captured
with other types of data. The work of Sidahmed et al. is one of the few
published articles on the subject and serves as a proof of concept that rich
information in the form of text is being underused by the industry. Although
very enlightening, the paper does not benefit from the recent advances in NLP
and deep learning, which can greatly scale information retrieval to complex
industrial settings.

Researchers with companies and universities, such as Google, Facebook, and
Stanford, have devoted a great deal of time considering how to process text
for information retrieval [2, 3, 4]. From the most basic task of text
classification to more elaborated summarization and question answering
software, researchers have developed novel algorithms capable of learning
semantics with limited or no human supervision. Such learning process depends
on the availability of large datasets in which deep neural networks are
trained on various NLP tasks.

In this work, recent developments in deep natural language processing (or deep
NLP) are applied to automatically classify sentences in thousands of drilling
reports and to identify companies’ behavior. This tool can be used offline by
an energy company interested in verifying old drilling reports for operation
patterns or by a government agency interested in investigating the aftermath
of environmental disasters. In the future, this tool could be used in real
time to enhance decision support systems, to help mitigate drilling costs
associated with non-productive time, and to reduce the risk of accidents.

This paper is organized as follows. Section II reviews state-of-the-art deep
natural language processing models for text classification. Section III
introduces the dataset and methodology used. Section IV presents the results
obtained with drilling reports and discusses the accuracy of the methods.
Section V describes conclusions about the applicability of the proposed tool
and points to future directions.

##  II Literature review

Natural language understanding and modeling has been a subject of research in
artificial intelligence since before the 1980s. In his article, “n-Gram
Statistics for Natural Language Understanding,” Suen [5] correctly predicts
the success of machine intelligence in NLP and presents classical n-gram
statistics of the English language based on a corpus of 1 million words.
Although more (memory-) efficient than vocabulary-based methods, the n-gram
statistics that were very powerful at the time are no longer the most
appropriate model to manage the massive amount of text generated daily on the
web (e.g., Google corpus, Wikipedia corpus) [6, 7, 8], nor can they easily
capture hidden semantic relationships needed for more challenging tasks, such
as information retrieval, summarization and question answering [3, 9].

Mikolov et al. [10] introduce a vector encoding for text words that
outperforms traditional NLP techniques based on co-occurrence counts and
n-gram statistics. The encoding effectively captures word semantics and shows
surprisingly linear relationships (e.g., “man” - “woman” + “queen” $\approx$
“king”) when trained on a sufficiently large corpus. These learned
relationships and word similarities are superior to those derived with
traditional minimum edit distance [11], and can be exploited to achieve much
deeper inference, as in the astonishing example “Russian” + “river” $\approx$
“Volga river” [10].

The continuous encoding proposed by Mikolov et al. is obtained by means of a
variation of cross-entropy minimization. To efficiently obtain an optimal
language model, the authors introduce a proxy logistic regression that
maximizes the probability of word pairs in the same context and minimizes the
probability of word pairs drawn at random from the unigram distribution. This
technique is called negative sampling, which is a particular case of noise-
contrastive estimation (NCE) of partition functions [12].

Two variants of the algorithm exist, depending on what is predicted and what
is given in the conditional probabilities: the continuous bag-of-words (CBOW)
and the skip-gram models. The CBOW model is trained to predict the center (or
target) word in the context window, whereas the skip-gram model predicts
context words given the center word in the window. The choice between these
two models is usually made by considering that the latter is better suited for
learning representations of infrequent words in the corpus.

In their article, “Glove: Global Vectors for Word Representation”, Pennington
et al. [13] discuss the properties that an encoding algorithm must obey for
word vectors to display semantic relationships similar to those introduced by
Mikolov et al. The authors propose a slightly different objective function,
inspired by simple experiments with probability ratios, and demonstrate
superior accuracy in word analogy tasks. These accuracy improvements, however,
are rarely manifested in practice with other NLP tasks.

In deep NLP, trained word vectors are input features to non-linear
classifiers/regressors. These vectors represent weights in a deep neural
network trained for a particular task [14, 15, 16]. The overall accuracy of
deep learning methods in NLP depends on the neural network architecture and
its capability of memorizing context information.

Deep network architectures have been successfully applied to different areas
of computer vision, such as face detection and recognition [17, 18], human
action recognition [19], and handwriting recognition [20, 21]. Mostly
convolutional, these networks compose input features (or words) into high-
level signals (or sentences), and can be exhaustively trained for solving
challenging tasks, such as dense captioning [22].

Recurrent neural network (RNN) architectures have been consistently
investigated in deep NLP because of their resemblance to linguistic grammars
[23, 15, 24] and superior classification accuracy in language-related tasks.
RNN architectures with memory cells have been developed that are capable of
retaining long sequence patterns. As the most prominent member of this
category, long short-term memory (LSTM) networks are suitable for text
classification and language comprehension [25].

Recently, deep neural networks have gained attention in NLP because of their
state-of-the-art accuracy, as compared to systems carefully designed for
specific languages and tasks [26]. Their adoption in information retrieval
systems of technical endeavor, such as the present work, has just started.

##  III Methodology

This section begins with an introduction to the dataset from an actual
petroleum field. Basic summary statistics and traditional NLP views of the
data are presented to illustrate the challenges with this technical language.
This section also includes a detailed description of the workflow with other
modeling decisions.

###  III-A Data

An energy company provided 9670 daily reports written by drilling engineers in
a field with 303 oil wells. These reports can be classified into two groups
according to the drilling state: productive and non-productive time. A
drilling operation is considered to be in non-productive time (NPT) when
actions are taken to solve an issue or accident. For example, drilling tools
can sometimes become stuck as a result of miscalculations and limited
understanding about the subsurface. In this case, the NPT includes the fishing
(or rescue) of the tool, as well as other minor adjustments before drilling
resumes.

In this study, 112 out of 303 wells contain NPT reports. The percentage
$1-\frac{112}{303}\approx 63\%$ is a rough estimate of the drilling operations
performance, which is considered to be low. Figure 1 illustrates the total NPT
duration and number of reports for the top 10 most inactive wells in the
field.

Figure 1: NPT duration (red) and number of reports (green) for the top 10 most
inactive wells in the field.

In Figure 1, well 44 was shut in as a result of unusually high pressure. This
action resulted in a multi-million dollar cost to the company and explains the
small number of NPT reports for this well. Well 83 provides an example of good
reporting practices given its short NPT duration. In general, a positive
correlation is expected between the number of reports and NPT duration, as
depicted with a simple regression model in Figure 2.

Figure 2: Simple regression model showing expected positive correlation.

The remaining 303 - 112 = 191 wells in the field contain thousands of
productive time (PT) daily reports with information that include equipment
inspection and physical measurements at depth.

Figure 3 illustrates the distribution of report lengths for both PT and NPT
calculated as the number of words in the text. At least two modes are observed
in the length distribution: the first mode at approximately 100 words is
associated with reports that only contain a summary paragraph; the second mode
at approximately 500 words is associated with reports that contain additional
remarks after a summary.

Figure 3: Distribution of reports lengths measured by number of words.

PT and NPT reports differ in their format, as shown by the samples in Figure
4. PT reports contain a short summary of the activity, followed by technical
remarks. NPT reports usually consist of a long summary that describes the
accident or issue in the drilling operation. Despite the format, these two
types of reports share context and therefore meaningful relationships.

Figure 4: Samples of PT and NPT reports illustrating their different format.

Figure 5 illustrates the word frequency cloud for all reports. The total
number of words (or tokens) is 859,361, and the vocabulary size is 22,337.
Several units, such as “psi”, “mTVD” and “mMD” are quite frequent in the
language, and target words, such as “incident” and “accident” are consistently
reported. Figure 6 shows the top 10 most frequent 3-grams and 4-grams in the
corpus. Sentences that contain shortcuts adopted by drillers in their daily
reports are highlighted.

Figure 5: Word cloud for all drilling reports showing high frequency of
physical units, acronyms, and abbreviations. The words “incident” and
“accident” are consistently reported. Figure 6: Top 10 most frequent 3-grams
and 4-grams showing unstructured sentences as the result of shortcuts adopted
by drillers in their day-to-day reporting activities.

Sentences extracted from NPT reports are labeled as EVENT, SYMPTOM, or ACTION
by a drilling expert. An event is defined as a major accident or failure in
the operation. For example, the drilling tool may become stuck in the pipe.
The driller can either take various actions after an event has happened, or be
proactive and modify the drilling plan before failure, based on observable
symptoms. Examples of symptoms include erratic torque during drilling, fluid
leakage, and over pressure. Inevitably, the labeling is unbalanced with 28%
EVENT, 15% SYMPTOM, and 57% ACTION, which reflects that events and symptoms
are usually followed by multiple actions during drilling. It also reflects
companies’ behavior in reporting operations. In this field, 20 companies co-
operate in the drilling cycle.

###  III-B Methods

Figure 7 illustrates the workflow for classification of sentences in drilling
reports. The process begins by extracting the daily operational notes from the
database of 303 wells and by concatenating the text in chronological order.
Next, the reports are cleaned with the regular expressions listed in Table I.
These expressions purge symbols that are considered to be meaningless in the
context in which they appear, and can be interpreted as a denoising layer.
After cleaning, the total number of tokens and the vocabulary size in the
corpus are reduced to T = 810,375 and V = 17,623, respectively.

Figure 7: Workflow for classification of sentences in drilling reports. TABLE I: Regular Expression Substitutions in Order of Execution. (Python syntax) FROM | TO | PURPOSE
---|---|---
`’,\s’` | `’ ’` | Commas at end of words
`’,([a-zA-Z])’` | `’ \1’` | Commas at end of words
`’\((.*?)\)’` | `’ \1 ’` | Enclosing parenthesis
`’\xe2\x80\xa2’` | `’ ’` | Bullet marks
`’-\s’` | `’ ’` | Dashes
`’==+|\*\*+’` | `’ ’` | Horizontal bars
`’\[(.*?)\]’` | ’ ` \1 ’` | Enclosing brackets
`’#|;’` | `’ ’` | Pounds and semicolons
`’_’` | `’ ’` | Underscores
`’\s/\s’` | `’ ’` | Orphan forward slashes

The Mikolov et al. methodology is strictly followed. The corpus is scanned
with a fixed window of size $m=3$, and each word $w_{i},\ i=1,2,\ldots,V$ in
the vocabulary is assigned two random vectors $u_{i},v_{i}\in[-1,1]^{d}$ with
$d=300$ the embedding dimension. The word $w_{i}$ can either be in the center
of a window, in which case $v_{i}$ is the associated vector representation, or
an outer (or target) word for which $u_{i}$ is looked up likewise. Within a
context window centered at a word $w_{c}$, a correct outer word $w_{o}$ is
sampled. In addition, $k=64$ words $w_{1},w_{2},\ldots,w_{k}$ are sampled from
the vocabulary at random from the unigram distribution $P(w)$. The probability
of the pair $(w_{c},w_{o})$ is maximized and the probability of the pairs
$(w_{c},w_{i}),\ i=1,2,\ldots,k$ is minimized, simultaneously, with the
objective function:

$J_{t}=\log{\sigma(u_{o}^{\top}v_{c})}+\sum_{i=1}^{k}\mathbb{E}_{w_{i}\sim
P(w)}\left[\log{\sigma(-u_{i}^{\top}v_{c})}\right]$ (1)

with $\sigma(x)=\frac{1}{1+e^{-x}}$ the sigmoid function. This process is
repeated for all context windows throughout the corpus leading to the total
objective $J=\sum_{t=1}^{T}J_{t}$. A batch of $b=128$ word pairs is processed
at a time during minimization. Word vectors are updated iteratively with
stochastic gradient descent and a learning rate of $lr=1.0$. At the end of the
minimization, the average loss is approximately $2.02$, and the resulting
embedding is illustrated in Figure 8 by means of a t-SNE projection [27]. All
hyper-parameters in the model were tuned by trial and error.

Figure 8: t-SNE projection of top 500 most frequent words showing concepts
like “integers”, “issues”, and “well trajectories”.

Vectors learned in this unsupervised stage cluster into concepts. As shown in
Figure 8, words such as “incident”, “accident”, “issues”, and “environmental”
appear together. Interesting patterns were observed, such as integers grouped
in ascending order. The word embedding is robust to noise, as shown by the
proximity of the words “remarks”, “remark”, “remarks:” Abbreviations are also
captured as in “circulate” and “circ”. All of these properties are extremely
relevant for overcoming nuances of technical languages.

To classify sentences in drilling reports, three neural network architectures
were tested: simple network with arithmetic averaging, convolutional neural
network (CNN), and long short-term memory network (LSTM).

#### Simple network with arithmetic averaging

In this architecture, fixed-length features are assigned to sentences by
averaging (i.e. reduction operation) their constituent word vectors. This
feature is then passed to a fully connected hidden layer with 20 _tanh_
neurons followed by a _softmax_ classification layer. Words that are not
present in the vocabulary are assigned the zero vector. The architecture is
illustrated in Figure 9.

#### Convolutional neural network

Input sentences are padded to have at least the maximum sentence length in all
drilling reports. The padding consists of introducing a special padding token
not present in the corpus. In this architecture, pre-trained word vectors are
passed to an embedding layer, which converts words in the vocabulary into the
corresponding word vectors. Next, convolution and max pooling layers are
interleaved twice in a total of four additional layers. The two convolutional
layers consist of 128 filters of length 3 and the two max pooling layers halve
their inputs. The architecture ends with a fully connected layer composed of
128 _ReLU_ neurons and a fully connected _softmax_ layer, as shown in Figure
10.

#### Long short-term memory network

Similar to the convolutional neural network, this architecture begins with an
embedding layer on padded input sentences. A long short-term memory layer with
100 neurons is appended, followed by 0.5 dropout. The architecture ends with a
fully connected _softmax_ layer, as shown in Figure 11.

Figure 9: Simple network with arithmetic averaging. Figure 10: Convolutional
neural network. Figure 11: Long short-term memory network.

The neural networks are trained on a set of labeled sentences provided by a
drilling expert. These sentences were extracted from NPT reports only. Each
labeled sentence is then preprocessed with the same regular expressions used
for cleaning the entire corpus; 80% of them are used for training and 20% are
saved for testing.

##  IV Results and discussion

In this section, sentences that are present in drilling reports are classified
as EVENT, SYMPTOM, or ACTION. Classification accuracy with 5-fold cross-
validation is presented in Table II for different neural network
architectures. The results confirm that LSTM is superior in the task, followed
by CNN and simple network with arithmetic averaging. The superior accuracy of
LSTM is explained by its capability of memorizing context information in the
input text, and is consistent with the deep NLP literature.

In Figure 12, classification of unseen sentences extracted from the test set
are shown along with the corresponding LSTM confusion matrix, which is a
better evaluation metric than accuracy for classification tasks with
unbalanced labels. Most misclassifications occur between symptoms and events,
which corroborates that the training set contains few examples of these
classes.

TABLE II: Sentence Classification Accuracy. Neural network architecture |  | Mean accuracy
---
(5-fold cross-validation)
Simple network with arithmetic averaging | 76.0%
Convolutional neural network | 77.3%
Long short-term memory network | 82.7%
Figure 12: Classification of a few unseen sentences extracted from the test
set and corresponding confusion matrix.

After training the LSTM network and testing it on the unseen labeled
sentences, the next step consists of feeding the network with all sentences in
NPT reports for the 303 wells in the field. The classification enables
queries, such as the example shown in Figure 13. In this plot, the most
problematic wells are selected according to the number of sentences classified
as events. In practice and more interestingly, the company that owns the field
or an agency investigating an accident can retrieve all wells for which a
specific symptom is followed by a specific action. The success rate of a
remediation action can be estimated, for example, by analyzing the various
outcomes in the classified sequence.

Figure 13: Selective extraction of most problematic wells per reporting
operator based on proposed classification.

In Figure 14, the NPT of two problematic wells is classified. Different
companies were confirmed to have different interests and roles when reporting
the same well. For example, on a normalized scale, the operator 19 reports
less actions than the other operators on the same well, which corroborates
internal business decisions. These illustrated differences in reporting
behavior are also affected by the drilling stage, depth at which reports were
written, and well location.

Figure 14: NPT sequencing for wells 83 and 23. Operators show different
reporting behavior according to their interests and roles.

##  V Conclusion

In this work, a methodology was presented for information retrieval in
drilling reports with deep natural language processing. The methodology was
tested with 9670 reports from 303 wells in an actual field, and promising NPT
sequencing results were obtained.

The skip-gram model with noise-contrastive estimation is capable of embedding
words from drilling reports successfully. The algorithm produces good
encodings, even with a small corpus, because of the repetitiveness of
sentences in the reports and the limited vocabulary of the language.

Sentence classification can be performed accurately while drilling. Among all
neural network architectures used in this work, LSTM is optimal for the task
given its ability to memorize context information. More effort is required
from energy companies, however, to refine the labeling and to begin
considering a collaborative project for corpus enlargement.

Having drilling reports classified automatically for thousands of wells can
help to mitigate accidents and improve our understanding of the sequences of
actions taken by drilling companies. The methodology can be used to establish
connections between classified text and other well data (e.g., well logs, rock
samples), and can be applied with little or no modification to other fields
that have a comparable number of technical reports and terms.

## Acknowledgment

The authors would like to thank the energy company that provided the data for
this study and the many developers leading the open source projects
TensorFlow111https://www.tensorflow.org and Keras222https://keras.io, both
used for the implementation of this work.

## References

  * [1] M. Sidahmed, C. J. Coley, and S. Shirzadi, “Augmenting operations monitoring by mining unstructured drilling reports,” _SPE Digital Energy Conference and Exhibition_ , 2015. [Online]. Available: http://dx.doi.org/10.2118/173429-MS
  * [2] M. Sanderson and W. B. Croft, “The history of information retrieval research,” _Proceedings of the IEEE_ , vol. 100, no. Special Centennial Issue, p. 1444–1451, May 2012. [Online]. Available: http://dx.doi.org/10.1109/JPROC.2012.2189916
  * [3] C. D. Manning, P. Raghavan, and H. Schütze, _Introduction to Information Retrieval_. Cambridge University Press, 2008. [Online]. Available: http://www.cambridge.org/gb/academic/subjects/computer-science/knowledge-management-databases-and-data-mining/introduction-information-retrieval
  * [4] S. Büttcher, C. L. A. Clarke, and G. V. Cormack, _Information Retrieval: Implementing and Evaluating Search Engines (MIT Press)_. The MIT Press, 2010.
  * [5] C. Y. Suen, “n-gram statistics for natural language understanding and text processing,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. PAMI-1, no. 2, p. 164–172, Apr 1979. [Online]. Available: http://dx.doi.org/10.1109/TPAMI.1979.4766902
  * [6] Y. Bengio, R. Ducharme, and P. Vincent, “A neural probabilistic language model,” in _NIPS_ , T. K. Leen, T. G. Dietterich, and V. Tresp, Eds. MIT Press, 2000, pp. 932–938.
  * [7] H. Schwenk, “Continuous space language models,” _Computer Speech & Language_, vol. 21, no. 3, p. 492–518, Jul 2007. [Online]. Available: http://dx.doi.org/10.1016/j.csl.2006.09.003
  * [8] T. Mikolov, S. Kombrink, L. Burget, J. Cernocky, and S. Khudanpur, “Extensions of recurrent neural network language model,” _2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2011. [Online]. Available: http://dx.doi.org/10.1109/ICASSP.2011.5947611
  * [9] S. Tellex, B. Katz, J. Lin, A. Fernandes, and G. Marton, “Quantitative evaluation of passage retrieval algorithms for question answering,” _Proceedings of the 26th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR ’03_ , 2003. [Online]. Available: http://dx.doi.org/10.1145/860435.860445
  * [10] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in _Advances in Neural Information Processing Systems 26_ , C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Eds. Curran Associates, Inc., 2013, pp. 3111–3119. [Online]. Available: http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
  * [11] G. Navarro, “A guided tour to approximate string matching,” _ACM Comput. Surv._ , vol. 33, no. 1, pp. 31–88, Mar. 2001. [Online]. Available: http://doi.acm.org/10.1145/375360.375365
  * [12] M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.” _Journal of Machine Learning Research_ , vol. 13, pp. 307–361, 2012. [Online]. Available: http://dblp.uni-trier.de/db/journals/jmlr/jmlr13.html#GutmannH12
  * [13] J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014. [Online]. Available: http://dx.doi.org/10.3115/v1/D14-1162
  * [14] N. Kalchbrenner, E. Grefenstette, and P. Blunsom, “A convolutional neural network for modelling sentences,” _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2014. [Online]. Available: http://dx.doi.org/10.3115/v1/P14-1062
  * [15] O. Irsoy and C. Cardie, “Opinion mining with deep recurrent neural networks,” _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014. [Online]. Available: http://dx.doi.org/10.3115/v1/D14-1080
  * [16] Y. Kim, “Convolutional neural networks for sentence classification,” _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014. [Online]. Available: http://dx.doi.org/10.3115/v1/D14-1181
  * [17] H. Rowley, S. Baluja, and T. Kanade, “Neural network-based face detection,” _IEEE Trans. Pattern Anal. Machine Intell._ , vol. 20, no. 1, p. 23–38, 1998\. [Online]. Available: http://dx.doi.org/10.1109/34.655647
  * [18] S. Lawrence, C. Giles, A. C. Tsoi, and A. Back, “Face recognition: a convolutional neural-network approach,” _IEEE Transactions on Neural Networks_ , vol. 8, no. 1, p. 98–113, 1997. [Online]. Available: http://dx.doi.org/10.1109/72.554195
  * [19] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 35, no. 1, p. 221–231, Jan 2013. [Online]. Available: http://dx.doi.org/10.1109/TPAMI.2012.59
  * [20] A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, and J. Schmidhuber, “A novel connectionist system for unconstrained handwriting recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 31, no. 5, p. 855–868, May 2009. [Online]. Available: http://dx.doi.org/10.1109/TPAMI.2008.137
  * [21] V. Frinken, A. Fischer, R. Manmatha, and H. Bunke, “A novel word spotting method based on recurrent neural networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 34, no. 2, p. 211–224, Feb 2012. [Online]. Available: http://dx.doi.org/10.1109/TPAMI.2011.113
  * [22] J. Johnson, A. Karpathy, and L. Fei-Fei, “Densecap: Fully convolutional localization networks for dense captioning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [23] R. Józefowicz, W. Zaremba, and I. Sutskever, “An empirical exploration of recurrent network architectures.” in _ICML_ , ser. JMLR Workshop and Conference Proceedings, F. R. Bach and D. M. Blei, Eds., vol. 37. JMLR.org, 2015, pp. 2342–2350. [Online]. Available: http://dblp.uni-trier.de/db/conf/icml/icml2015.html#JozefowiczZS15
  * [24] I. Sutskever, J. Martens, and G. E. Hinton, “Generating text with recurrent neural networks,” in _Proceedings of the 28th International Conference on Machine Learning (ICML-11)_ , 2011, pp. 1017–1024.
  * [25] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, p. 1735–1780, Nov 1997. [Online]. Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735
  * [26] T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Cernocký, “Empirical evaluation and combination of advanced language modeling techniques.” in _INTERSPEECH_. ISCA, 2011, pp. 605–608. [Online]. Available: http://dblp.uni-trier.de/db/conf/interspeech/interspeech2011.html#MikolovDKBC11
  * [27] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of Machine Learning Research_ , vol. 9, pp. 2579–2605, Nov. 2008\. [Online]. Available: http://www.jmlr.org/papers/v9/vandermaaten08a.html

|  Júlio Hoffimann received a B.Sc. degree in mechanical engineering
(computational mechanics) and a M.Sc. degree in civil engineering from Federal
University of Pernambuco, Brazil in 2011 and 2014, respectively. He is
currently a Ph.D. candidate in the Energy Resources Engineering department at
Stanford University where he is investigating (under a SwB fellowship)
stochastic models of landscape evolution. His research interests include
geostatistics, machine learning, convex optimization, computer vision, and
high-performance computing.
---|---
|  Youli Mao received a BS degree in applied mathematics from Sichuan
University, China, in 2009 and a PhD degree in applied mathematics from Texas
A&M University in 2014. He is currently a graduate student in the Geology and
Geophysics department at Texas A&M University. His research interests include
the integration of fluid mechanics and geomechanics, machine learning in
geophysics and data science.
---|---
|  Avinash Wesley received a BS degree in computer science from St. Stephen’s
College, Delhi India, in 2006, a MS degree in computer science from the
University of Houston in 2010, and a PhD degree in computer science from the
University of Houston in 2015. He is currently a senior technologist at
Halliburton. His interests are in data science, IoT, and deep learning. He
also develops real-time solutions for drilling and production operations in
the oil and gas industry.
---|---
|  Aimee Taylor received a BA degree in economics from University of Leeds and
a BSc degree in geology from Durham University. She is currently a senior
product specialist for real-time analytics and predictive analytics at
Halliburton, focusing on developing automated processes to facilitate and
streamline operations.
---|---