# Identifying and Understanding User Reactions to
Deceptive and Trusted Social News Sources

Maria Glenski Tim Weninger
Computer Science and Engineering
University of Notre Dame
Notre Dame, IN 46556
{mglenski<EMAIL_ADDRESS>
Svitlana Volkova
Data Sciences and Analytics Group
Pacific Northwest National Laboratory
Richland, WA 99354
<EMAIL_ADDRESS>

###### Abstract

In the age of social news, it is important to understand the types of
reactions that are evoked from news sources with various levels of
credibility. In the present work we seek to better understand how users react
to trusted and deceptive news sources across two popular, and very different,
social media platforms. To that end, (1) we develop a model to classify user
reactions into one of nine types, such as answer, elaboration, and question,
etc, and (2) we measure the speed and the type of reaction for trusted and
deceptive news sources for 10.8M Twitter posts and 6.2M Reddit comments. We
show that there are significant differences in the speed and the type of
reactions between trusted and deceptive news sources on Twitter, but far
smaller differences on Reddit.

##  1 Introduction

As the reliance on social media as a source of news increases and the
reliability of sources is increasingly debated, it is important to understand
how users react to various sources of news. Most studies that investigate
misinformation spread in social media focus on individual events and the role
of the network structure in the spread Qazvinian et al. (2011); Wu et al.
(2015); Kwon et al. (2017) or detection of false information Rath et al.
(2017). These studies have found that the size and shape of misinformation
cascades within a social network depends heavily on the initial reactions of
the users. Other work has focused on the language of misinformation in social
media Rubin et al. (2016); Rashkin et al. (2017); Mitra et al. (2017); Wang
(2017); Karadzhov et al. (2017); Volkova et al. (2017) to detect types of
deceptive news.

As an alternative to studying newsworthy events one at a time Starbird (2017),
the current work applies linguistically-infused models to predict user
reactions to deceptive and trusted news sources. Our analysis reveals
differences in reaction types and speed across two social media platforms —
Twitter and Reddit.

The first metric we report is the reaction type. Recent studies have found
that 59% of bitly-URLs on Twitter are shared without ever being read Gabielkov
et al. (2016), and 73% of Reddit posts were voted on without reading the
linked article Glenski et al. (2017). Instead, users tend to rely on the
commentary added to retweets or the comments section of Reddit-posts for
information on the content and its credibility. Faced with this reality, we
ask: what kind of reactions do users find when they browse sources of varying
credibility? Discourse acts, or speech acts, can be used to identify the use
of language within a conversation, e.g., agreement, question, or answer.
Recent work by Zhang et al. Zhang et al. (2017) classified Reddit comments by
their primary discourse act (e.g., question, agreement, humor), and further
analyzed patterns from these discussions.

The second metric we report is reaction speed. A study by Jin et al. Jin et
al. (2013) found that trusted news stories spread faster than misinformation
or rumor; Zeng et al. Zeng et al. (2016) found that tweets which deny rumors
had shorter delays than tweets of support. Our second goal is to determine if
these trends are maintained for various types of news sources on Twitter and
Reddit.

Hence, the contributions of this work are two-fold: (1) we develop a
linguistically-infused neural network model to classify reactions in social
media posts, and (2) we apply our model to label 10.8M Twitter posts and 6.2M
Reddit comments in order to evaluate the speed and type of user reactions to
various news sources.

##  2 Reaction Type Classification

In this section, we describe our approach to classify user reactions into one
of eight types of discourse: agreement, answer, appreciation, disagreement,
elaboration, humor, negative reaction, or question, or as none of the given
labels, which we call “other”, using linguistically-infused neural network
models.

###  2.1 Reddit Data

We use a manually annotated Reddit dataset from Zhang et al. Zhang et al.
(2017) to train our reaction classification model. Annotations from 25 crowd-
workers labelled the primary discourse act for 101,525 comments within 9,131
comment threads on Reddit. The Reddit IDs, but not the text content of the
comments themselves, were released with the annotations. So we collected the
content of Reddit posts and comments from a public archive of Reddit posts and
comments.111bigquery.cloud.google.com/dataset/fh-
bigquery:reddit_posts|reddit_comments Some content was deleted prior to
archival, so the dataset shown in Table 1 is a subset of the original content.
Despite the inability to capture all of the original dataset, Table 1 shows a
similar distribution between our dataset and the original.

| Zhang et al. | Present work
---|---|---
Reaction Type | # | % | # | %
agreement | 5,054 | 4.73 | 3,857 | 4.61
answer | 41,281 | 38.63 | 32,561 | 38.94
appreciation | 8,821 | 8.25 | 6,973 | 8.34
disagreement | 3,430 | 3.21 | 2,654 | 3.17
elaboration | 19,315 | 18.07 | 14,966 | 17.90
humor | 2,358 | 2.21 | 1,878 | 2.25
negative reaction | 1,901 | 1.78 | 1,473 | 1.76
other | 1,979 | 1.85 | 1,538 | 1.84
question | 10,568 | 9.89 | 8,194 | 9.80
no majority label | 12,162 | 11.38 | 9532 | 11.40
Total | 106,869 | 100 | 83, 626 | 100
Table 1: Summary of the training data we recovered compared to the data
collected by Zhang et al. Zhang et al. (2017) reported as distributions of
comments across reaction types.

###  2.2 Model

We develop a neural network architecture that relies on content and other
linguistic signals extracted from reactions and parent posts, and takes
advantage of a “late fusion” approach previously used effectively in vision
tasks Karpathy et al. (2014); Park et al. (2016). More specifically, we
combine a text sequence sub-network with a vector representation sub-network
as shown in Figure 1. The text sequence sub-network consists of an embedding
layer initialized with 200-dimensional GloVe embeddings Pennington et al.
(2014) followed by two 1-dimensional convolution layers, then a max-pooling
layer followed by a dense layer. The vector representation sub-network
consists of two dense layers. We incorporate information from both sub-
networks through concatenated padded text sequences and vector representations
of normalized Linguistic Inquiry and Word Count (LIWC) features Pennebaker et
al. (2001) for the text of each post and its parent.

Predicted LabelProbability Activation Layer (soft max)Dense Layer(100
units)Tensor ConcatenationMax Pooling Layer(1D, Pool Size=3)Convolutional
Layer (100 units)Convolutional Layer (100 units)Embedding Layer(200 units)Word
SequencesDense Layer(100 units)Dense Layer(100 units)LIWC FeaturesVector Sub-
NetworkText Sub-Network Figure 1: Architecture of neural network model used to
predict reaction types.

###  2.3 Reaction Type Classification Results

agreementanswerappreciationdisagreementelaborationhumornegative
reactionquestion$0$$0.25$$0.5$$0.75$$1$F1 ScoreF1 Scores by Reaction-Type

Zhang et al. Zhang et al. (2017)Present work

Figure 2: Comparison of our model’s performance, measured using F1 score,
trained only on content features, with the performance reported by Zhang et
al. Zhang et al. (2017) trained on content, author, thread, structure, and
community features.

As shown in Figure 2, our linguistically-infused neural network model that
relies solely on the content of the reaction and its parent has comparable
performance to the more-complex CRF model by Zhang et al. Zhang et al. (2017),
which relies on content as well as additional metadata like the author, thread
(e.g., the size of the the thread, the number of branches), structure (e.g.,
the position within the thread), and community (i.e., the subreddit in which
the comment is posted).

##  3 Measuring Reactions to Trusted and Deceptive News Sources

In this section, we present key results of our analysis of how often and how
quickly users react to content from sources of varying credibility using the
reaction types predicted by our linguistically-infused neural network model.

###  3.1 Twitter and Reddit News Data

We focus on trusted news sources that provide factual information with no
intent to deceive and deceptive news sources. Deceptive sources are ranked by
their intent to deceive as follows: clickbait (attention-grabbing, misleading,
or vague headlines to attract an audience), conspiracy theory (uncorroborated
or unreliable information to explain events or circumstances), propaganda
(intentionally misleading information to advance a social or political
agenda), and disinformation (fabricated or factually incorrect information
meant to intentionally deceive readers).

Trusted, clickbait, conspiracy, and propaganda sources were previously
compiled by Volkova et al. Volkova et al. (2017) through a combination of
crowd-sourcing and public resources. Trusted news sources with Twitter-
verified accounts were manually labeled and clickbait, conspiracy, and
propaganda news sources were collected from several public resources that
annotate suspicious news accounts222Example resources used by Volkova et al
Volkova et al. (2017) to compile deceptive news sources:
http://www.fakenewswatch.com/, http://www.propornot.com/p/the-list.html and
others.. We collected news sources identified as spreading disinformation by
the European Union’s East Strategic Communications Task Force from
euvsdisinfo.eu. In total, there were 467 news sources: 251 trusted and 216
deceptive.

We collected reaction data for two popular platforms, Reddit and Twitter,
using public APIs over the 13 month period from January 2016 through January
2017. For our Reddit dataset, we collected all Reddit posts submitted during
the 13 month period that linked to domains associated with one of our labelled
news sources. Then we collected all comments that directly responded to those
posts. For our Twitter dataset, we collected all tweets posted in the 13 month
period that explicitly @mentioned or directly retweeted content from a source
and then assigned a label to each tweet based on the class of the source
@mentioned or retweeted. A breakdown of each dataset by source type is shown
in Table 2. Figure 3 illustrates the distribution of deceptive news sources
and reactions across the four sub-categories of deceptive news sources. In our
analysis, we consider the set of all deceptive sources and the set excluding
the most extreme (disinformation).

Reddit Dataset
---
Type | # Sources | # Comments
Trusted | 169 | 5,429,694
Deceptive (no disinfo) | 128 | 664,670
Deceptive | 179 | 795,591
Total | 348 | 6,225,285
Twitter Dataset
Type | # Sources | # Tweets
Trusted | 182 | 6,567,002
Deceptive (no disinfo) | 100 | 775,844
Deceptive | 150 | 4,263,576
Total | 232 | 10,830,578
Table 2: Summary of Twitter and Reddit datasets used to measure the speed and
types of reactions to Trusted and Deceptive news sources excluding (no
disinformation) or including (All) the most extreme of the deceptive sources —
those identified as spreading disinformation.

$20$$40$$60$$80$# sources

ClickbaitConspiracyPropagandaDisinformation$10^{5}$$10^{6}$# reactions

Reddit DatasetTwitter Dataset

Figure 3: Distributions of Deceptive news sources and reactions to those
sources (Reddit comments or tweets, respectively) for the Reddit and Twitter
datasets across the four sub-categories of deceptive news sources.

###  3.2 Methodology

We use the linguistically-infused neural network model from Figure 1 to label
the reaction type of each tweet or comment. Using these labels, we examine how
often response types occur when users react to each type of news source. For
clarity, we report the five most frequently occurring reaction types
(expressed in at least 5% of reactions within each source type) and compare
the distributions of reaction types for each type of news source.

To examine whether users react to content from trusted sources differently
than from deceptive sources, we measure the reaction delay, which we define as
the time elapsed between the moment the link or content was posted/tweeted and
the moment that the reaction comment or tweet occurred. We report the
cumulative distribution functions (CDFs) for each source type and use Mann
Whitney U (MWU) tests to compare whether users respond with a given reaction
type with significantly different delays to news sources of different levels
of credibility.

$0$$20$$40$$60$% commentsReddit

answerappreciationelaborationquestionother$0$$20$$40$$60$% tweetsTwitter

TrustedDeceptive (no Disinformation)Deceptive

Figure 4: Distributions of five most frequently occurring reaction types
within comments on Reddit and tweets on Twitter for each news source type (MWU
$p<0.01$).

$1$$5$$10$$15$$20$$25$$30$$35$$0.2$$0.4$$0.6$$0.8$$1$% reactionsAnswer

$1$$5$$10$$15$$20$$25$$30$$35$$0.2$$0.4$$0.6$$0.8$$1$% reactionsAppreciation

$1$$5$$10$$15$$20$$25$$30$$35$$0.2$$0.4$$0.6$$0.8$$1$% reactionsElaboration

$1$$5$$10$$15$$20$$25$$30$$35$$0.2$$0.4$$0.6$$0.8$$1$% reactionsQuestion

$1$$5$$10$$15$$20$$25$$30$$35$$0.2$$0.4$$0.6$$0.8$$1$Reaction Delay in Hours%
reactionsOther

TrustedDeceptive (no Disinformation)Deceptive

RedditTwitter

Figure 5: CDF plots of the volumes of reactions by reaction delays for the
frequently occurring reactions (i.e., , reactions that occur in at least 5% of
comments) for each source-type, using a step size of one hour. The CDF for
Elaboration-reactions to Deceptive (no disinformation) Twitter news sources is
occluded by the CDF for Deceptive Twitter news sources. This figure is best
viewed in color.

###  3.3 Results and Discussion

For both Twitter and Reddit datasets, we found that the primary reaction types
were answer, appreciation, elaboration, question, or “other” (no label was
predicted). Figure 4 illustrates the distribution of reaction types among
Reddit comments (top plot) or tweets (bottom plot) responding to each type of
source, as a percentage of all comments/tweets reacting to sources of the
given type (i.e., trusted, all deceptive, and deceptive excluding
disinformation sources).

For Twitter, we report clear differences in user reactions to trusted vs.
deceptive sources. Deceptive (including disinformation) sources have a much
higher rate of appreciation reactions and a lower rate of elaboration
responses, compared to trusted news sources. Differences are still significant
($p<0.01$) but the trends reverse if we do not include disinformation sources.
We also see an increase in the rate of question-reactions compared to trusted
news sources if we exclude disinformation sources.

For Reddit, there appears to be a very similar distribution across reaction
types for trusted and deceptive sources. However, MWU tests still found that
the differences between trusted and deceptive news sources were statistically
significant ($p<0.01$) — regardless of whether we include or exclude
disinformation sources. Posts that link to deceptive sources have higher rates
of question, appreciation, and answering reactions, while posts that link to
trusted sources have higher rates of elaboration, agreement, and disagreement.

Next, we compared the speed with which users reacted to posts of sources of
varying credibility. Our original hypothesis was that users react to posts of
trusted sources faster than posts of deceptive sources. The CDFs for each
source type and platform (solid and dashed lines represent Reddit and Twitter
respectively) are shown in Figure 5. We observe that the lifetime of direct
reactions to news sources on Twitter is often more extended than for sources
on Reddit. One exception is answer reactions which almost always occur within
the first hour after the Twitter new source originally posted the tweet being
answered. This may be due to the different ways that users consume content on
the two platforms. Users follow accounts on Twitter, whereas on Reddit users
“follow” topics through their subscriptions to various subreddits. Users can
view the news feeds of individual sources on Twitter and view all of the
sources’ posts. Reddit, on the other hand, is not designed to highlight
individual users or news sources; instead new posts (regardless of the source)
are viewed based on their hotness score within each subreddit.

In addition, we observe that reactions to posts linked to trusted sources are
less heavily concentrated within the first 12 to 15 hours of the post’s
lifetime on Reddit. The opposite is found on Twitter. Twitter sources may have
a larger range of reaction delays, but they are also more heavily concentrated
in the lower end of that range ($p<0.01$).

##  4 Related Work

As we noted above, most studies that examine misinformation spread focus on
individual events such as natural disasters Takahashi et al. (2015), political
elections Ferrara (2017), or crises Starbird et al. (2014) and examine the
response to the event on social media. A recent study by Vosoughi et al.
Vosoughi et al. (2018) found that news stories that were fact-checked and
found to be false spread faster and to more people than news items found to be
true. In contrast, our methodology considers immediate reactions to news
sources of varying credibility, so we can determine whether certain reactions
or reactions to trusted or deceptive news sources evoke more or faster
responses from social media users.

##  5 Conclusion

In the current work, we have presented a content-based model that classifies
user reactions into one of nine types, such as answer, elaboration, and
question, etc., and a large-scale analysis of Twitter posts and Reddit
comments in response to content from news sources of varying credibility.

Our analysis of user reactions to trusted and deceptive sources on Twitter and
Reddit shows significant differences in the distribution of reaction types for
trusted versus deceptive news. However, due to differences in the user
interface, algorithmic design, or user-base, we find that Twitter users react
to trusted and deceptive sources very differently than Reddit users. For
instance, Twitter users questioned disinformation sources less often and more
slowly than they did trusted news sources; Twitter users also expressed
appreciation towards disinformation sources more often and faster than towards
trusted sources. Results from Reddit show similar, but far less pronounced,
reaction results.

Future work may focus on analysis of reaction behavior from automated (i.e.,
’bot’), individual, or organization accounts; on additional social media
platforms and languages; or between more fine-grained categories of news
source credibility.

## Acknowledgments

The research described in this paper is based on Twitter and Reddit data
collected by the University of Notre Dame using public APIs. The research was
supported by the Laboratory Directed Research and Development Program at
Pacific Northwest National Laboratory, a multiprogram national laboratory
operated by Battelle for the U.S. Department of Energy. This research is also
supported by the Defense Advanced Research Projects Agency (DARPA), contract
W911NF-17-C-0094. The U.S. Government is authorized to reproduce and
distribute reprints for Governmental purposes notwithstanding any copyright
annotation thereon. The views and conclusions contained herein are those of
the authors and should not be interpreted as necessarily representing the
official policies or endorsements, either expressed or implied, of DARPA or
the U.S. Government.

## References

  * Ferrara (2017) Emilio Ferrara. 2017.  Disinformation and social bot operations in the run up to the 2017 French presidential election.  First Monday 22(8).
  * Gabielkov et al. (2016) Maksym Gabielkov, Arthi Ramachandran, Augustin Chaintreau, and Arnaud Legout. 2016\.  Social clicks: What and who gets read on Twitter?  In ACM SIGMETRICS/IFIP Performance.
  * Glenski et al. (2017) Maria Glenski, Corey Pennycuff, and Tim Weninger. 2017.  Consumers and curators: Browsing and voting patterns on Reddit.  IEEE Transactions on Computational Social Systems 4(4):196–206.
  * Jin et al. (2013) Fang Jin, Edward Dougherty, Parang Saraf, Yang Cao, and Naren Ramakrishnan. 2013\.  Epidemiological modeling of news and rumors on Twitter.  In Proceedings of the 7th Workshop on Social Network Mining and Analysis. page 8.
  * Karadzhov et al. (2017) Georgi Karadzhov, Pepa Gencheva, Preslav Nakov, and Ivan Koychev. 2017.  We built a fake news & click-bait filter: What happened next will blow your mind!  In Proceedings of the International Conference on Recent Advances in Natural Language Processing.
  * Karpathy et al. (2014) Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014.  Large-scale video classification with convolutional neural networks.  In Proceedings of CVPR. pages 1725–1732.
  * Kwon et al. (2017) Sejeong Kwon, Meeyoung Cha, and Kyomin Jung. 2017.  Rumor detection over varying time windows.  PloS one 12(1):e0168344.
  * Mitra et al. (2017) Tanushree Mitra, Graham P Wright, and Eric Gilbert. 2017.  A parsimonious language model of social media credibility across disparate events.  In Proceedings of the ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW). ACM, pages 126–145.
  * Park et al. (2016) Eunbyung Park, Xufeng Han, Tamara L Berg, and Alexander C Berg. 2016.  Combining multiple sources of knowledge in deep CNNs for action recognition.  In Proceedings of AWACV. pages 1–8.
  * Pennebaker et al. (2001) James W Pennebaker, Martha E Francis, and Roger J Booth. 2001.  Linguistic inquiry and word count: Liwc.  Mahway: Lawrence Erlbaum Associates .
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global vectors for word representation.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pages 1532–1543.
  * Qazvinian et al. (2011) Vahed Qazvinian, Emily Rosengren, Dragomir R Radev, and Qiaozhu Mei. 2011.  Rumor has it: Identifying misinformation in microblogs.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pages 1589–1599.
  * Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017\.  Truth of varying shades: Analyzing language in fake news and political fact-checking.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing. pages 2921–2927.
  * Rath et al. (2017) Bhavtosh Rath, Wei Gao, Jing Ma, and Jaideep Srivastava. 2017.  From retweet to believability: Utilizing trust to identify rumor spreaders on Twitter.  In Advances in Social Networks Analysis and Mining (ASONAM).
  * Rubin et al. (2016) Victoria L Rubin, Niall J Conroy, Yimin Chen, and Sarah Cornwell. 2016.  Fake news or truth? Using satirical cues to detect potentially misleading news.  In Proceedings of NAACL-HLT. pages 7–17.
  * Starbird (2017) Kate Starbird. 2017.  Examining the alternative media ecosystem through the production of alternative narratives of mass shooting events on Twitter.  In Proceedings of International AAAI Conference on Web and Social Media. pages 230–239.
  * Starbird et al. (2014) Kate Starbird, Jim Maddock, Mania Orand, Peg Achterman, and Robert M Mason. 2014\.  Rumors, false flags, and digital vigilantes: Misinformation on twitter after the 2013 boston marathon bombing.  iConference 2014 Proceedings .
  * Takahashi et al. (2015) Bruno Takahashi, Edson C Tandoc, and Christine Carmichael. 2015.  Communicating on twitter during a disaster: An analysis of tweets during typhoon haiyan in the philippines.  Computers in Human Behavior 50:392–398.
  * Volkova et al. (2017) Svitlana Volkova, Kyle Shaffer, Jin Yea Jang, and Nathan Hodas. 2017.  Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on twitter.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. pages 647–653.
  * Vosoughi et al. (2018) Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018.  The spread of true and false news online.  Science 359(6380):1146–1151.
  * Wang (2017) William Yang Wang. 2017.  ”Liar, liar pants on fire”: A new benchmark dataset for fake news detection.
  * Wu et al. (2015) Ke Wu, Song Yang, and Kenny Q Zhu. 2015.  False rumors detection on sina weibo by propagation structures.  In 31st International Conference on Data Engineering (ICDE). IEEE, pages 651–662.
  * Zeng et al. (2016) Li Zeng, Kate Starbird, and Emma S Spiro. 2016.  Rumors at the speed of light? modeling the rate of rumor transmission during crisis.  In 49th Hawaii International Conference on System Sciences (HICSS). IEEE, pages 1969–1978.
  * Zhang et al. (2017) Amy Zhang, Bryan Culbertson, and Praveen Paritosh. 2017.  Characterizing online discussion using coarse discourse sequences.  In Proceedings of International AAAI Conference on Web and Social Media.


# Toward the Automated Construction of Probabilistic Knowledge Graphs for the
Maritime Domain

1st Fatemeh Shiri, 4th Teresa Wang
5th Shirui Pan, 6th Xiaojun Chang
7th Yuan-Fang Li, 8th Reza Haffari  Department of Data Science and AI
Faculty of Information Technology
Monash University
Melbourne, Australia
<EMAIL_ADDRESS>2nd Van Nguyen  Intelligence, Surveillance and
Space Division
Defence Science and Technology Group
Adelaide, Australia
<EMAIL_ADDRESS>3rd Shuang Yu * now with IBM Australia Monash
University*
Melbourne, Australia
<EMAIL_ADDRESS>
###### Abstract

International maritime crime is becoming increasingly sophisticated, often
associated with wider criminal networks. Detecting maritime threats by means
of fusing data purely related to physical movement (i.e., those generated by
physical sensors, or hard data) is not sufficient. This has led to research
and development efforts aimed at combining hard data with other types of data
(especially human-generated or soft data). Existing work often assumes that
input soft data is available in a structured format, or is focused on
extracting certain relevant entities or concepts to accompany or annotate hard
data. Much less attention has been given to extracting the rich knowledge
about the situations of interest implicitly embedded in the large amount of
soft data existing in unstructured formats (such as intelligence reports and
news articles). In order to exploit the potentially useful and rich
information from such sources, it is necessary to extract not only the
relevant entities and concepts, but also their semantic relations, together
with the uncertainty associated with the extracted knowledge (i.e., in the
form of probabilistic knowledge graphs). This will increase the accuracy of,
and confidence in, the extracted knowledge and facilitate subsequent reasoning
and learning. To this end, we propose Maritime DeepDive, an initial prototype
for the automated construction of probabilistic knowledge graphs from natural
language data for the maritime domain. In this paper, we report on the current
implementation of Maritime DeepDive, together with preliminary results on
extracting probabilistic events from maritime piracy incidents. This pipeline
was evaluated on a manually crafted gold standard, yielding promising results.

###### Index Terms:

Knowledge Graph, Maritime, DeepDive, Information Extraction, Relation
Extraction

Figure 1: Overview of the proposed framework, Maritime DeepDive.

##  I Introduction and Background

Maritime security has increasingly been recognised as a major global security
concern. Due to the increased shipping traffic (i.e., 80$\%$ of all world
trade is transported by sea, and seaborn trade has doubled every decade since
1945 [1]) and lack of agreed jurisdiction for maritime enforcement, maritime
security threats such as piracy have been on the rise. This poses a
significant threat to the economy (e.g., piracy worldwide has cost an average
of tens of billion US dollars a year), to humanitarianism (e.g., 3,862
seafarers were attacked and 561 hostages captured in Somali in 2011), and to
our environment (e.g., pollution damage caused by oil and chemical spills)
[2].

Due to the vast territorial waters involved, patrolling using military
resources (such as the EU’s Operation Atlanta111Operation ATALANTA.
https://eunavfor.eu/., NATO’s Operation Ocean Shield222Operation OCEAN SHIELD.
https://mc.nato.int/missions/operation-oceanshield. and the US-led Combined
Task Force-151333CTF 151: Counter Piracy.
https://combinedmaritimeforces.com/ctf-151-counter-piracy/.) is only partially
successful [3]. In addition, organised maritime crime is often associated with
wider criminal networks, e.g., there are connections between organised piracy
and wider criminal networks and corruption on land [4]. As such, many
associated criminal activities can be partially or completely carried out
through natural language communications (e.g., information manipulation or
attack planning) [5], and thus may not be detectable by purely physical
movement data provided by physical sensors.

Much research and development has been directed toward the automated
processing, fusion and reasoning with diverse data sources for maritime
surveillance to assist human analysts with decision making. In order to make
sense of complex scenarios to detect maritime threats, and for situational
awareness in general, it has been increasingly recognised that the fusion of
data relating to physical movement, such as that generated by physical sensors
(i.e., hard data, which has been extensively studied for sensor data fusion),
is not sufficient, but requires all available information, including data
provided by humans and intelligence (i.e., soft data) as well as other
operational and contextual information [5, 6]. Efforts to incorporate soft
data into fusion frameworks for situational awareness have considered various
types of soft data, including extant textual data in a structured or logical
format [7, 8]; text annotations associated with hard data that potentially
provide additional attributes for the relevant entities [9]; certain entities
and concepts of interest [10, 11, 12, 13, 14] and a few simple relations
between them [15] extracted from short human statements [12], social network
data [13], semi-structured synthesised reports [14], maritime incident reports
[10] and news articles [15]; and also semantic knowledge constructed from
natural language text (albeit, without taking into account the _uncertainty_
associated with the extracted knowledge) [16, 17].

To this end, Reasoning under Uncertainty with Soft and Hard data (RUSH) [18,
19] was proposed as a framework for reasoning and learning from diverse
sources for real-world situational awareness. With respect to soft data in
particular, RUSH is motivated by harnessing human knowledge implicitly
embedded in abundant natural language data often existing in many real-world
scenarios (e.g., news articles and intelligence reports). This form of complex
soft data is unstructured and potentially contains substantial knowledge about
a (maritime) situation. This includes certain entities or spatial/temporal
concepts of interest, and expressing the relationships between them with a
clear and coherent semantics. This facilitates subsequent reasoning and
learning while managing the associated uncertainty (crucial for handling noisy
data such as natural language), and enables explanation and provenance
(essential for informing decision-making). In other words, we are interested
in constructing _probabilistic knowledge graphs_ , where each piece of
extracted knowledge is associated with a marginal probability. This paper
discusses our initial effort toward achieving this goal, that is, extracting
probabilistic knowledge for maritime events from piracy incident reports.

##  II Knowledge graph construction

Knowledge graphs are used by many information systems that require access to
structured knowledge. Currently, the most popular knowledge graphs in the
research context (e.g. Google [20], Freebase [21], YAGO [22], and DBpedia
[23]) are designed to be generic. These general-purpose knowledge graphs
consist of several million entities and relations. They are typically wide-
breadth, large-scale, and exhibit a high level of automation and openness.
Despite these favourable characteristics, their application to specific
domains is limited due to a lack of domain-specific knowledge, resulting in a
low degree of accuracy and consequently an inability to reason effectively.

Due to the limitations of generic knowledge graphs, domain-specific knowledge
graphs are necessary for critical applications. Such knowledge graphs are
based on industry-specific ontologies and data, and generally achieve higher
accuracy in their target domains. They are typically designed for knowledge
management, data governance and aggregation of industry information, and
assist with various AI and machine learning tasks. For instance, in the field
of medical sciences, well-known domain-specific knowledge graphs include
“Knowlife” [24] and a knowledge graph of disease symptoms [25]. There have
also been knowledge graphs constructed in other industries, such as an
academic knowledge graph[26], and the knowledge graph on education “KnowEdu”
[27].

In respect to the maritime domain specifically, there are a few studies on
knowledge graphs and their construction [28, 29]; for instance [28] is
concerned with knowledge graphs for maritime dangerous goods, and for the
prediction of vessel locations. However, these efforts are either aimed at
constructing knowledge graphs from hard (i.e., AIS) data (as in [29]); or
focused on extracting information from soft data in a structured format (e.g.,
the International Maritime Dangerous Goods Code) rather than from unstructured
data such as textual reports (as in [28]). In addition, due to the need for
domain-specific data and the strong dependence on experts in the industry, the
construction of comprehensive domain-specific knowledge graphs is not
straightforward and can be very time consuming. To this end, we propose
Maritime DeepDive, a framework that facilitates the construction of a
(probabilistic) knowledge graph from natural language data. Maritime DeepDive
improves on existing work in that the system extracts not only maritime
entities of interest and their attributes, but also their semantic relations,
dealing with uncertainty and computing the degree of confidence associated
with the extracted knowledge. We envisage that Maritime Deepdive will be able
to assist human experts by automating many tasks through machine-human
interactions.

##  III Maritime DeepDive

In this section we describe our approach, Maritime DeepDive, and provide a
demonstration of events and activities of interest extracted from maritime
piracy incident reports.

To build a comprehensive knowledge graph, we employ the publicly available
platform DeepDive [30] which was developed for efficient and interpretable
machine learning and statistical inference to facilitate explainability and
provide provenance. DeepDive can effectively address the problems of
extraction, cleaning, and integration jointly, and treat them as a single
probabilistic inference problem that takes all available information into
account. Specifically, DeepDive was formulated on top of a large-scale
inference engine based on Markov Logic [31], currently used as part of the
high-level fusion and reasoning capability within RUSH for real–world
situational awareness [31, 19]. This in effect allows Maritime DeepDive to (i)
model the complexity and uncertainty of the extracted knowledge from natural
language, (ii) combine learning from both expert knowledge and available data
(via distant supervision), and (iii) facilitate future combination with
probabilistic knowledge and events generated from hard data within RUSH [19]
in a consistent and coherent manner. Fig 1 demonstrates the architecture of
our end-to-end framework.

To build the maritime knowledge graph in Maritime DeepDive, we carry out a few
data preprocessing steps. The information from different sources is collated
and preprocessed to ensure integrity, and then merged. The next step is to add
Natural Language Processing (NLP) markups and extract candidate relation
mentions together with a sparse feature representation for each candidate. The
extracted candidate relation mentions include mentions associated with various
entities of interest, their roles and potential relationships between
candidates. This information is used in the heuristic rules we developed to
perform distant supervision. Finally, a high-level configuration of our model
is specified, so the maritime probabilistic knowledge graph can be
(automatically) constructed. The following subsections provide details of the
components of this architecture and discuss how they are applied to build a
knowledge graph for the maritime domain.

Figure 2: An example graph of a piracy report modeled in SEM including role
types

###  III-A Data Sources

We obtained relevant data from two main sources: (a) the Worldwide Threats To
Shipping (WWTTS)444https://msi.nga.mil/Piracy and (b) the Regional Cooperation
Agreement on Combating Piracy and Armed Robbery against Ships in Asia
(ReCAAP)555https://www.recaap.org/. The WWTTS reports are obtained from the
Office of Naval Intelligence, which compiles and publishes the reports on a
weekly basis. They contain summaries of piracy and hostility acts towards
worldwide commercial shipping since 1994. The content in these reports is
unstructured data and/or semi-structured or structured data (e.g. HTML, XML or
CSV files). The ReCAAP documents report piracy and armed robbery against ships
weekly from 2006 up to the present. The reports contain several types of
criminal activities, including _attacking_ , _boarding_ , _firing upon_ ,
_kidnapping_ , _hijacking_ , _robbery_ and _suspiciously approaching_.

These data repositories have been widely used as sources of soft/natural
language data to assist with the detection and analysis of maritime piracy
events and behaviours. However, unlike the majority of the existing work where
the structured data (i.e., in the form of a database) provided by the
repositories are used as the input, the input data in our work includes the
textual incident reports. We use the structured data (together with domain and
expert knowledge) as sources of distant supervision for training the
probabilistic labels, from which knowledge and events related to maritime
piracy can be extracted from the unstructured data.

We extracted all incidents from the WWTTS and ReCAAP documents for the most
recent five years (2015–2020) where the incident includes a paragraph
describing piracy activity. We sorted all these incidents, removed duplicates
and maintained them in a relational database (Postgres). In total, we obtained
1,940 incidents. We randomly selected 75 incidents as the test dataset and the
rest (1865 incidents) were used for training. For each incident we applied a
markup produced by standard NLP preprocessing
tools666http://nlp.stanford.edu/software/, including speech tagging, named-
entity mentions, coreference resolution and dependency parsing.

Moreover, we constructed a structured database from semi–structured HTML
documents, the “WWTTS–Maritime database”, containing 7,846 unique incidents,
and utilised it as ground-truth data for distant supervision as discussed in
Section III-E. The WWTTS–Maritime database provides additional information
regarding date and aggressor entities in each incident. We also used the
“WWTTS–Piracy database”, containing 7,895 unique incidents, which can be
useful as an external database for the victim and incident type extractions
(Section III-E).

###  III-B The Ontology

A key step in the design of our knowledge graph is to define an ontology
specifying which target relations to extract. We utilize the Simple Event
Model (SEM) [32], which is a graph-based specification for events and related
concepts, including actors, places and times. Inspired by the SEM ontology, we
designed a relatively small maritime piracy event ontology with four main
classes, _Incident_ (what happens), _Actor_ (who participated), _Location_
(where) and _Date_ (when).

The SEM ontology represents not only the description of who did what, when and
where, but also the roles played by the _Actors_. Each incident can have
multiple actor types or incident types. Fig. 2 shows how the SEM ontology can
be applied to the analysis of maritime data. In this example, we focus on a
container ship being attacked by pirates near position 05:28N –- 002:21E. In
this maritime ontology, classes and sub-classes are nodes, while properties
are edges of a graph. The _Actor_ has sub-classes including _Individuals_ ,
_Organizations_ , _Transport ships_ , _Passenger ships_ , _Fishing ships_ ,
_Navy ships_ and _Other ships_. For instance, in Fig. 2, the “container ship”
is a subclass of _Transport ship_ , which in turn, is a subclass of _Actor_.

In this work, we mainly define two roles for the extracted _Actors_ , which
are Victim and Aggressor. Based on the context constraints, each _Actor_ has
one of these roles. In the next section, we explain how we map the data to
this ontology.

###  III-C Entity Extraction and Role Assignment

Maritime Deepdive seeks to extract different types of objects from input
documents, namely _entities_ , _entity mentions_ , _roles_ , _relations_ and
_relation mentions_. An entity is a real-world place, actor or thing. In this
work, _Actor, Date, Location_ and _Incident–Type_ are the entities of
interest. An entity mention is a span of text in an input document that refers
to an entity. We assign a role of _Aggressor_ or _Victim_ to any actor
mention. For example, in Fig. 2 “container ship” represents a mention for the
actor entity which has the role of victim. A relation associates two or more
entities, and represents the fact that there exists a relationship between the
participating entities. A relation mention is a phrase that connects two
entity mentions that participate in a relation. For instance, “container ship”
and “pirates” participate in a Victim–Aggressor relation, which indicates that
the container ship has been victimized by pirates.

Figure 3: An illustration of entity extraction/linking and actor role
assignments in Maritime DeepDive. The arrows denote coreferent mentions.

Entity Extraction: Maritime DeepDive uses Stanford’s CoreNLP natural language
processing to extract useful markups and structure from the input data we
provide. This step splits our incidents into sentences and their component
tokens. Additionally, it returns lemmas, part-of-speech (POS) tags, named
entity recognition (NER) tags, and a dependency parse of the sentence. For
each incident, we extract a set of possible _mentions_ based on signals, e.g.,
positions, contained words, matched regular expressions and SQL queries. For
example, a simple User Define Function (UDF) written in Python or DDlog can
tag spans of continuous tokens with the NER tag DATE as date entities. By
involving a set of regular expressions we can extract the longitude and
latitude of each incident as a location entity. For some of the incidents,
there are no explicit coordinates of the location, but there is a textual
description, such as in Example 1 below. For these incidents we either look up
the coordinates of locations in the Google Maps geocoding
API777https://developers.google.com/maps/documentation/geocoding/ or write a
UDF to find the correct location entity mention using heuristic rules. We
apply the same procedure to map all candidates for actor mentions and incident
type entities, as seen in Fig. 3.

Example 1 – BRAZIL: On 29 October, robbers boarded a passenger ferry on the
Amazon River in the state of Para and robbed the passengers and crew.

TABLE I: Some extracted features for a sample Victim–Date relation candidate.
REPUBLIC OF THE CONGO: On 7 January, 12 seafarers, who were kidnapped from the
Panama-flagged tanker Anuket Amber
---
and the Singapore-flagged anchor handling and supply vessel ARK TZE in October
2018 off the country’s coast, have been released
and are all safe.
Description | Features
The set of POS tags for the words between the mentions in the relation | $\begin{aligned} POS\\_SEQ\\_[\textrm{, CD NNS , WP VBD VBN IN DT JJ}]\end{aligned}$
The set of NER tags for the words between the mentions in the relation | $\begin{aligned} NER\\_SEQ\\_[\textrm{O NUMBER O O O O O O O MISC}]\end{aligned}$
The set of lemmas of the words between the mentions in the relation | $\begin{aligned} LEMMA\\_SEQ\\_[\textrm{, 12 seafarer , who be kidnap from the panama-flagged}]\end{aligned}$
The set of words between the mentions in the relation | $\begin{aligned} WORD\\_SEQ\\_[\textrm{, 12 seafarers , who were kidnapped from the Panama-flagged}]\end{aligned}$
The sum of the lengths of the words in the mentions | $\begin{aligned} LENGTHS\\_[1\\_2]\end{aligned}$
Indicator feature for whether the mentions start with a capital letter | $\begin{aligned} STARTS\\_WITH\\_CAPITAL\\_[False\\_False]\end{aligned}$
The lemmas in a window of size 1 around the mentions | $W\\_LEMMA\\_L\\_1\\_R\\_1\\_[\textrm{on}]\\_[\textrm{Anuket}]$
The lemmas in a window of size 2 around the mentions | $W\\_LEMMA\\_L\\_2\\_R\\_2\\_[\textrm{: on}]\\_[\textrm{Anuket Amber}]$
The NERs in a window of size 1 around the mentions | $W\\_NER\\_L\\_1\\_R\\_1\\_[\textrm{O}]\\_[\textrm{O}]$
The NERs in a window of size 3 around the mentions | $W\\_NER\\_L\\_3\\_R\\_3\\_[\textrm{LOCATION O O}]\\_[\textrm{O O O}]$

Entity Linking: Entity linking is the process of mapping a textual mention to
a real-world entity. We use coreference resolution with the CoreNLP coref
annotator to link the
entities888https://stanfordnlp.github.io/CoreNLP/coref.html. For example, in
Fig. 3, two mentions of “the general cargo ship OYA” and “the vessel” refer to
the same entity. Since the CoreNLP annotator mostly links noun entities, it is
used for linking actor mentions. Entities can also be linked manually by
adding heuristic rules. For instance, “robbers” and “steal” both indicate the
incident type “robbery”, and therefore consider the same entity for both
mentions.

Actor Role Assignment: In this step, we assign roles to the actor mentions
based on the incident context or distant supervision. Fig. 3 indicates the
roles assigned to different actor mentions. The rules are created by the user,
and the process of creating these rules is based on the semantic context of
the incident. For instance, in Fig. 3, the verb “board” appears immediately
after the actor mentions (“armed pirates”). Hence, we will assign the
aggressor role to the “armed pirates” mention. Furthermore, the verb “kidnap”
appears immediately before the actor mention (“five crewmen”). Therefore, we
assign the victim role to this actor mention.

###  III-D Relation candidate mapping and feature extraction

The next step within Maritime DeepDive is the extraction of relation mention
candidates and features. For each binary relation, we take all pairs of non-
overlapping entity mentions (e.g., victim and aggressor) that co-occur in an
incident paragraph, and form Victim–Aggressor relation mention candidates. We
declare eight binary relations, namely Victim–Aggressor, Victim–Date,
Victim–IncidentType, Victim–Location, IncidentType–Date,
Aggressor–IncidentType, Aggressor–Location and Aggressor–Date, and extract all
the corresponding relation mention candidates.

We also need to extract features associated with relation candidates. Maritime
DeepDive includes an automatic feature generation library, DDlib. One baseline
feature that is often used in DDlib is the word sequence between mention pairs
in an incident. Table I shows a set of generic features extracted from Example
2 for a Victim–Date relation candidate.

Example 2 – REPUBLIC OF THE CONGO: On _7 January_ , 12 seafarers, who were
kidnapped from the Panama-flagged _tanker_ Anuket Amber and the Singapore-
flagged anchor handling and supply vessel ARK TZE in October 2018 off the
country’s coast, have been released and are all safe.

###  III-E Distant supervision

One challenge with building a machine learning system for knowledge graph
construction is that it generally requires a large amount of labeled training
data. However annotated data for maritime piracy incidents is not available,
and manually labelling textual data is tedious and expensive. With Maritime
DeepDive, we take the approach referred to as distant supervision, where we
generate a noisy set of labels for candidate relation mentions using a mix of
mappings from secondary datasets and heuristic rules. We implement distant
supervision in two separate levels, entity-level and relation-level. We
illustrate these procedures using examples.

Distant supervision with secondary data: We utilise the WWTTS–Piracy and
WWTTS–Maritime databases by assuming that they represent “ground truth”. Some
true and some false victim, aggressor, date and incident type entity mentions
are generated by comparing the mention text to the ground truth data. For
instance, Fig. 3 describes an incident from our training dataset where its
date, latitude and longitude match one or more incidents from the WWTTS–Piracy
database. If the actor mention “the general cargo ship OYA” matches the ship
type in the WWTTS–Piracy database, then we consider this mention as a true
candidate for victim role in this incident. Moreover, in similar fashion, if
the date, location and first 50 characters of this incident match an incident
in the WWTTS–Maritime database, and the actor mention candidate “armed
pirates”, match the aggressor in the WWTTS–Maritime database, then we consider
this mention as a true candidate for the aggressor role. These are examples of
entity-level distant supervision. If in an incident both victim and aggressor
candidates are labeled as true, then the binary relation of Victim–Aggressor
is also labeled as true. Moreover, if any of victim or aggressor mentions in
an incident are labeled as false, the binary relation between them are labeled
as false. For instance, the relation mention “the general cargo ship OYA–
armed pirates” is a true candidate and the relation mention “the general cargo
ship OYA – five crewmen” is a false candidate for the Victim–Aggressor
relation.

Distant supervision with rules: We create supervision rules that do not rely
on any secondary structured dataset, but instead just use some heuristic rules
over the entities and relations. For instance in the entity level, based on
the knowledge domain, we create a list of aggressor keywords such as _robber,
intruder, pirate, terrorist, gunman, assailant_ and when an actor mention
matches one of these keywords, the actor is very likely the aggressor.
Furthermore, we create a list of aggressive acts such as _attack, fire upon,
hijack, board, kidnap_. Some incidents contain more than one date mention, as
shown in Example 2. In this case we wrote a heuristic rule (UDF) to label the
true date of the incident (for instance, the closest date to the victim
mention is likely to be the true date).

We implemented this heuristic rule for the Victim–Aggressor relation: If an
aggressive act comes via lemmas between the aggressor and victim mentions, the
relation between them is marked as true. It can be observed in Fig. 3 that the
aggressive verb, “board”, appears between “armed pirates” and “the general
cargo ship OYA”. Technically, “board” returns an identifier that determines a
heuristically true label for the given relation mention.

The distant supervision generates noisy, imperfect labels for some candidates.
Machine learning techniques are able to exploit redundancy to cope with this
noise and learn the relevant phrases (e.g., attack, board and kidnap).
Negative examples are also generated by distant supervision to construct a
balanced training set.

Finally, we describe a simple majority-vote approach to resolving multiple
labels per example, which can be implemented within DDlog. These final labels
train a machine learning classification model (here, as true or false mentions
of a relation). A fraction (one tenth) of these labeled data is considered as
a development dataset for the analysis and evaluation purpose in Section IV.

Figure 4: A sample calibration plot for the Aggressor–Location relation.

###  III-F Learning and Inference

Maritime DeepDive generates a factor graph that describes a set of random
variables and how they are correlated. It can learn the weights of the factor
graph from training data that can be either obtained through distant
supervision or specified by the user while populating the database during the
extraction phase. Maritime DeepDive performs statistical inference on the
factor graph using standard techniques, such as Gibbs sampling [33, 34]. At
the end of learning and inference, Maritime DeepDive outputs a marginal
probability for every candidate in the output knowledge base. As shown in Fig.
1, the probabilistic labels generated by Maritime DeepDive for the
Victim–Aggressor relation candidates are 0.94 (container ship–pirates) and
0.11 (Oil tanker–crewman).

Maritime Deepdive learns the parameters of the model (the weights of the
features and potentially the connections between variables), and then performs
statistical inference over the test dataset to estimate the probability that
each variable of interest is true.

TABLE II: Evaluation of our knowledge graph with respect to binary relation extraction performance.  Relation | Candidates | F1 | ROC-AUC | Recall | Precision
---|---|---|---|---|---
Victim–Date | 142 | 0.96 | 0.97 | 0.96 | 0.96
Victim–IncidentType | 233 | 0.73 | 0.88 | 0.75 | 0.72
Victim–Aggressor | 196 | 0.72 | 0.82 | 0.75 | 0.80
Victim–Location | 154 | 0.90 | 0.88 | 0.92 | 0.88
IncidentType–Date | 114 | 0.85 | 0.75 | 0.86 | 0.85
Aggressor–IncidentType | 175 | 0.74 | 0.74 | 0.73 | 0.74
Aggressor–-Location | 115 | 0.72 | 0.67 | 0.81 | 0.67
Aggressor-–Date | 100 | 0.82 | 0.87 | 0.79 | 0.85
TABLE III: Evaluation of our knowledge graph on the binary relation extraction with distant supervision by only secondary databases, by only heuristic rules and with both distant supervision methods. Relation | F1-Score
---|---
| Second Data | Heuristic | Heu+Data
Victim–Date | 0.51 | 0.76 | 0.96
Victim–IncidentType | 0.43 | 0.51 | 0.73
Victim–Aggressor | 0.42 | 0.63 | 0.72
Victim–Location | 0.12 | 0.75 | 0.90
IncidentType–Date | 0.11 | 0.69 | 0.85
Aggressor–IncidentType | 0.22 | 0.67 | 0.74
Aggressor–Location | 0.44 | 0.71 | 0.72
Aggressor–Date | 0.52 | 0.79 | 0.82

##  IV Experimental evaluation

In this section, we evaluate the performance of Maritime DeepDive. In many
real-world (Defence) scenarios, annotated text data is scarce. Relying on
human experts or crowdsourcing to manually annotate a large amount of text
data is expensive or not always possible. As such, our major evaluation focus
is on investigating the feasibility of automatically extracting events and
activities of interest from maritime piracy reports without any annotated data
(which is a challenging task). We organise this section into two stages.
First, we carry out an evaluation on the quality (e.g., effectiveness and
accuracy) of the constructed knowledge graph, where we measure F1, recall,
precision and Receiver Operating Characteristics–Area Under the Curve
(ROC–AUC) scores associated with the extracted binary relations. Next, we
describe our process for the analysis and improvement of Maritime DeepDive’s
performance using _calibration plots_ without annotated data.

Relation extraction evaluation. For the evaluation of our model, we also focus
on relation extraction within maritime data. By dynamically setting a
threshold for probabilistic labels of each binary relation (i.e., only if the
probability of a relation candidate is higher than the threshold can it be
considered true), we can measure the F1 score, ROC–AUC, the curves of the ROC
and the precision–recall of the output for each relation, as shown in Fig. 5.
We can see that in general, Maritime Deepdive is very effective at classifying
all binary relations being investigated. Although there are certain relations
where the classification results demonstrate excellent performance (e.g.
Victim–Date), there are others where the classification results are good but
not exceptional (e.g Aggressor–Location). In our experience, low precision
occurs when Maritime DeepDive outputs a high probability for a fact, but it is
an incorrect one or is not supported by the corresponding incident. We expect
that this issue can be satisfactorily addressed by providing sufficient
negative examples for training, or adding more distant supervision rules. Low
recall occurs when Maritime DeepDive fails to extract a fact. This usually
means that a relation candidate is not assigned a probability that is
sufficiently high by Maritime DeepDive. In this case, one solution is to add
more distant supervision rules to increase the number of positive examples.

We apply a relation–specific threshold to the generated probabilistic labels
for all relation candidates. The threshold is set via maximizing the
classification accuracy on the validation dataset. As mentioned in Section
III-A, our test dataset contains 75 incidents. The number of candidates and
the predictive performance for each binary relation of Maritime DeepDive are
listed in Table II. Considering that we have a large unlabeled corpus, we are
able to achieve satisfactory performance (the averages are 0.81 F1, 0.82 ROC-
AUC, 0.82 recall and 0.81 precision scores) in the relation extraction task.

Figure 5: Receiver Operating Characteristic (ROC) and precision–recall curves
for the binary relations extracted using Maritime Deepdive.

In addition, an important question is the significance of distant supervision
on the predictive performance of Maritime DeepDive, where we only consider the
effect of secondary databases as ground-truth for distant supervision. We
compare the results of Maritime DeepDive with a simpler pipeline, which is
trained via distant supervision using only secondary databases (the second
column of Table III). Alternatively, we also trained Maritime Deepdive with
heuristic rules for distant supervision (the third column of Table III). The
last column of Table III shows that the probabilistic labels of Maritime
DeepDive with distant supervision from both heuristic rules and secondary
databases consistently performs better than with either. We observe a
significant improvement of $45\%$ on average over distant supervision with
only secondary databases. These results demonstrate that Maritime DeepDive
more effectively learns from the heuristic and secondary datasets’
probabilistic training labels over the simpler pipeline without heuristic or
secondary datasets. Given that we did not use any annotated (labelled) text as
training data, the overall results obtained in Tables II and III are very
promising. We expect these results to be significantly enhanced once we add
labelled training data and/or improve the labels generated by distant
supervision using MindBender999Mindbender is an interactive search interface
to browse data produced by DeepDive.
https://github.com/HazyResearch/mindbender and MindTagger101010Mindtagger is a
graphical user interface tool for labeling data.
http://deepdive.stanford.edu/labeling.

Iterative improvement. Users can develop an intuitive sense of how to improve
Maritime DeepDive from the high level summary of model performance provided by
the calibration plots. Ideally, in Maritime DeepDive, the probability
associated with a given fact should equal the empirical probability that this
fact is correct, i.e., an extraction with a probability 0.95 should be correct
95$\%$ of the time when inspected in the original source.

A calibration plot contains three components as a function of probability: (a)
Accuracy, which measures the development set accuracy of a prediction; (b)
$\\#$ Predictions (Development set), which measures the number of extractions
in the development set with a certain probability; and (c) $\\#$ Predictions
(Whole Set), which measures the number of extractions in the whole set with a
certain probability. For example, Fig. 4.a shows the accuracy for the target
relation Aggressor–Location (red line). In a well-tuned Maritime DeepDive
system, this would coincide with the blue (0,0)-(1,1) line. The deviation from
the (0,0)-(1,1) line can be caused by noise in the training data, quantization
error, or sparsity in the training data. The histograms in Fig. 4.b-c show
satisfying results with U-shaped curves, with the majority of extractions
concentrated at high probability and low probability and a few being
indeterminate. In general, the goal of improving the accuracy of knowledge
graph construction is to “push” these probabilities closer to either (0, 0.1)
or (0.9, 1.0). The masses around 0.4-0.6 suggest that there are some types of
candidates for which Maritime DeepDive has insufficient features or labels,
resulting in low confidence/high uncertainty. Such results indicate relations
for which additional distant supervision rules need to be added to resolve
this uncertainty.

##  V Conclusion

We have presented our initial approach to the automated construction of
probabilistic knowledge graphs for the maritime domain, with a use case
illustration for maritime piracy events. As many types of maritime incidents
are recorded in a textual format (e.g., post-incident reports) and other
associated activities such as pirates’ communications and planning are
entirely or partially conducted through words, being able to extract and
construct a knowledge base from such textual data would not only provide an
effective method for query answering, but also facilitate further analysis and
forecasting (e.g., identifying dangerous zones, spatial-temporal patterns,
predicting trends and potentially discovering plans for attacks). More
significantly, the probabilistic knowledge graph constructed from text data
can be combined with those constructed from other sources of data (e.g., rich
contextual/domain knowledge and movement data generated by sensors as in [19])
in order to provide comprehensive situational awareness and assist with the
detection of illegal maritime activities.

In future work, we plan to incorporate deep learning methods within our
framework to increase robustness and generalisation on unseen data. Moreover,
our proposed approach can be applied to threat detection in other domains,
which we plan to investigate in the near future.

## References

  * [1] D. Rosenberg, “The political economy of piracy in the south china sea,” _Naval War College Review_ , vol. 62, no. 3, pp. 43–58, 2009.
  * [2] K. Hurlburt, “The human cost of somali piracy,” _Piracy at Sea_ , pp. 289–310, 2013.
  * [3] J. J. Dabrowski, “Contextual behavioural modelling and classification of vessels in a maritime piracy situation,” Ph.D. dissertation, University of Pretoria, 2014.
  * [4] G. Rogova and J. Garcia, “Contextual knowledge and information fusion for maritime piracy surveillance,” _Prediction and Recognition of Piracy Efforts Using Collaborative Human-Centric Information Systems_ , vol. 109, 2013\.
  * [5] K. Rein and J. Biermann, “Beyond situation awareness: Considerations for sense-making in complex intelligence operations,” in _2018 21st International Conference on Information Fusion (FUSION)_. IEEE, 2018.
  * [6] V. Dragos and K. Rein, “Integration of soft data for information fusion: Pitfalls, challenges and trends,” in _17th International Conference on Information Fusion (FUSION)_. IEEE, 2014\.
  * [7] T. L. Wickramarathne, K. Premaratne, M. N. Murthi, M. Scheutz, S. Kübler, and M. Pravia, “Belief theoretic methods for soft and hard data fusion,” in _2011 International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2011.
  * [8] R. C. Núnez, B. Samarakoon, K. Premaratne, and M. N. Murthi, “Hard and soft data fusion for joint tracking and classification/intent-detection,” in _Proceedings of the 16th International Conference on Information Fusion_. IEEE, 2013.
  * [9] V. Dragos, X. Lerouvreur, and S. Gatepaille, “A critical assessment of two methods for heterogeneous information fusion,” in _2015 18th International Conference on Information Fusion (FUSION)_. IEEE, 2015, pp. 42–49.
  * [10] A. Plachkov, “Soft Data-Augmented Risk Assessment and Automated Course of Action Generation for Maritime Situational Awareness,” Ph.D. dissertation, Université d’Ottawa/University of Ottawa, 2016.
  * [11] K. Wu, W. Tang, K. Mao, G.-W. Ng, and L. O. Mak, “Semantic-level fusion of heterogenous sensor network and other sources based on Bayesian network,” in _17th International Conference on Information Fusion (FUSION)_. IEEE, 2014.
  * [12] A. Panchapakesan, R. Abielmona, and E. Petriu, “Optimizing Maritime Vessel Service Time with Adaptive Quay Crane Deployment Through Level 4 Hard-Soft Information Fusion,” in _2019 22th International Conference on Information Fusion (FUSION)_. IEEE, 2019\.
  * [13] T. Abirami, E. Taghavi, R. Tharmarasa, T. Kirubarajan, and A. Boury-Brisset, “Fusing social network data with hard data,” in _2015 18th International Conference on Information Fusion (FUSION)_ , 2015\.
  * [14] L. Burks and N. Ahmed, “Collaborative semantic data fusion with dynamically observable decision processes,” in _2019 22th International Conference on Information Fusion (FUSION)_. IEEE, 2019\.
  * [15] S. Buffett, C. Cherry, C. Dai, A. Désilets, H. Guo, D. McDonald, J. Su, and D. Tulpan, “Arctic Maritime Awareness for Safety and Security (AMASS). Final Report,” _Canadian National Research Council_ , 2017.
  * [16] S. C. Shapiro and D. R. Schlegel, “Use of background knowledge in natural language understanding for information fusion,” in _2015 18th International Conference on Information Fusion (FUSION)_. IEEE, 2015.
  * [17] Q. Zhang, Y. Q. Wen, D. Han, F. Zhang, and C. S. Xiao, “Construction of knowledge graph of maritime dangerous goods based on IMDG code,” _The Journal of Engineering_ , vol. 2020, 2020.
  * [18] V. Nguyen, “On combining probabilistic and semantic similarity-based methods toward off-domain reasoning for situational awareness,” in _2019 22th International Conference on Information Fusion (FUSION)_. IEEE, 2019.
  * [19] V. Nguyen and L. Mellor, “Fuzzy MLNs and QSTAGs for Activity Recognition and Modelling with RUSH,” in _2020 IEEE 23rd International Conference on Information Fusion (FUSION)_. IEEE, 2020, pp. 1–8.
  * [20] Z. Pelikánová, “Google knowledge graph,” 2014.
  * [21] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Freebase: a collaboratively created graph database for structuring human knowledge,” in _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , 2008.
  * [22] T. Rebele, F. Suchanek, J. Hoffart, J. Biega, E. Kuzey, and G. Weikum, “YAGO: A multilingual knowledge base from wikipedia, wordnet, and geonames,” in _International semantic web conference_. Springer, 2016.
  * [23] J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P. N. Mendes, S. Hellmann, M. Morsey, P. Van Kleef, S. Auer _et al._ , “DBpedia–a large-scale, multilingual knowledge base extracted from Wikipedia,” _Semantic web_ , vol. 6, 2015.
  * [24] P. Ernst, A. Siu, and G. Weikum, “Knowlife: a versatile approach for constructing a large knowledge graph for biomedical sciences,” _BMC bioinformatics_ , vol. 16, 2015.
  * [25] M. Rotmensch, Y. Halpern, A. Tlimat, S. Horng, and D. Sontag, “Learning a health knowledge graph from electronic medical records,” _Scientific reports_ , vol. 7, 2017.
  * [26] M. Färber, “The microsoft academic knowledge graph: a linked data source with 8 billion triples of scholarly data,” in _International Semantic Web Conference_. Springer, 2019.
  * [27] P. Chen, Y. Lu, V. W. Zheng, X. Chen, and B. Yang, “KnowEdu: a system to construct knowledge graph for education,” _IEEE Access_ , vol. 6, 2018.
  * [28] Q. Zhang, Y. Wen, C. Zhou, H. Long, D. Han, F. Zhang, and C. Xiao, “Construction of knowledge graphs for maritime dangerous goods,” _Sustainability_ , vol. 11, 2019.
  * [29] J. Everwyn, A.-I. Mouaddib, B. Zanuttini, S. Gatepaille, and S. Brunessaux, “Link Prediction on Dynamic Attributed Knowledge Graphs for Maritime Situational Awareness,” in _Conférence Nationale sur les Applications Pratiques de l’Intelligence Artificielle (APIA 2019)_ , 2019.
  * [30] C. De Sa, A. Ratner, C. Ré, J. Shin, F. Wang, S. Wu, and C. Zhang, “Deepdive: Declarative knowledge base construction,” _ACM SIGMOD Record_ , vol. 45, 2016.
  * [31] F. Niu, C. Ré, A. Doan, and J. Shavlik, “Tuffy: Scaling up Statistical Inference in Markov Logic Networks using an RDBMS,” _Proceedings of the VLDB Endowment_ , vol. 4, 2011.
  * [32] W. R. van Hage and D. Ceolin, “The simple event model,” in _Situation awareness with systems of systems_. Springer, 2013, pp. 149–169.
  * [33] M. Wick, A. McCallum, and G. Miklau, “Scalable probabilistic databases with factor graphs and MCMC,” _arXiv preprint arXiv:1005.1934_ , 2010.
  * [34] C. Zhang and C. Ré, “Towards high-throughput Gibbs sampling at scale: A study across storage managers,” in _Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data_ , 2013.