# A Survey on Online User Aggression: Content Detection and Behavioural
Analysis on Social Media Platforms

Swapnil Mane<EMAIL_ADDRESS>0000-0002-8234-4557 ,  Suman Kundu
<EMAIL_ADDRESS>0000-0002-7856-4768 Indian Institute of Technology
JodhpurRajasthanIndia342030 and  Rajesh Sharma  Institute of Computer Science,
University of TartuEstonia<EMAIL_ADDRESS>0000-0003-3581-1332

(2023)

###### Abstract.

The rise of social media platforms has led to an increase in cyber-aggressive
behavior, encompassing a broad spectrum of hostile behavior, including
cyberbullying, online harassment, and the dissemination of offensive and hate
speech. These behaviors have been associated with significant societal
consequences, ranging from online anonymity to real-world outcomes such as
depression, suicidal tendencies, and, in some instances, offline violence.
Recognizing the societal risks associated with unchecked aggressive content,
this paper delves into the field of Aggression Content Detection and
Behavioral Analysis of Aggressive Users, aiming to bridge the gap between
disparate studies. In this paper, we analyzed the diversity of definitions and
proposed a unified cyber-aggression definition. We examine the comprehensive
process of Aggression Content Detection, spanning from dataset creation,
feature selection and extraction, and detection algorithm development.
Further, we review studies on Behavioral Analysis of Aggression that explore
the influencing factors, consequences, and patterns associated with cyber-
aggressive behavior. This systematic literature review is a cross-examination
of content detection and behavioral analysis in the realm of cyber-aggression.
The integrated investigation reveals the effectiveness of incorporating
sociological insights into computational techniques for preventing cyber-
aggressive behavior. Finally, the paper concludes byidentifying research gaps
and encouraging further progress in the unified domain of socio-computational
aggressive behavior analysis.

Cyber-aggression, Cyberbullying, Hate speech, Aggression detection, Aggressive
behavior analysis, Behavioural Analysis of User Aggression, Social Media

††copyright: rightsretained††journalyear: 2023††doi: XXXXXXX.XXXXXXX††journal:
CSUR††journalvolume: 37††journalnumber: 4††article: 1††publicationmonth: 11

##  1\. Introduction

In the age of rapid technological advancement, the landscape of global
communication has undergone a profound transformation. Social media platforms
serve a crucial role in shaping human interaction and information
dissemination (De Souza and Da Costa-Abreu, 2020; Saha et al., 2023). These
platforms transcend geographical boundaries, enabling individuals to connect,
share experiences, and engage in discussions on a global scale. However, this
digital evolution has also given rise to a complex challenge: the prevalence
of aggressive behavior and hostile communication within these virtual spaces
(Bhattacharya et al., 2020). Online aggression encompasses a diverse range of
hostile behaviors, such as cyberbullying, online harassment, and the
propagation of offensive and hate speech. This broad spectrum of aggressive
behavior manifests online, spanning from subtle forms of verbal aggression to
explicit instances of hate speech and threats (Mladenović et al., 2021;
Chatzakou et al., 2017). One of the reasons for this behavior is the shroud of
anonymity provided by the digital environment has encouraged individuals to
express opinions and engage in behavior that might be restrained in
traditional face-to-face interactions (Bazarova et al., 2013). This phenomenon
has created a digital environment where aggression manifests itself in diverse
and often insidious ways, challenging the very essence of open and
constructive communication.

Detecting aggressive content on social media is the most important first step
in developing any aggressive activity preventive system. Similarly, behavioral
analysis of user’s aggressiveness is equally important in order to identify
causes, contextual factors, and patterns associated with such behavior. Both
issues have a substantial impact on society, hence finding solutions to these
problems is essential. It has been reported that aggressive content can exert
detrimental psychological effects on both individual users and the broader
online community (Bouhnik and Deshen, 2014). Exposure to aggressive language
can evoke feelings of distress, anxiety, and fear, ultimately compromising the
sense of safety that should ideally characterize social media interactions. To
summarise, the unchecked propagation of aggressive behavior can result in the
erosion of the digital environment’s potential as a space for meaningful
discourse and collective problem-solving (Rawat et al., 2023).

Researchers have increasingly turned to computational methods for aggressive
content detection (Chatzakou et al., 2019; Mladenović et al., 2021). The
investigations have been conducted on prominent platforms like Facebook (Chen
et al., 2017; Kumar et al., 2018b; Kumari and Singh, 2022; Sharif and Hoque,
2022), Instagram (Kumari and Singh, 2022), X formerly known as Twitter (Kumar
et al., 2018b; Chen et al., 2017; Chatzakou et al., 2019; Sadiq et al., 2021;
Torregrosa et al., 2022; Kumari and Singh, 2022; Gattulli et al., 2022),
YouTube (Bhattacharya et al., 2020; Kumar et al., 2020; Sharif and Hoque,
2022), and From-spring.me (Reynolds et al., 2011). These approaches leverage
the power of natural language processing (NLP), machine learning (ML), and
deep learning (DL), including transformers to automatically identify and
categorize aggressive posts (Burnap and Williams, 2015). By promptly flagging
aggressive content, these techniques strive to mitigate the harmful
consequences of such behavior and promote a more positive online culture.
Simultaneously, researchers recognize that a comprehensive behavioral
understanding of users’ aggressiveness necessitates a deeper analysis of the
underlying user behaviors contributing to these interactions. Studies on
behavioral analysis of user aggressiveness have been carried out on platforms
like X (Chatzakou et al., 2017; Aragón et al., 2019; Poiitis et al., 2021;
Torregrosa et al., 2022; Mane et al., 2023), Gab.com (Mathew et al., 2019),
and others (Borraccino et al., 2022; Eraslan and Kukuoglu, 2019; Balci and
Salah, 2015). A holistic perspective requires exploring the triggers, factors,
and patterns associated with aggressive behavior. This nuanced comprehension
can inform the development of more effective prevention and intervention
strategies, addressing the issue at its roots.

Figure 1. Temporal trends in research publications focused on investigating
online aggression behavior. The trendline represents a clear upward
trajectory, indicating a significant increase in scholarly interest and
publications related to this topic.

Despite considerable progress in aggressive content detection and behavioral
analysis of user aggressiveness, a notable gap persists in integrating these
two domains (Poiitis et al., 2021; Terizi et al., 2021). This literature
review aims to bridge this gap by systematically analyzing prior research on
explicitly aggressive content detection methods and behavioral analysis of
aggressive users within the realm of social media platforms. By synthesizing
these two critical aspects, the review seeks to shed light on a combination of
these two aspects which can yield a more comprehensive understanding of online
aggression and its underlying dynamics. In Figure 1, we analyzed the trend in
research publications in Computer Science (CS) and Sociology in this domain.
The increasing number of publications over time indicates the current
significance and relevance of the topic. In this survey, we have focused on
research papers published over the past six years, starting from 2018.

Figure 2. The taxonomy of research focused on the aggressive behavior of the
user. This representation provides a systematic classification of studies and
methods employed to investigate various aspects of cyber-aggression.

The structure of this literature review is as follows (shown in Figure 2):
Subsequent sub-sections, 1.1 and 1.2, present the contributions of this paper
and the motivations behind conducting research in this area, respectively.
Section 2, analyzed theoretical aspects of this topic, including the
exploration of different definitions, while Section 3 presents an analysis of
a previous survey on the same topic. Section 4 reviews aggressive content
detection that outlines the datasets, features, and algorithms employed.
Furthermore, Section 5 explores approaches to the behavioral analysis of user
aggressiveness, including qualitative and quantitative studies conducted.
Section 6 highlights the main challenges faced in this domain and presents
potential research opportunities. Finally, Section 7 provides a comprehensive
conclusion to this survey by summarizing the key contributions.

###  1.1. Contributions

This literature review presents a contribution to the growing field of
aggressive behavior analysis, addressing persistent gaps in current research
by comprehensively examining approaches and providing potential research
directions. Following are the highlights of this work:

  1. (1)

Proposed unified and comprehensive definition of cyber-aggression by analyzing
the diverse range of existing definitions.

  2. (2)

Thoroughly examines the entire process of Aggression Content Detection,
including dataset creation, feature selection, extraction, and detection
algorithm development.

  3. (3)

Reviews studies related to the Behavioral Analysis of Aggression, shedding
light on the influencing factors, consequences, and patterns associated with
cyber-aggressive behavior, thus contributing to a deeper understanding of this
phenomenon.

  4. (4)

Emphasizes the significance of incorporating sociological findings into
automatic computational prevention systems for cyber-aggressive behavior,
highlighting the need for a multidisciplinary approach to address this complex
issue.

###  1.2. Motivation

Aggressive behavior analysis on social media has received significant
attention in recent years, evident through increased media coverage and
political attention. The importance of this issue can be emphasized by several
effective reasons, which we listed as following:

  * •

Effects on psychological well-being and social relationships. Aggressive
behavior displayed on social media platforms has a profound impact on
individuals’ psychological well-being, resulting in elevated levels of stress,
anxiety, depression, and even thoughts of suicide. (Hinduja and Patchin, 2018)
has demonstrated the detrimental effects of such aggressive behavior on mental
health. Moreover, aggressive behavior on social media can inflict significant
harm on social relationships, leading to conflicts, mistrust, and the
fragmentation of online communities. Studies conducted by (Vazsonyi et al.,
2012; Pantic et al., 2012) highlights how this behavior contributes to
anonymity and encourages the creation of a hostile online environment.

  * •

Fueling anonymity in a diverse landscape. The presence of aggressive behavior
on social media platforms can lead to the creation of anonymity among
different groups. This anonymity is often perpetuated by individuals engaging
in aggressive online behavior, which can exacerbate social tensions and
divisions. The study by (Zimmerman and Ybarra, 2016) highlighted that
aggressive behavior on social media platforms could result in the polarization
of communities and the amplification of social biases. It is to be noted that
online aggression not only undermines the potential for constructive dialogue
but also poses a threat to social cohesion in diverse societies. Moreover,
anonymity on social media platforms can encourage individuals to engage in
aggressive behavior without fear of real-world consequences.

  * •

Impact on the safety of vulnerable groups. Online aggression often targets
individuals or communities who are often vulnerable, such as children,
adolescents, and are more prone to face a threat, such as sexual and religious
minorities. These vulnerable populations are at a heightened risk of
experiencing various forms of online victimization, including harassment,
stalking, and other harmful behaviors, which can have severe offline
consequences. To ensure the protection of these groups, it is essential to
address the problem of aggressive behavior on social media (Błachnio and
Przepiórka, 2018; Livingstone et al., 2011).

  * •

Lack of study on low-resource languages. The study of aggressive behavior
detection and analysis in social media is essential, particularly in
addressing the lack of research on low-resource languages. While numerous
studies have focused on understanding and detecting aggressive behavior in
widely spoken languages, there is a notable dearth of research in low-resource
languages (Ranasinghe and Zampieri, 2021; Saeed et al., 2023) such as Punjabi,
Marathi, Tamil, Urdu, Italian, Slovenian, Latvian, Turkish, Ukrainian,
Vietnamese, etc. These languages represent important linguistic landscapes in
specific regions. Low-resource languages are characterized by limited
linguistic resources, including insufficient annotated data, linguistic tools,
and computational models. This lack of attention towards low-resource
languages impedes to effectively combat aggressive behavior and its
consequences within these linguistic communities. Additionally, the presence
of anonymity in low-resource languages further exacerbates the need for
research and intervention in understanding and addressing aggressive behavior
on social media platforms (Peng et al., 2003; Corcoran et al., 2015).

  * •

Increasing legal regulations around the world. Governments across various
countries have recognized the harmful impact of online aggressive behavior and
have responded by enacting laws and regulations to combat its spread and
consequences. For instance, in Germany, the Network Enforcement Act (NetzDG)
was introduced to address aggression including hate speech and illegal content
on social media platforms (Lee, 2022). Similarly, India has enacted the
Information Technology (Intermediary Guidelines and Digital Media Ethics Code)
Rules,
2021111https://meity.gov.in/writereaddata/files/Intermediary_Guidelines_and_Digital_Media_Ethics_Code_Rules-2021.pdf,
which aim to regulate online content, including aggressive behavior (The
Ministry of Electronics and Information Technology, 2021). United Kingdom has
also implemented legal measures, such as the Online Harms White Paper222
https://www.gov.uk/government/consultations/online-harms-white-paper, to
ensure online platforms take responsibility for tackling harmful content and
protecting users (Department for Digital, Culture, Media & Sport, 2019).
Singapore has enacted the Protection from Online Falsehoods and Manipulation
Act (POFMA)333 https://www.mlaw.gov.sg/news/others/pofmb-videos/ to address
the spread of false information and malicious online behavior (Ministry of
Law, Singapore, 2019). These legal developments illustrate the growing concern
of governments and policymakers regarding aggressive behavior in online
spaces. Therefore, studying and analyzing this problem is crucial to inform
effective detection and intervention strategies that align with legal
requirements and promote safer digital environments.

##  2\. Understanding Cyber-Aggression

Interactions on social media have given rise to harmful and widespread
aggressive behaviors with severe consequences. This section provides a
thorough analysis of cyber-aggression, aiming to establish a comprehensive
definition for its diverse expressions in virtual environments. The term
“cyber-aggression” has been given various interpretations within scholarly
discourse. The initial definitions primarily focused on instances of
hostility, harassment, or harm that occur through digital channels (Roy et
al., 2018; Díaz-Torres et al., 2020). While these initial definitions offer a
basic understanding, they may unintentionally limit the broad scope of cyber-
aggression to only overtly aggressive behaviors. Efforts have been made to
establish a comprehensive definition of cyber-aggression in order to encompass
its various nuanced manifestations. There is a need to emphasize that not all
digital conflicts involve aggression, despite the presence of stress. This
presents difficulties in accurately determining intent solely through online
interactions. In order to offer a thorough comprehension of cyber-aggression,
a collection of definitions from scholarly discourse has been compiled. These
definitions reflect the diverse nuances and interpretations associated with
this phenomenon. These definitions collectively illustrate various ways in
which aggression can be expressed through digital communication. For instance:

  * •

Language targeting individuals to cause harm (Anderson and Bushman, 2002).

  * •

Verbal attacks, abusive language, derogatory comparisons (Roy et al., 2018).

  * •

Language causing harm or inciting violence (Díaz-Torres et al., 2020).

  * •

Content containing aggression, credible threats, misinformation (Facebook, [n.
d.]).

  * •

Text glorifying violent actions, fostering hostility (Fortuna and Nunes,
2018).

  * •

Communication undermining based on attributes (Nobata et al., 2016).

  * •

Intent to harm via verbal, physical, psychological means (Baron and
Richardson, 1994).

  * •

Distinct forms: physical, verbal, relational (Buss, 1962).

  * •

Deliberate electronic harm perceived as offensive (Mishna et al., 2023).

  * •

Internet-driven harm (Grigg, 2010; Tonglin et al., 2018; Zhao and Gao, 2012).

  * •

Peer-to-peer online aggression (Kelly and Arnold, 2016).

  * •

Aggressive electronic behavior (Kowalski et al., 2014).

  * •

Digital technology fostering hostility (Patchin and Hinduja, 2015).

  * •

Digital aggression, harm via technology (Völlink et al., 2015).

  * •

Electronic devices, online platforms for harm (Al-Alosi, 2017).

  * •

Online aggressive acts, intent to harm (Barlett and Coyne, 2014).

The definitions of cyber-aggression highlight its complex nature, which
encompasses a range of behaviors intended to cause harm, abuse, derogatory
remarks, threats, and hostility within the digital domain. Various definitions
of cyber-aggression illustrate online aggression in distinct ways, often
tailored to specific contexts or applications, with a particular focus on
explicit acts of aggression. Nevertheless, it is crucial to acknowledge that
contemporary online expressions of aggression often adopt a covert tone,
including humor and sarcasm. This highlights the significance of taking into
account covert or satirical manifestations of aggression. However, this
perspective presents the difficulty of accurately identifying aggression in
the context of online interactions. In addition, other perspectives highlight
the power dynamics that are inherent in cyber-aggression, revealing situations
in which one party utilizes digital platforms to assert control or dominance
over another (Kowalski and Limber, 2013). Although this approach effectively
reveals manipulation and coercion as integral to certain aggressive behaviors,
it may fall short of understanding situations in which aggression arises from
misinterpreted or misconstrued intentions.

Figure 3. Comprehensive analysis of the dimensions characterizing cyber-
aggression, focusing on the various possible behaviors rather than their
underlying causes. It provides insight into the diverse ways in which
aggression can manifest within the digital realm, spanning from overt and
explicit actions to subtler, covert expressions.

The dimensions of cyber-aggression encompass a range of expressions, spanning
from explicitly aggressive behavior to subtler, covert behaviors, as
illustrated in Figure 3. In this dimensions explicit aggression involves
direct and overt harmful behaviors, while passive-aggressive behavior employs
indirect expressions of hostility. Microaggressions are subtle actions that
undermine or diminish individuals based on their characteristics. Exclusionary
practices involve deliberate isolation, and identity-based attacks target
personal attributes. Manipulative coercion employs emotional tactics,
trolling, and provocation incites reactions, and revengeful actions aim at
retribution.

Based on this comprehensive understanding of cyber-aggression. This study
proposed a unified definition, “Cyber-aggression refers to harmful intentional
online behavior, irrespective of whether it is overt or covert. It includes
the use of hostile, offensive, or abusive language, insults, threats, and
abusive comments intended to cause discomfort, distress, or harm to
individuals or communities.” This unified definition effectively captures the
wide range of aggressive behaviors encountered in the digital realm. It
includes not only overt and explicit attacks but also actions that operate
with certain levels of subtlety, resulting in psychological harm rather than
physical harm. To illustrate the real-world implications of this unified
definition, consider the following scenario: An individual consistently posts
derogatory comments on a social media platform, specifically targeting
someone’s physical appearance. These comments, while not explicitly violent or
threatening, nonetheless result in emotional distress and a profound sense of
humiliation for the targeted individual. This real-world example shows that
cyber-aggression behavior, by unified definition, encompasses a variety of
behaviors that ultimately lead to harm or discomfort in the digital space.

##  3\. Related Work

This section explores the existing literature surveys conducted in this field.
The related work encompasses studies focused on aggressive content detection
and behavioral analysis of user aggressiveness, particularly in the context of
social media platforms. The 2020 survey conducted by (Mladenović et al., 2021)
examined the detection of objectionable content including aggression,
bullying, and grooming. The survey encompassed the analysis of different
datasets, types of features, and methods utilized for detecting objectionable
content and assessing their effectiveness. Furthermore, the survey delved into
the exploration of diverse definitions and understanding of ambiguity in
various forms of objectionable content. Another survey (Langham and Gosha,
2018) reviews previous work on aggression detection and emphasizes the
necessity of aggressive behavior analysis in social media. This survey
recognizes the importance of analyzing aggressive language and the homogeneous
nature of users rather than focusing solely on hate speech. A review conducted
by (Anderson and Bushman, 2001) focuses on the effects of violent video games
on aggressive behavior. The study finds that exposure to violent video games
is associated with increase in aggressive behavior, thoughts, and affect.
Authors suggest that parents and educators should be aware of the potential
negative effects of violent video games and should limit children’s exposure
to them. In another survey (Martins, 2020), the impact of media use on social
aggression in children and adolescents is examined. By exploring the
relationship between media consumption and the development of social
aggression, the research contributes to the field of media psychology. The
survey highlights the potential influence of media exposure on the
manifestation of aggressive behaviors, emphasizing the significance of
understanding and addressing this issue. Additionally, (Corcoran et al., 2015)
provides a comprehensive analysis of current definitions related to cyber-
based peer-to-peer aggression. The study evaluates the similarities and
differences between traditional bullying and cyberbullying, emphasizing the
challenges of applying traditional bullying criteria to the online context.
The paper underscores the need for diverse theoretical perspectives and
alternative approaches to conceptualizing cyber aggression.

Previous surveys have predominantly reviewed either aggression detection or
behavioural analysis of user’s aggressiveness in isolation. However, our
comprehensive survey encompasses both aspects, providing a detailed review of
each dimension. Our survey explores diverse definitions of “Cyber-aggression”
and its various manifestations. We delve into aggressive content detection,
analyzing different datasets from various social media platforms, encompassing
different languages. We investigate the features employed in feature
engineering, word embeddings, and multimodal features. Additionally, we review
various detection techniques, including machine learning, deep learning,
transformer-based, and ensemble-based approaches. Furthermore, we conduct a
comprehensive behavioural analysis of user’s aggressiveness, encompassing both
qualitative and quantitative studies. This survey allows for a more
comprehensive understanding of the phenomenon and offers valuable insights
contributing to a deeper understanding of this complex issue. Moreover, our
findings motivate further computational research on aggressive behavior.

##  4\. Aggressive Content Detection

In this section, we provide an overview of the research papers that have
focused on the domain of aggression detection. Aggression detection system
encompasses various aspects such as data annotation, datasets, features, and
detection algorithms. The process of data annotation involves expert
annotation to establish ground truth labels for aggression detection. Machine
learning algorithms play a pivotal role in identifying aggression within
textual content, leveraging the annotated data. These algorithms encompass
both traditional and deep learning techniques. The conventional aggression
detection process is depicted in Figure 4. The following subsections provide
in-depth analytical insights by delving deeply into the annotation process,
available datasets, text features, and machine learning models used for
aggression detection.

Figure 4. Illustrates the step-by-step process of traditional aggression
detection, encompassing key stages such as data preparation, pre-processing,
feature selection, classification algorithms, model evaluation, and iterative
refinement in cases of suboptimal performance. The ultimate goal is to develop
a state-of-the-art final model for aggression detection.

###  4.1. Annotations

This section examines the diverse strategies employed by researchers to
annotate datasets for aggressive content detection. Various studies have
introduced tagsets with different attributes and levels of granularity. For
instance, some studies have introduced binary labels, categorizing content as
either Aggressive or Non-aggressive (Chen et al., 2017; Sadiq et al., 2021;
Gattulli et al., 2022; Sharif and Hoque, 2022; Aragón et al., 2020). Others
have extended the tagset to encompass different levels of aggression,
differentiating between Overt, Covert, and Non-aggression (Kumar et al., 2022,
2018b; Bhattacharya et al., 2020; Rawat et al., 2023). To further capture the
complex nature of aggressive behaviors, subtags have been introduced under the
“Aggression” category. These include Aggression Intensity Level, which
includes attributes such as physical threat and sexual threat (Kumar et al.,
2022), Discursive Effects, comprising actions like attack, defend,
counterspeech, abet/instigate, and gaslight (Kumar et al., 2022, 2018b), and
misogyny, categorized as gendered and non-gendered (Bhattacharya et al.,
2020), with strict annotation guidelines. In contrast, some studies have
employed tags for high aggressive, medium aggressive, or non-aggressive
content (Kumari et al., 2019; Kumari and Singh, 2022). Moreover, (Chatzakou et
al., 2019) introduced tagsets for user labeling as bullies, aggressors, normal
users, or spammers, each category with specific criteria for annotation.

Inter-rater agreement (IAA) measures the consensus among annotators and it is
crucial for ensuring the reliability of the annotated datasets. Various
metrics, such as Cohen’s kappa (annotators == 2) (Sane et al., [n. d.]; Sharif
and Hoque, 2022; Rawat et al., 2023), Fleiss’ kappa (annotators $\geq 2$)
(Chatzakou et al., 2019; Srivastava et al., 2020), or Krippendorff’s alpha
(annotators $\geq 2$) (Kumar et al., 2022, 2018b; Bhattacharya et al., 2020),
are used for IAA evaluation. These metrics consistently show agreement scores
ranging from moderate to almost perfect levels in respective studies,
demonstrating the robustness of the annotation process. Reliability levels are
categorized as poor agreement (0-0.20) indicates low reliability, while
substantial agreement (0.81-0.99) indicates a high degree of consensus, with
perfect agreement (1.00) indicating perfect reliability. The IAA score was
measured by (Kumar et al., 2022) in two phases involving multiple annotators
for different languages. The substantial improvement in IAA scores during the
second phase, which involved discussions and refinements, highlights the
iterative nature of the annotation process. (Kumar et al., 2018b) conducted an
IAA experiment using a crowdsourcing platform, which enhanced agreement scores
for top-level (for OAG, CAG, and NAG) annotations. (Bender and Friedman, 2018)
and (Binns et al., 2017) highlighted the influence of annotators’ identity and
demographics on annotations. To address this concern, (Sharif and Hoque, 2022)
deliberately select annotators from diverse racial, residential, and religious
backgrounds. (Kumari and Singh, 2022) selected annotators with balanced
demographics, aiming to minimize bias in religion and gender. Furthermore,
(Gattulli et al., 2022; Chatzakou et al., 2019; Rawat et al., 2023)
incorporated multiple annotators to ensure a variety of perspectives and
reduce individual biases. Despite these efforts, (Chatzakou et al., 2019;
Kumar et al., 2022) highlighted the challenge of annotating discursive effects
due to their subjective nature, resulting in revisions of annotation
guidelines. (Gattulli et al., 2022), and (Kumari and Singh, 2022) revealed the
difficulty of detecting bullying and aggressive behaviors, even during manual
inspection. Overall, dataset annotation requires careful consideration of
tagsets, inter-rater agreement, and annotator diversity. The evolution of
annotations and continuous revisions of guidelines ultimately contributed to
the development of robust datasets.

###  4.2. Datasets

The availability and diversity of datasets play a significant role in
advancing research on aggression detection and behavior analysis within the
realm of Online Social Media (OSM) platforms. A multitude of studies have
contributed to the development and curation of these datasets, enabling
researchers to explore aggressive post detection methods and user behavior
analysis. This section provides an overview of the datasets used in prior
research efforts, encompassing various OSM platforms and languages, as well as
the methodologies employed for data collection. Researchers have put forth
training datasets tailored to specific OSM platforms, including but not
limited to X (Chen et al., 2017; Rawat et al., 2023; Kumar et al., 2018b;
Bhattacharya et al., 2020; Kumar et al., 2022; Kumari and Singh, 2021;
Gattulli et al., 2022; Chatzakou et al., 2019; Sadiq et al., 2021; Mane et
al., 2023), Facebook (Kumar et al., 2018b; Bhattacharya et al., 2020; Kumar et
al., 2022; Sharif and Hoque, 2022; Kumari and Singh, 2021), Youtube
(Bhattacharya et al., 2020; Kumar et al., 2022; Sharif and Hoque, 2022),
Telegram (Kumar et al., 2022), Instagram (Kumari and Singh, 2021), and others
(Reynolds et al., 2011). These datasets accommodate posts predominantly in the
English language (Reynolds et al., 2011; Chen et al., 2017; Chatzakou et al.,
2019; Kumar et al., 2018b; Bhattacharya et al., 2020; Sadiq et al., 2021;
Kumari and Singh, 2021; Kumar et al., 2022; Mane et al., 2023) and extend
their reach to encompass low-resource languages, such as Spanish (Torregrosa
et al., 2022; Aragón et al., 2020), Italian (Gattulli et al., 2022), Hindi
(Kumar et al., 2018b; Bhattacharya et al., 2020; Kumar et al., 2022; Rawat et
al., 2023), Bangla (Kumar et al., 2018b; Bhattacharya et al., 2020; Kumar et
al., 2022; Sharif and Hoque, 2022), Meitei (Kumar et al., 2022), and Code-
mixed (Kumar et al., 2018b; Bhattacharya et al., 2020; Kumar et al., 2022;
Rawat et al., 2023), where the Code-mixed represents low-resource languages
expressed in English.

(a) Distribution of language-specific datasets in previous research. (b)
Distribution of data instances in language-specific datasets with comparative
percentage values.

Figure 5. Analysis of language-specific datasets.

Data extraction from OSM platforms is accomplished through the use of APIs
(Rawat et al., 2023; Kumar et al., 2022; Gattulli et al., 2022; Chatzakou et
al., 2019; Mane et al., 2023), or web crawlers (Kumar et al., 2022; Sharif and
Hoque, 2022). This data acquisition method is aligned with the particular
requirements of individual study, taking into consideration the unique use
case, problem statement, and overarching research objectives. For instance, a
study by (Rawat et al., 2023) aimed to analyze aggression during Indian
general elections. They scraped approximately 10,000 tweets from the public
Twitter accounts of 110 influential Indian personalities during the 2014 and
2019 elections. Subsequent keyword-based manual pruning reduced the dataset to
2000 instances, including English and Hindi-English code-mixed data. Another
comprehensive dataset, curated by (Kumar et al., 2022), contains diverse OSM
platforms and languages. The dataset was constructed by identifying videos and
channels that attracted hate speech and aggressive comments, particularly
targeting women and minority religious groups, especially Muslims. Keywords
and hashtags related to socio-political, religious, or cultural events were
employed to sample videos and channels in four languages.

Furthermore, a preliminary analysis (Kumar et al., 2018b) of this annotated
dataset revealed intriguing differences in communication styles between
Facebook and Twitter users. The majority of Facebook comments were shorter
than 150 characters, similar to Twitter’s 140 character limit (now expanded to
280 characters). However, the aggression levels on both platforms demonstrated
significant disparities, highlighting the platform’s influence on aggression
dynamics. An analysis of code-mixing and aggression reveled that code-mixed
comments and tweets tended to be more aggressive. In contrast, Hindi posts
displayed a more even distribution of aggression, while English posts were
largely non-aggressive.

Figure 6. Distribution of aggression detection datasets across OSM platforms.

Additional datasets, such as the one introduced by (Bhattacharya et al.,
2020), includes over 25,000 comments in three languages. A significant
proportion of these comments exhibit gendered or misogynistic content. In the
case of Bengali social media users, (Sharif and Hoque, 2022) compiled dataset
instances from Facebook and YouTube, targeting threads with at least 200
reactions. Italian aggressive Twitter dataset was proposed by (Gattulli et
al., 2022), while user aggressive behavior dataset was created by (Chatzakou
et al., 2019), categorizing user behaviors as normal, aggressor, bully, or
spammer based on tweet analysis. Beyond text, (Kumari and Singh, 2021)
introduced a dataset comprising symbolic images alongside textual content,
collected from various social media sites, including Twitter, Instagram, and
Facebook. These images were processed to remove duplicates and low-resolution
content, resulting in posts labeled as non-aggressive, medium-aggressive, or
high-aggressive.

Chatzakou et al. (Chatzakou et al., 2019)
(97) Kumar et al. (Kumar et al., 2018b)
(157) Kumar et al.
(Kumar et al., 2022)
(14) Sharif and
Hoque (Sharif and Hoque, 2022)
(26) Sadiq et al. (Sadiq et al., 2021)
(89) Bhattacharya et al.
(Bhattacharya et al., 2020) (71)

Figure 7. Researcher preferences and utilization of datasets in relation to
other datasets.

Table 1. Presents a comprehensive overview of various datasets, including the
year of creation, data source, languages covered, total data instances, non-
aggressive and aggressive instances, number of classes, IAA if available,
dataset accessibility status, and corresponding citations.

Ref. | Year | Source | Language | Total | Non-Aggressive | Aggressive | Classes | IAA (0-1) | Avail.? | Cites
---|---|---|---|---|---|---|---|---|---|---
(Chen et al., 2017) | 2017 | Tw and Fb | En | - | - | - | NAG, AG | - | N | $\geq 8$
(Kumar et al., 2018b) | 2018 | Tw and Fb | En, Hi |  | Tw: 18000
---
Fb: 21000
| Tw: 8982
---
FB: 8946
| Tw: {CAG: 7938,
---
OAG: 1080}
Fb: {CAG: 6279,
OAG: 5775}
| OAG, CAG,
---
NAG
KA: 0.49 | Y | $\geq 157$
(Chatzakou et al., 2019) | 2019 | Tw-user | En | 4954 | 3562 |  | AG: 59, Bully: 24,
---
Spam: 1300
| AG, Bully,
---
Spam, NAG
FK: 0.67 | Y | $\geq 77$
(Bhattacharya et al., 2020) | 2020 |  | YT, Fb,
---
and Tw
| En, Hi,
---
and Ba
20000 |  | Ba : 3312
---
Hi: 3148
En: 4901
| Ba: {CAG: 1341,
---
OAG: 1318}
Hi: {CAG: 1231,
OAG: 1802}
En: {CAG: 794,
OAG: 834}
| OAG, CAG,
---
NAG
KA: 0.75 | Y | $\geq 71$
(Srivastava et al., 2020) | 2020 | Tw | En | 1001 | 597 |  | CAG: 140,
---
OAG: 264
| OAG, CAG,
---
NAG
FK: 0.64 | Y | $\geq 1$
(Sadiq et al., 2021) | 2021 | Tw | En | 20001 | 12,179 | 7822 | AG, NAG | - | Y | $\geq 89$
(Torregrosa et al., 2022) | 2022 | Tw | Es | - | - | - |  | - | N | $\geq 4$
(Gattulli et al., 2022) | 2022 | Tw | It | 3028 | 1514 | 1514 | AG, NAG | - | N | $\geq 2$
(Sharif and Hoque, 2022) | 2022 | Fb and YT | Ba | 14158 | 7351 | 6807 | AG, NAG | CK: 0.87 | Y | $\geq 9$
(Aragón et al., 2020) | 2022 | Tw | Es | 7332 | 5222 | 2110 | AG, NAG | - | N | $\geq 25$
(Kumar et al., 2022) | 2022 |  | Tw, Fb
---
YT, and Tel
| En, Hi, Ba,
---
Mn, and Cm
59152 |  | Mn: 3221
---
Ba: 5875
Hi: 3684
En: 5798
Cm: 652
| OAG: {Mn: 7677,
---
Ba: 7523, Hi: 9866,
En: 2959, Cm: 1176},
CAG: {Mn: 4103,
Ba: 2360, Hi: 2482,
En: 1299, Cm: 477}
| NAG, OAG,
---
CAG
| KA:
---
0.27 - 0.93
Y | $\geq 14$
(Kumari and Singh, 2021) | 2022 |  | Tw, Fb,
---
and Insta
En with Img | 3600 | 1804 |  | MAG: 1327,
---
HAG: 469
| HAG, MAG,
---
NAG
- | N | $\geq 6$
(Mane et al., 2023) | 2023 | Tw | En | 6000 | 3398 | 2602 | AG, NAG | FK: 0.78 | Y | $\geq 0$
(Rawat et al., 2023) | 2023 | Tw | Cm-Hi | 2000 | 992 | CAG: 519, OAG: 489 |  | OAG, CAG,
---
NAG
CK: 0.76 | Y | $\geq 0$

In our dataset analysis, we provide key insights into the landscape of
aggression detection datasets within the realm of OSM platforms. As presented
in Figure 6, our observations reveal that a predominant portion of available
datasets pertains to Twitter, followed by Facebook and Youtube. Subsequently,
we conducted an analysis to determine which dataset has carried the most
attention and usage among researchers in their work. Our findings, illustrated
in Figure 7, highlight that the dataset that Kumar et al. (2018) (Kumar et
al., 2018b) introduced is the most widely adopted. Chatzakou et al. (2019)
(Chatzakou et al., 2019) is the second most referenced dataset, followed by
Sadiq et al. (2021) (Sadiq et al., 2021), Bhattacharya et al. (2020)
(Bhattacharya et al., 2020), Sharif and Hoque (2022) (Sharif and Hoque, 2022),
Kumar et al. (2022) (Kumar et al., 2022), and Rawat et al. (2023) (Rawat et
al., 2023). This analysis underscores the prevalence of research efforts
concentrated on Twitter and/or Facebook OSM platforms, predominantly utilizing
English and/or Hindi language datasets. Moreover, our language-specific
dataset analysis revealed that English language datasets are the most
prevalent, followed by code-mixed, Hindi, Bangla and Spanish datasets. Among
these top five languages, dataset availability is more extensive than others,
as illustrated in Figure 5a. Additionally, our analysis, as shown in Figure
5b, presents the size of language-specific datasets. An observation emerged,
with code-mixed datasets exhibiting considerable size, comparable to English
datasets. This observation aligns with prior research indicating a higher
incidence of aggressive behaviors in code-mixed contexts. Consequently, many
researchers are turning their attention towards developing code-mixed datasets
to enhance the effectiveness of aggression detection within this linguistic
realm. To provide an extensive and detailed overview of the datasets under
consideration, we present a comprehensive summary in Table 1. In conclusion,
it is imperative to expand the scope of research to include other major OSM
platforms and low resource languages. Recognizing the important role of
dataset size in the robustness of aggression detection models, there is a need
to increase dataset size in low-resource languages by incorporating different
contextual nuances.

###  4.3. Features

The identification of appropriate features is crucial in the advancement of
machine learning algorithms in the field of aggression detection. Feature
engineering, which involves identifying and leveraging specific properties, is
critical to enhancing the performance of aggression detection models. This
section review the various features utilized by researchers in the field.
Researchers have utilized many features to handle nuances in text, even when
dealing with low-source languages. They have also used a combination of
different features to capture the multidimensional aspects of a text. A
detailed breakdown of features used for aggression detection is provided in
Figure 8 and Table LABEL:tab:detection_table.

####  4.3.1. Stylistic Features

Stylistic features, often seen as crucial components of aggression detection,
offer valuable insights into the writing style, expression, and tone of
textual content. Research highlights that aggressive texts tend to exhibit
shorter and more fragmented sentences (Herodotou et al., 2021; Pascucci et
al., 2020; Golem et al., 2018). Additionally, they often employ simpler and
more repetitive language, as reflected by metrics like the type-token ratio
(TTR) (Pascucci et al., 2020). TTR measures the diversity of unique words
(types) in a text in relation to the total number of words (tokens). Word
length distribution analysis reveals that aggressive texts are characterized
by a prevalence of short and simple words (Herodotou et al., 2021; Singh et
al., 2018; Pascucci et al., 2020; Maitra and Sarkhel, 2018). Punctuation marks
(Pascucci et al., 2020; Fortuna et al., 2018; Ramiandrisoa and Mothe, 2018;
Tommasel et al., 2018), particularly excessive exclamation marks (Risch and
Krestel, 2018), asterisks (Risch and Krestel, 2018) capitalization (Singh et
al., 2018; Maitra and Sarkhel, 2018; Fortuna et al., 2018; Ramiandrisoa and
Mothe, 2018; Golem et al., 2018), and question marks, offer insight into the
emotional intensity or aggression of the content (Herodotou et al., 2021;
Huerta-Velasco and Calvo, 2022; Herodotou et al., 2021; Gattulli et al., 2022;
Singh et al., 2018; Sane et al., [n. d.]). Distinct linguistic patterns, such
as the frequent use of aggressive or negative adjectives and verbs, are
indicative of aggression in text (Herodotou et al., 2021; Pascucci et al.,
2020). Readability indices, including the Gunning Fog Index and Flesch Reading
Ease, further support this distinction, with aggressive content being less
complex and more straightforward (Pascucci et al., 2020; Ramiandrisoa and
Mothe, 2018). Figurative language and metaphors have been identified as
expressive of emotional states, including aggression (Pascucci et al., 2020).
In addition, the presence of swear words (Ramiandrisoa and Mothe, 2018; Golem
et al., 2018), and abusive/aggressive tokens (Singh et al., 2018; Gómez-Adorno
et al., 2018; Gattulli et al., 2022; Si et al., 2019; Mundra and Mittal, 2021)
serves as a potent signal of aggression within the text. Beyond linguistic
features, researchers have explored the inclusion of URLs (Singh et al., 2018;
Herodotou et al., 2021), hashtags (Singh et al., 2018; Sane et al., [n. d.];
Herodotou et al., 2021; Maitra and Sarkhel, 2018; Ramiandrisoa and Mothe,
2020a), mentions (Sane et al., [n. d.]; Herodotou et al., 2021), numbers
(Singh et al., 2018; Golem et al., 2018), and lexicons (Sane et al., [n. d.])
in the analysis. The inclusion of these elements, in combination with
linguistic and contextual features, contributes to the development of machine
learning models for aggression detection. In the pursuit of nuanced linguistic
style analysis, researchers have often combined these features. For instance,
one study integrated stylistic features, specifically the mean number of words
per sentence and mean word length, into their framework (Herodotou et al.,
2021). Another research effort leveraged COGIT, Expert System’s semantic
intelligence software, for word-sense disambiguation, adding to the evolving
landscape of aggression detection in text (Pascucci et al., 2020).

Sentence Length: (Herodotou et al., 2021; Pascucci et al., 2020; Golem et al.,
2018)
Word Length: (Herodotou et al., 2021; Singh et al., 2018; Pascucci et al.,
2020; Maitra and Sarkhel, 2018)
Special Symbol: (Pascucci et al., 2020; Fortuna et al., 2018; Ramiandrisoa and
Mothe, 2018; Tommasel et al., 2018; Risch and Krestel, 2018; Singh et al.,
2018; Maitra and Sarkhel, 2018; Golem et al., 2018; Herodotou et al., 2021;
Huerta-Velasco and Calvo, 2022; Gattulli et al., 2022; Sane et al., [n. d.])
Linguistic Patterns: (Herodotou et al., 2021; Pascucci et al., 2020;
Ramiandrisoa and Mothe, 2018)
Metaphor: (Pascucci et al., 2020)
Swear Words: (Ramiandrisoa and Mothe, 2018; Golem et al., 2018; Singh et al.,
2018; Gómez-Adorno et al., 2018; Gattulli et al., 2022; Si et al., 2019;
Mundra and Mittal, 2021) POS: (Huerta-Velasco and Calvo, 2022; Herodotou et
al., 2021; Iqbal and Keshtkar, 2019; Gómez-Adorno et al., 2018; Fortuna et
al., 2018; Si et al., 2019; Tommasel et al., 2019; Ramiandrisoa and Mothe,
2018; Golem et al., 2018)
N-grams: (Iqbal and Keshtkar, 2019; Gómez-Adorno et al., 2018; Sane et al.,
[n. d.]; Baruah et al., 2020; Tanase et al., 2020; Pascucci et al., 2020;
Risch and Krestel, 2018)
Emoticons: (Huerta-Velasco and Calvo, 2022; Si et al., 2019; Ramiandrisoa and
Mothe, 2018, 2020a)
LIWC: (Iqbal and Keshtkar, 2019; Samghabadi et al., 2020)
Emotional Traits: (Huerta-Velasco and Calvo, 2022; Orašan, 2018; Ramiandrisoa
and Mothe, 2018) One-hot: (Kumari et al., 2021; Srivastava et al., 2018;
Aroyehun and Gelbukh, 2018b; Raiyani et al., 2018; Kumari and Singh, 2020b)
TF-IDF: (Baruah et al., 2020; Datta et al., 2020; Chen et al., 2018; Tanase et
al., 2020; Tommasel et al., 2018; Shrivastava et al., 2021; Sadiq et al.,
2021; Orašan, 2018)
Word2Vec: (Tommasel et al., 2019; Khan et al., 2022; Sharif and Hoque, 2022;
Ramiandrisoa and Mothe, 2020a)
Doc2Vec: (Ramiandrisoa and Mothe, 2018)
Glove: (Tommasel et al., 2019; Orašan, 2018; Roy et al., 2018; Orabi et al.,
2018; Tommasel et al., 2018; Golem et al., 2018; Patwa et al., 2021;
Khandelwal and Kumar, 2020; Kumari and Singh, 2022)
FastText: (Modha et al., 2022; Srivastava and Khurana, 2019; Galery et al.,
2018; Modha et al., 2018; Aroyehun and Gelbukh, 2018a; Kumari and Singh,
2020b; Khandelwal and Kumar, 2020; Risch and Krestel, 2018) User Profile:
(Herodotou et al., 2021)
Network: (Herodotou et al., 2021)
Image: (Kumari and Singh, 2022) Stylistic Features Syntactic Features Token
Representations Sentiment Features Multimodal Features Features (Datta et al.,
2020; Huerta-Velasco and Calvo, 2022; Herodotou et al., 2021; Gattulli et al.,
2022; Iqbal and Keshtkar, 2019; Maitra and Sarkhel, 2018; Tommasel et al.,
2019; Orašan, 2018; Fortuna et al., 2018; Si et al., 2019; AGBAJE and Afolabi,
2022; Ramiandrisoa and Mothe, 2018; Tommasel et al., 2018; Golem et al., 2018)

Figure 8.  Taxonomy of features used for detect aggression in the literature.

####  4.3.2. Syntactic Features

Syntactic features focus on the structural aspects of language, shedding light
on how sentences are constructed and how language is employed to convey
aggression. This narrative outlines key syntactic features employed for
aggression detection, supported by relevant academic references. Analyzing the
distribution of various parts of speech, such as nouns, verbs, adjectives, and
Named Entity Recognition (NER) in a text can reveal patterns indicative of
aggression (Huerta-Velasco and Calvo, 2022; Herodotou et al., 2021; Iqbal and
Keshtkar, 2019; Gómez-Adorno et al., 2018; Fortuna et al., 2018; Si et al.,
2019; Tommasel et al., 2019; Ramiandrisoa and Mothe, 2018; Golem et al.,
2018). A higher frequency of aggressive verbs or adjectives can identify the
presence of aggressive content. The effectiveness of n-gram language models
that analyze combinations of words or characters has been demonstrated in the
identification of linguistic patterns associated with aggression. This
approach provides a comprehensive understanding of the text’s underlying
structure (Iqbal and Keshtkar, 2019; Gómez-Adorno et al., 2018; Sane et al.,
[n. d.]; Baruah et al., 2020; Tanase et al., 2020; Pascucci et al., 2020;
Risch and Krestel, 2018). In addition, the inclusion of parenthesis,
emoticons, or emojis within a written communication can serve as cues of
emotional or aggressive expression. The combination of syntactic analysis with
these formatting features has been shown to offer valuable insights (Huerta-
Velasco and Calvo, 2022; Si et al., 2019; Ramiandrisoa and Mothe, 2018,
2020a). Linguistic Inquiry and Word Count (LIWC2007) has been adapted to focus
on around 70-word categories associated with positive and negative emotions
and self-references, aiding in the nuanced exploration of the emotional
context within the aggressive text (Iqbal and Keshtkar, 2019; Samghabadi et
al., 2020). Various emotional traits, such as anger, disgust, and fear, have
been investigated in previous studies, focusing on classifying them as either
positive or negative discrete emotions. The inclusion of this category
enhances the comprehension of emotional aspects of aggression in texts
(Huerta-Velasco and Calvo, 2022; Orašan, 2018; Ramiandrisoa and Mothe, 2018;
Khan et al., 2022). The importance of identifying negation in assessing
aggression is well recognized since it can serve to moderate its expression
(Ramiandrisoa and Mothe, 2018). The incorporation of these syntactic features,
in conjunction with additional linguistic and contextual features, serves a
substantial role in the advancement of machine learning models that possess
the ability to detect linguistic patterns associated with aggressiveness in
text (Huerta-Velasco and Calvo, 2022). In an effort to deepen the
understanding of aggression, researchers have introduced affective features
and dimensional features, including valence, pleasantness, and attention, into
the analysis (Huerta-Velasco and Calvo, 2022). Gender-related information was
also incorporated into feature vectors in certain studies. (Waseem, 2016)
introduced the concept of gender probability, calculated using a Twitter-based
lexicon by (Sap et al., 2014). This probability was then transformed into
binary gender, categorizing cases as either female or male. These gender-
related features were integrated into the feature vectors, enhancing the
features used for aggression detection.

####  4.3.3. Sentiment Features

Sentiment features play a crucial role in distinguishing the emotional tone of
the content, thereby determining whether it conveys positive, negative, or
neutral sentiments. Several studies have contributed to the ongoing
advancements in this field (Datta et al., 2020; Huerta-Velasco and Calvo,
2022; Herodotou et al., 2021; Gattulli et al., 2022; Iqbal and Keshtkar, 2019;
Maitra and Sarkhel, 2018; Tommasel et al., 2019; Orašan, 2018; Fortuna et al.,
2018; Si et al., 2019; AGBAJE and Afolabi, 2022; Ramiandrisoa and Mothe, 2018;
Tommasel et al., 2018; Golem et al., 2018). Sentiment analysis was used to
obtain fine-grained sentiment distributions for textual comments (Samghabadi
et al., 2020). They employed sentiment distribution mean and standard
deviation for every message as feature vectors for aggression detection.
Several research combined sentiment features with other textual qualities.
According to (Maitra and Sarkhel, 2018), a K-competitive autoencoder named
Emoti-KATE increased aggression detection by using sentiment scores, word
count, capitalization, and hashtag analysis as initial features. It is
essential to note that while these approaches were effective for specific
language corpora, such as English, they faced challenges when applied to
languages like Hindi, as observed in a study by (Samghabadi et al., 2020).
This underscores the importance of considering linguistic and cultural nuances
in aggression detection.

####  4.3.4. Token Embedding and Representation

Word and character embedding features have emerged as highly promising tools
in the field of aggression detection. Many researchers have explored the
utility of various word embedding techniques to enhance the performance of
machine learning models in identifying aggressive language. Notably, pre-
trained word embeddings such as FastText (Bojanowski et al., 2017), Glove
(Pennington et al., 2014), Word2Vec (Mikolov et al., 2013), Doc2Vec (Le and
Mikolov, 2014), as well as traditional text representations like TF-IDF
(Aizawa, 2003) and one-hot encoding, play a pivotal role in this context.
Although not typically fine-tuned for specific aggression detection tasks,
these embeddings and representations prove effective due to their ability to
capture essential semantic and syntactic information from the text. One-hot
encoding, for instance, transforms words into binary vectors to indicate their
presence or absence in a document, making it effective at identifying isolated
words or phrases indicative of aggression (Kumari et al., 2021; Srivastava et
al., 2018; Aroyehun and Gelbukh, 2018b; Raiyani et al., 2018; Kumari and
Singh, 2020b). TF-IDF assigns weights to words based on their occurrence
frequency in a document relative to the entire corpus, thereby highlighting
distinctive aggressive terms within specific texts (Baruah et al., 2020; Datta
et al., 2020; Chen et al., 2018; Tanase et al., 2020; Tommasel et al., 2018;
Shrivastava et al., 2021). It can consider word unigrams, word bigrams, and
even terms such as emoticons (Sadiq et al., 2021; Orašan, 2018). Word2Vec
excels in capturing semantic information and recognizing subtle cues and
similarities in the usage of aggressive terms. It aids in identifying
aggressive language that might share semantic characteristics with known
aggressive terms, even if not explicitly labeled (Tommasel et al., 2019; Khan
et al., 2022; Sharif and Hoque, 2022; Ramiandrisoa and Mothe, 2020a; Raman et
al., 2022). Doc2Vec, on the other hand, generates document-level embeddings
that encapsulate the semantic content of entire documents or text segments,
offering valuable insights into the overall tone and sentiment of the text in
aggression detection (Ramiandrisoa and Mothe, 2018). Glove word embeddings
provide a contextual understanding of words within a document, enabling the
capture of nuances in aggression by considering the relationships between
words in a sentence or text segment. This contextual comprehension proves
essential in identifying aggressive language across varied contexts (Tommasel
et al., 2019; Orašan, 2018; Roy et al., 2018; Orabi et al., 2018; Tommasel et
al., 2018; Golem et al., 2018; Patwa et al., 2021; Khandelwal and Kumar, 2020;
Kumari and Singh, 2022). FastText, with its capability to capture subword
information, is particularly valuable for handling colloquial language often
found in aggressive online communication. It can understand and classify
aggressive language, even when it involves misspelled or combined words (Modha
et al., 2022; Srivastava and Khurana, 2019; Galery et al., 2018; Modha et al.,
2018; Aroyehun and Gelbukh, 2018a; Kumari and Singh, 2020b; Khandelwal and
Kumar, 2020; Risch and Krestel, 2018; Mane et al., 2023). Combining FastText
with HurtLex, a multilingual lexicon of offensive words (Galery et al., 2018),
can further enhance its effectiveness. (Mut Altın et al., 2020) harnessed word
vectors from a pre-trained model based on tokens, emojis, and hashtags from 20
million English tweets, while (Tawalbeh et al., 2020) utilized pre-trained
sentence Universal Sentence Encoder (USE) embeddings, and (Kumar et al.,
2018a) leveraged pre-trained embeddings from Deepmoji (Felbo et al., 2017).

(a) Utilization of traditional token embeddings and representations. (b)
Utilization of types of features.

Figure 9. Distribution analysis of features utilized in previous studies for
aggression analysis.

####  4.3.5. Data augmentation and Mulitmodal Featuers

To address data unavailability and data balancing challenges, researchers have
turned to data augmentation techniques, including machine-translated data
(Akter et al., 2022; Risch and Krestel, 2018; Ta et al., 2022b), GPT-2
(Shrivastava et al., 2021) and pseudo-labeling (Aroyehun and Gelbukh, 2018a).
Multimodal features for aggression detection, which combine Image, network,
and social media user profile features with text features. These features
include in-degree and out-degree user ego networks, account age, number of
user posts, and the number of lists subscribed to (Herodotou et al., 2021).
Hybrid feature sets, combining image and text features, have been optimized
using algorithms such as binary firefly optimization (BFFO), with VGG16 used
for image features and Glove for text features (Kumari and Singh, 2022). This
approach shows the wide range of features employed in previous studies to
develop robust aggression detection models on social media platforms.

Our analysis of previous research on features used in the field of aggression
detection highlights the prevalent use of traditional token embeddings and
representations. It clearly emphasizes FastText and Glove, which account for
approximately 50% of the selected features. TF-IDF, although less dominant,
still has a significant percentage at 22%, as shown in Figure 9a. Moreover, we
observed a clear hierarchy after analyzing the popularity of different
features in the reviewed studies. Token embeddings and representations claim
the foremost position, accounting for about 37% of feature selections,
followed by emotion features, which contribute about 20% of the feature
landscape. In third place, syntactic features account for approximately 19% of
the total feature usage, as shown in Figure 9b. These observations highlight
the importance of token embedding, emotion features, and syntactic properties
in detecting affective aggression and provide valuable insights for future
research in this domain.

(a) Distribution of aggression detection studies across languages with
comparative percentage values. (b) Comparative percentage distribution of
machine learning algorithms employed in aggression detection studies.

Figure 10. Distribution analysis of detection studies across various languages
and ML algorithms.

###  4.4. Algorithms

In this section, we explore various algorithms for detecting aggressiveness in
diverse languages and code-mixed content. Algorithms have been developed to
address aggressiveness in multiple languages, such as English (En) (Samghabadi
et al., 2018; Ramiandrisoa and Mothe, 2018; Raiyani et al., 2018; Nikhil et
al., 2018; Kumar et al., 2018a; Roy et al., 2018; Galery et al., 2018; Maitra
and Sarkhel, 2018; Aroyehun and Gelbukh, 2018a; Srivastava et al., 2018;
Orašan, 2018; Aroyehun and Gelbukh, 2018b; Fortuna et al., 2018; Arroyo-
Fernández et al., 2018; Risch and Krestel, 2018; Orabi et al., 2018; Tommasel
et al., 2018; Golem et al., 2018; Modha et al., 2018; Singh et al., 2018;
Patwa et al., 2021; Modha et al., 2022; Tommasel et al., 2019; Srivastava and
Khurana, 2019; Ramiandrisoa and Mothe, 2020a, b; Liu et al., 2020; Pascucci et
al., 2020; Mut Altın et al., 2020; Tawalbeh et al., 2020; Risch and Krestel,
2020; Baruah et al., 2020; Datta et al., 2020; Gordeev and Lykova, 2020;
Koufakou et al., 2020; Kumari and Singh, 2020b; Mishra et al., 2020;
Samghabadi et al., 2020; Khandelwal and Kumar, 2020; Chen et al., 2018; Zhao
et al., 2021; Kumari et al., 2021; Dutta et al., 2021, 2021; Herodotou et al.,
2021; Akter et al., 2022; Ramiandrisoa, 2022; Khan et al., 2022; Kumari and
Singh, 2022; AGBAJE and Afolabi, 2022; Iqbal and Keshtkar, 2019; Ghosh et al.,
2023; Rawat et al., 2023), Hindi (Hi) (Ramiandrisoa and Mothe, 2018; Raiyani
et al., 2018; Nikhil et al., 2018; Roy et al., 2018; Maitra and Sarkhel, 2018;
Risch and Krestel, 2018; Modha et al., 2018; Risch and Krestel, 2020; Baruah
et al., 2020; Datta et al., 2020; Gordeev and Lykova, 2020; Koufakou et al.,
2020; Kumari and Singh, 2020b; Mishra et al., 2020; Samghabadi et al., 2020;
Khandelwal and Kumar, 2020; Akter et al., 2022; Ghosh et al., 2023), Bangla
(Bn) (Risch and Krestel, 2020; Baruah et al., 2020; Datta et al., 2020;
Gordeev and Lykova, 2020; Koufakou et al., 2020; Kumari and Singh, 2020b;
Mishra et al., 2020; Samghabadi et al., 2020; Khandelwal and Kumar, 2020;
Akter et al., 2022; Sharif and Hoque, 2022; Ghosh et al., 2023), Spanish (Es)
(Gómez-Adorno et al., 2018; Gutiérrez-Esparza et al., 2019; Tanase et al.,
2020; Guzman-Silverio et al., 2020; Tonja et al., 2022; Ta et al., 2022b, a;
Huerta-Velasco and Calvo, 2022), Italian (It) (Gattulli et al., 2022), Russian
(Ru) (Shulginov et al., 2021), and even in code-mixed versions like Hindi-
English (Hi-En) (Aroyehun and Gelbukh, 2018a; Si et al., 2019; Shrivastava et
al., 2021; Mundra and Mittal, 2021; Sengupta et al., 2022) and Tamil-English
(Ta-En) (Sane et al., [n. d.]). In the recent past, some events have attracted
the attention of the research community towards the detection of aggression.
These include the First and Second Workshop on Trolling, Aggression, and
Cyberbullying (TRAC-2018(Kumar et al., 2018b) and TRAC-2020 (Bhattacharya et
al., 2020)) and the DA-VINCIS competition (Detection of Aggressive and Violent
INCIdents from Social Media in Spanish) at IberLEF 2022 (Arellano et al.,
2022). In Figure 11, we present the breakdown of algorithms used in previous
research for the detection of aggressive content. Further, Table
LABEL:tab:detection_table provide a detailed year-wise analysis of aggression
detection algorithms, describing their respective datasets, performance
metrics, and their status in the research community as estimated by citation
counts. A distribution analysis of previous studies on aggression detection in
different languages is presented in Figure 10a. Our analysis reveals that the
majority of aggression detection studies are in English, representing 53%.
Hindi follows at 18%, Bangla at 12%, Spanish at 8%, and Hindi-English code-
mixed at 5%. This data highlights a clear preference for English in the
research on aggression detection.

LR: (Samghabadi et al., 2018; Ramiandrisoa and Mothe, 2018; Golem et al.,
2018; Shrivastava et al., 2021)
GBM: (Datta et al., 2020; Si et al., 2019)
XGBoost: (Datta et al., 2020; Si et al., 2019; Tawalbeh et al., 2020;
Herodotou et al., 2021; Dutta et al., 2021)
RF: (Ramiandrisoa and Mothe, 2018; Orašan, 2018; Fortuna et al., 2018;
Gattulli et al., 2022; Iqbal and Keshtkar, 2019; Kumari and Singh, 2022)
SVM: (Orašan, 2018; Tommasel et al., 2018; Golem et al., 2018; Arroyo-
Fernández et al., 2018; Gómez-Adorno et al., 2018; Si et al., 2019; Tommasel
et al., 2019; Sane et al., [n. d.]; Pascucci et al., 2020; Baruah et al.,
2020; Shrivastava et al., 2021; Huerta-Velasco and Calvo, 2022)
NB: (Shrivastava et al., 2021)
OneR: (Gutiérrez-Esparza et al., 2019) DNN: (Khan et al., 2022; Raiyani et
al., 2018; Maitra and Sarkhel, 2018; Tommasel et al., 2018; Sadiq et al.,
2021)
CNN: (Roy et al., 2018; Aroyehun and Gelbukh, 2018a, b; Orabi et al., 2018;
Modha et al., 2018; Singh et al., 2018; Patwa et al., 2021; Aroyehun and
Gelbukh, 2018a; Modha et al., 2022; Kumari et al., 2019; Kumari and Singh,
2020b; Mundra and Mittal, 2021; Sharif and Hoque, 2022; Sengupta et al., 2022;
Kumari and Singh, 2022; Chen et al., 2018; Shrivastava et al., 2021;
Khandelwal and Kumar, 2020; Raman et al., 2022)
LSTM: (Nikhil et al., 2018; Aroyehun and Gelbukh, 2018a, b; Tommasel et al.,
2018; Golem et al., 2018; Modha et al., 2018; Aroyehun and Gelbukh, 2018a;
Koufakou et al., 2020; Kumari and Singh, 2020b; Kumari et al., 2021; Mundra
and Mittal, 2021; Sengupta et al., 2022, 2022; Shrivastava et al., 2021;
Khandelwal and Kumar, 2020)
BiLSTM: (Kumar et al., 2018a; Aroyehun and Gelbukh, 2018b; Golem et al., 2018;
Srivastava and Khurana, 2019; Mut Altın et al., 2020; Sharif and Hoque, 2022;
Mane et al., 2023)
GRU: (Risch and Krestel, 2018; Raman et al., 2022)
BiGRU: (Galery et al., 2018) Traditional ML DL Ensemble Approaches
Transformer-based Algorithms CNN+Traditional ML: (Tommasel et al., 2018; Si et
al., 2019; Patwa et al., 2021)
Bi-/LSTM + CNN: (Golem et al., 2018; Shrivastava et al., 2021; Khandelwal and
Kumar, 2020; Aroyehun and Gelbukh, 2018a)
BERTs: (Ramiandrisoa and Mothe, 2020b; Liu et al., 2020; Risch and Krestel,
2018; Sharif and Hoque, 2022; Guzman-Silverio et al., 2020)
CNN+Img-Transformer: (Kumari and Singh, 2022)
BERT: (Ramiandrisoa and Mothe, 2020a; Liu et al., 2020; Risch and Krestel,
2018; Baruah et al., 2020; Mishra et al., 2020; Samghabadi et al., 2020;
Guzman-Silverio et al., 2020; Dutta et al., 2021; Akter et al., 2022)
m-BERT: (Risch and Krestel, 2018; Gordeev and Lykova, 2020; Mishra et al.,
2020; Samghabadi et al., 2020; Akter et al., 2022; Sharif and Hoque, 2022; Ta
et al., 2022b)
DistilBERT: (Sharif and Hoque, 2022)
RoBERTa: (Ramiandrisoa, 2022; Rawat et al., 2023)
XLMRoBERTa: (Baruah et al., 2020; Tanase et al., 2020; Zhao et al., 2021;
Sharif and Hoque, 2022; Ghosh et al., 2023)
Lang-BERT: (Sharif and Hoque, 2022; Rawat et al., 2023; Shulginov et al.,
2021; Tonja et al., 2022)

Figure 11.  Taxonomy of algorithms utilized for aggression detection in the
literature. Transformer-based algorithms are a subset of deep learning
algorithms. The Lang-BERT represents language-specific BERT variants.

####  4.4.1. Machine learning

Machine learning (ML) algorithms have emerged as a foundation of aggression
detection methodologies. Traditional ML algorithms, recognized for their
adaptability to language nuances and the intricacies of online communication,
have been extensively analyzed. These algorithms include Logistic Regression
(LR) (Samghabadi et al., 2018; Ramiandrisoa and Mothe, 2018; Golem et al.,
2018; Shrivastava et al., 2021), Gradient Boosting Machine (GBM) (Datta et
al., 2020; Si et al., 2019), Random Forest (RF) (Ramiandrisoa and Mothe, 2018;
Orašan, 2018; Fortuna et al., 2018; Gattulli et al., 2022; Iqbal and Keshtkar,
2019; Kumari and Singh, 2022), eXtreme Gradient Boosting (XGBoost) (Datta et
al., 2020; Si et al., 2019; Tawalbeh et al., 2020; Herodotou et al., 2021;
Dutta et al., 2021), Support Vector Machine (SVM) (Orašan, 2018; Tommasel et
al., 2018; Golem et al., 2018; Arroyo-Fernández et al., 2018; Gómez-Adorno et
al., 2018; Si et al., 2019; Tommasel et al., 2019; Sane et al., [n. d.];
Pascucci et al., 2020; Baruah et al., 2020; Shrivastava et al., 2021; Huerta-
Velasco and Calvo, 2022), Naive Bayes (NB) (Shrivastava et al., 2021), and One
Rule (OneR) (Gutiérrez-Esparza et al., 2019). Researchers have employed LR for
multi-class classification tasks, using the one-versus-rest classifier to
predict the most probable class for each comment (Samghabadi et al., 2018).
SVM has been widely adopted, with variations such as the Radial Basis Function
(RBF) Kernel playing a pivotal role in aggressive content identification
(Tommasel et al., 2019; Sane et al., [n. d.]). Passive-aggressive learning
algorithms have also made their mark with SVM and SGD. The Passive-Aggressive
Learning algorithm updates weight vectors that parameterize a linear decision
boundary. This algorithm is considered ‘aggressive’ when it introduces
training errors and ‘passive’ when it does not (Arroyo-Fernández et al.,
2018). Furthermore, sequential minimal optimization (SMO) algorithms have been
deployed for classification, notably using 10-fold cross-validation to ensure
the robustness of aggression detection models (Pascucci et al., 2020). Real-
time streams of Twitter data have been subjected to the exploration of
streaming ML algorithms, including Hoeffding Tree (HT), Adaptive Random Forest
(ARF), and Streaming Logistic Regression (SLR) (Herodotou et al., 2021).

Further, the distribution analysis of machine learning algorithms employed in
previous aggression detection studies is presented in Figure 10b. Our analysis
reveals that SVMs are the most commonly employed machine learning algorithm at
38%, underscoring their effectiveness in aggression detection. RF is the
second choice at 19%, indicating its popularity for this task. XGBoost and LR
follow with 16% and 12% usage, respectively, while GBM are used in 6% of
studies. NB and OneR are less frequent choices, both at 3%.

####  4.4.2. Deep learning

Deep learning techniques have significantly advanced aggression detection by
harnessing the capabilities of neural networks. The Deep Neural Network (DNN)
plays a fundamental role, and its architecture has been explored with
variations in the number of layers used (Khan et al., 2022; Raiyani et al.,
2018; Maitra and Sarkhel, 2018; Tommasel et al., 2018; Sadiq et al., 2021).
These DNN structures provide the foundation for the development of more
advanced models. Researchers have explored various models with a focus on
sequence-to-sequence architectures. The Long Short-Term Memory (LSTM) model
has been a prominent choice due to its recurrent design, which is adept at
capturing temporal dependencies (Nikhil et al., 2018; Aroyehun and Gelbukh,
2018a, b; Tommasel et al., 2018; Golem et al., 2018; Modha et al., 2018;
Aroyehun and Gelbukh, 2018a; Koufakou et al., 2020; Kumari and Singh, 2020b;
Kumari et al., 2021; Mundra and Mittal, 2021; Sengupta et al., 2022, 2022;
Shrivastava et al., 2021; Khandelwal and Kumar, 2020; Karan and Kundu, 2023).
Variations such as the Bidirectional LSTM (BiLSTM) have also been studied,
leveraging both forward and backward information flow to enhance contextual
understanding (Kumar et al., 2018a; Aroyehun and Gelbukh, 2018b; Golem et al.,
2018; Srivastava and Khurana, 2019; Mut Altın et al., 2020; Sharif and Hoque,
2022; Mane et al., 2023). Gated Recurrent Unit (GRU) (Risch and Krestel, 2018;
Raman et al., 2022), and Bidirectional GRU(BiGRU) (Galery et al., 2018) models
have also been utilized. Beyond recurrent models, traditional Convolutional
Neural Networks (CNNs) have proven effective in processing and extracting
features from textual data in aggression detection tasks (Roy et al., 2018;
Aroyehun and Gelbukh, 2018a, b; Orabi et al., 2018; Modha et al., 2018; Singh
et al., 2018; Patwa et al., 2021; Aroyehun and Gelbukh, 2018a; Modha et al.,
2022; Kumari et al., 2019; Kumari and Singh, 2020b; Mundra and Mittal, 2021;
Sharif and Hoque, 2022; Sengupta et al., 2022; Kumari and Singh, 2022; Chen et
al., 2018; Shrivastava et al., 2021; Khandelwal and Kumar, 2020; Raman et al.,
2022). Additionally, the incorporation of autoencoders (Maitra and Sarkhel,
2018; Kumari et al., 2021) and attention (Nikhil et al., 2018; Mut Altın et
al., 2020; Sengupta et al., 2022) mechanisms has further enriched the field.
The utilization of DNN with the significance of layer counts and activation
function order is essential for aggression identification (Khan et al., 2022;
Sadiq et al., 2021; Raiyani et al., 2018). LSTM models, with their recurrent
architecture, were introduced to capture temporal dependencies, and an
attention-based classifier was incorporated to determine critical features
(Maitra and Sarkhel, 2018; Nikhil et al., 2018). LSTM autoencoder module,
achieving state-of-the-art results (Kumari et al., 2021). BiLSTM layers with
global max pooling and dropout management offered a nuanced understanding of
word context in aggression (Kumar et al., 2018a; Galery et al., 2018). LSTM
established efficiency in analyzing Facebook data, while the combination of
CNN and LSTM proved superior for code-mixed Hindi-English aggression detection
(Aroyehun and Gelbukh, 2018a). (Srivastava et al., 2018) proposed a capsule
network with focal loss, and (Patwa et al., 2021) utilized capsule networks
followed by CNN for aggression detection. Similarly, (Srivastava and Khurana,
2019) combined BiLSTM with Capsule Networks to expand the array of
possibilities for aggression detection. Character-based C-hierarchical
attention networks (Sengupta et al., 2022) and proposed approaches featuring
Deep Pyramid CNN, Pooled LSTM, and Disconnected RNN (Khandelwal and Kumar,
2020). Multi-convolutional neural network with pooling (MultiCNNPooling),
diversifying architectural strategies (Orabi et al., 2018). To address
character-level embeddings for code-mixed Hindi-English text, (Mundra and
Mittal, 2021) suggested a character-level embedding approach using CNN and
LSTM. Researchers addressed aggression in a multimodal context by combining
text features with LSTM and image features with CNN (Sengupta et al., 2022).
Additionally, another study utilized VGG-16 for extracting image features and
CNN for extracting text features (Kumari and Singh, 2022). The aim was to
tackle the complexities posed by the multi-dimensional nature of the digital
environment.

Figure 12a reveals the distribution of deep learning algorithms in aggression
detection studies. CNNs are the most favored at 37%, emphasizing their
proficiency in processing textual cues. LSTMs are the second choice at 31%,
indicating the importance of temporal dynamics. DNNs, used in 14% of studies,
offer a more generalized approach. BiLSTMs at 12% capture contextual nuances
effectively. GRUs and BiGRUs, together at 2%, are less common, possibly due to
their reduced complexity. These insights aid researchers and practitioners in
algorithm selection for aggression detection tasks.

(a) Utilization of Deep learning algorithms. (b) Utilization of Transformer-
based algorithms.

Figure 12. Comparative percentage distribution of DL and transformer-based
algorithms used in studies focused on aggression detection.

####  4.4.3. Transformer-based

The increasing prevalence of Transformer-based models is leading to a
significant transformation in the landscape of aggression studies and
understood on social media platforms. Researchers are progressively utilizing
the capabilities of these models to detect various forms of aggressive
language across multiple languages. Transformer-based models are commonly pre-
trained using large datasets to enhance their abilities in identifying
aggression. This pre-training process helps fine-tune the models specifically
for the task of aggression detection. In the current era of advancements in
transformer-based models, Bidirectional Encoder Representations from
Transformers (BERT) have gained significant prominence in aggression detection
(Ramiandrisoa and Mothe, 2020a; Liu et al., 2020; Risch and Krestel, 2018;
Baruah et al., 2020; Mishra et al., 2020; Samghabadi et al., 2020; Guzman-
Silverio et al., 2020; Dutta et al., 2021; Akter et al., 2022). The
Multilingual BERT (m-BERT) extends the utility of these models across
linguistic boundaries (Risch and Krestel, 2018; Gordeev and Lykova, 2020;
Mishra et al., 2020; Samghabadi et al., 2020; Akter et al., 2022; Sharif and
Hoque, 2022; Ta et al., 2022b). Distilled versions of BERT (DistilBERT) offer
a more lightweight alternative (Sharif and Hoque, 2022). Robustly Optimized
BERT Approach (RoBERTa) (Ramiandrisoa, 2022; Rawat et al., 2023), Cross-
lingual Language Model RoBERTa (XLMRoBERTa) (Baruah et al., 2020; Tanase et
al., 2020; Zhao et al., 2021; Sharif and Hoque, 2022; Ghosh et al., 2023), and
language-specific variants of BERT, such as BERT pre-trained on Bangla
(BanglaBERT) (Sharif and Hoque, 2022), BERT pre-trained on code-mixed Hindi-
English (Hing-BERT) (Rawat et al., 2023), BERT pre-trained on Russian (RuBERT)
(Shulginov et al., 2021), and DistilBERT pre-trained on Spanish (DistilBETO)
(Tonja et al., 2022) extend the applicability of transformer-based models,
catering to languages and code-mixed text with unique linguistic traits. These
variants prove invaluable in understanding aggression in diverse linguistic
landscapes. The landscape of transformer-based models also includes Generative
Pretrained Transformer 2 (GPT-2), which explores text generation and provides
valuable insights into understanding and detecting aggressive language (Akter
et al., 2022). Incorporating attention mechanisms into transformer models
enhances the depth of comprehension, enabling a concentrated focus on critical
aspects of aggression detection (Samghabadi et al., 2020; Ghosh et al., 2023).
Additionally, researchers find that maintaining class balance through the
inclusion of instances from external datasets consistently improves model
performance, highlighting the adaptability of transformer-based models
(Ramiandrisoa and Mothe, 2020b). In multi-task learning, (Ramiandrisoa, 2022)
presents a model designed simultaneously to identify English aggression and
hate speech using Roberta models. While each function retains its dedicated
model, using shared encoder weights achieves a unified approach to input
handling, enriching the text with a multidimensional perspective. Similarly,
the study by (Ghosh et al., 2023) tackles multi-task learning challenges
associated with the identification of aggression and offensive content. The
shared XLMRoBERTa model captures common task features, while individual multi-
head self-attention networks define task-specific characteristics, showcasing
versatility in handling diverse dimensions of aggression and offensive
language. (Bansal et al., 2022) introduced XLM-RoBERTa with BiGRU,
incorporating emoji embedding, for aggressive (abusive) language detection
across 13 indic code-mixed languages. Emphasizing the global applicability of
transformer-based models, (Ta et al., 2022a) used GAN-BERT for Spanish
aggression detection. This underscores the ongoing evolution of these models
as they continuously evolve to address the complexities of online aggression.
In Figure 12b, BERT stands out as the most widely used transformer-based
algorithm in aggression detection studies, representing 26% of the total.
M-BERT follows closely at 20%, and XLMRoBERTa at 18%. RoBERTa is employed in
7% of the studies. These findings highlight the prevalence of transformer-
based models, especially BERT, in this research domain.

####  4.4.4. Ensemble-based

Ensemble-based models have gained prominence in aggression detection due to
their ability to enhance classification accuracy by combining multiple models.
This surge in interest illustrates the dynamic nature of this field,
particularly in the pursuit of effective solutions for identifying and
categorizing aggressive text, especially in multilingual and code-mixed
contexts. The ensemble approach by (Tommasel et al., 2018) utilized three
layers of LSTM, DNN, and SVM, effective for aggression identification.
Ensemble methods expanded further with (Golem et al., 2018), proposing a
comprehensive model combining BiLSTM, CNN, LR, and SVM for aggression
detection. Similarly, (Si et al., 2019) introduced a majority voting ensemble
model, employing XGBoost, GBM, and SVM for code-mixed Hindi-English
aggression. (Shrivastava et al., 2021) adopted a similar approach,
incorporating SVM, NB, LR, CNN, and LSTM into their ensemble model, extending
its utility to code-mixed Hindi-English aggression detection. A combination of
a three-capsule network with CNN was proposed for aggression detection (Patwa
et al., 2021). (Ramiandrisoa and Mothe, 2020b) integrated the three uncased-
BERT-base variants, each along with a low-dimensional multi-head attention
layer known as Projected Attention Layers (PALs)(Stickland and Murray, 2019),
further they fine-tuned three uncased-BERT-base pre-trained transformers using
majority voting strategies for English language aggression detection. To
enhance comprehensiveness, (Liu et al., 2020) proposed an ensemble model
featuring three uncased-BERT-base variants, each with different feature sets.
(Risch and Krestel, 2018) achieved notable results by combining 20 uncased-
BERT models tailored for specific tasks and languages, recognizing linguistic
diversity in social media conversations. In the context of code-mixed
languages, (Khandelwal and Kumar, 2020) addressed the challenge with an
average ensemble model that leveraged the Deep Pyramid CNN, Pooled LSTM, and
Disconnected RNN, offering a holistic solution for code-mixed aggression
detection. (Sharif and Hoque, 2022) proposed a weighted-ensemble model that
amalgamated m-BERT, distill-BERT, Bangla-BERT, and XLM-R for Bangla aggression
detection through a weighted F1 score and softmax probability-driven approach.
For Spanish aggression detection, (Guzman-Silverio et al., 2020) employed a
weighted ensemble of 20 pre-trained BERT models, each fine-tuned to the
specific task, demonstrating the adaptability of ensemble techniques to
different languages. The multifaceted nature of aggression detection extended
to the fusion of image and text features in the work of (Kumari and Singh,
2022). Their model incorporated VGG-16 for image features and CNN for text
features, with Random Forest (RF) used for aggression identification in
English text. (Aroyehun and Gelbukh, 2018a) integrated a CNN-LSTM model,
trained on a combination of augmented training data and pseudo-labeled Twitter
data, demonstrating the adaptability of ensemble techniques in utilizing
diverse sources of information.

###  4.5. State-of-the-art Approaches

In this section, we discussed the leading-edge methods in aggression
detection. These state-of-the-art (SOTA) approaches are the ones that have
demonstrated outstanding performance on relevant aggression detection
datasets, highlighting the latest progress in this field. We have examined the
state-of-the-art approaches for the three most widely utilized datasets.
(Kumari et al., 2021) has contributed significantly to the state-of-the-art by
achieving remarkable performance on the (Kumar et al., 2018b) dataset,
including English and Hindi data. Their approach relies on a sophisticated
LSTM autoencoder module, augmented with one-hot text representations with
randomly initialized embeddings. The module comprises three essential units:
the Encoder, Repeat Vector, and Decoder. The Encoder includes three LSTM
layers (LSTM1, LSTM2, and LSTM3), progressively enhancing the complexity of
the representation. Similarly, the decoder consists of LSTM4, LSTM5, and
LSTM6, reflecting the complexity of the encoder layers. The Repeat Vector
ensures uniform representation across the decoder’s LSTM nodes. They achieved
an exceptional performance, quantified by the weighted F1-score (W-F1):
Facebook (Fb) - English: 81%, Hindi: 72%, Twitter (Tw) - English: 71%, Hindi:
0.58%. (Risch and Krestel, 2020) has established state-of-the-art performance
on (Bhattacharya et al., 2020) dataset, which include English, Hindi, and
Bangla languages. Their method employs an ensemble-based BERT model,
incorporating numerous models customized for different linguistic and task
contexts. They use 20 uncased-BERT models for English aggression detection and
25 cased BERT models for English misogyny detection. Multilingual aspects are
addressed with 15 uncased-m-BERT models for Hindi aggression and misogyny
detection. Likewise, 15 uncased-m-BERT models are employed for Bangla
aggression detection, while 25 uncased-m-BERT models are used for Bangla
misogyny detection. Their achievement is reflected in a comprehensive
performance measure: W-F1, showcasing excellence in tasks: English -
Aggression: 80.29%, Misogyny: 85.14%; Hindi - Aggression: 81.28%, Misogyny:
87.81%; Bangla - Aggression: 82.19%, Misogyny: 93.85%. (Sharif and Hoque,
2022) has made notable advancements in the domain of Bangla aggression
detection using Facebook and YouTube data. They introduce a weighted-ensemble
model, combining m-BERT, distill-BERT, Bangla-BERT, and XLM-R. The weighted
ensemble aggregates the weighted F1 score and softmax probabilities from each
model within the ensemble. The outcome is remarkable, measured by a W-F1 score
of 93.43% for aggression detection and 93.11% for misogyny detection. Our
observations highlight the prevalent use of transformer-based models,
particularly BERT and its multilingual variants, which have shown remarkable
effectiveness in handling various linguistic nuances. Furthermore, ensemble-
based approaches have gained prominence, combining different models tailored
for specific tasks and languages. This reflects a commitment to addressing the
multifaceted nature of aggression detection and achieving high performance
across a range of datasets. The rigorous evaluation of performance, as
indicated by metrics such as the weighted F1-score. This metrics underscores
the importance of precise and comprehensive evaluation in this domain.

Figure 13 presents the evolution of aggression detection from 2018 to 2023 and
reveals a progression from traditional models to transformative shift
transformers. Significant improvements in performance were achieved through
ensemble methods, holistic approaches, and further multi-task learning. The
refinement in 2023 with multi-head self-attention and the dominance of BERT-
based algorithms marks an evolved state in aggression detection.

Figure 13. Evolution of aggressive content detection over the last six years: Analyzing progress and trends in computational approaches. The approximate ($\sim$) performance is shown in percentage (%). Table 2. Provides a detailed overview of research efforts to detect aggressive content for specific datasets. The table includes information on the targeted languages, the effective features employed, the proposed algorithm and its performance performance, and the number of citations. For clarity on abbreviations, refer to Appendix 3. |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
Ref. | Dataset | Lang. | Year | Features | Algorithms | Results | Cites
(Samghabadi et al., 2018) | (Kumar et al., 2018b) |  | En,
---
Hi
2018 |  | En: Lexical, Semantic
---
Hi: Lexical
LR |  | W-F1: {Fb: 0.5921,
---
Tw: 0.6451}
$\geq 41$
(Ramiandrisoa and Mothe, 2018) | (Kumar et al., 2018b) | En | 2018 |  | Lexical, POS, Negation,
---
Capitalized, Sentiment,
Emotions, Swear words,
Doc2Vec, Others
| RF (features) +
---
LR (Doc2vec)
| W-F1: {Fb: 0.576,
---
Tw: 0.511}
$\geq 11$
(Raiyani et al., 2018) | (Kumar et al., 2018b) |  | En,
---
Hi
2018 |  | One-hot
---
encoding
DNN - 3 layers |  | W-F1: {
---
Fb: {En: 0.5813, Hi: 0.5951},
Tw: {En: 0.6009,Hi: 0.4830}}
$\geq 38$
(Nikhil et al., 2018) | (Kumar et al., 2018b) |  | En,
---
Hi
2018 | - | LSTM + attention |  | W-F1: {
---
Fb: {En: 0.5746, Hi: 0.6032},
Tw: {En: 0.5548,Hi: 0.4703}}
$\geq 29$
(Kumar et al., 2018a) | (Kumar et al., 2018b) | En | 2018 | Deepmoji | LSTM |  | W-F1: {Fb: 0.3572,
---
Tw: 0.1960}
$\geq 29$
(Roy et al., 2018) | (Kumar et al., 2018b) |  | En,
---
Hi
2018 |  | CNN: {En: Glove,
---
Hi: Pretrained on Wikipedia}
SVM: Unigram and TF-IDF
CNN + SVM |  | W-F1: {
---
Fb: {En: 0.5151, Hi: 0.5599},
Tw: {En: 0.5099,Hi: 0.3790}}
$\geq 17$
(Galery et al., 2018) | (Kumar et al., 2018b) | En | 2018 | FastText | Bi-GRU |  | W-F1: { Fb: 0.5315,
---
Tw: 0.4389}
$\geq 17$
(Maitra and Sarkhel, 2018) | (Kumar et al., 2018b) |  | En,
---
Hi
2018 |  | Wordcount, Sentiment,
---
Capitalization, Hashtags
| DNN - 3 layers
---
\+ autoencoder
| W-F1: {
---
Fb: {En: 0.5694, Hi: 0.4189},
Tw: {En: 0.3379,Hi: 0.3132}}
$\geq 6$
(Aroyehun and Gelbukh, 2018a) | (Kumar et al., 2018b) | En | 2018 |  | DA: Translation
---
Pseudo labeling
LSTM, CNN-LSTM |  | W-F1: {
---
Fb: 0.6425 (LSTM),
Tw: 0.5920 (LSTM-CNN)}
$\geq 155$
(Srivastava et al., 2018) | (Kumar et al., 2018b) | En | 2018 | One-hot encoding |  | CapsuleNet with
---
Focal Loss
| W-F1: {Fb: 0.6343,
---
Tw: 0.5941}
$\geq 92$
(Orašan, 2018) | (Kumar et al., 2018b) | En | 2018 |  | Glove, Emoticons TF-IDF,
---
Sentiment
SVM, RF |  | W-F1: {Fb: 0.5830 (RF),
---
Tw: 0.5074 (SVM)}
$\geq 34$
(Aroyehun and Gelbukh, 2018b) | (Kumar et al., 2018b) | En | 2018 | One-hot encoding |  | Majority voting: {CNN,
---
LSTM, and Bi-LSTM}
| W-F1: {Fb: 0.604,
---
Tw: 0.508}
$\geq 43$
(Fortuna et al., 2018) | (Kumar et al., 2018b) | En | 2018 |  | POS, Insults, Punctuation,
---
Sentiment, Capitalization
RF |  | W-F1: {Fb: 0.5288,
---
Tw: 0.3633}
$\geq 33$
(Arroyo-Fernández et al., 2018) | (Kumar et al., 2018b) | En | 2018 | N-gram TF-IDF |  | Passive-aggressive
---
algorithm and SVM
| W-F1: {Fb: 0.6315,
---
Tw: 0.3633}
$\geq 31$
(Risch and Krestel, 2018) | (Kumar et al., 2018b) |  | En,
---
Hi
2018 |  | English: FastText
---
Hindi: Wikipedia embeddings
Word-ngram, Char-ngram,
Hancrafted features
| Ensemble of GRU
---
with diffrent features
| W-F1: {
---
Fb: {En: 0.6011, Hi: 0.6311},
Tw: {En: 0.5995,Hi: 0.3835}}
$\geq 88$
(Orabi et al., 2018) | (Kumar et al., 2018b) | En | 2018 | Glove | MultiCNNPooling |  | W-F1: {Fb: 0.5974,
---
Tw: 0.5690}
$\geq 7$
(Tommasel et al., 2018) | (Kumar et al., 2018b) | En | 2018 |  | LSTM: Glove, LSTM: Sentiment,
---
NN: Composed features (SVM:
Tf-IDF, Sentiment, Punctuation)
| Avg (SVM +
---
NN (2LSTM+NN))
| W-F1: {Fb: 0.5948,
---
Tw: 0.5480}
$\geq 25$
(Golem et al., 2018) | (Kumar et al., 2018b) | En | 2018 |  | POS, NER, Bad, Capitalization,
---
Numerical tokens, Text length,
Sentiment
BiLSTM, CNN: GloVe
| (BiLSTM+CNN+LR)
---
\+ SVM
| W-F1: { Fb: 0.642,
---
Tw: 0.601}
$\geq 17$
(Modha et al., 2018) | (Kumar et al., 2018b) |  | En,
---
Hi
2018 | FastText | CNN, LSTM |  | W-F1: {
---
Fb: {En: 0.6178 (LSTM),
Hi: 0.6081},
Tw: {En: 0.5520,Hi: 0.4992}}
$\geq 54$
(Singh et al., 2018) | (Kumar et al., 2018b) | En | 2018 |  | Total, Avg words, Punctuation,
---
Abusive, URLs, Hashtags, Single
letters, Uppercase tokens, Ph. no.
CNN | F1: 0.58 | $\geq 43$
(Patwa et al., 2021) | (Kumar et al., 2018b) | En | 2018 | Glove | CNN-CapsNet | W-F1: 0.6520 | $\geq 4$
(Aroyehun and Gelbukh, 2018a) | (Kumar et al., 2018b) | En-Hi | 2018 | FastText |  | Fb: LSTM
---
Overall: CNN-LSTM
| W-F1: {Fb: 0.6425 ,
---
Overall: 0.5920}
$\geq 155$
(Gómez-Adorno et al., 2018) | (Aragón et al., 2020) | Es | 2018 |  | N-grams (Char, words, POS),
---
Aggressive tokens
SVM | F1: 0.4285 | $\geq 21$
(Si et al., 2019) | (Kumar et al., 2018b) | En-Hi | 2019 |  | POS, Emojis, Aggressive
---
tokens, Sentiment
| Majority voting:
---
(XGBoost, GBM, SVM)
F1: 0.6813 | $\geq 21$
(Modha et al., 2022) | (Kumar et al., 2018b) | En | 2019 | FastText | CNN |  | W-F1: {
---
Fb: {En: 0.5638, Hi: 0.6081},
Tw: {En: 0.5377,Hi: 0.4992}}
$\geq 16$
(Tommasel et al., 2019) | (Kumar et al., 2018b; Reynolds et al., 2011) | En | 2019 |  | TF-ID, Char tokens, Lemma,
---
NER, POS, Sentiment, Word2Vec,
GloVe, (Barbieri and Saggion, 2014),
(Farías et al., 2016)
SVM-RBF |  | F1: {
---
(Kumar et al., 2018b)\- Facebook: 0.6879,
(Kumar et al., 2018b)\- Twitter: 0.5529,
(Reynolds et al., 2011): 0.8416}
$\geq 6$
(Srivastava and Khurana, 2019) |  |  (Kumar et al., 2018b) and
---
Toxic
En | 2019 | FastText |  | BiLSTM followed
---
Capsule Network
| F1: {Fb: 0.6353,
---
Tw: 0.5795}
$\geq 12$
(Gutiérrez-Esparza et al., 2019) | (Gutiérrez-Esparza et al., 2019) | Es | 2019 | VIMs | OneR | Acc.: 0.90 | $\geq 37$
(Kumari et al., 2019) | (Kumari et al., 2019) | Img | 2019 | - | CNN |  | P: 90, R, F1: 89
---
$\geq 23$
(Sane et al., [n. d.]) | (Sane et al., [n. d.]) | En-Ta | 2019 |  | Char, word n-grams,
---
Lexicon, Punctuations,
Uppercase, Hashtags,
Mentions
SVM-RBF | F1: 0.67 | 1
(Ramiandrisoa and Mothe, 2020a) | (Kumar et al., 2018b) | En | 2020 | Emoticon, Hashtags | BERT-base |  | W-F1: {Fb: 0.627, Tw: 0.595}
---
$\geq 5$
(Kumar et al., 2020) | (Bhattacharya et al., 2020) | En | 2020 | Tokenizer |  | BERT-large +
---
Project attention layers
| W-F1: {AG: 0.6367,
---
GEN: 0.8202}
$\geq 114$
(Liu et al., 2020) | (Bhattacharya et al., 2020) | En | 2020 | - |  | Majority voting - 3
---
uncased-BERT-base
| W-F1: {AG: 0.664,
---
GEN: 0.851}
$\geq 2$
(Pascucci et al., 2020) | (Bhattacharya et al., 2020) | En | 2020 | Stylistic, Linguistic rules | SMO |  | W-F1: {AG: 0.6291,
---
GEN: 0.6733}
$\geq 5$
(Mut Altın et al., 2020) | (Bhattacharya et al., 2020) | En | 2020 |  Word embedding (Barbieri et al., 2016) | BiLSTM with attention |  | W-F1: {AG: 0.7246,
---
GEN: 0.8199}
$\geq 7$
(Tawalbeh et al., 2020) | (Bhattacharya et al., 2020) | En | 2020 |  USE (Cer et al., 2018) | XGBoost |  | W-F1: {AG: 0.6075,
---
GEN: 0.8567}
$\geq 5$
(Risch and Krestel, 2020) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 | - |  | BERT, m-BERT (15-25)
---
Ensamble
| W-F1: {
---
En: {AG: 0.8029, GEN: 0.8514},
Hi: {AG: 0.8128, GEN: 0.8781},
Bn: {AG: 0.8219, GEN: 0.9385}}
$\geq 62$
(Baruah et al., 2020) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 |  | TF-IDF of word (1-3),
---
char (1-6) -ngram
| SVM (TF-IDF), BERT,
---
XLMRoBERTa
| W-F1: {En: {AG: 0.73 (BERT),
---
GEN: 0.87 (SVM)}
Hi: {AG: 0.79 (SVM),
GEN: 0.87 (XLMRoBERTa)},
Bn: {AG: 0.81, GEN: 0.93} (SVM)}
$\geq 25$
(Datta et al., 2020) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 | TF-IDF, Sentiment |  | XGBoost, Gradient
---
Boosting
| W-F1: {En: 0.43 (XGBoost),
---
Hi: 0.59 (GBM),
Bn: 0.44 (GBM)}
$\geq 7$
(Gordeev and Lykova, 2020) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 | - | m-BERT-base |  | W-F1: {
---
En: {AG: 0.7568, GEN: 0.8716},
Hi: {AG: 0.7761, GEN: 0.7683},
Bn: {AG: 0.7761, GEN: 0.9297}}
$\geq 15$
(Koufakou et al., 2020) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 | FastText, HurtLex | LSTM |  | W-F1: {
---
En: {AG: 0.677, GEN: 0.838},
Hi: {AG: 0.726, GEN: 0.771},
Bn: {AG: 0.746, GEN: 0.869}}
$\geq 12$
(Kumari and Singh, 2020a) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 |  | FastText,
---
One-hot embedding
LSTM, CNN |  | W-F1: {
---
FastText-CNN: {
En: {AG: 0.6602, GEN: 0.8227}},
FastText-LSTM: {
Hi: {AG: 0.6547, GEN: 0.7363},
Bn: {AG: 0.7175, GEN: 0.8793}}
$\geq 25$
(Mishra et al., 2020) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 | - |  | uncased-m-bert-base,
---
uncased-BERT-base
| W-F1: {BERT: {
---
En: {AG: 0.759, GEN: 0.857},
m-BERT: {
Hi: {AG: 0.779, GEN: 0.849},
Bn: {AG: 0.780, GEN: 0.927}}}}
$\geq 26$
(Samghabadi et al., 2020) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2020 | - |  | cased-m-bert-base,
---
uncased-BERT-base,
along with attention
| W-F1: {BERT: {
---
En: {AG: 0.7143, GEN: 0.8579}},
m-BERT: {
Hi: {AG: 0.7183, GEN: 0.8008},
Bn: {AG: 0.7369, GEN: 0.9206}}}
$\geq 96$
(Khandelwal and Kumar, 2020) | (Kumar et al., 2018b; Sadiq et al., 2021) |  | En,
---
En-Hi
2020 | Glove and FastText |  | Avg (Deep Pyramid
---
CNN + Pooled LSTM
\+ Disconnected RNN)
| W-F1: {
---
(Kumar et al., 2018b)-Facebook: 0.6770,
(Kumar et al., 2018b)-Twitter: 0.6480 ,
(Sadiq et al., 2021): 0.9023}
$\geq 7$
(Chen et al., 2018) | (Chen et al., 2017) | En | 2020 | TF-IDF | CNN |  | Acc.: 0.92,
---
Micro-AUC: 0.98
Macro-AUC: 0.97
$\geq 87$
(Tanase et al., 2020) | (Aragón et al., 2020) | Es | 2020 | N-grams, TF-IDF | XLM-RoBERTa | F1: 0.7969 | $\geq 19$
(Guzman-Silverio et al., 2020) | (Aragón et al., 2020) | Es | 2020 | - | W-ensamble: 20 BERT | F1: 0.80 | $\geq 24$
(Shrivastava et al., 2021) | (Kumar et al., 2018b) | En-Hi | 2021 |  | TF-IDF followed
---
Data Balancing with
GPT-2
| Voting-Ensemble:
---
{SVM, NB, LR,
CNN, LSTM}
F1: 0.65 | $\geq 9$
(Mundra and Mittal, 2021) | (Kumar et al., 2018b) | En-Hi | 2021 |  | Char-One-hot embedding,
---
Aggressive tokens
CNN |  | P: 0.60, R: 0.59,
---
F1: 0.59, AUC: 0.59
$\geq 2$
(Zhao et al., 2021) | (Kumar et al., 2018b) | Fb: En | 2021 | - | XLM-e100 | micro-F1: 0.6196 | $\geq 24$
(Kumari et al., 2021) | (Kumar et al., 2018b) | En | 2021 |  | Randomly initialised
---
embedding
LSTM-autoencoders |  | W-F1:{ Fb: {En: 0.81, Hi: 0.72},
---
Tw: {En: 0.71, Hi: 0.58}}
$\geq 20$
(Dutta et al., 2021) | (Bhattacharya et al., 2020) | En | 2021 | - |  | Data augmentation:
---
BERT, TF-IDF-XgBoost
| W-F1: {AG: 0.735,
---
GEN: 0.852}
$\geq 2$
(Sadiq et al., 2021) | (Sadiq et al., 2021) | En | 2021 | TF-IDF (Uni+Bi-gram) | DNN - 3 layers |  | Acc: 0.92, P, R, F1: 0.90
---
$\geq 92$
(Shulginov et al., 2021) | (Shulginov et al., 2021) | Ru | 2021 | - | RuBERT | F1: 0.66 | $\geq 4$
(Sharif and Hoque, 2022) | (Sharif and Hoque, 2022) | Bn | 2021 | Word2Vec | CNN-BiLSTM | W-F1: 0.87 | $\geq 14$
(Herodotou et al., 2021) | (Founta et al., 2018) | En | 2021 |  | User-Profile, User-post (text),
---
Network Features
| Streeming-ML
---
(HT, ARF, SLR)
Batch-ML (DT)
| F1: {Streeming: 0.88,
---
Batch: 0.91}
$\geq 6$
(Akter et al., 2022) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2022 | DA: Machine Translation |  | BERT: En,
---
m-BERT: Hi, GPT2: Bn
| F1: {En: .77,
---
Hi:0.68, Bn: 0.70}
$\geq 6$
(Sengupta et al., 2022) | (Kumar et al., 2018b) | En-Hi | 2022 | - | Char-HAL | W-F1: 0.909 | $\geq 20$
(Ramiandrisoa, 2022) | (Kumar et al., 2018b; Bhattacharya et al., 2020) | En | 2022 | - | MTL-RoBERTa |  |  W-F1: { (Kumar et al., 2018b): 0.64, (Bhattacharya et al., 2020): 0.73}
---
$\geq 1$
(Sharif and Hoque, 2022) | (Sharif and Hoque, 2022) | Bn | 2022 | - |  | W-ensemble: {m-BERT,
---
distil-BERT, Bangla-BERT,
XLM-R}
| W-F1: {AG: 0.9343,
---
GEN: 0.9311}
$\geq 28$
(Khan et al., 2022) | (Sadiq et al., 2021) | En | 2022 | 7-Emotions + Word2Vec | DNN | F1: 0.97 | $\geq 12$
(Kumari and Singh, 2022) | (Kumari and Singh, 2022) | En | 2022 | Glove, BFFO |  | Text: CNN +
---
Img: VGG-16 +RF
| W-P: 0.80, R, F1: 0.79
---
$\geq 7$
(Tonja et al., 2022) | (Arellano et al., 2022) | Es | 2022 | - | DistilBETO |  | F1: {AG: 0.7455,
---
AG-Cat: 0.4903}
$\geq 8$
(Ta et al., 2022b) | (Arellano et al., 2022) | Es | 2022 | DA: Machine Translation | m-BERT-base |  | P: 0.7552, R: 0.7409,
---
F1: 0.7480
$\geq 2$
(Ta et al., 2022a) | (Arellano et al., 2022) | Es | 2022 | DA: Machine Translation |  GAN-BERT (Croce et al., 2020) |  | P: 0.7408, R: 0.7479,
---
F1: 0.7443
$\geq 4$
(Huerta-Velasco and Calvo, 2022) | (Aragón et al., 2020) | Es | 2022 |  | Structural, Affective,
---
Dimensional, Emotional,
Word2Vec
SVM | macro-F1: 0.7921 | $\geq 1$
(AGBAJE and Afolabi, 2022) | (Go et al., 2009) | En | 2022 | Sentiments |  | Text: LSTM
---
Img: CNN
| F1: {Tw: 0.91,
---
Img: 0.89}
$\geq 1$
(Gattulli et al., 2022) | (Gattulli et al., 2022) | It | 2022 | Syntactic, Sentiment | RF | Avg-Acc: 0.9125 | $\geq 2$
(Iqbal and Keshtkar, 2019) | (Kumar et al., 2018b) | En | 2023 |  | N-grams, LIWC,
---
POS and Sentiments
RF | F1: 0.67 | $\geq 2$
(Mane et al., 2023) | (Mane et al., 2023) | En | 2023 | FastText | BiLSTM |  | Acc: 0.81, P:0.77,
---
R:0.83, W-F1:0.81
$\geq 0$
(Ghosh et al., 2023) | (Bhattacharya et al., 2020) |  | En, Hi,
---
Bn
2023 | - |  | XLM-RoBERTa (XLMR),
---
Multi-head self-attention
| W-F1: {
---
En: {AG: 0.6453, GEN: 0.8612},
Hi: {AG: 0.7776, GEN: 0.8657},
Bn: {AG: 0.7835, GEN: 0.9053}}
$\geq 0$
(Rawat et al., 2023) | (Kumar et al., 2018b; Rawat et al., 2023) | En | 2023 | - |  | Political: RoBERTa-base
---
(Kumar et al., 2018b): Hing-BERT
| Macro-F1: { Political: 0.66,
---
Political+(Kumar et al., 2018b): 0.92}
$\geq 0$

##  5\. Behavioural Analysis of User Aggressiveness

Behavioral analysis of users’ aggressiveness on social media is essential for
a comprehensive understanding, and it requires qualitative and quantitative
approach studies. Qualitative research delves into the emotional and
psychological nuances, uncovering the ‘why’ behind aggressive behavior. This
provides valuable insights into underlying motivations and contextual factors.
In contrast, quantitative studies offer a broader perspective by quantifying
the prevalence and patterns of online aggression. This will helps to establish
empirical evidence, monitor trends, and inform policies and strategies. The
combination of both approaches is crucial for addressing and mitigating
aggressive behavior effectively in the digital realm, with qualitative
research exploring the ‘why’ and quantitative research revealing the ‘what’
and ‘how’ of such behavior. We discuss these approaches in detail in the
following sections. Figure 14 presents distributions of behavioral studies
conducted by different countries. This analysis offers insights into the
global distribution and contribution of research in behavioral studies.

Figure 14. Illustrates the distribution of studies on the behavioral analysis
of aggressive users conducted by various countries. The United States
conducted the most studies, followed by Greece and Russia. This analysis
highlights the limited research conducted in other regions.

###  5.1. Qualitative Studies

Qualitative studies are essential for in-depth exploration and understanding
of the underlying motives, contexts, and nuances of aggressive behavior in
online social media. Qualitative studies rely on data collected from surveys
and interviews involving diverse participants. As discussed in the following
subsections, this study reveals underlying human and cultural factors.

####  5.1.1. Influencing factors

Various studies have been conducted to examine the factors that influence
aggressive behavior on social media. Notable factors such as counter-comments
(Eraslan and Kukuoglu, 2019), specific topics or events (Pascual-Ferrá et al.,
2021), peer or friend influence (Sobkin and Fedotova, 2021; Henneberger et
al., 2017), anonymity (Zimmerman and Ybarra, 2016), religious influence (Boadi
and Kolog, 2021), the impact of fake media (Galyashina et al., [n. d.]), moral
emotions (Adinugroho et al., 2022), social media usage patterns (Wong et al.,
2022; Borraccino et al., 2022; Ferguson, 2021; Adinugroho et al., 2022),
gender stereotype traits (Wright, 2020), and person and situational factors
(Allen et al., 2018; Weingartner and Stahel, 2019). Study by (Eraslan and
Kukuoglu, 2019) revealed that counter-comments about a user’s values
substantially affect the user’s exhibition of aggressive tendencies.
Surprisingly, most users were found to be unaware of their rights in the
presence of social media aggression. Religion can also have an impact on
online aggression behaviors, according to (Boadi and Kolog, 2021). The
researchers used Contemporary Deterrence theory to examine how religion
influences an individual’s daily life and captures behavioral outcomes or its
direct influences on online aggression behaviors. Additionally, (Borraccino et
al., 2022) investigated the relationship between cyber aggression and
problematic social media use (PSMU), as well as the role of social support in
this relationship. They found that cyber aggression was more prevalent among
girls, and PSMU was more prevalent among 11-year-old boys and 13-year-old
girls. Social support was found to be highly protective against cyber
aggression and PSMU in all age groups and genders. In (Wright, 2020), it has
been shown that boys and girls with more feminine characteristics are more
likely to engage in cyber-aggression. In contrast, those endorsing more
masculine traits reported cyber relational aggression and cyber verbal
aggression more frequently through online gaming. (Sobkin and Fedotova, 2021)
examined the behavior of high school students on social media, considering
their role as aggressor or victim, nature of aggression (public or private),
characteristics of aggressors, and recommended actions for victims. They
revealed that those who have experienced real-life bullying are more likely to
be involved in aggressive situations on both sides of social media.

A study conducted by (Ferguson, 2021) delved into the connection between
social media and negative social outcomes, such as political polarization,
depression, suicide, aggression, and cyberbullying. Findings suggests that
social media may contribute to adverse outcomes, but the relationship is
complex and dependent on individual motivation and situational factors.
Similarly, (Sengupta et al., 2022) studied the relationship among five
offensive traits namely, aggression, hate, sarcasm, humor, and stance in
Hinglish social media code-mixed texts. They proposed a causal importance
score to quantify the effect of different abusive keywords and the overall
context on the offensiveness of the texts. In (Chaturvedi and Sharma, 2022),
the study qualitatively analyzed transitive and lattice graphs representing
the co-occurrence of aggressive (hate) terms in contextual data related to
Women and Regions. (Pascual-Ferrá et al., 2021) investigated how aggressive
language, particularly related to the wearing of face masks during the
COVID-19 pandemic, was used in online discourse. The study found that tweets
with anti-mask hashtags used more aggressive language, while tweets with pro-
mask hashtags displayed a relatively less toxic or aggressive tone, except
#WearADamnMask. Moreover, (Galyashina et al., [n. d.]) examined the potential
aggressive impact of fake media on social media users, raising questions about
the safety of social network communication. Study (Adinugroho et al., 2022)
identified two predictors of cyber aggression: moral emotions and the
frequency of social media usage. Their study revealed that guilt inhibits
cyber-aggression, while shame encourages it. (Weingartner and Stahel, 2019)
developed a comprehensive theoretical framework considering various factors
that contribute to cyber-aggressive behavior. This model includes socio-
structural determinants, individual determinants, and situational
determinants. (Runions and Bak, 2015) disclosed that cyber-aggressive behavior
can be influenced by various factors such as the influence of aggressive
peers, the influence of the initial aggressor on their network, and the
cascading diffusion trend from the source of the behavior to its connections.
Furthermore, (Zimmerman and Ybarra, 2016) found that anonymous users were more
aggressive than non-anonymous users, highlighting the impact of anonymity and
positive or negative social modeling. (Zeng et al., 2023) explored the effect
of an interpretation bias modification program on cyber-aggression among
Chinese middle school students, specifically focusing on modifying hostile
interpretation bias. Results indicated that the program effectively reduced
reactive cyber-aggression, but this effect was more pronounced in females than
males. The factors contributing to aggressive online commenting behavior are
complex and multifaceted, influenced by peer dynamics, social modeling,
anonymity, and subcultural norms.

####  5.1.2. Consequences

The consequences of aggressive behavior on social media platforms are of
paramount concern due to its potential to disrupt public stability and, in
some cases, escalate into real-world violence and aggression (Quang-Loc,
2021). A study (Torregrosa et al., 2022) demonstrated that the tone of
electoral campaign discourse may not be as negative as perceived by citizens.
The study confirms that the most ideologically extreme parties tend to have a
more aggressive language than the normal or moderate ones. This research used
both qualitative and quantitative analyses, including sentiment analysis of
Spanish Twitter data, to reach this conclusion. The experience of online
aggression among university students was investigated by (Mishna et al.,
2018), and it was found that a minority of students experienced online
aggression. The issue of aggression on social media platforms is especially
concerning for teenagers, particularly in the form of cyberbullying. The study
conducted by (Wong et al., 2022) found a relationship between the motivation
to seek rewards through aggressive behavior and the emergence of
cyberbullying, which can lead to social media addiction. Their findings
indicated that inclusive norms may reduce cyberbullying but increase social
media addiction. Cyberbullying has been found to be indirectly associated with
social media addiction to increased aggression, an indirect effect that is
intensified with increasingly inclusive anti-cyberbullying regulations.
Furthermore, (Vladimirou et al., 2021) explored how complaining is expressed
on social media and how it can become more aggressive due to complex
participation and multi-modality. A case study centered on an online protest
against a waste management policy in London’s Borough of Merton exemplified
this phenomenon. Initially, users expressed their dissatisfaction with the
policy on social media. However, as the protest continued, the tone of the
posts became increasingly aggressive, with participants resorting to
profanity, insults, and threats. This aggressive behavior on social media
eventually spilled over into the real world, disrupting public stability. This
case underscores the significant consequences of aggressive behavior on social
media, highlighting the importance of being mindful of the impact of online
interactions. These broad and diverse studies provide a rich and complex
research landscape on aggressive behavior analysis, providing critical
insights into the complex world of online aggression on social media
platforms.

###  5.2. Quantitative Studies

Quantitative studies have gained importance due to the gradual growth of
social media data with aggressive behaviour. Quantitative studies offer
statistical insights into the prevalence and trends of online aggressive
behavior. They enable data-driven strategies for addressing and mitigating
such behavior effectively in a large-scale dataset. Researchers are exploring
more abstract findings, such as behavior patterns, diffusion of behavior, and
identification of influential aggressive users, to reduce such behavior on a
larger scale. To conduct these studies, comprehensive frameworks, models, and
network and user-based approaches are used, as explained in the following sub-
sections.

####  5.2.1. Comprehensive Approaches

Researchers have developed frameworks and models to understand and prevent
user aggressive behavior on social media. One of these frameworks is the
General Aggression Model (GAM), developed by (Anderson and Bushman, 2002). The
GAM is a comprehensive framework that explores the person and situational
factors that influence cognitions, feelings, and arousal, leading to
aggressive behavior. (Allen et al., 2018) applied the GAM to understand
aggressive behavior on social media and found that factors such as anonymity,
disinhibition, and social comparison contribute to aggressive behavior. In
addition to the GAM, (Mathew et al., 2019) utilized the DeGroot model to
identify aggressive and non-aggressive users on social media. The DeGroot
model is a mathematical framework that analyzes the spread of information
within a network. (Mathew et al., 2019) found that the posts of aggressive
users tend to spread faster and wider, indicating the potential for hate
speech to have a significant impact on social media. Other frameworks and
models have also been developed to understand aggressive behavior on social
media. For example, the Social Information Processing Theory (SIPT) (Walther,
2008) suggests that aggressive behavior is a result of a lack of social cues
on social media, leading to misinterpretation and misunderstanding. The
Frustration-Aggression Theory suggests that frustration leads to aggression,
and this can be exacerbated by social media platforms that provide a platform
for anonymous venting. (Chaturvedi and Sharma, 2022) identified severe
aggressive terms quantitatively by applying a proposed threshold of an
offensive score derived from inter-agreement measures. This method effectively
contributes to the detection of aggressive language by identifying terms with
heightened offensive impact.

####  5.2.2. Network and User based Approaches

The rise of social media has also led to an increase in aggressive behavior on
these platforms, which can quickly spread through a network of users. To study
the propagation of aggression on social media, researchers have proposed
various methods that use opinion dynamics and network-based approaches. One
such study was conducted by (Terizi et al., 2021), which proposed a method
that uses opinion dynamics based on network dynamics to study the propagation
of aggression on social media. They analyzed how users’ connections to other
aggressive or non-aggressive users affected the spread of aggression on the
platform. Similarly, (Gallo et al., 2020) used a network knowledge-based model
to identify online users’ reactions on Twitter using a machine learning
approach. It explored user beliefs after exposure to their news feed using a
Network Knowledge Base (NKB) and identified several social cues as features to
predict whether a user would react in a given time period. Another study by
(Chatzakou et al., 2017) proposed a methodology to detect cyberbullying and
aggressive behavior users based on text, user, and network-based features.
They analyzed user behavior and found that aggressors are relatively popular
and tend to include more negative posts. (Ali et al., 2023) conducted a study
to identify influential users in the aggressive community using the Girvan
Newman algorithm. In their method, they utilized an LSTM-GRU model to
effectively detect aggressive tweets. (Karan and Kundu, 2023) identified
patterns related to various attributes, including followers, friends, number
of posts, and the percentage of aggressive posts among bully and target users
using LSTM. Similarly, (Basak et al., 2019) focused on detecting, analyzing,
and mitigating online public shaming on Twitter. Their study identified the
characteristics of online shaming, such as the use of negative words and
retweeting, and proposed a machine learning algorithm to detect it. (Chatzakou
et al., 2017) analyzed the behavioral patterns exhibited by aggressive users
on Twitter and their differences with other users. They utilized additional
network features for effective aggressive language detection and analyzed
Twitter’s mechanism for suspending users. By understanding the differences
between aggressive users and other user categories, they were able to propose
a machine learning approach for detecting aggressive behavior. Their analysis
of aggressive behavior in a network perspective provided valuable insights for
improving their approach.

Researchers have investigated different methods to identify and mitigate the
diffusion of such content. A study (Balci and Salah, 2015) analyzed verbal
aggression and abusive behaviors of users on online social games. They
analyzed user behavior and characteristics of offending users to identify
patterns and potential solutions. Further, (Aragón et al., 2019) provided an
overview of MEX-A3T, a shared task for identifying user profile and aggressive
tweets in Mexican Spanish. The researchers developed the user profile and
aggression detection dataset with baseline results from (Ortega-Mendoza and
López-Monroy, 2018) and (Graff et al., 2018), respectively. This task aimed to
improve the identification of aggressive content and help mitigate its spread.
(Mane et al., 2023) conducted a user aggressive behavior analysis on Twitter,
employing the proposed aggression Twitter dataset and content detection
algorithm. They introduced a novel metric, “user aggression intensity,” to
measure overall aggressive activity. Utilizing this metric, users are profiled
as aggressive or non-aggressive. Their findings indicate that the content in
social media feeds can influence aggressive behavior, especially with event-
specific feeds leading to increased aggression. Additionally, the study found
users supporting and encouraging aggressive content on social media,
contributing to the spread of such behavior. (Poiitis et al., 2021) focused on
developing a model for predicting the diffusion of aggressive tweets on
Twitter. Their proposed solution aimed to minimize the spread of such content
by identifying influential users and selectively blocking or limiting their
actions. Together, these studies highlight the importance of identifying and
mitigating aggressive content diffusion. By understanding user behavior,
characteristics, and influential users.

##  6\. Future Scope

Despite the considerable research that has been conducted on aggression
content detection and behavioral analysis of user aggression on social media,
there remain several challenges. Following, we discuss potential areas for
further research:

  * •

Limited Annotated Data for Robust Algorithms: A prominent challenge is the
lack of effective, robust, and generalized aggression detection algorithms,
mainly due to the limited size of human-annotated datasets. While some
datasets exist, they are often limited in size and may not cover the full
spectrum of aggression types and contexts, as shown in Table 1. Moreover, the
specific characteristics of various social media platforms, such as writing
style, post length, etc., require a specific social media generated dataset.

  * •

Multi-modal, Multilingual and Multicultural Contexts: Existing research in
aggression detection predominantly focuses on English text social media
platforms (see Table LABEL:tab:detection_table). Future research should
prioritize the analysis of aggression across various languages along with
other modalities and cultural contexts to ensure the effectiveness and
generalizability of detection models. Therefore, multilingual aggression
datasets are needed, especially considering the prevalence of code-mixed
languages on social media platforms.

  * •

Contextual Understanding: Aggression detection algorithms often face the
challenge of extracting the underlying context in user interactions. While
current approaches mainly rely on text analysis, integrating contextual
information such as user relationships, previous interactions, and post
history can increase the accuracy of aggression detection.

  * •

Detection of Covert Aggression: Current research addresses overt and covert
forms of aggression on social media. However, covert aggression, including
passive-aggressive behavior, sarcasm, and microaggressions, remains
challenging to detect effectively, as indicated in Table
LABEL:tab:detection_table. Enhancing algorithms to identify these covert forms
of aggression represents a critical challenge, given their prevalence.

  * •

Multi-modal and Multilingual Aggression Detection: While ensemble algorithms
have demonstrated effectiveness for aggression content detection, they have
predominantly focused on English text modality. However, some recent work has
explored multi-modal approaches, incorporating images, network properties, and
user traits alongside English textual data (Kumari and Singh, 2022, 2021;
Herodotou et al., 2021). There is a noticeable gap in research dedicated to
multi-modal and multilingual aggression detection. Multi-modal approaches are
anticipated to achieve enhanced accuracy by considering diverse perspectives
beyond text.

  * •

Explainable Algorithms: This domain lacks explainable aggression content
detection algorithms that offer interpretability for understanding decision-
making processes and identifying potential biases. The development of more
transparent and fair models is essential in this regard.

  * •

Leveraging Large-Scale Language Models (LLMs): While large-scale language
models (LLMs) have achieved remarkable success in various domains, their
potential for addressing aggression identification problems has not been
widely explored. Previous studies have effectively utilized LLMs in other
domains (Sun et al., 2023; Hartvigsen et al., 2022; Li et al., 2023).

  * •

Cultural and Behavioral Variations: Most of the research has been done in
specific countries on the behavioral analysis of aggressive users on social
media. Further cross-cultural research is needed to understand how cultural
differences or diversity, including religious beliefs, personality traits,
gender, age, and cultural background, influence aggressive behavior on social
media.

  * •

Longitudinal Studies: Prior quantitative studies focusing on the behavioral
analysis of aggressive users on social media are often cross-sectional,
capturing data at a single point in time. Longitudinal studies analyzing users
over extended periods of time are needed to gain a deeper understanding of how
aggressive behavior exhibit in response to different events and over time.

  * •

Comprehensive Quantitative Studies: There is a lack of quantitative studies
that comprehensively analyze the nature, propagation, influencing power,
social impact, and potential causes of user aggressive behavior on social
media platforms. Given the vast amount of data generated daily on these
platforms, this presents a challenging yet crucial area of research.

  * •

Reconciliation of Qualitative and Quantitative Studies: Qualitative studies in
sociology have provided insights into the factors influencing aggression.
These factors can be validated through quantitative research and leveraged in
quantitative studies for analyzing behavior propagation and can predict future
aggressive behavior based on network dynamics and qualitative findings.

##  7\. Conclusion

In this systematic literature review, we have effectively bridged the gap
between two disparate yet inherently interconnected studies, namely Aggression
Content Detection and Behavioural Analysis of Aggressive Users. In this
literature, we comprehensively analyzed various cyber-aggression definitions
and proposed a unified cyber-aggression definition. We thoroughly examined the
components of Aggression Content Detection, including datasets, features, and
detection algorithms. Notably, for dataset annotation, researchers have
consistently included a diverse range of aggression types, from explicit to
implicit. The assessment of inter-annotator agreement has predominantly
employed metrics such as Krippendorff’s alpha and Fleiss’ kappa. Despite the
progress made, our review underscores the scarcity of multi-modal,
multilingual, and social media-specific datasets. Our analysis has revealed
that FastText and Glove, non-contextual statistical features are effective
token representation methods. The incorporation of multi-modal features has
shown promise in enhancing aggression detection. An important observation is
the dominance of research in English-language aggression detection, leaving a
notable gap in the study of low-resource language aggression detection. In ML
algorithms, SVM proves effective, while DL algorithms, such as CNN and LSTM,
demonstrate effectiveness. Transformer-based DL algorithms, particularly
various BERT variants, perform better than traditional ML and DL algorithms.
Overall, ensemble-based approaches, including transformer algorithms,
consistently achieved higher F1 scores in aggression detection. In this paper,
we systematically present meta-information of aggression detection studies in
Table LABEL:tab:detection_table. In our exploration of aggressive user
behavior analysis, we extensively investigate both qualitative and
quantitative studies. This literature review reveals significant sociological
insights that can enhance computational techniques, potentially improving the
effectiveness of cyber-aggression prevention methods. Interestingly, most
qualitative and quantitative studies have been conducted in North America and
Europe, revealing a significant gap in research conducted in other regions.
Finally, we have identified challenges and provided future research
directions, highlighting a shift toward socio-computational approaches. This
work encourages researchers to adopt a socio-computational perspective in
addressing the widespread issue of cyber-aggressive behavior.

###### Acknowledgements.

This research was supported by the Prime Minister Research Fellowship funded
by the Ministry of Education (MOE), India. Rajesh is supported by EU H2020
program under the SoBigData++ project (grant agreement No. 871042), by the
CHIST-ERA grant No. CHIST-ERA-19-XAI-010 (ETAg grant No. SLTAT21096), and
partially funded by CHIST-ERA project HAMISON.

## References

  * (1)
  * Adinugroho et al. (2022) Indro Adinugroho, Priska Kristiani, and Nani Nurrachman. 2022\.  Understanding Aggression in Digital Environment: Relationship between Shame and Guilt and Cyber Aggression in Online Social Network.  _Makara Human Behavior Studies in Asia_ 26, 2 (2022), 105–113.
  * AGBAJE and Afolabi (2022) MICHAEL AGBAJE and Oreoluwa Afolabi. 2022.  Neural Network-Based Cyber-Bullying and Cyber-Aggression Detection Using Twitter Text.  _Research Square_ 1 (2022).  https://doi.org/10.21203/rs.3.rs-1878604/v1
  * Aizawa (2003) Akiko Aizawa. 2003\.  An information-theoretic perspective of tf–idf measures.  _Information Processing & Management_ 39, 1 (2003), 45–65.
  * Akter et al. (2022) M. Akter, H. Shahriar, N. Ahmed, and A. Cuzzocrea. 2022\.  Deep Learning Approach for Classifying the Aggressive Comments on Social Media: Machine Translated Data Vs Real Life Data. In _2022 IEEE International Conference on Big Data (Big Data)_. IEEE Computer Society, Los Alamitos, CA, USA, 5646–5655.  https://doi.org/10.1109/BigData55660.2022.10020249
  * Al-Alosi (2017) Hadeel Al-Alosi. 2017\.  Cyber-violence: digital abuse in the context of domestic violence.  _TheUNIVERSITY OF NEW SOUTH WALES LAW JOURNAL_ 40, 4 (2017), 1573–1603.
  * Ali et al. (2023) Mohsan Ali, Mehdi Hassan, Kashif Kifayat, Jin Young Kim, Saqib Hakak, and Muhammad Khurram Khan. 2023.  Social media content classification and community detection using deep learning and graph analytics.  _Technological Forecasting and Social Change_ 188 (2023), 122252.
  * Allen et al. (2018) Johnie J Allen, Craig A Anderson, and Brad J Bushman. 2018\.  The general aggression model.  _Current opinion in psychology_ 19 (2018), 75–80.