# The Evolution of Language in Social Media Comments

Niccolò Di Marco  Department of Computer Science, Sapienza University of Rome
Edoardo Loru  Department of Computer, Control and Management Engineering,
Sapienza University of Rome  Anita Bonetti  Department of Communication and
Social Research (CoRiS), Sapienza University of Rome  Alessandra Olga Grazia
Serra  Tuscia University  Matteo Cinelli  Department of Computer Science,
Sapienza University of Rome  Walter Quattrociocchi  Department of Computer
Science, Sapienza University of Rome

###### Abstract

Understanding the impact of digital platforms on user behavior presents
foundational challenges, including issues related to polarization,
misinformation dynamics, and variation in news consumption. Comparative
analyses across platforms and over different years can provide critical
insights into these phenomena. This study investigates the linguistic
characteristics of user comments over 34 years, focusing on their complexity
and temporal shifts. Utilizing a dataset of approximately 300 million English
comments from eight diverse platforms and topics, we examine the vocabulary
size and linguistic richness of user communications and their evolution over
time. Our findings reveal consistent patterns of complexity across social
media platforms and topics, characterized by a nearly universal reduction in
text length, diminished lexical richness, but decreased repetitiveness.
Despite these trends, users consistently introduce new words into their
comments at a nearly constant rate. This analysis underscores that platforms
only partially influence the complexity of user comments. Instead, it reflects
a broader, universal pattern of human behaviour, suggesting intrinsic
linguistic tendencies of users when interacting online.

Keywords: Social Media $|$ Language Evolution $|$ Social Dynamics

## Introduction

The rapid expansion of social media platforms has revolutionized how we
connect and communicate, fundamentally altering the landscape of human
interaction. These platforms have become integral to our daily lives as
primary sources of information, entertainment, and personal communication [53,
3, 15]. While they offer unprecedented opportunities for connectivity and
interaction, they also intertwine entertainment-driven business models with
complex social dynamics, raising substantial concerns about their impact on
users and society at large [26]. The influence of social media on public
discourse and individual behavior has become a pressing concern within the
scientific community, particularly regarding issues of polarization,
misinformation [52, 11, 24, 26, 20], and hate speech [8, 31, 48]. Recent
studies have explored their potential effects on user behavior, uncovering
complex interactions and identifying key unresolved questions [5, 24, 26, 39,
27]. They confirm that online users often select information that aligns with
their preferences, overlook dissenting information, and form homophilic
communities [11], which may influence their belief formation and communication
methods. In this landscape, investigating the textual properties and the
vocabulary of user-generated content is crucial for understanding evolving
social dynamics in terms of lexicon influence [25, 57, 4, 56].

Historically, measuring vocabulary has posed a significant challenge in
psychology, considering its close association with other cognitive skills such
as reading comprehension and information processing [49, 51]. While previous
studies have extensively examined the impact of vocabulary size on academic
success, revealing substantial individual differences and emphasizing the
pivotal role of lexical knowledge in educational settings [36], there is still
a gap in understanding how these dynamics adapt to the digital era. Indeed,
linguistics is increasingly focusing on social media, spurred by concerns that
internet language may corrupt traditional writing practices and face-to-face
communication [6]. Language adapts to changes in culture, society, and
technology, leading to the emergence of novel linguistic forms like
abbreviations, phonetic spellings, neologisms, and multimedia elements such as
hashtags and emojis [18].

However, a systematic understanding of language complexity, despite extensive
discourse, still needs to be improved. Current research often addresses
specific elements of complexity but needs to provide a holistic view, making
it challenging to develop a unified theoretical framework. The definition of
complexity within linguistic studies is often ambiguous [23], and the role of
quantitative metrics in assessing complexity has not been fully explored, with
only limited studies addressing this approach [17, 37]. Scholarly debate on
linguistic complexity tends to adopt two distinct perspectives. The first
views complexity as a theoretical abstraction with little direct application
to real-world linguistic scenarios. The second treats it as an empirical
phenomenon that can be quantified and analyzed using theoretical tools.
Miestamo introduces the concepts of ‘absolute complexity’ and ‘relative
complexity’ [35]. Absolute complexity is considered an intrinsic property of
language systems, independent of user interaction. In contrast, relative
complexity involves the user’s perspective, measuring complexity as the cost
or difficulty encountered by language users.

Our research primarily explores relative complexity, focusing on how it
manifests in user interactions on social media platforms. In more detail, this
study investigates whether and how social media platforms have influenced the
linguistic constructs of user communications online. We analyze a large
dataset of nearly 300 million English comments across eight major social media
platforms—Facebook, Twitter, YouTube, Voat, Reddit, Usenet, Gab, and
Telegram—covering nearly three decades and several topics. We begin by
assessing the average vocabulary size of users according to their activity
level. We then provide a framework to evaluate the speed at which users reach
their maximum vocabulary, offering new insights into how digital environments
shape communication norms and user engagement. Finally, we map text complexity
by applying two established lexical richness and repetitiveness metrics and we
explore their evolution over several years.

## Users vocabulary in social media

In the following sections, we conduct a comparative analysis of 8 different
social media platforms: Facebook, Twitter, YouTube, Voat, Reddit, Usenet, Gab
and Telegram. Each social media contains comments related to different topics
(see Data Collection section for further details).

### The vocabulary of users

To measure the vocabulary size on social media platforms, we aggregate all the
comments from each user into one document. We then perform text preprocessing,
including tokenization and stemming, to facilitate accurate word counting (see
further details about this procedure in Materials and Methods section). In
this work, we refer to tokens as instances of individual words as they appear
in the text, whereas types represent distinct words without repetition.
Therefore, we associate to each user a couple of integer values containing the
number of tokens (i.e., the number of their total words) and the number of
types (i.e., the number of their unique words).

Figure 1 displays the complementary cumulative distribution functions (CCDF)
of the number of tokens and types within the documents of each user in each
dataset.

Figure 1: CCDF of the distributions of number of $(a)$ tokens and $(b)$ types
used by each user.

The distributions display general consistency across different social media
platforms and topics, albeit with varying magnitudes. Their behaviors are
almost identical, with the primary differences manifesting in their tails,
since tokens exhibit longer tails. This pattern aligns with the expectation
that shorter texts typically have nearly identical numbers of words and unique
words, whereas longer ones can present richer vocabularies. This observation
is further supported by Figure 6 in Supplementary Information (SI), which
illustrates the distributions of Type-Token Ratios (TTR). These distributions
are predominantly peaked at 1, indicating a high similarity between types and
tokens in most cases, with only a minority of users exhibiting lower TTR
values. Nonetheless, consistently across various topics and social media, most
users typically employ up to 10 unique words, indicating a relatively small
vocabulary size.

Obviously, this observation is somewhat dependent on the user activity, which
is known to follow a heavy-tailed distribution [5] (i.e., only a few users
exhibit high activity, while the majority show very low participation),
potentially skewing the observed vocabulary sizes. To disentangle the possible
effect of user activity, we categorize users into four classes —low, medium,
high, and very high—based on the number of comments they have posted on each
specific dataset (more details about the classification criteria are provided
in the Materials and Methods section). Figure 2 illustrates the distribution
of types within each class except Low, thus considering only users with
sufficient activity.

Figure 2: Distribution of the number of types (i.e., unique words) employed by
users according to their activity class, which is determined by the number of
comments they left in each specific dataset.

As expected, higher activity levels among users are associated with vocabulary
distributions centered around higher values, indicating a more diverse lexical
usage. Interestingly, while vocabulary size distributions generally show
consistency across various social media platforms, it seems specific topics
demand larger vocabularies. However, the shifting in the distributions could
be a consequence of the different sizes of each dataset, resulting in some
users having a larger number of comments and thus a larger number of
types/tokens. To control for this effect, Figure 7 in SI represents the same
plots but in which activity classes are computed according to the whole
distribution of comments of a specific social. Thus, we are able to compare
the behaviour of users with approximately the same number of comments. The
results show that the shifting disappear, providing evidence that neither the
topic of discussion can influence the total number of types used by users.

Furthermore, we examine whether users’ vocabulary distributions adhere to
Zipf’s Law [61, 62], a principle that suggests a predictable frequency
distribution of words in human languages. The results, in SI, indicate
consistent exponents across social media and topics, supporting the hypothesis
that the form of Zipf’s Law is a valid approximation of the frequency term
observed in online social media communications.

### Vocabulary Evolution

In the previous section we have explored the aggregated users’ production
without examining their evolution. To address this gap, we analyze how the
vocabularies of individual users evolve over time by tracking the rate at
which they introduce new words in their time-ordered comments.

We chronologically arrange each user’s comments and apply the same
tokenization process used for our previous analyses.

For a user $u$ with $n$ comments, we compute the vector
$\mathbf{v}^{u}\in\mathbb{N}^{n}$, where each entry $i$ represents the
cumulative count of unique words up to the $i$-th comment.

For example, if a user $u$ writes two comments, the first containing the words
“politics,” “health,” “comics,” and the second “politics,” “left,” “right,”
then $\mathbf{v}^{u}$ would be (3, 5). We calculate $\mathbf{v}^{u}$ for all
users having 25 to 100 comments (for manageability we consider 50 to 100
comments in Facebook News). This range ensures that we focus on users with a
reasonable level of activity and excludes accounts that may be malicious or
not genuine, often exhibiting excessively high comment counts.

We employ a linear interpolation on the scaled values of $\mathbf{v}$ in
$[0,1]$ to track the evolution of users’ vocabularies. Since $v_{i}\leq
v_{i+1},i=1,\ldots,N$, a possible measure of the speed at which each user
reaches its maximum vocabulary is the area under this curve, similar to
methodologies used in previous studies [19]. This value, which ranges from 0
to 1, provides insights into vocabulary usage patterns: values close to 0
indicate a late expansion of vocabulary, values close to 1 suggest rapid
saturation of vocabulary usage early on, and values around 0.5 indicate a
steady increase in vocabulary across comments.

Figure 3 illustrates the distribution of these measurements across different
topics and social media platforms.

Figure 3: Distribution of the area under the curves determined by users’
progressive exploration of their vocabulary.

We observe a general consistency across all distributions, peaking at values
around $0.6$; this suggests, on average, a continuous but modest addition of
new words to a user’s vocabulary with a minority of users reaching their whole
vocabulary in the first comments.

The consistency of the findings underscores universal behaviors that appear to
be largely independent of the specific platforms and topics involved.

### Comments complexity

Beyond the size of users’ vocabularies, exploring the general complexity of
comments adds a significant dimension to our analysis. The complexity of texts
can be approached from various perspectives, and the literature made numerous
metrics available [22, 58, 54, 16, 61, 45, 50]. After a careful review of the
available measures and previous works, we decided to rely upon two of them
that are able to provide a rather orthogonal perspective in terms of text
complexity, namely Yule’s $K$-complexity and gzip complexity $g$. The detailed
methodologies are described in the Materials and Methods section. We recall
that a high $K$ suggests a small lexical complexity, whereas it assumes its
minimum (i.e., $K=0$) for a text in which only distinct words are used. On the
other hand, values of $g$ close to $0$ (or even negative) suggest texts with
low repetitiveness, while values close to $1$ indicate texts with high
repetitive patterns.

As with our previous analysis, we compile the complete set of comments from
each user into single documents. $K$-complexity is calculated following the
previously outlined preprocessing steps, whereas gzip complexity is assessed
using the raw texts, as done by previous works [30]. Figure 4 illustrates the
distribution of both complexity measures among users who have posted at least
20 comments. For datasets comprising over 50,000 users, Yule’s $K$-complexity
was calculated on a sample of 50,000 users to ensure manageability and
computational efficiency.

Figure 4: Distribution of $(a)$gzip complexity and $(b)$ K-complexity for
users having at least 20 comments. For the larger dataset, we selected a
sample of 50000 users to compute $K-$complexity. For visual reasons, we add 1
to all the values of $K-$complexity.

We observe consistency in the distributions of both $K$ and $g$ on different
social media platforms and topics. Generally, the texts produced by users
display moderate lexical complexity and repetitiveness. However, a minority of
users produce highly repetitive texts and exhibit low complexity, which may
indicate the presence of automated or coordinated accounts [14, 10, 38, 29,
41, 30].

In SI, we present an analysis where each user’s comments are randomized before
being aggregated into documents, i.e. the document associated with a user is
made of random comments from other users. The results show distributions with
a lower variance, suggesting much more uniform behaviors. Although
distributions look similar, Mann-Whitney tests detect that the real and null
distributions are different in almost all cases, suggesting that users adopt
their vocabulary, that is not replicable with a random assignment of comments.

### Evolution of complexity

In previous sections, we have explored the complexity of texts written by
users online. However, an interesting point to investigate regards the
evolution of the complexity of comments over time as a proxy for the use of
social media as a public square for opinion sharing and active debate. Thus,
our focus now shifts to determining whether the complexity of comments has
changed over time. To achieve this, we select subsets of datasets with a
sufficiently broad time span. Specifically, we include Facebook News, Facebook
Vaccines, Twitter Vaccines, Usenet News, Usenet Politics, Usenet Talk, Voat
News, Voat Politics, and YouTube News and we analyze the whole set of
comments, without aggregating by user.

First, we examine the evolution of the number of types (i.e., unique words),
being the simplest complexity measure. To mitigate any potential bias from
users’ activity levels, for each year we classify each user into one of four
activity classes —low, mid, high, or very high— as previously established.
Figure 5 illustrates the annual progression of the average number of types
across all activity classes, computed for all years with a minimum number of
100 comments.

Figure 5: Evolution of the mean number of types in each dataset. The smooth
curves are obtained using a loess regression

The number of unique words appears to decrease in all user classes except for
those on Twitter and YouTube. Additionally, Figure 10 in SI reveals that the
Type-Token Ratio (TTR) remains relatively stable across all platforms,
suggesting a concurrent reduction in the total number of words (i.e., tokens)
used. This trend reflects also a decrease in user activity, characterized by
shorter comments that contain fewer unique words.

To further quantify the relationship between time and text complexity, we
implement a regression model with interaction terms to account for the
specific social media platform where each comment was posted. For this
analysis, we use a sample of $6000$ comments per year from each platform and
topic, ignoring all years having less than this number of comments.

We provide a detailed discussion of the model in the Materials and Methods
section. Moreover, Table S1 contains a breakdown of the dataset used for this
experiment.

We recall here that our regression model employs complexity measures as
regressors and the year of the comments as the dependent variable. To allow
for a better comparison between measures, we first normalize each regressor
and, after detecting a heteroskedasticity problem, we correct standard errors
with their robust version. The model, whose coefficients are detailed in Table
1, achieves an adjusted $R^{2}$ value of approximately 0.87.

Variables | Estimate | Total estimate | Standard error | $p$
---|---|---|---|---
$\beta_{0}$ | 2013.091 | 2013.091 | 0.010 | $<0.001$
$\beta_{1}$ | -0.367 | -0.367 | 0.034 | $<0.001$
$\beta_{2}$ | 0.079 | 0.079 | 0.007 | $<0.001$
$\beta_{3}$ | -0.228 | -0.228 | 0.007 | $<0.001$
$\beta_{0,tw}$ | 7.344 | 2020.435 | 0.036 | $<0.001$
$\beta_{0,un}$ | -13.644 | 1999.447 | 0.013 | $<0.001$
$\beta_{0,vt}$ | 4.376 | 2017.467 | 0.013 | $<0.001$
$\beta_{0,yt}$ | 2.076 | 2015.167 | 0.021 | $<0.001$
$\beta_{1,tw}$ | 4.548 | 4.181 | 0.136 | $<0.001$
$\beta_{1,un}$ | 0.314 | -0.054 | 0.035 | $<0.001$
$\beta_{1,vt}$ | 0.357 | -0.011 | 0.039 | $<0.001$
$\beta_{1,yt}$ | 1.031 | 0.664 | 0.064 | $<0.001$
$\beta_{2,tw}$ | -0.071 | 0.008 | 0.009 | $<0.001$
$\beta_{2,un}$ | -0.067 | 0.012 | 0.012 | $<0.001$
$\beta_{2,vt}$ | -0.067 | 0.012 | 0.010 | $<0.001$
$\beta_{2,yt}$ | -0.070 | 0.009 | 0.019 | $<0.001$
$\beta_{3,tw}$ | -0.098 | -0.326 | 0.032 | $0.002$
$\beta_{3,un}$ | -0.540 | -0.769 | 0.015 | $<0.001$
$\beta_{0,vt}$ | 0.047 | -0.181 | 0.012 | $<0.001$
$\beta_{0,yt}$ | 0.169 | -0.059 | 0.021 | $<0.001$
Table 1: Results of regression model.

We use Facebook as the baseline, i.e. the estimates of coefficients
$\beta_{i}$ refer to Facebook comments. Moreover, the ‘Total Estimate’ column
reports the sum of the coefficient with the relative baseline, i.e. if
$\beta_{k,j}$ is considered, the column reports the value
$\beta_{k,j}+\beta_{k}$, thus describing the change in $y$ after a unit change
of regressor $k$ if all the other variables are fixed and the comment comes
from social $j$. In other words, $\beta_{k,j}$ is the deviation from the
behavior observed on Facebook.

The results suggest that the number of unique words is positively correlated
with more recent years on Twitter and YouTube, as previously observed.
Conversely, there is a negative relation on other platforms. Regarding Yule’s
K-complexity, higher values are associated with more recent years, thus
suggesting that comments exhibit a decrease in lexical complexity over time.
Finally, $g$-complexity consistently shows negative associations with time
across all platforms, indicating that recent comments tend to have lower
repetitiveness, despite their lower lexical complexity.

To check the robustness of the results, we conduct a similar analysis in SI
using a logistic regression. Notably, despite the use of a different
methodology, the results show consistency.

Overall, the data indicates that comments across platforms exhibit lower
lexical complexity and repetitiveness as time progresses. Furthermore, in
nearly all cases, comments become shorter and contain fewer unique words.

## Conclusions

Our comprehensive analysis across eight major social media platforms reveals
consistent patterns in user behavior and language complexity. This study
clarifies how users’ linguistic behavior has adapted to the digital era and
evolved over nearly three decades of internet use. Despite the diversity of
topics and platforms analyzed, which encompass approximately 300 million
English comments over nearly three decades, we find a general decrease in the
length of comments and a reduction in lexical richness. Notably, our findings
suggest that language complexity is not influenced by the platform, but rather
reflect a broader, universal aspect of human communication. We have also
explored how individual users evolve their vocabulary over time, discovering
that most users gradually introduce new words, with distributions peaking at
relatively low levels of vocabulary expansion. This study underscores the
persistent nature of linguistic behavior across digital communication
platforms, emphasizing the influence of inherent human communication traits
over technological or topical variations. Our results provide valuable
insights into the dynamics of language use in digital environments, supporting
the hypothesis of universal linguistic patterns among social media users.

## Materials and Methods

### Data collection

Below we report the detailed procedure to gather each dataset.

Facebook \- We utilize datasets from prior studies on discussions concerning
Vaccines [46], News [47], and Brexit [12]. For the vaccine topic, the dataset
comprises approximately 2 million comments from public groups and pages
collected over the period from January 2, 2010, to July 17, 2017. For the News
topic, we selected pages from the Europe Media Monitor that reported the news
in English, resulting in a dataset containing roughly 362 million comments
between September 9, 2009, and August 18, 2016. Additionally, this dataset
includes approximately 4.5 billion likes associated with posts and comments on
these pages. Lastly, for the Brexit topic, the dataset encompasses around
460,000 comments from December 31, 2015, to July 29, 2016.

Gab \- We collected data from the Pushshift.io archive
(https://files.pushshift.io/gab/) on discussions from the platform’s inception
on August 10, 2016, until it temporarily went offline on October 29, 2018,
following the Pittsburgh shooting [28]. The dataset includes approximately 14
million comments.

Reddit \- Data was collected from the Pushshift.io archive
(https://pushshift.io/) covering the period from January 1, 2018, to December
31, 2022. We manually identified and selected subreddits for each topic that
best represented the targeted discussions. From this process, we gathered
approximately 800,000 comments from the r/conspiracy subreddit for the
Conspiracy topic. For the Vaccines topic, we collected about 70,000 comments
from the r/DebateVaccines subreddit, focusing on the COVID-19 vaccine debate.
The r/News subreddit provided roughly 400,000 comments for the News topic.
From the r/environment subreddit, we obtained approximately 70,000 comments
related to the Climate Change topic. Lastly, the r/science subreddit yielded
about 550,000 comments for the Science topic.

Telegram \- We compiled a list of 14 channels, each linked to one of the
study’s topics. We manually collected messages and their associated comments
from each channel. From the 4 channels related to the News topic
(news_notiziae, news_ultimora, news_edizionestraordinaria,
news_covidultimora), we gathered approximately 724,000 comments from posts
dated between April 9, 2018, and December 20, 2022. For the Politics topic,
the 2 channels (politics_besttimeline, politics_polmemes) yielded a total of
about 490,000 comments from the period between August 4, 2017, and December
19, 2022. Lastly, the 8 channels focused on the Conspiracy topic
(conspiracy_bennyjhonson, conspiracy_tommyrobinsonnews,
conspiracy_britainsfirst, conspiracy_loomeredofficial,
conspiracy_thetrumpistgroup, conspiracy_trumpjr, conspiracy_pauljwatson,
conspiracy_iononmivaccino) produced approximately 1.4 million comments from
August 30, 2019, to December 20, 2022.

Twitter \- We utilized datasets from previous research that include
discussions on Vaccines [55], Climate Change [20], and News [44]. We collected
approximately 50 million comments on the Vaccines topic from January 23, 2010,
to January 25, 2023. For the News topic, we expanded the dataset used in [44]
by including all threads with fewer than 20 comments, resulting in a total of
approximately 9.5 million comments collected from January 1, 2020, to November
29, 2022. Lastly, we gathered about 9.7 million comments on the Climate Change
topic from January 1, 2020, to January 10, 2023.

Usenet \- We collected data from the Usenet discussion system using the Usenet
Archive (https://archive.org/details/usenet?tab=about). We identified a range
of topics, including extensive, broad, and heterogeneous discussions within
active and populated newsgroups. As a result, we selected conspiracy,
politics, news, and talk as the topic candidates for our analysis. We gathered
approximately 280,000 comments from the alt for the Conspiracy
topic.conspiracy newsgroup, from September 1, 1994, to December 30, 2005.
About 2.6 million comments were collected from the alt for the Politics
topic.politics newsgroup between June 29, 1992, and December 31, 2005. We
obtained approximately 620,000 comments for the News topic from the alt.news
newsgroup, from December 5, 1992, to December 31, 2005. Finally, we collected
all discussions from the alt for the Talk topic.talk newsgroup, totaling about
2.1 million contents, from February 13, 1989, to December 31, 2005.

Voat \- We utilized a dataset described in [34] that encompasses the entire
lifespan of the platform from January 9, 2018, to December 25, 2020. This
dataset includes approximately 16.2 million posts and comments from around
113,000 users across approximately 7,100 subverses (Voat’s equivalent of a
subreddit). Similarly to previous platforms, we linked topics to specific
subverses. As a result, for the Conspiracy topic, we collected about 1 million
comments from the greatawakening subverse during the platform’s operational
period. For the Politics topic, we gathered roughly 1 million comments from
the politics subverse between June 16, 2014, and December 25, 2020. Lastly, we
amassed approximately 1.4 million comments from the news subverse for the News
topic between November 21, 2013, and December 25, 2020.

YouTube \- We utilized a dataset referenced in previous research that
initially focused on Climate Change discussions [20]. This dataset has been
expanded to include conversations on Vaccines and News topics, following the
approach used for other platforms. The data collection for YouTube was
conducted using the YouTube Data API
(https://developers.google.com/youtube/v3). For the Climate Change topic, we
collected approximately 840,000 comments from March 16, 2014, to February 28,
2022. For the Vaccines topic, we gathered comments between January 31, 2020,
and October 24, 2021, that include keywords related to COVID-19 vaccines, such
as _Sinopharm, CanSino, Janssen, Johnson &Johnson, Novavax, CureVac, Pfizer,
BioNTech, AstraZeneca, Moderna_, resulting in about 2.6 million comments.
Finally, for the News topic, we collected roughly 20 million comments from
February 13, 2006, to February 8, 2022, including videos and comments from a
list of UK-based news outlets, provided by Newsguard, a fact-checking agency.

We select only English text from this data as detected by the cld3 package of
R [40]. Table 2 shows a breakdown of the resulting dataset.

Dataset | Time range | Comments | Users
---|---|---|---
Facebook Brexit | 2015-12-31 2016-07-29 | 322365 | 171054
Facebook Vaccines | 2010-01-02 2017-07-17 | 1590907 | 304706
Facebook News | 2009-09-09 2016-08-17 | 229915622 | 36096691
Gab Feed | 2016-08-10 2018-10-29 | 10799968 | 126351
Reddit Climate change | 2018-01-01 2022-12-12 | 60113 | 22822
Reddit Conspiracy | 2018-01-01 2022-10-31 | 649054 | 82733
Reddit News | 2018-01-01 2018-12-31 | 358594 | 102982
Reddit Science | 2018-01-01 2022-12-11 | 488963 | 192675
Reddit Vaccines | 2018-01-21 2022-11-06 | 59980 | 4866
Telegram Conspiracy | 2019-08-30 2022-12-20 | 1111479 | 107009
Telegram News | 2018-05-24 2022-12-16 | 4493 | 1674
Telegram Politics | 2017-08-05 2022-12-19 | 20851 | 2013
Twitter Climate Change | 2020-01-01 2023-01-10 | 3562447 | 1467783
Twitter News | 2020-01-01 2022-11-29 | 5882655 | 1237679
Twitter Vaccines | 2010-01-23 2023-01-25 | 17682887 | 4916617
Usenet Conspiracy | 1994-09-01 2005-12-30 | 160632 | 30223
Usenet News | 1992-12-05 2005-12-30 | 385404 | 51204
Usenet Politics | 1992-06-29 2005-12-30 | 1541803 | 142469
Usenet Talk | 1989-02-13 2005-12-30 | 1390574 | 128321
Voat Conspiracy | 2018-01-09 2020-12-25 | 828018 | 24666
Voat News | 2013-11-21 2020-12-25 | 1164549 | 78199
Voat Politics | 2014-06-19 2020-12-25 | 902419 | 59442
Youtube News | 2006-02-13 2022-02-10 | 20946472 | 5623730
Table 2: Data breakdown of the dataset. We consider approximately 300M
comments wrote by 50M users.

### Preprocessing of comments

Before the main analysis, we tokenize the comments removing punctuation,
symbols (and emojis), numbers, URL, hashtags, and English stopwords.
Subsequently, we apply a stemmer to the resulting tokens, reducing them to
their root form. After these steps, a text may remain empty (for example if it
contains only tags or hashtags), therefore we remove all the comments
containing 0 tokens.

The preprocessing and the main analysis were conducted using the quanteda
package of R [7, 1].

### Measures of text complexity

In linguistics, special attention has always been posed to develop measures
capable of detecting the complexity of text. In particular, a text can be
considered complex from a variety of points of view, such as lexical,
readability, or repetitiveness. Many measures have been proposed (see [1] for
a collection of measures provided by quanteda.textstats package in R [7]).
Here, we have focused on lexical complexity and repetitiveness measures, using
Yule’s $K$ and $gzip$ complexity.

For what concerns the former, previous works have highlighted that many
lexical complexity measures are incapable of being independent of text length
[50, 54]. The same studies also highlight that, even if with some limitations,
the well-known Yule’s K-complexity [58] seems to be almost independent of text
length. Given a text of length $N$ with $V$ unique words, Yule’s K is defined
as

$K=10^{4}\cdot\left[-\frac{1}{N}+\sum_{i=1}^{V}V(i,N)\left(\frac{i}{N}\right)^{2}\right]$
(1)

where $V(i,N)$ denotes the number of words appearing $i$ times in the text. In
particular, the larger is $K$, the less rich the vocabulary is.

For what concerns the repetitiveness of text, we use an approach akin to
previous works [42, 13], i.e. we compress the raw texts using gzip and compare
their dimensions with the original ones. We define the gzip complexity $g$ as

$g=\frac{s_{raw}-s_{compressed}}{s_{raw}}$ (2)

where $s_{raw}$ is the size of the raw text and $s_{compressed}$ is the size
of the compressed text. Note that if the text is highly repetitive
$s_{compressed}\ll s_{raw}$ and therefore $g\approx 1$. On the other hand, low
values characterize texts with low repetitiveness. In particular, for very
short texts the compressor may increase the size, therefore $g$ can also
assume negative values.

### Classification of users

We adopt a non-parametric method developed by [21] to partition heavy-tailed
distributions, and recently employed in different domains [2, 9, 15] to divide
users into classes according to the number of comments they left. In detail,
we consider four classes of activity, namely $\\{low,mid,high,very\,high\\}$,
and adopt the following procedure: firstly, we compute the mean number of
comments $\bar{x}$ and assign to class low all users that have left less than
$\bar{x}$ comments; then, we delete these users from the distribution and
recursively repeat the procedure until each user is assigned to one of the
four classes $\\{low,mid,high,very\;high\\}$.

### Regression model

Since we are interested in obtaining a unique model capable of detecting the
overall relationship between time, social media and the complexity of
comments, we consider

$y_{i}\sim[1\;w_{i}\;K_{i}\;g_{i}]\cdot{\bf B}\cdot\begin{bmatrix}1\\\
tw_{i}\\\ vt_{i}\\\ yt_{i}\\\ un_{i}\end{bmatrix}$ (3)

where $\cdot$ is the standard matrix product and

  * •

$w_{i}$ is the number of types of comment $i$;

  * •

$K_{i}$ is the $K-$complexity of comment $i$;

  * •

$g_{i}$ is the $g$-complexity of comment $i$;

  * •

$y_{i}$ is the year in which comment $i$ has been created;

  * •

$tw_{i},vt_{i},yt_{i}$ and $un_{i}$ are dummy variables that are equal to one
if and only comment $i$ has been written in Twitter, Voat, YouTube or Usenet,
respectively.

Finally, ${\bf B}$ is the matrix of the coefficients, defined as follows:

$B=\begin{bmatrix}\beta_{0}&\beta_{0,tw}&\beta_{0,vt}&\beta_{0,yt}&\beta_{0,un}\\\
\beta_{1}&\beta_{1,tw}&\beta_{1,vt}&\beta_{1,yt}&\beta_{1,un}\\\
\beta_{2}&\beta_{2,tw}&\beta_{2,vt}&\beta_{2,yt}&\beta_{2,un}\\\
\beta_{3}&\beta_{3,tw}&\beta_{3,vt}&\beta_{3,yt}&\beta_{3,un}\\\
\end{bmatrix}$

We use Facebook as the baseline, i.e. the estimates of coefficients
$\beta_{i}$ refer to Facebook comments. Therefore, the generic coefficient
$\beta_{k}+\beta_{k,j}$ describes the change in $y$ due to a unit increase of
regressor $k$ if all the other variables are kept constant and comment $i$
comes from a social $j$ different from Facebook. To obtain more interpretable
estimates, we first normalize all the regressors. Therefore, the parameters
quantify changes in the dependent variable in standard deviations. Moreover,
since we detect heteroskedasticity, we correct the errors and tests using the
sandwich package [60, 59] in R.

## Acknowledgements

The work is supported by IRIS Infodemic Coalition (UK government, grant no.
SCH-00001-3391), SERICS (PE00000014) under the NRRP MUR program funded by the
European Union - NextGenerationEU, project CRESP from the Italian Ministry of
Health under the program CCM 2022, PON project “Ricerca e Innovazione”
2014-2020, and PRIN Project MUSMA for Italian Ministry of University and
Research (MUR) through the PRIN 2022. This work was supported by the PRIN 2022
“MUSMA” - CUP G53D23002930006 - Funded by EU - Next-Generation EU – M4 C2
I1.1.

## Supplementary Information

### Type-Token ratio of users’ comments

We used the Type-Token ratio (TTR), a simple measure defined as the ratio of
unique words over the total words of a text, to evaluate the lexical
complexity of users’ comments. Figure 6 shows the distribution of TTR values
obtained from the documents created with text of each user.

The distribution are mostly peaked at one, thus indicating a high similarity
between the number of types and tokens.

### Zipf’s law on comments

Zipf’s law was initially designed and studied in [61, 62] to model the
frequency at which words are observed in texts. In particular, it simply
states that the $r-$th most frequent word has frequency $f(r)$ that scales
according to

$f(r)\approx r^{-\alpha}$ (4)

In particular, real texts often show $\alpha\approx 1$.

Although many alternatives have been proposed (see [33, 32] for example) and
some works suggest that the right form decay of $f(r)$ may depend on the
context [43], here we rely on the simplest form of (4).

In particular, we aim to understand if it is reasonable to suppose that the
comments generated by users follow (4). As explained in the main text, for
each user we paste together her/his generated texts, obtaining a document for
each of them. We then apply the standard tokenization and stemming procedure
and keep only users with at least 1000 tokens. In this way, we select a subset
of active users for which the fitting may be reliable. We then apply a linear
regression to the logged values of $r$ and $f(r)$ and store the results.
Figure 8 $(a)$ shows the distribution of $\alpha$ for all the significant fit
($p<0.001$). Note that Telegram News was not included since no users have a
sufficient number of tokens.

Interestingly, we observe very similar distributions centered in $\approx
0.7$. Moreover, the majority of regressions explain a very large fraction of
the variance, as depicted in Figure 8 $(b)$. The results thus suggest that
equation (4) may be a reliable approximation for the frequency of terms used
by online users, even if the exponents show lower values than the ones
commonly observed.

## Null model of complexity measures

To detect if each user possesses her/his distinct vocabulary, we select and
randomize the comments of users (with more than $20$ comments) before the
aggregation into documents in each distinct dataset. Doing that, each user is
associated with a set of random comments, having the same cardinality as the
original document.

To ensure consistent statistics and manageability, we select a sample of $15K$
users for bigger datasets.

We then compute gzip complexity and Yule’s $K$ for each document. The
resulting distributions are shown in Figure 9.

We observe narrower distributions than the real case, suggesting even more
uniform behaviours. To check if statistically significant differences arise
between the real and the null distributions, we apply $2-$sided Mann-Whitney
tests, whose results are shown in Table 4. Interestingly, we observe that, in
almost all cases, the distributions are significantly different thus
suggesting that each user has her/his distinct vocabulary that is not possible
to reply with a random assignment of texts.

### Logistic regression

In this section, we define a logistic regression model employing the same data
used in the main paper (Table 3 shows its data breakdown).

Let’s consider the dataset comprising the comments from social $j$ and let $d$
be its distribution of years. We say that a comment $i$ is recent if
$year(i)>median(d)$. Then, we employ a logistic regression model to predict if
a comment is recent based on their measures of complexity. More in detail, the
model has the following form:

$p=\frac{1}{1+e^{-z}},$ (5)

or equivalently

$z=\ln\frac{p}{1-p},$ (6)

where $z=\beta_{0}+\beta_{1}w+\beta_{2}K+\beta_{3}g$ and $w,K,g$ are the
number of types, the $K$-complexity and $g-$complexity respectively.

Thus, a positive estimate of parameter $\beta_{j},j=1,2,3$ indicates that an
increase in complexity $j$ increases the logit of being a recent comment. The
opposite result holds if $\beta_{j}$ is negative.

We construct a model for each social separately. The results of the analysis
are reported in Table 5, where we show only the values of the parameter for
each social and their significativity.

Although not all the coefficients are significant, we can see a general
coherence with the result obtained by the model presented in the main paper,
with the only differences arising for Twitter.

Moreover, by applying the same methodology to the whole dataset without
distinguishing for the social of provenience, we obtain the following
estimates (all significant at the $0.001$ level):
$\beta_{0}=0.744,\beta_{1}=-0.028,\beta_{2}=2.489\cdot 10^{-4}$ and
$\beta_{3}=-1.044$, coherent with our results.

Overall, the analysis confirms that low values of types and $g$ are linked to
recent comments, while low values of $K$ are linked to less recent texts.

### Supplementary Figures

Figure 6: Distribution of TTR values of comments of users. To consider
consistent cases, we only compute TTR for users having at least 50 tokens.

Figure 7: Distribution of the number of types (i.e., unique words) employed by
users according to their activity class. In this case, the activity classes
are computed using the comments distribution of the whole social platform.

Figure 8: Distribution of the $(a)$ exponent of the Zipf’s Law and $(b)$
$R^{2}$ of the regressions. Telegram News was excluded since no user had more
than 1000 tokens.

Figure 9: Distribution of gzip complexity and Yule’s $K$ over documents
obtained in the case in which the comments of users are randomized.

Figure 10: Evolution of TTR values across datasets and classes of activities.

### Supplementary tables

Table 3: Data breakdown of the dataset used in the regression analysis Social | Topic | Min year | Max year | Comments | Users
---|---|---|---|---|---
Facebook | News | 2010 | 2016 | 42000 | 39096
Facebook | Vaccines | 2010 | 2017 | 48000 | 27742
Twitter | Vaccines | 2015 | 2022 | 36000 | 33042
Usenet | News | 1994 | 2005 | 72000 | 21103
Usenet | Politics | 1994 | 2005 | 72000 | 26739
Usenet | Talk | 1992 | 2005 | 84000 | 27416
Voat | News | 2015 | 2020 | 36000 | 9727
Voat | Politics | 2015 | 2020 | 36000 | 8991
Youtube | News | 2008 | 2022 | 90000 | 77682

Table 4: Results of Mann-Whitney tests comparing $g$ and $K$ distributions from real and null model case.  Dataset | $p_{g}$ | $p_{k}$
---|---|---
Facebook Brexit | ¡0.001 | ¡0.001
Facebook Vaccines | ¡0.001 | ¡0.001
Gab Feed | ¡0.001 | ¡0.001
Reddit Climate Change | ¡0.001 | ¡0.001
Reddit Conspiracy | ¡0.001 | ¡0.001
Reddit News | ¡0.001 | ¡0.001
Reddit Science | ¡0.001 | ¡0.001
Reddit Vaccines | ¡0.001 | ¡0.001
Telegram Conspiracy | ¡0.001 | ¡0.001
Telegram News | 0.650 | 0.001
Telegram Politics | 0.717 | ¡0.001
Twitter Climate Change | ¡0.001 | ¡0.001
Twitter News | ¡0.001 | ¡0.001
Twitter Vaccines | ¡0.001 | ¡0.001
Usenet Conspiracy | ¡0.001 | ¡0.001
Usenet News | ¡0.001 | ¡0.001
Usenet Politics | ¡0.001 | ¡0.001
Usenet Talk | ¡0.001 | ¡0.001
Voat Conspiracy | ¡0.001 | ¡0.001
Voat News | ¡0.001 | ¡0.001
Voat Politics | ¡0.001 | ¡0.001
Youtube News | ¡0.001 | ¡0.001
Facebook News | ¡0.001 | ¡0.001

Table 5: Result of logistic regression for each dataset. $*,**,***$ indicate that the coefficient is significant at $0.05,0.01,0.001$ level respectively. Social | $\beta_{0}$ | $\beta_{1}$ | $\beta_{2}$ | $\beta_{3}$
---|---|---|---|---
Usenet | $-0.103^{***}$ | $-0.013^{*}$ | $0.015^{***}$ | $-0.369^{***}$
Facebook | $-0.132^{***}$ | $-0.096^{***}$ | $0.084^{***}$ | $-0.298^{***}$
YouTube | $-0.133^{***}$ | $0.069^{***}$ | $0.004$ | $-0.026^{**}$
Voat | $0.001$ | $-0.006$ | $0.007$ | $-0.164^{***}$
Twitter | $-0.275^{***}$ | $0.029^{***}$ | ${-2.047\cdot 10^{-4}}^{***}$ | $0.036$

## References

  * [1] quanteda.textstats.  https://cran.r-project.org/web/packages/quanteda.textstats/quanteda.textstats.pdf, 2024.  Accessed: 2024-05-30.
  * [2] Giovanni Abramo, Ciriaco Andrea D’Angelo, and Anastasiia Soldatenkova.  An investigation on the skewness patterns and fractal nature of research productivity distributions at field and discipline level.  Journal of Informetrics, 11(1):324–335, 2017.
  * [3] Thomas Aichner, Matthias Grünfelder, Oswin Maurer, and Deni Jegeni.  Twenty-five years of social media: a review of social media applications and definitions from 1994 to 2019.  Cyberpsychology, behavior, and social networking, 24(4):215–222, 2021.
  * [4] Abdulrahman Alrumaih, Ali Al-Sabbagh, Ruaa Alsabah, Harith Kharrufa, and James Baldwin.  Sentiment analysis of comments in social media.  International Journal of Electrical & Computer Engineering (2088-8708), 10(6), 2020.
  * [5] Michele Avalle, Niccolò Di Marco, Gabriele Etta, Emanuele Sangiorgio, Shayan Alipour, Anita Bonetti, Lorenzo Alvisi, Antonio Scala, Andrea Baronchelli, Matteo Cinelli, and Walter Quattrociocchi.  Persistent interaction patterns across social media platforms and over time.  Nature, 628(8008):582–589, March 2024.
  * [6] Naomi S Baron.  Always on: Language in an online and mobile world.  Oxford University Press, 2008.
  * [7] Kenneth Benoit, Kohei Watanabe, Haiyan Wang, Paul Nulty, Adam Obeng, Stefan Müller, and Akitaka Matsuo.  quanteda: An r package for the quantitative analysis of textual data.  Journal of Open Source Software, 3(30):774, 2018.
  * [8] Sergio Andrés Castaño-Pulgarín, Natalia Suárez-Betancur, Luz Magnolia Tilano Vega, and Harvey Mauricio Herrera López.  Internet, social media and online hate speech. systematic review.  Aggression and violent behavior, 58:101608, 2021.
  * [9] Matteo Cinelli.  Ambiguity of network outcomes.  Journal of Business Research, 129:555–561, 2021.
  * [10] Matteo Cinelli, Stefano Cresci, Walter Quattrociocchi, Maurizio Tesconi, and Paola Zola.  Coordinated inauthentic behavior and information spreading on twitter.  Decision Support Systems, 160:113819, 2022.
  * [11] Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini.  The echo chamber effect on social media.  Proceedings of the National Academy of Sciences, 118(9):e2023301118, 2021.
  * [12] Michela Del Vicario, Fabiana Zollo, Guido Caldarelli, Antonio Scala, and Walter Quattrociocchi.  Mapping social dynamics on facebook: The brexit debate.  Social Networks, 50:6–16, 2017.
  * [13] Antonio Desiderio, Anna Mancini, Giulio Cimini, and Riccardo Di Clemente.  Recurring patterns in online social media interactions during highly engaging events.  arXiv preprint arXiv:2306.14735, 2023.
  * [14] Niccolò Di Marco, Sara Brunetti, Matteo Cinelli, and Walter Quattrociocchi.  Post-hoc evaluation of nodes influence in information cascades: the case of coordinated accounts.  arXiv preprint arXiv:2401.01684, 2024.
  * [15] Niccolò Di Marco, Matteo Cinelli, Shayan Alipour, and Walter Quattrociocchi.  Users volatility on reddit and voat.  IEEE Transactions on Computational Social Systems, page 1–9, 2024.
  * [16] Daniel Dugast.  Vocabulaire et stylistique, volume 8.  Slatkine, 1979.
  * [17] Katharina Ehret, Aleksandrs Berdicevskis, Christian Bentz, and Alice Blumenthal-Dramé.  Measuring language complexity: challenges and opportunities.  Linguistics Vanguard, 9(s1):1–8, 2023.
  * [18] Jacob Eisenstein, Brendan O’Connor, Noah A Smith, and Eric P Xing.  Diffusion of lexical change in social media.  PloS one, 9(11):e113114, 2014.
  * [19] Gabriele Etta, Emanuele Sangiorgio, Niccolò Di Marco, Michele Avalle, Antonio Scala, Matteo Cinelli, and Walter Quattrociocchi.  Characterizing engagement dynamics across topics on facebook.  PLOS ONE, 18(6):1–12, 06 2023.
  * [20] Max Falkenberg, Alessandro Galeazzi, Maddalena Torricelli, Niccolò Di Marco, Francesca Larosa, Madalina Sas, Amin Mekacher, Warren Pearce, Fabiana Zollo, Walter Quattrociocchi, and Andrea Baronchelli.  Growing polarization around climate change on social media.  Nature Climate Change, 12(12):1114–1121, November 2022.
  * [21] Wolfgang Glänzel and András Schubert.  Characteristic scores and scales in assessing citation impact.  Journal of Information Science, 14(2):123–127, 1988.
  * [22] Felix Golcher.  A new text statistical measure and its application to stylometry.  In Proceedings of Corpus Linguistics, 2007.
  * [23] Tao Gong and Christophe Coupé.  A report on the workshop on complexity in language: developmental and evolutionary perspectives.  Biolinguistics, 5(4):370–380, 2011.
  * [24] Sandra González-Bailón, David Lazer, Pablo Barberá, Meiqing Zhang, Hunt Allcott, Taylor Brown, Adriana Crespo-Tenorio, Deen Freelon, Matthew Gentzkow, Andrew M Guess, et al.  Asymmetric ideological segregation in exposure to political news on facebook.  Science, 381(6656):392–398, 2023.
  * [25] Felix Greaves, Daniel Ramirez-Cano, Christopher Millett, Ara Darzi, Liam Donaldson, et al.  Use of sentiment analysis for capturing patient experience from free-text comments posted online.  Journal of medical Internet research, 15(11):e2721, 2013.
  * [26] Andrew M Guess, Neil Malhotra, Jennifer Pan, Pablo Barberá, Hunt Allcott, Taylor Brown, Adriana Crespo-Tenorio, Drew Dimmery, Deen Freelon, Matthew Gentzkow, et al.  How do social media feed algorithms affect attitudes and behavior in an election campaign?  Science, 381(6656):398–404, 2023.
  * [27] Andrew M Guess, Neil Malhotra, Jennifer Pan, Pablo Barberá, Hunt Allcott, Taylor Brown, Adriana Crespo-Tenorio, Drew Dimmery, Deen Freelon, Matthew Gentzkow, et al.  Reshares on social media amplify political news but do not detectably affect beliefs or opinions.  Science, 381(6656):404–408, 2023.
  * [28] Trevor Hunnicutt and Paresh Dave.  Gab.com goes offline after pittsburgh synagogue shooting, 2018.
  * [29] Franziska B. Keller, David Schoch, Sebastian Stier, and JungHwan Yang.  Political astroturfing on twitter: How to coordinate a disinformation campaign.  Political Communication, 37(2):256–280, October 2019.
  * [30] Kyumin Lee, Brian Eoff, and James Caverlee.  Seven months with the devils: A long-term study of content polluters on twitter.  Proceedings of the International AAAI Conference on Web and Social Media, 5(1):185–192, Aug. 2021.
  * [31] Yonatan Lupu, Richard Sear, Nicolas Velásquez, Rhys Leahy, Nicholas Johnson Restrepo, Beth Goldberg, and Neil F Johnson.  Offline events and online hate.  PLoS one, 18(1):e0278511, 2023.
  * [32] Benoit Mandelbrot.  On the theory of word frequencies and on related markovian models of discourse.  Structure of language and its mathematical aspects, 12:190–219, 1961.
  * [33] Benoit Mandelbrot et al.  An informational theory of the statistical structure of language.  Communication theory, 84:486–502, 1953.
  * [34] Amin Mekacher and Antonis Papasavva.  “i can’t keep it up” a dataset from the defunct voat.co news aggregator.  Proceedings of the International AAAI Conference on Web and Social Media, 16(1):1302–1311, May 2022.
  * [35] Matti Miestamo et al.  Grammatical complexity in a cross-linguistic perspective.  Language Complexity, Miestamo et al.(eds), John Benjamins, 2008.
  * [36] James Milton and Jeanine Treffers-Daller.  Vocabulary size revisited: the link between vocabulary size and academic achievement.  Applied Linguistics Review, 4(1):151–172, 2013.
  * [37] Mehdi Mirzapour, Jean-Philippe Prost, and Christian Retoré.  Measuring linguistic complexity: introducing a new categorial metric.  Logic and Algorithms in Computational Linguistics 2018 (LACompLing2018), pages 95–123, 2020.
  * [38] Alexander C. Nwala, Alessandro Flammini, and Filippo Menczer.  A language framework for modeling social media account behavior.  EPJ Data Science, 12(1), August 2023.
  * [39] Brendan Nyhan, Jaime Settle, Emily Thorson, Magdalena Wojcieszak, Pablo Barberá, Annie Y Chen, Hunt Allcott, Taylor Brown, Adriana Crespo-Tenorio, Drew Dimmery, et al.  Like-minded sources on facebook are prevalent but not polarizing.  Nature, 620(7972):137–144, 2023.
  * [40] Jeroen Ooms.  cld3: Google’s Compact Language Detector 3, 2024.  R package version 1.6.0.
  * [41] Diogo Pacheco, Pik-Mai Hui, Christopher Torres-Lugo, Bao Tran Truong, Alessandro Flammini, and Filippo Menczer.  Uncovering coordinated networks on social media: Methods and case studies.  Proceedings of the International AAAI Conference on Web and Social Media, 15:455–466, May 2021.
  * [42] Emilia Parada-Cabaleiro, Maximilian Mayerl, Stefan Brandl, Marcin Skowron, Markus Schedl, Elisabeth Lex, and Eva Zangerle.  Song lyrics have become simpler and more repetitive over the last five decades.  Scientific Reports, 14(1):5531, 2024.
  * [43] Steven T Piantadosi.  Zipf’s word frequency law in natural language: a critical review and future directions.  Psychon. Bull. Rev., 21(5):1112–1130, October 2014.
  * [44] Alessandro Quattrociocchi, Gabriele Etta, Michele Avalle, Matteo Cinelli, and Walter Quattrociocchi.  Reliability of news and toxicity in twitter conversations.  In Frank Hopfgartner, Kokil Jaidka, Philipp Mayr, Joemon Jose, and Jan Breitsohl, editors, Social Informatics, pages 245–256, Cham, 2022. Springer International Publishing.
  * [45] Alfréd Rényi.  On measures of entropy and information.  In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pages 547–562. University of California Press, 1961.
  * [46] Ana Lucía Schmidt, Fabiana Zollo, Antonio Scala, Cornelia Betsch, and Walter Quattrociocchi.  Polarization of the vaccination debate on facebook.  Vaccine, 36(25):3606–3612, 2018.
  * [47] Ana Lucía Schmidt, Fabiana Zollo, Michela Del Vicario, Alessandro Bessi, Antonio Scala, Guido Caldarelli, H. Eugene Stanley, and Walter Quattrociocchi.  Anatomy of news consumption on facebook.  Proceedings of the National Academy of Sciences, 114(12):3035–3039, 2017.
  * [48] Alexandra A Siegel.  Online hate speech.  Social media and democracy: The state of the field, prospects for reform, pages 56–88, 2020.
  * [49] Mary K Smith.  Measurement of the size of general english vocabulary through the elementary grades and high school.  Genetic Psychology Monographs, 24:311–345, 1941.
  * [50] Kumiko Tanaka-Ishii and Shunsuke Aihara.  Computational constancy measures of texts—yule’skand rényi’s entropy.  Computational Linguistics, 41(3):481–502, September 2015.
  * [51] Erwin Tschirner.  Breadth of vocabulary and advanced english study: An empirical investigation.  Electronic Journal of Foreign Language Teaching, 1(1):27–39, 2004.
  * [52] Joshua Tucker, Andrew Guess, Pablo Barbera, Cristian Vaccari, Alexandra Siegel, Sergey Sanovich, Denis Stukal, and Brendan Nyhan.  Social media, political polarization, and political disinformation: A review of the scientific literature.  SSRN Electronic Journal, 2018.
  * [53] Joshua A Tucker, Andrew Guess, Pablo Barberá, Cristian Vaccari, Alexandra Siegel, Sergey Sanovich, Denis Stukal, and Brendan Nyhan.  Social media, political polarization, and political disinformation: A review of the scientific literature.  Political polarization, and political disinformation: a review of the scientific literature (March 19, 2018), 2018.
  * [54] Fiona J. Tweedie and R. Harald Baayen.  How variable may a constant be? measures of lexical richness in perspective.  Computers and the Humanities, 32(5):323–352, 1998.
  * [55] Carlo M. Valensise, Matteo Cinelli, Matthieu Nadini, Alessandro Galeazzi, Antonio Peruzzi, Gabriele Etta, Fabiana Zollo, Andrea Baronchelli, and Walter Quattrociocchi.  Lack of evidence for correlation between covid-19 infodemic and vaccine acceptance, 2021.  Preprint at https://arxiv.org/abs/2107.07946.
  * [56] Svitlana Volkova, Yoram Bachrach, Michael Armstrong, and Vijay Sharma.  Inferring latent user properties from texts published in social media.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  * [57] Guixian Xu, Yueting Meng, Xiaoyu Qiu, Ziheng Yu, and Xu Wu.  Sentiment analysis of comment texts based on bilstm.  Ieee Access, 7:51522–51532, 2019.
  * [58] C Udny Yule.  The statistical study of literary vocabulary.  Cambridge University Press, 2014.
  * [59] Achim Zeileis.  Econometric computing with HC and HAC covariance matrix estimators.  Journal of Statistical Software, 11(10):1–17, 2004.
  * [60] Achim Zeileis, Susanne Köll, and Nathaniel Graham.  Various versatile variances: An object-oriented implementation of clustered covariances in R.  Journal of Statistical Software, 95(1):1–36, 2020.
  * [61] George Kingsley Zipf.  The psycho-biology of language: An introduction to dynamic philology.  Routledge, 2013.
  * [62] George Kingsley Zipf.  Human behavior and the principle of least effort: An introduction to human ecology.  Ravenio books, 2016.


# CUSBoost: Cluster-based Under-sampling with Boosting for Imbalanced
Classification

Farshid Rayhan, Sajid Ahmed, Asif Mahbub, Md. Rafsan Jani,
Swakkhar Shatabda, and Dewan Md. Farid  Department of Computer Science &
Engineering, United International University, Bangladesh
Email<EMAIL_ADDRESS>
###### Abstract

Class imbalance classification is a challenging research problem in data
mining and machine learning, as most of the real-life datasets are often
imbalanced in nature. Existing learning algorithms maximise the classification
accuracy by correctly classifying the majority class, but misclassify the
minority class. However, the minority class instances are representing the
concept with greater interest than the majority class instances in real-life
applications. Recently, several techniques based on sampling methods (under-
sampling of the majority class and over-sampling the minority class), cost-
sensitive learning methods, and ensemble learning have been used in the
literature for classifying imbalanced datasets. In this paper, we introduce a
new clustering-based under-sampling approach with boosting (AdaBoost)
algorithm, called CUSBoost, for effective imbalanced classification. The
proposed algorithm provides an alternative to RUSBoost (random under-sampling
with AdaBoost) and SMOTEBoost (synthetic minority over-sampling with AdaBoost)
algorithms. We evaluated the performance of CUSBoost algorithm with the state-
of-the-art methods based on ensemble learning like AdaBoost, RUSBoost,
SMOTEBoost on 13 imbalance binary and multi-class datasets with various
imbalance ratios. The experimental results show that the CUSBoost is a
promising and effective approach for dealing with highly imbalanced datasets.

###### Index Terms:

Boosting; Class imbalance; Clustering; Ensemble classifier; Sampling; RUSBoost

##  I Introduction

In machine learning (ML) for data mining (DM) applications, supervised
learning (or classification) is the process of identifying new/ unknown
instances employing classifiers (or classification algorithms) based on a
group of instances with known class membership (training data) [1, 2, 3, 4].
Often real-world data sets are multi-class, high-dimensional and class-
imbalanced, which fall-off the classification accuracy of many ML algorithms.
Therefore, number of ensemble classifiers with sampling techniques have been
proposed for classifying binary-class low-dimensional imbalanced data in the
last decade [5, 6, 7]. Ensemble classifiers use multiple ML algorithms to
improve the performance of individual classifiers that combine multiple
hypotheses to form an advance hypothesis [3]. The sampling methods use under-
sampling (under-sampling of the majority class instances) and over-sampling
(over-sampling the minority class instances) techniques to alter the original
class distribution of imbalanced data. The under-sampling methods with random
sampling of the majority class might suffer from the loss of potentially
useful training instances. On the other hand, over-sampling with replacement
doesn’t significantly improve minority class recognition and increase the
likelihood of overfitting [8].

In real-world class imbalance data sets, the minority class instances are
outnumbered by the majority class instances. However, the minority class
instances are representing the concept with greater interest than the majority
class instances [9]. The traditional ML for DM algorithms, such as decision
tree (DT) [1, 3], naïve Bayes (NB) classifier [2], and k-nearest neighbors
(kNN) [1], build the classification models that maximise the classification
rate, but ignore the minority class. The most approved methods for dealing
with the class imbalance problems are sampling techniques, ensemble methods,
and cost-sensitive learning methods. The sampling techniques (under-sampling
and over-sampling) either remove the majority class instances from the
imbalanced data or add the minority class instances into the imbalanced data
to get the balanced data. The ensemble methods such as Bagging and Boosting
are also widely used for classifying imbalanced data. Basically, the ensemble
methods use sampling technique in each iteration. The cost-sensitive learning
is also applied for solving the class imbalance problems, which assigns
different cost of misclassification errors for different classes. Usually,
high cost for the minority class and low cost for the majority class. But, the
classification results are not stable in cost-sensitive learning methods, as
it is difficult to get the accurate misclassification cost and different
misclassification cost might result in different inductions.

The methods for dealing with class imbalance problems are divided into two
categories: (a) external methods and (b) internal methods. The external
methods are also known as data balancing methods, which preprocess the
imbalanced data to get the balanced data. The internal methods modify the
existing learning algorithms for reducing their sensitiveness to the class
imbalance when learning from the imbalanced data. In this paper, we present a
new clustering-based under-sampling approach with boosting (AdaBoost), called
CUSBoost algorithm. We divide the imbalanced dataset into two part: majority
class instances and minority class instances. Then, we cluster the majority
class instances into several clusters using k-means clustering algorithm and
select the majority class instances from each cluster to form a balanced
dataset, where the majority and minority class instances are almost equal.
Clustering helps us to group the majority class instances in such a way that
instances in the same cluster are more similar to each other than to those in
other clusters. So, instead of randomly removing the majority class instances
we used clustering technique to select the majority class instances. CUSBoost
combines the sampling and boosting methods to form an efficient and effective
algorithm for class imbalance learning. We tested the performance of CUSBoost
algorithm with AdaBoost, RUSBoost, and SMOTEBoost algorithms on 13 imbalance
datasets. Based on the experimental results, we can validate that combining
clustering-based under-sampling approach with AdaBoost algorithm is a
promising technique for alleviating class imbalance problem.

The remainder of this paper is organised as follows. Section II presents
related works. Section III describes the data balancing methods, and Section 3
presents the CUSBoost algorithm. Section V provides the experimental results.
Finally, we conclude in Section VI.

##  II Related work

In the last decade, sampling methods, bagging and boosting based ensemble
methods, and cost-sensitive learning methods, have been used to deal with the
imbalanced binary classification problems [10, 11]. Fig. 1 shows the process
of classifying imbalanced data using sampling with boosting approach.

Figure 1: Sampling with boosting for classifying imbalanced data.

Sun et al. [8] proposed an ensemble method for addressing binary-class
imbalance problems by converting an imbalanced binary learning process into
multiple balanced learning processes. In this method, the majority class
instances were divided into several groups/ sub-data sets, where each sub-set
has the similar number of minority class instances. So, several balanced data
sets were generated. Then, each balanced data set was employed to build a
binary classifier. Finally, these binary classifiers were combined into an
ensemble classifier to classify new data.

Chawla et al. [12] proposed an over-sampling approach called SMOTE (Synthetic
Minority Over-sampling TEchnique) algorithm in which the minority class is
over-sampled by creating synthetic minority class instances rather than by
over-sampling with replacement. SMOTE generated synthetic instances by
operating in feature space rather than data space employing $k$ minority class
nearest neighbors. Their result showed that the combination of over-sampling
with under-sampling performed better in Receiver Operating Characteristic
(ROC) space. Santos et al. [13] implemented a cluster-based (k-means) over-
sampling approach where SMOTE was adapted to oversample clusters with reduced
sizes. This work considered merging the minority class instances from the
multiple over-sampled datasets. Blagus and Lusa [14] investigated the behavior
of SMOTE on high-dimensional imbalanced data, where the number of features
greatly exceeds the number of training instances. They found that feature
selection is necessary for SMOTE with k-nearest neighbors (kNN), as SMOTE
strongly biases the classifier towards the minority class.

Seiffert et al. [15] presented a new hybrid sampling/boosting algorithm,
called RUSBoost, which applied random under-sampling (RUS) with AdaBoost
algorithm. RUS randomly removes the majority class instances to form a
balanced data. RUSBoost was built based on the SMOTEBoost (synthetic minority
over-sampling with AdaBoost) algorithm [16]. SMOTEBoost was built upon over-
sampling approach with AdaBoost algorithm. Galar et al. [17] presented an
ensemble algorithm by evolutionary under-sampling (EUA) approach, called
EUSBoost, to classify highly imbalanced datasets. EUS generated several sub-
datasets using randomly under-sampling technique and obtained a best under-
sampled dataset of the original dataset that cannot be improve further.
RUSBoost, SMOTEBoost, and EUSBoost applied data sampling techniques into the
AdaBoost.M2 algorithm by considering the minority class instances and the
majority class instances. Yen and Lee [18] presented a cluster-based under-
sampling approach to cluster all the training instances (majority class
instances and minority class instances) into some clusters. This approach
selected a suitable number of majority class instances from each cluster by
considering the ratio of the number of majority class instances to the number
of minority class instances in the cluster.

##  III Data Balancing Methods

Researchers have previously proposed various ways of dealing with imbalanced
datasets. The following section describes a few of those methods.

###  III-A Sampling Techniques

####  III-A1 Under-sampling

Under-sampling involves the removal of some majority class instances to result
in a balanced distribution of all classes [9]. However, this can result in the
removal of informative instances from the majority class especially if the
number of instances of the minority class is very small (for example majority
to minority ratio is 100:1). Thus, there will be a huge loss of information
from the majority class instances, leading to a non-optimal classification.
Fig. 2 shows the random under-sampling process to select the majority class
instances randomly.

Figure 2: Random under-sampling (RUS) approach. Here the red dots are selected
instances from the majority class, where black and red dots are representing
all the majority class instances.

####  III-A2 Oversampling

In case of oversampling, we increase the number of minority class instances
and this can be done in different ways. The most common of these methods is
random oversampling, where minority class instances are duplicated until both
class instances are balanced. The issue with this method, however, is that
there is a high probability of over-fitting due to the same instances
occurring multiple times. SMOTE (Synthetic Minority class Oversampling
TechniquE) is another oversampling technique which handles this problem by
creating synthetic instances of the minority class instead of duplicating the
minority class instances [12]. This is done by by interpolating several
minority class instances that lie together. Nevertheless, this problem can be
present to some extent because the newly generated samples could be almost
identical to the existing minority class instances. Modified synthetic
minority oversampling technique (MSMOTE) is a modified version of SMOTE that
divides the instances of the minority class into three groups: safe, border,
and latent noise instances. This is done by the calculation of the distances
among all instances.

Under-sampling method generally works better than oversampling methods as long
as the imbalance ratio of the dataset is not very high [9].

###  III-B Cost-sensitive learning

This method tackles the class imbalance problem by assigning more
misclassification cost to minority class instances for the underlying
classifier. The misclassification cost can be used in the cost function and
the cost function can be optimised by the classifier. The classifier will now
treat both majority and minority classes equally due to their underlying
costs. But in this case, the greatest challenge is in assigning the costs as
they are difficult to be derived from datasets.

###  III-C Ensemble Learning

Ensemble learning methods combine multiple base learners which may be the same
or different. This usually increases the predictive capability of the
individual base classifiers, thus making it adaptive to different datasets.
Ensemble methods find their usage in a wide variety of problems.

####  III-C1 Bagging

Bagging creates multiple sub-sets of the original dataset through sampling
with replacement or without replacement. These sub-sets are used by the base
learners while considering each instance with equal weight. The output of the
individual base learners are considered as votes to determine the final
prediction.

####  III-C2 Boosting

Boosting is similar to bagging in that it combines multiple base learners to
obtain a result based on voting technique. However, it’s differences lie in
that boosting assigns weight to instances according to how hard they are to
classify, thus setting high weight of instances that are difficult to
classify. One base learner contributes to the weight used by the next base
learner. Weights are assigned to the base learners as well according to their
predictive accuracy and this is taken into consideration when classifying a
new test instance. Although boosting was not created specifically for
imbalanced dataset problems, it’s characteristic of assigning weights to
examples that are relatively harder to classify makes it ideal for dealing
with class imbalance problems. It can be somewhat referred to as a cost-
sensitive method.

##  IV CUSBoost Algorithm

CUSBoost is based on the combination of cluster-based sampling and Adaboost
algorithm. It is similar to RUSBoost and SMOTEBoost with the critical
difference occurring in the sampling technique. SMOTEBoost uses SMOTE method
to oversample the minority class instances, while RUSBoost uses random under-
sampling on the majority class. In comparison, our proposed CUSBoost uses
cluster-based sampling from the majority class. CUSBoost separates the
majority and minority class instances from the original dataset and clusters
the majority class instances into $k$ clusters using k-means clustering
algorithm. Here, the parameter $k$ is determined by hyper-parameter
optimisation. After that, random under-sampling is performed on each of the
created clusters by randomly selecting 50% of the instances (but it can be
tuned according to the domain problem or dataset) and removing the rest. As
clustering is used before sampling, theoretically this algorithm will perform
best when the dataset is highly cluster-able. These representative samples are
then combined with the minority class instances to obtain balanced datasets.
Our algorithm’s strength lies in the fact that it considers examples from all
subspaces of the majority class, since k-means clustering puts each instance
in some cluster. Other similar methods often fail in obtaining proper
representatives of the majority class. Fig. 3 shows the proposed cluster-based
under-sampling technique to select the majority class instances.

Figure 3: Cluster-based under-sampling (CUS) approach. Here the red dots are
selected instances from the majority class, where black and red dots are
representing all the majority class instances.

CUSBoost considers a series of decision trees using C4.5 algorithm and
combines the votes of each individual tree to classify new instances.
Initially, each instance is initialised with an equal weight, $\frac{1}{d}$,
where $d$ is the total number of training instances. The weights of instances
are adjusted according to how they were classified. If an instance was
correctly classified then its weight is decreased, or if misclassified then
its weight is increased. The weight of an instance reflects how difficult it
is to classify. To compute the error rate of model $M_{i}$, we sum the weights
of misclassified instances in $D_{i}$ that is shown in Eq. 1. If an instance,
$x_{i}$ is misclassified, then $err(x_{i})$ is one. Otherwise, $err(x_{i})$ is
zero (when $x_{i}$ is correctly classified).

$error(M_{i})=\sum_{i=1}^{d}w_{i}*err(x_{i})$ (1)

If an instance, $x_{i}$ in $i$th iteration is correctly classified, it’s
weight is multiplied by error $(\frac{error(M_{i})}{1-error(M_{i})})$. Then
the weights of all instances (including misclassified instances) are
normalised. To normalise a weight, we multiply it by the sum of old weights,
divided by the sum of new weights. As a result, the weights of misclassified
instances are increased and the weights of correctly classified instances are
decreased. If the error rate of model $M_{i}$ exceeds 0.5 then we abandon
$M_{i}$ and derive a new $M_{i}$ by generating a new sub-data set $D_{i}$. The
CUSBoost algorithm is summarised in Algorithm 1.

Algorithm 1 CUSBoost Algorithm

Input: Imbalanced data, $D$, number of iterations, $k$, and C4.5 decision tree
induction algorithm.
Output: An ensemble model.
Method:

1: initialize weight, $x_{i}\in D$ to $\frac{1}{d}$;

2: for i = 1 to k do

3: create balanced dataset $D_{i}$ with distribution $D$ using cluster-based
under-sampling;

4: derive a tree, $M_{i}$ from $D_{i}$ employing C4.5 algorithm;

5: compute the error rate of $M_{i}$, $error(M_{i})$;

6: if $error(M_{i})$ $\geq$ 0.5 then

7: go back to step 3 and try again;

8: end if

9: for each $x_{i}\in D_{i}$ that correctly classified do

10: multiply weight of $x_{i}$ by $(\frac{error(M_{i})}{1-error(M_{i})})$; //
update weights

11: end for

12: normalise the weight of each instances, $x_{i}$;

13: end for

To use the ensemble to classify instance, $x_{New}$:

1: initialise weight of each class to 0;

2: for i = 1 to k do

3: $w_{i}=log\frac{1-error(M_{i})}{error(M_{i})}$; // weight of the
classifier’s vote

4: $c=M_{i}(x_{New})$; // class prediction by $M_{i}$

5: add $w_{i}$ to weight for class $c$;

6: end for

7: return the class with largest weight;

##  V Experimental Results

In this section, we present the experimental analysis to examine the
performance of our proposed CUSBoost algorithm. We have used datasets from
KEEL-dataset repository [19] with different imbalance ratio. Table I show the
datasets details .

TABLE I: Imbalanced datasets description. No. | Datasets | Instances | Features | Class | Imbalance
---|---|---|---|---|---
|  |  |  | Values | Ratio
1 | pima | 768 | 8 | 2 | 1.87
2 | dermatology | 366 | 34 | 6 | 5.55
3 | segment0 | 2308 | 19 | 2 | 6.02
4 | led7digit | 443 | 7 | 2 | 10.97
5 | abalone9-18 | 731 | 8 | 2 | 16.4
6 | yeast | 1484 | 8 | 10 | 23.15
7 | $poker-9_{v}s_{7}$ | 244 | 10 | 2 | 29.5
8 | kddcup-guess | 1642 | 41 | 2 | 29.98
| passwd_vs_satan |  |  |  |
9 | yeast5 | 1484 | 8 | 2 | 38.73
10 | ecoli | 336 | 7 | 8 | 71.5
11 | abalone19 | 4174 | 8 | 2 | 129.44
12 | Page Blocks | 548 | 10 | 5 | 164
13 | Statlog (Shuttle) | 2175 | 9 | 7 | 853

###  V-A Evaluation Methods

ROC curves can be thought of as representing the family of best decision
boundaries for relative costs of true positive (TP) and false positive (FP).
On an ROC curve the X-axis represents and the Y-axis represents.

$TP={TP\over TP+FN}$ (2)

$FP={FP\over FP+TN}$ (3)

The ideal point on the ROC curve would be (0,100), that is all positive
instances are classified correctly and no negative instances are misclassified
as positive. The line y = x represents the scenario of randomly guessing the
class. Area Under the ROC Curve (AUC) is a useful metric for classifier
performance as it is independent of the decision criterion selected and prior
probabilities.The AUC comparison can establish a dominance relationship
between classifiers. If the ROC curves are intersecting, the total AUC is an
average comparison between models. However, for some specific cost and class
distributions, the classifier having maximum AUC may in fact be suboptimal.
Hence, we also compute the ROC convex hulls, since the points lying on the ROC
convex hull are potentially optimal.

###  V-B Results

In this experiment, we have compared the proposed CUSBoost method with
AdaBoost, RUSboost, and SMOTEBoost methods. Each dataset was validated using
Area Under the ROC Curve (AUC). As for base learner we used C4.5 decision tree
induction in boosting. Keel-dataset repository’s implementation was used for
the AdaBoost, RUSBoost, and SMOTEBoost algorithms [19]. Each of these
experiments are done 5 times with 10 fold cross validation and their mean
scores are shown in Table II.

TABLE II: Average performance of the AdaBoost, RUSboost, SMOTEBoost, and CUSBoost methods on 13 imbalanced datasets. Datasets | AdaBoost | RUSBoost | SOMTEBoost | CUSBoost
---|---|---|---|---
pima | 0.6223 | 0.6376 | 0.6597 | 0.6679
dermatology | 0.9342 | 0.43 | 0.632 | 0.54
segment0 | 0.996 | 0.957 | 0.951 | 0.943
led7digit | 0.8937 | 0.8279 | 0.9373 | 0.9412
abalone9-18 | 0.6934 | 0.7051 | 0.7195 | 0.7243
yeast | 0.7589 | 0.7382 | 0.741 | 0.7603
$poker-9_{v}s_{7}$ | 0.642 | 0.589 | 0.6708 | 0.967
kddcup-guess | 0.8324 | 0.8681 | 0.8513 | 0.8745
passwd_vs_satan |  |  |  |
yeast5 | 0.9231 | 0.887 | 0.9253 | 0.9
ecoli | 0.6354 | 0.517 | 0.6597 | 0.6589
abalone19 | 0.5723 | 0.5923 | 0.5583 | 0.609
Page Blocks | 0.7605 | 0.5975 | 0.8123 | 0.8981
Statlog (Shuttle) | 0.8331 | 0.727 | 0.8445 | 0.8847

From the results we can see that the CUSBoost algorithm performs best most of
the time when the imbalance ratio is in specific range. As the imbalance ratio
gets high CUSBoost starts to outperform all the other methods significantly.
This happens because our method does not focus on making the ratio of majority
and minority class examples $1:1$. So for this reason the sub sampled training
dataset holds a better representation of the majority class while remaining
imbalance itself. RUSBoost shows a performance with high variance especially
with highly imbalance datasets, thus it shows a poor performance if the mean
results are chosen. But, if the best results are chosen from 10 experiments
then RUSBoost outperforms the other methods including the proposed method in
many datasets those are shown in Table III. The best result of proposed method
is usually quite close to the average result thus proving low variance in its
performance .

TABLE III: The best result using AdaBoost, RUSboost, SMOTEBoost, and CUSBoost methods in each dataset is stressed in bold-face. Datasets | AdaBoost | RUSBoost | SOMTEBoost | CUSBoost
---|---|---|---|---
pima | 0.6653 | 0.6796 | 0.667 | 0.6679
dermatology | 0.9 | 0.6935 | 0.632 | 0.54
segment0 | 0.998 | 0.9805 | 0.9642 | 0.9523
led7digit | 0.9157 | 0.886 | 0.9485 | 0.9544
abalone9-18 | 0.707 | 0.805 | 0.7029 | 0.7434
yeast | 0.7839 | 0.801 | 0.7653 | 0.7895
$poker-9_{v}s_{7}$ | 0.657 | 0.7395 | 0.69 | 0.9685
kddcup-guess | 0.8578 | 0.9123 | 0.8602 | 0.8849
passwd_vs_satan |  |  |  |
yeast5 | 0.9324 | 0.9515 | 0.9391 | 0.9126
ecoli | 0.6455 | 0.677 | 0.6753 | 0.6656
abalone19 | 0.5796 | 0.6498 | 0.5876 | 0.6176
Page Blocks | 0.78 | 0.937 | 0.8592 | 0.9007
Statlog (Shuttle) | 0.8567 | 0.8989 | 0.8516 | 0.8967

##  VI Conclusion

Existing classification algorithms generally focus on majority class instances
and ignore the minority class instances. So, it is a challenging task to
construct an effective classifier that can correctly classify the instances of
the minority class. This is ever so pertinent as class imbalance problems
affect a vast range of domains. Recently, computational intelligence
researchers have proposed several hybrid techniques by combining sampling with
ensemble classifiers for dealing with class imbalance problems. The purpose of
this paper is to present a new algorithm called CUSBoost, or Cluster-based
Under-sampling with Boosting, in order to alleviate the problem of class
imbalance. We have compared the performance of CUSBoost algorithm with the
most effective boosting techniques like AdaBoost, RUSBoost, and SMOTEBoost
algorithms. Based on experimental results, we have found that CUSBoost
performed favourably when compared to these popular techniques on datasets
having high class imbalance ratio.

Rather than simply randomly choosing samples from the dataset, CUSBoost first
clusters the majority class instances and then performs random under sampling
so that the boosting algorithm (Adaboost) can use examples from all regions of
the dataset. Thus an advantage it holds over RUSBoost is that the variance of
the results are low, which leads to stable performance. Additionally, the
performance of this algorithm has been shown to be relatively proportional to
the imbalance ratio of the dataset, where higher imbalanced ratio produces
greater performance. It has shown to outperform the other algorithms in a
certain range of imbalanced ratio. In this paper, we have compared the
performance of CUSBooost with RUSBoost, SMOTEBoost and Adaboost. In future
work, we intend to perform extensive experiments to continue investigating the
performance of CUSBoost with other established sampling methods and ensemble
methods.

## References

  * [1] D. M. Farid, M. A. Al-Mamun, B. Manderick, and A. Nowe, “An adaptive rule-based classifier for mining big biological data,” _Expert Systems with Applications_ , vol. 64, pp. 305–316, December 2016.
  * [2] D. M. Farid, L. Zhang, C. M. Rahman, M. Hossain, and R. Strachan, “Hybrid decision tree and naïve bayes classifiers for multi-class classification tasks,” _Expert Systems with Applications_ , vol. 41, no. 4, pp. 1937–1946, March 2014.
  * [3] D. M. Farid, L. Zhang, A. Hossain, C. M. Rahman, R. Strachan, G. Sexton, and K. Dahal, “An adaptive ensemble classifier for mining concept drifting data streams,” _Expert Systems with Applications_ , vol. 40, no. 15, pp. 5895–5906, November 2013.
  * [4] D. M. Farid and C. M. Rahman, “Assigning weights to training instances increases classification accuracy,” _International Journal of Data Mining & Knowledge Management Process_, vol. 3, no. 1, pp. 13–25, January 2013\.
  * [5] H. He and E. A. Garcia, “Learning from imbalanced data,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 21, no. 9, pp. 1263–1284, June 2009.
  * [6] Y. Sun, A. K. C. Wong, and M. S. Kamel, “Classification of imbalance data: A review,” _International Journal of Pattern Recognition and Artificial Intelligence_ , vol. 23, no. 4, pp. 687–719, June 2009.
  * [7] M. Wasikowski and X. wen Chen, “Combating the small sample class imbalance problem using feature selection,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 22, no. 10, pp. 1388–1400, October 2009.
  * [8] Z. Sun, Q. Song, X. Zhu, H. Sun, B. Xu, and Y. Zhou, “A novel ensemble method for classifying imbalanced data,” _Pattern Recognition_ , vol. 48, no. 5, pp. 1623–1637, May 2015.
  * [9] D. M. Farid, A. Nowé, and B. Manderick, “A new data balancing method for classifying multi-class imbalanced genomic data,” _25th Belgian-Dutch Conference on Machine Learning (Benelearn)_ , pp. 1–2, 12-13 September 2016.
  * [10] C. Beyan and R. Fisher, “Classifying imbalanced data sets using similarity based hierarchical decomposition,” _Pattern Recognition_ , vol. 48, no. 5, pp. 1653–1672, May 2015.
  * [11] A. D. Pozzolo, O. Caelen, and G. Bontempi, “When is undersampling effective in unbalanced classification tasks?” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 2015, pp. 200–215.
  * [12] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority over-sampling technique,” _Journal of Artificial Intelligence Research_ , vol. 16, pp. 321–357, June 2002.
  * [13] M. S. Santos, P. H. Abreu, P. J. García-Laencina, A. Simão, and A. Carvalho, “A new cluster-based oversampling method for improving survival prediction of hepatocellular carcinoma patients,” _Journal of Biomedical Informatics_ , vol. 58, pp. 49–59, September 2015.
  * [14] R. Blagus and L. Lusa, “SMOTE for high-dimensional class-imbalanced data,” _BMC Bioinformatics_ , vol. 14, no. 106, pp. 1–16, March 2013.
  * [15] C. Seiffert, T. M. Khoshgoftaar, J. V. Hulse, and A. Napolitano, “Rusboost: A hybrid approach to alleviating class imbalance,” _IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans_ , vol. 40, no. 1, pp. 185–197, January 2010.
  * [16] N. V. Chawla, A. Lazarevic, L. O. Hall, and K. W. Bowyer, “Smoteboost: Improving prediction of the minority class in boosting,” _7th European Conference on Principles and Practice of Knowledge Discovery in Databases_ , pp. 107–109, 22-26 September 2003.
  * [17] M. Galar, A. Fernández, E. Barrenechea, and F. Herrera, “Eusboost: Enhancing ensembles for highly imbalanced data-sets by evolutionary undersampling,” _Pattern Recognition_ , vol. 46, no. 12, pp. 3460–3471, December 2013.
  * [18] S.-J. Yen and Y.-S. Lee, “Cluster-based under-sampling approaches for imbalanced data distributions,” _Expert Systems with Applications_ , vol. 36, no. 3 (Part 1), pp. 5718–5727, April 2009.
  * [19] J. Alcalá-Fdez, A. Fernandez, J. Luengo, J. Derrac, S. García, L. Sánchez, and F. Herrera, “Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework,” _Journal of Multiple-Valued Logic and Soft Computing_ , vol. 17, no. 2-3, pp. 255–287, 2011.