# Mathematical Modeling Insights into Improving CAR T cell Therapy for Solid
Tumors: Antigen Heterogeneity and Bystander Effects

Erdi Kara,, T. L. Jackson, , Chartese Jones, , Reginald L. McGee II, Rockford
Sison Corresponding Author<EMAIL_ADDRESS>of Mathematics,
Spelman CollegeDepartment of Mathematics, University of MichiganDepartment of
Mathematics, University of MissouriDepartment of Mathematics, College of the
Holy CrossDepartment of Mathematics, Spelman College

###### Abstract

As an adoptive cellular therapy, Chimeric Antigen Receptor T-cell (CAR T-cell)
therapy has shown remarkable success in hematological malignancies, but only
limited efficacy against solid tumors. Compared with blood cancers, solid
tumors present a unique set of challenges that ultimately neutralize the
function of CAR T-cells. One such barrier is antigen heterogeneity -
variability in the expression of the antigen on tumor cells. Success of CAR
T-cell therapy in solid tumors is unlikely unless almost all the tumor cells
express the specific antigen that CAR T-cells target. A critical question for
solving the heterogeneity problem is whether CAR T therapy induces bystander
effects, such as antigen spreading. Antigen spreading occurs when CAR T-cells
activate other endogenous antitumor CD8 T cells against antigens that were not
originally targeted. In this work, we develop a mathematical model of CAR
T-cell therapy for solid tumors that takes into consideration both antigen
heterogeneity and bystander effects. Our model is based on in vivo treatment
data that includes a mixture of target antigen-positive and target antigen-
negative tumor cells. We use our model to simulate large cohorts of virtual
patients to gain a better understanding of the relationship between bystander
killing. We also investigate several strategies for enhancing the bystander
effect and thus increasing the overall efficacy of CAR T-cell therapy for
solid tumor.

##  1 Introduction

Chimeric antigen receptor T‐cell (CAR‐T) therapies are one of the most
innovative cancer treatment modalities available today and have radically
altered the treatment landscape for multiple malignancies [1]. Chimeric
antigen receptors (CARs) are engineered fusion proteins that allow T cells to
recognize specific antigens expressed on cancer cells [2]. By removing immune
cells, manipulating their genes, and implanting them back into the body,
scientists have created a way to “teach” our immune system to identify and
destroy cancer cells. This procedure got its first two FDA approvals in 2017
[3]. Since then, four more therapies have been approved[4, 2]. These therapies
have demonstrated repeated success in handling liquid tumors including certain
leukemias, lymphomas, and most recently, myeloma [5]. Given the promising
clinical responses for several hematological malignancies, there is a vested
interest in examining the applicability to solid tumors. Unfortunately, CAR
T-cell therapy has not generated even close to the same level of success in
the fight against solid tumors due to several challenges that are often
interconnected [6]. Some difficulties in applying CAR T-cell therapy to solid
tumors are tumor antigen heterogeneity, trafficking and infiltration into
tumor tissue, and immunosuppressive tumor micro-environment[7]. Of these,
antigen heterogeneity - variability in the expression of the antigen on tumor
cells is particularly challenging [8]. The ideal tumor associated antigen
(TAA) would be expressed on all tumor cells because CARs can only attack cells
having the targeted antigen. Therefore, antigen heterogeneity impairs the
detection of cancer cells by T cells and therapuetic success is unlikely
unless almost all the tumor cells express the TAA [9].

A critical question for solving the heterogeneity problem is can CAR T-cell
therapy induce ”bystander effects” [10, 11]. The term bystander effect refers
to the phenomenon where CAR T-cells not only target cancer cells expressing a
specific antigen but also stimulate cytotoxic activity against nearby cancer
cells that lack the targeted antigen expression. In the same context,
bystander cells refer to immune cells triggered by CAR T-cells playing a role
in the killing of tumor cells. These bystander cells can include CD8 T cells,
innate immune cells (such as macrophages, neutrophils, or natural killer (NK)
cells), or even antigen-presenting cells (APCs) involved in antigen cross-
presentation. The bystander effect allows CAR T-cells to exert their
therapeutic impact beyond the targeted antigen-expressing cells, contributing
to the eradication of heterogeneous cancer cell populations commonly found in
solid tumors. Two types of bystander effects are indirect tumor cell killing
and antigen spreading. Indirect killing could result from activation
macrophages and/or natural killer cells by cytokines released after CAR
engagement. Antigen spreading occurs when CAR T-cells induce the activation of
other endogenous antitumor CD8 T cells against other antigens that were not
originally targeted [11]. It usually occurs following initial therapy-mediated
tumor destruction, which leads to the release of secondary tumor antigens. The
extent to which this occurs will be critical in setting the thresholds of
antigen expression that will be used to define eligible candidates.

Experimentally evaluating the bystander effect in CAR T-cell therapy presents
several challenges. Firstly, relevant experimental models are required to
accurately assess immunologic bystander effects. While many preclinical
studies have utilized immunodeficient mouse models with human tumors,
assessing the bystander effect necessitates intact immune systems in mouse
models. Intact immune system is essential to evaluate the extent to which CAR
T-cells can effectively cure antigen-positive and target antigen-negative
tumor types and thus define the presence and impact of bystander effects.
Secondly, defining and quantifying bystander effects can be technically
challenging. Precise characterization of antigen expression profiles on tumor
cells, designing well-defined mixtures of antigen-positive and antigen-
negative tumor cell populations, and accurately measuring the impact of CAR
T-cells on these populations require sophisticated techniques and
methodologies [12].

Increasingly, mathematical models have been used to study CAR T-cell therapy
[13, 14, 15]. However, to the best of our knowledge, there is no mathematical
model investigating bystander effect of CAR T-cells in solid tumors. In this
paper, we build a mathematical model based on an in-vivo experiment [12] that
includes a mixture of target antigen-positive and target antigen-negative
tumor cells. In our model, we investigate the interaction of four distinct
cell populations; antigen-positive tumor cells, antigen-negative tumor cells,
CAR T-cells and bystander cells. Our results indicate that augmenting the
number of CAR T-cells reaching the tumor site does not enhance the therapeutic
outcome, even in cases of high antigen expressivity, suggesting the need for
bystander effect . However, our virtual patient analysis reveals that
enhancing the cytotoxic capability of bystander cells can generate
significantly improved outcomes.

The plan for the paper is as follows. In Sec-2 we describe the mathematical
model designed to investigate antigen heterogeneity and bystander effects and
explain the key steps of parameter estimation using in-vivo data provided in
[12]. In section Sec-3, we develop a heterogeneous virtual patient population
that reflects the variability in the experimental data to simulate different
treatment scenarios to gain a better understanding about bystander effect on
therapeutic outcome. Finally, in Sec-4, we explore the potential extension of
our model to address various scenarios regarding the bystander phenomenon.

##  2 Model Formation and Calibration

###  2.1 Model Equations

Our mathematical model investigates the evolution of four distinct cell
populations over the course of treatment. Specifically, we consider the total
tumor population as consisting of two sub-populations denoted by $T_{s}$ and
$T_{r}$, which respectively represent the quantities of antigen-positive and
antigen-negative tumor cells, measured in units of $\unit{mm^{3}}$.
Additionally, we denote by $C$ and $B$ the quantities of CAR T-cells and
bystander cells present within the tumor microenvironment , also measured in
units of $\unit{mm^{3}}$.

$\displaystyle\dfrac{dT_{s}}{dt}=r_{1}T_{s}\Big{(}1-\dfrac{T_{s}+T_{r}}{K_{1}}\Big{)}-D_{B}T_{s}-D_{C}T_{s}$
(1)
$\displaystyle\dfrac{dT_{r}}{dt}=r_{2}T_{r}\Big{(}1-\dfrac{T_{s}+T_{r}}{K_{1}}\Big{)}-D_{R}T_{r}$
(2)
$\displaystyle\dfrac{dC}{dt}=v_{C}(t)-\gamma_{C}C-\mu_{C}\log\Big{(}\dfrac{B+C}{K_{2}}\Big{)}\dfrac{D_{C}^{2}}{k+D_{C}^{2}}C-\omega_{C}CT_{s}$
(3)
$\displaystyle\dfrac{dB}{dt}=b-\gamma_{B}B-\mu_{B}\log\Big{(}\dfrac{B+C}{K_{2}}\Big{)}\dfrac{D_{B}^{2}}{k+D_{B}^{2}}B-\omega_{B}B(T_{s}+T_{r})$
(4)

where

$\displaystyle D_{B}$
$\displaystyle=d_{B}\dfrac{(B/T_{s})^{l}}{s+(B/T_{s})^{l}}$ (5a)
$\displaystyle D_{C}$
$\displaystyle=d_{C}\dfrac{(C/T_{s})^{l}}{s+(C/T_{s})^{l}}$ (5b)
$\displaystyle D_{R}$
$\displaystyle=d_{B}\dfrac{(B/T_{r})^{l}}{s+(B/T_{r})^{l}}$ (5c)

In the absence of CAR T-cells and bystander cells, our model accounts for
tumor growth through competition-based dynamics. The first two terms in
equations (1) and (2) respectively represent the logistic growth of the
antigen-positive and antigen-negative tumor sub-populations, each towards a
carrying capacity $K_{1}$ with its own growth rate $r_{1}$ and $r_{2}$.

In Eq.(1), the killing of the antigen-positive tumor population is modeled
through two ratio-dependent cell-kill terms denoted by $D_{B}$ and $D_{C}$.
Those terms were originally proposed by de Pillis et al. in [16] to capture
the interaction between tumor cells and immunological components, including
CD8 positive T cells and natural killer cells. Therein, they discovered that
tumor cell lysis by tumor-specific CD8-T cells is better modelled by using a
term that is a function of the ratio of T cells to tumor cells rather than a
usual linear cell-kill term. We also tested linear cell-kill and confirmed
that it does not capture the dynamics of our data and thus we adopt the ratio-
dependent functional form for cell kill for our model.

Ratio dependent killing terms are given in Eq.(5). The parameter $l$ describes
the degree of dependence of the killing rate on the ratio of CAR T-cells or
bystander cells, while the parameter $s$ characterizes the steepness of
killing of the corresponding terms. The parameters $d_{B}$ and $d_{C}$
represent the maximum killing rate that can be achieved when the
concentrations of bystander and CAR T-cells are at their respective saturation
levels. In Eq.(1), CAR T-cells selectively recognize and target the antigen-
positive tumor cell population, leading to their death. Additionally, the same
antigen-positive cell population can be killed by bystander cells in the
presence of a bystander effect. In Eq.(2), the antigen-negative tumor cell
population is only inhibited by bystander cells, since these cells are not
recognized by CAR T-cells due to the absence of the target antigen.

In the context of the equations (3) and (4), the parameters $\gamma_{C}$ and
$\gamma_{B}$ represent the natural death rates of CAR T-cells and bystander
cells, respectively. It has been observed that CAR-T cells, like T cells
responding to chronic viral infections or certain cancers, can experience a
loss of proliferative capacity due to prolonged exposure to antigen
stimulation. In this sense, the parameter $w_{C}$ represents the rate of
exhaustion of CAR T-cells resulting from exposure to the antigen-positive
tumor population, while $w_{B}$ represents the rate of exhaustion of bystander
cells due to their interaction with both tumor cell populations. We assume
that the recruitment of CAR T-cells and bystander cells follows the Michaelis-
Menten dynamics, as outlined in [17], which are represented by the terms
$\dfrac{D_{C}^{2}}{k+D_{C}^{2}}$ and $\dfrac{D_{B}^{2}}{k+D_{B}^{2}}$,
respectively. Here, the maximum recruitment of CAR T-cells and bystander cells
are denoted by $\mu_{C}$ and $\mu_{B}$, respectively. Note that both terms are
scaled by the immune cell competition term $\log{\dfrac{B+C}{K_{2}}}$, which
limits the proliferation of immune cell components beyond the carrying
capacity $K_{2}$ [18].

CAR T-cell injection $v_{C}(t)$ is modelled as a one-time increase in the
amount of CAR T-cells on the days of injection.

$v_{C}(t)=\begin{cases}I_{C},&\text{if }t=\text{injection day}\\\
0,&\text{otherwise}\end{cases}$

where $I_{C}$ represents the amount of CAR T-cell injected at time day of
treatment.

###  2.2 Parameter Estimation

We provide the parameter values associated with our model in Table-1. To
determine the key parameters in our model, we utilize data generated in
Klampatsa et.al [12] where the bystander effect is investigated in-vivo in a
systematic way.

As defined in the introduction, bystander effects refer to the phenomenon
where CAR T-cells are able to eliminate tumor cells that do not express the
target antigen via a variety of mechanisms including Fas-FasL death receptor
engagement and endogenous CD8 T activation [11]. Klampatsa et.al describes a
mixing model developed using CAR T-cells that react against a human
mesothelin-expressing solid murine tumor cell line, allowing for direct
testing of the bystander hypothesis. Specifically, in their experiment,
$2\times 10^{6}$ human mesothelin (AE17om) were seeded into the right flanks
of the mice and tumor population reached to approximately 50 $\unit{mm^{3}}$
by the end of 3rd day. The treatment scheme is implemented via administration
of two doses of $10^{7}$ transduced M11 CAR T-cells on day 7 and day 9. It was
reported that while M11 CAR T-cells could cure tumors that were 100%
mesothelin positive, they could not cure tumor that were 90%, 75% or 50%
mesothelin positive, indicating the lack of bystander effect. However, they
found that administering low-dose(100 mg/kg) cyclophosphamide (CTX) to the
mice prior to treatment resulted in a bystander effect and the cure of tumor
mixtures. Each of these experiments included control groups, which means that
some subjects were not treated and served as a comparison to the treated
group. We will use their data to sequentially calibrate some of our model
parameters.

In the subsequent analysis, we used free online tool Plot Digitizer to extract
the data provided in the aforementioned publication [19]. In our model, we
work with the amount of cells in mm^3, which is more convenient for our
purposes than the original paper’s use of cell number. To convert between the
two units, we assume that $10^{6}$ cells can be approximated by 1mm^3 of
cells. Parameters that are not discussed below are taken from [20].

Parameter |  Description |  Value |  Units |  Source
---|---|---|---|---
$r_{1}$ |  $T_{s}$ proliferation rate |  0.151 |  day^-1 |  fitted
$r_{2}$ |  $T_{r}$ proliferation rate |  0.180 |  day^-1 |  fitted
$K_{1}$ |  $T_{s}$ and $T_{r}$ carrying capacity |  $5.9\times 10^{3}$ |  mm^3 |  [20]
$l$ |  exponent of tumor lysis by either C or B |  1.291 |  unit-less |  fitted
$\mu_{C}$ |  max recruitment rate of CARTs by antigen-positive tumor lysis |  0.6 |  day^-1 |  fitted
$d_{C}$ |  maximum killing rate of antigen-positive cells via CAR-T cells |  0.27 |  day^-1 |  fitted
$s$ |  steepness of fractional antigen-negative tumor kill by bystanders |  $3.05\times 10^{-1}$ |  unit-less |  [20]
$\gamma_{C}$ |  CARTs death rate |  $2.93\times 10^{-2}$ |  day^-1 |  [20]
$k$ |  steepness of cart/bystander recruitment |  $2.019\times 10^{-7}$ |  day^-2 |  [20]
$\omega_{C}$ |  cart exhaustion due to antigen-positive cells |  $3\times 10^{-5}$ |  mm^-3day^-1 |  [20]
$K_{2}$ |  immune cell carrying capacity |  $1.65\times 10^{3}$ |  mm^3 |  [20]
$d_{B}$ |  maximum killing rate of antigen-positive/antigen-negative cells via bystanders |  0.27 |  day^-1 |  fitted
$\mu_{B}$ |  max recruitment rate of bystanders by antigen-positive tumor lysis |  0.82 |  day^-1 |  fitted
$b$ |  base recruitment rate of bystanders |  $5\times 10^{-2}$ |  mm^3day^-1 |  [20]
$\gamma_{B}$ |  bystander death rate |  $2\times 10^{-2}$ |  day^-1 |  [20]
$\omega_{B}$ |  bystander exhaustion due to antigen-positive/antigen-negative cells |  $3.42\times 10^{-6}$ |  mm^3day^-1 |  [20]
Table 1: Baseline parameters.

Several studies have shown that CAR T-cells face several obstacles in
penetrating solid tumors and reaching their intended targets within the tumor
site [21]. Thus, we assume that (after converting to mm^3), only 1% of the CAR
T-cells reaches the target tumor and set $I_{C}=0.1$ based on this discussion.
At the parameter fitting stage, which is described in the subsequent
paragraphs, initial conditions are set as $T_{s}(0)=50p$, $T_{r}(0)=50(1-p)$,
$C(0)=0$ and $B(0)=0.1$ where $p$ is the percentage of the antigen-positive
tumor cell population. All units are given in mm^3.

The control group associated with 100% mesothelin positive case allows us to
calibrate the growth rate $r_{1}$ of antigen-positive tumor cell population in
Eq.(1). We then fix this value and calibrate the growth rate $r_{2}$ of
antigen-negative tumor cell population using the control data associated with
90% and 75% mesothelin positive cases. Through our analysis, we determined
that $r_{1}=0.151$ and $r_{2}=0.180$. Our analysis indicates that, in this
case, the tumor population lacking CART antigen expression exhibits a
significantly higher growth rate compared to the population expressing the
CART antigen, which also aligns with the experimental data plotted in Figure 1
of [12].

The good agreement with the model and the data can be seen in Fig-1. Note that
only first two ODE terms in Eq-(1) and Eq-(2) are active in case of control
data since there are no treatment and bystander effect present.

Figure 1: The image depicts a comparison between the model fit and the actual
data in the control group, with the graph representing 100%, 90%, and 75%
antigen-positive cases on the left, middle, and right, respectively. The graph
demonstrates a good agreement between the model fit and the observed data,
indicating that the model accurately represents the behavior of the control
group across varying levels of antigen positivity.

To calibrate the parameters associated with CAR T-cell treatment without
bystander effect, we use the results of the two-dose CAR T-cell regimen. In
this way, we calibrated the exponent of tumor lysis $l$, maximum lysis rate of
antigen-positive tumor cells due to CARTs $d_{C}$ and the maximum recruitment
of CAR T-cells $\mu_{C}$. The results can be observed in Fig-2.

Figure 2: Comparison of model fit and data for CAR T-cell treatment. The
figure on the left demonstrates the complete tumor elimination for 100%
antigen positive case. The figure in the middle and figure on the right are
90% and 75% antigen positive cells. We observe the absence of bystander effect
in both cases. Although CAR T-cell treatment could not slowed tumor growth
temporarily, it could not cure the tumor progression. 75%(left), 90%(middle)
and 75%(right) antigen positive cases.

As mentioned above, CTX administered as a single dose(100mg/kg) a day before
the CAR T-cell treatment can induce bystander effect leading to tumor cure.
This data allows us to calibrate maximum lysis rate of tumor cells due to
bystander cells $d_{B}$ and the maximum recruitment of bystander cells
$\mu_{B}$. Fig-3 displays these results.

Figure 3: Comparison of model fit and data for CTX induced bystander effect on
experiments conducted on 90% and 75% antigen positive tumor types.
Administration of CTX along with M11 CAR T-cells cured the tumors in both
cases proving the bystander effect.

##  3 Results

###  3.1 Tumor Response to Therapy is Largely Independent of CAR T-Cell Dose

In Sec-2.2, we demonstrated that our model is good agreement with the
experimental data. We will now simulate different scenarios to gain a better
understanding about bystander effect of CAR T-cells.

In Figure-4, we present the results of a study evaluating the therapeutic
responses to a range of CAR T-cell dosages and levels of antigen positivity
over a period of 45 days. Our analysis involved classifying the response of
each tumor based on whether the final tumor burden was less than the initial
seed size of the solid tumor (2 mm^3) prescribed in [12]. If the final tumor
population is below this threshold, we label it as a responder(R), denoted by
orange markers, otherwise we label the tumor as a non-responder(NR) and
denoted by blue markers. Our findings indicate that simply increasing the
dosage of CAR T-cells does not guarantee a favorable therapeutic outcome.
Specifically, in cases where the levels of CAR T-cell antigen expression are
approximately 55% or lower, augmenting the quantity of injected CAR T-cells
does not result in a responsive treatment.

Figure 4: Therapeutic responses for varying cart cell dosages and antigen
positive percentages. We can observe that increasing the dosage of CAR T-cells
does not guarantee a favorable therapeutic outcome. Specifically, in cases
where the levels of CAR T-cell antigen expression are approximately 55% or
lower, augmenting the quantity of injected CAR T-cells does not result in a
responsive treatment. We can also observe that the minimum dose required to
elicit a response is around 0.05 mm^3, which is below the baseline dose 0.1
mm^3 we utilized in the model.

###  3.2 Virtual Patient Analysis Predicts Tumor Response Patterns

In this section, we generate a virtual patient (VP) cohort to ensure the
exclusion of that reflects the variability in the data to examine the impact
of the variability on treatment outcomes. Utilizing virtual patients allow us
to conduct statistical analysis and draw conclusions on the effectiveness of
various treatments or treatment combinations. We apply the treatment regimen
described above to each VP in the cohort and identify any patterns or trends
that may contribute to successful therapeutic outcome.

To generate virtual patients (VPs), we begin by considering the parameters
$r_{1}$, $r_{2}$, and $K_{1}$, and generate values for each parameter such
that they fall within a 0.5 relative error band of each data point with
respect to the control data for the 100%, 90%, and 75% cases. By imposing this
acceptance criteria we aim to obtain a set of parameters, $r_{1}$, $r_{2}$,
and $K_{1}$, that capture the dynamics in the experimental data. In total, we
generated 6433 such values using this methodology, as illustrated in Figure-5.
Parameters of interests fall in the following ranges;
$r1\in[0.11,0.16],r2\in[0.14,0.22],K1\in[2\times 10^{3},7\times 10^{3}]$.

Figure 5: Comparison of VPs and the control data. Trajectories of 6433 control
VPs are illustrated in region while the control data regarding 75%,90% and
100% antigen-positive tumor population are represented with black marker. We
can observe that the generated VPs cohort is a good representative of the
actual experimental data.

In order to establish a VP cohort that encompasses all 16 variables, we fix
the values of $r_{1},r_{2},K_{1}$ obtained above, and for each of the control
VP, we generate the remaining 13 parameters by sampling from a uniform
distribution within the following parameter ranges.

$\displaystyle l$ $\displaystyle\in[0.5,1.8],\quad
d_{C}\in[0,1],\quad\mu_{C}\in[0,1],\quad s\in[10^{-1},5\times
10^{-1}],\quad\gamma_{C}\in[10^{-2},5\times 10^{-2}],$ $\displaystyle\quad k$
$\displaystyle\in[10^{-7},3\times 10^{-7}],\quad w_{C}\in[10^{-5},5\times
10^{-5}],\quad K_{2}\in[2\times 10^{3},7\times 10^{3}],\quad
d_{B}\in[0,1],\quad\mu_{B}\in[0,1],$ $\displaystyle b$
$\displaystyle\in[10^{-2},10^{-1}],\quad\gamma_{B}\in[10^{-2},10^{-1}],\quad
w_{B}\in[10^{-6},10^{-5}].$

We will then submit these VPs to standard treatment regimen and investigate
which patients responds to the treatment. We also fix the antigen positive
percentage to $75\%$. As previously mentioned, the initial seed size of the
solid tumor for all experiments in [12] was 2 mm^3, and the CAR T-cell
treatment was initiated when the tumor size reached 50 mm^3. Thus, we will use
the following criteria to classify the outcome of the treatment for each VP.

$response=\begin{cases}R,&T_{45}\leq 2\unit{mm^{3}}\\\ PR,&2\unit{mm^{3}}\leq
T_{45}\leq 50\unit{mm^{3}}\\\ NR,&T_{45}\geq 50\unit{mm^{3}}\end{cases}$

where R, PR and NR represent response, partial-response and no-response,
respectively. We display the trajectory of each VP in Fig-6 where blue, green
and magenta trajectories represent R, PR and NR categories, respectively.
$T_{45}$ is the size of the final tumor population at day $t=45$, given in
$\unit{mm^{3}}$. Out of 6433 VPs, the response distribution in terms of
percentage occurs as follows

$\mbox{R}:53\%,\mbox{PR}:11\%,\mbox{NR}:36\%$

Figure 6: Trajectories of 6433 VPs are illustrated based on the final
therapeutic outcome. Blue, green and magenta trajectories represent R,PR and
NR,respectively. Out of 6433 VPs, the response distribution in terms of
percentage occurs as 53% R, 11 PR and 36% NR.

###  3.3 Enhancing Bystander Cell Cytotoxicity Increases Survival Rate

We can now assess the statistical significance between these three groups and
determine which parameters have a greater impact on the treatment outcome. One
way of achieving this task would be to inspect the empirical cumulative
distribution function(ECDF) of each parameter across these three classes. ECDF
is constructed by ordering the data points from smallest to largest and
plotting the proportion of data points that are less than or equal to each
value. The resulting curve shows how the data is distributed, with a steeper
curve indicating a higher concentration of data points at lower values. We use
ECDF because it is non-parametric, meaning the method does not make any
assumptions about the underlying distribution of the data, which matches our
VP set-up. Through empirical cumulative distribution function (ECDF), we
observe that the distributions of $d_{B}$, $d_{C}$, and $l$ are significantly
different across the three classes, R, PR, and NR, as seen in Fig-7. For
example, the left figure associated with $d_{B}$, maximum killing rate of
bystander cells, shows that responders are strongly associated with higher
$d_{B}$ values while this case is the opposite for non-responders.

Figure 7: Empirical cumulative distribution plots for $d_{B}$(left),
$d_{C}$(middle) and $l$(right). We observe that the distributions of these
parameters are significantly different across the three classes, R, PR, and
NR. For example, the left figure associated with $d_{B}$ shows that responders
are strongly associated with higher $d_{B}$ values while this case is the
opposite for non-responders.

To gain a deeper understanding, we can also visualize box plots that show the
distribution of values of the corresponding parameter within each class. The
box plots reveal that the distribution of $d_{B}$ values is noticeably
different across the three classes. Specifically, the box plot for the
responder class shows a higher median and a larger spread of values compared
to the other two classes, indicating that the $d_{B}$ values tend to
accumulate towards the upper range for this class. In contrast, the box plot
for the non-responder class shows a lower median and a smaller spread of
values, indicating that $d_{B}$ values for this class tend to be lower and are
more tightly clustered towards the lower range. These differences suggest that
$d_{B}$ may be a key parameter in distinguishing between the R and NR classes.
Similarly, we observe that the $l$ values for the non-responders tend to be
higher compared to the other two classes.

Figure 8: Distribution profile of $d_{B}$(bottom left), $l$(bottom right) and
$d_{C}$(middle) for the categories of interests. Notice that the box plot for
the responder class regarding $d_{B}$ shows a higher median and a larger
spread of values compared to the other two classes, indicating that responders
are associated with higher $d_{B}$ values. This suggest that higher maximum
killing rates may result in an improved therapeutic outcome.

Recall that $d_{B}$ represents the maximum killing rate of bystander cells
once they are activated. Through our VP cohort, we can investigate whether a
higher maximum killing rate would result in an improved therapeutic outcome.
To achieve this, we systematically increased the value of $d_{B}$ by varying
percentages while keeping all other parameters constant. We then submitted
each virtual patient to the same standard therapy described above. As depicted
in Figure-9, each incremental increase in $d_{B}$ resulted in a higher
proportion of responders compared to the baseline scenario. Based on this
observation, our model suggests that maximizing the bystander effect may be a
more effective approach to achieving successful outcomes in CAR T-cell therapy
compared to solely increasing the number of CAR T-cells.

Figure 9: Each incremental increase in $d_{B}$ results in a higher proportion
of responders compared to the baseline scenario. This suggests that enhancing
the cytotoxic capacity of bystander cells is is a crucial aspect to maximize
the bystander effect.

##  4 Discussion

In this study, we developed a mathematical model to investigate the bystander
effect in CAR T-cell therapy for solid tumors. Our model exhibits a remarkably
accurate fit to the unconventional data obtained from an in-vivo experiment.
To the best of our knowledge, this is the first mathematical investigation
specifically focused on the bystander effect in CAR T-cell treatment. The
findings of our study are consistent with key observations made by [12]. In
their study, the authors observed that the bystander effect was not influenced
by the augmented persistence of administered CAR T-cells. Instead, they
provided evidence demonstrating that the presence of endogenous CD8 T cells
was a necessary factor for the occurrence of the bystander effect. Our results
also indicate that augmenting the number of CAR T-cells reaching the tumor
site does not enhance the therapeutic outcome, even in cases of high antigen
expressivity, suggesting the absence of bystander effect. However, our model
further reveals that enhancing the cytotoxic capability of bystander cells
yield significantly improved outcomes, as revealed through virtual patient
analysis.

It is important to highlight that our model incorporates components from well-
established works cited in the model development section. In particular, our
modeling approach aligns with a similar philosophy with [20], which explores
the impact of CAR T-cell therapy in the presence of various chemotherapy
regimens, primarily for hematologic malignancies. The key variables in this
context are tumor cells, CAR T-cells, effector T-cells, and chemotherapy
drugs. In our work, we break down the tumor population based on whether they
can be targeted by CAR T-cells. Moreover, our model is primarily applicable to
solid tumors since a mixing model is feasible only for solid tumors. Here,
mixing refers to the experiment in which a mixture of tumor cells with
different antigen expression profiles is used. Also, we do not specifically
model drug infusion in our model due to the the nature of bystander experiment
discussed in [12]. The authors therein did not administer a high dose of CTX
since the goal was to investigate the effect of low-dose CTX on bystander
effect, and determine if this dose could enhance the anti-tumor immune
response without causing anti-tumor effects or lymphodepletion which
essentially invalidates the observablity of bystander effect.

We would like to emphasize that our model is driven by the experimentally
available data. Therefore, one natural limitation of the current model is that
we can investigate the bystander phenomena primarily from the perspective of
CAR T-cells and bystander CD8 T-cells. However, given the good agreement with
the experimental data, our model offers a foundation for further improvements
and provides valuable insights into existing theories or the development of
new hypotheses regarding this phenomenon. It is important to note that the
study of bystander phenomena remains an active research area with ongoing
advancements. Building upon the existing research, below we outline several
potential research questions that can be explored within the same modeling
framework. Interested readers can refer to [12] for a more comprehensive
discussion.

It is clear that the bystander effect requires the presence of CD8 positive T
cells. However, it is not clear exactly by which mechanism those effector T
cells are activated. In this sense, one can investigate the role of CD4 helper
T-cells or regulatory T cells(Tregs). In the context of the data we use, the
latter is of special importance. Tregs are a specialized subset of T cells
that play a crucial role in immune regulation and maintaining immune
tolerance. In the context of cancer immunotherapy, Tregs can have a
suppressive effect on the immune response against tumors [22]. This
suppression occurs through various mechanisms, including the inhibition of
anti-tumor responses by endogenous CD8 T cells. On the other hand, CTX has
been shown to have the ability to reduce the population or function of Tregs
[23, 24]. By reducing the number of Tregs, CTX can potentially alleviate their
suppressive effects on immune responses, particularly those directed against
tumors. This pathway can be explored via mathematical modelling by
incorporating Tregs and CTX into the existing framework.

Cytokines produced by CAR T-cells are another direction that can be explored.
Upon activation, CAR T-cells can release cytokines such as IFN-gamma (IFNg)
and tumor necrosis factor-alpha (TNF-a) [25, 26]. These cytokines have the
potential to directly kill tumor cells. Additionally, the released cytokines
can activate the components of innate immune system, including macrophages,
neutrophils, or natural killer (NK) cells. Once activated, these immune cells
can induce tumor destruction through mechanisms such as phagocytosis. Although
a fully coupled modelling scenario is possible, the terms related to cytokines
and innate immune system can be potentially decoupled from the other model
variables and directly linked to CAR-T dynamics since this pathway is
primarily triggered by CAR-T activation.

Exploring the concept of cross-presentation, also known as T-cell cross
priming, could also be a promising line of investigation. This refers to the
process by which antigen-presenting cells, particularly dendritic cells (DCs),
present exogenous antigens(antigens that originate from outside the body) to
CD8 T cells [10]. DCs are immune cells that are crucial for orchestrating
immune responses. They have a pivotal role in determining whether CD8 T cells,
which are important for anti-tumor immune responses, will mount an immune
response against tumor antigens or exhibit tolerance. DCs can present antigens
from engulfed or captured tumor cells to CD8 T cells leading to the activation
of CD8 T cells, enabling them to recognize and target tumor cells. Thus, a CAR
T-cell therapy combined with a strong activation of DCs could potentially
amplify the bystander effects, where the activated CD8 T cells also target
neighboring tumor cells that do not express the specific antigen recognized by
CAR T-cell. Reader can refer to [27] for similar experimental study. Similar
to previous modelling scenario, DC population, in addition to agents
activating DCs, can be linked to bystander cells to create a similar
framework.

So far, our emphasis regarding the bystander dynamics has been based on ODE
modelling. However, one can also consider PDE modelling to account for spatial
dynamics as well. In fact, PDE modelling can be useful for several reasons.
Firstly, bystander effect often involves interactions and diffusion of
signaling molecules, cytokines, and other mediators within the tumor
microenvironment. Thus, PDE models can capture the spatial distribution and
diffusion processes more accurately, allowing for a better representation of
the complex dynamics occurring in the tumor. Moreover, as outlined in the
introduction, solid tumors are characterized by spatial and cellular
heterogeneity, with different regions exhibiting distinct characteristics. PDE
models can incorporate spatial variations in cell density, antigen expression,
and other factors relevant to the bystander effect. For example, in the
context of our study, distribution of antigen-positive and negative tumor
cells withing the solid tumor bulk would likely have a major impact on the
efficacy of the CAR T-cell treatment as well as emergence of bystander
effects. However, it is widely recognized that obtaining spatial data is often
more challenging than acquiring temporal data in biological and clinical
setting. This challenge becomes particularly pronounced when investigating the
bystander effect, as outlined in the introduction, due to the complexities
involved in setting up and conducting experiments in spatially resolved
systems. Consequently, PDE modeling, which incorporates spatial dynamics, may
necessitate a more ”mechanistic model” approach rather than relying on data-
driven modeling, as exemplified by our ODE model.

Upon the publication of our study, the codes, datasets, and major results of
this study will be made publicly available in our GitHub repository.

## References

  * [1] Robert C Sterner and Rosalie M Sterner.  Car-t cell therapy: current limitations and potential strategies.  Blood cancer journal, 11(4):69, 2021.
  * [2] Kathryn M Cappell and James N Kochenderfer.  Long-term outcomes following car t cell therapy: What we know so far.  Nature Reviews Clinical Oncology, pages 1–13, 2023.
  * [3] Peter Braendstrup, Bruce L. Levine, and Marco Ruella.  The long road to the first fda-approved gene therapy: chimeric antigen receptor t cells targeting cd19.  Cytotherapy, 22(2):57–69, 2020.
  * [4] Approved cellular and gene therapy products.  https://www.fda.gov/vaccines-blood-biologics/cellular-gene-therapy-products/approved-cellular-and-gene-therapy-products.  Accessed: 2023-05-01.
  * [5] Pouya Safarzadeh Kozani, Pooria Safarzadeh Kozani, and Fatemeh Rahbarizadeh.  Car-t cell therapy in t-cell malignancies: Is success a low-hanging fruit?  Stem cell research & therapy, 12(1):1–17, 2021.
  * [6] Pouya Safarzadeh Kozani, Pooria Safarzadeh Kozani, Milad Ahmadi Najafabadi, Fatemeh Yousefi, Seyed Mohamad Javad Mirarefin, and Fatemeh Rahbarizadeh.  Recent advances in solid tumor car-t cell therapy: driving tumor cells from hero to zero?  Frontiers in Immunology, 13:1947, 2022.
  * [7] Faroogh Marofi, Roza Motavalli, Vladimir Safonov, Lakshmi Thangavelu, Alexei Valerievich Yumashev, Markov Alexander, Navid Shomali, Navid Shomali, Max Stanley Chartrand, Yashwant Pathak, Mostafa Jarahian, Mostafa Jarahian, Sepideh Izadi, Sepideh Izadi, Sepideh Izadi, Ali Hassanzadeh, Ali Hassanzadeh, Ali Hassanzadeh, Naghmeh Shirafkan, Naghmeh Shirafkan, Safa Tahmasebi, Safa Tahmasebi, and Farhad Motavalli Khiavi.  Car t cells in solid tumors: challenges and opportunities.  Stem Cell Research & Therapy, 2021.
  * [8] Qingzhu Jia, Aoyun Wang, Yixiao Yuan, Bo Zhu, and Haixia Long.  Heterogeneity of the tumor immune microenvironment and its clinical relevance.  Experimental Hematology & Oncology, 11(1):24, 2022.
  * [9] Faroogh Marofi, Roza Motavalli, Vladimir A Safonov, Lakshmi Thangavelu, Alexei Valerievich Yumashev, Markov Alexander, Navid Shomali, Max Stanley Chartrand, Yashwant Pathak, Mostafa Jarahian, et al.  Car t cells in solid tumors: challenges and opportunities.  Stem cell research & therapy, 12(1):1–16, 2021.
  * [10] AR Sánchez-Paulete, A Teijeira, Francisco J Cueto, Saray Garasa, José L Pérez-Gracia, A Sánchez-Arráez, David Sancho, and Ignacio Melero.  Antigen cross-presentation and t-cell cross-priming in cancer immunology and immunotherapy.  Annals of Oncology, 28:xii44–xii55, 2017.
  * [11] Ranjan Upadhyay, Jonathan A Boiarsky, Gvantsa Pantsulaia, Judit Svensson-Arvelund, Matthew J Lin, Aleksandra Wroblewska, Sherry Bhalla, Nathalie Scholler, Adrian Bot, John M Rossi, et al.  A critical role for fas-mediated off-target tumor killing in t-cell immunotherapyfas mediates bystander tumor killing by t cells.  Cancer discovery, 11(3):599–613, 2021.
  * [12] Astero Klampatsa, Michael S Leibowitz, Jing Sun, Maria Liousia, Evguenia Arguiri, and Steven M Albelda.  Analysis and augmentation of the immunologic bystander effects of car t cell therapy in a syngeneic mouse cancer model.  Molecular Therapy-Oncolytics, 18:360–371, 2020.
  * [13] Prativa Sahoo, Xin Yang, Daniel Abler, Davide Maestrini, Vikram Adhikarla, David Frankhouser, Heyrim Cho, Vanessa Machuca, Dongrui Wang, Michael Barish, Margarita Gutova, Sergio Branciamore, Christine E. Brown, and Russell C. Rockne.  Mathematical deconvolution of car t-cell proliferation and exhaustion from real-time killing assay data.  J. R. Soc. Interface., 17, 2020.
  * [14] Odelaisy León-Triana, Soukaina Sabir, Gabriel F. Calvo, Juan Belmonte-Beitia, Salvador Chulián, Álvaro Martínez-Rubio, María Rosa, Antonio Pérez-Martínez, Manuel Ramirez-Orellana, and Víctor M. Pérez-García.  Car t cell therapy in b-cell acute lymphoblastic leukaemia: Insights from mathematical models.  Communications in Nonlinear Science and Numerical Simulation, 94:105570, 2021.
  * [15] LRC Barros, EA Paixão, AMP Valli, GT Naozuka, AC Fassoni, and RC Almeida.  Cartmath—a mathematical model of car-t immunotherapy in preclinical studies of hematological cancers.  Cancers, 13(12), 2021.
  * [16] Lisette G de Pillis, Ami E Radunskaya, and Charles L Wiseman.  A validated mathematical model of cell-mediated immune response to tumor growth.  Cancer research, 65(17):7950–7958, 2005.
  * [17] Vladimir A Kuznetsov, Iliya A Makalkin, Mark A Taylor, and Alan S Perelson.  Nonlinear dynamics of immunogenic tumors: parameter estimation and global bifurcation analysis.  Bulletin of mathematical biology, 56(2):295–321, 1994.
  * [18] Gregory J Kimmel, Frederick L Locke, and Philipp M Altrock.  Response to car t cell therapy can be explained by ecological cell dynamics and stochastic extinction events.  bioRxiv, page 717074, 2019.
  * [19] Joseph A. Huwaldt.  Plot digitizer, 2016.
  * [20] Katherine Owens and Ivana Bozic.  Modeling car t-cell therapy with patient preconditioning.  Bulletin of Mathematical Biology, 83:1–36, 2021.
  * [21] Marina Martinez and Edmund Kyung Moon.  Car t cells for solid tumors: new strategies for finding, infiltrating, and surviving in the tumor microenvironment.  Frontiers in immunology, 10:128, 2019.
  * [22] Theresa L Whiteside.  The role of regulatory t cells in cancer immunology.  ImmunoTargets and therapy, pages 159–171, 2015.
  * [23] Takeshi Nakahara, Hiroshi Uchi, Alexander M Lesokhin, Francesca Avogadri, Gabrielle A Rizzuto, Daniel Hirschhorn-Cymerman, Katherine S Panageas, Taha Merghoub, Jedd D Wolchok, and Alan N Houghton.  Cyclophosphamide enhances immunity by modulating the balance of dendritic cell subsets in lymphoid organs.  Blood, The Journal of the American Society of Hematology, 115(22):4384–4392, 2010.
  * [24] William Humphries, Jun Wei, John H Sampson, and Amy B Heimberger.  The role of tregs in glioma-mediated immunosuppression: potential target for intervention.  Neurosurgery Clinics, 21(1):125–137, 2010.
  * [25] Caio Raony Farina Silveira, Amanda Cristina Corveloni, Sâmia Rigotto Caruso, Nathália Araújo Macêdo, Natália Moscheta Brussolo, Felipe Haddad, Taisa Risque Fernandes, Pamela Viani de Andrade, Maristela Delgado Orellana, and Renato Luiz Guerino-Cunha.  Cytokines as an important player in the context of car-t cell therapy for cancer: Their role in tumor immunomodulation, manufacture, and clinical implications.  Frontiers in Immunology, page 5244, 2022.
  * [26] Maria Cosenza, Stefano Sacchi, and Samantha Pozzi.  Cytokine release syndrome associated with t-cell-based therapies for hematological malignancies: pathophysiology, clinical presentation, and treatment.  International Journal of Molecular Sciences, 22(14):7652, 2021.
  * [27] Junyun Lai, Sherly Mardiana, Imran G House, Kevin Sek, Melissa A Henderson, Lauren Giuffrida, Amanda XY Chen, Kirsten L Todd, Emma V Petley, Jack D Chan, et al.  Adoptive cellular therapy with t cells expressing the dendritic cell growth factor flt3l drives epitope spreading and antitumor immunity.  Nature immunology, 21(8):914–926, 2020.


# Topic and Sentiment Analysis on OSNs:
a Case Study of Advertising Strategies on Twitter

Shana Dacres  Hamed Haddadi  Matthew Purver

Cognitive Science Research Group  School of Electronic Engineering and
Computer Science  Queen Mary University of London, UK
<EMAIL_ADDRESS>
###### Abstract

Social media have substantially altered the way brands and businesses
advertise: Online Social Networks provide brands with more versatile and
dynamic channels for advertisement than traditional media (e.g., TV and
radio). Levels of engagement in such media are usually measured in terms of
content adoption (e.g., _likes_ and _retweets_) and sentiment, around a given
topic. However, sentiment analysis and topic identification are both non-
trivial tasks.

In this paper, using data collected from Twitter as a case study, we analyze
how engagement and sentiment in promoted content spread over a 10-day period.
We find that promoted tweets lead to higher positive sentiment than promoted
trends; although promoted trends pay off in response volume. We observe that
levels of engagement for the brand and promoted content are highest on the
first day of the campaign, and fall considerably thereafter. However, we show
that these insights depend on the use of robust machine learning and natural
language processing techniques to gather focused, relevant datasets, and to
accurately gauge sentiment, rather than relying on the simple keyword- or
frequency-based metrics sometimes used in social media research.

###### category:

H.3.1 Information Storage and Retrieval Content Analysis and Indexing

###### keywords:

Linguistic processing

###### category:

I.2.7 Artificial Intelligence Natural Language Processing

###### keywords:

Text analysis

###### keywords:

Sentiment analysis, Topic analysis, Social Media, Online Advertising, User
Engagement

††terms: Algorithms, Measurement

##  1 Introduction

Online Social Networks (OSNs) such as Facebook, Twitter, and YouTube have
emerged as highly engaging marketing and influence tools, increasingly used by
advertisers to promote brand awareness and catalyze word-of-mouth marketing.
Researchers have also long recognised the effectiveness of OSNs as a rich
source for understanding the spread of information about the real world [20].
For example, Asur et al. [1] analyzed Twitter messages (_tweets_) to predict
box-office ratings for newly released movies. Their findings shows that OSNs
can be used to make quantitative predictions that outperform those of markets
forecasts, by focusing on the sentiment expressed in the tweets. Brands also
now recognise the potential of OSNs for gathering market intelligence and
insight. In 2012, Twitter announced that 79% of people follow brands to get
exclusive content.111http://advertising.twitter.com/2012/05/twitter4brands-
event-in-nyc.html This provides the opportunity for brands to participate in
real-time conversations to listen to and engage users, respond to complaints
and feedback, drive consumer action and broadcast content. Understanding the
real engagement of the end users with the brands and their OSN presence has
given rise to a number of data analytics, sentiment analysis and social media
optimisation startups and academic research projects. However, the techniques
required pose a number of challenges and pitfalls often ignored by researchers
and analysts, and adopting a particular method naively can lead to problems.
Significant progress in Natural Language Processing (NLP) and Machine Learning
(ML) has produced models for topic modelling designed for social media [23],
and high accuracies in sentiment detection (e.g. [26]), even with the
possibility of detecting sarcasm [11]; but care still needs to be taken while
using and relying on the relevant tools and techniques straight out of the box
[8].

In this study we present a focused case study by examining the content and
volume of users’ brand engagement on OSNs to determine the effect of choice of
promotion channel on a brand’s influence.222In this work, engagement is
defined as adoption of the content by e.g., replying to a tweet, mentioning
the brand name, or including the hashtag in a tweet. We are not able to
measure external engagement such as sharing content on other OSNs, or clicking
on the links in the tweet. We do this by analysing the engagement level of
Twitter users, their adoption of brand hashtags, and the sentiment they
express, to determine the similarities and differences between two separate
advertising strategies on this network: promoted tweets, and promoted trends.
We pose a number of questions regarding brands and advertising on OSNs: How
does the sentiment for a promotion strategy spread over time? What are the
engagement levels for each day of promotion? What is the engagement level
(e.g. _retweets_ and _mentions_) for promoted brands and how do these affect
the sentiments expressed towards a brand?

In order to answer these questions, we use Twitter’s Streaming API service to
collect engaged users’ profiles and tweets in regards to promoted influences
(tweets and trends) over a busy 10 day shopping period for a selection of
brands across different industries. We observe the need to accurately filter
the resulting tweets for topical relevance, and compare simple keyword-based
methods with a discrimative Machine Learning (ML) approach. We then classify
the tweets by sentiment (positive, negative or neutral), and again compare a
range of existing methods and tools. We then use this data to establish the
driving factors behind the success of promoted influences and differences
between advertising strategies. For both tasks, the choice of classification
method makes a significant difference, highlighting the care that must be
taken when choosing techniques for this kind of analysis.

The rest of the paper is organized as follows: In Section 2 we present the
some recent related studies. In Section 3 we describe our case study, dataset
and its characteristics. In Section 4 we briefly discuss our sentiment
analysis and text classification methodology and the challenges which only
become apparent upon thorough manual inspection of the data. Section 5
presents our results and the insights gained from our analysis. We conclude
the paper and present potential future directions in sentiment and content
analysis in Section 6.

##  2 Related Work

### Influence on OSNs

Our primary interest in this work is in understanding the factors which govern
the effectiveness and influence of campaigns on OSNs. Several recent studies
have examined individuals’ influence on OSNs [3], and the effectiveness of
online advertising [4, 2], but little attention has been paid to identifying
the driving factors behind a brand’s influence on their social audience
(although it has been noted that brand names are more important online for
some categories [6]). Cheung et al. [5] examined the way information spreads
differently within social networks as opposed to word-of-mouth (WOM)
broadcasting, by focusing on electronic word-of-mouth (eWOM), showing
comprehensiveness and relevance to be the key influences of information
adoption. The closest work to ours in understanding brands on Twitter is the
study by Jansen et al. [10], who found that 20% of tweets that mentioned a
brand expressed a sentiment or opinion concerning that company, product or
service. Here, we examine and compare such mentions and sentiments across
different promotion strategies available to brands on Twitter, thus
specifically investigating advertising effectiveness (see Section 3).333Data
availability limits us to effects within OSNs; we cannot determine effects on
actual clicks or sales.

In a study on the spread of hashtags within Twitter, Romero et al. [24] used
over 3 billion tweets 2009-2010 to analyze sources of variation in how the
most widely used hashtags spread within its user population. Their results
suggested that the mechanism that controls the spread of hashtags related to
sports or politics tends to be more persistent than average; repeated exposure
to users who use these hashtags affects the probability that a person will
eventually use the hashtag more positively than average. However, they only
examined hashtags that succeeded in reaching a large number of users. In
regards to the focus of promoted influences within Twitter, this raises the
question; what distinguishes a promoted item that spreads widely, possibly
with positive sentiment, from one that fails to attract attention or is
associated with mainly negative sentiment? Our study aims to answer this by
examining the sentiment and spread of tweets in relation to brands’ promoted
items.

### Analysis Methods

Sentiment analysis has been approached across many domains, including
products, movie reviews and newspaper articles as well as social media (see
e.g [18] for a comprehensive overview). Typically, the methods employed depend
either on existing language resources (e.g. sentiment dictionaries or
ontologies) or on machine learning from annotated datasets. The former can
provide deep insight, but are somewhat inflexible in the face of the non-
standard and rapidly changing language used on OSNs, for which few suitable
linguistic resources currently exist. The latter are more scalable and can be
trained on relevant data (e.g. [14]), but generally depend on large amounts of
manual annotation (expensive and often problematic in terms of accuracy) and
in some cases the existence of grammatical resources for the language and text
domain in question (e.g. [26]). However, some approaches leverage the
existence of implicit labelling in the datasets available (_distant
supervision_), to avoid the necessity for manual annotation: for example, user
ratings provided with movie or product reviews [19, 4]); or author conventions
such as emoticons and hashtags on OSNs [7, 17, 21]). Hybrid approaches also
exist, e.g. the use of predefined sentiment dictionaries with weights learned
from data (e.g. [27]).

Identifying the topic of text has also received much attention in NLP
research, with methods ranging from the use of existing topic resources or
ontologies (e.g. [12]) to unsupervised models for discovery of topics (e.g.
[23]). The use of machine learning to detect the relevance (or otherwise) of
text to a known topic also has a long history, perhaps most well-known in the
form of Naïve Bayes filtering for spam filtering [25].

However, research into OSN behaviour or influence sometimes ignores the spread
of sophisticated methods available. Sentiment analysis is often performed
based on defined dictionaries (e.g. [28]), and topic identification is often
ignored, with datasets filtered purely on keywords or simple Boolean queries.
Recently, Goncalves et al. [8] examined the difference in performance across
various sentiment analysis approaches on online text, finding significant
variations. The effect of these variations in a specific analysis problem is
less clear, though: how much does the variation in sophistication (and
accuracy) of these methods actually matter? [22] compared statistical and
lexicon-based methods and found significant differences at the level of
individual messages, although a correlation at the level of their intended
analysis (user profiles). Here, we investigate the effect when considering
individual advertising campaigns (promoted items). For text relevance, we
compare the use of keywords to Naïve Bayes classification via Weka [9]. For
sentiment analysis, we examine three existing and freely available tools: the
widely-used Data Science Toolkit’s
text2sentiment444http://www.datasciencetoolkit.org/ based on a sentiment
lexicon [16]; the lexicon-based but data-driven hybrid SentiStrength [27]; and
a statistical machine-learning-based approach, Chatterbox’s
Sentimental555http://sentimental.co/ (see [21]).

##  3 Data collection

We set up a crawler to use the Twitter Streaming
API666https://dev.twitter.com/docs/streaming-apis to collect the tweets of
interest and all associated metadata (e.g., ID, username, user’s social
graph), with details stored in a MySQL database. In this section we briefly
describe our dataset and data collection strategy.

### Identifying promoted brands

Twitter distinguishes promoted tweets and trends by the use of a _Promoted_
tag. We collected tweets from 11 brands with an active advertising campaign
during our study period, across different industry domains, ranging from
entertainment to health-care. For each promoted item, the brand names was used
to crawl Twitter for tweet data posted in English for a 10 day period. If the
promoted item also included a hashtag, the hashtag was also included in the
parameters of the crawl’s GET function. This included all tweets that
contained keywords such as @BrandName, #BrandName, BrandName, #PromotedHashtag
and other brand related terms. These parameter values were selected to keep
the dataset both relevant to brand-related tweets, and also manageable for
searching purposes. Followers and following information was also tracked on a
daily basis for each brand.

Industry | Promotion type | Brand
---|---|---
Electronics | Promoted tweet | International CES
Promoted tweet | SONY
Promoted trend | Nintendo UK
Travel | Promoted tweet | Marriot
Entertainment | Promoted tweet | BBC One
Automobile | Promoted trend | Vauxhall
Heath Care | Promoted tweet | Paints like Me
Retail | Promoted trend | ASOS
Promoted trend | PespiMax
Promoted tweet | JRebel
Telecomms | Promoted trend | O2 Network
Table 1: Industry sectors and sample brands

Details of the selected brands and their promoted type are provided in Table
1. Given that we were interested in promoted items for branding purposes, a
range of different brands from different industries were selected. The aim was
to include both major, and small brands when selecting promoted items. In
addition, a major brand and a small brand enable a comparison of sentiment
while weakly controlling for follower count.

### Dataset

We identified different industries’ promoted items for 10 day periods between
$17^{th}$ December 2012 and $7^{th}$ January 2013. We used non-parallel
crawling periods in order to avoid the query limits set by the Twitter API. In
total, around 180,000 individual tweets were collected by crawling Twitter
continuously, excluding December $21^{st}$ 2012 when there was a 6 hour outage
in the crawler API. The crawler collected tweets from around 120,000 different
Twitter users engaged in spreading the promoted tweets and trends. Tweets
across all topics and with no geographical limits were gathered, as long as
they featured the brand’s name/hashtag. When a brand contained more than one
directly relevant hashtag, e.g., #Coke and #CocaCola, we included all the
relevant hashtags.

Twitter users do often repeat their tweets to benefit from repeated exposure.
However, in order to remove noise and bias in analysis caused by spam tweets,
we removed users who had posted the exact same tweet more than 20 times during
our measurement periods, along with their tweets. Twitter users, tweets and
tweet timestamps were also cross-analysed to check for spamming accounts. In
one case a single user was removed for adding over 8,000 spam tweets to the
database. After manual inspection of many tweets and accounts, we are
confident that nearly all spam has been removed from our dataset.

##  4 Text Processing & Classification

In this section we present the details of our tweet classification (using ML)
and sentiment analysis (using existing NLP tools).

###  4.1 Topic Classification

One of the major challenges during cleaning the dataset and removing spam was
ensuring topic relevance. Our expectation was that this would not be an issue:
as in much previous work, our study is looking at all sentiment expressed
towards the brands, as long as the tweet matched the parameters of the tweet
selection as explained in Section 3. However, whilst sampling tweets for
spammers, a general problem surfaced. We found that a keyword-based approach
tends to be too broad to accurately identify tweets referring to a particular
brand, _O2_ (a UK mobile telecommunications provider and network). Our
parameters for collecting tweets for this brand were to match tweets
containing O2WhatWouldYouDo and O2 (the hashtag being promoted was
#O2WhatWouldYouDo and @O2 is the official brand Twitter handle). Over the 10
day period, 90,000 tweets were collected that matched these keywords. However,
examining a random sample of 200 tweets from this dataset showed that over 70%
were not referring to the O2 Network brand; many were referring to the “O2
Academy” (a chain of concert venues), the “O2 Arena” (a dome-shaped
monstrosity in London), or other senses of ‘O2’ such as oxygen. We also
noticed that Twitter users have recently established a new way of using the
letter sequence ‘O2’ as a replacement for the letters ‘to’: e.g.
“@CokeWave_Thang What Picture You Want Me O2 Put As My BackGround”, “what im
goin o2 do o2day”. Experiments with boolean combinations of O2 with other
keywords were not successful. A major challenge therefore becomes to filter
out non-brand-related tweets automatically: the problem is not trivial, given
the variability and unpredictability of language, vocabulary and spelling on
Twitter, and the short length of tweets (up to 140 characters); and manual
removal of approximately 70% of large datasets is prohibitively labour-
intensive.

We therefore approached this as a text classification problem and investigated
various supervised machine learning approaches using the Weka toolkit [9].
First, we performed a pilot study over a 200-tweet development set to
determine a suitable feature representation and classification method; the
data was manually labelled as O2-related or otherwise to give a binary
decision problem. We tested a variety of classifiers including Naive Bayes,
Naive Bayes Multinomial, ID3, IBK and J48 decision trees; features were based
on the tweet text using a standard bag-of-words representation (see e.g. [13])
with various scaling methods,777We used Weka’s StringToWordVector filter for
text feature extraction and scaling. with the addition of user ID and date of
tweet. Given the small size of the dataset, we restricted the feature space to
be based on the most common 100 words. We also tested using a simple manual
keyword-based filter to remove some common negative instances (using keywords
arena, academy, etc) before training (see “manually filtered” results in the
figures). Tests were performed using ten-fold cross-validation in order to
simulate performance on unseen data. Best performance (overall accuracy) was
obtained using only bag-of-words text features, with stopwords removed and a
TF-IDF weighting, after manual filtering. The best performing classifiers in
cross-validation were J48 and Naive Bayes (NB), with 71% and 91% accuracy
respectively. We then compared their performance on a held-out test set: the
NB model outperformed the J48 model with 84% accuracy compared to 71% for J48,
with training and prediction also noticeably faster for NB (the tree structure
of the J48 model made it very slow with larger training sets).

Figure 1: NB accuracy with increasing training data.

To determine a suitable training set size, we then varied the training set
while testing performance on a held-out test dataset of 30 manually labelled
tweets. Increasing training set size improved performance (see Figure 1): we
tested up to a 2,000-tweet training set; while the curve suggests performance
may improve beyond this point, the accuracy on the held-out test set is
approaching that on the training set so large improvements are unlikely. The
NB classifier trained on 2,000 tweets was therefore used for the experiments
below. Figure 2 shows results when tested on a larger, unseen, randomly
selected test set of 100 tweets; the version with manual filtering achieves
78% accuracy, 77% recall and 66% precision. Figure 3 gives details of the per-
class predictions: without manual filtering, false positives are more common
than false negatives (i.e. too much irrelevant data is slipping through);
levels are much closer with filtering.

Figure 2: Classification results using Naive Bayes. Figure 3: Classification
details per class using Naive Bayes.

###  4.2 Sentiment Analysis

Having identified tweets with relevant content, we now required a method for
sentiment analysis – determining the positive or negative stance of the
writer. As discussed in Section 2 above, many methods for sentiment detection
exist, with the major distinction being between lexicon-based and machine
learning-based approaches. We examined existing tools for Twitter sentiment
analysis using both of these approaches in order to determine the most
suitable for our data.

As a baseline lexicon-based tool we used the freely available Data Science
Toolkit888http://www.datasciencetoolkit.org/. The sentiment analyser is based
on a sentiment lexicon [16]; we therefore anticipate its coverage to be low
but take it to be representative of commonly-used lexicon-based approaches.

For a more robust tool for comparison, we examined two alternatives. As a
hybrid lexicon/machine-learning tool we chose SentiStrength [27]. This method
uses a predetermined list of words commonly associated with negative or
positive sentiment, which are given an empirically determined weight (learned
from data); new texts are classified by summing the weights of the words they
contain. Thelwall et al. [27] report accuracy on Twitter data of 63.7% for
positive sentiment and 67.8% for negative when predicting ratings on a 1-5
scale, and accuracies near 95% when predicting a simple binary
positive/negative label. However, even though their word lists and weightings
are determined for OSN data (including Twitter), this approach may suffer when
faced with social text with new words, unexpected spellings and context-
dependent language and meaning (see [15]).

For a purely ML-based option we used Chatterbox’s Sentimental
API,999http://mashape.com/sentimental/sentiment-analysis-for-social-media
based on statistical machine learning over large, distantly labelled datasets
[21]. This data-based approach means it might be expected to handle slang,
errorful or abbreviated text better. Purver & Battersby [21] report accuracies
approaching 80% using a similar technique on smaller datasets; Chatterbox
report 83.4% accuracy in an independent study.101010See
http://content.chatterbox.co/Sentiment\%20Analysis\%20Case\%20Study\%20-\%20Chatterbox\%20and\%20IDL.pdf.

Before applying the sentiment analysis tool, and in order to compare the two
approaches, a few hundred random tweets were selected from the database and
were read and manually labelled for positive or negative sentiment, and both
tools were tested on the resulting set. Results showed accuracy below 50% for
the lexicon-based Data Science Toolkit, 63% for the hybrid SentiStrength
approach, and 84% for the ML-based Chatterbox approach. Error analysis showed
one significant source of the latter difference to be sentiment expressed in
hashtags (e.g. the negative #shambles), which were detected better by the ML-
based approach, presumaby due to their absence from SentiStrength’s
predetermined lexicon. We therefore use Chatterbox as the “robust” tool in our
experiments below, and compare to the Data Science Toolkit as a purely
lexicon-based baseline.

##  5 Results

Figure 4: Distribution of promoted tweets volumes over time. Figure 5:
Distribution of promoted trends volumes over time.

### Response volume over time

To examine the spread of engagement for each promoted item over the 10 day
period, we analysed the volume of unique tweets each day in response to each
promoted item, then averaged the results across all brands. Figures 4 and 5
display the distribution of this volume in response to _promoted tweets_ (4)
and _promoted trends_ (5) per brand. On average, promoted trends led to much
higher response volumes. However, the highest percentages of _mentions_ within
responses were from promoted tweets, where an average of 18% of tweets each
day included an ‘@’ mention to the brand; promoted trends had an average of
only 15% mentions per day. This indicates that for a brand to successfully
engage users in the content of the promoted item, a promoted tweet is better
for this purpose. For example, out of the O2 Network’s $\sim$30,000 tweets,
7,965 included an ‘@’mention to the brand (25%).

Results confirmed that the greatest percentage of engagement for a brand’s
promoted item takes place on the first day of promotion. On average, 24% of
engagements around the promoted item take place on the first day. The effect
is most pronounced for _promoted trends_ , with 34% of engagement on average
on the first day of promotion, after which the engagement falls dramatically
by an average of 25% to 9% by day two and continues to fall thereafter, even
if the item is promoted for several days. For _promoted tweets_ , the effect
is less pronounced: 19% of the engagement takes place on the first day of
promotion, with engagement decreasing by 8% by the second day of promotion.
However, it does not continue on a steady decline thereafter, but it rises and
falls over the next 8 days, although never again reaching the peak of the
first day of promotion. This could be due to the fact that a promoted tweet is
usually promoted for several days on Twitter where it occasionally appears at
the top of different user’s timeline were users are repeatedly exposed to the
item. This finding can be said to conform to Romero et al.’s theory of
repeated exposure [24].111111Also see
http://advertising.twitter.com/2013/03/Nielsen-Brand-Effect-for-Twitter-How-
Promoted-Tweets-impact-brand-metrics.html They found that repeated exposure to
a hashtag within Twitter had a significant marginal effect on the probability
of adoption of that hashtag.

In general, though, these results show that adoption of a promoted item is not
a slow gradual shift over several days (as might be assumed) but rather an
immediate incline when exposure to the item is new to users.

### Effects on user sentiment

The sentiment breakdown for each promoted brand item can be observed in
Figures 6 and 7, with Figure 6 showing the results obtained using our chosen
machine learning method and Figure 7 those obtained using a keyword-based
method (see section 4 above). We observe that in most cases, the percentage of
positive sentiment was higher than that of negative and neutral for promoted
items. Notable exceptions are the results for two brands, NiveaUK and O2,
where neutral and/or negative levels outweigh positive; the ASOS brand also
shows little difference between negative and positive levels. However,
comparison of the figures that would have been gained using a keyword-based
approach (Figure 7) shows misleading results in precisely these interesting
cases: apparent positive levels are higher than negative in all cases. Neutral
cases also appear much more common; this is due to the low coverage of the
keyword lexicon causing large numbers of results with apparently zero
sentiment. Use of the more accurate tool (as objectively assessed – see
section 4) therefore does appear crucial.

Figure 6: Sentiment analysis by brand - machine learning Figure 7: Sentiment
analysis by brand - keywords

On average, across all brands (promoted tweets and trends), the average
percentage of tweets and retweets121212We assume that retweeting users share
the same sentiment as the original tweet. which contained a positive sentiment
is 50%, that which contained a negative sentiment is 12%, and 38% of tweets
had a neutral tone.

Figures 8 and 9 then show the distribution of positive and negative sentiments
in this response traffic over time. On average, positive sentiment outweighs
negative sentiment; on the first day, 49% of the tweets were positive. In
general, _promoted tweets_ lead to more positive sentiment and less negative
sentiment than _promoted trends_.

Figure 8: Positive sentiment distribution over time Figure 9: Negative
sentiment distribution over time.

In total, 47% of tweets relating to a _promoted tweet_ are positive in
sentiment. Day one received the highest percentage of positive sentiment
tweets (58%); positive sentiment then continues to dominate over the 10 day
period, never falling below 36% of the tweets. Examining _promoted trends_ ,
we found that, on average, only 37% of tweets relating to a promoted trend
contained a positive sentiment. On the first day of promotion, 26% of tweets
expressed a negative sentiment, 32% expressed a positive sentiment and 42%
expressed no sentiment at all. This shows that Twitter users do not tweet as
positively about a promoted trend as they would about a promoted tweet.
Instead, a large proportion of tweets relating to a promoted trend contained
no emotional words, or if they did, the positive and negative sentiments
balanced each other out. They generally contained just the promoted hashtag or
generally had an objective, matter-of-fact tone (e.g., - “Get 3G where I
live... #O2WhatWouldYouDo”).

Taken together with the analysis of engagement volume, these results show that
when an item is promoted, the brand and the item get adopted immediately and
regarded quite positively by the engaged users. Twitter users welcome the
promoted item on Twitter, which has a positive effect on the tweets expressed.
The engagement level reduces to an average of 10% of the total tweets on day
two, when the item is no longer being promoted, or is no longer seen as “new
and interesting”. However, on average, the positive sentiment expressed still
outperforms that of negative sentiment and neutral sentiment each day.

### Effect of hashtags on engagement and sentiment

Figure 10: Hashtag related engagements for ASOS. Figure 11: Hashtag related
engagements for Vauxhall.

We then performed two example case studies, using the ASOS and Vauxhall
brands, to examine the use of hashtags within promoted items. Figures 10 and
11 show the results. ASOS promoted a trend, #AsosSale, on the $19^{th}$ and
$20^{th}$ of December to highlight their Boxing Day sale on the $26^{th}$ of
December (day 8 of data collection). Although the promoted hashtag was
virtually discarded by day two of data collection, we found that user
engagement (use of hashtag, mentions and tweets) for the forthcoming sale
continued. This trend is also apparent in Vauxhall’s tweet volumes for their
sale which stated on the $27^{th}$ of December (day one of promotion), and
ended the day after our 10 day data collection period. The engagement for
Vauxhall remained at a consistent level throughout the event (see Figures 5
and 11), despite the rapid drop-off in use of the promoted hashtag.

##  6 Conclusions & Future directions

In this paper we present a measurement-driven study of the effects of promoted
tweets and trends on Twitter on the engagement level of users, using a number
of ML and NLP techniques in order to detect relevant tweets and their
sentiments. Our results indicate that use of accurate methods for sentiment
analysis, and robust filtering for topical content, is crucial. Given this, we
then see that promoted tweets and trends differ considerably in the form of
engagement they produce and the overall sentiment associated with them. We
found that promoted trends lead to higher engagement volumes than promoted
tweets. However, although promoted tweets obtain less engagement than promoted
trends, their engagement forms are often more brand inclusive (more direct
mentions); and while engagement volumes drop for both forms of promoted items
after the first day, this effect is less pronounced for promoted tweets. We
also found that although the volume of tweets is highest in promoted trends,
they do not lead to the same level of positive sentiment that promoted tweets
do. Hence advertisers should carefully assess the trade-offs between high
level of engagement, drop-off rate, direct mentions, and positive user
sentiment.

In the next stage of this study we will investigate the effect of individuals’
influence on the take-up of promoted tweets and trends by their social graph.
We will investigate new data at finer granularity (hourly) for events that are
time-sensitive, such as major concert ticket sales. This is our first attempt
at understanding this space. The advertising campaigns have very different
structure and we need to understand these in details. Promoted trends
typically stay on the trends list for a day, and promoted tweets are
selectively shown to a subset of users for a period of time selected by the
advertiser. Without accounting for such nuances, broad statements on the
impact of the two forms of advertising are not conclusive. However in this
paper we focussed on insights in using sentiment analysis methods and accurate
data labelling. We believe our findings could provide a new insight for social
network marketing and advertisements strategies, in addition to comparing
different methods of classifying and filtering relevant content.

## Acknowledgments

The authors thank Chatterbox131313http://chatterbox.co for providing unlimited
access to the Sentimental API.

## References

  * [1] S. Asur and B. A. Huberman.  Predicting the future with social media.  CoRR, abs/1003.5699, 2010.
  * [2] T. Blake, C. Nosko, and S. Tadelis.  Consumer heterogeneity and paid search effectiveness: A large scale field experiment.  2013\.
  * [3] M. Cha, H. Haddadi, F. Benevenuto, and K. P. Gummadi.  Measuring user influence in twitter: The million follower fallacy.  In in ICWSM 10: Proceedings of international AAAI Conference on Weblogs and Social, 2010.
  * [4] T. Y. Chan, C. Wu, and Y. Xie.  Measuring the lifetime value of customers acquired from google search advertising.  Marketing Science, 30(5):837–850, Sept. 2011.
  * [5] C. M. Cheung and D. R. Thadani.  The effectiveness of electronic word-of-mouth communication: A literature analysis.  Proceedings of the 23rd Bled eConference eTrust: Implications for the Individual, Enterprises and Society, 2010.
  * [6] A. M. Degeratu, A. Rangaswamy, and J. Wu.  Consumer choice behavior in online and traditional supermarkets: The effects of brand name, price, and other search attributes.  International Journal of Research in Marketing, 17(1):55 – 78, 2000\.
  * [7] A. Go, R. Bhayani, and L. Huang.  Twitter sentiment classification using distant supervision.  Master’s thesis, Stanford University, 2009.
  * [8] P. Goncalves, M. Araujo, F. Benevenuto, and M. Cha.  Comparing and combining sentiment analysis methods.  In Proceedings of the 1st ACM Conference on Online Social Networks (COSN’13), 2013.
  * [9] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten.  The WEKA data mining software: An update.  SIGDKDD Explorations, 11(1):10–18, 2009.
  * [10] B. J. Jansen, M. Zhang, K. Sobel, and A. Chowdury.  Twitter power: Tweets as electronic word of mouth.  Journal of the American Society for Information Science and Technology, 60(11):2169–2188, 2009.
  * [11] C. Liebrecht, F. Kunneman, and A. Van den Bosch.  The perfect solution for detecting sarcasm in tweets #not.  In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 29–37, Atlanta, Georgia, June 2013. Association for Computational Linguistics.
  * [12] P. Malo, P. Siitari, O. Ahlgren, J. Wallenius, and P. Korhonen.  Semantic content filtering with Wikipedia and ontologies.  In IEEE International Conference on Data Mining Workshops (ICDMW), pages 518–526. IEEE, 2010.
  * [13] C. Manning and H. Schütze.  Foundations of Statistical Natural Language Processing.  MIT Press, 1999.
  * [14] Y. Mejova and P. Srinivasan.  Crossing media streams with sentiment: Domain adaptation in blogs, reviews and twitter.  Proc. ICWSM, 2012.
  * [15] A. Naradhipa and A. Purwarianti.  Sentiment classification for indonesian message in social media.  In Cloud Computing and Social Networking (ICCCSN), 2012 International Conference on, pages 1–5, 2012.
  * [16] F. Å. Nielsen.  A new ANEW: evaluation of a word list for sentiment analysis in microblogs.  In Proceedings of the ESWC2011 Workshop ‘Making Sense of Microposts’: Big things come in small packages, pages 93–98, May 2011.
  * [17] A. Pak and P. Paroubek.  Twitter as a corpus for sentiment analysis and opinion mining.  In Proceedings of the 7th conference on International Language Resources and Evaluation, 2010.
  * [18] B. Pang and L. Lee.  Opinion mining and sentiment analysis.  Foundations and Trends in Information Retrieval, 2(1–2):1–135, 2008.
  * [19] B. Pang, L. Lee, and S. Vaithyanathan.  Thumbs up?: sentiment classification using machine learning techniques.  In Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10, EMNLP ’02, pages 79–86, Stroudsburg, PA, USA, 2002. Association for Computational Linguistics.
  * [20] M. J. Paul and M. Dredze.  You are what you tweet: Analyzing twitter for public health.  In Int’l Conference on Weblogs and Social Media, ICWSM, 2011.
  * [21] M. Purver and S. Battersby.  Experimenting with distant supervision for emotion classification.  In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 482–491, Avignon, France, Apr. 2012. Association for Computational Linguistics.
  * [22] D. Quercia, J. Ellis, L. Capra, and J. Crowcroft.  Tracking “gross community happiness” from tweets.  Technical Report RN/11/20, University College London, Nov. 2011.
  * [23] A. Ritter, C. Cherry, and B. Dolan.  Unsupervised modeling of Twitter conversations.  In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 172–180, Los Angeles, California, June 2010. Association for Computational Linguistics.
  * [24] D. M. Romero, B. Meeder, and J. Kleinberg.  Differences in the mechanics of information diffusion across topics: idioms, political hashtags, and complex contagion on twitter.  In Proceedings of the 20th international conference on World wide web, WWW ’11, pages 695–704, New York, NY, USA, 2011. ACM.
  * [25] M. Sahami, S. Dumais, D. Heckerman, and E. Horvitz.  A Bayesian approach to filtering junk email.  In AAAI Workshop on Learning for Text Categorization, Madison, WI, July 1998.  AAAI Technical Report WS-98-05.
  * [26] R. Socher, A. Perelygin, J. Y. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts.  Recursive deep models for semantic compositionality over a sentiment treebank.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013.  To appear.
  * [27] M. Thelwall, K. Buckley, and G. Paltoglou.  Sentiment strength detection for the social web.  J. Am. Soc. Inf. Sci. Technol., 63(1):163–173, Dec. 2012.
  * [28] A. Tumasjan, T. O. Sprenger, P. G. Sandner, and I. M. Welpe.  Predicting elections with twitter: What 140 characters reveal about political sentiment.  In Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media, 2010.