# Alternative approaches for analysing
repeated measures data that are
missing not at random

Oliver Dukes1, David Richardson2, and Eric Tchetgen Tchetgen1
1: University of Pennsylvania, Philadelphia, PA, USA
2: University of California Irvine, Irvine, CA, USA

###### Abstract

We consider studies where multiple measures of an outcome variable are
collected over time, but some subjects drop out before the end of follow up.
Analyses of such data often proceed under either a ‘last observation carried
forward’ or ‘missing at random’ assumption. We consider two alternative
strategies for identification; the first is closely related to the difference-
in-differences methodology in the causal inference literature. The second
enables correction for violations of the parallel trend assumption, so long as
one has access to a valid ‘bespoke instrumental variable’. These are compared
with existing approaches, first conceptually and then in an analysis of data
from the Framingham Heart Study.

## Introduction

In this paper, we consider the analysis of data where a variable of interest
is measured at multiple time points, and at certain time points is subject to
missing entries. We will consider in particular settings where there is drop-
out without re-entry, such that once an individual has missing data at one
time point, then data at all subsequent time points are also missing. Analyses
with drop-out sometimes proceeds under a Missing At Random (MAR) assumption,
whereby one assumes that missingness depends on variables which are fully
observed under a given missing data pattern. Such an assumption is convenient,
as one can then employ methods such as multiple imputation or mixed models for
repeated measures to remove potential biases; these are now widely available
in standard software packages.

Nevertheless, in many settings MAR may not be plausible. In that case,
estimation and inference under an alternative Missing Not At Random (MNAR)
assumption may be possible, albeit more challenging. Some examples of possible
alternative assumptions are given in Wu and Carroll, (1988); Diggle and
Kenward, (1994); Rotnitzky and Robins, (1997); Little and Rubin, (2019). Much
of the literature on analyses under MNAR mechanisms has focused on sensitivity
analyses (Robins et al.,, 2000; Carpenter et al.,, 2007), rather than (or in
addition to) reporting a single estimate and confidence interval for a
parameter of interest. Typically, one fluctuates a sensitivity parameter in
order to assess the consequences of violating a ‘reference’ missing data
assumption.

In this article, we will consider two alternative missing data assumptions,
which may hold under certain MNAR mechanisms. Both harness the fact that the
data include repeated measures from the same individuals. The first approach
is an extension of the difference-in-differences (DiD) approach that is well
known in the causal inference literature (Card and Krueger,, 1993; Lechner et
al.,, 2011). This design makes a comparison between treatment arms in terms of
a difference in the end of study outcome from baseline, and can recover a
causal effect under a ‘parallel trends’ assumption. We therefore extend this
strategy to the missing data setting, and compare the parallel trends
assumption both to MAR and to the controversial Last Observation Carried
Forward (LOCF) assumption (Kenward and Molenberghs,, 2009; Lachin,, 2016).
After this, we consider a distinct approach based on harnessing a ‘bespoke’
Instrumental Variable (IV). This strategy was recently described for inferring
causal effects in Richardson and Tchetgen Tchetgen, (2021) and Dukes et al.,
(2022). We show how a bespoke IV can be used to correct for violation of the
parallel trends assumption, so long as the association between the bespoke
instrument and the outcome is stable across time.

Both the DiD and bespoke IV strategies are relatively straightforward to
compute using existing software. Although we first describe them in a simple
two-time point setting, they can be extended to longitudinal settings with
many repeated measures. In Section 5 we describe their use for estimating the
effect of a treatment in a clinical trial where outcomes are subject to
monotone missingness. The DiD strategy we propose is closely related to the
changes-in-changes approach described in Ghanem et al., (2022); the proposal
made here is generally simpler to implement, and we show how it can be
extended to more general longitudinal data structures (those authors focus on
a two-time point setting). Interestingly, methods closely related to the
proposals described here have been implemented before in the analysis of
clinical trials (Aisen et al.,, 2003; Petersen et al.,, 2005; Molnar et al.,,
2008), although the connection with the DiD design does not seem to have been
made, nor have the assumptions been rigorously stated. As far as we are aware,
this is the first work to extend the bespoke IV strategy to missing data
problems.

##  1 Setting and existing methods

To illustrate the methods and develop intuition for the assumptions, we will
first focus on a simple setting, where data are collected on an outcome $Y$ at
two consecutive time points. Consider the following data structure
$\\{Y_{0},R,RY_{1}\\}$ where $R$ is a missingness indicator, and $Y_{t}$,
$t=0,1$ is an outcome of interest measured at time $t$. Suppose that there is
interest in the full-data population mean of the final outcome $E(Y_{1})$;
then one can show that

$\displaystyle E(Y_{1})$
$\displaystyle=E(Y_{1}|R=1)-\\{E(Y_{1}|R=1)-E(Y_{1}|R=0)\\}Pr(R=0)$

where the conditional expectation $E(Y_{1}|R=0)$ is unknown. If we use the
mean of $Y_{1}$ in the complete cases in place of $E(Y_{1})$, then the bias is
generally equal to

$\\{E(Y_{1}|R=1)-E(Y_{1}|R=0)\\}Pr(R=0)$

If data are “missing completely at random” (MCAR), then $R\perp\\!\\!\\!\perp
Y_{1}$; in other words, the missingness indicator does not depend on the
variable $Y_{1}$ which is subject to missingness. Under this assumption, it
follows that the bias term given above is equal to zero. In contrast, if the
data are MAR or MNAR, analysing data by restricting to the complete cases will
generally lead to invalid results.

When one has access to a baseline measurement $Y_{0}$, an approach that is
sometimes leveraged to identify $E(Y_{1})$ is LOCF. The underlying assumption
can be phrased as

$\displaystyle E(Y_{1}|R=0)=E(Y_{0}|R=0)$ (1)

e.g. that the average of $Y_{1}$ in those with missing data is equal to the
average of the earlier measurement in the same group. Under LOCF, one can show
that $E(Y_{1})$ can be identified as

$\displaystyle E(Y_{1})$
$\displaystyle=E(Y_{1}|R=1)-\\{E(Y_{1}|R=1)-E(Y_{0}|R=0)\\}Pr(R=0)$

One can show that the right hand side of the above is equal to

$E(Y_{1}|R=1)Pr(R=1)+E(Y_{0}|R=0)Pr(R=0)$

which means that $E(Y_{1})$ can straightforwardly be estimated using the
observed data by substituting the missing outcomes with values of $Y_{0}$.
Nevertheless, condition (1) is often deemed as implausible (Shao and Zhong,,
2003; Kenward and Molenberghs,, 2009), since it requires there to be no time
trend in $Y_{t}$ in those with missing data. Despite criticism, this approach
has been widely used in practice (Lachin,, 2016), likely due to its
computational simplicity.

A second assumption often made that leverages the baseline $Y_{0}$ is MAR. In
this context, the MAR assumption can be expressed as

$R\perp\\!\\!\\!\perp Y_{1}|Y_{0}$

e.g. the missing data mechanism is independent of $Y_{1}$, within levels of
the baseline measurement. This condition is often considered as a relaxation
of the previous MCAR assumption. Repeating the previous decomposition of the
mean of $Y_{1}$ whilst now conditioning on $Y_{0}$ gives

$\displaystyle E(Y_{1}|Y_{0})=$ $\displaystyle
E(Y_{1}|R=1,Y_{0})-\\{E(Y_{1}|R=1,Y_{0})-E(Y_{1}|R=0,Y_{0})\\}Pr(R=0|Y_{0})$

and the MAR assumption implies that

$E(Y_{1}|R=1,Y_{0})-E(Y_{1}|R=0,Y_{0})$

is equal to zero. Hence, one can identify $E(Y_{1})$ by averaging the
predictions $E(Y_{1}|R=1,Y_{0})$ made in the complete cases over the
distribution of $Y_{0}$ in the full sample. More generally, under the MAR
assumption, one can then estimate parameters of interest using maximum
likelihood (Allison,, 2012), imputation approaches (Little and Rubin,, 2019),
inverse probability weighting (Horvitz and Thompson,, 1952; Robins et al.,,
1994), or combinations of the above e.g. doubly robust methods (Robins et
al.,, 1994, 1995). These all rely on the same missing data conditions for
validity, but may require distinct parametric modelling assumptions (and
possess differing statistical properties).

##  2 Difference in differences

The DiD assumption for missing data can be stated as :

$\displaystyle E(Y_{1}-Y_{0}|R=1)=E(Y_{1}-Y_{0}|R=0)$ (2)

The above condition assumes the trend in outcomes in the unobserved group
equals the trend in the observed group. Such an assumption is illustrated in
Figure 1(a), where the black line indicates the trend in average outcome
values between time points in those with observed data. The dashed line is the
trend in values in those with missing data; note that the point where the line
crosses the $Y$ axis represents the measurement $E(Y_{0}|R=0)$ which is
observed. However, because $E(Y_{1}|R=0)$ is unobserved, one would not observe
this trend in those with $R=0$. Note that MCAR is not satisfied here, as
$E(Y=1|R=1)\neq E(Y=1|R=0)$. Assumption (2) would nevertheless be satisfied,
since the observed and unobserved trend curves are parallel, so that one can
use the average difference in $Y_{1}-Y_{0}$ in the complete cases to predict
$E(Y_{1}|R=0)$. Indeed, one can show that the full population mean $E(Y_{1})$
is identified as

$\displaystyle E(Y_{1})$
$\displaystyle=E(Y_{1}|R=1)-\\{E(Y_{0}|R=1)-E(Y_{0}|R=0)\\}Pr(R=0).$

One can construct an estimator of $E(Y_{1})$ based on the above expression,
replacing the population expectations with the sample analogues.
Alternatively, one can impute the missing outcomes with the baseline $Y_{0}$
plus the average change $Y_{1}-Y_{0}$ as estimated in the complete cases,
before calculating the mean of the observed and imputed $Y_{1}$ values.
Estimation of a standard error can be done using the nonparametric bootstrap
(the same holds for other approaches discussed in this article).

The DiD assumption (2) is equivalent to the condition that

$E(Y_{1}|R=1)-E(Y_{1}|R=0)=E(Y_{0}|R=1)-E(Y_{0}|R=0)$

(Sofer et al.,, 2016). Note that this is no longer phrased in terms of the
outcome trends, but rather concerns the bias in the complete case analysis.
The assumption therefore states that this bias is exactly equal to the
association between the baseline outcome and the missingness indicator, and
therefore can be learnt from the data. In other words, the bias from
restricting to complete cases is reflected by the baseline differences (in
terms of the outcome) between those with and without missing data.

Although as far as we are aware, the DiD strategy has received limited
attention from missing data methodologists, versions of it nevertheless appear
to have been applied in previous analyses of clinical trials. For example, in
a trial comparing treatment with naproxen or rofecoxib versus placebo in
patients with mild-to-moderate Alzheimer disease, the primary outcome was the
Alzheimer Disease Assessment Scale-Cognitive (ADAS-Cog) subscale score (a
measure of cognitive decline) at 12 months after randomisation (Aisen et al.,,
2003). To impute missing outcome data, the estimated score change for the
unobserved period in the relevant treatment arm complete cases was added to a
participant’s last observed ADAS-Cog score. The approach is described as
‘slope imputation’, and is used in the primary analysis as an alternative to
LOCF.

###  2.1 Comparison with LOCF

(a)

(b)

Figure 1: Two figures where the observed data is the same, but the missing
data mechanism changes. The bold circles indicate the observed values of
$E(Y_{t}|R=r)$ at $t=0$ and $t=1$ (if $R=1$). The faint circle indicates the
unobserved $E(Y_{1}|R=0)$. Figure 1(a) represents a data generating mechanism
where the parallel trends assumption (2) holds, whereas Figure 1(b) represents
a setting where (2) is violated but (1) holds.

Assumption (2) is closely related to the LOCF assumption (1): it can be
further rephrased as

$E(Y_{1}|R=0)=E(Y_{0}|R=0)+E(Y_{1}-Y_{0}|R=1)$

in other words, that the mean of $Y_{1}$ in the group with missing data is
equal to the baseline mean in that group (as in LOCF), plus the average trend
in outcomes in the complete cases. The DiD assumption may often be more
plausible than LOCF, if one believes that a time trend in the outcome is
likely. However, the fact that $E(Y_{1}-Y_{0}|R=1)\neq 0$ (which can be
checked from the data) does not necessarily support the DiD assumption over
LOCF. This is because (2) is untestable; it may be that in truth there was no
time trend in those with $R=0$. This is illustrated in Figure 1(b), where one
sees that the unobserved time trend in those with $R=0$ is equal to zero.
However, since there was a trend in the complete cases, a DiD analysis would
erroneously use this trend to impute $Y_{1}$. In summary, a DiD analysis
corrects the bias from LOCF only when the (untestable) parallel trends
assumption holds.

One major criticism of LOCF is that even when the data is MCAR, it can still
yield biased results. To make this concrete, suppose that that MCAR assumption
$R\perp\\!\\!\\!\perp Y_{1}$ holds. The LOCF analysis then has a bias equal to

$\\{E(Y_{1}|R=1)-E(Y_{0}|R=0)\\}Pr(R=0)$

which is not equal to zero unless $E(Y_{1}|R=1)=E(Y_{0}|R=0)$. This is not
implied either by MCAR or the LOCF assumption alone. On the other hand, note
that the DiD analysis has a bias of

$\\{E(Y_{0}|R=1)-E(Y_{0}|R=0)\\}Pr(R=0);$

if one strengthens the previous MCAR condition to
$R\perp\\!\\!\\!\perp(Y_{1},Y_{0})$, then it follows that this analysis in
unbiased. In practice, it is difficult to think of settings where MCAR holds
but the stronger condition does not, as $Y_{1}$ and $Y_{0}$ will usually be
strongly correlated, so $R$ will typically depend on both (or neither).

$(a)$$Y_{0}$$U$$Y_{1}$$R$

$(b)$$Y_{0}$$U$$Y_{1}$$R$

$(c)$$Y_{0}$$U$$Y_{1}$$R$

$(d)$$Y_{0}$$U$$Y_{1}$$R$

Figure 2: Four causal diagrams illustrating different missing data mechanisms.

###  2.2 Comparison with MAR

This assumption is also distinct from MAR, as it neither implies it nor is
implied by it. The comparison is similar to the distinction between the
parallel trends assumption and conditional mean exchangeability (given the
baseline outcome) in the causal inference literature. To see a setting where
MAR may be preferable, consider the causal diagram in Figure 2(a); drop-out is
directly influenced by the baseline outcome $Y_{0}$, which is associated with
the outcome of interest via an unmeasured latent variable $U$. It follows from
the diagram that conditioning on $Y_{0}$ renders $Y_{1}$ and the missingnness
indicator independent. However, in Figure 2(b), drop-out is influenced instead
by the unmeasured $U$, rather than $Y_{0}$; in this case there would still be
residual dependence between $R$ and $Y_{1}$. This figure corresponds to an
MNAR mechanism.

Still focusing on Figure 2(b), suppose that the following linear models hold:

$\displaystyle E(Y_{t}|R,U)$ $\displaystyle=\alpha_{t}+\beta U$ $\displaystyle
E(U|R)$ $\displaystyle=\delta+\gamma R$

The first equation encodes the restriction that $R$ is independent of $Y_{1}$
given $U$; importantly, the association between $U$ and the outcome is also
constant across time points. Then by averaging out $U$, we have

$\displaystyle E(Y_{t}|R)$ $\displaystyle=\alpha_{t}+\beta\\{\delta+\gamma
R\\}$

so although $E(Y_{1}|R)$ and $E(Y_{0}|R)$ both depend on $R$, we see that
$E(Y_{1}-Y_{0}|R)=\alpha_{1}-\alpha_{0}$ does not. Hence in this setting,
whilst an analysis under MAR may be misleading, the DiD assumption (2) holds.
We note that (2) can be used as a primitive assumption, and one does not need
the previous system of linear equations to hold; nor does one need to make
assumptions about the stationarity of the association between $Y_{t}$ and $U$.
Nevertheless, this example may help to reason about settings where this
assumption is plausible and others may fail.

Figures 2(c) and 2(d) illustrate other settings where MAR would generally
fail, but a DiD analysis may be unbiased so long as (2) holds. A property
shared by Figures 2(b)- 2(d) is that $Y_{0}$ is not a common cause of $R$ and
$Y_{1}$, but rather causes either one or the other (or neither). In these
cases $Y_{0}$ is not itself responsible for the missing data bias. This is a
distinction with the MAR analysis, where $Y_{0}$ is conditioned on to close
the ‘backdoor path’ between $R$ and $Y_{1}$ (Pearl,, 1995).

To summarise, the MAR and DiD assumptions are in general distinct and non-
overlapping. There are however settings where both may hold, although these
are perhaps artificial. One setting where both would coincide is if we were to
assume that the following linear model holds:

$\displaystyle E(Y_{1}|R,Y_{0})=\alpha^{*}_{1}+Y_{0}$ (3)

Note that this implies the MAR assumption as the right hand side does not
depend on $R$; however, stronger parametric restrictions are made, namely that
the coefficient in the linear model for $Y_{0}$ is exactly equal to 1. One can
show that this model also implies (2). We emphasise that in a DiD analysis,
one does not require (3) to hold; rather, the parallel trends assumption (2)
and MAR coincide under (3).

##  3 Bespoke instrumental variables

In certain cases, the parallel trends assumption (2) will not be plausible.
For example, in the trial evaluating different treatments for Alzheimer
Disease, it may be that the change in ADAS-Cog scores in the active
participants who dropped out tended to be less pronounced than the change
observed in the complete cases. Assumption (2) may fail when there are
unmeasured factors which are common causes of $Y_{0}$ and $R$ which differ
from the unmeasured common causes of $Y_{1}$ and $R$. The bias of the DiD
estimate of $E(Y^{1})$ can be shown to equal

$\\{E(Y_{1}-Y_{0}|R=1)-E(Y_{1}-Y_{0}|R=0)\\}Pr(R=0).$

It can be easily verified that this expression equals zero when either (2)
holds or $Pr(R=0)=0$ (there is no missing data). In what follows, we will look
at how collecting data on an additional auxiliary variable $Z$ can help to
relax (2). This comes at the cost of some alternative assumptions made between
the outcome process and $Z$, which in certain cases may be more plausible.

Suppose a binary variable $Z$ satisfies the following key conditions

  * •

(Stability of association)

$\displaystyle E(Y_{1}|Z=1)-E(Y_{1}|Z=0)=E(Y_{0}|Z=1)-E(Y_{0}|Z=0)$ (4)

  * •

(Bias homogeneity)

$\displaystyle E(Y_{1}-Y_{0}|Z=1,R=1)-E(Y_{1}-Y_{0}|Z=1,R=0)$
$\displaystyle=E(Y_{1}-Y_{0}|Z=0,R=1)-E(Y_{1}-Y_{0}|Z=0,R=0)$ (5)

  * •

(Predictive of missingness)

$\displaystyle Pr(R=1|Z=1)-Pr(R=1|Z=0)\neq 0$ (6)

The first condition requires that the association between $Y_{1}$ and $Z$ in
the full population is the same as the association between $Y_{0}$ and $Z$.
Similar to (2), this assumption cannot be tested, since we do not have access
to $Y_{1}$ for everyone. It requires choosing a covariate for which the
association with the outcome is likely to be stable over time, and is not
subject to temporal trends.

Assumption (4) is a type of ‘exclusion restriction’, since we exclude an
association between the mean of $Y_{1}-Y_{0}$ and $Z$. It is related to the
assumption in the causal inference literature that an instrumental variable
(IV) cannot have a direct causal effect on the outcome. Tchetgen Tchetgen and
Wirth, (2017) previously considered the setting with an outcome missing not at
random, and showed how an IV for missing outcome can be used to correct for
bias. They relied on the exclusion restriction that $E(Y_{1}|Z=1)=(Y_{1}|Z=0)$
e.g. that it is not associated with the outcome in the full data. However, in
practice it might be hard to find variables for which this assumption is
satisfied. Our assumption is close to Tchetgen Tchetgen and Wirth’s, but is
made after ‘differencing’ the outcome; hence we can allow for $Z$ to be
associated with $Y_{1}$ in the full data, so long as its association matches
that with $Y_{0}$. The link with the causal inference literature has lead to
this type of approach being described a ‘bespoke IV’ strategy, where a
standard confounder $Z$ measured at baseline is rendered as an IV (Richardson
and Tchetgen Tchetgen,, 2021; Dukes et al.,, 2022). We use the same
terminology here.

Assumption (• ‣ 3) states that the bias in the DiD analysis is the same for
those with $Z=1$ as for those with $Z=0$. Since we do not know the selection
bias, this is also not an assumption that can be empirically verified. Hence,
the assumption (• ‣ 3) allows for deviations from the parallel trends
assumption (2), so long as these deviations do not depend on $Z$. Assumption
(6) requires $Z$ to be predictive of missingness; unlike the other
assumptions, this condition can be verified empirically. Therefore, a
candidate $Z$ could potentially be a covariate that would usually be
conditioned on in a MAR analysis, that is associated with both the outcome and
the probability of missingness. A good choice of $Z$ is additionally required
to have a stable association with the outcome over time, and for the DiD bias
to be the same within each of its levels. It may be that assumption (• ‣ 3) is
the hardest to justify on scientific grounds; assumption (4) could potentially
be motivated by external data or existing scientific knowledge.

In the Appendix, we show that assumptions (4)-(6) are sufficient to identify
$E(Y_{1})$. This result depends on the fact that we are able (under these
assumptions) to represent the observed data as

$\displaystyle E(Y_{1}-Y_{0}|Z,R=1)$
$\displaystyle=E(Y_{1}-Y_{0})+\\{E(Y_{1}-Y_{0}|R=1)-E(Y_{1}-Y_{0}|R=0)\\}Pr(R=0|Z)$

Notice that the left hand side is a conditional expectation that involves only
the observed data. The right hand side involve two quantities that are
unknown: $E(Y_{1}-Y_{0})$ and $E(Y_{1}-Y_{0}|R=1)-E(Y_{1}-Y_{0}|R=0)$ (one can
straightforwardly calculate $Pr(R=0|Z)$ without using data on $Y_{1}$ or
$Y_{0}$). This is tantamount to a regression problem, where $Y_{1}-Y_{0}$ is
the outcome, $Pr(R=0|Z)$ is the regressor, and we restrict only to the
complete cases; $E(Y_{1}-Y_{0})$ and $E(Y_{1}-Y_{0}|R=1)-E(Y_{1}-Y_{0}|R=0)$
are the intercept and slope parameters respectively. This motivates the
following procedure for estimating $E(Y_{1})$:

  1. 1.

Obtain the predictions $Pr(R=1|Z)$ using e.g. logistic regression.

  2. 2.

Fit a linear regression model using the complete cases, with $Y_{1}-Y_{0}$ as
the outcome, $Pr(R=1|Z)$ as a predictor, and $-E(Y_{0})$ as an offset:

$\displaystyle E(Y_{1}-Y_{0}|Z,R=1)=-E(Y_{0})+\tau_{0}+\tau_{1}Pr(R=0|Z)$ (7)

  3. 3.

Estimate $E(Y_{1})$ as the the estimated value of $\tau_{0}$.

In the regression model (7), the parameters $\tau_{0}$ and $\tau_{1}$
correspond with $E(Y_{1})$ and the DiD bias respectively.

##  4 Data analysis: Framingham Heart Study

In order to illustrate the aforementioned methods we considered a subsample of
data from the Framingham Heart Study. This is a landmark prospective study in
cardiovascular epidemiology, conducted using individuals from Framingham,
Massachusetts; participants are followed up every 2 to 8 years. Our sample
included measurements on clinical examination data from when the first group
of subjects were enrolled in 1956, as well as measurements on the same
variables from their third examination in 1968. Out of the 5,209 participants
who initially enrolled, the subsample consisted of 400 individuals, who were
sampled such that the dataset was balanced in terms of sex and smoking status;
the data set was obtained from the ‘sur’ package in R (Weinberg et al.,,
2020).

Our parameter of interest was average systolic blood pressure (SBP) at the
third examination; this was missing for 92 individuals (23%). Complete data
was collected on SBP at the first examination. We compare several approaches;
an analysis under a MCAR assumption, based on comparing average outcomes in
the complete cases; a LOCF analysis, where missing outcomes in each arm are
imputed using baseline outcome; a MAR analysis, where baseline outcomes are
assumed to render missingness and outcomes independent; a DiD analysis, based
on (2); and a bespoke IV analysis. For the final approach, we chose age as a
bespoke instrument. Age was highly correlated with missingness; participants
with missing data were on average 4.49 years older than the complete cases
(95% CI: 2.56, 6.41). We implemented two bespoke IV analyses, where we varied
the specification of the model for $Pr(R=1|Z)$. The first approach involved a
logistic regression with a linear term for age (BSIV), whereas the second
included an additional quadratic term in the model (BSIV-Q)

Figure 3: Estimates of mean SBP from the Framingham Heart Study for those with
and without missing data at third examination. The black circles and line
describe the trajectory for the complete cases ($R=1$). The other lines
indicate the different trajectories for those with $R=0$, as the missing data
assumptions are varied.

In Figure 3, we give a graphical illustration of how the different approaches
estimate $E(Y_{1}|R=0)$. The bold black line indicates the change in outcomes
between first and third examination in the complete cases. It is evident that
baseline values tended to be higher in those who then had missing data at
follow up. One can see that the LOCF approach would give similar estimates of
$E(Y_{1}|R=1)$ as an MCAR analysis, since mean SBP at third examination is
close to the mean of SBP at baseline in those with missing data at follow up.
The MAR and DiD analysis suggest that SBP would tend to increase over time in
those with $R=0$; the DiD approach yields the highest estimate of
$E(Y_{1}|R=0)$, and may be considered most plausible if one expects that SBP
typically increases in individuals over time. A test of the null hypothesis
that $E(Y_{1}-Y_{0}|R=1)$ equals zero yielded a $p$-value $<0.01$, suggesting
that (under a parallel trends assumption) a LOCF analysis is biased. The
bespoke IV analysis yielded the lowest estimate of $E(Y_{1}|R=0)$,
illustrating how this approach may sometimes yield results that contradict
those of the DiD methods. Here, bespoke IV assumes that the trend in SBP is
the same (in the full population) does not depend on age; this would be
violated if in truth older patients tended to see a more dramatic rise in SBP
compared with younger patients. It may be in this case that the parallel
trends assumption is more plausible. However, including a quadratic term led
to results more in line with those for the other methods, suggesting some
sensitivity to model specification.

The estimates of mean SBP at third examination, along with nonparametric
bootstrap 95% confidence intervals, are provided in Table 1. The point
estimates are fairly close in magnitude, likely due to the relatively modest
proportion of missing values. One can see that the BSIV intervals are wider
than for the other approaches, overlapping with those from the MAR and DiD
analyses; this should caveat the observed differences in the point estimates.

Table 1: Estimates of mean SBP at third examination under different missing data assumptions in the Framingham Heart Study. 95% CI: 95% Confidence Interval (based on the nonparametric bootstrap - 2,000 bootstrap samples used) Analysis | Estimate |  $95\%$ CI
---|---|---
MCAR | 137.99 | (135.48,140.51)
LOCF | 138.49 | (136.23,140.76)
MAR | 139.92 | (137.31,142.53)
DiD | 140.92 | (138.32,143.52)
BSIV | 136.86 | (129.5,144.22)
BSIV-Q | 138.91 | (131.63,146.19)

##  5 Longitudinal clinical trials with monotone missing data

###  5.1 Difference-in-Differences

Having introduced the DiD and bespoke IV strategies for handling a single
missing outcome, we will now extend our reasoning for clinical trials subject
to monotone missing outcome data. Let $A$ now denote a binary treatment
variable of interest. Then we will consider a clinical trial subject to drop-
out, where repeated measurements have been collected of the outcome at $T$
time points, in addition to a baseline measurement $Y_{0}$. Then the observed
data takes the form $\\{Y_{0},A,R_{1},R_{1}Y_{1},...,R_{T},R_{T}Y_{T}\\}$; in
what follows, let $\bar{R}_{t}=\\{R_{1},...,R_{t}\\}$. We will assume a
monotone missing data structure, such that once an individual drops out of the
study, no further data is collected on them; in other words, if $R_{j}=0$ at
time $t=j$ for a participant, then $R_{k}=0$ for all $k>j$. Interest is in the
causal contrast

$E(Y^{1}_{T}-Y^{0}_{T})$

where $Y_{T}^{a}$ is the potential outcome at time $T$ if an individual were
given treatment $A=a$. By virtue of randomisation, this quantity is equal to
$E(Y_{T}|A=1)-E(Y_{T}|A=0)$. However, due to the presence of missing data, one
can still not readily compute this from the observed data without further
assumptions. In what follows, for simplicity we will consider identification
of $E(Y^{1})=E(Y_{T}|A=1)$; results along the same lines follow for
$E(Y^{0})$.

One can show that the generalised version of the DiD parallel trend assumption
can be given as

$\displaystyle
E(Y_{T}-Y_{t-1}|\bar{R}_{t}=1,A=1)=E(Y_{T}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,A=1)$
(8)

for $t=1,...,T$. Interpretation of this assumption is more subtle than in the
previous sections, because at each time $t<T$, the group for whom
$\bar{R}_{t}=1$ will include a mix of the complete cases $(\bar{R}_{T}=1)$ and
individuals who subsequently drop-out. As a result, neither the left (unless
$t=T$) nor the right hand side of (8) can be directly calculated from the
observed data.

Figure 4: The top line indicates the outcome trajectory in the complete cases.
The middle line indicates the trajectory in participants for whom
$R_{2}=1,R_{1}=1$ under (2); the full black (dashed grey) line represents the
observed (unobserved) data. The bottom line represents the trajectory for
participants for whom data was missing subsequent to baseline under (2). Here,
$Pr(R_{2}=1|R_{1}=1,A=1)=Pr(R_{2}=0|R_{1}=1,A=1)=0.5$.

To develop some understanding of this assumption, consider first the group of
patients with missing data only at the very end of follow up. For the sake of
simplicity, we let $T=2$. For these individuals, the above restriction implies
that

$\displaystyle E(Y_{2}|R_{2}=0,R_{1}=1,A=1)=$ $\displaystyle
E(Y_{1}|R_{2}=0,R_{1}=1,A=1)+E(Y_{2}-Y_{1}|\bar{R}_{2}=1,A=1)$ (9)

such that one can impute $Y_{T}$ in these individuals based on adding the
trend $Y_{2}-Y_{1}$ (as observed in the complete cases) to their outcomes at
time 1. So far, this straightforwardly follows what was described in Section
2. For individuals who drop-out before $Y_{1}$ is measured ($R_{1}=0$), one
can show that assumption (8) implies that

$\displaystyle E(Y_{2}|R_{1}=0,A=1)$ $\displaystyle=E(Y_{0}|R_{1}=0,A=1)$
$\displaystyle\quad+E(Y_{2}-Y_{0}|\bar{R}_{2}=1,A=1)Pr(R_{2}=1|R_{1}=1,A=1)$
$\displaystyle\quad+\\{E(Y_{2}-Y_{1}|\bar{R}_{2}=1,A=1)+E(Y_{1}-Y_{0}|R_{2}=0,R_{1}=1,A=1)\\}Pr(R_{2}=0|R_{1}=1,A=1)$

Hence we impute their outcomes using the baseline $Y_{0}$, plus a weighted
average of the observed and predicted trends $Y_{2}-Y_{0}$. The parallel
trends assumption allows us to predict the trend in those for whom $Y_{1}$ was
observed but $Y_{2}$ was not as
$E(Y_{2}-Y_{1}|\bar{R}_{2}=1,A=1)+E(Y_{1}-Y_{0}|R_{2}=0,R_{1}=1,A=1)$, where
information is again borrowed from the complete cases. Figure 4 displays some
potential observed and predicted trajectories in the complete cases (top),
those with $R_{2}=0,R_{1}=1$ (middle) and $R_{2}=R_{1}=0$ (bottom). In
particular, the bottom dashed line illustrates the predicted trajectory in the
group with data missing subsequent to baseline; it is a weighted average of
the trajectories in the two other missing data groups.

There are multiple ways to implement the sequential DiD approach. In the
Appendix we show that under assumption (8),

$\displaystyle E(Y_{T}|A=1)$ $\displaystyle=E(Y_{T}|\bar{R}_{T}=1,A=1)$
$\displaystyle\quad-\sum^{T}_{t=1}\\{E(Y_{t-1}|\bar{R}_{t}=1,A=1)-E(Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,A=1)\\}Pr(R_{t}=0|\bar{R}_{t-1}=1,A=1)$

where the right hand side involves terms that can be readily computed from the
data. Loosely, we subtract an estimate of the bias (using the immediate
outcomes across different time points) from the naïve complete case analysis.
Alternatively, one can impute the outcomes working backwards in time using
predicted trajectories as hinted at in the previous paragraph. A third way to
implement the method is via the representation

$E(Y_{T}|A=1)=E(Y_{0}|A=1)+\sum^{T}_{t=1}E(Y_{t}-Y_{t-1}|\bar{R}_{t}=1,A=1)$

such that one can calculate the average difference $Y_{t}-Y_{t-1}$ in those
with $R_{t}=1$ across different time points and then sum them, adding on
$E(Y_{0}|A=1)$.

An alternative assumption to (8) would be

$\displaystyle
E(Y_{T}-Y_{t-1}|\bar{R}_{T}=1,A=1)=E(Y_{T}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,A=1)$
(10)

where in the conditional expectation on the left hand side, we now condition
on being a complete case. Note that the left hand side can now be readily
computed from the data, which was not the case with (8). An analysis based on
(10) would impute missing outcomes using the slopes based on the complete
cases alone, rather than harnessing additional information from individuals
who drop-out at later time points. In general, we view assumption (8) as more
plausible; nevertheless, it is (10) that appears to have been applied before
(Aisen et al.,, 2003), perhaps because it is more intuitive and can be easily
implemented. Referring back to Figure 4, under (10), the bottom dashed line
would be flatter, as it would be based entirely on the trajectory in the
complete cases, rather than borrowing information from the steeper trajectory
in participants with partially missing data.

A weakness of analyses developed either under assumptions (8) and (10) is that
they may lack plausibility when there are big gaps in time between measurement
of $Y_{t}$ and earlier time points. We note that there is a growing literature
on the failure of parallel trends in the causal inference literature on the
DiD design (Roth et al.,, 2022), and many of these ideas may also be useful in
this context.

###  5.2 Bespoke instrumental variables

The bespoke instrumental variable method can also be extended to the
longitudinal set-up. We will now assume that one has access to a time-
dependent variable $Z=\\{Z_{1},...,Z_{T}\\}$, where we assume the temporal
ordering $\\{Z_{t},R_{t},R_{t}Y_{t}\\}$ for all $t=1,...,T$. We also need to
extend the three previous bespoke IV conditions from Section 3:

  * •

(Stability of association) For all $t=0,...,T$,

$\displaystyle
E(Y_{T}|\bar{R}_{t-1}=1,Z_{t}=1,\bar{Z}_{t-1},A=1)-E(Y_{T}|\bar{R}_{t-1}=1,Z_{t}=0,\bar{Z}_{t-1},A=1)$
$\displaystyle=E(Y_{t-1}|\bar{R}_{t-1}=1,Z_{t}=1,\bar{Z}_{t-1},A=1)-E(Y_{t-1}|\bar{R}_{t-1}=1,Z_{t}=0,\bar{Z}_{t-1},A=1)$
(11)

  * •

(Bias homogeneity) For all $t=0,...,T$,

$\displaystyle E(Y_{T}-Y_{t-1}|\bar{R}_{t}=1,Z_{t}=1,\bar{Z}_{t-1},A=1)$
$\displaystyle\quad-E(Y_{T}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,Z_{t}=1,\bar{Z}_{t-1},A=1)$
$\displaystyle=E(Y_{T}-Y_{t-1}|\bar{R}_{t}=1,Z_{t}=0,\bar{Z}_{t-1},A=a)$
$\displaystyle\quad-E(Y_{T}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,Z_{t}=0,\bar{Z}_{t-1},A=a)$
(12)

  * •

(Predictive of missingness) For all $t=0,...,T$,

$\displaystyle
Pr(R_{t}=1|\bar{R}_{t-1}=1,Z_{t}=1,\bar{Z}_{t-1},A=a)-Pr(R_{t}=1|\bar{R}_{t-1}=1,Z_{t}=0,\bar{Z}_{t-1},A=a)$
$\displaystyle\neq 0$ (13)

Conditions (11), (• ‣ 5.2) and (• ‣ 5.2) generalise (4), (• ‣ 3) and (6) in
the same way that the sequential parallel trend assumption (8) generalises
(2). As in that case, for any $t$ up to $T-1$, the assumptions refer to a
subgroup of participants, some of whom will subsequently drop-out. Assumption
(• ‣ 5.2) can be see as a weakening of the sequential parallel trends
assumption (8), where parallel trends at time $t$ is allowed to be violated so
long as the violation does not depend on $Z_{t}$.

In the Appendix, we show that when assumptions (11)-(• ‣ 5.2) hold,

$\displaystyle E(Y_{T}|A=1)$
$\displaystyle\quad+\sum^{T}_{t=1}\\{E(Y_{t}-Y_{t-1}|\bar{R}_{t}=1,\bar{Z}_{t-1}=\bar{z}_{t-1},A=1)-E(Y_{t}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,\bar{Z}_{t-1}=\bar{z}_{t-1},A=1)\\}$
$\displaystyle\quad\times
Pr(R_{t}=1|\bar{R}_{t-1}=1,\bar{Z}_{t}=\bar{z}_{t},A=1)$
$\displaystyle=E(Y_{0}|A=1)+\sum^{T}_{t=1}E(Y_{t}-Y_{t-1}|\bar{R}_{t}=1,\bar{Z}_{t}=\bar{z}_{t},A=1)$

The left hand side of the equality involves the unknown $E(Y_{T}|A=1)$ and
$E(Y_{t}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,\bar{Z}_{t-1}=\bar{z}_{t-1},A=1)$,
whereas the right hand side involves quantities that can be obtained from the
data. Below, we use this result as a recipe for implementation:

  1. 1.

For each $t=1,...,T-1$, fit a model for

$E(Y_{t}-Y_{t-1}|\bar{R}_{t}=1,\bar{Z}_{t},A=1)$

by regressing $Y_{t}-Y_{t-1}$ on $\bar{Z}_{t}$ in those with $\bar{R}_{t}=1$
and $A=1$.

  2. 2.

In the complete cases ($\bar{R}_{T}=1$), add together the model predictions in
the following way:

$\omega(\bar{Z}_{T-1})=E(Y_{0}|A=1)+\sum^{T-1}_{t}E(Y_{t}-Y_{t-1}|\bar{R}_{t}=1,\bar{Z}_{t},A=1)$

  3. 3.

For each $t=1,...,T$, fit a regression model for the probability of remaining
in the risk set:

$Pr(R_{t}=1|\bar{R}_{t-1}=1,\bar{Z}_{t},A=1)$

  4. 4.

Fit the following regression model in the complete cases in the treated arm:

$\displaystyle E(Y_{T}-Y_{T-1}|\bar{R}_{T}=1,\bar{Z}_{t},A=1)$
$\displaystyle=-\omega(\bar{Z}_{T-1})+\tau_{0}+\sum^{T}_{t}\tau_{t}Pr(R_{t}=0|\bar{R}_{t-1}=1,\bar{Z}_{t}=\bar{z}_{t},A=1)$

where $-\omega(\bar{Z}_{t})$ is included as an offset. Then $\tau_{0}$ is an
estimate of $E(Y_{T}|A=1)$.

At the final stage, one can also allow for interactions between the regressor
$Pr(R_{t}=0|\bar{R}_{t-1}=1,\bar{Z}_{t}=\bar{z}_{t},A=1)$ and $\bar{Z}_{t-1}$
for any $t=1,...,T$, but the interaction cannot involve $Z_{t}$ by assumption
(• ‣ 5.2).

##  6 Discussion

In this approaches, we have discussed two alternatives to the MAR assumption
as a means of obtaining identification in settings with missing data. The
first approach extends the DiD design from causal inference, via a missing
data parallel trends type assumption. The second approach allows for a
violation of this assumption, and is valid under alternative conditions. It
requires collecting data on an auxiliary covariate, but such covariate data is
often collected in order to make the MAR assumption more plausible. We have
limited our attention to settings with monotone missing data. Non-monotone
missingness structures pose additional challenges for identification, which
are beyond the scope of this article (Robins and Gill,, 1997).

In DiD analyses, one often makes a parallel trends assumptions conditional on
measured covariates (Abadie,, 2005). This enables one to utilise the DiD
approach whilst controlling for measured confounders. A similar approach could
be taken in the analyses here; indeed, in the appendix we have generalised the
results in the main paper to allow for covariate adjustment. A full study of
the different estimators that may be available in this setting - such as
likelihood-based, weighting-based and doubly robust methods - forms an
interesting topic for additional work.

## References

  * Abadie, (2005) Abadie, A. (2005).  Semiparametric difference-in-differences estimators.  The Review of Economic Studies, 72(1):1–19.
  * Aisen et al., (2003) Aisen, P. S., Schafer, K. A., Grundman, M., Pfeiffer, E., Sano, M., Davis, K. L., Farlow, M. R., Jin, S., Thomas, R. G., Thal, L. J., et al. (2003).  Effects of rofecoxib or naproxen vs placebo on alzheimer disease progression: a randomized controlled trial.  Jama, 289(21):2819–2826.
  * Allison, (2012) Allison, P. D. (2012).  Handling missing data by maximum likelihood.  In SAS global forum, volume 2012, pages 1038–21.
  * Card and Krueger, (1993) Card, D. and Krueger, A. B. (1993).  Minimum wages and employment: A case study of the fast food industry in new jersey and pennsylvania.
  * Carpenter et al., (2007) Carpenter, J. R., Kenward, M. G., and White, I. R. (2007).  Sensitivity analysis after multiple imputation under missing at random: a weighting approach.  Statistical methods in medical research, 16(3):259–275.
  * Diggle and Kenward, (1994) Diggle, P. and Kenward, M. G. (1994).  Informative drop-out in longitudinal data analysis.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 43(1):49–73.
  * Dukes et al., (2022) Dukes, O., Richardson, D., Shahn, Z., and Tchetgen, E. T. (2022).  Semiparametric bespoke instrumental variables.  arXiv preprint arXiv:2204.04119.
  * Ghanem et al., (2022) Ghanem, D., Hirshleifer, S., Kedagni, D., and Ortiz-Becerra, K. (2022).  Correcting attrition bias using changes-in-changes.  arXiv preprint arXiv:2203.12740.
  * Horvitz and Thompson, (1952) Horvitz, D. G. and Thompson, D. J. (1952).  A generalization of sampling without replacement from a finite universe.  Journal of the American statistical Association, 47(260):663–685.
  * Kenward and Molenberghs, (2009) Kenward, M. G. and Molenberghs, G. (2009).  Last observation carried forward: a crystal ball?  Journal of biopharmaceutical statistics, 19(5):872–888.
  * Lachin, (2016) Lachin, J. M. (2016).  Fallacies of last observation carried forward analyses.  Clinical trials, 13(2):161–168.
  * Lechner et al., (2011) Lechner, M. et al. (2011).  The estimation of causal effects by difference-in-difference methods, volume 4.  Now Hanover, MA.
  * Little and Rubin, (2019) Little, R. J. and Rubin, D. B. (2019).  Statistical analysis with missing data, volume 793.  John Wiley & Sons.
  * Molnar et al., (2008) Molnar, F. J., Hutton, B., and Fergusson, D. (2008).  Does analysis using “last observation carried forward” introduce bias in dementia research?  Cmaj, 179(8):751–753.
  * Pearl, (1995) Pearl, J. (1995).  Causal diagrams for empirical research.  Biometrika, 82(4):669–688.
  * Petersen et al., (2005) Petersen, R. C., Thomas, R. G., Grundman, M., Bennett, D., Doody, R., Ferris, S., Galasko, D., Jin, S., Kaye, J., Levey, A., et al. (2005).  Vitamin e and donepezil for the treatment of mild cognitive impairment.  New England Journal of Medicine, 352(23):2379–2388.
  * Richardson and Tchetgen Tchetgen, (2021) Richardson, D. B. and Tchetgen Tchetgen, E. J. (2021).  Bespoke instruments: A new tool for addressing unmeasured confounders.  American journal of epidemiology.
  * Robins, (1986) Robins, J. (1986).  A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect.  Mathematical modelling, 7(9-12):1393–1512.
  * Robins and Gill, (1997) Robins, J. M. and Gill, R. D. (1997).  Non-response models for the analysis of non-monotone ignorable missing data.  Statistics in medicine, 16(1):39–56.
  * Robins et al., (2000) Robins, J. M., Rotnitzky, A., and Scharfstein, D. O. (2000).  Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models.  In Statistical models in epidemiology, the environment, and clinical trials, pages 1–94. Springer.
  * Robins et al., (1994) Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1994).  Estimation of regression coefficients when some regressors are not always observed.  Journal of the American statistical Association, 89(427):846–866.
  * Robins et al., (1995) Robins, J. M., Rotnitzky, A., and Zhao, L. P. (1995).  Analysis of semiparametric regression models for repeated outcomes in the presence of missing data.  Journal of the american statistical association, 90(429):106–121.
  * Roth et al., (2022) Roth, J., Sant’Anna, P. H., Bilinski, A., and Poe, J. (2022).  What’s trending in difference-in-differences? a synthesis of the recent econometrics literature.  arXiv preprint arXiv:2201.01194.
  * Rotnitzky and Robins, (1997) Rotnitzky, A. and Robins, J. (1997).  Analysis of semi-parametric regression models with non-ignorable non-response.  Statistics in medicine, 16(1):81–102.
  * Shao and Zhong, (2003) Shao, J. and Zhong, B. (2003).  Last observation carry-forward and last observation analysis.  Statistics in medicine, 22(15):2429–2441.
  * Sofer et al., (2016) Sofer, T., Richardson, D. B., Colicino, E., Schwartz, J., and Tchetgen Tchetgen, E. J. (2016).  On negative outcome control of unobserved confounding as a generalization of difference-in-differences.  Statistical science: a review journal of the Institute of Mathematical Statistics, 31(3):348.
  * Tchetgen Tchetgen and Wirth, (2017) Tchetgen Tchetgen, E. J. and Wirth, K. E. (2017).  A general instrumental variable framework for regression analysis with outcome missing not at random.  Biometrics, 73(4):1123–1131.
  * Weinberg et al., (2020) Weinberg, S. L., Harel, D., and Abramowitz, S. K. (2020).  Statistics using R: An integrative approach.  Cambridge University Press.
  * Wu and Carroll, (1988) Wu, M. C. and Carroll, R. J. (1988).  Estimation and comparison of changes in the presence of informative right censoring by modeling the censoring process.  Biometrics, pages 175–188.

##  Appendix A Justification of approaches

###  A.1 Difference in differences

In what follows, we will give results for a general data structure
$\\{Y_{0},L_{1},A,R_{1},R_{1}Y_{1},...,,L_{T},R_{T},R_{T}Y_{T}\\}$, where
$\bar{L}_{T}=\\{L_{1},...,L_{T}\\}$ is a time-varying variable that satisfies
the conditional parallel trends assumption:

$\displaystyle
E(Y_{T}-Y_{t-1}|\bar{R}_{t}=1,\bar{L}_{t},A=a)=E(Y_{T}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,\bar{L}_{t},A=a)$
(14)

for all $t=1,...,T$ and $a=0,1$. Then we will show how assumption (14) can be
used in order to identify $E(Y_{T}^{a})$. In addition to (14), we will invoke
the following assumptions that are standard in the causal inference/missing
data literature:

  * •

(Consistency) If $A=a$, then $Y=Y^{a}$.

  * •

(Positivity) $Pr(A=a|L_{1}=l_{1})>0$ for all $a$ and $l_{1}$ where
$f(l_{1})\neq 0$. Also if
$Pr(R_{t}=1|\bar{R}_{t-1}=1,\bar{L}_{t}=\bar{l}_{t},A=a)>0$ for all $a$ and
$l_{t}$ where $f(\bar{R}_{t-1}=1,l_{t},a)\neq 0$, at each $t=1,..,T$.

  * •

(Conditional exchangeability) $A\perp\\!\\!\\!\perp Y^{a}|L_{1}=l_{1}\forall
a$.

These results allow for $A$ to be non-randomised, so long as the baseline
covariates $L_{1}$ are sufficient to account for confounding.

Using a decomposition of the conditional mean
$E(Y_{T}|\bar{R}_{T}=1,\bar{L}_{T}=\bar{l}_{T},A=a)$ similar to that in
Tchetgen Tchetgen and Wirth, (2017), we have

$\displaystyle E(Y_{T}|\bar{R}_{T}=1,\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle=\\{E(Y_{T}|\bar{R}_{T}=1,\bar{L}_{T}=\bar{l}_{T},A=a)-E(Y_{T}|R_{T}=0,\bar{R}_{T-1}=1,\bar{L}_{T}=\bar{l}_{T},A=a)\\}\\{1-\pi_{T}(\bar{l}_{T})\\}$
$\displaystyle\quad+\delta_{T}(\bar{l}_{T})-E\\{\delta_{T}(\bar{L}_{T})|\bar{R}_{T-1}=1,\bar{L}_{T-1}=\bar{l}_{T-1}\\}$
$\displaystyle\quad+E(Y_{T}|\bar{R}_{T-1}=1,\bar{L}_{T-1}=\bar{l}_{T-1},A=a)$

where we use the notation

$\displaystyle\pi_{t}(\bar{L}_{t})$
$\displaystyle=Pr(R_{t}=1|\bar{R}_{t-1}=1,\bar{L}_{t},A=a)$
$\displaystyle\delta_{t}(\bar{l}_{t})$
$\displaystyle=E(Y_{T}|\bar{R}_{t-1}=1,\bar{L}_{t}=l_{t},A=a)-E(Y_{T}|\bar{R}_{t-1}=1,L_{t}=0,\bar{L}_{T-1}=\bar{l}_{T-1},A=a)$

Note that (14) implies that

$\displaystyle E(Y_{T}|\bar{R}_{T}=1,\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle=\\{E(Y_{T-1}|\bar{R}_{T}=1,\bar{L}_{T}=\bar{l}_{T},A=a)-E(Y_{T-1}|R_{T}=0,\bar{R}_{T-1}=1,\bar{L}_{T}=\bar{l}_{T},A=a)\\}\\{1-\pi_{T}(\bar{l}_{T})\\}$
$\displaystyle\quad+\delta_{T}(\bar{l}_{T})-E\\{\delta_{T}(\bar{L}_{T})|\bar{R}_{T-1}=1,\bar{L}_{T-1}=\bar{l}_{T-1}\\}$
$\displaystyle\quad+E(Y_{T}|\bar{R}_{T-1}=1,\bar{L}_{T-1}=\bar{l}_{T-1},A=a)$

and one can use a similar expansion for
$E(Y_{T}|\bar{R}_{T-1}=1,\bar{L}_{T-1}=\bar{l}_{T-1},A=a)$. Working backwards
in time down to $t=1$, we have

$\displaystyle E(Y_{T}|R_{1}=1,\bar{L}_{1}=\bar{l}_{1},A=a)$
$\displaystyle=\\{E(Y_{0}|R_{1}=1,L_{1}=l_{1},A=a)-E(Y_{0}|R_{1}=0,L_{1}=l_{1},A=a)\\}\\{1-\pi_{1}(l_{1})\\}$
$\displaystyle\quad+E(Y_{T}|L_{1}=l_{1},A=a)$

Then adding all the preceding terms in the series together and re-arranging
gives

$\displaystyle E(Y_{T}|L_{1}=l_{1},A=a)$
$\displaystyle=E(Y_{T}|\bar{R}_{T}=1,\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle\quad-\sum^{T}_{t=1}\\{E(Y_{t-1}|\bar{R}_{t}=1,\bar{L}_{t}=\bar{l}_{t},A=a)-E(Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,\bar{L}_{t}=\bar{l}_{t},A=a)\\}\\{1-\pi_{t}(\bar{l}_{t})\\}$
$\displaystyle\quad-\sum^{T}_{t=2}[\delta_{t}(\bar{l}_{t})-E\\{\delta_{t}(\bar{L}_{t})|\bar{R}_{t-1}=1,\bar{L}_{T-1}=\bar{l}_{T-1},A=a\\}]$

where one can verify that the right hand side depends only on the observed
data. Under the standard structural assumptions given above, $E(Y_{T}^{a})$
can then be identified as

$E(Y_{T}^{a})=E\\{E(Y_{T}|L_{1},A=a)\\}$

(Robins,, 1986).

###  A.2 Bespoke instrumental variables

Consider now a general data structure
$\\{Y_{0},L_{1},Z_{1},A,R_{1},R_{1}Y_{1},...,,L_{T},Z_{T},R_{T},R_{T}Y_{T}\\}$,
where $\bar{Z}_{t}$ satisfies the following restrictions:

  * •

(Stability of association) For all $t=0,...,T$ and $a=0,1$,

$\displaystyle
E(Y_{T}-Y_{t-1}|\bar{R}_{t-1}=1,Z_{t}=1,\bar{Z}_{t-1},\bar{L}_{t},A=a)$
$\displaystyle=E(Y_{T}-Y_{t-1}|\bar{R}_{t-1}=1,Z_{t}=0,\bar{Z}_{t-1},\bar{L}_{t},A=a)$
(15)

  * •

(Bias homogeneity) For all $t=0,...,T$ and $a=0,1$,

$\displaystyle
E(Y_{T}-Y_{t-1}|\bar{R}_{t}=1,Z_{t}=1,\bar{Z}_{t-1},\bar{L}_{t},A=a)$
$\displaystyle\quad-E(Y_{T}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,Z_{t}=1,\bar{Z}_{t-1},\bar{L}_{t},A=a)$
$\displaystyle=E(Y_{T}-Y_{t-1}|\bar{R}_{t}=1,Z_{t}=0,\bar{Z}_{t-1},\bar{L}_{t},A=a)$
$\displaystyle\quad-E(Y_{T}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,Z_{t}=0,\bar{Z}_{t-1},\bar{L}_{t},A=a)$
(16)

  * •

(Predictive of missingness) For all $t=0,...,T$ and $a=0,1$,

$\displaystyle
Pr(R_{t}=1|\bar{R}_{t-1}=1,Z_{t}=1,\bar{Z}_{t-1},\bar{L}_{t},A=a)-Pr(R_{t}=1|\bar{R}_{t-1}=1,Z_{t}=0,\bar{Z}_{t-1},\bar{L}_{t},A=a)$
$\displaystyle\neq 0\quad\textrm{ almost surely.}$ (17)

Then using a similar expansion as in the previous proof,

$\displaystyle
E(Y_{T}-Y_{T-1}|\bar{R}_{T}=1,\bar{Z}_{T}=\bar{z}_{T},\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle=\\{E(Y_{T}-Y_{T-1}|\bar{R}_{T}=1,\bar{Z}_{T}=\bar{z}_{T},\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle\quad-E(Y_{T}-Y_{T-1}|R_{T}=0,\bar{R}_{T-1}=1,\bar{Z}_{T}=\bar{z}_{T},\bar{L}_{T}=\bar{l}_{T},A=a)\\}\\{1-\pi_{T}(\bar{z}_{T},\bar{l}_{T})\\}$
$\displaystyle\quad+E(Y_{T}-Y_{T-1}|\bar{R}_{T-1}=1,\bar{Z}_{T}=\bar{z}_{T},\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle=\\{E(Y_{T}-Y_{T-1}|\bar{R}_{T}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle\quad-E(Y_{T}-Y_{T-1}|R_{T}=0,\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T}=\bar{l}_{T},A=a)\\}\\{1-\pi_{T}(\bar{z}_{T},\bar{l}_{T})\\}$
$\displaystyle\quad+E(Y_{T}-Y_{T-1}|\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T}=\bar{l}_{T},A=a)$

by application of (15)-(• ‣ A.2). As in Tchetgen Tchetgen and Wirth, (2017),
this suffices to identify
$E(Y_{T}-Y_{T-1}|\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T}=\bar{l}_{T},A=a)$,
and therefore
$E(Y_{T}|\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T}=\bar{l}_{T},A=a)$.
This is because for fixed $\bar{L}_{T}$ and $\bar{Z}_{T-1}$,
$E(Y_{T}-Y_{T-1}|\bar{R}_{T}=1,\bar{Z}_{T}=\bar{z}_{T},\bar{L}_{T}=\bar{l}_{T},A=a)$
has two degrees of freedom, as does the right hand side of the second equality
above. Note that $\pi_{T}(\bar{z}_{T},\bar{l}_{T})$ is considered as known
here. Further,

$\displaystyle
E(Y_{T}-Y_{T-1}|\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T}=\bar{l}_{T},A=a)$
$\displaystyle=\delta^{*}_{T,T-1}(\bar{z}_{T-1},\bar{l}_{T})-E\\{\delta^{*}_{T,T-1}(\bar{z}_{T-1},\bar{L}_{T})|\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T-1}=\bar{l}_{T-1},A=a\\}$
$\displaystyle\quad+E(Y_{T}-Y_{T-1}|\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T-1}=\bar{l}_{T-1},A=a)$

where

$\displaystyle\delta^{*}_{T,t}(\bar{z}_{T-1},\bar{l}_{t})=$ $\displaystyle
E(Y_{T}-Y_{t}|\bar{R}_{t-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{t}=l_{t},A=a)$
$\displaystyle-E(Y_{T}-Y_{t}|\bar{R}_{t-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},L_{t}=0,\bar{L}_{T-1}=\bar{l}_{T-1},A=a)$

One can then extend this reasoning to identify
$E(Y_{T}|\bar{R}_{t}=1,\bar{Z}_{t}=\bar{z}_{t},\bar{L}_{t}=\bar{l}_{t},A=a)$
for all $t=2,...,T$. At $t=1$, we have

$\displaystyle E(Y_{T}-Y_{0}|R_{1}=1,Z_{1}=z_{1},L_{1}=l_{1},A=a)$
$\displaystyle=\\{E(Y_{T}-Y_{0}|R_{1}=1,L_{1}=l_{1},A=a)-E(Y_{T}-Y_{0}|R_{1}=0,L_{1}=l_{1},A=a)\\}$
$\displaystyle\quad\times\\{1-\pi_{1}(z_{1},l_{1})\\}$
$\displaystyle\quad+E(Y_{T}-Y_{0}|L_{1}=l_{1},A=a)$

and by combining all previous terms

$\displaystyle
E(Y_{0}|L_{1}=l_{1},A=A)+\sum^{T}_{t=1}E(Y_{t}-Y_{t-1}|\bar{R}_{t}=1,\bar{Z}_{t}=\bar{z}_{t},\bar{L}_{t}=\bar{l}_{t},A=a)$
$\displaystyle+\sum^{T}_{t=1}\delta^{*}_{T,t}(\bar{z}_{T-1},\bar{l}_{t})-E\\{\delta^{*}_{T,t}(\bar{z}_{T-1},\bar{L}_{t})|\bar{R}_{T-1}=1,\bar{Z}_{T-1}=\bar{z}_{T-1},\bar{L}_{T-1}=\bar{l}_{T-1},A=a\\}$
$\displaystyle=E(Y_{T}|L_{1}=l_{1},A=a)$
$\displaystyle\quad+\sum^{T}_{t=1}\\{E(Y_{t}-Y_{t-1}|\bar{R}_{t}=1,\bar{Z}_{t-1}=\bar{z}_{t-1},\bar{L}_{t}=\bar{l}_{t},A=a)$
$\displaystyle\quad-E(Y_{t}-Y_{t-1}|R_{t}=0,\bar{R}_{t-1}=1,\bar{Z}_{t-1}=\bar{z}_{t-1},\bar{L}_{t}=\bar{l}_{t},A=1)\\}\pi_{t}(\bar{z}_{t},\bar{l}_{t})$

which one can leverage to obtain $E(Y_{T}|L_{1}=l_{1},A=a)$. Under the
structural assumptions in the previous subsection, this can be used in order
to identify $E(Y^{a}_{T})$.


# Stanceosaurus 2.0: Classifying Stance Towards Russian and
Spanish Misinformation

Anton Lavrouk, Ian Ligon, Tarek Naous, Jonathan Zheng, Alan Ritter, Wei Xu
College of Computing
Georgia Institute of Technology
{antonlavrouk, iligon3, tareknaous<EMAIL_ADDRESS>{alan.ritter,
<EMAIL_ADDRESS>
###### Abstract

The Stanceosaurus corpus (Zheng et al., 2022) was designed to provide high-
quality, annotated, 5-way stance data extracted from Twitter, suitable for
analyzing cross-cultural and cross-lingual misinformation. In the
Stanceosaurus 2.0 iteration, we extend this framework to encompass Russian and
Spanish. The former is of current significance due to prevalent misinformation
amid escalating tensions with the West and the violent incursion into Ukraine.
The latter, meanwhile, represents an enormous community that has been largely
overlooked on major social media platforms. By incorporating an additional
3,874 Spanish and Russian tweets over 41 misinformation claims, our objective
is to support research focused on these issues. To demonstrate the value of
this data, we employed zero-shot cross-lingual transfer on multilingual BERT,
yielding results on par with the initial Stanceosaurus study with a macro F1
score of 43 for both languages. This underlines the viability of stance
classification as an effective tool for identifying multicultural
misinformation.

Stanceosaurus 2.0: Classifying Stance Towards Russian and
Spanish Misinformation

Anton Lavrouk, Ian Ligon, Tarek Naous, Jonathan Zheng, Alan Ritter, Wei Xu
College of Computing Georgia Institute of Technology {antonlavrouk, iligon3,
tareknaous<EMAIL_ADDRESS>{alan.ritter<EMAIL_ADDRESS>

##  1 Introduction

Misinformation on social media is a highly multicultural phenomenon
(Roozenbeek et al., 2020). In the ongoing Russia-Ukraine conflict, Russian-
language misinformation and propaganda are important weapons used by both
sides to influence the opinions of Internet users across the globe. Meanwhile,
Spanish-language misinformation is surging unchecked through virtually every
online community.111The Guardian With these issues in mind, we seek to create
a dataset that can help identify Spanish and Russian misinformation beyond a
binary yes/no approach. We do this by expanding the Stanceosaurus dataset
(Zheng et al., 2022) to include Spanish and Russian tweets annotated using a
5-way stance labeling schema (Gorrell et al. 2018, Schiller et al. 2021), thus
creating Stanceosaurus 2.0. By fine-tuning multilingual BERT (Devlin et al.,
2019), we experiment with zero-shot cross-lingual transfer, demonstrating the
potential for Stanceosaurus 2.0 to help drive forward misinformation research
on Spanish and Russian. Furthermore, recent Twitter policies have made it
clear that the site is moving away from account-based labeling of
misinformation.222Twitter Our dataset presents the opportunity to identify
potential misinformation on a per-tweet basis, allowing users to see relevant
context for potentially misleading tweets. Some may argue that in recent
times, Twitter (now X at the time of revision) has taken a far more "hands-
off" approach to misinformation. While this may or may not be true, this
dataset can be used on social media platforms that are different from
Twitter/X. One can get around the tweet length limit by simply concatenating
various tweets, etc. In the following sections, we discuss what exactly
Russian and Spanish misinformation entail and why they are so important.

Figure 1: Example of a data point (tweet and context) in the Russian
Stanceosaurus dataset. For the claim "NATO forces are currently fighting in
Ukraine", we have an example tweet chain demonstrating various stances.

#### Russian Misinformation

Misinformation and propaganda are crucial to Russian political warfare. Part
of so-called “active measures”, they are designed to “weaken the West [and] to
drive wedges in the Western community alliances of all sorts, particularly
NATO …” (Alexander, 2017). When Russia launched a full-scale invasion of
Ukraine in February of 2022,333CNBC both sides of the conflict engaged in
hybrid warfare, putting an equal focus on the information front and global
deception.444The Atlantic With propaganda machines in full force, the war in
Ukraine has spawned many new misinformation claims. In this context, although
a Russian stance dataset is present in Lozhnikov et al. (2018), it is limited,
and our research aims to modernize Russian stance data to include wartime
misinformation. This is because Russian misinformation "then" and "now" are
two different beasts. Potential feasibility for the idea of Russian
Stanceosaurus can be seen via the findings in Park et al. (2022) which, among
many interesting things, identified Twitter as a platform with a significant
amount of Russian-language discussion regarding wartime events. The series of
Solopova et al. (2023a) and Solopova et al. (2023b), which showcases great
results on pro-Kremlin propaganda detection, also potentially implies
feasibility of our stance-based approach.

#### Spanish Misinformation

Misinformation is rampant in the Spanish-speaking world, surging through
various online communities (Bonnevie et al., 2023). Despite being the fourth
most spoken language in the world and an enormous medium for the spread of
information worldwide both true and false, misinformation in Spanish is far
more of a problem than in English555Washington Post. This problem is further
exacerbated when it is ignored; Facebook whistle-blower Frances Haugen
revealed an enormous disconnect between the proportion of users who speak
Spanish and the amount of spending committed to anti-misinformation resources
in this language666The Guardian. Unsurprisingly, works such as Posadas-Durán
et al. (2019) and Abonizio et al. (2020) attempt to help solve this crucial
problem, particularly framing the problem as detecting fake news. These
studies show that this kind of claim-based misinformation detection works
quite well. Our approach was inspired by such studies. To our knowledge, there
are two existing Spanish stance datasets. One is Zotova et al. (2020), a
valuable but singular claim-limited collection of Spanish-language stance
data. The other is Toledo-Ronen et al. (2020), which creates a wonderful
Spanish stance dataset, but based on arguments and not misinformation claims.
We aim to expand the set of Spanish misinformation via the five-way and three-
way classification framework of Stanceosaurus.

##  2 Stanceosaurus 2.0: Details

In order to facilitate the study of Russian and Spanish misinformation, we
have created a 5-way stance classification dataset in accordance with the
guidelines established by Zheng et al. (2022). These stance categories are
Irrelevant, Supporting, Refuting, Querying, and Discussing. The stance of a
tweet is based on the misinformation claim it is discussing. An example of
various misinformation claim related tweets and their stance categorization
can be seen in Figure 1. Details on the five stance categories (and how they
can be merged to 3 categories) are listed in appendix C.

###  2.1 Data Collection

#### Misinformation Claims

We derived 18 examples of Russian-language misinformation, with 13 from the
European Union initiative, euvsdisinfo, and manually translated them into
Russian using a bilingual Russian/English speaker (both fluent). Despite
criticism of fact-checking methodology (Giorio, 2018), euvsdisinfo is to our
knowledge the best source of prominent misinformation which can be found on
Russian-language Twitter, especially considering that there is no reliable
Russian fact-checking website (this is the reason why we had to translate the
claims to Russian). Nonetheless, to mitigate this bias, we supplemented these
misinformation claims with 5 claims from the Western media. Again, the absence
of claims from Russian sources or Russian-language fact-checking sites is
notable. We re-iterate that identifying misinformation is challenging when
Russian media is largely state controlled777ForeignPolicy.com. Any source that
does fact-checking that might disagree with the state media would most likely
get taken down or blocked by Roskomnadzor. Therefore, for this study, we
selected a ground truth based on western-leaning sources to assess Russian
misinformation claims. For the Spanish corpus, we collected 23 misinformation
claims from reputable Spanish-language fact-checking websites Verificado,
Chequeado, Newtral, and ChequeaBolivia, including claims with various veracity
ratings. The selection of both Spanish and Russian claims was guided by the
volume of relevant Twitter discourse. The detailed Russian and Spanish claims
are listed in Appendix A and B, respectively.

#### Tweet Collection & Reply Chains

For both languages, we collect tweets using the Twitter API. Queries (Appendix
A and B) are manually curated and iteratively refined in order to capture as
many relevant tweets as possible while allowing for diversity in stance
categories. This refinement was done by scraping data for a claim using a
certain query, and then sampling 20 tweets and hand annotating them. Queries
were then modified, and the process was repeated until a reasonably equal
distribution of stances was achieved. Alongside tweets from queries, we also
collect additional tweets for context, including those “above”(preceding) and
“below”(following) in the reply chain. Similar to the original Stanceosaurus
paper, context tweets were included to potentially help models make
classification decisions. Annotation was done by sampling 50 tweets for each
claim, adding up to 100 context tweets, which were then evaluated. These
quantities were chosen based on the number of available annotators. All three
annotators are college-educated students who are fluent in their respective
languages. There were two annotators for Russian and one for Spanish. Future
versions of this work will include another annotator for Spanish. The 5-class
Cohen Kappa for the Russian data is 77.4. Reproducibility criteria for this
process are discussed in Appendix D. A basic overview of corpus statistics is
in Table 1. More detailed corpus statistics can be found in F. As mentioned in
D, the dataset can be requested directly from the authors. Since the tweet
ID’s it contains can link tweets back to their authors, it is important that
these tweets are used only for academic purposes, especially given that some
of them present controversial political opinions.

#tweets Refute Support Irrel. Query Discuss Total Russian 119 332 999 50 407
1907 Spanish 270 302 1036 16 342 1966

Table 1: Corpus statistics of Stanceosaurus 2.0.

###  2.2 Russian Corpus

#### Russian Twitter

According to a Statista study (Statista, 2023a), only 5% of Russians surveyed
reported using Twitter. This makes sense, as before this study, “Facebook,
Instagram, and Twitter were all blocked by the Russian state in early March
2022 when the laws on antiwar activity came into force” (McCarthy et al.,
2023). Thus, the only way to access Twitter in Russia is through a VPN. From
this information, we gather that Russian speakers on Twitter either use a VPN,
are native to another country where Russian is a common language, or live
abroad. Acknowledging this is important, as it provides a population for the
Twitter users that were sampled and presents a limitation of the data to be
addressed with future work.

#### Code Switching

There are instances of tweets that contain different languages. It is fairly
common to see acronyms such as “HIMARS” and “NATO” being written in both
English and Russian interchangeably. A brief analysis using regular
expressions finds that tweets containing characters from the English alphabet
make up about 12 percent of all tweets. Furthermore, sometimes the Russian
language is phonetically written in the Latin alphabet. This poses a challenge
when querying for relevant Tweets, which we addressed by accounting for as
many code-switched variations as was reasonable. Furthermore, in the sampled
reply chains, it was common to see the mixing of languages, especially between
Russian and Ukrainian. Fortunately, since every single tweet was hand
annotated by a fluent Russian speaker with proficient knowledge of Ukrainian,
it was not difficult to differentiate between the two languages, as well as
any other language that uses the Cyrillic alphabet.

#### Obscenities

Due to the politically charged nature of the Russian misinformation claims,
many tweets contain large amounts of cursing, which is known as mat
(pronounced maht). It is argued that mat “is not merely an accumulation of
obscenities, but rather constitutes a set of refined, complex structures”,
hinting at a “potentially limitless quantity of expressions” (Dreizin and
Priestly, 1982). A cursory analysis indicates that around 10 percent of all
Russian tweets collected contain some sort of obscenity. Context is key when
trying to understand Russian obscenities, and this may prove to be quite
confusing for a language model to interpret.

###  2.3 Spanish Corpus

#### Circumventing Filters

Particularly when discussing the COVID-19 vaccine, many tweets include
language that is most likely obscured to circumvent misinformation filters.
When searching “vacuna” (Spanish for “vaccine”), Twitter sends users a warning
to check federal websites for information related to the pandemic.888This is
no longer the case following recent changes to Twitter policy. Accordingly,
the queries had to be adjusted to include numerous alternative spellings for
the word for vaccine (including ‘vacuno’, ‘vacun@’, ‘vakuna’, ‘cacunados’,
‘v@cunad0s’, ‘kakuna’, etc.).

#### Social Media Usage

The decision to utilize Twitter for this corpus was driven by its accessible
API and publicly shareable text-centric content for open and ethical NLP
research. It is worth noting that within the Spanish-speaking realm, Twitter
ranks behind Facebook, Instagram, and TikTok in terms of social media usage
(Statista 2023b, StatCounter 2023). Additionally, more Hispanics use WhatsApp
than any other race or ethnicity,999Insider Intelligence and significant
volumes of misinformation spread on private channels such as
WhatsApp101010Harvard Kennedy School where misinformation detection is much
more difficult and misinformation is less likely to be corrected by the
public.

#### Code Switching

Mixing Spanish and English together in a single tweet is common, particularly
in Spanish-speaking communities in Northern Mexico and the USA. The spread of
misinformation in bilingual communities is a unique challenge of particular
importance in the United States, where more than one-third of all Hispanic
adults self-identify as bilingual in English and Spanish (Pew, 2015).

##  3 Automatic Stance Detection Using Stanceosaurus 2.0

#### Zero-Shot Cross-Lingual Transfer

In accordance with the original Stanceosaurus paper (Zheng et al., 2022), we
conduct a zero-shot cross-lingual transfer experiment on our data. This
entails training a model on the English Stanceosaurus dataset of 20,707 tweets
and then evaluating it on the Russian and Spanish sets. We believe that this
is the best way to evaluate Stanceosaurus 2.0 since we assume that there is
little to no stance-based training data available for Russian and Spanish
(something we observed during our research, and can be seen in Section 1 where
we discuss related work). Also, various studies such as Pires et al. (2019)
and Artetxe et al. (2020) have shown zero-shot cross-lingual transfer to be an
effective approach in many languages, including Russian and Spanish.

#### Multilingual BERT

Multilingual BERT (Devlin et al., 2019), or mBERT, has been shown to be very
competitive in the zero-shot setting that we have described (Wu and Dredze
2019, Libovický et al. 2019). We believe that mBERT is a simple baseline that
indicates the quality of our dataset and model performance. For our
experiments, we follow the original Stanceosaurus paper (Zheng et al., 2022)
and use the five stance label schema. To create model input, we format our
strings using special tokens as follows: “[CLS] claim [SEP] text”.

#### Loss Functions

Similar to the original Stanceosaurus (Zheng et al., 2022), we examine three
different loss functions: cross-entropy loss, weighted cross-entropy loss (Cui
et al., 2019), and class-balanced focal loss (Baheti et al., 2021). While the
cross-entropy loss is a baseline commonly used in classification tasks, we use
weighted cross-entropy to modify this baseline to account for imbalanced
classes by assigning more weights to classes with fewer samples. Class-
balanced focal loss is an alternative method to account for imbalanced
classes. It down-weights easy examples and focuses more on difficult ones (Cui
et al., 2019).

#### Results

The results of our experiments can be seen in Table 2. One can compare these
results to English performance on BERTBASE for unseen claims from the original
Stanceosaurus paper (Zheng et al., 2022), as well as the same zero-shot cross-
lingual transfer experiment on Hindi and Arabic. These extra experiments are
also shown in 2, but they are clearly marked as the contribution of the
authors of the original Stanceosaurus paper. Both Russian and Spanish datasets
performed similarly to models for English to Hindi and English to Arabic
transfer experiments in the original Stanceosaurus (Zheng et al., 2022). The
weighted loss functions performed better overall, and both languages achieved
an F1 score of around $43$. Reproducibility criteria for our experiments can
be seen in appendix E.

Russian (our contribution)
---
Loss | Precision | Recall | F1
CE | 53.55$\pm$0.8 | 35.33$\pm$0.7 | 36.15$\pm$1.3
Weighted CE | 44.38$\pm$0.2 | 42.84$\pm$0.5 | 42.09$\pm$0.1
CBFL | 45.60$\pm$1.5 | 46.98$\pm$2.0 | 43.94$\pm$0.2
Spanish (our contribution)
Loss | Precision | Recall | F1
CE | 50.26$\pm$1.9 | 40.86$\pm$0.7 | 41.81$\pm$1.0
Weighted CE | 54.12$\pm$0.4 | 42.65$\pm$0.5 | 43.75$\pm$0.4
CBFL | 51.26$\pm$2.2 | 44.15$\pm$0.9 | 43.83$\pm$1.0
Hindi (Zheng et al., 2022)
Loss | Precision | Recall | F1
CE | 52.1$\pm$2.9 | 39.4$\pm$2.0 | 40.8$\pm$2.5
Weighted CE | 55.0$\pm$4.2 | 42.4$\pm$1.4 | 44.3$\pm$1.8
CBFL | 53.0$\pm$3.4 | 44.1$\pm$1.7 | 45.3$\pm$1.5
Arabic (Zheng et al., 2022)
Loss | Precision | Recall | F1
CE | 44.8$\pm$4.0 | 40.1$\pm$2.5 | 40.0$\pm$2.0
Weighted CE | 44.1$\pm$3.3 | 40.7$\pm$1.6 | 39.7$\pm$1.7
CBFL | 46.1$\pm$2.6 | 44.7$\pm$1.1 | 43.1$\pm$0.2
English on BERTBASE (Zheng et al., 2022)
Loss | Precision | Recall | F1
CE | 51.1$\pm$1.1 | 50.5$\pm$2.0 | 50.4$\pm$1.6
Weighted CE | 50.5$\pm$1.9 | 52.7$\pm$1.1 | 51.3$\pm$1.3
CBFL | 50.6$\pm$1.3 | 55.7$\pm$2.1 | 52.5$\pm$1.0
Table 2: Russian and Spanish experiments. Models are trained on English
Stanceosaurus and then evaluated on either Russian or Spanish in our work. F1
is measured as macro F1. Results are taken as the average of 3 experiments,
with error being one standard deviation. English, Arabic, and Hindi
experiments are taken directly from Stanceosaurus (Zheng et al., 2022) as a
comparison benchmark.

##  4 Conclusion

We introduce Stanceosaurus 2.0, an extension of the 5-way stance dataset
Stanceosaurus (Zheng et al., 2022). Our dataset includes 18 Russian
misinformation claims (1907 tweets) and 23 Spanish misinformation claims (1966
tweets). Our dataset is modern and up to date given the recent slough of
misinformation and current events. It also contains Russian and Spanish, which
as shown previously, are two languages in which misinformation thrives, and
efforts to combat it are limited. Our zero-shot cross-lingual transfer
experiments show that our dataset performs at similar levels to that of Hindi
and Arabic in the original Stanceosaurus, with a macro F1 score of about 43.
This means that there is potential to continue refining models and algorithms
to create a somewhat reliable stance classifier using transformer-based models
like mBERT. Future versions of this work will entail experiments on more
models, as well as a second annotator for the Spanish version.

## Limitations

#### The Veracity of Fact-Checked Claims

One of the biggest limitations of our work is the fact that fact-checking is
often not as black-and-white as it seems and is generally a practice that
suffers from many limitations (Uscinski and Butler, 2013). It is very
difficult to find objective truths that are verified to a degree of absolute
precision for a work like this. This is doubly so for political-leaning
claims, such as the claims in the Russian dataset.

#### Russian Misinformation Claims

An unfortunate limitation of the Russian language is that there are no Russian
fact-checking websites that would provide reasonably objective fact-checking,
at least as far as we are aware. This is most likely due to the level of
control that the Russian government has over the Russian internet (Polyakova
and Meserole, 2019). This lack of resources means that Russian claims were
hand-picked. This could introduce author bias, and may not be an accurate
representation of the Russian internet, as claims were mostly all found on the
heavily western-leaning website euvsdisinfo, as discussed in section 2.

#### Russian Twitter

As mentioned in section 2.2, Twitter is not the most used social media, and
this could introduce various biases into our data. Future work could involve
the social media website VKontakte, which as mentioned earlier, is the most
popular in Russia. However, some problems could arise due to state-owned
entities being shareholders111111Reuters.

#### Spanish Twitter

Likewise, Twitter is far from the most popular social media network in Latin
America. More work should be done to analyze misinformation on Facebook and
WhatsApp in the Hispanosphere. Despite favoring small-group communication,
WhatsApp persists as a medium for rapid misinformation dissemination in Latin
America (Nobre et al., 2022).

#### Spanish Queries

As mentioned in section 2.3, numerous obstacles made it difficult to query for
relevant Tweets in Spanish. From properties inherent to the Spanish language
like a highly inflectional morphology to broader social factors including the
prevalence of code-mixing and filter circumvention, care had to be taken when
querying Twitter’s API to find relevant Tweets without biasing the data in any
one direction (Pfaff, 1979). Future work might include broad queries to
procure larger datasets that can then be manually cleaned to include more
relevant Tweets.

#### Code Switching

As mentioned in both sections 2.2 and 2.3, both languages experienced a decent
amount of code-switching, whether it be in the context or the tweet itself. It
has been shown before that dealing with code-switching is not an easy task
(Winata et al., 2021). However, recently there has been a large number of
code-switching datasets that have become available (Jose et al., 2020).
Potential further research may include creating stance datasets exclusively on
code-switched datasets.

#### Tweet Deletion

A feature of the obscured version of the dataset (the version we plan on
giving out in most cases) is that it only features tweet IDs. However, if
someone deletes a tweet, that tweet will be gone from the obscured dataset.
This maintains the user’s right to remove their content without it still being
a database. However, this may be an issue for researchers using this dataset a
long time after the tweets were originally collected.

## Ethics Statement

#### Working With Social Media Data

Mining social media data from Twitter users without their consent is at best
ethically problematic (Taylor and Pagliari, 2018). Unfortunately, this kind of
data would not exist without this technique. However, our publicly available
dataset only contains tweet IDs and does not include actual tweets and
usernames. Furthermore, social media data can contain harmful biases towards
certain groups, as moderating social media can be extremely difficult (Ganesh
and Bright, 2020). We encourage a thorough review of the data and its context
before deploying in a production environment.

#### Data Annotation

We recognize that some of the tweets that have been annotated deal with
sensitive topics and contain some hateful language, especially in the Russian
dataset, given its political nature. We recognize that annotators need to be
warned of this before they start annotating.

#### Propaganda Analysis

An issue with analyzing propaganda and misinformation is that this analysis
can potentially fall into the wrong hands. For example, using this dataset to
analyze the effectiveness of Russian propaganda can inform the source of the
propaganda exactly what they could improve on.

## Acknowledgments

The authors would like to thank Dennis Pozhidaev for his help with data
annotation and evaluation. This research is supported by the NSF (IIS-2052498)
and IARPA via the HIATUS program (2022-22072200004). The views and conclusions
contained herein are those of the authors and should not be interpreted as
necessarily representing the official policies, either expressed or implied,
of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized
to reproduce and distribute reprints for governmental purposes notwithstanding
any copyright annotation therein.

## References

  * Abonizio et al. (2020) Hugo Queiroz Abonizio, Janaina Ignacio de Morais, Gabriel Marques Tavares, and Sylvio Barbon Junior. 2020.  Language-independent fake news detection: English, portuguese, and spanish mutual features.  _Future Internet_ , 12(5).
  * Alexander (2017) Keith B. Alexander. 2017.  Disinformation: A primer in russian active measures and influence campaigns.  Prepared statement, United States Senate Select Committee on Intelligence.
  * Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Baheti et al. (2021) Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021.  Just say no: Analyzing the stance of neural dialogue generation in offensive contexts.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4846–4862, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bonnevie et al. (2023) E Bonnevie, V Ricciulli, M Fields, and R O’Neill. 2023.  Lessons learned from monitoring spanish-language vaccine misinformation during the covid-19 pandemic.  _Public Health Rep_ , 138(4):586–592.  PMID: 37102367; PMCID: PMC10140774.
  * Cui et al. (2019) Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Dreizin and Priestly (1982) F. Dreizin and T. Priestly. 1982.  A systematic approach to russian obscene language.  _Russ Linguist_ , 6:233–249.
  * Ganesh and Bright (2020) Bharath Ganesh and Jonathan Bright. 2020.  Countering extremists on social media: Challenges for strategic communication and content moderation.
  * Giorio (2018) L. Giorio. 2018.  _War on Propaganda or PRopaganda War?: A case study of fact-checking and (counter)propaganda in the EEAS project EUvsDisinfo_.  Dissertation, Uppsala University, Jagiellonian University.
  * Gorrell et al. (2018) Genevieve Gorrell, Kalina Bontcheva, Leon Derczynski, Elena Kochkina, Maria Liakata, and Arkaitz Zubiaga. 2018.  Rumoureval 2019: Determining rumour veracity and support for rumours.
  * Jose et al. (2020) Navya Jose, Bharathi Raja Chakravarthi, Shardul Suryawanshi, Elizabeth Sherly, and John P. McCrae. 2020.  A survey of current datasets for code-switching research.  In _2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS)_ , pages 136–141.
  * Libovický et al. (2019) Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2019.  How language-neutral is multilingual bert?
  * Lozhnikov et al. (2018) Nikita Lozhnikov, Leon Derczynski, and Manuel Mazzara. 2018.  Stance prediction for russian: Data and analysis.
  * McCarthy et al. (2023) Lauren A. McCarthy et al. 2023.  Four months of “discrediting the military”: Repressive law in wartime russia.  _Demokratizatsiya: The Journal of Post-Soviet Democratization_.
  * Nobre et al. (2022) Gabriel Peres Nobre, Carlos H.G. Ferreira, and Jussara M. Almeida. 2022.  A hierarchical network-oriented analysis of user participation in misinformation spread on whatsapp.  _Information Processing & Management_, 59(1):102757.
  * Park et al. (2022) Chan Young Park, Julia Mendelsohn, Anjalie Field, and Yulia Tsvetkov. 2022.  Challenges and opportunities in information manipulation detection: An examination of wartime russian media.
  * Pew (2015) Pew. 2015.  A majority of english-speaking hispanics in the u.s. are bilingual.  Accessed: 2023-06-11.
  * Pfaff (1979) Carol W. Pfaff. 1979.  Constraints on language mixing: Intrasentential code-switching and borrowing in spanish/english.  _Language_ , 55(2):291–318.
  * Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019.  How multilingual is multilingual bert?
  * Polyakova and Meserole (2019) Alina Polyakova and Chris Meserole. 2019.  Exporting digital authoritarianism: The russian and chinese models.  _Policy Brief, Democracy and Disorder Series_ , pages 1–22.
  * Posadas-Durán et al. (2019) Juan-Pablo Posadas-Durán et al. 2019.  Detection of fake news in a new corpus for the spanish language.  _Journal of Intelligent & Fuzzy Systems_, 36(5):4869–4876.
  * Roozenbeek et al. (2020) Jon Roozenbeek, Claudia R. Schneider, Sarah Dryhurst, John Kerr, Alexandra L. J. Freeman, Gabriel Recchia, Anne Marthe van der Bles, and Sander van der Linden. 2020.  Susceptibility to misinformation about covid-19 around the world.  _R. Soc. open sci._ , 7:201199.
  * Schiller et al. (2021) B. Schiller, J. Daxenberger, and I. Gurevych. 2021.  Stance detection benchmark: How robust is your stance detection? _Künstl Intell_ , 35:329–341.
  * Solopova et al. (2023a) V. Solopova, OI. Popescu, C. Benzmüller, et al. 2023a.  Automated multilingual detection of pro-kremlin propaganda in newspapers and telegram posts.  _Datenbank Spektrum_ , 23:5–14.
  * Solopova et al. (2023b) Veronika Solopova, Christoph Benzmüller, and Tim Landgraf. 2023b.  The evolution of pro-kremlin propaganda from a machine learning and linguistics perspective.  In _Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP)_ , pages 40–48, Dubrovnik, Croatia. Association for Computational Linguistics.
  * StatCounter (2023) StatCounter. 2023.  Social media stats spain.  Accessed: June 11, 2023.
  * Statista (2023a) Statista. 2023a.  Ranking of social media platforms in russia q3 2022, by user share.  Accessed: 2023-05-26.
  * Statista (2023b) Statista. 2023b.  Social media usage in latin america - statistics & facts.  Accessed: June 11, 2023.
  * Taylor and Pagliari (2018) Joanna Taylor and Claudia Pagliari. 2018.  Mining social media data: How are research sponsors and researchers addressing the ethical challenges?  _Research Ethics_ , 14(2):1–39.
  * Toledo-Ronen et al. (2020) Orith Toledo-Ronen, Matan Orbach, Yonatan Bilu, Artem Spector, and Noam Slonim. 2020\.  Multilingual argument mining: Datasets and analysis.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 303–317, Online. Association for Computational Linguistics.
  * Uscinski and Butler (2013) Joseph E. Uscinski and Ryden W. Butler. 2013.  The epistemology of fact checking.  _Critical Review_ , 25(2):162–180.
  * Winata et al. (2021) Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2021.  Are multilingual models effective in code-switching?
  * Wu and Dredze (2019) Shijie Wu and Mark Dredze. 2019.  Beto, bentz, becas: The surprising cross-lingual effectiveness of bert.
  * Zheng et al. (2022) Jonathan Zheng, Ashutosh Baheti, Tarek Naous, Wei Xu, and Alan Ritter. 2022.  Stanceosaurus: Classifying stance towards multilingual misinformation.
  * Zotova et al. (2020) Elena Zotova, Rodrigo Agerri, Manuel Nuñez, and German Rigau. 2020.  Multilingual stance detection in tweets: The Catalonia independence corpus.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 1368–1375, Marseille, France. European Language Resources Association.

##  Appendix A Russian Claims and Queries

Russian claims and queries can be found in figure 3.

##  Appendix B Spanish Claims and Queries

Spanish claims and queries can be found in figures 4 and 5.

##  Appendix C Stance Categorization

The following is a description of each stance:

  * •

Supporting: Tweets that directly support the fact that a claim is true.

  * •

Refuting: Tweets that refute the veracity of a claim.

  * •

Querying: Questions the veracity of a claim.

  * •

Discussing: Provides neutral information on the context or truth of a claim.

  * •

Irrelevant: Not relevant to the given claim.

If a tweet is labeled as discussing, then to enable 3-way stance
classification, the tweet is also given a leaning. The following is a
description of each leaning:

  * •

Supporting: The tweet has an indirect positive bias when discussing the claim.

  * •

Refuting: The tweet has an indirect negative bias when discussing the claim.

  * •

Other: The tweet does not have any sort of bias.

With this information, we can now construct our guidelines for 3-way stance
categorization as well:

  * •

Supporting: Merge supporting with discussingsupporting.

  * •

Refuting: Merge refuting with discussingrefuting.

  * •

Other: Merge irrelevant, querying, and discussingother.

##  Appendix D Dataset Reproducibility Criteria

  * •

Using the twitter API, up to 150 tweets were pulled for each claim using the
queries listed in figures 3, 4, and 5. Context for each tweet was also
retrieved. Context in this case means the entire reply chain from the root
tweet down to the pulled tweet, as well as any immediate replies.

  * •

Quality control was done by an extensive iteration of Twitter API queries. We
aimed to make queries such that the distribution of stance categories was
reasonably even, although this proved to be difficult with the "Querying"
category.

  * •

With these tweets in hand, up to 50 tweets were sampled for each claim for
annotation. Context tweets were also annotated. Up to 50 parent context tweets
were sampled and up to 50 context children tweets were sampled for each claim.

  * •

Claims were annotated in accordance with details given in appendix C. Russian
tweets were double annotated, while Spanish tweets currently only have a
single annotator, but we are working to find another annotator at the moment.

  * •

Tweets were pre-processed to remove duplicates using lexical similarity.

  * •

The context chains were then reconstructed and formatted in json to match the
original Stanceosaurus paper (Zheng et al., 2022).

  * •

The dataset can be requested from the authors using the emails given in the
paper. Since the data is potentially sensitive (tweets of political nature) we
need to make sure that anyone who uses these tweets is doing so solely out of
academic intent.

##  Appendix E Experiment Reproducibility Criteria

  * •

Model: bert-base-multilingual-uncased

  * •

Computing Infrastructure: 4 Nvidia Titan X GPUs. NVIDIA-SMI 460.84. Driver
Version 460.84. CUDA version 11.2. Running on CentOS linux 7. Conda version 7.
Package versions listed in requirements.txt file in code used.

  * •

Average Training Time: Per experiment, around 40 minutes

  * •

Evaluation Metrics: Best evaluation of the development set per training run

  * •

Number of Experiments: Each row in 2 was done 3 times. Results are the mean
$\pm$ the standard deviation. Random seeds for the three runs were 10, 20, and
30.

  * •

Hyperparameters: Hyperparameters were chosen based off of best performing
hyper parameters in the original Stanceosaurus model, and then manually tuned.

    * –

Learning Rate: 3e-5

    * –

Batch Size: $8$ per GPU, so $32$ total

    * –

Class Balanced Focal Loss: Similar to the original paper, we tune $\beta$ and
$\gamma$ between $[0.1,1)$ and $[0.1,1.1]$ respectively.

    * –

The rest are defaulted to what is used in the code. Run commands are included
with code.

  * •

Code zip file can be accessed upon request.

##  Appendix F Corpus Statistics

The distribution of labels and tweet types for Russian Spanish are shown in
tables 3 and 4 respectively. A visual representation of the tweets (not
context or replies) for Russian Spanish is shown in figures 2 and 2
respectively.

##  Appendix G Annotation Logistics

Annotators were American college students paid 18 dollars an hour. Each
annotator was fluent in the language they were annotating. All annotators were
recruited as people the authors directly knew. Verbally, annotators were told
the scope of the paper and given the abstract.

##  Appendix H Use of AI assistants

AI assistants were used by the authors of this paper in order to proofread the
paper. Occasionally, an AI assistant was asked to rephrase some text, just to
generate some ideas on sentence flow. Work was never directly copied, and
model output was used as inspiration.

| Refuting | Supporting | Irrelevant | Querying | Dsupporting | Drefuting | Dother | Total
---|---|---|---|---|---|---|---|---
Tweets | 109 | 315 | 149 | 39 | 77 | 169 | 41 | 899
Context | 5 | 15 | 738 | 9 | 51 | 40 | 7 | 865
Replies | 5 | 2 | 112 | 2 | 6 | 15 | 1 | 143
Total | 119 | 332 | 999 | 50 | 134 | 224 | 49 | 1907
Table 3: Russian Corpus Statistics. | Refuting | Supporting | Irrelevant | Querying | Dsupporting | Drefuting | Dother | Total
---|---|---|---|---|---|---|---|---
Tweets | 228 | 269 | 418 | 12 | 85 | 52 | 60 | 1124
Context | 15 | 21 | 370 | 2 | 18 | 13 | 4 | 443
Replies | 27 | 12 | 248 | 2 | 76 | 18 | 16 | 399
Total | 270 | 302 | 1036 | 16 | 179 | 83 | 80 | 1966
Table 4: Spanish Corpus Statistics.

Figure 2: Label distribution for tweets (by query, not context) in the (a)
Russian dataset and (b) Spanish dataset. Figure 3: Russian Claims and Queries
Figure 4: Part 1 of Spanish Claims and Queries Figure 5: Part 2 of Spanish
Claims and Queries