Synthetic Controls in Action

Alberto Abadie | Jaume Vives-i-Bastida
---|---
MIT | MIT

August 27, 2024

###### Abstract

In this article we propose a set of simple principles to guide empirical
practice in synthetic control studies. The proposed principles follow from
formal properties of synthetic control estimators, and pertain to the nature,
implications, and prevention of over-fitting biases within a synthetic control
framework, to the interpretability of the results, and to the availability of
validation exercises. We discuss and visually demonstrate the relevance of the
proposed principles under a variety of data configurations.

00footnotetext: Alberto Abadie, Department of Economics, MIT<EMAIL_ADDRESS>Jaume Vives-i-Bastida, Department of Economics, MIT<EMAIL_ADDRESS>We greatly
benefited form comments by Kaspar Wuthrich and Yinchu Zhu. NSF support through
grant SES-1756692 is gratefully acknowledged.

##  1 Introduction

Synthetic control methods have become widely applied in empirical research in
economics and other disciplines. Recent empirical applications of the
synthetic control framework include studies of the effects of political
connections (Acemoglu et al.,, 2016), legalized prostitution (Cunningham and
Shah,, 2018), right-to-carry laws (Donohue et al.,, 2019), and many other (see
Abadie,, 2021, for a more elaborate review of the applications of synthetic
control estimators). With the increasing popularity of synthetic control
estimators, it has become particularly important to understand the settings
where synthetic controls produce reliable estimates and those where they do
not.

In this article, we explore in detail how the properties of synthetic controls
translate to actual performance. From the properties of synthetic control
estimators we distill a set of simple principles to guide empirical practice,
and demonstrate the practical relevance of our proposed principles via
simulation exercises.

First, we use a variety of data configurations to demonstrate the crucial role
of pre-treatment fit in the performance of synthetic control estimators. Close
pre-treatment fit, however, does not guarantee good performance of synthetic
control estimators because of the possibility of over-fitting. We describe the
nature of over-fitting within the context of synthetic control estimation,
explain how over-fitting induces biases, demonstrate the practical relevance
of these biases, and discuss how a careful study design can minimize or avoid
them. Finally, we discuss validation exercises that can be applied to assess
the validity of synthetic control estimators in empirical applications.

The rest of the article is organized as follows. Section 2 introduces
synthetic control estimators and discusses their formal properties under a
linear factor structure in the data generating process. Section 3 employs a
grouped version of the linear factor model of Section 3 to study how the
performance of synthetic controls is related to pre-treatment fit and over-
fitting. Section 4 discusses how to validate synthetic control estimates.
Section 5 discusses trimming as a way to alleviate over-fitting and
interpolation biases. Section 6 discusses the role of covariates. The
applicability of synthetic control estimators is not confined, however, to the
linear factor model. In Section 7, we use a simple auto-regressive model to
illustrate this point. Section 8 concludes.

##  2 A Primer on Synthetic Controls

Suppose we observe $j=1,\ldots,J+1$ aggregate units, such as states or
countries for $t=1,\ldots,T$ periods. The first unit (that is, $j=1$) is
exposed to a policy intervention, or some other event or treatment of
interest, at time $t=T_{0}+1$, with $T_{0}+1\leq T$. The remaining $J$ units
are not exposed to the event or intervention of interest. We aim to estimate
the effect of the treatment on some outcome of interest during the post-
treatment periods, $T_{0}+1,\ldots,T$. We will use the terms “intervention”,
“event”, and “treatment” interchangeably.

To define treatment effects, we formally adopt a model of potential outcomes
(Rubin,, 1974). Let $Y^{N}_{jt}$ be the potential outcome observed for unit
$j\in\\{1,\ldots,J+1\\}$ and time $t=\\{1,\ldots,T\\}$ in the absence of the
intervention. Let $Y^{I}_{1t}$ be the potential outcome observed for the
treated unit at time $t=T_{0}+1,\ldots,T$ under the intervention. For each
unit and time period, $Y_{jt}$ is the observed outcome. Therefore, observed
outcomes for untreated units, $j=2,\ldots,,J+1$, are equal to $Y^{N}_{jt}$.
For the treated unit, the observed outcome is equal to $Y^{N}_{1t}$ for
$t=1,\ldots,T_{0}$, and equal to $Y^{I}_{1t}$ for $t=T_{0}+1,\ldots,T$. The
object of interest is the treatment effect on the treated unit,

$\tau_{t}=Y^{I}_{1t}-Y_{1t}^{N},$

for $t=T_{0}+1,\ldots,T$. Because $Y^{I}_{1t}=Y_{1t}$ in the post-treatment
periods, we obtain

$\tau_{t}=Y_{1t}-Y_{1t}^{N},$

for $t=T_{0}+1,\ldots,T$. That is, because $Y^{I}_{1t}$ is observed in the
post-treatment periods, estimating $\tau_{1t}$ for $t=T_{0}+1,\ldots,T$ boils
down to estimating $Y_{1t}^{N}$. A synthetic control estimator of $Y_{1t}^{N}$
is a weighted average of the outcomes for the “donor pool” of $J$ untreated
units,

$\widehat{Y}_{1t}^{N}=\sum_{j=2}^{J+1}W_{j}Y_{jt},$

where $W_{2},\ldots,W_{J+1}$ are non-negative and sum to one. A synthetic
control estimator of $\tau_{1t}$ is equal to the difference between the
outcome values for the treated units and the outcomes values for the synthetic
control,

$\widehat{\tau}_{t}=Y_{1t}-\sum_{j=2}^{J+1}W_{j}Y_{jt}.$

The weights, $W_{2},\ldots,W_{J+1}$, represent the contribution of each
untreated observation to the estimate of the counterfactual of interest,
$\widehat{Y}^{N}_{1t}$. While these weights could, in principle, be directly
chosen by the analyst, the synthetic control literature provides a host of
data-driven weight selectors. For $j=1,\ldots,J+1$, let
$\boldsymbol{X}_{j}=(X_{1j},\ldots,X_{kj})^{\prime}$ be a $(k\times 1)$-vector
of pre-intervention values of predictors of $Y^{N}_{jt}$, with
$t=T_{0}+1,\ldots,T$. Let $\boldsymbol{X}_{0}$ be the $(k\times J)$-matrix
that concatenates $\boldsymbol{X}_{2},\ldots,\boldsymbol{X}_{J+1}$. A simple
data-driven selector
$\boldsymbol{W}^{*}=(W^{*}_{2},\ldots,W^{*}_{J+1})^{\prime}$ of the synthetic
control weights minimizes

$\|\boldsymbol{X}_{1}-\boldsymbol{X}_{0}\boldsymbol{W}\|=\left(\sum_{h=1}^{k}v_{h}(X_{h1}-W_{2}X_{h2}-\dots-
W_{J+1}X_{hJ+1})^{2}\right)^{1/2},$ (1)

subject to the constraints that the weights are non-negative and sum up to
one. The non-negative constants, $v_{1},\ldots,v_{k}$, can be used to
standardize the predictors (for example, by making $v_{h}$ equal to the
inverse of the variance of the $h$-th predictor) or chosen to reflect their
predictive power on the outcome of interest.

Synthetic control estimates, as defined above, are typically sparse, meaning
that only a few units in the donor pool obtain weigths, $W^{*}_{j}$, different
from zero. Figure 1 provides a geometric interpretation of this property. The
synthetic control $\boldsymbol{X}_{0}\boldsymbol{W}^{*}$ is given by the
projection of $\boldsymbol{X}_{1}$ on the convex hull of the columns of
$\boldsymbol{X}_{0}$. The curse of dimensionality implies that, even in
moderately large dimension, $\boldsymbol{X}_{1}$ may often be outside the
convex hull of the columns of $\boldsymbol{X}_{0}$. This is the case depicted
in Figure 1. As indicated in the figure, when $\boldsymbol{X}_{1}$ is outside
the convex hull of the columns of $\boldsymbol{X}_{0}$, the synthetic control
$\boldsymbol{X}_{0}\boldsymbol{W}^{*}$ is unique and sparse provided that the
columns of $\boldsymbol{X}_{0}$ are in general position (see Abadie and
L’Hour,, 2021, for the meaning of “general position”). Then, the number of
non-zero weights, $W^{*}_{j}$, is not larger than $k$, the dimension of
$\boldsymbol{X}_{j}$. In Figure 1, the only untreated units that contribute to
the synthetic control are represented by the three vertices of the shaded
facet, the one that contains $\boldsymbol{X}_{0}\boldsymbol{W}^{*}$. If
$\boldsymbol{X}_{1}$ is in the convex hull of the columns of
$\boldsymbol{X}_{0}$, then the synthetic control does not need to be unique or
sparse. However, sparse solutions with no more than $k+1$ non-zero weights
exist by Carathéodory’s theorem. Abadie and L’Hour, (2021) provide a penalized
synthetic control estimator that is unique and sparse even in the case that
$\boldsymbol{X}_{1}$ is in the convex hull of the columns of
$\boldsymbol{X}_{0}$. The weight selector in Abadie and L’Hour, (2021) favors
synthetic controls composed by untreated units with values of the predictors
in $\boldsymbol{X}_{j}$ close to the values of the predictors for the treated
unit, $\boldsymbol{X}_{1}$. The result is a procedure that not only produces
unique and sparse estimates, but also ameliorates potential interpolation
biases that could emerge from averaging outcomes between untreated units that
are far from the treated unit in the space of the predictors,
$\boldsymbol{X}_{j}$. Sparsity plays an important role in facilitating the
interpretability of synthetic control estimates. The exact nature of the
synthetic control estimate, the contribution of each unit in the donor pool to
this estimate, and the size and direction of potential biases that may arise
if the untreated units contributing to a synthetic control are indirectly
affected by the treatment or by other known idiosyncratic shocks to their
outcomes, are greatly facilitated by the non-negativity and sparsity of the
weights and by the fact that they sum up to one, so they can be interpreted as
proper weights.

To study the properties of synthetic control estimators, we will posit a
generative model for the outcomes, $Y_{jt}^{N}$. Abadie et al., (2010)
consider the linear factor model,

$Y^{N}_{jt}=\delta_{t}+\boldsymbol{\theta}_{t}\boldsymbol{Z}_{j}+\boldsymbol{\lambda}_{t}\boldsymbol{\mu}_{j}+\epsilon_{jt},$
(2)

where $\delta_{t}$ is a time trend, and $\boldsymbol{Z}_{j}$ and
$\boldsymbol{\mu}_{j}$ are vectors of observed and unobserved predictors,
respectively, with time varying coefficients, $\boldsymbol{\theta}_{t}$ and
$\boldsymbol{\lambda}_{t}$. We will also use the term “covariates” to refer to
$\boldsymbol{Z}_{j}$ and $\boldsymbol{\mu}_{j}$. The variable $\epsilon_{jt}$
is a transitory shock that we will model as white noise. We will take all the
components on the right-hand side of equation (2) as given (conditioned on)
except for $\epsilon_{jt}$.

Equation (2) imposes a zero mean restriction on $\epsilon_{jt}$ regardless of
treatment status, which can be interpreted as a form of unconfoundedness
conditional on $\boldsymbol{Z}_{j}$ and $\boldsymbol{\mu}_{j}$. However,
equation (2) does not directly restrict the process that determines selection
for treatment. In the contexts where synthetic controls are typically applied
in practice—with only one or a small number of treated units—a precise
knowledge of the selection mechanism may not be particularly useful in the
absence of a model for the outcomes. Consider the usual setting with only one
treated unit. Randomization of the treatment–something that is extremely
unlikely in settings with aggregate units, like states or countries—implies
that a simple difference in means between outcomes of treated and untreated
units is ex-ante unbiased for the average treatment effect (that is, unbiased
before randomization). However, a simple difference in means could be heavily
biased ex-post if randomization produces large differences between the
characteristics of the treated unit and the units that are not exposed to the
treatment. Equation (2) accounts for these differences by modeling the effect
of observables, $\boldsymbol{Z}_{j}$, and unobservables,
$\boldsymbol{\mu}_{j}$, on the potential outcomes, $Y^{N}_{jt}$.

Suppose that, with probability one, the solution to the minimization of (1)
subject to the weight constraints yields

$\sum_{j=2}^{J}W_{j}^{*}\boldsymbol{Z}_{j}=\boldsymbol{Z}_{1}\quad\mbox{ and
}\quad\sum_{j=2}^{J}W_{j}^{*}Y_{jt}=Y_{1t},$ (3)

for $t=1,\ldots,T_{0}$. Under these conditions, a bound can be established on
the magnitude of the bias, $|E[\widehat{\tau}_{t}-\tau_{t}]|$, of the
synthetic control estimator. Abadie et al., (2010) provide a precise
expression for this bound. For the purpose of the present article, however, it
is enough to know that the bound increases with (i) the ratio between the
scale of the transitory shocks, $\epsilon_{jt}$, and the number of pre-
intervention periods, $T_{0}$, (ii) the number of units in the donor pool,
$J$, and (iii) the dimension of $\boldsymbol{\mu}_{j}$ (i.e., the number of
unobserved factors). The bias bound and equation (3) motivate including
$\boldsymbol{Z}_{j}$ and pre-treatment values of the outcomes in the vector of
matching variables, $\boldsymbol{X}_{j}$. This equation may only hold
approximately in practice if $\boldsymbol{X}_{1}$ is outside the convex hull
of the columns of $\boldsymbol{X}_{0}$.

Figure 1: Sparsity of Synthetic Controls: A Geometric Interpretation.

Note: Projection of $\boldsymbol{X}_{1}$ on the convex hull of the columns of
$\boldsymbol{X}_{0}$.

We next provide intuition for why (i) to (iii) may affect the size of the
bias. Notice that, under the model in (2), synthetic control weights that
reproduce the values of observed and unobserved covariates,

$\sum_{j=2}^{J}W_{j}^{*}\boldsymbol{Z}_{j}=\boldsymbol{Z}_{1}\quad\mbox{ and
}\quad\sum_{j=2}^{J}W_{j}^{*}\boldsymbol{\mu}_{j}=\boldsymbol{\mu}_{1},$ (4)

would yield an unbiased estimator of the treatment effect. Although the
unobserved covariates, $\boldsymbol{\mu}_{j}$, cannot be fitted directly, a
synthetic control such that (3) holds employs pre-treatment outcomes as
proxies for the unobserved factors. This would clearly be justified if the
scale of the transitory shocks is small, so most of the heterogeneity in pre-
treatment outcomes that does not come from $\boldsymbol{Z}_{j}$ is generated
by heterogeneity in $\boldsymbol{\mu}_{j}$. In that case, fitting pre-
treatment outcomes, as in (3), comes very close to fitting the unobserved
factors, as in (4). However, a substantial amount of variability in
$\epsilon_{jt}$ opens the door to over-fitting: the possibility that pre-
treatment outcomes are fitted in (3) out of variation in $\epsilon_{jt}$. In
that case, the fitted value for the unobserved factors given by the synthetic
control,

$\sum_{j=2}^{J+1}W^{*}_{j}\boldsymbol{\mu}_{j},$

may substantially differ from $\boldsymbol{\mu}_{1}$, inducing bias. The
probability that (3), or an approximate version of it, is mainly produced by
over-fitting is small when the scale of $\epsilon_{jt}$ is small and the
number of pre-treatment periods, $T_{0}$, to be fitted is large. A large donor
pool, however, increases the chances that the pre-treatment outcomes are
fitted out of variation in $\epsilon_{jt}$. Moreover, given that the linear
model in (2) is nothing but a local approximation to a data generating process
that could be non-linear, fitting the values of the treated units with
untreated units that are far from each other in the space of the predictors
may result in sizable interpolation biases. Finally, the dimension of
$\boldsymbol{\mu}_{j}$ reflects the variability induced by unobserved
covariates. In the absence of unobserved covariates, a synthetic control that
reproduces the value of $\boldsymbol{Z}_{1}$ only (as in the first part of
(4)) would be exactly unbiased.

From our discussion of the synthetic control design and the bias bound, we can
now distill seven guiding principles for empirical practice with synthetic
control estimators:

  1. 1.

Low volatility. Closely fitting a highly volatile series is likely the result
of over-fitting at work, especially if it happens over a short pre-
intervention period. Synthetic controls were designed for settings with
aggregate series, where aggregation attenuates the magnitude of the noise.

  2. 2.

Extended pre-intervention period. A good control unit must closely reproduce
the trajectory of the outcome variable for the treated unit over an extended
pre-intervention period. A good fit, if it is the result of a secular
agreement between the treated and the synthetic control units in their
responses to unobserved factors, should persist in time.

  3. 3.

Small donor pool. A larger donor pool is not necessarily better than a smaller
one. Adopting a small donor pool of untreated units that are close to the
treated unit in the space of the predictors helps reduce over-fitting and
interpolation biases.

  4. 4.

Sparsity makes synthetic controls interpretable.

  5. 5.

Covariates matter. A component of $\boldsymbol{Z}_{i}$ that is not controlled
for (that is, not included in $\boldsymbol{X}_{j}$) is effectively thrown into
$\boldsymbol{\mu}_{j}$. Fitting $\boldsymbol{Z}_{j}$ is easier than fitting
$\boldsymbol{\mu}_{j}$.

  6. 6.

Fit matters. The bound on the bias is predicated on close fit. A deficient fit
raises concerns about the validity of a synthetic control (see, however,
Ferman,, 2021; Ferman and Pinto,, 2021, for exceptions and qualifications on
this rule).

  7. 7.

Out-of-sample validation is key. The goal of synthetic controls is to predict
the trajectory of the outcome variable for the treated unit in the absence of
the intervention of interest. The quality of a synthetic control can be
assessed by measuring predictive power in pre-intervention periods that are
left out of the sample used to calculate the synthetic control weights.

We will study and illustrate the empirical relevance of the seven principles
described above. Principles 1-6 derive directly from the bias bound in Abadie
et al., (2010). Because the formal validity of the bias bound rests on strong
assumptions, it is useful to investigate the extent to which these principles
effectively translate to empirical practice. We will also illustrate the
effectiveness of validation techniques derived from the last principle on the
list.

We will employ simulations under a variety of data generating processes to
investigate how features of the data affect the performance of synthetic
control estimators in ways that can be predicted from the seven guiding
principles above. An important take-away of our analysis is that, as
previously argued in Abadie, (2021) and others, contextual and data
requirements are key for the performance of synthetic control estimators, and
should be carefully checked in empirical applications. Mechanistic
applications of the method, without regard to the guiding principles described
above, leave the results of the empirical exercise vulnerable to biases
created by over-fitting and by discrepancies between the values of the
predictors for the treated units and for the units that contribute to the
synthetic control. When conscientiously applied, however, synthetic controls
become powerful and transparent design tools for estimating the effects of
aggregate interventions.

##  3 Performance of the Synthetic Control Estimator

To begin our investigation of the performance of the synthetic control
estimator under a variety of features in the data, we adopt the grouped factor
model of Ferman, (2021), which is a special case of (2). We will use a
generative model with $F$ equally-sized groups of units denoted
$f(i)\in\\{1,\dots,F\\}$ and $F$ common factors, with each unit $i$ loading
exclusively on factor $f(i)$,

$\displaystyle Y_{it}^{N}=\delta_{t}+\lambda_{f(i)t}+\epsilon_{it}.$ (5)

The common factors, $\lambda_{ft}$, follow an $AR(1)$ processes with
autoregressive coefficient $\rho$ and standard Gaussian innovations.
$\epsilon_{it}$ follows a Gaussian distribution, with mean zero and variance
$\sigma^{2}$, independent of any other component.

We assume that only the first unit is treated and that only the second unit
loads on the same factor as the treated unit, so $f(1)=f(2)$. The other
untreated units—that is, units $j=3,\ldots,J+1$—are also grouped into pairs
that load on a pair-specific factor, $f(3)=f(4)$, $f(5)=f(6)$, and so on. An
unbiased synthetic control in this setting is one with $W^{*}_{2}=1$ and
$W_{3}^{*}=W^{*}_{4}=\cdots=W^{*}_{J+1}=0$. This paired design represents the
least favorable case for synthetic control estimators under the grouped factor
model of Ferman, (2021): there is only one unit that reproduces the trajectory
of $\lambda_{f(i)t}$ for the treated unit in the absence of the intervention.

Unless we state otherwise, we will impose $Y^{I}_{1t}=Y^{N}_{1t}$ for
$t=T_{0}+1,\ldots,T$, so the treatment effect on the treated unit is equal to
zero. This allows us to interpret deviations from zero in treatment effect
estimates as reflective of lack of accuracy or precision. We fix $T=30$ and
study the performance of the synthetic control estimator (and the extent to
which the estimated weights approximate $W^{*}_{2}=1$) under a variety of
values for $\sigma$, $T_{0}$, $\rho$ and $F$. We will also investigate the
extent to which variations on the basic estimator improve performance or
diminish it.

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure 2: Pre-treatment fit and estimation error.

Note: Synthetic control estimates for the grouped factor model in equation
(2), with $\rho=0.5$, $T_{0}=20$, and $J+1=20$ ($F=10$). Each panel reports
results for a different value of $\sigma$.

Figure 2 reports simulations of the synthetic control estimator for the case
$T_{0}=20$, $\rho=0.5$, and $J=19$ (so $F=10$). Each panel contains results
for a different value for $\sigma$. With a large value of $\sigma$ in panel
(a), the pre-treatment fit is poor, which translates in large estimation
errors during the post-treatment periods. As $\sigma$ decreases in the
subsequent panels, the pre-treatment fit improves noticeably and it translates
into similar precision gains during the post-intervention periods. These gains
in estimation accuracy are reflected in substantial increases in the value of
$W^{*}_{2}$ as $\sigma$ decreases. With $\sigma=2$ in panel (a), the synthetic
control assigns less than half of the weight the the “correct” unit, $j=2$.
When $\sigma=0.25$ in panel (d), the weight of unit $j=2$ on the synthetic
control estimator is almost one. The pattern of results in Figure 2 is
reflective of the fact that the bias of the synthetic control estimator and
the quality of the pre-treatment fit both depend on the scale of the
individual transitory shocks, $\epsilon_{jt}$. Under the generative process in
(2), lack of pre-treatment fit can arise because of noise in the series (large
$\sigma$) or because the values of $\boldsymbol{Z}_{1}$ and
$\boldsymbol{\mu}_{1}$ cannot be closely reproduced with a convex combination
of the values of $\boldsymbol{Z}_{j}$ and $\boldsymbol{\mu}_{j}$ for the units
in the donor pool. In both cases, post-treatment synthetic control estimates
could incorporate sizable biases. The results in Figure 2 underscore the
importance of a good pre-treatment fit.

In order to visually demonstrate the performance of synthetic controls “in
action”, Figure 2 reports the results for one random realization in our
simulation design only. One could, instead, run a large number of simulations
and plot or tabulate results on pre-treatment fit and post-treatment bias.
Figure 3 plots results for 10000 simulations, with the same simulation design
as in Figure 2. Like Figure 2, the new Figure 3 indicates substantial
increases in estimation accuracy that are associated with pre-intervention
fit. For the rest of this article, we report results of single random
simulations in the main text of the article. In the appendix we report the
analogous results calculated over a large number of simulations.

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure 3: Prediction error over 10000 simulations

Note: 95% bands for the simulation design of Figure 2.

Figure 4 shows the results of a simulation that employs the same data
generating process as in Figure 2 with the exception that now the series
$\lambda_{ft}$ incorporate stochastic trends. As for Figure 2, the magnitude
of the noise in $\epsilon_{jt}$ is key for pre-treatment fit and post-
treatment prediction error. Notice, however, that heterogeneity in the
trending behavior of the series helps with the selection of a synthetic
control heavily based on the outcome series of unit 2, which is affected by
the same stochastic trend as the treated unit. In addition to the outcome
series for the treated unit and the synthetic controls, Figure 4 also reports
the average outcome path for all untreated units. Figure 4 shows large gains
from synthetic control estimation, relative to a simple average of the outcome
for the untreated units. These gains are created by heterogeneity across units
in the outcome trends, something to which the synthetic control method is able
to adapt.

The result of Figure 4 is rather general. In a model like (2), non-stationary
factor components help identify synthetic controls that reproduce the values
of the $\boldsymbol{\mu}_{j}$ for the treated. This is again the case in
Figure 5, which repeats the analysis of Figure 2 but with $\rho=1$, so the
series $\lambda_{ft}$ are non-stationary. As in Figure 4, the large variation
in factor components cannot be compensated by variation in $\epsilon_{jt}$.
This results in minimal over-fitting in Figure 5, with $W^{*}_{2}$ close to
one for all values of $\sigma$ in the figure.

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure 4: Pre-treatment fit and estimation error with a stochastic trend

Note: Synthetic control estimates for the grouped factor model in equation
(2), with $\rho=0.5$, $T_{0}=20$, and $J+1=20$ ($F=10$). Relative to Figure 2,
the series $\lambda_{ft}$ incorporate stochastic trends. The average control
assigns equal weights to all the untreated units.

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure 5: Pre-treatment fit and estimation error with $\rho=1$

Note: Synthetic control estimates for the grouped factor model in equation
(2), with $\rho=1$, $T_{0}=20$, and $J+1=20$ ($F=10$). Each panel reports
results for a different value of $\sigma$. The average control assigns equal
weights to all the untreated units.

Obtaining a good pre-treatment fit in the outcome variable (but also in other
predictors of the outcome, as we will discuss later) is an important requisite
for the performance of synthetic control estimators. However, good pre-
treatment fit is not in itself a guarantee of good performance. The reason is
that, in some scenarios, a good pre-treatment fit may be obtained from
variation in the individual transitory shocks, $\epsilon_{jt}$, even when the
selected synthetic control does not come close to reproducing the values of
$\boldsymbol{\mu}_{j}$ in equation (2) for the treated. This is what we have
previously referred to as over-fitting.

A researcher employing synthetic controls should be able to identify settings
conductive of over-fitting, and to modify the design of a study in order to
avoid or attenuate over-fitting biases. We devote much of the rest of the
article to discussing prevention, detection, and correction of over-fitted
estimators. We discuss the scenarios where over-fitting is likely to bias
synthetic control estimators and how to avoid those biases.

(a) $T_{0}=20$, $F=10$ $(J=19)$

(b) $T_{0}=4$, $F=10$ $(J=19)$

(c) $T_{0}=20$, $F=100$ $(J=199)$

(d) $T_{0}=4$, $F=100$ $(J=199)$

Figure 6: Over-fitting with a short pre-treatment period and many donor units

Note: Synthetic control estimates for different values of $T_{0}$ and $J$,
with $\rho=1$ and $\sigma=0.5$.

Figure 6 illustrates the effect of small $T_{0}$ and large $J$ on over-
fitting. Panel (a) of Figure 6 reproduces panel (c) in Figure 5, so $\rho=1$
and $\sigma=0.5$. In panel (a), the number of pre-treatment periods is
$T_{0}=20$. Heterogeneity in the processes that govern the trajectories of the
factors, $\lambda_{ft}$, identifies the ideal synthetic control with
$W^{*}_{2}=1$. That is, panel (a) depicts a setting with no over-fitting or
bias. Variation in the transitory shocks, $\epsilon_{jt}$, is equally well
represented in the pre-treatment and post-treatment differences between the
outcomes for the treated unit and the outcomes for the synthetic control. As
we will see next, however, a small number of pre-intervention periods,
$T_{0}$, or a larger number of donor pool units, $J$, may create over-fitting,
breaking the close correspondence between pre-treatment fit and post-treatment
error. Consider now panel (b), where the design of the simulation is the same
as in panel (a) but with only $T_{0}=4$ pre-intervention periods. The fit in
pre-intervention outcomes is now perfect. However, the small number of pre-
intervention periods induces over-fitting, which creates large post-treatment
estimation errors and a reduced role of the second sample unit in the
synthetic control, with $W^{*}=0.79$. Panel (c) goes back to $T_{0}=20$, as in
panel (a), but increases the number of units in the donor pool to $J=199$
(that is, $F=100$). Similar to the effect of a small number of pre-treatment
periods in panel (b), the increased number of units in the donor pool opens
the door to over-fitting in panel (c). The post-treatment estimation error in
panel (c) is much larger than in panel (a), even when the pre-treatment fit is
better in panel (c). Moreover, the role of the second unit in the synthetic
control is reduced to $W^{*}_{2}=0.53$. Finally, panel (d) depicts the setting
with $T_{0}=4$ and $J=199$. As in panel (b), the pre-treatment fit is perfect.
However, post-treatment estimation error is very large and the second unit
does not contribute to the synthetic control, $W_{2}^{*}=0$. On the whole,
Figure 6 illustrates how the number of pre-treatment periods and the number of
untreated units crucially affect the bias of the synthetic control estimator.

Apart from the possibility of over-fitting bias created by an excessively
large $J$, preventing interpolation bias is another good reason to place
restrictions on the untreated units that are allowed to enter the donor pool.
Consider the factor model in equation (2). Even if $\epsilon_{jt}$ lacks
variation and, therefore, over-fitting bias is not a concern, the linearity
assumption in equation (2) may only be a local approximation. If that is the
case, reproducing the predictor values for the treated unit by averaging
untreated units that are far from the treated unit in the space of the
predictors may result in interpolation biases. To attenuate interpolation
biases, it is useful to restrict the units allowed in the donor pool to
untreated units with similar values in the predictors, $\boldsymbol{X}_{j}$,
as the treated unit.

As we have discussed above, researchers applying synthetic control estimators
should recognize settings conducive to biases—because of small values of
$T_{0}$, large values of $J$, or because the units in the donor pool have
values of the predictor that are substantially different from those of the
treated unit—and adapt their designs to ameliorate bias concerns. In Section
4, we will discuss validation techniques to assess the potential for bias in
concrete empirical settings.

We finish this section with a discussion of how modifications of the synthetic
control design may affect the performance of the estimates. Panel (a) of
Figure 7 plots the outcome values for a treated unit and its synthetic
control, for a simulation design that is the same as for panel (d) or Figure 2
except that $T_{0}=15$. Because this is a simple setting without covariates,
the synthetic control estimate of $Y^{N}_{1t}$ in panel (a) is

$\sum_{j=2}^{J+1}W^{*}_{j}Y_{jt},$

with $W^{*}_{2},\ldots,W^{*}_{J+1}$ chosen to minimize

$\sum_{t=1}^{T_{0}}\Big{(}Y_{1t}-\sum_{j=2}^{J+1}W_{j}Y_{jt}\Big{)}^{2}$

with respect to $W_{2},\ldots,W_{J+1}$, subject to the constrain that the
weights, $W_{2},\ldots,W_{J+1}$, are non-negative and sum up to one. The pre-
treatment root-mean-square error (pre-RMSE),

$\left(\frac{1}{T_{0}}\sum_{t=1}^{T_{0}}\Big{(}Y_{1t}-\sum_{j=2}^{J+1}W^{*}_{j}Y_{jt}\Big{)}^{2}\right)^{1/2},$
(6)

is equal to 0.23. The post-treatment root-mean-square error (post-RMSE)–which
is analogous to (6), but calculated for the post-treatment periods—is somewhat
higher, 0.33. Still, in this example, pre-intervention fit comes reasonably
close to estimation accuracy in the post-intervention periods. As we will see
next, adding extra flexibility to the synthetic control estimator may break
down the correspondence between pre-RMSE and post-RMSE. Consider now panel (b)
of Figure 7. It reports the results of a synthetic control estimator with an
added constant shift (Doudchenko and Imbens,, 2016). That is, panel (b) plots
the result of estimating of $Y^{N}_{1t}$ as

$\widehat{Y}^{N}_{1t}=\widehat{\alpha}+\sum_{j=2}^{J+1}\widehat{W}_{j}Y_{jt},$
(7)

where $\widehat{\alpha},\widehat{W}_{2},\ldots,\widehat{W}_{J+1}$ minimize

$\sum_{t=1}^{T_{0}}\Big{(}Y_{1t}-\alpha-\sum_{j=2}^{J+1}W_{j}Y_{jt}\Big{)}^{2}$
(8)

with respect to $\alpha,W_{2},\ldots,W_{J+1}$, subject to the constrain that
the weights, $W_{2},\ldots,W_{J+1}$, are non-negative and sum up to one.
Introducing the constant shift $\alpha$, as in (8) is equivalent to computing
a synthetic control with the outcome variables measured in deviations with
respect to their pre-treatment means

$Y_{jt}-\frac{1}{T_{0}}\sum_{k=1}^{T_{0}}Y_{jk}.$

The availability of the constant $\alpha$ increases the degrees of freedom of
the synthetic control estimator. This extra flexibility could be useful in
certain instances, especially when the trajectory of the outcome for the
treated unit is extreme relative to the trajectory of the outcomes in the
donor pool, so a good fit is not attainable on the levels of $Y_{jt}$, but may
be attainable on the demeaned outcomes. It is good to remember, however, that
additional flexibility increases the potential for over-fitting. In the
example of Figure 7, introducing a constant shift in the estimate may seem to
be warranted at first sight. After all, the pre-treatment trajectory of the
synthetic control seems to over-estimate the values of the treated outcomes.
Indeed, panel (b) shows that the inclusion of the constant shift noticeably
improves pre-treatment fit, with pre-RMSE dropping to 0.14. However, post-RMSE
increases to 0.44. Panel (c) allows for additional flexibility relaxing the
constraints on the weights (Hsiao et al.,, 2012; Doudchenko and Imbens,,
2016). A simple regression estimator of this model takes the same form as
$\widehat{Y}^{N}_{1t}$ in equation (7), but with parameters
$\widehat{\alpha},\widehat{W}_{2},\ldots,\widehat{W}_{J+1}$ that minimize (8)
for unrestricted values of $\alpha,W_{2},\ldots,W_{J+1}$. Panel (c) in Figure
7 plots the result. Now, pre-RMSE is equal to zero, the unrestricted
regression perfectly fits the trajectory of the outcome variable for the
treated in the pre-treatment periods. However, the post-RMSE is 0.54,
indicating a larger out-of-sample estimation error than in panels (a) and (b).

The increased risk of over-fitting that comes from increased flexibility in
synthetic control estimation is well-recognized in the literature. At least
since Doudchenko and Imbens, (2016), researchers extending synthetic controls
beyond the convex hull of the predictors’ values for donor pool units have
often combined the extra flexibility in the parameters with some
regularization device—usually, but not exclusively, in the form of $L_{1}$
and/or $L_{2}$ penalizations on the synthetic control parameters (see, in
particular, Amjad et al.,, 2018; Agarwal et al.,, 2021; Arkhangelsky et al.,,
2019; Athey et al.,, 2021; Ben-Michael et al.,, 2021; Chernozhukov et al.,,
2021; Doudchenko and Imbens,, 2016).

Figure 7 illustrates the effectiveness of restricting synthetic control
weights to be non-negative and sum up to one as a regularization device.
Relaxing the restrictions on the synthetic control weights, or adopting bias-
correction techniques, as in Abadie and L’Hour, (2021) and Ben-Michael et al.,
(2021), extends the applicability of the synthetic control method to settings
where the values of the predictors for the treated unit cannot be closely
fitted by a weighted average of the units in the donor pool. However, the
increased flexibility in the synthetic control weights is obtained at the
expense of the straightforward interpretability of the estimates. In addition,
relaxing the restrictions on the synthetic control weights allows for
unchecked extrapolation outside the support of the data, increasing the risk
that the estimates reflect extreme counterfactuals, for which the data contain
little information (King and Zeng,, 2006).

(a) Synthetic control

(b) Synthetic control with constant shift

(c) Synthetic control via unrestricted regression

Figure 7: Over-fitting with added flexibility

Note: Synthetic control estimates with added flexibility. Panel (a) reports
results for the standard synthetic control estimator with weight constraints.
Panel (b) reports results for a synthetic control model that includes a
constant shift. Panel (c) reports results for a synthetic control estimated
via unrestricted regression. $\sigma=0.25$, $T_{0}=15$, $\rho=0.5$ and $J=19$.

##  4 Validating the Synthetic Control Estimator

In Section 3 we have discussed how the scale of the transitory shocks, the
number of pre-treatment periods, and the size of the donor pool influence the
bias of the synthetic control estimator. In any particular empirical setting,
however, a researcher could be uncertain about potential exposure to over-
fitting and interpolation biases. In this section, we discuss validation
techniques to help assess the magnitude of potential biases in concrete
empirical applications.

An effective and visually interpretable validation exercise for synthetic
controls can be obtained by artificially backdating the treatment of interest
in the data. Figure 8 reports the results of backdating the treatment in the
simulation setting of Figure 2. Recall that in this simulated scenario, the
last pre-treatment period is $T_{0}=20$. In Figure 8, we have backdated the
treatment so that the last pre-treatment period is now $T_{0}^{\,b}=10$. This
implies that we calculate the synthetic control weights using data for
$t=1,\ldots,T_{0}^{\,b}$ only. Backdating the treatment to $T_{0}^{\,b}=10$
creates ten hold-out periods, $t=11,\ldots,20$, available to validate the
predictions of the synthetic control estimator. Notice that this validation
exercise could be performed at $t=T_{0}$, before post-treatment data can
possibly be observed, to inform study design decisions. An analyst carrying
out this validation exercise at $t=T_{0}$ would observe large out-of-sample
prediction errors for $\sigma=2$ and $\sigma=1$ in panels (a) and (b), but
much more reliable predictions for $\sigma=0.5$ and especially for
$\sigma=0.25$, in panels (c) and (d) respectively. In Figure 8, the quality of
the prediction in the post-treatment periods, $t=21,\ldots,30$ closely mirrors
the prediction error for the hold-out periods, $t=11,\ldots,20$.

Figure 9 employs the same simulation design as in Figure 8, but with a
positive treatment effect in the post-treatment periods. The effect of the
treatment is detected in panel (a), where the synthetic control is trained
using data up to period $T_{0}$. Panel (b) reports the result of the same
backdating exercise as in Figure 8. Not only is the out-of-sample fit good
during the hold-out periods, $t=11,\ldots,20$, but the fact that an effect
emerges at $T_{0}$, exactly the time when the true (in this case, simulated)
treatment takes place and that the magnitudes of the estimates are similar in
panels (a) and (b) provides additional credibility to the synthetic control
estimates of panel (a).

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure 8: Synthetic control validation

Note: Synthetic control estimates with treatment backdated to $T_{0}^{b}=10$
for different values of $\sigma$. $T_{0}=20$, $\rho=1$ and $J=19$.

(a) Intervention at $T_{0}=20$

(b) Backdating to $T_{0}^{\,b}=10$

Figure 9: Backdating with a treatment effect.

Note: Synthetic control estimates with a non-zero treatment effect and
treatment backdated to $T_{0}^{b}=10$. $\sigma=0.25$, $T_{0}=20$, $\rho=0.5$
and $J=19$.

##  5 Trimming the Donor Pool

In Sections 2 and 3, we have discussed the importance controlling the size of
the donor pool. In this section, we will show how actively trimming the donor
pool to units with values of the predictors that are close to the values of
the predictors for the treated unit can substantially improve the performance
of synthetic control estimators.

(a) Without trimming.

(b) 75% trimming.

(c) 90% trimming.

(d) 95% trimming.

Figure 10: Synthetic control with trimming.

Note: Synthetic control with and without trimming for $\sigma=0.25$,
$T_{0}=20$, $\rho=1$ and $J=199$. Panel (a) shows the standard synthetic
control. Panel (b) shows the synthetic control with the donor pool trimmed to
the $25\%$ closest units to the treated unit according to the Euclidean
distance. Panel (c) and (d) are like panel (b) with trimming to $10\%$ and
$5\%$ of the units respectively.

The first panel of Figure 10 reports the result of a simulation based on the
grouped factor model in equation (5), with $\rho=1$, $\sigma=0.25$,
$T_{0}=20$, and $F=100$ ($J=199$). In this simulation, the large size of the
donor pool promotes over-fitting resulting in large post-treatment estimation
errors. In panels (b)-(d) of Figure 10 we have only retained the 50, 20, and
10 donor pool units, respectively, that best fit the values of the predictors
for the treated (in this simple case, pre-treatment outcomes only). Trimming
mechanically increases pre-treatment differences between the treated unit and
the synthetic control, but noticeably improves the performance of the
estimator in the post-treatment periods.

##  6 The Role of Observed Covariates

Thus far, we have only considered settings without covariates,
$\boldsymbol{Z}_{j}$, in our simulations. In this section we discuss how the
inclusion of observed covariates influences the performance of synthetic
control estimators.

To illustrate the role of observed covariates, we consider the linear factor
model in (2), with $\delta_{t}=100$, $\boldsymbol{Z}_{j}$ and
$\boldsymbol{\mu}_{j}$ uniformly distributed on $[0,20]$, and where the
components of $\boldsymbol{\theta}_{t}$ and $\boldsymbol{\lambda}_{t}$ follow
independent random walks with standard Gaussian innovations. The
idiosynchratic errors, $\epsilon_{jt}$, are modelled as Gaussian noise with
standard deviation equal to five. In Figure 11 we adopt a setting with
$J=1000$ and $T_{0}=4$ and five covariates. Panel (a) considers the case where
the five covariates are unobserved (so $F=5$) and there are no observed
covariates, so a synthetic control estimator is based on pre-treatment
outcomes only. The large value of $J$, the small value of $T_{0}$, and the
presence of unobserved factors creates substantial over-fitting and estimation
bias. In panels (b)-(f), we incrementally shift covariates from
$\boldsymbol{\mu}_{j}$ to $\boldsymbol{Z}_{j}$, by including these covariates
in the vectors $\boldsymbol{X}_{j}$ of predictive variables. That is, in
contrast to panel (a), panels (b)-(f) use incrementally larger numbers of
observed covariates, in addition to pre-treatment outcomes, to fit a synthetic
control. In panel (f) there are no unobserved factors ($F=0$): all five
covariates are included in $\boldsymbol{Z}_{j}$ (and, therefore, in
$\boldsymbol{X}_{j}$). The patterns of fit and estimation error in Figure 11
illustrate the importance of including predictive covariates in
$\boldsymbol{X}_{j}$. Adding covariates to $\boldsymbol{X}_{j}$ induces a
slight deterioration in pre-treatment fit in the outcomes, given that other
covariates (apart from pre-treatment outcomes) enter $\boldsymbol{X}_{j}$.
However, reductions in $F$ are associated with substantial decreases in
estimation error. As we increase the number of observed covariates, we rely
less on pre-treatment outcomes as imperfect proxies for the values of the
covariates, obtaining more reliable estimates. Figure 11 underscores the
potential importance of including observed predictors of the outcomes, beyond
pre-treatment outcomes, in the set of variables used to fit a synthetic
control.

(a) $F=5$

(b) $F=4$

(c) $F=3$

(d) $F=2$

(e) $F=1$

(f) $F=0$

Figure 11: Including observed covariates

Note: Synthetic control under different values of $F$, the number of
covariates not included in $\boldsymbol{X}_{j}$. See main text for a detailed
description of the data generating process.

##  7 An Auto-Regressive Model

Although we have thus far used a linear factor structure in our simulations,
it is important to notice that the validity of synthetic control estimators is
not limited to the linear factor model in (2). In particular, Abadie et al.,
(2010) study the bias behavior of synthetic control estimators under a
generative model with a vector auto-regressive structure in covariates and
outcomes. In this section, we illustrate the applicability of synthetic
control estimators under an auto-regressive data generating process. For that
purpose, we consider a simple version, without covariates, of the auto-
regressive model in Abadie et al., (2010),

$Y_{jt}^{N}=\alpha_{jt}Y_{jt-1}^{N}+\epsilon_{jt},$

for $j=1,\ldots,J+1$ and $t=2,\ldots,T$, and $Y_{1t}^{I}=Y_{1t}^{N}$, for
$t=T_{0}+1,\ldots,T$. This is an $AR(1)$ model with coefficients $\alpha_{jt}$
that are allowed to vary in time and across units. For the simulation results,
we adopt $J+1=50$ and, as before, $T_{0}=20$ and $T=30$. We model
$Y^{N}_{j1}$—the initial value of the outcome—for units $j=1,\ldots J+1$, as a
Gaussian variable with mean 100 and standard deviation equal to 20. The error
terms, $\epsilon_{jt}$, are standard Gaussian and independent of each other,
for $j=1,\ldots J+1$ and $t=1,\ldots,T$. For $\alpha_{jt}$, we adopt a grouped
structure, with five groups of 10 units. Inside the groups,
$\alpha_{j2},\ldots,\alpha_{jT}$ are constant across units and drawn as
independent Gaussian variables with mean equal to one and standard deviation
equal to 0.1. The series $\alpha_{j2},\ldots,\alpha_{jT}$ are independent
across groups.

The result of a random simulation under this data generating process is
reported in Figure 12. In this simulation, because of the assumed group-level
heterogeneity in the trajectory of the outcome, the average of the outcome
variable among all the untreated units is not able to reproduce the outcome
for the treated unit, before or after the treatment is implemented. In
contrast, the synthetic control estimates closely follow the trajectory of the
outcome for the treated, demonstrating the applicability of synthetic control
estimators outside the linear factor model considered in previous sections.

Figure 12: Synthetic control with an auto-regressive process.

Note: Synthetic control under an auto-regressive model for the outcome
variable. $T_{0}=20$, $J+1=50$. See text of Section 7 for a detailed
description of the design of the simulation.

##  8 Conclusion

Synthetic controls are intuitive, transparent, and produce reliable estimates
for a variety of data generating processes. However, like for any other
statistical or econometric method to estimate treatment effects, there are
settings where synthetic controls may fail. In this article, we have described
the settings where the synthetic control method provides reliable estimates,
and those where it does not. Of particular importance, we have discussed the
nature of over-fitting in synthetic control estimators, when it arises, how to
recognize it, and ways to alleviate or eliminate over-fitting biases.

Moreover, we have described intuitive and effective diagnosis checks for the
validity of synthetic control estimates that can be easily carried out in
empirical research. They pertain to the quality of the fit and length of the
pre-intervention period, the ability to fit the treated unit using a
relatively small donor pool of similar units, the ability to validate out-of-
sample, and that a treatment effect estimate emerges at the time of the actual
treatment implementation, even when the treatment is artificially backdated in
the data.

Our recommendations are summarized in seven guiding principles for the
research design of empirical studies that employ synthetic controls.

## References

  * Abadie, (2021) Abadie, A. (2021).  Using synthetic controls: Feasibility, data requirements, and methodological aspects.  Journal of Economic Literature, 59(2):391–425.
  * Abadie et al., (2010) Abadie, A., Diamond, A., and Hainmueller, J. (2010).  Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program.  Journal of the American Statistical Association, 105(490):493–505.
  * Abadie and L’Hour, (2021) Abadie, A. and L’Hour, J. (2021).  A penalized synthetic control estimator for disaggregated data.  Journal of the American Statistical Association.  Forthcoming.
  * Acemoglu et al., (2016) Acemoglu, D., Johnson, S., Kermani, A., Kwak, J., and Mitton, T. (2016).  The value of connections in turbulent times: Evidence from the United States.  Journal of Financial Economics, 121:368–391.
  * Agarwal et al., (2021) Agarwal, A., Shah, D., and Shen, D. (2021).  Synthetic interventions.  arXiv e-print, 2006.07691.
  * Amjad et al., (2018) Amjad, M., Shah, D., and Shen, D. (2018).  Robust synthetic control.  Journal of Machine Learning Research, 19(22):1–51.
  * Arkhangelsky et al., (2019) Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., and Wager, S. (2019).  Synthetic difference in differences.  NBER Working Paper, 25532.
  * Athey et al., (2021) Athey, S., Bayati, M., Doudchenko, N., Imbens, G., and Khosravi, K. (2021).  Matrix completion methods for causal panel data models.  Journal of the American Statistical Association.  Forthcoming.
  * Ben-Michael et al., (2021) Ben-Michael, E., Feller, A., and Rothstein, J. (2021).  The augmented synthetic control method.  Journal of the American Statistical Association.  Forthcoming.
  * Chernozhukov et al., (2021) Chernozhukov, V., Wüthrich, K., and Zhu, Y. (2021).  An exact and robust conformal inference method for counterfactual and synthetic controls.  Journal of the American Statistical Association.  Forthcoming.
  * Cunningham and Shah, (2018) Cunningham, S. and Shah, M. (2018).  Decriminalizing indoor prostitution: Implications for sexual violence and public health.  The Review of Economic Studies, 85(3):1683–1715.
  * Donohue et al., (2019) Donohue, J. J., Aneja, A., and Weber, K. D. (2019).  Right-to-carry laws and violent crime: A comprehensive assessment using panel data and a state-level synthetic control analysis.  Journal of Empirical Legal Studies, 16(2):198–247.
  * Doudchenko and Imbens, (2016) Doudchenko, N. and Imbens, G. W. (2016).  Balancing, regression, difference-in-differences and synthetic control methods: A synthesis.  NBER Working Paper, 22791.
  * Ferman, (2021) Ferman, B. (2021).  On the properties of the synthetic control estimator with many periods and many controls.  Journal of the American Statistical Association.  Forthcoming.
  * Ferman and Pinto, (2021) Ferman, B. and Pinto, C. (2021).  Synthetic controls with imperfect pre-treatment fit.  Quantitative Economics.  Forthcoming.
  * Hsiao et al., (2012) Hsiao, C., Steve Ching, H., and Ki Wan, S. (2012).  A panel data approach for program evaluation: Measuring the benefits of political and economic integration of Hong Kong with mainland China.  Journal of Applied Econometrics, 27(5):705–740.
  * King and Zeng, (2006) King, G. and Zeng, L. (2006).  The dangers of extreme counterfactuals.  Political Analysis, 14(2):131–159.
  * Rubin, (1974) Rubin, D. B. (1974).  Estimating causal effects of treatments in randomized and nonrandomized studies.  Journal of Educational Psychology, 66(5):688.

## Appendix

Table A.1 summarizes the results for all figures by providing the average pre-
RMSE, post-RMSE and $W_{2}$ (the weight assigned to the second unit) over
10000 simulations. Figures A.4-12 show 95% bands over 10000 simulations of the
estimation error (prediction minus actual) for the same simulation designs as
in Figures 4-12, respectively.

Table A.1: Simulation results for all figures. Figure | $J$ | $T_{0}$ | $\sigma$ | $\rho$ | post-RMSE | pre-RMSE | $W^{*}_{2}$
---|---|---|---|---|---|---|---
Figure 2 (a) | 19 | 20 | 0.25 | 0.5 | 0.365 | 0.315 | 0.890
Figure 2 (b) | 19 | 20 | 0.5 | 0.5 | 0.723 | 0.588 | 0.730
Figure 2 (c) | 19 | 20 | 1 | 0.5 | 1.362 | 1.032 | 0.429
Figure 2 (d) | 19 | 20 | 2 | 0.5 | 2.421 | 1.782 | 0.171
Figure 4 (a) | 19 | 20 | 0.25 | 1 | 0.387 | 0.341 | 0.983
Figure 4 (b) | 19 | 20 | 0.5 | 1 | 0.795 | 0.675 | 0.953
Figure 4 (c) | 19 | 20 | 1 | 1 | 1.642 | 1.315 | 0.882
Figure 4 (d) | 19 | 20 | 2 | 1 | 3.272 | 2.513 | 0.763
Figure 5 (a) | 19 | 20 | 0.25 | 1 | 0.439 | 0.320 | 0.920
Figure 5 (b) | 19 | 20 | 0.5 | 1 | 0.921 | 0.609 | 0.799
Figure 5 (c) | 19 | 20 | 1 | 1 | 1.799 | 1.108 | 0.564
Figure 5 (d) | 19 | 20 | 2 | 1 | 3.117 | 1.965 | 0.316
Figure 6 (a) | 19 | 4 | 0.5 | 1 | 2.595 | 0.282 | 0.397
Figure 6 (b) | 199 | 4 | 0.5 | 1 | 3.305 | 0.057 | 0.080
Figure 6 (c) | 19 | 20 | 0.5 | 1 | 0.921 | 0.609 | 0.799
Figure 6 (d) | 199 | 20 | 0.5 | 1 | 1.308 | 0.484 | 0.529
Figure 7 (a) | 19 | 15 | 0.25 | 0.5 | 0.381 | 0.304 | 0.874
Figure 7 (b) | 19 | 15 | 0.25 | 0.5 | 0.405 | 0.289 | 0.860
Figure 7 (c) | 19 | 15 | 0.25 | 0.5 | 1.019 | 0.000 |
Figure 8 (a) | 19 | 20 | 0.25 | 0.5 | 0.415 | 0.356 | 0.841
Figure 8 (b) | 19 | 20 | 0.5 | 0.5 | 0.829 | 0.692 | 0.631
Figure 8 (c) | 19 | 20 | 1 | 0.5 | 1.515 | 1.234 | 0.320
Figure 8 (d) | 19 | 20 | 2 | 0.5 | 2.579 | 2.114 | 0.131
Figure 9 (a) | 19 | 20 | 0.25 | 0.5 | 1.694 | 0.356 | 0.841
Figure 9 (b) | 19 | 20 | 0.25 | 0.5 | 1.682 | 0.315 | 0.890
Figure 10 (d) | 199 | 20 | 0.25 | 1 | 0.495 | 0.315 | 0.875
Figure 10 (c) | 199 | 20 | 0.25 | 1 | 0.524 | 0.305 | 0.853
Figure 10 (b) | 199 | 20 | 0.25 | 1 | 0.562 | 0.291 | 0.826
Figure 10 (a) | 199 | 20 | 0.25 | 1 | 0.637 | 0.274 | 0.786
Figure 11 (a) | 1000 | 4 | 5 | 1 | 38.153 | 0.661 |
Figure 11 (b) | 1000 | 4 | 5 | 1 | 34.967 | 1.080 |
Figure 11 (c) | 1000 | 4 | 5 | 1 | 26.531 | 1.250 |
Figure 11 (d) | 1000 | 4 | 5 | 1 | 20.531 | 1.438 |
Figure 11 (e) | 1000 | 4 | 5 | 1 | 12.776 | 1.675 |
Figure 11 (f) | 1000 | 4 | 5 | 1 | 7.616 | 1.811 |
Figure 12 | 49 | 20 | 1 |  | 4.093 | 2.073 |

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure A.4: Synthetic control error simulations with a stochastic trend.

Note: 95% bands for the simulation design of Figure 4.

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure A.5: Synthetic control error simulations when $\rho=1$.

Note: 95% bands for the simulation design of Figure 5.

(a) $T_{0}=20$, $F=10$ $(J=19)$

(b) $T_{0}=4$, $F=10$ $(J=19)$

(c) $T_{0}=20$, $F=100$ $(J=199)$

(d) $T_{0}=4$, $F=100$ $(J=199)$

Figure A.6: Synthetic control error simulations and over-fitting.

Note: 95% bands for the simulation design of Figure 6.

(a) Standard SC

(b) SC with constant shift

(c) SC via regression

Figure A.7: Synthetic control error simulations for different weight
restrictions.

Note: 95% bands for the simulation design of Figure 7.

(a) $\sigma=2$

(b) $\sigma=1$

(c) $\sigma=0.5$

(d) $\sigma=0.25$

Figure A.8: Synthetic control error simulations with a validation period.

Note: 95% bands for the simulation design of Figure 8.

(a) Intervention at $T_{0}=20$

(b) Backdating to $T=10$

Figure A.9: Synthetic control error simulations with treatment effect.

Note: 95% bands for the simulation design of Figure 9.

(a) Without trimming.

(b) 75% trimming.

(c) 90% trimming.

(d) 95% trimming.

Figure A.10: Synthetic control error simulation with trimming.

Note: 95% bands for the simulation design of Figure 10

(a) $F=5$

(b) $F=4$

(c) $F=3$

(d) $F=2$

(e) $F=1$

(f) $F=0$

Figure A.11: Synthetic control error simulations with observed covariates.

Note: 95% bands for the simulation design of Figure 11.

Figure A.12: Synthetic control error simulations with an auto-regressive
process.

Note: 95% bands for the simulation design of Figure 12.