# Asymptotic normality of a change plane estimator in fixed dimension with
near-optimal rate

Debarghya Mukherjee†, Moulinath Banerjee†,
Debasri Mukherjee∗ and Ya’acov Ritov†

†Department of Statistics, University of Michigan
∗Department of Economics, Western Michigan University

###### Abstract

Linear thresholding models postulate that the conditional distribution of a
response variable in terms of covariates differs on the two sides of a
(typically unknown) hyperplane in the covariate space. A key goal in such
models is to learn about this separating hyperplane. Exact likelihood or least
squares methods to estimate the thresholding parameter involve an indicator
function which make them difficult to optimize and are, therefore, often
tackled by using a surrogate loss that uses a smooth approximation to the
indicator. In this paper, we demonstrate that the resulting estimator is
asymptotically normal with a near optimal rate of convergence: $n^{-1}$ up to
a log factor, in both classification and regression thresholding models. This
is substantially faster than the currently established convergence rates of
smoothed estimators for similar models in the statistics and econometrics
literatures. We also present a real-data application of our approach to an
environmental data set where $CO_{2}$ emission is explained in terms of a
separating hyperplane defined through per-capita GDP and urban agglomeration.

##  1 Introduction

The simple linear regression model assumes a uniform linear relationship
between the covariate and the response, in the sense that the regression
parameter $\beta$ is the same over the entire covariate domain. In practice,
the situation can be more complicated: for instance, the regression parameter
may differ from sub-population to sub-population within a large (super-)
population. Some common techniques to account for such heterogeneity include
mixed linear models, introducing an interaction effect, or fitting different
models among each sub-population which corresponds to a supervised
classification setting where the true groups (sub-populations) are _a priori
known_.

A more difficult scenario arises when the sub-populations are unknown, in
which case regression and classification must happen simultaneously. Consider
the scenario where the conditional mean of $Y_{i}$ given $X_{i}$ is different
for different unknown sub-groups. A well-studied treatment of this problem –
the so-called change point problem – considers a simple thresholding model
where membership in a sub-group is determined by whether a real-valued
observable $X$ falls to the left or right of an unknown parameter $\gamma$.
More recently, there has been work for multi-dimensional covariates, namely
when the membership is determined by which side a random vector $X$ falls with
respect to an hyperplane with unknown normal vector $\theta_{0}$. A concrete
example appears in [26] who extend the linear thresholding model due to [12]
to general dimensions:

$\displaystyle Y=\mu_{1}\cdot 1_{X^{\top}\theta_{0}\geq 0}+\mu_{2}\cdot
1_{X^{\top}\theta_{0}<0}+\varepsilon\,,$ (1.1)

and studied computational algorithms and consistency of the same. This model
and others with similar structure, called _change plane models_ , are useful
in various fields of research, e.g. modeling treatment effect heterogeneity in
drug treatment ([11]), modeling sociological data on voting and employment
([11]), or cross country growth regressions in econometrics ([21]).

Other aspects of this model have also been investigated. [8] examined the
change plane model from the statistical testing point of view, with the null
hypothesis being the absence of a separating hyperplane. They proposed a test
statistic, studied its asymptotic distribution and provided sample size
recommendations for achieving target values of power. [15] extended the change
point detection problem in the multi-dimensional setup by considering the case
where $X^{\top}\theta_{0}$ forms a multiple change point data sequence. The
key difficultly with change plane type models is the inherent discontinuity in
the optimization criteria involved where the parameter of interest appears as
an argument to some indicator function, rendering the optimization extremely
hard. To alleviate this, one option is to kernel smooth the indicator
function, an approach that was adopted by Seo and Linton [21] in a version of
the change-plane problem, motivated by earlier results of Horowitz [10] that
dealt with a smoothed version of the maximum score estimator. Their model has
an additive structure of the form:

$Y_{t}=\beta^{\top}X_{t}+\delta^{\top}\tilde{X}_{t}\mathds{1}_{Q_{t}^{\top}\boldmath\psi>0}+\epsilon_{t}\,,$

where $\psi$ is the (fixed) change-plane parameter, and $t$ can be viewed as a
time index. Under a set of assumptions on the model (Assumptions 1 and 2 of
their paper), they showed asymptotic normality of their estimator of $\psi$
obtained by minimizing a smoothed least squares criterion that uses a
differentiable distribution function $\mathcal{K}$. The rate of convergence of
$\hat{\psi}$ to the truth was shown to be $\sqrt{n/\sigma_{n}}$ where
$\sigma_{n}$ was the bandwidth parameter used to smooth the least squares
function. As noted in their Remark 3, under the special case of i.i.d.
observations, their requirement that $\log n/(n\sigma_{n}^{2})\rightarrow 0$
translates to a maximal convergence rate of $n^{3/4}$ up to a logarithmic
factor. The work of [15] who considered multiple parallel change planes
(determined by a fixed dimensional normal vector) and high dimensional linear
models in the regions between consecutive hyperplanes also builds partly upon
the methods of [21] and obtains the same (almost) $n^{3/4}$ rate for the
normal vector (as can be seen by putting Condition 6 in their paper in
conjunction with the conclusion of Theorem 3).

While it is established that the condition $n\sigma_{n}^{2}\to\infty$ is
sufficient (upto a log factor) for achieving asymptotic normality of the
smoothed estimator, there is no result in the existing literature to ascertain
whether its necessity. Intuitively speaking, the necessary condition for
asymptotic normality ought to be $n\sigma_{n}\to 0$, as this will ensure a
growing number of observations in a $\sigma_{n}$ neighborhood around the true
hyperplane, allowing the central limit theorem to kick in. In this paper we
_bridge this gap_ by proving that asymptotic normality of the smoothed change
point estimator is, in fact, achievable with $n\sigma_{n}\to\infty$. This
implies that the best possible rate of convergence of the smoothed estimator
can be arbitrarily close to $n^{-1}$, the minimax optimal rate of estimation
for this problem. To demonstrate this, we focus on two change plane estimation
problems, one with a continuous and another with a binary response. The
continuous response model we analyze here is the following:

$Y_{i}=\beta_{0}^{\top}X_{i}+\delta_{0}^{\top}X_{i}\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}+{\epsilon}_{i}\,.$
(1.2)

for i.i.d. observations $\\{(X_{i},Y_{i},Q_{i}\\}_{i=1}^{n}$, where the zero-
mean transitory shocks
${\epsilon}_{i}\rotatebox[origin={c}]{90.0}{$\models$}(X_{i},Q_{i})$. Our
calculation can be easily extended to the case when the covariates on the
either side of the change hyperplane are different and
$\mathbb{E}[{\epsilon}\mid X,Q]=0$ with more tedious bookkeeping. As this
generalization adds little of interest, conceptually, to our proof, we posit
the simpler model for ease of understanding. As the parameter $\psi_{0}$ is
only identifiable upto its norm, we assume that the first co-ordinate is $1$
(along the lines of [21]) which removes one degree of freedom and makes the
parameter identifiable.

To illustrate that a similar phenomenon transpires with binary response, we
also study a canonical version of such a model which can be briefly described
as follows: The covariate $Q\sim P$ where $P$ is distribution on
$\mathbb{R}^{d}$ and the conditional distribution of $Y$ given $Q$ is modeled
as follows:

$P(Y=1|Q)=\alpha_{0}\mathds{1}(Q^{\top}\psi_{0}\leq
0)+\beta_{0}\mathds{1}(Q^{\top}\psi_{0}>0)$ (1.3)

for some parameters $\alpha_{0},\beta_{0}\in(0,1)$ and
$\psi_{0}\in\mathbb{R}^{d}$ (with first co-ordinate being one for
identifiability issue as for the continuous response model), the latter being
of primary interest for estimation. This model is identifiable up to a
permutation of $(\alpha_{0},\beta_{0})$, so we further assume
$\alpha_{0}<\beta_{0}$. For both models, we show that
$\sqrt{n/\sigma_{n}}(\hat{\psi}-\psi_{0})$ converges to zero-mean normal
distribution as long as $n\sigma_{n}\to\infty$ but the calculations for the
binary model are completely relegated to Appendix C.

Organization of the paper: The rest of the paper is organized as follows: In
Section 2 we present the methodology, the statement of the asymptotic
distributions and a sketch of the proof for the continuous response model
(1.2). In Section 3 we briefly describe the binary response model (1.3) and
related assumptions, whilst the details can be found in the supplementary
document. In Section 5 we present some simulation results, both for the binary
and the continuous response models to study the effect of the bandwidth on the
quality of the normal approximation in finite samples. In Section 6, we
present a real data analysis where we analyze the effect of income and
urbanization on the $CO_{2}$ emission in different countries.

Notations:  Before delving into the technical details, we first setup some
notations here. We assume from now on, $X\in{\mathbb{R}}^{p}$ and
$Q\in{\mathbb{R}}^{d}$. For any vector $v$ we define by $\tilde{v}$ as the
vector with all the co-ordinates expect the first one. We denote by $K$ the
kernel function used to smooth the indicator function. For any matrix $A$, we
denote by $\|A\|_{2}$ (or $\|A\|_{F}$) as its Frobenious norm and $\|A\|_{op}$
as its operator norm. For any vector, $\|\cdot\|_{2}$ denotes its $\ell_{2}$
norm.

##  2 Methodology and Theory for Continuous Response Model

In this section we present our analysis for the continuous response model.
Without smoothing, the original estimating equation is:

$f_{\beta,\delta,\psi}(Y,X,Q)=\left(Y-X^{\top}\beta-X^{\top}\delta\mathds{1}_{Q^{\top}\psi>0}\right)^{2}$

and we estimate the parameters as:

$\displaystyle\left(\hat{\beta}^{LS},\hat{\delta}^{LS},\hat{\psi}^{LS}\right)$
$\displaystyle={\arg\min}_{(\beta,\delta,\psi)\in\Theta}\mathbb{P}_{n}f_{\beta,\delta,\psi}$
$\displaystyle:={\arg\min}_{(\beta,\delta,\psi)\in\Theta}\mathbb{M}_{n}(\beta,\delta,\psi)\,.$
(2.1)

where $\mathbb{P}_{n}$ is empirical measure based on i.i.d. observations
$\\{(X_{i},Y_{i},Q_{i})\\}_{i=1}^{n}$ and $\Theta$ is the parameter space.
Henceforth, we assume $\Theta$ is a compact subset of dimension
${\mathbb{R}}^{2p+d}$. We also define $\theta=(\beta,\delta,\psi)$, i.e. all
the parameters together as a vector and by $\theta_{0}$ is used to denote the
true parameter vector $(\beta_{0},\delta_{0},\psi_{0})$. Some modification of
equation (2) leads to the following:

$\displaystyle(\hat{\beta}^{LS},\hat{\delta}^{LS},\hat{\psi}^{LS})$
$\displaystyle={\arg\min}_{\beta,\delta,\psi}\sum_{i=1}^{n}\left(Y_{i}-X_{i}^{\top}\beta-
X_{i}^{\top}\delta\mathds{1}_{Q_{i}^{\top}\psi>0}\right)^{2}$
$\displaystyle={\arg\min}_{\beta,\delta,\psi}\sum_{i=1}^{n}\left[\left(Y_{i}-X_{i}^{\top}\beta\right)^{2}\mathds{1}_{Q_{i}^{\top}\psi_{0}\leq
0}\right.$ $\displaystyle\hskip
140.00021pt\left.+\left(Y_{i}-X_{i}^{\top}\beta-
X_{i}^{\top}\delta\right)^{2}\mathds{1}_{Q_{i}^{\top}\psi>0}\right]$
$\displaystyle={\arg\min}_{\beta,\delta,\psi}\sum_{i=1}^{n}\left[\left(Y_{i}-X_{i}^{\top}\beta\right)^{2}+\left\\{\left(Y_{i}-X_{i}^{\top}\beta-
X_{i}^{\top}\delta\right)^{2}\right.\right.$ $\displaystyle\hskip
170.00026pt\left.\left.-\left(Y_{i}-X_{i}^{\top}\beta\right)^{2}\right\\}\mathds{1}_{Q_{i}^{\top}\psi>0}\right]$

Typical empirical process calculations yield under mild conditions:

$\|\hat{\beta}^{LS}-\beta_{0}\|^{2}+\|\hat{\delta}^{LS}-\delta_{0}\|^{2}+\|\hat{\psi}^{LS}-\psi_{0}\|_{2}=O_{p}(n^{-1})$

but inference is difficult as the limit distribution is unknown, and in any
case, would be a highly non-standard distribution. Recall that even in the
one-dimensional change point model with fixed jump size, the least squares
change point estimator converges at rate $n$ to the truth with a non-standard
limit distribution, namely a minimizer of a two-sided compound Poisson process
(see [13] for more details). To obtain a computable estimator with tractable
limiting distribution, we resort to a smooth approximation of the indicator
function in (2) using a distribution kernel with suitable bandwidth, i.e we
replace $\mathds{1}_{Q_{i}^{\top}\psi>0}$ by $K(Q_{i}^{\top}\psi/\sigma_{n})$
for some appropriate distribution function $K$ and bandwidth $\sigma_{n}$,
i.e.

$\displaystyle(\hat{\beta}^{S},\hat{\delta}^{S},\hat{\psi}^{S})$
$\displaystyle={\arg\min}_{\beta,\delta,\psi}\left\\{\frac{1}{n}\sum_{i=1}^{n}\left[\left(Y_{i}-X_{i}^{\top}\beta\right)^{2}+\left\\{\left(Y_{i}-X_{i}^{\top}\beta-
X_{i}^{\top}\delta\right)^{2}\right.\right.\right.$ $\displaystyle\hskip
150.00023pt\left.\left.\left.-\left(Y_{i}-X_{i}^{\top}\beta\right)^{2}\right\\}K\left(\frac{Q_{i}^{\top}\psi}{\sigma_{n}}\right)\right]\right\\}$
$\displaystyle={\arg\min}_{(\beta,\delta,\psi)\in\Theta}\mathbb{P}_{n}f^{s}_{(\beta,\delta,\psi)}(X,Y,Q)$
$\displaystyle:={\arg\min}_{\theta\in\Theta}\mathbb{M}^{s}_{n}(\theta)\,.$

Define $\mathbb{M}$ (resp. $\mathbb{M}^{s}$) to be the population counterpart
of $\mathbb{M}_{n}$ and $\mathbb{M}_{n}^{s}$ respectively which are defined
as:

$\displaystyle\mathbb{M}(\theta)$
$\displaystyle=\mathbb{E}\left(Y-X^{\top}\beta\right)^{2}+\mathbb{E}\left(\left[-2\left(Y_{i}-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right]\mathds{1}_{Q^{\top}\psi>0}\right)\,,$
$\displaystyle\mathbb{M}^{s}(\theta)$
$\displaystyle=\mathbb{E}\left[(Y-X^{\top}\beta)^{2}+\left\\{-2(Y-X^{\top}\beta)(X^{\top}\delta)+(X^{\top}\delta)^{2}\right\\}K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right]\,.$

As noted in the proof of Seo and Linton, the assumption
$\log{n}/n\sigma_{n}^{2}\to 0$ was only used to show:

$\frac{\left\|\hat{\psi}^{s}-\psi_{0}\right\|}{\sigma_{n}}=o_{p}(1)\,.$

In this paper, we show that one can achieve the same conclusion as long as
$n\sigma_{n}\to\infty$. The rest of the proof for the normality is similar to
that of [21], we will present it briefly for the ease the readers. The proof
is quite long and technical, therefore we break the proof into several lemmas.
We, first, list our assumptions:

######  Assumption 2.1.

  1. 1.

Define $f_{\psi}(\cdot\mid\tilde{Q})$ to be the conditional distribution of
$Q^{\top}\psi$ given $\tilde{Q}$. (In particular we will denote by
$f_{0}(\cdot\mid\tilde{q})$ to be conditional distribution of
$Q^{\top}\psi_{0}$ given $\tilde{Q}$ and $f_{s}(\cdot\mid\tilde{q})$ to be the
conditional distribution of $Q^{\top}\psi_{0}^{s}$ given $\tilde{Q}$. Assume
that there exists $F_{+}$ such that $\sup_{t}f_{0}(t|\tilde{Q})\leq F_{+}$
almost surely on $\tilde{Q}$ and for all $\psi$ in a neighborhood of
$\psi_{0}$ (in particular for $\psi_{0}^{s}$). Further assume that $f_{\psi}$
is differentiable and the derivative is bounded by $F_{+}$ for all $\psi$ in a
neighborhood of $\psi_{0}$ (again in particular for $\psi_{0}^{s}$).

  2. 2.

Define $g(Q)={\sf var}(X\mid Q)$. There exists $c_{-}$ and $c_{+}$ such that
$c_{-}\leq\lambda_{\min}(g(Q))\leq\lambda_{\max}(g(Q))\leq c_{+}$ almost
surely. Also assume that $g$ is a Lipschitz with constant $G_{+}$ with respect
to $Q$.

  3. 3.

There exists $p_{+}<\infty$ and $p_{-}>0,r>0$ such that:

$p_{-}\|\psi-\psi_{0}\|\leq\mathbb{P}\left(\text{sign}\left(Q^{\top}\psi\right)\neq\text{sign}\left(Q^{\top}\psi_{0}\right)\right)\leq
p_{+}\|\psi-\psi_{0}\|\,,$

for all $\psi$ such that $\|\psi-\psi_{0}\|\leq r$.

  4. 4.

For all $\psi$ in the parameter space
$0<\mathbb{P}\left(Q^{\top}\psi>0\right)<1$.

  5. 5.

Define $m_{2}(Q)=\mathbb{E}\left[\|X\|^{2}\mid Q\right]$ and
$m_{4}(Q)=\mathbb{E}\left[\|X\|^{4}\mid Q\right]$. Assume $m_{2},m_{4}$ are
bounded Lipschitz function of $Q$.

###  2.1 Sufficient conditions for above assumptions

We now demonstrate some sufficient conditions for the above assumptions to
hold. The first condition is essentially a condition on the conditional
density of the first co-ordinate of $Q$ given all other co-ordinates. If this
conditional density is bounded and has bounded derivative, then first
assumption is satisfied. This condition is satisfied in fair generality. The
second assumption implies that the conditional distribution of X given Q has
variance in all the direction over all $Q$. This is also very weak condition,
as is satisfied for example if X and Q and independent (with $X$ has non-
degenerate covariance matrix) or $(X,Q)$ are jointly normally distributed to
name a few. This condition can further be weaken by assuming that the maximum
and minimum eigenvalues of $\mathbb{E}[g(Q)]$ are bounded away from $\infty$
and $0$ respectively but it requires more tedious book-keeping. The third
assumption is satisfied as long as as $Q^{\top}\psi$ has non-zero density near
origin, while the fourth assumption merely states that the support of $Q$ is
not confined to one side of the hyperplane for any hyperplane and a simple
sufficient condition for this is $Q$ has continuous density with non-zero
value at the origin. The last assumption is analogous to the second assumption
for the conditional fourth moment which is also satisfied in fair generality.

Kernel function and bandwidth:  We take $K(x)=\Phi(x)$ (distribution of
standard normal random variable) for our analysis. For the bandwidth we assume
$n\sigma_{n}^{2}\to 0$ and $n\sigma_{n}\to\infty$ as the other case, (i.e.
$n\sigma_{n}^{2}\to\infty$) is already established in [21].

Based on Assumption 2.1 and our choice of kernel and bandwidth we establish
the following theorem:

######  Theorem 2.2.

Under Assumption 2.1 and the above choice of kernel and bandwidth we have:

$\sqrt{n}\begin{pmatrix}\begin{pmatrix}\hat{\beta}^{s}\\\
\hat{\delta}^{s}\end{pmatrix}-\begin{pmatrix}\beta_{0}\\\
\delta_{0}\end{pmatrix}\end{pmatrix}\overset{\mathscr{L}}{\implies}\mathcal{N}(0,\Sigma_{\beta,\delta})$

and

$\sqrt{n/\sigma_{n}}\left(\hat{\psi}^{s}-\psi_{0}\right)\overset{\mathscr{L}}{\implies}\mathcal{N}(0,\Sigma_{\psi})\,,$

for matrices $\Sigma_{\beta,\delta}$ and $\Sigma_{\psi}$ mentioned explicitly
in the proof. Moreover they are asymptotically independent.

The proof of the theorem is relatively long, so we break it into several
lemmas. We provide a roadmap of the proof in this section while the elaborate
technical derivations of the supporting lemmas can be found in Appendix. Let
$\nabla\mathbb{M}_{n}^{s}(\theta)$ and $\nabla^{2}\mathbb{M}_{n}^{s}(\theta)$
be the gradient and Hessian of $\mathbb{M}_{n}^{s}(\theta)$ with respect to
$\theta$. As $\hat{\theta}^{s}$ minimizes $\mathbb{M}_{n}^{s}(\theta)$, we
have from the first order condition,
$\nabla\mathbb{M}_{n}^{s}(\hat{\theta}^{s})=0$. Using one step Taylor
expansion we have:

$\displaystyle\nabla\mathbb{M}_{n}^{s}(\hat{\theta}^{s})=\nabla\mathbb{M}_{n}^{s}(\theta_{0})+\nabla^{2}\mathbb{M}_{n}^{s}(\theta^{*})\left(\hat{\theta}^{s}-\theta_{0}\right)=0$

i.e.

$\left(\hat{\theta}^{s}-\theta_{0}\right)=-\left(\nabla^{2}\mathbb{M}_{n}^{s}(\theta^{*})\right)^{-1}\nabla\mathbb{M}_{n}^{s}(\theta_{0})$
(2.2)

for some intermediate point $\theta^{*}$ between $\hat{\theta}^{s}$ and
$\theta_{0}$. Following the notation of [21], define a diagonal matrix $D_{n}$
of dimension $2p+d$ with first $2p$ elements being 1 and the last $d$ elements
being $\sqrt{\sigma_{n}}$. we can write:

$\displaystyle\sqrt{n}D_{n}^{-1}(\hat{\theta}^{s}-\theta_{0})$
$\displaystyle=-\sqrt{n}D_{n}^{-1}\nabla^{2}\mathbb{M}_{n}^{s}(\theta^{*})^{-1}\nabla\mathbb{M}_{n}^{s}(\theta_{0})$
$\displaystyle=\begin{pmatrix}\nabla^{2}\mathbb{M}_{n}^{s,\gamma}(\theta^{*})&\sqrt{\sigma_{n}}\nabla^{2}\mathbb{M}_{n}^{s,\gamma\psi}(\theta^{*})\\\
\sqrt{\sigma_{n}}\nabla^{2}\mathbb{M}_{n}^{s,\gamma\psi}(\theta^{*})&\sigma_{n}\nabla^{2}\mathbb{M}_{n}^{s,\psi}(\theta^{*})\end{pmatrix}^{-1}\begin{pmatrix}\sqrt{n}\nabla\mathbb{M}_{n}^{s,\gamma}(\theta_{0})\\\
\sqrt{n\sigma_{n}}\nabla\mathbb{M}_{n}^{s,\psi}(\theta_{0})\end{pmatrix}$
(2.3)

where $\gamma=(\beta,\delta)\in{\mathbb{R}}^{2p}$. The following lemma
establishes the asymptotic properties of
$\nabla\mathbb{M}_{n}^{s}(\theta_{0})$:

######  Lemma 2.3 (Asymptotic Normality of
$\nabla\mathbb{M}_{n}^{s}(\theta_{0})$).

Under assumption 2.1 we have:

$\displaystyle\sqrt{n}\nabla\mathbb{M}_{n}^{s,\gamma}(\theta_{0})\implies\mathcal{N}\left(0,4V^{\gamma}\right)\,,$
$\displaystyle\sqrt{n\sigma_{n}}\nabla\mathbb{M}_{n}^{s,\psi}(\theta_{0})\implies\mathcal{N}\left(0,V^{\psi}\right)\,.$

for some n.n.d. matrices $V^{\gamma}$ and $V^{\psi}$ which is mentioned
explicitly in the proof. Further more
$\sqrt{n}\nabla\mathbb{M}_{n}^{s,\gamma}(\theta_{0})$ and
$\sqrt{n\sigma_{n}}\nabla\mathbb{M}_{n}^{s,\psi}(\theta_{0})$ are
asymptotically independent.

Next, we analyze the convergence of $\nabla^{2}\mathbb{M}_{n}^{s}(\theta^{*})$
which is stated in the following lemma:

######  Lemma 2.4 (Convergence in Probability of
$\nabla^{s}\mathbb{M}_{n}^{s}(\theta^{*})$).

Under Assumption (2.1), for any random sequence
$\breve{\theta}=\left(\breve{\beta},\breve{\delta},\breve{\psi}\right)$ such
that
$\breve{\beta}\overset{p}{\to}\beta_{0},\breve{\delta}\overset{p}{\to}\delta_{0},\|\breve{\psi}-\psi_{0}\|/\sigma_{n}\overset{P}{\rightarrow}0$,
we have:

$\displaystyle\nabla^{2}_{\gamma}\mathbb{M}_{n}^{s}(\breve{\theta})$
$\displaystyle\overset{p}{\longrightarrow}2Q^{\gamma}\,,$
$\displaystyle\sqrt{\sigma_{n}}\nabla^{2}_{\psi\gamma}\mathbb{M}_{n}^{s}(\breve{\theta})$
$\displaystyle\overset{p}{\longrightarrow}0\,,$
$\displaystyle\sigma_{n}\nabla^{2}_{\psi}\mathbb{M}_{n}^{s}(\breve{\theta})$
$\displaystyle\overset{p}{\longrightarrow}Q^{\psi}\,.$

for some matrices $Q^{\gamma},Q^{\psi}$ mentioned explicitly in the proof.
This, along with equation (2.3), establishes:

$\displaystyle\sqrt{n}\left(\hat{\gamma}^{s}-\gamma_{0}\right)$
$\displaystyle\overset{\mathscr{L}}{\implies}\mathcal{N}\left(0,Q^{\gamma^{-1}}V^{\gamma}Q^{\gamma^{-1}}\right)\,,$
$\displaystyle\sqrt{n/\sigma_{n}}\left(\hat{\psi}^{s}-\psi_{0}\right)$
$\displaystyle\overset{\mathscr{L}}{\implies}\mathcal{N}\left(0,Q^{\psi^{-1}}V^{\psi}Q^{\psi^{-1}}\right)\,.$

where as before $\hat{\gamma}^{s}=(\hat{\beta}^{s},\hat{\delta}^{s})$.

It will be shown later that the condition
$\|\breve{\psi}_{n}-\psi_{0}\|/\sigma_{n}\overset{P}{\rightarrow}0$ needed in
Lemma C.9 holds for the (random) sequence $\psi^{*}$, the intermediate point
in the Taylor expansion. Then, combining Lemma C.8 and Lemma C.9 we conclude
the proof of Theorem 2.2. Observe that, to show
$\left\|\psi^{*}-\psi_{0}\right\|=o_{P}(\sigma_{n})$, it suffices to to prove
that $\left\|\hat{\psi}^{s}-\psi_{0}\right\|=o_{P}(\sigma_{n})$. Towards that
direction, we have following lemma:

######  Lemma 2.5 (Rate of convergence).

Under Assumption 2.1 and our choice of kernel and bandwidth,

$n^{2/3}\sigma_{n}^{-1/3}d^{2}_{*}\left(\hat{\theta}^{s},\theta_{0}^{s}\right)=O_{P}(1)\,,$

where

$\displaystyle d_{*}^{2}(\theta,\theta_{0}^{s})$
$\displaystyle=\|\beta-\beta_{0}^{s}\|^{2}+\|\delta-\delta_{0}^{s}\|^{2}$
$\displaystyle\qquad\qquad+\frac{\|\psi-\psi_{0}^{s}\|^{2}}{\sigma_{n}}\mathds{1}_{\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}}+\|\psi-\psi_{0}^{s}\|\mathds{1}_{\|\psi-\psi_{0}^{s}\|>\mathcal{K}\sigma_{n}}\,.$

for some specific constant $\mathcal{K}$. (This constant will be mentioned
precisely in the proof). Hence as $n\sigma_{n}\to\infty$, we have
$n^{2/3}\sigma_{n}^{-1/3}\gg\sigma_{n}^{-1}$ which implies
$\|\hat{\psi}^{s}-\psi_{0}^{s}\|/\sigma_{n}\overset{P}{\longrightarrow}0\,.$

The above lemma establishes
$\|\hat{\psi}^{s}-\psi_{0}^{s}\|/\sigma_{n}=o_{p}(1)$ but our goal is to show
that $\|\hat{\psi}^{s}-\psi_{0}\|/\sigma_{n}=o_{p}(1)$. Therefore, we further
need $\|\psi^{s}_{0}-\psi_{0}\|/\sigma_{n}\rightarrow 0$ which is demonstrated
in the following lemma:

######  Lemma 2.6 (Convergence of population minimizer).

Under Assumption 2.1 and our choice of kernel and bandwidth, we have:
$\|\psi^{s}_{0}-\psi_{0}\|/\sigma_{n}\rightarrow 0$.

Hence the final roadmap is the following: Using Lemma C.12 and Lemma 2.5 we
establish that $\|\hat{\psi}^{s}-\psi_{0}\|/\sigma_{n}=o_{p}(1)$ if
$n\sigma_{n}\to 0$. This, in turn, enables us to prove Lemma C.9, i.e.
$\sigma_{n}\nabla^{2}\mathbb{M}_{n}^{s}(\theta^{*})\overset{P}{\rightarrow}Q$,which,
along with Lemma C.8, establishes the main theorem.

##  3 Binary response model

Recall our binary response model in equation (1.3). To estimate $\psi_{0}$, we
resort to the following loss (without smoothing):

$\mathbb{M}(\psi)=\mathbb{E}\left((Y-\gamma)\mathds{1}(Q^{\top}\psi\leq
0)\right)$ (3.1)

with $\gamma\in(\alpha_{0},\beta_{0})$, which can be viewed as a variant of
the square error loss function:

$\mathbb{M}(\alpha,\beta,\psi)=\mathbb{E}\left(\left(Y-\alpha\mathds{1}(Q^{\top}\psi<0)-\beta\mathds{1}(Q^{\top}\psi>0)\right)^{2}\right)\,.$

We establish the connection between these losses in sub-section C.2. It is
easy to prove that under fairly mild conditions (discussed later)
$\psi_{0}={\arg\min}_{\psi\in\Theta}\mathbb{M}(\psi)$, uniquely. Under the
standard classification paradigm, when we know a priori that
$\alpha_{0}<1/2<\beta_{0}$, we can take $\gamma=1/2$, and in the absence of
this constraint, $\bar{Y}$, which converges to some $\gamma$ between
$\alpha_{0}$ and $\beta_{0}$, may be substituted in the loss function. In the
rest of the paper, we confine ourselves to a known $\gamma$, and for technical
simplicity, we take $\gamma=\frac{(\beta_{0}+\alpha_{0})}{2}$, but this
assumption can be removed with more mathematical book-keeping. Thus,
$\psi_{0}$ is estimated by:

$\hat{\psi}={\arg\min}_{\psi\in\Theta}\mathbb{M}_{n}(\psi)={\arg\min}_{\psi\in\Theta}\frac{1}{n}\sum_{\i=1}^{n}(Y_{i}-\gamma)\mathds{1}(Q_{i}^{\top}\psi\leq
0)\,.$ (3.2)

We resort to a smooth approximation of the indicator function in (3.2) using a
distribution kernel with suitable bandwidth. The smoothed version of the
population score function then becomes:

$\mathbb{M}^{s}(\psi)=\mathbb{E}\left((Y-\gamma)\left(1-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right)\right)$
(3.3)

where as in the continuous response model, we use $K(x)=\Phi(x)$, and the
corresponding empirical version is:

$\mathbb{M}^{s}_{n}(\psi)=\frac{1}{n}\sum_{i=1}^{n}\left((Y_{i}-\gamma)\left(1-K\left(\frac{Q_{i}^{\top}\psi}{\sigma_{n}}\right)\right)\right)$
(3.4)

Define $\hat{\psi}^{s}$ and $\psi_{0}^{s}$ to be the minimizer of the smoothed
version of the empirical (equation (3.4)) and population score (equation
(3.3)) function respectively. Here we only consider the choice of bandwidth
$n\sigma_{n}\to\infty$ and $n\sigma_{n}^{2}\to 0$. Analogous to Theorem 2.2 we
prove the following result for binary response model:

######  Theorem 3.1.

Under Assumptions (C.1 \- C.5):

$\sqrt{\frac{n}{\sigma_{n}}}\left(\hat{\psi}_{n}-\psi_{0}\right)\Rightarrow
N(0,\Gamma)\,,$

for some non-stochastic matrix $\Gamma$, which will be defined explicitly in
the proof.

We have therefore established that in the regime $n\sigma_{n}\to\infty$ and
$n\sigma_{n}^{2}\to 0$, it is possible to attain asymptotic normality using a
smoothed estimator for binary response model.

##  4 Inferential methods

We draw inferences on $(\beta_{0},\delta_{0},\psi_{0})$ by resorting to
similar techniques as in [21]. For the continuous response model, we need
consistent estimators of $V^{\gamma},Q^{\gamma},V^{\psi},Q^{\psi}$ (see Lemma
C.9 for the definitions) for hypothesis testing. By virtue of the
aforementioned Lemma, we can estimate $Q^{\gamma}$ and $Q^{\psi}$ as follows:

$\displaystyle\hat{Q}^{\gamma}$
$\displaystyle=\nabla^{2}_{\gamma}\mathbb{M}_{n}^{s}(\hat{\theta})\,,$
$\displaystyle\hat{Q}^{\psi}$
$\displaystyle=\sigma_{n}\nabla^{2}_{\psi}\mathbb{M}_{n}^{s}(\hat{\theta})\,.$

The consistency of the above estimators is established in the proof of Lemma
C.9. For the other two parameters $V^{\gamma},V^{\psi}$ we use the following
estimators:

$\displaystyle\hat{V}^{\psi}$
$\displaystyle=\frac{1}{n\sigma_{n}^{2}}\sum_{i=1}^{n}\left(\left(Y_{i}-X_{i}^{\top}(\hat{\beta}+\hat{\delta})\right)^{2}-\left(Y_{i}-X_{i}^{\top}\hat{\beta}\right)^{2}\right)^{2}\tilde{Q}_{i}\tilde{Q}_{i}^{\top}\left(K^{\prime}\left(\frac{Q_{i}^{\top}\hat{\psi}}{\sigma_{n}}\right)\right)^{2}$
$\displaystyle\hat{V}^{\gamma}$
$\displaystyle=\hat{\sigma}^{2}_{\epsilon}\begin{pmatrix}\frac{1}{n}X_{i}X_{i}^{\top}&\frac{1}{n}X_{i}X_{i}^{\top}\mathds{1}_{Q_{i}^{\top}\hat{\psi}>0}\\\
\frac{1}{n}X_{i}X_{i}^{\top}\mathds{1}_{Q_{i}^{\top}\hat{\psi}>0}&\frac{1}{n}X_{i}X_{i}^{\top}\mathds{1}_{Q_{i}^{\top}\hat{\psi}>0}\end{pmatrix}$

where $\hat{\sigma}^{2}_{\epsilon}$ can be obtained as
$(1/n)(Y_{i}-X_{i}^{\top}\hat{\beta}-X_{i}^{\top}\hat{\delta}\mathds{1}(Q_{i}^{\top}\hat{\psi}>0))^{2}$,
i.e. the residual sum of squares. The explicit value of $V_{\gamma}$ (as
derived in equation (B.23) in the proof Lemma C.8) is:

$V^{\gamma}=\sigma_{\epsilon}^{2}\begin{pmatrix}\mathbb{E}\left[XX^{\top}\right]&\mathbb{E}\left[XX^{\top}\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\\\
\mathbb{E}\left[XX^{\top}\mathds{1}_{Q^{\top}\psi_{0}>0}\right]&\mathbb{E}\left[XX^{\top}\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\end{pmatrix}$

Therefore, the consistency of $\hat{V}_{\gamma}$ is immediate from the law of
large numbers. The consistency of $\hat{V}^{\psi}$ follows via arguments
similar to those employed in proving Lemma C.9 but under somewhat more
stringent moment conditions: in particular, we need
$\mathbb{E}[\|X\|^{8}]<\infty$ and $\mathbb{E}[(X^{\top}\delta_{0})^{k}\mid
Q]$ to be Lipschitz functions over $Q$ for $1\leq k\leq 8$. The inferential
techniques for the classification model are similar and hence skipped, to
avoid repetition.

##  5 Simulation studies

In this section, we present some simulation results to analyse the effect of
the choice of $\sigma_{n}$ on the finite sample approximation of asymptotic
normality, i.e. Berry-Essen type bounds. If we choose a smaller sigma, the
rate of convergence is accelerated but the normal approximation error at
smaller sample sizes will be higher, as we don’t have enough observations in
the vicinity of the change hyperplane for the CLT to kick in. This problem is
alleviated by choosing $\sigma_{n}$ larger, but this, on the other hand,
compromises the convergence rate. Ideally, a Berry-Essen type of bound will
quantify this, but this will require a different set of techniques and is left
as an open problem. In our simulations, we generate data from following setup:

  1. 1.

Set $N=50000,p=3,\alpha_{0}=0.25,\beta=0.75$ and some
$\theta_{0}\in\mathbb{R}^{p}$ with first co-ordinate $=1$.

  2. 2.

Generate $X_{1},\dots,X_{n}\sim\mathcal{N}(0,I_{p})$.

  3. 3.

Generate
$Y_{i}\sim\textbf{Bernoulli}\left(\alpha_{0}\mathds{1}_{X_{i}^{\top}\theta_{0}\leq
0}+\beta_{0}\mathds{1}_{X_{i}^{\top}\theta_{0}>0}\right)$.

  4. 4.

Estimate $\hat{\theta}$ by minimizing $\mathbb{M}_{n}(\theta)$ (replacing
$\gamma$ by $\bar{Y}$) based on $\\{(X_{i},Y_{i})\\}_{i=1}^{n}$ for different
choices of $\sigma_{n}$.

We repeat Step 2 - Step 4 a hundred times to obtain
$\hat{\theta}_{1},\dots,\hat{\theta}_{100}$. Define $s_{n}$ to be the standard
deviation of $\\{\hat{\theta}_{i}\\}_{i=1}^{100}$. Figures reffig:co2 and 2
show the qqplots of $\tilde{\theta}_{i}=(\hat{\theta}_{i}-\theta_{0})/s_{n}$
against the standard normal for four different choices of
$\sigma_{n}=n^{-0.6},n^{-0.7},n^{-0.8},n^{-0.9}$.

Figure 1: In this figure, we present qqplot for estimating second co-ordinate
of $\theta_{0}$ with different choices of $\sigma_{n}$ mentioned at the top of
each plots. Figure 2: In this figure, we present qqplot for estimating third
co-ordinate of $\theta_{0}$ with different choices of $\sigma_{n}$ mentioned
at the top of each plots.

It is evident that smaller value of $\sigma_{n}$ yield a poor normal
approximation. Although our theory shows that asymptotic normality holds as
long as $n\sigma_{n}\to\infty$, in practice we recommend choosing $\sigma_{n}$
such that $n\sigma_{n}\geq 30$ for the central limit of theorem to take
effect.

##  6 Real data analysis

We illustrate our method using cross-country data on pollution (carbon-
dioxide), income and urbanization obtained from the World Development
Indicators (WDI), World Bank. The Environmental Kuznets Curve hypothesis (EKC
henceforth), a popular and ongoing area of research in environmental
economics, posits that at an initial stage of economic development pollution
increases with economic growth, and then diminishes when society’s priorities
change, leading to an inverted U-shaped relation between income (measured via
real GDP per capita) and pollution. The hypothesis has led to numerous
empirical papers (i) testing the hypothesis (whether the relation is inverted
U-shaped for countries/regions of interest in the sample), (ii) exploring the
threshold level of income at which pollution starts falling, as well as (iii)
examining the countries/regions which belong to the upward rising part versus
the downward sloping part of the inverted U-shape, if at all. The studies have
been performed using US state level data or cross-country data (e.g. [22],
[19], [1], [14],[4], [18], [9], [3], [2], [24] to name a few). While some of
these papers have found evidence in favor of the EKC hypothesis (inverted
U-shaped income-pollution relation), others have found evidence against it
(monotonically increasing or other shapes for the relation). The results often
depend on countries/regions in the sample, period of analysis, as well as the
pollutant studied.

While income-pollution remains the focal point of most EKC studies, several of
them have also included urban agglomeration (UA) or some other measures of
urbanization as an important control variable especially while investigating
carbon emissions.111Although income growth is connected to urbanization,
countries are heterogenous and follow different growth paths due to their
varying geographical structures, population densities, infrastructures,
ownerships of resources making a case for using urbanization as another
control covariate in the income-pollution study. The income growth paths of
oil rich UAE, manufacturing based China, serviced based Singapore, low
population density Canada (with vast land) are all different. (see for
example, [22], [4]and [16]). The theory of ecological economics posits
potentially varying effects of increased urbanization on pollution– (i)
urbanization leading to more pollution (due to its close links with
sanitations, dense transportations, and proximities to polluting manufacturing
industries), (ii) urbanization potentially leading to less pollution based on
‘compact city theory’ (see [5], [6], [20]) that explains the potential
benefits of increased urbanization in terms of economies of scale (for
example, replacing dependence on automobiles with large scale subway systems,
using multi-storied buildings instead of single unit houses, keeping more open
green space). [17], using 17 developed countries, find a positive and
significant effect of urbanization on pollution. On the contrary, using a set
of 69 countries [23] find a negative and significant effect of urbanization on
pollution while [7] find an insignificant effect of urbanization on carbon
emission. Using various empirical strategies [20] conclude that the positive
and negative effects of urbanization on carbon pollution may cancel out
depending on the countries involved often leaving insignificant effects on
pollution. They also note that many countries are yet to achieve a sizeable
level of urbanization which presumably explains why many empirical works using
less developed countries find insignificant effect of urbanization. In
summary, based on the existing literature, both the relationship between
urbanization and pollution as well as the relationship between income and
pollution appear to depend largely on the set of countries considered in the
sample. This motivates us to use UA along with income in our change plane
model for analyzing carbon-dioxide emission to plausibly separate the
countries into two regimes.

Following the broad literature we use pollution emission per capita (carbon-
dioxide measured in metric tons per capita) as the dependent variable and real
GDP per capita (measured in 2010 US dollars), its square (as is done commonly
in the EKC literature) and a popular measure of urbanization, namely urban
agglomeration (UA)222The exact definition can be found in the World
Development Indicators database from the World Bank website. as covariates (in
our notation $X$) in our regression. In light of the preceding discussions we
fit a change plane model comprising real GDP per capita and UA (in our
notation $Q$). To summarize the setup, we use the continuous response model as
described in equation (1.2), i.e

$\displaystyle Y_{i}$
$\displaystyle=X_{i}^{\top}\beta_{0}+X_{i}^{\top}\delta_{0}\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}+{\epsilon}_{i}$
$\displaystyle=X_{i}^{\top}\beta_{0}\mathds{1}_{Q_{i}^{\top}\psi_{0}\leq
0}+X_{i}^{\top}(\beta_{0}+\delta_{0})\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}+{\epsilon}_{i}$

with the per capita $CO_{2}$ emission in metric ton as $Y$, per capita GDP,
square of per capita GDP and UA as $X$ (hence $X\in\mathbb{R}^{3}$) and
finally, per capita GDP and UA as $Q$ (hence $Q\in\mathbb{R}^{2}$). Observe
that $\beta_{0}$ represents the regression coefficients corresponding to the
countries with $Q_{i}^{\top}\psi_{0}\leq 0$ (henceforth denoted by Group 1)
and $(\beta_{0}+\delta_{0})$ represents the regression coefficients
corresponding to the countries with $Q_{i}^{\top}\psi_{0}\geq 0$ (henceforth
denoted by Group 2). As per our convention, in the interests of
identifiability we assume $\psi_{0,1}=1$, where $\psi_{0,1}$ is the change
plane parameter corresponding to per capita GDP. Therefore the only change
plane coefficient to be estimated is $\psi_{0,2}$, the change plane
coefficient for UA. For numerical stability, we divide per capita GDP by
$10^{-4}$ (consequently square of per capital GDP is scaled by
$10^{-8}$)333This scaling helps in the numerical stability of the gradient
descent algorithm used to optimize the least squares criterion.. After some
pre-processing (i.e. removing rows consisting of NA and countries with $100\%$
UA) we estimate the coefficients $(\beta_{0},\delta_{0},\psi_{0})$ of our
model based on data from 115 countries with $\sigma_{n}=0.05$ and test the
significance of the various coefficients using the methodologies described in
Section 4. We present our findings in Table 1.

Coefficients | Estimated values | p-values
---|---|---
$\beta_{0,1}$ (RGDPPC for Group 1) | 6.98555060 | 4.961452e-10
$\beta_{0,2}$ (squared RGDPPC for Group 1) | -0.43425991 | 7.136484e-02
$\beta_{0,3}$ (UA for Group 1) | -0.02613813 | 1.066065e-01
$\beta_{0,1}+\delta_{0,1}$ (RGDPPC for Group 2) | 2.0563337 | 0.000000e+00
$\beta_{0,2}+\delta_{0,2}$ (squared RGDPPC for Group 2) | -0.1866490 | 4.912843e-04
$\beta_{0,3}+\delta_{0,3}$ (UA for Group 2) | 0.1403171 | 1.329788e-05
$\psi_{0,2}$ (Change plane coeff for UA) | -0.07061785 | 0.000000e+00
Table 1: Table of the estimated regression and change plane coefficients along
with their p-values.

From the above analysis, we find that GDP has significantly positive effect on
pollution for both groups of countries. The effect of its squared term is
negative for both groups; but the effect is significant for Group-2 consisting
of mostly high income countries whereas its effect is insignificant (at the 5%
level) for the Group-1 countries (consisting of mostly low or middle income
and few high income countries). Thus, not surprisingly, we find evidence in
favor of EKC for the developed countries, but not for the mixed group.
Notably, Group-1 consists of a mixed set of countries like Angola, Sudan,
Senegal, India, China, Israel, UAE etc., whereas Group-1 consists of rich and
developed countries like Canada, USA, UK, France, Germany etc. The urban
variable, on the other hand, is seen to have insignificant effect on Group-1
which is in keeping with [7], [20]. Many of them are yet to achieve
substantial urbanization and this is more true for our sample period 444We use
6 years average from 2010-2015 for GDP and pollution measures. Such averaging
is in accordance with the cross-sectional empirical literature using cross-
country/regional data and helps avoid business cycle fluctuations in GDP. It
also minimizes the impacts of outlier events such as the financial crisis or
great recession period. The years that we have chosen are ones for which we
could find data for the largest number of countries.. In contrast, UA has a
positive and significant effect on Group-2 (developed) countries which is
consistent with the findings of [17], for example. Note that UA plays a
crucial role in dividing the countries into different regimes, as the
estimated value of $\psi_{0,2}$ is significant. Thus, we are able to partition
countries into two regimes: a mostly rich and a mixed group.

Note that many underdeveloped countries and poorer regions of emerging
countries are still swamped with greenhouse gas emissions from burning coal,
cow dung etc., and usage of poor exhaust systems in houses and for transport.
This is more true for rural and semi-urban areas of developing countries. So
even while being less urbanized compared to developed nations, their overall
pollution load is high (due to inefficient energy usage and higher dependence
on fossil fuels as pointed out above) and rising with income and they are yet
to reach the descending part of the inverted U-shape for the income-pollution
relation. On the contrary, for countries in Group-2, the adoption of more
efficient energy and exhaust systems are common in households and
transportations in general, leading to eventually decreasing pollution with
increasing income (supporting EKC). Both the results are in line with the
existing EKC literature. Additionally we find that the countries in Group 2
are yet to achieve ‘compact city’ and green urbanization. This is a stylized
fact that is confirmed by the positive and significant effect of UA on
pollution in our analysis.

There are many future potential applications of our method in economics.
Similar analyses can be performed for other pollutants (such as sulfur
emission, electrical waste/e-waste, nitrogen pollution etc.). While income/GDP
remains a common, indeed the most crucial variable in pollution studies, other
covariates (including change plane defining variables) may vary, depending on
the pollutant of interest. Another potential application can be that of
identifying the determinants of family health expenses in household survey
data. Families are often asked about their health expenses incurred in the
past one year. An interesting case in point may be household surveys collected
in India where one finds numerous (large) joint families with several children
and old people residing in the same household and most families are uninsured.
It is often seen that health expenditure increases with income with a major
factor being the costs associated with regularly performed preventative
medical examinations which are affordable only once a certain income level is
reached. The important covariates here are per capita family income, family
wealth, ‘dependency ratio’ (number of children and old to the total number of
people in the family) and the binary indicator of any history of major
illness/hospitalizations in the family in the past year. Family income per
capita and history of major illness are natural candidate covariates for
defining the change plane.

##  7 Conclusion

In this paper we have established that under some mild assumptions the kernel-
smoothed change plane estimator is asymptotically normal with near optimal
rate $n^{-1}$. To the best of our knowledge, the state of the art result in
this genre of problems is due to [21], where they demonstrate a best possible
rate about $n^{-3/4}$ for i.i.d. data. The main difference between their
approach and ours is mainly the proof of Lemma C.12. Our techniques are based
upon modern empirical process theory which allow us to consider much smaller
bandwidths $\sigma_{n}$ compared to those in [21], who appear to require
larger values to achieve the result, possibly owing to their reliance on the
techniques developed in [10]. Although we have established it is possible to
have asymptotic normality with really small bandwidths, we believe that the
finite sample approximation (e.g. Berry-Essen bound) to normality could be
poor, which is also evident from our simulation.

##  Appendix A Appendix

In this section, we present the proof of Lemma 2.5, which lies at the heart of
our refined analysis of the smoothed change plane estimator. Proofs of the
other lemmas and our results for the binary response model are available in
the Appendix B.

###  A.1 Proof of Lemma 2.5

###### Proof.

The proof of Lemma 2.5 is quite long, hence we further break it into few more
lemmas.

######  Lemma A.1.

Under Assumption (2.1), there exists $u_{-},u_{+}>0$ such that:

$u_{-}d^{2}(\theta,\theta_{0})\leq\mathbb{M}(\theta)-\mathbb{M}(\theta_{0})\leq
u_{+}d^{2}(\theta,\theta_{0})\,,$

for $\theta$ in a (non-srinking) neighborhood of $\theta_{0}$, where:

$d(\theta,\theta_{0}):=\sqrt{\|\beta-\beta_{0}\|^{2}+\|\delta-\delta_{0}\|^{2}+\|\psi-\psi_{0}\|}\,.$

######  Lemma A.2.

Under Assumption 2.1 the smoothed loss function $\mathbb{M}^{s}(\theta)$ is
uniformly close to the non-smoothed loss function $\mathbb{M}(\theta)$:

$\sup_{\theta\in\Theta}\left|\mathbb{M}^{s}(\theta)-\mathbb{M}(\theta)\right|\leq
K_{1}\sigma_{n}\,,$

for some constant $K_{1}$.

######  Lemma A.3.

Under certain assumptions:

$\displaystyle\mathbb{M}^{s}(\theta)-\mathbb{M}^{s}(\theta_{0}^{s})$
$\displaystyle\gtrsim\|\beta-\beta_{0}^{s}\|^{2}+\|\delta-\delta_{0}^{s}\|^{2}$
$\displaystyle\qquad\qquad+\frac{\|\psi-\psi_{0}^{s}\|^{2}}{\sigma_{n}}\mathds{1}_{\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}}+\|\psi-\psi_{0}^{s}\|\mathds{1}_{\|\psi-\psi_{0}^{s}\|>\mathcal{K}\sigma_{n}}$
$\displaystyle:=d_{*}^{2}(\theta,\theta_{0}^{s})\,.$

for some constant $\mathcal{K}$ and for all $\theta$ in a neighborhood of
$\theta_{0}$, which does not change with $n$.

The proofs of the three lemmas above can be found in Appendix B. We next move
to the proof of Lemma 2.5. In Lemma A.3 we have established the curvature of
the smooth loss function $\mathbb{M}^{s}(\theta)$ around $\theta_{0}^{s}$. To
determine the rate of convergence of $\hat{\theta}^{s}$ to $\theta_{0}^{s}$,
we further need an upper bound on the modulus of continuity of our loss
function. Towards that end, first recall that our loss function is:

$f_{\theta}(Y,X,Q)=\left(Y-X^{\top}\beta\right)^{2}+\left[-2\left(Y-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right]K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)$

The centered loss function can be written as:

$\displaystyle f_{\theta}(Y,X,Q)-f_{\theta_{0}^{s}}(Y,X,Q)$
$\displaystyle=\left(Y-X^{\top}\beta\right)^{2}+\left[-2\left(Y-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right]K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)$
$\displaystyle\qquad\qquad\qquad\qquad-\left(Y-X^{\top}\beta_{0}^{s}\right)^{2}-\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)$
$\displaystyle=\left(Y-X^{\top}\beta\right)^{2}+\left[-2\left(Y-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right]K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)$
$\displaystyle\qquad\qquad\qquad\qquad-\left(Y-X^{\top}\beta_{0}^{s}\right)^{2}-\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad-\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}$
$\displaystyle=\underbrace{\left(Y-X^{\top}\beta\right)^{2}-\left(Y-X^{\top}\beta_{0}^{s}\right)^{2}}_{M_{1}}$
$\displaystyle\qquad+\underbrace{\left\\{\left[-2\left(Y-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right]-\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\right\\}K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)}_{M_{2}}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad-\underbrace{\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}}_{M_{3}}$
$\displaystyle:=M_{1}+M_{2}+M_{3}$ (A.1)

For the rest of the analysis, fix $\zeta>0$ and consider the collection of
functions $\mathcal{F}_{\zeta}$ which is defined as:

$\mathcal{F}_{\zeta}=\left\\{f_{\theta}-f_{\theta^{s}}:d_{*}(\theta,\theta^{s})\leq\zeta\right\\}\,.$

First note that $\mathcal{F}_{\zeta}$ has bounded uniform entropy integral
(henceforth BUEI) over $\zeta$. To establish this, it is enough to argue that
the collection $\mathcal{F}=\\{f_{\theta}:\theta\in\Theta\\}$ is BUEI. Note
that the functions $X\mapsto X^{\top}\beta$ has VC dimension $p$ and so is the
map $X\mapsto X^{\top}(\beta+\delta)$. Therefore the functions
$(X,Y)\mapsto(Y-X^{\top}(\beta+\delta))^{2}-(Y-X^{\top}\beta)^{2}$ is also
BUEI, as composition with monotone function (here $x^{2}$) and taking
difference keeps this property. Further by the hyperplane $Q\mapsto
Q^{\top}\psi$ also has finite dimension (only depends on the dimension of $Q$)
and the VC dimension does not change by scaling it with $\sigma_{n}$.
Therefore the functions $Q\mapsto Q^{\top}\psi/sigma_{n}$ has same VC
dimension as $Q\mapsto Q^{\top}\psi$ which is independent of $n$. Again, as
composition of monotone function keeps BUEI property, the functions $Q\mapsto
K(Q^{\top}\psi/\sigma_{n})$ is also BUEI. As the product of two BUEI class is
BUEI, we conclude that $\mathcal{F}$ (and hence $\mathcal{F}_{\zeta}$) is
BUEI.

Now to bound the modulus of continuity we use Lemma 2.14.1 of [25]:

$\sqrt{n}\mathbb{E}\left[\sup_{\theta:d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left(\mathbb{P}_{n}-P\right)\left(f_{\theta}-f_{\theta_{0}^{s}}\right)\right|\right]\lesssim\mathcal{J}(1,\mathcal{F}_{\zeta})\sqrt{\mathbb{E}\left[F_{\zeta}^{2}(X,Y,Q)\right]}$

where $F_{\zeta}$ is some envelope function of $\mathcal{F}_{\zeta}$. As the
function class $\mathcal{F}_{\zeta}$ has bounded entropy integral,
$\mathcal{J}(1,\mathcal{F}_{\zeta})$ can be bounded above by some constant
independent of $n$. We next calculate the order of the envelope function
$F_{\zeta}$. Recall that, by definition of envelope function is:

$F_{\zeta}(X,Y,Q)\geq\sup_{\theta:d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|f_{\theta}-f_{\theta_{0}}\right|\,.$

and we can write $f_{\theta}-f_{\theta_{0}^{s}}=M_{1}+M_{2}+M_{3}$ which
follows from equation (A.1). Therefore, to find the order of the envelope
function, it is enough to find the order of bounds of $M_{1},M_{2},M_{3}$ over
the set $d_{*}(\theta,\theta_{0}^{s})\leq\zeta$. We start with $M_{1}$:

$\displaystyle\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}|M_{1}|$
$\displaystyle=\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\delta}\left|\left(Y-X^{\top}\beta\right)^{2}-\left(Y-X^{\top}\beta_{0}^{s}\right)^{2}\right|$
$\displaystyle=\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|2YX^{\top}(\beta_{0}^{s}-\beta)+(X^{\top}\beta)^{2}-(X^{\top}\beta_{0}^{S})^{2}\right|$
$\displaystyle\leq\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\|\beta-\beta_{0}^{s}\|\left[2|Y|\|X\|+(\|\beta_{0}^{s}\|+\zeta)\|X\|^{2}\right]$
$\displaystyle\leq\zeta\left[2|Y|\|X\|+(\|\beta_{0}^{s}\|+\zeta)\|X\|^{2}\right]:=F_{1,\zeta}(X,Y,Q)\hskip
7.22743pt[\text{Envelope function of }M_{1}]$ (A.2)

and the second term:

$\displaystyle\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}|M_{2}|$
$\displaystyle=\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{\left[-2\left(Y-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right]\right.\right.$
$\displaystyle\qquad\qquad\qquad\qquad\left.\left.-\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\right\\}\right|K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)$
$\displaystyle\leq\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{\left[-2\left(Y-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right]\right.\right.$
$\displaystyle\qquad\qquad\qquad\qquad\left.\left.-\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\right\\}\right|$
$\displaystyle=\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{\left[2Y(X^{\top}\delta_{0}^{s}-X^{\top}\delta)+2[(X^{\top}\beta)(X^{\top}\delta)\right.\right.\right.$
$\displaystyle\qquad\qquad\qquad\qquad\left.\left.\left.-(X^{\top}\beta_{0}^{s})(X^{\top}\delta_{0}^{s})]+(X^{\top}\delta)^{2}-(X^{\top}\delta_{0}^{s})^{2}\right]\right\\}\right|$
$\displaystyle\leq\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left\\{\|\delta-\delta_{0}^{s}\|2|Y|\|X\|+2\|\beta-\beta_{0}\|\|X\|\|\delta\|\right.$
$\displaystyle\qquad\qquad\qquad\qquad\left.+2\|\delta-\delta_{0}^{s}\|\|X\|\|\beta_{0}^{s}\|+2\|X\|\|\delta+\delta_{0}^{s}\|\|\delta-\delta_{0}^{s}\|\right\\}$
$\displaystyle\leq\zeta\left[2|Y|\|X\|+2\|X\|(\|\delta_{0}^{s}\|+\|\zeta\|)+2\|X\|\|\beta_{0}^{s}\|+2\|X\|(\|\delta_{0}^{s}\|+\zeta)\right]$
$\displaystyle=\zeta\times
2\|X\|\left[2|Y|+2(\|\delta_{0}^{s}\|+\|\zeta\|)+\|\beta_{0}^{s}\|\right]:=F_{2,\zeta}(X,Y,Q)\hskip
7.22743pt[\text{Envelope function of }M_{2}]$ (A.3)

For the third term, note that:

$\displaystyle\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}|M_{3}|$
$\displaystyle\leq\left|\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\right|\times\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}\right|$
$\displaystyle:=F_{3,\zeta}(X,Y,Q)$

Henceforth, we define the envelope function to be
$F_{\zeta}=F_{\zeta,1}+F_{\zeta,2}+F_{\zeta,3}$. Hence we have by triangle
inequality:

$\sqrt{\mathbb{E}\left[F_{\zeta}^{2}(X,Y,Q)\right]}\leq\sum_{i=1}^{3}\sqrt{\mathbb{E}\left[F_{i,\zeta}^{2}(X,Y,Q)\right]}$

From equation (A.2) and (A.3) we have:

$\sqrt{\mathbb{E}\left[F_{1,\zeta}^{2}(X,Y,Q)\right]}+\sqrt{\mathbb{E}\left[F_{2,\zeta}^{2}(X,Y,Q)\right]}\lesssim\zeta\,.$
(A.4)

For $F_{3,\zeta}$, first note that:

$\displaystyle\mathbb{E}\left[\left|\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\right|^{2}\mid
Q\right]$ $\displaystyle\leq
8\mathbb{E}\left[\left(Y-X^{\top}\beta_{0}^{s}\right)^{2}(X^{\top}\delta_{0})^{2}\mid
Q\right]+2\mathbb{E}[(X^{\top}\delta_{0}^{s})^{4}\mid Q]$
$\displaystyle\leq\left\\{8\|\beta-\beta_{0}^{s}\|^{2}\|\delta_{0}\|^{2}+8\|\delta_{0}\|^{4}+2\|\delta_{0}^{s}\|^{4}\right\\}m_{4}(Q)\,.$

where $m_{4}(Q)$ is defined in Assumption 2.1. In this part, we have to tackle
the dichotomous behavior of $\psi$ around $\psi_{0}^{s}$ carefully. Henceforth
define $d_{*}^{2}(\psi,\psi_{0}^{s})$ as:

$\displaystyle d_{*}^{2}(\psi,\psi_{0}^{s})=$
$\displaystyle\frac{\|\psi-\psi_{0}^{s}\|^{2}}{\sigma_{n}}\mathds{1}_{\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}}+\|\psi-\psi_{0}^{s}\|\mathds{1}_{\|\psi-\psi_{0}^{s}\|>\mathcal{K}\sigma_{n}}$

This is a slight abuse of notation, but the reader should think of it as the
part of $\psi$ in $d_{*}^{2}(\theta,\theta_{0}^{s})$. Define
$B_{\zeta}(\psi_{0}^{s})$ to be set of all $\psi$’s such that
$d^{2}_{*}(\psi,\psi_{0}^{s})\leq\zeta^{2}$. We can decompose
$B_{\zeta}(\psi_{0}^{s})$ as a disjoint union of two sets:

$\displaystyle B_{\zeta,1}(\psi_{0}^{s})$
$\displaystyle=\left\\{\psi:d^{2}_{*}(\psi,\psi_{0}^{s})\leq\zeta^{2},\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}\right\\}$
$\displaystyle=\left\\{\psi:\frac{\|\psi-\psi_{0}^{s}\|^{2}}{\sigma_{n}}\leq\zeta^{2},\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}\right\\}$
$\displaystyle=\left\\{\psi:\|\psi-\psi_{0}^{s}\|\leq\zeta\sqrt{\sigma_{n}},\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}\right\\}$
$\displaystyle B_{\zeta,2}(\psi_{0}^{s})$
$\displaystyle=\left\\{\psi:d^{2}_{*}(\psi,\psi_{0}^{s})\leq\zeta^{2},\|\psi-\psi_{0}^{s}\|>\mathcal{K}\sigma_{n}\right\\}$
$\displaystyle=\left\\{\psi:\|\psi-\psi_{0}^{s}\|\leq\zeta^{2},\|\psi-\psi_{0}^{s}\|>\mathcal{K}\sigma_{n}\right\\}$

Assume $\mathcal{K}>1$. The case where $\mathcal{K}<1$ follows from similar
calculations and hence skipped for brevity. Consider the following two cases:

Case 1: Suppose $\zeta\leq\sqrt{\mathcal{K}\sigma_{n}}$. Then
$B_{\zeta,2}=\phi$. Also as $\mathcal{K}>1$, we have:
$\zeta\sqrt{\sigma_{n}}\leq\mathcal{K}\sigma_{n}$. Hence we have:

$\sup_{d_{*}^{2}(\psi,\psi_{0}^{s})\leq\zeta^{2}}\|\psi-\psi_{0}^{s}\|=\sup_{B_{\zeta,1}}\|\psi-\psi_{0}^{s}\|=\zeta\sqrt{\sigma_{n}}\,.$

This implies:

$\displaystyle\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}\right|^{2}$
$\displaystyle\leq\max\left\\{\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}+\|\tilde{Q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)\right\\}\right|^{2},\right.$
$\displaystyle\qquad\qquad\qquad\left.\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}-\|\tilde{Q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)\right\\}\right|^{2}\right\\}$
$\displaystyle:=\max\\{T_{1},T_{2}\\}\,.$

Therefore we have:

$\mathbb{E}\left[F^{2}_{3,\zeta}(X,Y,Q)\right]\leq\mathbb{E}[m_{4}(Q)T_{1}]+\mathbb{E}[m_{4}(Q)T_{2}]\,.$

Now:

$\displaystyle\mathbb{E}[m_{4}(Q)T_{1}]$
$\displaystyle=\mathbb{E}\left[m_{4}(Q)\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}+\|\tilde{Q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)\right\\}\right|^{2}\right]$
$\displaystyle=\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\left|K\left(t\right)-K\left(t+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)\right|^{2}\
f_{s}(\sigma_{n}t\mid\tilde{q})\ dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\left|K\left(t\right)-K\left(t+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)\right|\
f_{s}(\sigma_{n}t\mid\tilde{q})\ dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\int_{t}^{t+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}K^{\prime}(s)\
ds\ f_{s}(\sigma_{n}t\mid\tilde{q})\ dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}K^{\prime}(s)\int_{s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{s}m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})f_{s}(\sigma_{n}t\mid\tilde{q})\
dt\ ds\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=\zeta\sqrt{\sigma_{n}}\mathbb{E}[\|\tilde{Q}\|m_{4}(-\tilde{Q}^{\top}\psi_{0}^{s},\tilde{Q})f_{s}(0\mid\tilde{Q})]+R$

where as before we split $R$ into three parts $R=R_{1}+R_{2}+R_{3}$.

$\displaystyle\left|R_{1}\right|$
$\displaystyle=\left|\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}K^{\prime}(s)\int_{s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{s}m_{4}(-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})(f_{s}(\sigma_{n}t\mid\tilde{q})-f_{s}(0\mid\tilde{q}))\
dt\ ds\ f(\tilde{q})\ d\tilde{q}\right|$
$\displaystyle\leq\sigma_{n}^{2}\int_{\mathbb{R}^{p-1}}m_{4}(-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\dot{f}_{s}(\tilde{q})\int_{-\infty}^{\infty}K^{\prime}(s)\int_{s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{s}|t|dt\
ds\ f(\tilde{q})\ d\tilde{q}$ (A.5)

We next calculate the inner integral (involving $(s,t)$) of equation (A.5):

$\displaystyle\int_{-\infty}^{\infty}K^{\prime}(s)\int_{s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{s}|t|dt\
ds$
$\displaystyle=\left(\int_{-\infty}^{0}+\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}+\int_{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{\infty}\right)K^{\prime}(s)\int_{s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{s}|t|dt\
ds$
$\displaystyle=\frac{1}{2}\int_{-\infty}^{0}K^{\prime}(s)\left[\left(s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)^{2}-s^{2}\right]\
ds+\frac{1}{2}\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}K^{\prime}(s)\left[\left(s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)^{2}+s^{2}\right]\
ds$
$\displaystyle\qquad\qquad\qquad\qquad+\frac{1}{2}\int_{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{\infty}K^{\prime}(s)\left[s^{2}-\left(s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)^{2}\right]\
ds$
$\displaystyle=-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\int_{-\infty}^{0}K^{\prime}(s)s\
ds+\|\tilde{q}\|^{2}\frac{\zeta^{2}}{2\sigma_{n}}\int_{-\infty}^{0}K^{\prime}(s)\
ds+\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}s^{2}K^{\prime}(s)\
ds$
$\displaystyle\qquad\qquad-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}sK^{\prime}(s)\
ds+\|\tilde{q}\|^{2}\frac{\zeta^{2}}{2\sigma_{n}}\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}K^{\prime}(s)\
ds$
$\displaystyle\qquad\qquad\qquad+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\int_{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{\infty}sK^{\prime}(s)\
ds-\|\tilde{q}\|^{2}\frac{\zeta^{2}}{2\sigma_{n}}\int_{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{\infty}K^{\prime}(s)\
ds$
$\displaystyle=\|\tilde{q}\|^{2}\frac{\zeta^{2}}{2\sigma_{n}}\left[2K\left(\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)-1\right]+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\left[-\int_{-\infty}^{0}K^{\prime}(s)s\
ds-\right.$
$\displaystyle\qquad\qquad\left.\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}K^{\prime}(s)s\
ds+\int_{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{\infty}sK^{\prime}(s)\
ds\right]+\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}s^{2}K^{\prime}(s)\
ds$
$\displaystyle=\|\tilde{q}\|^{2}\frac{\zeta^{2}}{\sigma_{n}}\left[K\left(\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)-K(0)\right]+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\left[-\int_{-\infty}^{-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}K^{\prime}(s)s\
ds+\int_{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{\infty}sK^{\prime}(s)\
ds\right]$
$\displaystyle\qquad\qquad+\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}s^{2}K^{\prime}(s)\
ds$
$\displaystyle=\|\tilde{q}\|^{2}\frac{\zeta^{2}}{\sigma_{n}}\left[K\left(\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)-K(0)\right]+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\int_{-\infty}^{\infty}K^{\prime}(s)|s|\mathds{1}_{|s|\geq\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}\
ds+\int_{0}^{\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}s^{2}K^{\prime}(s)\
ds$
$\displaystyle\leq\dot{K}_{+}\|\tilde{q}\|^{3}\frac{\zeta^{3}}{\sigma^{3/2}_{n}}+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\int_{-\infty}^{\infty}K^{\prime}(s)|s|\
ds+\|\tilde{q}\|^{2}\frac{\zeta^{2}}{\sigma_{n}}\left(K\left(\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)-K(0)\right)$
$\displaystyle\lesssim\|\tilde{q}\|^{3}\frac{\zeta^{3}}{\sigma^{3/2}_{n}}+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}$

Putting this bound in equation (A.5) we obtain:

$\displaystyle|R_{1}|$
$\displaystyle\leq\frac{\sigma_{n}^{2}}{2}\int_{\mathbb{R}^{p-1}}m_{4}(-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\dot{f}_{s}(\tilde{q})\left(\|\tilde{q}\|^{3}\frac{\zeta^{3}}{\sigma^{3/2}_{n}}+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)\
f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\frac{\zeta^{3}}{2\sqrt{\sigma_{n}}}\mathbb{E}\left[m_{4}(-\tilde{Q}^{\top}\tilde{\psi}_{0}^{s},\tilde{Q})\dot{f}_{s}(\tilde{Q})\|\tilde{Q}\|^{3}\right]+\frac{\zeta\sqrt{\sigma_{n}}}{2}\mathbb{E}\left[m_{4}(-\tilde{Q}^{\top}\tilde{\psi}_{0}^{s},\tilde{Q})\dot{f}_{s}(\tilde{Q})\|\tilde{Q}\|\right]$

and

$\displaystyle\left|R_{2}\right|$
$\displaystyle=\left|\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}K^{\prime}(s)\int_{s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{s}\left(m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})-m_{4}(-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\right)f_{s}(0\mid\tilde{q})\
dt\ ds\ f(\tilde{q})\ d\tilde{q}\right|$
$\displaystyle\leq\sigma_{n}^{2}\int_{\mathbb{R}^{p-1}}\dot{m}_{4}(\tilde{q})f_{s}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime}(s)\int_{s-\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}}^{s}|t|dt\
ds\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\sigma_{n}^{2}\int_{\mathbb{R}^{p-1}}\dot{m}_{4}(\tilde{q})f_{s}(0\mid\tilde{q})\left(\|\tilde{q}\|^{3}\frac{\zeta^{3}}{\sigma^{3/2}_{n}}+\|\tilde{q}\|\frac{\zeta}{\sqrt{\sigma_{n}}}\right)\
f(\tilde{q})\ d\tilde{q}$
$\displaystyle=\zeta\sigma_{n}^{3/2}\mathbb{E}\left[\dot{m}_{4}(\tilde{Q})f_{s}(0\mid\tilde{Q})\|\tilde{Q}\|\right]+\zeta^{3}\sqrt{\sigma_{n}}\mathbb{E}\left[\dot{m}_{4}(\tilde{Q})f_{s}(0\mid\tilde{Q})\|\tilde{Q}\|^{3}\right]$

The third residual $R_{3}$ is even higher order term and hence skipped. It is
immediate that the order of the remainders are equal to or smaller than
$\zeta\sqrt{\sigma_{n}}$ which implies:

$\mathbb{E}[m_{4}(Q)T_{1}]\lesssim\zeta\sqrt{\sigma_{n}}\,.$

The calculation for $T_{2}$ is similar and hence skipped for brevity.
Combining conclusions for $T_{1}$ and $T_{2}$ we conclude when
$\zeta\leq\sqrt{\mathcal{K}\sigma_{n}}$:

$\displaystyle\mathbb{E}\left[F^{2}_{3,\zeta}(X,Y,Q)\right]$
$\displaystyle\mathbb{E}\left[\left|\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\right|^{2}\times\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}\right|^{2}\right]$
$\displaystyle\lesssim\mathbb{E}\left[m_{4}(Q)\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}\right|^{2}\right]$
$\displaystyle\lesssim\zeta\sqrt{\sigma_{n}}\,.$ (A.6)

Case 2:  Now consider $\zeta>\sqrt{\mathcal{K}\sigma_{n}}$. Then it is
immediate that:

$\sup_{d_{*}^{2}(\psi,\psi^{s}_{0})\leq\zeta^{2}}\|\psi-\psi^{s}_{0}\|=\zeta^{2}\,.$

Using this we have:

$\displaystyle\mathbb{E}[m_{4}(Q)T_{1}]$
$\displaystyle=\mathbb{E}\left[m_{4}(Q)\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}+\|\tilde{Q}\|\frac{\zeta^{2}}{\sqrt{\sigma_{n}}}\right)\right\\}\right|^{2}\right]$
$\displaystyle=\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\left|K\left(t\right)-K\left(t+\|\tilde{q}\|\frac{\zeta^{2}}{\sigma_{n}}\right)\right|^{2}\
f_{s}(\sigma_{n}t\mid\tilde{q})\ dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\left|K\left(t\right)-K\left(t+\|\tilde{q}\|\frac{\zeta^{2}}{\sigma_{n}}\right)\right|\
f_{s}(\sigma_{n}t\mid\tilde{q})\ dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}m_{4}(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})\|\tilde{q}\|\frac{\zeta^{2}}{\sigma_{n}}\
f_{s}(\sigma_{n}t\mid\tilde{q})\ dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=\zeta^{2}\int_{\mathbb{R}^{p-1}}m_{4}(-\tilde{q}^{\top}\tilde{\psi}_{0}^{s},\tilde{q})f_{s}(0\mid\tilde{q})\|\tilde{q}\|\
f(\tilde{q})\ d\tilde{q}+R$
$\displaystyle\leq\zeta^{2}\mathbb{E}\left[\|\tilde{Q}\|m_{4}\left(-\tilde{Q}^{\top}\tilde{\psi}_{0}^{s},\tilde{Q}\right)f_{s}(0\mid\tilde{Q})\right]+R$

The analysis of the remainder term is similar and if is of higher order. This
concludes when $\zeta>\sqrt{K\sigma_{n}}$:

$\displaystyle\mathbb{E}\left[F^{2}_{3,\zeta}(X,Y,Q)\right]$
$\displaystyle\mathbb{E}\left[\left|\left[-2\left(Y-X^{\top}\beta_{0}^{s}\right)X^{\top}\delta_{0}^{s}+(X^{\top}\delta_{0}^{s})^{2}\right]\right|^{2}\times\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}\right|^{2}\right]$
$\displaystyle\lesssim\mathbb{E}\left[m_{4}(Q)\sup_{d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)\right\\}\right|^{2}\right]$
$\displaystyle\lesssim\zeta^{2}$ (A.7)

Combining (A.6), (A.7) with equation (A.4) we have:

$\displaystyle\sqrt{n}\mathbb{E}\left[\sup_{\theta:d_{*}(\theta,\theta_{0}^{s})\leq\zeta}\left|\left(\mathbb{P}_{n}-P\right)\left(f_{\theta}-f_{\theta_{0}^{s}}\right)\right|\right]$
$\displaystyle\lesssim\sqrt{\zeta}\sigma_{n}^{1/4}\mathds{1}_{\zeta\leq\sqrt{\mathcal{K}\sigma_{n}}}+\zeta\mathds{1}_{\zeta>\sqrt{\mathcal{K}\sigma_{n}}}$
$\displaystyle:=\phi_{n}(\zeta)\,.$

Hence to obtain rate we have to solve
$r_{n}^{2}\phi_{n}(1/r_{n})\leq\sqrt{n}$, i.e. (ignoring $\mathcal{K}$ as this
does not affect the rate)

$r_{n}^{3/2}\sigma_{n}^{1/4}\mathds{1}_{r_{n}\geq\sigma_{n}^{-1/2}}+r_{n}\mathds{1}_{r_{n}\leq\sigma_{n}^{-1/2}}\leq\sqrt{n}\,.$

Now if $r_{n}\leq\sigma_{n}^{-1/2}$ then $r_{n}=\sqrt{n}$ which implies
$\sqrt{n}\leq\sigma_{n}^{-1/2}$ i.e. $n\sigma_{n}\to 0$ and hence
contradiction. On the other hand, if $r_{n}\geq\sigma_{n}^{-1/2}$ then
$r_{n}=n^{1/3}\sigma_{n}^{-1/6}$. This implies
$n^{1/3}\sigma_{n}^{-1/6}\geq\sigma_{n}^{-1/2}$, i.e.
$n^{1/3}\geq\sigma_{n}^{-1/3}$, i.e. $n\sigma_{n}\to\infty$ which is okay.
This implies:

$n^{2/3}\sigma_{n}^{-1/3}d^{2}(\hat{\theta}^{s},\theta_{0}^{s})=O_{p}(1)\,.$

Now as $n^{2/3}\sigma_{n}^{-1/3}\gg\sigma_{n}^{-1}$, we have:

$\frac{1}{\sigma_{n}}d^{2}(\hat{\theta}^{s},\theta_{0}^{s})=o_{p}(1)\,.$

which further indicates $\|\hat{\psi}^{s}-\psi_{0}^{s}\|/\sigma_{n}=o_{p}(1)$.
This, along with the fact that $\|\psi_{0}^{s}-\psi_{0}\|/\sigma_{n}=o(1)$
(from Lemma C.12), establishes that
$\|\hat{\psi}_{0}^{s}-\psi_{0}\|/\sigma_{n}=o_{p}(1)$. This completes the
proof. ∎

##  Appendix B Supplementary Lemmas for the proof of Theorem 2.2

###  B.1 Proof of Lemma C.12

###### Proof.

First we establish the fact that $\theta_{0}^{s}\to\theta_{0}$. Note that for
all $n$, we have:

$\mathbb{M}^{s}(\theta_{0}^{s})\leq\mathbb{M}^{s}(\theta_{0})$

Taking $\limsup$ on the both side we have:

$\limsup_{n\to\infty}\mathbb{M}^{s}(\theta_{0}^{s})\leq\mathbb{M}(\theta_{0})\,.$

Now using Lemme A.2 we have:

$\limsup_{n\to\infty}\mathbb{M}^{s}(\theta_{0}^{s})=\limsup_{n\to\infty}\left[\mathbb{M}^{s}(\theta_{0}^{s})-\mathbb{M}(\theta_{0}^{s})+\mathbb{M}(\theta_{0}^{s})\right]=\limsup_{n\to\infty}\mathbb{M}(\theta_{0}^{s})\,.$

which implies
$\limsup_{n\to\infty}\mathbb{M}(\theta_{0}^{s})\leq\mathbb{M}(\theta_{0})$ and
from the continuity of $\mathbb{M}(\theta)$ and $\theta_{0}$ being its unique
minimizer, we conclude the proof. Now, using Lemma A.1 and Lemma A.2 we
further obtain:

$\displaystyle u_{-}d^{2}(\theta_{0}^{s},\theta_{0})$
$\displaystyle\leq\mathbb{M}(\theta_{0}^{s})-\mathbb{M}(\theta_{0})$
$\displaystyle=\mathbb{M}(\theta_{0}^{s})-\mathbb{M}^{s}(\theta^{s}_{0})+\underset{\leq
0}{\underline{\mathbb{M}^{s}(\theta_{0}^{s})-\mathbb{M}^{s}(\theta_{0})}}+\mathbb{M}^{s}(\theta_{0})-\mathbb{M}(\theta_{0})$
$\displaystyle\leq\sup_{\theta\in\Theta}\left|\mathbb{M}^{s}(\theta)-\mathbb{M}(\theta)\right|\leq
K_{1}\sigma_{n}\,.$ (B.1)

Note that we neeed consistency of $\theta_{0}^{s}$ here as the lower bound in
Lemma A.1 is only valid in a neighborhood around $\theta_{0}$. As
$\theta_{0}^{s}$ is the minimizer of $\mathbb{M}^{s}(\theta)$, from the first
order condition we have:

$\displaystyle\nabla_{\beta}\mathbb{M}^{s}_{n}(\theta_{0}^{s})$
$\displaystyle=-2\mathbb{E}\left[X(Y-X^{\top}\beta_{0}^{s})\right]+2\mathbb{E}\left\\{\left[X_{i}X_{i}^{\top}\delta_{0}^{s}\right]K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right\\}=0$
(B.2) $\displaystyle\nabla_{\delta}\mathbb{M}^{s}_{n}(\theta_{0}^{s})$
$\displaystyle=\mathbb{E}\left\\{\left[-2X_{i}\left(Y_{i}-X_{i}^{\top}\beta_{0}^{s}\right)+2X_{i}X_{i}^{\top}\delta_{0}^{s}\right]K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right\\}=0$
(B.3) $\displaystyle\nabla_{\psi}\mathbb{M}^{s}_{n}(\theta_{0}^{s})$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left\\{\left[-2\left(Y_{i}-X_{i}^{\top}\beta_{0}^{s}\right)X_{i}^{\top}\delta_{0}^{s}+(X_{i}^{\top}\delta_{0}^{s})^{2}\right]\tilde{Q}_{i}K^{\prime}\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right\\}=0$
(B.4)

We first show that $(\tilde{\psi}^{s}_{0}-\tilde{\psi}_{0})/\sigma_{n}\to 0$
by _reductio ab absurdum_. From equation (B.1), we know
$\|\psi_{0}^{s}-\psi_{0}\|/\sigma_{n}=O(1)$. Hence it has a convergent
subsequent $\psi^{s}_{0,n_{k}}$, where
$(\tilde{\psi}^{s}_{0,n_{k}}-\tilde{\psi}_{0})/\sigma_{n}\to h$. If we can
prove that $h=0$, then we establish every subsequence of
$\|\psi_{0}^{s}-\psi_{0}\|/\sigma_{n}$ has a further subsequence which
converges to $0$ which further implies $\|\psi_{0}^{s}-\psi_{0}\|/\sigma_{n}$
converges to $0$. To save some notations, we prove that if
$(\psi_{0}^{s}-\psi_{0})/\sigma_{n}\to h$ then $h=0$. We start with equation
(B.4). Define
$\tilde{\eta}=(\tilde{\psi}^{s}_{0}-\tilde{\psi}_{0})/\sigma_{n}=(\psi_{0}^{s}-\psi_{0})/\sigma_{n}$
where $\tilde{\psi}$ is all the co-ordinates of $\psi$ except the first one,
as the first co-ordinate of $\psi$ is always assumed to be $1$ for
identifiability purpose.

$\displaystyle 0$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left\\{\left[-2\left(Y_{i}-X_{i}^{\top}\beta_{0}^{s}\right)X_{i}^{\top}\delta_{0}^{s}+(X_{i}^{\top}\delta_{0}^{s})^{2}\right]\tilde{Q}_{i}K^{\prime}\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right\\}$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left[\left(-2\delta_{0}^{s}XX^{\top}(\beta_{0}-\beta^{s}_{0})-2\delta_{0}^{s}XX^{\top}\delta_{0}\mathds{1}_{Q^{\top}\delta_{0}>0}+(X_{i}^{\top}\delta_{0}^{s})^{2}\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right]$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left[\left(-2\delta_{0}^{s}XX^{\top}(\beta_{0}-\beta^{s}_{0})-2\delta_{0}^{s}XX^{\top}(\delta_{0}-\delta_{0}^{s})\mathds{1}_{Q^{\top}\delta_{0}>0}\right.\right.$
$\displaystyle\hskip
100.00015pt\left.\left.+(X_{i}^{\top}\delta_{0}^{s})^{2}\left(1-2\mathds{1}_{Q^{\top}\delta_{0}>0}\right)\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right]$
$\displaystyle=\frac{-2}{\sigma_{n}}\mathbb{E}\left[\left(\delta_{0}^{s^{\top}}g(Q)(\beta_{0}-\beta^{s}_{0})\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right]$
$\displaystyle\qquad\qquad\qquad-\frac{2}{\sigma_{n}}\mathbb{E}\left[\left(\delta_{0}^{s^{\top}}g(Q)(\delta_{0}-\delta^{s}_{0})\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\delta_{0}>0}\right]$
$\displaystyle\hskip
150.00023pt+\frac{1}{\sigma_{n}}\mathbb{E}\left[\left(\delta_{0}^{s^{\top}}g(Q)\delta^{s}_{0}\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\left(1-2\mathds{1}_{Q^{\top}\delta_{0}>0}\right)\right]$
$\displaystyle=-\underbrace{\frac{2}{\sigma_{n}}\mathbb{E}\left[\left(\delta_{0}^{s^{\top}}g(Q)(\beta_{0}-\beta^{s}_{0})\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right]}_{T_{1}}$
$\displaystyle\qquad\qquad-\underbrace{\frac{2}{\sigma_{n}}\mathbb{E}\left[\left(\delta_{0}^{s^{\top}}g(Q)(\delta_{0}-\delta^{s}_{0})\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\delta_{0}>0}\right]}_{T_{2}}$
$\displaystyle\qquad\qquad\qquad+\underbrace{\frac{1}{\sigma_{n}}\mathbb{E}\left[\left(\delta_{0}{\top}g(Q)\delta_{0}\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\left(1-2\mathds{1}_{Q^{\top}\delta_{0}>0}\right)\right]}_{T_{3}}$
$\displaystyle\qquad\qquad\qquad\qquad+\underbrace{\frac{2}{\sigma_{n}}\mathbb{E}\left[\left((\delta_{0}-\delta_{0}^{s})^{\top}g(Q)\delta_{0}\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\left(1-2\mathds{1}_{Q^{\top}\delta_{0}>0}\right)\right]}_{T_{4}}$
$\displaystyle=T_{1}+T_{2}+T_{3}+T_{4}$ (B.5)

As mentioned earlier, there is a bijection between $(Q_{1},\tilde{Q})$ and
$(Q^{\top}\psi_{0},\tilde{Q})$. The map of one side is obvious. The other side
is also trivial as the first coordinate of $\psi_{0}$ is 1, which makes
$Q^{\top}\psi_{0}=Q_{1}+\tilde{Q}^{\top}\tilde{\psi}_{0}$:

$(Q^{\top}\psi_{0},\tilde{Q})\mapsto(Q^{\top}\psi_{0}-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})\,.$

We first show that $T_{1},T_{2}$ and $T_{4}$ are $o(1)$. Towards that end
first note that:

$\displaystyle|T_{1}|$
$\displaystyle\leq\frac{2}{\sigma_{n}}\mathbb{E}\left[\|g(Q)\|_{op}\
\|\tilde{Q}\|\
\left|K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right|\right]\|\delta_{0}^{s}\|\|\beta_{0}-\beta_{0}^{s}\|$
$\displaystyle|T_{2}|$
$\displaystyle\leq\frac{2}{\sigma_{n}}\mathbb{E}\left[\|g(Q)\|_{op}\
\|\tilde{Q}\|\
\left|K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right|\right]\|\delta_{0}^{s}\|\|\delta_{0}-\delta_{0}^{s}\|$
$\displaystyle|T_{4}|$
$\displaystyle\leq\frac{2}{\sigma_{n}}\mathbb{E}\left[\|g(Q)\|_{op}\
\|\tilde{Q}\|\
\left|K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right|\right]\|\delta_{0}^{s}\|\|\delta_{0}-\delta_{0}^{s}\|$

From the above bounds, it is immediate that to show that above terms are
$o(1)$ all we need to show is:

$\frac{1}{\sigma_{n}}\mathbb{E}\left[\|g(Q)\|_{op}\ \|\tilde{Q}\|\
\left|K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right|\right]=O(1)\,.$

Towards that direction, define
$\eta=(\tilde{\psi}_{0}^{s}-\tilde{\psi}_{0})/\sigma_{n}$:

$\displaystyle\frac{1}{\sigma_{n}}\mathbb{E}\left[\|g(Q)\|_{op}\
\|\tilde{Q}\|\
\left|K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right|\right]$
$\displaystyle\leq c_{+}\frac{1}{\sigma_{n}}\mathbb{E}\left[\|\tilde{Q}\|\
\left|K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\right|\right]$
$\displaystyle=c_{+}\frac{1}{\sigma_{n}}\int\int\|\tilde{q}\|\left|K^{\prime}\left(\frac{t}{\sigma_{n}}+\tilde{q}^{\top}\eta\right)\right|f_{0}\left(t\mid\tilde{q}\right)f(\tilde{q})\
dt\ d\tilde{q}$
$\displaystyle=c_{+}\int\int\|\tilde{q}\|\left|K^{\prime}\left(t+\tilde{q}^{\top}\eta\right)\right|f_{0}\left(\sigma_{n}t\mid\tilde{q}\right)f(\tilde{q})\
dt\ d\tilde{q}$
$\displaystyle=c_{+}\int\|\tilde{q}\|f_{0}\left(0\mid\tilde{q}\right)\int\left|K^{\prime}\left(t+\tilde{q}^{\top}\eta\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}+R_{1}$
$\displaystyle=c_{+}\int\left|K^{\prime}\left(t\right)\right|dt\
\mathbb{E}\left[\|\tilde{Q}\|f_{0}(0\mid\tilde{Q})\right]+R_{1}=O(1)+R_{1}\,.$

Therefore, all it remains to show is $R_{1}$ is also $O(1)$ (or of smaller
order):

$\displaystyle|R_{1}|$
$\displaystyle=\left|c_{+}\int\int\|\tilde{q}\|\left|K^{\prime}\left(t+\tilde{q}^{\top}\eta\right)\right|\left(f_{0}\left(\sigma_{n}t\mid\tilde{q}\right)-f_{0}(0\mid\tilde{q})\right)f(\tilde{q})\
dt\ d\tilde{q}\right|$ $\displaystyle\leq
c_{+}F_{+}\sigma_{n}\int\|\tilde{q}\|\int_{-\infty}^{\infty}|t|\left|K^{\prime}\left(t+\tilde{q}^{\top}\eta\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=c_{+}F_{+}\sigma_{n}\int\|\tilde{q}\|\int_{-\infty}^{\infty}|t-q^{\top}\eta|\left|K^{\prime}\left(t\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}$ $\displaystyle\leq
c_{+}F_{+}\sigma_{n}\left[\int\|\tilde{q}\|\int_{-\infty}^{\infty}|t|\left|K^{\prime}\left(t\right)\right|\
dt\ f(\tilde{q})\
d\tilde{q}+\int\|\tilde{q}\|^{2}\|\eta\|\int_{-\infty}^{\infty}\left|K^{\prime}\left(t\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}\right]$
$\displaystyle=c_{+}F_{+}\sigma_{n}\left[\left(\int_{-\infty}^{\infty}|t|\left|K^{\prime}\left(t\right)\right|\
dt\right)\times\mathbb{E}[\|\tilde{Q}\|]+\left(\int_{-\infty}^{\infty}\left|K^{\prime}\left(t\right)\right|\
dt\right)\times\|\eta\|\ \mathbb{E}[\|\tilde{Q}\|^{2}]\right]$
$\displaystyle=O(\sigma_{n})=o(1)\,.$

This completes the proof. For $T_{3}$, the limit is non-degenerate which can
be calculated as follows:

$\displaystyle T_{3}$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left[\left(\delta_{0}{\top}g(Q)\delta_{0}\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi^{s}_{0}}{\sigma_{n}}\right)\left(1-2\mathds{1}_{Q^{\top}\delta_{0}>0}\right)\right]$
$\displaystyle=\frac{1}{\sigma_{n}}\int\int\left(\delta_{0}{\top}g(t-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q})\delta_{0}\right)\tilde{q}K^{\prime}\left(\frac{t}{\sigma_{n}}+\tilde{q}^{\top}\eta\right)\left(1-2\mathds{1}_{t>0}\right)\
f_{0}(t\mid\tilde{q})\ f(\tilde{q})\ dt\ d\tilde{q}$
$\displaystyle=\int\int\left(\delta_{0}{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q})\delta_{0}\right)\tilde{q}K^{\prime}\left(t+\tilde{q}^{\top}\eta\right)\left(1-2\mathds{1}_{t>0}\right)\
f_{0}(\sigma_{n}t\mid\tilde{q})\ f(\tilde{q})\ dt\ d\tilde{q}$
$\displaystyle=\int\int\left(\delta_{0}{\top}g(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q})\delta_{0}\right)\tilde{q}K^{\prime}\left(t+\tilde{q}^{\top}\eta\right)\left(1-2\mathds{1}_{t>0}\right)\
f_{0}(0\mid\tilde{q})\ f(\tilde{q})\ dt\ d\tilde{q}+R$
$\displaystyle=\int\left(\delta_{0}{\top}g(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q})\delta_{0}\right)\tilde{q}f_{0}(0\mid\tilde{q})\left[\int_{-\infty}^{0}K^{\prime}\left(t+\tilde{q}^{\top}\eta\right)\
dt-\int_{0}^{\infty}K^{\prime}\left(t+\tilde{q}^{\top}\tilde{\eta}\right)\
dt\right]\ f(\tilde{q})\ d\tilde{q}+R$
$\displaystyle=\int\left(\delta_{0}{\top}g(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q})\delta_{0}\right)\tilde{q}f_{0}(0\mid\tilde{q})\left(2K\left(\tilde{q}^{\top}\eta\right)-1\right)\
f(\tilde{q})\ d\tilde{q}+R$
$\displaystyle=\mathbb{E}\left[\tilde{Q}f(0\mid\tilde{Q})\left(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})\delta_{0}\right)\left(2K(\tilde{Q}^{\top}\eta)-1\right)\right]+R$

That the remainder $R$ is $o(1)$ again follows by similar calculation as
before and hence skipped. Therefore we have when
$\eta=(\tilde{\psi}_{0}^{s}-\psi_{0})/\sigma_{n}\to h$:

$T_{3}\overset{n\to\infty}{\longrightarrow}\mathbb{E}\left[\tilde{Q}f(0\mid\tilde{Q})\left(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})\delta_{0}\right)\left(2K(\tilde{Q}^{\top}h)-1\right)\right]\,,$

which along with equation (B.1) implies:

$\mathbb{E}\left[\tilde{Q}f(0\mid\tilde{Q})\left(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})\delta_{0}\right)\left(2K(\tilde{Q}^{\top}h)-1\right)\right]=0\,.$

Taking inner product with respect to $h$ on both side of the above equation we
obtain:

$\mathbb{E}\left[\tilde{Q}^{\top}hf(0\mid\tilde{Q})\left(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})\delta_{0}\right)\left(2K(\tilde{Q}^{\top}h)-1\right)\right]=0$

Now from the symmetry of our Kernel $K$ we have
$\left(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})\delta_{0}\right)\tilde{Q}^{\top}hf(0\mid\tilde{Q})(2K(\tilde{Q}^{\top}\tilde{h})-1)\geq
0$ almost surely. As the expectation is $0$, we further deduce that
$\tilde{Q}^{\top}hf(0\mid\tilde{Q})(2K(\tilde{Q}^{\top}\tilde{h})-1)=0$ almost
surely, which further implies $h=0$.

We next prove that $(\beta_{0}-\beta^{s}_{0})/\sqrt{\sigma_{n}}\to 0$ and
$(\delta_{0}-\delta^{s}_{0})/\sqrt{\sigma_{n}}\to 0$ using equations(B.2) and
(B.3). We start with equation (B.2):

$\displaystyle 0$
$\displaystyle=-\mathbb{E}\left[X(Y-X^{\top}\beta_{0}^{s})\right]+\mathbb{E}\left\\{\left[X_{i}X_{i}^{\top}\delta_{0}^{s}\right]K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right\\}$
$\displaystyle=-\mathbb{E}\left[XX^{\top}(\beta_{0}-\beta_{0}^{s})\right]-\mathbb{E}[XX^{\top}\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}]+\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right]\delta_{0}^{s}$
$\displaystyle=-\Sigma_{X}(\beta_{0}-\beta_{0}^{s})-\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right](\delta_{0}-\delta_{0}^{s})+\mathbb{E}\left[g(Q)\left\\{K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi_{0}>0}\right\\}\right]\delta_{0}^{s}$
$\displaystyle=\Sigma_{X}\frac{(\beta_{0}^{2}-\beta_{0})}{\sigma_{n}}+\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\frac{(\delta_{0}^{2}-\delta_{0})}{\sigma_{n}}+\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)\left\\{K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi_{0}>0}\right\\}\right]\delta_{0}^{s}$
$\displaystyle=\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\Sigma_{X}\frac{(\beta_{0}^{2}-\beta_{0})}{\sigma_{n}}+\frac{\delta_{0}^{s}-\delta_{0}}{\sigma_{n}}$
$\displaystyle\qquad\qquad\qquad\qquad+\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)\left\\{K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi_{0}>0}\right\\}\right]\delta_{0}^{s}$
(B.6)

From equation (B.3) we have:

$\displaystyle 0$
$\displaystyle=\mathbb{E}\left\\{\left[-X\left(Y-X^{\top}\beta_{0}^{s}\right)+XX^{\top}\delta_{0}^{s}\right]K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right\\}$
$\displaystyle=-\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right](\beta_{0}-\beta_{0}^{s})-\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\delta_{0}$
$\displaystyle\hskip
200.0003pt+\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right]\delta_{0}^{s}$
$\displaystyle=-\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right](\beta_{0}-\beta_{0}^{s})-\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right](\delta_{0}-\delta_{0}^{s})$
$\displaystyle\hskip
200.0003pt+\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\left(1-\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\right]\delta_{0}^{s}$
$\displaystyle=\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right]\frac{(\beta_{0}^{s}-\beta_{0})}{\sigma_{n}}+\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\frac{(\delta^{s}_{0}-\delta_{0})}{\sigma_{n}}$
$\displaystyle\hskip
200.0003pt+\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\left(1-\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\right]\delta_{0}^{s}$
$\displaystyle=\left(\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right]\frac{(\beta_{0}^{s}-\beta_{0})}{\sigma_{n}}+\frac{(\delta^{s}_{0}-\delta_{0})}{\sigma_{n}}$
$\displaystyle\qquad\qquad\qquad+\left(\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\left(1-\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\right]\delta_{0}^{s}$
(B.7)

Subtracting equation (B.1) from (B.1) we obtain:

$0=A_{n}\frac{(\beta_{0}^{s}-\beta_{0})}{\sigma_{n}}+b_{n}\,,$

i.e.

$\lim_{n\to\infty}\frac{(\beta_{0}^{s}-\beta_{0})}{\sigma_{n}}=\lim_{n\to\infty}-A_{n}^{-1}b_{n}\,.$

where:

$\displaystyle A_{n}$
$\displaystyle=\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\Sigma_{X}$
$\displaystyle\qquad\qquad-\left(\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right]$
$\displaystyle b_{n}$
$\displaystyle=\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)\left\\{K\left(\frac{Q_{i}^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi_{0}>0}\right\\}\right]\delta_{0}^{s}$
$\displaystyle\qquad-\left(\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\left(1-\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\right]\delta_{0}^{s}$

It is immediate via DCT that as $n\to\infty$:

$\displaystyle\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\right]$
$\displaystyle\longrightarrow\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\,.$
(B.8)
$\displaystyle\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]$
$\displaystyle\longrightarrow\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\,.$
(B.9)

From equation (B.8) and (B.9) it is immediate that:

$\displaystyle\lim_{n\to\infty}A_{n}$
$\displaystyle=\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\Sigma_{X}-I$
$\displaystyle=\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}\leq
0}\right]\right):=A\,.$

Next observe that:

$\displaystyle\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)\left\\{K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi_{0}>0}\right\\}\right]$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)\left\\{K\left(\frac{Q^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)-\mathds{1}_{Q^{\top}\psi_{0}>0}\right\\}\right]$
$\displaystyle=\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q})\left[K\left(t+\tilde{q}^{\top}\tilde{\eta}\right)-\mathds{1}_{t>0}\right]f(\sigma_{n}t\mid\tilde{q})\
dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\longrightarrow\mathbb{E}\left[g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})f(0\mid\tilde{Q})\right]\cancelto{0}{\int_{-\infty}^{\infty}\left[K\left(t\right)-\mathds{1}_{t>0}\right]\
dt}\,.$ (B.10)

Similar calculation yields:

$\displaystyle\frac{1}{\sigma_{n}}\mathbb{E}\left[g(Q)K\left(\frac{Q^{\top}\psi_{0}^{s}}{\sigma_{n}}\right)\left(1-\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\right]$
$\displaystyle\longrightarrow\mathbb{E}[g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})f_{0}(0\mid\tilde{Q})]\int_{-\infty}^{\infty}\left[K\left(t\right)\mathds{1}_{t\leq
0}\right]\ dt\,.$ (B.11)

Combining equation (B.10) and (B.1) we conclude:

$\displaystyle\lim_{n\to\infty}b_{n}$
$\displaystyle=\left(\mathbb{E}\left[g(Q)\mathds{1}_{Q^{\top}\psi_{0}>0}\right]\right)^{-1}\mathbb{E}[g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})\delta_{0}f_{0}(0\mid\tilde{Q})]\int_{-\infty}^{\infty}\left[K\left(t\right)\mathds{1}_{t\leq
0}\right]\ dt$ $\displaystyle:=b\,.$

which further implies,

$\lim_{n\to\infty}\frac{(\beta_{0}^{s}-\beta_{0})}{\sigma_{n}}=-A^{-1}b\implies(\beta_{0}^{s}-\beta_{0})=o(\sqrt{\sigma_{n}})\,,$

and by similar calculations:

$(\delta_{0}^{s}-\delta_{0})=o(\sqrt{\sigma_{n}})\,.$

This completes the proof. ∎

###  B.2 Proof of Lemma A.1

###### Proof.

From the definition of $M(\theta)$ it is immediate that
$\mathbb{M}(\theta_{0})=\mathbb{E}[{\epsilon}^{2}]=\sigma^{2}$. For any
general $\theta$:

$\displaystyle\mathbb{M}(\theta)$
$\displaystyle=\mathbb{E}\left[\left(Y-X^{\top}\left(\beta+\delta\mathds{1}_{Q^{\top}\psi>0}\right)\right)^{2}\right]$
$\displaystyle=\sigma^{2}+\mathbb{E}\left[\left(X^{\top}\left(\beta+\delta\mathds{1}_{Q^{\top}\psi>0}-\beta_{0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\right)^{2}\right]$
$\displaystyle\geq\sigma^{2}+c_{-}\mathbb{E}_{Q}\left[\left\|\beta-\beta_{0}+\delta\mathds{1}_{Q^{\top}\psi>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right\|^{2}\right]$

This immediately implies:

$\mathbb{M}(\theta)-\mathbb{M}(\theta_{0})\geq
c_{-}\mathbb{E}\left[\left\|\beta-\beta_{0}+\delta\mathds{1}_{Q^{\top}\psi>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right\|^{2}\right]\,.$

For notational simplicity, define $p_{\psi}=\mathbb{P}(Q^{\top}\psi>0)$.
Expanding the RHS we have:

$\displaystyle\mathbb{E}\left[\left\|\beta-\beta_{0}+\delta\mathds{1}_{Q^{\top}\psi>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right\|^{2}\right]$
$\displaystyle=\|\beta-\beta_{0}\|^{2}+2(\beta-\beta_{0})^{\top}\mathbb{E}\left[\delta\mathds{1}_{Q^{\top}\psi>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right]+\mathbb{E}\left[\left\|\delta\mathds{1}_{Q^{\top}\psi>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right\|^{2}\right]$
$\displaystyle=\|\beta-\beta_{0}\|^{2}+2(\beta-\beta_{0})^{\top}\mathbb{E}\left[\delta\mathds{1}_{Q^{\top}\psi>0}-\delta\mathds{1}_{Q^{\top}\psi_{0}>0}+\delta\mathds{1}_{Q^{\top}\psi_{0}>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right]$
$\displaystyle\qquad\qquad\qquad\qquad\qquad+\mathbb{E}\left[\left\|\delta\mathds{1}_{Q^{\top}\psi>0}-\delta\mathds{1}_{Q^{\top}\psi_{0}>0}+\delta\mathds{1}_{Q^{\top}\psi_{0}>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right\|^{2}\right]$
$\displaystyle=\|\beta-\beta_{0}\|^{2}+2(\beta-\beta_{0})^{\top}(\delta-\delta_{0})p_{\psi_{0}}+\|\delta-\delta_{0}\|^{2}p_{\psi_{0}}$
$\displaystyle\qquad\qquad\qquad+2(\beta-\beta_{0})^{\top}\delta\left(p_{\psi}-p_{\psi_{0}}\right)+\|\delta\|^{2}\mathbb{P}\left(\text{sign}(Q^{\top}\psi)\neq\text{sign}(Q^{\top}\psi_{0})\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad-2\delta^{\top}(\delta-\delta_{0})\mathbb{P}\left(Q^{\top}\psi_{0}>0,Q^{\top}\psi<0\right)$
(B.12)

Using the fact that $2ab\geq(a^{2}/c)+cb^{2}$ for any constant $c$ we have:

$\displaystyle\|\beta-\beta_{0}\|^{2}+2(\beta-\beta_{0})^{\top}(\delta-\delta_{0})p_{\psi_{0}}+\|\delta-\delta_{0}\|^{2}p_{\psi_{0}}$
$\displaystyle\geq\|\beta-\beta_{0}\|^{2}+\|\delta-\delta_{0}\|^{2}p_{\psi_{0}}-\frac{\|\beta-\beta_{0}\|^{2}p_{\psi_{0}}}{c}-c\|\delta-\delta_{0}\|^{2}p_{\psi_{0}}$
$\displaystyle=\|\beta-\beta_{0}\|^{2}\left(1-\frac{p_{\psi_{0}}}{c}\right)+\|\delta-\delta_{0}\|^{2}p_{\psi_{0}}(1-c)\,.$

for any $c$. To make the RHS non-negative we pick $p_{\psi_{0}}<c<1$ and
concludes that:

$\|\beta-\beta_{0}\|^{2}+2(\beta-\beta_{0})^{\top}(\delta-\delta_{0})p_{\psi_{0}}+\|\delta-\delta_{0}\|^{2}p_{\psi_{0}}\gtrsim\left(\|\beta-\beta_{0}\|^{2}+\|\delta-\delta_{0}\|^{2}\right)\,.$
(B.13)

For the last 3 summands of RHS of equation (B.12):

$\displaystyle
2(\beta-\beta_{0})^{\top}\delta\left(p_{\psi}-p_{\psi_{0}}\right)+\|\delta\|^{2}\mathbb{P}\left(\text{sign}(Q^{\top}\psi)\neq\text{sign}(Q^{\top}\psi_{0})\right)$
$\displaystyle\qquad\qquad-2\delta^{\top}(\delta-\delta_{0})\mathbb{P}\left(Q^{\top}\psi_{0}>0,Q^{\top}\psi<0\right)$
$\displaystyle=2(\beta-\beta_{0})^{\top}\delta\mathbb{P}\left(Q^{\top}\psi>0,Q^{\top}\psi_{0}<0\right)-2(\beta-\beta_{0})^{\top}\delta\mathbb{P}\left(Q^{\top}\psi<0,Q^{\top}\psi_{0}>0\right)$
$\displaystyle\qquad\qquad+|\delta\|^{2}\mathbb{P}\left(\text{sign}(Q^{\top}\psi)\neq\text{sign}(Q^{\top}\psi_{0})\right)-2\delta^{\top}(\delta-\delta_{0})\mathbb{P}\left(Q^{\top}\psi_{0}>0,Q^{\top}\psi<0\right)$
$\displaystyle=\left[\|\delta\|^{2}-2(\beta-\beta_{0})^{\top}\delta-2\delta^{\top}(\delta-\delta_{0})\right]\mathbb{P}\left(Q^{\top}\psi_{0}>0,Q^{\top}\psi<0\right)$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad+\left[\|\delta\|^{2}+2(\beta-\beta_{0})^{\top}\delta\right]\mathbb{P}\left(Q^{\top}\psi>0,Q^{\top}\psi_{0}<0\right)$
$\displaystyle=\left[\|\delta_{0}\|^{2}-2(\beta-\beta_{0})^{\top}(\delta-\delta_{0})-2(\beta-\beta_{0})^{\top}\delta_{0}-\|\delta-\delta_{0}\|^{2}\right]\mathbb{P}\left(Q^{\top}\psi_{0}>0,Q^{\top}\psi<0\right)$
$\displaystyle\qquad+\left[\|\delta_{0}\|^{2}+\|\delta-\delta_{0}\|^{2}+2(\delta-\delta_{0})^{\top}\delta_{0}+2(\beta-\beta_{0})^{\top}(\delta-\delta_{0})+2(\beta-\beta_{0})^{\top}\delta_{0}\right]\mathbb{P}\left(Q^{\top}\psi>0,Q^{\top}\psi_{0}<0\right)$
$\displaystyle\geq\left[\|\delta_{0}\|^{2}-2\|\beta-\beta_{0}\|\|\delta-\delta_{0}\|-2\|\beta-\beta_{0}\|\|\delta_{0}\|-\|\delta-\delta_{0}\|^{2}\right]\mathbb{P}\left(Q^{\top}\psi_{0}>0,Q^{\top}\psi<0\right)$
$\displaystyle\qquad+\left[\|\delta_{0}\|^{2}+\|\delta-\delta_{0}\|^{2}+2\|\delta-\delta_{0}\|\|\delta_{0}\|+2\|\beta-\beta_{0}\|\|\delta-\delta_{0}\|+2\|\beta-\beta_{0}\|\|\delta_{0}\|\right]\mathbb{P}\left(Q^{\top}\psi>0,Q^{\top}\psi_{0}<0\right)$
$\displaystyle\gtrsim\|\delta_{0}\|^{2}\mathbb{P}\left(\text{sign}(Q^{\top}\psi)\neq\text{sign}(Q^{\top}\psi_{0})\right)\gtrsim\|\psi-\psi_{0}\|\hskip
14.45377pt[\text{By Assumption }\ref{eq:assm}]\,.$ (B.14)

Combining equation (B.13) and (B.14) we complete the proof of lower bound. The
upper bound is relatively easier: note that by our previous calculation:

$\displaystyle\mathbb{M}(\theta)-\mathbb{M}(\theta_{0})$
$\displaystyle=\mathbb{E}\left[\left(X^{\top}\left(\beta+\delta\mathds{1}_{Q^{\top}\psi>0}-\beta_{0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\right)^{2}\right]$
$\displaystyle\leq
c_{+}\mathbb{E}\left[\left\|\beta-\beta_{0}+\delta\mathds{1}_{Q^{\top}\psi>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right\|^{2}\right]$
$\displaystyle=c_{+}\mathbb{E}\left[\left\|\beta-\beta_{0}+\delta\mathds{1}_{Q^{\top}\psi>0}-\delta\mathds{1}_{Q^{\top}\psi_{0}>0}+\delta\mathds{1}_{Q^{\top}\psi_{0}>0}-\delta_{0}\mathds{1}_{Q^{\top}\psi_{0}>0}\right\|^{2}\right]$
$\displaystyle\lesssim\left[\|\beta-\beta_{0}\|^{2}+\|\delta-\delta_{0}\|^{2}+\mathbb{P}\left(\text{sign}(Q^{\top}\psi)\neq\text{sign}(Q^{\top}\psi_{0})\right)\right]$
$\displaystyle\lesssim\left[\|\beta-\beta_{0}\|^{2}+\|\delta-\delta_{0}\|^{2}+\|\psi-\psi_{0}\|\right]\,.$

This completes the entire proof. ∎

###  B.3 Proof of Lemma A.2

###### Proof.

The difference of the two losses:

$\displaystyle\left|\mathbb{M}^{s}(\theta)-\mathbb{M}(\theta)\right|$
$\displaystyle=\left|\mathbb{E}\left[\left\\{-2\left(Y_{i}-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right\\}\left(K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi>0}\right)\right]\right|$
$\displaystyle\leq\mathbb{E}\left[\left|-2\left(Y_{i}-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right|\left|K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi>0}\right|\right]$
$\displaystyle:=\mathbb{E}\left[m(Q)\left|K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi>0}\right|\right]$

where
$m(Q)=\mathbb{E}\left[\left|-2\left(Y_{i}-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right|\mid
Q\right]$. This function can be bounded as follows:

$\displaystyle m(Q)$
$\displaystyle=\mathbb{E}\left[\left|-2\left(Y_{i}-X^{\top}\beta\right)X^{\top}\delta+(X^{\top}\delta)^{2}\right|\mid
Q\right]$ $\displaystyle\leq\mathbb{E}[(X^{\top}\delta)^{2}\mid
Q]+2\mathbb{E}\left[\left|(\beta-\beta_{0})^{\top}XX^{\top}\delta\right|\right]+2\mathbb{E}\left[\left|\delta_{0}^{\top}XX^{\top}\delta\right|\right]$
$\displaystyle\leq
c_{+}\left(\|\delta\|^{2}+2\|\beta-\beta_{0}\|\|\delta\|+2\|\delta\|\|\delta_{0}\|\right)\lesssim
1\,,$

as our parameter space is compact. For the rest of the calculation define
$\eta=(\tilde{\psi}-\tilde{\psi}_{0})/\sigma_{n}$. The definition of $\eta$
may be changed from proof to proof, but it will be clear from the context.
Therefore we have:

$\displaystyle\left|\mathbb{M}^{s}(\theta)-\mathbb{M}(\theta)\right|$
$\displaystyle\lesssim\mathbb{E}\left[\left|K\left(\frac{Q^{\top}\psi}{\sigma_{n}}\right)-\mathds{1}_{Q^{\top}\psi>0}\right|\right]$
$\displaystyle=\mathbb{E}\left[\left|\mathds{1}\left(\frac{Q^{\top}\psi_{0}}{\sigma_{n}}+\eta^{\top}\tilde{Q}\geq
0\right)-K\left(\frac{Q^{\top}\psi_{0}}{\sigma_{n}}+\eta^{\top}\tilde{Q}\right)\right|\right]$
$\displaystyle=\sigma_{n}\int_{\mathbb{R}^{p-1}}\int_{-\infty}^{\infty}\left|\mathds{1}\left(t\geq
0\right)-K\left(t\right)\right|f_{0}(\sigma_{n}(t-\eta^{\top}\tilde{q})|\tilde{q})\
dt\ dP(\tilde{q})$ $\displaystyle\leq
f_{+}\sigma_{n}\int_{-\infty}^{\infty}\left|\mathds{1}\left(t\geq
0\right)-K\left(t\right)\right|\ dt\lesssim\sigma_{n}\,.$

where the integral over $t$ is finite follows from the definition of the
kernel. This completes the proof. ∎

###  B.4 Proof of Lemma A.3

###### Proof.

First note that we can write:

$\displaystyle\mathbb{M}^{s}(\theta)-\mathbb{M}^{s}(\theta_{0}^{s})$
$\displaystyle=\underbrace{\mathbb{M}^{s}(\theta)-\mathbb{M}(\theta)}_{\geq-
K_{1}\sigma_{n}}+\underbrace{\mathbb{M}(\theta)-\mathbb{M}(\theta_{0})}_{\underbrace{\geq
u_{-}d^{2}(\theta,\theta_{0})}_{\geq\frac{u_{-}}{2}d^{2}(\theta,\theta_{0}^{s})-u_{-}\sigma_{n}}}+\underbrace{\mathbb{M}(\theta_{0})-\mathbb{M}(\theta_{0}^{s})}_{\geq-
u_{+}d^{2}(\theta_{0},\theta_{0}^{s})\geq-
u_{+}\sigma_{n}}+\underbrace{\mathbb{M}(\theta_{0}^{s})-\mathbb{M}^{s}(\theta_{0}^{s})}_{\geq-
K_{1}\sigma_{n}}$
$\displaystyle\geq\frac{u_{-}}{2}d^{2}(\theta,\theta_{0}^{s})-(2K_{1}+\xi)\sigma_{n}$
$\displaystyle\geq\frac{u_{-}}{2}\left[\|\beta-\beta^{s}_{0}\|^{2}+\|\delta-\delta^{s}_{0}\|^{2}+\|\psi-\psi^{s}_{0}\|\right]-(2K_{1}+\xi)\sigma_{n}$
$\displaystyle\geq\left[\frac{u_{-}}{2}\left(\|\beta-\beta^{s}_{0}\|^{2}+\|\delta-\delta^{s}_{0}\|^{2}\right)+\frac{u_{-}}{4}\|\psi-\psi^{s}_{0}\|\right]\mathds{1}_{\|\psi-\psi^{s}_{0}\|>\frac{4(2K_{1}+\xi)}{u_{-}}\sigma_{n}}$
$\displaystyle\gtrsim\left[\|\beta-\beta^{s}_{0}\|^{2}+\|\delta-\delta^{s}_{0}\|^{2}+\|\psi-\psi^{s}_{0}\|\right]\mathds{1}_{\|\psi-\psi^{s}_{0}\|>\frac{4(2K_{1}+\xi)}{u_{-}}\sigma_{n}}$
(B.15)

where $\xi$ can be taken as close to $0$ as possible. Henceforth we set
$\mathcal{K}=4(2K_{1}+\xi)/u_{-}$. For the other part of the curvature (i.e.
when $\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}$) we start with a two
step Taylor expansion of the smoothed loss function:

$\displaystyle\mathbb{M}^{s}(\theta)-\mathbb{M}^{s}(\theta_{0}^{s})=\frac{1}{2}(\theta_{0}-\theta^{0}_{s})^{\top}\nabla^{2}\mathbb{M}^{s}(\theta^{*})(\theta_{0}-\theta^{0}_{s})$

Recall the definition of $\mathbb{M}^{s}(\theta)$:

$\mathbb{M}^{s}_{n}(\theta)=\mathbb{E}\left(Y-X^{\top}\beta\right)^{2}+\mathbb{E}\left\\{\left[-2\left(Y_{i}-X_{i}^{\top}\beta\right)X_{i}^{\top}\delta+(X_{i}^{\top}\delta)^{2}\right]K\left(\frac{Q_{i}^{\top}\psi}{\sigma_{n}}\right)\right\\}$

The partial derivates of $\mathbb{M}^{s}(\theta)$ with respect to
$(\beta,\delta,\psi)$ was derived in equation (B.2) - (B.4). From there, we
calculate the hessian of $\mathbb{M}^{s}(\theta)$:

$\displaystyle\nabla_{\beta\beta}\mathbb{M}^{s}(\theta)$
$\displaystyle=2\Sigma_{X}$
$\displaystyle\nabla_{\delta\delta}\mathbb{M}^{s}(\theta)$
$\displaystyle=2\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi}{\sigma_{n}}\right)\right]=2\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right]$
$\displaystyle\nabla_{\psi\psi}\mathbb{M}^{s}(\theta)$
$\displaystyle=\frac{1}{\sigma_{n}^{2}}\mathbb{E}\left\\{\left[-2\left(Y_{i}-X_{i}^{\top}\beta\right)X_{i}^{\top}\delta+(X_{i}^{\top}\delta)^{2}\right]\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle\nabla_{\beta\delta}\mathbb{M}^{s}(\theta)$
$\displaystyle=2\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi}{\sigma_{n}}\right)\right]=2\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right]$
$\displaystyle\nabla_{\beta\psi}\mathbb{M}^{s}(\theta)$
$\displaystyle=\frac{2}{\sigma_{n}}\mathbb{E}\left(g(Q)\delta\tilde{Q}^{\top}K^{\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right)$
$\displaystyle\nabla_{\delta\psi}\mathbb{M}^{s}(\theta)$
$\displaystyle=\frac{2}{\sigma_{n}}\mathbb{E}\left\\{\left[-X_{i}\left(Y_{i}-X_{i}^{\top}\beta\right)+X_{i}X_{i}^{\top}\delta\right]\tilde{Q}_{i}^{\top}K^{\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}\,.$

where we use $\tilde{\eta}$ for a generic notation for
$(\tilde{\psi}-\tilde{\psi}_{0})/\sigma_{n}$. For notational simplicity, we
define $\gamma=(\beta,\delta)$ and $\nabla^{2}\mathbb{M}^{s,\gamma}(\theta)$,
$\nabla^{2}\mathbb{M}^{s,\gamma\psi}(\theta),\nabla^{2}\mathbb{M}^{s,\psi\psi}(\theta)$
to be corresponding blocks of the hessian matrix. We have:

$\displaystyle\mathbb{M}^{s}(\theta)-\mathbb{M}^{s}(\theta_{0}^{s})$
$\displaystyle=\frac{1}{2}(\theta-\theta^{0}_{s})^{\top}\nabla^{2}\mathbb{M}^{s}(\theta^{*})(\theta-\theta^{0}_{s})$
$\displaystyle=\frac{1}{2}(\gamma-\gamma_{0}^{s})^{\top}\nabla^{2}\mathbb{M}^{s,\gamma}(\theta^{*})(\gamma-\gamma^{0}_{s})+(\gamma-\gamma_{0}^{s})^{\top}\nabla^{2}\mathbb{M}^{s,\gamma\psi}(\theta^{*})(\psi-\psi^{0}_{s})$
$\displaystyle\qquad\qquad\qquad\qquad+\frac{1}{2}(\psi-\psi_{0}^{s})^{\top}\nabla^{2}\mathbb{M}^{s,\psi\psi}(\theta^{*})(\psi-\psi^{0}_{s})$
$\displaystyle:=\frac{1}{2}\left(T_{1}+2T_{2}+T_{3}\right)$ (B.16)

Note that we can write:

$\displaystyle T_{1}$
$\displaystyle=(\gamma-\gamma_{0}^{s})^{\top}\nabla^{2}\mathbb{M}^{s,\gamma}(\tilde{\theta})(\gamma-\gamma^{0}_{s})$
$\displaystyle=(\gamma-\gamma_{0}^{s})^{\top}\nabla^{2}\mathbb{M}^{s,\gamma}(\theta_{0})(\gamma-\gamma^{0}_{s})+(\gamma-\gamma_{0}^{s})^{\top}\left[\nabla^{2}\mathbb{M}^{s,\gamma}(\tilde{\theta})-\nabla^{2}\mathbb{M}^{s,\gamma}(\theta_{0})\right](\gamma-\gamma^{0}_{s})$

The operator norm of the difference of two hessians can be bounded as:

$\left\|\nabla^{2}\mathbb{M}^{s,\gamma}(\theta^{*})-\nabla^{2}\mathbb{M}^{s,\gamma}(\theta_{0})\right\|_{op}=O(\sigma_{n})\,.$

for any $\theta^{*}$ in a neighborhood of $\theta_{0}^{s}$ with
$\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}$. To prove this note that for
any $\theta$:

$\nabla^{2}\mathbb{M}^{s,\gamma}(\theta^{*})-\nabla^{2}\mathbb{M}^{s,\gamma}(\theta_{0})=2\begin{pmatrix}0&A\\\
A&A\end{pmatrix}=\begin{pmatrix}0&1\\\ 1&1\end{pmatrix}\otimes A$

where:

$A=\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi}{\sigma_{n}}\right)\right]-\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}\right)\right]$

Therefore it is enough to show $\|A\|_{op}=O(\sigma_{n})$. Towards that
direction:

$\displaystyle A$
$\displaystyle=\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi}{\sigma_{n}}\right)\right]-\mathbb{E}\left[g(Q)K\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}\right)\right]$
$\displaystyle=\sigma_{n}\int\int
g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\left(K(t+\tilde{q}^{\top}\eta)-K(t)\right)f_{0}(\sigma_{n}t\mid\tilde{q})\
f(\tilde{q})\ dt\ d\tilde{q}$ $\displaystyle=\sigma_{n}\left[\int\int
g(-\tilde{q}^{\top}\tilde{\psi}_{0})\left(K(t+\tilde{q}^{\top}\eta)-K(t)\right)f_{0}(0\mid\tilde{q})\
f(\tilde{q})\ dt\ d\tilde{q}+R\right]$ $\displaystyle=\sigma_{n}\left[\int\int
g(-\tilde{q}^{\top}\tilde{\psi}_{0})f_{0}(0\mid\tilde{q})\int_{t}^{t+\tilde{q}^{\top}\eta}K^{\prime}(s)\
ds\ f(\tilde{q})\ dt\ d\tilde{q}+R\right]$ $\displaystyle=\sigma_{n}\left[\int
g(-\tilde{q}^{\top}\tilde{\psi}_{0})f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime}(s)\int_{s-\tilde{q}^{\top}\eta}^{s}\
dt\ ds\ f(\tilde{q})\ d\tilde{q}+R\right]$ $\displaystyle=\sigma_{n}\left[\int
g(-\tilde{q}^{\top}\tilde{\psi}_{0})f_{0}(0\mid\tilde{q})\tilde{q}^{\top}\eta\
f(\tilde{q})\ d\tilde{q}+R\right]$
$\displaystyle=\sigma_{n}\left[\mathbb{E}\left[g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})f_{0}(0\mid\tilde{Q})\tilde{Q}^{\top}\eta\right]+R\right]$

using the fact that
$\left\|\mathbb{E}\left[g(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q})f_{0}(0\mid\tilde{Q})\tilde{Q}^{\top}\eta\right]\right\|_{op}=O(1)$
and $\|R\|_{op}=O(\sigma_{n})$ we conclude the claim. From the above claim we
conclude:

$T_{1}=(\gamma-\gamma_{0}^{s})^{\top}\nabla^{2}\mathbb{M}^{s,\gamma}(\theta^{*})(\gamma-\gamma^{s}_{0})\geq\|\gamma-\gamma^{s}_{0}\|^{2}(1-O(\sigma_{n}))\geq\frac{1}{2}\|\gamma-\gamma_{0}^{s}\|^{2}$
(B.17)

for all large $n$.

We next deal with the cross term $T_{2}$ in equation (B.16). Towards that end
first note that:

$\displaystyle\frac{1}{\sigma_{n}}\mathbb{E}\left((g(Q)\delta)\tilde{Q}^{\top}K^{\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\eta^{*}\right)\right)$
$\displaystyle=\int_{\mathbb{R}^{(p-1)}}\left[\int_{-\infty}^{\infty}\left(g\left(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)\delta\right)K^{\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)f_{0}(\sigma_{n}t\mid\tilde{q})\
dt\right]\tilde{q}^{\top}\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=\int_{\mathbb{R}^{(p-1)}}\left[\int_{-\infty}^{\infty}\left(g\left(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)\delta\right)K^{\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)f_{0}(0\mid\tilde{q})\
dt\right]\tilde{q}^{\top}\ f(\tilde{q})\ d\tilde{q}+R_{1}$
$\displaystyle=\mathbb{E}\left[\left(g\left(-\tilde{Q}^{\top}\tilde{\psi}_{0},\tilde{Q}\right)\delta\right)\tilde{Q}^{\top}f_{0}(0\mid\tilde{Q})\right]+R_{1}$

where the remainder term $R_{1}$ can be further decomposed
$R_{1}=R_{11}+R_{12}+R_{13}$ with:

$\displaystyle\left\|R_{11}\right\|$
$\displaystyle=\left\|\int_{\mathbb{R}^{(p-1)}}\left[\int_{-\infty}^{\infty}\left(g\left(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)\delta\right)K^{\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)(f_{0}(\sigma_{n}t\mid\tilde{q})-f_{0}(0\mid\tilde{q}))\
dt\right]\tilde{q}^{\top}\ f(\tilde{q})\ d\tilde{q}\right\|$
$\displaystyle\leq\left\|\int_{\mathbb{R}^{(p-1)}}\left[\int_{-\infty}^{\infty}\left\|g\left(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)\right\|_{op}\|\delta\|\left|K^{\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\right|\left|f_{0}(\sigma_{n}t\mid\tilde{q})-f_{0}(0\mid\tilde{q})\right|\
dt\right]\left|\tilde{q}\right|\ f(\tilde{q})\ d\tilde{q}\right\|$
$\displaystyle\leq\sigma_{n}\dot{f}^{+}c_{+}\|\delta\|\int_{\mathbb{R}^{(p-1)}}\|\tilde{q}\|\int_{-\infty}^{\infty}|t|\left|K^{\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\sigma_{n}\dot{f}^{+}c_{+}\|\delta\|\int_{\mathbb{R}^{(p-1)}}\|\tilde{q}\|\int_{-\infty}^{\infty}|t-\tilde{q}^{\top}\eta^{*}|\left|K^{\prime}\left(t\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\sigma_{n}\dot{f}^{+}c_{+}\|\delta\|\left[\int_{\mathbb{R}^{(p-1)}}\|\tilde{q}\|\int_{-\infty}^{\infty}|t|\left|K^{\prime}\left(t\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}\right.$
$\displaystyle\qquad\qquad\qquad\left.+\int_{\mathbb{R}^{(p-1)}}\|\tilde{q}\|^{2}\|\eta^{*}\|\int_{-\infty}^{\infty}|K^{\prime}(t)|\
dt\ f(\tilde{q})\ d\tilde{q}\right]$
$\displaystyle\leq\sigma_{n}\dot{f}^{+}c_{+}\|\delta\|\left[\int_{\mathbb{R}^{(p-1)}}\|\tilde{q}\|\int_{-\infty}^{\infty}|t|\left|K^{\prime}\left(t\right)\right|\
dt\ f(\tilde{q})\
d\tilde{q}+\mathcal{K}\int_{\mathbb{R}^{(p-1)}}\|\tilde{q}\|^{2}\int_{-\infty}^{\infty}|K^{\prime}(t)|\
dt\ f(\tilde{q})\ d\tilde{q}\right]$ $\displaystyle\lesssim\sigma_{n}\,.$

where the last bound follows from our assumptions using the fact that:

$\displaystyle\|R_{12}\|$
$\displaystyle=\left\|\int_{\mathbb{R}^{(p-1)}}\left[\int_{-\infty}^{\infty}\left(\left(g\left(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)-g\left(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)\right)\delta\right)K^{\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)f_{0}(0\mid\tilde{q})\
dt\right]\tilde{q}^{\top}\ f(\tilde{q})\ d\tilde{q}\right\|$
$\displaystyle\leq\int\|\tilde{q}\|\|\delta\|f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}\left\|g\left(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)-g\left(-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q}\right)\right\|_{op}\left|K^{\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle\leq\dot{c}_{+}\sigma_{n}\int\|\tilde{q}\|\|\delta\|f_{0}(0\mid\tilde{q})\dot{\int}_{-\infty}^{\infty}|t|\left|K^{\prime}\left(t+\tilde{q}^{\top}\tilde{\eta}\right)\right|\
dt\ f(\tilde{q})\ d\tilde{q}\hskip 14.45377pt[\text{Assumption
}\ref{eq:assm}]$ $\displaystyle\lesssim\sigma_{n}\,.$

The other remainder term $R_{13}$ is the higher order term and can be shown to
be $O(\sigma_{n}^{2})$ using same techniques. This implies for all large $n$:

$\displaystyle\left\|\nabla_{\beta\psi}\mathbb{M}^{s}(\theta)\right\|_{op}$
$\displaystyle=O(1)\,.$

and similar calculation yields
$\left\|\nabla_{\delta\psi}\mathbb{M}^{s}(\theta)\right\|_{op}=O(1)$. Using
this we have:

$\displaystyle T_{2}$
$\displaystyle=(\gamma-\gamma_{0}^{s})^{\top}\nabla^{2}\mathbb{M}^{s,\gamma\psi}(\tilde{\theta})(\psi-\psi^{0}_{s})$
$\displaystyle=(\beta-\beta_{0}^{s})^{\top}\nabla_{\beta\psi}^{2}\mathbb{M}^{s}(\tilde{\theta})(\psi-\psi^{0}_{s})+(\delta-\delta_{0}^{s})^{\top}\nabla_{\delta\psi}^{2}\mathbb{M}^{s}(\tilde{\theta})(\psi-\psi^{0}_{s})$
$\displaystyle\geq-C\left[\|\beta-\beta_{0}^{s}\|+\|\delta-\delta_{0}^{s}\|\right]\|\psi-\psi^{0}_{s}\|$
$\displaystyle\geq-C\sqrt{\sigma_{n}}\left[\|\beta-\beta_{0}^{s}\|+\|\delta-\delta_{0}^{s}\|\right]\frac{\|\psi-\psi^{0}_{s}\|}{\sqrt{\sigma_{n}}}$
$\displaystyle\gtrsim-\sqrt{\sigma_{n}}\left(\|\beta-\beta_{0}^{s}\|^{2}+\|\delta-\delta_{0}^{s}\|^{2}+\frac{\|\psi-\psi^{0}_{s}\|^{2}}{\sigma_{n}}\right)$
(B.18)

Now for $T_{3}$ note that:

$\displaystyle\sigma_{n}\nabla_{\psi\psi}\mathbb{M}^{s}_{n}(\theta)$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left\\{\left[-2\left(Y_{i}-X_{i}^{\top}\beta\right)X_{i}^{\top}\delta+(X_{i}^{\top}\delta)^{2}\right]\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left\\{\left[-2\left(Y_{i}-X_{i}^{\top}\beta\right)X_{i}^{\top}\delta\right]\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle\qquad\qquad\qquad+\frac{1}{\sigma_{n}}\mathbb{E}\left\\{(\delta^{\top}g(Q)\delta)\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle=\frac{1}{\sigma_{n}}\mathbb{E}\left\\{\left[-2X_{i}^{\top}\left(\beta_{0}-\beta\right)X_{i}^{\top}\delta-2(X_{i}^{\top}\delta_{0})(X_{i}^{\top}\delta)\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}\right]\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle\qquad\qquad\qquad+\frac{1}{\sigma_{n}}\mathbb{E}\left\\{(\delta^{\top}g(Q)\delta)\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle=\frac{-2}{\sigma_{n}}\mathbb{E}\left\\{((\beta_{0}-\beta)^{\top}g(Q)\delta)\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle\qquad\qquad\qquad+\frac{-2}{\sigma_{n}}\mathbb{E}\left\\{(\delta_{0}^{\top}g(Q)\delta)\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}\right\\}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad+\frac{1}{\sigma_{n}}\mathbb{E}\left\\{(\delta^{\top}g(Q)\delta)\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle=\underbrace{\frac{-2}{\sigma_{n}}\mathbb{E}\left\\{((\beta_{0}-\beta)^{\top}g(Q)\delta)\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}}_{M_{1}}$
$\displaystyle\qquad\qquad\qquad+\underbrace{\frac{-2}{\sigma_{n}}\mathbb{E}\left\\{(\delta_{0}^{\top}g(Q)\delta_{0})\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}\right\\}}_{M_{2}}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad+\underbrace{\frac{-2}{\sigma_{n}}\mathbb{E}\left\\{(\delta_{0}^{\top}g(Q)(\delta-\delta_{0}))\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}\right\\}}_{M_{3}}$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+\underbrace{\frac{1}{\sigma_{n}}\mathbb{E}\left\\{(\delta^{\top}g(Q)\delta)\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}}_{M_{4}}$
$\displaystyle:=M_{1}+M_{2}+M_{3}+M_{4}$

We next show that $M_{1}$ and $M_{4}$ are $O(\sigma_{n})$. Towards that end
note that for any two vectors $v_{1},v_{2}$:

$\displaystyle\frac{1}{\sigma_{n}}\mathbb{E}\left\\{(v_{1}^{\top}g(Q)v_{2})\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\right\\}$
$\displaystyle=\int\tilde{q}\tilde{q}^{\top}\int_{-\infty}^{\infty}(v_{1}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\eta},\tilde{q})v_{2})K^{\prime\prime}(t+\tilde{q}^{\top}\tilde{\eta})f(\sigma_{n}t\mid\tilde{q})\
dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=\int\tilde{q}\tilde{q}^{\top}(v_{1}^{\top}g(-\tilde{q}^{\top}\tilde{\eta},\tilde{q})v_{2})f(0\mid\tilde{q})f(\tilde{q})\
d\tilde{q}\cancelto{0}{\int_{-\infty}^{\infty}K^{\prime\prime}(t)\ dt}+R=R$

as $\int K^{\prime\prime}(t)\ dt=0$ follows from our choice of kernel
$K(x)=\Phi(x)$. Similar calculation as in the case of analyzing the remainder
of $T_{2}$ yields $\|R\|_{op}=O(\sigma_{n})$. This immediately implies
$\|M_{1}\|_{op}=O(\sigma_{n})$ and $\|M_{4}\|_{op}=O(\sigma_{n})$. Now for
$M_{2}$:

$\displaystyle M_{2}$
$\displaystyle=\frac{-2}{\sigma_{n}}\mathbb{E}\left\\{(\delta_{0}^{\top}g(Q)\delta_{0})\tilde{Q}_{i}\tilde{Q}_{i}^{\top}K^{\prime\prime}\left(\frac{Q_{i}^{\top}\psi_{0}}{\sigma_{n}}+\tilde{Q}^{\top}\tilde{\eta}\right)\mathds{1}_{Q_{i}^{\top}\psi_{0}>0}\right\\}$
$\displaystyle=-2\int\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}f_{0}(\sigma_{n}t\mid\tilde{q})\
dtf(\tilde{q})\ d\tilde{q}$
$\displaystyle=-2\int(\delta_{0}^{\top}g(-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}+R$
$\displaystyle=2\mathbb{E}\left[(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{Q}\tilde{Q}^{\top}f_{0}(0\mid\tilde{Q})K^{\prime}(\tilde{Q}^{\top}\eta^{*})\right]+R$
(B.19)

where the remainder term R is $O_{p}(\sigma_{n})$ can be established as
follows:

$\displaystyle R$
$\displaystyle=-2\left[\int\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}f_{0}(\sigma_{n}t\mid\tilde{q})\
dtf(\tilde{q})\ d\tilde{q}\right.$
$\displaystyle\qquad\qquad-\left.\int(\delta_{0}^{\top}g(-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}\right]$
$\displaystyle=-2\left\\{\left[\int\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}f_{0}(\sigma_{n}t\mid\tilde{q})\
dtf(\tilde{q})\ d\tilde{q}\right.\right.$
$\displaystyle\qquad\qquad-\left.\left.\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}\right]\right.$
$\displaystyle\left.+\left[\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}\right.\right.$
$\displaystyle\qquad\qquad\left.\left.-\int(\delta_{0}^{\top}g(-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}\right]\right\\}$ $\displaystyle=-2(R_{1}+R_{2})\,.$

For $R_{1}$:

$\displaystyle\left\|R_{1}\right\|_{op}$
$\displaystyle=\left\|\left[\int\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}f_{0}(\sigma_{n}t\mid\tilde{q})\
dtf(\tilde{q})\ d\tilde{q}\right.\right.\,.$
$\displaystyle\qquad\qquad-\left.\left.\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}\right]\right\|_{op}$ $\displaystyle\leq
c_{+}\int\int\|\tilde{q}\|^{2}|K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)||f_{0}(\sigma_{n}t\mid\tilde{q})-f_{0}(0\mid\tilde{q})|\
dt\ f(\tilde{q})\ d\tilde{q}$ $\displaystyle\leq
c_{+}F_{+}\sigma_{n}\int\|\tilde{q}\|^{2}\int|t||K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)|\
dt\ f(\tilde{q})\ d\tilde{q}$
$\displaystyle=c_{+}F_{+}\sigma_{n}\int\|\tilde{q}\|^{2}\int|t-\tilde{q}^{\top}\eta^{*}||K^{\prime\prime}\left(t\right)|\
dt\ f(\tilde{q})\ d\tilde{q}$ $\displaystyle\leq
c_{+}F_{+}\sigma_{n}\left[\mathbb{E}[\|\tilde{Q}\|^{2}]\int|t||K^{\prime\prime}(t)|\
dt+\|\eta^{*}\|\mathbb{E}[\|\tilde{Q}\|^{3}]\int|K^{\prime\prime}(t)|\
dt\right]=O(\sigma_{n})\,.$

and similarly for $R_{2}$:

$\displaystyle\|R_{2}\|_{op}$
$\displaystyle=\left\|\left[\int(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}\right.\right.$
$\displaystyle\qquad\qquad\left.\left.-\int(\delta_{0}^{\top}g(-\tilde{q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{q}\tilde{q}^{\top}f_{0}(0\mid\tilde{q})\int_{-\infty}^{\infty}K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)\mathds{1}_{t>0}\
dtf(\tilde{q})\ d\tilde{q}\right]\right\|_{op}$ $\displaystyle\leq
F_{+}\|\delta_{0}\|^{2}\int\left\|g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0})-g(-\tilde{q}^{\top}\tilde{\psi}_{0})\right\|_{op}\|\tilde{q}\|^{2}\int_{-\infty}^{\infty}|K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)|\
dt$ $\displaystyle\leq
G_{+}F_{+}\sigma_{n}\int\|\tilde{q}\|^{2}\int_{-\infty}^{\infty}|t||K^{\prime\prime}\left(t+\tilde{q}^{\top}\eta^{*}\right)|\
dt=O(\sigma_{n})\,.$

Therefore from (B.19) we conclude:

$M_{2}=2\mathbb{E}\left[(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0})\delta_{0})\tilde{Q}\tilde{Q}^{\top}f_{0}(0\mid\tilde{Q})K^{\prime}(\tilde{Q}^{\top}\eta^{*})\right]+O(\sigma_{n})\,.$
(B.20)

Similar calculation for $M_{3}$ yields:

$M_{3}=2\mathbb{E}\left[(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0})(\delta-\delta_{0}))\tilde{Q}\tilde{Q}^{\top}f_{0}(0\mid\tilde{Q})K^{\prime}(\tilde{Q}^{\top}\eta^{*})\right]+O(\sigma_{n})\,.$

i.e.

$\|M_{3}\|_{op}\leq
c_{+}\mathbb{E}\left[\|\tilde{Q}\|^{2}f_{0}(0\mid\tilde{Q})K^{\prime}(\tilde{Q}^{\top}\eta^{*})\right]\|\delta_{0}\|\|\delta-\delta_{0}\|\,.$
(B.21)

Now we claim that for any $\mathcal{K}<\infty$, $\lambda_{\min}(M_{2})>0$ for
all $\|\eta^{*}\|\leq\mathcal{K}$. Towards that end, define a function
$\lambda:B_{\mathbb{R}^{2d}}(1)\times
B_{\mathbb{R}^{2d}}(\mathcal{K})\to\mathbb{R}_{+}$ as:

$\lambda:(v,\eta)\mapsto
2\mathbb{E}\left[(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0})\delta_{0})\left(v^{\top}\tilde{Q}\right)^{2}f_{0}(0\mid\tilde{Q})K^{\prime}(\tilde{Q}^{\top}\eta)\right]$

Clearly $\lambda\geq 0$ and is continuous on a compact set. Hence its infimum
must be attained. Suppose the infimum is $0$, i.e. there exists
$(v^{*},\eta^{*})$ such that:

$\mathbb{E}\left[(\delta_{0}^{\top}g(-\tilde{Q}^{\top}\tilde{\psi}_{0})\delta_{0})\left(v^{*^{\top}}\tilde{Q}\right)^{2}f_{0}(0\mid\tilde{Q})K^{\prime}(\tilde{Q}^{\top}\eta^{*})\right]=0\,.$

as $\lambda_{\min}(g(\dot{)})\geq c_{+}$, we must have
$\left(v^{*^{\top}}\tilde{Q}\right)^{2}f_{0}(0\mid\tilde{Q})K^{\prime}(\tilde{Q}^{\top}\eta^{*})=0$
almost surely. But from our assumption,
$\left(v^{*^{\top}}\tilde{Q}\right)^{2}>0$ and
$K^{\prime}(\tilde{Q}^{\top}\eta^{*})>0$ almost surely, which implies
$f_{0}(0\mid\tilde{q})=0$ almost surely, which is a contradiction. Hence there
exists $\lambda_{-}$ such that:

$\lambda_{\min}(M_{2})\geq\lambda_{-}>0\ \ \forall\ \
\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\,.$

Hence we have:

$\lambda_{\min}\left(\sigma_{n}\nabla_{\psi\psi}\mathbb{M}^{2}(\theta)\right)\geq\frac{\lambda_{-}}{2}(1-O(\sigma_{n}))$

for all theta such that $d_{*}(\theta,\theta_{0}^{s})\leq{\epsilon}\,.$

$\displaystyle\frac{1}{\sigma_{n}}(\psi-\psi_{0}^{s})^{\top}\sigma_{n}\nabla^{\psi\psi}\mathbb{M}^{s}(\tilde{\theta})(\psi-\psi^{0})\gtrsim\frac{\|\psi-\psi^{s}_{0}\|^{2}}{\sigma_{n}}\left(1-O(\sigma_{n})\right)$
(B.22)

From equation (B.17), (B.18) and (B.22) we have:

$\displaystyle\frac{1}{2}(\theta_{0}-\theta^{0}_{s})^{\top}\nabla^{2}\mathbb{M}^{s}(\theta^{*})(\theta_{0}-\theta^{0}_{s})$
$\displaystyle\qquad\qquad\gtrsim\left[\|\beta-\beta^{s}_{0}\|^{2}+\|\gamma-\gamma^{s}_{0}\|^{2}+\frac{\|\psi-\psi^{s}_{0}\|^{2}}{\sigma_{n}}\right]\mathds{1}_{\|\psi-\psi_{0}^{s}\|\leq\mathcal{K}\sigma_{n}}\,.$

This, along with equation (B.15) concludes the proof. ∎

###  B.5 Proof of Lemma C.8

We start by proving that analogues of Lemma 2 of [21]: we show that:

$\displaystyle\lim_{n\to\infty}\mathbb{E}\left[\sqrt{n\sigma_{n}}\nabla\mathbb{M}_{n}^{s,\psi}(\theta_{0})\right]$
$\displaystyle=0$ $\displaystyle\lim_{n\to\infty}{\sf
var}\left[\sqrt{n\sigma_{n}}\nabla\mathbb{M}_{n}^{s,\psi}(\theta_{0})\right]$
$\displaystyle=V^{\psi}$

for some matrix $V^{\psi}$ which will be specified later in the proof. To
prove the limit of the expectation:

$\displaystyle\mathbb{E}\left[\sqrt{n\sigma_{n}}\nabla\mathbb{M}_{n}^{s,\psi}(\theta_{0})\right]$
$\displaystyle=\sqrt{\frac{n}{\sigma_{n}}}\mathbb{E}\left[\left\\{(Y-X^{\top}(\beta_{0}+\delta_{0}))^{2}-(Y-X^{\top}\beta_{0})^{2}\right\\}\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi_{0}}{\sigma_{n}}\right)\right]$
$\displaystyle=\sqrt{\frac{n}{\sigma_{n}}}\mathbb{E}\left[\left(\delta_{0}^{\top}g(Q)\delta_{0}\right)\left(1-2\mathds{1}_{Q^{\top}\psi_{0}>0}\right)\tilde{Q}K^{\prime}\left(\frac{Q^{\top}\psi_{0}}{\sigma_{n}}\right)\right]$
$\displaystyle=\sqrt{\frac{n}{\sigma_{n}}}\times\sigma_{n}\int\int\left(\delta_{0}^{\top}g(\sigma_{n}t-\tilde{q}^{\top}\tilde{\psi}_{0},\tilde{q})\delta_{0}\right)\left(1-2\mathds{1}_{t>0}\right)\tilde{q}K^{\prime}\left(t\right)\