# Adapting a Language Model While Preserving its General Knowledge

Zixuan Ke1, Yijia Shao2, Haowei Lin2, Hu Xu3, Lei Shu1 and Bing Liu1
1Department of Computer Science, University of Illinois at Chicago
2Wangxuan Institute of Computer Technology, Peking University
3Meta AI
<EMAIL_ADDRESS>
2{shaoyj<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Now at Google Research<EMAIL_ADDRESS>
###### Abstract

Domain-adaptive pre-training (or DA-training for short), also known as post-
training, aims to train a pre-trained general-purpose language model (LM)
using an unlabeled corpus of a particular domain to adapt the LM so that end-
tasks in the domain can give improved performances. However, existing DA-
training methods are in some sense blind as they do not explicitly identify
what knowledge in the LM should be preserved and what should be changed by the
domain corpus. This paper shows that the existing methods are sub-optimal and
proposes a novel method to perform a more informed adaptation of the knowledge
in the LM by (1) soft-masking the attention heads based on their importance to
best preserve the general knowledge in the LM and (2) contrasting the
representations of the general and the full (both general and domain
knowledge) to learn an integrated representation with both general and domain-
specific knowledge. Experimental results will demonstrate the effectiveness of
the proposed approach.111https://github.com/UIC-Liu-Lab/DGA

##  1 Introduction

Pre-trained general-purpose language models (LMs) like BERT Devlin et al.
(2019), RoBERTa Liu et al. (2019), and GPT-3 Brown et al. (2020) have become a
standard component in almost all NLP applications. Researchers have also found
that domain-adaptive pre-training (or DA-training for short) using an
unlabeled corpus in a specific domain to adapt an LM can further improve the
end-task performance in the domain Gururangan et al. (2020); Xu et al. (2019a,
b); Sun et al. (2019); Alsentzer et al. (2019). Note that domain-adaptive pre-
training is also called post-training Xu et al. (2019a).

Existing DA-training methods simply apply the same pre-training objective,
i.e., the mask language model (MLM) loss, to further train an LM using a
domain corpus. These methods are sub-optimal because they do not explicitly
identify what should be preserved and what should be updated in the LM by the
domain corpus.

This paper argues that a good DA-training method has two needs. On the one
hand, the general language knowledge learned in the LM should be preserved as
much as possible because the target domain data is typically not large enough
to be sufficient to learn the general knowledge well. For example, some words
and their contexts may appear infrequently in a particular domain. The
knowledge about them cannot be learned accurately based on the domain data
alone. When these words and contexts appear in an end-task, the system will
have difficulties. Thus, we need to rely on the knowledge about them in the
LM. Since existing DA-training updates the LM with little guidance, such
useful general knowledge may be corrupted. On the other hand, due to polysemy
(same word with different meanings in different domains) and the fact that
different domains also have their special word usages and contexts, the LM
should be specialized or adapted to the target domain. A good DA-training
should balance these two needs to adapt the LM to the target domain with
minimal corruption to the good general knowledge in the LM.

This paper proposes a novel technique to enable a more informed adaptation to
(1) preserve the general knowledge in the LM as much as possible, and (2)
update the LM to incorporate the domain-specific knowledge of the target
domain as needed. The focus of the existing DA-training research has been on
(2). As we argued above, (1) is also important as focusing only on (2) may
destroy some useful general knowledge and produce sub-optimal results for end-
tasks. To achieve (1), the system should constrain the gradient update of each
attention head222We will see in Sec. 4 that constraining the neurons in other
layers is unnecessary. based on its importance to the general knowledge so
that the general knowledge in LM can be preserved as much as possible. With
(1), (2) will be able to change the part of the general knowledge that needs
to be updated to adapt the LM to suit the target domain.333This is very
different from continual learning (CL) Chen and Liu (2018) as CL needs to
preserve the past knowledge to deal with catastrophic forgetting McCloskey and
Cohen (1989). DA-training can and should change/adapt the general knowledge in
the original LM to suit the target domain.

In this paper, we propose a novel model called DGA (DA-training - General
knowledge preservation and LM Adaptation) for the purpose. The key idea of the
proposed method is to preserve the general language knowledge in the LM while
adapting the LM to a specific domain. However, it is not obvious how this can
be done, i.e., how to find those parameters that are important for the general
knowledge and how to protect them. This paper proposes a novel proxy-based
method to achieve the objectives. It works as follows. DGA first estimates the
importance of each attention head in the LM via the newly proposed proxy KL-
divergence loss (Sec. 3.1). This importance score reflects how important each
attention head is to the general knowledge. Based on the importance scores, it
performs two key functions: The first function uses the scores to soft-mask
(rather than binary-mask or completely block) the gradient update to prevent
important general knowledge in LM from being unnecessarily corrupted. This is
related to pruning of unimportant attention heads Michel et al. (2019).
However, pruning is not directly applicable to DA-training as we will show in
Sec. 2. The proposed soft-masking constrains only the backward gradient flow
in training. It is not necessary to soft-mask the forward pass in either
training or inference. This is important because using the knowledge in the
full network encourages maximal integration of pre-trained general knowledge
and the target domain-specific knowledge. The second function contrasts the
representation for the general knowledge in the LM and the full (including
both the general and the domain-specific) knowledge to learn an integrated
representation (Sec. 3.2).444Contrasting the general and only the domain-
specific knowledge gives poorer results (see Sec. 4.3) as it causes the two
types of knowledge to split rather than to integrate.

In summary, this paper makes two key contributions.

(1). It proposes the idea of informed adaptation to integrate the specialized
knowledge in the target domain into the LM with minimal corruption to the
useful general knowledge in the original LM.

(2). It proposes a new model DGA with two novel functions to enable better DA-
training. DGA estimates the attention head importance to protect the important
general knowledge in the LM and integrates the specialized knowledge in the
target domain into the LM through contrasting the general and the full
knowledge.

To the best of our knowledge, none of these has been reported in the
literature before.

Extensive experiments have been conducted in 6 different domains and on 10
baselines to demonstrate the effectiveness of the proposed DGA.

##  2 Related Work

Domain-adaptive pre-training (DA-training). Researchers have applied DA-
training to many domains, e.g., reviews Xu et al. (2019a, b), biomedical text
Lee et al. (2020), news and papers Gururangan et al. (2020), and social media
Chakrabarty et al. (2019). However, they all use the same mask language model
(MLM) loss. We argue that it is sub-optimal and it is also important to
preserve the general knowledge in the LM as much as possible and integrate it
with the target domain knowledge.

Network pruning as importance computation. It is known that many parameters in
a neural network are redundant and can be pruned Li et al. (2021); Lai et al.
(2021). This has also been shown for pre-trained Transformer Chen et al.
(2020a); Lin et al. (2020); Gao et al. (2021b); Michel et al. (2019); Voita et
al. (2019). A popular pruning method is to discard the parameters with small
absolute values Han et al. (2015); Guo et al. (2016). Other methods prune the
network at a higher level. In a Transformer-based model, these include pruning
the attention head Michel et al. (2019); Voita et al. (2019); McCarley et al.
(2019) and pruning sub-layers in a standard Transformer layer Fan et al.
(2020); Sajjad et al. (2020). However, the above methods are not directly
applicable to us as we need to compute the head importance for the LM using
unlabeled domain data, while the above approaches are all for supervised end-
tasks. We propose to use a proxy KL-divergence loss for our purpose. Note that
it is possible to prune other sub-layers in the Transformer. However, as shown
in Sec. 4.3, estimating the importance for other layers does not improve the
performance.

Contrastive learning. Contrastive learning Chen et al. (2020b); He et al.
(2020) can learn good representations by maximizing the similarity of positive
pairs and minimizes that of negative pairs:

$\mathcal{L}_{\text{contrast}}=-\log\frac{e^{(\text{sim}(q_{i},q^{+}_{i})/\tau)}}{\sum_{j=1}^{N}e^{(\text{sim}(q_{i},q^{+}_{j})/\tau)}},$
(1)

where $N$ is the batch size, $\tau$ is a temperature parameter,
$\text{sim}(\cdot)$ is a similarity metric, and $q_{i}$ and $q^{+}_{i}$ are
representations for positive pairs $x_{i}$ and $x_{i}^{+}$ (typically,
$x_{i}^{+}$ is an augmented sample of $x_{i}$, e.g., generated via cropping,
deletion or synonym replacement Gao et al. (2021a)). In the unsupervised
contrastive loss, the negative samples are the other samples in the batch,
indicated in the denominator.

We mainly use contrasive loss to contrast the representations of the important
general knowledge in the original LM and the full knowledge (both the general
and domain-specific knowledge) to achieve a good integration of the general
knowledge and the domain specific knowledge.

##  3 Proposed DGA System

As discussed earlier, DGA goes beyond the MLM loss to perform two more
functions: (1) preserving the important general knowledge in the LM by soft-
masking the attention heads based on their importance. This helps avoid
potential corruption of the general knowledge in the LM in DA-training (Sec.
3.1). However, the challenge is how to identify the general knowledge in the
LM and how to protect it. We will propose a method to do that. (2) encouraging
the model to learn integrated representations of the target domain and the
general knowledge in the LM (Sec. 3.2). It is also not obvious how this can be
done. We propose a contrastive learning based method to do it. Figure 1 gives
an overview of DGA.

Figure 1:  Illustration of DGA. (A) shows the importance computation. This is
done by adding a gate vector $\bm{g}_{l}$ multiplying with the multi-head
attention (Eq. 5) and averaging its training gradients (Eq. 6). (B) shows DGA
training. In backward pass, attention heads are soft-masked based on their
importance $\bm{I}$ (Eqs. 9 and 10) to try to preserve the general knowledge
in the LM as much as possible. In forward pass, the added gate vector is
removed except for feature learning in the contrastive loss. The contrastive
loss is computed by contrasting the general knowledge with importance
($\bm{o}^{\text{gen}}$ in Eq. 12) applied and the full knowledge without
applying the importance ($\bm{o}^{\text{full}}$ in Eq. 14). The final
objective of DGA consists of MLM loss and contrastive loss. Note that we omit
the details of other parts of Transformer and only focus on the multi-head
attention mechanism.

###  3.1 Preserving General Knowledge by Soft-Masking Attention Heads

Multi-head attention. Multi-head attention is arguably the most important
component in the Transformer model Vaswani et al. (2017). We omit details of
other parts and refer the reader to the original paper. Formally, let
$\bm{x}=x^{(1)},...,x^{(T)}$ be a sequence of $T$ real vectors where
$x^{(t)}\in\mathbb{R}^{d}$ and let $q\in\mathbb{R}^{d}$ be a query vector. The
attention mechanism is defined as

$\text{att}(\bm{x},q)=W_{o}\sum_{t=1}^{T}\alpha^{(t)}(q)W_{v}x^{(t)},$ (2)

where

$\alpha^{(t)}(q)=\text{softmax}(\frac{q^{T}W_{q}^{T}W_{k}x^{(t)}}{\sqrt{d}}).$
(3)

The projection matrices $W_{o},W_{v},W_{q},W_{k}\in\mathbb{R}^{d\times d}$ are
learnable parameters. The query vector is from the same sequence as $\bm{x}$
in self-attention. A Transformer contains $L$ identical layers. For layer $l$,
$H_{l}$ different attention heads are applied in parallel to enable the
Transformer to be trained on more data. Simply put, multi-head attention
(mhatt) is the simultaneous application of multiple attention heads in a
single Transformer architecture. They are then applied in parallel to obtain
multi-head attention.555We follow the notation in Michel et al. (2019), where
the notation in Eq. 4 is equivalent to the “concatenation” formulation in
Vaswani et al. (2017).

$\text{mhatt}_{l}(\bm{x},q)=\sum_{h=1}^{H_{l}}\text{att}_{lh}(\bm{x},q),$ (4)

where $h$ indicates the $h^{th}$ attention head. Note that the input $\bm{x}$
is different in each layer since the input of a given layer is the output of
last layer. To ease the notation, we use the input $\bm{x}$ for all layers.

Head importance. Researchers have found that not all attention heads are
important Michel et al. (2019). We introduce a gate vector, $\bm{g}_{l}$,
where each cell is a gate variable, $g_{lh}$, to the attention head summation
for detecting the importance of attention heads. The resulting importance
scores are used to soft-mask the heads in DA-training.

$\text{gmhatt}_{l}(\bm{x},q)=\sum_{h=1}^{H_{l}}g_{lh},\otimes\text{att}_{lh}(\bm{x},q)$
(5)

where $\otimes$ is the element-wise multiplication. A gradient-based head
importance detection method is proposed in Michel et al. (2019). Given a
dataset $D=\\{(\bm{y}_{m},\bm{x}_{m})\\}_{m=1}^{M}$ of $M$ samples
($\bm{y}_{m}$ is the label of $\bm{x}_{m}$ as Michel et al. (2019) worked on
supervised learning), the importance of a head is estimated with a gradient-
based proxy score

$I_{lh}=\frac{1}{M}\sum_{m=1}^{M}|\nabla_{g_{lh}}|,$ (6)

where $\nabla_{g_{lh}}$ is the gradient of gate variable $g_{lh}$,

$\nabla_{g_{lh}}=\frac{\partial\mathcal{L}_{\text{impt}}(\bm{y}_{m},\bm{x}_{m})}{\partial
g_{lh}},$ (7)

where $\mathcal{L}_{\text{impt}}$ is a task-specific/domain-specific loss
function. The gradient can be used as the importance score because changing
${g_{lh}}$ is liable to have a large effect on the model if $I_{lh}$ has a
high value.

Although Eq. 6 offers a way to compute the importance of attention heads
w.r.t. a given loss $\mathcal{L}_{\text{impt}}$, we are unable to directly
apply it: If we use the domain data at hand and the MLM loss as
$\mathcal{L}_{\text{impt}}$, $\nabla_{g_{lh}}$ only indicates the importance
score for domain-specific knowledge. However, our goal is to estimate the
attention heads importance for the general knowledge in LM which requires the
data used in training the LM to compute the $\mathcal{L}_{\text{impt}}$. In
practice, such data is not accessible to users of the LM. Further, label is
needed in Eq. 6 but our domain corpus is unlabeled in DA-training. To address
these issues, we propose to compute a proxy KL-divergence loss for
$\mathcal{L}_{\text{impt}}$.

Proxy KL-divergence loss. We need a proxy for $\mathcal{L}_{\text{impt}}$ such
that its gradient ($\nabla_{g_{lh}}$) can be used to compute head importance
without using the LM’s original pre-training data. We propose to use model
robustness as the proxy, i.e., we try to detect heads that are important for
LM’s robustness. Its gradient, $\nabla_{g_{lh}}$, then indicates the
robustness and thus the importance to the LM model. Our rationale is as
follows: If an $I_{lh}$ (the average of $|\nabla_{g_{lh}}|$, see Eq. 6) has a
high value, it indicates that it is important to the LM’s robustness because
its change can cause the LM to change a great deal. It is thus an important
head to the LM. In contrast, if $I_{lh}$ has a small value, it is a less or
not important head to the LM.

To compute the robustness of the LM, we take a subset (a hyper-parameter) of
the target domain data $\\{\bm{x}^{\text{sub}}_{m}\\}$ (no label in DA-
training) and input $\bm{x}^{\text{sub}}_{m}$ twice to the LM and compute the
KL-divergence of the two resulting representations,

$\mathcal{L}_{\text{impt}}=\text{KL}(f_{1}(\bm{x}^{\text{sub}}_{m}),f_{2}(\bm{x}^{\text{sub}}_{m})),$
(8)

where $f_{1}$ and $f_{2}$ are the LM with different dropout masks. Note that
we don’t need to add any additional dropouts to implement $f$ because
independently sampled dropout masks are used as input in the Transformer. In
training a Transformer, there are dropout masks placed on fully-connected
layers and attention probabilities. Thus, simply feeding the same input to the
Transformer twice will get two representations with different dropout masks.
Since dropout is similar to adding noise, the difference between the two
representations can be regarded as the robustness of the Transformer model.
Figure 1 (A) shows how we compute the importance of each attention head using
the gradient of the gate vector $\bm{g}_{l}$.

Soft-masking attention heads in DA-training. Recall we want to preserve the
general knowledge in the LM during DA-training using head importance $I_{lh}$.
Given the attention head $\text{att}(\bm{x},q)$ and DA-training loss
$\mathcal{L}_{\text{DA-train}}$ (typically the MLM loss; we also propose an
additional loss in Sec. 3.2), we can “soft mask” its corresponding gradient
($\nabla_{\text{att}_{lh}}$ 666$\nabla_{\text{att}_{lh}}$ indicates the
gradient for attention head $\text{att}_{lh}(\bm{x},q)$, distinguished from
$\nabla_{g_{lh}}$ in Eq. 6 which is the gradient for the gate variable
$g_{lh}$) using the head importance value $I_{lh}$,

$\nabla^{\prime}_{\text{att}_{lh}}=(1-I^{\text{norm}}_{lh})\otimes\nabla_{\text{att}_{lh}},$
(9)

where $I^{\text{norm}}_{lh}$ is from $I_{lh}$ via normalization

$I^{\text{norm}}_{lh}=|\text{Tanh}(\text{Normalize}(I_{lh}))|.$ (10)

Normalize makes the $I_{lh}$ have a mean of 0 and standard deviation of 1.
Absolute value of Tanh ensures that $I_{lh}$ takes values in the interval
$[0,1]$.  Eq. 9 means to constrain the gradient of the corresponding head
$\text{att}_{lh}(\bm{x},q)$ by element-wise multiplying one minus the head
importance to the gradient. It is “soft-masking” because $I_{lh}$ is a real
number in $[0,1]$ (instead of binary {0, 1}), which gives the model the
flexibility to adjust the attention head. This is useful because although some
heads are important to the LM, they may conflict with the knowledge in the
target domain and thus need adjusting. Also note that the soft masks here
affect only the backward pass and are not used in forward pass (so that
forward pass can use the full network and encourage maximal integration of
pre-trained general and domain-specific knowledge) except for feature learning
using contrastive learning (see below). Figure 1 (B) shows that attention
heads are soft-masked during training.

###  3.2 Contrasting General and Full Knowledge

We now present how to integrate the general knowledge in the LM and the
domain-specific knowledge in the target domain by contrasting the general
knowledge and the full knowledge (both general and domain-specific). We first
introduce how we obtain such knowledge from the LM for the input $\bm{x}$, and
then discuss how we contrast them.

Obtaining the general knowledge for the input sequence $\bm{x}$ from the LM is
by extracting the representation of combining the attention heads and their
importance scores ($I^{\text{norm}}_{lh}$ in Eq. 10) in the forward pass. The
intuition is that since the importance scores show how important each
attention head is to the general knowledge, the resulting representation
reflects the main general knowledge used by $\bm{x}$. Formally, we plug
$I^{\text{norm}}_{lh}$ (soft-masks) as the gate variable $g_{lh}$ in Eq. 5,

$\text{gmhatt}^{\text{gen}}_{l}(\bm{x},q)=\sum_{h=1}^{H_{l}}I^{\text{norm}}_{lh}\otimes\text{att}_{lh}(\bm{x},q).$
(11)

Given the attention heads for general knowledge, we can plug it into the whole
Transformer to obtain the final general knowledge (taking the average of each
token’s output in the input sequence).

$\bm{o}^{\text{gen}}=\text{Transformer}(\text{gmhatt}^{\text{gen}}(\bm{x},q)).$
(12)

(See $\bm{o}^{\text{gen}}$ also in Figure 1 (B)).

Obtaining the full (both general and domain-specific) knowledge in $\bm{x}$ is
similar. The only difference is that we extract the representation of $\bm{x}$
without applying the importance (soft-masks) on attention heads in the forward
pass,

$\text{gmhatt}^{\text{full}}_{l}(\bm{x},q)=\sum_{h=1}^{H_{l}}\text{att}_{lh}(\bm{x},q).$
(13)

Similarly, we can plug it into the Transformer,

$\bm{o}^{\text{full}}=\text{Transformer}(\text{gmhatt}^{\text{full}}(\bm{x},q)).$
(14)

(See $\bm{o}^{\text{full}}$ also in Figure 1 (B)). Note that it is possible to
use $(1-I_{lh}^{\text{norm}})$ as the importance of domain-specific knowledge
and contrast it with the general knowledge. However, this produces poorer
results (see Table 3) as explained in footnote 4.

Contrasting general and full knowledge. It is known that contrastive learning
helps learn a good isotropic representation that is good for down-stream
tasks, with the help of positive and negative instances. We contrast the
general ($\bm{o}^{\text{gen}}$) and full ($\bm{o}^{\text{full}}$)
representations (as positive and negative instances) for the same input
$\bm{x}$ to make them different, which encourages the learning of domain-
specific knowledge in $\bm{o}^{\text{full}}$ that is not already in the
general knowledge and yet related to and integrated with the general knowledge
($\bm{o}^{\text{gen}}$) of the input.

We construct contrastive instances as follows: for an input $\bm{x}_{m}$,
three contrastive instances are produced. Anchor $\bm{o}_{m}$ and positive
instance $\bm{o}_{m}^{+}$ are both full knowledge from Eq. 14, obtained based
on two independently sampled dropout masks in the Transformer (recall that
this can be achieved by inputting $\bm{x}_{m}$ twice (see Sec. 3.1). We regard
$\bm{o}_{m}^{+}$ and $\bm{o}_{m}$ as positive instances because the dropout
noise has been shown to be good positive instances for improving alignment in
training sentence embedding Gao et al. (2021a). Negative instance
$\bm{o}_{m}^{-}$ is the general knowledge for $\bm{x}_{m}$ from the LM
obtained via Eq. 12.  With $\bm{o}_{m}$, $\bm{o}_{m}^{+}$, and
$\bm{o}_{m}^{-}$, our contrastive loss is ($\text{sim}(\cdot)$ is the cosine
similarity),

$\mathcal{L}_{\text{contrast}}=-\text{log}\frac{e^{\text{sim}(\bm{o}_{m},\bm{o}_{m}^{+})}/\tau}{\sum_{j=1}^{N}(e^{\text{sim}(\bm{o}_{m},\bm{o}_{j}^{+})/\tau}+e^{\text{sim}(\bm{o}_{m},\bm{o}_{j}^{-})/\tau})}.$
(15)

Compared to Eq. 1, the second term is added in the denominator, i.e., general
knowledge representations as additional negative samples/instances. Figure 1
(B) shows a red arrow pointed from $\bm{o}^{\text{full}}$ to itself,
indicating the positive instances are from inputting twice. The dashed red
arrow pointing to $\bm{o}^{\text{gen}}$ indicates the negative instances
contrasting the specialized and general knowledge.

Unlabeled Domain Datasets | End-Task Classification Datasets
---|---
Source | Dataset | Size | Dataset | Task | #Training | #Testing | #Classes
Reviews | Yelp Restaurant | 758MB | Restaurant | Aspect Sentiment Classification (ASC) | 3,452 | 1,120 | 3
Amazon Phone | 724MB | Phone | Aspect Sentiment Classification (ASC) | 239 | 553 | 2
Amazon Camera | 319MB | Camera | Aspect Sentiment Classification (ASC) | 230 | 626 | 2
Academic Papers | ACL Papers | 867MB | ACL | Citation Intent Classification | 1,520 | 421 | 6
AI Papers | 507MB | AI | Relation Classification | 2,260 | 2,388 | 7
PubMed Papers | 989MB | PubMed | Chemical-protein Interaction Prediction | 2,667 | 7,398 | 13

Table 1:  Statistics for domain post-training datasets and end task supervised
classification datasets (more detail of each task is given in Appendix A).

###  3.3 DGA Objectives

DGA is a pipelined model: First, a subset of the domain data is used to
estimate the attention head importance ($I_{lh}$ in Sec. 3.1). Second, given
the attention head importance, we compute the final domain-adaptive loss by
combining the conventional Masked Language Model (MLM) loss (include the
proposed soft-masking for general knowledge) and the proposed contrastive
loss:

$\mathcal{L}_{\text{DA-
train}}=\mathcal{L}_{\text{MLM}}+\lambda_{1}\mathcal{L}_{\text{contrast}},$
(16)

where $\lambda_{1}$ is the hyper-parameter to adjust the impact of the added
term.

##  4 Experiments

We follow the experiment setup in Gururangan et al. (2020). RoBERTa Liu et al.
(2019)777https://huggingface.co/roberta-base is used as the LM. In each
experiment, we first DA-train the LM and then fine-tune it on the end-task.
The final evaluation is based on the end-task results.

###  4.1 Datasets and Baselines

Datasets: Table 1 shows the statistics of the unlabeled domain datasets for
DA-training and their corresponding end-task classification datasets. We use 6
unlabeled domain datasets:888We down-sampled the PubMed due to its huge
original size. In general, our datasets are much smaller comparing to previous
work Gururangan et al. (2020) (which used more than 11GB of data for each
domain). Our experiments showed that a smaller dataset is sufficient and more
data does not help. It also requires much less computation resource. 3 of them
are about reviews: Yelp Restaurant Xu et al. (2019a), Amazon Phone Ni et al.
(2019), Amazon Camera Ni et al. (2019); 3 of them are academic papers: ACL
Papers Lo et al. (2020), AI Papers Lo et al. (2020), and PubMed
Papers999https://pubmed.ncbi.nlm.nih.gov/. Each unlabeled domain dataset has a
corresponding end-task classification dataset101010Note that our results are
different from those presented in Table 5 of Gururangan et al. (2020) because
we observe very high variances due to very small original test sets and thus
re-partition the training and test set (by enlarging the test set and reducing
the training set slightly): Restaurant111111To be consistent with existing
research Tang et al. (2016), examples with conflict polarities (both positive
and negative sentiments are expressed about an aspect term) are not used. Xu
et al. (2019a), Phone Ding et al. (2008); Hu and Liu (2004), Camera Ding et
al. (2008); Hu and Liu (2004)121212Note that Ding et al. (2008) and Hu and Liu
(2004) contain 9 and 5 domains, respectively. We extract those domains related
to “Phone” and “Camera” from them., ACL (ACL-ARC in Jurgens et al. (2018)), AI
(SCIERC in Luan et al. (2018)), and PubMed (CHEMPORT in Kringelum et al.
(2016)).

Baselines. We consider 10 baselines.

(1). Non-DA-training (RoBERTa) Liu et al. (2019) uses the original RoBERTa for
the end-task fine-tuning without any DA-training.

(2). DA-training using masked language model loss (MLM) is the existing DA-
training method. To our knowledge, existing DA-training systems are all based
on the MLM loss.

(3). DA-training using adapter-tuning (MLM (Adapter)) adds adapter layers
between layers of Transformer for DA-training. An adapter Houlsby et al.
(2019) has two fully connected layers and a skip connection. During DA-
training, the Transformer is fixed, only the adapters are trained. The
bottleneck (adapter) size is set to 64 Houlsby et al. (2019). During end-task
fine-tuning, both RoBERTa and adapters are trainable for fair comparison.

(4). DA-training using prompt-tuning (MLM (Prompt)) Lester et al. (2021) adds
a sequence of prompt tokens to the end of the original sequence. In DA-
training, RoBERTa (the LM) is fixed and only the prompt tokens are trained. In
end-task fine-tuning, both LM and the trained prompt are trainable. We
initialize 100 tokens and set the learning rate of the prompt token to 0.3 in
DA-training, following the setting in Lester et al. (2021).

(5). Knowledge distillation (MLM+KD) Hinton et al. (2015) minimizes the
representational deviation between the general knowledge in LM and the
specialized knowledge in DA-training. We compute the KL divergence between the
representations (the output before the masked language model prediction head)
of each word of the two models (LM and DA-trained) as the distillation loss.

(6). Adapted distillation through attention (MLM+AdaptedDeiT) is derived from
DeiT Touvron et al. (2021), a distillation method for visual Transformer (ViT)
Dosovitskiy et al. (2020). We adapt DeiT to a text-based and unsupervised
model by distilling the LM representation131313We take the average of its
token’s output as sequence’s representation. The same for SimCSE baseline. to
the added distillation token and change ViT to RoBERTa.

(7, 8). DA-training using sequence-level contrastive learning (MLM+SimCSE and
MLM+InfoWord). SimCSE is a contrastive learning method for sentence embedding
Gao et al. (2021a). We use its unsupervised version where positive samples are
from the same input with different dropout masks and negative samples are
other instances in the same batch. InfoWord Kong et al. (2020) is another
contrastive learning method contrasts the span-level local representation and
sequence-level global representation.

(9, 10). DA-training using token-aware contrastive learning (MLM+TaCL and
MLM+TaCO). TaCL Su et al. (2021) and TaCO Fu et al. (2022) are two recent
methods to improve BERT pre-training with token-aware contrastive
loss.141414TaCL and TaCO are not a DA-training model. It pre-trains an LM to
improve it using the same data as that for training the LM. We switch the data
to our target domain data. We change the backbone to RoBERTa for fair
comparison.

###  4.2 Implementation Detail

Architecture. We adopt $\text{RoBERTa}_{\textbf{BASE}}$ as our backbone LM (12
layers and 12 attention heads in each layer). A masked language model head is
applied for DA-training. The end-task fine-tuning of RoBERTa follows the
standard practice. For the three ASC tasks (see Table 1), we adopt the ASC
formulation in Xu et al. (2019a), where the aspect (e.g., “sound”) and review
sentence (e.g., “The sound is great”) are concatenated via </s>.

Hyperparameters. Unless otherwise stated, the same hyper-parameters are used
in all experiments. The maximum input length is set to 164 which is long
enough for all datasets. Adam optimizer is used for both DA-training and end-
task fine-tuning. The max sequence length is set to 164, which is long enough
for our end-tasks and only needs moderate computational resources.

Domain-adaptive pre-training (DA-training). The learning rate is set to 1e-4
and batch size is 256. We train 2.5K steps for each domain, roughly a full
pass through the domain data, following Gururangan et al. (2020); Xu et al.
(2019a). The subset of data $\\{\bm{x}^{\text{sub}}_{m}\\}$ for computing
$\mathcal{L}_{\text{impt}}$ to determine head importance in Sec. 3.1 is set to
1.64 Million tokens, which is sufficient in our experiments. $\lambda_{1}$ in
Eq. 16 is set to 1 and $\tau$ in Eq. 15 is set to 0.05.

End-task fine-tuning. The learning rate is set to 1e-5 and batch size to 16.
We train on end-task fine-tuning datasets for 5 epochs for Restaurant; 10
epochs for ACL, AI and PubMed; and 15 epochs for Phone and Camera. We simply
take the results for the last epoch as we empirically found that the above
number of epochs gives us stable and convergence results.

Domain | Camera | Phone | Resturant | AI | ACL | PubMed | Avg
---|---|---|---|---|---|---|---
Model | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | Micro-F1
RoBERTa | 78.82 | 87.03 | 83.75 | 86.08 | 79.81 | 87.00 | 60.98 | 71.85 | 66.11 | 71.26 | 72.38 | 73.64
MLM | 84.39 | 89.90 | 82.59 | 85.50 | 80.84 | 87.68 | 68.97 | 75.95 | 68.75 | 73.44 | 72.84 | 76.40
MLM (Adapter) | 83.62 | 89.23 | 82.71 | 85.35 | 80.19 | 87.14 | 60.55 | 71.38 | 68.87 | 72.92 | 71.68 | 74.60
MLM (Prompt) | 85.52 | 90.38 | 84.17 | 86.53 | 79.00 | 86.45 | 61.47 | 72.36 | 66.66 | 71.35 | 73.09 | 74.98
MLM+KD | 82.79 | 89.30 | 80.08 | 83.33 | 80.40 | 87.25 | 67.76 | 75.46 | 68.19 | 72.73 | 72.35 | 75.26
MLM+AdaptedDeiT | 86.86 | 91.37 | 83.08 | 85.64 | 79.70 | 86.84 | 69.72 | 76.83 | 69.11 | 73.35 | 72.69 | 76.86
MLM+SimCSE | 84.91 | 90.35 | 83.46 | 86.08 | 80.88 | 87.59 | 69.10 | 76.25 | 69.89 | 74.30 | 72.77 | 76.84
MLM+TaCL | 81.98 | 88.88 | 81.87 | 84.92 | 81.12 | 87.50 | 64.04 | 73.18 | 63.18 | 70.31 | 69.46 | 73.61
MLM+TaCO | 84.50 | 90.22 | 82.63 | 85.32 | 79.27 | 86.68 | 59.73 | 71.22 | 63.66 | 70.36 | 72.38 | 73.69
MLM+InfoWord | 87.95 | 91.92 | 84.58 | 86.84 | 81.24 | 87.82 | 68.29 | 75.92 | 68.58 | 73.68 | 73.21 | 77.31
DGA | 88.52 | 92.49 | 85.47 | 87.45 | 81.83 | 88.20 | 71.99 | 78.06 | 71.01 | 74.73 | 73.65 | 78.74

Table 2: We report the macro-F1 (MF1) and accuracy results for all datasets,
except for CHEMPORT in the PubMed domain, for which we use micro-F1 following
Gururangan et al. (2020); Dery et al. (2021); Beltagy et al. (2019). The
results are averages of 5 random seeds (the standard deviation is reported in
Appendix B).  The average column (Avg) is the average over the MF1 (or
Micro-F1 for PubMed) for all datasets.

###  4.3 Evaluation Results and Ablation Study

We report the end-task results of the 10 baselines on the 6 datasets in Table
2.

Superiority of DGA. Our DGA consistently outperforms all baselines. Thanks to
the proposed more informed adaptation, DGA improves over the widely used
traditional DA-training baseline MLM. We also see that MLM markedly
outperforms RoBERTa (non-DA-training) on average (see the last column). We
discuss more observations about the results bellow.

(1). Training the entire LM in DGA helps achieve much better results. Using
adapter (MLM (adapter)) and prompt (MLM (prompt)) have mixed results. This is
because adapter and prompt do not have sufficient trainable parameters, which
are also randomly initialized and can be difficult to train.

(2). DGA is also better than distillation-based systems: MLM+AdaptedDeiT and
MLM+KD, which try to preserve the past knowledge. This is not surprising
because the goal of DA-training is not simply preserving the previous
knowledge but also to adapt/change it as needed to suit the target domain. DGA
is specifically designed for this with soft-masking and contrasting of
knowledge.

(3). The contrastive learning in DGA is more effective than the other
contrastive alternatives (MLM+SimCSE, MLM+TaCL, MLM+TaCO and MLM+InfoWord).
This indicates contrasting the general and full knowledge for knowledge
integration is important.

Domain | Camera | Phone | Resturant | AI | ACL | PubMed | Avg
---|---|---|---|---|---|---|---
Model | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | Micro-F1
RoBERTa | 78.82 | 87.03 | 83.75 | 86.08 | 79.81 | 87.00 | 60.98 | 71.85 | 66.11 | 71.26 | 72.38 | 73.64
MLM | 84.39 | 89.90 | 82.59 | 85.50 | 80.84 | 87.68 | 68.97 | 75.95 | 68.75 | 73.44 | 72.84 | 76.40
DGA (H, I) | 86.79 | 91.60 | 84.21 | 86.40 | 81.32 | 87.91 | 71.07 | 77.36 | 69.50 | 73.82 | 73.34 | 77.71
DGA (H, I, O) | 88.04 | 92.01 | 85.85 | 87.63 | 81.45 | 87.79 | 71.54 | 77.61 | 70.52 | 74.58 | 73.10 | 78.42
DGA (H, I, O, E) | 87.05 | 91.60 | 83.74 | 86.11 | 80.64 | 87.61 | 72.64 | 78.17 | 71.24 | 74.96 | 73.54 | 78.14
DGA (w/o contrast) | 86.19 | 90.89 | 84.48 | 86.65 | 81.70 | 87.93 | 68.25 | 75.49 | 69.31 | 73.73 | 72.72 | 77.11
DGA (random mask) | 82.07 | 89.30 | 83.86 | 86.33 | 80.60 | 87.52 | 69.51 | 76.64 | 69.59 | 73.73 | 72.92 | 76.43
Ensemble (LM+MLM) | 85.22 | 90.64 | 85.15 | 87.23 | 79.86 | 86.98 | 65.10 | 74.43 | 68.56 | 73.44 | 72.60 | 76.08
DGA (domain-specific) | 88.06 | 92.04 | 83.45 | 85.82 | 81.72 | 87.90 | 68.00 | 75.57 | 70.91 | 75.06 | 73.17 | 77.55
DGA | 88.52 | 92.49 | 85.47 | 87.45 | 81.83 | 88.20 | 71.99 | 78.06 | 71.01 | 74.73 | 73.65 | 78.74

Table 3: Ablation results - averages of 5 random seeds. The standard
deviations are reported in Appendix B.

Effectiveness of the proxy KL-divergence loss. We use the proposed proxy KL-
divergence loss to compute the head importance to identify the general
language knowledge in the LM without using the LM’s original pre-training data
(Sec. 3.1).

For evaluation, we are interested in how good the proxy is. Since we don’t
have the data that pre-trains RoBERTa, it is not obvious how to assess the
quality of the proxy directly. Here, we provide some indirect evidences to
show the effectiveness of the proxy for computing the importance of units to
the general knowledge in the LM.

We conduct a separate experiment to compare the attention heads’ importance
score vectors after applying the proxy using the data from different domains.
For each domain $i$, we compare its importance vector with the importance
vector of every other domain, and then average the cosine similarities to get
the value for domain $i$. We get 0.92 for Restaurant, the same 0.91 for ACL,
AI, and Phone, 0.89 for PubMed and 0.92 for Camera. We see that different
domains give similar importance values, which indirectly show that our proxy
can identify the common general knowledge.

We also compute the importance score distributions of the proxy. For each of
the 6 domains, after applying the proxy, around 20% of the attention heads are
heavily protected ($0.8\leq I^{\text{norm}}_{lh}\leq 1.0$) and another 20%
moderately protected ($0.6\leq I^{\text{norm}}_{lh}<0.8$), which indicate the
general knowledge. While Phone, AI, Camera and Restaurant share a similar
distribution, ACL and PubMed protect slightly less. This is understandable as
PubMed and ACL (medical or NLP publications) are probably less common than the
other domains and the general knowledge in the LM covers them less.

Ablation study. To better understand DGA, We want to know (1) whether
constraining the neurons in other layers are helpful (the proposed DGA only
constrains the attention heads), and (2) where the gain of DGA is from. To
answer (1), we constrain the training of different layers in a standard
Transformer. In Table 3 (rows 3-5), “H”, “I”, and “O” refer to attention head,
intermediate layer, output layer in a standard Transformer layer,
respectively. “E” refers to the embedding layers. The brackets with
combination of “H, I, O, E” indicate the location we apply the soft-masking
(DGA only applies soft-masking in the attention head). We can see their
results are similar or worse than DGA, implying that attention heads are more
indicative of important knowledge. To answer (2), we conduct the following
ablation experiments: (i) DGA (w/o contrast), without the contrastive loss,
but only soft-masking the backward pass according to the attention head
importance. (ii) DGA (random masking) with randomly generated attention head
importance scores and using them to do soft-masking and contrastive learning.
(iii) Ensemble (LM+MLM) performs the end-task fine-tuning on both the MLM DA-
trained RoBERTa (conventional DA-training) and the original RoBERTa (LM) by
concatenating their outputs and taking the average. (iv) DGA (domain-specific)
refers to the variant that contrasts domain-specific and general knowledge
(see Sec. 3.2).151515We don’t have DGA(w/o soft-masking) because our
contrastive learning relies on soft-masking. If removed, contrastive loss will
not have the additional negative samples and our DGA becomes MLM+SimCSE.

Table 3 shows that the full DGA always gives the best result, indicating every
component contributes. Additional observations are as follows:

(1) DGA’s gain is partially from the novel soft-masking: we can see that on
average, DGA (w/o contrast) outperforms conventional DA-training (MLM).
Besides, our gradient-based mask is informative: we can see DGA (random mask)
is worse than DGA (w/o contrast) on all datasets. DGA (w/o contrast) is even
better than Ensemble, which directly combines the information given by both
the original LM and the traditional DA-trained model during end-task fine-
tuning

(2) Besides soft-masking, contrasting the general and full knowledge also
helps. We can see DGA outperforms DGA (w/o contrast) and DGA (domain-specific)
in all datasets.

##  5 Conclusion

This paper argued that an effective DA-training method should effectively
integrate the target domain knowledge to the general knowledge in the LM.
Existing approaches do not explicitly do this. This paper proposed a novel
method DGA to achieve it (1) by estimating the attention heads importance in
LM and using the importance scores to soft-mask the attention heads in DA-
training to preserve the important knowledge in LM as much as possible, and
(2) by contrasting the general and the full knowledge. Extensive experiment
results demonstrated the effectiveness of the proposed approach DGA.

##  6 Limitations

While effective, DGA has some limitations. First, the main focus of DGA is to
adapt an LM to a given target domain. It does not consider the generalization
to other domains. For example, it will be interesting to incrementally or
continually adapt an LM to more and more domains to make the LM more useful Ke
et al. (2020, 2021, 2022). Second, the importance of parameters for general
knowledge in the LM is computed using a proxy method based on model
robustness. Although it is quite effective, it is interesting to explore other
approaches to further improve it. We will work on these in our future work as
specializing and improving an LM is an important problem.

## Acknowledgments

The work of Zixuan Ke and Bing Liu was supported in part by three National
Science Foundation (NSF) grants (IIS-1910424, IIS-1838770, and CNS-2225427).

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical bert embeddings.  _arXiv preprint arXiv:1904.03323_.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_.
  * Chakrabarty et al. (2019) Tuhin Chakrabarty, Christopher Hidey, and Kathleen McKeown. 2019.  Imho fine-tuning improves claim detection.  _arXiv preprint arXiv:1905.07000_.
  * Chen et al. (2020a) Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020a.  The lottery ticket hypothesis for pre-trained bert networks.  _Advances in neural information processing systems_ , 33:15834–15846.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR.
  * Chen and Liu (2018) Zhiyuan Chen and Bing Liu. 2018.  Lifelong machine learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ , 12(3):1–207.
  * Dery et al. (2021) Lucio M Dery, Paul Michel, Ameet Talwalkar, and Graham Neubig. 2021.  Should we be pre-training? an argument for end-task aware training as an alternative.  _arXiv preprint arXiv:2109.07437_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_.
  * Ding et al. (2008) Xiaowen Ding, Bing Liu, and Philip S Yu. 2008.  A holistic lexicon-based approach to opinion mining.  In _Proceedings of the 2008 international conference on web search and data mining_.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Fan et al. (2020) Angela Fan, Edouard Grave, and Armand Joulin. 2020.  Reducing transformer depth on demand with structured dropout.  In _International Conference on Learning Representations_.
  * Fu et al. (2022) Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, and Lei Li. 2022.  Contextual representation learning beyond masked language modeling.  In _ACL_.
  * Gao et al. (2021a) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021a.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_.
  * Gao et al. (2021b) Yang Gao, Nicolo Colombo, and Wei Wang. 2021b.  Adapting by pruning: A case study on bert.  _arXiv preprint arXiv:2105.03343_.
  * Guo et al. (2016) Yiwen Guo, Anbang Yao, and Yurong Chen. 2016.  Dynamic network surgery for efficient dnns.  _Advances in neural information processing systems_ , 29.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _ACL_.
  * Han et al. (2015) Song Han, Jeff Pool, John Tran, and William Dally. 2015.  Learning both weights and connections for efficient neural network.  _Advances in neural information processing systems_ , 28.
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9729–9738.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7).
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _ICML_.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of ACM SIGKDD_.
  * Jurgens et al. (2018) David Jurgens, Srijan Kumar, Raine Hoover, Daniel A. McFarland, and Dan Jurafsky. 2018.  Measuring the evolution of a scientific field through citation frames.  _TACL_.
  * Ke et al. (2022) Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu. 2022.  Continual training of language models for few-shot learning.  In _Empirical Methods in Natural Language Processing (EMNLP)_.
  * Ke et al. (2020) Zixuan Ke, Bing Liu, and Xingchang Huang. 2020.  Continual learning of a mixed sequence of similar and dissimilar tasks.  In _NeurIPS_.
  * Ke et al. (2021) Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. 2021.  Achieving forgetting prevention and knowledge transfer in continual learning.  _NeurIPS_.
  * Kong et al. (2020) Lingpeng Kong, Cyprien de Masson d’Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2020.  A mutual information maximization perspective of language representation learning.  In _ICLR_.
  * Kringelum et al. (2016) Jens Kringelum, Sonny Kim Kjaerulff, Søren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. 2016.  Chemprot-3.0: a global chemical biology diseases mapping.  _Database_ , 2016.
  * Lai et al. (2021) Cheng-I Jeff Lai, Yang Zhang, Alexander H Liu, Shiyu Chang, Yi-Lun Liao, Yung-Sung Chuang, Kaizhi Qian, Sameer Khurana, David Cox, and Jim Glass. 2021\.  Parp: Prune, adjust and re-prune for self-supervised speech recognition.  _NeurIPS_ , 34.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _EMNLP_.
  * Li et al. (2021) Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2021.  Differentiable subset pruning of transformer heads.  _Transactions of the Association for Computational Linguistics_ , 9:1442–1459.
  * Lin et al. (2020) Zi Lin, Jeremiah Zhe Liu, Zi Yang, Nan Hua, and Dan Roth. 2020.  Pruning redundant mappings in transformer models via spectral-normalized identity prior.  _arXiv preprint arXiv:2010.01791_.
  * Liu (2015) Bing Liu. 2015.  _Sentiment analysis: Mining opinions, sentiments, and emotions_.  Cambridge University Press.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. 2020.  S2ORC: the semantic scholar open research corpus.  In _ACL_.
  * Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018.  Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction.  In _ACL_.
  * McCarley et al. (2019) JS McCarley, Rishav Chakravarti, and Avirup Sil. 2019.  Structured pruning of a bert-based question answering model.  _arXiv preprint arXiv:1910.06360_.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one?  _Advances in neural information processing systems_ , 32.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In _EMNLP_ , pages 188–197. Association for Computational Linguistics.
  * Sajjad et al. (2020) Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2020.  On the effect of dropping layers of pre-trained transformer models.  _arXiv preprint arXiv:2004.03844_.
  * Su et al. (2021) Yixuan Su, Fangyu Liu, Zaiqiao Meng, Lei Shu, Ehsan Shareghi, and Nigel Collier. 2021.  Tacl: Improving bert pre-training with token-aware contrastive learning.  _arXiv preprint arXiv:2111.04198_.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019.  How to fine-tune bert for text classification?  In _China national conference on Chinese computational linguistics_ , pages 194–206. Springer.
  * Tang et al. (2016) Duyu Tang, Bing Qin, and Ting Liu. 2016.  Aspect level sentiment classification with deep memory network.  In _EMNLP_.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , pages 10347–10357. PMLR.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_.
  * Voita et al. (2019) Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019.  Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned.  _arXiv preprint arXiv:1905.09418_.
  * Xu et al. (2019a) Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019a.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  In _NAACL-HLT_.
  * Xu et al. (2019b) Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019b.  Review conversational reading comprehension.  _arXiv preprint arXiv:1902.00821_.

Domain | Camera | Phone | Resturant | AI | ACL | PubMed
---|---|---|---|---|---|---
Model | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | Micro-F1
RoBERTa |  $\pm$0.0403 |  $\pm$0.0179 |  $\pm$0.0210 |  $\pm$0.0154 |  $\pm$0.0117 |  $\pm$0.0049 |  $\pm$0.0646 |  $\pm$0.0347 |  $\pm$0.0192 |  $\pm$0.0096 |  $\pm$0.0071
MLM |  $\pm$0.0479 |  $\pm$0.0298 |  $\pm$0.0165 |  $\pm$0.0103 |  $\pm$0.0096 |  $\pm$0.0056 |  $\pm$0.0117 |  $\pm$0.0086 |  $\pm$0.0218 |  $\pm$0.0118 |  $\pm$0.0035
MLM (adapter) |  $\pm$0.0165 |  $\pm$0.0110 |  $\pm$0.0265 |  $\pm$0.0181 |  $\pm$0.0102 |  $\pm$0.0068 |  $\pm$0.0551 |  $\pm$0.0288 |  $\pm$0.0142 |  $\pm$0.0099 |  $\pm$0.0055
MLM (prompt) |  $\pm$0.0243 |  $\pm$0.0138 |  $\pm$0.0126 |  $\pm$0.0087 |  $\pm$0.0060 |  $\pm$0.0035 |  $\pm$0.0301 |  $\pm$0.0124 |  $\pm$0.0068 |  $\pm$0.0108 |  $\pm$0.0028
MLM+KD |  $\pm$0.0295 |  $\pm$0.0158 |  $\pm$0.0320 |  $\pm$0.0230 |  $\pm$0.0099 |  $\pm$0.0070 |  $\pm$0.0345 |  $\pm$0.0224 |  $\pm$0.0292 |  $\pm$0.0155 |  $\pm$0.0093
MLM+AdaptedDeiT |  $\pm$0.0187 |  $\pm$0.0122 |  $\pm$0.0160 |  $\pm$0.0101 |  $\pm$0.0048 |  $\pm$0.0022 |  $\pm$0.0250 |  $\pm$0.0179 |  $\pm$0.0065 |  $\pm$0.0079 |  $\pm$0.0086
MLM+SimCSE |  $\pm$0.0114 |  $\pm$0.0077 |  $\pm$0.0098 |  $\pm$0.0065 |  $\pm$0.0029 |  $\pm$0.0016 |  $\pm$0.0086 |  $\pm$0.0056 |  $\pm$0.0054 |  $\pm$0.0071 |  $\pm$0.0027
MLM+TaCL |  $\pm$0.0218 |  $\pm$0.0103 |  $\pm$0.0230 |  $\pm$0.0159 |  $\pm$0.0105 |  $\pm$0.0059 |  $\pm$0.0275 |  $\pm$0.0156 |  $\pm$0.0713 |  $\pm$0.0394 |  $\pm$0.0118
MLM+TaCO |  $\pm$0.0456 |  $\pm$0.0232 |  $\pm$0.0166 |  $\pm$0.0134 |  $\pm$0.0077 |  $\pm$0.0052 |  $\pm$0.0675 |  $\pm$0.0380 |  $\pm$0.0207 |  $\pm$0.0128 |  $\pm$0.0099
MLM+InfoWord |  $\pm$0.0267 |  $\pm$0.0139 |  $\pm$0.0272 |  $\pm$0.0191 |  $\pm$0.0170 |  $\pm$0.0089 |  $\pm$0.0344 |  $\pm$0.0219 |  $\pm$0.0070 |  $\pm$0.0079 |  $\pm$0.0072
DGA |  $\pm$0.0095 |  $\pm$0.0047 |  $\pm$0.0127 |  $\pm$0.0094 |  $\pm$0.0052 |  $\pm$0.0040 |  $\pm$0.0127 |  $\pm$0.0081 |  $\pm$0.0079 |  $\pm$0.0080 |  $\pm$0.0034

Table 4: Standard deviations of the corresponding metrics of the proposed DGA
model and the baselines on the six experiments.

Domain | Camera | Phone | Resturant | AI | ACL | PubMed
---|---|---|---|---|---|---
Model | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | MF1 | Acc. | Micro-F1
RoBERTa |  $\pm$0.0403 |  $\pm$0.0179 |  $\pm$0.0210 |  $\pm$0.0154 |  $\pm$0.0117 |  $\pm$0.0049 |  $\pm$0.0646 |  $\pm$0.0347 |  $\pm$0.0192 |  $\pm$0.0096 |  $\pm$0.0071
MLM |  $\pm$0.0479 |  $\pm$0.0298 |  $\pm$0.0165 |  $\pm$0.0103 |  $\pm$0.0096 |  $\pm$0.0056 |  $\pm$0.0117 |  $\pm$0.0086 |  $\pm$0.0218 |  $\pm$0.0118 |  $\pm$0.0035
DGA (H, I) |  $\pm$0.0373 |  $\pm$0.0210 |  $\pm$0.0032 |  $\pm$0.0039 |  $\pm$0.0054 |  $\pm$0.0045 |  $\pm$0.0095 |  $\pm$0.0048 |  $\pm$0.0094 |  $\pm$0.0073 |  $\pm$0.0049
DGA (H, I, O) |  $\pm$0.0167 |  $\pm$0.0092 |  $\pm$0.0182 |  $\pm$0.0155 |  $\pm$0.0055 |  $\pm$0.0033 |  $\pm$0.0093 |  $\pm$0.0075 |  $\pm$0.0080 |  $\pm$0.0070 |  $\pm$0.0056
DGA (H, I, O, E) |  $\pm$0.0237 |  $\pm$0.0123 |  $\pm$0.0270 |  $\pm$0.0187 |  $\pm$0.0099 |  $\pm$0.0050 |  $\pm$0.0109 |  $\pm$0.0089 |  $\pm$0.0067 |  $\pm$0.0057 |  $\pm$0.0079
DGA (w/o contrast) |  $\pm$0.0433 |  $\pm$0.0251 |  $\pm$0.0135 |  $\pm$0.0106 |  $\pm$0.0060 |  $\pm$0.0040 |  $\pm$0.0197 |  $\pm$0.0119 |  $\pm$0.0132 |  $\pm$0.0093 |  $\pm$0.0050
DGA (random mask) |  $\pm$0.0879 |  $\pm$0.0413 |  $\pm$0.0335 |  $\pm$0.0235 |  $\pm$0.0096 |  $\pm$0.0044 |  $\pm$0.0153 |  $\pm$0.0090 |  $\pm$0.0105 |  $\pm$0.0059 |  $\pm$0.0052
Ensemble |  $\pm$0.0332 |  $\pm$0.0178 |  $\pm$0.0199 |  $\pm$0.0139 |  $\pm$0.0035 |  $\pm$0.0031 |  $\pm$0.0236 |  $\pm$0.0103 |  $\pm$0.0061 |  $\pm$0.0028 |  $\pm$0.0046
DGA (domain-specific) |  $\pm$0.0137 |  $\pm$0.0070 |  $\pm$0.0259 |  $\pm$0.0200 |  $\pm$0.0031 |  $\pm$0.0018 |  $\pm$0.0128 |  $\pm$0.0071 |  $\pm$0.0108 |  $\pm$0.0067 |  $\pm$0.0043
DGA |  $\pm$0.0095 |  $\pm$0.0047 |  $\pm$0.0127 |  $\pm$0.0094 |  $\pm$0.0052 |  $\pm$0.0040 |  $\pm$0.0127 |  $\pm$0.0081 |  $\pm$0.0079 |  $\pm$0.0080 |  $\pm$0.0034

Table 5: Standard deviations of the corresponding metrics of the proposed DGA
model and the ablation on the six experiments.

##  Appendix A Datasets Details

Table 2 in the main paper has given the number of examples in each dataset.
Here we provide additional details about the 4 types of end-tasks.

(1) (Phone, Camera and Restaurant) Aspect Sentiment Classification (ASC) is
defined as follows Liu (2015): given an aspect or product feature (e.g.,
picture quality in a camera review) and a review sentence containing the
aspect in a domain or product category (e.g., camera), classify if the
sentence expresses a positive, negative, or neutral (no opinion) sentiment or
polarity about the aspect (for Phone and Camera, there are only negative and
positive polarities in the data).

(2) (ACL) Citation Intent Classification is defined as follows: given a citing
sentence (a sentence contains a citation), classify if the sentence expresses
a citation function among “background”, “motivation”, “uses”, “extension” and
“comparison or contrast future”.

(3) (AI) Relation Classification is defined as follows: given a within-
sentence word sequence spans containing a pair of entities, classify if the
span expresses a relation among “feature of”, “conjunction”, “evaluate for”,
“hyponym of”, “used for”, “part of” and “compare”.

(4) (PubMed) Chemical-protein Interaction Classification is defined as
follows: given a span containing a pair of chemical and protein, classify if
the span expresses a chemical-protein interaction among “downregulator”,
“substrate”, “indirect-upregulator”, “indirect-downregulator”, “agnonist”,
“activator”, “product of”, “agonist-activator”, “inhibitor”, “upregulator”,
“substrate product of”, “agonist-inhibitor”and “antagonist”.

##  Appendix B Standard Deviations

Table 4 reports the standard deviations of the corresponding results in Table
2 (in the main paper) of DGA and the considered baselines over 5 runs with
random seeds. We can see the results of DGA are stable. Some baselines (e.g.,
RoBERTa in AI, MLM in Camera and MLM+TaCL in ACL) can have quite large
standard deviations.

Table 5 reports the standard deviations of the corresponding results in Table
3 (in the main paper) of DGA and the considered baselines over 5 runs with
random seeds. We can see the results of DGA are stable. Some baselines (e.g.,
DGA (random mask) and DGA (w/o contrast) in Camera) can have quite large
standard deviations.


# Quantum Privacy Aggregation of Teacher Ensembles (qPATE)
for privacy preserving quantum machine learning

###### Abstract

The utility of machine learning has rapidly expanded in the last two decades
and presents an ethical challenge. Papernot et. al. developed a technique,
known as Private Aggregation of Teacher Ensembles (PATE) to enable federated
learning in which multiple _teacher_ models are trained on disjoint data sets.
This study is the first to apply PATE to an ensemble of quantum neural
networks (QNN) to pave a new way of ensuring privacy in quantum machine
learning (QML) models.

##  1 Introduction

Machine Learning (ML) is being utilized in a wide range of applications [1, 2,
3, 4, 5, 6] and often raises privacy and ethical concerns. For instance,
privacy leaking is a major concern as demonstrated in large language models
like GPT trained with sensitive texts [7] and many applications deploy the
model directly on the device, which would allow an adversary to use the model
parameters directly to release private data. This is true especially in mobile
applications in an effort to reduce communication with the server [8].
_Differential Privacy_ (DP) seeks to address these privacy concerns through
its privacy-loss framework. One of the newer techniques to ensure differential
privacy is known as _Privacy-Aggregation of Teacher Ensembles_ , i.e. PATE. It
was pioneered by Nicholas Papernot et al. in 2017 [9]. PATE provides strong
privacy guarantees by training in a two-layer fashion. The training set is
divided between $N$ _teacher models_ , who each train independently of each
other. Then, the _teacher ensembles_ predict the labels of a disjoint training
set through a noisy aggregation. The disjoint training set with newly
aggregated noisy labels is used to train _student_. The _student_ , never
having access to the original training set or any _teacher ensembles_ model
parameters, is deployed into the application.

On the other hand, with the recent advancements of quantum computing hardware
[10], it is a natural next step to investigate _Quantum Machine Learning_
(QML). Currently, there have been some investigations at the intersection of
quantum computing and privacy-preserving machine learning [11, 12], yet to our
best knowledge, there is no research implementing PATE with _variational
quantum circuits_ (VQC). This study aims to implement an ensemble of hybrid
quantum-classical classifiers and train them using privacy-aggregation of
teacher ensembles (PATE). After training, the student model will satisfy
privacy loss limits. Classical classifiers with PATE training will be used as
controls in the study. We demonstrate that the privacy-preserving VQC on MNIST
handwritten digits has over 99% accuracy with significant privacy guarantees.

##  2 Privacy-Aggregation of Teacher Ensembles (PATE)

###  2.1 Differential Privacy

Differential Privacy (DP) has manifested as the standard tool for gauging
privacy loss [13, 14]. The notion of a privacy budget determines the amount of
information that an adversary can extract. Information can be divided into two
groups, _general information_ and _private information_. The former refers to
a general property of the dataset, whereas the latter refers to entry-specific
information. DP puts limits on how much _private information_ can be
ascertained from querying a database, or in the case of machine learning, a
classifier [15].

DP is designated by two hyper-parameters, $\epsilon$ and $\delta$. The first
parameter $\epsilon$ assigns the _privacy budget_ to the system and $\delta$
denotes the probability of leaking more information than allowed by the
privacy budget [16]. The _privacy budget_ also known as the _privacy loss_ ,
$\mathcal{L}$, explicates the differences in the distributions characterized
by two similar queries, $\mathcal{M}(x)$ and $\mathcal{M}(y)$. The privacy
loss for a given observation, $\xi\in\mathrm{range}(\mathcal{M})$, quantifies
the likelihood of observing $\xi$ from $\mathcal{M}(x)$ versus
$\mathcal{M}(y)$.
$\mathcal{L}^{(\xi)}_{\mathcal{M}(x)||\mathcal{M}(y)}=\ln\left(\frac{Pr[\mathcal{M}(x)=\xi]}{Pr[\mathcal{M}(y)=\xi]}\right)$.
Privacy-preserving algorithms are by definition non-deterministic, and an
algorithm $\mathcal{M}$ is $(\epsilon,\delta)$-_differentially private_ if it
satisfies, $Pr[\mathcal{M}(x)\in
S]\leq\mathrm{exp}(\varepsilon)Pr[\mathcal{M}(y)\in S]+\delta$, for all
datasets $x,y$ as the domain of $\mathcal{M}$ with condition $\|x-y\|_{1}\leq
1$ where $S$ denotes all possible events as outcomes of $\mathcal{M}$.
Additionally, $\epsilon$ is the privacy loss for the randomized algorithm, and
$\delta$ is the privacy cutoff, the probability that the model does not
preserve privacy. Differential privacy gives an upper-bound on the privacy
loss and ($\epsilon,\delta)$-differential privacy guarantees a privacy budget
of $\epsilon$.

###  2.2 Differential Privacy in Machine Learning

For a machine learning context, the randomized algorithm, $\mathcal{M}$, can
be cast as a training algorithm, which takes in a training dataset, $x\in X$,
and leads a model, $\theta\in\Theta$ [8].

The most basic technique to ensure DP is the _Gaussian Mechanism_ as defined
in [15, 17], which is applied to machine learning in the following way. The
gradient for all batches, $B$, is uniformly clipped to a length of $S$ and
Gaussian noise is added to each element of the gradient, with a variance of
$\sigma^{2}S^{2}$, where $\sigma$ is the _noise multiplier_. Higher levels of
$\sigma$ produce less accurate yet more secure classifiers.
$\mathbf{g}_{B}\leftarrow\left[\mathbf{g}_{B}*\mathrm{min}\left(1,\frac{S}{||\mathbf{g}_{B}||}\right)+\mathcal{N}(0,\sigma^{2}S^{2}I)\right]$.
The gradient contains all the information the classifier will ever see of the
training data, so by manipulating that, one achieves privacy guarantees on the
model. By clipping the batch loss gradient, one reduces the impact a given set
of training datum can have on the training of the model parameters. The
hyperparamters associated with the _Gaussian mechanism_ are the noise
multiplier, $\sigma$, and the $\ell_{2}$-norm cutoff, $S$ [18].

More information about the Gaussian mechanism as applied to quantum machine
learning can be found in [12]. This modification to the optimizer algorithm
can be applied to any ML optimizer (SGD, Adam, RMSprop, etc.).

Nicholas Papernot et al. employed a new technique to achieve differential
privacy, in their models, called _Privacy-aAggregation of Teacher Ensembles_
(PATE). Their method builds off of the work on knowledge aggregation and
transfer [19, 20, 21]. First an ensemble of models, known as the _teachers_ ,
are trained on disjoint private data. Then a _student_ model trains with
public data, which is labeled by the output of the _teacher ensemble_. The
_teacher ensemble_ labels the data according to a noisy argmax voting system.
PATE can provide privacy estimates that rival privacy-preserving techniques
employed in real-life situations, such as Google’s RAPPOR [22]. Papernot et
al. achieved an accuracy of 98% for classifying MNIST data with
$(2.04,10^{-5})$-differential privacy, which was a tighter privacy bound than
Abadi et al. MNIST classification [23].

First, the training data $X$ and labels $Y$ are split into $n$ disjoint sets.
Each teacher is trained on one of the disjoint subsets, $(X_{i},Y_{i})$ such
that $(X,Y)=\bigsqcup_{i}^{n}\\{(X_{i},Y_{i})\\}$ to produce a model
characterized by parameters $\phi_{i}$. The function $f_{\phi_{i}}(x)$ then
predicts the label of the input data $x$. Next, the ensemble of teachers is
queried about non-sensitive data, which aggregates the individual teacher
predictions.

The aggregation of the _teacher ensembles’_ predictions forms the backbone of
the PATE algorithm, by providing clear privacy guarantees. Consider a
classification task with $m$ different labels
$f(x)=\operatorname*{arg\,max}_{j\in[m]}\biggl{\\{}n_{j}(x)+Lap\biggl{(}\frac{1}{\gamma}\biggr{)}\biggr{\\}}$
where $n_{j}(x)=|\\{i:i\in[n],f_{\phi_{i}}(x)=j\\}|$, or in other words,
$n_{j}(x)$ is the number of teachers that predict label $j$ given an input
$x$. $Lap(b)$ is a Laplacian distribution with location $0$ and scale $b$, and
$\gamma$ is a privacy parameter [9]. The addition of Laplacian noise
introduces a privacy guarantee, by providing ambiguity in the number of
teacher models that voted for a given label prediction.

Making predictions directly from $f$ would limit our privacy guarantee since
our privacy loss would increase with each query. Additionally, ‘white-box’
access breaks any claims of privacy on $f$. This reasoning provides a
motivation to use the _teacher ensemble_ to train another model, the
_student_.

The _student_ model is released to the public, and it has a well-defined
privacy loss because the student needs to only query the _teacher ensemble_ a
finite number of times during training. Thus, the privacy loss does not
increase as end-users query the _student_ model [9].

Following Papernot et al., this study will use the momentum-accountant
developed in Abadi et al. in 2016 [23] to calculate the privacy loss of the
_student_ model. One could calculate the privacy loss from each label query
and then use Dwork’s _Composition Theorem_ to calculate a total bound on the
privacy loss, but this bound is too loose to be useful. Therefore, the
momentum-accountant is used.

PATE has huge utility since there are no assumptions about the workings of the
teachers or students. This is the first study to apply PATE to variational
quantum classifiers.

##  3 Quantum PATE (qPATE)

We propose a hybrid quantum-classical framework that will train a _student_
classifier using a private-aggregation of _teacher_ classifiers. A hybrid
quantum-classical framework means that a quantum measurement is performed on a
state generated by a quantum circuit. The quantum circuit output is compared
to the ground truth on a classical computer, which will calculate the
difference between the two using a _loss function_. The gradient of the loss
function, with respect to the parameters of the quantum circuit, determines
the direction that the parameters should evolve to reduce the loss and improve
the accuracy of the machine. After updating the parameters in the circuit
according to the loss gradient, the process repeats in an iterative fashion,
_training_ the quantum circuit to replicate the desired function. Unlike
Watkins et al. [12], this study does not alter the gradients in the update-
parameter step to achieve privacy guarantees. Instead, the labels in the
training set of the student model are generated by the privacy-preserving
assembly of teacher models. The ‘fuzziness’ in the truth of the training
labels creates an assurance of privacy.

##  4 Variational Quantum Circuits

Modern deep learning relies on gradient-based optimization, while Quantum
Machine Learning (QML) employs the parameter-shift rule for quantum model
gradients [24, 25]. VQCs are fundamental for trainable quantum classifiers,
with adaptable parameters updated during training [25, 26]. VQCs are resilient
to noise, making them valuable for NISQ devices [27], and have been used for
computing chemical ground states [28]. The generic VQC consists of
$R_{y}(\arctan(x_{i}))$ and $R_{z}(\arctan(x_{i}^{2}))$ rotations for
_variational encoding_ and general single-qubit unitary gate
$R(\alpha,\beta,\gamma)$ as the trainable components. CNOT gates are employed
to establish qubit entanglement. The circuit’s ultimate output is represented
by the $\sigma_{z}$ measurement outcome

Our motivation arises from compelling evidence that hybrid VQC-deep neural
networks (DNNs) outperform DNNs in terms of efficiency and convergence [29,
30, 31, 32]. Recent studies also show quantum circuits achieving faster
learning and higher accuracy than classical counterparts, as in quantum CNN
(QCNN) [33, 34], quantum long short-term memory (QLSTM) networks [35] and
natural language processing [36], highlighting their potential in advancing
machine learning.

##  5 Experiment Methodology

This study demonstrates a privacy-preserving hybrid classical-quantum
classifier using the PATE technique developed by Papernot et al. [9]. We
illustrate the efficacy of PATE with hybrid VQC-DNNs (quantum PATE) by
comparing it to PATE with DNNs (classical PATE). A binary classification of
MNIST digits is used as a benchmark for quantum PATE, which was also used in
Watkins et al. [12] in their study of privacy-preserving classifications with
quantum circuits.

###  5.1 MNIST Binary Classification

MNIST is a representative benchmark dataset in deep learning and quantum
machine learning studies [37]. MNIST includes $28\times 28$ pixel black-white
images of handwritten digits. Because of the computational complexity of
simulating large quantum systems, the MNIST classification task is reduced to
a binary classification of distinguishing the handwritten digits of ‘0’ and
‘1.’ The original inputs are padded with additional zeros to be images of size
$32\times 32$.

###  5.2 Model Architecture and Hyperparameters

As part of the investigation into differentially private QML, classical PATE
and quantum PATE are compared. In classical PATE, both teacher and student
networks are made out of four convolution blocks. As depicted in Figure 1, the
former two convolution blocks consist of a 3x3 convolutional layer, batch
normalization, and ReLU activation, and the latter two convolution blocks
consist of a 1x1 convolutional layer, batch normalization, and ReLU
activation. On the other hand, in quantum PATE (Figure 2), both teacher and
student networks are made out of two convolution blocks and an additional VQC
block. Two convolution blocks consist of a 3x3 convolutional layer, batch
normalization, and leaky ReLU activation.

Fig. 1: Classical PATE network architecture. Classical PATE uses four
convolution blocks.

Fig. 2: Quantum PATE network architecture. Quantum PATE uses two convolution
blocks with the additional VQC block.

The VQC block within quantum PATE has two subcircuits for variational
encoding, following two subcircuits for angle encoding [38, 39]. After
$512$-dimensional latent embeddings from classical convolution blocks are
reduced to $10$-dimensional latent embeddings, two angle encoding subcircuits
encode these latent embeddings into a $10$-qubit state. Subsequently, the
outputs from angle encoding pass through two variational encoding subcircuits,
each consisting of a rotation gate and a CNOT (controlled-NOT) gate as
described in Figure 3. Since the rotation gate assigned to each qubit is
parameterized with three parameters, and the subcircuit consists of 10 qubits,
there are $3\times 10\times 2=60$ parameters in the whole circuit.

Fig. 3: VQC block for MNIST classification. The VQC block encodes latent
embeddings from convolution blocks within quantum PATE into quantum states
represented by $10$-qubits. $U(\mathbf{x})$ denotes the quantum algorithm for
angle encoding. $\phi_{i}$, $\theta_{i}$, and $\omega_{i}$ are the parameters
to optimize. The dashed box denotes one subcircuit of the VQC block that is
repeated two times. The dial to the far right represents that the circuit has
two outputs. The expectation of $\sigma_{z}$ is measured on two qubits.

The experiments are characterized by the hyperparameters of the neural network
training process: the optimizer, number of epochs, number of training samples,
learning rate, batch size, and weight penalty. When differentially private
optimizers are used, the additional hyperparameters needed are the noise
multiplier and $\epsilon$. After preliminary experiments, the AdamW optimizer
was selected for use in all of the experiments presented in this paper. Most
of the model’s hyperparameters were the same for both classical PATE and
quantum PATE; the learning rate was set to $0.001$, while the number of train,
validation, and test datasets were 1,000, 100, and 100, randomly assigned to
each teacher and student networks respectively. The batch size was $64$ with a
weight decay $10^{-4}$.

An $\epsilon$ was calculated from the DP hyperparameters, $S,\sigma$. Because
all tasks were classifications, cross-entropy was used as the loss function
for all training. According to [16, 15], the probability of breaking
$\epsilon$-DP should be $\delta\sim\mathcal{O}(1/n)$ for $n$ samples. A
$\delta$ larger than $1/n$ always will be able to satisfy DP simply by
releasing $n\delta$ complete records. Therefore, $\epsilon$ was determined by
hyperparameter choice, and $\delta$ was set to be $10^{-5}$ for the entire
study. Various noise multipliers were employed for comparing differentially
private networks, with the “varies” noise parameter indicating the use of
multiple noise values in both classical PATE and quantum PATE experiments.
$\epsilon$ also varied across DP experiments as it directly depends on the
noise multiplier.

##  6 Results

$\epsilon$ | $\delta$ | classical PATE | quantum PATE
---|---|---|---
0.01 | $10^{-5}$ | 0.534 $\pm$ 0.0992 |  0.688 $\pm$ 0.0163
0.1 | $10^{-5}$ | 0.985 $\pm$ 0.0215 |  0.992 $\pm$ 0.0098
1.0 | $10^{-5}$ |  0.997 $\pm$ 0.0046 | 0.99 $\pm$ 0.0134
10.0 | $10^{-5}$ |  0.997 $\pm$ 0.0046 | 0.991 $\pm$ 0.0137
Table 1: Results from binary MNIST classification. Accuracies of classical
PATE and quantum PATE after 20 epochs. The private quantum classifier is more
accurate and successful for $\epsilon$ between 0.01 and 0.1. The number of
teachers is set as 4.

Fig. 4: Accuracy vs. Epsilon Plot for 4 Teachers in classical PATE and quantum
PATE. We averaged the results of 10 experiments, and the error bar denotes the
standard deviation. (A) is the result of 1 epoch training, (B) is the result
of 10 epoch training, (C) is the result of 20 epoch training.

Since $\epsilon$ measures the amount of privacy loss at a differential change
in data, the lower the $\epsilon$ value, the stronger the privacy protection
is. As seen in Table 1, quantum PATE showed 28.84% higher performance than
classical PATE at $\epsilon$ value of 0.01. While at $\epsilon$ value of
greater than 0.1, the prediction performance of both classical PATE and
quantum PATE was comparable.

Furthermore, we observed that quantum PATE achieved quite a good test
performance only after 1 epoch of training at various $\epsilon$ values
(Figure 4). After longer epochs of training, the accuracy of classical PATE
and quantum PATE became comparable at high $\epsilon$ values, i.e., 0.1, 1,
10. However, at a low $\epsilon$ value, quantum PATE still outperforms
classical PATE. These results indicate that quantum PATE outperformed
classical PATE in terms of accuracy and convergence while preserving high
privacy protection compared to classical PATE.

##  7 Conclusion

This is the first implementation of a hybrid quantum-classical classifier that
preserves privacy through privacy-aggregation of teacher ensembles (PATE).
Furthermore, our results demonstrate a quantum advantage since the models are
of the same order of complexity.

On the other hand, we highlight the potential of a hybrid quantum-classical
framework in implementing accurate and privacy-protecting machine learning
algorithms. Though it’s crucial to attain high prediction accuracy while
keeping $\epsilon$ values low to ensure privacy protection, the trade-off
between prediction accuracy and $\epsilon$ value makes it hard to achieve
these goals simultaneously. Nevertheless, our results conclusively demonstrate
that a hybrid approach, combining Variational Quantum Circuits (VQC) with Deep
Neural Networks (DNNs), has the capability to significantly enhance prediction
accuracy at low $\epsilon$ values when compared to using only DNNs. In future
works, we will test that our results can be generalized to more complex image
classification tasks, e.g., CIFAR10, ImageNet21k.

## References

  * [1] Ilya Sutskever, Oriol Vinyals, and Quoc V Le,  “Sequence to sequence learning with neural networks,”  in Advances in neural information processing systems, 2014, pp. 3104–3112.
  * [2] Mei Wang and Weihong Deng,  “Deep face recognition: A survey,”  Neurocomputing, vol. 429, pp. 215–244, mar 2021.
  * [3] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich,  “Going deeper with convolutions,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
  * [4] Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis,  “Deep Learning for Computer Vision: A Brief Review,”  Computational Intelligence and Neuroscience, vol. 2018, pp. 1–13, 2018.
  * [5] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun, Eds., 2015.
  * [6] Babak Alipanahi, Andrew Delong, Matthew Weirauch, and Brendan Frey,  “Predicting the sequence specificities of dna- and rna-binding proteins by deep learning,”  Nature biotechnology, vol. 33, 07 2015.
  * [7] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel,  “Extracting training data from large language models,” 2020.
  * [8] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang,  “Deep learning with differential privacy,”  Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Oct 2016.
  * [9] Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar,  “Semi-supervised knowledge transfer for deep learning from private training data,” 2016.
  * [10] Sergey Bravyi, Oliver Dial, Jay M Gambetta, Darío Gil, and Zaira Nazario,  “The future of quantum computing with superconducting qubits,”  Journal of Applied Physics, vol. 132, no. 16, 2022.
  * [11] Wooyeong Song, Youngrong Lim, Hyukjoon Kwon, Gerardo Adesso, Marcin Wieśniak, Marcin Pawłowski, Jaewan Kim, and Jeongho Bang,  “Quantum secure learning with classical samples,”  Physical Review A, vol. 103, no. 4, Apr 2021.
  * [12] William M Watkins, Samuel Yen-Chi Chen, and Shinjae Yoo,  “Quantum machine learning with differential privacy,”  Scientific Reports, vol. 13, no. 1, pp. 2453, 2023.
  * [13] Cynthia Dwork,  “Differential privacy,”  in Automata, Languages and Programming, Michele Bugliesi, Bart Preneel, Vladimiro Sassone, and Ingo Wegener, Eds., Berlin, Heidelberg, 2006, pp. 1–12, Springer Berlin Heidelberg.
  * [14] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Rich Zemel,  “Fairness through awareness,” 2011.
  * [15] Cynthnia Dwork and Aaron Roth,  “The algorithmic foundations of diﬀerential privacy,”  Foundations and Trends in Theoretical Computer Science, vol. 9, no. 3,4, pp. 211,407, 2014.
  * [16] Zhanglong Ji, Zachary C. Lipton, and Charles Elkan,  “Differential privacy and machine learning: a survey and review,”  arXiv:1607.00133v2 [stat.ML], 2014.
  * [17] H. Brendan McMahan and Galen Andrew,  “A general approach to adding differential privacy to iterative training procedures,”  CoRR, vol. abs/1812.06210, 2018.
  * [18] Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov,  “Differential privacy has disparate impact on model accuracy,”  33rd Conference on Neural Information Processing Systems, 2019.
  * [19] Leo Breiman,  “Bagging predictors,”  Machine Learning, pp. 123–140, 1996.
  * [20] Manas Pathak, Shantanu Rane, and Bhiksha Raj,  “Multiparty differential privacy via aggregation of locally trained classifiers,”  in Advances in Neural Information Processing Systems, J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta, Eds. 2010, vol. 23, Curran Associates, Inc.
  * [21] Jihun Hamm, Paul Cao, and Mikhail Belkin,  “Learning privately from multiparty data,” 2016.
  * [22] Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova,  “RAPPOR,”  in Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security. nov 2014, ACM.
  * [23] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang,  “Deep learning with differential privacy,”  in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. oct 2016, ACM.
  * [24] Maria Schuld, Ville Bergholm, Christian Gogolin, Josh Izaac, and Nathan Killoran,  “Evaluating analytic gradients on quantum hardware,”  Physical Review A, vol. 99, no. 3, pp. 032331, 2019.
  * [25] Kosuke Mitarai, Makoto Negoro, Masahiro Kitagawa, and Keisuke Fujii,  “Quantum circuit learning,”  Physical Review A, vol. 98, no. 3, pp. 032309, 2018.
  * [26] Maria Schuld, Alex Bocharov, Krysta M. Svore, and Nathan Wiebe,  “Circuit-centric quantum classifiers,”  Physical Review A, vol. 101, no. 3, Mar 2020.
  * [27] M Cerezo, Andrew Arrasmith, Ryan Babbush, Simon C Benjamin, Suguru Endo, Keisuke Fujii, Jarrod R McClean, Kosuke Mitarai, Xiao Yuan, Lukasz Cincio, et al.,  “Variational quantum algorithms,”  arXiv preprint arXiv:2012.09265, 2020.
  * [28] Alberto Peruzzo, Jarrod McClean, Peter Shadbolt, Man-Hong Yung, Xiao-Qi Zhou, Peter J Love, Alán Aspuru-Guzik, and Jeremy L O’brien,  “A variational eigenvalue solver on a photonic quantum processor,”  Nature communications, vol. 5, pp. 4213, 2014.
  * [29] Sukin Sim, Peter D Johnson, and Alán Aspuru-Guzik,  “Expressibility and entangling capability of parameterized quantum circuits for hybrid quantum-classical algorithms,”  Advanced Quantum Technologies, vol. 2, no. 12, pp. 1900070, 2019.
  * [30] Trevor Lanting, Anthony J Przybysz, A Yu Smirnov, Federico M Spedalieri, Mohammad H Amin, Andrew J Berkley, Richard Harris, Fabio Altomare, Sergio Boixo, Paul Bunyk, et al.,  “Entanglement in a quantum annealing processor,”  Physical Review X, vol. 4, no. 2, pp. 021041, 2014.
  * [31] Yuxuan Du, Min-Hsiu Hsieh, Tongliang Liu, and Dacheng Tao,  “The expressive power of parameterized quantum circuits,”  arXiv preprint arXiv:1810.11922, 2018.
  * [32] Amira Abbas, David Sutter, Christa Zoufal, Aurélien Lucchi, Alessio Figalli, and Stefan Woerner,  “The power of quantum neural networks,”  arXiv preprint arXiv:2011.00027, 2020.
  * [33] Samuel Yen-Chi Chen, Tzu-Chieh Wei, Chao Zhang, Haiwang Yu, and Shinjae Yoo,  “Quantum convolutional neural networks for high energy physics data analysis,”  Physical Review Research, vol. 4, no. 1, pp. 013231, 2022.
  * [34] Samuel Yen-Chi Chen, Tzu-Chieh Wei, Chao Zhang, Haiwang Yu, and Shinjae Yoo,  “Hybrid quantum-classical graph convolutional network,”  arXiv preprint arXiv:2101.06189, 2021.
  * [35] Samuel Yen-Chi Chen, Shinjae Yoo, and Yao-Lung L Fang,  “Quantum long short-term memory,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 8622–8626.
  * [36] Chao-Han Huck Yang, Jun Qi, Samuel Yen-Chi Chen, Pin-Yu Chen, Sabato Marco Siniscalchi, Xiaoli Ma, and Chin-Hui Lee,  “Decentralizing feature extraction with quantum convolutional neural network for automatic speech recognition,”  arXiv preprint arXiv:2010.13309, 2020.
  * [37] Li Deng,  “The mnist database of handwritten digit images for machine learning research,”  IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
  * [38] Mikko Möttönen, Juha J Vartiainen, Ville Bergholm, and Martti M Salomaa,  “Transformation of quantum states using uniformly controlled rotations,”  Quant. Inf. Comp., vol. 5, no. 6, pp. 467–473, 2005.
  * [39] Maria Schuld and Francesco Petruccione,  “Information encoding,”  in Supervised Learning with Quantum Computers, pp. 139–171. Springer International Publishing, Cham, 2018.