# Domain Generalization Using Large Pre-
trained Models with Mixture-of-Adapters

Gyuseong Lee, Wooseok Jang††footnotemark: , Jin Hyeon Kim, Jaewoo Jung,
Seungryong Kim
Department of Computer Science and Engineering
Korea University, Seoul, Korea
<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Learning a robust vision model despite large distribution shift is essential
for model deployment in real-world settings. Especially, domain generalization
(DG) algorithm aims to maintain the performance of a trained model on
different distributions which were not seen during training. One of the most
effective methods has been leveraging the already learned rich knowledge of
large pretrained models. However, naively fine-tuning large models to DG tasks
is often practically infeasible due to memory limitations, extensive time
requirements for training, and the risk of learned knowledge deterioration.
Recently, parameter-efficient fine-tuning (PEFT) methods have been proposed to
reduce the high computational cost during training and efficiently adapt large
models to downstream tasks. In this work, for the first time, we find that the
use of adapters in PEFT methods not only reduce high computational cost during
training but also serve as an effective regularizer for DG tasks.
Surprisingly, a naive adapter implementation for large models achieve superior
performance on common datasets. However, in situations of large distribution
shifts, additional factors such as optimal amount of regularization due to the
strength of distribution shifts should be considered for a sophisticated
adapter implementation. To address this, we propose a mixture-of-expert based
adapter fine-tuning method, dubbed as mixture-of-adapters (MoA). Specifically,
we employ multiple adapters that have varying capacities, and by using
learnable routers, we allocate each token to a proper adapter. By using both
PEFT and MoA methods, we effectively alleviate the performance deterioration
caused by distribution shifts and achieve state-of-the-art performance on
diverse DG benchmarks.

##  1 Introduction

The goal of domain generalization (DG) is to well predict on domains that were
unavailable during training (a.k.a unseen domains) (Gulrajani & Lopez-Paz,
2020). In DG settings, the model is trained on multiple source domains and
evaluated on one unseen target domain (Zhou et al., 2021; Gulrajani & Lopez-
Paz, 2020; Cha et al., 2021). Unlike domain adaptation setting, domain
generalization is unable to access any information about the target domain.
Therefore, DG algorithms should fully exploit domain invariant features
underlying in the source domain to well predict on the target domain (Seo et
al., 2020; Gulrajani & Lopez-Paz, 2020; Arjovsky et al., 2019).

In recent times, the usage of large pretrained models is gaining popularity in
domain generalization fields (Cha et al., 2022; Mao et al., 2022; Lew et al.,
2023; Li et al., 2023). Since large pretrained models already possess some
extent of domain invariant knowledge (Cha et al., 2022), exploiting this
knowledge into domain generalization has become a popular choice. There are
few studies that have tried to train these models directly with empirical risk
minimization (ERM) (Vapnik, 1998). Angarano et al. (2022) shows that ERM
algorithm performs competitively well when accompanied with proper backbones
like EfficientNet (Tan & Le, 2019), ViT (Dosovitskiy et al., 2020), DeiT
(Touvron et al., 2021), LeViT (Graham et al., 2021) and ConViT (d’Ascoli et
al., 2021). This inspired us to leverage large pretrained models on DG
settings.

However it is widely known that naively fine-tuning a large model is
impractical, not only does it demand a large size VRAM caused by high peak
memory in training phase, but it also requires a significant amount of
training time due to numerous amount of parameters. Additionally, the
pretrained feature extractor can be distorted by overfitting on source domains
(Gao et al., 2021) and its contextual information can be harmed (Kumar et al.,
2022; Mao et al., 2022; Wortsman et al., 2022), which overall results in a
generalization failure to out-of-distribution data. Fig. 1 compares the
accuracy of various fine-tuning methods, including naive full fine-tuning,
linear probing, and other partial fine-tuning methods,which also highlights
their incapability to handle such distribution shifts.

Figure 1: Results on domain generalization benchmark with varying trainable
parameters in ViT-Base (Dosovitskiy et al., 2020) pretrained with CLIP of
OpenAI (Radford et al., 2021). Y axis is accuracy. We use linear probing
(denoted as Linear), bias tuning in attention layer (Bias (MSA)), bias tuning
in both attention and MLP layer (Bias (MSA+MLP)), and full fine-tuning to show
the accuracy change according to the trainable parameter change when using
PEFT methods with large models. OH, TI, DN denotes OfficeHome (Venkateswara et
al., 2017), TerraIncognita (Beery et al., 2018), and DomainNet (Peng et al.,
2019), respectively.

In this paper, for the first time, we propose to adopt parameter-efficient
fine-tuning (PEFT) methods in the context of domain generalization, expanding
its usage beyond its traditional domains on transfer learning. PEFT methods
(Houlsby et al., 2019; Karimi Mahabadi et al., 2021; He et al., 2022; Zaken et
al., 2021; Hu et al., 2021; Ryu, 2023; Jia et al., 2022; Chen et al., 2022; Li
& Liang, 2021; Sung et al., 2022a) mitigate the high cost of full fine-tuning
of large pretrained models and aims to reach or exceed the performance of full
fine-tuning or zero-shot performance by tuning only some parts of the model
(Zaken et al., 2021) or by employing a small number of external learnable
parameters (Hu et al., 2021; Ryu, 2023; Houlsby et al., 2019; Karimi Mahabadi
et al., 2021; He et al., 2022; Jia et al., 2022). We compare various aspects
of different trainable parameter settings (e.g., full fine-tuning, and adapter
fine-tuning) and empirically verify that PEFT methods can act as an effective
regularization during the training process and show that it can largely
outperform a fully fine-tuned model and reach comparable performance with
recent state-of-the-art methods (Cha et al., 2021; 2022; Arpit et al., 2022;
Li et al., 2023) in DG settings. We also discover that the optimal strength of
PEFT regularization differs, depending on the amount of a distribution shift.

As shown in Fig. 1, linear probing (Linear), bias tuning in attention layer
(Bias (MSA)), bias tuning in both attention and MLP layer (Bias (MSA+MLP))
show comparable performance on PACS (Li et al., 2017) and VLCS Fang et al.
(2013) dataset. However on datasets with large distribution shifts, say
TerraIncognita (Beery et al., 2018), Bias (MSA+MLP) shows significant
improvement compared to the Bias (MSA) and Linear method. This shows that
determining the proper amount of regularization by adjusting the trainable
parameter is crucial to deal with different distribution shift. To handle
this, we introduce mixture-of-expert based adapter architecture, called
mixture-of-adapters (MoAs), that adequately manipulate the magnitude of
regularization by employing adapters that have different capacity and routing
each token to proper adapters. By doing so, we can further improve the
performance on DG tasks by employing MoAs and learnable routers to handle the
different intensity of distribution shift among various datasets and achieve
state-of-the-art performance on DG benchmark.

##  2 Related Work

#### Domain generalization.

For the past decade, numerous learning methods on how to learn domain
invariant representations have been proposed in the domain generalization
field. Empirical risk minimization (ERM) (Vapnik, 1998) which is one of the
most simplest approaches, just minimizes the loss on each domains and trains
the model. In DomainBed (Gulrajani & Lopez-Paz, 2020), ERM still remains
effective within the benchmark’s restricted hyperparameter search space and
model selection methods. DomainBed tested many DG methods such as IRM
(Arjovsky et al., 2019), GroupDRO (Sagawa et al., 2019), MixUp (Xu et al.,
2020), DANN (Ganin et al., 2016), and CORAL (Sun & Saenko, 2016) in a unified
and contained experimental setup. SWAD (Cha et al., 2021) explored the
relationship between flat loss surfaces and DG performance, achieving superior
results. Ensemble-of-Averages (EoA) (Arpit et al., 2022) perform model
averaging in training time and ensemble them in test time. Current research is
focused on leveraging knowledge from large pretrained models, with MIRO (Cha
et al., 2022) introducing an oracle model approximated by a large pretrained
model to maximize the mutual information between the target model. Recently,
methods that ensemble diverse models show remarkable performance in the DG
benchmark. SIMPLE (Li et al., 2023) utilizes many different pretrained models
from a ModelPool and extracts outputs from the frozen pretrained models, and
trains a shallow dispatcher using these outputs.

#### Parameter efficient fine-tuning.

Leveraging large pretrained models for specific tasks involves fine-tuning,
but fine-tuning all parameters is impractical. Recent approaches such as
Parameter-efficient Fine-Tuning (PEFT) (Houlsby et al., 2019; He et al., 2021;
Paul et al., 2022), focus on efficient fine-tuning by freezing most of the
model parameters and optimizing only a few for a given task. Many successful
PEFT approaches (Zaken et al., 2021; Hu et al., 2021; Ryu, 2023; Gao et al.,
2020; He et al., 2021; Jia et al., 2022; Lester et al., 2021; Sung et al.,
2022a; b; Zhang et al., 2022; 2023) adopt popular pretrained models to various
downstream tasks. Among these methods, the use of adapters (Hu et al., 2021;
Karimi Mahabadi et al., 2021; He et al., 2022) are adopted because of its high
performance and efficient computation cost. Adapters are small modules trained
on specific tasks which are inserted between network layers, where all the
layers except for the adapters are frozen during training. LoRA (Hu et al.,
2021), Compacter (Karimi Mahabadi et al., 2021) and KAdaptation (He et al.,
2022) greatly reduced the number of trainable parameters by attaching a low-
rank hypercomplex adapter layers in the transformer model and by decomposing
the updated weight matrix into low rank matrices respectively.

#### Mixture-of-Experts.

Mixture-of-Experts (MoEs) models are proposed to improve model performance by
incorporating multiple subsets of parameters called ‘experts’ with routing
algorithms conditioned by its input (Jacobs et al., 1991; Jordan & Jacobs,
1994; Eigen et al., 2013). Evolving from this, a type of model called sparse
MoEs has become popular in both NLP (Shazeer et al., 2017; Lepikhin et al.,
2020; Zoph et al., 2022; Fedus et al., 2022; Du et al., 2022) and vision
(Ahmed et al., 2016; Gross et al., 2017; Yang et al., 2019; Wang et al., 2020;
Riquelme et al., 2021) tasks lately. This is due to its capability to enhance
model capacity while simultaneously reducing the substantial increase in the
computational resources required for training. In the field of domain
generalization, Li et al. (2022) incorporated MoE design into DG tasks, and
proposed a Generalizable Mixture-of-Experts (GMoE) architecture to effectively
handle distribution shifts.

##  3 Preliminaries

###  3.1 Domain generalization

Let us denote the set of training domains as
$\mathcal{D}=\\{\mathcal{D}^{i}\\}_{i=1}^{K}$ where $K$ implies the total
number of training domains, and $\mathcal{D}_{i}$ is the distribution over the
input space. Also, define the set of target domains as
$\mathcal{T}=\\{\mathcal{T}^{i}\\}_{i=1}^{K^{\prime}}$ where $K^{\prime}$
denotes the total number of training domains. The training dataset is composed
of $n_{\mathcal{D}^{i}}$ data points denoted as
$(x^{i}_{j},y^{i}_{j})_{j=1}^{n_{\mathcal{D}^{i}}}\sim\mathcal{D}^{i}$ where
$x$ is the input and $y$ is the target label for each training domain. In DG
settings, the goal is to find the model parameter $\theta$ of a classifier
$f_{\theta}$ which generalizes well on both $\mathcal{D}$ and $\mathcal{T}$.
To be more precise, for the ERM algorithm, we define

$\mathcal{E}_{\mathcal{D}}(\theta)={\frac{1}{K}}\sum^{K}_{i=1}\mathbb{E}_{(x^{i},y^{i})\sim\mathcal{D}^{i}}[l(f_{\theta}(x^{i}),y^{i})]$
(1)

where $l(\cdot,\cdot)$ denote a loss function and define
$\mathcal{E}_{\mathcal{T}}$ in the similar manner. We minimize empirical risk
$\hat{\mathcal{E}}_{\mathcal{D}}(\theta)={\frac{1}{K}}\sum^{K}_{i=1}{\frac{1}{n_{\mathcal{D}^{i}}}}\sum^{n_{\mathcal{D}^{i}}}_{j=1}[l(f_{\theta}(x^{i}_{j}),y^{i}_{j})]$
during training and expect the optimal parameter
$\hat{\theta}=\arg\min_{\theta}\hat{\mathcal{E}}_{\mathcal{D}}(\theta)$ to be
optimal for $\mathcal{E}_{\mathcal{T}}(\theta)$.

###  3.2 Parameter efficient adapter

Parameter-efficient fine-tuning (PEFT) efficiently handles enormous pretrained
models to a downstream task and achieves significant performance unlike the
direct fine-tuning methods where high cost of memory and computation are
required. Previous methods of fine-tuning large models have been accomplished
by partial fine-tuning (Touvron et al., 2022) methods. However, recent works
(Houlsby et al., 2019; Hu et al., 2021; Karimi Mahabadi et al., 2021) reveal
that employing a learnable layer in the frozen pretrained weights can surpass
the performance of the partial fine-tuning method.

More specifically, Aghajanyan et al. (2020) verifies that dense neural network
layers with full-rank matrices can be reduced to lower-rank subspaces. From
this finding, Hu et al. (2021) constrains the updated weight
$\Delta\mathbf{W}=\mathbf{BA}\in\mathbb{R}^{k\times d}$ to have a low
intrinsic rank which can be expressed as
$\mathbf{h}=\mathbf{W_{0}}\mathbf{x}+\Delta\mathbf{W}\mathbf{x}=\mathbf{W_{0}}\mathbf{x}+\mathbf{BA}\mathbf{x}$,
where $\mathbf{x}$ denotes input sequences of each module,
$\mathbf{W_{0}}\in\mathbb{R}^{k\times d}$ denotes the frozen weight matrix of
a pretrained model, and $\mathbf{A}\in\mathbb{R}^{k\times r}$, and
$\mathbf{B}\in\mathbb{R}^{r\times d}$ $(r<k,d)$ are the trainable parameters
during training.

In addition, KAdaptation (He et al., 2022) and Compacter (Karimi Mahabadi et
al., 2021) both exploit Kronecker products to decompose the weight matrices
into reduce trainable parameters. The Kroneker product between matrix
$\mathbf{A}\in\mathbb{R}^{m\times n}$ and $\mathbf{B}\in\mathbb{R}^{p\times
q}$ is denoted by $\mathbf{A}\otimes\mathbf{B}\in\mathbb{R}^{mp\times nq}$ and
can be expressed as the following:

$\mathbf{A}\otimes\mathbf{B}=\begin{pmatrix}a_{11}\mathbf{B}&\textbf{\ldots}&a_{1n}\mathbf{B}\\\
\vdots&\ddots&\vdots\\\ a_{m1}\mathbf{B}&\textbf{\ldots}&a_{mm}\mathbf{B}\\\
\end{pmatrix}$ (2)

where $a_{ij}$ indicates the element in the $i$-th row and the $j$-th column
of $\mathbf{A}$. It decomposes the update weight matrix
$\Delta\mathbf{W}=\sum_{i=1}^{t}\mathbf{A}_{i}\otimes\mathbf{B}_{i}\in\mathbb{R}^{k\times
d}$ where $t$ is a hyperparameter that decides the number of Kronecker
products. Furthermore, $\mathbf{W}$ can be expressed as the following:
$\mathbf{W}=\sum_{i=1}^{t}\mathbf{A}_{i}\otimes\mathbf{B}_{i}=\sum_{i=1}^{n}\mathbf{A}_{i}\otimes(\mathbf{u}_{i}\mathbf{v}_{i}^{\top})$
where slow-weights $\mathbf{A}_{i}\in\mathbb{R}^{t\times t}$ is shared across
all layers, and fast-weights
$\mathbf{B}_{i}\in\mathbb{R}^{\frac{k}{t}\times\frac{d}{t}}$ is decomposed
into low-rank matrices $\mathbf{u}_{i}\in\mathbb{R}^{\frac{k}{t}\times r}$ and
$\mathbf{v}_{i}\in\mathbb{R}^{r\times\frac{d}{t}}$ with $i\in\\{1,\dots,t\\}$.
Compacter decomposes the weight matrix to the additional adapter layer whereas
KAdaptation decomposes the update matrix $\Delta\mathbf{W}$ in the original
layer.

##  4 Analysis of Adapters for Domain Generalization

(a) Fully fine-tuned

(b) LoRA

(c) KAdaptation

(d) KMoA (Ours)

Figure 2: Flatness comparison of loss surfaces from models trained with full
fine-tuning, LoRA, KAdaptation, and KAdaptation with Mixture-of-Adapter
(denoted as KMoA) on PACS dataset Li et al. (2017). All visualizations are
computed from test environment 0 (Art) domain.

In this section, we investigate how adapters affect loss landscape flatness by
analyzing the maximum Hessian eigenvalue spectra and loss landscapes of
various trained models. It is widely known that finding a flat minima during
optimization is closely related to the generalization performance and
robustness of trained models, as documented in prior studies (Izmailov et al.,
2018; Keskar et al., 2016; Garipov et al., 2018; Foret et al., 2020; Cha et
al., 2021; Park & Kim, 2022). Therefore, by comparing loss landscapes along
with maximum Hessian eigenvalues, we can anticipate which model will
generalize better to unseen domains. We show the results of our analysis in
the following sections.

###  4.1 Loss Landscapes

As shown in (Park & Kim, 2022; Garipov et al., 2018; Li et al., 2018) randomly
perturbing the trained weights with random direction vectors, we can obtain
variations of the loss value, which can be used to predict the models
generalization ability. We evaluate on PACS (Li et al., 2017) dataset, one of
the dataset in the DG benchmark (Gulrajani & Lopez-Paz, 2020). As shown in
Fig. 2, compared to the case of full fine-tuning (Fig. 2(a)) where we trained
all of the model parameters, model parameters tuned with LoRA (Hu et al.,
2021; Ryu, 2023) (Fig. 2(b)) and KAdaptation (He et al., 2022) (Fig. 2(c))
displays a more flatter loss surface. Additionally, KAdaptation shows a
flatter loss surface than LoRA. We show further visualizations of the loss
landscape on additional test environments on PACS in Appendix A.1.

###  4.2 Maximum Hessian Eigenvalue Spectra

Figure 3: Hessian max eigenvalue spectra of test environment 0 (Art) in PACS
dataset. KA and KMoA denotes KAdaptation and Mixture-of-Adapter with
KAdaptation method, respectively. In the order of w/o LoRA, w/LoRA, w/KA, and
w/KMoA, the hessian eigenvalue spectrum concentrates to zero and eigenvalues
are suppressed accordingly.

Drawing loss landscapes from the optimal point to random directions sometimes
fail to fully represent the shape of loss surfaces due to the high
dimensionality of the loss surface (Garipov et al., 2018; Li et al., 2018).
Therefore, following Park & Kim (2022) we calculate the top-5 Hessian
eigenvalues and show their spectra.

Analogous to loss landscape results in the previous section, we observe the
same phenomena with the use of adapters, say LoRA and KAdaptation. As depicted
in Fig. 3, the top-5 Hessian eigenvalues are further zero-concentrated than
the fully fine-tuned model. These discrepancy implies that employing adapters
with large pretrained models results in a flatter loss surface around the
optimal point. This is because the Hessian matrix serves as an indicator of
the curvature of the loss surface. Since the flatness of loss surfaces has a
substantial impact on model performance in DG tasks Cha et al. (2021), it can
be concluded that adapter fine-tuning with large pretrained models effectively
facilitate the performance in domain generalization. We show visualizations of
max Hessian eigenvalue spectra on additional test environments on PACS dataset
in Appendix A.2.

###  4.3 Parameter-Efficient Adapter for Domain Generalization

Table 1: Quantitative evaluation on domain generalization datasets with
different PEFT methods. Following DomainBed (Gulrajani & Lopez-Paz, 2020), we
evaluate our algorithm on PACS (Li et al., 2017), VLCS (Fang et al., 2013),
OfficeHome (Venkateswara et al., 2017), TerraIncognita (Beery et al., 2018)
and DomainNet (Peng et al., 2019). For fair comparison, we use training-
domain-validation model selection strategy. Reported performance of each
algorithms are from their papers. We indicate the pretraining method and
dataset as the form of $\mathtt{PRETRAIN}_{\mathtt{DATASET}}$, and IG3B
denotes Instagram-3B dataset from Zellers et al. (2018). Results with our
method are colored in blue.

Algorithm | Architecture | Pretraining | PACS | VLCS | OfficeHome | TerraIn. | DomainNet | Avg. | #Param. | Trainable
---|---|---|---|---|---|---|---|---|---|---
#Param.
MIRO | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ |  97.4 $\pm$ 0.2 | 79.9 $\pm$ 0.6 | 80.4 $\pm$ 0.2 | 58.9 $\pm$ 1.3 | 53.8 $\pm$ 0.1 | 74.1 | 167.2M | 83.6M
SMA | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 95.5 $\pm$ 0.0 | 80.7 $\pm$ 0.1 | 82.0 $\pm$ 0.0 |  59.7 $\pm$ 0.0 | 60.0 $\pm$ 0.0 | 75.6 | 83.6M | 83.6M
MIRO+SWAD | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 96.8 $\pm$ 0.2 | 81.7 $\pm$ 0.1 | 83.3 $\pm$ 0.1 |  64.3 $\pm$ 0.3 | 60.7 $\pm$ 0.0 | 77.3 | 167.2M | 83.6M
Methods with Parameter-Efficient Fine-Tuning (PEFT)
ERM (Baseline) | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 85.8 $\pm$ 2.1 | 78.5 $\pm$ 0.9 | 78.1 $\pm$ 0.8 | 41.0 $\pm$ 1.6 | 52.2 $\pm$0.1 | 67.1 | 85.8M | 85.8M
ERM${}_{\textrm{LoRA},r=2}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 96.4 $\pm$ 0.6 |  82.6 $\pm$ 0.6 |  86.7 $\pm$ 0.3 | 46.1 $\pm$1.7 |  61.5 $\pm$0.1 | 74.7 | 85.9M | 0.11M
ERM${}_{\textrm{KAdaptaion}}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 97.5 $\pm$ 0.1 | 83.0 $\pm$ 0.1 | 90.3 $\pm$ 0.1 | 51.9 $\pm$0.5 | 62.7 $\pm$0.0 | 77.1 | 85.9M | 0.14M

Previously, we demonstrated that models trained with parameter-efficient fine-
tuning methods have the potential to reach a more generalizable optimization
point. Nevertheless, it’s challenging to come to a conclusive decision that a
flatter loss surface directly translates to better generalization performance.
Consequently, in the following section, we conduct a comprehensive evaluation
of various fine-tuning methods and make comparisons between them. Finally, we
discuss about the most effective practical fine-tuning approach and explore
strategies to further enhance performance.

In Tab. 1 we show results from various paper on five datasets, namely PACS (Li
et al., 2017), VLCS (Fang et al., 2013), OfficeHome (Venkateswara et al.,
2017), TerraIncognita Beery et al. (2018) and DomainNet (Peng et al., 2019),
and from the observations above, we evaluate ERM with various PEFT adapter
methods in standard DG benchmark form Gulrajani & Lopez-Paz (2020). Some works
like Cha et al. (2022) report the results of full fine-tuning a ViT-B/16 model
with an ERM algorithm. However just by training a small adapter layer or an
attention layer, we observe a greater performance improvement. Compared to the
methods that use additional regularization or ensembling, our parameter
efficient training approach with naive ERM algorithm reaches comparable
accuracy and even state-of-the-art for the OfficeHome dataset. Additionally
KAdaptation method achieves the best average accuracy of all the methods.
These results validate our observation in Sec. 4.1 and 3, demonstrating that a
proper selection of an adapter method can boost the performance adequately.

##  5 Mixture-of-Adapters (MoA)

In this section we show the effectiveness of our proposed method, mixture-of-
adapter, which can tune large models with low cost and achieve great
performance on DG tasks. As revealed in Li et al. (2022), mixture-of-expert
model architectures with the use of cosine routing is more effective on DG
tasks. Experts also handle some parts of the visual attributes, therefore
reducing incorrect token allocation caused by intra-domain similarities. We
bring it to our adapter-based method, which maintains the computational
efficiency and achieves better results in domain generalization.

(a) Adapter PEFT methods

(b) Mixture-of-Adapter method

Figure 4: Architecture of proposed Mixture-of-Adapters (MoA).
$\mathbf{W}_{0}$, $\mathbf{x}_{\mathrm{in}}$, and $\mathbf{x}_{\mathrm{out}}$
denotes original pretrained weight, input, and output tokens in multi-head
self-attention (MHSA). Adapter in Fig. 4(a) can be any adapter-based PEFT
methods like LoRA, Compacter, KAdatptaion, etc, and Router in Fig. 4(b) can be
linear or cosine router that commonly used in Mixture-of-Expert methods.

As demonstrated in Sec. 1, we utilize adapters to handle various amounts of
distribution shift among the domains. Specifically, adapters, each with
distinct capacities at every MoA layer, are employed to select the top-k
outputs which are averaged and then integrated with the original layer’s
outputs. We manipulate the capacities of each adapter by adjusting its inner
rank. The mixture-of-experts (MoE) layer utilizing the cosine router
$\mathrm{G}(\mathbf{x})$ with embedding $\mathbf{E}$ and adapter
$\mathrm{A}_{r_{i}}$ with inner rank $r_{i}$ can be denoted as:

$\begin{split}f_{\mathrm{MoE}}(\mathbf{x})&=\sum_{i=1}^{N}\mathrm{G}(\mathbf{x})_{i}\mathrm{A}_{r_{i}}(\mathbf{x})\\\
&=\sum_{i=1}^{N}\mathrm{TOP}_{k}\left(\texttt{Softmax}\left(\frac{\mathbf{E}^{\mathrm{T}}\mathbf{W}\mathbf{x}}{{\tau}\|\mathbf{W}\mathbf{x}\|\|\mathbf{E}\|}\right)\right)_{i}\mathrm{A}_{r_{i}}(\mathbf{x})\end{split}$
(3)

where $\mathrm{A}_{r_{i}}(\mathbf{x})$ is the output of an adapter with a
different inner rank $r_{i}$ and $\tau$ is a learnable temperature term of a
softmax layer. In detail, we incorporate adapters in the large pretrained
model by attaching them to attention sub modules. Following the implementation
of traditional mixture-of-expert (MoE) methods, routers are adopted to
dispatch tokens to the appropriate expert adapters. An adapter can be of any
form such as LoRA (Hu et al., 2021), one of the most popular method in PEFT,
Compacter (Karimi Mahabadi et al., 2021), and KAdaptation (He et al., 2022).
In the case of KAdaptation, we choose the inner rank $r_{i}$ as the additional
decomposition of $\mathbf{B}_{\mathrm{i}}$. Visualizations of our architecture
is shown in Fig. 4

##  6 Experiments

#### Experimental details.

We use the standard benchmark DomainBed (Gulrajani & Lopez-Paz, 2020) for
training and evaluating the performance on the domain generalization task.
Following this, we use fixed hyperparameters within the same backbone model
for all the experiments. We train five types of models which are full fine-
tuning, attention tuning, LoRA (Hu et al., 2021), KAdaptation (He et al.,
2022), and Compacter (Karimi Mahabadi et al., 2021). We employ CLIP (Radford
et al., 2021) trained ViT (Dosovitskiy et al., 2020) as our initialization
model because CLIP carries strong zero-shot ability which can be used to
obtain favorable generalization performance. In detail we use OpenCLIP
(Ilharco et al., 2021), an open-source re-implementation of CLIP model with
LAION-2B dataset (Schuhmann et al., 2022), for all our experiments. We
describe additional implementation details in Appendix B.1

#### Evaluation protocols and datasets.

For a fair comparison, we employ DomainBed evaluation protocols (Cha et al.,
2021; Gulrajani & Lopez-Paz, 2020). The following five benchmark datasets are
used: PACS (Li et al., 2017), VLCS (Fang et al., 2013), OfficeHome
(Venkateswara et al., 2017), TerraIncognita (Beery et al., 2018), and
DomainNet (Peng et al., 2019). Using a leave-one-out cross-validation, all
performance scores are evaluated by averaging all the cases that use a single
domain as the target domain and the others as the source domains. Experiment
is repeated three times and 20% percent of source domain data is left out for
validation purposes. Lastly model selection (training-domain validation) and
hyperparameter search procedure is referenced from DomainBed (Gulrajani &
Lopez-Paz, 2020). We perform three runs with different random seeds for each
setting and report their mean and standard deviation to show the training
randomness. In ablation studies, we keep all the random seeds fixed and
conduct the experiment.

###  6.1 Results for mixture-of-adapter

Table 2:  Quantitative evaluation on domain generalization datasets with
Mixture-of-Adapter (MoA) methods. The numbers marked with † are reported
results from Cha et al. (2022). OpenAI denotes the private dataset used to
train CLIP (Radford et al., 2021), and ModelPool-A and B denotes the set of
pretrained models used in Li et al. (2023). Results with ensemble methods are
colored in gray, and results with our method are colored in blue.

Algorithm | Architecture | Pretraining | PACS | VLCS | OfficeHome | TerraIn. | DomainNet | Avg. | #Param. | Trainable
---|---|---|---|---|---|---|---|---|---|---
#Param.
ERM† | ViT-B/16 | CLIP${}_{\textrm{OpenAI}}$ | 83.4 $\pm$ 0.5 | 75.9 $\pm$ 1.3 | 66.4 $\pm$ 0.5 | 35.3 $\pm$ 0.8 | 44.4 $\pm$ 0.6 | 61.1 | 85.8M | 85.8M
MIRO† | ViT-B/16 | CLIP${}_{\textrm{OpenAI}}$ | 95.6 $\pm$ 0.8 | 82.2 $\pm$ 0.3 | 82.5 $\pm$ 0.1 | 54.3 $\pm$ 0.4 | 54.0 $\pm$ 0.3 | 73.7 | 172M | 85.8M
ERM† | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 89.6 $\pm$ 0.4 | 78.6 $\pm$ 0.3 | 71.9 $\pm$ 0.6 | 51.4 $\pm$ 1.8 | 48.5 $\pm$ 0.6 | 68.0 | 83.6M | 83.6M
MIRO† | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 97.4 $\pm$ 0.2 | 79.9 $\pm$ 0.6 | 80.4 $\pm$ 0.2 | 58.9 $\pm$ 1.3 | 53.8 $\pm$ 0.1 | 74.1 | 167.2M | 83.6M
ERM+SWAD† | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 94.7 $\pm$ 0.2 | 79.7 $\pm$ 0.2 | 80.0 $\pm$ 0.1 | 57.9 $\pm$ 0.7 | 53.6 $\pm$ 0.6 | 73.2 | 83.6M | 83.6M
MIRO+SWAD† | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 96.8 $\pm$ 0.2 | 81.7 $\pm$ 0.1 | 83.3 $\pm$ 0.1 | 64.3 $\pm$ 0.3 | 60.7 $\pm$ 0.0 | 77.3 | 167.2M | 83.6M
SMA | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 95.5 $\pm$ 0.0 | 80.7 $\pm$ 0.1 | 82.0 $\pm$ 0.0 | 59.7 $\pm$ 0.0 | 60.0 $\pm$ 0.0 | 75.6 | 83.6M | 83.6M
Methods with Parameter-Efficient Fine-Tuning (PEFT)
ERM (Baseline) | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 85.8 $\pm$ 2.1 | 78.5 $\pm$ 0.9 | 78.1 $\pm$ 0.8 | 41.0 $\pm$ 1.6 | 52.2 $\pm$0.1 | 67.1 | 85.8M | 85.8M
ERM${}_{\textrm{Compacter}}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 94.1 $\pm$ 0.4 | 81.0 $\pm$ 0.5 | 83.0 $\pm$ 0.1 | 35.9 $\pm$0.7 | 56.2 $\pm$1.2 | 70.0 | 85.9M | 0.10M
ERM${}_{\textrm{Attention}}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 93.8 $\pm$ 0.6 | 82.0 $\pm$ 0.3 | 85.9 $\pm$ 0.4 | 51.4 $\pm$0.8 | 57.2 $\pm$0.1 | 74.1 | 85.8M | 28.4M
ERM${}_{\textrm{LoRA},r=2}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 96.4 $\pm$ 0.6 | 82.6 $\pm$ 0.6 | 86.7 $\pm$ 0.3 | 46.1 $\pm$1.7 | 61.5 $\pm$0.1 | 74.7 | 85.9M | 0.11M
ERM${}_{\textrm{KAdaptaion}}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 97.5 $\pm$ 0.1 | 83.0 $\pm$ 0.1 | 90.3 $\pm$ 0.1 | 51.9 $\pm$0.5 |  62.7 $\pm$0.0 | 77.1 | 85.9M | 0.14M
Methods with Mixture-of-Adapter
ERM${}_{\textrm{LoRA-MoA}}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 96.9 $\pm$ 0.3 | 82.8 $\pm$ 0.7 | 89.5 $\pm$ 0.2 | 49.2 $\pm$ 2.4 | 62.2 $\pm$ 0.0 | 75.9 | 87.2M | 1.5M
ERM${}_{\textrm{KAdaptaion-MoA}}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 97.4 $\pm$ 0.2 |  83.1 $\pm$ 0.3 |  90.6 $\pm$ 0.0 | 52.8 $\pm$ 1.4 |  62.7 $\pm$ 0.1 | 77.3 | 87.3M | 1.5M
Methods Using Ensemble
EoA | RegNetY-16GF | SWAG${}_{\textrm{IG3B}}$ | 95.8 | 81.1 | 83.9 | 61.1 | 60.9 | 76.6 |  $>$ 500M | -
SIMPLE | ModelPool-A | ModelPool-A | 88.6 $\pm$ 0.4 | 79.9 $\pm$ 0.5 | 84.6 $\pm$ 0.5 | 57.6 $\pm$ 0.8 | 49.2 $\pm$ 1.1 | 72.0 |  $>$ 1,000M | 0.9M
SIMPLE+ | ModelPool-B | ModelPool-B | 99.0 $\pm$ 0.1 | 82.7 $\pm$ 0.4 | 87.7 $\pm$ 0.4 |  59.0 $\pm$ 0.6 | 61.9 $\pm$ 0.5 | 78.1 |  $>$ 1,000M | 0.9M
ERM${}_{\textrm{KAdaptaion-MoA-Ensemble}}$ | ViT-B/16 | CLIP${}_{\textrm{LAION2B}}$ | 97.6 | 83.4 | 90.9 | 54.3 | 63.1 | 77.9 | 261.3M | -

As described in Sec. 5, we implement Mixture-of-Adapter (MoA) with LoRA (Hu et
al., 2021) and KAdaptation, and report the benchmark result in Table 2. In
addition to the performance gains by applying PEFT, results after employing
MoA are increased consistently at all datasets. In the case of LoRA-MoA,
average performance is increased by 1.2pp compared to the results when using
only LoRA. Also, KAdaptation-MoA method achieves state-of-the-art results on
VLCS, OfficeHome, DomainNet datasets, exceeding the result of the original
KAdaptation by 0.2pp in average, validating the analyses about loss surface in
Fig. 2. Additionally, following Arpit et al. (2022), ensembling the three
weights obtained from different seeds, we further enhance the performace by
0.6pp in average, achieving best results in VLCS, OfficeHome, and DomainNet.

#### Analysis on the router and experts.

(a) Location 100 (class dog)

(b) Location 38 (class cat)

Figure 6: Visualizations of routed indices of each patch in TerraIncognita
(Beery et al., 2018) dataset. Left column is original image, and in right
column we indicate where each patch is routed. Upper and lower images were
took from same location but different time, therefore they have same
background but different object (dog, cat) shape and location.

To understand why using multiple adapters with a router improves performance,
we conduct experiments on what the router see and what each experts learn. To
investigate the role of an adapter, we monitor the routing path of each token
and identify the expert to which it is directed. This visualization is
performed on TerraIncognita dataset. In Fig. 6, the image is divided into
patches and each patch is assigned a number, where the number corresponds to
the routed expert. We also reveal that the router tends to cluster tokens in
areas with semantic information (e.g., object foreground or object outlines).
As an example, consider the dog depicted in Fig. 6(a) and the cat in Fig.
6(b), which exhibit varying positions between the first and second rows.
Images sharing the same location have consistent backgrounds, resulting in
shared expert routing for background tokens. However, objects and their
positions can differ, causing object-related tokens to be routed to different
experts. In conclusion, the importance of routing numerous tokens to their
respective adapters greatly enhances the model’s ability to capture semantic
information in images, thereby enabling the model to effectively navigate
challenging distribution shift scenarios in domain generalization tasks. We
show more visualizations of routed patches in Appendix A.3.

###  6.2 Ablation Study

(a) PACS (Li et al., 2017)

(b) VLCS (Fang et al., 2013)

(c) OfficeHome (Venkateswara et al., 2017)

(d) TerraIncognita (Beery et al., 2018)

(e) DomainNet (Peng et al., 2019)

(f) Averaged over 5 datasets

Figure 8: Performance comparison between original (not fine-tuned) CLIP and
ImageNet fine-tuned CLIP from timm (Wightman, 2019) (denoted as CLIP-FT) on
different fine-tuning strategies. Full, Att., LoRA, and KA denotes full fine-
tuning, attention-only tuning (Touvron et al., 2022), LoRA, and KAdaptation,
respectively.

#### Effect of fine-tuning in pretrained models.

We observe that using a fine-tuned model with a smaller dataset shows degraded
performance. Specifically, LAION-2B (Schuhmann et al., 2022) pretrained CLIP-
ViT model almost consistently outperforms the LAION-2B pretrained, ImageNet
Deng et al. (2009) fine-tuned CLIP-ViT model across all the adapter methods in
terms of accuracy except for TerraIncognita dataset, as illustrated in Fig. 8.
This can be attributed to the superior generalization capabilities of larger
models and the adapter’s capacity to preserve the knowledge from the
pretrained model. These findings align with previous literature, such as Kumar
et al. (2022), which suggests that fine-tuning entire large models can degrade
the learned representations.

#### Ablation study on routing strategy in Mixture-of-Adapter.

Table 3: Performance comparison on different mixture-of-adapter settings. We
perform experiments with dropping each components independently from our best
setting which is highlighted in blue. ‘w/o $\mathcal{L}_{\mathrm{aux}}$’
denotes the results without auxiliary loss that used in (Li et al., 2022),
‘Cosine$\rightarrow$Linear’ denotes the result when we changed the Cosine
router to a Linear router, and ‘Every$\rightarrow$Last’ denotes the result
when we changed the adapter attached layer from Every, which attaches adapters
in every two layer, to Last, which attaches adapters in last two layers.

Adapter | Changed component | PACS | VLCS | OfficeHome | TerraIn. | DomainNet | Avg. | #Param. | Trainable
---|---|---|---|---|---|---|---|---|---
#Param.
KAdaptaion-MoA | Original | 97.5 | 82.8 | 90.6 | 53.1 | 62.6 | 77.3 | 87.3M | 1.5M
w/o $\mathcal{L}_{\mathrm{aux}}$ | 97.3 | 82.6 | 90.5 | 54.0 | 62.6 | 77.4 | 87.3M | 1.5M
Cosine $\rightarrow$ Linear | 97.5 | 82.3 | 90.3 | 51.5 | 62.6 | 76.9 | 86.1M | 0.33M
Every $\rightarrow$ Last | 97.2 | 82.2 | 90.2 | 47.5 | 62.4 | 75.9 | 86.3M | 0.51M

We conduct an ablation study on three components proposed in Li et al. (2022),
which are cosine router, auxiliary loss $\mathcal{L}_{\mathrm{aux}}$, and the
layer configuration on the location of the MoE layer. Li et al. (2022)
demonstrates that employing a cosine router in their Generalizable Mixture-of-
Experts (GMoE) architecture yields better performance when contrasted with
employing a linear router. Our approach significantly deviates from theirs, as
each of our expert adapters exhibit distinct capacities, and our MoA is
attached to an attention layer. Thus we test both the linear and cosine
routers. The auxiliary loss function ($\mathcal{L}_{\mathrm{aux}}$) balances
the amount of token allocation to each experts, and the layer configuration
called ‘Every 2’ or ‘Last 2’ in their work denotes the attachment of MoE layer
with every two layers or only on the last two layers. Therefore we also
conduct experiments on these settings and report the whole results in Table 3.
In our setting, employing a cosine router showed a consistent performance
increase for all domains. Also attaching an adapter with every two layers
brought better performance than using MoA only on the last two layers. While
there was a minimal difference between using and not using
$\mathcal{L}_{\mathrm{aux}}$ (within the error margin), using the auxiliary
loss yielded more better results for the four datasets (PACS, VLCS,
OfficeHome, DomainNet). Thus we decided to incorporate the auxiliary loss
$\mathcal{L}_{\mathrm{aux}}$ when conducting our main experiment.

##  7 Conclusion

We have shown that using only parameter-efficient fine-tuning can outperform
or be competitive with previous state-of-the-art domain generalization
algorithms. Additionally, we propose integrating extra adapters with learnable
routers to handle various distribution shift situations. This allows us to
achieve state-of-the-art results compared to methods that do not utilize
ensembling, and when we do employ ensembling, our results are on par with
those obtained through ensemble based approaches. In this work we have
witnessed the remarkable effectiveness of parameter-efficient fine-tuning and
the utilization of large models for domain generalization. We anticipate that
these findings can serve as a source of inspiration for future research in the
domain generalization field, especially encouraging the robust fine-tuning of
large pretrained models.

## References

  * Aghajanyan et al. (2020) Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  _arXiv preprint arXiv:2012.13255_ , 2020.
  * Ahmed et al. (2016) Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani.  Network of experts for large-scale image categorization.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14_ , pp. 516–532. Springer, 2016.
  * Angarano et al. (2022) Simone Angarano, Mauro Martini, Francesco Salvetti, Vittorio Mazzia, and Marcello Chiaberge.  Back-to-bones: Rediscovering the role of backbones in domain generalization.  _arXiv preprint arXiv:2209.01121_ , 2022.
  * Arjovsky et al. (2019) Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Arpit et al. (2022) Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong.  Ensemble of averages: Improving model selection and boosting performance in domain generalization.  _Advances in Neural Information Processing Systems_ , 35:8265–8277, 2022.
  * Beery et al. (2018) Sara Beery, Grant Van Horn, and Pietro Perona.  Recognition in terra incognita.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 456–473, 2018.
  * Cha et al. (2021) Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  _Advances in Neural Information Processing Systems_ , 34:22405–22418, 2021.
  * Cha et al. (2022) Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun.  Domain generalization by mutual-information regularization with pre-trained models.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII_ , pp. 440–457. Springer, 2022.
  * Chen et al. (2022) Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision transformer adapter for dense predictions.  _arXiv preprint arXiv:2205.08534_ , 2022.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Du et al. (2022) Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al.  Glam: Efficient scaling of language models with mixture-of-experts.  In _International Conference on Machine Learning_ , pp. 5547–5569. PMLR, 2022.
  * d’Ascoli et al. (2021) Stéphane d’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun.  Convit: Improving vision transformers with soft convolutional inductive biases.  In _International Conference on Machine Learning_ , pp. 2286–2296. PMLR, 2021.
  * Eigen et al. (2013) David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever.  Learning factored representations in a deep mixture of experts.  _arXiv preprint arXiv:1312.4314_ , 2013.
  * Fang et al. (2013) Chen Fang, Ye Xu, and Daniel N Rockmore.  Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1657–1664, 2013.
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _The Journal of Machine Learning Research_ , 23(1):5232–5270, 2022.
  * Foret et al. (2020) Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.  Sharpness-aware minimization for efficiently improving generalization.  _arXiv preprint arXiv:2010.01412_ , 2020.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Gao et al. (2021) Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv preprint arXiv:2110.04544_ , 2021.
  * Gao et al. (2020) Tianyu Gao, Adam Fisch, and Danqi Chen.  Making pre-trained language models better few-shot learners.  _arXiv preprint arXiv:2012.15723_ , 2020.
  * Garipov et al. (2018) Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson.  Loss surfaces, mode connectivity, and fast ensembling of dnns.  _Advances in neural information processing systems_ , 31, 2018.
  * Graham et al. (2021) Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze.  Levit: a vision transformer in convnet’s clothing for faster inference.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 12259–12269, 2021.
  * Gross et al. (2017) Sam Gross, Marc’Aurelio Ranzato, and Arthur Szlam.  Hard mixtures of experts for large scale weakly supervised vision.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 6865–6873, 2017.
  * Gulrajani & Lopez-Paz (2020) Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  In _International Conference on Learning Representations_ , 2020.
  * He et al. (2021) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  _arXiv preprint arXiv:2110.04366_ , 2021.
  * He et al. (2022) Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang.  Parameter-efficient fine-tuning for vision transformers.  _arXiv preprint arXiv:2203.16329_ , 2022.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning_ , pp. 2790–2799. PMLR, 2019.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2021.
  * Ilharco et al. (2021) Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Izmailov et al. (2018) Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  _arXiv preprint arXiv:1803.05407_ , 2018.
  * Jacobs et al. (1991) Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton.  Adaptive mixtures of local experts.  _Neural computation_ , 3(1):79–87, 1991.
  * Jia et al. (2022) Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ , pp. 709–727. Springer, 2022.
  * Jordan & Jacobs (1994) Michael I Jordan and Robert A Jacobs.  Hierarchical mixtures of experts and the em algorithm.  _Neural computation_ , 6(2):181–214, 1994.
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34:1022–1035, 2021.
  * Keskar et al. (2016) Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  _arXiv preprint arXiv:1609.04836_ , 2016.
  * Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kumar et al. (2022) Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  In _International Conference on Learning Representations_ , 2022.
  * Lepikhin et al. (2020) Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.  Gshard: Scaling giant models with conditional computation and automatic sharding.  _arXiv preprint arXiv:2006.16668_ , 2020.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_ , 2021.
  * Lew et al. (2023) Byunggyu Lew, Donghyun Son, and Buru Chang.  Gradient estimation for unseen domain risk minimization with pre-trained models.  _arXiv preprint arXiv:2302.01497_ , 2023.
  * Li et al. (2022) Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu.  Sparse mixture-of-experts are domain generalizable learners.  _arXiv preprint arXiv:2206.04046_ , 2022.
  * Li et al. (2017) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5542–5550, 2017.
  * Li et al. (2018) Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein.  Visualizing the loss landscape of neural nets.  _Advances in neural information processing systems_ , 31, 2018.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ , 2021.
  * Li et al. (2023) Ziyue Li, Kan Ren, Xinyang Jiang, Yifei Shen, Haipeng Zhang, and Dongsheng Li.  Simple: Specialized model-sample matching for domain generalization.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Mao et al. (2022) Xiaofeng Mao, Yuefeng Chen, Xiaojun Jia, Rong Zhang, Hui Xue, and Zhao Li.  Context-aware robust fine-tuning.  _arXiv preprint arXiv:2211.16175_ , 2022.
  * Park & Kim (2022) Namuk Park and Songkuk Kim.  How do vision transformers work?  In _International Conference on Learning Representations_ , 2022.
  * Paul et al. (2022) Lysandre Debut Younes Belkada Sayak Paul, Sourab Mangrulkar, and Sylvain Gugger.  Peft: State-of-the-art parameter-efficient fine-tuning methods, 2022.
  * Peng et al. (2019) Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 1406–1415, 2019.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Riquelme et al. (2021) Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby.  Scaling vision with sparse mixture of experts.  _Advances in Neural Information Processing Systems_ , 34:8583–8595, 2021.
  * Ryu (2023) Simo Ryu.  lora.  https://github.com/cloneofsimo/lora, 2023.
  * Sagawa et al. (2019) Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.  _arXiv preprint arXiv:1911.08731_ , 2019.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _arXiv preprint arXiv:2210.08402_ , 2022.
  * Seo et al. (2020) Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han.  Learning to optimize domain specific normalization for domain generalization.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_ , pp. 68–83. Springer, 2020.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * Sun & Saenko (2016) Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_ , pp. 443–450. Springer, 2016.
  * Sung et al. (2022a) Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  Lst: Ladder side-tuning for parameter and memory efficient transfer learning.  _Advances in Neural Information Processing Systems_ , 35:12991–13005, 2022a.
  * Sung et al. (2022b) Yi-Lin Sung, Jaemin Cho, and Mohit Bansal.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5227–5237, 2022b.
  * Tan & Le (2019) Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pp. 6105–6114. PMLR, 2019.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pp. 10347–10357. PMLR, 2021.
  * Touvron et al. (2022) Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou.  Three things everyone should know about vision transformers.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV_ , pp. 497–515. Springer, 2022.
  * Vapnik (1998) Vladimir Vapnik.  Statistical learning theory wiley.  _New York_ , 1998.
  * Venkateswara et al. (2017) Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5018–5027, 2017.
  * Wang et al. (2020) Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez.  Deep mixture of experts via shallow embedding.  In _Uncertainty in artificial intelligence_ , pp. 552–562. PMLR, 2020.
  * Wightman (2019) Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7959–7971, 2022.
  * Xu et al. (2020) Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang.  Adversarial domain adaptation with domain mixup.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 6502–6509, 2020.
  * Yang et al. (2019) Brandon Yang, Gabriel Bender, Quoc V Le, and Jiquan Ngiam.  Condconv: Conditionally parameterized convolutions for efficient inference.  _Advances in neural information processing systems_ , 32, 2019.
  * Zaken et al. (2021) Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.  Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  _arXiv preprint arXiv:2106.10199_ , 2021.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  _arXiv preprint arXiv:1808.05326_ , 2018.
  * Zhang et al. (2022) Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  Pointclip: Point cloud understanding by clip.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8552–8562, 2022.
  * Zhang et al. (2023) Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, and Peng Gao.  Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners.  _arXiv preprint arXiv:2303.02151_ , 2023.
  * Zhou et al. (2021) Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain generalization with mixstyle.  _arXiv preprint arXiv:2104.02008_ , 2021.
  * Zoph et al. (2022) Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus.  St-moe: Designing stable and transferable sparse expert models.  _arXiv preprint arXiv:2202.08906_ , 2022.

Appendix

In this appendix, we provide a more detailed explanations of the
implementation details, results and visualizations of additional analyses, and
discuss limitations of this work.

##  Appendix A Additional analysis

In this section, we show additional analysis about loss landscapes, max
Hessian eigenvalue spectra, and analyses about routed tokens.

###  A.1 Comparisons of loss landscape visualizations

We show loss landscapes for all test environments in PACS dataset (Li et al.,
2017) in Fig. 10. Similar with the visualizations in main paper, the other
test environments have a tendency that fully fine-tuned models show most sharp
loss landscape. But trained models with LoRA and KAdaptation shows much more
flatter loss landscapes, especially KAdaptation have most flat loss landscape.

TE0

TE1

TE2

TE3

(a) Full fine-tuning

TE0

TE1

TE2

TE3

(b) LoRA (Hu et al., 2021; Ryu, 2023)

TE0

TE1

TE2

TE3

(c) KAdaptation (He et al., 2022)

TE0

TE1

TE2

TE3

(d) KAdaptation with Mixture-of-Adapter (Ours)

Figure 10: Flatness comparison of loss surfaces trained with full fine-tuning,
LoRA, KAdaptation, and KAdaptation with mixture-of-expert on PACS dataset (Li
et al., 2017).

###  A.2 Comparisons of max Hessian eigenvalue spectra

We show the max Hessian eigenvalue spectra evaluated on the all domains from
PACS dataset (Li et al., 2017) in Fig. 12. Every domain shows consistent shape
of max Hessian eigenvalue spectra that they shows more zero-concentrated shape
in the order of w/o LoRA, w/LoRA, w/ KA and w/KMoA.

Figure 12: Comparison of max Hessian eigenvalue spectra trained with full
fine-tuning, LoRA, and KAdaptation on PACS dataset (Li et al., 2017). In all
test environments we can observe that our proposed KMoA shows the most zero-
concentrated eigenvalue specturm, and other two adapter method, KA and LoRA
also shows smaller max Hessian eigenvalue distribution compared to the fully
fine-tuned case (w/o LoRA).

###  A.3 Visualizations of routed patches in PACS and TerraIncognita dataset.

We additionally show the visualizations of routed patch indices in Fig. 13 on
PACS dataset (Li et al., 2017), and Fig. 15, 17 on TerraIncognita dataset
(Beery et al., 2018). All images are visualizations from the last adapter-
attached transformer layer, layer 10. Similar with the findings from main
paper, we can observe that same indices are clustered at the regions where
having semantic meanings.

Figure 13: Visualizations of routed indices of each patch. We show a total of
seven classes in PACS dataset (Li et al., 2017), with one class per row in the
order of ‘Dog’, ‘Elephant’, ‘Giraffe’, ‘Guitar’, ‘Horse’, ‘House’, ‘Person’.
Also, in each column, the same domains are located in the order of ‘Art
Painting’, ‘Cartoon’, ‘Photo’, and ‘Sketch’.

Location 43 Figure 15: Visualizations of routed indices of each patch in
TerraIncognita (Beery et al., 2018) dataset. Left column is original image,
and in right column we indicate where each patch is routed. Upper and lower
images were took from same location but different time, therefore they have
same background but different object (bird) shape and location.

Location 46
Figure 17: Visualizations of routed indices of each patch in TerraIncognita
(Beery et al., 2018) dataset. Left column is original image, and in right
column we indicate where each patch is routed. Upper and lower images were
took from same location but different time, therefore they have same
background but different object (opossum) shape and location.

##  Appendix B Additional experimental details

###  B.1 Implementation details

Pretrained vision transformer models are obtained from the timm (Wightman,
2019) library. Adam (Kingma & Ba, 2014) optimizer is used for model
optimization along with a learning rate of $5e-5$. A batch size of 32 per
domain is used for the ViT-Base model. We run 15,000 iterations on DomainNet
and 5,000 for others, and evaluate at every 500 iteration steps for DomainNet,
200 steps for others. For the adapters, we arbitrarily choose the inner rank
so that all adapter methods have approximately the same number of trainable
parameters ($\sim$0.1M). The LoRA (Hu et al., 2021) and KAdaptation (He et
al., 2022) weights are attached to the QKV projection matrix and projection
weight matrix in the self-attention layer. Also for Compacter (Karimi Mahabadi
et al., 2021), we attach the adapter after the MLP layer in every blocks.
Every adapter except LoRA are implemented using their official repository, and
for LoRA, we implement it using unofficial implementation (Ryu, 2023). For
pretrained models, we use models from timm (Wightman, 2019) library for all
our experiments, and we perform all experiment on one server with 8 NVIDIA
RTX3090 GPUs. We’ll release training code for all our experiments and trained
weights in public.

Table 4: List of hyperparameters used in experiments on domain generalization benchmarks. Hyperparameter | LoRA | KAdaptation | LoRA-MoA | KMoA
---|---|---|---|---
# of Experts | N/A | $4$ | $4$
Scale of $\mathcal{L}_{\mathrm{aux}}$ | N/A | $0.01$ | $0.01$
Router | N/A | Cosine
Router Top-k | N/A | Top-1
Rank of adapter ($r_{i}$) | $2$ | $1$ | $[1,2,4,8]$
# of Kroneker products ($t$) | N/A | 64 | N/A | 64
Batch size |  $160$ (DomainNet), $96$ (Otherwise)
Learning rate | $5e-5$
Optimizer | Adam

###  B.2 Details about hyperparameters

We report all the hyperparameters that are used in our experiments and
analysis in Table 4.

##  Appendix C Limitations

Our method heavily relies on the performance of large pretrained models, hence
using a better pretrained model can lead to improved performance. But, such
models are limited and require a substantial amount of time and cost for
training. These weakness also exist in methods like MIRO (Cha et al., 2022) or
SIMPLE (Li et al., 2023), and the availability of high-performance open-source
models like OpenCLIP (Ilharco et al., 2021) can alleviate these drawbacks. Our
approach may not yield significantly better performance on datasets that are
more challenging than TerraIncognita (Beery et al., 2018) since it has fewer
trainable parameters compared to fully fine-tuned DG algorithms. However, our
method offers flexibility in adjusting trainable parameters through the
control of inner rank $r_{i}$ and allows for obtaining the optimal rank
through hyperparameter search, addressing this limitation effectively.


Nb-Ti   9.2 (8.1)   3.4   45.2 (63.9)   660 (660)   18.9   5200   900   100   60   -200

Nb-Ti   REBCO   9.2 (9.1)   3.4   45.2 (79.0)   660 (660)   19.3   5280   900   100   60   -200

REBCO   REBCO   10.1 (13.2)   3.7   47.5 (91.5)   660 (660)   20.8   6470   1410   280   920   100

Nb-Ti   [l]Quat.
Nb-Ti   9.7 (10.3)   3.6   46.8 (79.2)   660 (660)   20.0   6150   1250   230   630   -40

Nb-Ti   REBCO   10.1 (13.0)   3.7   47.5 (91.6)   660 (660)   20.8   6350   1400   280   900   100

Upgraded tokamaks designed for 100  net electricity and H$_\text{98}$ = 1.6: Performance and cost data for a tokamak optimised for Nb-Ti toroidal field and central solenoid coils with minimised capital cost (*) and upgraded magnets of different superconductors (with maximised net electricity yield). Also shown are the power values that would result, were a reduced H$_\text{98}$ = 1.0 to occur in practice.

2cH$_\text{98}$ = 1.6     2cH$_\text{98}$ = 1.0

TF s.c.    CS s.c.   [l]TF (CS)
peak field
()   [l]Field on
()   [l]TF (CS)
steel %
(%)   [l]TF (CS)
peak stress
()   [l]Plasma
(M)   [l]Capital
(1990 M$)   [l]Fusion
(M)   [l]Net.
(M)   [l]Fusion
(M)   [l]Net.

Nb-Ti   [l]Commer.*
Nb-Ti   9.0 (7.3)   3.0   45.5 (58.8)   660 (660)   15.9   4470   870   100   20   -200

Nb-Ti   REBCO   9.0 (7.2)   3.0    45.5 (69.5)   660 (660)   15.9   4530   870   100   20   -200

REBCO   REBCO   10.2 (14.4)   3.3   46.7 (90.8)   660 (660)   18.1   5280   1190   210   30   -200

Nb-Ti   [l]Quat.
Nb-Ti   9.7 (10.9)   3.2   45.8 (71.0)   660 (660)   17.3   5180   1070   170   30   -200

Nb-Ti   REBCO   10.2 (14.2)   3.4   46.7 (90.7)   660 (660)   18.1   4890   1200   210   30   -200

Upgraded tokamaks designed for 500  net electricity and H$_\text{98}$ = 1.2: Performance and cost data for a tokamak optimised for Nb-Ti toroidal field and central solenoid coils with minimised capital cost (*) and upgraded magnets of different superconductors (with maximised net electricity yield). Also shown are the power values that would result, were a reduced H$_\text{98}$ = 1.0 to occur in practice.

2cH$_\text{98}$ = 1.2     2cH$_\text{98}$ = 1.0

TF s.c.    CS s.c.   [l]TF (CS)
peak field
()   [l]Field on
()   [l]TF (CS)
steel %
(%)   [l]TF (CS)
peak stress
()   [l]Plasma
(M)   [l]Capital
(1990 M$)   [l]Fusion
(M)   [l]Net.
(M)   [l]Fusion
(M)   [l]Net.

Nb-Ti   [l]Commer.*
Nb-Ti   9.2 (8.7)   3.4   48.3 (81.6)   660 (660)   29.4   7730   2180   500   1500   270

Nb-Ti   REBCO   9.2 (9.3)   3.4   48.3 (93.4)   660 (660)   29.6   7910   2220   510   1570   290

REBCO   REBCO   10.0 (11.9)   3.6   48.8 (92.2)   660 (660)   32.2   9790   2730   680   1730   340

Nb-Ti   [l]Quat.
Nb-Ti   9.4 (9.1)   3.4   47.8 (83.3)   660 (660)   30.1   9180   2290   540   1560   290

Nb-Ti   REBCO   10.0 (9.4)   3.6   48.8 (92.2)   660 (660)   32.2   9600   2730   670   1730   340

Performance and cost of a steady-state H$_\text{98}$ = 1.6,  100 $_\text{e}$ reactor with copper TF, CS and PF coils (with minimised capital cost). Based on STPP <cit.>.

2cH$_\text{98}$ = 1.6    2cH$_\text{98}$ = 1.0
$R_\text{major}$   $A$   $\kappa$   $\delta$   [l]Field on
TF coil
()   [l]Field on
()   [l]TF steel
(%)   [l]Plasma
(M)   [l]Capital
(1990 M$)        [l]Fusion
(M)   [l]Net.
(M)   [l]Fusion
(M)   [l]Net.

3.42   1.40   3.20   	0.55   	7.56   	2.50   5   	19.4   	5200        		1670   	100   530   -370

A comparison between experimentally measured plasma energy confinement times from various tokamaks and the confinement time predicted by the IPB98(y,2) scaling law and adapted from <cit.>. The solid line is $H_{98} = 1.0$ dashed line is $H_{98} = 1.2$ and the dotted line is $H_{98} = 2.0$. Inset: H-factor as a function of toroidal field on the plasma axis for a tokamak with $I_P$ = 8.20 MA, $P_L$ = 46  , $n$ = 1.8$\times{10}^{20}m^{-3}$, $M$ = 2.5, $R_{major}$ = 2 m, $A$ = 1.8, $\kappa$ = 3 using the NSTX Gyro-Bohm <cit.> confinement time scaling.

Top: Total central solenoid coil (CS) and poloidal field coil (PF) flux, central solenoid bore and thickness, and (Bottom:) plasma major radius and plant total capital cost as a function of plasma burn time, of 100 $_\text{e}$ tokamak power plants with REBCO CS and TF coils and Nb-Ti PF coils (i.e. magnet materials as per the preferred tokamak design) optimised for minimum capital cost. Note that the preferred reactor has a 7200 s plasma burn time.

Cross-sections and inboard mid-plane radial build of the preferred reactor (a cost-optimised H$_\text{98}$ = 1.2, 100  net electricity tokamak with REBCO toroidal field and central solenoid coils) with an optimised 25.0  radiation shield. Details of the layer constituents can be found in table <ref>. The inboard blanket is 53  in radial thickness and based on the EU-DEMO helium-cooled pebble bed design <cit.>, guaranteeing a tritium breeding ratio $>$ 1.1.

Nuclear heating in the TF coil system and number of full-power operation years until the Weber dose limit <cit.> is achieved for the preferred reactor (a cost-optimised H$_\text{98}$ = 1.2, 100  net electricity tokamak with REBCO toroidal field and central solenoid coils) as a function of the thickness of its tungsten carbide radiation shield as calculated by  (closed squares) and the benchmarking calculation (open diamonds). The material layers between the plasma and the TF coil are given in table <ref>. The dotted black lines indicate the minimum 40 year conductor lifetime limit (and corresponding minimum shield thickness as calculated by ). The dotted red lines indicate the maximum 10  heating limit on the TF system (and corresponding minimum shield thickness as calculated by our benchmarking calculations).

calculated (a) fast neutron flux through the poloidal cross-section of the preferred reactor (a cost-optimised H$_\text{98}$ = 1.2, 100  net electricity tokamak with REBCO toroidal field and central solenoid coils) with an optimised 25.0  radiation shield detailed in the right hand column of table <ref>. (b) The neutron flux spectrum and (c) photon flux spectrum as a function of distance into inboard mid plane of the preferred reactor with a 25.0  radiation shield. These spectra were converted to flux density per unit lethargy by multiplying the spectral histogram fluxes by the ratio between the energy bin average energies and the bin widths.

Neutron and photon induced wall loading along the mid-plane within the preferred reactor (a cost-optimised H$_\text{98}$ = 1.2, 100  net electricity tokamak with REBCO toroidal field and central solenoid coils) with an optimised 25.0  radiation shield. The radial positions of the central solenoid coil (light pink), toroidal field coil legs (blue), vacuum vessel (green), radiation shield (black) and blanket (deep pink) are shown.

Whole strand/tape critical current density of commercial ITER specification Nb-Ti (Comm. Nb-Ti), quaternary (Quat.) Nb-Ti,  internal tin Nb$_3$Sn, and REBa$_2$Cu$_3$O$_{7}$ (REBCO where RE: rare-earth) at 4.5 , used in this work. Quaternary Nb-Ti is not commercially available (but could be optimised for fusion applications).

Toroidal magnetic field through the midplane of the preferred reactor (a cost-optimised H$_\text{98}$ = 1.2, 100  net electricity tokamak with REBCO toroidal field and central solenoid coils).

Fusion power, gross electric power and net electric power as a function of operating H$_\text{98}$-factor for two reactors with REBCO CS and TF coils and Nb-Ti PF coils. The reactors were optimised for minimum capital cost (open data points) at H$_\text{98}$-factors of H$_\text{98}$ = 1.6 (black) and H$_\text{98}$ = 1.2 (red) and produce 100 $_\text{e}$. Reactors at higher or lower than expected H$_\text{98}$-factors were optimised to produce maximum net electricity (solid data points). The discontinuities in fusion power that occur between high and low H$_\text{98}$ factors are due to a loss of energy confinement at H$_\text{98}$ factors that are too low.

(a) Change in cost-optimal field on coil and (b) capital cost of the preferred reactor (a cost-optimised H$_\text{98}$ = 1.2, 100  net electricity tokamak with REBCO toroidal field and central solenoid coils) as a function of the allowable maximum of the shear stresses (as used for the Tresca yield criterion) on the central solenoid and inboard toroidal field coil mid-planes. Different data sets correspond to different costs of steel components (standard, 1.5 $\times$ standard etc.), representative of either more expensive steels or larger steel volumes.

(a) Reactor power balance (where “Core Systems” includes the cryo-system (46 $_\text{e}$) and the tritium handling system (15 $_\text{e}$)); (b) direct capital cost breakdown of our preferred 100  net electricity producing REBCO based tokamak power plant. Costs are in 1990 US M$.