# TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial
Robustness and Generalization Ability

Fengji Ma1, Li Liu1†, Hei Victor Cheng2 1The Hong Kong University of Science
and Technology (Guangzhou) 2Aarhus University

(2018)

###### Abstract.

This work addresses the challenge of achieving zero-shot adversarial
robustness while preserving zero-shot generalization in large-scale foundation
models, with a focus on the popular Contrastive Language-Image Pre-training
(CLIP). Although foundation models were reported to have exceptional zero-shot
generalization, they are highly vulnerable to adversarial perturbations.
Existing methods achieve a comparable good tradeoff between zero-shot
adversarial robustness and generalization under small adversarial
perturbations. However, they fail to achieve a good tradeoff under large
adversarial perturbations. To this end, we propose a novel Text-Image Mutual
Awareness (TIMA) method that strikes a balance between zero-shot adversarial
robustness and generalization. More precisely, we propose an Image-Aware Text
(IAT) tuning mechanism that increases the inter-class distance of text
embeddings by incorporating the Minimum Hyperspherical Energy (MHE).
Simultaneously, fixed pre-trained image embeddings are used as cross-modal
auxiliary supervision to maintain the similarity between the MHE-tuned and
original text embeddings by the knowledge distillation, preserving semantic
information between different classes. Besides, we introduce a Text-Aware
Image (TAI) tuning mechanism, which increases inter-class distance between
image embeddings during the training stage by Text-distance based Adaptive
Margin (TAM). Similarly, a knowledge distillation is utilized to retain the
similarity between fine-tuned and pre-trained image embeddings. Extensive
experimental results demonstrate the effectiveness of our approach, showing
impressive zero-shot performance against a wide range of adversarial
perturbations while preserving the zero-shot generalization capabilities of
the original CLIP model.

Adversarial Robustness, Multimodal Learning, Zero-Shot Learning, Vision-
Language Models

${\dagger}$ Corresponding author (avrillliu@hkust-gz.edu.cn).

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Computing methodologies Computer vision tasks

(a) Clean Accuracy

(b) Small perturbation radius 1/255

(c) Large perturbation radius 4/255

(d) Large perturbation radius 8/255

Figure 1. Zero-shot accuracies of compared methods and our proposed method. We
show zero-shot robust accuracy of different methods under different
perturbation radius. Numbers in the bracket denote the best accuracy among
different methods. The accuracy of all methods is normalized with the highest
method in the corresponding dataset. Our method (TIMA) achieves better results
than existing methods (TeCoA (Mao et al., 2022), PMG (Wang et al., 2024), and
LAAT (Li et al., 2023b)), under both small and large adversarial
perturbations.

##  1\. Introduction

Large-scale foundation models (Radford et al., 2021; Jia et al., 2021; Ahn et
al., 2022; Ramesh et al., 2022) have recently garnered significant attention
due to their excellent zero-shot generalization ability. Contrastive Language-
Image Pre-training (CLIP) (Radford et al., 2021), the most widely used vision-
language model, demonstrates the ability to accurately classify new classes
with only simple text captions, even those that have not been encountered
before. However, recent studies (Mao et al., 2022; Zhao et al., 2023; Yang and
Mirzasoleiman, 2023; Michels et al., 2023; Noever and Noever, 2021; Wang et
al., 2023; Inkawhich et al., 2023) revealed that while these foundation models
exhibit strong generalization performance, they are extremely vulnerable to
adversarial perturbation. Notably, existing adversarial methods (Madry et al.,
2017; Carlini and Wagner, 2017; Croce and Hein, 2020) have effectively
attacked these foundation models, resulting in performance degradation of up
to 90% on different datasets, even with minor attacks.

Traditionally, time-intensive and resource-heavy adversarial training is
needed to achieve adversarial robustness. To bypass the retraining across new
tasks or datasets, and to harvest the full potential of foundation models, the
concept of zero-shot adversarial robustness in foundation models (Mao et al.,
2022; Li et al., 2023b; Wang et al., 2024) has emerged as a new and urgent
research topic. The objective of zero-shot adversarial robustness is twofold:
to transfer adversarial robustness in a zero-shot learning manner, and to
preserve zero-shot generalization capabilities of foundational models. Here,
adversarial robustness refers to a model’s ability to withstand adversarial
attacks with various perturbations. Zero-shot generalization, on the other
hand, denotes the model’s capacity to achieve high accuracy on previously
unseen classes without prior exposure.

In pursuit of a zero-shot adversarial robust CLIP model, the pioneer work
TeCoA (Mao et al., 2022) proposes to align adversarial image embeddings with
their corresponding text embeddings. This eliminates the need for retraining
in subsequent visual tasks. Building on this idea, PMG (Wang et al., 2024)
introduces additional constraints from pre-trained models and clean samples
into the objective function, encouraging the preservation of generalized pre-
trained information.

Despite these advancements, TeCoA (Mao et al., 2022) and PMG (Wang et al.,
2024) fall short under larger adversarial attacks, as their effectiveness
wanes significantly with increased perturbation radius, which can be seen from
Fig 1. The reason is that both methods fail to enlarge the embedding distances
between classes. To address this, LAAT (Li et al., 2023b) proposes an
algorithm to expand the distance between fixed textual embeddings in
hyperspherical space, showing promise against a larger perturbation radius.
Unfortunately, disrupting the semantic information within the text embedding
space and the contrastive relationship between text and image embeddings
sacrifices zero-shot generalization on clean samples.

To address the challenge of balancing adversarial robustness and
generalization in the zero-shot setting, this work proposes the Text-Image
Mutual Awareness (TIMA) mechanism. TIMA achieves state-of-the-art results on
clean datasets and datasets under both small and large perturbations, as
demonstrated in Fig. 1. TIMA overcomes the limitation identified in PMG (Wang
et al., 2024), TeCoA (Mao et al., 2022), and (Li et al., 2023b) by ensuring
robustness against larger perturbations while preserving zero-shot
generalization, achieving a satisfactory balance between zero-shot adversarial
robustness and generalization. Central to TIMA’s innovation is the hypothesis:

Increasing the inter-class distances within the pretrained CLIP text and image
embeddings is the key to improving zero-shot adversarial robustness,
especially under large perturbation.

Building on this, we propose two corresponding modules for enhancing the
inter-class distance for the text and image embeddings, respectively. First,
for the text embeddings, unlike LAAT (Li et al., 2023b), TIMA employs a
learning-based approach named Image-Aware Text (IAT) tuning mechanism during
adversarial training. This approach applies the Minimum Hyperspherical Energy
(MHE) principle (Liu et al., 2018, 2021; Lin et al., 2020; Liu et al., 2022;
Wang et al., 2020) to increase the distances between classes in text
embeddings, meanwhile ensuring the preservation of the semantic information
and coherent relationship between text and image embeddings inherent in the
original pre-trained CLIP model using an image-to-text distillation module.

Additionally, we illustrate the importance of adjusting the inter-class
distance of image embeddings, which has been ignored in prior work. TIMA
proposes a Text-Aware Image (TAI) tuning mechanism, which adds text distance
based adaptive margins to different classes based on the classification
results and hence enlarges the inter-class distance of the image embeddings
during contrastive adversarial fine-tuning. Similarly, an image-to-text
distillation module is used to transfer the semantic information and
interaction dynamics from the original CLIP model.

Combining IAT with TAI mechanisms, TIMA achieves effective multimodal training
by enlarging the inter-class distance in image and text embeddings and
utilizing the contrastive and interactive information between text and image
modalities to strike an optimal balance between zero-shot robustness and
generalization.

Our main contributions can be summarized as follows:

  * •

The proposed MHE method finds the largest inter-class distances for text
embeddings, making CLIP more resilient to both small and large perturbations.

  * •

This is the first work to propose a working mechanism for enlarging the inter-
class distance in image embeddings. The proposed TAI can be applied in a plug-
and-play manner to combine with any text embedding adjustment schemes.

  * •

We design novel knowledge distillation methods for text and image embeddings
to keep semantic and interactive information between the modalities after
enlarging inter-class distances.

  * •

Extensive experiments show the proposed TIMA’s superiority in zero-shot robust
accuracy and clean accuracy across multiple datasets and under different
perturbations, establishing its excellent efficacy.

Figure 2. The framework of our proposed method Text-Image Mutual Awareness
(TIMA). $\mathcal{L}_{\mathrm{MHE}}$ is Minimum Hyperspherical Energy loss,
$\mathcal{L}_{\mathrm{IAKD}}$ is Image-Aware Knowledge Distillation loss,
$\mathcal{L}_{\mathrm{TAW}}$ is Text-distance Adaptive Margin loss and
$\mathcal{L}_{\mathrm{TAKD}}$ is Text-Aware Knowledge Distillation loss.

##  2\. Related Works

This paper explores the trade-off between zero-shot adversarial robustness and
zero-shot generalization in vision-language models (VLMs), particularly CLIP.
Numerous studies (Li et al., 2023a; Radenovic et al., 2023; Zhou et al., 2024;
Huang et al., 2023; Zang et al., 2023; Yang et al., 2023) have shown that
knowledge distillation preserves the zero-shot generalization of VLMs. In this
paper, our focus is primarily on improving zero-shot adversarial robustness.

Existing zero-shot adversarial robustness approaches (Mao et al., 2022; Wang
et al., 2024; Li et al., 2023b) fall into two categories based on the
adjustment of inter-class distances of pretrained CLIP text embeddings. The
first category does not adjust these distances and includes methods like TeCoA
(Mao et al., 2022), which uses adversarial fine-tuning to align adversarial
image embeddings with fixed CLIP text embeddings, and PMG (Wang et al., 2024),
which improves the similarity between clean and adversarial image embeddings
through knowledge distillation. These methods, however, fail under large-scale
perturbation adversarial attacks. The second category, exemplified by LAAT (Li
et al., 2023b), increasing text embedding distances but often at the cost of
losing the original semantic relationships, leading to zero-shot
generalization significant decrease.

Our proposed method diverges from these by hypothesizing that increasing
inter-class distance of both text and image embedding can improve zero-shot
adversarial robustness, especially under large perturbations. By increasing
both types of inter-class distances and leveraging cross-modal auxiliary
supervision information, we aim to preserve the semantic information of
pretrained CLIP, thereby sustaining zero-shot generalization.

##  3\. Methods

###  3.1. Preliminaries and Problem Setup

We choose CLIP, a popular large-scale pre-trained vision-language model for
zero-shot recognition, as our base model. Let $F_{\theta}(\cdot)$ be the CLIP
image encoder parameterized by $\theta$ and $F_{\phi}(\cdot)$ be the CLIP text
encoder parameterized by $\phi$. Given an input image $x$ and a textual
description $\mathcal{T}$ about its class, such as ”This is a photo of a ”,
denoted as $t$, the model will output an image embedding $F_{\theta}(x)$ and a
text embedding $F_{\phi}(\mathcal{T})$. We denote the image embedding as
$z=F_{\theta}(x)$ and text embedding as $t=F_{\phi}(\mathcal{T})$. The
similarity between image embedding and each candidate class of text embedding
is computed. For the $j$-th class, the text embedding can be described as
$z_{j}=F_{\phi}(\mathcal{T}_{j})$. For the CLIP model, the generation of
adversarial examples can be described as:

(1) $\displaystyle
x^{(a)}=\underset{x^{(a)}}{\mathrm{argmax}}\;\mathcal{L}_{\mathrm{CE}}(x^{(a)},\mathcal{T},y),\qquad
s.t.\>||x^{(a)}-x||_{p}\leq\varepsilon,$

where $\mathcal{L}_{\mathrm{CE}}$ represents cross-entropy loss. The additive
noise is bounded within a $p$-norm ball of radius $\varepsilon$. For an
adversarial example $x^{(a)}$, the adversarial image embedding is given by
$z^{(a)}=F_{\theta}(x^{(a)})$. The generation of adversarial examples is by
PGD (Madry et al., 2017), which is shown in Fig. 2.

We follow the definitions of zero-shot adversarial robustness and
generalization as outlined in existing studies (Mao et al., 2022; Wang et al.,
2024; Li et al., 2023b). Our experiments use zero-shot robust accuracy as the
metric for assessing zero-shot adversarial robustness. The framework of the
proposed method, TIMA, is shown in Fig. 2.

###  3.2. Image-Aware Text Tuning Mechanism

(a) CLIP (Radford et al., 2021)

(b) LAAT (Li et al., 2023b)

(c) Ours

Figure 3. Text-text embedding pairs and (clean) image-text embedding pairs
cosine similarity matrix on CIFAR10. The first row shows the text-text
embedding pairs similarity of different methods. The second row shows the
similarity among clean image-text embedding pairs for different methods.

Prior research (Li et al., 2023b) demonstrated that increasing the inter-class
distance of text embeddings can significantly enhance a model’s zero-shot
adversarial robustness. Inspired by it, we aim to strengthen zero-shot
adversarial robustness under the large perturbation radius by increasing the
inter-class distance of text embeddings. The idea of MHE (Minimum
Hyperspherical Energy) (Liu et al., 2018, 2021; Lin et al., 2020; Liu et al.,
2022; Wang et al., 2020) is applied to fine-tune text embeddings, enforcing
embeddings from different classes to be distributed uniformly over the
hypersphere representing the text embedding space, thus maximizing their
inter-class distances. On the other hand, as illustrated in Fig. 3, the text
embedding expansion algorithm in LAAT disrupts the semantic relationships
between similar classes. To address this, we utilize the (fixed) pre-trained
image embeddings as auxiliary supervisory information for cross-modal
supervision distillation.

####  3.2.1. MHE-Guided Text Distance Enlargement

In pursuit of amplifying inter-class distances, thereby improving the
separability of text embeddings, we introduce MHE, which can be described as:

(2) $\displaystyle\mathcal{L}_{\mathrm{MHE}}=\mathbb{E}_{j\neq
k}\left[\frac{1}{d^{\alpha}(t_{j},t_{k})}\right],$

where $t_{j},t_{k}$ represents the text embedding of the text encoder, i.e.,
$t_{j}=F_{\phi}(\mathcal{T}_{j})$, and $d(t_{j},t_{k})$ is the Euclidean
distance between text embeddings of different classes with $\alpha=2$.
Different from the original form of MHE loss, to prevent gradient explosion,
we adopted the MHE loss as $\mathcal{L}_{\mathrm{MHE}}=\mathbb{E}_{j\neq
k}\left[\frac{1}{1+d^{\alpha}(t_{j},t_{k})}\right]$.

####  3.2.2. Image-Aware Knowledge Distillation for Text Semantic Consistency

Previous research (Li et al., 2023b) solely focuses on increasing the inter-
class distances of text embeddings to improve zero-shot adversarial
robustness, which destroys semantic information between different classes of
text embeddings and sacrifices the generalization performance on clean
datasets. We propose an Image-Aware Knowledge Distillation (IAKD) to
circumvent the issue of semantic information destruction inherent in enlarging
the inter-class distance in text embeddings. The idea is to employ the image
embeddings from fixed pretrained CLIP as supervision information, we distill
text embeddings to maintain their semantic consistency, thereby preserving
zero-shot generalization. It’s important to note that our approach does not
directly distill between the pretrained CLIP text embeddings and the fine-
tuned text embeddings. Denote the teacher contrastive distribution as $p_{ij}$
and the student contrastive distribution as $q_{ij}$, they are given by:

(3) $\displaystyle
p_{ij}=\frac{\mathrm{exp}({s(\hat{z}_{i},\hat{t}_{j})}/\tau)}{{\sum_{k}}\mathrm{exp}(s(\hat{z}_{i},\hat{t}_{k})/\tau)},\qquad
q_{ij}=\frac{\mathrm{exp}({s(\hat{z}_{i},t_{j})}/\tau)}{{\sum_{k}}\mathrm{exp}(s(\hat{z}_{i},t_{k})/\tau)}.$

where $\tau$ is the temperature of CLIP, $\hat{t}_{j}$ represents the $j$-th
text embedding of the fixed pretrained CLIP text encoder, $t$ is the text
embedding of our fine-tuning CLIP text encoder and $\hat{z}$ is the image
embedding of fixed pre-trained CLIP image encoder. Specifically, we first
calculate the softmax distribution of the cosine similarity between the fixed
pretrained CLIP image embeddings and the fixed pretrained CLIP text embeddings
to serve as the teacher contrastive distribution. Subsequently, we compute the
softmax distribution of the cosine similarity between the fixed pretrained
CLIP image embeddings and the fine-tuned text embeddings as the student
contrastive distribution. Then, we compute KL divergence between student
contrastive distribution and teacher contrastive distribution. The loss
function $\mathcal{L}_{\mathrm{IAKD}}$ of image-awareness supervision text
embedding distillation is described as:

(4)
$\displaystyle\mathcal{L}_{\mathrm{IAKD}}=\mathrm{D}_{\mathrm{KL}}(p_{ij}||q_{ij}).$

Therefore, the loss function of our Image-Aware Text (IAT) tuning mechanism
can be described as
$\mathcal{L}_{\mathrm{IAT}}=\mathcal{L}_{\mathrm{MHE}}+\lambda_{\mathrm{T}}\mathcal{L}_{\mathrm{IAKD}}$.
where $\lambda_{\mathrm{T}}$ is a hyper-parameter that balances the text
embedding separation and zero-shot generalization.

As illustrated in Fig. 3, the text embedding expansion algorithm proposed by
LAAT (Li et al., 2023b) significantly diminishes the semantic relationship
between image and text embeddings on clean samples. Furthermore, the inherent
semantic relatedness between similar classes is lost due to the expansion
algorithm’s neglect of inter-class relationships. In this perspective, we
improve zero-shot adversarial robustness via increasing inter-class distance
of text embeddings without sacrificing the zero-shot generalization
performance. For a detailed analysis of why increasing the inter-class
distances of text embeddings by MHE principle improves CLIP’s zero-shot
adversarial robustness, please refer to the supplementary materials.

###  3.3. Text-Aware Image Tuning Mechanism

(a) TeCoA (Mao et al., 2022)

(b) LAAT (Li et al., 2023b)

(c) Ours

Figure 4. Adversarial image vs. adversarial image embedding pairs cosine
similarity matrix on CIFAR10 under large perturbation radius
($\varepsilon=8/255$) adversarial attack.

Previous research in unimodal image classification (Bui et al., 2021; Mustafa
et al., 2019; Li et al., 2021; Mao et al., 2019; Zhou and Patel, 2022; Hou et
al., 2023) underscores the importance of increasing inter-class distances in
image features to improve model robustness. However, existing methods for
adjusting CLIP such as TeCoA, PMG, and LAAT lack adjustments in the inter-
class distances of image embeddings. This oversight results in exaggerated
cosine distances between adversarial image embeddings of different classes
under large perturbation attacks, as shown in Fig. 4. The results of TeCoA and
LAAT from Fig. 4 show high cosine similarity between different classes of
adversarial image embeddings, i.e. small inter-class distance of adversarial
image embeddings, which leads to compromising model robustness under large
perturbation (shown in Fig. 1(d)). Similar to the IAT tuning mechanism, which
expands the inter-class distance of text embeddings while preserving semantic
relationships to boost zero-shot generalization and robustness, we introduce
Text-Aware Image (TAI) tuning mechanism, a novel strategy aiming to enhance
zero-shot adversarial robustness by increasing the inter-class distances of
fine-tuned adversarial image embeddings without sacrificing semantic
information. Our method involves the introduction of an adaptive margin based
on text distances to increase the inter-class distance of image embeddings,
complemented by the utilization of fixed pre-trained text embeddings for
cross-modal supervision in the distillation of image embeddings.

####  3.3.1. Text-distance based Adaptive Margin

Numerous studies (Mao et al., 2019; Zhou and Patel, 2022; Ding et al., 2019;
Liu and Chan, 2022; Hou et al., 2023; Liu et al., 2016) have demonstrated that
incorporating margin penalties can increase the distance between classes,
thereby enhancing adversarial robustness. However, these studies have applied
a fixed margin that treats all negative samples equally, overlooking the
intrinsic semantic relationships between classes. This oversight leads to
misclassifications among semantically similar classes under large perturbation
adversarial attacks. To make use of semantic information and the likelihood of
misclassification and potential misclassification among different classes, our
approach apply margins to incorrectly classified classes and potentially
misclassified, where the added margin increases as the semantic proximity
between the incorrect and correct classes increases. Considering rich semantic
information between different classes captured by pre-trained CLIP text
embeddings, we propose the text-distance based adaptive margin. When the
cosine similarity between the image embedding and the corresponding text
embedding falls below a threshold, indicating a significant semantic
difference, then no additional margin is added. Conversely, when the
similarity exceeds the threshold, indicating semantic proximity, an additional
margin is added to these classes. The margin size depends on the level of
semantic similarity to the correct class. Therefore, we design a margin is as
follows:

(5) $\displaystyle\mathcal{M}_{ik}=\left\\{\begin{matrix}m\circ
s(\hat{t}_{i},\hat{t}_{k}),&s(\hat{z}_{i},\hat{t}_{k})\geq\eta\cdot
s(\hat{z}_{i},\hat{t}_{j})\\\ 0,&s(\hat{z}_{i},\hat{t}_{k})<\eta\cdot
s(\hat{z}_{i},\hat{t}_{j})\end{matrix},\right.$

where $\circ$ is element-wise multiplication and $\hat{z}_{i}$ are the frozen
image embeddings of the original CLIP image encoder. Margin value $m$ and
threshold $\eta$ are hyper-parameters of adaptive margin. Specifically, when
the cosine similarity between image embedding and the incorrect class text
embedding is higher than that between image embedding and the corresponding
correct class text embedding, $\textit{i.e.},$
$s(\hat{z}_{i},\hat{t}_{k})>s(\hat{z}_{i},\hat{t}_{j})$, which means that is a
misclassified class of text embedding. The value $\eta$ should be set as
$\eta<1$. When $\eta\cdot s(\hat{z}_{i},\hat{t}_{j})\leq
s(\hat{z}_{i},\hat{t}_{k})<s(\hat{z}_{i},\hat{t}_{j})$, the text embedding
class represents the potential class that may be classified incorrectly. Such
a class is semantically close to the correct class, and the text distance
adaptive margin needs to be added. From the experimental results in Sec. 4,
the hyper-parameters, margin value $m$ and threshold $\eta$ are set as 0.1 and
0.95. respectively. Putting together, our proposed Text-distance Adaptive
Margin (TAM) can be combined with adversarial fine-tuning:

(6)
$\displaystyle\mathcal{L}_{\mathrm{TAM}}=-\mathbb{E}_{i,j}\left[y_{ij}\mathrm{log}\frac{\mathrm{exp}({[s(z^{(a)}_{i},\hat{t}_{j})-\mathcal{M}_{ij}]}/\tau)}{{\sum_{k}}\mathrm{exp}([s(z^{(a)}_{i},\hat{t}_{k})-\mathcal{M}_{ik}]/\tau)}\right],$

where $\mathcal{M}$ is the text-distance adaptive margin. $\tau$ is the
temperature of CLIP. $\hat{t}$ is the fixed text embedding from the pre-
trained CLIP. $z^{(a)}$ is the adversarial image embedding, i.e.,
$z^{(a)}=F_{\theta}(x^{(a)})$, where $x^{(a)}$ is the adversarial image
generated by Eq. 1. The proposed text distance adaptive margin for cross-modal
image-text alignment leverages pre-trained text embeddings to increase the
inter-class distance of image embeddings (which can be seen in Fig. 4(c)),
thereby enhancing adversarial robustness, particularly under large
perturbations.

####  3.3.2. Text-Aware Knowledge Distillation for Image Semantic Consistency

Similar to Image-Aware Knowledge Distillation (IAKD) for text semantic
consistency, we propose a Text-Aware Knowledge Distillation (TAKD) for image
semantic consistency. In this paper, we denote the teacher contrastive
distribution as $p_{ij}$ in TAI tuning mechanism (which is the same as teacher
contrastive distribution of Eq. 3 in IAT tuning mechanism) and the student
contrastive distribution as $r_{ij}$ in tuning mechanism which can be
described as:

(7) $\displaystyle
r_{ij}=\frac{\mathrm{exp}({s(z^{(a)}_{i},\hat{t}_{j})}/\tau)}{{\sum_{k}}\mathrm{exp}(s(z^{(a)}_{i},\hat{t}_{k})/\tau)},$

where $z^{(a)}$ is the adversarial image embedding of the fine-tuning CLIP
image encoder and $\hat{t}$ is the text embedding of fixed pre-trained CLIP
text encoder. Specifically, we first calculate the softmax distribution of the
cosine similarity between the fixed pretrained CLIP clean image embeddings and
the fixed pretrained CLIP text embeddings to serve as the teacher contrastive
distribution. Subsequently, we compute the softmax distribution of the cosine
similarity between the adversarial image embeddings and the fixed pretrained
text embeddings as the student contrastive distribution. Then, we conduct KL
divergence between student contrastive distribution and teacher contrastive
distribution. The loss function $\mathcal{L}_{\mathrm{TAKD}}$ of Text-Aware
Knowledge Distillation for image semantic consistency is described as:

(8)
$\displaystyle\mathcal{L}_{\mathrm{TAKD}}=\mathrm{D}_{\mathrm{KL}}(p_{ij}||r_{ij}).$

Thus the loss function of our Text-Aware Image (TAI) tuning mechanism can be
described as
$\mathcal{L}_{\mathrm{TAI}}=\mathcal{L}_{\mathrm{TAM}}+\lambda_{\mathrm{V}}\mathcal{L}_{\mathrm{TAKD}}$.
$\lambda_{\mathrm{V}}$ is a hyper-parameter, which balances the text distance
adaptive margin loss and knowledge distillation loss. The total loss function
is:

$\displaystyle\mathcal{L}_{\mathrm{TIMA}}$
$\displaystyle=\mathcal{L}_{\mathrm{TAI}}+\lambda\mathcal{L}_{\mathcal{\mathrm{IAT}}}$
(9)
$\displaystyle=\mathcal{L}_{\mathrm{TAM}}+\lambda_{\mathrm{V}}\mathcal{L}_{\mathrm{TAKD}}+\lambda(\mathcal{L}_{\mathrm{MHE}}+\lambda_{\mathrm{T}}\mathcal{L}_{\mathrm{IAKD}}),$

where $\lambda$ is the hyper-parameter, which balances the
$\mathcal{L}_{\mathrm{TAI}}$ and $\mathcal{L}_{\mathrm{IAT}}$,
$\lambda_{\mathrm{V}}$ and $\lambda_{\mathrm{T}}$ are hyper-paeameters in TAI
and IAT tuning.

Table 1. Result of zero-shot accuracies. To evaluate robust accuracy, we utilize two white-box attacks, PGD (Madry et al., 2017) and AutoAttack (Croce and Hein, 2020), by adversarial fine-tuning the model on ImageNet. The average performance of each method is presented in the last column. The best accuracy among SOTA methods (TeCoA, PMG, LAAT) and our proposed method (TIMA) are in bold. |  |  ImageNet |  CIFAR10 |  CIFAR100 |  STL10 |  Tiny-ImageNet |  OxfordPet |  Food101 |  EuroSAT |  SUN |  DTD |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---
PGD | CLIP | 1.72 | 9.57 | 4.55 | 35.40 | 3.50 | 2.72 | 3.95 | 0.11 | 1.02 | 2.50 | 6.51
TeCoA | 41.00 | 60.61 | 34.17 | 83.49 | 40.99 | 62.31 | 28.16 | 14.72 | 31.13 | 22.77 | 41.93
PMG | 45.53 | 62.08 | 35.02 | 84.66 | 45.59 | 64.60 | 27.81 | 17.07 | 31.92 | 23.40 | 43.77
LAAT | 40.73 | 58.54 | 35.02 | 33.43 | 70.56 | 41.57 | 21.11 | 8.86 | 31.35 | 13.14 | 35.43
TIMA (Ours) | 46.09 | 65.43 | 43.95 | 84.96 | 47.85 | 67.92 | 28.59 | 15.69 | 33.88 | 29.68 | 46.41
AutoAttack | CLIP | 0.00 | 2.54 | 2.34 | 2.93 | 0.20 | 0.00 | 0.00 | 0.26 | 0.11 | 0.00 | 0.84
TeCoA | 40.88 | 59.28 | 32.13 | 82.45 | 41.95 | 61.61 | 27.99 | 12.69 | 29.34 | 22.18 | 41.05
PMG | 41.03 | 60.33 | 36.89 | 82.98 | 42.16 | 61.05 | 29.86 | 13.97 | 30.59 | 25.56 | 42.44
LAAT | 38.75 | 51.56 | 33.20 | 77.34 | 41.21 | 23.05 | 24.77 | 4.69 | 32.90 | 15.89 | 34.34
TIMA (Ours) | 46.17 | 64.06 | 42.19 | 84.96 | 46.48 | 65.31 | 28.67 | 15.17 | 33.86 | 28.13 | 44.54
Clean Accuracy | CLIP | 59.12 | 89.06 | 62.32 | 97.16 | 60.74 | 85.82 | 83.15 | 38.26 | 57.67 | 40.52 | 67.38
TeCoA | 63.21 | 78.63 | 50.07 | 93.66 | 61.37 | 81.58 | 55.93 | 26.29 | 53.01 | 34.40 | 59.85
PMG | 63.27 | 78.23 | 48.94 | 92.83 | 61.93 | 80.00 | 48.83 | 27.28 | 49.36 | 32.45 | 58.31
LAAT | 62.66 | 71.66 | 49.90 | 91.86 | 62.34 | 50.75 | 41.48 | 9.53 | 54.20 | 21.65 | 51.60
TIMA (Ours) | 69.25 | 85.74 | 63.87 | 93.36 | 68.36 | 84.52 | 61.88 | 20.38 | 57.63 | 38.67 | 64.37

##  4\. Experiments

###  4.1. Experimental Setup

####  4.1.1. Datasets and Baselines

In our study, we assess the zero-shot adversarial robustness of our proposed
method, TIMA, alongside several state-of-the-art (SOTA) methods, TeCoA (Mao et
al., 2022), PMG (Wang et al., 2024), and LAAT (Li et al., 2023b), adversarial
fine-tuned on both ImageNet (Deng et al., 2009) and Tiny-ImageNet (Deng et
al., 2009) as two different versions. The two datasets were selected to
evaluate the adaptability of TIMA to both large-scale datasets, like ImageNet,
and smaller-scale datasets, such as Tiny-ImageNet. Performance will be
analyzed using a variety of zero-shot test datasets besides the two datasets.
Specifically, we include CIFAR10, CIFAR100 (Krizhevsky et al., 2009), STL10
(Coates et al., 2011) for generic classification; OxfordPets (Parkhi et al.,
2012), Food101 (Bossard et al., 2014) for fine-grained classification; SUN397
(Xiao et al., 2010) for scene recognition; DTD (Cimpoi et al., 2014) for
texture recognition, and finally, EuroSAT (Helber et al., 2019) for domain-
specialized task.

For baselines, we utilize the foundational CLIP model and the aforementioned
SOTA methods under the same zero-shot settings previously established by these
methodologies. We report the performance of these models after fine-tuning on
both ImageNet and Tiny-ImageNet, ensuring that learning rate and adversarial
training duration are consistent with our experimental setup for fair
comparison. Detailed experimental results are presented in Tab. 1 and Tab. 2,
with additional information available in the supplementary materials to
provide a comprehensive evaluation of both our method and the compared methods
across the selected datasets.

####  4.1.2. Implementation Details

We utilized the CLIP-B/32 architecture for our experiments, with the prompt
text configured as ”This is a photo of a ”. The model was optimized using an
SGD optimizer with a momentum of 0.9. We conducted adversarial fine-tuning
using $l_{\infty}$ bounded attacks with $\varepsilon=1/255$ through a 2-step
PGD attack (Madry et al., 2017), also with a step size of 1/255. The
hyperparameters $m$ and $\eta$ for the Text-distance adaptive margin in the
TAI tuning mechanism were uniformly set at 0.1 and 0.95. The hyper-parameters
$\lambda,\lambda_{\mathrm{T}},\lambda_{\mathrm{V}}$ were set at $10^{-3},1,1$
for ImageNet and $1,1,1$ for Tiny-ImageNet. For adversarial fine-tuning on
ImageNet, we trained the model for 5 epochs using a learning rate of $10^{-4}$
and a batch size of 256. In the case of Tiny-ImageNet, the model was trained
over 10 epochs with a learning rate $10^{-4}$ but a batch size of 512.

Table 2. Result of zero-shot accuracies. Robust accuracy is under PGD-10 (Madry et al., 2017). All SOTA methods and our proposed method TIMA are adversarial fine-tuned on Tiny-ImageNet. We show the average performance of each method in the last column. The best accuracy is in bolded. |  |  ImageNet |  CIFAR10 |  CIFAR100 |  STL10 |  Tiny-ImageNet |  OxfordPet |  Food101 |  EuroSAT |  SUN |  DTD |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---
Robust Accuracy | TeCoA | 12.98 | 54.91 | 30.15 | 75.14 | 56.63 | 29.76 | 9.59 | 14.95 | 13.86 | 10.77 | 30.87
PMG | 15.17 | 56.67 | 34.19 | 76.89 | 52.52 | 31.45 | 13.10 | 15.11 | 16.30 | 10.82 | 32.22
LAAT | 13.30 | 57.50 | 33.15 | 76.54 | 56.72 | 28.35 | 12.83 | 12.33 | 13.53 | 8.51 | 31.28
TIMA (Ours) | 15.65 | 58.70 | 33.37 | 77.40 | 56.41 | 38.87 | 14.23 | 15.21 | 13.52 | 8.31 | 33.17
Clean Accuracy | TeCoA | 43.87 | 76.10 | 46.00 | 91.39 | 76.00 | 75.33 | 43.11 | 17.83 | 46.15 | 29.73 | 54.55
PMG | 46.08 | 78.47 | 50.31 | 92.10 | 70.98 | 76.62 | 53.45 | 23.72 | 45.85 | 31.12 | 56.87
LAAT | 42.72 | 68.08 | 42.72 | 87.75 | 75.84 | 51.21 | 34.81 | 9.44 | 43.92 | 18.14 | 47.46
TIMA (Ours) | 47.33 | 82.92 | 55.92 | 92.99 | 76.98 | 77.30 | 55.64 | 24.42 | 47.33 | 32.39 | 59.32

###  4.2. Comparison Results

We present an assessment of the robustness of our proposed method against
prevalent white-box adversarial attacks, encompassing PGD-10 (Madry et al.,
2017) (10 steps, step size of 1/255), and AutoAttack (Croce and Hein, 2020)
(AA), an effective attack used for evaluating adversarial robustness. Tab. 1
and 2 display the robust accuracy (zero-shot adversarial robustness) and clean
accuracy (zero-shot generalization) of state-of-the-art methods and our
proposed approach under the $l_{\infty}$ setting, with the maximum adversarial
perturbation set to $\varepsilon=1/255$. The best results for clean accuracy
and adversarial robustness under various attack methods for each dataset are
bolded. The final column in the table showcases the average adversarial
robustness (and clean accuracy) results across these datasets.

Tab. 1 and 2 present the clean accuracy and robust accuracy of adversarial
fine-tuning on ImageNet and Tiny-ImageNet, comparing existing methods with the
proposed TIMA. In Tab. 1, we showcase the robust accuracy under PGD-10 (Madry
et al., 2017) (step size as 1/255) and AutoAttack (Croce and Hein, 2020), with
a perturbation radius of $1/255$. Due to space constraints, Tab. 2 only
displays robust accuracy at a perturbation budget of $1/255$ under the PGD-10
attack. For AutoAttack robust accuracy of existing methods and our TIMA after
adversarial fine-tuning on Tiny-ImageNet, please refer to the supplementary
materials.

From Tab. 1, it is evident that our method, despite a marginal 3.01% (from
67.38% to 64.37%) decline in zero-shot clean accuracy (generalization)
compared to CLIP, achieves a remarkable 39.9% (from 6.51% to 46.41%)
improvement in performance under PGD-10 attacks and a substantial 43.7% (from
0.84% to 44.54%) enhancement against AutoAttack. In comparison with existing
SOTA methods, our method improves zero-shot clean accuracy by 4.52% (from
59.85% to 64.37%) over TeCoA, 6.06% (from 58.31% to 64.37%) over PMG, and by a
significant 12.77% (from 51.60% to 64.37%) over LAAT.

Regarding zero-shot robust accuracy, under PGD attack, our proposed method’s
zero-shot robust accuracy exceeds TeCoA by 4.48% (from 41.93% to 46.41%), PMG
by 2.64% (from 43.77% to 46.41%), and LAAT by 10.98% (from 35.43% to 46.41%).
Under AutoAttack, our method’s zero-shot robust accuracy is 3.49% higher than
TeCoA’s, 2.1% more than PMG’s, and 10.2% above LAAT’s. The data from Tab. 1
leads to the conclusion that, when adversarially fine-tuned on the ImageNet
dataset, our method achieves a favorable trade-off between zero-shot
adversarial robustness and zero-shot generalization. Moreover, the results
from Tab. 2 show that adversarial training on a smaller scale of dataset also
works our proposed method. From Tab. 2, we conclude that our proposed method
also a favorable trade-off between zero-shot adversarial robustness and zero-
shot generalization, when adversarially fine-tuned on the Tiny-ImageNet
dataset. The scale of adversarial fine-tuning dataset is not a constraint of
our proposed method.

Table 3. Zero-shot robust accuracy under different perturbation radius $\varepsilon$. We adversarial fine-tune the model on Tiny-ImageNet. The average performance of each method is shown in the last column. The best accuracies are bolded. |  |  ImageNet |  CIFAR10 |  CIFAR100 |  STL10 |  Tiny-ImageNet |  OxfordPet |  Food101 |  EuroSAT |  SUN |  DTD |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---
$\varepsilon=4/255$ | TeCoA | 1.99 | 10.62 | 6.71 | 42.91 | 17.97 | 6.62 | 1.40 | 0.53 | 1.22 | 1.54 | 9.15
PMG | 0.72 | 11.68 | 6.26 | 34.34 | 8.69 | 2.34 | 1.09 | 0.00 | 0.57 | 1.06 | 6.68
LAAT | 1.25 | 24.22 | 10.16 | 58.34 | 16.99 | 4.23 | 2.60 | 0.00 | 1.55 | 1.03 | 12.04
Ours | 11.89 | 32.76 | 15.80 | 69.26 | 44.94 | 35.11 | 11.49 | 0.00 | 7.04 | 3.30 | 23.16
$\varepsilon=8/255$ | TeCoA | 1.85 | 7.06 | 3.95 | 40.93 | 15.29 | 6.54 | 1.45 | 0.00 | 1.10 | 0.53 | 7.87
PMG | 0.54 | 5.51 | 2.07 | 30.06 | 4.13 | 2.34 | 1.13 | 0.00 | 0.42 | 0.37 | 4.66
LAAT | 1.04 | 23.44 | 9.34 | 58.59 | 14.67 | 4.11 | 2.67 | 0.00 | 1.62 | 0.32 | 11.58
Ours | 11.89 | 30.74 | 14.80 | 68.59 | 44.41 | 34.67 | 11.55 | 0.00 | 7.00 | 2.71 | 22.64

###  4.3. Result Analysis on $\varepsilon$ and $\tau$

####  4.3.1. Zero-shot Adversarial Robustness under Larger Perturbations

Tab. 3 illustrates the zero-shot adversarial robustness under the larger
perturbation radius ($\varepsilon=4/255,8/255$) with the robust accuracy by
PGD-10 (Madry et al., 2017) attack. For results on zero-shot adversarial
robustness with larger perturbations using AutoAttack, please refer to the
supplementary materials. The adversarial robustness at a smaller perturbation
radius of $1/255$ and the clean sample accuracy (zero-shot generalization)
results are shown in Tab. 2. From Tab. 3, under large perturbation
$\varepsilon=8/255$, our zero-shot robust accuracy improved by 14.77% (from
7.87% to 22.64%) compared to the TeCoA method and by 11.06% (from 11.58% to
22.64%) relative to the LAAT.

(a) Robust Accuracy

(b) Clean Accuracy

Figure 5. Zero-shot robust/clean accuracy of the proposed method under the CLIP temperature $\tau=1$. Table 4. Zero-shot accuracies of TeCoA (Mao et al., 2022) and LAAT (Li et al., 2023b) with our proposed tuning mechanism. We adversarial fine-tune the method on Tiny-ImageNet. CA and RA represent clean accuracy and robust accuracy. See 4/255 case in appendix. |  |  ImageNet |  CIFAR10 |  CIFAR100 |  STL10 |  Tiny-ImageNet |  OxfordPet |  Food101 |  EuroSAT |  SUN |  DTD |  Average
---|---|---|---|---|---|---|---|---|---|---|---|---
TeCoA | CA | 43.87 | 76.10 | 46.00 | 91.39 | 76.00 | 75.33 | 43.11 | 17.83 | 46.15 | 29.73 | 54.55
RA ($\varepsilon=1/255$) | 12.98 | 54.91 | 30.15 | 75.14 | 56.63 | 62.61 | 9.59 | 10.77 | 13.86 | 14.95 | 30.87
| RA ($\varepsilon=8/255$) | 1.85 | 7.06 | 3.95 | 40.93 | 15.29 | 6.54 | 1.45 | 0.00 | 1.10 | 0.53 | 7.87
TeCoA + IAT | CA | 44.50 | 76.74 | 45.92 | 91.94 | 77.11 | 76.75 | 41.47 | 20.16 | 44.18 | 28.62 | 54.74
RA ($\varepsilon=1/255$) | 12.34 | 55.16 | 30.19 | 75.11 | 58.07 | 28.73 | 9.21 | 14.99 | 13.04 | 13.51 | 31.04
RA ($\varepsilon=8/255$) | 5.64 | 23.62 | 14.08 | 48.97 | 19.19 | 6.49 | 1.65 | 0.00 | 1.93 | 0.78 | 12.24
TeCoA +TAI | CA | 46.26 | 83.80 | 55.50 | 93.13 | 77.20 | 77.19 | 55.73 | 19.63 | 47.77 | 33.46 | 58.97
RA ($\varepsilon=1/255$) | 15.00 | 60.11 | 33.98 | 77.44 | 56.43 | 37.89 | 13.79 | 1.39 | 13.47 | 15.43 | 32.49
RA ($\varepsilon=8/255$ ) | 8.36 | 22.66 | 14.06 | 66.02 | 41.89 | 19.92 | 10.61 | 0.00 | 8.08 | 2.93 | 19.45
LAAT | CA | 42.72 | 68.08 | 42.72 | 87.75 | 75.84 | 51.21 | 34.81 | 9.44 | 43.92 | 18.14 | 47.46
RA ($\varepsilon=1/255$) | 13.30 | 57.50 | 33.15 | 76.54 | 56.72 | 28.35 | 12.83 | 12.33 | 13.53 | 8.51 | 31.28
RA ($\varepsilon=8/255$) | 1.04 | 23.44 | 9.34 | 58.59 | 14.67 | 4.11 | 2.67 | 0.00 | 1.62 | 0.32 | 11.58
LAAT + TAI | CA | 44.96 | 74.51 | 49.27 | 89.26 | 77.34 | 73.63 | 55.80 | 23.07 | 45.36 | 25.12 | 55.84
RA ($\varepsilon=1/255$) | 13.56 | 57.81 | 31.64 | 77.15 | 58.20 | 32.37 | 15.35 | 14.76 | 12.82 | 8.79 | 32.25
RA ($\varepsilon=8/255$) | 10.78 | 30.04 | 12.85 | 69.17 | 45.22 | 29.34 | 7.40 | 0.00 | 10.39 | 1.37 | 21.66
TIMA (Ours) | CA | 47.33 | 82.92 | 55.92 | 92.99 | 76.98 | 77.30 | 55.64 | 24.42 | 47.33 | 32.39 | 59.32
RA ($\varepsilon=1/255$) | 15.65 | 58.70 | 33.37 | 77.40 | 56.41 | 38.87 | 14.23 | 15.21 | 13.52 | 8.31 | 33.17
RA ($\varepsilon=8/255$) | 11.89 | 30.74 | 14.80 | 68.59 | 44.41 | 34.67 | 11.55 | 0.00 | 7.00 | 2.71 | 22.64

####  4.3.2. Temperature Effect for Zero-shot Adversarial Robustness and
Generalization

In addition to its effectiveness against large perturbative adversarial
attacks, we observe that the proposed TIMA method remains efficacious across
various CLIP temperature $\tau$. As depicted in Fig. 5, an increase in the
CLIP temperature $\tau$ to 1 (contrary to other experiments within this study
where the default temperature coefficients for existing methods and our
proposed TIMA method are maintained at 0.01), the results demonstrate that our
TIMA notably excels over current SOTA methods in both robust and clean
accuracy. This result affirms the versatility and superior performance of the
TIMA method across a broad range of CLIP temperature $\tau$.

(a) Robust Accuracy vs. $m$

(b) Robust Accuracy vs. $\eta$

Figure 6. Zero-shot robust accuracy of our proposed method with different
hyper-parameter $m$ and $\eta$ in our proposed Text-distance Adaptive Margin.

###  4.4. Ablation Study

The ablation study is based on the proposed model which is adversarial fine-
tuning on Tiny-ImageNet. It should be noted that the robust accuracy in Sec.
4.4 is evaluated by PGD-10.

####  4.4.1. Contribution of Each Tuning Mechanism

Tab. 4 presents the results of an ablation study on tuning mechanisms (IAT and
TAI), showing the zero-shot clean accuracy and zero-shot robust accuracy zero-
shot clean accuracy across multiple test datasets under various perturbation
budgets. The results clearly demonstrate that IAT markedly improves
adversarial robustness under various perturbations. TAI is a plug-and-play
tuning mechanism which can be introduced into TeCoA and LAAT, leading to
increasing both zero-shot adversarial robustness and generalization. It should
be noted that IAT tuning mechanism is a learn-based text emmbedding adjustment
algorithm, while LAAT has fixed text expansion algorithms, so IAT cannot be
directly introduced into LAAT.

Comparing the results between TeCoA (LAAT) and TeCoA+TAI (LAAT+TAI), it is
evident that increasing the inter-class distances of image embeddings improves
zero-shot adversarial robustness. Further, the results between TeCoA+IAT,
TeCoA+TAI and TIMA demonstrate that increasing both text and image embedding
inter-class distances improves zero-shot adversarial robustness, particularly
under large perturbations, which can be the evidence of our hypothesis in Sec.
1.

####  4.4.2. Hypermeters $\beta,m$ in TAI Tuning Mechanism

We also present experimental results for the remaining hyper-parameters of
Text-distance Adaptive Margin in the TAI tuning mechanism in Fig. 6(a) and
6(b), which showcase the outcomes of experiments concerning the hyper-
parameters $\eta$ and $m$ in the Text-distance Adaptive Margin. As depicted in
Fig. 6(a), the optimal performance for zero-shot robust accuracy is observed
when the value of $m$ is set to 0.1. Similarly, from Fig. 6(b), it is evident
that the optimal and relatively stable performance is achieved when $\eta$
falls within the range of 0.9 to 0.95. Therefore, in our experiments, we set
$\eta$ and $m$ to 0.95 and 0.1.

##  5\. Conclusion

This study proposes an adversarial fine-tuning approach with two tuning
mechanisms (IAT and TAI) to balance zero-shot adversarial robustness and
generalization. IAT tuning achieves a balance between increasing inter-class
distance and semantic consistency between text embeddings by combining MHE and
IAKD. TAI tuning balances the enhanced inter-class distance and semantic
consistency between image embeddings by combining TAW and TAKD. Our results
affirm the importance at the inter-class distance for adversarial robustness
and highlight the role of semantic alignments for generalization. Both aspects
deserve future investigation.

## References

  * (1)
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022\.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ (2022).
  * Bossard et al. (2014) Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014.  Food-101–mining discriminative components with random forests. In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_. Springer, 446–461.
  * Bui et al. (2021) Anh Bui, Trung Le, He Zhao, Paul Montague, Seyit Camtepe, and Dinh Phung. 2021.  Understanding and achieving efficient robustness with adversarial supervised contrastive learning.  _arXiv preprint arXiv:2101.10027_ (2021).
  * Carlini and Wagner (2017) Nicholas Carlini and David Wagner. 2017.  Towards evaluating the robustness of neural networks. In _2017 ieee symposium on security and privacy (sp)_. Ieee, 39–57.
  * Cimpoi et al. (2014) Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. 2014.  Describing textures in the wild. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3606–3613.
  * Coates et al. (2011) Adam Coates, Andrew Ng, and Honglak Lee. 2011.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In _Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics_ _(Proceedings of Machine Learning Research, Vol. 15)_ , Geoffrey Gordon, David Dunson, and Miroslav Dudík (Eds.). PMLR, Fort Lauderdale, FL, USA, 215–223.  https://proceedings.mlr.press/v15/coates11a.html
  * Croce and Hein (2020) Francesco Croce and Matthias Hein. 2020.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In _International conference on machine learning_. PMLR, 2206–2216.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Ding et al. (2019) Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. 2019.  MMA Training: Direct Input Space Margin Maximization through Adversarial Training. In _International Conference on Learning Representations_.
  * Helber et al. (2019) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. 2019.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ 12, 7 (2019), 2217–2226.
  * Hou et al. (2023) Pengyue Hou, Jie Han, and Xingyu Li. 2023.  Improving adversarial robustness with self-paced hard-class pair reweighting. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 37. 14883–14891.
  * Huang et al. (2023) Zeyi Huang, Andy Zhou, Zijian Ling, Mu Cai, Haohan Wang, and Yong Jae Lee. 2023.  A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 11685–11695.
  * Inkawhich et al. (2023) Nathan Inkawhich, Gwendolyn McDonald, and Ryan Luley. 2023.  Adversarial Attacks on Foundational Vision Models.  _arXiv preprint arXiv:2308.14597_ (2023).
  * Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021.  Scaling up visual and vision-language representation learning with noisy text supervision. In _International conference on machine learning_. PMLR, 4904–4916.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * Li et al. (2023a) Xuanlin Li, Yunhao Fang, Minghua Liu, Zhan Ling, Zhuowen Tu, and Hao Su. 2023a.  Distilling large vision-language model with out-of-distribution generalizability. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2492–2503.
  * Li et al. (2021) Xin Li, Xiangrui Li, Deng Pan, and Dongxiao Zhu. 2021.  Improving adversarial robustness via probabilistically compact loss with logit constraints. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 8482–8490.
  * Li et al. (2023b) Xiao Li, Wei Zhang, Yining Liu, Zhanhao Hu, Bo Zhang, and Xiaolin Hu. 2023b.  Anchor-Based Adversarially Robust Zero-Shot Learning Driven by Language.  _arXiv preprint arXiv:2301.13096_ (2023).
  * Lin et al. (2020) Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng, Zhiding Yu, James M Rehg, Li Xiong, and Le Song. 2020.  Regularizing neural networks via minimizing hyperspherical energy. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6917–6927.
  * Liu et al. (2018) Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. 2018.  Learning towards minimum hyperspherical energy.  _Advances in neural information processing systems_ 31 (2018).
  * Liu et al. (2021) Weiyang Liu, Rongmei Lin, Zhen Liu, Li Xiong, Bernhard Schölkopf, and Adrian Weller. 2021.  Learning with hyperspherical uniformity. In _International Conference On Artificial Intelligence and Statistics_. PMLR, 1180–1188.
  * Liu et al. (2016) Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. 2016.  Large-margin softmax loss for convolutional neural networks.  _arXiv preprint arXiv:1612.02295_ (2016).
  * Liu et al. (2022) Weiyang Liu, Longhui Yu, Adrian Weller, and Bernhard Schölkopf. 2022.  Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap. In _The Eleventh International Conference on Learning Representations_.
  * Liu and Chan (2022) Ziquan Liu and Antoni B Chan. 2022.  Boosting adversarial robustness from the perspective of effective margin regularization.  _arXiv preprint arXiv:2210.05118_ (2022).
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ (2017).
  * Mao et al. (2022) Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. 2022.  Understanding zero-shot adversarial robustness for large-scale models.  _arXiv preprint arXiv:2212.07016_ (2022).
  * Mao et al. (2019) Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, and Baishakhi Ray. 2019.  Metric learning for adversarial robustness.  _Advances in neural information processing systems_ 32 (2019).
  * Michels et al. (2023) Felix Michels, Nikolas Adaloglou, Tim Kaiser, and Markus Kollmann. 2023.  Contrastive Language-Image Pretrained (CLIP) Models are Powerful Out-of-Distribution Detectors.  _arXiv preprint arXiv:2303.05828_ (2023).
  * Mustafa et al. (2019) Aamir Mustafa, Salman Khan, Munawar Hayat, Roland Goecke, Jianbing Shen, and Ling Shao. 2019.  Adversarial defense by restricting the hidden space of deep neural networks. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3385–3394.
  * Noever and Noever (2021) David A Noever and Samantha E Miller Noever. 2021.  Reading Isn’t Believing: Adversarial Attacks On Multi-Modal Neurons.  _arXiv preprint arXiv:2103.10480_ (2021).
  * Parkhi et al. (2012) Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. 2012.  Cats and dogs. In _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 3498–3505.
  * Radenovic et al. (2023) Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. 2023.  Filtering, distillation, and hard negatives for vision-language pre-training. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6967–6977.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ 1, 2 (2022), 3.
  * Wang et al. (2023) Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. 2023\.  On the robustness of chatgpt: An adversarial and out-of-distribution perspective.  _arXiv preprint arXiv:2302.12095_ (2023).
  * Wang et al. (2024) Sibo Wang, Jie Zhang, Zheng Yuan, and Shiguang Shan. 2024.  Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness.  _arXiv preprint arXiv:2401.04350_ (2024).
  * Wang et al. (2020) Zhennan Wang, Canqun Xiang, Wenbin Zou, and Chen Xu. 2020.  MMA regularization: Decorrelating weights of neural networks by maximizing the minimal angles.  _Advances in Neural Information Processing Systems_ 33 (2020), 19099–19110.
  * Xiao et al. (2010) Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010.  Sun database: Large-scale scene recognition from abbey to zoo. In _2010 IEEE computer society conference on computer vision and pattern recognition_. IEEE, 3485–3492.
  * Yang et al. (2023) Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, and Yongjun Xu. 2023.  CLIP-KD: An Empirical Study of Distilling CLIP Models.  _arXiv preprint arXiv:2307.12732_ (2023).
  * Yang and Mirzasoleiman (2023) Wenhan Yang and Baharan Mirzasoleiman. 2023.  Robust Contrastive Language-Image Pretraining against Adversarial Attacks.  _arXiv preprint arXiv:2303.06854_ (2023).
  * Zang et al. (2023) Yuhang Zang, Hanlin Goh, Joshua M Susskind, and Chen Huang. 2023.  Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization. In _The Twelfth International Conference on Learning Representations_.
  * Zhao et al. (2023) Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, and Min Lin. 2023.  On evaluating adversarial robustness of large vision-language models.  _arXiv preprint arXiv:2305.16934_ (2023).
  * Zhou et al. (2024) Andy Zhou, Jindong Wang, Yu-Xiong Wang, and Haohan Wang. 2024.  Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Zhou and Patel (2022) Mo Zhou and Vishal M Patel. 2022.  Enhancing adversarial robustness for deep metric learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15325–15334.