capbtabboxtable[][]

# Consistency-guided Prompt Learning for Vision-Language Models

Shuvendu Roy, Ali Etemad
Dept. ECE and Ingenuity Labs Research Institute
Queen’s University, Kingston, Canada
{shuvendu.roy<EMAIL_ADDRESS>
###### Abstract

We propose Consistency-guided Prompt learning (CoPrompt), a new fine-tuning
method for vision-language models that addresses the challenge of improving
the generalization capability of large foundation models while fine-tuning
them on downstream tasks in a few-shot setting. The basic idea of CoPrompt is
to enforce a consistency constraint in the prediction of the trainable and
pre-trained models to prevent overfitting on the downstream task.
Additionally, we introduce the following two components into our consistency
constraint to further boost the performance: enforcing consistency on two
perturbed inputs and combining two dominant paradigms of tuning, prompting and
adapter. Enforcing consistency on perturbed input further regularizes the
consistency constraint, effectively improving generalization, while tuning
additional parameters with prompting and adapters improves the performance on
downstream tasks. Extensive experiments show that CoPrompt outperforms
existing methods on a range of evaluation suites, including base-to-novel
generalization, domain generalization, and cross-dataset evaluation tasks. On
the generalization task, CoPrompt improves the state-of-the-art by 2.09% on
the zero-shot task and 1.93% on the harmonic mean over 11 recognition
datasets. Detailed ablation studies show the effectiveness of each of the
components in CoPrompt.

##  1 Introduction

Figure 1: Comparison between the CoPompt and existing prompting approach.

Vision-Language foundation models (e.g., CLIP [1]) that are trained on large-
scale datasets of image-text pairs have demonstrated excellent generalization
capabilities. However, the sheer size of these models can make it challenging
to fine-tune them for downstream tasks, especially for small downstream tasks
(e.g., few-shot learning), while preserving their ability to generalize [2,
3]. To overcome this challenge, various methods have been proposed to fine-
tune these large foundation models by adding and tuning extra parameters (such
as prompting [3] or adapter[4]), while keeping the pre-trained weights frozen.
Prompt-based methods introduce additional tunable weights in the input space
(i.e., with image [5], text [3], or both [6] embeddings), whereas adapter-
based methods [4] add learnable weights inside the network, typically near the
prediction head.

Despite these advancements in few-shot fine-tuning, it is still a challenge to
maintain the generalization capability of the original models such as CLIP,
let alone improve them. In fact, it has been shown that improvements in few-
shot performance often result in a drop in zero-shot capabilities (e.g., CoOp
[2]). This is caused by severe overfitting on newly introduced parameters
during few-shot fine-tuning, resulting in a significant deviation from the
foundation model’s original behaviour.

Figure 2: Performance comparison on base-to-novel generalization.

Method | Base | Novel | HM
---|---|---|---
CLIP | 69.34 | 74.22 | 71.70
CoOp | 82.69 | 63.22 | 71.66
Co-CoOp | 80.47 | 71.69 | 75.83
ProGrad | 82.48 | 70.75 | 76.16
KgCoOp | 80.73 | 73.60 | 77.00
MaPLe | 82.28 | 75.14 | 78.55
CoPrompt | 84.00 | 77.23 | 80.48

Figure 3: Average accuracy on 11 datasets for base-to-novel generalization
task.

In this work, we propose Consistency-guided Prompt learning (CoPrompt), a new
fine-tuning method for vision-language models that reduces the overfitting
problem by preventing the trainable model’s embeddings from deviating too far
from the pre-trained model’s embedding. More specifically, we enforce a
consistency constraint on both the language and image branches between the
trainable and pre-trained models, as illustrated in Figure 1. Furthermore, we
introduce two additional components to improve the proposed consistency
constraint. First, we enforce consistency on two perturbed inputs instead of
the same input to further regularize the consistency constraint, effectively
improving generalization. On the ‘text’ branch, we use a pre-trained large
language model (LLM), GPT [7], to generate a more detailed and descriptive
sentence from an input prompt text of a generic format (a photo of a ‘class’).
We then enforce consistency between the learnable and pre-trained text encoder
on the representations of these two sentences. On the image branch, we apply
augmentations on an input image to generate two perturbed images. Second, we
add more trainable parameters to increase the model’s learning capacity on the
new task. To this end, we combine the two dominant paradigms of tuning,
prompting [6], and adapters [4]. By combining prompt and adapter, we can tune
more parameters to improve the performance on the new task while the
consistency constraint maintains or possibly improves the model’s zero-shot
capability.

Extensive experiments on three common evaluation settings, including base-to-
novel generalization, domain generalization, and cross-dataset evaluation,
demonstrate the strong performances of CoPrompt. In the base-to-novel
generalization task, CoPrompt outperforms existing methods on 11 benchmark
datasets (as illustrated in Figure 3), achieving a 2.09% improvement on novel
classes and a 1.93% improvement on the harmonic mean (as shown in Table 3)
over MaPLe, the previous SOTA. Importantly, our improvements do not come at
the expense of the performance of the base class, which also achieves a 1.72%
improvement over MaPLe. Additionally, CoPrompt achieves considerable
improvements over existing methods on cross-dataset evaluation and domain
generalization. An extensive ablation study confirms the importance of each
component of the proposed method. In summary, this paper makes the following
contributions: (1) We propose a consistency-enforced fine-tuning method for
large foundation models that enables learning a new task from a few samples
without losing its zero-shot generalizability. (2) Our method incorporates the
knowledge of a pre-trained LLM with consistency constraints on the text branch
and data augmentations on the image branch to improve the generalization
further. (3) Our method combines the two strong paradigms of tuning foundation
models, prompting and adapter, into a single framework to improve performance
on new tasks. (4) We set a new state-of-the-art for a range of evaluation
suites, including base-to-novel generalization, cross-dataset recognition, and
domain generalization.

##  2 Related Work

Recent developments in vision-language models, such as CLIP [1], ALIGN [8],
LiT [9], FILIP [10], and Florence [11], have exhibited impressive performance
in various vision tasks, including few-shot and zero-shot learning. However,
the enormous size of these models makes it challenging to fine-tune them
without losing their generalization. The two commonly used approaches for
using a pre-trained model for a downstream task are (a) full fine-tuning and
(b) linear probing. However, neither of these methods performs well for
foundation models. Full fine-tuning results in a loss of generalization, while
linear probing often leads to poor performance on downstream tasks [6].
Consequently, many recent studies have focused on adapting large foundation
models on downstream tasks without changing the pre-trained weights [2].
Existing works in this direction can be categorized into two main groups:
Prompting [3, 6] and Adapter [4].

Prompts are typically instructions in the form of text that guide the
downstream task. They can either be manually crafted for a specific task or
learned automatically. The latter method is called prompt tuning, which was
initially introduced in [12, 13, 14]. In this context, CoOp [2] introduced a
set of continuous vectors into the language branch’s input, which is optimized
with the final loss. However, this approach demonstrated poor performance on
unseen classes, indicating poor generalization on zero-shot tasks. CoCoOp [3]
improved CoOp’s zero-shot performance by explicitly conditioning on the image
inputs. ProGrad [15] only updated the prompts where the gradients aligned with
the original prompt’s defined general knowledge. Prompting techniques have
also been utilized for dense prediction tasks [16]. While the earlier works on
prompting added prompts only to the text input, some recent works have also
explored prompting on the image inputs [5]. Later, MaPLe took a multi-modal
approach that used prompting on both image and text inputs. This method
explicitly ensured mutual synergy between the language and vision prompts to
discourage learning from unimodal data. This multi-modal prompting design
forms the backbone of our proposed method, CoPrompt, on which we introduce the
proposed consistency constraint to improve generalization.

Another method for tuning foundation models is Adapters. This approach
introduces learnable parameters to one or multiple layers of the pre-trained
model to transform features [4]. Adapters are typically added to the upper
layers of the network, which can be seen as a learnable transformation module
for the pre-trained model. Adapters have also been studied in vision-only
models, including dense prediction tasks [17]. In our work, we combine both
prompting and adapter into a single framework that enhances downstream task
performance. The additional tunable parameters allow for better adaptation to
downstream tasks, while the consistency constraint avoids overfitting and
ensures better generalization.

##  3 Method

In this section, we present the details of the proposed CoPormpt method.
First, we discuss the preliminaries on vision-language models and prompting
required for our method, followed by the specifics of the proposed CoPrompt
method.

###  3.1 Preliminaries

We adopt CLIP [1] as the pre-trained vision-language foundation model in our
method. CLIP consists of a transformer-based image encoder, $\theta$, and a
transformer-based text encoder, $\phi$. CLIP performs zero-shot prediction by
freezing the pre-trained encoders and searching for the highest similarity
between the embedding of an input image and the embeddings of hand-crafted
text prompts for all class names. CLIP generates the hand-crafted text prompt
following the template ‘a photo of a [category]’. With $C$ being the number of
classes, the text embedding for all class names can be represented as
$W=\\{w_{k}\\}_{k}^{C}$, where $w_{k}=\phi$(‘a photo of a [category]k’). For
an input image $x$, the image embedding is extracted by the image encoder as
$z=\theta(x)$. Finally, CLIP makes a zero-shot prediction as follows:

$p({y}|x)=\frac{\text{exp}(sim(z,w_{y})/\tau)}{\sum_{k=1}^{C}\text{exp}(sim(z,w_{k})/\tau)},$
(1)

where, $\tau$ is a temperature parameter, and $sim(.)$ is the cosine
similarity.

Even though CLIP shows strong zero-shot performance, it requires further
tuning to perform well on new downstream tasks. Additionally, the hand-crafted
template approach does not perform well universally across domains. To this
end, CoOp [2] proposed a solution by replacing the hand-crafted prompt with a
set of learnable continuous context vectors that generate task-specific text
embeddings. More specifically, CoOp replaced the fixed sentence ‘a photo of a
[category]’ with $m$ learnable prompt vectors ($u_{k}$) of the same dimension
as the word embedding of CLIP. These are concatenated with the word embedding
of the class name $c_{k}$, yielding $t_{k}=\\{u_{1},u_{2},....u_{m},c_{k}\\}$.
In multi-modal promoting methods [6], an input image is first projected to
patch embeddings ({$p_{1},p_{2},...p_{d}$}), and then learnable context
vectors are concatenated, yielding
$i=\\{v_{1},v_{2},..v_{m},p_{1},p_{2},...p_{d}\\}$, where $d$ is the number of
image patches. Consequently, the modified prediction objective of CLIP with
learnable prompt can be expressed as:

$p({y}|x)=\frac{\text{exp}(sim(\theta(i),\phi(t_{y}))/\tau)}{\sum_{k=1}^{C}\text{exp}(sim(\theta(i),\phi(t_{k}))/\tau)}.$
(2)

Figure 4: Overview of the proposed CoPrompt.

In this work, we build upon the multi-modal prompting concept of MaPLe [6] to
fine-tune CLIP for downstream tasks. A major limitation of the existing fine-
tuning methods is that the zero-shot generalizability is severely hampered
when new learnable parameters are fine-tuned for a specific downstream task
which is caused by overfitting on the downstream dataset. In the following
section, we introduce CoPrompt, a novel approach that addresses this problem
and enhances the performance for both downstream tasks and zero-shot
prediction.

###  3.2 CoPrompt: Consistency-guided Prompt Learning

CoPrompt tackles the issue of reduced generalization due to overfitting on the
downstream task, by implementing a consistency constraint that ensures the
text and image embeddings produced by the trainable model (tunable prompt
parameters in both the image and text branches) are not significantly
different from those generated by the pre-trained CLIP. To further impose
regularization in the consistency constraint, we utilize perturbations to the
input for the trainable and pre-trained models. On the language branch, a pre-
trained LLM is used to generate more descriptive sentences from the template
text input, while on the image branch, we use augmentations. In addition,
CoPrompt includes additional trainable parameters by adding adapters on the
image and text branches to enhance performance on new downstream tasks. The
consistency constraint, input perturbation, and adapters are discussed in
further detail below.

Consistency constraint. We use cosine distance as the consistency constraint
between the embeddings of the pre-trained encoder and the learnable encoder.
However, other similar criteria, like Euclidean distance, can also be used as
a constraint. We empirically observe that cosine distance as the consistency
constraint yields the best performance. This constraint is applied on both
image and text branches. Following the notation introduced in the
preliminaries, we can denote the consistency constraint as:

$\mathcal{L}_{cc}=2-\frac{w_{y}\cdot\phi(t_{y})}{||w_{y}||~{}||\phi(t_{y})||}-\frac{z\cdot\theta(i)}{||z||~{}||\theta(i)||}.$
(3)

Here, $y$ is the class label of the input image.

Input perturbation. Given the template text ‘a photo of a [category]’, we use
a pre-trained LLM, GPT($\phi_{GPT}$), to generate a more descriptive sentence
as $s_{k}=\phi_{GPT}$(‘a photo of a [category]k’). For this, we follow the
training setup of KgCoOP [18]. But unlike KgCoOp, we generate a single
sentence on the fly rather than generating a pre-defined number of sentences
and averaging their embedding. On the image branch, we use an augmentation
module $\delta$ to generate perturbed image $x^{\prime}=\delta(x)$. Now we
enforce the consistency between the embedding of the perturbed input to the
pre-trained model and the learnable model as:

$\mathcal{L}_{cc}=2-\frac{\phi(s_{y})\cdot\phi(t_{y})}{||\phi(s_{y})||~{}||\phi(t_{y})||}-\frac{\theta(x^{\prime})\cdot\theta(i)}{||\theta(x^{\prime})||~{}||\theta(i)||}.$
(4)

Adapters. We incorporate more trainable parameters for better adaptation to
the new task. We do so by adding adapters [4] to both the vision and language
branches. Adapters are trainable parameters that are added on top of the
encoder to transform the embedding vector. Following [4], we define our
adapter as two linear layers with non-linearity in between. But unlike [4], we
do not restrict the adapter only to the text branch; rather use it on both.
Let $\phi^{a}$ be the text adapter that takes a text embedding $w_{k}$ as
input and transforms it as $\phi^{a}(w_{k})$. Similarly, $\theta^{a}$ is the
image adapter. Incorporating that into our proposed consistency constraint,
the proposed consistency constraint loss can be represented as:

$\mathcal{L}_{cc}=2-\frac{\phi(s_{y})\cdot\phi^{a}(\phi(t_{y}))}{||\phi(s_{y})||~{}||\phi^{a}(\phi(t_{y}))||}-\frac{\theta(x^{\prime})\cdot\theta^{a}(\theta(i))}{||\theta(x^{\prime})||~{}||\theta^{a}(\theta(i))||}.$
(5)

Final loss.  The proposed consistency constraint loss is combined with a
supervised loss to form the final loss. We represent the supervised loss as:

$\mathcal{L}_{ce}=-log\frac{\text{exp}(sim(z,w_{y})/\tau)}{\sum_{k=1}^{C}\text{exp}(sim(z,w_{k})/\tau)}.$
(6)

Adding both losses with a balancing factor $\lambda$, we get the final loss
function of CoPrompt is:

$\mathcal{L}=\mathcal{L}_{ce}+\lambda\mathcal{L}_{cc}$ (7)

##  4 Experiments

###  4.1 Experiment Setup

To evaluate the proposed method, we follow the experiment setup and protocols
established in CoOp [2] and subsequent works, such as CoCoOp [3], and MaPLe
[6]. Here, we first describe the datasets and training details, followed by a
discussion on the evaluation protocol.

####  4.1.1 Datasets and Training details

We evaluate our model’s performance on 11 datasets that cover various
recognition tasks, including generic object classification datasets such as
ImageNet [19] and Caltech101 [20], fine-grained recognition datasets such as
OxfordPets [21], StanfordCars [22], Flowers102 [23], Food101 [24], and
FGVCAircraft [25], as well as scene recognition dataset SUN397 [26], action
recognition dataset UCF101 [27], satellite-image classification dataset
EuroSAT [28], and texture recognition dataset DTD [29]. Moreover, to evaluate
domain generalization, we utilize four variants of the ImageNet dataset,
including ImageNetV2 [30], ImageNet-Sketch [31], ImageNet-A [32], and
ImageNet-R [33].

We adopt CLIP (ViT-B/16) [1] as the pre-trained vision-language backbone to
evaluate our proposed method. We fine-tune the model in few-shot settings with
16 samples per class for all known classes. The model is trained with an SGD
optimizer for 8 epochs, using a batch size of 4 and a learning rate of 0.035.
We report the average accuracy over 3 runs on the individual experiments. The
training is conducted on a single Nvidia V100 GPU, and the experiment takes
slightly less than a day to train and evaluate on all 11 datasets.

####  4.1.2 Evaluation Protocols

Base-to-novel class generalization. The most prevalent approach for evaluating
performance in zero-shot scenarios while fine-tuning on a few-shot setting is
through base-to-novel generalization testing. This involves partitioning the
dataset into known (base) and unseen (novel) classes, followed by training the
model using a small number of samples from the base classes. Finally, the
model’s performance is evaluated on both base (few-shot performance) and novel
(zero-shot performance) categories. We also report the harmonic mean [34] over
known and unknown classes.

Cross-dataset evaluation. In this experiment, we assess the zero-shot ability
of the model on a cross-dataset evaluation setup. Specifically, we train the
model in a few-shot setting on one dataset and subsequently evaluate its
performance on ten other unseen datasets that contain unknown categories. We
adopt the evaluation protocol from CoCoOp [3] and MaPLe [6] for this
experiment.

Domain generalization. In addition, we assess the model’s performance on out-
of-distribution generalization. For this experiment, we first fine-tune the
model on one dataset and then test the model on different datasets that
contain the same classes but from different distributions.

Table 1: Comparison with state-of-the-art methods on base-to-novel
generalization. The best accuracies are bolded and the absolute gains over the
previous SOTA (MaPLe) are indicated in blue. HM indicates the harmonic mean.

(a) Average

| Base | Novel | HM
---|---|---|---
CLIP | 69.34 | 74.22 | 71.70
CoOp | 82.69 | 63.22 | 71.66
Co-CoOp | 80.47 | 71.69 | 75.83
ProGrad | 82.48 | 70.75 | 76.16
KgCoOp | 80.73 | 73.60 | 77.00
MaPLe | 82.28 | 75.14 | 78.55
CoPrompt | 84.00 | 77.23 | 80.48
| +1.72 | +2.09 | +1.93

(b) ImageNet

| Base | Novel | HM
---|---|---|---
CLIP | 72.43 | 68.14 | 70.22
CoOp | 76.47 | 67.88 | 71.92
Co-CoOp | 75.98 | 70.43 | 73.10
ProGrad | 77.02 | 66.66 | 71.46
KgCoOp | 75.83 | 69.96 | 72.78
MaPLe | 76.66 | 70.54 | 73.47
CoPrompt | 77.67 | 71.27 | 74.33
| +1.01 | +0.73 | +0.86

(c) Caltech101

| Base | Novel | HM
---|---|---|---
CLIP | 96.84 | 94.00 | 95.40
CoOp | 98.00 | 89.81 | 93.73
Co-CoOp | 97.96 | 93.81 | 95.84
ProGrad | 98.02 | 93.89 | 95.91
KgCoOp | 97.72 | 94.39 | 96.03
MaPLe | 97.74 | 94.36 | 96.02
CoPrompt | 98.27 | 94.90 | 96.55
| +0.53 | +0.54 | +0.53

(d) OxfordPets

| Base | Novel | HM
---|---|---|---
CLIP | 91.17 | 97.26 | 94.12
CoOp | 93.67 | 95.29 | 94.47
Co-CoOp | 95.20 | 97.69 | 96.43
ProGrad | 95.07 | 97.63 | 96.33
KgCoOp | 94.65 | 97.76 | 96.18
MaPLe | 95.43 | 97.76 | 96.58
CoPrompt | 95.67 | 98.10 | 96.87
| +0.24 | +0.34 | +0.29

(e) StanfordCars

| Base | Novel | HM
---|---|---|---
CLIP | 63.37 | 74.89 | 68.65
CoOp | 78.12 | 60.40 | 68.13
Co-CoOp | 70.49 | 73.59 | 72.01
ProGrad | 77.68 | 68.63 | 72.88
KgCoOp | 71.76 | 75.04 | 73.36
MaPLe | 72.94 | 74.00 | 73.47
CoPrompt | 76.97 | 74.40 | 75.66
| +4.03 | +0.40 | +2.19

(f) Flowers102

| Base | Novel | HM
---|---|---|---
CLIP | 72.08 | 77.80 | 74.83
CoOp | 97.60 | 59.67 | 74.06
Co-CoOp | 94.87 | 71.75 | 81.71
ProGrad | 95.54 | 71.87 | 82.03
KgCoOp | 95.00 | 74.73 | 83.65
MaPLe | 95.92 | 72.46 | 82.56
CoPrompt | 97.27 | 76.60 | 85.71
| +1.35 | +4.14 | +3.15

(g) Food101

| Base | Novel | HM
---|---|---|---
CLIP | 90.10 | 91.22 | 90.66
CoOp | 88.33 | 82.26 | 85.19
Co-CoOp | 90.70 | 91.29 | 90.99
ProGrad | 90.37 | 89.59 | 89.98
KgCoOp | 90.50 | 91.70 | 91.09
MaPLe | 90.71 | 92.05 | 91.38
CoPrompt | 90.73 | 92.07 | 91.4
| +0.02 | +0.02 | +0.02

(h) FGVCAircraft

| Base | Novel | HM
---|---|---|---
CLIP | 27.19 | 36.29 | 31.09
CoOp | 40.44 | 22.30 | 28.75
Co-CoOp | 33.41 | 23.71 | 27.74
ProGrad | 40.54 | 27.57 | 32.82
KgCoOp | 36.21 | 33.55 | 34.83
MaPLe | 37.44 | 35.61 | 36.50
CoPrompt | 40.2 | 39.33 | 39.76
| +2.76 | +3.72 | +3.26

(i) SUN397

| Base | Novel | HM
---|---|---|---
CLIP | 69.36 | 75.35 | 72.23
CoOp | 80.60 | 65.89 | 72.51
Co-CoOp | 79.74 | 76.86 | 78.27
ProGrad | 81.26 | 74.17 | 77.55
KgCoOp | 80.29 | 76.53 | 78.36
MaPLe | 80.82 | 78.70 | 79.75
CoPrompt | 82.63 | 80.03 | 81.31
| +1.81 | +1.33 | +1.56

(j) DTD

| Base | Novel | HM
---|---|---|---
CLIP | 53.24 | 59.90 | 56.37
CoOp | 79.44 | 41.18 | 54.24
Co-CoOp | 77.01 | 56.00 | 64.85
ProGrad | 77.35 | 52.35 | 62.45
KgCoOp | 77.55 | 54.99 | 64.35
MaPLe | 80.36 | 59.18 | 68.16
CoPrompt | 83.13 | 64.73 | 72.79
| +2.77 | +5.55 | +4.63

(k) EuroSAT

| Base | Novel | HM
---|---|---|---
CLIP | 56.48 | 64.05 | 60.03
CoOp | 92.19 | 54.74 | 68.69
Co-CoOp | 87.49 | 60.04 | 71.21
ProGrad | 90.11 | 60.89 | 72.67
KgCoOp | 85.64 | 64.34 | 73.48
MaPLe | 94.07 | 73.23 | 82.35
CoPrompt | 94.6 | 78.57 | 85.84
| +0.53 | +5.34 | +3.49

(l) UCF101

| Base | Novel | HM
---|---|---|---
CLIP | 70.53 | 77.50 | 73.85
CoOp | 84.69 | 56.05 | 67.46
Co-CoOp | 82.33 | 73.45 | 77.64
ProGrad | 84.33 | 74.94 | 79.35
KgCoOp | 82.89 | 76.67 | 79.65
MaPLe | 83.00 | 78.66 | 80.77
CoPrompt | 86.9 | 79.57 | 83.07
| +3.90 | +0.91 | +2.30

###  4.2 Base to Novel Generalization

In this section, we present the results of our proposed method on the base-to-
novel generalization task. Table 1 presents a detailed comparison of our
method with CLIP [1], CoOp [2], CoCoOp [3], ProGrad [15], KgCoOp [18], and
MaPLe [6]. We have highlighted the best results in bold and marked the
improvement over the previous state-of-the-art (MaPLe) in blue. As we see from
the average over all datasets (Table 1(a)), CoPrompt outperforms all existing
methods for both novel and base categories by a large margin. Our method
demonstrates strong zero-shot generalization, with an improvement of 2.09% on
novel categories, increasing the SOTA from 75.14% to 77.23%. Apart from MaPLe,
no existing method outperformed the pre-trained CLIP (which was not fine-
tuned), indicating the difficulty of maintaining zero-shot performance while
learning a new task in a few-shot setting. Along with a large improvement in
zero-shot performance, CoPormpt also improves the few-shot performance by
1.72% on base categories. This confirms that improvement in zero-shot
performance does not come at the cost of few-shot performance or vice versa.
In fact, CoPrompt is the first method to beat all existing methods in both
base and novel categories since the introduction of prompt tuning of VLMs in
CoOp. On a harmonic mean, CoPrompt provides a 1.93% improvement over existing
methods.

Observing the performance of individual datasets, we find that CoPrompt
outperforms MaPLe on all the datasets. Furthermore, it beats all existing
methods on 8 out of 11 datasets. The best accuracy for the remaining three
datasets is held by CoOp. It is important to note that CoOp was heavily over-
fitted on the base classes, resulting in the worst average accuracy (63.22%)
on novel categories, which is 14.34% less than CoPrompt. Overall, CoPrompt
still shows an improvement of 3.01% for the average of the 11 datasets,
demonstrating its ability to learn new tasks while maintaining strong
generalizability. On the novel categories, CoPrompt outperforms all existing
methods except the Flower102 dataset, where the accuracy of CoPrompt is
slightly behind CLIP. CoPrompt improves the zero-shot performance on novel
categories by up to 6.34% (on the EuroSAT dataset). Furthermore, the
performance of CoPrompt is unchallenged on the harmonic mean across all
datasets. The highest improvement of 4.64% over the previous SOTA in the
harmonic mean is observed for DTD.

Table 2:  Performance of CoPrompt on cross-dataset evaluation and its
comparison to existing methods. Here, the model is trained on the ImageNet
dataset and evaluated on ten other datasets in a zero-shot setting

| Source | Target
---|---|---
|  ImNet |  Caltech |  Pets |  Cars |  Flowers |  Food |  Aircraft |  SUN397 |  DTD |  EuroSAT |  UCF |  _Ave._
CoOp | 71.51 | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 | 63.88
Co-CoOp | 71.02 | 94.43 | 90.14 | 65.32 | 71.88 | 86.06 | 22.94 | 67.36 | 45.73 | 45.37 | 68.21 | 65.74
MaPLe | 70.72 | 93.53 | 90.49 | 65.57 | 72.23 | 86.20 | 24.74 | 67.01 | 46.49 | 48.06 | 68.69 | 66.30
CoPrompt | 70.80 | 94.50 | 90.73 | 65.67 | 72.30 | 86.43 | 24.00 | 67.57 | 47.07 | 51.90 | 69.73 | 67.00

Table 3: Performance on domain generalization.

| Source | Target |
---|---|---|---
| ImNet | ImNetV2 | ImNetS | ImNetA | ImNetR | Ave.
CLIP | 66.73 | 60.83 | 46.15 | 47.77 | 73.96 | 57.17
UPT | 72.63 | 64.35 | 48.66 | 50.66 | 76.24 | 59.98
CoOp | 71.51 | 64.20 | 47.99 | 49.71 | 75.21 | 59.28
Co-CoOp | 71.02 | 64.07 | 48.75 | 50.63 | 76.18 | 59.90
ProGrad | 72.24 | 64.73 | 47.61 | 49.39 | 74.58 | 59.07
KgCoOp | 71.20 | 64.10 | 48.97 | 50.69 | 76.70 | 60.11
MaPLe | 70.72 | 64.07 | 49.15 | 50.90 | 76.98 | 60.26
CoPrompt | 70.80 | 64.25 | 49.43 | 50.50 | 77.51 | 60.42

###  4.3 Cross-dataset Evaluation

In Table 2, we present the results for cross-dataset evaluation. Here, the
model is fine-tuned on a source dataset (ImageNet) and evaluated on target
datasets in a zero-shot manner. As we see, CoPrompt shows improvements on 9
out of 10 datasets. Overall, CoPromt provides an average accuracy of 67.0%,
which is 0.70% higher than the previous state-of-the-art, MaPLe. We obtain the
highest improvement of 3.84% over MaPLe on the EuroSAT dataset.

###  4.4 Domain Generalization

We present the results for domain generalization in Table 3. Here, the
original ImageNet dataset is used as the source dataset to fine-tune the
model. The model is then tested on four other variants of ImageNet that come
from different distributions. CoPrompt shows strong domain generalization by
outperforming MaPLe on all datasets except for ImageNet-A. It achieves a new
state-of-the-art average accuracy of 60.43% on this task.

###  4.5 Ablation Study

####  4.5.1 Main ablation

In this section, we present an ablation study by removing different components
of the proposed method to understand the importance of each of them. We show
the results of these experiments in Table 4, where ‘Cons.’, ‘In. Pert.’, and
‘Adp.’ represent the consistency constraint, input perturbations, and adapter,
respectively. For reference, in the first row of the table, we present the
final performance of CoPrompt,

Table 4: Ablation Study

Cons. | In. Pert. | Adp. | Accuracy
---|---|---|---
✓ | ✓ | ✓ | 80.48
✓ | ✓ | ✗ | 80.02
✓ | ✗ | ✓ | 79.56
✓ | ✗ | ✗ | 79.50
✗ | ✗ | ✓ | 78.45
✗ | ✗ | ✗ | 78.55

which has a harmonic mean of 80.48%. In the first ablation experiment, we
eliminate the adapters from CoPrompt, leading to an accuracy of 80.02% (a
performance drop of 0.46%). This highlights the importance of adapters in
CoPrompt. Next, we remove the input perturbations, effectively enforcing
consistency between the trainable and pre-trained encoder for the same image
and text input. This results in an accuracy of 79.56%, which is a 0.92% drop
in performance, suggesting the high importance of input perturbations in
CoPrompt. Finally, while we are interested in understanding the importance of
the consistency constraint, we can not only remove this component since the
input perturbation is also a part of it. So, to understand the impact of this
component, we perform two separate studies. First, we remove both the input
perturbations and the adapters, which results in an average accuracy of
79.50%. This shows that utilizing the consistency constraint alone provides a
0.95% improvement over removing all three components (as shown in the last row
of the table). In the second study, we remove the consistency constraint along
with the input perturbations, effectively training the adapters and prompts
without enforcing consistency. Surprisingly, this results in an accuracy of
78.45%, even lower than when all three components are removed. This is caused
by the fact that the adapters introduce new parameters to train, which causes
the trainable model to overfit to the few training samples when the
consistency constraint is not enforced. These two experiments clearly indicate
the significance of the consistency constraint in CoPrompt. In the following
section, we present an in-depth analysis of each of these components and their
performance for different plausible alternates.

####  4.5.2 Analysis of consistency constraints

Here, we present an analysis of the key aspects of the consistency constraint
in CoPrompt. While the consistency constraint is applied to both image and
text modalities, we focus on understanding the impact of the consistency
constraint on image and text modalities individually. The results of applying
consistency on individual modalities are presented in Table
LABEL:tab:ablation_consistency_modality. The findings reveal that enforcing
consistency on text representations has a greater significance compared to
that on image representations. Specifically, using a text-only constraint
results in a 0.46% drop in performance, while an image-only constraint leads
to a 0.89% decrease. The highest performance is achieved when consistency is
enforced on both modalities. Next, we investigate the influence of different
consistency criteria. Table LABEL:tab:ablation_consistency compares the
performance when using cosine distance, MSE, and L1 as consistency
constraints. The results demonstrate that cosine distance performs the best as
a consistency loss, while L1 shows a very similar performance. However,
employing MSE leads to a drop in performance.

####  4.5.3 Analysis of input perturbations

In this section, we explore the impact of different input perturbations on the
performance of the final model. Table LABEL:tab:ablation_text_input presents
the results for using the same text as input compared to using more
descriptive text generated by the LLM as input. The findings reveal a 0.39%
decrease in performance when the same input is used for both the learnable and
frozen text encoders. This emphasizes the importance of utilizing LLMs to
generate more descriptive text on which to enforce consistency.

Similarly, we conduct a study on the image inputs, as shown in Table
LABEL:tab:ablation_image_input. We compare the results when using the same
image, simple augmented image, and hard augmented image as inputs. Consistent
with the observations on text analysis, using the same image input shows a
decline in performance since it fails to provide a sufficiently discriminative
signal for learning. On the contrary, we expect hard augmentations to yield
the best discriminative features for learning and, consequently, the highest
accuracy. However, the results indicate that simple augmentations (random
resized crop, horizontal flip) outperform hard augmentations (RandAug [35]).
We believe the images generated by hard augmentations are too different for
the consistency constraint to provide the most optimal learning signals,
resulting in performances that are slightly less than using no augmentations
at all.

####  4.5.4 Analysis of adapters

Lastly, we delve into several important factors concerning the design of the
adapters. First, we present the results of incorporating adapters on different
modalities, as shown in Table LABEL:tab:ablation_adapter_modality. Consistent
with previous findings [4], we observe that adding an adapter to the text
branch yields more benefits compared to the image branch (80.35% compared to
80.1%). However, contrary to the findings in [4], we do not observe a drop in
performance when adapters are employed on both modalities. In fact, the
highest accuracy is achieved when adapters are used on both the image and text
branches. This emphasizes that while naive few-shot tuning does not benefit
from utilizing adapters on both branches, the proposed consistency-guided
tuning approach facilitates learning via more tunable parameters on both
modalities.

We also explore the impact of the number of linear layers used in the adapter
design. We evaluate three different configurations: a single-layer adapter, a
2-layer adapter as proposed in [4], and a 3-layers adapter. The results are
presented in Table LABEL:tab:ablation_adapter_layers. The findings indicate
that the 2-layer adapter slightly outperforms the single-layer design. This
suggests that incorporating an additional layer allows for capturing more
complex relationships and improving performance to some extent. However, using
a 3-layer adapter exhibits a significant drop in performance. We believe this
is because adding too many linear layers in the adapters can introduce an
excessive number of parameters, which may re-introduce the overfitting problem
to the limited training examples available in few-shot settings.

Consistency | Accuracy
---|---
Image only | 79.59
Text only | 80.02
Both | 80.48

(a)

Criterion | Accuracy
---|---
Cosine | 80.48
L1 | 80.40
MSE | 79.33

(b)

Input | Accuracy
---|---
Same Text | 80.09
LLM generated | 80.48

(c)

Input | Accuracy
---|---
Same Image | 80.16
Simple Aug. | 80.48
Hard Aug. | 79.90
(d)

Adapter | Accuracy
---|---
Text only | 80.35
Image only | 80.10
Both | 80.48
(e)

Layers | Accuracy
---|---
Single layer | 80.40
2 layers | 80.48
3 layers | 79.75
(f)

Table 5: Analysis of different components of CoPrompt.

####  4.5.5 Sensitivity study

Table 6:  Performance of CoPrompt for different values of $\lambda$.

|  ImNet |  Caltech |  Pets |  Cars |  Flowers |  Food |  Aircraft |  SUN397 |  DTD |  EuroSAT |  UCF
---|---|---|---|---|---|---|---|---|---|---|---
0.1 | 73.82 | 96.22 | 96.87 | 74.44 | 84.15 | 91.73 | 37.60 | 79.95 | 67.42 | 85.84 | 81.93
1.0 | 74.05 | 96.44 | 96.86 | 75.43 | 84.99 | 91.43 | 38.69 | 80.40 | 70.09 | 84.04 | 82.57
2.0 | 74.14 | 96.41 | 96.77 | 75.28 | 84.89 | 91.40 | 39.76 | 80.72 | 72.25 | 81.46 | 83.07
8.0 | 74.33 | 96.55 | 96.84 | 75.66 | 85.71 | 91.43 | 39.37 | 81.31 | 72.79 | 78.63 | 82.77

In this section, we present a sensitivity analysis of the proposed method to
some of its key parameters. First, we investigate the impact of the weight
factor ($\lambda$) of the consistency constraint loss. The results are
reported in Table 6. We observe that higher values of $\lambda$ lead to better
accuracy, indicating the high importance of the consistency constraint.
Specifically, we achieve the best accuracy for $\lambda=8$ on 6 out of 11
datasets and close to the best accuracy on the remaining datasets, except for
EuroSAT dataset. Factors such as data distribution and its similarity to the
pre-training dataset of CLIP influence the value of $\lambda$, resulting in
different optimal values for different datasets.

Table 7: Sensitivity studies.

Layers | Acc.
---|---
3 | 78.77
6 | 79.05
9 | 80.15
12 | 80.48

(g)

Epochs | Acc.
---|---
3 | 79.54
5 | 80.24
8 | 80.48
10 | 80.02

(h)

Next, we explore how the performance of CoPrompt varies with different numbers
of prompt layers and present the results in Table
LABEL:tab:sensitivity_prompt_layers. While previous works like MaPLe [6] have
shown the best results with 9 prompt layers, we observe that adding prompts on
all 12 layers of the encoder leads to the best accuracy. The consistency
constraint introduced in CoPrompt enables us to train more parameters without
overfitting. Table LABEL:tab:sensitivity_train_epochs shows the results for
different numbers of training epochs. We find that the best performance is
achieved with 8 epochs.

##  5 Conclusion

We present a novel tuning method for large vision-language foundation models
that enhances their performance in downstream tasks and also improve zero-shot
generalization. CoPrompt is a carefully designed method with three important
components that reduce the overfitting problem during fine-tuning. Through
extensive evaluations across three different tasks, CoPrompt demonstrated its
effectiveness in few-shot learning, zero-shot learning, cross-dataset, and
domain generalization tasks, surpassing the existing state-of-the-art by a
significant margin. Additionally, the study includes extensive ablation
analysis to confirm the effectiveness of each proposed component and explore
feasible alternatives. We believe that our consistency-guided tuning approach
will have a substantial impact on tuning vision and vision-language models for
various applications.

Limitations.  Firstly, the training of CoPrompt is slightly more
computationally expensive since the forward pass needs to compute the
embedding for both pre-trained and trainable encoders. Secondly, CoPrompt
utilizes CLIP’s text encoder to predict the class name, limiting its
predictions to the vocabulary of the text encoder.

## Acknowledgements

We would like to thank Bank of Montreal and Mitacs for funding this research.
We are also thankful to SciNet HPC Consortium for helping with the computing
resources.

## References

  * [1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [2] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130(9):2337–2348, 2022\.
  * [3] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825, 2022.
  * [4] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.  Clip-adapter: Better vision-language models with feature adapters.  arXiv preprint arXiv:2110.04544, 2021.
  * [5] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.  Exploring visual prompts for adapting large-scale models.  arXiv preprint arXiv:2203.17274, page 4, 2022.
  * [6] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan.  Maple: Multi-modal prompt learning.  arXiv preprint arXiv:2210.03117, 2022.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
  * [8] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [9] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022.
  * [10] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [11] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al.  Florence: A new foundation model for computer vision.  arXiv preprint arXiv:2111.11432, 2021.
  * [12] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * [13] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  arXiv preprint arXiv:2101.00190, 2021.
  * [14] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  arXiv preprint arXiv:2110.07602, 2021.
  * [15] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang.  Prompt-aligned gradient for prompt tuning.  arXiv preprint arXiv:2205.14865, 2022.
  * [16] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18082–18091, 2022.
  * [17] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao.  Vision transformer adapter for dense predictions.  arXiv preprint arXiv:2205.08534, 2022.
  * [18] Hantao Yao, Rui Zhang, and Changsheng Xu.  Visual-language prompt tuning with knowledge-guided context optimization.  arXiv preprint arXiv:2303.13283, 2023.
  * [19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248–255. Ieee, 2009.
  * [20] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 178–178. IEEE, 2004.
  * [21] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3498–3505. IEEE, 2012.
  * [22] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 554–561, 2013.
  * [23] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008.
  * [24] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In The European Conference on Computer Vision, pages 446–461, 2014\.
  * [25] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [26] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3485–3492, 2010.
  * [27] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [28] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.
  * [29] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014.
  * [30] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In International Conference on Machine Learning, pages 5389–5400. PMLR, 2019.
  * [31] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing.  Learning robust global representations by penalizing local predictive power.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [32] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262–15271, 2021.
  * [33] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021.
  * [34] Yongqin Xian, Bernt Schiele, and Zeynep Akata.  Zero-shot learning-the good, the bad and the ugly.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4582–4591, 2017.
  * [35] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020.