# Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal
Large Language Models

Yue Zhang Hehe Fan Yi Yang
School of Computer Science and Technology
Zhejiang University

###### Abstract

To bridge the gap between vision and language modalities, Multimodal Large
Language Models (MLLMs) usually learn an adapter that converts visual inputs
to understandable tokens for Large Language Models (LLMs). However, most
adapters generate consistent visual tokens, regardless of the specific objects
of interest mentioned in the prompt. Since these adapters distribute equal
attention to every detail in the image and focus on the entire scene, they may
increase the cognitive load for LLMs, particularly when processing complex
scenes. To alleviate this problem, we propose prompt-aware adapters. These
adapters are designed with the capability to dynamically embed visual inputs
based on the specific focus of the prompt. Specifically, prompt-aware adapters
utilize both global and local textual features to capture the most relevant
visual clues from the prompt at both coarse and fine granularity levels. This
approach significantly enhances the ability of LLMs to understand and
interpret visual content. Experiments on various visual question answering
tasks, such as counting and position reasoning, demonstrate the effectiveness
of prompt-aware adapters.

##  1 Introduction

Recent advances in Large Language Models (LLMs) brown2020language ;
chung2022scaling ; openai2022chatgpt ; touvron2023llama ; chiang2023vicuna
have significantly enhanced their performance across various natural language
processing tasks. These models are able to perform language comprehension and
logical reasoning, enabling them to handle complex linguistic functions such
as summarizing texts, answering questions, processing dialogues, and composing
new essays or articles. However, LLMs are inherently limited by their
inability to process visual information. This has led to the development of
Multimodal Large Language Models (MLLMs) driess2023palm ; yang2023mm ;
li2023videochat ; doveh2024dense ; peng2023kosmos , which expand upon LLMs by
incorporating visual processing abilities. MLLMs offer a more holistic
understanding and interaction by synthesizing both textual and visual data,
thereby broadening their utility in diverse real-world applications.

Figure 1: Illustration comparing prompt-unaware and prompt-aware adapters.
Left: Prompt-unaware adapter treats visual patches as a kind of words and
directly converts these patches into “readable” tokens for LLMs, without
considering the specific objects of interest. In this case, whether the
question involves “pool” or “drinks,” they consistently generate the same
tokens and allocate equal attention to every detail in the scene, which may
increase the cognitive load for LLMs. Right: Prompt-aware adapter leverages
prompts to collect the most relevant visual clues and generate adaptive tokes,
thus enhancing the ability of LLMs to understand and interpret visual content.

To equip LLMs with visual perception ability, MLLMs typically employ a
trainable adapter that connects a frozen visual encoder and a frozen LLM.
Adapters play a crucial role in bridging the gap between vision and language,
enabling profound visual understanding while leveraging the powerful reasoning
capabilities of LLMs. However, most existing adapters liu2023visual ;
chen2023minigpt ; chen2023shikra ; zeng2023matters ; su2023pandagpt ;
peng2023kosmos ; zhou2023infmllm ; cha2023honeybee approach visual patches as
if they were words, and directly translate these patches into tokens
comprehensible to LLMs (e.g., through linear projection), regardless of the
specific objects of interest in the prompt. These prompt-unaware adapters may
enable LLMs to correctly analyze simple images but could struggle to
understand complex scenes. As shown in Fig. 1, no matter the query is “is
there a pool in this image” or “how many drinks are on the table”, these
adapters consistently convert the image into the same tokens. Consequently,
the subsequent LLMs have to independently parse the scene to deduce the
spatial context and shift attention to “pool” or “drinks” entirely on their
own.

A few studies have explored using prompts to guide the behavior of adapters.
For instance, VisionLLM wang2023visionllm and Flamingo alayrac2022flamingo
employs the cross-attention mechanism to learn adaptive visual tokens, where
prompt words serve as the query and image patches are treated as the key and
value. InstructBLIP dai2024instructblip first injects the prompt information
into learned queries li2023blip (via self-attention) and then uses cross-
attention to gather visual clues. These cross-attention-based adapters
encounter two main challenges. First, they search for visual clues at the word
level, thereby neglecting global information for capturing an overview of the
prompt-related region. Second, the use of the softmax function in cross-
attention normalizes attention distribution from word to patch such that the
total attention allocated to each word equals 1. This implies that every word
in the prompt, including function words like “a”, “the” and “is”, is forced to
correspond to a specific region in the image. Because these irrelevant words
may correspond to different regions each time, this unrealistic assumption may
lead adapters to produce unstable visual tokens, inevitably causing
significant confusion for LLMs.

In this paper, we produce a prompt-aware adapter to adaptively embed visual
inputs based on the prompt’s global and local representations. Our adapter
comprises two key components: prompt-aware global attention and prompt-aware
local attention. The global attention is designed to capture coarse-grained,
prompt-related visual perceptions, while the local attention focuses on
refining responses to specific, fine-grained regions of interest. This dual
approach allows adapters to effectively uncover visual contexts and shift
attention to relevant areas as needed. We conducted extensive experiments on
the COCO-QA ren2015exploring and MME fu2023mme datasets. Compared to the
prompt-unaware baseline, our method shows significant improvements on COCO-QA,
enhancing object classification, counting, color recognition, and positional
reasoning by 7.71%, 18.42%, 12.84%, and 9.51%, respectively. On the MME
dataset, it boosts the total scores for perception tasks and cognition tasks
by 59.43% and 46.91%, respectively. Our method demonstrates superior
performance in handling complex scenarios and parsing intricate problems,
effectively sensing prompts and capturing informative details necessary for
answering questions. The contributions are as follows:

  * •

Among the early efforts, we conduct a comprehensive study on the impact of
prompts on adapters. Our research reveals that prompt-independent adapters may
be not sufficient to capture the most informative visual clues for visual
understanding. Furthermore, existing prompt-relevant adapters still suffer
from issues such as failing to focus on crucial words in prompts or adequately
aligning visual and semantic elements.

  * •

We propose prompt-aware adapters that dynamically embed visual inputs based on
prompts at both coarse and fine granularities. Our method provides effective
and convenient attention mechanisms for existing adapter-empowered MLLMs.

  * •

Experiments on complex scene understanding applications demonstrate that the
proposed method effectively improves the visual perception and reasoning
capabilities of MLLMs.

##  2 Related Work

Multimodal Large Language Models and Adapters. The integration of the
perceptual abilities of vision models kirillov2023segment ; mokady2021clipcap
; oquab2023dinov2 ; radford2021learning with the reasoning capabilities of
LLMs brown2020language ; ouyang2022training ; chowdhery2022palm ;
thoppilan2022lamda ; zhang2022opt ; openai2022chatgpt has given rise to
Multimodal Large Language Models (MLLMs) yang2024doraemongpt ; hu2024minicpm ;
chen2023sharegpt4v ; zheng2024dreamlip ; li2024mini ; li2024cir . In MLLMs,
visual signals are converted into tokens that LLMs can understand, typically
using adapters. For example, MLLMs such as LLaVA liu2023visual , Shikra
chen2023shikra , MiniGPT chen2023minigpt , Matters zeng2023matters , PandaGPT
su2023pandagpt , Kosmos peng2023kosmos , and InfMLLM zhou2023infmllm utilize
linear projection adapters. QWEN-VL bai2023qwen and mPlUG-Owl ye2023mplug
employ Perceiver-like architectures jaegle2021perceiver as adapters. Since the
introduction of the Q-Former in BLIP-2 li2023blip , many MLLMs zhu2023minigpt
; yu2023reformulating have adopted it as an adapter for modal alignment. The
visual descriptions generated by the aforementioned prompt-independent
adapters typically encompass the entire image, without emphasizing the
specific visual details relevant to the prompt. VisionLLM wang2023visionllm ,
Flamingo alayrac2022flamingo , and subsequent studies gong2023multimodal ;
li2023otter utilize cross-attention adapters to improve prompt-aware
capabilities. InstructBLIP dai2024instructblip and Cheetor li2023fine
incorporate prompt information into learned queries using self-attention and
then employ cross-attention to collect visual clues. These adapters often
struggle with issues such as failing to focus on crucial words in prompts or
adequately aligning visual and semantic elements. In this paper, we introduce
a new type of adapter designed to align visual signals with textual semantics
for MLLMs.

Attention Mechanisms. The attention mechanism DBLP:journals/corr/BahdanauCB14
enables vision-language models to focus on relevant information within the
input while minimizing focus on irrelevant details. Its effectiveness and
interpretability have inspired numerous subsequent studies lu2023multi ;
zheng2023less ; yao2023attention . For example, xu2015show introduces both
hard and soft attention mechanisms that automatically learn to describe the
contents of images. Additionally, in the Visual Question Answering (VQA) task,
it is crucial to focus on the relevant parts of both the image and the
question. Consequently, Co-Attention lu2016hierarchical was developed to
facilitate simultaneous attention learning across multiple inputs (i.e., image
and text). Self-Attention yang2016hierarchical enables interactions between
different positions within a single input sequence, thereby capturing content
that is relatively more important. Drawing from sources such as
yang2016hierarchical ; lin2017structured ; rocktaschel2017frustratingly , the
Transformer vaswani2017attention , recognized as a foundational component of
large models, incorporates Scaled Dot-Product Attention and Multi-Head
Attention. Cross-Attention wei2020multi has been developed within the
Transformer vaswani2017attention architecture to facilitate interaction
between two different sequences. To enhance the interaction between images and
text, we propose global and local attention mechanisms that automatically
focus on the visual content of interest mentioned in the text.

##  3 Proposed Method

In this section, We first provide a brief overview of existing adapters for
the visual perception of Multimodal Large Language Models (MLLMs). Then, we
describe how the proposed prompt-aware adapter enables MLLMs to effectively
uncover visual context and adaptively shift attention for enhancing visual
reasoning.

###  3.1 Preliminary: MLLM and Adapter

MLLM is an advanced type of artificial intelligence model that processes and
understands information from multiple types of data, such as text, images, and
sometimes audio or video. This allows the model to perform tasks that involve
more than one mode of communication, providing a richer understanding of the
content than the text-only LLM. MLLMs integrate capabilities from different
domains. For example, they can extract meaning from both the text in an image
caption and the visual content of the image itself. This is particularly
useful in applications such as image captioning, Visual Question Answering
(VQA) and video understanding.

In MLLMs, a specialized module called adapter is commonly employed to enhance
the model’s capability to process and integrate information from different
modalities. Adapters are particularly valuable because they allow pre-trained
models to be adapted to new tasks or modalities without the need for extensive
retraining of the entire model. Formally, for a given visual input, a frozen
visual encoder (such as ViT and Q-Former) is typically employed to extract the
visual features ${\bm{X}}\in\mathbb{R}^{N\times C}$, where $N$ represents the
number of patches and $C$ denotes the number of feature channels. Because
these visual features are not directly comprehensible to the subsequent LLM,
the adapter is trained to translate them into tokens that the LLM can
understand as follows,

${\bm{X}}^{\prime}=\mathrm{Adapter}({\bm{X}};\theta),$ (1)

where $\theta$ is the learnable parameters of the adapter,
${\bm{X}}^{\prime}\in\mathbb{R}^{N^{\prime}\times C^{\prime}}$ and
$N^{\prime}$ represents the number of converted tokens and $C^{\prime}$
denotes the dimension of the LLM’s input. In most MLLMs, the adapter is
implemented as a linear projection layer that projects visual features into
the textual feature space liu2023visual ; chen2023minigpt ; zeng2023matters ;
su2023pandagpt ; chen2023shikra . Additionally, the adapter in Honeybee
cha2023honeybee first applies a convolution to convert visual features and
then downsamples them, resulting in $N^{\prime}<N$. However, no matter linear
projection or convolution, these adapters distribute equal attention to every
detail in the visual input and focus on the entire scene, regardless of the
specific objects of interest mentioned in the prompt, converting visual input
to invariant tokens. In this case, these prompt-unaware adapters may struggle
to effectively convert complex scenes for its subsequent LLM.

Figure 2: Illustration comparing the cross-attention (left) and proposed
(right) adapters. (a) Methods like VisionLLM wang2023visionllm and Flamingo
alayrac2022flamingo utilize text features as queries and visual features as
keys and values in cross-attention. It assumes that each word in the prompt
corresponds to specific regions. The number of converted visual tokens is
equal to that of text features. (b) InstructBLIP dai2024instructblip first
injects prompt information into learnable queries via self-attention, and then
employs cross-attention. It assumes that each query in the learnable queries
corresponds to specific regions. The number of converted visual tokens is
equal to that of learnable queries. (c) Our adapter comprises global and local
attention components. Due to the new attention calculation mechanism used in
local attention, the number of converted visual tokens remains unchanged.

###  3.2 Prompt-Aware Adapter

To translate informative visual tokens for LLMs, we can use prompts to guide
the behavior of adapters. Suppose ${\bm{Y}}\in\mathbb{R}^{M\times D}$ is the
prompt, where $M$ represents the number of words in the prompt and $D$ denotes
the dimension of the word embedding. This mechanism can be formulated as
follows,

${\bm{X}}^{\prime}=\mathrm{Adapter}({\bm{X}},{\bm{Y}};\theta).$ (2)

There are many ways to implement Eq. (2). As shown in Fig 2, VisionLLM
wang2023visionllm and Flamingo alayrac2022flamingo utilize text features as
queries and visual features as keys and values in cross-attention processes to
facilitate modal interaction. InstructBLIP dai2024instructblip first
incorporates prompt information into learned queries through self-attention,
and then employs cross-attention to collect visual clues. In summary, these
methods are based on cross-attention, which can be formulated as follows,

${\bm{X}}^{\prime}=\mathrm{softmax}(\frac{{\bm{Q}}\cdot{\bm{K}}^{T}}{\sqrt{E}})\cdot{\bm{V}},~{}~{}~{}~{}~{}~{}~{}{\bm{Q}}={\bm{Y}}\cdot{\bm{W}}_{q},~{}~{}{\bm{K}}={\bm{X}}\cdot{\bm{W}}_{k},~{}~{}{\bm{V}}={\bm{X}}\cdot{\bm{W}}_{v},$
(3)

where ${\bm{W}}_{q}\in\mathbb{R}^{D\times E}$,
${\bm{W}}_{k}\in\mathbb{R}^{C\times E}$, ${\bm{W}}_{v}\in\mathbb{R}^{C\times
E}$ and $\cdot$ denotes matrix multiplication. In this paper, we use the
lowercase $\mathrm{softmax}$ to indicate that the function is applied at each
row. This mechanism results in two outcomes:

  * •

Attention. Because the use of the softmax function in cross-attention
normalizes the attention distribution from each word to all patches, the total
attention allocated to each word equals 1. Specifically, if ${\bm{a}}_{ij}$
represents the attention of the $i$-th word to the $j$-th patch, then
$\sum_{j=1}^{N}{\bm{a}}_{ij}=1$.

  * •

Output. Because the prompt ${\bm{Y}}$ serves as the query in cross-attention,
it leads to ${\bm{X}}^{\prime}\in\mathbb{R}^{M\times E}$. This indicates that
the number of converted visual tokens is $M$ (the number of words in the
prompt), rather than $N$ (the number of patches in the visual input).

This suggests that every word in the prompt, including function words like
“a”, “an”, “the”, “is” and “are”, is forced to correspond to a specific region
in the image. Given that these irrelevant words might correspond to different
regions each time, this unrealistic assumption can cause adapters to generate
unstable visual tokens, leading to significant confusion for LLMs.

Figure 3:  Illustration of the proposed prompt-aware adapter. The adapter
consists of a global attention component and a local attention component. The
global attention, integrated into the visual encoder, is designed to capture
coarse-grained, prompt-related visual perceptions. Meanwhile, the local
attention focuses on refining responses to specific, fine-grained regions of
interest.

Global Attention. To extract the most informative clues, we propose the
prompt-aware adapter (shown in Fig. 3). The adapter consists of a prompt-aware
global attention component and a prompt-aware local attention component. The
global attention component searches for visual clues at the scene level,
aiming to capture an overview of the region related to the prompt. To this
end, we first utilize CLIP’s text encoder to extract the global feature of the
prompt. Second, we employ a learned projection layer to map the global feature
into the same space as the visual patch, i.e., ${\bm{y}}\in\mathbb{R}^{1\times
C}$. Third, the global feature of the prompt is appended to the visual
patches, resulting in $[{\bm{X}};y]\in\mathbb{R}^{(N+1)\times C}$. Fourth,
self-attention is employed to allow the adapter to incorporate the global
prompt information, which is as follows,

${\bm{I}}=\mathrm{SelfAttention}([{\bm{y}};{\bm{X}}]).$ (4)

In this way, a coarse extraction of visual clues mentioned in the prompt is
conducted. Finally, the global attention discards the prompt representation at
the last position, yielding $N$ visual features, i.e.,
${\bm{I}}\in\mathbb{R}^{N\times C_{i}}$, where $C_{i}$ is the feature
dimension.

Local Attention. Global attention reflects the overall correlation between the
visual input and prompt. To capture local details, we design a local attention
component. First, local attention calculates a similarity matrix
${\bm{S}}\in\mathbb{R}^{N\times M}$ between text and visual features, as
follows,

${\bm{S}}=\mathrm{SOFTMAX}\Big{(}\frac{\big{(}{\bm{I}}\cdot{\bm{W}}_{i})\cdot({\bm{Y}}\cdot{\bm{W}}_{y})^{T}}{\sqrt{E}}\Big{)}$
(5)

where ${\bm{W}}_{i}\in\mathbb{R}^{C_{i}\times E}$ and
${\bm{W}}_{y}\in\mathbb{R}^{D\times E}$. Here, we use the capital
$\mathrm{SOFTMAX}$ to denote that the function is applied to the entire
matrix. Compared to the $\mathrm{softmax}$ function used in Eq. (3),
$\mathrm{SOFTMAX}$ suggests that each word may or may not correspond to a
visual patch in the scene, and similarly, each patch may or may not correspond
to a word in the prompt. This assumption is more flexible and realistic than
that of cross-attention.

Second, the sum of each row ($\mathbb{R}^{1\times M}$) in ${\bm{S}}$ can be
interpreted as the correlation between a specific patch and the entire prompt
description. Therefore, we calculate the weight of each visual patch by
summing all attention focused on it, as follows,

${\bm{a}}_{i}=\sum_{j=1}^{M}{\bm{S}}_{ij},~{}~{}~{}{\bm{X}}^{\prime}=\mathrm{MLP}({\bm{a}}^{T}\cdot{\bm{I}}),$
(6)

where ${\bm{a}}\in\mathbb{R}^{1\times N}$. Then, local attention is applied to
visual features. This process produces prompt-aware visual tokens. Finally, a
multilayer perceptron (MLP) is employed to transform them.

Our method intrinsically differs from the cross-attention mechanism described
in Eq. (3). In cross-attention, prompt features ${\bm{Y}}$ function as
${\bm{Q}}$ (queries) and visual features ${\bm{X}}$ as ${\bm{K}}$ (keys) and
${\bm{V}}$ (values). Here, each text token distributes its attention across
visual patches such that the total sums to 1. This means that even function
words (like conjunctions and prepositions) must have their similarity scores
with all visual tokens collectively equal 1. This requirement is impractical,
as it assumes that every text token is equally significant in directing
attention to visual cues. The proposed local attention mechanism ensures that
$\sum_{i=1}^{N}\sum_{j=1}^{M}{\bm{S}}_{i,j}=1$. As a result, text tokens that
are more relevant to the visual context (and have higher similarity scores)
exert greater influence on visual encoding. This approach, akin to emphasizing
certain words in a sentence when reading aloud, allows for the extraction of
prompt-related visual features in a more detailed and nuanced manner. In
summary, there are two differences between cross-attention and the proposed
local attention.

  * •

Attention. The cross-attention implies that each word corresponds to a
specific region, whereas local attention does not enforce such a
correspondence.

  * •

Output. The cross-attention produces $M$ (the number of words in the prompt)
visual tokens, whereas local attention generates $N$ (the number of patches in
the visual input) tokens.

##  4 Experiments

###  4.1 Implementation Details

Network Details. In this paper, we utilize the open-sourced LLaMA2 (7B) model
touvron2023llama2 as our Large Language Model (LLM). We employ the text
encoder from CLIP radford2021learning to ensure that the extracted text
features closely align with the corresponding visual features in the embedding
space. ViT-g/14 from EVA-CLIP fang2023eva is used as the vision encoder. The
global text token is injected into the vision encoder to conduct prompt-aware
global attention. The use AdamW loshchilov2017decoupled as the optimizer, with
$\beta_{1}=0.9$, $\beta_{2}=0.999$, and a weight decay rate of $0.05$. The
learning rate is linearly warmed up from $10^{-6}$ to $8\times 10^{-5}$ over
the first $1,000$ steps to accelerate model convergence. This is followed by a
cosine decay of the learning rate to a minimum of $10^{-5}$. The training is
set to a maximum of $50$ epochs, with $1,000$ iterations per epoch. The model
is trained with a batch size of 4, across a period of 3 days on a single
NVIDIA RTX A6000 GPU.

Training Details and Datasets. Following touvron2023llama2 ; chen2023minigpt ,
the conversation template is as follows,

[INST] <Img> <Image Feature> </Img> [Task Identifier] <Prompt Feature> [/INST]
---

where [INST] and [/INST] represent the user role and chat assistant,
respectively. During training, the <Image Feature> is replaced with the visual
embeddings, and the <Prompt Feature> is substituted with textual prompt
embeddings. The [Task Identifier] is replaced based on specific circumstances
(e.g., [vqa] and [caption]), making our model more adept at understanding
multiple tasks.

We initialize our model with the pre-trained parameters of MiniGPT-V2
chen2023minigpt . MiniGPT-V2 has demonstrated impressive performance across
multiple tasks, having undergone three stages of training on various fine-
grained datasets schuhmann2021laion ; sharma2018conceptual2 ;
ordonez2011im2text ; lin2014microsoft ; sidorov2020textcaps ;
kazemzadeh2014referitgame ; yu2016modeling ; mao2016generation ; hudson2019gqa
; goyal2017making ; mishra2019ocr ; marino2019ok ; schwenk2022okvqa ;
plummer2015flickr30k . We fine-tune the model on downstream tasks at a low
cost to validate the effectiveness of our method. Our model is trained on the
COCO-QA dataset ren2015exploring , which includes a mix of question-answer
pairs covering object classification, color recognition, counting, and
positional reasoning. The training image-text pairs account for around $67\%$,
and the remaining pairs are used for the zero-shot image-to-text generation
task. Throughout the entire training process, the text encoder, global prompt-
aware vision encoder, and LLM remain frozen. Only the local prompt-aware
vision adapter and additional projection layers are fine-tuned. Evaluation is
primarily conducted on the COCO-QA ren2015exploring test dataset and the MME
fu2023mme benchmark. The latter includes $10$ perceptual (i.e., existence,
count, position, color, poster, celebrity, scene, landmark, artwork, and OCR)
and $4$ cognitive (i.e., commonsense reasoning, numerical calculation, text
translation, and code reasoning) tasks.

Figure 4: Visualization of prompt-aware global and local attention. Global
attention spans the entire prompt content, while local attention concentrates
predominantly on the specific object in question.

###  4.2 Evaluation

Attention Visualization. To demonstrate the effectiveness of the proposed
method, we visualize global and local attention in Fig. 4. The global
attention map effectively targets entire regions of the image that
semantically align with the entire prompt sentence. The local attention map,
associated with the question, highlights its ability to explore fine-grained
semantics when extracting visual clues.

Table 1: Quantitative results of $10$ perception tasks on the MME fu2023mme
benchmark. Evaluation metric is scores for correct answers, with higher scores
indicating better performance. $\ast$: frozen, $\checkmark$: fine-tuned. The
best and second-best scores are highlighted in bold and underlined,
respectively.

Method | Existence | Count | Position | Color | Poster | Celebrity | Scene | Landmark | Artwork | OCR | Total
---|---|---|---|---|---|---|---|---|---|---|---
MiniGPT-4 zhu2023minigpt | 68.33 | 55.00 | 43.33 | 75.00 | 41.84 | 54.41 | 71.75 | 54.00 | 60.50 | 57.50 | 581.66
PandaGPT su2023pandagpt | 70.00 | 50.00 | 50.00 | 50.00 | 76.53 | 57.06 | 118.00 | 69.75 | 51.25 | 50.00 | 642.59
ImageBind-LLM zhang2023llama | 128.33 | 60.00 | 46.67 | 73.33 | 64.97 | 76.47 | 113.25 | 62.00 | 70.75 | 80.00 | 775.77
VPGTrans zhang2024vpgtrans | 70.00 | 85.00 | 63.33 | 73.33 | 84.01 | 53.53 | 141.75 | 64.75 | 77.25 | 77.50 | 790.45
LaVIN luo2024cheap | 185.00 | 88.33 | 63.33 | 75.00 | 79.59 | 47.35 | 136.75 | 93.50 | 87.25 | 107.50 | 963.60
mPLUG-Owl ye2023mplug | 120.00 | 50.00 | 50.00 | 55.00 | 136.05 | 100.29 | 135.50 | 159.25 | 96.25 | 65.00 | 967.34
LRV-Instruction liu2023mitigating | 165.00 | 111.67 | 86.67 | 165.00 | 139.04 | 112.65 | 147.98 | 160.53 | 101.25 | 110.00 | 1299.79
BLIP-2 li2023blip | 160.00 | 135.00 | 77.33 | 148.33 | 141.84 | 105.59 | 145.25 | 138.00 | 136.50 | 110.00 | 1293.84
Flamingo alayrac2022flamingo | 70.00 | 60.00 | 46.67 | 73.33 | 41.84 | 57.06 | 68.00 | 62.00 | 60.50 | 65.00 | 604.40
Multimodal-GPT gong2023multimodal | 61.67 | 55.00 | 58.33 | 68.33 | 57.82 | 73.82 | 68.00 | 69.75 | 59.50 | 82.50 | 654.72
InstructBLIP dai2024instructblip | 185.00 | 143.33 | 66.67 | 153.33 | 123.81 | 101.18 | 153.00 | 79.75 | 134.25 | 72.50 | 1212.82
Linear$\checkmark$ | 88.33 | 35.00 | 20.00 | 43.33 | 24.82 | 93.82 | 74.75 | 108.75 | 41.50 | 27.5 | 557.82
Q-Former$\ast$ \+ Linear$\checkmark$ | 83.18 | 49.45 | 50.66 | 68.95 | 55.05 | 55.68 | 49.93 | 53.02 | 30.29 | 49.54 | 545.78
Q-Former$\checkmark$ \+ Linear$\checkmark$ | 65.00 | 33.22 | 51.66 | 54.21 | 42.70 | 52.60 | 73.75 | 73.04 | 52.13 | 72.50 | 570.84
Cross-Attention$\checkmark$ | 91.66 | 63.33 | 39.99 | 21.66 | 57.14 | 38.23 | 110.25 | 99.25 | 76.75 | 70.00 | 668.29
Ours | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02

Quantitative Results. For quantitative evaluation, we exclusively consider
questions with precise and concise answers, due to the statistical challenges
posed by open-ended responses from MLLMs fu2023mme ; openai2022chatgpt ;
liu2023visual . First, we compare our model with MLLMs equipped with prompt-
unware adapters, such as MiniGPT-4 zhu2023minigpt , mPLUG-Owl ye2023mplug ,
BLIP-2 li2023blip , and MLLMs that consider prompts during visual encoding,
like Multimodal-GPT gong2023multimodal , InstructBLIP dai2024instructblip .
The above comparison models are zero-shot evaluation results. Second, we
compare the proposed prompt-aware adapters with other popular adapters. To
ensure fairness, we strive to exclude the influence of training data and the
number of parameters on MLLM performance. Hence, we use the same dataset
ren2015exploring to train MLLMs with MiniGPT-4/v2 zhu2023minigpt ;
chen2023minigpt as the uniform backbone, changing only the adapters, including
linear projection, Q-Former family, cross-attention, and our prompt-aware
adapter. Results are from the MME
leaderboards111https://github.com/BradyFU/Awesome-Multimodal-Large-Language-
Models/tree/Evaluation

Table 2: Quantitative results of object classification, counting, color
recognition, and position reasoning on the COCO-QA ren2015exploring test
dataset. Evaluation metric is accuracy (%). $\ast$: frozen parameters,
$\checkmark$: fine-tuned parameters. The best results are highlighted in bold.

Method | Object | Count | Color | Position | Total
---|---|---|---|---|---
MiniGPT-4 zhu2023minigpt | 73.69 | 36.01 | 63.32 | 57.23 | 68.42
Linear$\checkmark$ | 74.86 | 62.15 | 69.87 | 63.97 | 72.44
Q-Former$\ast$ \+ Linear${\checkmark}$ | 56.50 | 43.04 | 29.75 | 35.23 | 49.73
Q-Former${\checkmark}$ \+ Linear${\checkmark}$ | 28.46 | 38.83 | 32.09 | 26.43 | 29.67
Cross-Attention${\checkmark}$ | 76.47 | 71.79 | 69.95 | 65.98 | 74.39
Ours | 81.12 | 76.19 | 80.17 | 70.70 | 79.95

Tab. 2 shows the quantitative results on the COCO-QA ren2015exploring text
dataset. The proposed method is capable of selectively focusing on objects
mentioned in the question during visual feature extraction, leading to a
noticeable improvement (approximately 5.78%) in quantity perception compared
to cross-attention adapters. In the object classification, color recognition
and positional reasoning tasks, our prompt-aware adapters surpass cross-
attention adapters by more than 5.73%, 12.75%, and 6.95%, respectively. As
depicted in Tab.s 1 and 3, compared to prompt-unaware adapters, our method
excels in both perception tasks ($1375.02$ vs $1299.79$) and cognition tasks
($289.28$ vs $210.31$) on the MME fu2023mme benchmark.

Qualitative Results. We qualitatively compare our model with several popular
MLLMs on perception and cognition tasks with more diverse visual inputs and
prompts. Among the compared methods, LLaVA liu2023visual , MiniGPT-4
zhu2023minigpt , BLIP-2 li2023blip adopt prompt-unaware adapters. While
Flamingo alayrac2022flamingo and InstructBLIP dai2024instructblip inherently
extract prompt-related visual signals through cross-attention. For visual
results, refer to Fig.s 6$\sim$5 and Appendix. Thanks to the prompt-aware
global and local attention, our model shows a clear improvement in following
prompts to focus on specific visual clues.

Figure 5: Qualitative results of the proposed method on diverse perception and
cognition tasks.

###  4.3 Ablation Study

Table 3: Quantitative results of four cognition tasks on the MME fu2023mme
benchmark. Evaluation metric is scores for correct answers, with higher scores
indicating better performance. $\ast$: frozen, $\checkmark$: fine-tuned.

Method | Common | Calculate | Translate | Code | Total
---|---|---|---|---|---
MiniGPT-4 zhu2023minigpt | 59.29 | 45.00 | 0.00 | 40.00 | 144.29
Linear$\checkmark$ | 58.57 | 42.50 | 5.00 | 47.50 | 153.57
Q-Former$\ast$ \+ Linear$\checkmark$ | 37.24 | 78.07 | 50.00 | 45.00 | 210.31
Q-Former$\checkmark$ \+ Linear$\checkmark$ | 38.45 | 0.00 | 47.50 | 5.00 | 90.95
Cross-Attention$\checkmark$ | 40.00 | 52.50 | 15.00 | 57.50 | 165.00
Ours | 99.28 | 50.00 | 87.50 | 52.50 | 289.28

Ablation Details. In this section, we analyze the necessity of global and
local attention. During validation for the former, the global text token is
not added to the visual encoder for guidance, while for the latter, a simple
linear projection layer is employed to replace the designed local attention
component. We also test the baseline that does not incorporate any prompt-
related attention mechanism during visual feature extraction. To ensure
fairness, the other experimental settings are identical.

Prompt-Aware Global Attention. Results (see Tab.s 4$\sim$6) suggest that the
prompt-unaware method struggles to perceive visual inputs and make reasonable
inferences, as it does not focus on visual signals emphasized by prompts. The
global attention component directs attention allocation with complete prompt
embeddings in the initial phase, thus extracting semantically aligned visual
embeddings. Without prompt-aware global attention, the answer accuracy of MLLM
drops obviously.

Prompt-Aware Local Attention. We evaluate the local prompt-aware adapter
against linear projection without local attention. According to the results in
Tab.s 4$\sim$6, the local prompt-aware attention component effectively
contributes to the perceptual and cognitive capabilities of our MLLM.
Especially in counting and position reasoning tasks, the correct scores are
enhanced by $16.05\%$ and $31.66\%$ (shown in Tab. 4), respectively.
Responding to fine-grained prompt guidance during the visual translation
process relieves the comprehension load of LLMs.

Figure 6: Comparison of qualitative results between our method and other
popular MLLMs. Table 4: Ablation study of prompt-aware global and local
attention on MME fu2023mme perception tasks. The best and second-best scores
are highlighted in bold and underlined, respectively.

Method | Existence | Count | Position | Color | Posters | Celebrity | Scene | Landmark | Artwork | OCR | Total
---|---|---|---|---|---|---|---|---|---|---|---
prompt-unware | 88.33 | 35.00 | 20.00 | 43.33 | 24.82 | 93.82 | 74.75 | 108.75 | 41.50 | 27.5 | 557.82
w/o global-atten | 175.00 | 126.66 | 91.66 | 90.00 | 152.04 | 129.11 | 147.25 | 117.50 | 122.75 | 132.50 | 1284.49
w/o local-atten | 175.00 | 113.33 | 68.33 | 99.99 | 152.38 | 131.17 | 155.25 | 119.75 | 122.75 | 80.00 | 1217.97
w/ global + local | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02

Table 5: Ablations on COCO-QA ren2015exploring test set.  Method | Object | Count | Color | Position | Total
---|---|---|---|---|---
prompt-unware | 74.86 | 62.15 | 69.87 | 63.97 | 72.44
w/o global-atten | 78.67 | 73.94 | 79.12 | 68.40 | 77.75
w/o local-atten | 77.08 | 74.59 | 76.06 | 67.83 | 76.15
w/ global + local | 81.12 | 76.19 | 80.17 | 70.70 | 79.95

Table 6: Ablations on MME fu2023mme cognition tasks.  Method | Comm. | Cal. | Trans. | Code | Total
---|---|---|---|---|---
prompt-unware | 58.57 | 42.50 | 5.00 | 47.50 | 153.57
w/o global-atten | 96.42 | 50.00 | 55.00 | 55.00 | 256.42
w/o local-atten | 99.28 | 47.50 | 50.00 | 52.50 | 249.28
w/ global + local | 99.28 | 50.00 | 87.50 | 52.50 | 289.28

##  5 Conclusions

In this paper, we introduce novel prompt-aware adapters designed to adaptively
embed visual inputs based on the given prompt. Our method aims to extract the
most informative visual clues relevant to the prompt, thereby enhancing the
visual understanding capabilities of LLMs. We first present a global attention
mechanism that utilizes a global text token for coarse-grained visual
guidance. Then, we introduce a local attention mechanism that leverages local
textual features to extract highly relevant visual clues with fine
granularity. The resulting prompt-aware global and local visual tokens
significantly reduce the visual perception burden on LLMs. Experimental
results show that our method achieves competitive accuracy on multiple visual
perception and cognitive tasks. We discuss the limitations and broader impacts
of our method in Appendix.

## References

  * [1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [2] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models.  CoRR, abs/2210.11416:1–12, 2022.
  * [3] OpenAI.  Introducing chatgpt, 2023.  https://openai.com/blog/chatgpt.
  * [4] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, abs/2302.13971:1–12, 2023.
  * [5] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
  * [6] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378, 2023.
  * [7] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.  Mm-react: Prompting chatgpt for multimodal reasoning and action.  arXiv preprint arXiv:2303.11381, 2023.
  * [8] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao.  Videochat: Chat-centric video understanding.  arXiv preprint arXiv:2305.06355, 2023.
  * [9] Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al.  Dense and aligned captions (dac) promote compositional reasoning in vl models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [10] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.  Kosmos-2: Grounding multimodal large language models to the world.  arXiv preprint arXiv:2306.14824, 2023.
  * [11] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [12] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny.  Minigpt-v2: large language model as a unified interface for vision-language multi-task learning.  arXiv preprint arXiv:2310.09478, 2023.
  * [13] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.  Shikra: Unleashing multimodal llm’s referential dialogue magic.  arXiv preprint arXiv:2306.15195, 2023.
  * [14] Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong.  What matters in training a gpt4-style language model with multimodal inputs?  arXiv preprint arXiv:2307.02469, 2023.
  * [15] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  arXiv preprint arXiv:2305.16355, 2023.
  * [16] Qiang Zhou, Zhibin Wang, Wei Chu, Yinghui Xu, Hao Li, and Yuan Qi.  Infmllm: A unified framework for visual-language tasks.  arXiv preprint arXiv:2311.06791, 2023.
  * [17] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh.  Honeybee: Locality-enhanced projector for multimodal llm.  arXiv preprint arXiv:2312.06742, 2023.
  * [18] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al.  Visionllm: Large language model is also an open-ended decoder for vision-centric tasks.  Advances in Neural Information Processing Systems, 36:1–13, 2024.
  * [19] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  * [20] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [21] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [22] Mengye Ren, Ryan Kiros, and Richard Zemel.  Exploring models and data for image question answering.  Advances in neural information processing systems, 28, 2015.
  * [23] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al.  Mme: A comprehensive evaluation benchmark for multimodal large language models.  arXiv preprint arXiv:2306.13394, abs/2306.13394:1–12, 2023.
  * [24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick.  Segment anything.  In IEEE/CVF International Conference on Computer Vision, ICCV 2023, October 1-6, 2023, pages 3992–4003, Paris, France, 2023. IEEE.
  * [25] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [26] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [29] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  Journal of Machine Learning Research, 24(240):1–113, 2023.
  * [30] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  arXiv preprint arXiv:2201.08239, 2022.
  * [31] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, abs/2205.01068:1–12, 2022.
  * [32] Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang.  Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified as a video agent).  In ICML, 2024.
  * [33] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al.  Minicpm: Unveiling the potential of small language models with scalable training strategies.  arXiv preprint arXiv:2404.06395, 2024.
  * [34] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.  Sharegpt4v: Improving large multi-modal models with better captions.  arXiv preprint arXiv:2311.12793, 2023.
  * [35] Kecheng Zheng, Yifei Zhang, Wei Wu, Fan Lu, Shuailei Ma, Xin Jin, Wei Chen, and Yujun Shen.  Dreamlip: Language-image pre-training with long captions.  arXiv preprint arXiv:2403.17007, 2024.
  * [36] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia.  Mini-gemini: Mining the potential of multi-modality vision language models.  arXiv preprint arXiv:2403.18814, 2024.
  * [37] Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan Kankanhalli.  Improve context understanding in multimodal large language models via multimodal composition learning.  In ICML, 2024.
  * [38] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  2023\.
  * [39] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.  mplug-owl: Modularization empowers large language models with multimodality.  arXiv preprint arXiv:2304.14178, 2023.
  * [40] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  * [41] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.
  * [42] Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, et al.  Reformulating vision-language foundation models and datasets towards universal multimodal assistants.  arXiv preprint arXiv:2310.00653, 2023.
  * [43] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.  Multimodal-gpt: A vision and language model for dialogue with humans.  arXiv preprint arXiv:2305.04790, 2023.
  * [44] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.  Otter: A multi-modal model with in-context instruction tuning, 2023.
  * [45] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang.  Fine-tuning multimodal llms to follow zero-shot demonstrative instructions.  In The Twelfth International Conference on Learning Representations, 2023.
  * [46] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [47] Siyu Lu, Mingzhe Liu, Lirong Yin, Zhengtong Yin, Xuan Liu, and Wenfeng Zheng.  The multi-modal fusion in visual question answering: a review of attention mechanisms.  PeerJ Computer Science, 9:e1400, 2023.
  * [48] Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang.  Less is more: Focus attention for efficient detr.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6674–6683, 2023.
  * [49] Man Yao, Guangshe Zhao, Hengyu Zhang, Yifan Hu, Lei Deng, Yonghong Tian, Bo Xu, and Guoqi Li.  Attention spiking neural networks.  IEEE transactions on pattern analysis and machine intelligence, 2023.
  * [50] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057. PMLR, 2015.
  * [51] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh.  Hierarchical question-image co-attention for visual question answering.  Advances in neural information processing systems, 29, 2016.
  * [52] Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy.  Hierarchical attention networks for document classification.  In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489, 2016.
  * [53] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio.  A structured self-attentive sentence embedding.  arXiv preprint arXiv:1703.03130, 2017.
  * [54] Tim Rocktäschel, Johannes Welbl, and Sebastian Riedel.  Frustratingly short attention spans in neural language modeling, 2017.
  * [55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [56] Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu.  Multi-modality cross attention network for image and sentence matching.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10941–10950, 2020.
  * [57] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [58] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358–19369, BC, Canada, 2023. IEEE.
  * [59] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [60] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [61] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [62] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2text: Describing images using 1 million captioned photographs.  Advances in neural information processing systems, 24, 2011.
  * [63] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [64] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 742–758. Springer, 2020.
  * [65] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  Referitgame: Referring to objects in photographs of natural scenes.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014.
  * [66] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg.  Modeling context in referring expressions.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. Springer, 2016.
  * [67] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20, 2016.
  * [68] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  * [69] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * [70] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In 2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019.
  * [71] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
  * [72] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In European Conference on Computer Vision, pages 146–162. Springer, 2022.
  * [73] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [74] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  arXiv preprint arXiv:2303.16199, abs/2303.16199:1–12, 2023.
  * [75] Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua.  Vpgtrans: Transfer visual prompt generator across llms.  Advances in Neural Information Processing Systems, 36, 2024.
  * [76] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji.  Cheap and quick: Efficient vision-language instruction tuning for large language models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [77] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.  Mitigating hallucination in large multi-modal models via robust instruction tuning.  In The Twelfth International Conference on Learning Representations, 2023.
  * [78] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin.  Movienet: A holistic dataset for movie understanding.  CoRR, abs/2007.10937, 2020.
  * [79] Hui Mao, Ming Cheung, and James She.  Deepart: Learning joint representations of visual arts.  In Qiong Liu, Rainer Lienhart, Haohong Wang, Sheng-Wei "Kuan-Ta" Chen, Susanne Boll, Yi-Ping Phoebe Chen, Gerald Friedland, Jia Li, and Shuicheng Yan, editors, Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017, pages 1183–1191. ACM, 2017.
  * [80] Hui Mao, James She, and Ming Cheung.  Visual arts search on mobile devices.  ACM Trans. Multim. Comput. Commun. Appl., 15(2s):60:1–60:23, 2019.
  * [81] Tobias Weyand, André Araújo, Bingyi Cao, and Jack Sim.  Google landmarks dataset v2 - A large-scale benchmark for instance-level recognition and retrieval.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2572–2581. Computer Vision Foundation / IEEE, 2020.
  * [82] Bolei Zhou, Àgata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva.  Learning deep features for scene recognition using places database.  In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 487–495, 2014.
  * [83] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang.  Curved scene text detection via transverse and longitudinal sequence connection.  Pattern Recognit., 90:337–345, 2019.

##  Appendix A Appendix Overview

In summary, the appendix is organized as follows:

Appendix B: Additional experiments and analysis.

Appendix C: More visualization results of prompt-aware global and local
attention.

Appendix D: More qualitative results and analysis.

Appendix E: More details of COCO-QA dataset and MME benchmark.

Appendix F: Prompts used for fine-tuning.

##  Appendix B Additional Experiments and Analysis

###  B.1 Comparison of Text Encoders

During the implementation of the global and local attention mechanisms, we
compare the quantitative results of text feature extraction by different text
encoders. Tab.s 7$\sim$9 show comparison results of adopting CLIP [27] and
Llama 2 [4] to extract prompt features for global and local attention. G-Num
refers to the number of text tokens used in global attention. $0$ means no
global attention is used in the model architecture, $1$ means only one global
text token is used, and $L$ means the total number of text tokens.

The experimental results show that using global text tokens performs better
than not using them, confirming the effectiveness of our prompt-aware global
attention. Besides, CLIP [27] generally performs better than Llama 2 [4] on
perceptual tasks, while Llama 2 [4] excels in cognitive tasks. This may be
because, in perceptual tasks, the text features extracted by CLIP can better
align with visual features. In cognitive tasks, Llama 2 [4], having been
trained on vast amounts of data, possesses superior reasoning abilities.

Table 7: Comparison results of using different text encoders (CLIP [27] and
Llama 2 [4]) to extract prompt features for global and local attention. G-Num
refers to the number of text tokens used in global attention, and $L$ is the
total amount of text tokens. The evaluation is built on the MME [23]
perception tasks, the evaluation metric is correct answer scores, with higher
scores being better. The best and second-best scores are highlighted in bold
and underlined, respectively.

Method | Perception Tasks |
---|---|---
G-Num | Global | Local | Existence | Count | Position | Color | Poster | Celebrity | Scene | Landmark | Artwork | OCR | Total
$0$ | - | - | 88.33 | 35.00 | 20.00 | 43.33 | 24.82 | 93.82 | 74.75 | 108.75 | 41.50 | 27.5 | 557.82
$0$ | - | CLIP | 175.00 | 126.66 | 91.66 | 90.00 | 152.04 | 129.11 | 147.25 | 117.50 | 122.75 | 132.50 | 1284.49
$1$ | CLIP | - | 175.00 | 113.33 | 68.33 | 99.99 | 152.38 | 131.17 | 155.25 | 119.75 | 122.75 | 80.00 | 1217.97
$1$ | CLIP | Llama 2 | 170.00 | 121.66 | 48.33 | 99.99 | 130.27 | 127.94 | 149.50 | 134.25 | 119.50 | 125.00 | 1226.46
$L$ | CLIP | Llama 2 | 170.00 | 106.66 | 48.33 | 70.00 | 128.23 | 118.23 | 154.25 | 139.50 | 119.25 | 102.50 | 1156.96
$L$ | CLIP | CLIP | 160.0 | 111.66 | 73.33 | 90.00 | 133.33 | 131.76 | 158.00 | 146.50 | 119.25 | 95.00 | 1218.84
$1$ | CLIP | CLIP | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02

Table 8: Comparison results of using different text encoders (CLIP [27] and
Llama 2 [4]) to extract prompt features for global and local attention. G-Num
refers to the number of text tokens used in global attention, and $L$ is the
total amount of text tokens. The evaluation is built on the MME [23] cognition
tasks, the evaluation metric is correct answer scores, with higher scores
being better. The best and second-best scores are highlighted in bold and
underlined, respectively.

Method | Cognition Tasks
---|---
G-Num | Global | Local | Common Sense | Calculation | Translation | Code | Total
$0$ | - | - | 58.57 | 42.50 | 5.00 | 47.50 | 153.57
$0$ | - | CLIP | 96.42 | 50.00 | 55.00 | 55.00 | 256.42
$1$ | CLIP | - | 99.28 | 47.50 | 50.00 | 52.50 | 249.28
$1$ | CLIP | Llama 2 | 83.57 | 57.50 | 57.50 | 47.50 | 246.07
$L$ | CLIP | Llama 2 | 92.14 | 50.00 | 65.00 | 47.50 | 254.64
$L$ | CLIP | CLIP | 96.42 | 45.00 | 57.50 | 47.50 | 246.42
$1$ | CLIP | CLIP | 99.28 | 50.00 | 87.50 | 52.50 | 289.28

Table 9: Comparison results of using different text encoders (CLIP [27] and
Llama 2 [4]) to extract prompt features for global and local attention. G-Num
refers to the number of text tokens used in global attention, and $L$ is the
total amount of text tokens. The evaluation is built on the COCO-QA [22]
cognition tasks, the evaluation metric is correct answer scores, with higher
scores being better. The best and second-best scores are highlighted in bold
and underlined, respectively.

Method | Perception Tasks
---|---
G-Num | Global | Local | Object | Count | Color | Position | Total
$0$ | - | - | 74.86 | 62.15 | 69.87 | 63.97 | 72.44
$0$ | - | CLIP | 78.67 | 73.94 | 79.12 | 68.40 | 77.75
$1$ | CLIP | - | 77.08 | 74.59 | 76.06 | 67.83 | 76.15
$1$ | CLIP | Llama 2 | 80.57 | 76.13 | 78.64 | 68.76 | 79.25
$L$ | CLIP | Llama 2 | 80.41 | 76.62 | 77.41 | 70.94 | 79.07
$L$ | CLIP | CLIP | 80.51 | 76.40 | 79.07 | 69.00 | 79.25
$1$ | CLIP | CLIP | 81.12 | 76.19 | 80.17 | 70.70 | 79.95

###  B.2 Residual Visual Details

We also try to fuse the visual features obtained through global attention and
the visual features obtained through local attention. $Ratio$ represents the
retention ratio of local visual features, and the retention ratio of global
visual features is (1 - $Ratio$). For example, when $Ratio$=0.2, it means that
the local features are multiplied by 0.2 and the global features are
multiplied by 0.8 and then added.

Tab.s 10$\sim$11 show the impact of using different $Ratio$ values on the
model’s perceptual and cognitive tasks. Experimental results show that the
weighted sum of global and local features helps to improve model performance
to a certain extent but not very much. In addition, when local features
account for greater weight, the model seems to perform better.

###  B.3 Exploration of Prompt-Aware Q-Former

Inspired by InstructBLIP [21], we further explore novel architectures for
prompt-aware Q-Former. In this section, we implement a Q-Former-based prompt-
aware adapter. Specifically, we try to add prompt cross-attention to different
blocks in Q-Former. Table 12 shows the quantitative evaluation results of
various model-specific implementations. Through extensive experiments, we
found that the Q-former is difficult to train, and it tends to lose previously
learned knowledge when fine-tuning on downstream tasks. Additionally, the
effects of injecting text features into different layers of the Q-Former vary
significantly.

Table 10: Comparison results of using different $Ratio$ to combine global and
local visual features. The value range of $Ratio$ is 0-1. The evaluation is
built on the MME [23] perception tasks, the evaluation metric is correct
answer scores, with higher scores being better. The best and second-best
scores are highlighted in bold and underlined, respectively.

$Ratio$ | Existence | Count | Position | Color | Posters | Celebrity | Scene | Landmark | Artwork | OCR | Total
---|---|---|---|---|---|---|---|---|---|---|---
0.2 | 85.00 | 65.00 | 58.33 | 51.66 | 71.42 | 40.29 | 52.25 | 99.75 | 56.25 | 70.00 | 649.97
0.4 | 175.00 | 108.33 | 68.33 | 73.33 | 130.27 | 136.17 | 156.75 | 148.25 | 136.00 | 95.00 | 1227.44
0.6 | 175.00 | 126.66 | 91.66 | 90.00 | 152.04 | 129.11 | 147.25 | 117.50 | 122.75 | 132.50 | 1284.49
0.8 | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02
1.0 | 190.0 | 128.33 | 91.66 | 150.00 | 152.72 | 144.41 | 155.50 | 141.00 | 139.75 | 80.00 | 1373.38

Table 11: Comparison results of using different $Ratio$ to combine global and
local visual features. The value range of $Ratio$ is 0-1. The evaluation is
built on the MME [23] cognition tasks. the evaluation metric is correct answer
scores, with higher scores being better. The best and second-best scores are
highlighted in bold and underlined, respectively.

$Ratio$ | Commonsense | Calculation | Translation | Code | Total
---|---|---|---|---|---
0.2 | 67.14 | 65.00 | 5.00 | 47.50 | 184.64
0.4 | 97.14 | 47.50 | 50.00 | 47.50 | 242.14
0.6 | 96.42 | 50.00 | 55.00 | 55.00 | 256.42
0.8 | 92.85 | 57.50 | 57.50 | 45.00 | 252.85
1.0 | 103.57 | 55.00 | 72.50 | 47.50 | 278.57

Table 12: Quantitative evaluation results of various model-specific
implementations. The “No." is the number of the experimental setup. The
“BlockNum" denotes the number of attention blocks in which the text cross-
attention layer is inserted. The “OC", “CON", “CR", and “PR" represent the
four tasks of object classification, counting, color recognition, and position
reasoning, respectively. The “Acc (%)“ denotes accuracy and “Total Acc (%)"
refers to the overall accuracy on four tasks.

No. | BlockNum | OC | CON | CR | PR | Total
---|---|---|---|---|---|---
Acc (%) | Acc (%) | Acc (%) | Acc (%) | Acc (%)
1 | 0 | 78.09 | 73.22 | 74.23 | 65.54 | 76.33
2 | 0+1 | 65.46 | 70.84 | 51.55 | 52.31 | 62.73
3 | 0+1+2 | 77.67 | 65.48 | 73.49 | 63.08 | 75.23
4 | 0+1+2+3 | 78.20 | 69.95 | 69.89 | 62.47 | 75.29
5 | 0+1+2+3+4 | 77.72 | 69.35 | 79.06 | 62.47 | 76.38
6 | 0+1+2+3+4+5 | 78.52 | 71.43 | 74.85 | 67.39 | 76.73
7 | 1 | 29.31 | 36.31 | 8.68 | 10.47 | 25.23
8 | 1+3+5+7+11 | 33.21 | 29.47 | 9.79 | 22.77 | 28.50
9 | 0+2+4+6+8+10 | 77.98 | 69.05 | 72.37 | 64.31 | 75.59
10 | 11 | 77.02 | 68.46 | 60.97 | 62.77 | 72.93
11 | 10+11 | 75.15 | 71.73 | 72.37 | 64.62 | 73.79
12 | 9+10+11 | 74.89 | 69.95 | 72.87 | 66.47 | 73.69
13 | 8+9+10+11 | 75.09 | 69.95 | 73.49 | 65.85 | 73.89
14 | 7+8+9+10+11 | 78.65 | 72.03 | 78.31 | 67.69 | 77.26
15 | 6+7+8+9+10+11 | 77.72 | 68.76 | 75.97 | 64.92 | 76.1
16 | 0+1+2+3+4+5+6+7+8+9+10+11 | 76.70 | 69.05 | 74.73 | 64.62 | 75.09

##  Appendix C More Visualization Results of Prompt-Aware Global and Local
Attention

In this section, we present more results of global and local attention
visualization. Fig. 7 aims to verify the role of the proposed attention
mechanism in object recognition tasks. Fig.s 8 to 10 re-validate the
superiority of the proposed prompt-aware adapter in counting. Fig. 11 and Fig.
12 illustrate the focal points of the attention mechanism in color recognition
and position reasoning tasks, respectively. A large number of visualization
results illustrate the effectiveness of our method and provide a degree of
interpretability.

Figure 7: Visualization of prompt-aware global and local attention on the
object perception task. Visualization results show that global attention spans
the entire prompt content, while local attention concentrates predominantly on
the specific object in question. Figure 8: Visualization of prompt-aware
global and local attention on counting task. Visualization results show that
global attention spans the entire prompt content, while local attention
concentrates predominantly on the specific object in question. Our method
performs well in accurately identifying and counting objects. Figure 9:
Visualization of prompt-aware global and local attention on counting task.
Visualization results show that global attention spans the entire prompt
content, while local attention concentrates predominantly on the specific
object in question. Our method performs well in accurately identifying and
counting objects. Figure 10: Visualization of prompt-aware global and local
attention on counting task. Visualization results show that global attention
spans the entire prompt content, while local attention concentrates
predominantly on the specific object in question. Our method performs well in
accurately identifying and counting objects. Figure 11: Visualization of
prompt-aware global and local attention on the color recognition task.
Visualization results show that global attention spans the entire prompt
content, while local attention concentrates predominantly on the specific
object in question. Our method achieves impressive performance in accurately
identifying and distinguishing colors. Figure 12: Visualization of prompt-
aware global and local attention on the position reasoning task. Visualization
results show that global attention spans the entire prompt content, while
local attention concentrates predominantly on the specific object in question.
Our method has the capability of location recognition.

##  Appendix D More Qualitative Results and Analysis

We also provide more qualitative results with a wider range of image and
prompt inputs (as shown in Fig.s13$\sim$26). Specifically, Fig. 13 focuses on
the color recognition task, Fig. 14 on the counting task, Fig. 15 on the
position reasoning task, Fig. 17 on the OCR task, Fig. 18 on the landmark
recognition task, Fig. 19 on the celebrity recognition task, and Fig. 20 on
the poster recognition task. Additionally, Fig. 21, Fig. 22and Fig. 23
respectively showcase the model’s capabilities in computation, coding, and
common sense reasoning. Finally, Fig.s 24$\sim$26 provide visualization
results for mixed perception tasks.

Figure 13: Zero-shot image-to-text generation using an MLLM based on the
prompt-aware adapter, where it shows favorable visual perception ability on
the color recognition task. Figure 14: Zero-shot image-to-text generation
using an MLLM based on the prompt-aware adapter, where it shows favorable
visual perception ability on the counting task. Figure 15: Zero-shot image-to-
text generation using an MLLM based on the prompt-aware adapter, where it
shows favorable visual perception ability on the position reasoning task.
Figure 16: Zero-shot image-to-text generation using an MLLM based on the
prompt-aware adapter, where it shows favorable visual perception ability on
the object detection task. Figure 17: Zero-shot image-to-text generation using
an MLLM based on the prompt-aware adapter, where it shows favorable visual
perception ability on the OCR task. Figure 18: Zero-shot image-to-text
generation using an MLLM based on the prompt-aware adapter, where it shows
favorable visual perception ability on the landmark recognition task. Figure
19: Zero-shot image-to-text generation using an MLLM based on the prompt-aware
adapter, where it shows favorable visual perception ability on the celebrity
recognition task. Figure 20: Zero-shot image-to-text generation using an MLLM
based on the poster recognition task. Figure 21: Zero-shot image-to-text
generation using an MLLM based on the prompt-aware adapter, where it shows
favorable visual cognition ability on the calculation task. Figure 22: Zero-
shot image-to-text generation using an MLLM based on the prompt-aware adapter,
where it shows favorable visual cognition ability on the code reasoning task.
Figure 23: Zero-shot image-to-text generation using an MLLM based on the
prompt-aware adapter, where it shows favorable visual cognition ability on the
commonsense reasoning task. Figure 24: Zero-shot image-to-text generation
using an MLLM based on the prompt-aware adapter, where it shows favorable
visual perception ability. Figure 25: Zero-shot image-to-text generation using
an MLLM based on the prompt-aware adapter, where it shows favorable visual
perception ability. Figure 26: Zero-shot image-to-text generation using an
MLLM based on the prompt-aware adapter, where it shows favorable visual
perception ability.

##  Appendix E More Details of COCO-QA Dataset and MME Benchmark

###  E.1 COCO-QA Dataset

Object Classification. The primary objective of the object classification task
is to accurately recognize objects in the given visual input and determine the
category to which they belong. The metric used to evaluate this task is
accuracy. For this task, we utilize $82,198$ question-answer pairs related to
objects from the COCO-QA dataset [22], with $78,666$ used for fine-tuning and
$3,532$ for testing. The questions typically involve the use of “what" to
inquire about the object type. In addition, a succinct single-word answer is
employed to precisely indicate the response.

Counting. The main goal of counting is to recognize and count the number of
questioned objects from the given visual input. The metric employed to assess
this task is accuracy. In the counting task, we use $19,568$ question-answer
pairs related to the number of objects from the COCO-QA dataset [22], with
$1,8761$ used for fine-tuning and $807$ for testing. When posing questions,
the inquiry typically involves the use of “how many" to inquire about the
object amount. Likewise, we only use a single word describing the quantity in
the response.

Color Recognition. The color recognition task is designed to detect objects in
questions and perceive color information from the input visual signal. The
metric employed to assess this task is also accuracy. In this task, we utilize
$8,640$ question-answer pairs related to objects from the COCO-QA dataset
[22], with $8,304$ used for fine-tuning and $336$ for testing. Questions
related to color are relatively straightforward, usually beginning with “What
is the color of". To facilitate quantitative evaluation, responses responses
are kept as brief as possible for color-related words.

Position Reasoning. In the position reasoning task, our primary goal is to
infer the location information of the queried object based on the input visual
content. Similarly, we adopt the accuracy as the evaluation metric. We adopt
$7,278$ question-answer pairs related to objects from the COCO-QA dataset
[22], with $6,953$ used for fine-tuning and $325$ for testing. In this task,
questions are formatted to start with “where" to ask for the position of an
object. Answers mostly be places, scenes, or large objects that contain
smaller objects.

Examples of questions and answers from four perceptual tasks are shown in
Table 13.

Table 13: Examples of questions and answers from four visual perception tasks.

Visual Perception Task | Question | Answer
---|---|---
Object Classification | what are sitting down on the ground | bears
what is parked on the side of the grass | motorcycle
what are two mhttps://www.overleaf.com/project/64eb0aa69f2b1a658d35a15ben playing with some elephants | ball
what is the color of the shirt | blue
what is laying on the bed next to some pillows | cat
Counting | how many men is sitting on the street in front of a building | two
how many red velvet cup cakes with no frosting on a flowered plate | three
how many pairs of shoes on a mat with a cat is sitting in the middle | eight
how many dessert treats in the white cardboard box | six
how many trays of itallian food are in large pans | four
Color Recognition | what is the color of the airplane | black
what is the color of the motorcycle | orange
what is the color of the brush | green
what is the color of the bird | white
what is the color of the flowers | red
Position Reasoning | where is the cat lounging | chair
where do the mother and son make sundaes | kitchen
where is the cheese pizza | box
where is the person sitting | bed
where do the large and over-sized stuffed teddy bear sitting | chair

###  E.2 MME Benckmark

MME [23] benchmark evaluates both the perception and cognition abilities of
MLLMs. In addition to OCR, perception encompasses the recognition of both
coarse-grained and fine-grained objects. Coarse-grained recognition includes
the identification of objects’ existence, count, position, and color. Fine-
grained recognition involves identifying movie posters, celebrities, scenes,
landmarks, and artworks. Cognition tasks include commonsense reasoning,
numerical calculation, text translation, and code reasoning. There are a total
of 14 sub-tasks, detailed as follows.

Coarse-Grained Recognition. The contents of coarse-grained recognition include
identifying the existence of common objects and determining their count,
color, and position. Images are sourced from the COCO dataset [63], but all
instruction-answer pairs are manually constructed rather than directly using
available annotations. Even if MLLMs have previously encountered these COCO
images, the manually prepared pairs are not included in their training sets.
This setup requires MLLMs to comprehend the instructions and infer the correct
answers. For each perception subtask of existence, count, color, and position,
30 images are used, with a total of 60 instruction-answer pairs prepared.

Fine-Grained Recognition. Fine-grained recognition evalutates the knowledge
resources of MLLMs, encompassing subtasks such as recognizing movie posters,
celebrities, scenes, landmarks, and artworks. These subtasks include 147, 170,
200, 200, and 200 images respectively. For the celebrity subtask, a red box is
placed around a person with a clearly visible face in the image, with the
instruction being, “Is the actor inside the red box named [celebrity name]?
Please answer yes or no.” Similar to the coarse-grained recognition, images
for these subtasks are sourced from publicly available datasets [78, 79, 80,
81, 82], and all instructions are manually crafted.

Optical Character Recognition. Optical Character Recognition (OCR) is a
foundational capability of MLLMs, supporting subsequent text-based tasks such
as text translation and text understanding. Images are sampled from [83], with
all instruction-answer pairs manually designed. Given that MLLMs are still in
their early stages, only relatively simple samples are chosen for this version
of MME. This section includes 20 images and 40 instruction-answer pairs.

All image-prompt-answer pairs are manually created. The few public datasets
used in this benchmark utilize only the images without relying on their
original annotations. Additionally, images are collected through real
photographs and image generation. The prompts in MME are designed to be
concise to minimize the influence of prompt engineering on the model’s output.
Due to the instruction design of “please answer yes or no” quantitative
statistics can be easily performed based on the “yes” or “no” responses from
MLLMs, ensuring accuracy and objectivity.

##  Appendix F Prompts used for Fine-tuning

The prompt templates employed in the end-to-end prompt fine-tuning process
across various tasks are illustrated in Table 14.

Table 14: Prompt templates used for converting datasets into prompt-tuning
data. In VQA tasks, two formats of prompts are available. Note that the
“{Question}" format is mandatory for the prompt-aware adapter, and needs to be
replaced with the specific question in Table 13.

Task | Prompt Template
---|---
Image Captioning |  [INST] <Img> <Image Feature> </Img> [Caption] <Describe this image in detail.> [/INST]
[INST] <Img> <Image Feature> </Img> [Caption] <Take a look at this image and
describe what you notice.> [/INST]
[INST] <Img> <Image Feature> </Img> [Caption] <Please provide a detailed
description of the picture.> [/INST]
[INST] <Img> <Image Feature> </Img> [Caption] <Could you describe the contents
of this image for me?> [/INST]
[INST] <Img> <Image Feature> </Img> [Caption] <Is this image common in real
world?> [/INST]
Visual Question Answering |  [INST] <Img> <Image Feature> </Img> [vqa] <Describe this image in detail. > [/INST]
[INST] <Img> <Image Feature> </Img> [vqa] <Take a look at this image and
describe what you notice. > [/INST]
[INST] <Img> <Image Feature> </Img> [vqa] <Could you describe the contents of
this image for me? > [/INST]
[INST] <Img> <Image Feature> </Img> [vqa] <Is this image common in real world?
> [/INST]
[INST] <Img> <Image Feature> </Img> [vqa] <Question > [/INST]