# MaXM: Towards Multilingual Visual Question Answering

Soravit Changpinyo  Linting Xue  Idan Szpektor  Ashish V. Thapliyal  Julien
Amelot  Michal Yarom  Xi Chen  Radu Soricut

###### Abstract

Visual Question Answering (VQA) has been primarily studied through the lens of
the English language. Yet, tackling VQA in other languages in the same manner
would require a considerable amount of resources. In this paper, we propose
scalable solutions to multilingual visual question answering (mVQA), on both
data and modeling fronts. We first propose a translation-based framework to
mVQA data generation that requires much less human annotation efforts than the
conventional approach of directly collection questions and answers. Then, we
apply our framework to the multilingual captions in the Crossmodal-3600
dataset and develop an efficient annotation protocol to create
$\mathrm{MaXM}$, a test-only VQA benchmark in 7 diverse languages. Finally, we
propose an approach to unified, extensible, open-ended, and end-to-end mVQA
modeling and demonstrate strong performance in 13 languages.

Multilinguality, Visual Question Answering

Figure 1: Multilingual VQA Data in 7 languages. Our data is automatically
generated from multilingual captions and then verified and adjusted by humans.
One example per language, from top to bottom: English (en), French (fr), Hindi
(hi), Hebrew (iw), Romanian (ro), Thai (th), and Chinese (zh).

##  1 Introduction

Visual Question Answering (VQA), the task of answering visual questions
grounded in images, is key to human-machine interaction in the visual world.
In particular, the natural language interface in VQA makes it easy for lay
people to express their needs and benefit from its applications, including
accessibility, education, and search. Yet, most attention to VQA so far has
been on English-based VQA. The promise of VQA, in other words, only applies to
a privileged subset of human populations.

Arguably, the English language has dominated the field mainly because of the
availability of English VQA benchmarks. These benchmarks are diverse,
permitting studies of different aspects of VQA, from general VQA (Zhu et al.,
2016; Kafle & Kanan, 2017; Krishna et al., 2017; Antol et al., 2015; Goyal et
al., 2017; Changpinyo et al., 2022), robust VQA (Agrawal et al., 2018),
compositional visual reasoning-based VQA (Hudson & Manning, 2019), useful VQA
for the blind and the visually-impaired (Gurari et al., 2018), VQA for scene-
text understanding (Singh et al., 2019; Biten et al., 2019), to VQA that
requires external, commonsense, or world knowledge (Marino et al., 2019;
Zellers et al., 2019; Schwenk et al., 2022). While useful, these benchmarks
require considerable amount of resources to create. For example, the most
prevalent approach to VQA benchmark creation is to laboriously collect and
verify the questions and the answers for each image by leveraging human
annotators.

To extend VQA to all languages in the world, we must make data creation more
automatic than the status-quo (Sect. 2.2). In this paper, we take a step
toward addressing this problem. Building on recent work on automatic data
creation for English VQA from captions (Changpinyo et al., 2022), we propose a
translation-based framework for multilingual visual question answering (mVQA)
data creation. Our framework automates much of the task of generating
questions and answers, thus providing a scalable path to mVQA.

As a concrete implementation of this idea, we apply our framework to the
generation of question-answer pairs from the multilingual captions of the
recently-proposed Crossmodal-3600 dataset ($\mathrm{XM3600}$) (Thapliyal et
al., 2022). Combined this with an efficient human annotation protocol, we
arrive at $\mathrm{MAVERICS}$-$\mathrm{XM3600}$ ($\mathrm{MaXM}$), a test-only
benchmark for multilingual visual question answering in 7 languages. See Fig.
1 for examples.

Finally, we use this benchmark to drive progress in mVQA modeling. We leverage
advances in image modeling and multilingual text-to-text modeling: ViT
(Dosovitskiy et al., 2021) and mT5 (Xue et al., 2021) and propose unified,
extensible, open-ended VQA models. We also showcase the advantage of our
models in the cross-lingual VQA setting (Pfeiffer et al., 2022).

In summary, our main contributions are (i) We propose a scalable translation-
based framework for multilingual VQA data generation based on captions (Sect.
3); (ii) We develop an efficient annotation protocol and create multilingual
test-only benchmarks called $\mathrm{MAVERICS}$-$\mathrm{XM3600}$
($\mathrm{MaXM}$) in 7 diverse languages: English (en), French (fr), Hindi
(hi), Hebrew (iw), Romanian (ro), Thai (th), and Chinese (zh) (Sect. 4); (iii)
We propose an approach to unified, extensible, open-ended, and end-to-end mVQA
modeling (Sect. 5) and demonstrate strong performance on both existing and our
proposed benchmarks (Sect. 6).

##  2 Related Work

###  2.1 VQA and Multilingual Multimodal Benchmarks

English has been the primary language in which vision-and-language researchers
study the VQA task, driven by the availability of data and benchmarks (Zhu et
al., 2016; Kafle & Kanan, 2017; Krishna et al., 2017; Antol et al., 2015;
Goyal et al., 2017; Agrawal et al., 2018; Gurari et al., 2018; Hudson &
Manning, 2019; Marino et al., 2019; Singh et al., 2019; Biten et al., 2019;
Sheng et al., 2021; Li et al., 2021; Changpinyo et al., 2022). The only
exception is xGQA (Pfeiffer et al., 2022), an extension of the English GQA
dataset (Hudson & Manning, 2019). xGQA consists of human translations of the
12,578 English questions from 398 images in the balanced testdev split of GQA
in 8 typologically diverse languages: English (en), German (de), Portuguese
(pt), Russian (ru), Indonesian (id), Bengali (bn), Korean (ko), and Chinese
(zh). Besides the differences in the languages considered, our proposed
approach to multilingual VQA data creation and benchmark complement xGQA, as
we will discuss in Sect. 4.4.

Beyond VQA, training and evaluation data for multilingual multimodal models
are limited. For a review of previous work, we refer the reader to the Image-
Grounded Language Understanding Evaluation (IGLUE) benchmark (Bugliarello et
al., 2022), where xGQA is a part of. In general, early attempts often focus on
Chinese (Li et al., 2019; Wang et al., 2019), Japanese (Yoshikawa et al.,
2017; Aggarwal & Kale, 2020) and several Indo-European languages (e.g.,
German, French, and Czech) (Elliott et al., 2016, 2017; Barrault et al.,
2018). However, the trend moves toward a wider variety of both languages and
tasks. For instance, XTD (Aggarwal & Kale, 2020) for test-only image retrieval
also considers Russian, Korean, and Turkish; XVNLI (Bugliarello et al., 2022)
for cross-lingual visual natural language inference also considers Arabic;
MaRVL (Liu et al., 2021) for mutli-lingual visual reasoning also considers
Indonesian, Swahili, Tamil, and Turkish; Room-Across-Room (Ku et al., 2020)
for vision-and-Language navigation also considers Hindi and Telugu. Notably,
Wikipedia Image Text (WIT) (Srinivasan et al., 2021) provides a large-scale
image-text dataset in 108 languages, automatically collected form Wikipedia,
and Crossmodal-3600 ($\mathrm{XM3600}$) (Thapliyal et al., 2022) provides
human-curated test-only image captions in 36 languages. Our work builds on top
of $\mathrm{XM3600}$ and the 7 languages that we consider are typologically,
genealogically, and geographically diverse.

###  2.2 VQA Data Creation

Previous work on VQA data creation relies heavily on humans: a set of
annotators ask questions, the same or the other set answer (Zhu et al., 2016;
Krishna et al., 2017; Goyal et al., 2017; Gurari et al., 2018; Marino et al.,
2019). Some work makes this process entirely or partially automatic. CLEVR
(Johnson et al., 2017a) uses a template-based approach to create a diagnostic
VQA dataset but it is based on synthetic images for which ground-truth image
annotations are available. GQA (Hudson & Manning, 2019) follows a similar
approach but instead starts from Visual Genome scene graphs (Krishna et al.,
2017), which themselves require large annotation efforts.

The most relevant is the line of work that rewrites image captions or video
transcripts as question-answer pairs for VQA data creation (Ren et al., 2015;
Banerjee et al., 2021; Yang et al., 2021; Lee et al., 2021; Changpinyo et al.,
2022). For instance, COCOQA (Ren et al., 2015) uses a template-based approach
that can only generate questions with one-word answers. WeaQA (Banerjee et
al., 2021) improves upon this with question-answer driven semantic role
labeling, paraphrasing, and backtranslation. Recently, Changpinyo et al.
(2022) and Yang et al. (2021) leverage T5 (Raffel et al., 2020) fine-tuned on
textual question answering datasets with success, generating large-scale VQA
datasets for images and videos, respectively. Our approach to mVQA data
creation leverages $\mathrm{VQ}^{2}\\!\mathrm{A}$, the approach in (Changpinyo
et al., 2022), which we will review in more detail in Sect. 3.1. To the best
of our knowledge, none of the previous work on VQA data generation (besides
xGQA) considers languages beyond English.

Figure 2: Our approach to multilingual VQA data generation, which is easy to
scale, highly automatic and only requiring humans to modify “Almost Correct”
questions or correct/expand answers (left) or filter out “Incorrect”
questions(right). MT is short for automatic machine translation.

##  3 Multilingual VQA Data Creation

Like in many other machine learning tasks, the main bottleneck to mVQA is
obtaining high-quality labeled data. The most popular data collection
framework to English VQA is to ask a set of human annotators to come up with
visual questions, and another set of annotator to answer them (Sect. 2.2). To
scale VQA for all languages in the world, we believe that mVQA data creation
must step away from this resource-intensive paradigm, and only involve humans
as necessary. To this end, we resort to recent advances in automatic question
generation. In particular, we extend an automatic English VQA data creation
method called Visual Question Generation with Question Answering validation,
or $\mathrm{VQ}^{2}\\!\mathrm{A}$ (Changpinyo et al., 2022) and study how to
leverage it for the purpose of mVQA data creation.

###  3.1 Background: $\mathrm{VQ}^{2}\\!\mathrm{A}$

A cost-effective approach to VQA data generation is to leverage the dense
information in image-text data sources, often available at scale (Ordonez et
al., 2011; Chen et al., 2015; Sharma et al., 2018; Pont-Tuset et al., 2020;
Changpinyo et al., 2021; Desai et al., 2021; Schuhmann et al., 2021) and
beyond English (Srinivasan et al., 2021; Gu et al., 2022). In particular, one
can often rewrite a declarative image caption into multiple interrogative
question-answer pairs. One effective method is $\mathrm{VQ}^{2}\\!\mathrm{A}$
(Changpinyo et al., 2022). It consists of three main steps: (i) _Candidate
Answer Extraction_ extracts candidate answers based on syntactic and semantic
analysis of an input caption, (ii) _Question Generation_ generates candidate
questions based on the caption and its candidate answers, (iii) _Question
Answering and Answer Validation_ filters generated candidate questions that do
not pass the “round-trip” consistency check (Alberti et al., 2019; Honovich et
al., 2021), which involves answering the questions based on the caption and
compares those answers to the original ones (Wang et al., 2020). We refer the
reader to (Changpinyo et al., 2022) for details.

Note that each step in $\mathrm{VQ}^{2}\\!\mathrm{A}$ is optimized for
English; Step (i) uses English spaCy and both Step (ii) and Step (iii)
leverage high-capacity English-pre-trained T5 models (Raffel et al., 2020)
fine-tuned on English textual question answering datasets (Rajpurkar et al.,
2016, 2018; Kwiatkowski et al., 2019).

###  3.2 Translation-based-$\mathrm{VQ}^{2}\\!\mathrm{A}$ (TransVQ2A)

Inspired by $\mathrm{VQ}^{2}\\!\mathrm{A}$ (Changpinyo et al., 2022), our goal
is to generate multilingual VQA data at scale based on (multilingual) image
captions. Multilingualizing each step in $\mathrm{VQ}^{2}\\!\mathrm{A}$ can be
a non-trivial and resource-intensive task due to the heavy reliance of
$\mathrm{VQ}^{2}\\!\mathrm{A}$ on English tools, models, and data (Sect. 3.1).
For this reason, we propose to study a translation-based extension of
$\mathrm{VQ}^{2}\\!\mathrm{A}$ as an initial step toward this direction.

Given an input caption $c$, not necessarily in English, and a target language
$\langle\mathrm{lang}\rangle$, we want to generate question-answer pairs in
$\langle\mathrm{lang}\rangle$. We propose Translation-
based-$\mathrm{VQ}^{2}\\!\mathrm{A}$ ($\mathrm{TransVQ}^{2}\\!\mathrm{A}$), as
follows: Step 1: Caption Translation. If the caption $c$ is not in English, we
automatically translate the input caption to English $c_{e}$. Step 2: Apply
$\mathbf{\mathrm{VQ}^{2}\\!\mathrm{A}}$. We apply
$\mathrm{VQ}^{2}\\!\mathrm{A}$ to the (translated) English caption $c_{e}$ to
get a set of English question-answer pairs $\\{q_{e},a_{e}\\}$. Step 3:
Question-Answer Translation. We automatically translate each English question-
answer pair ($q_{e},a_{e}$) to $\langle\mathrm{lang}\rangle$ ($q,a$). Step 4:
Question-Answer Validation. We check111Excluding boolean questions. if the
translated answer $a$ is in the original caption $c$. If not, we filter the
question-answer pair out.

Note that we add the final validation step in favor of precision. One major
reason is that Step 3 is especially error-prone; translating from English
tends to be more problematic than to English and translating answers more
problematic than translating captions, since answers are often short and
questions may not provide enough context.

Another thing to note is that our pipeline can be relaxed so that the caption
$c$ does not have to be in the same target language as the output question-
answer pair, allowing for any caption to be used for a target language.
However, the final validation step would require appropriate adjustments, such
as applying an additional backtranslation step to the caption into the target
language, or an additional backtranslation step to the question-answer pair
into English and verify the answer with the English caption. We leave this
exploration for future work.

###  3.3 Direct Question Generation (DirectQG) for Answer Bias Mitigation

One drawback of $\mathrm{TransVQ}^{2}\\!\mathrm{A}$ is the low coverage of
particular types of answers, such as “no”. This is because the captions
generally do not indicate the absence of objects or properties (e.g., _“There
is no dog”_ , _“The dog is not white”_). To mitigate this, we train a
multilingual question generation model as a complementary VQA data generator.
The model takes in an answer and a caption in a target language as input and
generates candidate questions in the same language. Specifically, we used the
model to generate questions for “yes”, “no”, or “none” as answers in each
target language.

Concretely, we fine-tune mT5-XXL (Xue et al., 2021) on large-scale translated
COCO Captions (Chen et al., 2015) and its corresponding VQA data
$\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO (Changpinyo et al., 2022) (with images
discarded). For validation set, we use the subset of generated multilingual
VQA data in Sect. 3.2, with around 300 golden examples for each language. The
best checkpoints are selected based on ROUGE-L scores. Then we run the
inference on selected checkpoint to generate additional questions given
caption-answer pairs.

###  3.4 Discussion

Fig. 2 summarizes our proposed pipeline, using a Chinese caption in
Crossmodal-3600 (Thapliyal et al., 2022) as an example. We highlight that the
approach we have described so far is purely automatic, making our data
generation pipeline scalable. In the next section, we involve human annotators
to clean the generated data, mainly to perform a slight modification of the
Almost Correct question (the left example) or filter the Incorrect question
out (the right example). The human annotators will also be involved in
expanding answers for some question types.

One main advantage of $\mathrm{TransVQ}^{2}\\!\mathrm{A}$ is its applicability
to a huge set of languages for which automatic translation to and from English
is possible. Additionally, it requires less human annotation efforts than it
would need with the traditional pipeline (i.e., (Goyal et al., 2017)). On the
other hand, as mentioned above, $\mathrm{TransVQ}^{2}\\!\mathrm{A}$ provides
low coverage for some answer types, which we mitigate via a direct question
generation approach (Sect. 3.3). In addition, another possible limitation of
$\mathrm{TransVQ}^{2}\\!\mathrm{A}$ is the problematic artifacts of the so-
called Translationese texts (Lembersky et al., 2012; Volansky et al., 2013),
which we will mitigate directly via human validation and modification; see
concrete details in the next section.

Figure 3: The diversity of multilingual captions in $\mathrm{XM3600}$. We show the captions (their English translations) from 4 languages for the images of a snow cannon (left) and xiao long bao (right). Label | Question | Answer
---|---|---
Correct | Makes sense AND is relevant to the image. | Satisfies the question’s intent wrt the image.
Almost Correct | Correct but its surface form can be improved (syntactic errors or awkward/uncommon usages.)
Incorrect | NOT Correct.
Table 1: Description of “Correct”, “Almost Correct”, or “Incorrect” for questions and answers in our annotation protocol. Detailed protocol in the Appendix. | Number of Instances (% of Previous Stage)
---|---
| en | fr | hi | iw | ro | th | zh
Captions | 7200 | 8562 | 8503 | 7200 | 7123 | 7200 | 7174
English QAs | 373248 | 499900 | 520080 | 544268 | 516604 | 415180 | 524252
Validated | 264930 | 343621 | 346948 | 375629 | 346887 | 286024 | 362304
English QAs | (71.0%) | (68.7%) | (66.7%) | (69.0%) | (67.1%) | (68.9%) | (69.1%)
Validated | 264724 | 122644 | 153465 | 128613 | 121221 | 95531 | 182095
Multilingual QAs | (99.92%) | (33.85%) | (53.67%) | (37.08%) | (32.27%) | (27.53%) | (52.99%)
Table 2: Statistics of automatically-generated question-answer (QA) pairs based on Crossmodal-3600 captions. Validated English QA pairs are w.r.t the QG-QA consistency filter. Validated multilingual QA pairs are w.r.t the caption-answer consistency filter.  | en | fr | hi | iw | ro | th | zh
---|---|---|---|---|---|---|---
# of questions evaluated | 377 | 389 | 400 | 365 | 440 | 401 | 391
% Correct | 62.6% | 65.8% | 66.5% | 61.6% | 59.1% | 47.4% | 51.9%
% Almost Correct | 17.5% | 12.3% | 9.0% | 22.5% | 9.3% | 27.9% | 25.1%
% Incorrect | 19.9% | 21.9% | 24.50% | 15.9% | 31.6% | 24.7% | 23.02%
# of answers evaluated | 302 | 304 | 302 | 307 | 301 | 302 | 301
% Correct | 66.2% | 72.0% | 77.8% | 73.9% | 76.2% | 82.2% | 81.9%
% Almost Correct | 26.5% | 24.0% | 17.9% | 24.1% | 16.9% | 7.9% | 9.2%
% Incorrect | 7.3% | 3.9% | 4.3% | 2.0% | 7.0% | 9.9% | 8.9%
Table 3: Human evaluation of the generated questions and answers.  Question | Percentage
---|---
Prefix | en | fr | hi | iw | ro | th | zh
_“is”_ | 22.8 | 21.2 | 21.4 | 17.8 | 19.5 | 16.2 | 20.2
_“what is”_ | 15.8 | 11.3 | 16.0 | 15.2 | 13.5 | 11.6 | 13.7
_“how many”_ | 15.1 | 11.3 | 13.9 | 10.2 | 14.1 | 15.2 | 12.1
_“where”_ | 6.7 | 9.2 | 7.5 | 8.6 | 6.6 | 9.3 | 5.9
_“what kind”_ | 6.0 | 7.2 | 1.4 | 3.8 | 6.6 | 4.0 | 3.6
_“what are”_ | 3.4 | 1.4 | 1.0 | 2.2 | 2.4 | 2.3 | 2.3
_“who”_ | 3.4 | 3.1 | 9.2 | 2.5 | 1.5 | 2.6 | 2.3
_“are”_ | 3.4 | 0.7 | 4.1 | 3.8 | 3.6 | 3.3 | 2.0
_“what color”_ | 3.4 | 7.2 | 5.1 | 9.2 | 6.9 | 8.6 | 7.5
_“a”_ | 3.0 | 3.1 | 1.4 | 3.5 | 2.1 | 3.0 | 2.9
_“what type”_ | 2.7 | 2.4 | 0.3 | 0.3 | 2.1 | 1.0 | 4.6
_“what was”_ | 1.0 | 0.3 | 0.0 | 0.6 | 1.2 | 3.3 | 0.7
_“do”_ | 0.7 | 0.3 | 0.0 | 0.3 | 0.6 | 0.7 | 2.3
_“in”_ | 0.7 | 1.7 | 0.3 | 0.3 | 2.4 | 1.7 | 2.0
_“besides”_ | 0.3 | 1.0 | 4.4 | 3.2 | 1.2 | 0.3 | 2.3
_“does”_ | 0.3 | 2.7 | 1.7 | 2.5 | 1.5 | 3.3 | 0.7
other | 11.4 | 16.0 | 12.2 | 15.9 | 14.1 | 13.6 | 15.3
Table 4: The distribution of question types in $\mathrm{MaXM}$ across
languages. Approximated by their corresponding English question prefixes.

Figure 4: Word clouds of the answers for different question types in
$\mathrm{MaXM}$. Top cloud for _“What”_ questions (excluding _“What color”_).
Bottom clouds from left to right for _“What color”_ , Boolean
_“Is/Are/Was/Were/Do/Does/Did”_ , and _“How many”_ questions, respectively.

##  4 MaXM: Multilingual VQA Benchmark

In this section, we leverage our approach for creating a multilingual VQA
test-only benchmark, called MaXM. In what follows, we describe our data
sources, how to leverage our approach to generate candidate data, human
annotation protocol, and an analysis and a discussion of our benchmark.
Following the naming convention in (Changpinyo et al., 2022), we call our
benchmark MAVERICS-XM3600, or $\mathrm{MaXM}$ in short. We will release our
benchmark to foster research in this area.

###  4.1 Data Sources

Language Selection. We choose 7 languages in which we have access to native
annotators while ensuring that they are 1) typologically diverse, 2)
genealogically diverse, and 3) geographically diverse: English (en), French
(fr), Hindi (hi), Hebrew (iw), Romanian (ro), Thai (th), and Chinese (zh).

Image and Caption Selection. We choose a subset of the 3600 images of
Crossmodal-3600 ($\mathrm{XM3600}$) (Thapliyal et al., 2022), in which high-
quality multilingual image captions are available. For each language, a subset
of 100 validation and test images of Open Images (Krasin et al., 2017;
Kuznetsova et al., 2020) that were taken in the region(s) in which those
languages were spoken are selected. As a result, these images cover a wide
range of visual concepts in different cultural contexts.

Starting from the captions in a target language, like the ones in
$\mathrm{XM3600}$, makes the constructed VQA examples diverse and specific to
that language. For example, in Fig. 3, unlike French and Romanian language
speakers, Hebrew and Thai speakers are less likely to understand what a snow
cannon exactly is. On the other hand, Thai and Chinese language speakers are
more likely to understand what xiao long bao is, whereas in French or Hindi,
it would be referred to as dim-sum ravioli or Chinese dim sum. Another benefit
of $\mathrm{XM3600}$ is that the Open Images images are out-of-domain with
respect to most widely-used VQA benchmarks (Ren et al., 2015; Antol et al.,
2015; Zhu et al., 2016; Krishna et al., 2017; Goyal et al., 2017; Agrawal et
al., 2018; Hudson & Manning, 2019; Marino et al., 2019), which are often based
on MS-COCO images (Lin et al., 2014).

###  4.2 Large-Scale mVQA Data Creation

We apply our approach described in Sect. 3 to the $\mathrm{XM3600}$ captions
to generate a large number of question-answer pairs for each language.

TransVQ2A. Table 2 reports the number of question-answer pairs at different
stages in our pipeline. Overall, we are able to generate a large number of
question-answer pairs in all languages. Further, we find that, across
languages, approximately 30% of (translated) English question-answer pairs are
filtered out according to the $\mathrm{VQ}^{2}\\!\mathrm{A}$ pipeline. In
contrast, different percentages of translated answers across languages are
filtered out based on the caption-answer consistency filter. A main reason for
this is due to the limitation of question-answer translation. For instance,
68% of questions with “alb” (masculine “white” in Romanian) is filtered out
because it is not in the correct feminine form “alba” with respect to the
corresponding object in the question.

DirectQG. We augment our questions with additional candidate questions
generated by the Direct Question Generation model (Sect. 3.3), using the
$\mathrm{XM3600}$ captions paired with “yes”, “no”, or “none” in their
corresponding language as input. See Appendix for more details.

###  4.3 Human Annotation

We use native speakers of the selected 7 languages to annotate and create our
benchmark. We design an annotation protocol to balance efficiency and
accuracy. In particular, we keep human-in-the-loop brief and only when it
would be difficult for the machine to do the job (e.g., correcting translation
artifacts, expanding answers, identifying sensitive questions). In addition,
whenever the task of correction is not worth the effort (e.g., when the
question does not make sense), the respective example is discarded. We provide
more details below and in the Appendix.

Question and Answer Validation. We first use a 3-way rating system of
“Correct”, “Almost Correct”, “Incorrect” in Table 1 for both the questions and
the answers. More specifically, “Correct” questions are kept unchanged,
“Almost Correct” questions are rewritten, and “Incorrect” questions are
discarded. Given Correct and Almost Correct questions, the annotator rates the
answer and corrects it in the cases of both Almost Correct and Incorrect.
Table 3 reports the breakdown of Correct, Almost Correct, and Incorrect
questions and answers randomly-sampled from those generated by
$\mathrm{TransVQ}^{2}\\!\mathrm{A}$222For the questions and answers generated
by the DirectQG approach, we do not perform exhaustive verification; we ask
the annotators to focus on annotating “Correct” and “Almost Correct” questions
with “no” answers.. Across languages, we observe at least 75% Correct or
Almost Correct questions and, given these questions, at least 90% Correct or
Almost Correct answers. This highlights the effectiveness of our approach.

Answer Expansion and Standardization. We separate the questions into 4
categories: yes/no, number, color, and others. We ask the annotators to
perform standardization of yes/no, number, and color questions based on each
language’s guideline. For the rest of the questions, we ask another set of at
least 2 annotators per language to answer the questions with as many correct
(but not overly lengthy) answers as they can.

Additional Filtering. We perform additional round of verification and answer
standardization. Presented with expanded/standardized answers along with the
questions, we further filter out examples with “Ambiguous” and “Responsible-
AI-sensitive” questions and/or with inappropriate image content. We also mark
questions that are likely to lead to long answers that are difficult to
evaluate (e.g., “What is on the table?” when there are multiple items) as
“Collection” but do not filter them out.

###  4.4 Analysis and Discussion

Size and Question Type and Answer Distributions. $\mathrm{MaXM}$ v1 has a
total of 2,142 questions in 7 languages: English (en, 298), French (fr, 293),
Hindi (hi, 294), Hebrew (iw, 315), Romanian (ro, 333), Thai (th, 302), and
Chinese (zh, 307).

Table 4 shows a breakdown of question types from $\mathrm{MaXM}$. Since
question prefixes in some languages are not indicative of question types
(e.g., unlike English, Thai does not always begin the ”What” questions with
the Thai ”What”), we approximate the question types using the prefixes of the
English questions before translations. We observe diverse question types and a
high degree of linguistic variations.

Finally, Fig. 4 presents word clouds for answers for selected question types:
_”What”_ (excluding _”What color”_), _”What color”_ , Boolean
_”Is/Are/Was/Were/Do/Does/Did”_ , and _”How many”_. Again, we observe diverse
answers for each question type.

Comparison to xGQA. In terms of settings, one difference is the languages of
the answers; xGQA operates in the “cross-lingual” setting where the input
question can be non-English but the output answer is always English. While
this simplifies the evaluation process, we argue that the “multilingual”
setting with non-English answers considered in this paper is more practical.
Another difference is the definition of the zero-shot setting; xGQA refers to
unseen languages (not images) whereas our setting is more general, referring
to both unseen images and languages. Finally, the type of translated data and
how it is used for training are different; we only consider the zero-shot
setting and always use machine-translated questions for training, while xGQA
considers both the zero-shot and few-shot settings with human-translated
questions involved only in the few-shot case.

In terms of the datasets, xGQA inherits the characteristics of GQA, whose
questions are restricted in style (e.g., generated by a probabilistic
template-based question engine) and in the skills required (e.g., reasoning-
based with multi-step inference of object attributes and relationships). In
contrast, the questions in our dataset are more general. Across languages,
xGQA considers the same set of questions for all languages, whereas our
dataset considers different sets of questions guided by the captions in each
language.

Figure 5: Our unified and open-ended 2$\langle\mathrm{lang}\rangle$ VQA model
used in our experiments. We leverage ViT (Dosovitskiy et al., 2021) and mT5
(Xue et al., 2021) and train them together end-to-end.

##  5 Modeling

Much of the previous work on VQA is built for English. Further, VQA is often
formulated as _vocab-based VQA_ , a classification task into a pre-defined
space of top (English) answer vocabulary; see, e.g., (Antol et al., 2015;
Goyal et al., 2017). The main drawback of this approach is its inability to
deal with rare answers through language compositionality. Recent work
considers VQA as generation (Cho et al., 2021; Wang et al., 2022b; Alayrac et
al., 2022; Wang et al., 2022a), capable of _open-ended VQA_. We adopt this as
a scalable and flexible modeling approach to mVQA as the language coverage
increases. In particular, we propose a _single_ open-ended VQA model for
multiple languages. Our proposed formulation is more desirable than existing
ones since it takes advantage of both compositionality in individual languages
and the relationship among related languages.

To this end, we first describe an encoder-decoder architecture for VQA in the
open-ended generation setting. Then, we describe how we train this model for
multiple languages. This is summarized in Fig. 5.

###  5.1 Open-Ended VQA

Our starting architecture is mT5 (Xue et al., 2021), a multilingual variant of
T5 (Raffel et al., 2020). mT5 is an encoder-decoder transformer-based
architecture, pre-trained on a Common Crawl-based dataset covering 101
languages. This allows us to leverage multilingual language understanding (for
the questions) and generation (for the answers) from the get-go. To adapt mT5
to the VQA task, we prepend patch embeddings from the image to the question
tokens. In particular, we encode the image pixels using Vision Transformers
(ViT) (Dosovitskiy et al., 2021). We use ViT-L16 and mT5-Large in all of our
experiments. Both mT5 and ViT are trained together in an end-to-end fashion to
predict the target answer for each image-question pair, using the standard
cross-entropy loss.

###  5.2 Multi-Language Prompted Training

We resort to multi-task prompted/instruction training (Sanh et al., 2022; Wei
et al., 2022), where a task corresponds to VQA for a particular language. For
the input question $\langle$question$\rangle$ in language
$\langle$lang$\rangle$, we construct the prompt “Answer in
$\langle$lang$\rangle$: $\langle$question$\rangle$” and use it as the text
input to our model, similar to a modification to the input in Google’s
Multilingual Neural Machine Translation System (Johnson et al., 2017b). Such a
design for mult-task learning makes extending VQA to multiple languages
simple; as data for additional languages become available, one can simply add
them to the pool without the need for architecture changes.

##  6 Experiments

The goal of our experiments is to assess the quality of our proposed approach
to multilingual VQA modeling and establish strong baseline performance on our
proposed benchmark $\mathrm{MaXM}$.

|  | Training | Question & Answer Language
---|---|---|---
Metric | Model | Dataset | en | fr | hi | iw | ro | th | zh
Accuracy |  Single-Lang 2$\langle\mathrm{lang}\rangle$ | VQA2.0 | 37.6 | 33.8 | 53.7 | 35.6 | 36.0 | 50.0 | 29.0
2$\langle\mathrm{lang}\rangle$ | VQA2.0 | 36.6 | 36.2 | 55.1 | 40.6 | 42.3 | 50.0 | 30.3
2$\langle\mathrm{lang}\rangle$ |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO | 48.0 | 43.3 | 56.8 | 42.2 | 45.6 | 52.3 | 34.5
2$\langle\mathrm{lang}\rangle$ |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-CC3M | 38.3 | 34.1 | 45.6 | 34.9 | 36.9 | 45.0 | 29.0
CIDEr |  Single-Lang 2$\langle\mathrm{lang}\rangle$ | VQA2.0 | 90.2 | 92.4 | 59.2 | 72.4 | 76.3 | 89.1 | 64.0
2$\langle\mathrm{lang}\rangle$ | VQA2.0 | 91.5 | 102.0 | 62.0 | 78.6 | 89.6 | 86.7 | 68.4
2$\langle\mathrm{lang}\rangle$ |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO | 115.3 | 134.3 | 66.7 | 86.2 | 97.6 | 88.9 | 70.6
2$\langle\mathrm{lang}\rangle$ |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-CC3M | 103.3 | 107.5 | 53.1 | 75.2 | 86.7 | 85.1 | 66.6
ROUGE-L |  Single-Lang 2$\langle\mathrm{lang}\rangle$ | VQA2.0 | 46.3 | 42.7 | 56.9 | 38.4 | 40.4 | 57.0 | 34.9
2$\langle\mathrm{lang}\rangle$ | VQA2.0 | 45.0 | 47.9 | 57.9 | 42.8 | 49.4 | 57.7 | 37.7
2$\langle\mathrm{lang}\rangle$ |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO | 60.5 | 55.2 | 61.4 | 45.8 | 52.5 | 59.1 | 41.6
2$\langle\mathrm{lang}\rangle$ |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-CC3M | 49.8 | 43.5 | 49.0 | 36.8 | 44.6 | 51.5 | 36.4
Table 5: Main Results on $\mathrm{MaXM}$. Accuracy (%), CIDEr ($\times$ 100), and ROUGE-L ($\times$ 100) of our unified 2$\langle\mathrm{lang}\rangle$ models trained on different training datasets as well as our Single-Lang 2$\langle\mathrm{lang}\rangle$ baselines. | Training | Question Language
---|---|---
Model | Dataset | en | bn | de | id | ko | pt | ru | zh
M3P (Pfeiffer et al., 2022) | GQA | 58.4 | 17.6 | 24.8 | 18.7 | 19.7 | 26.7 | 24.3 | 19.7
mBERT${}^{\textrm{Ada}}$ (Pfeiffer et al., 2022) | GQA | 56.3 | 13.4 | 32.4 | 19.8 | 19.9 | 31.5 | 25.5 | 26.2
Single-Lang 2en | VQA2.0 | 43.1 | 37.9 | 39.6 | 40.4 | 38.9 | 40.3 | 39.3 | 39.7
2en | VQA2.0 | 41.5 | 38.6 | 40.5 | 39.5 | 38.7 | 39.8 | 39.5 | 39.5
2en |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO | 36.6 | 34.3 | 36.1 | 35.5 | 35.1 | 34.6 | 34.5 | 35.4
2en |  $\mathrm{VQ}^{2}\\!\mathrm{A}$-CC3M | 34.0 | 30.9 | 33.3 | 33.2 | 32.5 | 32.1 | 32.0 | 32.7
Table 6: Zero-Shot Results on xGQA. Accuracy (%) of our unified 2en models
trained on different training datasets as well as our Single-Lang 2en
baselines and the baselines from (Pfeiffer et al., 2022).

###  6.1 Experimental Setup

Evaluation Datasets. We consider two evaluation datasets: (i) Our proposed
benchmark $\mathrm{MaXM}$ consisting of 300 question-answer pairs per language
from approximately 300 images in 7 languages (en,fr,hi,iw,ro,th,zh, Sect. 4);
(ii) xGQA (Pfeiffer et al., 2022), the testdev split of 12,578 question-answer
pairs per language from 398 images in 8 languages (en,bn,de,id,ko,pt,ru,zh).

Training Datasets. We consider three datasets for training our models, all are
translations of existing large-scale English VQA datasets: VQA2.0 (Goyal et
al., 2017), $\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO and
$\mathrm{VQ}^{2}\\!\mathrm{A}$-CC3M (Changpinyo et al., 2022) to the 13
languages covered by $\mathrm{MaXM}$ and xGQA. We use the Karpathy training
split (Karpathy & Fei-Fei, 2015) for VQA2.0 and
$\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO and the standard training split for
$\mathrm{VQ}^{2}\\!\mathrm{A}$-CC3M.

Our Main Models. We have two main models: “2$\langle\mathrm{lang}\rangle$” and
“2en” models, where the main difference lies in whether the answers are
$\langle\mathrm{lang}\rangle$ or English. In particular, for each question in
$\langle\mathrm{lang}\rangle$, our 2$\langle\mathrm{lang}\rangle$ model is
trained and evaluated with “Answer in $\langle$lang$\rangle$:
$\langle$question$\rangle$”, whereas our 2en model is trained and evaluated
with the English-only prompt “Answer in en: $\langle$question$\rangle$”. The
2en model is used to evaluate our approach on xGQA, which requires the answer
to be in English.

Baselines. We train single-language baselines (Single-Lang in our tables), in
which only the data in $\langle\mathrm{lang}\rangle$ are considered. This is
to see if multi-language prompted training for a unified model is beneficial
or detrimental. We also include zero-shot baselines for our 2en model: M3P and
mBERT${}^{\textrm{Ada}}$ from (Pfeiffer et al., 2022). Note that both M3P and
mBERT${}^{\textrm{Ada}}$ have access to the (English) GQA training data
(Hudson & Manning, 2019), where our model does not. On the other hand, they do
not use translated data as in our case.

Evaluation Metrics. We use Exact Match _Accuracy_ for xGQA, following
(Pfeiffer et al., 2022). For $\mathrm{MaXM}$, we report Exact Match _Accuracy_
as well as _soft_ text similarity metrics _CIDEr_ (Vedantam et al., 2015) and
_ROUGE-L_ (Lin, 2004). For _Accuracy_ , the answer is correct if it belongs to
any of the ground-truth answers. For _CIDEr_ and _ROUGE-L_ , we treat each of
the ground-truth answers equally as one of the references (as if each of them
was answered by an annotator).

###  6.2 Results

Table 5 shows our main results on $\mathrm{MaXM}$. First, on VQA2.0, our
unified 2$\langle\mathrm{lang}\rangle$ model performs similarly or better than
each of the Single-Lang 2$\langle\mathrm{lang}\rangle$ baselines,
demonstrating the utility of our modeling choice of learning from related
languages. Second, we find that translated COCO is overall the best training
data source. We attribute this to (i) the fact that
$\mathrm{VQ}^{2}\\!\mathrm{A}$ was used to generate
$\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO, and (ii)
$\mathrm{VQ}^{2}\\!\mathrm{A}$-COCO is generally more robust in the cross-
dataset setting (Changpinyo et al., 2022).

Surprisingly, even though mT5 training data is biased towards English
(Kreutzer et al., 2022), English is not the highest-performing language using
the unified model in the generative VQA task. Depending on the metric, other
languages perform similar to or better than English: Hindi and Thai for
Accuracy, French and Romanian for CIDEr, and Hindi and Thai for ROUGE-L.

Table 6 shows additional results on xGQA. We outperform _multilingual_ zero-
shot baselines on all non-English languages, without access to English GQA
labeled data. This further confirms that our unified approach to mVQA is
effective. In addition, unlike on $\mathrm{MaXM}$, VQA2.0 is the best pre-
training data source. We attribute this to the fact that VQA2.0 and xGQA share
COCO images (Lin et al., 2014). This highlights the utility of $\mathrm{MaXM}$
as additional out-of-domain test-only VQA evaluation data.

##  7 Conclusions

We take initial steps toward multilingual VQA by proposing scalable solutions
on both data creation and modeling fronts. We create a multilingual VQA
benchmark in 7 diverse languages to drive modeling progress on multilingual
VQA. We establish strong unified and open-ended VQA models that work well on
13 languages.

In this work, we used native-language question generation in a limited scope.
We would like to expand this and have a single native-language question
generation for all target answers.

Acknowledgments. We would like to thank Jialin Wu, Kenton Lee, Tomer
Levinboim, Nan Ding, Sarah Laszlo, Doron Kukliansky, and Tania Bedrax-Weiss
for their feedback and discussion. Word clouds are generated from
https://www.jasondavies.com/wordcloud/.

## References

  * Aggarwal & Kale (2020) Aggarwal, P. and Kale, A.  Towards zero-shot cross-lingual image retrieval.  _arXiv preprint arXiv:2012.05107_ , 2020.
  * Agrawal et al. (2018) Agrawal, A., Batra, D., Parikh, D., and Kembhavi, A.  Don’t just assume; look and answer: Overcoming priors for visual question answering.  In _CVPR_ , 2018.
  * Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.  Flamingo: a visual language model for few-shot learning.  _arXiv preprint arXiv:2204.14198_ , 2022.
  * Alberti et al. (2019) Alberti, C., Andor, D., Pitler, E., Devlin, J., and Collins, M.  Synthetic QA corpora generation with roundtrip consistency.  In _ACL_ , 2019.
  * Antol et al. (2015) Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., and Parikh, D.  VQA: Visual question answering.  In _ICCV_ , 2015.
  * Banerjee et al. (2021) Banerjee, P., Gokhale, T., Yang, Y., and Baral, C.  WeaQA: Weak supervision via captions for visual question answering.  In _Findings of ACL-IJCNLP_ , 2021.
  * Barrault et al. (2018) Barrault, L., Bougares, F., Specia, L., Lala, C., Elliott, D., and Frank, S.  Findings of the third shared task on multimodal machine translation.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , 2018.
  * Biten et al. (2019) Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D.  Scene text visual question answering.  In _ICCV_ , 2019.
  * Bradbury et al. (2018) Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Bugliarello et al. (2022) Bugliarello, E., Liu, F., Pfeiffer, J., Reddy, S., Elliott, D., Ponti, E. M., and Vulić, I.  IGLUE: A benchmark for transfer learning across modalities, tasks, and languages.  In _ICML_ , 2022.
  * Changpinyo et al. (2021) Changpinyo, S., Sharma, P., Ding, N., and Soricut, R.  Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _CVPR_ , 2021.
  * Changpinyo et al. (2022) Changpinyo, S., Kukliansky, D., Szpektor, I., Chen, X., Ding, N., and Soricut, R.  All you may need for VQA are image captions.  In _NAACL_ , 2022.
  * Chen et al. (2015) Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L.  Microsoft COCO Captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Cho et al. (2021) Cho, J., Lei, J., Tan, H., and Bansal, M.  Unifying vision-and-language tasks via text generation.  In _ICML_ , 2021.
  * Desai et al. (2021) Desai, K., Kaul, G., Aysola, Z., and Johnson, J.  RedCaps: Web-curated image-text data created by the people, for the people.  In _NeurIPS_ , 2021.
  * Dosovitskiy et al. (2021) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Elliott et al. (2016) Elliott, D., Frank, S., Sima’an, K., and Specia, L.  Multi30K: Multilingual English-German image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language, ACL_ , 2016.
  * Elliott et al. (2017) Elliott, D., Frank, S., Barrault, L., Bougares, F., and Specia, L.  Findings of the second shared task on multimodal machine translation and multilingual image description.  In _Proceedings of the Second Conference on Machine Translation_ , Sep 2017.
  * Goyal et al. (2017) Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D.  Making the V in VQA matter: Elevating the role of image understanding in visual question answering.  In _CVPR_ , 2017.
  * Gu et al. (2022) Gu, J., Meng, X., Lu, G., Hou, L., Niu, M., Xu, H., Liang, X., Zhang, W., Jiang, X., and Xu, C.  Wukong: 100 million large-scale chinese cross-modal pre-training dataset and a foundation framework.  _arXiv preprint arXiv:2202.06767_ , 2022.
  * Gurari et al. (2018) Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P.  VizWiz Grand Challenge: Answering visual questions from blind people.  In _CVPR_ , 2018.
  * Honovich et al. (2021) Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., and Abend, O.  $Q^{2}$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.  In _EMNLP_ , 2021.
  * Hudson & Manning (2019) Hudson, D. A. and Manning, C. D.  GQA: A new dataset for real-world visual reasoning and compositional question answering.  In _CVPR_ , 2019.
  * Johnson et al. (2017a) Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R.  CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.  In _CVPR_ , 2017a.
  * Johnson et al. (2017b) Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., Hughes, M., and Dean, J.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  _TACL_ , 5:339–351, 2017b.
  * Kafle & Kanan (2017) Kafle, K. and Kanan, C.  An analysis of visual question answering algorithms.  In _ICCV_ , 2017.
  * Karpathy & Fei-Fei (2015) Karpathy, A. and Fei-Fei, L.  Deep visual-semantic alignments for generating image descriptions.  In _CVPR_ , 2015.
  * Krasin et al. (2017) Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-El-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Kamali, S., Malloci, M., Pont-Tuset, J., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., and Murphy, K.  OpenImages: A public dataset for large-scale multi-label and multi-class image classification.  _Dataset available from https://g.co/dataset/openimages_ , 2017.
  * Kreutzer et al. (2022) Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., Tapo, A., Subramani, N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A., Papadimitriou, I., Osei, S., Suarez, P. O., Orife, I., Ogueji, K., Rubungo, A. N., Nguyen, T. Q., Müller, M., Müller, A., Muhammad, S. H., Muhammad, N., Mnyakeni, A., Mirzakhalov, J., Matangira, T., Leong, C., Lawson, N., Kudugunta, S., Jernite, Y., Jenny, M., Firat, O., Dossou, B. F. P., Dlamini, S., de Silva, N., Çabuk Ballı, S., Biderman, S., Battisti, A., Baruwa, A., Bapna, A., Baljekar, P., Azime, I. A., Awokoya, A., Ataman, D., Ahia, O., Ahia, O., Agrawal, S., and Adeyemi, M.  Quality at a glance: An audit of web-crawled multilingual datasets.  _Transactions of the Association for Computational Linguistics_ , 10:50–72, 2022.  doi: 10.1162/tacl˙a˙00447.  URL https://aclanthology.org/2022.tacl-1.4.
  * Krishna et al. (2017) Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., Bernstein, M., and Fei-Fei, L.  Visual Genome: Connecting language and vision using crowdsourced dense image annotations.  _IJCV_ , 123(1):32–73, 2017.
  * Ku et al. (2020) Ku, A., Anderson, P., Patel, R., Ie, E., and Baldridge, J.  Room-Across-Room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.  In _EMNLP_ , 2020.
  * Kuznetsova et al. (2020) Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J. R. R., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Duerig, T., and Ferrari, V.  The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale.  _IJCV_ , 128(7):1956–1981, 2020.
  * Kwiatkowski et al. (2019) Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., , Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S.  Natural Questions: a benchmark for question answering research.  _TACL_ , 7:453–466, 2019.
  * Lee et al. (2021) Lee, H., Scialom, T., Yoon, S., Dernoncourt, F., and Jung, K.  QACE: Asking questions to evaluate an image caption.  In _Findings of EMNLP_ , 2021.
  * Lembersky et al. (2012) Lembersky, G., Ordan, N., and Wintner, S.  Language models for machine translation: Original vs. translated texts.  _Computational Linguistics_ , 38(4):799–825, 2012\.
  * Li et al. (2021) Li, L., Lei, J., Gan, Z., and Liu, J.  Adversarial VQA: A new benchmark for evaluating the robustness of vqa models.  In _ICCV_ , 2021.
  * Li et al. (2019) Li, X., Xu, C., Wang, X., Lan, W., Jia, Z., Yang, G., and Xu, J.  COCO-CN for cross-lingual image tagging, captioning, and retrieval.  _IEEE Transactions on Multimedia_ , 21(9):2347–2360, 2019.
  * Lin (2004) Lin, C.-Y.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , 2004.
  * Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P.  Microsoft COCO: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. (2021) Liu, F., Bugliarello, E., Ponti, E. M., Reddy, S., Collier, N., and Elliott, D.  Visually grounded reasoning across languages and cultures.  In _EMNLP_ , 2021.
  * Marino et al. (2019) Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R.  OK-VQA: A visual question answering benchmark requiring external knowledge.  In _CVPR_ , 2019.
  * Ordonez et al. (2011) Ordonez, V., Kulkarni, G., and Berg, T.  Im2Text: Describing images using 1 million captioned photographs.  In _NIPS_ , 2011.
  * Pfeiffer et al. (2022) Pfeiffer, J., Geigle, G., Kamath, A., Steitz, J.-M. O., Roth, S., Vulić, I., and Gurevych, I.  xGQA: Cross-lingual visual question answering.  In _Findings of ACL_ , 2022.
  * Pont-Tuset et al. (2020) Pont-Tuset, J., Uijlings, J., Changpinyo, S., Soricut, R., and Ferrari, V.  Connecting vision and language with localized narratives.  In _ECCV_ , 2020.
  * Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_ , 2020.
  * Rajpurkar et al. (2016) Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P.  SQuAD: 100,000+ questions for machine comprehension of text.  In _EMNLP_ , 2016.
  * Rajpurkar et al. (2018) Rajpurkar, P., Jia, R., and Liang, P.  Know what you don’t know: Unanswerable questions for SQuAD.  In _ACL_ , 2018.
  * Ren et al. (2015) Ren, M., Kiros, R., and Zemel, R.  Exploring models and data for image question answering.  In _NIPS_ , 2015.
  * Sanh et al. (2022) Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Fevry, T., Fries, J. A., Teehan, R., Scao, T. L., Biderman, S., Gao, L., Wolf, T., and Rush, A. M.  Multitask prompted training enables zero-shot task generalization.  In _ICLR_ , 2022.
  * Schuhmann et al. (2021) Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A.  LAION-400M: Open dataset of CLIP-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schwenk et al. (2022) Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R.  A-OKVQA: A benchmark for visual question answering using world knowledge.  _arXiv preprint arXiv:2206.01718_ , 2022.
  * Sharma et al. (2018) Sharma, P., Ding, N., Goodman, S., and Soricut, R.  Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _ACL_ , 2018.
  * Shazeer & Stern (2018) Shazeer, N. and Stern, M.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _ICLR_ , 2018.
  * Sheng et al. (2021) Sheng, S., Singh, A., Goswami, V., Magana, J. A. L., Galuba, W., Parikh, D., and Kiela, D.  Human-adversarial visual question answering.  In _NeurIPS_ , 2021.
  * Singh et al. (2019) Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M.  Towards VQA models that can read.  In _CVPR_ , 2019.
  * Srinivasan et al. (2021) Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M.  WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning.  In _SIGIR_ , 2021.
  * Thapliyal et al. (2022) Thapliyal, A. V., Pont-Tuset, J., Chen, X., and Soricut, R.  Crossmodal-3600: A massively multilingual multimodal evaluation dataset.  _arXiv preprint arXiv:2205.12522_ , 2022.
  * Vedantam et al. (2015) Vedantam, R., Lawrence Zitnick, C., and Parikh, D.  CIDEr: Consensus-based image description evaluation.  In _CVPR_ , 2015.
  * Volansky et al. (2013) Volansky, V., Ordan, N., and Wintner, S.  On the features of translationese.  _Digital Scholarship in the Humanities_ , 30(1):98–118, 2013.
  * Wang et al. (2020) Wang, A., Cho, K., and Lewis, M.  Asking and answering questions to evaluate the factual consistency of summaries.  In _ACL_ , 2020.
  * Wang et al. (2022a) Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L.  GIT: A generative image-to-text transformer for vision and language.  _arXiv preprint arXiv:2205.14100_ , 2022a.
  * Wang et al. (2019) Wang, X., Wu, J., Chen, J., Li, L., Wang, Y.-F., and Wang, W. Y.  VaTeX: A large-scale, high-quality multilingual dataset for video-and-language research.  In _ICCV_ , 2019.
  * Wang et al. (2022b) Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y.  SimVLM: Simple visual language model pretraining with weak supervision.  In _ICLR_ , 2022b.
  * Wei et al. (2022) Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V.  Finetuned language models are zero-shot learners.  In _ICLR_ , 2022.
  * Xue et al. (2021) Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _NAACL_ , 2021.
  * Yang et al. (2021) Yang, A., Miech, A., Sivic, J., Laptev, I., and Schmid, C.  Just ask: Learning to answer questions from millions of narrated videos.  In _ICCV_ , 2021.
  * Yoshikawa et al. (2017) Yoshikawa, Y., Shigeto, Y., and Takeuchi, A.  STAIR captions: Constructing a large-scale Japanese image caption dataset.  In _ACL_ , 2017.
  * Zellers et al. (2019) Zellers, R., Bisk, Y., Farhadi, A., and Choi, Y.  From recognition to cognition: Visual commonsense reasoning.  In _CVPR_ , 2019.
  * Zhu et al. (2016) Zhu, Y., Groth, O., Bernstein, M., and Li, F.-F.  Visual7W: Grounded question answering in images.  In _CVPR_ , 2016.

Figure 6: Detailed Instructions on Question Annotation

Figure 7: Detailed Instructions on Answer Annotation as well as Answer
Expansion and Standardization.

Figure 8: Additional MaXM examples.

Figure 9: Additional examples on our approach to multilingual VQA data
generation. Green, yellow, and red texts correspond to “Correct”, “Almost
Correct”, and “Incorrect,” respectively.

Figure 10: Examples of Collection questions.

Figure 11: Examples of Ambiguous questions that we flagged and filtered out.

Figure 12: Examples of Responsible-AI-sensitive questions that we flagged and
filtered out. Faces are hidden.

##  Appendix A Considerations and Limitations

Our dataset is intended to be used for research-only purposes.

Our pipeline takes in an image caption as input. Image captions may have
mistakes and biases, which could be further amplified by machine learning
models used by our approach. In particular, we use generative models for
automatic question generation and machine translation that may create outputs
with incorrect or nonfactual contents or outputs with Translationese artifacts
(cf., Sect. 3.4). We have mitigated this manually via human in the loop and
automatically via the caption-answer consistency check (cf., Sect. 3.2). Note
that the English $\mathrm{VQ}^{2}\\!\mathrm{A}$ (Changpinyo et al., 2022) that
we leverage in our pipeline also has similar filtering using the round-trip
consistency check via question answering. Together these significantly improve
the correctness and fluency of our pipeline. In addition, we explicitly mark
examples that can be considered Responsible-AI-sensitive, but not necessarily
incorrect; see Sect. B for details and examples.

Another type of biases is the low coverage of particular types of answers,
resulting from the image captions not mentioning the absences of objects or
properties. We have also taken a step toward mitigating this. See Sect. 3.3.

Finally, we select a diverse set of languages, alleviating typological,
genealogical, and geographical language biases presented in the VQA research
community.

We mainly use Crossmodal-3600 ($\mathrm{XM3600}$) (Thapliyal et al., 2022).
Open Images (Krasin et al., 2017; Kuznetsova et al., 2020) and the
multilingual captions in $\mathrm{XM3600}$ are human-curated and cleaned,
which mitigates the risks that $\mathrm{MaXM}$ would contain information that
names or uniquely identifies individual people or offensive content.

##  Appendix B Human Verification and Modification

###  B.1 Annotation Guideline

We provide our general instructions and detailed instructions on question
annotation in Fig. 6, where we explicitly ask the annotators to be wary of
Responsible-AI-sensitive questions. Fig. 7 provides detailed instructions on
answer annotation and on answer expansion and standardization.

###  B.2 Additional Examples

Additional Examples. Fig. 8 provides additional examples to the ones in Fig.
1. Again, we highlight the richness and diversity of our questions. For
instance, it requires recognizing a cross under occlusion (French), a type of
vegetables (Hindi), the Arabic language (Hebrew), and a type of flowers
(Romanian). Some of these examples are specific to particular languages; it
would be difficult for other language speakers to answer the Hebrew example
(or the Chinese example in Fig. 1, which requires OCR).

We also highlight the richness of our candidate answers. For the “where”
question in Thai, 10 answers count as correct. Similarly, the Romanian example
in Fig. 1 provides multiple diverse surface forms for “coffee with cream.”

Fig. 9 additional examples to the Chinese one in Fig 2. These examples
showcase the efficiency of our annotation process. They also provide concrete
examples of “Almost Correct.” For instance, in the middle example, the Thai
translation of “What leaves are in the photo?” is not _neutral_ because it
contains an Honorific particle
333https://en.wikipedia.org/wiki/Thai_honorifics; it ends with “khá” which
signifies a sign of respect to the addressee and indicates that the sex of the
speaker is female. Finally, these examples provide a glimpse of sources of
errors. For instance, it is $\mathrm{VQ}^{2}\\!\mathrm{A}$ that hallucinates
“in the video” in the Hindi example on the right.

Collection Examples. Fig. 11 provides examples of “Collection” questions. We
keep these questions as we believe they are useful in practice and as a way to
encourage the community to work on better automatic evaluation metrics for
this type of questions.

Ambiguous Examples. Fig. 10 provides examples of “Ambiguous” questions that we
filter out. Reasons include object being too small (English) or irregular
(Chinese), determining sizes being subjective (French), and not enough context
(Hebrew, Romanian). “What kind/What type” questions are particularly difficult
to answer and tend to be ambiguous.

Responsible-AI-sensitive Examples. Fig. 12 provides examples of Responsible-
AI-sensitive questions that we filter out. These cases are often associated
with directly asking for the information about or describing a particular
gender or race, or involving an incorrect assumption about such protected
attributes (e.g., girl vs. woman in the Hebrew example).

##  Appendix C Implementation Details

As mentioned in Sect. 5.1, our architecture is based on ViT-L16 (Dosovitskiy
et al., 2021) and mT5-Large (Xue et al., 2021), trained to generate an answer
text in an end-to-end fashion. We use the Flax implementation (Bradbury et
al., 2018). For training both our 2$\langle\mathrm{lang}\rangle$ and 2en
models, we use Adafactor (Shazeer & Stern, 2018) with a $\beta_{1}$ of 0 and a
second-moment exponential decay of 0.8. We use a linear warmup of 1K steps
with a peak learning of learning rate of 1e-3 and inverse square-root decay.
We set the ViT dropout rate to 0 and the mT5 dropout rate to 0.1. We train
each model with data parallelism using 16 Cloud TPU
Pods444https://cloud.google.com/tpu, each with a batch size of 512, for 100K
steps. We use standard image resolution of 224x224. We use the maximum input
length of 24 and the target output length of 8.