# Can Vision-Language Models be a Good Guesser?
Exploring VLMs for Times and Location Reasoning

Gengyuan Zhang
LMU Munich
Munich Center for Machine Learning
<EMAIL_ADDRESS>Yurui Zhang
TU Munich
<EMAIL_ADDRESS>Kerui Zhang
LMU Munich
<EMAIL_ADDRESS>Volker Tresp
LMU Munich
Munich Center for Machine Learning
<EMAIL_ADDRESS>
###### Abstract

Vision-Language Models (VLMs) are expected to be capable of reasoning with
commonsense knowledge as human beings. One example is that humans can reason
where and when an image is taken based on their knowledge. This makes us
wonder if, based on visual cues, Vision-Language Models that are pre-trained
with large-scale image-text resources can achieve and even outperform human’s
capability in reasoning times and location. To address this question, we
propose a two-stage Recognition and Reasoning probing task, applied to
discriminative and generative VLMs to uncover whether VLMs can recognize times
and location-relevant features and further reason about it. To facilitate the
investigation, we introduce WikiTiLo, a well-curated image dataset
compromising images with rich socio-cultural cues. In the extensive
experimental studies, we find that although VLMs can effectively retain
relevant features in visual encoders, they still fail to make perfect
reasoning. We will release our dataset and codes to facilitate future studies.

##  1 Introduction

Figure 1: An example image of times and location reasoning in WikiTiLo: Can
you tell where and when is this picture taken?  Figure 2: We apply a two-stage
probing task Recognition and Reasoning to both discriminative and generative
VLMs. We evaluate Recognition on discriminative VLMs and Reasoningon
generative VLMs with the visual encoder intact.

Vision-Language Models (VLMs) have exhibited remarkable advancements in
enhancing multi-modal learning. On one hand, discriminative VLMs such as CLIP
[25], BLIP [17], and ImageBind [10] offer powerful encoders that possess
excellent representation and demonstrate significant transferability on
recognition and understanding tasks. On the other hand, generative VLMs like
LLaMA Adapter [41, 9], BLIP2 [16], Flamingo [2], and LLaVA [22] combine visual
encoders from discriminative VLMs with Large Language Models (LLMs) such as
LLaMA[35] and GPT3 [5] to further bridge visual features and leverage the
exceptional reasoning abilities of LLMs.

With VLMs trained on extensive knowledge corpora comprising multi-modal data,
an intriguing question arises: can VLMs reason about the implicit
sociocultural backgrounds associated with an image, such as its times and
location? Just as in the popular game ‘GeoGuesser,’ where players are tasked
with locating an image based on visual cues, we contemplate whether VLMs can
also exhibit a similar aptitude for being a ‘good guesser.’ This kind of
commonsense understanding necessitates that the model can infer the times and
location of an image by comprehending not only the local visual evidence but
also the underlying concepts. For example, from Fig. 1, humans with related
experience can easily infer that the image was most likely taken in Germany
and during the famous Oktoberfest based on the cues such as people wearing
traditional Bavarian clothes and celebrating with beers and barrels.

Under this background, our work aims to investigate the times and location
reasoning capabilities of VLMs. Consequently, we put forward two research
questions:
RQ1: Can discriminative VLMs recognize times and location-relevant features
from visual input?
RQ2: Can generative VLMs reason about times and locations associated with
images based on visual cues?

To address these questions step by step, we devise a two-stage probing task:
Recognition and Reasoning, applied to two categories of Vision-Language Models
(VLMs) as illustrated in Fig. 2. In the first stage Recognition, we assess
whether the visual encoders trained within discriminative VLMs can
successfully identify distinctive features for location and times reasoning,
using a classification task. In the second stage Reasoning, assuming that the
visual encoder can identify salient features within its pre-trained
representations, we evaluate the ability of generative VLMs, with the visual
encoder intact, to conduct times and location reasoning through an open-ended
question-answering task.

We also construct a new dataset for times and location reasoning, WikiTiLo
(WikiCommon Times and Location), which comprises images captured over a broad
time range and is geographically balanced to mitigate cultural bias. The
dataset has been carefully curated to ensure that each image contains distinct
visual cues that align with human expert knowledge.

This is one of the first works to investigate whether state-of-the-art large
pre-trained Vision-Language Models, enhanced with Large Language Model
techniques, are capable of times and location reasoning.

Our contributions to this work includes:

  1. 1.

we construct a new dataset, WikiTiLo, for times and location reasoning, with a
focus on the socio-cultural background behind images, and annotated with
curated times and location information;

  2. 2.

we propose a two-stage probing task, Recognition and Reasoning, to evaluate
the ability of VLMs to recognize times and location-relevant visual cues on
classification tasks, and subsequently reason about times and location on
question answering tasks;

  3. 3.

we evaluate three discriminative VLMs and two generative VLMs on this same
benchmark. Experiments show that visual encoders in large-scale pre-trained
discriminative VLMs can effectively recognize visual cues that are beneficial
for times and location reasoning. However, we also identify a gap in the
ability of generative VLMs to fully leverage these visual cues.

##  2 Related Work

Vision-Language Model Vision-Language Models bridge the gap between
flourishing studies in computer vision and natural language studies. Recent
works, including [6, 14, 25, 17, 38, 23, 18, 7, 40, 15, 39, 24, 10] aim to
learn a representation of images and texts in a joint feature space and unify
vision and language tasks like and visual question answering [15, 14, 33].
Along with the boost of pre-training generative Large Language Models (LLM)
like GPT-3 [5], LLaMA [35] manifest their excellent commonsense reasoning
ability in broad tasks and are fastly adapted to multi-modal scenarios as in
works including Flamingo [2], LLaMA-adpter [9], BLIP-2 [16].

Times and Location Reasoning Recent studies on image times and geo-location
prediction [37, 13, 30] show an interesting capacity of deep learning models.
CLIP [25] also reports a similar Geolocalization classification. Some
geolocation datasets such as Cross-View Time Dataset (CVT) [28], GT-CrossView
[36], and Cross-View Image Geolocalization (CVIG) [19] have been provided for
the downstream task as Geolocalization. But unlike geolocation estimation, our
work is more focused on commonsense knowledge-based reasoning and pays more
attention to the sociocultural perspective of images when constructing the
dataset. TARA [8] is a preceding work on exploring CLIP’s commonsense
knowledge in this regard; compared to TARA, we construct a dataset consisting
of images with a wider timespan and unbiased location distribution and
evaluate on a wider range of VLMs.

Model Probing Model probing is first proposed in Natural Language Processing
to investigate whether representations of Language Models have already learned
specific linguistic properties [32, 12]. [4] proposes a framework to probe
visual superpixels as an equivalence of word tokens. However, probing
multimodal models is a relatively new area. [21, 29, 26, 25] attempt to
investigate multimodal probing tasks such as object counting and position
identification to learn whether VLMs are capable of understanding multimodal
concepts. Our work aims to utilize a simple linear probing [1] method to probe
whether VLMs can reason commonsense knowledge like times and location.

(a) Mexico in 1890

(b) Kazakhstan in 1890

(c) Germany in 1930

(d) Vietnam in 1940

(e) Afghanistan in 1980

(f) India in 1980

(g) South Africa in 2000

(h) China in 2010

Figure 3: Some example images in the WikiTiLo exhibit abundant visual cues
with a sociocultural background, such as stylish buildings, text on images,
and traditional clothing. These visual cues enable humans to reason and draw
conclusions based on the provided evidence.

(a) Location Distribution

(b) Times Distribution

Figure 4: The WikiTiLo dataset exhibits a diverse distribution of times and
locations. In terms of time, the images range from 1827 to the post-2000 era.
Regarding location, we selected 30 countries whose images met our filtering
criteria, representing eight regions. Due to the development of Internet
media, images taken after 2000 constitute a significant portion of the
dataset. Conversely, images taken before 1900 only account for 10% of the
dataset, primarily due to limited data availability and poor quality.

##  3 Dataset: WikiTiLo

We construct the WikiTiLo (WikiCommon Times and Location) dataset, which
consists of 6296 images with annotation of the specific time and country where
images are taken. The dataset covers 30 countries in Europe, Asia, America,
and Africa and the years from 1826 to 2021.

For some experimental settings, we also use regions as a higher level of
geological concept other than countries since the country border does not
equate with cultural borders [34]. We divide these countries in 8 regions
according to UNESCO based on geography and cultural proximity. The designation
can be found in Supplementary material.

###  3.1 Data Curation

Wikimedia Commons111https://commons.wikimedia.org/wiki/Main_Page which is a
project of the Wikimedia Foundation, contains a media repository of open
images, sounds, videos, and other media. The images of Wikimedia Commons are
accumulated from different countries and regions around the world, from
different historical periods, and in various categories.

We select the images manually based on such a criterion that the location
identity and time period features of each image can be distinguished from
architectural patterns, costume styles, language types, movement postures,
photo colors, and quality or other fine-grained features by humans. In
particular, to avoid image distribution bias, we attempt to control image
origins and balance the ratio of countries that range from more visible areas
like Europe and America to less attended areas like Africa and Central Asia,
according to socio-cultural characteristics and geographical regions. A
detailed selection paradigm can be found in Supplementary Material.

In total, we have included 30 countries from which images can meet our
criteria to represent different regions. Fig. 3 presents some example images
in WikiTiLo.

###  3.2 Statistics

The times and location proportions of WikiTiLo are demonstrated in Fig. 4. All
images are nearly evenly distributed in 8 regions. Half of the images are
taken after 2000 due to the multimedia era and 10% are before 1900 due to
limited resources and bad quality. The entire dataset contains 6296 images.
For training tasks in some experimental settings, we use $80\%$ as the
training set, $10\%$ as the validation set, and $10\%$ for evaluation
reported.

##  4 Methodology

We explore the times and location reasoning of Vision-Language Models in two
stages: Recognition and Reasoning. In the first step Recognition, we
investigate whether the visual encoders of discriminative VLMs can effectively
recognize visual cues relevant to times and location reasoning. In the second
step Reasoning, we further explore whether with the intactvisual encoders,
LLM-enhanced generative VLMs can reason about times and location based on
these visual cues.

###  4.1 Recognition

In the Recognition probing stage, we test whether discriminative VLMs can
recognize prominent times/location-relevant visual features via probing the
features of visual encoders on multi-class classification tasks.
Recognition${}_{\textsc{Times}}$ is a fine-grained image classification task
where models predict the occurrence time of an image. This task evaluates
whether visual representations of a model contain commonsense knowledge of
historical times. Recognition${}_{\textsc{Location}}$ is a fine-grained image
classification task where models predict the location where an image is taken.
We test on both country level and region level that aligns with the dataset
information. We attempt to evaluate the visual encoders of discriminative VLMs
in two settings: zero-shot with a prediction calculated as in Eq. 1 and linear
probing as in Eq. 2.

$\mathbb{P}_{zs}(c_{i}|v)=\frac{\exp(\mathrm{sim}(\mathrm{VE}(v),\mathrm{TE}(p\circ
c_{i})))}{\sum_{j}^{C}\exp(\mathrm{sim}(\mathrm{VE}(v),\mathrm{TE}(p\circ
c_{j})))}$ (1)

$\mathbb{P}_{lp}(c_{i}|v)=\mathrm{softmax}(\mathrm{linear}(\mathrm{VE}(v)))_{i}$
(2)

where $p$ is hard prompts we use, $\mathrm{VE}$ and $\mathrm{TE}$ are visual
and textual encoders in VLMs, and $c$ is the class label.

###  4.2 Reasoning

Upon the hypothesis that visual encoders extract recognizable visual features
for times and location cues, we further evaluate generative VLMs that can give
us good reasoning about times and location based on the cues. We propose using
open-ended visual question answering tasks Reasoning${}_{\textsc{Times}}$ and
Reasoning${}_{\textsc{Location}}$ to reason times and location.

OpenFlamingo Cloze Test <image> Output: This is a local photo taken in country
China. <|endofchunk|>
Open Flamingo VQA
The photograph was taken in one of the following 30 countries. These 30
countries are … <image> Question: In which country was this photograph taken?
Short answer: China. <|endofchunk|>
OpenFlamingo VQA - CoT The photograph was taken in one of the following 30
countries. These 30 countries are … <image> Question: In which country was
this photograph taken? Answer: Because in the photo, there is an elderly
person wearing a turban standing in a barren land, with the houses built on a
hill and a snow-capped mountain in the background, which are consistent with
the attire and landscape of Afghan, this photo was taken in
Afghanistan.<|endofchunk|>

Figure 5: We list templates for each protocol for
Reasoning${}_{\textsc{Location}}$. For OpenFlamingo, few-shot examples are
used to facilitate in-context learning. In the case of the zero-shot setting,
two text samples are retained but without the inclusion of image tokens
<image>. We also list the templates for Reasoning${}_{\textsc{Times}}$ in the
Supplementary Material.

##  5 Experiments

In Recognition task, we choose three state-of-the-art Vision-Language models:
ViLT [14], CLIP [25], and BLIP [17], for comparison. In the Linear Probing
setting, we also choose one pre-trained ResNet-50 [11] as a pure vision
baseline. For Reasoning tasks, we evaluate two generative VLMs, OpenFlamingo
[3] and LLaMA-adapter V2 [9], both with CLIP-ViT-L/14 as the backbone visual
encoder, which is one of the best-performing encoders in the Recognition task.

LLaMA-Adapter V2 Instructiona Instruction: The photograph was taken in one of
the following 30 countries. These 30 countries are … In which country was this
photo taken? LLaMA-Adapter V2 Instructionb Instruction: In which country was
this photo taken?

Figure 6: For the LLaMA-Adapter V2, we use a simple question as the
instruction. For the case with labels, the label sentence was added in the
instruction before the question.

llcccccc[colortbl-like] 7-12,15-17,20 [l]1-1 [c]1-1 [c]1-3Times
[c]1-3Location${}_{\text{country}}$
Model Accuracy Precision F1-score Accuracy Precision F1-score
[c]12-1  Recognition CLIP-ViT-B/32 78.57% 70.66% 70.66% 44.28% 43.11% 40.19%
CLIP-ViT-B/16 75.71% 67.94% 70.08% 55.23% 54.16% 53.71%
CLIP-ViT-L/14 76.03% 69.73& 69.62% 68.25% 60.77% 61.87%
CLIP-ViT-L/14@336px 79.05% 73.66% 73.75% 72.85% 64.52% 65.79%
BLIP-129M 30.95% 46.81% 46.14% 35.23% 35.30% 30.07%
BLIP-129M-Coco 32.22% 50.13% 43.49% 35.23% 34.25% 28.74%
BLIP-129M-Flickr 58.57% 54.33% 56.19% 33.49% 29.84% 27.32%
BLIP-ViT-L 44.44% 54.97% 52.86% 42.22% 43.30% 40.10%
BLIP-ViT-L-Coco 42.06% 51.75% 49.40% 39.20% 38.93% 34.61%
BLIP-ViT-L-Flickr 66.51% 58.37% 60.14% 40.47% 41.24% 37.54%
ViLT-Coco 70.16% 58.51%  57.78% 3.65% 3.60% 3.10%
ViLT-Flickr30K 51.90% 48.84% 49.21% 7.93% 4.00% 4.80%
[c]5-1  Reasoning OpenFlamingo-Cloze Test 27.70% 26.36% 11.49% 3.89% 3.69%
2.18%
OpenFlamingo-VQA 31.59% 30.36% 28.60% 48.88% 53.78% 41.19%
OpenFlamingo-VQA CoT 35.21% 29.36% 28.42% 40.3% 45.24% 33.17%
LLaMA-Adapter V2-Instructiona 58.02% 28.04% 32.88% 23.05% 52.64% 18.66%
LLaMA-Adapter V2-Instructionb 34.34% 58.59% 37.70% 45.62% 51.57% 35.50%
Frequency baseline 25.07% 25.29% 23.27% 3.33% 2.95% 2.88%
Human baseline 67.42% - - 48.30% - -

Table 1: Results of performance without training for Recognition and
Reasoning. The highest performance among all models is highlighted in bold,
and the best performance of each VLM is marked in italic.

###  5.1 Experimental Setups

Zero-shot Setting To evaluate the zero-shot performance of discriminative
VLMs, we adopt a prompt “A historical / recent / contemporary image with
black-and-white/color taken in [times period]” for
Recognition${}_{\textsc{Times}}$ classification and “a local photograph from
[country] in [region]” for Recognition${}_{\textsc{Location}}$ classification.
We calculate the image-text similarities across all candidates and retrieve
the top rank as predicted classes.

Linear Probing Setting We used a single linear layer with vision encoders’
embedding size as input size and 8 for Recognition${}_{\textsc{Location}}$ on
regions, and 4 for Recognition${}_{\textsc{Times}}$ as output size. The visual
encoders of the model were frozen during training. Each linear probe was
trained for 18 epochs. We search the hyperparameter, including learning rate
in the range from $0.05$ to $0.0001$ and learning rate decay among
$[0.1,0.2,0.5]$ with ray.tune222https://www.ray.io/ray-tune. For each task, we
adopt cross-entropy loss for classification.

Generation Setting We employ two evaluation protocols for OpenFlamingo: VQA
and Cloze Test. In both protocols, we utilize either in-context demonstrations
or explicit prompts to constrain the generated answer to be within the set of
candidate answers. If the generated answer fails to directly address the given
question, it is considered incorrect.

We also conduct tests with varying numbers of in-context shots in the range of
$[0,4,8,16,32]$. In the zero-shot setting, we retain two text samples while
excluding image tokens <image> and image samples. To solicit rationale, we
also attempt to output Chain-of-Thought(CoT) for prediction by annotating a
subset of samples with rationales, which is showcased in Supplementary
Material. For LLaMA-Adapter V2, we use two different instructions for question
answering on times and location reasoning. Details of all the prompts used can
be found in Fig. 5 to Fig. 6. For each task, we report the results averaged on
four trials with different random seeds.

###  5.2 Evaluation

Metrics For all experimental settings, we evaluate the performance using
accuracy, precision, and F1 score averaged across classes. To adapt generative
models for our tasks, we employ prompts to guide the model to generate answers
with in-context demonstrations and instructions. In order to compare the
results with the Recognition capability of discriminative VLMs, we remap the
randomly formatted outputs to the label space. Any generated answer that does
not directly address the question is considered a mismatch.

Human performance serves as an important baseline. Human common sense
knowledge about times and location reasoning varies substantially between
individuals depending on their background and education. Hence we conducted
tests with 12 participants, each assigned 60 randomly sampled images from the
test set. All participants have varied backgrounds from six countries across
four regions and do not receive any pre-training prior to the test. We report
the average accuracy as the human-level baseline.

lccc[colortbl-like] 3-6 13-14 Model Accuracy Precision F1-score
ResNet-50 80.63% 73.71% 71.89%
CLIP-ViT-B/32 89.37% 85.67% 85.04%
CLIP-ViT-B/16 88.25% 83.04% 82.95%
CLIP-ViT-L/14 90.63% 86.20% 86.14%
CLIP-ViT-L/14@336px 92.70% 89.40% 89.64%
BLIP-129M 86.51% 81.36% 80.51%
BLIP-129M-Coco 85.87% 81.16% 80.77%
BLIP-129M-Flickr 86.67% 82.42% 80.91%
BLIP-ViT-L 88.41% 84.72% 84.02%
BLIP-ViT-L-Coco 86.67% 82.45% 81.47%
BLIP-ViT-L-Flickr 87.94% 83.73% 83.23%
ViLT-Coco 80.00% 75.05% 72.72%
ViLT-Flickr30K 80.79% 75.86% 73.19%

Table 2: Results of Linear Probing performance for Times

##  6 Results

###  6.1 Recognition

Zero-shot Performance Tab. 5 shows the results of the zero-shot performance of
VLMs on Recognition${}_{\textsc{Times}}$ and
Recognition${}_{\textsc{Location}}$ tasks. We notice that CLIP variants
achieve impressive performance on both tasks and outperform other models by a
large margin, which corresponds with the findings of excellent zero-shot
performance in [25]. Amongst all variants, model performance on times
classification does not vary much from model architecture. By contrast, BLIP
performs poorly on both tasks.

We also find that among each VLM family, models with a larger visual encoder
perform slightly better than others on times classification but have a bigger
advantage on location classification. Datasets also are critical for model
performance. CLIP is trained on large-scale image-text data and manifests the
best capabilities of reasoning times and location. However, we still cannot
ground the failure of ViLT and BLIP fully since the domain shift between pre-
training datasets and WikiTiLo might be huge.

Compared to human performance, all CLIP models outperform our testees on both
tasks. Noteworthily, human performance varies individually according to their
background, with a variance of $10.41\%$ in times classification and $8.69\%$
in location classification.

Linear Probing Performance Tab. 5.2-6.1 shows the results of model performance
on Recognition in the linear probing setting. We report the models’ best
performance after the hyperparameter search. We use pre-trained ResNet-50 as a
pure-vision baseline. ResNet-50 shows non-inferior performance on the
Recognition${}_{\textsc{Times}}$ task but cannot compare to most Vision-
Language models on the Recognition${}_{\textsc{Location}}$ except ViLT.

lccc[colortbl-like] 3-6 13-14 Model Accuracy Precision F1-score
ResNet-50 54.13% 54.85% 54.45%
CLIP-ViT-B/32 79.37% 79.36% 79.11%
CLIP-ViT-B/16 84.60% 85.18% 84.88%
CLIP-ViT-L/14 90.95% 91.36% 90.92%
CLIP-ViT-L/14@336px 93.33%  93.90% 93.37%
BLIP-129M 75.08% 75.10% 75.26%
BLIP-129M-Coco 77.14% 76.77% 76.99%
BLIP-129M-Flickr 76.98% 77.01% 76.81%
BLIP-ViT-L 76.51% 77.37% 76.77 %
BLIP-ViT-L-Coco 78.10% 77.92% 78.04%
BLIP-ViT-L-Flickr 78.73% 78.64% 78.53%
ViLT-Coco 42.70% 45.61% 42.12%
ViLT-Flickr30K 45.08% 42.91% 43.01%

Table 3: Results of Linear Probing performance for
Location${}_{\text{region}}$

All models enjoy a prominent performance improvement compared to their zero-
shot baselines. CLIP models still achieve much better performance than other
VLMs. In particular, BLIP models also enjoy a large improvement in both tasks.
Thus, we deduce that both CLIP and BLIP have learned comparably informative
and discriminative features for these two tasks in pre-training already, which
profits from their pre-training data scale. By contrast, ViLT shows
unsatisfactory results and is even worse than the baseline model ResNet-50,
which might be caused by dataset distribution shifts.

We conclude that location classification is a harder and also more interesting
task than times classification. This could be explained by the fact that
location classification is a more fine-grained task, which requires the model
to distinguish more detailed visual cues at a commonsense level, like
understanding distinct geological and cultural elements.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 7: Visualization of transportation plan of word patch alignment on
times classification. Best viewed zoomed in. Rows from top to bottom: ViLT,
CLIP, and BLIP. Columns from left to right: Afghanistan(Middle East) in 2000,
Argentina(Latin America) in 1980, Japan(Eastern Asia) in 1940, and
Germany(Europe) in 1880.

Visualization Inspired by [14], we adopt a cross-modal alignment of
times/location tokens and image patches on linear probing models for
visualization, to show that visual encoders can recognize relevant
discriminative features. We select the first $25\%$ of visual patches that
have more mass transported from word tokens of labels in prompts to visual
patches to emphasize. Fig. 7 and Fig. 9 show the visualization of times and
location classification. For Times Classification, the attended patches seem
less specific. Generally, visual tokens in the background instead of
foreground objects have seemingly dominant contributions. We find that in the
human evaluation tests, our participants also tend to give predictions majorly
based on image styles instead of visual details.

Interestingly, we find that for Recognition${}_{\textsc{Location}}$, models
tend to attend to specific regions like scene texts and human clothing. CLIP
can locate the Chinese characters on the board, as shown in Fig. 99(i) and
banner texts reading “Malaysian” in Fig. 99(f). When the model performs
poorly, as shown in Fig. 99(b), attended patches are less meaningful. This
shows us that VLMs can recognize distinct and discriminative visual cues on an
image that helps reason commonsense knowledge about the location.

Figure 8: Impact of different shot numbers for OpenFlamingo. More in-context
shots do not substantially increase the performance on
Reasoning${}_{\textsc{Location}}$, but achieve a higher accuracy on
Reasoning${}_{\textsc{Times}}$.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 9: Visualization of transportation plan of word patch alignment on
location classification. Best viewed zoomed in. Rows from top to bottom: ViLT,
CLIP, and BLIP. Columns from left to right: China (Eastern Asia) in 2010,
Malaysia (South-Eastern Asia) in 1990, Tajikistan (Central Asia) in 2000,
Bangladesh (Southern Asia) in 1970.

###  6.2 Reasoning

Figure 10: Example cases of interest from OpenFlamingo VQA with Chain of
Thought, and LLaMA Adapter V2.

Performance of In-Context Learning-based Reasoning From Tab. 5, we also find
that OpenFlamingo Cloze Test only achieves frequency baseline results. This
reveals that in-context demos can teach the generative LLMs the mapping to
output label space but do not leverage any semantic information from visual
features. In the VQA protocol, OpenFlamingo has higher accuracy compared to
the Cloze Test protocol. However, enforcing the model to output Chain-of-
Thought does not improve the performance. Regarding the impact of the number
of shots in in-context learning, we show that more shots do not improve
Reasoning${}_{\textsc{Location}}$ performance substantially as in Fig.8.
Especially for Reasoning${}_{\textsc{Times}}$, we find the output prediction
is more unstable when having more in-context shots and deteriorate the
performance.

Performance of Instruction-based Reasoning LLaMA-Adapter V2 exhibits
insatisfying performance compared to other models in general. With
Instructiona, the model’s performance declines when provided with label
instructions. With Instructionb, the model achieves performance that is
comparable to OpenFlamingo VQA as well as the averaged human baselines. We
also find different instructions have an big impact on the model performance
on Reasoning.

Gap between Recognition and Reasoning We notice the performance gap between
Recognition and Reasoning on the same benchmark. Generative models also can
hardly outperform averaged human baselines, which cannot live up to the
expectations since large-scale VLMs should benefit more from a massive
knowledge base in our assumption. We hypothesize the performance gap exists in
two aspects: 1) in the open-ended question answering (QA) protocol, generative
VLMs often fail to generate a relevant answer, resulting in inaccuracies; 2)
generative VLMs fail to utilize features of the visual encoder effectively and
consequently encounter difficulties in reasoning. Both expose the shortcomings
of VLMs in understanding and reasoning about times and locations. To validate
our hypotheses, we conduct a case study on the failure cases.

Failure Cases We investigate the failure cases in the Reasoning phase of
generative VLMs. In Fig. 10, we analyze different types of failure cases for
OpenFlamingo and LLaMA-Adapter V2. By requiring OpenFlamingo to output Chain-
of-Thought and providing clear instructions to LLaMA-Adapter V2, both models
can make grounded guesses. However, failure guesses can occur in the following
scenarios: 1) when OpenFlamingo fails to generate a useful reason for
prediction, and 2) when OpenFlamingo cannot generate truthful reasoning based
on an image. We find that these failures are primarily caused by OpenFlamingo
being heavily influenced by the few-shot demos, which limits its ability to
extract relevant and useful information to a significant extent. On the other
hand, for LLaMA-Adapter V2, predictions are rejected when the model fails to
locate relevant visual cues in an image, despite humans being able to reason
effectively. This highlights that generative VLMs still struggle to fully
leverage the visual cues in images for times and location reasoning.

##  7 Conclusion

In this work, we propose a two-stage probing task Recognition and Reasoning
and introduce a new dataset WikiTiLo, to explore Vision Language Models’ times
and location reasoning ability Based on the findings of Recognition and
Reasoning tasks on the WikiTiLo dataset, we have probed the capabilities of
VLMs in times and location reasoning. Results show that large-scale visual
encoders of discriminative VLMs like CLIP have excellent performance on times
and location recognition. Our analysis reveals that discriminative VLMs can
effectively retain times and location-relevant features for recognition.
However, LLMs of generative VLMs fail to extract useful information for
reasoning. Further studies are needed to enhance the ability of generative
VLMs to effectively utilize visual features to unlock LLMs’ full reasoning
potential.

## References

  * [1] Guillaume Alain and Yoshua Bengio.  Understanding intermediate layers using linear classifier probes.  arXiv preprint arXiv:1610.01644, 2016.
  * [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  * [3] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt.  Openflamingo, Mar. 2023.
  * [4] Dominika Basaj, Witold Oleszkiewicz, Igor Sieradzki, Michał Górszczak, Barbara Rychalska, Tomasz Trzciński, and Bartosz Zieliński.  Explaining self-supervised image representations with visual probing.  Freiburg, Germany: International Joint Conferences on Artificial Intelligence, 2021.
  * [5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX, pages 104–120. Springer, 2020.
  * [7] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu.  Violet: End-to-end video-language transformers with masked visual-token modeling.  arXiv preprint arXiv:2111.12681, 2021.
  * [8] Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Vondrick, and Dan Roth.  There’sa time and place for reasoning beyond the image.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1138–1149, 2022.
  * [9] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al.  Llama-adapter v2: Parameter-efficient visual instruction model.  arXiv preprint arXiv:2304.15010, 2023.
  * [10] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] John Hewitt and Percy Liang.  Designing and interpreting probes with control tasks.  arXiv preprint arXiv:1909.03368, 2019.
  * [13] Hyo Jin Kim, Enrique Dunn, and Jan-Michael Frahm.  Predicting good features for image geo-localization using per-bundle vlad.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1170–1178, 2015.
  * [14] Wonjae Kim, Bokyung Son, and Ildoo Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In International Conference on Machine Learning, pages 5583–5594. PMLR, 2021.
  * [15] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi.  Align and prompt: Video-and-language pre-training with entity prompts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4953–4963, 2022.
  * [16] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [17] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [18] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [19] Tsung-Yi Lin, Serge Belongie, and James Hays.  Cross-view image geolocalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 891–898, 2013.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [21] Adam Dahlgren Lindström, Suna Bensch, Johanna Björklund, and Frank Drewes.  Probing multimodal embeddings for linguistic properties: the visual-semantic case.  arXiv preprint arXiv:2102.11115, 2021.
  * [22] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [23] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [24] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, pages 529–544. Springer, 2022.
  * [25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [26] Philipp J Rösch and Jindřich Libovickỳ.  Probing the role of positional information in vision-language models.  arXiv preprint arXiv:2305.10046, 2023.
  * [27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * [28] Tawfiq Salem, Scott Workman, and Nathan Jacobs.  Learning a dynamic map of visual appearance.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12435–12444, 2020.
  * [29] Emmanuelle Salin, Badreddine Farah, Stéphane Ayache, and Benoit Favre.  Are vision-language transformers learning multimodal representations? a probing perspective.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11248–11257, 2022.
  * [30] Paul Hongsuck Seo, Tobias Weyand, Jack Sim, and Bohyung Han.  Cplanet: Enhancing image geolocalization by combinatorial partitioning of maps.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 536–551, 2018.
  * [31] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [32] Xing Shi, Inkit Padhi, and Kevin Knight.  Does string-based neural mt learn source syntax?  In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1526–1534, 2016.
  * [33] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15638–15650, 2022.
  * [34] Vas Taras, Piers Steel, and Bradley L Kirkman.  Does country equate with culture? beyond geography in the search for cultural boundaries.  Management International Review, 56:455–487, 2016.
  * [35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [36] Nam N Vo and James Hays.  Localizing and orienting street views using overhead imagery.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 494–509. Springer, 2016.
  * [37] Tobias Weyand, Ilya Kostrikov, and James Philbin.  Planet-photo geolocation with convolutional neural networks.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 37–55. Springer, 2016.
  * [38] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao.  Unified contrastive learning in image-text-label space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19163–19173, 2022.
  * [39] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [40] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  arXiv preprint arXiv:2205.01917, 2022.
  * [41] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  arXiv preprint arXiv:2303.16199, 2023.

##  Appendix A List of Countries in WikiTiLo

The countries included in WikiTiLo and their regions are listed in Tab. 4.
These countries are almost evenly distributed in 7 regions defined by their
cultural and geographical affinity with reference of
UNESCO333https://population.un.org/wpp/DefinitionOfRegions/ and sorted
alphabetically.

$\mathbf{Country}$ | $\mathbf{Region}$ | $\mathbf{Country}$ | $\mathbf{Region}$
---|---|---|---
Afghanistan | Middle East | Argentina | Latin America
Australia | NA, EU and OC | Brazil | Latin America
Bangladesh | Southern Asia | China | Eastern Asia
Germany | NA, EU and OC | India | Southern Asia
Indonesia | South-Eastern Asia | Iran | Middle East
Japan | Eastern Asia | Kazakhstan | Central Asia
Kenya | Sub-Saharan Africa | Kyrgyzstan | Central Asia
Malaysia | South-Eastern Asia | Mexico | Latin America
Nigeria | Sub-Saharan Africa | North Korea | Eastern Asia
Pakistan | Middle East | Rwanda | Sub-Saharan Africa
Saudi Arabia | Middle East | South Africa | Sub-Saharan Africa
South Korea | Eastern Asia | Sri Lanka | Southern Asia
Tajikistan | Central Asia | Thailand | South-Eastern Asia
Turkmenistan | Central Asia | United States | NA, EU and OC
Uzbekistan | Central Aisa | Vietnam | South-Eastern Asia

Table 4: Countries with corresponding regions (NA, EU and OC is the
abbreviation of North America, Europa and Oceania).

##  Appendix B Data Curation

In order to guarantee that WikiTiLo comprises images that are characteristics
of socio-cultural visual hints, we conduct a manual image curation based on
image visual cues on raw images in Wikimedia Commons, as in Fig. 11. We try to
ensure that the space identity and time period features of each image can be
distinguished from the architectural patterns, costume styles, language types,
movement postures, photo colors, and quality, or other fine- grained features.

Figure 11: An example of manual image curating on to determine its time and
location. By doing this, the images of WikiTiLo are grounded by multiple scene
text, faces, objects segments from the image, and colors and resolution of the
image.

##  Appendix C Visual encoders of discriminative VLMs

We compare all the visual encoders of discriminative Vision Lanague Models and
Vision Models we used for references in the paper in the dimension of dataset,
visual encoder, textual encoder in Tab. 5.

Model | Visual Encoder | Text Encoder | Dataset | Multimodal
---|---|---|---|---
ResNet-50 | ResNet | - |  ImageNet [27] | ✘
ViLT | Patch Emb. | BERT |  MSCOCO [20], GCC [31] | ✔
CLIP-ViT | ViT | Transf. | Online data | ✔
CLIP-RN | ResNet | Transf. | Online data | ✔
BLIP | ViT | BERT |  CapFilt [17],14M[17], 129M[17] | ✔

Table 5: Comparison of disminitative VLMs. Variants of different datasets and
encoders will be denoted by suffixes.

lcccccc[colortbl-like] [l]1-1 [c]1-3Times [c]1-3Location
Model Accuracy Precision F1-score Accuracy Precision F1-score
BLIP-129M 22.06%(-8.89%) 30.33%(-16.48%) 32.15%(-13.99%) 47.94%(+1.43%)
49.28%(-7.74%) 47.40%(-1.65%)
CLIP-ViT-B/32 34.92%(-43.65%) 54.13%(-16.53%) 51.87%(-18.88%) 59.52%(-4.13%)
61.72%(-5.64%) 59.46%(-4.96%)
CLIP-ViT-B/16 38.89%(-36.82%) 51.93%(-16.01%) 51.67%(-18.41%) 70.32%(-7.3%)
69.95%(-7.28%) 69.18%(-7.68%)
CLIP-ViT-L/14 47.78%(-28.25%) 56.42%(-13.31%) 56.36%(-13.56%) 79.84%(-5.72%)
80.08%(-8.84%) 78.96%(-6.76%)
ViLT-Coco 27.14%(-43.02%) 42.70%(-15.81%) 42.37%(-15.41%) 16.03%(-0.95%)
19.18%(+1.79%) 17.21%(-0.11%)
ViLT-Flickr30k 23.33%(-28.57%) 24.97%(-23.87%) 29.91%(-19.41%) 18.10%(-1.52%)
19.61%(-0.76%) 17.82%(-1.99%)

Table 6: Zero-shot performance on the grayscale test set in Recognition. The
difference between the original test set and the grayscale test set is shown
in the bracelet. red figures show a performance drop and green figures show a
performance increase.

lcccccc[colortbl-like] [l]1-1 [c]1-3Times [c]1-3Location
Model Accuracy Precision F1-score Accuracy Precision F1-score
ResNet-50 39.05%(-41.58%) 53.49%(-20.22%) 51.82%(-20.07%) 40.16%(-13.97%)
42.64%(-12.21%) 39.93%(-14.52%)
BLIP-129M 72.70%(-13.81%) 69.57%(-11.79%) 71.08%(-9.43%) 66.53%(-8.73%)
66.78%(-8.32%) 65.35%(-9.91%)
CLIP-ViT-B/32 72.54%(-16.83%) 71.56%(-14.11%) 72.49%(-12.55%) 70.32%(-9.05%)
70.02%(-9.34%) 69.47%(-9.64%)
CLIP-ViT-B/16 81.75%(-6.50%) 76.54%(-6.50%) 77.00%(-5.95%) 70.32%(-7.93%)
76.67%(–8.85%) 75.44%(-9.44%)
CLIP-ViT-L/14 87.62%(-3.01%) 83.13%(-3.07%) 83.97%(-2.17%) 87.14%(-3.81%)
87.41%(-3.95%) 86.59%(-4.33%)
ViLT-Coco 49.21%(-30.79%) 57.68%(-17.37%) 54.77%(-17.95%) 36.98%(-5.72%)
31.81%(-13.80%) 32.16%(-9.96%)
ViLT-Flickr30k 44.44%(-36.35%) 57.20%(-18.66%) 54.59%(-18.60%) 37.78%(-7.30%)
34.79%(-8.12%) 34.24%(-8.77%)

Table 7: Linear probing performance on the grayscale test set in Recognition.
The difference between the original test set and the grayscale test set is
shown in the bracelet.

##  Appendix D Results on Grayscale Test Set

To investigate the model’s robustness to image styles, we convert all images
in the test set to grayscale and evaluate the model performance on the
grayscale test set as a control task. We find that models are not robust to
grayscale images on the Recognition${}_{\textsc{times}}$ task as the
Recognition${}_{\textsc{Location}}$ task, in particular for ResNet-50 and
ViLT, which suffer $30\%$-$50\%$ performance drop, as shown in Tab. C; CLIP
and BLIP still manifest strong robustness to color conversion. In contrast,
model performance on the location classification is less influenced by the
image conversion. This proves our assumption that the times classification
task is more based on the image style instead of more local evidence in images
for ResNet and ViLT and the times and location ability of CLIP and BLIP is
further proven.

##  Appendix E Prompts used for generative VLMs on
reasoning${}_{\textsc{Times}}$

We also list all the prompts used for OpenFlamingo and LLaMA-Adapter V2 used
for reasoning${}_{\textsc{Times}}$ in Fig. 12 as in the paper for references.

OpenFlamingo Cloze Test <image> Output: This is a historical photo taken in
the 19th Century. <|endofchunk|>
Open Flamingo VQA
We divide time into 4 eras. These 4 eras are in the 19th Century, between 1900
and 1950, between 1950 and 2000, in the 21st Century. <image> Question: When
was this photo taken? Short answer: in the 21st Century <|endofchunk|>
OpenFlamingo VQA - CoT We divide time into 4 eras. These 4 eras are in the
19th Century, between 1900 and 1950, between 1950 and 2000, in the 21st
Century. <image> Question: When was this photo taken? Answer: Because the
people in this photograph are dressed in attire typical of the Qing Dynasty in
China. Therefore, it can be inferred that this photograph was taken during the
Qing Dynasty,this photo was taken in the 19th Century.<|endofchunk|>

(a)

LLaMA-Adapter V2 Instructiona Instruction: This photograph was taken during
one of the following 4 periods. We divide these 4 periods as in the 19th
Century, between 1900 and 1950, between 1950 and 2000, in the 21st Century. In
which period was this photo taken? LLaMA-Adapter V2 Instructionb Instruction:
In which period was this photo taken?

(b)

Figure 12: We list respectively the prompt templates we used in OpenFlamingo
for each protocol for Reasoning for times in (a), instructions for LLaMA-
Adapter V2 for times in (b).

##  Appendix F Rationale examples for Chain-of-Thought

Figure 13: Rationale examples we used as few shot samples in protocol
OpenFlamingo VQA with CoT.

We annotate a subset of images with rationale in Reasoning tasks for
OpenFlamingo Chain-of-Thought. Here we showcase some examples of images and
the rationale associated. We attempt to include visual details that are
relevant for reasoning about times and locations for humans.