# Distributed Attention for Grounded Image Captioning

Nenglun Chen  The University of Hong Kong Youtu Lab, Tencent ,  Xingjia Pan
Youtu Lab, Tencent ,  Runnan Chen  The University of Hong Kong ,  Lei Yang
The University of Hong Kong ,  Zhiwen Lin  Youtu Lab, Tencent ,  Yuqiang Ren
Youtu Lab, Tencent ,  Haolei Yuan  Youtu Lab, Tencent ,  Xiaowei Guo  Youtu
Lab, Tencent ,  Feiyue Huang  Youtu Lab, Tencent and  Wenping Wang  The
University of Hong Kong

(2021)

###### Abstract.

We study the problem of weakly supervised grounded image captioning. That is,
given an image, the goal is to automatically generate a sentence describing
the context of the image with each noun word grounded to the corresponding
region in the image. This task is challenging due to the lack of explicit
fine-grained region word alignments as supervision. Previous weakly supervised
methods mainly explore various kinds of regularization schemes to improve
attention accuracy. However, their performances are still far from the fully
supervised ones. One main issue that has been ignored is that the attention
for generating visually groundable words may only focus on the most
discriminate parts and can not cover the whole object. To this end, we propose
a simple yet effective method to alleviate the issue, termed as partial
grounding problem in our paper. Specifically, we design a distributed
attention mechanism to enforce the network to aggregate information from
multiple spatially different regions with consistent semantics while
generating the words. Therefore, the union of the focused region proposals
should form a visual region that encloses the object of interest completely.
Extensive experiments have demonstrated the superiority of our proposed method
compared with the state-of-the-arts.

Image captioning; Visual grounding; Weakly supervised learning; Neural
networks

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
29th ACM International Conference on Multimedia; October 20–24, 2021; Virtual
Event, China††booktitle: Proceedings of the 29th ACM International Conference
on Multimedia (MM ’21), October 20–24, 2021, Virtual Event, China††price:
15.00††doi: 10.1145/3474085.3475354††isbn: 978-1-4503-8651-7/21/10††ccs:
Computing methodologies Natural language generation††ccs: Computing
methodologies Object detection

##  1\. Introduction

Describing a given scene is considered an essential ability for humans to
perceive the world. It is thus important to develop intelligent agents with
such an ability, formally known as image captioning that generates the natural
language descriptions based on the given images (Kulkarni et al., 2013;
Vinyals et al., 2015; Cornia et al., 2020; Guo et al., 2020). A further
pursuit in the image captioning task has emerged: not only the captions should
semantically reflect the content of the image, but also the words in the
generated captions should be grounded in the image, i.e., finding visual
regions in the image that corresponds to the words. This way, more
interpretable captions are expected to be generated.

Figure 1. The partial grounding problem. The captions and the corresponding
attention were generated with the baseline method. The attended region of the
word ’woman’ only covers the ’face’.

To this end, the mainstream works leverage attention mechanisms that focus on
certain regions of images to generate more grounded captions. Pioneering works
(Liu et al., 2017; Yu et al., 2017; Zhou et al., 2019) have explored using
annotated bounding box as attention supervision for each visually groundable
word or noun phrase in the training stage, and led to desirable improvement on
the interpretability of the generated captions. However, acquiring such
region-word alignment annotations is expensive and time-consuming.

Recently, several attempts have been made to generate more grounded attention
with weak supervision (Zhou et al., 2020; Liu et al., 2020a; Ma et al., 2020).
These methods mainly focus on designing various regularization schemes to
produce attended regions to enclose the visual entities corresponding to the
words in the generated captions. Despite the promising results achieved by the
weakly supervised methods, their performances are still far from the fully
supervised baselines. One main issue is that the attention for generating the
visually groundable word without ground-truth grounding supervision may only
focus on the most discriminant regions, failing to cover the entire object.
See Fig. 1 for example, the grounded region of the word ’woman’ generated by
our baseline network only covers the facial part. While such grounding region
identify the correct person (‘the woman in blue shirt’), the network largely
ignores the body parts presented in the image, which means that it fails to
visually distinguish the word ‘woman’ and the word ‘face’. This problem is
important, especially when the input images are not accessible to the users
(e.g. voice assist system for blind person). In such cases, whether grounding
on partial or the whole body may affect some follow-up decisions.

Such a problem, termed as the partial grounding problem in our paper, is
usually seen in the grounded image captioning task with weak supervision. This
is primarily attributed to the fact that the objects of interest in the images
are usually partially covered by many redundant region proposals, and choosing
the most salient proposal inevitably leads to the described problem. In Fig.
2, we statistically analyze the grounding error of the baseline architecture
on Flickr 30k Entities (Plummer et al., 2015) dataset by casting it into three
categories including partial grounding, enlarged grounding, and deviated
grounding. As we can see that, the partial grounding problem contributes to a
large portion of the grounding error. This further confirms the necessity to
solve the partial grounding problem.

(a) Statistics (b) Enlarged Grounding

(c) Partial Grounding

(d) Deviated Grounding

Figure 2. Error analysis of the baseline method.

Based on the observation, we propose a simple yet effective method to
alleviate the partial grounding problem. The ingredient of our method is a
distributed attention mechanism that enforces the network to attend to
multiple semantically consistent regions. In this way, the union of the
attended regions should form a visual region that encloses the object
completely. We follow the previous method (Zhou et al., 2019) and build our
baseline network based on the widely used up-down architecture (Anderson et
al., 2018). The attention module in the baseline network is augmented with
multiple branches, and region proposal elimination is used to enforce
different attention branches to focus on different regions. We conduct various
experiments to verify the effectiveness of our method. From our experiments,
we illustrate that the partial grounding problem is a crucial issue that
produces inferior grounding results, and adding our distributed attention to
the baseline method can effectively achieve a large performance gain on
grounding accuracy than those produced by the state-of-the-art methods relying
on sophisticated network designs.

In summary, our technical contributions are summarized:

  * •

We find that alleviating the partial grounding issue is critical to promote
grounding performance, which is ignored in all previous methods.

  * •

We propose a novel solution to the partial grounding issue by introducing a
distributed attention mechanism that enforces the network to aggregate the
semantically consistent regions across the entire image.

  * •

By testing our method on the Flickr30k Entities dataset and ActivityNet
Entities dataset, we achieve significant gain on grounding accuracy compared
with state-of-the-art methods.

##  2\. Related Work

#### Grounded Image Captioning

Image captioning is an important task at the juncture of computer vision and
natural language processing. Traditional methods (Das et al., 2013; Kulkarni
et al., 2013; Mitchell et al., 2012; Zhang et al., 2020a) mainly adopts the
pre-defined templates with slots that are filled with visual cues detected on
the images. The performance of these methods is limited. With the advent of
deep neural networks, the caption quality has been greatly improved. State-of-
the-art methods (Yang et al., 2019; You et al., 2016; Lu et al., 2017;
Anderson et al., 2018; Huang et al., 2019; Guo et al., 2020; Cornia et al.,
2020) make use of various form of the attention mechanism for attending to
different regions of the images while generating captions. The attention
mechanism can be applied to various types of data including spatial grids (Lu
et al., 2017), semantic metadata (Yang et al., 2019; You et al., 2016) as well
as object region proposals (Anderson et al., 2018; Huang et al., 2019).
Despite the great success achieved in terms of captioning performance of these
methods, the grounding accuracy is still not satisfactory.

Several attempts have been made for improving the grounding accuracy. Some
pioneering works (Liu et al., 2017; Yu et al., 2017; Zhou et al., 2019; Zhang
et al., 2020a) obtained improved grounding results with the help of the
attention mechanism in a fully supervised manner. However, labeling the word-
region correspondences is time-consuming and difficult to scale up to large
datasets. Recently, researchers started to use different kinds of
regularization schemes to improve the grounding accuracy in a weakly
supervised manner. Zhou et al. (Zhou et al., 2020) adopted the teacher-student
scheme and distilled the knowledge learned by a Part-of-Speech enhanced image-
text matching network (Lee et al., 2018) to improve the grounding accuracy of
the image captioning model. Ma et al. (Ma et al., 2020) employed the cyclical
training regimen as a regularization, in which a generator and a localizer are
trained jointly to regularize the attention. Liu et al. (Liu et al., 2020a)
proposed the _Prophet Attention_ , which uses future information to compute
the ideal attention and then takes the ideal attention as regularization to
alleviate the deviated attention. The abovementioned methods utilized
different regularization schemes to improve the grounding accuracy, however,
they all neglect the partial grounding issue as described previously. To this
end, we propose a distributed attention mechanism for aggregating semantically
consistent regions to alleviate the partial grounding problem.

#### Visual Grounding

The visual grounding task aims at learning fine-grained correspondences
between image regions and visually groundable noun phrases. Existing works can
be roughly divided into two categories: supervised and weakly supervised. The
supervised methods (Plummer et al., 2018; Liu et al., 2020b; Dogan et al.,
2019; Yu et al., 2018) took bounding boxes as supervision to enforce the
alignment between image regions and noun phrases, and have achieved remarkable
success. However, annotating the region word alignments is expensive and time-
consuming, which makes it difficult to scale to large datasets.

Weakly supervised methods (Liu et al., 2019, 2021; Rohrbach et al., 2016; Chen
et al., 2018; Gupta et al., 2020; Wang et al., 2020) aimed at learning the
correspondence with only image-caption pairs. Some studies utilize
reconstruction regularization (Rohrbach et al., 2016; Liu et al., 2019) to
learn visual grounding. Rohrbach et al. (Rohrbach et al., 2016) first generate
a candidate bounding box by the attention mechanism and then reconstruct the
phrase based on the selected bounding box. Liu et al. (Liu et al., 2019)
further boost the grounding accuracy by introducing the contextual entity and
model the relationship between the target entity (subject) and contextual
entity (object). Contrastive learning is also exploited in several method
(Wang et al., 2020; Gupta et al., 2020). Wang et al. (Wang et al., 2020) learn
a score function between the region-phrase pairs for distilling knowledge from
a generic object detector for weakly supervised grounding. Gupta et al. (Gupta
et al., 2020) maximize a lower bound on mutual information between sets of the
region features extracted from an image and contextualized word
representations. Chen et al. (Chen et al., 2018) facilitate weakly supervised
grounding by considering both visual and language consistency and leveraging
complementary knowledge from the feature extractor. Our task is different from
visual grounding in that our caption is to be generated rather than given for
grounding.

#### Weakly Supervised Object Detection

WSOD aims to learn the localization of objects with only image-level labels
(e.g., ones given for classification) and can be roughly divided into methods
based on region proposals or class activation maps (CAMs) (Zhou et al., 2016).
Proposal-based approaches (Song et al., 2014; Gokberk Cinbis et al., 2014;
Bilen et al., 2015) formulate this task as multiple instance learning. Bilen
et al. (Bilen and Vedaldi, 2016) select proposals by parallel detection and
classification branches in deep convolutional networks. Contextual information
(Kantorov et al., 2016), attention mechanism (Teh et al., 2016), gradient map
(Shen et al., 2019) and semantic segmentation (Wei et al., 2018) are leveraged
to learn accurate object proposals. CAM-based methods (Singh and Lee, 2017,
2017; Zhang et al., 2018a, 2020b; Pan et al., 2021) produce localization maps
by aggregating deep feature maps using a class-specific fully connected layer.
Despite the simplicity and effectiveness of CAM-based methods, they suffer
from identifying small discriminative parts of objects. To improve the
activation of CAMs, several methods (Singh and Lee, 2017; Yun et al., 2019;
Choe and Shim, 2019; Zhang et al., 2018a) adopted adversarial erasing on input
images or feature maps to drive localization models focusing on extended
object parts. SPG (Zhang et al., 2018b) and I2C (Zhang et al., 2020b)
increased the quality of localization maps by introducing the constraint of
pixel-level correlations into the network. SPA (Pan et al., 2021) and TS-CAM
(Gao et al., 2021) obtain accurate localization maps with the help of long-
range structural information. Different from the WSOD task, our task leverages
the correctly attended regions for caption generation.

##  3\. Methodology

(a)  (b)

Figure 3. The pipeline of our proposed method (a), and the detailed framework
of the baseline network (b). The attention branches in the distributed
attention module are applied iteratively to discover semantically consistent
regions. Note that, we do not show all the region proposals for a clear
illustration.

In this section, we first give the notations of image captioning (Sec 3.1). We
then briefly introduce the attention based encoder-decoder baseline network as
our image captioning baseline (Sec 3.2). In Sec 3.3, we describe in detail the
proposed distributed attention mechanism for aggregating semantically
consistent partial regions for more grounded image captioning.

###  3.1. Notations

The generic image captioning task aims at generating a sentence $Y$ to
describe the context of a given image $I$. We follow the pioneering works
(Zhou et al., 2019; Ma et al., 2020) on weakly supervised image captioning and
present the input image as a set of region proposals
$R=\\{r_{1},r_{2},...,r_{N}\\}(r_{i}\in\mathbb{R}^{d})$ and the image feature
map $f_{c}$, where $N$ and $d$ are the number and the feature dimension of the
region proposals, respectively. We generate $r_{i}$ with Faster RCNN (Ren et
al., 2015) while $f_{c}$ is extracted from a pretrained ResNet (He et al.,
2016) model. The generated sentence is represented as a sequence of one-hot
vectors $Y=\\{y_{1},y_{2},...,y_{T}\\}(y_{t}\in\mathbb{R}^{s})$, where $T$ is
the length of the sequence and $s$ is the size of the vocabulary.

###  3.2. Revisit the Baseline Method

Our method is built on GVD (Zhou et al., 2019) except for the self-attention
loss for region feature embedding. Fig. 3(b) shows the core part of the GVD,
named language generating module, which is an extension of the widely used
Bottom-up and top-down attention network (Anderson et al., 2018) for the image
captioning task. Specifically, the language generating model consists of two
LSTM layers, the attention LSTM layer, and the language LSTM layer. The
attention LSTM is used to identify the importance of different regions in the
input image for generating the next word. It takes as input the global image
feature vector $v_{g}$, the previous word embedding vector
$e_{t-1}=W_{e}y_{t-1}$ and the previous hidden state $h_{t-1}^{1}$, and encode
them into current hidden state $h_{t}^{1}$:

(1) $h_{t}^{1}=LSTM_{1}([v_{g};e_{t-1}],h_{t-1}^{1}),$

where $W_{e}$ is the learnable word embedding matrix, $[\cdot;\cdot]$ denotes
the concatenation operation and we obtain $v_{g}$ by applying global average
pooling on the image feature map $f_{c}$. Note that, the description of the
cell states for both LSTM layers is omitted for notational clarity.

To obtain the attention weight $\alpha_{t}$ over the image regions, the hidden
state $h_{t}^{1}$ produced by the above-mentioned attention LSTM layer is
further fed into an attention module followed by a softmax layer as:

(2) $\displaystyle z_{i,t}$ $\displaystyle=W_{a}tanh(W_{r}r_{i}\oplus
W_{h}h_{t}^{1}),$ $\displaystyle\alpha_{t}$ $\displaystyle=softmax(z_{t}),$

where $W_{a}$, $W_{r}$ and $W_{h}$ are the learnable weight matrices and
$\oplus$ is the element-wise addition operation. Given the attention weight
$\alpha_{t}$, the attended region feature $\hat{r}_{t}$ can be calculated as
the weighted combination of the regions features:

(3) $\hat{r}_{t}=\alpha_{t}^{\top}R$

where $R\in\mathbb{R}^{N\times d}$ is a feature matrix whose column represents
the feature of a region proposal. Following (Zhou et al., 2019; Ma et al.,
2020), we also enhance the attended region feature $\hat{r}_{t}$ with the
image features $f_{c}$ to get the attended image feature $\hat{f}_{t}$:

(4) $\displaystyle\hat{f}_{t}=\hat{r}_{t}+att_{img}(f_{c})$

where $att_{img}$ is the attention block for aggregating the image feature map
$f_{c}$ into a feature vector.

Finally, the attended image feature $\hat{f}_{t}$ is fed into the language
LSTM for producing the conditional probability distribution
$p(y_{t}|y_{1:t-1})$ of the next word $y_{t}$ over all possible outputs as:

(5) $\displaystyle h_{t}^{2}=LSTM_{2}([h_{t}^{1},\hat{f}_{t}],h_{t-1}^{2}),$
$\displaystyle p(y_{t}|y_{1:t-1})=softmax(W_{o}h_{t}^{2}),$

where $y_{1:t-1}$ denotes the sequence of the previously predicted words,
$p(y_{t}|y_{1:t-1})$ denotes the conditional probability of the word $y_{t}$
given the previous words, $h^{2}_{t}$ is the hidden state of the language LSTM
and $W_{o}$ is the learnable weight.

In the training phase, the network is optimized by adopting the teacher
forcing strategy given the ground truth caption sequence
$Y^{*}=\\{y_{1}^{*},y_{2}^{*},...,y_{T}^{*}\\}$. The training objective is to
minimize the following cross-entropy loss $L_{CE}(\theta)$:

(6) $\displaystyle
L_{CE}(\theta)=-\sum_{t=1}^{T}{log(p_{\theta}(y^{*}_{t}|y^{*}_{1:t-1}))}$

where $\theta$ denotes the trainable parameters. The region with the maximum
attention weight will be selected as the grounding result.

Figure 4. The extracted region proposals (white bounding boxes) are redundant
and the objects are partially covered by many region proposals.

###  3.3. Region Aggregation with Distributed Attention

In this section, we introduce our distributed attention mechanism for
aggregating semantically consistent regions to resolve the partial grounding
problem in detail.

As shown in Fig. 4, the extracted region proposals by a pre-trained Faster-
RCNN are redundant with each object partially covered by many region
proposals. It is likely that the attention mechanism used in the baseline
network will only identify parts of the objects which are the most
discriminative.

Based on the above discussion, we propose the distributed attention mechanism
for aggregating semantically consistent regions for generating the words. Fig.
3(a) illustrates the overall pipeline of our proposed method. Specifically, to
make the attention focus on different regions, we augment the attention module
in the baseline network with $K$ attention branches, and enforce the focused
regions of all the attention branches to be mutually exclusive by iteratively
applying the attention branches on the region proposals:

(7) $\displaystyle a_{t}^{k}$
$\displaystyle=Attention^{k}(\hat{R}^{k},h^{1}_{t-1}),$
$\displaystyle\hat{R}^{k}$ $\displaystyle=\\{r_{i}\in R,r_{i}\notin
M^{k-1}\\}$

where $Attention^{k}()$ is the function for computing the attention weights
with $k$-th attention branch over the region proposals as described in Equ. 2,
$k\in[1,K]$ is the branch index, $\hat{R}^{k}$ denotes the set of region
proposals used in the attention branch $k$, $a_{t}^{k}$ is the attention
weights predicted by the $k$-th attention branch and $M^{k-1}$ denotes $k-1$
region proposals selected by previous $k-1$ attention branches. Note that
these attention branches do not share the same weights. The attended features
of all $K$ attention branches are further passed through language LSTM (Equ.
3, 4, 5) to get $K$ outputs.

At the training stage, we use the teacher-forcing strategy and apply the
cross-entropy loss to all the $K$ outputs. During testing, at each time step,
we select the word predicted by the most number of attention branches and
combine the corresponding regions together into a single region and use it as
the grounding output.

Through our method, the distributed attention model will learn to aggregate
semantically consistent regions when generating the word to alleviate the
partial grounding problem. The union of the attended regions should form a
visual region that encloses the object of interest completely.

##  4\. Experiments

###  4.1. Experimental Settings

Datasets. We conducted the main experiments on the widely used
Flickr30k-Entities dataset (Plummer et al., 2015). It contains $31k$ images in
total, and each image is annotated with $5$ sentences. In addition, it also
contains $275k$ bounding box annotations and each bounding box corresponds to
a visually groundable noun phrase in the caption. Following GVD (Zhou et al.,
2019), we used the data split setting from Karpathy et al. (Karpathy and Fei-
Fei, 2015), which has $29k$ images for training, $1k$ images for validation,
and $1k$ for testing. The vocabulary size of the dataset is $8639$.

Evaluation Metrics. We used the standard captioning evaluation toolkit (Chen
et al., 2015) to measure the captioning quality. Four commonly used language
evaluation metrics are reported, i.e., BLEU (Papineni et al., 2002), METEOR
(Denkowski and Lavie, 2014), CIDEr (Vedantam et al., 2015) and SPICE (Anderson
et al., 2016).

To evaluate the grounding performance, we used the metrics $F1_{all}$ and
$F1_{loc}$ defined in GVD (Zhou et al., 2019). For $F1_{all}$, a prediction is
considered as correct if the object word is correctly generated and the
Intersection-over-Union (IOU) of the predicted bounding box and the ground
truth bounding box is larger than $0.5$. In $F1_{loc}$, only the correctly
generated words are considered. Both metrics were averaged over classes.

Implementation Details. We implemented our method with PyTorch, and all the
experiments were conducted on two V100 GPUs. Following the previous works
(Zhou et al., 2019; Ma et al., 2020; Liu et al., 2020a), we adopted the widely
used Faster R-CNN network (Ren et al., 2015) pre-trained on Visual Genomes
(Krishna et al., 2017) by GVD (Zhou et al., 2019) to extract $100$ region
proposals for each image. The word embedding dimension in the captions was set
to $512$, and the word embedding layer was trained from scratch. The dimension
of the hidden states for both Attention and Language LSTM was set to $1024$.

The network was optimized with Adam Optimizer (Kingma and Ba, 2014), with an
initial learning rate set to $5e^{-4}$ and decayed by a factor of $0.8$ for
every three epochs. The batch size was set to $64$. We trained the network
with a single attention branch for $20$ epoches and included the other
branches in the training loop afterward. The whole training process takes less
than one day.

Figure 5. Results of the generated captions and the grounding regions by
comparing our method with the baseline.

###  4.2. Comparison with Prior Work

Quantitative Comparison. We compared our method with state-of-the-art weakly
supervised grounded image captioning methods, including GVD (Zhou et al.,
2019), Prophet (Liu et al., 2020a), Zhou et al. (Zhou et al., 2020) and
Cyclical (Ma et al., 2020), to see the effectiveness of our method on the
grounding accuracy. Note that, in Zhou et al. (Zhou et al., 2020), various
techniques are used to improve the captioning performance including SCST
(Rennie et al., 2017) and scheduled sampling (Bengio et al., 2015), while we
only employ the standard cross-entropy loss used in the baseline network
during training.

As shown in Table. 1, by applying our proposed distributed attention module on
the baseline method, we achieved significant improvement on the grounding
accuracy ($F1_{all}$ and $F1_{loc}$) compared with other weakly supervised
grounded image captioning methods. By comparing our method with the supervised
baseline method in Cyclical, the grounding performance of our method only
inferiors a little. It further confirmed the necessity of solving the partial
grounding problem in this task.

Table 1. Comparison with other state-of-the-art methods on the Flickr30k
Entities test set.

Method | Captioning Eval | Grounding Eval
---|---|---
| B@1 | B@4 | M | C | S | $F1_{all}$ | $F1_{loc}$
Supervised methods |  |  |  |  |  |
Cyclical(Ma et al., 2020) | 69.0 | 26.8 | 22.4 | 61.1 | 16.8 | 8.44 | 22.78
GVD(Zhou et al., 2019) | 69.9 | 27.3 | 22.5 | 62.3 | 16.5 | 7.55 | 22.2
Weakly supervised methods |  |  |  |  |  |
GVD(Zhou et al., 2019) | 69.2 | 26.9 | 22.1 | 60.1 | 16.1 | 3.88 | 11.70
Prophet(Liu et al., 2020a) | - | 27.2 | 22.3 | 60.8 | 16.3 | 5.45 | 15.30
Zhou et al.(Zhou et al., 2020) | 71.4 | 28.0 | 22.6 | 66.2 | 17.0 | 6.53 | 15.79
Cyclical(Ma et al., 2020) | 69.9 | 27.4 | 22.3 | 61.4 | 16.6 | 4.98 | 13.53
Ours | 69.2 | 27.2 | 22.5 | 62.5 | 16.5 | 7.91 | 21.54

Qualitative Comparison. We qualitatively compare our method with the baseline
method to see how the partial grounding problem can be alleviated. As shown in
Fig. 5, the predicted grounding regions of our method are more accurate and
can cover the whole objects, while the selected regions of the baseline method
may only cover the most salient parts.

Performance on ActivityNet-Entities. We further tested the performance of our
method on the ActivityNet-Entities dataset (Zhou et al., 2019) to see if our
method works or not on the grounded video description task. By extending GVD
with our distributed attention module, we achieved significant improvement on
the validation set for both $F1_{all}$(from $3.70$ to $5.32$) and
$F1_{loc}$(from $12.70$ to $21.06$) with the captioning performance keeping
comparable. This further confirmed the effectiveness of our proposed method.

###  4.3. Qualitative Results.

Example Results. In Fig. 6, we present some example results including the
generated captions as well as the corresponding grounding results. As we can
see that, our method is able to generate fine grained image captions with
accurate groundings for both foreground objects (e.g., ’man’) and the
background (e.g., ’beach’).

Visualization of the Distributed Attention. We visualize the distributed
attention in Fig. 7 to see how the region proposals are selected and fused.
When generating a word, the distributed attention will attend to multiple
semantically consistent regions with different locations, and the attended
region proposals with the same semantic meanings will be fused to get the
grounding results. We can see that our distributed attention has the ability
to aggregate semantically consistent regions to enclose the object of interest
completely and thus alleviate the partial grounding problem.

Figure 6. Visualization of our generated captions as well as the corresponding
grounded regions.

###  4.4. Ablation Study

Effect of $K$. We tested the performance of our method with a varied number of
attention branches $K$. As shown in Table 2, the grounding performance of our
method outperformed the state-of-the-art methods by a large margin when the
number of attention branches $K$ is larger than $3$. When $K$ is smaller than
$3$, the grounding accuracy is degraded. The main reason is that, due to the
redundant region proposals extracted with Faster-RCNN, the distributed
attention module is not able to discover enough regions when $K$ is too small.

Effect of Region Proposal Elimination. As shown in Table 3 we tested the
performance of our method without region proposal elimination. Our method can
also improve the grounding accuracy without region proposal elimination.
That’s mainly because, different initialization of the attention branches will
lead to slightly different attention results, and the partial grounding
problem can also be alleviated. The use of the region proposal elimination is
to explicitly enforce the distributed attention module to focus on different
semantically consistent regions to further boost the grounding performance.

Relationship with Multi-head Attention. Our method has a close relationship
with multi-head attention proposed in (Vaswani et al., 2017). Here, we
compared our distributed attention with multi-head attention by replacing the
attention module in our baseline network with multi-head attention and tested
it by varying the number of attention heads from $2$ to $6$. The best
performing result is achieved with the number of attention head setting to
$4$, that is $4.75$ for $F1_{all}$ and $13.19$ for $F1_{loc}$, which is lower
than our result as shown in Table 1. One possible reason is that, in our work,
we explicitly enforce the attention branches to focus on multiple regions with
consistent semantics, which is not guaranteed for multi-head attention.

Table 2. Effect of the number of attention branches $K$. K | Captioning Eval | Grounding Eval
---|---|---
| B@1 | B@4 | M | C | S | $F1_{all}$ | $F1_{loc}$
2 | 68.8 | 26.5 | 22.4 | 60.9 | 16.6 | 5.2 | 15.90
3 | 69.3 | 27.0 | 22.4 | 61.6 | 16.7 | 6.78 | 20.10
4 | 69.2 | 27.2 | 22.5 | 62.5 | 16.5 | 7.91 | 21.54
5 | 68.9 | 26.6 | 22.4 | 62.4 | 16.7 | 7.62 | 20.45
Table 3. Results without region proposal elimination. K | Captioning Eval | Grounding Eval
---|---|---
| B@1 | B@4 | M | C | S | $F1_{all}$ | $F1_{loc}$
2 | 68.9 | 27.0 | 22.4 | 61.1 | 16.5 | 5.48 | 15.80
3 | 68.6 | 27.1 | 22.51 | 61.85 | 16.41 | 5.48 | 16.14
4 | 68.9 | 26.8 | 22.4 | 61.5 | 16.6 | 6.82 | 18.91
5 | 68.9 | 26.8 | 22.3 | 61.4 | 16.3 | 6.27 | 17.36

###  4.5. Error Analysis

In this section, to better understand the grounding performance, we analyzed
the grounding accuracy detailly by classifying all the predictions on
Flickr30k Entities test set into five categories:

Figure 7. Visualization of the distributed attention. For a noun word (red) in
a generated caption, the corresponding attention weights and the selected
regions are shown.

Mis Cls: Missing Class computes the ratio of noun words that are in the ground
truth reference captions but not in the corresponding predicted captions.

Hallu Cls: Hallucinated Class computes the ratio of noun words that are in the
predicted captions but not in the corresponding ground truth reference
captions.

Corr Grd: Correct Grounding computes the ratio of noun words that are
correctly generated and grounded ($IOU>0.5$).

Part Grd: Partial Grounding computes the ratio of noun words that are
correctly generated but partially grounded (the predicted bounding box is
contained in the ground truth bounding boxes, but the $IOU$ of them is less
than $0.5$).

Ohter Err: Other Error Grounding computes the ratio of noun words that are
correctly generated but are neither in Part Grd nor in Corr Grd.

Table 4. Detailed analysis of the grounding accuracy by classifying all the predictions on Flickr30k Entities test set into five types. The ratio of each type is computed. | Err Cls | Corr Cls
---|---|---
| Mis Cls | Hallu Cls | Corr Grd | Part Grd | Ohter Err
Baseline | 33.26% | 44.92% | 7.12% | 7.94% | 6.76%
Ours | 32.76% | 45.32% | 10.92% | 2.03% | 8.97%

As shown in Table 4, the classification error (missing class and hallucinated
class) contributes to the most portion of the error when computing the
grounding accuracy. One main reason is that, for a given image, there exist
various ways for describing its content, and directly comparing the generated
captions with the corresponding references may lead to errors in computing the
classification accuracy. That’s why the metric $F1_{all}$, which considers the
classification accuracy, is always much lower than $F1_{loc}$. And it’s also
the limitation of current evaluation metrics in this task.

When only considering the predictions whose classification labels are correct,
our proposed method achieved much more correctly grounded predictions than
that of the baseline method. It further confirmed the importance of solving
the partial grounding issue and the superiority of our proposed method.

We also measure the grounding accuracy of the baseline method if regarding all
the predictions which are partially grounded as correct predictions to see the
upper bound. In this case, the $F1_{all}$ and $F1_{loc}$ are $10.65$ and
$30.28$ correspondingly. By comparing it with our results ($F1_{all}:7.91$,
$F1_{loc}:21.5$), we can see that, there is still a large space for
improvement.

###  4.6. Limitations

The main limitation of our proposed method is that the predicted grounding
regions might be larger than the target objects as shown in Fig. 8. One
important reason might be that the attention weights are calculated based on
contextual information from the previous words rather than the one to be
generated. Thus, the attention weights predicted by some attention branches
might have the ’deviated focus’ problem, and fusing the deviated regions
together may lead to enlarged grounding regions. One possible solution is to
combine our work with other state-of-the-arts (e.g. (Liu et al., 2020a; Ma et
al., 2020)). As it is not the main focus of our work, we leave it as future
work.

Figure 8. Limitation of our method.

##  5\. Conclusion

In this paper, we study the problem of generating image captions with accurate
grounding regions without any grounding annotations. We unveil the fact that
alleviating the partial grounding issue is critical to improve the grounding
performance, which has been overlooked in many previous works. To this end, we
propose the distributed attention mechanism to enforce the network to
aggregate different regions with consistent semantics while generating the
words. Extensive experiments have shown the effectiveness of our proposed
method. By incorporating our proposed module into the baseline network, we
achieved significant improvement on the grounding accuracy compared with the
state-of-the-arts.

## References

  * (1)
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  Spice: Semantic propositional image caption evaluation. In _European conference on computer vision_. Springer, 382–398.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6077–6086.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  _arXiv preprint arXiv:1506.03099_ (2015).
  * Bilen et al. (2015) Hakan Bilen, Marco Pedersoli, and Tinne Tuytelaars. 2015\.  Weakly supervised object detection with convex clustering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1081–1089.
  * Bilen and Vedaldi (2016) Hakan Bilen and Andrea Vedaldi. 2016.  Weakly supervised deep detection networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2846–2854.
  * Chen et al. (2018) Kan Chen, Jiyang Gao, and Ram Nevatia. 2018.  Knowledge aided consistency for weakly supervised phrase grounding. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4042–4050.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ (2015).
  * Choe and Shim (2019) Junsuk Choe and Hyunjung Shim. 2019.  Attention-based dropout layer for weakly supervised object localization. In _IEEE CVPR_. 2219–2228.
  * Cornia et al. (2020) Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020.  Meshed-memory transformer for image captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10578–10587.
  * Das et al. (2013) Pradipto Das, Chenliang Xu, Richard F Doell, and Jason J Corso. 2013.  A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2634–2641.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language. In _Proceedings of the ninth workshop on statistical machine translation_. 376–380.
  * Dogan et al. (2019) Pelin Dogan, Leonid Sigal, and Markus Gross. 2019\.  Neural sequential phrase grounding (seqground). In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4175–4184.
  * Gao et al. (2021) Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, and Qixiang Ye. 2021\.  TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.  _arXiv preprint arXiv:2103.14862_ (2021).
  * Gokberk Cinbis et al. (2014) Ramazan Gokberk Cinbis, Jakob Verbeek, and Cordelia Schmid. 2014\.  Multi-fold mil training for weakly supervised object localization. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2409–2416.
  * Guo et al. (2020) Longteng Guo, Jing Liu, Xinxin Zhu, Peng Yao, Shichen Lu, and Hanqing Lu. 2020\.  Normalized and Geometry-Aware Self-Attention Network for Image Captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10327–10336.
  * Gupta et al. (2020) Tanmay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. 2020\.  Contrastive learning for weakly supervised phrase grounding.  _arXiv preprint arXiv:2006.09920_ (2020).
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Huang et al. (2019) Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. 2019\.  Attention on attention for image captioning. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 4634–4643.
  * Kantorov et al. (2016) Vadim Kantorov, Maxime Oquab, Minsu Cho, and Ivan Laptev. 2016\.  Contextlocnet: Context-aware deep network models for weakly supervised localization. In _European Conference on Computer Vision_. Springer, 350–365.
  * Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015.  Deep visual-semantic alignments for generating image descriptions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3128–3137.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ 123, 1 (2017), 32–73.
  * Kulkarni et al. (2013) Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. 2013\.  Babytalk: Understanding and generating simple image descriptions.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 35, 12 (2013), 2891–2903.
  * Lee et al. (2018) Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018.  Stacked cross attention for image-text matching. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 201–216.
  * Liu et al. (2017) Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. 2017\.  Attention correctness in neural image captioning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 31.
  * Liu et al. (2020a) Fenglin Liu, Xuancheng Ren, Xian Wu, Shen Ge, Wei Fan, Yuexian Zou, and Xu Sun. 2020a.  Prophet Attention: Predicting Attention with Future Attention.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Liu et al. (2019) Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Li Su, and Qingming Huang. 2019\.  Knowledge-guided pairwise reconstruction network for weakly supervised referring expression grounding. In _Proceedings of the 27th ACM International Conference on Multimedia_. 539–547.
  * Liu et al. (2021) Yongfei Liu, Bo Wan, Lin Ma, and Xuming He. 2021\.  Relation-aware Instance Refinement for Weakly Supervised Visual Grounding.  _arXiv preprint arXiv:2103.12989_ (2021).
  * Liu et al. (2020b) Yongfei Liu, Bo Wan, Xiaodan Zhu, and Xuming He. 2020b.  Learning cross-modal context graph for visual grounding. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 11645–11652.
  * Lu et al. (2017) Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017\.  Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 375–383.
  * Ma et al. (2020) Chih-Yao Ma, Yannis Kalantidis, Ghassan AlRegib, Peter Vajda, Marcus Rohrbach, and Zsolt Kira. 2020.  Learning to generate grounded visual captions without localization supervision. In _Proceedings of the European Conference on Computer Vision (ECCV)_ , Vol. 2. Springer.
  * Mitchell et al. (2012) Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alexander Berg, Tamara Berg, and Hal Daumé III. 2012\.  Midge: Generating image descriptions from computer vision detections. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_. 747–756.
  * Pan et al. (2021) Xingjia Pan, Yingguo Gao, Zhiwen Lin, Fan Tang, Weiming Dong, Haolei Yuan, Feiyue Huang, and Changsheng Xu. 2021\.  Unveiling the Potential of Structure-Preserving for Weakly Supervised Object Localization.  _arXiv preprint arXiv:2103.04523_ (2021).
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. 311–318.
  * Plummer et al. (2018) Bryan A Plummer, Paige Kordas, M Hadi Kiapour, Shuai Zheng, Robinson Piramuthu, and Svetlana Lazebnik. 2018.  Conditional image-text embedding networks. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 249–264.
  * Plummer et al. (2015) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In _Proceedings of the IEEE international conference on computer vision_. 2641–2649.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _arXiv preprint arXiv:1506.01497_ (2015).
  * Rennie et al. (2017) Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017.  Self-critical sequence training for image captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 7008–7024.
  * Rohrbach et al. (2016) Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. 2016.  Grounding of textual phrases in images by reconstruction. In _European Conference on Computer Vision_. Springer, 817–834.
  * Shen et al. (2019) Yunhang Shen, Rongrong Ji, Kuiyuan Yang, Cheng Deng, and Changhu Wang. 2019.  Category-aware spatial constraint for weakly supervised detection.  _IEEE Transactions on Image Processing_ 29 (2019), 843–858.
  * Singh and Lee (2017) Krishna Kumar Singh and Yong Jae Lee. 2017.  Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In _IEEE ICCV_. 3544–3553.
  * Song et al. (2014) Hyun Oh Song, Ross Girshick, Stefanie Jegelka, Julien Mairal, Zaid Harchaoui, and Trevor Darrell. 2014.  On learning to localize objects with minimal supervision. In _International Conference on Machine Learning_. PMLR, 1611–1619.
  * Teh et al. (2016) Eu Wern Teh, Mrigank Rochan, and Yang Wang. 2016.  Attention Networks for Weakly Supervised Object Localization.. In _BMVC_. 1–11.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ (2017).
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015\.  Cider: Consensus-based image description evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4566–4575.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3156–3164.
  * Wang et al. (2020) Liwei Wang, Jing Huang, Yin Li, Kun Xu, Zhengyuan Yang, and Dong Yu. 2020\.  Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation.  _arXiv preprint arXiv:2007.01951_ (2020).
  * Wei et al. (2018) Yunchao Wei, Zhiqiang Shen, Bowen Cheng, Honghui Shi, Jinjun Xiong, Jiashi Feng, and Thomas Huang. 2018.  Ts2c: Tight box mining with surrounding segmentation context for weakly supervised object detection. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 434–450.
  * Yang et al. (2019) Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019\.  Auto-encoding scene graphs for image captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10685–10694.
  * You et al. (2016) Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016.  Image captioning with semantic attention. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4651–4659.
  * Yu et al. (2018) Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018.  Mattnet: Modular attention network for referring expression comprehension. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1307–1315.
  * Yu et al. (2017) Youngjae Yu, Jongwook Choi, Yeonhwa Kim, Kyung Yoo, Sang-Hun Lee, and Gunhee Kim. 2017\.  Supervising neural attention models for video captioning by human gaze data. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 490–498.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019\.  Cutmix: Regularization strategy to train strong classifiers with localizable features. In _Proceedings of the IEEE International Conference on Computer Vision_. 6023–6032.
  * Zhang et al. (2020a) Wenqiao Zhang, Xin Eric Wang, Siliang Tang, Haizhou Shi, Haochen Shi, Jun Xiao, Yueting Zhuang, and William Yang Wang. 2020a.  Relational Graph Learning for Grounded Video Description Generation. In _Proceedings of the 28th ACM International Conference on Multimedia_. 3807–3828.
  * Zhang et al. (2018a) Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, and Thomas S Huang. 2018a.  Adversarial complementary learning for weakly supervised object localization. In _IEEE CVPR_. 1325–1334.
  * Zhang et al. (2018b) Xiaolin Zhang, Yunchao Wei, Guoliang Kang, Yi Yang, and Thomas Huang. 2018b.  Self-produced guidance for weakly-supervised object localization. In _ECCV_. 597–613.
  * Zhang et al. (2020b) Xiaolin Zhang, Yunchao Wei, and Yi Yang. 2020b.  Inter-image communication for weakly supervised localization. In _ECCV_ , Vol. 12364. 271–287.
  * Zhou et al. (2016) Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016.  Learning deep features for discriminative localization. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2921–2929.
  * Zhou et al. (2019) Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. 2019.  Grounded video description. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6578–6587.
  * Zhou et al. (2020) Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. 2020.  More grounded image captioning by distilling image-text matching model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4777–4786.


11institutetext: University of Bordeaux, Bordeaux, Nouvelle-Aquitaine, France
11email<EMAIL_ADDRESS>
# Mean Opinion Score as a New Metric for User-Evaluation of XAI
Methods††thanks: Supported by organization Laboratoire Bordelais de Recherche
en Informatique.

Hyeon Yu  Jenny Benois-Pineau 11 0000-0003-0659-8894 Romain Bourqui 11
0000-0002-1847-2589 Romain Giot 11 0000-0002-0638-7504 Alexey Zhukov 11
0009-0009-0518-5474

###### Abstract

This paper investigates the use of Mean Opinion Score (MOS), a common image
quality metric, as a user-centric evaluation metric for XAI post-hoc
explainers. To measure the MOS, a user experiment is proposed, which has been
conducted with explanation maps of intentionally distorted images. Three
methods from the family of feature attribution methods – Gradient-weighted
Class Activation Mapping (Grad-CAM), Multi-Layered Feature Explanation Method
(MLFEM), and Feature Explanation Method (FEM) – are compared with this metric.
Additionally, the correlation of this new user-centric metric with automatic
metrics is studied via Spearman’s rank correlation coefficient. MOS of MLFEM
shows the highest correlation with automatic metrics of Insertion Area Under
Curve (IAUC) and Deletion Area Under Curve (DAUC). However, the overall
correlations are limited, which highlights the lack of consensus between
automatic and user-centric metrics.

###### Keywords:

Explainable AI Quality Metric User-Centric.

##  1 Introduction

Artificial intelligence (AI) systems are rapidly becoming ubiquitous,
influencing decisions in diverse domains ranging from medicine to finance.
These models function as “black-box” due to their inherent complexity, making
it difficult to explain their reasoning and potentially perpetuating biases in
the decision-making process. This growing reliance, however, has exposed a
critical gap in transparency and interpretability.

Regulations such as the General Data Protection Regulation (GDPR) [2], and the
need to address algorithmic bias and fairness highlight the importance of
understanding how these complex systems, often deep learning models, arrive at
their conclusions [10]. The Defense Advanced Research Projects Agency (DARPA)
[11] recognizes the critical role of eXplainable AI (XAI) in the success of
deep learning. Users may reject models they don’t comprehend, leading to
decreased system adoption and potential ethical concerns [6], [7].

While various explanation methods for image classification have emerged (e.g.
[22], [5], [21]), their effectiveness in fostering human understanding remains
uncertain. The contributions of this work are the following:

  * •

We propose the use of Mean Opinion Score (MOS), which has been widely used in
image quality assessment [3], as a new metric for post-hoc XAI evaluation
involving user experience;

  * •

We design the protocol of its collection for a “hard” case of distorted images
and well- and poorly classified situations with a trained model;

  * •

We apply this metric to three explainers from the feature attribution family,
one of which being popular Grad-CAM [22] and two are statistical explainers
proposed in previous works;

  * •

Finally, we study the correlation of this metric with automatic metrics
“Insertion Area Under Curve” (IAUC) and “Deletion Area Under Curve” (DAUC)
[20].

The remainder of the paper is organized as follows. In section 2 related work
is reviewed. In section 3 the design of the assessment experiment is proposed.
Section 4 is devoted to statistical analysis of experimental data, while in
section 5 we compare both the methods in terms of the MOS metric and present
the results of the study of the correlation of MOS with IAUC and DAUC. Section
6 concludes our work and outlines its perspectives.

##  2 Related Work

Nowadays, there is no consensus on the evaluation metrics for explanation
methods. We propose to devise them into i)automatic or no-reference metrics,
ii) semi-user-centered or reference-based metrics, and iii) fully user-
centered metrics.

####  2.0.1 No-Reference Metrics.

No-reference metrics do not depend on human judgments [25]. Examples of these
metrics are Insertion Area Under the Curve (IAUC) and Deletion Area Under
Curve (DAUC) [20]. DAUC evaluates explanation maps by progressively masking
image regions based on their importance supplied by the explanation method
(highest score pixels masked first). It calculates the area under the curve of
the image’s classification score as the masked proportion increases. A lower
DAUC score indicates a better explanation because masking supposedly important
regions should significantly reduce the initial score. IAUC works similarly,
but in the opposite direction: it measures how revealing important areas
increases the score.

####  2.0.2 Reference-Based Metrics.

Reference-based metrics serve for evaluation of a XAI method by human
interpretations or comparisons with human interest expressed a priori by Gaze
Fixation Density Maps (GFDMs) [5]. Such metrics are Pearson correlation
coefficient (PCC) and similarity (SIM) measured between two maps. In the paper
[25], the authors evaluate the explanation methods Grad-CAM [22], MLFEM [5],
and FEM [4] using both reference-based and no-reference metrics. We qualify
“reference-based metrics” from [25] as semi-automatic, as they do not require
the assessment of explanation maps by the user himself, but a comparison of
the explanation maps with the importance of pixels in the input image for the
user.

####  2.0.3 Fully User-Centered Metrics.

On the contrary, [15] focuses on assessing explanation methods through user
perceptions of the accuracy of AI classifications. This approach is therefore
fully user-centered. The authors conduct user studies focused on both item-
and system-level metrics to evaluate the performance and acceptance of AI
classifications. Item-level measures focus on individual classifications.
Participants judge the correctness and reasonableness of each classification
on a 5-point scale, using their understanding of these terms. These ratings
provide a detailed view of how participants perceive the accuracy and logic
behind each classification. System-level measures evaluate users’ overall
trust and satisfaction with the entire AI system. These metrics align with
approaches used in the DARPA XAI program [11], where user experience with
explanations is assessed to gauge trust and satisfaction in the system itself.
Standardized questionnaires are employed for this purpose.

####  2.0.4 Mean Opinion Score.

The Recommendation of International Telecommunication Union Telecommunication
Standardization Sector (ITU-T) P.800.2 [1] provides a detailed discussion on
the differential perception of audio and video quality by humans, highlighting
the absence of a singular definition for a ’good’ quality. To evaluate user
experience with new audio or video systems, subjective experiments are
conducted in which participants rate the performance of these systems. These
experiments employ rating scales to convert user opinions into numerical
scores, opinion scores (OS), which can then be averaged to produce a Mean
Opinion Score (MOS) per content item. The most widely utilized scale is the
5-point Absolute Category Rating (ACR) scale, where participants independently
rate each sample without any reference. Alternatively, participants may be
asked to compare a processed sample to an original one. Furthermore, the
5-point ACR scale serves as the primary quality metric in the single- and
double-stimulus subjective assessment of television picture quality [3]. ACR
is typically the same scale as the famous Likert scale in its 5-point version
[17], which is employed for user studies. Hence, in the following design of
our experiment for MOS computation, we will employ the Likert scale.

##  3 MOS as a Quality Metric for XAI Methods

Drawing inspiration from the established Mean Opinion Score (MOS) metric used
in image quality assessment [3], we explore the potential of applying MOS to
evaluate the quality of explanation methods in XAI.

####  3.0.1 Mean Opinion Score (MOS).

MOS provides a standardized way to quantify perceived quality through
subjective human judgments. By adapting this approach to the realm of
explanation methods, we aim to develop a reliable metric that involves user
feedback to assess the effectiveness and clarity of explanations generated by
different methods.

MOS in the realm of explanation methods, refers to the mean rating
participants provided for the explanation map given by the explanation method
for each image. For all participants $j$, $j\in\\{1,2,3,...,N\\}$, MOS of
image $I_{i}$ and explanation method $k$ can be expressed as:

$MOS_{{I_{i}}{k}}=\frac{1}{N}\sum_{j=1}^{N}o_{ijk}$ (1)

where $o_{ijk}$ is the opinion score of participant $j$ for image $I_{i}$ and
explanation method $k$.

The opinion score $o$ ranges from 1 to 5 with 5 corresponding to “excellent”.

To obtain the MOS of the explanation maps of explanation methods of images, we
have to design a user-centric experiment, which we present in the following.

####  3.0.2 Design of Psycho-Visual Experiment to Measure MOS of Explanation
Maps of XAI Methods.

This user study departs from traditional approaches that focus on usefulness
and satisfaction for users [9]. Instead, it adopts a methodology akin to
visual quality assessment, commonly employed for evaluating images and videos
[3]. The goal of the psycho-visual experiment is to evaluate the quality of
explanation maps generated by different explanation methods. Participants are
thus tasked with rating the explanation maps associated with distorted images,
all of which were classified by the same Deep Neural Network (DNN) model. We
have taken the ResNet50 classifier as in [25]. The experiment consisted of
sessions, conducted either online or offline. Participants chose their
preferred format at the outset and remained in that format for the entire
experiment. For online sessions, screen sharing was utilized for participant
supervision. The offline sessions were conducted under supervision in the
presence of the coordinator.

####  3.0.3 Data.

From the whole set of distorted images from [25], which we used only two
levels of distortions, we qualify as “weak” and “strong” (see example in Fig.
2). The types of distortions were: additive Gaussian noise, Gaussian blur, and
uniform random brightness shift. This subset selection aimed to manage the
experiment’s duration. In addition, only distorted images were used since we
are specifically interested in the performance of the explanation methods on
perturbed data. These images were pushed through the trained model and
explanation maps obtained by XAI methods were submitted to user evaluation. In
total, 300 distorted images were used, and three different XAI methods were
applied to each image to obtain the explanation maps, resulting in a total of
900 maps to evaluate.

(a) Weak distortion

(b) Strong distortion

Figure 2: Weakly and strongly distorted images of SALICON dataset: additive
Gaussian noise (left), Gaussian blur (center), and uniform random brightness
shift (right)

####  3.0.4 Explanation Methods.

Among explainable AI methods for image classification, Gradient-weighted Class
Activation Mapping (Grad-CAM) [22] is the most widely used approach. Recently,
Feature-wise Explanation Maps (FEM) and Multi-layer Feature-wise Explanation
Maps (MLFEM) were proposed as alternative techniques, demonstrating superior
performance to Grad-CAM [5]. These methods fall under the category of Feature
Attribution methods. Specifically, they assign scores to individual features
or pixels within the input image to show their contribution into the
classification decision. These scores represent the relative importance of
each feature in influencing the model’s final prediction. While Grad-CAM
relies on gradients to evaluate feature significance, FEM employs a k-sigma
rule on the activation values of the last convolutional layer. MLFEM, on the
other hand, extends this analysis to all convolutional layers within the
model.

####  3.0.5 Experimental Protocol.

The ITU-R BT.500-15 recommendation [3] for single-stimulus image quality
assessment was followed. Indeed, the users had to evaluate the quality of only
one image - the explanation map overlaid on the original image as a heat map.
The recommendation ITU-R BT.500-15 considers visual fatigue and memory
limitations, advising a 10-second exposure time per image. To allow the
participants to take in the information on the screen and the time to click on
the map quality, the exposure time was extended to 20 seconds per image pair.
Between each map screen, a 5-second inter-stimulus interval (ISI) with a mid-
gray screen was used. ISI minimizes carry-over effects, allowing participants
to reset their visual attention and reduce fatigue. Each session presented
participants with 225 images. The duration of sessions was designed to avoid
visual fatigue of the subjects. Thus, the time spent for one session per
subject was approximately 1 hour 20 minutes if the subjects worked without
interruption. There was also an option to interrupt the session and restart it
with unseen images. The overall participation time per subject was about 5
hours and 20 min. The experiment screen is depicted in Fig. 3.

Figure 3: Evaluation screen. From left to right: distorted image, importance
heat map, ground truth class labels, model prediction result, and Likert scale
for explanation quality.

####  3.0.6 Evaluation Rules.

As illustrated in Fig. 3, adapting the commonly used ITU-R scale [3], the
experiment employed a 5-point Likert scale with options: Excellent, Good,
Fair, Poor, and Bad.

An explanation is considered good if the explanation map and the
classification are consistent: the classification corresponds to the image’s
content highlighted by the XAI method. Another case of a good map is when the
classification is incorrect (poorly classified) and the heat map highlights
pixels outside the ground truth objects.

An explanation is considered bad if the explanation map and the classification
are inconsistent: the classification is correct (well-classified), but the
relevant pixels are not highlighted or the classification is incorrect (poorly
classified) but the pixels corresponding to the true class are highlighted.

####  3.0.7 Participants Recruitment.

Participants have to have acute visual attention and a quick processing speed
to evaluate explanation maps. These abilities decline with age [8], [23]. To
align with the young adult group from the mentioned papers, participants aged
18 to 29 were recruited for the experiment. To ensure participants could
accurately assess the heat map of the explanation maps, which rely on color
differentiation, all underwent an Ishihara test [13] for red-green color
deficiencies. Only those who passed proceeded to the image evaluation task.

The experiment involved 31 participants. Fifteen participated in person
(average age: 25.14 $\pm$ 2.16 y.o.). Sixteen participated online (average
age: 25.31 $\pm$ 1.53 y.o.). The online participants were in various
locations: France (2), Netherlands (2), Hungary (1), Spain (2), Canada (7),
and India (2).

##  4 Statistical Analysis of Experiment Data

This section presents the statistical analysis of the user experiment, with
the goals outlined as follows:

  * •

To filter out the most discrepant subjects from the experimental data;

  * •

To test the homogeneity of the two groups: online and offline;

  * •

To evaluate a set of XAI methods with the proposed MOS metric;

  * •

To study the correlation of our fully user-centered metric with automatic
metrics IAUC and DAUC.

The results of the statistical analysis are presented in Section 5.

###  4.1 Filtering of Outliers

We define outliers as subjects whose average opinion scores for all the
explanation methods deviate significantly from the average scores of the
entire group for that explanation method. Filtering outliers helps to ensure
that the evaluation reflects the opinions of a more representative sample and
reduces the influence of potentially biased or inattentive participants.
Subjects (participants) are excluded if their average opinion score of all the
explanation methods falls outside two standard deviations of the average
opinion score of the method for the subjects’ group.

The filtering process is conducted separately for each group: i) offline
$g^{\circ}_{offline}$ and ii) online $g^{\circ}_{online}$. As detailed in
Section 3.0.2, the offline group consists of participants who joined the study
in person, and in the online group, subjects participated in online sessions.

For each subject within a group, we iterate through each explanation method
and perform the following steps to identify outliers:

  1. 1.

Calculate individual’s statistics: For a subject $s^{\circ}_{j}$ in the group
$g^{\circ}$, the average opinion score for the current explanation methods
($e_{k}$) over $N_{I}$ images is,

$\bar{o}(g^{\circ},s^{\circ}_{j},e_{k})=\frac{1}{N_{I}}\sum_{i=1}^{N_{I}}o_{ijk}$
(2)

  2. 2.

Calculate group statistics: The mean and the standard deviation of the average
opinion scores $\bar{o}(g^{\circ},s^{\circ}_{j},e_{k})$ for the current
explanation method are calculated across all subjects of the group.

  3. 3.

Identify outliers: If the subject’s average opinion score falls outside the
two standard deviations of the mean of the average opinion scores, the subject
is identified as an outlier for that particular method and thus removed from
the group.

Subjects identified as outliers are removed from further analysis. Groups with
removed outliers will be denoted as $g_{offline}$ and $g_{online}$.

###  4.2 Test of Homogeneity of MOS for Two Groups

The objective of this test is to evaluate the feasibility of merging two
groups of subjects: the “online” and “offline” participants. Pearson’s chi-
square test of homogeneity [19] was performed. If the results indicate
homogeneity between these groups, we can compute the Mean Opinion Score (MOS)
from the combined opinion scores of all participants for a given image.

While the test itself can handle unequal group sizes, the accuracy and
reliability of the results can be impacted. To mitigate this concern, the
number of subjects in each group was set to the minimum number $N$ of
participants in either group ($N=min(|g_{offline}|,|g_{online}|)$). $N$
subjects were randomly selected for the group with a larger sample size
($|g|>N$).

Furthermore, as the chi-squared test requires categorical data, the continuous
MOS values were quantized by a uniform quantizer. For a given image $I_{i}$ a
uniform quantizer with a quantization step size, $\triangle$ (in our case,
0.0001) for $MOS_{{I_{i}}{k}}$ can be expressed as:

$MOSQ_{{I_{i}}{e_{k}}}=\triangle\lfloor\frac{MOS_{{I_{i}}{k}}}{\triangle}+\frac{1}{2}\rfloor$
(3)

###  4.3 Analysis of the Performance of Explanation Methods

To evaluate the performance of Grad-CAM, MLFEM, and FEM methods with the new
metric MOS, a non-parametric statistical analysis was employed on the mean
opinion scores (MOS) of explanation maps generated by these methods. The
Kruskal-Wallis H test [16] was utilized to assess the overall existence of
statistically significant differences in MOS ratings across the three methods.
Subsequently, pairwise comparisons were conducted using the Mann-Whitney U
test [18] to pinpoint specific differences in methods.

The post hoc pairwise Mann-Whitney U test was done with unadjusted p-values
rather than employing multiple comparison corrections like Bonferroni.
Although common, such corrections can be overly conservative in this context.
Since the Kruskal-Wallis test already establishes an overall difference
between explanation methods, unadjusted p-values in the pairwise tests allow
for a more detailed exploration of these variations. This approach helps
identify specific explanation method pairs with significant MOS discrepancies.
The analysis result can be seen in Section 5.

###  4.4 Study of correlation of MOS with Automatic Metrics

It is necessary to position our new user-centric metric with regard to purely
automatic metrics, such as IAUC and DAUC. Xu-Darme et al. [24] have shown that
semi-automatic metrics, which compare explanation maps with gaze fixation
density maps did not correlate with DAUC. Hence, our study also examines the
correlation between our proposed user-centric metric, MOS, and the automatic
metrics IAUC and DAUC. For this, Spearman’s rank correlation coefficient is
computed between MOS and IAUC and DAUC for the same set of images and across
the three XAI methods. The results are detailed in Section 5.0.5.

##  5 Results and Discussion

This section presents the results of the statistical analysis of MOS conducted
to evaluate the performance of the explanation methods (Grad-CAM [22], MLFEM
[5], and FEM [4]) accordingly to the methodology presented in Section 4. Then
we present and discuss the results of the study of positioning of our new
user-involvement metric with regard to automatic metrics IAUC and DAUC.

####  5.0.1 Outlier Filtering

The results of outlier filtering from the participants are illustrated in Fig.
4. It represents the box-plots of each subject’s average opinion score of each
explanation method Subject 37 from the offline group and subject 24 from the
online group have been considered as outliers, denoted by red ellipses.

Figure 4: Box Plot of MOS of Individuals in Each Group. The outliers’ box-
plots have been identified by the red circles. The first row shows the offline
group and the second row shows the online group. From left-to-right: Grad-CAM,
MLFEM, FEM.

####  5.0.2 Homogeneity of Offline and Online Group

By applying Pearson’s chi-squared test of homogeneity [19], with a $\chi^{2}$
value of 1.24 and a p-value of 0.74, we can infer that the observed
distribution of MOS scores between the offline and online groups with removed
outliers ($g_{offline}$ and $g_{online}$) does not significantly differ from
what would be expected by chance. Therefore, it is statistically justifiable
to merge the data from both groups for further analysis. The following results
are presented for the merged groups, totaling 29 participants.

####  5.0.3 Tests of Statistically Significance of Differences of All
Explanation Methods with MOS Metrics.

The result of the Kruskal-Wallis H test is shown in Tab. 5.0.3. We have
considered the performances of the explanations on well-classified images and
poorly classified images from, each distortion together and separately,
independently of distortion level. The bold p-values in the table indicate
significant differences in MOS ratings between explanation methods.

This experiment confirms that our newly introduced user-centric metric MOS
differentiates between the explanation methods. Note that these methods were
already shown to give different results when evaluated with other metrics in
[25] and [24]. We illustrate MOS values together with the standard deviation
of opinion scores in Fig. 5 for all 900 maps we have tested our methods on.
Nevertheless, for the distortion such as Gaussian blur on well-classified
images and uniform random brightness shift on poorly classified images, there
is no significant difference. Therefore, we exclude them from further
analysis. It is interesting to note that the MOS values of explanation methods
are globally lower for poorly classified images, while the standard deviation
of each image is bigger. This can suggest disagreements in rating among the
participants.

[HTML]C0C0C0  Classification Distortion | All |  | Additive Gaussian
---
Noise
| Gaussian
---
Blur
| Uniform Random
---
Brightness Shift
Poorly | 0.0012 | 0.0125 | 0.0004 | 0.4111
Well | 0.0006 | 0.0148 | 0.2899 | 0.0412

Table 1: The p-value of the Kruskal-Wallis H test for testing the differences
of the XAI methods with MOS metric.

(a) MOS of poorly classified images. Blue: Grad-CAM, green: FEM, orange:
MLFEM.

(b) MOS of well-classified images. Blue: Grad-CAM, green: FEM, orange: MLFEM.

Figure 5: MOS and the standard deviation of opinion scores of poorly
classified and well-classified images. X-axis: map number, Y-axis: MOS.

####  5.0.4 Pair-wise Comparison of XAI Methods with MOS metric.

The results of the pairwise Mann-Whitney U test are presented in Tab. 2. The
bold p-value indicates that there exists a statistically significant
difference between the methods.

Here once more, we distinguish between well-classified and poorly classified
images and perform computations for all images, see column three, and for each
considered distortion, see columns 4,5,6 of Tab. 2.

The post-hoc Mann-Whitney U test indicates significant differences in MOS
between Grad-CAM and MLFEM (p = 0.0007) and between Grad-CAM and FEM (p =
0.0041) for poorly classified images. Here, the average MOS of Grad-CAM is
$2.0940\pm 0.3685$. For MLFEM, it is $1.9284\pm 0.2635$ and for FEM,
$1.9611\pm 0.2627$. Therefore, for poorly classified images, Grad-CAM is a
better explainer than FEM and MLFEM.

For well-classified images, the post-hoc test revealed significant differences
between MLFEM and both Grad-CAM (p = 0.0008) and FEM (p = 0.0009). MLFEM
obtained a statistically higher mean MOS ($3.2423\pm 0.9088$) compared to
Grad-CAM ($2.9190\pm 0.9368$) and FEM ($2.9360\pm 0.9171$). Hence, in the case
of well-classified images, MLFEM is the best explainer according to the metric
MOS.

More detailed performances of explanation methods on different distortions can
be seen in Tab. 3. For poorly classified images, Grad-CAM exhibits higher mean
MOS, achieving $2.1304\pm 0.4061$ for images with additive Gaussian noise and
$2.2714\pm 0.3689$ for images with Gaussian blur, which further supports the
Grad-CAM’s superior performance on poorly classified images. In addition, the
superiority of MLFEM on well-classified images is demonstrated, as it
outperforms Grad-CAM and FEM on well-classified images with additive Gaussian
noise ($3.2471\pm 0.8902$) and uniform random brightness shift ($3.3508\pm
0.8353$).

[HTML]C0C0C0 Classification |  | Explanation
---
Methods
All |  | Additive Gaussian
---
Noise
| Gaussian
---
Blur
| Uniform Random
---
Brightness Shift
Poorly Classified |  | Grad-CAM/
---
MLFEM
0.0007 | 0.1677 | 0.0003 | N/A
| Grad-CAM/
---
FEM
0.0041 | 0.0042 | 0.0476 | N/A
| MLFEM/
---
FEM
0.4954 | 0.0837 | 0.0160 | N/A
Well Classified |  | Grad-CAM/
---
MLFEM
0.0008 | 0.0068 | N/A | 0.0340
| Grad-CAM/
---
FEM
0.9174 | 0.4664 | N/A | 0.8683
| MLFEM/
---
FEM
0.0009 | 0.0294 | N/A | 0.0255
Table 2: The p-value of the Mann-Whitney U test. If the Kruskal-Wallis H test suggested no significant difference between the explanation methods, the post-hoc Mann-Whitney U test was not conducted hence the “N/A” [HTML]C0C0C0  Distortion + Classification Explanation Methods | Grad-CAM | MLFEM | FEM
---|---|---|---
Combined Distortion + Poor | $\mathbf{2.0940\pm 0.3685}$ | $1.9284\pm 0.2635$ | $1.9611\pm 0.2627$
Combined Distortion + Well | $2.9190\pm 0.9368$ | $\mathbf{3.2423\pm 0.9088}$ | $2.9360\pm 0.9171$
Additive Gaussian Noise + Poor | $\mathbf{2.1304\pm 0.4061}$ | $1.9910\pm 0.2665$ | $1.9108\pm 0.2452$
Additive Gaussian Noise + Well | $2.7688\pm 0.8902$ | $\mathbf{3.2471\pm 0.8902}$ | $2.8870\pm 0.8670$
Gaussian Blur + Poor | $\mathbf{2.2714\pm 0.3689}$ | $1.8681\pm 0.2490$ | $2.0510\pm 0.2490$
Gaussian Blur + Well | $2.9848\pm 0.9706$ | $3.1657\pm 0.9367$ | $2.9521\pm 0.9672$
Uni. Rand. Bright. Shift + Poor | $1.9741\pm 0.2775$ | $1.8973\pm 0.2551$ | $1.9662\pm 0.2834$
Uni. Rand. Bright. Shift + Well | $2.9775\pm 0.9151$ | $\mathbf{3.3508\pm 0.8353}$ | $2.9629\pm 0.8890$
Table 3: Mean MOS of explanation methods for each distortion type and
classification category. The bold values indicate the best-performing
explanation method based on the mean of MOS. Rows without bold values indicate
no statistical significance.

####  5.0.5 Correlation Between MOS and Automatic Metrics.

The Spearman’s rank correlation coefficient (SRCC) values between MOS and
automatic metrics IAUC and DAUC can be seen in Tab. 4. For IAUC and DAUC, 1%
of the pixels were inserted or deleted at each deletion or insertion step. Row
1 contains SRCC on the whole set of maps (900) and we state that there exists
a positive correlation between MOS and IAUC and DAUC. The row 2, 3, and 4
contain the SRCC values per method. The highest values are highlighted in
bold. MLFEM’s MOS shows the highest correlation with both metrics. However,
the correlations are not very strong, especially for DAUC, which aligns with
the findings in [24] that indicate no significant correlation between
automatic and user-involved metrics. Users tend to trust more on metrics that
incorporate their input than automatic metrics. This lack of consensus
underscores that identifying the best evaluation metrics for explanation
methods remains an open question.

[HTML]C0C0C0 | DAUC | IAUC
---|---|---
Overall | 0.4406 (0.0) | 0.6078 (0.0)
Grad-CAM | 0.3765 (0.0) | 0.5448 (0.0)
MLFEM | 0.4979 (0.0) | 0.6843 (0.0)
FEM | 0.4541 (0.0) | 0.5836 (0.0)
Table 4: Spearman’s rank correlation coefficient between explanation methods
and DAUC and IAUC, with p-values in parentheses. The coefficients are rounded
to four decimal places, and the p-values are rounded to three decimal places.
The highest correlation is shown in bold.

##  6 Conclusion and Future Work

In this work, we proposed the use of the Mean Opinion Score (MOS), a metric
commonly used in image quality assessment, as a user-centric metric for
evaluating the quality of explanation maps of XAI methods. We compared MOS
with “automatic metrics” IAUC and DAUC. MOS involves collecting subjective
assessments from users, capturing their opinions on the explanations’ clarity,
usefulness, and satisfaction. Additionally, MOS is quick to compute and does
not require users to answer extensive or complicated questionnaires.

Based on the preliminary work [25] and the work of Ford et al. [9], we
developed a methodology and conducted a psycho-visual experiment with 31
subjects, including both online and offline participants. We evaluated three
feature attribution methods: Grad-CAM [22], MLFEM [5], and FEM [4]. These
methods were applied to intentionally distorted images (with additive Gaussian
noise, Gaussian blur, and uniform random brightness shift) classified using
ResNet50 [12] on a subsample of the SALICON dataset [14].

Our statistical analysis allowed us to filter out subjects with the most
discrepant opinions. It then revealed that both offline and online
participants exhibited similar behavior, allowing us to combine their
responses for the MOS computation after filtering out the outliers. The
comparison of MOS indicates the possibility of differentiating the
performances of explanation methods. Therefore, we conclude that it can be
used as the evaluation metric for XAI methods.

Building on previous work [5] which demonstrates the MLFEM’s effectiveness
compared to Grad-CAM and FEM, our findings further reveal MLFEM’s superior
ability to explain well-classified images in terms of MOS. Interestingly,
Grad-CAM consistently performed well for poorly classified images.

Additionally, Spearman’s rank correlation coefficient showed that MOS measured
for the MLFEM explainer had the highest correlation with automatic metrics
IAUC and DAUC. Overall, the positive correlation between this user-centric
metric and automatic metrics remains limited, as it was also shown in [24] for
other user-centered semi-automatic metrics involving Gaze Fixation Density
Maps. This lack of consensus highlights that finding the evaluation metrics
that users will trust for explanation methods remains an open question that
requires further investigation.

## References

  * [1] Mean opinion score interpretation and reporting. Tech. Rep. P.800.2, ITU-T Recommendation P.800.2, Geneva, Switzerland (jul 2016), https://handle.itu.int/11.1002/1000/12973-en
  * [2] Regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (general data protection regulation) (119), 1–88 (May 2016), http://data.europa.eu/eli/reg/2016/679/oj
  * [3] Methodologies for the subjective assessment of the quality of television images. Tech. rep., ITU-R Recommendation BT.500-15, Geneva, Switzerland (May 2023), https://www.itu.int/rec/R-REC-BT.500/en
  * [4] Ahmed Asif Fuad, K., Martin, P.E., Giot, R., Bourqui, R., Benois-Pineau, J., Zemmari, A.: Features understanding in 3d cnns for actions recognition in video. In: 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA). pp. 1–6 (2020). https://doi.org/10.1109/IPTA50016.2020.9286629
  * [5] Bourroux, L., Benois-Pineau, J., Bourqui, R., Giot, R.: Multi Layered Feature Explanation Method for Convolutional Neural Networks. In: International Conference on Pattern Recognition and Artificial Intelligence (ICPRAI). Paris, France (Jun 2022). https://doi.org/10.1007/978-3-031-09037-0_49
  * [6] Burton, J.W., Stein, M.K., Jensen, T.B.: A systematic review of algorithm aversion in augmented decision making. Journal of Behavioral Decision Making 33(2), 220–239 (2020). https://doi.org/10.1002/bdm.2155
  * [7] Dietvorst, B.J., Simmons, J.P., Massey, C.: Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General 144(1), 114–126 (2015). https://doi.org/10.1037/xge0000033
  * [8] Ebaid, D., Crewther, S.G.: Visual information processing in young and older adults. Frontiers in Aging Neuroscience 11 (May 2019). https://doi.org/10.3389/fnagi.2019.00116
  * [9] Ford, C., Keane, M.T.: Explaining classifications to non-experts: An xai user study of post-hoc explanations for a classifier when people lack expertise. In: Rousseau, J.J., Kapralos, B. (eds.) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. pp. 246–260. Springer Nature Switzerland, Cham (2023). https://doi.org/10.1007/978-3-031-37731-0_15
  * [10] Goodman, B., Flaxman, S.: European union regulations on algorithmic decision making and a “right to explanation”. AI Magazine 38(3), 50–57 (Sep 2017). https://doi.org/10.1609/aimag.v38i3.2741
  * [11] Gunning, D., Aha, D.W.: Darpa’s explainable artificial intelligence program. AI Magazine 40(2), 44–58 (2019). https://doi.org/10.1609/aimag.v40i2.2850
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
  * [13] Ishihara, Shinobu, ..: Ishihara’s tests for colour-blindness. Kanehara, Tokyo, Japan, 38 plates ed edn. (1988, 1980), https://worldcat.org/title/50437309
  * [14] Jiang, M., Huang, S., Duan, J., Zhao, Q.: Salicon: Saliency in context. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1072–1080 (2015). https://doi.org/10.1109/CVPR.2015.7298710
  * [15] Kenny, E.M., Ford, C., Quinn, M., Keane, M.T.: Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and error-rates in xai user studies. Artificial Intelligence 294, 103459 (2021). https://doi.org/10.1016/j.artint.2021.103459
  * [16] Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association 47(260), 583–621 (1952). https://doi.org/10.1080/01621459.1952.10483441
  * [17] Likert, R.: A technique for the measurement of attitudes. Archives of Psychology 22(140), 5–55 (1932), https://legacy.voteview.com/pdf/Likert_1932.pdf
  * [18] Mann, H.B., Whitney, D.R.: On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics 18(1), 50 – 60 (1947). https://doi.org/10.1214/aoms/1177730491
  * [19] McHugh, M.L.: The chi-square test of independence. Biochemia Medica (Zagreb) 23(2), 143–149 (2013). https://doi.org/10.11613/bm.2013.018
  * [20] Petsiuk, V., Das, A., Saenko, K.: Rise: Randomized input sampling for explanation of black-box models (2018), https://api.semanticscholar.org/CorpusID:49324724
  * [21] Ribeiro, M.T., Singh, S., Guestrin, C.: "why should i trust you?": Explaining the predictions of any classifier. p. 1135–1144. KDD ’16, Association for Computing Machinery, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939778
  * [22] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74
  * [23] Wang, Y., Chubarau, A., Yoo, H., Akhavan, T., Clark, J.: Age-specific perceptual image quality assessment. Electronic Imaging 35(8), 302–1–302–1 (2023). https://doi.org/10.2352/EI.2023.35.8.IQSP-302
  * [24] Xu-Darme, R., Benois-Pineau, J., Giot, R., Quénot, G., Chihani, Z., Rousset, M.C., Zhukov, A.: On the stability, correctness and plausibility of visual explanation methods based on feature importance (2023), https://arxiv.org/abs/2311.12860
  * [25] Zhukov, A., Benois-Pineau, J., Giot, R.: Evaluation of explanation methods of ai - cnns in image classification tasks with reference-based and no-reference metrics. Advances in Artificial Intelligence and Machine Learning 03(01), 620–646 (Jan 2023). https://doi.org/10.54364/aaiml.2023.1143