# IG2: Integrated Gradient on Iterative Gradient Path for Feature Attribution

Yue Zhuo, Zhiqiang Ge  This paper has been accepted by IEEE Transactions on
Pattern Analysis and Machine Intelligence. This work was supported in part by
the National Natural Science Foundation of China (NSFC) (92167106, 62103362
and 61833014) and the Natural Science Foundation of Zhejiang Province
(LR18F030001). (Corresponding author: Zhiqiang Ge) Yue Zhuo is with the State
Key Laboratory of Industrial Control Technology, College of Control Science
and Engineering, Zhejiang University, Hangzhou, 310027, China. Zhiqiang Ge is
with Peng Cheng Laboratory, Shenzhen, 518000, China. (E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Feature attribution explains Artificial Intelligence (AI) at the instance
level by providing importance scores of input features’ contributions to model
prediction. Integrated Gradients (IG) is a prominent path attribution method
for deep neural networks, involving the integration of gradients along a path
from the explained input (explicand) to a counterfactual instance (baseline).
Current IG variants primarily focus on the gradient of explicand’s output.
However, our research indicates that the gradient of the counterfactual output
significantly affects feature attribution as well. To achieve this, we propose
Iterative Gradient path Integrated Gradients (IG2), considering both
gradients. IG2 incorporates the counterfactual gradient iteratively into the
integration path, generating a novel path (_GradPath_) and a novel baseline
(_GradCF_). These two novel IG components effectively address the issues of
attribution noise and arbitrary baseline choice in earlier IG methods. IG2, as
a path method, satisfies many desirable axioms, which are theoretically
justified in the paper. Experimental results on XAI benchmark, ImageNet,
MNIST, TREC questions answering, wafer-map failure patterns, and CelebA face
attributes validate that IG2 delivers superior feature attributions compared
to the state-of-the-art techniques. The code is released at:
https://github.com/JoeZhuo-ZY/IG2.

###### Index Terms:

Feature Attribution, Integrated Gradient, eXplainable Artificial Intelligence
(XAI), Counterfactual Explanation

##  1 Introduction

AI models are becoming increasingly prevalent in critical fields, such as
industrial control and biomedical analysis. Consequently, the need for
research into their explainability (XAI) has become urgent. This is essential
to keep humans in the loop and help people understand, explain, and control
the models [1, 2, 3]. Given an input instance (e.g., an image), feature
attribution for deep neural networks quantifies the contributions of
individual features, such as pixels, to the model output. These results can
support the users in reasoning which input elements drive model predictions.

The gradient is a basic form of feature attribution that analogizes the
model’s coefficients for a deep network. Early local gradient methods such as
Vanilla Gradient [4], Grad-CAM [5], and Guided Backpropagation [6] suffer from
the gradient saturation, a problem that the gradients in the input
neighborhood are misleading [7, 8]. Recently, for solving this problem,
Integrated Gradients (IG) [9] was proposed as a path method that integrates
gradient (of model output) along a path between the explained instance (i.e.,
explicand) and baseline.

IG methods introduce the concept of counterfactual explanation, which
contrastively explain the models by answering: “Which features cause the model
output prediction A (of explicand) rather than counterfactual prediction B (of
baseline)111Note that the terms _counterfactuals_ [11], _contrastive facts_
[15], and _counter (class) example_ [16] are used interchangeably in prior
explanation works. In the context of Shapley value [12] and path methods [9],
they are commonly referred to as _background_ and _baseline_.?” From the
perspectives of philosophy and psychology, the counterfactuals align with
human cognition to explain unexpected events [10, 11], and have been implied
in many attribution methods, such as SHapley Additive exPlanations [12],
DeepLIFT [7], SCOUT [13], and sub-region interpretation [14].

The focus of this study is on IG, a well-known path (attribution) method for
deep neural networks. Path methods, rooted in Aumann-Shapley game theory,
adhere to many describable axioms [17, 9]. IG’s attribution performance
depends on two essential hyperparameters: path and baseline. The conventional
approach, Vanilla IG, typically employs a zero baseline (e.g., all-black
image) along with a straight-line path. However, this choice is arbitrary and
agnostic to the model and explicand, leading to several shortcomings. For
instance, the straight-line path can introduce noise into the attribution due
to the saturation effect [18] and the use of a black baseline will result in
incomplete attributions [19, 20].

Figure 1: Illustration of IG2 with GradPath and GradCF, compared with Vanilla
IG (with a zero baseline) and Guided IG (counterfactual baseline). The
explicand is a sample [Doberman] from ImageNet [21] classified by Inception-v3
[22]. The graphs of logit predictions for [Doberman] w.r.t. $\alpha$ value are
plotted. The attributions snapshots of each method are shown at $\alpha$
values of 0.02, 0.5 and 1.0. At $\alpha=1.0$, the attributions are decomposed
into two multipliers: explicand’s gradient (small images in red boxes) and
integration path direction (in blue boxes). Integration path of IG2 optimally
aligns the explicand’s gradient with the counterfactual gradient. This
alignment results in feature attributions that are less noisy and more
complete (on the body of Doberman). Comprehensive attribution results on
ImageNet can be found in Section 6.3.

Recently, with different paths and baselines, many variant path methods have
been proposed for improving attributions. For integration paths, Guided IG
[23] adaptively chooses the path by selecting features with the smallest
partial derivatives; Blur IG [24] integrated the gradients on the gradually
blurred image path. For baselines, Expected IG [19] sampled the baselines from
the data distribution; Sturmfels et al. [20] discussed different baselines’
impacts on the path methods.

Notably, despite these advancements, the majority of existing baselines and
paths are considered model-agnostic and explicand-agnostic. We believe the
excellent baseline and path should contain the information of both explicand
and model, which motivates the idea of IG2. Table I summarizes the existing
IG-based methods from paths and baselines, and Section 5.2 contrasts them with
our proposal in detail.

IG2: Fig. 1 depicts IG2. Starting from the explicand, IG2 iteratively searches
the instances in the descent direction of the counterfactual gradient,
minimizing the representation distance between the explicand and
counterfactual. The set of all samples searched at each step is denoted as
GradPath, and its endpoint is the baseline GradCF, a counterfactual (CF)
example. As the name suggests, IG2 multiplies two gradients during the
integration: one of explicand’s prediction and another of the counterfactual
class, the latter of which is implied in the GradPath. IG2 provides superior
attributions over existing techniques, due to the distinctions on two
essentials of path methods: the integration path and baseline.

GradPath in IG2 effectively mitigates saturation effects [23, 18] (in
Definition 3). This is achieved by its alignment with the counterfactual
gradients, leading to a rapid decrease in the model’s prediction of the
explicand. Fig. 1 show the merit of GradPath by the attributions at
$\alpha=1$, where three path share an identical explicand’s gradient. The
GradPath, by directing itself towards the salient features that distinguish
the explicand and counterfactual, which “filters” the noise in explicand’s
gradient by path integration. In contrast, the straight-line in Vanilla IG is
on the dissimilar direction to the explicand’s gradient, causing the noise in
image background to accumulate along the integration path, leading to
saturation effects; Guided IG shares a similar idea with IG2 but its path
direction is constrained, which results in less complete attributions than
IG2.

GradCF is a novel baseline proposed in IG2, and its advantages are illustrated
in Fig. 2, contrasting the explicand with different baselines. GradCF can
significantly highlight the critical features. For [Doberman] example, GradCF
accurately highlights the dog’s body with the counterfactual gradients. On the
contrary, using a black baseline disregards the black pixels on dog and
adversely attributes to the white background instead. While Expected IG
employs counterfactual data as the baseline to address this issue, but this
naive contrast in the input space remains irrelevant to the explicand,
resulting in noises [20]. For other two datasets, GradCF also precisely
highlights the key features, distinguishing between digital 5 and digital 6,
as well as the defective wafer map in contrast to normal ones.

Figure 2: Differences between explicand and three baselines: the black image,
counterfactuals and GradCF. Three samples from ImageNet, MNIST [25] and wafer
map [25] datasets are plotted. For the non-black baselines of counter class,
the ImageNet sample is contrasted with randomly sampled images, MNIST’s
digital 5 is contrasted with digitals 6, and the central failure wafer map is
contrasted with normal wafer maps. The critical features in explicands are
accurately highlighted by contrasting with GradCF. Detailed discussions of
GradCF on MNIST examples are in Section 4.2.

In this study, our primary contribution is the introduction of IG2, a novel
path attribution method. IG2 comprises both a novel baseline, GradCF, and a
novel integration path, GradPath. To the best of our knowledge, it is the
first time that counterfactual gradients are integrated into the path
attribution methods. Through extensive experiments on datasets from diverse
domains, both qualitative and quantitative results consistently demonstrate
IG2’s superiority over existing state-of-the-art attribution methods.
Furthermore, we substantiate the individual effectiveness of GradCF and
GradPath through an ablation study.

The remainder of the paper is organized as: Section 2 introduces the
preliminary about path methods; Section 3 demonstrates our proposal in detail;
Section 4 gives a deep insight into IG2 and theoretically justify its axioms;
Section 5 contrasts our method with related works across different fields;
Section 6 presents experimental results to verify the attribution performances
of IG2; Section 7 presents the IG2 implementation details; Section 8 concludes
the paper.

TABLE I: Summary of existing path methods from the aspects of path and
baseline

Methods | Path | M-s$\dagger$ | Baseline | E-s$\S$ | M-s$\dagger$
---|---|---|---|---|---
[20]* | straight line |  | maximal distance | ✓ |
noised data | ✓ |
uniform noise |  |
Vanilla IG [9] | straight line | zero vector |  |
Expected IG [19] | straight line |  | train data |  |
XRAI [26] | straight line |  | black+white images |  |
Blur IG [24] | blur path |  | blurred image | ✓ |
Guided IG [23] |  | straight line’s
---
projection
✓ | zero vector |  |
IG2 (ours) | GradPath | ✓ | GradCF | ✓ | ✓

  * *

Work [20] discussed all the other baselines in Vanilla IG, Expected IG, XRAI,
Blur IG, and Guided IG (except ours), which is not listed for clarity.

  * $\dagger$

M-s (Model-specific): The path (or baseline) is specifically designed for
explained models.

  * $\S$

E-s (Explicand-specific): The baseline is specifically designed for explained
sample.

##  2 Preliminary of path methods

Path methods are based on the Aumann-Shapley theory from cost-sharing with
many desirable properties. Given input instances of $n$ dimension
$x\in\mathbb{R}^{n}$, the path of gradient integral is formally defined as
$\gamma(\alpha)$ for $\alpha\in[0,1]$. Path
$\gamma(\alpha):[0,1]\rightarrow\mathbb{R}^{n}$ consists of a set of points in
$\mathbb{R}^{n}$, from the baseline $x^{\prime}$ to the explicand $x$ (i.e.,
$\gamma(0)=x^{\prime}$ and $\gamma(1)=x$).

Given a path $\gamma$ and model $f:\mathbb{R}^{n}\rightarrow\mathbb{R}$ (in
classification models, only considering the output of explicand’s class
label), path integrated gradient attributes the $i^{th}$ feature by
integrating the gradients of the model output w.r.t the $i^{th}$ feature value
along the path $\gamma(\alpha)_{i}$, which is defined as [9]:

$\displaystyle\phi^{Path}_{i}=\int_{0}^{1}\frac{\partial
f(\gamma(\alpha))}{\partial\gamma(\alpha)_{i}}\frac{\partial\gamma(\alpha)_{i}}{\partial\alpha}d\alpha,$
(1)

where the first multiplier is the explicand’s gradient of model prediction and
the second one is the path direction.

Baselines: The choices of baseline $x^{\prime}$ are various, and there is
currently no consensus on which baseline is the best. Work [20] carefully
researched the mainstream baselines, and we summarized them in Table I. The
zero (black) instance $x^{\prime}=\vec{0}$, one (white) instance
$x^{\prime}=\vec{1}$ and train data $x^{\prime}\sim D_{train}$ are three
commonly used baselines.

Straight-line path: The commonly used path is the straight line from the
explicand $x$ to baseline $x^{\prime}$, which is specified
$\gamma(\alpha)=x^{\prime}+\alpha\times(x-x^{\prime})$ for $\alpha\in[0,1]$.

In practice, it is intractable to directly compute the integral in Eq. 1.
Instead, a Riemann sum is used for a discrete approximate with $k$ points in
the integral interval. On a straight-line path, the IG [9] is computed by:

$\displaystyle\phi^{IG}_{i}=(x_{i}-x^{\prime}_{i})\times\sum_{i=1}^{k}\frac{\partial
f(x^{\prime}+\frac{k}{m}\times(x-x^{\prime}))}{\partial
x_{i}}\times\frac{1}{m}.$ (2)

Figure 3: Illustration for building GradPath at each step. From the explicand,
the direction of GradPath is iteratively built on the gradient direction for
minimizing the model representation distance to the reference.

##  3 Methodologies of IG2

IG2 is a path method that extends IG [9] by introducing a novel baseline
(GradCF) and a novel integration path (GradPath). As the name suggests, IG2
accumulates not only the gradient of the explicand’s prediction but also the
counterfactual gradient, the latter of which is contained in the direction of
GradPath.

Given an explicand $x$, IG2 first sample a counterfactual reference $x^{r}$
with the different class label to $x$. We name $x^{r}$ reference instead of
baseline since it is not the integration endpoint of IG2.222Counterfactual
sample $x^{r}$ directly serves as the baseline in Expected IG [19].

######  Definition 1.

_(IG 2 attribution)_ Let $x^{r}$ denote reference, $\gamma^{G}$ denote
GradPath, $f$ and $\tilde{f}$ denote the prediction and representation layer
of the explained model, IG2 attribution for the $i^{th}$ feature of explicand
$x$ is defined as:

$\displaystyle\phi^{IG^{2}}_{i}=\sum_{j=0}^{k-1}\underbrace{\frac{\partial
f(\gamma^{G}(\frac{j}{k}))}{\partial x_{i}}}_{\text{\emph{explicand's
}}}\times\underbrace{\frac{\partial\|\tilde{f}(\gamma^{G}(\frac{j}{k}))-\tilde{f}(x^{r})\|_{2}^{2}}{\partial
x_{i}}\times\frac{\eta}{W_{j}}}_{\text{\emph{counterfactual} }},$ (3)

where $W_{j}$ is the normalization coefficient and $\eta$ is the step size
hyperparameter.

Eq. 3 explicitly reveals the nature of IG2: the multiplication of two
gradients. Compared to Vanilla IG in Eq. 2, the major distinction is the
Riemann summation weight of explicand’s gradients: the weight of
counterfactual gradient can highlight the critical features, whereas that in
Vanilla IG is a constant. Intuitively, IG2 introduces the information of
_model representation difference_ rather than _naive difference in the input
feature space_ (i.e., $x_{i}-x_{i}^{\prime}$).

Overall, IG2 is built on two stages: building GradPath and integrating
gradients on GradPath. The following sections respectively introduce them,
deriving IG2 in Eq. 3.

Algorithm 1 Compute GradPath and GradCF

0: representation layer: $\tilde{f}$ explicand: $x$ reference: $x^{r}$ step
size: $\eta$ steps: $k$

1: $\delta\Leftarrow\vec{0}\ \lhd$ initiate perturbation

2: $\gamma^{G}(1)=x\ \lhd$ initiate GradPath 333GradPath is built on the
opposite direction to the conventional path integration. To match path
methods, we set GradPath $\gamma^{G}(j)$ from $j=1$ to $j=0$, so that the
optimization iteration starts at the point $\gamma^{G}(1)$ (explicand) and
ends at the point $\gamma^{G}(0)$ (baseline, GradCF).

3: for $j=k-1$ to $0$ do

4: $g=\frac{\partial\|\tilde{f}(x+\delta)-\tilde{f}(x^{r})\|_{2}}{\partial x}$

5: $W=\|g\|_{1or2}\ \lhd$ $\ell_{1}$ or $\ell_{2}$ norm

6: $g=\eta\cdot\frac{g}{W}\lhd$ normalized iterative gradient

7: $\delta=\delta-g\lhd$ update total perturbation

8: $\gamma^{G}(j/k)=x+\delta\ \lhd$ store the CF at each step

9: end for

9: GradCF: $\gamma^{G}(0)$, GradPath: $\gamma^{G}$

###  3.1 Building GradPath

Fig. 3 illustrates how GradPath is built during the iterative search of
GradCF. The motivation of GradCF is to provide a counterfactual explanation:

_Given limited perturbation resource, perturbing which features on explicand
$x$ can make the model consider the perturbed explicand to be most similar to
the (counterfactual) reference $x^{r}$? _

This similarity can be measured by the distance between two model
representations444We use activations in the penultimate layer as the
representation. The choice of representation layer is discussed in Appendix
A.3. and the perturbation search can be converted to a minimization problem.
Denoting the network representation by $\tilde{f}$, the perturbation by
$\delta$ and Euclidean distance measure by $\|\cdot\|^{2}_{2}$, the
optimization objective for GradCF is:

$\displaystyle\min_{\delta}\|\tilde{f}(x+\delta)-\tilde{f}(x^{r})\|^{2}_{2}.$
(4)

We iteratively solve Eq. 4 using gradient descent with normalization at each
step. GradPath $\gamma^{G}$ is built during the iteration by the trajectory of
normalized gradient descent, and the endpoint $x+\delta$ is the target
counterfactual baseline, GradCF.

Algorithm 1 provides the pseudo-code for computing GradCF and GradPath.
GradPath can be defined as:

######  Definition 2.

_(GradPath)_ Given reference $x^{r}$ and model representation $\tilde{f}$,
GradPath is defined by a discrete function $\gamma^{G}$, on a feasible set
$\\{0,\frac{1}{k},\cdots,\frac{k-1}{k},1\\}$, for $0\leq j\leq k-1,\
j\in\mathbb{N}$:

$\displaystyle\gamma^{G}(\frac{j}{k})$
$=\gamma^{G}(\frac{j+1}{k})-\frac{\partial\|\tilde{f}(\gamma^{G}(\frac{j+1}{k}))-\tilde{f}(x^{r})\|_{2}}{\partial\gamma^{G}(\frac{j+1}{k})}\frac{\eta}{W_{j}},$
$\displaystyle\gamma^{G}(1)$ $\displaystyle=x,$ (5)

where $W_{j}$ is introduced in Line 5, Algorithm 1.

GradCF as explanation (GradCFE) We can utilize the difference between the
GradCF and explicand to provide a counterfactual feature attribution:

$\displaystyle\mathrm{GradCFE}=x-\gamma^{G}(0).$ (6)

###  3.2 Integrating gradients on GradPath

IG2 integrates feature gradients in the same approach as path methods (in Eq.
1). Since the GradPath $\gamma^{G}$ is a discrete function, which does not
have continuous gradients $\frac{\partial\gamma(\alpha)}{\partial\alpha}$, we
can operate finite difference to approximate the path gradient. Both forward
difference and backward difference are feasible. According to Eq. 5, the path
gradient of GradPath w.r.t $\alpha=\frac{j}{k}$ can be computed by both two
differences:

$\displaystyle\frac{\partial\gamma^{G}(\frac{j}{k})}{\partial\frac{j}{k}}=\begin{cases}\frac{\gamma^{G}(\frac{j+1}{k})-\gamma^{G}(\frac{j}{k})}{1/k}=k\frac{\partial\|\tilde{f}(\gamma^{G}(\frac{j+1}{k}))-\tilde{f}(x^{r})\|_{2}}{\partial\gamma^{G}(\frac{j+1}{k})}\frac{\eta}{W_{j}}\\\
\frac{\gamma^{G}(\frac{j}{k})-\gamma^{G}(\frac{j-1}{k})}{1/k}=k\frac{\partial\|\tilde{f}(\gamma^{G}(\frac{j}{k}))-\tilde{f}(x^{r})\|_{2}}{\partial\gamma^{G}(\frac{j}{k})}\frac{\eta}{W_{j}}\\\
\end{cases},$ (7)

where the first line is the forward difference and the second is the backward
difference.

Using the backward difference in Eq. 7 and Riemann sum of Eq. 1, we derive IG2
attribution (for the $i^{th}$ feature):

$\displaystyle\phi^{IG^{2}}_{i}$ $\displaystyle=\int_{0}^{1}\frac{\partial
f(\gamma^{G}(\alpha))}{\partial\gamma^{G}(\alpha)_{i}}\frac{\partial\gamma^{G}(\alpha)_{i}}{\partial\alpha}d\alpha$
$\displaystyle\approx\sum_{j=0}^{k-1}\frac{\partial
f(\gamma^{G}(\frac{j}{k}))}{\partial\gamma^{G}(\frac{j}{k})_{i}}\times\frac{\partial\|\tilde{f}(\gamma^{G}(\frac{j}{k}))-\tilde{f}(x^{r})\|_{2}}{\partial\gamma^{G}(\frac{j}{k})_{i}}$
$\displaystyle\quad\times\frac{k\eta}{W_{j}}\times(\frac{j+1-j}{k})$
$\displaystyle=\sum_{j=0}^{k-1}\frac{\partial
f(\gamma^{G}(\frac{j}{k}))}{\partial
x_{i}}\times\frac{\partial\|\tilde{f}(\gamma^{G}(\frac{j}{k}))-\tilde{f}(x^{r})\|_{2}}{\partial
x_{i}}\times\frac{\eta}{W_{j}},$ (8)

where the two denominators $\partial\gamma^{G}(\frac{j}{k})_{i}$ are
substituted as $\partial x_{i}$, due to $\gamma^{G}(\frac{j}{k})=x+\delta$
(Line 8, Algorithm 1).

Practically, the integration direction (from baseline to explicand) is
opposite to the GradPath search direction (from explicand to baseline). Thus,
backward difference requires the complete GradPath to be computed first, and
then the gradient is integrated. If using forward difference, the gradient
integral can be computed simultaneously with the GradPath.

For the implementation details of IG2, Section 7 comprehensively discusses the
hyperparameter impacts on IG2 attribution, including reference, step size,
step number, normalization, and similarity measures. The IG2 computational
cost is also analyzed and compared with other attribution methods.

Figure 4:  (a) Illustration for three counterfactual examples and three
integration paths: CF1 sampled from counterfactual data distribution, CF2
generated by an adversarial attack and the GradCF using CF1 as the reference.
The saturated area on straight-line path is marked in red. An MNIST explicand
(digital 5) and three CFs (digital 6) are plotted. (b) Graphs of explicand’s
Softmax prediction along integration paths on ImageNet, MNIST, and TREC,
averaged on 100 samples of each dataset. Figure 5: The toy example for feature
attribution with function $y=\max(x_{1},x_{2}-1)$ on explicand
$(x_{1},x_{2})=(3,3)$ with the zero reference $(0,0)$. (a) GradPath on
counterfactual gradients in blue line. (b) The attribution results of three
methods in the legend.

##  4 Interpreting IG2

The novel baseline (GradCF) and novel integration path (GradPath) are two
major contributions of IG2, so we discuss IG2 by interpreting the superiority
of these two components. Specifically, Section 4.2 and Section 4.1 answer the
questions: _Why GradCF and GradPath are better baseline and integration path
than the existing methods?_

Theoretically, the desirable axioms of IG2 and GradCFE are justified in
Section 4.3.

###  4.1 GradPath: mitigating saturation effects

The superiority of GradPath for feature attribution is discussed from the
perspective of saturation effects.

######  Definition 3.

_(Saturation effects)_ [23, 18] The straight-line path of IG is susceptible to
travel through the saturated area where model output is not changing
substantially with respect to $\alpha$. In this area, the feature gradient is
not pointing toward the integration path. This phenomenon leads to the
accumulation of noisy attributions, called saturation effects.

The integral value in Eq. 1 can be decomposed into two multipliers: (input)
feature gradient and path direction. The presence of a saturated area
indicates that the feature gradient and integration path are in dissimilar
directions (otherwise, the model prediction should drop quickly), which means
the path is not moving on the important features. This will result in
incorrect feature attributions, and a good integration path should avoid this
undesirable area.

The saturation effects were analyzed in works [23, 18]. Some techniques have
been proposed for this issue, for example, averaging over multiple straight
paths [26, 27] and splitting the straight paths into different segments [18].
Guided IG [23] explicitly avoids the saturated area by designing the path
based on the absolute values of feature gradients. However, the path of Guided
IG is still constrained in the hyper-rectangular with the straight-line path
as diagonal.

GradPath effectively mitigates the saturation effects. Recalling the objective
of GradCF in Eq. 4, we minimize the distance to the counterfactual model
representation, which implicitly means the explicand’s prediction is meanwhile
minimized. Each step of GradPath points toward the steep direction that
rapidly decreases the model prediction (shown in Fig. 4).

As a supplementary for graphs in Fig. 1, Fig. 4b displays the average output
curves on three datasets. Compared with straight line, Guided IG’s path and
GradPath both avoid the saturated area. Since Guided IG’s path is restricted,
GradPath can get out of the saturated area more quickly.

Toy example: Fig. 5 displays a toy example, showing GradPath contributes to
better feature attribution. Because $x_{2}$ is always smaller than $x_{1}$ on
the straight-line path, the $\max$ function signs zero gradients on the
smaller feature (right black arrows). This causes IG and gradient methods both
assign zero attribution on feature $x_{2}$, which is inconsistent with
intuition. Contrarily, integrated gradients on GradPath give nonzero
attribution on $x_{2}$ with a more reasonable baseline, GradCF $(0,1)$.

Figure 6:  Feature attributions of MNIST samples. (a) Shifted digital 1s
explained with the references of digitals 7. (b) Digitals 5 explained with the
references of different categories (digitals 3, 6, 9). The _most_ critical
areas that distinguish the explicand to reference are marked by red boxes,
e.g., the explained digitals 1 will become digital 7 if we filled these areas.
IG2 is compared with Vanilla IG (using black baseline) and Expected IG (using
references as the baselines).

###  4.2 GradCF

Based on works in the field of counterfactual explanation (CFE), we summarize
a good counterfactual baseline in path methods should have the following
desirable properties:

  * •

_Validity_ [11] The counterfactual baseline should be classified in the
desired class (different from the explicand).

  * •

_Data manifold closeness_ [28] It would be hard to trust a counterfactual if
it resulted in a combination of features that were utterly unlike any
observations the classifier has seen before. Therefore, a generated
counterfactual should be realistic in the sense that it is near the training
data.

  * •

_Explicand relevance_ A good counterfactual example should be related to the
explicand to directly contrast features. Though almost all the CFE methods
[28] generate counterfactual examples based on the explicand, the baselines of
most existing path methods are explicand-agnostic (see Table I).

The commonly used uninformative baselines (black and white) violate all three
properties, which are unrealistic and do not have any information on classes.
For instance, the black pixel will not be attributed with an all-black
baseline, even though it contributes to the model output.

Expected IG [19] solves this by using the samples from the data distribution
(CF1 in Fig. 4), which satisfies the validity and data manifold closeness.
However, the sampling procedure of train data is _explicand-agnostic_ , which
provides inaccurate contrast in the feature space (discussed in Fig. 6a).

A basic counterfactual example (CF2 in Fig. 4) is also compared. CF2 is
generated with projected gradient descent (PGD) [29] attack for the minimal
perturbation that causes the model to give a counterfactual prediction [30].
Though this basic CF is related to the explicand, it is usually unrealistic
and violates the _Data manifold closeness_.

Distinctively, GradCF satisfies all three describable properties, which
correlates with the explicand and stays on the manifold of counterfactual
data. In other words, the generation of GradCF simultaneously implies the
closeness to feature manifolds of explicand and counterfactual.

MNIST examples: Fig. 6a demonstrates the significance of _Explicand
relevance_. Using digitals 7 as the references, we explain shifted digitals 1
at different positions. To human intuition, the critical areas distinguishing
digital 1 to 7 are at the top left of digital 1 (see the red boxes). Fig. 6a
shows that the highlighted areas of IG2 are synchronized with the shift of
explicands and critical areas, that is _Explicand relevance_.

As for Expected IG, using reference samples as baselines only provides the
naive pixel contrast at the input feature space. This will result in
explanations that are irrelevant to the explicand, which is obviously
inconsistent with the intuition (see the last rows in Fig. 6).

Fig. 6b shows the impact of the different references, where the critical areas
are marked by red boxes. Attributed to the gradients of counterfactual
classes, the GradCF and IG2 significantly highlight the critical areas of all
three references. Due to the same issue of straight paths, attributions of
Expected IG are overly focused on the upper right corners of digital 5, which
is not the _most_ critical areas555The ablation study is conducted in Appendix
Section B.1 for validating the most critical areas that distinguishes digital
5 to digitals 3, 6, 9.. As for IG with the all-black baseline, any pixels out
of the explained digitals will not have the attributions, which is incomplete
for the explanation.

In Section 7.1, we will further demonstrate the effects of counterfactual
references on IG2 with multiple datasets.

###  4.3 Axioms of IG2

Works [17, 9, 12] claimed that path methods are the unique methods that
satisfy certain desirable axioms. As a subset of path methods, we justify IG2
satisfies the following four axioms: _Completeness_ , _Dummy_ ,
_Implementation Invariance_ and _Symmetry_ , and GradCFE also satisfies the
latter three.

######  Definition 4.

_(Completeness)_ For every explicand $x$, and baseline $x^{\prime}$, the
attributions $\phi_{i}$ add up to the prediction difference
$f(x)-f(x^{\prime})$:

$\displaystyle\phi_{i}=f(x)-f(x^{\prime}).$ (9)

######  Remark 1.

Like other path integral methods, IG2 also integrates the gradient in a
conservative (input) vector field. Since all the path methods satisfy
Completeness regardless of the path shape (see [9, 31]), IG2 satisfies
Completeness.

Due to our proposal utilizing the model representation, we extend the
definitions of the three remaining axioms to the representation layer version,
the extension of which is respectively (i.e., resp.) shown in the brackets.
Notably, if we use the output layer as representation, no extension is needed.

######  Definition 5.

_(Dummy)_ Dummy features get zero attributions. A feature $i$ is dummy in a
function $f$ (_resp. $\tilde{f}$_) if for any two values $x_{i}$ and
$x^{\prime}_{i}$ and every value $x_{N\setminus i}$ of the other features,
$f(x_{i};x_{N\setminus i})=f(x^{\prime}_{i};x_{N\setminus i})$ (_resp.
$\tilde{f}(x_{i};x_{N\setminus i})=\tilde{f}(x^{\prime}_{i};x_{N\setminus
i})$_). Conceptually, the feature that is not referenced by the model
naturally requires zero attributions.

######  Remark 2.

IG2 satisfies Dummy and GradCFE satisfies Dummy at the representation layer.
The latter is a sufficient condition for the former.

###### Proof.

According to Eq. 5, given any point $\gamma^{G}(\alpha)$ on GradPath, the
change on feature $\gamma^{G}(\alpha)_{i}$ is proportional the normalized
gradient:

$\displaystyle\Delta\gamma^{G}(\alpha)_{i}$
$\displaystyle\propto\frac{\partial\|\tilde{f}(\gamma^{G}(\alpha))-\tilde{f}(x^{r})\|_{2}^{2}}{\partial\gamma^{G}(\alpha)_{i}}\times\frac{1}{W}$
(10) $\displaystyle=\lim_{h\to
0}\Big{(}\partial\|\tilde{f}(\cdots,x_{i}+h,\cdots)-\tilde{f}(x^{r})\|_{2}^{2}$
$\displaystyle\quad-\|\tilde{f}(\cdots,x_{i},\cdots)-\tilde{f}(x^{r})\|_{2}^{2}\Big{)}\times\frac{1}{hW}.$
(11)

Based on the definition of Dummy,
$\tilde{f}(\cdots,x_{i}+h,\cdots)\equiv\tilde{f}(\cdots,x_{i},\cdots)$, so
that

$\displaystyle\Delta\gamma^{G}(\alpha)_{i}\equiv 0.$ (12)

According to the definitions in Eq.3 and Eq. 6, we get the zero attributions
of both IG2 and GradCFE for dummy features at the representation layer.

The Dummy axiom of IG2 does not require the feature to be dummy at the
representation layer. Suppose the feature is dummy only for the model output
(unextended definition). In that case, IG2 still assigns zero attribution on
this feature, the proof of which is similar (explicand’s gradient for dummy
features in Eq. 3 constantly equals to zero) and not repeated here. ∎

######  Definition 6.

_(Implementation Invariance)_ Two networks are functionally equivalent if
their outputs (_resp. representations_) are equal for all inputs, despite
having very different implementations. Attribution methods should satisfy
Implementation Invariance, i.e., the attributions are always identical for two
functionally equivalent networks and do not refer to implementation details.

######  Remark 3.

IG2 and GradCFE satisfy Implementation Invariance. The former is a sufficient
condition for the latter. IG2 and GradCFE only relies on the function
gradients of output and representation (GradCFE only concerns representation
gradient), which are invariant to the models’ internal implementation before
the representation layer.

######  Definition 7.

_(Symmetry)_ For every function $f$ (_resp. $\tilde{f}$_) is symmetric in two
variables $i$ and $j$, if
$f(\cdots,x_{i},x_{j},\cdots)=f(\cdots,x_{j},x_{i},\cdots)$ (_resp.
$\tilde{f}$_). If the explicand $x$ are such that $x_{i}=x_{j}$, the
attributions of symmetric function for features $i$ and $j$ should be equal.
Conceptually, under the symmetric function, the identical symmetric variables
receive identical attributions.

######  Remark 4.

IG2 and GradCFE preserve Symmetry. The former is a sufficient condition for
the latter. Notably, previous path methods are symmetry requires the variable
$i$ and $j$ of the baseline $x^{\prime}$ are also identical,
$x^{\prime}_{i}=x^{\prime}_{j}$, while our methods do not. The explicand with
identical symmetric variables will consequently lead to the identical
symmetric variables in the synthesized baseline GradCF.

######  Proposition 1.

(Guided IG [23]) If the values of symmetric variables are equal at every point
of the integration path, then their attributions are equal. Therefore, such a
path attribution method is symmetry preserving.

###### Proof.

According to Proposition 1, IG2 and GradCFE preserve Symmetry if the values of
symmetric features on GradPath $\gamma^{G}$ are equal at every point, the
proof of which is in the following.

Given explicand $x$, we only focus on the symmetric variables $i$ and $j$,
$x=(\cdots,x_{i},x_{j},\cdots)$, where $x_{i}=x_{j}$. According to Eq. 5, the
change on variable $x_{i}$ along GradPath $\gamma^{G}$ is:

$\displaystyle\Delta x_{i}$
$\displaystyle=-\frac{\partial\|\tilde{f}(\gamma^{G}(\alpha))-\tilde{f}(x^{r})\|_{2}^{2}}{\partial
x_{i}}\times\frac{\eta}{W}$ (13)
$\displaystyle=(\tilde{f}(\gamma^{G}(\alpha))-\tilde{f}(x^{r}))\times\frac{-2\eta}{W}\times\frac{\partial\tilde{f}(\gamma^{G}(\alpha))}{\partial
x_{i}}$ (14) $\displaystyle=\lim_{h\to
0}\frac{\tilde{f}(\cdots,x_{i}+h,x_{j},\cdots)-\tilde{f}(\cdots,x_{i},x_{j},\cdots)}{\partial
h}$
$\displaystyle\quad\times\frac{-2\eta}{W}\times(\tilde{f}(\gamma^{G}(\alpha))-\tilde{f}(x^{r})).$
(15)

and similarly, we can get the change on variable $x_{j}$:

$\displaystyle\Delta x_{j}$ $\displaystyle=\lim_{h\to
0}\frac{\tilde{f}(\cdots,x_{i},x_{j}+h,\cdots)-\tilde{f}(\cdots,x_{i},x_{j},\cdots)}{\partial
h}$
$\displaystyle\quad\times\frac{-2\eta}{W}\times(\tilde{f}(\gamma^{G}(\alpha))-\tilde{f}(x^{r})).$
(16)

Because $\tilde{f}$ is symmetric for variable $i$ and $j$:

$\displaystyle\tilde{f}(\cdots,x_{i}+h,x_{j},\cdots)=\tilde{f}(\cdots,x_{i},x_{j}+h,\cdots),$
(17)

two gradients in Eq. 15 and Eq. 17 are equal and we get the identical changes
on variable $i$ and $j$:

$\displaystyle\Delta x_{i}=\Delta x_{j}.$ (18)

Starting from the identical value $x_{i}=x_{j}$ of the explicand, we can get
the symmetric variables $x_{i}$ and $x_{j}$ are equal at every step of
GradPath. Thus, both IG2 and GradCFE provide identical attribution for the
symmetric variables.

∎

##  5 Related works

This section first introduces the previous attribution methods in the field of
XAI, and then systemically contrasts our proposal with the related works in
three sub-fields: path attribution, Shapley values and adversarially
counterfactual explanation.

###  5.1 Feature attribution

Feature attribution is a post-hoc method that explains AI models by scoring
the feature contributions to the model output [32].

Gradient-based methods are widely used for attribution. One of the earliest
successful work is DeconvNet [33], which applies a ReLU non-linearity to the
gradient computation. Based on Vanilla Gradient [4] and DeconvNet, Guided
Backpropagation [6] introduced an additional guidance signal from the higher
layers. Furthermore, Class Activation Map (CAM) was developed [34] and its
variants like Grad-CAM [5] also achieved success.

Another class of attribution methods is based on perturbation, which analyses
the model sensitivity by perturbing the input features. Occlusion sensitivity
maps [33] was one early method, which perturbs the input image with grey
squares and observes the model prediction. Not restricted to deep models, LIME
[35] can be applied to any prediction model by training a linear proxy model.

Shapley values can be considered a particular example of perturbation-based
methods and is justified to be the unique method that satisfies certain
desirable axioms [36]. However, computing the exact Shapley value is NP-hard
[37], which is prohibitive for deep neural networks. Thus, the related works
are dedicated to efficiently approximating the Shapley values with fewer model
evaluations. Some works are based on sampling: Strumbelj et al. [38] and
Mitchell et al. [39] respectively proposed Monte Carlo and quasi-Monte Carlo
for randomly sampling permutations; KernelSHAP [12] used LIME to reduce the
number of samples; Chen et al. [40] leveraged the underlying graph structure
for the structured data; Wang et al. [41] took the advantages of contributive
cooperator selection; Ancona et al. [42] introduced probabilistic deep network
to approximately propagate the Shapley value through the network layers. To
further accelerate the approximation, FastSHAP [43] trained a surrogate model
to fast generate the explanation, which avoids the expensive sampling
procedures; DeepSHAP [12] approximated the Shapley value at each layer and
merged them by DeepLIFT [7] in a backward fashion.

###  5.2 Path methods

Path method is a popular feature attribution that integrates the gradients
along a path. Table I has summarized the existing path methods to the best of
our knowledge. IG2 is the first path method with both model-specific path and
baseline.

Contrasting Blur IG [24] Blur IG resembles IG2 in using an iterative algorithm
to simultaneously construct the baseline and the integration path. The nature
of Blur IG is to iteratively build a path from the explicand to the baseline
of a totally blurred explicand. Since the blurred baseline fully depends on
the explicand, it lacks the counterfactual information within the model and
data distribution. Moreover, Blur IG is restricted to images and not
applicable to tabular data.

Contrasting Expected IG [19] Expected IG uses the informative baselines from
the data distribution, which is the reference of our proposal. Nevertheless,
the baseline is irrelevant to the explicand, and its straight-line path
naively contrasts the baseline and explicand by the difference in the feature
space, which still suffers from the saturation effects and noise problem.
Notably, the Expected IG2 (in Section 7.1.1) follows the Expected IG’s idea to
calculate average attributions by sampling references from the data
distribution.

Contrasting Guided IG [23] Guided IG is one close work to IG2. Firstly, our
proposal implies a similar motivation of Guided IG (discussed in Section 4.1).
Second, both integration paths are iteratively calculated based on the
gradient information to explicitly or implicitly avoid the saturated area. We
argue that the shape of GradPath is a generalization of Guided IG’s path. If
using $\ell_{1}$ normalization in Eq. 21, the GradPath has the identical shape
as Guided IG’s. We select the sparse features with the largest counterfactual
gradients, while Guided IG selects the smallest explicand’s gradients in the
converse direction (the identical shape does not guarantee the identical
integration path).

###  5.3 Shapley values

IG methods are the generalization of Aumann-Shapley value, an extension of
Shapley value to the continuous setting, which inherits desirable attribution
axioms [9]. Compared with the sampling-based Shapley value approximations, IG2
mainly advances in two aspects:

  * •

Scalability: Though recent algorithms can achieve efficient approximations,
they are still prohibitive for the high-dimensional input features (e.g.,
ImageNet samples). They have to apply the superpixel (group pixels) to reduce
the input dimension [39, 42, 12], which impairs the explanation quality. On
the large models, IG-based methods are much more efficient than most Shapley
value algorithms, the computational time comparison to sampling-based
KernelSHAP is reported in Appendix A.5.

  * •

Implicit zero baseline: Similar to Vanilla IG, many Shapley value methods need
to indicate the absence of features by replacing them with zero value, which
implicitly defines a zero baseline. The zero baseline’s adverse effects on
feature attributions have been discussed in the previous sections.

Contrasting DeepSHAP [12] Unlike other sampling-based approximations, DeepSHAP
is more related to IG methods. Its core part, DeepLIFT, replaces the gradient
at each nonlinear function with its average gradient, which is shown to be
most often a good approximation of IG [44].

In summary, though sharing the same theory fundamental, we argue that IG and
Shapley value approximations are on two different tracks: the former is mainly
designed for explaining large networks with accessible gradients, and the
latter is for accurately approximating the exact Shapley value, which is more
suitable for the small black-box models.

###  5.4 Adversarially counterfactual explanation

The adversarial learning shares the same optimization objective with
counterfactual explanation, and they are tightly related [30]. Some works [45,
46] utilized adversarial attack to explain the network. GradCF differs from
counterfactual explanation and adversarial attack in violating the principle
that counterfactual explanation should be the small perturbation on the
explicand [28]. Hence, we argue that GradCF is neither a canonical
counterfactual explanation (GradCF still provides a counterfactual
explanation) nor an adversarial attack.

From the perspective of methodologies, the iterative gradient descent
optimization method in Eq. 4 follows the adversarial attack method. The
optimization with $\ell_{2}$ normalization is from projected gradient descent
(PGD) [29] while $\ell_{1}$ normalization is from sparse adversarial attack
[47]. The only difference is that adversarial attack methods clamp the
computed instance within the neighborhood of explicand to guarantee
imperceptible perturbations.

Notably, the same $\ell_{2}$ normalized optimization method is also used in
work [48]. Nevertheless, they focus on searching the robust features under the
adversarial attack while the model explanation is out of their scope.

TABLE II: Evaluation of attribution methods on XAI-BENCH

| fai.($\uparrow$) | mon.($\uparrow$) | ROAR($\uparrow$) | G-S($\uparrow$) | inf.($\downarrow$)
---|---|---|---|---|---
Random | -0.033 | 0.458 | 0.332 | -0.060 | 0.034
Vanilla IG | 0.349 | 0.440 | 0.356 | 0.714 | 0.015
Guided IG | 0.342 | 0.463 | 0.359 | 0.681 | 0.021
Expected IG | 0.596 | 0.470 | 0.365 | 0.814 | 0.014
DeepSHAP | 0.380 | 0.488 | 0.357 | 0.821 | 0.014
KernelSHAP | 0.370 | 0.435 | 0.340 | 0.901 | 0.015
IG2(Ours) | 0.610 | 0.486 | 0.377 | 0.833 | 0.021

  * *

fai.:faithfulness mon.:monotonicity G-S:GT-Shapley inf.:infidelity

TABLE III: Evaluation of attribution methods on real-world datasets

|  | Ground truth | SIC AUC
---|---|---|---
Datasets | Explainers | AUC $\uparrow$ | SUM $\uparrow$ | ADD $\uparrow$ | DEL $\downarrow$
ImageNet | Gradient | 0.482 | 0.336 | 0.467 | 0.209
Vanilla IG | 0.536 | 0.327 | 0.476 | 0.205
Guided IG | 0.599 | 0.464 | 0.545 | 0.212
Expected IG | 0.666 | 0.431 | 0.557 | 0.116
DeepSHAP | 0.694 | 0.470 | 0.543 | 0.206
KernelSHAP | 0.747 | 0.498 | 0.561 | 0.274
IG2(Ours) | 0.805 | 0.516 | 0.656 | 0.115
TREC | Gradient | —* | 0.909 | 0.189
Vanilla IG | 0.937 | 0.159
Guided IG | 0.938 | 0.156
Expected IG | 0.940 | 0.141
DeepSHAP | 0.933 | 0.170
KernelSHAP | 0.933 | 0.172
IG2(Ours) | 0.942 | 0.140
Wafer map | Gradient | 0.570 | 0.226 | 0.661 | 0.216
Vanilla IG | 0.732 | 0.342 | 0.829 | 0.061
Guided IG | 0.758 | 0.450 | 0.789 | 0.050
Expected IG | 0.850 | 0.488 | 0.883 | 0.038
DeepSHAP | 0.863 | 0.528 | 0.890 | 0.029
KernelSHAP | 0.683 | 0.339 | 0.707 | 0.042
IG2(Ours) | 0.849 | 0.551 | 0.898 | 0.036
CelebA$\dagger$ | Gradient | 0.748 | 0.211 | 0.745 | 0.296
Vanilla IG | 0.705 | 0.175 | 0.740 | 0.314
Guided IG | 0.653 | 0.225 | 0.744 | 0.370
Expected IG | 0.698 | 0.189 | 0.737 | 0.293
DeepSHAP | 0.699 | 0.179 | 0.750 | 0.309
KernelSHAP | 0.788 | 0.212 | 0.765 | 0.197
IG2(Ours) | 0.795 | 0.224 | 0.815 | 0.205

  * *

The ground truth of TREC is not available.

  * $\dagger$

The ground truth of CelebA face attributes is generated by face parsing model
pretrained on CelebAMask-HQ [49], which is detailed in Appendix B.6.

##  6 Experiments

We conduct the attribution experiments on one synthetic dataset, XAI-BENCH,
and four real-world tasks: image classification on ImageNet, question
classification on TREC, anomaly classification on wafer map failure pattern,
and face attribute classification on CelebA. We compare IG2 with six methods:

  * •

(Vanilla) Gradient: The fundamental feature attribution method based on
backpropagation, using the input gradient w.r.t. the model’s prediction for
generating the saliency maps.

  * •

Vanilla IG, Guided IG, Expected IG: Three IG-based methods, which have been
discussed in the previous sections.

  * •

KernelSHAP: A basic sampling method for approximating Shapley values of black-
box models, which is a common baseline for approximation algorithms. Due to
the scalability limitation, the superpixel technique is applied when
attributing ImageNet and wafer map samples with KernelSHAP.

  * •

DeepSHAP: A high-speed Shapley value approximation for deep models based on
DeepLIFT.

For the baselines of compared methods, Vanilla IG, KernelSHAP, and Guided IG
all use the black image as the (implicit) baseline, Expected IG and DeepSHAP
samples the baselines from the same distribution as the counterfactual
references of IG2.

###  6.1 XAI Benchmark

First, we evaluate our proposal on the synthetic datasets and metrics released
by XAI-BENCH [50], a benchmark for feature attribution algorithms. Synthetic
datasets allow the efficient computation of the ground truth Shapley values
and other metrics, which is intractable on real-world datasets. We briefly
introduced the dataset and metrics, the details of which can be found in the
XAI-BENCH work [50].

####  6.1.1 Synthetic Dataset

The features are sampled from a multivariate normal distribution
$\mathbf{X}\sim\mathcal{N}(\mathbf{\mu},\mathbf{\Sigma})$, where
$\mathbf{\mu}$ is the mean vector and $\mathbf{\Sigma}$ is the covariance
matrix. The labels are binary (0 and 1) and defined over a _piecewise_
distribution with the function $\Psi(x)$. The explained model is a trained
three-layer perceptron for the regression task on the synthetic dataset. The
specification of the model and synthetic dataset are reported in Appendix B.2.

####  6.1.2 Metrics

We use five metrics from XAI-BENCH: (1) _faithfulness_ computes the Pearson
correlation coefficient between the attribution and the approximate marginal
contribution for each feature; (2)_monotonicity_ computes the fraction of the
marginal improvement for feature with attribution order $i$ is greater than
the marginal improvement for feature with attribution order $i+1$; (3)_ROAR_
is remove-and-retrain, which retrains the model with the features removed and
the area-under-the-curve (AUC) of the model is computed; (4) _GT-Shapley_
computes the Pearson correlation coefficient of the feature attribution to the
ground-truth Shapley values (ground-truth marginal improvement); (5)
_Infidelity_ is computed by considering the effects of replacing each feature
with a noisy baseline conditional expectation.

####  6.1.3 Results

Table II reports the five metrics for evaluating the feature attributions on
XAI-BENCH datasets. The Vanilla IG and Guided IG use zero baselines with
different paths, and the Expected IG uses data distribution as the baseline.
We also use the randomly generated attribution (Random) as a weak comparison.

The results evaluate the feature attributions of the small model and low input
dimensions. IG2 generally outperforms most other methods on the first three
metrics. Unsurprisingly, the Shapley value sampling method (KernelSHAP)
achieves the best performance on the GT-Shapley metric. The improvement of
Expected IG over other IG methods reveals that using the exception of
attribution from the data distribution baseline is effective.

Figure 7: Feature attributions on images from ImageNet dataset. The predicted
classes are listed in square brackets.

###  6.2 Metrics for real-world datasets

We adopt two types of quantitative metrics for evaluating the feature
attribution on real-world datasets:

Ground truth annotation [51] The first metric requires the ground truth
segmentation annotated by humans. The better attribution should be closer to
the ground truth annotations. Specifically, this metric treats the
attributions as binary classification prediction scores. With changing the
threshold of attribution scores to be negative class, the area under the
receiver operating characteristic curve is calculated [23], which is called
_ground truth-AUC_. We can also use the multiplication of (normalized)
attribution scores and ground truth to show the sum of attributions on the
annotated features, which is called _ground truth-SUM_.

Softmax Information Curve (SIC AUC) [26] This metric is free from the ground
truth annotation and measures how much the attributed features can influence
the model prediction, which is similar to the marginal contribution concept in
the Shapley theory. The better attribution should have a better focus on where
the model is truly looking. There are two metrics in different directions. The
first one gradually adds the feature values of the explicand to the
background. By sliding an attribution threshold, the feature with the largest
attribution is first added and the least the last. The better attribution
method should increase the model prediction more quickly, which can be
quantized by the area under the Softmax prediction curve w.r.t threshold, _SIC
AUC-ADD_. Conversely, another metric deletes the most important feature until
all the features are replaced by the background [19]. Similarly, the better
attribution method should decrease the model prediction quicker, where the AUC
w.r.t. threshold is called _SIC AUC-DEL_.

Table III summarizes four metrics of three real-world datasets, where IG2
significantly outperforms other methods in general.

###  6.3 Image classification explanation

####  6.3.1 Dataset

We validate IG2 on a standard image classification dataset, ImageNet. We take
the explained images from the ILSVRC [21] subset (1k classes) with the ground
truth annotations. We use the pre-trained classifier of Inception-v3 [22] with
input size 299$\times$299\. We only consider the images that are predicted as
one of the top 5 classes by the Inception-v3 classifier.

####  6.3.2 Attributions

Fig. 7 shows ImageNet images explained with four IG-based methods and two
Shapley value methods, where IG2 generally outperforms previous techniques.
Whereas guided IG efficiently reduces the noises by constructing the path that
avoids the saturated areas (discussed in Section 4.1), it gives incomplete
attributions due to the zero baseline, which is also a drawback of Vanilla IG
and KernelSHAP. This problem is especially evident in the images with black
subjects (images #1, #2, #4, and #5 in Fig. 7).

Expected IG and DeepSHAP achieve similar attribution results. They mitigate
the incomplete attribution problem by introducing informative baselines
providing more attributions on image subjects, but it still suffers from the
undesirable noise problem. This is especially obvious in the images with
interference objects (e.g., the dog in #6[basketball] and the player in
#7[sax]). Expected IG and DeepSHAP will incorrectly highlight these objects,
which are irrelevant to the explicand class label.

IG2 combines two advantages of Guided IG and Expected IG, perspectively
attributed to two techniques, GradPath and GradCF:

Less noise by GradPath: As discussed in Section 4.1, the integration path of
IG2 successfully mitigates the saturation effects. Image attributions in Fig.
7 also validate this superiority. IG2 provides significantly less noisy
attributions (less noise on background or irrelevant objects) over IG and
Expected IG that use the straight-line path. Compared with Guided IG, IG2 is
competitive and slightly better on some samples (e.g., image #7).

More complete attribution by GradCF: As discussed in Section 4.2, the
explicand-specific GradCF of IG2 can highlight the critical features that
distinguish the explicand from the counterfactual reference. As for images
from ImageNet, the critical areas should be the subjects of the explicand
label. Appendix Fig. B.6 shows the difference between the explicand and GradCF
(i.e., GradCFE). Based on this counterfactual contrast, IG2 attributions
highlight the critical features more completely than IG, Guided IG, and even
Expected IG.

Table III with quantitative metrics shows our proposal achieves the best
performances. Despite the incomplete attributions, the features highlighted by
Expected IG are enough to make the classifier give incorrect predictions. This
is why Expected IG achieves comparable SIC AUC-DEL value to IG2, but got much
worse performances on SIC AUC-ADD and ground truth metrics, which are more
dependent on attribution completeness. Since the superpixel technique makes
the high attributions concentrated on small areas, KernelSHAP achieves
relatively good performance on metrics about ground truth annotation, but its
SIC AUC metrics are not desirable.

###  6.4 Question classification explanation

####  6.4.1 Dataset

In the field of natural language processing (NLP), question answering is an
important task. Question classification can categorize the questions into
different semantic classes (whether the question is about location, person or
numeric information, etc.), which can impose constraints on potential answers.
For instance, the question–“ _Where did guinea pigs originate?_ ” should be
classified as having the answer type [location].

We use TREC question dataset [52] involving six semantic classes and train a
CNN-based classifier (TextCNN) [53]. We attribute word-level features in order
to seek the trigger words that contribute most to the answer type.

####  6.4.2 Attributions

Fig. 8 lists questions sampled from five classes from TREC dataset with word
attributions by IG2 and IG. IG uses the all-zero embedding vector as the
baseline. Compared to IG, the trigger words highlighted by IG2 are more
consistent with human grammatical perception. We summarize two advantages of
IG2 over IG.

Less attributions on weak interrogative words: Some initial interrogative
words are strongly associated with the question types, e.g., “ _where_ ”
indicates the question for [location] and “ _who_ ” indicates [human] (see
questions #2 and #7 in Fig. 8). In this case, these interrogative words should
be strongly attributed.

On the other hand, some interrogative words are weakly related. For instance,
“ _what_ ” and “ _which_ ” may indicate almost all the question types
(questions #1, #3, #4, #5, #8, and #9). Word “ _how_ ” itself is ambiguous,
which becomes a trigger phrase only when combined with other words (#6 and
#10). These weakly related interrogative words should be less attributed.

Shown in Fig. 8, Vanilla IG strongly attributes all the interrogative words,
whereas IG2 precisely attributes different interrogative words. IG2 keeps
large attributions on strong interrogative words (“where” and “who” in
questions #2 and #7), and provides much less attributions on weak
interrogative words (the remaining questions).

More attributions on critical phrases: Compared with IG, attributions of IG2
concentrate more on the critical phrases, such as, “ _drug_ ” and “ _name
revolt_ ” for [entity] (questions #3 and #4), “ _how do_ ” for
#6[description], “ _sought to_ ” for #7[human], and “ _lengths_ ” for
#9[numeric].

Table III validates that word attributions of IG2 are more consistent with the
model behavior than other methods. Since the number of input features in TREC
dataset is small (maximum 37 words), sampling-based KernelSHAP is competitive
with other path methods.

Figure 8: Word attributions for questions from TREC dataset. IG2 is compared
with Vanilla IG. The color depth indicates attribution strength, and the color
type indicates the attribution direction (red is negative, and green is
positive). The predicted classes are listed in the top of each question. The
complete attributions of all methods are reported in Appendix Fig. B.8. Figure
9: Feature attributions on wafer maps with eight different failure patterns.
The first row shows the original anomaly wafer maps with the red ground truth
annotation based on expert knowledge.

###  6.5 Wafer map failure pattern explanation

####  6.5.1 Dataset

Wafer map analysis is critical in daily semiconductor manufacturing
operations. Wafer maps provide visual details that are crucial for identifying
the stage of manufacturing at which wafer pattern failure occurs. Instead of
manual work, automatically identifying different types of wafer map failure
patterns can effectively improve the efficiency of the semiconductor
manufacturing process.

The explanation of classification deep neural network for wafer map failure
pattern can determine which parts (pixels) of the wafer maps are the cause
that leads to the failure. This explanation enhances the model’s ability to
automatically identify the cause of the anomaly wafer maps rather than only
recognizing the failure types.

Specifically, we use WM-811K dataset [25], which is the largest known wafer
map dataset available to the public. We use a subset of the whole dataset for
training the classification model based on convolution neural networks (CNN),
achieving a classification accuracy greater than 98.5% on both train and test
sets. The implementation detail of the classification model and sampled
instances from WM-811K are provided in Appendix B.5.

####  6.5.2 Attributions

First of all, Fig. 9 directly compares the different attributions on eight
samples with different patterns in WM-811K dataset. Compared to naive gradient
methods, integrated gradients significantly improves feature attribution.
Still, the Vanilla IG fails to completely highlight the failure patterns (as
the red ground truth) and suffers from the noise problem. Though Guided IG
efficiently reduces the noise on the irrelevant features by the designed path,
its attribution is still incomplete, caused by the arbitrary baseline
containing less counterfactual information. Expected IG and DeepSHAP solve
this by using the informative baselines over the data distribution of
[nonpattern] instances, but its straight-line path still introduces some
noises (especially on the circle edges).

Our GradCF solves the inaccurate counterfactual information problem in the
existing baselines, shown by GradCFE. It highlights the features contributing
to the model representation difference between the explicand and reference.
However, it simultaneously accumulates lots of undesirable noises on
irrelevant features. IG2 successfully solves this side effect by incorporating
the gradient of explicand’s prediction, which significantly reduces the noise
attributions by filtering out the features that have less contribution to the
output of explicand’s class.

Overall, compared with the advanced existing attribution methods, IG2 provides
less noisy attributions that are most consistent with expert knowledge and
human intuition. Moreover, Table III reports quantitative metrics to evaluate
the attributions. The results show that IG2 generally outperforms other
advanced path attribution methods and KernelSHAP. Our proposal significantly
improves the Vanilla IG’s performance and is competitive with Expected IG and
DeepSHAP.

Figure 10: Feature attributions on CelebA face attributes. The explained
labels appear at the top of each explicand. The critical facial areas
associated with the explained label are marked in red dashed lines. The
feature attributions are shown as the heatmaps on the images, where warmer
colors (reds and yellows) denote areas of higher importance.

###  6.6 Face attribute classification explanation

####  6.6.1 Dataset

Each face image in CelebA [54] has 40 binary face attribute labels, indicating
the presence or absence of specific facial attributes like smiling, wearing
earrings, or having a mustache. We train the face attribute classification
model on the CelebA dataset, based on MobileNet-v2 [55] with 40 output nodes
corresponding to each face attribute. The classification accuracy for each
face attributes are reported in Appendix B.6.

For multi-label classifier, we separately explains each output label, i.e.,
one face attribute at a time [16]. We use the counterfactual references that
are most relevant to the explicand, i.e., the faces with labels differ in the
explained attribute but are closet in other face attributes. The effect of
references on CelebA will be analyzed in Section 7.1.4.

####  6.6.2 Attributions

Fig. 10 showcases the feature attributions on the CelebA face images. The
effectiveness of a feature attribution method is determined by its ability to
accurately emphasize the image region associated with the explained label (as
indicated by red dashed line areas in Fig. 10). Using face image with label
[Black_Hair] as a case study, IG2 method demonstrates a more focused
attribution towards the hair region, with less noises than other methods.
Conversely, Vanilla IG and Guided IG encounter issues with black baselines in
face images, mistakenly ignoring black pixels. Although Vanilla Gradient and
Expected IG methods show relatively good performance on CelebA datasets, they
still suffer from the saturation effect, leading to undesirable attributions
on the irrelevant pixels, such as the image background.

Overall, the attributions by IG2 method are more precisely aligned with the
facial regions that are relevant to the explained labels, whereas other
methods tend to produce noisier and less accurate attributions on the critical
facial regions.

###  6.7 Ablation study

Furthermore, based on the wafer map dataset, we studied the impact of GradCF
and GradPath independently as the baseline and integration path for path
methods. Table IV reports three different baselines (black, train data, and
GradCF) under three different paths (straight-line, Guided IG’s, and
GradPath). As a baseline under the straight-line path and Guided-IG’s path,
GradCF achieves the best performance of most metrics compared with the other
two. On the other hand, GradPath outperforms than other two paths with the
GradCF baseline.

In general, we can conclude that: (1) GradCF is a good baseline for path
attribution, even independently combined with other paths. (2) GradPath
outperforms other paths on the GradCF baseline, which accomplishes IG2.

TABLE IV: Ablation study of GradCF and GradPath

|  | Ground truth | SIC AUC
---|---|---|---
Paths | Baselines | AUC $\uparrow$ | SUM $\uparrow$ | ADD $\uparrow$ | DEL $\downarrow$
Straight-line | Black | 0.732 | 0.342 | 0.829 | 0.061
Train data | 0.810 | 0.445 | 0.777 | 0.053
GradCF | 0.845 | 0.510 | 0.842 | 0.041
Guided-IG’s | Black | 0.758 | 0.450 | 0.789 | 0.050
Train data | 0.804 | 0.493 | 0.775 | 0.043
GradCF | 0.781 | 0.472 | 0.819 | 0.037
GradPath (IG2) |  GradCF* | 0.849 | 0.551 | 0.898 | 0.036

  * *

The baseline of GradPath is iteratively synthesized, so only GradCF is
available for GradPath.

##  7 Implementation details

We discuss the details of IG2, including the reference, step, normalization,
similarity measure, and computational cost. We provide in-depth analyses of
different choices on these hyper-parameters. Some supplementary experiments
are reported in Appendix A.

###  7.1 Reference

The choice of counterfactual reference is a major hyper-parameter of IG2.
Specifically, IG2 attributions are sensitive to the _category_ of
counterfactual reference (while relatively insensitive to different samples of
the same category).

Choice of reference category: For the classification tasks on different
datasets, the ways to sample references are also different:

  * •

Anomaly classification: For the dataset consisting of anomaly and normal
samples (e.g., wafer map failure patterns), it is natural to use the samples
of the normal category as the references for the anomaly explicand rather than
other anomaly categories.

  * •

General classification: Most datasets do not have such a natural category for
reference, such as ImageNet, TREC, etc. Without loss of generality, we
randomly sample the references from the different categories. Empirically, we
recommend uniformly sampling references from more categories and a few (1 or
2) samples per category.

  * •

Tricks for denoising: During experiments, we find using the references of
categories that are closely relevant to the explicand can reduce the noises in
attributions, but at the cost of losing completeness. This will be beneficial
for the explicands with interference terms, such as some ImageNet samples
(Section 7.1.2) and multi-label CelebA samples (Section 7.1.4).

Notably, without loss of generality, we take the second one (without denoising
tricks) as the default reference choice strategy. The effect of reference
category on IG2 attributions are discussed on four datasets, MNIST (Fig. 6),
ImageNet (Section 7.1.2), TREC (Section 7.1.3), and CelebA (Section 7.1.4).

Figure 11: Feature attributions w.r.t. different references. (a) Explained
image of class [sax] with interference object “player”. (b) Explained image of
class [Doberman] on pure background.

####  7.1.1 Expected IG2 over references

To eliminate the influence of the reference choice and reduce noise, we use
the expectation of IG2 as the final attribution. The Expected IG2 is averaged
over the references sampled from counterfactual categories, which is defined
as:

$\displaystyle\phi^{IG^{2}}_{i}=\mathbb{E}_{x^{r}\sim D^{r}}\
\phi^{IG^{2}}_{i}(x_{r}),$ (19)

where $D^{r}$ is the data distribution of the counterfactual categories to the
explicand and $\phi^{IG^{2}}_{i}(x_{r})$ is calculated by Eq. 3 as a function
of $x_{r}$. In the following, we make no distinction between IG2 and Expected
IG2, and use Expected IG2 as the practice for experiments.

TABLE V: Evaluation of IG2 reference choice strategies on ImageNet

|  | Ground truth | SIC AUC
---|---|---|---
Strategy | Set of explicand | AUC $\uparrow$ | SUM $\uparrow$ | ADD $\uparrow$ | DEL $\downarrow$
Relevant | W/ interference | 0.905 | 0.345 | 0.901 | 0.084
Pure background | 0.755 | 0.633 | 0.537 | 0.119
Whole | 0.798 | 0.551 | 0.641 | 0.109
Irrelevant | W/ interference | 0.864 | 0.273 | 0.876 | 0.086
Pure background | 0.781 | 0.613 | 0.568 | 0.126
Whole* | 0.805 | 0.516 | 0.656 | 0.115

  * *

The strategy reported in Table III.

####  7.1.2 References of ImageNet

During experiments of ImageNet, we find that the category of reference can
influence the IG2 attributions. We categorize the references into two types by
their labels: _relevant_ references and _irrelevant_ references. _Relevant_
references are the samples of categories that are closely related to the
explicand, e.g., the classes in model top 3 predictions, and _irrelevant_
references are the samples of other classes.

Generally, we find that only using _relevant_ references will lead to more
concentrated (less noisy) but less complete attributions. Contrarily, the
_irrelevant_ references can ensure the complete attribution covering the whole
image subject but at the cost of introducing more noise. Fig. 11 and Appendix
Fig. B.7 intuitively compares the IG2 attributions w.r.t. different
references. Moreover, we propose a trick to solve this trade-off in the choice
of references.

Trick of reference choice: Empirically, the explicands with interference
objects will be beneficial from the _relevant_ references (see Fig. 11a and
Appendix Fig. B.7a), the interference objects of which will be less
attributed. Contrarily, the image with the explained subject on the pure
background will be beneficial from the _irrelevant_ references (see Fig. 11b
and Appendix Fig. B.7b), where attributions on subjects will be more complete.
This rule allows us to sample the references from particular categories to
improve the attribution quality.

This trick is consistent with the intuition of counterfactual explanation:
_the contrast between ambiguous (hard-to-identify) classes will highlight the
most critical features of explicands._ Notably, this trick is only necessary
for some explicands in specific datasets. In more general cases, we just need
to sample the references uniformly from the counterfactual classes.

We conduct a quantitative evaluation for different reference choice strategies
on the ImageNet dataset, the results of which are reported in Table V. We
split the explained samples into two subsets: One subset is with interference
objects, and another one is on the pure background. We consider two
strategies: “ _Relevant_ ” selects references from the relevant categories of
the top 4 predictions; “ _Irrelevant_ ” randomly and uniformly samples
references from other categories.

By the denoising trick (using relevant references), the attributions are more
concentrated, which leads to higher _ground truth-SUM_ and _SIC AUC-DEL_ for
all images. For the interfered images, all four metrics are significantly
improved by this strategy. Since images on the pure background account for the
majority in ImageNet ($\sim 72\%$ in our test set), the two reference choice
strategies are competitive on the whole test set. Without loss of generality,
on ImageNet dataset, IG2 uses the strategy that randomly samples the
references by default.

Figure 12: Word attributions w.r.t. different references, for the question “
_What was the name of the Protestant revolt against the supremacy of the
Pope?_ ” of class [entity]. REFs #1 to #8 are used for calculating the
Expected IG2 reported in Fig. 8 #4.  Figure 13: (a) Curves of attribution
metrics w.r.t. CelebA label distance between reference and explicand. (b) A
CelebA example attributed by IG2 of different references (explained label:
[Wearing_Earrings]).

####  7.1.3 References of TREC

Fig. 12 lists IG2 attributions w.r.t. different references, for the question
#4 in Fig. 8. Fig. 12 intuitively shows that references of different
categories provide different counterfactual contrast, which makes IG2
explanation more in line with the human-authored grammar rules.

Based on grammar rules, the subject phrase of the question #4 (of class
[entity]) should be “name of revolt”. The two most attributed words by Vanilla
IG are “what” and “against” (see REF #0 in Fig. 12), which are inaccurate
since they do not determine the class [entity].

IG2 uses references #1 to #8, where some insights can be found into the
counterfactual explanation:

  * •

Compared to IG, all the references make IG2 have much less attribution on
“what”, which is irrelevant to the class [entity]. Notably, even if the
interrogative word of reference is not “what”, IG2 attribution is still
significantly reduced (see REFs #4 to #8).

  * •

Almost all the counterfactual references contrastively highlight “ _revolt_ ”,
since modifying this word is the fastest way to turn the explicand into
another class 666For instance, replacing “ _revolt_ ” with “ _country_ ” turns
the question into class [location], and replacing with “ _leader_ ” turns into
[human].. Reference #3 is an exception, an interpretation of which is: if we
only modify the word “ _revolt_ ”, the sentence will hardly become the class
[description]; but if removing the phrase “ _the name of_ ”, the explained
question will be similar to reference #3.

  * •

The word “ _name_ ” is also more highlighted by IG2, but the references of
classes [human] and [location] do not provide this contrast. This is because
the word “ _name_ ” is also related to the classes [human] and [location]
(e.g., “name of lawyer” and “name of country”).

Finally, REFs #9 and #10 in Fig. 12 list the attributions w.r.t. the
references of the same class, where the word “ _revolt_ ” is highlighted in
the opposite direction. The attributions w.r.t. non-counterfactual references
are confusing, which is why we only consider the references of different
classes when we compute the Expected IG2.

####  7.1.4 Reference of CelebA

We leverage a multi-label dataset to quantitatively evaluate the choice of
references. Thus we can use the $\ell_{1}$ distance of label vectors to
measure the similarity between references and explicand, and then build the
quantitative correlation between the reference categories and feature
Attributions.

Fig. 13a reports the feature attribution metric curves w.r.t. label distances,
averaged over 300 CelebA samples. Fig. 13b displays an explicand with the
explained label [Wearing_Earrings], showing its attributions w.r.t. different
references. The non-counterfactual references have the identical explained
label to the explicand, and _vice versa_.

The results show, the similar counterfactual references (with small label
distances) lead to better feature attributions (e.g., highlighting the earring
pixels). Higher label distances result in a decline in the attribution
metrics. The non-counterfactual references cannot contrast the explained
label, which leads to the worst feature attributions.

Based on the results, we make two conclusions for the reference choice on the
multi-label dataset: (1) References should be counterfactual on the explained
label; (2) Using relevant references will benefit the feature attributions of
multi-label samples. The second conclusion is consistent with the above
denoising tricks for images with interference objects. In the multi-label
classification, the not-explained labels (e.g., the face attributes other than
[Wearing_Earrings]) can be regarded as the interference objects, so relevant
references will make the feature attributions less noisy and more concentrated
on the explained image parts.

###  7.2 Step size and number

Appendix Fig. A.1 uses an ImageNet example to illustrate the effect of step
size and number on IG2 attribution. Appendix Fig. A.1a shows the objective
loss (Eq. 4) curves during the optimization for GradPath, and Appendix Fig.
A.1b shows IG2 attributions w.r.t. different step sizes and numbers. The step
sizes and numbers chosen for different datasets are reported in Appendix Table
A.1.

Step size can affect the IG2 attributions. There is a trade-off issue on step
size: small step sizes tend to result in less noisy but incomplete
attributions, whereas large step sizes result in complete but noisy
attributions. This is intuitive: when the total magnitude is more limited, the
perturbation will be concentrated on a few more important features. In
practice, we will choose a moderate step size that can well balance these two
sides.

Step number is not a critical hyper-parameter. Appendix Fig. A.1 shows that
IG2 attribution does not substantially change when objective loss approaches
convergence. Hence, we set the step number to a relatively large value that
can allow optimizations of most explicands to be converged.

###  7.3 Representation distance measure

IG2 constructs GradPath and searches GradCF by minimizing the distance between
model representations of reference and explicand. Eq. 4 shows this
optimization objective, where the distance between two vectors is based on the
Euclidean measure. The usage of Euclidean distance is inspired by the feature
matching trick in GAN training [56] and the adversarial robustness work [48].
Besides, the Cosine similarity and $\ell_{1}$ norm are two common measures for
the vector distance. We conduct an ablation study to compare the different
distance measures in IG2.

Appendix Table A.2 reports the performance gap between Cosine similarity and
$\ell_{1}$ norm to Euclidean distance, and Appendix Fig. A.4 displays ImageNet
samples attributed by three different measures. Based on the quantitative and
qualitative results, we argue that Euclidean distance is not significantly
better than Cosine similarity, while $\ell_{1}$ norm is not a good choice for
IG2 attributions. Empirically, IG2 uses the Euclidean distance by default,
achieving slightly higher evaluation metrics in experiments.

###  7.4 Computational cost

We analyze the computational cost of IG2. The major computational cost of path
methods is gradient calculation. Since GradPath requires the same gradient
calculation times as the gradient integration, IG2 requires at least twice the
computational cost over other methods. On the other hand, IG2 commonly
requires about 10 references to get the desired performance, while other
methods only require one. Hence, the number of gradient calculations in IG2 is
about 20 times more than other methods.

In practice, we can reduce the running time by calculating the gradients of
different references in one batch. The practical running time of IG2 is about
10 to 20 times that of (Expected) IG and about 3 times that of Guided IG.
Appendix Table A.3 reports the average explanation time per sample of
different methods. Despite the increased computational cost, the running time
of IG2 is still practically feasible. Compared with the sampling-based
KernelSHAP method, even with superpixel techniques, IG2 is significantly
faster on the high-dimensional datasets.

Besides above hyperparameters, the discussions about normalization and
representation layer are presented in Appendix A.2 and A.3

##  8 Conclusion

This paper proposes a novel feature attribution method, Iterative Gradient
path Integrated Gradients (IG2), which simultaneously incorporates two
gradients, the explicand’s and counterfactual. IG2 proposes two novel
essential components of path methods: baseline (GradCF) and integration path
(GradPath). GradPath incorporates the counterfactual gradient into its
direction and implicitly solves the saturation effects and noisy attributions.
GradCF is the first baseline that contains the information of both model and
explicand, avoiding the previous arbitrary baseline choice.

We contrast our work with path methods and the works in the field of
counterfactual explanation and adversarial learning. We argue that our work
can be regarded as a generalization of Guided IG. We intuitively interpret our
proposal and justify the desirable axioms of IG2 in theory. The effectiveness
of IG2 is verified by an XAI benchmark and multiple real-world datasets from
diverse domains with qualitative and quantitative results. Moreover, the
ablation study reveals that GradPath and GradCF individually improve the
attribution of IG methods, harmonized by IG2.

## References

  * [1] X.-H. Li, C. C. Cao, Y. Shi, W. Bai, H. Gao, L. Qiu, C. Wang, Y. Gao, S. Zhang, X. Xue, and L. Chen, “A survey of data-driven and knowledge-aware explainable ai,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 34, no. 1, pp. 29–49, 2022.
  * [2] Y. Shen, C. Yang, X. Tang, and B. Zhou, “Interfacegan: Interpreting the disentangled face representation learned by gans,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 4, pp. 2004–2018, 2022.
  * [3] S. A. Bargal, A. Zunino, V. Petsiuk, J. Zhang, K. Saenko, V. Murino, and S. Sclaroff, “Guided zoom: Zooming into network evidence to refine fine-grained model decisions,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 11, pp. 4196–4202, 2021.
  * [4] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” _arXiv preprint arXiv:1312.6034_ , 2013.
  * [5] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 618–626.
  * [6] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” 2014.
  * [7] A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” in _Proceedings of the 34th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 06–11 Aug 2017, pp. 3145–3153.
  * [8] A.-K. Dombrowski, M. Alber, C. J. Anders, M. Ackermann, K.-R. Müller, and P. Kessel, “Explanations can be manipulated and geometry is to blame,” 2019\.
  * [9] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in _International conference on machine learning_. PMLR, 2017, pp. 3319–3328.
  * [10] P. Lipton, “Contrastive explanation,” _Royal Institute of Philosophy Supplement_ , vol. 27, p. 247–266, 1990.
  * [11] S. Wachter, B. Mittelstadt, and C. Russell, “Counterfactual explanations without opening the black box: Automated decisions and the gdpr,” 2017.
  * [12] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [13] P. Wang and N. Vasconcelos, “Scout: Self-aware discriminant counterfactual explanations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [14] R. Chen, H. Zhang, S. Liang, J. Li, and X. Cao, “Less is more: Fewer interpretable region via submodular subset selection,” in _International Conference on Learning Representations_ , 2024.
  * [15] A. Dhurandhar, P. Chen, R. Luss, C. Tu, P. Ting, K. Shanmugam, and P. Das, “Explanations based on the missing: Towards contrastive explanations with pertinent negatives,” _CoRR_ , vol. abs/1802.07623, 2018.
  * [16] R. Chen, J. Li, H. Zhang, C. Sheng, L. Liu, and X. Cao, “Sim2word: Explaining similarity with representative attribute words via counterfactual explanations,” _ACM Trans. Multimedia Comput. Commun. Appl._ , vol. 19, no. 6, jul 2023.
  * [17] M. Sundararajan and A. Najmi, “The many shapley values for model explanation,” in _International conference on machine learning_. PMLR, 2020, pp. 9269–9278.
  * [18] V. Miglani, N. Kokhlikyan, B. Alsallakh, M. Martin, and O. Reblitz-Richardson, “Investigating saturation effects in integrated gradients,” _CoRR_ , vol. abs/2010.12697, 2020.
  * [19] G. Erion, J. D. Janizek, P. Sturmfels, S. M. Lundberg, and S.-I. Lee, “Improving performance of deep learning models with axiomatic attribution priors and expected gradients,” _Nature machine intelligence_ , vol. 3, no. 7, pp. 620–631, 2021.
  * [20] P. Sturmfels, S. Lundberg, and S.-I. Lee, “Visualizing the impact of feature attribution baselines,” _Distill_ , 2020, https://distill.pub/2020/attribution-baselines.
  * [21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” _International Journal of Computer Vision (IJCV)_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [22] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” _CoRR_ , vol. abs/1512.00567, 2015.
  * [23] A. Kapishnikov, S. Venugopalan, B. Avci, B. Wedin, M. Terry, and T. Bolukbasi, “Guided integrated gradients: An adaptive path method for removing noise,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5050–5058.
  * [24] S. Xu, S. Venugopalan, and M. Sundararajan, “Attribution in scale and space,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 9677–9686.
  * [25] M.-J. Wu, J.-S. R. Jang, and J.-L. Chen, “Wafer map failure pattern recognition and similarity ranking for large-scale data sets,” _IEEE Transactions on Semiconductor Manufacturing_ , vol. 28, no. 1, pp. 1–12, 2015\.
  * [26] A. Kapishnikov, T. Bolukbasi, F. Viegas, and M. Terry, “Xrai: Better attributions through regions,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. Los Alamitos, CA, USA: IEEE Computer Society, nov 2019, pp. 4947–4956.
  * [27] D. Smilkov, N. Thorat, B. Kim, F. B. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” _CoRR_ , vol. abs/1706.03825, 2017.
  * [28] S. Verma, J. Dickerson, and K. Hines, “Counterfactual explanations for machine learning: A review,” _arXiv preprint arXiv:2010.10596_ , 2020.
  * [29] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _International Conference on Learning Representations_ , 2018.
  * [30] A. Ignatiev, N. Narodytska, and J. Marques-Silva, “On relating explanations and adversarial examples,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [31] E. J. Friedman, “Paths and consistency in additive cost sharing,” _International Journal of Game Theory_ , vol. 32, no. 4, pp. 501–518, 2004\.
  * [32] K. Abhishek and D. Kamath, “Attribution-based xai methods in computer vision: A review,” _arXiv preprint arXiv:2211.14736_ , 2022.
  * [33] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in _European conference on computer vision_. Springer, 2014, pp. 818–833.
  * [34] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2921–2929.
  * [35] M. T. Ribeiro, S. Singh, and C. Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [36] “A value for n-person games,” _Classics in game theory_ , vol. 69.
  * [37] Y. Matsui and T. Matsui, “Np-completeness for calculating power indices of weighted majority games,” _Theoretical Computer Science_ , vol. 263, no. 1-2, pp. 305–310, 2001.
  * [38] E. Strumbelj and I. Kononenko, “An efficient explanation of individual classifications using game theory,” _The Journal of Machine Learning Research_ , vol. 11, pp. 1–18, 2010.
  * [39] R. Mitchell, J. Cooper, E. Frank, and G. Holmes, “Sampling permutations for shapley value estimation,” 2022.
  * [40] J. Chen, L. Song, M. J. Wainwright, and M. I. Jordan, “L-shapley and c-shapley: Efficient model interpretation for structured data,” _arXiv preprint arXiv:1808.02610_ , 2018.
  * [41] G. Wang, Y.-N. Chuang, M. Du, F. Yang, Q. Zhou, P. Tripathi, X. Cai, and X. Hu, “Accelerating shapley explanation via contributive cooperator selection,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 22 576–22 590.
  * [42] M. Ancona, C. Oztireli, and M. Gross, “Explaining deep neural networks with a polynomial time algorithm for shapley value approximation,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 272–281.
  * [43] N. Jethani, M. Sudarshan, I. C. Covert, S.-I. Lee, and R. Ranganath, “Fastshap: Real-time shapley value estimation,” in _International Conference on Learning Representations_ , 2021.
  * [44] M. Ancona, E. Ceolini, C. Öztireli, and M. Gross, “Towards better understanding of gradient-based attribution methods for deep neural networks,” _arXiv preprint arXiv:1711.06104_ , 2017.
  * [45] N. Akhtar, M. Jalwana, M. Bennamoun, and A. S. Mian, “Attack to fool and explain deep networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2021.
  * [46] Y. Dong, F. Bao, H. Su, and J. Zhu, “Towards interpretable deep neural networks by leveraging adversarial examples,” _CoRR_ , vol. abs/1901.09035, 2019.
  * [47] M. Zhu, T. Chen, and Z. Wang, “Sparse and imperceptible adversarial attack via a homotopy algorithm,” 2021.
  * [48] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, “Adversarial examples are not bugs, they are features,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [49] C.-H. Lee, Z. Liu, L. Wu, and P. Luo, “Maskgan: Towards diverse and interactive facial image manipulation,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [50] Y. Liu, S. Khandagale, C. White, and W. Neiswanger, “Synthetic benchmarks for scientific research in explainable machine learning,” _CoRR_ , vol. abs/2106.12543, 2021.
  * [51] Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand, “What do different evaluation metrics tell us about saliency models?” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 41, no. 3, pp. 740–757, 2019.
  * [52] X. Li and D. Roth, “Learning question classifiers,” in _The 19th International Conference on Computational Linguistics_ , 2002.
  * [53] Y. Kim, “Convolutional neural networks for sentence classification,” _CoRR_ , vol. abs/1408.5882, 2014.
  * [54] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * [55] M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen, “Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation,” _CoRR_ , vol. abs/1801.04381, 2018.
  * [56] T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” _CoRR_ , vol. abs/1606.03498, 2016\.

##  Appendix A Implementation details

###  A.1 Step size and number

Fig. A.1a shows the objective loss (Eq. 4 in main paper) curves during the
optimization for GradPath, and Fig. A.1b shows IG2 attributions w.r.t.
different step sizes and numbers.

We choose a moderate step size that can deal well with the trade-off between
attribution noise and completeness. The specific hyper-parameters about step
size and number for different datasets are reported in Table A.1.

###  A.2 Normalization

We consider two normalization methods: $\ell_{2}$ norm and $\ell_{1}$ norm.
Given the gradient
$g_{\alpha}=\frac{\partial\|\tilde{f}(\gamma^{G}(\alpha))-\tilde{f}(\alpha)\|_{2}}{\partial\gamma^{G}(\alpha)}$
at step $\alpha$, we can have $W^{\ell_{2}}_{\alpha}$ and
$W^{\ell_{1}}_{\alpha}$:

$\ell_{2}$ norm is simple:

$\displaystyle W^{\ell_{2}}_{\alpha}=\|g_{\alpha}\|_{2},$ (20)

which is used in previous works about adversarial robustness [29, 48].

$\ell_{1}$ norm is more tricky. Following a sparse adversarial attack work
[47], an additional parameter $k$ that controls the sparsity of gradient on
each step is introduced. Specifically, we select the features $i$ with top $k$
maximal absolute gradient value and then normalize the sparse vector with sign
function:

$\displaystyle W^{\ell_{1}}_{\alpha}(i)=\begin{cases}|S|g(i),\
i\in\mathcal{S}=\underset{|A|=k}{\mathrm{argmax}\ \Sigma_{i\in A}|g(i)|}\\\
0\end{cases},$ (21)

where $S$ is the set contains the index of top $k$ maximal absolute value in
vector $g$ and the set size $|S|$ is the $\ell_{1}$ norm of signed vector.

Both $l_{1}$ and $l_{2}$ normalization imply the feature importance on each
step perturbation, the comparison of which is presented below. In practice, we
use more general $\ell_{2}$ normalization by default.

We compare $\ell_{2}$ and $\ell_{1}$ normalization for the gradient of
GradPath (i.e., $W$). $\ell_{1}$ normalization has an additional parameter to
control sparsity, which is presented by the percentile of the gradient vector.
Fig. A.2 plots the IG2 attributions on wafer map under different
normalizations. When the percentile is low (90), the attribution under
$\ell_{1}$ normalization resembles $\ell_{2}$. When the attribution sparsity
increases, the noises significantly drop, but some attributions on relevant
features also get very small, which especially damages the performance of
certain classes (e.g., Donut, Near-full, and Random). This can be regarded as
a precision and recall trade-off.

TABLE A.1: IG2 hyper-parameters for different datasets  Datasets | Input region | Step size | Steps | Ref.
---|---|---|---|---
ImageNet | [0,255] | 1024 | 500 | random (trick)
TREC | [-1.156, 1.094] | 0.01 | 1000 | random
wafer map | [0,1] | 0.02 | 2000 | the normal
Figure A.1: (a) Curves of loss w.r.t. step numbers for different step sizes.
(b) IG2 attributions w.r.t. step sizes and numbers. [Doberman] is explained
with reference [gorilla]. Figure A.2: IG2 attributions of $\ell_{1}$ and
$\ell_{2}$ normalization. Five sparsity percentile parameters (90, 95, 97, 98,
98.5) of $\ell_{1}$ normalization are compared.  Figure A.3: IG2 with the
different representation layer (fc1 used in the paper).

###  A.3 Representation layer

We compare the IG2 with the different representation layers of wafer map
classifier (Table B.5), which is shown in Fig. A.3. GradCFE gets less noise
using the deeper layer as representation, while IG2 is slightly affected by
the choice of representation layer.

###  A.4 Representation distance measure

We compare the representation distance measures of IG2. Fig. A.4 shows the
ImageNet sample attributions by three measures, as an intuitive illustration
of main paper Table A.2. Generally, IG2 with Euclidean distance and Cosine
similarity provide very similar feature attributions, while $\ell_{l}$ norm
will introduce unpleasant noises.

TABLE A.2: Evaluation of model representation distance measures (the gap to Euclidean distance) |  | Ground truth | SIC AUC
---|---|---|---
Datasets | Measures | AUC $\uparrow$ | SUM $\uparrow$ | ADD $\uparrow$ | DEL $\downarrow$
ImageNet | Cosine | $-0.055$ | $-0.021$ | $+0.018$ | $+0.047$
$\ell_{1}$ norm | $-0.065$ | $-0.027$ | $-0.028$ | $+0.003$
TREC | Cosine | — | $+0.009$ | $+0.007$
$\ell_{1}$ norm | $-0.010$ | $+0.012$
Wafer map | Cosine | $-0.024$ | $-0.019$ | $-0.010$ | $-0.002$
$\ell_{1}$ norm | $-0.072$ | $-0.037$ | $-0.029$ | $+0.016$
Figure A.4: IG2 attributions of Euclidean, Cosine, and $\ell_{1}$ norm
measures (Euclidean distance used in the paper).

###  A.5 Computational cost

Table A.3 reports the average computation time per explained sample. For a
fair comparison, all the methods run with the same step number and batch size
(if available). We run the experiments with 1 NVIDIA GeForce GTX 1080 Ti GPU.
The KernelSHAP applies the superpixel technique for ImageNet and wafer map
datasets.

TABLE A.3: Average explanation time per sample (sec.). The input feature numbers of KernelSHAP are reduced.  | ImageNet | TREC | wafermap
---|---|---|---
Gradient | 0.13 | 0.003 | 0.03
Vanilla IG | 8.14 | 0.26 | 0.89
Guided IG | 37.0 | 4.48 | 6.39
Expected IG | 8.34 | 0.49 | 1.25
DeepSHAP | 111.0 | 0.32 | 0.95
KernelSHAP | 3150.2 ($13\times 13$) | 1.8 | 75.0 ($28\times 28$)
IG2 | 108.2 | 9.80 | 18.3
TABLE A.4: The structure of MLP to be explained for the regression on the synthetic dataset from XAI-BENCH  Layer | Configuration
---|---
input |  $5$-Dimension vector
fc1 | fc5,64+BN+Tanh
fc2 | conv64,16+Tanh
fc3 | conv16,1
Figure A.5: Ablation study for critical areas in the explained digital 5 with
references of digitals 3, 6, 9. The titles of each subfigure report the
predicted class with Softmax output. The areas in the first figure of each
block (with red titles) are used in main paper Fig. 6.

##  Appendix B Datasets and explained models

###  B.1 MNIST classification

Fig. A.5 validates the most critical areas in digital 5 that distinguishes the
explicand to references of digitals 3, 6, 9. We compare the areas highlighted
by IG2 (the areas in the figures with red titles) and the areas of Expected IG
(the right upper corners). We study the importance of different areas by the
permutations: We first remove the right upper pixels of digital 5, which does
not change model prediction. If we fill the highlighted areas of IG2 with
pixels, the model predicts the modified digitals as the class labels of
references. Based on this, if we further remove the right upper corners of
these digitals, the impact on model predictions is not significant.

In sum, compared with the right upper corners of Expected IG, the areas
highlighted by IG2 are more critical for distinguishing the explicand to
references. Besides, we believe areas of IG2 are also closer to the intuition.

Figure B.6: GradCFE for the ImageNet images in main paper Fig. 7, showing the
general counterfactual directions of GradPath (from the explicand to its
GradCF). Figure B.7: Feature attributions with regard to different references,
as a supplement for main paper Fig. 11. (a) Explained image of class
[basketball] with interference object “dog”. (b) Explained image of class
[killer whale] on pure background.

###  B.2 Synthetic dataset in XAI-BENCH

We synthesize the dataset $x$ from the 5-dimension Gaussian distribution with
$\mathbf{\mu}=\vec{0}$ and $\mathbf{\Sigma}=I_{5}$. The additive piecewise
function for labels are [50]:

$\displaystyle\psi_{1}(x_{1})$ $\displaystyle=1\ \text{if}\ x_{1}\geq 0,\
\text{otherwise}\ -1,$ $\displaystyle\psi_{2}(x_{2})$
$\displaystyle=\begin{cases}-2&,\ x_{2}<-0.5\\\ -1&,\ -0.5\leq x_{2}<0\\\ 1&,\
0\leq x_{2}<0.5\\\ 2&,\ x_{2}\geq 0.5\end{cases},$
$\displaystyle\psi_{3}(x_{3})$ $\displaystyle=\lfloor 2cos(\pi x_{3})\rfloor,$
$\displaystyle\psi_{i}(x_{i})$ $\displaystyle=0,\ i=4,5,$ (22)

and we cut off the normalized function output with a threshold to obtain the
binary label:

$\displaystyle y=\begin{cases}1&,\ \text{if}\
\mathrm{Norm}[\Sigma_{i=1}^{5}\psi_{i}]>0\\\ 0&,\ \text{if}\
\mathrm{Norm}[\Sigma_{i=1}^{5}\psi_{i}]\leq 0\end{cases},$ (23)

where $\mathrm{Norm}[\ ]$ makes the mean value of outputs zero.

The structure Multi-Layer Perceptron (MLP) trained on the synthetic dataset is
reported in Table A.4.

Figure B.8: Word attributions of all compared methods for TREC dataset, as a
complement for main paper Fig. 8.

###  B.3 ImageNet classification

Fig. B.6 shows the GradCFE for the explicand displayed in main paper Fig. 7.
GradCFE presents the difference between the explicand and its baseline,
GradCF. Fig. B.6 illustrates that the integration path of IG2 is in the
direction of critical features, which can mitigate the saturation effects.

As a supplement for main paper Fig. 11, Fig. B.7 shows the feature
attributions with regard to different categories of references for two more
ImageNet samples. This supports the analyses of reference impact and choice
trick.

###  B.4 TREC classification

Fig. B.8 shows the complete word attribution results of all compared methods.
The results are consistent with the conclusions made in the main paper that
IG2 makes less attributions on weak interrogative words and more attributions
on critical phrases.

###  B.5 Wafermap failure pattern classification

Fig. B.9 plots the wafer maps sampled from WM-811K dataset [25], including
three images per class. To achieve the task of failure pattern classification,
we select a small subset of WM-811K dataset with a total of 19000 wafer maps.
The backbone of the classification neural network is based on convolution, the
detailed structure of which is reported in Table B.5.

Figure B.9: The sampled wafer map images from WM-811K dataset, including eight failure patterns and one without any pattern: [Center], [Donut], [Edge-Loc], [Edge-Ring], [Loc], [Near-full], [Random], [Scratch] and [None]. The failure cause position (pixels) are also annotated in the red border area (except [Random] and [None]).  TABLE B.5: The structure of CNN to be explained for wafer map failure pattern classification  Layer | Configuration
---|---
input |  $56\times 56\times 3$ RGB images
conv1 | conv3,32,3+ReLU+Pool
conv2 | conv32,64,3+ReLU+Pool
conv3 | conv64,128,3+ReLU+Pool
fc1 | fc8192,1250+ReLU
fc2 | fc1250,9+Softmax

###  B.6 CelebA face attribute classification

Face attribute classification in CelebA is a multi-label task. There are
totally 40 face attributes with binary labels. The classification accuracy for
each face attributes are reported in Table B.6.

TABLE B.6: Classification accuracy on each face attribute of CelebA  Attributes |  Acc. (%) | Attributes |  Acc. (%)
---|---|---|---
5_o_Clock_Shadow | 94.4 | Male | 97.9
Arched_Eyebrows | 83.4 | Mouth_Slightly_Open | 93.6
Attractive | 82.5 | Mustache | 96.9
Bags_Under_Eyes | 85.0 | Narrow_Eyes | 87.7
Bald | 98.9 | No_Beard | 96.0
Bangs | 95.9 | Oval_Face | 75.4
Big_Lips | 71.3 | Pale_Skin | 97.0
Big_Nose | 84.6 | Pointy_Nose | 77.1
Black_Hair | 89.0 | Receding_Hairline | 93.7
Blond_Hair | 95.6 | Rosy_Cheeks | 94.9
Blurry | 95.9 | Sideburns | 97.7
Brown_Hair | 88.2 | Smiling | 92.7
Bushy_Eyebrows | 92.6 | Straight_Hair | 83.4
Chubby | 95.3 | Wavy_Hair | 83.4
Double_Chin | 96.2 | Wearing_Earrings | 90.4
Eyeglasses | 99.6 | Wearing_Hat | 99.0
Goatee | 97.5 | Wearing_Lipstick | 93.7
Gray_Hair | 98.3 | Wearing_Necklace | 87.5
Heavy_Makeup | 91.3 | Wearing_Necktie | 96.7
High_Cheekbones | 86.8 | Young | 88.2
|  | Average | 91.0

For the ground truth in quantitative evaluation of celebA attribution (in main
paper Table III), we utilize the face parsing model pretrained on CelebAMask-
HQ [49] to split the face image into different facial components, such as
hair, nose and eyes. The ground truth is chosen as the facial components that
are most related to the explained face attribute. Table B.7 gives the
correspondences between explained face attributes (of CelebA) and parsed
facial components (of CelebAMask-HQ). In quantitative evaluation, we only
consider the face attributes that have explicit corresponding components.

TABLE B.7: Correspondences between explained face attributes and parsed facial components Attributes | Comp. | Attributes | Comp.
---|---|---|---
5_o_Clock_Shadow | - | Male | -
Arched_Eyebrows | brows | Mouth_Slightly_Open | mouth
Attractive | - | Mustache | -
Bags_Under_Eyes | eyes | Narrow_Eyes | eyes
Bald | hair | No_Beard | -
Bangs | hair | Oval_Face | -
Big_Lips | lips | Pale_Skin | skin
Big_Nose | nose | Pointy_Nose | nose
Black_Hair | hair | Receding_Hairline | hair
Blond_Hair | hair | Rosy_Cheeks | -
Blurry | - | Sideburns | hair
Brown_Hair | hair | Smiling | mouth
Bushy_Eyebrows | brows | Straight_Hair | hair
Chubby | - | Wavy_Hair | hair
Double_Chin | - | Wearing_Earrings | ear_r
Eyeglasses | eye_g | Wearing_Hat | hat
Goatee | - | Wearing_Lipstick | lips
Gray_Hair | hair | Wearing_Necklace | neck_l
Heavy_Makeup | - | Wearing_Necktie | cloth
High_Cheekbones | - | Young | -