On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Neurons Neural networks Kernel functions Sensory perception Algorithms Coding mechanisms Imaging techniques Vision
Grégoire Montavon Frederick Klauschen Klaus-Robert Müller Wojciech Samek Sebastian Bach Alexander Binder
Understanding and interpreting classification decisions of automated image classification systems is of high value in many applications, as it allows to verify the reasoning of the system and provides additional information to the human expert. Although machine learning methods are solving very successfully a plethora of tasks, they have in most cases the disadvantage of acting as a black box, not providing any information about what made them arrive at a particular decision. This work proposes a general solution to the problem of understanding classification decisions by pixel-wise decomposition of nonlinear classifiers. We introduce a methodology that allows to visualize the contributions of single pixels to predictions for kernel-based classifiers over Bag of Words features and for multilayered neural networks. These pixel contributions can be visualized as heatmaps and are provided to a human expert who can intuitively not only verify the validity of the classification decision, but also focus further analysis on regions of potential interest. We evaluate our method for classifiers trained on PASCAL VOC 2009 images, synthetic image data containing geometric shapes, the MNIST handwritten digits data set and for the pre-trained ImageNet model available as part of the Caffe open source package.
/resource/img/favicon.ico
Layer-wise relevance propagation

We will introduce layer-wise relevance propagation as a concept defined by a set of constraints. Any solution satisfying the constraints will be considered to follow the concept of layer-wise relevance propagation. In later sections we will then derive solutions for two particular classifier architectures and evaluate these solutions experimentally for their meaningfulness. Layer-wise relevance propagation in its general form assumes that the classifier can be decomposed into several layers of computation. Such layers can be parts of the feature extraction from the image or parts of a classification algorithm run on the computed features. As shown later, this is possible for Bag of Words features with non-linear SVMs as well as for neural networks.

The first layer are the inputs, the pixels of the image, the last layer is the real-valued prediction output of the classifier f. The l-th layer is modeled as a vector with dimensionality V(l). Layer-wise relevance propagation assumes that we have a Relevance score for each dimension of the vector z at layer l + 1. The idea is to find a Relevance score for each dimension of the vector z at the next layer l which is closer to the input layer such that the following equation holds. (2)

Iterating Eq (2) from the last layer which is the classifier output f(x) down to the input layer x consisting of image pixels then yields the desired Eq (1). The Relevance for the input layer will serve as the desired sum decomposition in Eq (1). In the following we will derive further constraints beyond Eqs (1) and (2) and motivate them by examples. As we will show now, a decomposition satisfying Eq (2) per se is neither unique, nor it is guaranteed that it yields a meaningful interpretation of the classifier prediction.

We give here a simple counterexample. Suppose we have one layer. The inputs are x ∈ ℝV. We use a linear classifier with some arbitrary and dimension-specific feature space mapping ϕd and a bias b (3) Let us define the relevance for the second layer trivially as . Then, one possible layer-wise relevance propagation formula would be to define the relevance R(1) for the inputs x as (4) This clearly satisfies Eqs (1) and (2), however the Relevances R(1)(xd) of all input dimensions have the same sign as the prediction f(x). In terms of pixel-wise decomposition interpretation, all inputs point towards the presence of a structure if f(x) > 0 and towards the absence of a structure if f(x) < 0. This is for many classification problems not a realistic interpretation.

Let us discuss a more meaningful way of defining layer-wise relevance propagation. For this example we define (5) Then, the relevance of a feature dimension xd depends on the sign of the term in Eq (5). This is for many classification problems a more plausible interpretation. This second example shows that the layer-wise relevance propagation is able to deal with non-linearities such as the feature space mapping ϕd to some extent and how an example of layer-wise relevance propagation satisfying Formula (2) may look like in practice. Note that no regularity assumption on the feature space mapping ϕd is required here at all, it could be even non-continuous, or non-measurable under the Lebesgue measure. The underlying Formula (2) can be interpreted as a conservation law for the relevance R in between layers of the feature processing.

The above example gives furthermore an intuition about what relevance R is, namely, the local contribution to the prediction function f(x). In that sense the relevance of the output layer is the prediction itself f(x). This first example shows what one could expect as a decomposition for the linear case. The linear case is not a novelty, however, it provides a first intuition.

We give a second, more graphic and non-linear, example. The left panel of Fig 2 shows a neural network-shaped classifier with neurons and weights wij on connections between neurons. Each neuron i has an output ai from an activation function.

wij are connection weights. ai is the activation of neuron i. Right: The neural network-shaped classifier during layer-wise relevance computation time. is the relevance of neuron i which is to be computed. In order to facilitate the computation of we introduce messages . are messages which need to be computed such that the layer-wise relevance in Eq (2) is conserved. The messages are sent from a neuron i to its input neurons j via the connections used for classification, e.g. 2 is an input neuron for neurons 4, 5, 6. Neuron 3 is an input neuron for 5, 6. Neurons 4, 5, 6 are the input for neuron 7.

https://doi.org/10.1371/journal.pone.0130140.g002

The top layer consists of one output neuron, indexed by 7. For each neuron i we would like to compute a relevance Ri. We initialize the top layer relevance as the function value, thus . Layer-wise relevance propagation in Eq (2) requires now to hold (6) (7) We will make two assumptions for this example. Firstly, we express the layer-wise relevance in terms of messages between neurons i and j which can be sent along each connection. The messages are, however, directed from a neuron towards its input neurons, in contrast to what happens at prediction time, as shown in the right panel of Fig 2. Secondly, we define the relevance of any neuron except neuron 7 as the sum of incoming messages: (8) For example . Note that neuron 7 has no incoming messages anyway. Instead its relevance is defined as . In Eq (8) and the following text the terms input and source have the meaning of being an input to another neuron in the direction as defined during classification time, not during the time of computation of layer-wise relevance propagation. For example in Fig 2 neurons 1 and 2 are inputs and source for neuron 4, while neuron 6 is the sink for neurons 2 and 3. Given the two assumptions encoded in Eq (8), the layer-wise relevance propagation by Eq (2) can be satisfied by the following sufficient condition: (9) (10) (11) (12)

In general, this condition can be expressed as: (13) The difference between condition (13) and definition (8) is that in the condition (13) the sum runs over the sources at layer l for a fixed neuron k at layer l+1, while in the definition (8) the sum runs over the sinks at layer l+1 for a fixed neuron i at a layer l. When using Eq (8) to define the relevance of a neuron from its messages, then condition (13) is a sufficient condition in order to ensure that Eq (2) holds. Summing over the left hand side in Eq (13) yields One can interpret condition (13) by saying that the messages are used to distribute the relevance of a neuron k onto its input neurons at layer l. Our work in the following sections will be based on this notion and the more strict form of relevance conservation as given by definition (8) and condition (13). We set Eqs (8) and (13) as the main constraints defining layer-wise relevance propagation. A solution following this concept is required to define the messages according to these equations.

Now we can derive an explicit formula for layer-wise relevance propagation for our example by defining the messages . The layer-wise relevance propagation should reflect the messages passed during classification time. We know that during classification time, a neuron i inputs aiwik to neuron k, provided that i has a forward connection to k. Thus, we can rewrite the left hand sides of Eqs (9) and (10) so that it matches the structure of the right hand sides of the same equations by the following (14) (15) The match of the right hand sides of Eqs (9) and (10) against the right hand sides of (14) and (15) can be expressed in general as (16)

While this solution (16) for message terms still needs to be adapted such that it is usable when the denominator becomes zero, the example given in Eq (16) gives an idea what a message could be, namely the relevance of a sink neuron which has been already computed, weighted proportionally by the input of the neuron i from the preceding layer l. This notion holds in an analogous way when we use different classification architectures and replace the notion of a neuron by a dimension of a feature vector at a given layer.

The Formula (16) has a second property: The sign of the relevance sent by message becomes inverted if the contribution of a neuron ai wik has different sign then the sum of the contributions from all input neurons, i.e. if the neuron fires against the overall trend for the top neuron from which it inherits a portion of the relevance. Same as for the example with the linear mapping in Eq (5), an input neuron can inherit positive or negative relevance depending on its input sign. This is a difference to the Eq (4). While this sign switching property can be defined analogously for a range of architectures, we do not add it as a constraint for layer-wise relevance propagation.

One further property is visible here as well. The formula for distribution of relevance is applicable to non-linear and even non-differentiable or non-continuous neuron activations ak. An algorithm would start with relevances R(l+1) of layer l+1 which have been computed already. Then the messages would be computed for all elements k from layer l+1 and elements i from the preceding layer l—in a manner such that Eq (13) holds. Then definition (8) would be used to define the relevances R(l) for all elements of layer l.

The relevance conservation property can in principle be supplemented by other constraints that further reduce the set of admissible solutions. For example, one could constrain relevance messages that result from the redistribution of relevance onto lower-level nodes in a way that is consistent with the contribution of lower-level nodes to the upper layer during the forward pass. If a node i has a larger weighted activation zik = aiwik, then, in a qualitative sense, it should also receive a larger fraction of the relevance score of the node k. In particular, for all nodes k satisfying Rk, ∑i zik > 0, one can define the constraint . (The formulas provided in section Pixel-wise Decomposition for Multilayer Networks adhere to this ordering constraint.) Nevertheless, nothing is said about the exact relevance values beyond their adherence to the constraint stated above.

To summarize, we have introduced layer-wise relevance propagation in a feed-forward network. In our proposed definition, the total relevance is constrained to be preserved from one layer to another, and the total node relevance must be the equal to the sum of all relevance messages incoming to this node and also equal to the sum of all relevance messages that are outgoing to the same node. It is important to note that the definition is not given as an algorithm or a solution of an objective with a distinct minimum. Instead, it is given as a set of constraints that the solution should satisfy. Thus, different algorithms with different resulting solutions may be admissible under these constraints.

Taylor-type decomposition

One alternative approach for achieving a decomposition as in (1) for a general differentiable predictor f is first order Taylor approximation. (17) The choice of a Taylor base point x0 is a free parameter in this setup. As said above, in case of classification we are interested to find out the contribution of each pixel relative to the state of maximal uncertainty of the prediction which is given by the set of points f(x0) = 0, since f(x) > 0 denotes presence and f(x) < 0 absence of the learned structure. Thus, x0 should be chosen to be a root of the predictor f. For the sake of precision of the Taylor approximation of the prediction, x0 should be chosen to be close to x under the Euclidean norm in order to minimize the Taylor residuum according to higher order Taylor approximations. In case of multiple existing roots x0 with minimal norm, they can be averaged or integrated in order to get an average over all these solutions. The above equation simplifies to (18) The pixel-wise decomposition contains a non-linear dependence on the prediction point x beyond the Taylor series, as a close root point x0 needs to be found. Thus the whole pixel-wise decomposition is not a linear, but a locally linear algorithm, as the root point x0 depends on the prediction point x.

Several works have been using sensitivity maps [17–19] for visualization of classifier predictions which were based on using partial derivatives at the prediction point x. There are two essential differences between sensitivity maps based on derivatives at the prediction point x and the pixel-wise decomposition approach. Firstly, there is no direct relationship between the function value f(x) at the prediction point x and the differential Df(x) at the same point x. Secondly, we are interested in explaining the classifier prediction relative to a certain state given by the set of roots of the prediction function f(x0) = 0. The differential Df(x) at the prediction point does not necessarily point to a root which is close under the Euclidean norm. It points to the nearest local optimum which may still have the same sign as the prediction f(x) and thus be misleading for explaining the difference to the set of root points of the prediction function. Therefore derivatives at the prediction point x are not useful for achieving our aim. Fig 3 illustrates the qualitative difference between local gradients (black arrows) and the dimension-wise decomposition of the prediction (red arrow).

The blue dots are labeled negatively, the green dots are labeled positively. Left: Local gradient of the classification function at the prediction point. Right: Taylor approximation relative to a root point on the decision boundary. This figure depicts the intuition that a gradient at a prediction point x—here indicated by a square—does not necessarily point to a close point on the decision boundary. Instead it may point to a local optimum or to a far away point on the decision boundary. In this example the explanation vector from the local gradient at the prediction point x has a too large contribution in an irrelevant direction. The closest neighbors of the other class can be found at a very different angle. Thus, the local gradient at the prediction point x may not be a good explanation for the contributions of single dimensions to the function value f(x). Local gradients at the prediction point in the left image and the Taylor root point in the right image are indicated by black arrows. The nearest root point x0 is shown as a triangle on the decision boundary. The red arrow in the right image visualizes the approximation of f(x) by Taylor expansion around the nearest root point x0. The approximation is given as a vector representing the dimension-wise product between Df(x0) (the black arrow in the right panel) and x − x0 (the dashed red line in the right panel) which is equivalent to the diagonal of the outer product between Df(x0) and x − x0.

https://doi.org/10.1371/journal.pone.0130140.g003

One technical difficulty is to find a root point x0. For continuous classifiers we may use unlabeled test data in a sampling approach and perform a line search between the prediction point x and a set of candidate points {x′} such that their prediction has opposite sign: f(x)f(x′) < 0. It is clear that the line l(a) = ax + (1 − a)x′ must contain a root of f which can be found by interval intersection. Thus each candidate point x′ yields one root, and one may select a root point which minimizes the Taylor residuum or use an average over a subset of root points with low Taylor residues. A second possible solution would be, for example, to find the root point x0 that is the nearest to the data point x, or optimal in some measurable sense. However, while we have presented two types of solutions to finding a root point, in the most general case, the Taylor-type decomposition is best described as a constraint-based approach. In particular, the root point x0 at which the Taylor decomposition is computed is constrained to satisfy f(x0) = 0 and to lie not too far (e.g. within a fixed radius) from the actual data point x. Using this constraint-based definition, the most desirable properties of the Taylor decomposition are preserved, while the remaining specification is deferred to a later point in time.

Note that Taylor-type decomposition, when applied to one layer or a subset of layers, can be seen as an approximate way of relevance propagation when the function is highly non-linear. This holds in particular when it is applied to the output function f as a function of the preceding layer f = f(zi − 1), as Eq (18) satisfies approximately the propagation Eq (2) when the relevance of the output layer is initialized as the value of prediction function f(x). Unlike the Taylor approximation, layer-wise relevance propagation does not require to use a second point besides the input point. The formulas in Sections Pixel-wise Decomposition for Classifiers over Bag of Words Features and Pixel-wise Decomposition for Multilayer Networks will demonstrate that layer-wise relevance propagation can be implemented for a wide range of architectures without the need to approximate by means of Taylor expansion.

Related Work

Several works have been dedicated to the topic of explaining neural networks, kernel-based classifiers in general and classifiers over Bag of Words features in particular.

As for neural networks, [20] is dedicated towards analyzing classifier decisions at neurons which is applicable also to the pixel level. It performs a layer-wise inversion down from output layers towards the input pixels for the architecture of convolutional networks [21]. This work is specific to the architecture of convolutional neural networks. Compared to our approach it is based on a different principle. See [22] which establishes an interpretation of the work in [20] as an approximation to partial derivatives with respect to pixels in the input image. In a high-level sense, the work in [20] uses the method from their own predecessor work in [23] which solves optimization problems in order to reconstruct the image input, while our approach attempts to reconstruct the classifier decision. In a more technical sense, the difference between [20] and our approach can be seen by comparing how the responses are projected down towards the inputs. [20] uses rectified linear units to project information from the unfolded maps towards the inputs with the aim to ensure the feature maps to be non-negative. In our approach we use the signed activations of the neurons from the layer below for weighting the relevance quantity from the layer above with the aim of conserving the relevance quantity across layers. The sign of the activations of neurons of the layer below, for which the relevance is to be computed, is used in our work for encoding assumptions about the structure of the neural network. With respect to neural networks, our work is applicable to a wide range of architectures. A different type of analysis for understanding neural networks deals with creating counter-intuitive examples, see for example the work in [24, 25]. For a treatment on how to discriminate structures which influence predictions from correlations with hidden variables such that the hidden variables have an impact on the predictor, see [26].

Another approach which lies between partial derivatives at the input point x and a full Taylor series around a different point x0 is presented in [22]. This work uses a different point x0 than the input point x for computing the derivative and a remainder bias which both are not specified further but avoids for an unspecified reason to use the full linear weighting term x − x0 of a Taylor series. Besides deviating in the usage of a Taylor series, the work in [22] does not make use of the layer-wise propagation strategy for neural networks presented in this paper, which unlike a Taylor series does not rely on a local approximation. Explanation of neural network behavior on the level of single neurons is done in [27] and [28]. These works try to find inputs which maximize the activation of neurons by means of optimization problems which can be solved by gradient ascent. Our approach aims at explaining the decision for a given input rather than finding optimal stimuli for a particular neuron. Note that the approach presented in our paper is different from plotting the activations of neurons during a forward-pass of the input image. The latter is independent from the neural network properties in higher layers whereas in our approach we feed the classification score into the top neurons and use quantities computed by using properties of higher layers to obtain a representation at lower layers. Quantifying the importance of input variables using a neural network model has also been studied in specific areas such as ecological modeling, where [29, 30] surveyed a large ensemble of possible analyses, including, computing partial derivatives, perturbation analysis, weights analysis, and studying the effect of including and removing variables at training time. A different avenue to understanding decisions in neural network is to fit a more interpretable model (e.g. decision tree) to the function learned by the neural network [31], and extract the rules learned by this new model.

With respect to sensitivity of kernel-based classifiers to input dimensions, [32] yields sensitivity maps, which are invariant to the sign of the predictions. [17–19] consider explanation vectors based on local gradients which are sign-sensitive. Both approaches use partial derivatives at the point which is to be predicted. The first works to our knowledge to visualize BoW models are [33] and [34]. The former work finds regions with a large influence for classification for the special case of max pooling, sparse coding and a linear classifier. The latter work obtains an exact decomposition into local feature scores for the case of a histogram intersection kernel and zero-one coding of local features onto visual words.

We differ from the above works on kernel-based classifiers over Bag of Words features in the following sense: Our methodology is applicable to arbitrary Bag of Words models including various feature codings such as Fisher vectors and regularized codings and to a broader class of kernels, namely all differentiable or so-called sum decomposable kernels. When relying on Taylor decomposition, our work relies on a Taylor series around a root close to the prediction point rather than partial derivatives at the prediction point itself. The rationale for this choice was justified in the preceding Section Taylor-type decomposition. Finally, in contrast to the preceding publications, our work introduces layer-wise relevance propagation for neural networks and Bag of Words features.

Overview of the decomposition steps

The main contribution of this part is the formulation of a generic framework for retracing the origins of a decision made by the learned kernel-based classifier function for a BoW feature. This is achieved, in a broad sense as visualized in Fig 4, by following the construction of a BoW representation x of an image and the evaluation thereof by a classifier function in reverse direction. In this section we will derive a decomposition of a kernel-based classifier prediction into contributions of individual local features and finally single pixels. The proposed approach consists of three consecutive steps.

Each step taken towards the final pixel-wise decomposition has a complementing analogue within the Bag of Words classification pipeline. The calculations used during the pixel-wise decomposition process make use of information extracted by those corresponding analogues. Airplane image in the graphic by Pixabay user tpsdave.

https://doi.org/10.1371/journal.pone.0130140.g004

In the first step we will use, depending on the type of kernel, either the Taylor-type decomposition strategy or the layer-wise relevance propagation strategy. In the first step relevance scores for the third layer of the BoW feature extraction process are obtained, describing the influence of all BoW feature dimensions d by deconstructing the classifier prediction function f(x) such that . In other words, we gain a decomposition into contributions describing f(x) as a sum of individual predictions for all dimensions of x.

In the second step we will apply the layer-wise relevance propagation strategy in order to obtain relevance scores for the local features l from the relevance scores . Layer-wise relevance propagation ensures that holds.

The third step describes the computation of pixel-wise scores from local feature relevance scores , which are then visualized as heatmaps by color-coding.

Step one: relevance scores for the third layer of the BoW feature extraction process.

The third layer is the BoW feature itself. In the first step we would like to achieve a decomposition of the classifier prediction f(x) into relevance scores for BoW feature dimension d. (22)

The work of [34] has performed this step for the special case of one single histogram intersection kernel. Such a decomposition can be generalized naturally and performed without error for all kernel functions which are sum-decomposable along input dimensions. We define a kernel function k to be sum-decomposable if there exists kernel functions k(d) acting on single input feature dimensions such that (23) In this case, as with the linear and histogram intersection kernel, we can achieve Eq (22) with equality in the following definition by applying layer-wise relevance propagation which results in Eq (24).

Def. 1 Relevance scores for sum decomposable kernels (24)

For the case of a general differentiable kernel we apply the Taylor-type decomposition strategy in order to linearly approximate the dimensional contributions . The approximated dimensional contributions can be expressed as in Eq (25).

Def. 2 Relevance scores for differentiable kernels (25)

Step two: relevance scores for the second layer of the BoW feature extraction process.

The second layer are the local features extracted from many regions of the image. In the second step we would like to achieve a decomposition of the classifier prediction f(x) into relevance scores for the local features l based on the relevances from the third layer.

For the sake of clarity, we do for now start with the case of sum-pooled BoW aggregation, to later extend to a more general formulation for p-means pooling from this point on.

As introduced in context of Eq (19) m(d) denotes the mapping projecting to dimension d of the BoW space. We define the set of input dimensions Z(x) which are effectively not reached by the mappings of local features of an image. Applying the layer-wise propagation allows to define the local feature relevance score as given in Eq (27).

Def. 3 Local feature scores for sum pooling (26) (27)

The coarse structure of definition (27) can be explained as taking the relevances from the layer above and weighting them with the outputs from the layer below. The summations over the mappings m(d)(l) over the local features {l} in Eq (27) achieve a weighting of the relevance from the third layer proportional to the ratios of the mappings. The second part describes an equal distribution of those relevance scores which correspond to dimensions of the Bag of Words feature x which have a value of zero and yet may contribute to the classifier prediction. To see this consider computing the χ2-kernel value to a support vector which has a non-zero value in the same feature dimension. This is necessary to ensure that the propagation Eq (2) holds.

Summing the local feature relevance scores from Eq (27) yields the Taylor approximation of the prediction score f(x) in Eqs (24) or (25). This property is the key to our approach. We obtain exact summation to the prediction f(x) in the case of sum decomposable kernels and usage of Eq (24) in this special case. (28) We would like to point out that this property holds also in the case when mappings m(d) can become negative as a consequence of the definition used in (26) and (27), as can be seen from the summation given in the appendix. For that reason our approach is also applicable to Fisher vectors [11] and regularized coding approaches [12–14]. Furthermore note that definition (27) has no explicit dependence on the way how the local features are pooled in Eq (19) and this might be inappropriate weighting for max-pooling or general p-means pooling.

We can extend this definition to reflect the usage of p-means pooling (29) which may yield different results in prediction and local decomposition than sum pooling. The extension is well-defined for non-negative mappings m(d) ≥ 0 and any value of p and for arbitrary mappings when combined with a value of p from the natural numbers.

Def. 4 Local feature scores for p-means pooling (30) (31)

The first quotient in Eq (31) converges to an indicator function for the maximal mapping element in the limit p → ∞ which is consistent to max-pooling (32)

Step three: relevance scores for the first layer of the BoW feature extraction process.

The first layer are the pixels of the image. In order to calculate scores for each pixel we make use of information regarding local feature geometry and location known from the local feature extraction phase at the beginning of the image classification pipeline. The pixel score of an image coordinate q is calculated as a sum of local feature scores of all local features l covering q, weighted by the number of pixels covered by each local feature l. In terms of layer-wise relevance propagation a local feature is a computation unit which has as much inputs as the number of pixels it is covering. Without assumption of any further structure we distribute the relevance of the local feature equally to all its covered pixels. Layer-wise propagation yields Eq (34). (33) (34) The operator area(l) used Eqs (33) and (34) returns the set of pixel coordinates covered by a local feature l. Projecting the obtained scores to their respective image coordinates yields the pixel-wise decomposition representation h of the evaluated image.

For visualization in the sense of color coding, the pixel-wise decomposition R(1) is then normalized as (35) ensuring that . The normalized pixel-wise decomposition is then color coded by mapping the pixel scores to a color space of choice. In the case of individually classified sub-regions originating from the same image, a global pixel-wise decomposition can be constructed by averaging local pixel-wise decompositions scores.

Note that by choosing above normalization scheme, the assumption is made that at least one class is represented within the image. In case the assumption holds this might lead to prominent local predictions of even weakly projected features which we found suitable for the purpose of detecting class evidence. If this assumption does not hold, then images may display score artifacts dominated by the set of pixels covered by a small subset of local features which would otherwise be considered input noise. A solution for this problem is global normalization which uses a maximum over pixels over a set of images instead of one image. We found that a global normalization scheme can be more appropriate for visualizing the actual decision process of the classifier, as it preserves the relative order of magnitude of local feature scores in between pixel-wise decomposition tiles. Algorithm 1 gives an overview how to compute the pixel-wise decomposition for classifiers based on Bag of Words features and support vector machines.

Algorithm 1 Pixel-wise decomposition for BoW features with SVM classifiers

Inputs:

Image I

Local features L

BoW representation x (and Taylor root point x0)

model and mapping parameters

for d = 1 to V do

as in Eqs (24) or (25)

end for

for all l ∈ L do

as in Eqs (27) or (31)

end for

for all pixels q ∈ I do

as in Eq (34)

end for

Output:

Layer-wise relevance backpropagation

As an alternative to Taylor-type decomposition, it is possible to compute relevances at each layer in a backward pass, that is, express relevances as a function of upper-layer relevances , and backpropagating relevances until we reach the input (pixels). Fig 5 depicts a graphic example.

The method works as follows: Knowing the relevance of a certain neuron for the classification decision f(x), one would like to obtain a decomposition of such relevance in terms of messages sent to neurons of the previous layers. We call these messages Ri ← j. In particular, as expressed by Eqs (8) and (13), the conservation property (55) must hold. In the case of a linear network f(x) = ∑i zij where the relevance Rj = f(x), such decomposition is immediately given by Ri ← j = zij. However, in the general case, the neuron activation xj is a non-linear function of zj. Nevertheless, for the hyperbolic tangent and the rectifying function—two simple monotonically increasing functions satisfying g(0) = 0, —the pre-activations zij still provide a sensible way to measure the relative contribution of each neuron xi to Rj. A first possible choice of relevance decomposition is based on the ratio of local and global pre-activations and is given by: (56) These relevances Ri ← j are easily shown to approximate the conservation properties of Eq (2), in particular: (57) where the multiplier accounts for the relevance that is absorbed (or injected) by the bias term. If necessary, the residual bias relevance can be redistributed onto each neuron xi.

A drawback of the propagation rule of Eq (56) is that for small values zj, relevances Ri ← j can take unbounded values. Unboundedness can be overcome by introducing a predefined stabilizer ɛ ≥ 0: (58) The conservation law then becomes (59) where we can observe that some further relevance is absorbed by the stabilizer. In particular, relevance is fully absorbed if the stabilizer ɛ becomes very large.

An alternative stabilizing method that does not leak relevance consists of treating negative and positive pre-activations separately. Let and where “ − ” and “+” denote the negative and positive part of zij and bj. Relevance propagation is now defined as (60) where α + β = 1. For example, for α,β = 1/2, the conservation law becomes: (61) which has similar form to Eq (57). This alternate propagation method also allows to control manually the importance of positive and negative evidence, by choosing different factors α and β.

Once a rule for relevance propagation has been selected, the overall relevance of each neuron in the lower layer is determined by summing up the relevance coming from all upper-layer neurons in consistence with Eqs (8) and (13): (62) The relevance is backpropagated from one layer to another until it reaches the input pixels x(d), and where relevances provide the desired pixel-wise decomposition of the decision f(x). The complete layer-wise relevance propagation procedure for neural networks is summarized in Algorithm 2.

Algorithm 2 Pixel-wise decomposition for neural networks

Input: R(L) = f(x)

for l ∈ {L − 1, …, 1} do

as in Eqs (58) or (60)

end for

Output:

Above formulas (58) and (60) are directly applicable to layers which satisfy a certain structure. Suppose we have a neuron activation xj from one layer which is modeled as a function of inputs from activations xi from the preceding layer. Then layer-wise relevance propagation is directly applicable if there exists a function gj and functions hij such that (63) In such a general case, the weighting terms zij = xiwij from Eq (50) have to be replaced accordingly by a function of hij(xi). A key observation is that relevance propagation is invariant against the choice of function gj for computing relevances for the inputs xi conditioned on keeping the value of relevance Rj for xj fixed. This observation allows to deal with non-linear activation functions gj. The function gj does however exert influence on computing the relevance Ri by its influence on the relevance Rj for xj. This can be explained by the fact, that the choice of gj determines the value of xj and thus also relevance Rj for xj which gets assigned by the weights in the layer above.

We remark again, that even max pooling fits into this structure as a limit of generalized means, see Eq (32) for example. For structures with a higher degree of non-linearity, such as local renormalization [58, 59], Taylor approximation applied to neuron activation xj can be used again to achieve an approximation for the structure as given in Eq (63).

Finally, it can be seen from the formulas established in this section that layer-wise relevance propagation is different from a Taylor series or partial derivatives. Unlike Taylor series, it does not require a second point other than the input image. Layer-wise application of the Taylor series can be interpreted as a generic way to achieve an approximate version of layer-wise relevance propagation. Similarly, in contrast to any methods relying on derivatives, differentiability or smoothness properties of neuron activations are not a necessary requirement for being able to define formulas which satisfy layer-wise relevance propagation. In that sense it is a more general principle.

Neural Networks for MNIST digits

We would like to investigate the capacity of relevance propagation to find evidence for classification of MNIST handwritten digits. A particular advantage of this data set over many-class image data sets is that it is easy for humans to interpret both positive and negative evidence, because of the small number of classes. For example, evidence for the handwritten digit “1” comes from the presence of a vertical bar on the pixel grid, but also from the absence of horizontal bar starting from the top of the vertical bar, which would make it a “7”. Also, the data set is relatively simple and the relation between the training algorithm and the resulting pixel-wise relevances can be analyzed.

We perform three experiments for MNIST data, as we would like to demonstrate that the method is able to uncover properties specific to the way of training. One experiment is done with a smaller network which is trained without translations of digits for the sake of allowing a direct comparison of the pixel-wise decomposition results to class-densities for each pixel of digits and seeing the impact of artifacts in the training set. Two further experiments are done on a larger network with has been trained without artifacts and with translated versions of digits and more training iterations for the sake of a better response to digits. The latter two experiments intend to show the impact of non-digit pixels as positive and negative evidence for a class of digits.

MNIST experiments I.

The first set of experiments is done on a fully-connected neural network trained in the most common way: Input data is normalized so that the sum of pixels is on average zero, and the variance of pixel values is on average one. This setting implies that only black pixels yield strong inputs whereas white pixels fire only due to mean subtraction. The absence of translation invariance during training allows to uncover correlations of the pixel-wise decomposition to pixel-wise training densities and, as we will see in the experiments, allow to uncover artifacts in the data that may harm generalization.

Examples for pixel-wise decompositions for the first type of neural networks are given in Figs 11, 12 and 13. Multilayer neural networks were trained on the MNIST [57] data of handwritten digits and solve the posed ten-class problem with a prediction accuracy of 98.25% on the MNIST test set. Our network consists of three linear sum-pooling layers with a bias-inputs, followed by an activation or normalization step each. The first linear layer accepts the 28 × 28 pixel large images as a 784 dimensional input vector and produces a 400-dimensional tanh-activated output vector. The second layer projects those 400 inputs to equally many tanh-activated outputs. The last layer then transforms the 400-dimensional space to a 10-dimensional output space followed by a softmax layer for activation in order to produce output probabilities for each class. The network was trained using a standard error back-propagation algorithm using batches of 25 randomly drawn training samples with an added Gaussian noise layer per training iteration. The above prediction accuracy was achieved after terminating the training procedure after 50 000 iterations.

Each group of four horizontally aligned panels shows—from left to right—the input digit, the Taylor root point x0, the gradient of the prediction function f at x0 of a specific digit class indicated by the subscript next to f and the approximated pixel-wise contributions for x.

https://doi.org/10.1371/journal.pone.0130140.g011

When considering a digit from class i and a pixel-wise decomposition from class k ≠ i, it is observable that the pixel-wise decomposition shows frequently highly positive activations on pixels of the digit from class i which have high relative density dk for the digit class k ≠ i.

https://doi.org/10.1371/journal.pone.0130140.g013

Fig 11 shows pixel-wise relevances resulting from Taylor-type decomposition as defined in Eqs (18) and (53). The left column of the figure shows approximated pixel-wise relevances of the input digits, both times a 7, relative to a blank pixel tile. Relevances are calculated for the classifier output producing the prediction score for the class of digits 7. Evidently the upper part of the digit and most prominently the horizontal bar at the top are important characteristics for the predictor speaking for the digit’s class. The middle column of Fig 11 shows pixel-wise relevances for inputs from digit class 5 relative to a blank tile as interpreted by the classifier output reserved for the class of digit 3. As for digit class 7, the gradient at the root point x0 shows high positive prediction weights in areas corresponding to typical characteristics of the target class. We observe strong positive responses in areas shared by both digit classes 5 and 3 (indicated by annotation a2) and negative pixel-wise responses where the upper left vertical bar of the input digits is located, which is usually not present in a digit 3 (a1). The rightmost column of Fig 11 indicates distinguishing parts of the input digits from class 2 relative to their counter parts from digit class 6 chosen by minimal euclidean distance. High pixel scores highlight characteristics of the prediction point indicating membership in class 2, most notable the upper arc and the tail of the input digits 2, as marked with the annotation a2. Notably in the lower right image of the figure the absence of the leftmost vertical arc of the digit 6 (annotation a1) in the prediction point causes positive local predictions for class 2.

Pixel-wise predictions obtained via the layer-wise relevance propagation Formula (56) were calculated based on the output of the last linear layer without taking the succeeding softmax normalization layer into account. We can see pixel-wise decompositions for exemplary digits in Fig 12. Notably, for the pixel-wise decomposition of the digit 2, we see highly positive responses in lower parts of the digits for the classifiers for digits 2 and 6, however for the classifier for digit 6, the prediction score is negative, because the high positive responses in the lower part are suppressed by high negative responses in the upper part where the digit 2 has an arc which is not present in the digit 6.

Correlating the pixel-wise decompositions for classifier for digit k with the relative density of the pixels of the digits k in the training set demonstrates the plausibility of the decomposition. (65) This can be seen in Fig 13. When considering the decomposition for a classifier for digit k and using a digit i ≠ k from the wrong class as input, then we observe often that the decomposition shows high positive activations on those pixels of digit i which have high relative density dk(p) (Eq (65)) within the training set for digits of type k. Even when we use the classifier for the wrong class, it uncovers evidence on those pixels which overlap regions with high density of positively labeled digits of its training set.

MNIST experiments I I.

In this set of experiments, we train larger neural networks on the MNIST data set augmented translated copies of the digits. Neural networks are composed of three hidden layers of 1296 units each, where weight connections between layers are initialized at random. Neural networks are trained by backpropagation using stochastic gradient descent with mini batches of size 25 and using a softmax objective [6]. In order to force the neural network to make decisions based on the shape of digits rather than their absolute position on the pixel grid, we augment the training data with pixel-wise translations of ±4 pixels (both vertical and horizontal) and add 10% salt-and-pepper noise to the inputs. Also, in order to favor the construction of evidence on both black and white pixels, MNIST digits are re-scaled to take pixel values between −1 (white pixel) and 1 (black pixel).

We consider two types of non-linearities: (1) rectified linear units and (2) hyperbolic tangent sigmoids. These non-linearities are some of the most commonly used in neural networks and are plotted in Fig 14. Also, in order to study explanations on different level of detail, we consider (1) a network that has been trained for 1 000 000 iterations until it reaches 99.2% accuracy and (2) a network trained for only 10 000 iterations, at which point it reaches approximately 97.0% accuracy.

Figs 15 and 16 are two case studies of explanations produced by the neural networks we have trained. The number in brackets on the top-left corner of the pixel-wise decomposition indicates the class with respect to which evidence is measured. Similar explanations are produced by a network trained with rectified linear units. It is interesting to note that layer-wise relevance propagation, which is not making use of the gradient information—and therefore agnostic to the type of neurons in the network, —still produces similar results across different types of neurons. Explanations learned by the rectifying network are arguably more global, because the rectified linear units are prevented from saturating on both sides and therefore more linear. Explanations produced by a rectifying network trained for a shorter time (rect⋆) are clearly more global, as evidenced by the positive and negative parts of the pixel-wise decompositions spreading over larger pixel areas.

Strong positive evidence for “4” is allocated to the top part of the image for keeping it blank. If trying to interpret these digits as “9”, the open top-part of the image is perceived as negative evidence for this class, because a “9” would rather have a top-dash closing the upper loop of the “4”. Explanations are consistent across a variety of neural networks and samples.

https://doi.org/10.1371/journal.pone.0130140.g015

Classifying as “3” is supported by the middle horizontal stroke featured in this digit and the absence of vertical connections on the left of the image. Evidence for being a “8” feature again the middle horizontal stroke, however, the absence of connections on the left side of the digit constitutes negative evidence. Explanations are again stable for various models and samples.

https://doi.org/10.1371/journal.pone.0130140.g016

In order to compute the pixel-wise decompositions, we used Eq (56), which does not use numerical stabilizers. Use of numerical stabilizers causes relevance to leak from one layer to another, in particular, when the relevance of a particular handwritten digit is measured for another class. Fig 17 shows examples of pixel-wise decompositions for 16 randomly drawn digits sorted according to the digit class.

Results are obtained using the relevance propagation Formula (56) with the rectifying network trained for 1 000 000 iterations from Section MNIST experiments I I.

https://doi.org/10.1371/journal.pone.0130140.g017

Pixel flipping experiments using the rect-long model from Section MNIST experiments I I.

In this section we intend to make a semi-quantitative analysis of the pixel-wise decompositions. The basic idea is to compute a decomposition of a digit for a digit class and then flip pixels with highly positive, highly negative scores or pixels with scores close to zero and then to evaluate the impact of these flips onto the prediction scores. The advantage of demonstrating this on MNIST data is that a vast majority of pixels either has very high (black) or very low (white) values, with very few pixels having values in between. For results on photographic scenes one may need to resort to building masks which are specific to each category, for example, a gray box may be good for masking a flower, but it will not mask effectively a gray road or fur of a Koala bear.

In general: If we flip a pixel, we do flipped = pixel ⋅(−1). In further experiments we will consider the digits three and four as inputs. The test set contains 983 fours and 1011 threes, over which the mean prediction is calculated.

One apparent result from the preceding section is that we can observe non-digit pixels with highly positive pixel scores. In Fig 18 we evaluate the impact of flipping non-digit pixels with highly positive pixel scores on the resulting classifier prediction. We observe in Fig 18 that the classifier prediction for the true digit class, namely three and four, decreases fast when the non-digit pixels with high pixel scores are flipped. This shows, firstly, that having non-digit pixels with high positive pixel scores makes sense for a correct classification. We see also, that flipping the highest scoring pixels for a digit three turns it into a digit eight, which is consistent to the decompositions seen in Fig 16.

Pixels with highest positive scores are flipped first. The pixel-wise decomposition was computed for the true digit class, three (left) and four (right).

https://doi.org/10.1371/journal.pone.0130140.g018

Secondly, it shows that measuring the quality of a pixel-wise decomposition by an object segmentation mask is not always a good idea. When seeing the digit as an object, having non-object pixels with high scores can make sense in the case of geometric constraints for objects which are to be recognized, as we have in our case. For this reason we deliberately did not choose object segmentation masks for the digits as a basis for evaluation in the sense that digit-pixels should have highly positive scores, and non-digit pixels should have zero or negative scores.

For the same reason, namely the possibility of the presence of geometric constraints, pixel-wise decomposition is not always a good weak segmentation in contrast to the convincing results in the experiments of [22] on images from the ILSVRC data set. Another reason for a possible divergence between pixel-wise decomposition and segmentation is the possible influence of context in a scene.

Once we have established the reason why we do not use segmentation masks for evaluating the quality of pixel-wise prediction we can observe the effects of flipping the highest scoring pixels, independently of whether they are a digit or non-digit pixel. We can see from Fig 19 that the prediction of the correct digit class decreases sharply in this case as well. This decrease can be compared to the case when we flip those pixels whose absolute value of the decomposition score s is closest to zero by sorting the pixels according to 1 − ∣s∣. We see in Fig 20 that the classifier prediction for the true class decreases only very slowly, thus neutrally scored pixels are less relevant for the prediction. The comparison between Figs 19 and 20 shows that the decomposition score is able to identify those pixels which play little role in the classification of a digit and those pixels which are important for identifying a digit.

Pixels with highest positive scores are flipped first. The pixel-wise decomposition was computed for the true digit classes three (left) and four (right).

https://doi.org/10.1371/journal.pone.0130140.g019

Pixels with absolute value closest to zero are flipped first. Digit and non-digit pixels may be flipped. Pixel-wise decomposition have been computed for the true digit classes three (left) and four (right).

https://doi.org/10.1371/journal.pone.0130140.g020

Thus, the pixel-wise decomposition is not only intuitively appealing to a human but also makes sense for the representation used in classifier to make its decision. Using digits for demonstrating such a statement has a mild bias towards our method because for a geometric-driven task like digits we can expect that firstly the problem can be learned well by a classifier so that the resulting pixel-wise decompositions are very informative, and secondly what has been learned on digits might be more similar between humans and algorithms compared to complex natural scene recognition tasks. See, however, the experiments in the Section Neural Network for 1000 ILSVRC classes on the categories from ILSVRC challenge for results on an object recognition tasks on photographic images which also yield results plausible to a human. In general one can expect that a classifier with poor recognition ability will yield also pixel-wise decompositions with less informative scores.

Finally, we evaluate the influence of pixels with negative scores. For this, we take a digit, compute the pixel-wise decomposition for a wrong class, then flip the pixels which are marked negatively when trying to predict the wrong class. Results are shown in Fig 21. We observe for both example cases that the prediction for the true class declines sharply as we tweak the digits towards the wrong class. For digit 9 as target and digits 4 as inputs this flipping results in a noisy blob on top of a stick, such that the result is not cleanly predictable as a nine. However one can observe still a moderate rise of the prediction of the 4 as a 9. For digit 8 as target and digits 3 as inputs we can see, that for some intermediate percentage of flips, the 3 will be identified as an 8. In summary, negatively scored pixels are able to represent what is contradictory to classifying a digit to belong to a particular class.

The results in Figs 18, 19 and 20 are shown for the true class against one or more particular wrong classes. The results hold also when we show the behavior of the classifier predictions under flipping of the highest scoring and the most zero-like pixels when we compared for each digit against the maximum of predictions over all wrong classes. Furthermore we compute the average over all 10 digits, not only digits 3 and 4 as true classes as shown in Figs 18, 19 and 20. This result is shown in Fig 22.

Left: Pixels with highest positive scores are flipped first. Right: Flipping of neutrally predicted pixels, i.e. pixels with absolute value closest to zero are flipped first (solid lines), and flipping of randomly picked pixels (dashed lines). Results are averaged over digits from all digit classes in contrast to using only digit classes 3 and 4 in the preceding figures.

https://doi.org/10.1371/journal.pone.0130140.g022

On average over all digits, flipping the highest scoring pixels at first results in a fast decline of the prediction for the true class, and at some point another class is predicted. Flipping the pixels at first with scores close to zero results in a much slower decline of the prediction for the true class. This result demonstrates a quantifiable plausibility of the pixel-wise decomposition by layer-wise relevance prediction. In order to visualize the process of pixel flipping, Figs 23 and 24 provide examples of a digit, its heatmap and the resulting images with increasing amounts of flipped pixels. For each of the resulting images the softmax output y and the linear layer output yp are shown. The first difference between Figs 23 and 24 is that in the former the pixels were flipped according to the heatmap for the highest scoring class, and in the latter the pixels were flipped according to the heatmap for a random class which did not yield the highest score. The second difference is that in Fig 23 the highest scoring pixels were flipped first, which implies that evidence for the predicted class label is removed. As a consequence, the original prediction is gradually lost without giving a specific false class label as target. The digit in Fig 23 changes towards that class label which the classifier and the heatmapping implicitly perceive as the nearest neighbor. With respect to the second difference, in Fig 24 the lowest scoring pixels for the random false class label were flipped first, which implies that evidence against the false class label is removed. As a consequence, the digit is modified towards the specific false class label which is denoted in parentheses above the heatmap. In Fig 23 each of the digits 7 and 8 results into different class labels after flipping. We can see that destroying evidence for a given predicted class may result into differing final class labels. The nearest neighbor of a digit is not constant within the digit class, but depends on the style of the digit itself. Furthermore we can see that pixel-flipping sometimes removes and sometimes adds black pixels, depending on where the highest pixel scores are located. In Fig 24 we can see results of flipping towards a specified false class label. In both cases the resulting digits are plausible to the human observer. Note also small details, e.g. the 4 in Fig 24 is rounded at its lower left end during flipping, while the classifier regards the 7 in Fig 24 to be too short for being a 1 as seen on the blue spot above the vertical bar of the digit. Finally, Fig 17 provides examples of the pixel-wise decomposition for 16 randomly drawn digits and the predictors for all 10 digit classes.

Here pixels are flipped away from the class label given in parentheses above the heatmap. Pixels were flipped in steps of 1% of all pixels until the predicted class label changed. The plots show the output of the softmax function y and the output score of the preceding linear layer yp. The pixels were sorted before flipping in decreasing order of the pixel-wise score, i.e. highest scoring pixels were flipped first. In this panel the heatmap was computed for the classifier which produced the highest score, i.e. for the predicted class label. The originally predicted label is given on the leftmost image in parentheses, the predicted label after the switch of the prediction is given in the rightmost image.

https://doi.org/10.1371/journal.pone.0130140.g023

Here pixels are flipped towards the class label given in parentheses above the heatmap. Pixels were flipped in steps of 1% of all pixels until the predicted class label changed. The plots show the output of the softmax function y and the output score of the preceding linear layer yp. The pixels were sorted before flipping in increasing order of the pixel-wise score, i.e. lowest scoring pixels were flipped first. In this panel the heatmap was computed for a classifier which did not produce the highest score, i.e. for a random false class label. The originally predicted label is given on the leftmost image in parentheses, the predicted label after the switch of the prediction is given in the rightmost image.

https://doi.org/10.1371/journal.pone.0130140.g024

Neural Network for 1000 ILSVRC classes

We use here the pre-trained neural network which is provided by the Caffe open source package [60]. Fig 25 shows pixel-wise decompositions for example images from categories of the ILSVRC data set. Results are shown on images from Wikimedia Commons rather than the ILSVRC images in order to avoid license problems when showing images. Unsurprisingly the evidence is not as sharp as for the MNIST data set because the underlying classification problem is much more diverse with a higher complexity of classes and images to be processed. We can see for the rooster high scores at his cockscomb and parts of his feathers. The black widow spider has high evidence in her head section which is surprising because it is not so prominent to humans but is consistent among other images of black widow spiders as well. Further high evidence lies in parts of black widow legs, and, as expected, the characteristic red spots of a black widow. For the tabby cat we see evidence in his fur but also in the road surface under the cat which has similar color and texture to a cat’s fur. The espresso cup has high responses on the cup’s edges, the grip, some evidence in the espresso liquid, however only little response for the edges on the accompanying sweets which has the wrong color, gray, for an espresso. Note that a good classifier for a concept does not necessarily imply that an object can be localized very well by its pixel-wise decomposition. Since the classifier processes an image globally, it can perceive evidence away from the actual object, for example by taking into account typical background of objects, the context of an object or global statistics which are correlated to an object. In that sense, for example, bottles can serve as evidence for a dining table or sky can be a clue for photos with airplanes on the ground or photos of outside views of churches.

Second column shows decompositions computed by Formula (58) with stabilizers ɛ = 0.01, the third column with stabilizers ɛ = 100, the fourth column was computed by Formula (60) using α = +2, β = −1. The artifacts at the edges of the images are caused by filling the image with locally constant values which comes from the requirement to input square sub-parts of images into the neural net. Pictures in order of appearance from Wikimedia Commons by authors Jens Nietschmann, Shenrich91, Sandstein, Jörg Hempel.

https://doi.org/10.1371/journal.pone.0130140.g025

Fig 26 shows examples where the pixel-wise decomposition does not yield discriminative results. For the toilet paper, positive prediction scores were already low in both examples. This corresponds to our expectation that a less discriminative classifier results also in less convincing pixel-wise decompositions.

Left and Right: Failures to recognize toilet paper. The decompositions computed by Formula (58) with stabilizers ɛ = 0.01 The neural net is the pre-trained one on ILSVRC data from the Caffe package [60]. The computing methods for each column are the same as in Fig 25. Pictures in order of appearance from Wikimedia Commons by authors Robinhood of the Burger World and Taro the Shiba Inu.

https://doi.org/10.1371/journal.pone.0130140.g026

Comparing the different shown methods in Figs 25 and 27, one can see that the method (58) with a small value of the stabilizer ϵ produces results with the finest granularity and also with the highest amount of negative evidence. When increasing the values of the stabilizer ϵ, the method becomes less granular and shows less negative evidence. This result was also observed when comparing other small and large values of the stabilizer ϵ. The decrease of negative pixel-wise scores and of granularity can be both explained under some assumptions. Firstly, note that compared to Eq (58) with stabilizer equal to 0 the Eq (58) with stabilizer ϵ differs by a multiplicative factor of which goes to zero when ∣zj∣ goes to zero. Thus dampening by this factor increases when the absolute value of inputs ∣zj∣ decreases. Secondly note, that in Eq (58) the sign of the relevance Rj of input neuron j will be flipped in the messages for either the positive or the negative inputs zij (depending on the sign of ). When we assume that the neurons at layer l + 1 with small absolute value of inputs ∣zj∣ are responsible for generating a high proportion of all negative messages , then the reduction of negative pixel-wise scores by increasing stabilizer ϵ can be explained. Such an assumption can be reasonable because a small absolute value of inputs can sometimes be explained by the sum of positive inputs being close to the sum negative inputs, and noting that either the positive or the negative sets will produce negative messages . Therefore, in case of small inputs, sometimes a large fraction of the relevance Rj can be turned into negative messages. On the other side, one can expect that in a neural network which predicts a positive score f(x) > 0, a majority of network inputs is positive, and a majority of neuron relevances Rj is positive, since their sum within a layer equals f(x) > 0. In particular one may expect from that, that the majority of neurons with large inputs zj have positive relevances Rj > 0, and the messages for neurons with large inputs zj are mostly positive. In conclusion, positive scores are less affected by dampening when f(x) > 0. Canceling out negative scores by a large value of ϵ in intermediate layers leads also to more positive scores observed in the final heatmaps, and thus also to less visual granularity. We emphasize that this is a qualitative argument which we validated by checking a small number images rather than a universally holding claim. As for properties of Eq (60), we note that a choice of β < 0 such that α = 1 − β fixes the ratio of negative to positive messages to for each neuron. In particular, the sum of the negative messages is upper bounded by the sum of the positive messages. For rectified linear neurons this is a reasonable assumption because they fire only when their input is positive. In conclusion, our choice β = −1, yielding a ratio of 1:2 towards positive messages explains why the heatmaps for the method (60) in Figs 25 and 27 are dominantly positive.

Only a subset of strong edges and textures receive high scores. Panels show the original image on the left, and the decomposition on the right. The decompositions were computed twice for the classes table lamp and once for the class rooster. The neural net is the pre-trained one on ILSVRC data from the Caffe package [60]. The computing methods for each column are the same as in Fig 25. The last row shows the gradient norms normalized to lie in [0, 1] mapped by the same color scheme as for the heatmaps. Pictures in order of appearance from Wikimedia Commons by authors Wtshymanski, Serge Ninanne and Immanuel Clio.

https://doi.org/10.1371/journal.pone.0130140.g027

It is known from [21] that lower layers of deep networks which are very similar to the one from [60] used here may act like learned texture or gradient detectors. Fig 27 shows that the pixel-wise decomposition is, however, not equivalent to simply assigning scores to dominant edges. The original images from Fig 27 have many regions with strong gradients, however the pixel-wise decomposition assigns notable scores only to a small fraction of pixels with large gradients. Note for example that the highly textured and visually prominent wallpaper in the middle example with the table lamp receives only small scores. The same holds for the artifacts at the edges of the images which come from filling the image with locally constant values which is a compromise in order to make these images to be a square.