Keywords

1 Introduction

Artificial neural networks enjoy increasing popularity for image classification tasks. They have shown excellent performance in large scale competitions [4]. One reason is the ability to train neural networks with millions of training samples by parallelizing them on GPU hardware. This allows to use numbers of training samples which match the large number of parameters in deep neural networks. However, understanding what region of the image is important for a classification decision, is still an open question for neural networks, as well as for many other non-linear models. The work of [1] proposed Layer-wise Relevance Propagation (LRP) as a solution for explaining what pixels of an image are relevant for reaching a classification decision. This was done for neural networks, bag of word models [2, 10], and in a subsequent work [5], for Fisher vectors.

This paper proposes an approach to extend LRP to neural networks with nonlinearities beyond the commonly used neural network formulation. One example of such nonlinearities are local renormalization layers which can not be handled by standard LRP [1]. The presented approach is based on first (or higher) order Taylor expansion. We consider a classification setup with real-valued outputs. A classifier f is a mapping of an input space \(f: X \rightarrow \mathbb {R}\) such that \(f(x)>0\) denotes the presence of the class.

2 Layer-Wise Relevance Propagation for Neural Networks

In the following we consider neural networks consisting of layers of neurons. The output \(x_j\) of a neuron j is a non-linear activation function g as given by

$$\begin{aligned} x_j&= g\Big ( {\textstyle \sum _i} w_{ij} x_i +b \Big ) \end{aligned}$$
(1)

Given an image x and a classifier f the aim of layer-wise relevance propagation is to assign each pixel p of x a pixel-wise relevance score \(R^{(1)}_p\) such that

$$\begin{aligned} f(x) \approx {\textstyle \sum _p} R^{(1)}_p \end{aligned}$$
(2)

Pixels p with \(R^{(1)}_p<0\) contain evidence against the presence of a class, while \(R^{(1)}_p>0\) is considered as evidence for the presence of a class. These pixel-wise relevance scores can be visualized as an image called heatmap (see Fig. 1 for examples). Obviously, many possible such decompositions exist which satisfy Eq. 2. The work of [1] yield pixel-wise decompositions which are consistent with evaluation measures [8] and human intuition.

Fig. 1.
figure 1

Pixel-wise decompositions for classes wolf, frog and wolf using a neural network pretrained for the 1000 classes of the ILSVRC challenge.

Full size image

Assume that we know the relevance \(R^{(l+1)}_j\) of a neuron j at network layer \(l+1\) for the classification decision f(x), then we like to decompose this relevance into messages \(R^{(l,l+1)}_{i \leftarrow j}\) sent to those neurons i at the layer l which provide inputs to neuron j such that Eq. 3 holds.

$$\begin{aligned} R^{(l+1)}_j=\sum _{i \in (l)} R^{(l,l+1)}_{i \leftarrow j} \end{aligned}$$
(3)

We can then define the relevance of a neuron i at layer l by summing all messages from neurons at layer \(l+1\) as in Eq. 4

$$\begin{aligned} R^{(l)}_i=\sum _{j \in (l+1)} R^{(l,l+1)}_{i \leftarrow j} \end{aligned}$$
(4)

Equations 3 and 4 define the propagation of relevance from layer \(l+1\) to layer l. The relevance of the output neuron at layer M is \(R^{(M)}_1=f(x)\). The pixel-wise scores are the resulting relevances of the input neurons \(R^{(1)}_d\).

The work in [1] established two formulas for computing the messages \(R^{(l,l+1)}_{i \leftarrow j}\). The first formula called \(\epsilon \)-rule is given by

$$\begin{aligned} R^{(l,l+1)}_{i \leftarrow j}&= \frac{ z_{ij} }{ z_j + \epsilon \cdot \mathrm {sign}(z_j) }R^{(l+1)}_j \end{aligned}$$
(5)

with \(z_{ij} = (w_{ij} x_i )^p\) and \(z_j = \sum _{k: w_{kj} \ne 0 } z_{kj}\). The variable \(\epsilon \) is a “stabilizer” term whose purpose is to avoid numerical degenerations when \(z_j\) is close to zero, and which is chosen to be small. The second formula called \(\beta \)-rule is given by

$$\begin{aligned} R^{(l,l+1)}_{i \leftarrow j}&= \Big ((1+\beta )\frac{ z_{ij}^{+} }{ z_j^{+} }-\beta \frac{ z_{ij}^{-} }{ z_j^{-} } \Big ) R^{(l+1)}_j \end{aligned}$$
(6)

where the positive and negative weighted activations are treated separately. The variable \(\beta \) controls how much inhibition is incorporated in the relevance redistribution. A fairly large value for \(\beta \) (e.g. \(\beta = 1\)) leads to sharper heatmaps. In both formulas the message \(R^{(l,l+1)}_{i \leftarrow j}\) has the following structure

$$\begin{aligned} R^{(l,l+1)}_{i \leftarrow j}&= v_{ij} R^{(l+1)}_j \quad \mathrm {with} \quad {\textstyle \sum _i} v_{ij} = 1 \end{aligned}$$
(7)

The meaningfulness of the resulting pixel-wise decomposition for the input layer \(R_d^{(1)}\) comes from the fact that the terms \(v_{ij}\) are derived from the weighted activations \(w_{ij}x_i\) of the input neurons. Note that layer-wise relevance propagation does not use gradients in contrast to backpropagation during the training phase. For full details on layer-wise relevance propagation the reader is referred to [1].

3 Extending LRP to Local Renormalization Layers

We consider a general neuron j whose pooling and activation does not fit into the structure given by Eq. 1, and consequently, intuition for a possible redistribution formula is lacking. In this paper we propose a strategy for such neurons, based on the Taylor expansion of its activation function. A Taylor-based approach was used in [6] for decomposing ReLU neurons by exploiting their local linearity. Here, we consider instead fully nonlinear neurons.

Suppose we can define for each neuron i input to neuron j a term \(v_{ij}\) which is derived from its activation \(x_i\) such that \(\sum _i v_{ij}=1\). Then we can define a message \(R^{(l,l+1)}_{i \leftarrow j} = v_{ij} R^{(l+1)}_j \). Such messages were used in Eqs. 5 and 6 where the weighting \(v_{ij}\) was chosen to depend on the weighted activations of neuron i: \(v_{ij} = c \, (w_{ij} x_i )^p\) and \(v_{ij} = c_1 z_{ij}^{+}+c_2 z_{ij}^{-}\), respectively. For differentiable neurons, such weighting can be obtained by performing a first order Taylor expansion. Let \(x_j = g ( x_{h_1}, \ldots , x_{h_n})\) be a nonlinear activation function. Then, by Taylor expansion at some reference point \(( \widetilde{x}_{h_1}, \ldots , \widetilde{x}_{h_n} )\), we get

$$\begin{aligned} x_j \approx g( \widetilde{x}_{h_1}, \ldots , \widetilde{x}_{h_n} ) + \sum _{i \leftarrow j}\frac{\partial g}{\partial x_{h_i}}( \widetilde{x}_{h_1}, \ldots , \widetilde{x}_{h_n} ) (x_{h_i}-\widetilde{x}_{h_i}). \end{aligned}$$
(8)

Elements of the sum can be assigned to incoming neurons, and the zero-order term can be redistributed equally between them, leading to the decomposition

$$\begin{aligned} \forall _{i \leftarrow j}:~ z_{ij}&= \frac{1}{n} \,g( \widetilde{x}_{h_1}, \ldots , \widetilde{x}_{h_n} ) + \frac{\partial g}{\partial x_{h_i}}( \widetilde{x}_{h_1}, \ldots , \widetilde{x}_{h_n} ) (x_{h_i}-\widetilde{x}_{h_i}) \end{aligned}$$
(9)

of the neuron activation onto its input neurons. Local renormalization layers have been shown to improve the performance in deep neural networks [4]. Consider the local renormalization \(y_k\) of a neuron \(x_k\) by the set of its surrounding neurons \(\{x_1, \ldots , x_n\}\) as

$$\begin{aligned} y_k(x_1, \ldots , x_n)&= \frac{x_k}{ \left( 1+ b \sum _{i=1}^n x_i^2 \right) ^c } \end{aligned}$$
(10)

This interaction can be modeled by a layer in the network that has an activation function as given in Eq. 10. Local renormalization layers represent a non-linearity which cannot be tackled exactly by LRP as introduced in [1], however the strategy proposed above can be applied.

One choice to be made is the point at which to perform the Taylor expansion. There are two apparent candidates, firstly the actual input to the renormalization layer \(z_1=(x_1, \ldots , x_n)\) and, secondly, the input corresponding to the case when only the neuron k fires which is to be normalized \(z_2=(0, ... \ldots ,0, x_k, 0, \ldots , 0)\). The partial derivative of y at \(z_2\) is zero for all variables \(x_i\) with \(i \ne k\) due to

$$\begin{aligned} \frac{\partial y_k}{\partial x_j}&= \frac{\delta _{kj}}{ \left( 1+ b \sum _{i=1}^n x_i^2 \right) ^{c} }-2bc\frac{x_k x_j}{ \left( 1+ b \sum _{i=1}^n x_i^2 \right) ^{c+1} } \end{aligned}$$
(11)

This implies that the Taylor approximation has no off-diagonal contribution.

$$\begin{aligned} y_k(z_1) \approx y_k(z_2)+0=\frac{x_k}{(1+bx_k^2)^c} \end{aligned}$$
(12)

Therefore we apply the Taylor series around the point \(z_1\):

$$\begin{aligned} y_k(z_2)&\approx y_k(z_1) +\nabla y_k(z_1) \cdot (z_2-z_1)\end{aligned}$$
(13)
$$\begin{aligned} \Rightarrow y_k(z_1)&\approx y_k(z_2) +\nabla y_k(z_1) \cdot (z_1-z_2) \end{aligned}$$
(14)
$$\begin{aligned} \Rightarrow y_k(z_1)&\approx \frac{x_k}{(1+bx_k^2)^c}-2bc \sum _{j:j \ne k} \frac{x_k x_j^2}{ \left( 1+ b \sum _{i=1}^n x_i^2 \right) ^{c+1} } \end{aligned}$$
(15)

This weighting satisfies the following qualitative properties: for the neuron input \(x_k\) which is to be normalized, the sign of the relevance is kept. For suppressing neighboring neurons \(x_i\), \(i \ne k\), the sign of the relevance can be flipped in line with their suppressing property. The absolute value of the relevance received by the suppressing neurons is proportional to the square of their input. In the limits \(c \rightarrow 0\) and \(b \rightarrow 0\), the local renormalization converges against the identity, and the approximation recovers the identity. A baseline to compare against is to treat the normalization as constant. In that case the weights \(v_{ij}\) for the relevance propagation in Eq. 3 become a zero one vector, the relevance is propagated only to that neuron which is to be normalized: \(v_{ij}=1\) if and only if i is the neuron which is to be normalized by neuron j.

4 Experiments

We need to define a measure for meaningfulness and quality of a pixel-wise decomposition in order to evaluate the various strategies to compute it. Here we use an idea from [8]: A pixel p is considered highly relevant for the classification score f(x) of the image x if modifying it by assigning it a random RGB value \(\tilde{x}(p)\), and classifying the modified image \(\bar{x}_{p}= x \setminus \{x(p)\} \cup \{\tilde{x}(p)\}\) results in a strong decrease of the real-valued classification score \(f(\bar{x}_{p})\). This idea can be extended by sequentially modifying pixels from the most relevant to the least relevant. The result is a graph of the prediction score \(f(\bar{x})\) as a function of the number of modified pixels. An example for some sequences which will be explained below is shown in Fig. 2. We can use these graphs to evaluate the meaningfulness of a pixel-wise decomposition.

Fig. 2.
figure 2

Decrease of classification score as pixels are sequentially replaced by random noise on the CIFAR-10 dataset. Red curve: pixels with highest pixel-wise scores are flipped first. Blue curve: pixels are flipped in random order. Green curve: least relevant pixels are flipped first. A similar comparison for Imagenet is found in [8]. (Color figure online)

Full size image

In the first experiment we compare the measure when flipping highest-scoring pixels first, against flipping pixels in random order, and against flipping lowest scoring pixels first. If the classifier is able to identify pixels that are important for classification, then flipping highest scoring pixels first should result in the fastest decaying curve, while flipping lowest scoring pixels first should result in the slowest decrease. Figure 2 tests this property on the CIFAR-10 dataset [3] which consists of 50000 images of size \(32\times 32\) drawn from 10 object classes. Scores are averaged over the 5000 images of the test set of CIFAR-10 for a classifier in which local renormalization layers are treated as the identity during computation of pixel-wise scores. Experiments corroborate that flipping highest scoring pixels first results in the fastest decrease of the prediction score on average over the test set. The decrease is sharper compared to random flipping, or flipping lowest scoring pixels first.

In a second experiment we compare which treatment of the local renormalization layer is best to identify those pixels that are most relevant for classifying an image. The two tested approaches for treating the local renormalization are (1) like it would be the identity, (2) by first order Taylor expansion as given by Eq. 15. These approaches are furthermore tested when used in conjunction with the two methods proposed by [1], namely, the \(\epsilon \)-rule in Eq. 5 with a fixed value of the numerical stabilizer \(\epsilon \), and the \(\beta \)-rule shown in Eq. 6, with fixed \(\beta \).

Table 1. Comparison of different types of LRN layer treatments for two approaches of computing pixel-wise scores for CIFAR-10. Lower scores are better.
Full size table

We measure the quality of heatmaps by perturbing highest pixels first and computing the area under the curve (AUC). Lower AUC averaged over a large number of images indicates a better identification of pixel relevance by the heatmap. Results on CIFAR-10 are shown in Table 1. We observe that in all cases using first order Taylor in normalization layers improves the heatmap AUC score. This shows its effectiveness for dealing with non-linear neuron layers.

Table 2. Comparison of different types of heatmap computations for Imagenet and MIT Places. We use the shortcut notation \(\varDelta _{a}^{b}\) for expressing \(\mathrm {AUC}_\mathrm {a} - \mathrm {AUC}_\mathrm {b}\). Thus, a negative value indicates that the method produces better heatmaps with parameter a than with parameter b. Note that \(\epsilon \) refers to Eqs. 4 and 5; \(\beta \) refers to Eqs. 4 and 6.
Full size table
Table 3. Impact of using the Taylor method in various settings. Negative value indicates that using the Taylor expansion for the local renormalization is better in AUC terms (i.e. heatmaps are more representative of the importance of each pixel).
Full size table
Fig. 3.
figure 3

Top row shows original unwarped image. Remaining rows show heatmaps produced by various parameters of the LRP method.

Full size image

We perform the same experiments also with Imagenet [7] and MIT Places [12] datasets, each time evaluating results for 5000 images from their respective unlabeled test sets. Note that computing a heatmap requires only a predicted class label, not a ground truth. We evaluated results for the parameter settings \(\beta =0\), \(\beta =1\) in Eq. 6 and \(\epsilon =0.01\), \(\epsilon =1\), \(\epsilon =100\) in Eq. 5. Table 2 shows the difference of AUC between variants of LRP, when using either the identity or the Taylor expansion for local renormalization layers. We observe the following ordering starting with the lowest (best) AUC: \(\epsilon =1\), \(\epsilon =0.01\), \(\epsilon =100\), \(\beta =1\), \(\beta =0\). This order holds independent of whether we consider Imagenet or MIT places, when using Taylor for local renormalization layers. When using identity instead of Taylor, the order remain the same, except for \(\epsilon =100\) and \(\epsilon =0.01\) that are swapped. This is by itself an interesting result demonstrating that use of Taylor in the normalization layer does not disrupt the overall properties of relevance propagation techniques. For a comparison to other approaches such as heatmaps based on deconvolutions [11], or backpropagated gradients [9] we refer to [8].

Table 3 shows the difference of AUC between Taylor and identity for local renormalization layers, for various choices of datasets and LRP parameters. We observe that for the parameters with best AUC (\(\epsilon =1\) and \(\epsilon =0.01\)), using Taylor expansion for representing local renormalization layers further improves the AUC scores. For the remaining choices the results are on par or slightly worse. This is consistent with the interpretation of large values of \(\epsilon \) as smoothing out small contributions. It is also consistent with the observation that \(\beta =1\) and \(\beta =0\) yield both smooth heatmaps in general. Heatmaps for some parameters of interest are shown in Fig. 3. Taylor with \(\epsilon =1\) has both high pixel selectivity and low noise, which in agreement with its measured superiority in the quantitative experiments.

5 Conclusion

We have presented an extension of layer-wise relevance propagation (LRP) based on first-order Taylor expansions for product-type nonlinearities. Such nonlinearities occur in the local renormalization layers of deep convolutional neural networks. The proposed extension is evaluated on three popular datasets and it is shown to clearly outperform the original LRP method. In future work we will investigate the potential gain of using higher order Taylor expansions, and apply the method to a larger class of neural network layers.