Rethinking the Role of Gradient-Based Attribution Methods for Model
Interpretability
- URL: http://arxiv.org/abs/2006.09128v2
- Date: Wed, 3 Mar 2021 09:42:58 GMT
- Title: Rethinking the Role of Gradient-Based Attribution Methods for Model
Interpretability
- Authors: Suraj Srinivas, Francois Fleuret
- Abstract summary: Current methods for interpretability of discriminative deep neural networks rely on the model's input-gradients.
We show that these input-gradients can be arbitrarily manipulated without changing the discriminative function.
- Score: 8.122270502556374
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Current methods for the interpretability of discriminative deep neural
networks commonly rely on the model's input-gradients, i.e., the gradients of
the output logits w.r.t. the inputs. The common assumption is that these
input-gradients contain information regarding $p_{\theta} ( y \mid x)$, the
model's discriminative capabilities, thus justifying their use for
interpretability. However, in this work we show that these input-gradients can
be arbitrarily manipulated as a consequence of the shift-invariance of softmax
without changing the discriminative function. This leaves an open question: if
input-gradients can be arbitrary, why are they highly structured and
explanatory in standard models?
We investigate this by re-interpreting the logits of standard softmax-based
classifiers as unnormalized log-densities of the data distribution and show
that input-gradients can be viewed as gradients of a class-conditional density
model $p_{\theta}(x \mid y)$ implicit within the discriminative model. This
leads us to hypothesize that the highly structured and explanatory nature of
input-gradients may be due to the alignment of this class-conditional model
$p_{\theta}(x \mid y)$ with that of the ground truth data distribution
$p_{\text{data}} (x \mid y)$. We test this hypothesis by studying the effect of
density alignment on gradient explanations. To achieve this alignment we use
score-matching, and propose novel approximations to this algorithm to enable
training large-scale models.
Our experiments show that improving the alignment of the implicit density
model with the data distribution enhances gradient structure and explanatory
power while reducing this alignment has the opposite effect. Overall, our
finding that input-gradients capture information regarding an implicit
generative model implies that we need to re-think their use for interpreting
discriminative models.
Related papers
- Black-Box Anomaly Attribution [2.22999837458579]
When a black-box machine learning model deviates from the true observation, what can be said about the reason behind that deviation?
This is a fundamental and ubiquitous question that the end user in a business or industrial AI application often asks.
We propose a novel likelihood-based attribution framework we call the likelihood compensation''
arXiv Detail & Related papers (2023-05-29T01:42:32Z) - Variational Classification [51.2541371924591]
We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders.
Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency.
We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
arXiv Detail & Related papers (2023-05-17T17:47:19Z) - CEnt: An Entropy-based Model-agnostic Explainability Framework to
Contrast Classifiers' Decisions [2.543865489517869]
We present a novel approach to locally contrast the prediction of any classifier.
Our Contrastive Entropy-based explanation method, CEnt, approximates a model locally by a decision tree to compute entropy information of different feature splits.
CEnt is the first non-gradient-based contrastive method generating diverse counterfactuals that do not necessarily exist in the training data while satisfying immutability (ex. race) and semi-immutability (ex. age can only change in an increasing direction)
arXiv Detail & Related papers (2023-01-19T08:23:34Z) - The Manifold Hypothesis for Gradient-Based Explanations [55.01671263121624]
gradient-based explanation algorithms provide perceptually-aligned explanations.
We show that the more a feature attribution is aligned with the tangent space of the data, the more perceptually-aligned it tends to be.
We suggest that explanation algorithms should actively strive to align their explanations with the data manifold.
arXiv Detail & Related papers (2022-06-15T08:49:24Z) - Discovering Invariant Rationales for Graph Neural Networks [104.61908788639052]
Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph's features.
We propose a new strategy of discovering invariant rationale (DIR) to construct intrinsically interpretable GNNs.
arXiv Detail & Related papers (2022-01-30T16:43:40Z) - Discrete Denoising Flows [87.44537620217673]
We introduce a new discrete flow-based model for categorical random variables: Discrete Denoising Flows (DDFs)
In contrast with other discrete flow-based models, our model can be locally trained without introducing gradient bias.
We show that DDFs outperform Discrete Flows on modeling a toy example, binary MNIST and Cityscapes segmentation maps, measured in log-likelihood.
arXiv Detail & Related papers (2021-07-24T14:47:22Z) - Causality-based Counterfactual Explanation for Classification Models [11.108866104714627]
We propose a prototype-based counterfactual explanation framework (ProCE)
ProCE is capable of preserving the causal relationship underlying the features of the counterfactual data.
In addition, we design a novel gradient-free optimization based on the multi-objective genetic algorithm that generates the counterfactual explanations.
arXiv Detail & Related papers (2021-05-03T09:25:59Z) - Why do classifier accuracies show linear trends under distribution
shift? [58.40438263312526]
accuracies of models on one data distribution are approximately linear functions of the accuracies on another distribution.
We assume the probability that two models agree in their predictions is higher than what we can infer from their accuracy levels alone.
We show that a linear trend must occur when evaluating models on two distributions unless the size of the distribution shift is large.
arXiv Detail & Related papers (2020-12-31T07:24:30Z) - How do Decisions Emerge across Layers in Neural Models? Interpretation
with Differentiable Masking [70.92463223410225]
DiffMask learns to mask-out subsets of the input while maintaining differentiability.
Decision to include or disregard an input token is made with a simple model based on intermediate hidden layers.
This lets us not only plot attribution heatmaps but also analyze how decisions are formed across network layers.
arXiv Detail & Related papers (2020-04-30T17:36:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.