Towards falsifiable interpretability research

• URL: http://arxiv.org/abs/2010.12016v1
• Date: Thu, 22 Oct 2020 22:03:41 GMT
• Title: Towards falsifiable interpretability research
• Authors: Matthew L. Leavitt, Ari Morcos
• Abstract summary: We argue that interpretability research suffers from an over-reliance on intuition-based approaches. We examine two popular classes of interpretability methods-saliency and single-neuron-based approaches. We propose a strategy to address these impediments in the form of a framework for strongly falsifiable interpretability research.
• Score: 7.360807642941714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Methods for understanding the decisions of and mechanisms underlying deep neural networks (DNNs) typically rely on building intuition by emphasizing sensory or semantic features of individual examples. For instance, methods aim to visualize the components of an input which are "important" to a network's decision, or to measure the semantic properties of single neurons. Here, we argue that interpretability research suffers from an over-reliance on intuition-based approaches that risk-and in some cases have caused-illusory progress and misleading conclusions. We identify a set of limitations that we argue impede meaningful progress in interpretability research, and examine two popular classes of interpretability methods-saliency and single-neuron-based approaches-that serve as case studies for how overreliance on intuition and lack of falsifiability can undermine interpretability research. To address these concerns, we propose a strategy to address these impediments in the form of a framework for strongly falsifiable interpretability research. We encourage researchers to use their intuitions as a starting point to develop and test clear, falsifiable hypotheses, and hope that our framework yields robust, evidence-based interpretability methods that generate meaningful advances in our understanding of DNNs.

Related papers

• Statistical tuning of artificial neural network [0.0]
  This study introduces methods to enhance the understanding of neural networks, focusing specifically on models with a single hidden layer. We propose statistical tests to assess the significance of input neurons and introduce algorithms for dimensionality reduction. This research advances the field of Explainable Artificial Intelligence by presenting robust statistical frameworks for interpreting neural networks.
  arXiv  Detail & Related papers  (2024-09-24T19:47:03Z)
• Independence Constrained Disentangled Representation Learning from Epistemological Perspective [13.51102815877287]
  Disentangled Representation Learning aims to improve the explainability of deep learning methods by training a data encoder that identifies semantically meaningful latent variables in the data generation process. There is no consensus regarding the objective of disentangled representation learning. We propose a novel method for disentangled representation learning by employing an integration of mutual information constraint and independence constraint.
  arXiv  Detail & Related papers  (2024-09-04T13:00:59Z)
• The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms [3.3653074379567096]
  mechanistic interpretability (MI) has emerged as a distinct research area studying the features and implicit algorithms learned by foundation models such as large language models. We argue that current methods are ripe to facilitate a transition in deep learning interpretation echoing the "cognitive revolution" in 20th-century psychology. We propose a taxonomy mirroring key parallels in computational neuroscience to describe two broad categories of MI research.
  arXiv  Detail & Related papers  (2024-08-11T20:50:16Z)
• A Survey on Transferability of Adversarial Examples across Deep Neural Networks [53.04734042366312]
  adversarial examples can manipulate machine learning models into making erroneous predictions. The transferability of adversarial examples enables black-box attacks which circumvent the need for detailed knowledge of the target model. This survey explores the landscape of the adversarial transferability of adversarial examples.
  arXiv  Detail & Related papers  (2023-10-26T17:45:26Z)
• Adversarial Attacks on the Interpretation of Neuron Activation Maximization [70.5472799454224]
  Activation-maximization approaches are used to interpret and analyze trained deep-learning models. In this work, we consider the concept of an adversary manipulating a model for the purpose of deceiving the interpretation.
  arXiv  Detail & Related papers  (2023-06-12T19:54:33Z)
• Interpreting Neural Policies with Disentangled Tree Representations [58.769048492254555]
  We study interpretability of compact neural policies through the lens of disentangled representation. We leverage decision trees to obtain factors of variation for disentanglement in robot learning. We introduce interpretability metrics that measure disentanglement of learned neural dynamics.
  arXiv  Detail & Related papers  (2022-10-13T01:10:41Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks [1.5854438418597576]
  We present gradient-based interpretability methods for explaining decisions of deep neural networks. We discuss the role that adversarial robustness plays in having meaningful explanations. We conclude with the future directions for research in the area at the convergence of robustness and explainability.
  arXiv  Detail & Related papers  (2021-07-23T18:06:29Z)
• ACRE: Abstract Causal REasoning Beyond Covariation [90.99059920286484]
  We introduce the Abstract Causal REasoning dataset for systematic evaluation of current vision systems in causal induction. Motivated by the stream of research on causal discovery in Blicket experiments, we query a visual reasoning system with the following four types of questions in either an independent scenario or an interventional scenario. We notice that pure neural models tend towards an associative strategy under their chance-level performance, whereas neuro-symbolic combinations struggle in backward-blocking reasoning.
  arXiv  Detail & Related papers  (2021-03-26T02:42:38Z)
• Interpretable Deep Learning: Interpretations, Interpretability, Trustworthiness, and Beyond [49.93153180169685]
  We introduce and clarify two basic concepts-interpretations and interpretability-that people usually get confused. We elaborate the design of several recent interpretation algorithms, from different perspectives, through proposing a new taxonomy. We summarize the existing work in evaluating models' interpretability using "trustworthy" interpretation algorithms.
  arXiv  Detail & Related papers  (2021-03-19T08:40:30Z)
• Adversarial Examples on Object Recognition: A Comprehensive Survey [1.976652238476722]
  Deep neural networks are at the forefront of machine learning research. adversarial examples are intentionally designed to test the network's sensitivity to distribution drifts. We discuss the impact of adversarial examples on security, safety, and robustness of neural networks.
  arXiv  Detail & Related papers  (2020-08-07T08:51:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.