Explaining Deep Learning Hidden Neuron Activations using Concept Induction

• URL: http://arxiv.org/abs/2301.09611v1
• Date: Mon, 23 Jan 2023 18:14:32 GMT
• Title: Explaining Deep Learning Hidden Neuron Activations using Concept Induction
• Authors: Abhilekha Dalal, Md Kamruzzaman Sarker, Adrita Barua, and Pascal Hitzler
• Abstract summary: State of the art indicates that hidden node activations appear to be interpretable in a way that makes sense to humans. We show that we can automatically attach meaningful labels from the background knowledge to individual neurons in the dense layer of a Convolutional Neural Network.
• Score: 3.6223658572137825
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: One of the current key challenges in Explainable AI is in correctly interpreting activations of hidden neurons. It seems evident that accurate interpretations thereof would provide insights into the question what a deep learning system has internally \emph{detected} as relevant on the input, thus lifting some of the black box character of deep learning systems. The state of the art on this front indicates that hidden node activations appear to be interpretable in a way that makes sense to humans, at least in some cases. Yet, systematic automated methods that would be able to first hypothesize an interpretation of hidden neuron activations, and then verify it, are mostly missing. In this paper, we provide such a method and demonstrate that it provides meaningful interpretations. It is based on using large-scale background knowledge -- a class hierarchy of approx. 2 million classes curated from the Wikipedia Concept Hierarchy -- together with a symbolic reasoning approach called \emph{concept induction} based on description logics that was originally developed for applications in the Semantic Web field. Our results show that we can automatically attach meaningful labels from the background knowledge to individual neurons in the dense layer of a Convolutional Neural Network through a hypothesis and verification process.

Related papers

• On the Value of Labeled Data and Symbolic Methods for Hidden Neuron Activation Analysis [1.55858752644861]
  State of the art indicates that hidden node activations can, in some cases, be interpretable in a way that makes sense to humans. We introduce a novel model-agnostic post-hoc Explainable AI method demonstrating that it provides meaningful interpretations.
  arXiv  Detail & Related papers  (2024-04-21T07:57:45Z)
• Simple and Effective Transfer Learning for Neuro-Symbolic Integration [50.592338727912946]
  A potential solution to this issue is Neuro-Symbolic Integration (NeSy), where neural approaches are combined with symbolic reasoning. Most of these methods exploit a neural network to map perceptions to symbols and a logical reasoner to predict the output of the downstream task. They suffer from several issues, including slow convergence, learning difficulties with complex perception tasks, and convergence to local minima. This paper proposes a simple yet effective method to ameliorate these problems.
  arXiv  Detail & Related papers  (2024-02-21T15:51:01Z)
• Understanding CNN Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning [3.6223658572137825]
  State of the art indicates that hidden node activations can, in some cases, be interpretable in a way that makes sense to humans. We show that we can automatically attach meaningful labels from the background knowledge to individual neurons in the dense layer of a Convolutional Neural Network.
  arXiv  Detail & Related papers  (2023-08-08T02:28:50Z)
• NeuroExplainer: Fine-Grained Attention Decoding to Uncover Cortical Development Patterns of Preterm Infants [73.85768093666582]
  We propose an explainable geometric deep network dubbed NeuroExplainer. NeuroExplainer is used to uncover altered infant cortical development patterns associated with preterm birth.
  arXiv  Detail & Related papers  (2023-01-01T12:48:12Z)
• Mapping Knowledge Representations to Concepts: A Review and New Perspectives [0.6875312133832078]
  This review focuses on research that aims to associate internal representations with human understandable concepts. We find this taxonomy and theories of causality, useful for understanding what can be expected, and not expected, from neural network explanations. The analysis additionally uncovers an ambiguity in the reviewed literature related to the goal of model explainability.
  arXiv  Detail & Related papers  (2022-12-31T12:56:12Z)
• An Interpretable Neuron Embedding for Static Knowledge Distillation [7.644253344815002]
  We propose a new interpretable neural network method, by embedding neurons into the semantic space. The proposed semantic vector externalizes the latent knowledge to static knowledge, which is easy to exploit. Empirical experiments of visualization show that semantic vectors describe neuron activation semantics well.
  arXiv  Detail & Related papers  (2022-11-14T03:26:10Z)
• NELLIE: A Neuro-Symbolic Inference Engine for Grounded, Compositional, and Explainable Reasoning [59.16962123636579]
  This paper proposes a new take on Prolog-based inference engines. We replace handcrafted rules with a combination of neural language modeling, guided generation, and semi dense retrieval. Our implementation, NELLIE, is the first system to demonstrate fully interpretable, end-to-end grounded QA.
  arXiv  Detail & Related papers  (2022-09-16T00:54:44Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Expressive Explanations of DNNs by Combining Concept Analysis with ILP [0.3867363075280543]
  We use inherent features learned by the network to build a global, expressive, verbal explanation of the rationale of a feed-forward convolutional deep neural network (DNN) We show that our explanation is faithful to the original black-box model.
  arXiv  Detail & Related papers  (2021-05-16T07:00:27Z)
• This is not the Texture you are looking for! Introducing Novel Counterfactual Explanations for Non-Experts using Generative Adversarial Learning [59.17685450892182]
  counterfactual explanation systems try to enable a counterfactual reasoning by modifying the input image. We present a novel approach to generate such counterfactual image explanations based on adversarial image-to-image translation techniques. Our results show that our approach leads to significantly better results regarding mental models, explanation satisfaction, trust, emotions, and self-efficacy than two state-of-the art systems.
  arXiv  Detail & Related papers  (2020-12-22T10:08:05Z)
• Compositional Explanations of Neurons [52.71742655312625]
  We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts. We use this procedure to answer several questions on interpretability in models for vision and natural language processing.
  arXiv  Detail & Related papers  (2020-06-24T20:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.