ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations

• URL: http://arxiv.org/abs/2604.07019v1
• Date: Wed, 08 Apr 2026 12:37:39 GMT
• Title: ConceptTracer: Interactive Analysis of Concept Saliency and Selectivity in Neural Representations
• Abstract summary: ConceptTracer is an application for analyzing neural representations through the lens of human-interpretable concepts.<n>We show that ConceptTracer facilitates the discovery of interpretable neurons.
• Score: 1.338174941551702
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks deliver impressive predictive performance across a variety of tasks, but they are often opaque in their decision-making processes. Despite a growing interest in mechanistic interpretability, tools for systematically exploring the representations learned by neural networks in general, and tabular foundation models in particular, remain limited. In this work, we introduce ConceptTracer, an interactive application for analyzing neural representations through the lens of human-interpretable concepts. ConceptTracer integrates two information-theoretic measures that quantify concept saliency and selectivity, enabling researchers and practitioners to identify neurons that respond strongly to individual concepts. We demonstrate the utility of ConceptTracer on representations learned by TabPFN and show that our approach facilitates the discovery of interpretable neurons. Together, these capabilities provide a practical framework for investigating how neural networks like TabPFN encode concept-level information. ConceptTracer is available at https://github.com/ml-lab-htw/concept-tracer.

Related papers

• FaCT: Faithful Concept Traces for Explaining Neural Network Decisions [56.796533084868884]
  Deep networks have shown remarkable performance across a wide range of tasks, yet getting a global concept-level understanding of how they function remains a key challenge.<n>We put emphasis on the faithfulness of concept-based explanations and propose a new model with model-inherent mechanistic concept-explanations.<n>Our concepts are shared across classes and, from any layer, their contribution to the logit and their input-visualization can be faithfully traced.
  arXiv  Detail & Related papers  (2025-10-29T13:35:46Z)
• Concept-Guided Interpretability via Neural Chunking [64.6429903327095]
  We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract recurring chunks on a neural population level.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
  arXiv  Detail & Related papers  (2025-05-16T13:49:43Z)
• Neuro-Symbolic Concepts [72.94541757514396]
  This article presents a concept-centric paradigm for building agents that can learn continually and reason flexibly.<n>The concept-centric agent utilizes a vocabulary of neuro-symbolic concepts.<n>This framework offers several advantages, including data efficiency, compositional generalization, continual learning, and zero-shot transfer.
  arXiv  Detail & Related papers  (2025-05-09T17:02:51Z)
• Discovering Chunks in Neural Embeddings for Interpretability [53.80157905839065]
  We propose leveraging the principle of chunking to interpret artificial neural population activities.<n>We first demonstrate this concept in recurrent neural networks (RNNs) trained on artificial sequences with imposed regularities.<n>We identify similar recurring embedding states corresponding to concepts in the input, with perturbations to these states activating or inhibiting the associated concepts.
  arXiv  Detail & Related papers  (2025-02-03T20:30:46Z)
• Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning [2.9539724161670167]
  Deep reinforcement learning (DRL) has successfully addressed many complex control problems.<n>Current DRL interpretability methods largely treat neural networks as black boxes.<n>We propose a novel concept-based interpretability method that provides fine-grained explanations of DRL models at the neuron level.
  arXiv  Detail & Related papers  (2025-02-02T06:05:49Z)
• ConceptLens: from Pixels to Understanding [1.3466710708566176]
  ConceptLens is an innovative tool designed to illuminate the intricate workings of deep neural networks (DNNs) by visualizing hidden neuron activations. By integrating deep learning with symbolic methods, ConceptLens offers users a unique way to understand what triggers neuron activations.
  arXiv  Detail & Related papers  (2024-10-04T20:49:12Z)
• Contrastive Learning in Memristor-based Neuromorphic Systems [55.11642177631929]
  Spiking neural networks have become an important family of neuron-based models that sidestep many of the key limitations facing modern-day backpropagation-trained deep networks. In this work, we design and investigate a proof-of-concept instantiation of contrastive-signal-dependent plasticity (CSDP), a neuromorphic form of forward-forward-based, backpropagation-free learning.
  arXiv  Detail & Related papers  (2024-09-17T04:48:45Z)
• LLM-assisted Concept Discovery: Automatically Identifying and Explaining Neuron Functions [15.381209058506078]
  Prior works have associated concepts with neurons based on examples of concepts or a pre-defined set of concepts. We propose to leverage multimodal large language models for automatic and open-ended concept discovery. We validate each concept by generating examples and counterexamples and evaluating the neuron's response on this new set of images.
  arXiv  Detail & Related papers  (2024-06-12T18:19:37Z)
• Manipulating Feature Visualizations with Gradient Slingshots [53.94925202421929]
  Feature Visualization (FV) is a widely used technique for interpreting the concepts learned by Deep Neural Networks (DNNs)<n>We introduce a novel method, Gradient Slingshots, that enables manipulation of FV without modifying the model architecture or significantly degrading its performance.
  arXiv  Detail & Related papers  (2024-01-11T18:57:17Z)
• Finding Concept Representations in Neural Networks with Self-Organizing Maps [2.817412580574242]
  We show how self-organizing maps can be used to inspect how activation of layers of neural networks correspond to neural representations of abstract concepts. We show that, among the measures tested, the relative entropy of the activation map for a concept is a suitable candidate and can be used as part of a methodology to identify and locate the neural representation of a concept.
  arXiv  Detail & Related papers  (2023-12-10T12:10:34Z)
• Formal Conceptual Views in Neural Networks [0.0]
  We introduce two notions for conceptual views of a neural network, specifically a many-valued and a symbolic view. We test the conceptual expressivity of our novel views through different experiments on the ImageNet and Fruit-360 data sets. We demonstrate how conceptual views can be applied for abductive learning of human comprehensible rules from neurons.
  arXiv  Detail & Related papers  (2022-09-27T16:38:24Z)
• Human-Centered Concept Explanations for Neural Networks [47.71169918421306]
  We introduce concept explanations including the class of Concept Activation Vectors (CAV) We then discuss approaches to automatically extract concepts, and approaches to address some of their caveats. Finally, we discuss some case studies that showcase the utility of such concept-based explanations in synthetic settings and real world applications.
  arXiv  Detail & Related papers  (2022-02-25T01:27:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.