Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation

• URL: http://arxiv.org/abs/2603.24953v1
• Date: Thu, 26 Mar 2026 02:45:41 GMT
• Title: Select, Hypothesize and Verify: Towards Verified Neuron Concept Interpretation
• Abstract summary: Existing approaches describe neuron concepts by generating natural language descriptions.<n>In fact, some neurons may be redundant or may offer misleading concepts.<n>We introduce a verification of neuron functions, which checks whether the generated concept highly activates the corresponding neuron.
• Score: 24.76656635414083
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: It is essential for understanding neural network decisions to interpret the functionality (also known as concepts) of neurons. Existing approaches describe neuron concepts by generating natural language descriptions, thereby advancing the understanding of the neural network's decision-making mechanism. However, these approaches assume that each neuron has well-defined functions and provides discriminative features for neural network decision-making. In fact, some neurons may be redundant or may offer misleading concepts. Thus, the descriptions for such neurons may cause misinterpretations of the factors driving the neural network's decisions. To address the issue, we introduce a verification of neuron functions, which checks whether the generated concept highly activates the corresponding neuron. Furthermore, we propose a Select-Hypothesize-Verify framework for interpreting neuron functionality. This framework consists of: 1) selecting activation samples that best capture a neuron's well-defined functional behavior through activation-distribution analysis; 2) forming hypotheses about concepts for the selected neurons; and 3) verifying whether the generated concepts accurately reflect the functionality of the neuron. Extensive experiments show that our method produces more accurate neuron concepts. Our generated concepts activate the corresponding neurons with a probability approximately 1.5 times that of the current state-of-the-art method.

Related papers

• In Search of Grandmother Cells: Tracing Interpretable Neurons in Tabular Representations [1.503974529275767]
  We show that some neurons show moderate, statistically significant saliency and selectivity for high-level concepts.<n>These findings suggest that interpretable neurons can emerge naturally and that they can, in some cases, be identified without resorting to more complex interpretability techniques.
  arXiv  Detail & Related papers  (2026-01-07T07:13:01Z)
• Interpreting the Second-Order Effects of Neurons in CLIP [73.54377859089801]
  We interpret the function of individual neurons in CLIP by automatically describing them using text.<n>We present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output.<n>Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.
  arXiv  Detail & Related papers  (2024-06-06T17:59:52Z)
• Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment [65.268245109828]
  We examine algorithms for conducting credit assignment in artificial neural networks that are inspired or motivated by neurobiology. We organize the ever-growing set of brain-inspired learning schemes into six general families and consider these in the context of backpropagation of errors. The results of this review are meant to encourage future developments in neuro-mimetic systems and their constituent learning processes.
  arXiv  Detail & Related papers  (2023-12-01T05:20:57Z)
• Identifying Interpretable Visual Features in Artificial and Biological Neural Systems [3.604033202771937]
  Single neurons in neural networks are often interpretable in that they represent individual, intuitively meaningful features. Many neurons exhibit $textitmixed selectivity$, i.e., they represent multiple unrelated features. We propose an automated method for quantifying visual interpretability and an approach for finding meaningful directions in network activation space.
  arXiv  Detail & Related papers  (2023-10-17T17:41:28Z)
• Disentangling Neuron Representations with Concept Vectors [0.0]
  The main contribution of this paper is a method to disentangle polysemantic neurons into concept vectors encapsulating distinct features. Our evaluations show that the concept vectors found encode coherent, human-understandable features.
  arXiv  Detail & Related papers  (2023-04-19T14:55:31Z)
• Constraints on the design of neuromorphic circuits set by the properties of neural population codes [61.15277741147157]
  In the brain, information is encoded, transmitted and used to inform behaviour. Neuromorphic circuits need to encode information in a way compatible to that used by populations of neuron in the brain.
  arXiv  Detail & Related papers  (2022-12-08T15:16:04Z)
• POPPINS : A Population-Based Digital Spiking Neuromorphic Processor with Integer Quadratic Integrate-and-Fire Neurons [50.591267188664666]
  We propose a population-based digital spiking neuromorphic processor in 180nm process technology with two hierarchy populations. The proposed approach enables the developments of biomimetic neuromorphic system and various low-power, and low-latency inference processing applications.
  arXiv  Detail & Related papers  (2022-01-19T09:26:34Z)
• NeuroCartography: Scalable Automatic Visual Summarization of Concepts in Deep Neural Networks [18.62960153659548]
  NeuroCartography is an interactive system that summarizes and visualizes concepts learned by neural networks. It automatically discovers and groups neurons that detect the same concepts. It describes how such neuron groups interact to form higher-level concepts and the subsequent predictions.
  arXiv  Detail & Related papers  (2021-08-29T22:43:52Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Neuron-based explanations of neural networks sacrifice completeness and interpretability [67.53271920386851]
  We show that for AlexNet pretrained on ImageNet, neuron-based explanation methods sacrifice both completeness and interpretability.<n>We show the most important principal components provide more complete and interpretable explanations than the most important neurons.<n>Our findings suggest that explanation methods for networks like AlexNet should avoid using neurons as a basis for embeddings.
  arXiv  Detail & Related papers  (2020-11-05T21:26:03Z)
• Compositional Explanations of Neurons [52.71742655312625]
  We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts. We use this procedure to answer several questions on interpretability in models for vision and natural language processing.
  arXiv  Detail & Related papers  (2020-06-24T20:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.