Disentangling Polysemantic Neurons with a Null-Calibrated Polysemanticity Index and Causal Patch Interventions

• URL: http://arxiv.org/abs/2508.16950v1
• Date: Sat, 23 Aug 2025 08:48:59 GMT
• Title: Disentangling Polysemantic Neurons with a Null-Calibrated Polysemanticity Index and Causal Patch Interventions
• Abstract summary: Polysemanticity Index (PSI) is a null-calibrated metric that quantifies when a neuron's top activations decompose into semantically distinct clusters.<n>On a pretrained ResNet-50 evaluated with Tiny-ImageNet images, PSI identifies neurons whose activation sets split into coherent, nameable prototypes.
• Score: 4.032680910442999
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks often contain polysemantic neurons that respond to multiple, sometimes unrelated, features, complicating mechanistic interpretability. We introduce the Polysemanticity Index (PSI), a null-calibrated metric that quantifies when a neuron's top activations decompose into semantically distinct clusters. PSI multiplies three independently calibrated components: geometric cluster quality (S), alignment to labeled categories (Q), and open-vocabulary semantic distinctness via CLIP (D). On a pretrained ResNet-50 evaluated with Tiny-ImageNet images, PSI identifies neurons whose activation sets split into coherent, nameable prototypes, and reveals strong depth trends: later layers exhibit substantially higher PSI than earlier layers. We validate our approach with robustness checks (varying hyperparameters, random seeds, and cross-encoder text heads), breadth analyses (comparing class-only vs. open-vocabulary concepts), and causal patch-swap interventions. In particular, aligned patch replacements increase target-neuron activation significantly more than non-aligned, random, shuffled-position, or ablate-elsewhere controls. PSI thus offers a principled and practical lever for discovering, quantifying, and studying polysemantic units in neural networks.

Related papers

• Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits [0.0]
  This paper develops a structured pruning framework in which complete neurons are removed using multi-armed bandit (MAB) algorithms.<n>The framework supports policies, including Epsilon-Greedy, Softmax, UCB1 and Thompson Sampling, and multiplicative-weight policies, including Hedge-style multiplicative weights and EXP3.
  arXiv  Detail & Related papers  (2026-05-29T19:56:54Z)
• NeuronScope: A Multi-Agent Framework for Explaining Polysemantic Neurons in Language Models [24.550940304055562]
  NeuronScope is a multi-agent framework that reformulates neuron interpretation as an iterative, activation-guided process.<n>We show that NeuronScope uncovers hidden polysemanticity and produces explanations with significantly higher activation correlation compared to single-pass baselines.
  arXiv  Detail & Related papers  (2026-01-07T07:50:47Z)
• Neuronal Group Communication for Efficient Neural representation [85.36421257648294]
  This paper addresses the question of how to build large neural systems that learn efficient, modular, and interpretable representations.<n>We propose Neuronal Group Communication (NGC), a theory-driven framework that reimagines a neural network as a dynamical system of interacting neuronal groups.<n>NGC treats weights as transient interactions between embedding-like neuronal states, with neural computation unfolding through iterative communication among groups of neurons.
  arXiv  Detail & Related papers  (2025-10-19T14:23:35Z)
• Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPs [18.335932098245895]
  We introduce a mechanistic interpretability framework based on coalitional game theory.<n>We extract stable coalitions of neurons whose joint ablation has non-additive effects.<n>We then track their transitions across layers as persistence, splitting, merging, or disappearance.
  arXiv  Detail & Related papers  (2025-09-28T06:55:24Z)
• Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution [16.460751105639623]
  We show that even highly salient neurons consistently exhibit polysemantic behavior.<n>This observation motivates a shift from neuron attribution to range-based interpretation.<n>We introduce NeuronLens, a novel range-based interpretation and manipulation framework.
  arXiv  Detail & Related papers  (2025-02-04T03:33:55Z)
• Interpreting the Second-Order Effects of Neurons in CLIP [73.54377859089801]
  We interpret the function of individual neurons in CLIP by automatically describing them using text.<n>We present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output.<n>Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.
  arXiv  Detail & Related papers  (2024-06-06T17:59:52Z)
• What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes [14.623741848860037]
  Polysemantic neurons -- neurons that activate for a set of unrelated features -- have been seen as a significant obstacle towards interpretability of task-optimized deep networks. We show that polysemanticity can arise incidentally, even when there are ample neurons to represent all features in the data.
  arXiv  Detail & Related papers  (2023-12-05T19:29:54Z)
• Neuroevolutionary algorithms driven by neuron coverage metrics for semi-supervised classification [60.60571130467197]
  In some machine learning applications the availability of labeled instances for supervised classification is limited while unlabeled instances are abundant. We introduce neuroevolutionary approaches that exploit unlabeled instances by using neuron coverage metrics computed on the neural network architecture encoded by each candidate solution.
  arXiv  Detail & Related papers  (2023-03-05T23:38:44Z)
• Permutation Equivariant Neural Functionals [92.0667671999604]
  This work studies the design of neural networks that can process the weights or gradients of other neural networks. We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-02-27T18:52:38Z)
• Problem-Dependent Power of Quantum Neural Networks on Multi-Class Classification [83.20479832949069]
  Quantum neural networks (QNNs) have become an important tool for understanding the physical world, but their advantages and limitations are not fully understood. Here we investigate the problem-dependent power of QCs on multi-class classification tasks. Our work sheds light on the problem-dependent power of QNNs and offers a practical tool for evaluating their potential merit.
  arXiv  Detail & Related papers  (2022-12-29T10:46:40Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks [13.60023740064471]
  SeReNe is a method for learning sparse topologies with a structure. We define the sensitivity of a neuron as the variation of the network output. By including the neuron sensitivity in the cost function as a regularization term, we areable to prune neurons with low sensitivity.
  arXiv  Detail & Related papers  (2021-02-07T10:53:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.