Feature Superposition in Neural Networks: From Theory to Practice

• URL: http://arxiv.org/abs/2609.06862v1
• Date: Sun, 06 Sep 2026 22:24:25 GMT
• Title: Feature Superposition in Neural Networks: From Theory to Practice
• Abstract summary: Superposition refers to neural networks representing more features than they have dimensions.<n>We review the geometry, learning, and computation of superposed representations, explaining how feature statistics and decoder choice affect the conclusions.<n>We assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks.
• Score: 41.616730353916964
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Superposition refers to neural networks representing more features than they have dimensions. It offers a possible explanation for polysemantic neurons and motivates methods for recovering interpretable features from neural activations. Theoretical models typically start with a given set of input features and assumptions about how their values vary across inputs, then study how a network encodes those values in a lower-dimensional hidden representation. Empirical work, by contrast, seeks to identify the features encoded in trained networks and determine their role in computation. In this survey, we review the geometry, learning, and computation of superposed representations, explaining how feature statistics and decoder choice affect the conclusions. To connect these theoretical accounts with evidence from trained networks, we compare practical methods for recovering and analyzing features and examine what their evaluations establish. Since accurate activation reconstruction alone does not establish feature identity or causal use, we discuss the methods' documented failures and applications in light of the evidence available for these different claims. Finally, we assess previously stated open problems and identify remaining theoretical and empirical questions about superposition in trained networks. We hope our work can pave the way for a deeper understanding of superposition and more reliable methods for interpreting neural networks.

Related papers

• Concept-Guided Interpretability via Neural Chunking [64.6429903327095]
  We show that neural networks exhibit patterns in their raw population activity that mirror regularities in the training data.<n>We propose three methods to extract recurring chunks on a neural population level.<n>Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data.
  arXiv  Detail & Related papers  (2025-05-16T13:49:43Z)
• From superposition to sparse codes: interpretable representations in neural networks [3.6738925004882685]
  Recent evidence suggests that neural networks encode features in superposition, meaning that input concepts are linearly overlaid within the network's representations.<n>We present a perspective that explains this phenomenon and provides a foundation for extracting interpretable representations from neural activations.<n>Our arguments have implications for neural coding theories, AI transparency, and the broader goal of making deep learning models more interpretable.
  arXiv  Detail & Related papers  (2025-03-03T18:49:59Z)
• Understanding polysemanticity in neural networks through coding theory [0.8702432681310401]
  We propose a novel practical approach to network interpretability and theoretical insights into polysemanticity and the density of codes. We show how random projections can reveal whether a network exhibits a smooth or non-differentiable code and hence how interpretable the code is. Our approach advances the pursuit of interpretability in neural networks, providing insights into their underlying structure and suggesting new avenues for circuit-level interpretability.
  arXiv  Detail & Related papers  (2024-01-31T16:31:54Z)
• Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes [0.0]
  We propose utilizing the framework of processes, which has been underutilized thus far. We focus solely on activation frequency, leveraging neuroscience techniques used for real neuron spike trains. We derive parameters describing activation patterns in each network, revealing consistent differences across architectures and training sets.
  arXiv  Detail & Related papers  (2023-08-01T22:12:30Z)
• Topological Understanding of Neural Networks, a survey [0.0]
  We look at the internal structure of neural networks which is usually treated as a black box. We review the significance of different activation functions, types of network architectures associated to them, and some empirical data.
  arXiv  Detail & Related papers  (2023-01-23T22:11:37Z)
• Measures of Information Reflect Memorization Patterns [53.71420125627608]
  We show that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization. Importantly, we discover that information organization points to the two forms of memorization, even for neural activations computed on unlabelled in-distribution examples.
  arXiv  Detail & Related papers  (2022-10-17T20:15:24Z)
• Searching for the Essence of Adversarial Perturbations [73.96215665913797]
  We show that adversarial perturbations contain human-recognizable information, which is the key conspirator responsible for a neural network's erroneous prediction. This concept of human-recognizable information allows us to explain key features related to adversarial perturbations.
  arXiv  Detail & Related papers  (2022-05-30T18:04:57Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Analyzing Representations inside Convolutional Neural Networks [8.803054559188048]
  We propose a framework to categorize the concepts a network learns based on the way it clusters a set of input examples. This framework is unsupervised and can work without any labels for input features. We extensively evaluate the proposed method and demonstrate that it produces human-understandable and coherent concepts.
  arXiv  Detail & Related papers  (2020-12-23T07:10:17Z)
• Perceptron Theory Can Predict the Accuracy of Neural Networks [6.136302173351179]
  Multilayer neural networks set the current state of the art for many technical classification problems. But, these networks are still, essentially, black boxes in terms of analyzing them and predicting their performance. Here, we develop a statistical theory for the one-layer perceptron and show that it can predict performances of a surprisingly large variety of neural networks.
  arXiv  Detail & Related papers  (2020-12-14T19:02:26Z)
• Neuron-based explanations of neural networks sacrifice completeness and interpretability [67.53271920386851]
  We show that for AlexNet pretrained on ImageNet, neuron-based explanation methods sacrifice both completeness and interpretability.<n>We show the most important principal components provide more complete and interpretable explanations than the most important neurons.<n>Our findings suggest that explanation methods for networks like AlexNet should avoid using neurons as a basis for embeddings.
  arXiv  Detail & Related papers  (2020-11-05T21:26:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.