Revenge of Monosemanticity: Specialized Neurons Improve Data Efficiency in MLPs

• URL: http://arxiv.org/abs/2608.24007v1
• Date: Tue, 25 Aug 2026 02:54:58 GMT
• Title: Revenge of Monosemanticity: Specialized Neurons Improve Data Efficiency in MLPs
• Abstract summary: We show that multilayer perceptrons (MLPs) naturally develop monosemantic specialized neurons.<n>Rather than learning a single global low-dimensional representation, relevants learn a collection of local low-dimensional representations that can collectively span a high-dimensional space.
• Score: 17.43268776924599
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional predictive geometry. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) naturally develop monosemantic specialized neurons: individual neurons become strongly aligned with a specific predictive feature relevant to a particular region of the input space. Rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations that can collectively span a high-dimensional space. This specialization provably gives MLPs a data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.

Related papers

• Emergent Generalization by Representation Learning in Artificial Neural Networks [0.0]
  We show that an explicit information bottleneck forcing a recurrent neural network to learn a low-dimensional representation is necessary for generalisation.<n>Using information-theoretic measures of causal emergence, we characterise the dynamics of this representation across the memorisation-to-generalisation transition.<n>This trajectory scales with task complexity, and the magnitude of emergent structure reliably predicts generalisation performance.
  arXiv  Detail & Related papers  (2026-07-11T18:25:23Z)
• Hedonic Neurons: A Mechanistic Mapping of Latent Coalitions in Transformer MLPs [18.335932098245895]
  We introduce a mechanistic interpretability framework based on coalitional game theory.<n>We extract stable coalitions of neurons whose joint ablation has non-additive effects.<n>We then track their transitions across layers as persistence, splitting, merging, or disappearance.
  arXiv  Detail & Related papers  (2025-09-28T06:55:24Z)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization [66.03821840425539]
  In this paper, we investigate the training dynamics of $L$-layer neural networks using the tensor gradient program (SGD) framework.<n>We show that SGD enables these networks to learn linearly independent features that substantially deviate from their initial values.<n>This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum.
  arXiv  Detail & Related papers  (2025-03-12T17:33:13Z)
• You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Data [54.56492110703343]
  Data heterogeneity, stemming from local non-IID data and global long-tailed distributions, is a major challenge in federated learning (FL)<n>We propose FedYoYo to improve representation learning by distilling knowledge between weakly and strongly augmented local samples.<n>We show FedYoYo achieves state-of-the-art results, even surpassing centralized logit adjustment methods by 5.4% under global long-tailed settings.
  arXiv  Detail & Related papers  (2025-03-10T04:57:20Z)
• From MLP to NeoMLP: Leveraging Self-Attention for Neural Fields [26.659511924272962]
  We develop a new type of connectionism based on hidden and scalable nodes, called NeoMLP.<n>We demonstrate the effectiveness of our method by fitting high-resolution signals, including multi-modal audio-visual data.
  arXiv  Detail & Related papers  (2024-12-11T19:01:38Z)
• Generalization from Starvation: Hints of Universality in LLM Knowledge Graph Learning [8.025148264640967]
  We investigate how neural networks represent knowledge during graph learning. We find hints of universality, where equivalent representations are learned across a range of model sizes. We show that these attractor representations optimize generalization to unseen examples.
  arXiv  Detail & Related papers  (2024-10-10T16:23:42Z)
• Spatiotemporal Graph Learning with Direct Volumetric Information Passing and Feature Enhancement [62.91536661584656]
  We propose a dual-module framework, Cell-embedded and Feature-enhanced Graph Neural Network (aka, CeFeGNN) for learning.<n>We embed learnable cell attributions to the common node-edge message passing process, which better captures the spatial dependency of regional features.<n>Experiments on various PDE systems and one real-world dataset demonstrate that CeFeGNN achieves superior performance compared with other baselines.
  arXiv  Detail & Related papers  (2024-09-26T16:22:08Z)
• Self-Supervised Learning of Representations for Space Generates Multi-Modular Grid Cells [16.208253624969142]
  mammalian lineage has developed striking spatial representations. One important spatial representation is the Nobel-prize winning grid cells. Nobel-prize winning grid cells represent self-location, a local and aperiodic quantity.
  arXiv  Detail & Related papers  (2023-11-04T03:59:37Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks [80.55378250013496]
  We study how neural networks trained by gradient descent extrapolate what they learn outside the support of the training distribution. Graph Neural Networks (GNNs) have shown some success in more complex tasks.
  arXiv  Detail & Related papers  (2020-09-24T17:48:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.