Neuron Populations Exhibit Divergent Selectivity with Scale

• URL: http://arxiv.org/abs/2606.03990v1
• Date: Tue, 02 Jun 2026 17:59:52 GMT
• Title: Neuron Populations Exhibit Divergent Selectivity with Scale
• Abstract summary: We study Rosetta Neurons, a class of neurons whose activation patterns are similar across independently trained models.<n>In separate analyses of language models up to 30B parameters and vision models up to 5B parameters, we observe that the population of Rosetta Neurons follows a sublinear power law in model size.<n>We further observe a Neuron Polarization Effect: Rosetta Neurons become more selective and increasingly monosemantic with scale.
• Score: 46.27365621560269
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate whether neuron populations within neural networks evolve predictably with scale, extending scaling laws beyond macroscopic observables such as loss. To probe this question, we study Rosetta Neurons, a previously characterized class of neurons whose activation patterns are similar across independently trained models (Dravid et al., 2023). In separate analyses of language models up to 30B parameters and vision models up to 5B parameters, we observe that the population of Rosetta Neurons follows a sublinear power law in model size, growing in absolute number but occupying a shrinking fraction of the total neuron count. We further observe a Neuron Polarization Effect: Rosetta Neurons become more selective and increasingly monosemantic with scale, separating from a growing non-Rosetta population that remains less selective. An analytical model balancing feature utility against limited neuron capacity explains the sublinear power-law scaling and this polarization effect. Finally, we find that Rosetta Neurons become more domain-specialized with scale and illustrate their selectivity through a targeted data-filtering case study for continued pretraining. Our results point to a scaling law for interpretable, shared neuron-level structure, linking model size to systematic changes in neuron universality, selectivity, and specialization.

Related papers

• Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts [57.98275914877019]
  We show that across a wide range of open-weight Transformers, a subset of neurons remains consistently highly activated during inference across tasks of multiple capability dimensions.<n>By probing along the cross-task activation strength, an extremely sparse subset is isolated, whose removal causes a collapse in model behavior.<n>Our analysis reveals that keystone neurons are a stable and intrinsic neuron subset of the model that is largely established during pretraining.
  arXiv  Detail & Related papers  (2026-05-24T03:31:07Z)
• Updating the standard neuron model in artificial neural networks [2.2046235060588018]
  From their inception in the 1950s, artificial neural networks (ANNs) started using the so-called point neuron model.<n>Over the years the neuroscience literature has shown that the point neuron model is too simplistic to properly represent many fundamental neural processes.
  arXiv  Detail & Related papers  (2026-05-19T05:21:45Z)
• In Search of Grandmother Cells: Tracing Interpretable Neurons in Tabular Representations [1.503974529275767]
  We show that some neurons show moderate, statistically significant saliency and selectivity for high-level concepts.<n>These findings suggest that interpretable neurons can emerge naturally and that they can, in some cases, be identified without resorting to more complex interpretability techniques.
  arXiv  Detail & Related papers  (2026-01-07T07:13:01Z)
• Neuronal Group Communication for Efficient Neural representation [85.36421257648294]
  This paper addresses the question of how to build large neural systems that learn efficient, modular, and interpretable representations.<n>We propose Neuronal Group Communication (NGC), a theory-driven framework that reimagines a neural network as a dynamical system of interacting neuronal groups.<n>NGC treats weights as transient interactions between embedding-like neuronal states, with neural computation unfolding through iterative communication among groups of neurons.
  arXiv  Detail & Related papers  (2025-10-19T14:23:35Z)
• Neuromorphic Online Clustering and Its Application to Spike Sorting [0.783218941317936]
  We develop neuromorphic dendrites as basic neural building blocks capable of dynamic online clustering.<n>Features and capabilities of neuromorphic dendrites are demonstrated via a benchmark drawn from experimental neuroscience: spike sorting.<n>The accuracy of the proposed dendrite is compared with the more compute-intensive, offline k-means clustering approach.
  arXiv  Detail & Related papers  (2025-06-14T15:53:55Z)
• NOBLE -- Neural Operator with Biologically-informed Latent Embeddings to Capture Experimental Variability in Biological Neuron Models [63.592664795493725]
  NOBLE is a neural operator framework that learns a mapping from a continuous frequency-modulated embedding of interpretable neuron features to the somatic voltage response induced by current injection.<n>It predicts distributions of neural dynamics accounting for the intrinsic experimental variability.<n>NOBLE is the first scaled-up deep learning framework that validates its generalization with real experimental data.
  arXiv  Detail & Related papers  (2025-06-05T01:01:18Z)
• Confidence Regulation Neurons in Language Models [91.90337752432075]
  This study investigates the mechanisms by which large language models represent and regulate uncertainty in next-token predictions. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. token frequency neurons, which we describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution.
  arXiv  Detail & Related papers  (2024-06-24T01:31:03Z)
• Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles [3.625425081454343]
  Systems neuroscience relies on two complementary views of neural data, characterized by single neuron tuning curves and analysis of population activity. These two perspectives combine elegantly in neural latent variable models that constrain the relationship between latent variables and neural activity. We propose feature sharing across neural tuning curves, which significantly improves performance and leads to better-behaved optimization.
  arXiv  Detail & Related papers  (2022-10-06T18:37:49Z)
• Improving Spiking Neural Network Accuracy Using Time-based Neurons [0.24366811507669117]
  Research on neuromorphic computing systems based on low-power spiking neural networks using analog neurons is in the spotlight. As technology scales down, analog neurons are difficult to scale, and they suffer from reduced voltage headroom/dynamic range and circuit nonlinearities. This paper first models the nonlinear behavior of existing current-mirror-based voltage-domain neurons designed in a 28nm process, and show SNN inference accuracy can be severely degraded by the effect of neuron's nonlinearity. We propose a novel neuron, which processes incoming spikes in the time domain and greatly improves the linearity, thereby improving the inference accuracy compared to the
  arXiv  Detail & Related papers  (2022-01-05T00:24:45Z)
• The Neural Coding Framework for Learning Generative Models [91.0357317238509]
  We propose a novel neural generative model inspired by the theory of predictive processing in the brain. In a similar way, artificial neurons in our generative model predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality.
  arXiv  Detail & Related papers  (2020-12-07T01:20:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.