Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks

• URL: http://arxiv.org/abs/2602.15997v3
• Date: Tue, 24 Feb 2026 00:03:59 GMT
• Title: Anatomy of Capability Emergence: Scale-Invariant Representation Collapse and Top-Down Reorganization in Neural Networks
• Abstract summary: We track five geometric measures across five model scales.<n>We find that training begins with a universal representation collapse to task-specific floors.<n>On Pythia, global geometric patterns replicate but per-task precursor signals do not.
• Score: 1.5567685129899713
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Capability emergence during neural network training remains mechanistically opaque. We track five geometric measures across five model scales (405K--85M parameters), 120 task$\times$level$\times$ model combinations (119 achieving accuracy-based emergence) across eight algorithmic tasks, and three Pythia language models (160M--2.8B). We find: (1) training begins with a universal representation collapse to task-specific floors that are scale-invariant across a 210$\times$ parameter range (e.g., modular arithmetic collapses to RANKME $\,\approx\,$2.0 regardless of model size); (2) collapse propagates top-down through layers (28/32 task$ \times $model consistency), contradicting bottom-up feature-building intuition; (3) a geometric hierarchy in which representation geometry leads emergence (100% precursor rate for hard tasks across all model sizes), while the local learning coefficient is synchronous (0/24 precursor) and Hessian measures lag. We also delineate prediction limits: geometric measures encode coarse task difficulty but not fine-grained timing (within-class concordance ranges from 52% for easy tasks to 69% for hard tasks; when task ordering reverses across scales, prediction fails at 26%). On Pythia, global geometric patterns replicate but per-task precursor signals do not, as the precursor relationship requires task--training alignment that naturalistic pre-training does not provide. Our contribution is the geometric anatomy of emergence and its boundary conditions, not a prediction tool.

Related papers

• Geometric Self-Supervised Pre-training for Neural Combinatorial Optimization [40.74618635552097]
  We propose a geometric self-supervised pre-training framework for routing graphs.<n>The framework learns robust structural representations prior to the policy optimization phase.<n>The proposed model exhibits remarkable computational efficiency, delivering speedups of up to two orders of magnitude over the exact solver Concorde.
  arXiv  Detail & Related papers  (2026-07-31T20:23:21Z)
• AgentOmnia: Scaling Agentic Models for Full-Scenario Applications [36.656762500581216]
  Large language model agents have advanced rapidly, yet progress remains fragmented across domains, capabilities, task difficulty, and interaction settings.<n>We frame this as full-scenario agentic scaling and present Agent Omnia, a framework coordinating task-space definition, data synthesis, post-training, evaluation, and improvement across To-Consumer (ToC), To-Business (ToB), and To-Employee (ToE) applications.
  arXiv  Detail & Related papers  (2026-07-25T09:58:24Z)
• Euclean: Automated Geometry Problem Formalization with Unified Verification in Lean [7.28030691695892]
  Euclean is a framework for automatically formalizing geometry in native Mathlib.<n>Human evaluation shows 48.89% TOP1 and 73.33% TOP5 accuracy.
  arXiv  Detail & Related papers  (2026-06-17T15:27:17Z)
• Mesh Graph Neural Network Framework for Accelerating Finite Element Simulation for Arbitrary Geometries [0.126928839393823]
  This work presents a mesh graph network (MGN) for predicting von Mises stress fields in 2D structural components with arbitrary hole geometries.<n>Unlike traditional machine learning approaches that use absolute node coordinates as features, the proposed model builds on existing MGN frameworks.<n>The model achieves $R2 geq 0.97$ on an unseen geometry and unseen load, compared to $R2 approx 0.01$--$0.86$ for conventional models.
  arXiv  Detail & Related papers  (2026-06-06T18:17:08Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• MeshWeaver: Sparse-Voxel-Guided Surface Weaving for Autoregressive Mesh Generation [53.06620176910962]
  MeshWeaver is an autoregressive framework that treats mesh generation as a surface weaving process.<n>MeshWeaver achieves a state-of-the-art compression ratio of 18%, can generate meshes with up to 16K faces, and significantly improves geometric fidelity over prior approaches.
  arXiv  Detail & Related papers  (2026-06-03T10:15:43Z)
• What do Language Models Learn and When? The Implicit Curriculum Hypothesis [50.89698323917259]
  We propose the Implicit Curriculum Hypothesis: pretraining follows a compositional and predictable curriculum across models.<n>We track emergence points across four model families spanning sizes from 410M-13B parameters.<n>We find that emergence orderings of when models reach fixed accuracy thresholds are strikingly consistent.
  arXiv  Detail & Related papers  (2026-04-09T17:50:12Z)
• Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles [74.32932832937618]
  We introduce $textbfRigidSSL$ ($textitRigidity-Aware Self-Supervised Learning$), a geometric pretraining framework.<n>Phase I (RigidSSL-Perturb) learns geometric priors from 432K structures from the AlphaFold Protein Structure Database with simulated perturbations.<n>Phase II (RigidSSL-MD) refines these representations on 1.3K molecular dynamics trajectories to capture physically realistic transitions.
  arXiv  Detail & Related papers  (2026-03-02T21:32:30Z)
• Model Agreement via Anchoring [31.113945168635656]
  We prove disagreement bounds for four commonly used machine learning algorithms.<n>For clarity, we work out our initial bounds in the setting of one-dimensional regression with squared error loss.<n>All of our results generalize to multi-dimensional regression with any strongly convex loss.
  arXiv  Detail & Related papers  (2026-02-26T18:59:32Z)
• MagicAgent: Towards Generalized Agent Planning [73.21129030631421]
  We present textbfMagicAgent, a series of foundation models specifically designed for generalized agent planning.<n>We introduce a lightweight and scalable synthetic data framework that generates high-quality trajectories across diverse planning tasks.<n>We show that MagicAgent-32B and MagicAgent-30B-A3B achieve superior performance across diverse open-source benchmarks.
  arXiv  Detail & Related papers  (2026-02-22T01:39:16Z)
• Low-Dimensional Execution Manifolds in Transformer Learning Dynamics: Evidence from Modular Arithmetic Tasks [0.0]
  We investigate the structure of learning dynamics in transformer models through carefully controlled arithmetic tasks.<n>Our results suggest a unifying geometric framework for understanding transformer learning.
  arXiv  Detail & Related papers  (2026-02-11T03:57:46Z)
• The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models [0.0]
  We analyze 25,000+ chain-of-thought trajectories across four domains (Law, Science, Code, Math)<n>We discover that neural scaling laws trigger domain-specific phase transitions rather than uniform capability gains.
  arXiv  Detail & Related papers  (2026-01-19T19:53:37Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model [32.831576387973875]
  We propose a two-stage deterministic framework for stable, accurate and fine-grained geometric dense prediction.<n>Specifically, in the first stage, the core predictor employs a single-step deterministic formulation with a clean-data objective.<n>In the second stage, the detail sharpener performs a constrained multi-step rectified-flow refinement within the manifold defined by the core predictor.
  arXiv  Detail & Related papers  (2025-11-30T18:57:25Z)
• Agentic World Modeling for 6G: Near-Real-Time Generative State-Space Reasoning [70.56067503630486]
  We argue that sixth-generation (6G) intelligence is not fluent token prediction but calibrated the capacity to imagine and choose.<n>We show that WM-MS3M cuts mean absolute error (MAE) by 1.69% versus MS3M with 32% fewer parameters and similar latency, and achieves 35-80% lower root mean squared error (RMSE) than attention/hybrid baselines with 2.3-4.1x faster inference.
  arXiv  Detail & Related papers  (2025-11-04T17:22:22Z)
• Deep Hierarchical Learning with Nested Subspace Networks [53.71337604556311]
  We propose Nested Subspace Networks (NSNs) for large neural networks.<n>NSNs enable a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets.<n>We show that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier.
  arXiv  Detail & Related papers  (2025-09-22T15:13:14Z)
• Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training [68.94373533768501]
  We model knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training.<n>We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy.
  arXiv  Detail & Related papers  (2025-02-06T13:23:53Z)
• Disentangled Representation Learning with the Gromov-Monge Gap [65.73194652234848]
  Learning disentangled representations from unlabelled data is a fundamental challenge in machine learning.<n>We introduce a novel approach to disentangled representation learning based on quadratic optimal transport.<n>We demonstrate the effectiveness of our approach for quantifying disentanglement across four standard benchmarks.
  arXiv  Detail & Related papers  (2024-07-10T16:51:32Z)
• Efficiently Modeling Long Sequences with Structured State Spaces [15.456254157293836]
  We propose a new sequence model based on a new parameterization for the fundamental state space model. S4 achieves strong empirical results across a diverse range of established benchmarks, including (i) 91% accuracy on sequential CIFAR-10 with no data augmentation or auxiliary losses, on par with a larger 2-D ResNet.
  arXiv  Detail & Related papers  (2021-10-31T03:32:18Z)
• NeuroMorph: Unsupervised Shape Interpolation and Correspondence in One Go [109.88509362837475]
  We present NeuroMorph, a new neural network architecture that takes as input two 3D shapes. NeuroMorph produces smooth and point-to-point correspondences between them. It works well for a large variety of input shapes, including non-isometric pairs from different object categories.
  arXiv  Detail & Related papers  (2021-06-17T12:25:44Z)
• Meta-Generating Deep Attentive Metric for Few-shot Classification [53.07108067253006]
  We present a novel deep metric meta-generation method to generate a specific metric for a new few-shot learning task. In this study, we structure the metric using a three-layer deep attentive network that is flexible enough to produce a discriminative metric for each task. We gain surprisingly obvious performance improvement over state-of-the-art competitors, especially in the challenging cases.
  arXiv  Detail & Related papers  (2020-12-03T02:07:43Z)
• Primal-Dual Mesh Convolutional Neural Networks [62.165239866312334]
  We propose a primal-dual framework drawn from the graph-neural-network literature to triangle meshes. Our method takes features for both edges and faces of a 3D mesh as input and dynamically aggregates them. We provide theoretical insights of our approach using tools from the mesh-simplification literature.
  arXiv  Detail & Related papers  (2020-10-23T14:49:02Z)
• Geometric Prediction: Moving Beyond Scalars [4.702729080310267]
  We show that equivariant networks have the capability to predict real-world geometric tensors without the need for such approximations. We propose a novel formulation of an important task, biomolecular structure refinement, as a geometric prediction problem. In both settings, we find that our equivariant network is able to generalize to unseen systems, despite having been trained on small sets of examples.
  arXiv  Detail & Related papers  (2020-06-25T04:12:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.