Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic

• URL: http://arxiv.org/abs/2606.08985v1
• Date: Mon, 08 Jun 2026 03:30:56 GMT
• Title: Beyond Neural Collapse: Task-Intrinsic Geometry Governs Neural Representations in Modular Arithmetic
• Abstract summary: We find that a simplex ETF gains only an $O(1)$ advantage in cross-entropy, whereas the cyclic rank-2 solution enjoys a $(K)$ advantage under Schatten or weight-decay surrogates.<n>Our results show that grokking on modular arithmetic is governed not by maximal separation alone but by a task-structured trade-off between separation, symmetry, and complexity.
• Score: 18.72807692009739
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While neural collapse (NC) predicts that a $K$-class-balanced classifier should organize terminal representations as a $(K-1)$-dimensional simplex equiangular tight frame (ETF), modular addition consistently enters a different regime: networks compress to a two-dimensional cyclic geometry in which both classifier weights and token embeddings lie on circles. We refine the explanation of this phenomenon in three directions. First, we formalize a layerwise non-uniform training mechanism: downstream classifier weights are driven by dense cross-entropy gradients into a rank-2 equiangular configuration before upstream embeddings fully reorganize, and once this classifier plane forms, backpropagated feature gradients constrain embedding motion to the same plane while weight decay suppresses orthogonal components. Second, after this subspace locking, the induced in-plane dynamics admit an entropy-regularized transport interpretation on $S^1$; combined with modular-addition labels, this reduces embedding formation to phase alignment, whose minimizers are single-frequency characters of $\mathbb{Z}/P\mathbb{Z}$ and hence equal-angle points on a circle. Third, we quantify why this solution prevails over NC: a simplex ETF gains only an $O(1)$ advantage in cross-entropy, whereas the cyclic rank-2 solution enjoys a $Θ(K)$ advantage under Schatten or weight-decay surrogates, yielding a critical threshold $λ_{\mathrm{crit}} = Θ(1/K)$. Our results explain both why classifier weights move first and why embeddings subsequently align with them, showing that grokking on modular arithmetic is governed not by maximal separation alone but by a task-structured trade-off between separation, symmetry, and complexity.

Related papers

• $\mathbb{SL}(n)$ Representation Learning: An Intrinsic Mixed-Curvature Space with Higher Curvature Capacities and Deeper Order-Aware Composition [47.80325814873361]
  Mixed-curvature representation learning seeks to capture rich geometric structures that cannot be adequately modeled by a single curvature regime.<n>We introduce the $mathbbSL(n)$ space, a representation geometry defined by the simple $det(A)variant=1$ constraint and a left in Schatten-$p$ structure.<n>Despite this minimal construction, $mathbbSL(n)$ exhibits pointwise negative, zero, and flag curvature around a common flagpole.
  arXiv  Detail & Related papers  (2026-09-14T05:52:14Z)
• Dynamical splitting and a nodal Bose liquid in 2d chiral XYZ model [0.0]
  We study a class of Hamiltonians with a structure that we call "dynamical splitting"<n>This structure yields an algebraic duality to effective degrees of freedom on which the two parts of the Hamiltonian act disjointly.<n>We exploit it in the "chiral XYZ model", a geometrically frustrated spin-$1/2$ model on the triangular lattice.
  arXiv  Detail & Related papers  (2026-08-25T23:28:30Z)
• Nonlocality-induced critical-length hierarchy from non-Hermitian competition [48.9823527611878]
  In short-range systems formed by two coupled chains, competition leads to the critical-length law $N_csimln D$.<n>Here we show that long-range hoppings fundamentally reorganizes this critical behavior, producing a hierarchy of distinct scaling laws.<n>Our results offer a platform-independent framework testable in programmable topoelectrical circuits, photonic lattices and digital quantum simulators.
  arXiv  Detail & Related papers  (2026-08-03T18:00:18Z)
• $\mathbfλ$-VAE: Variance Equalization for Posterior Collapse [0.0]
  Variational Autoencoders (VAEs) frequently suffer from posterior collapse, a failure mode in which the approximate posterior converges to the prior.<n>We identify and formalize two logically independent but coupled causes.
  arXiv  Detail & Related papers  (2026-07-06T18:07:46Z)
• Imbuing Large Language Models with Bidirectional Logic for Robust Chain Repair [44.80087038178069]
  We introduce Teleological Reasoning Infilling (TRI), a training framework that endows decoder-only transformers with a native emphgoal-conditioned bridging capability.<n>At inference, TRI operates as a surgical repair module within a dual-system loop.<n>Experiments on three benchmarks demonstrate that TRI achieves state-of-the-art performance across all tasks, while reducing per-problem token expenditure by 31.2%.
  arXiv  Detail & Related papers  (2026-06-03T15:58:48Z)
• Group-Algebraic Tensors: Provably-optimal Equivariant Learning and Physical Symmetry Discovery [17.340037184664208]
  We introduce the $star_G$ tensor algebra, in which any finite group $G$ defines the algebraic multiplication rule.<n>The framework provides capabilities that redesign neural networks (ENNs) structurally cannot: a closed-form per-irreducible equi- decomposition of every prediction, and data-driven discovery of the symmetry group that best fits a dataset.
  arXiv  Detail & Related papers  (2026-05-19T19:47:40Z)
• Information-Geometric Decomposition of Generalization Error in Unsupervised Learning [0.0]
  We decompose the Kullback--Leibler generalization error (GE) of unsupervised learning into three non-negative components: model error, data bias, and variance.<n>The decomposition is exact for any e-flat model class.
  arXiv  Detail & Related papers  (2026-04-14T06:23:18Z)
• Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures [45.88028371034407]
  Modern automatic differentiation frameworks (JAX, PyTorch) return the Hessian of the loss function as a monolithic tensor, without exposing the internal structure of inter-layer interactions.<n>The theoretical analysis explains exponential decay of resonance in empirical networks and its connections under skip skip.
  arXiv  Detail & Related papers  (2026-04-13T15:48:41Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Group Representational Position Encoding [66.33026480082025]
  We present GRAPE, a unified framework for positional encoding based on group actions.<n>Two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in $mathrmSO(d)$ and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group $mathrmGL$.
  arXiv  Detail & Related papers  (2025-12-08T18:39:13Z)
• Measurement-induced Lévy flights of quantum information [35.31418199674737]
  We explore a model of free fermions in one dimension subject to frustrated local measurements across adjacent sites.<n>For maximal misalignment, superdiffusive behavior emerges from the vanishing of the measurement-induced quasiparticle decay rate.<n>Our findings show how intricate fractal-scaling entanglement can be produced for local Hamiltonians.
  arXiv  Detail & Related papers  (2025-01-22T14:29:13Z)
• Entanglement renormalization of fractonic anisotropic $\mathbb{Z}_N$ Laplacian models [4.68169911641046]
  Gapped fracton phases constitute a new class of quantum states of matter which connects to topological orders but does not fit easily into existing paradigms.<n>We investigate the anisotropic $mathbbZ_N$ Laplacian model which can describe a family of fracton phases defined on arbitrary graphs.
  arXiv  Detail & Related papers  (2024-09-26T18:36:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.