Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks

• URL: http://arxiv.org/abs/2606.10324v1
• Date: Tue, 09 Jun 2026 02:19:26 GMT
• Title: Rank Collapse, Fixed Points, and the Renormalization Group Structure of MLP Residual Networks
• Abstract summary: We study the simplest architecture for which the analogy is tractable.<n>The effective rank of the residual stream decreases monotonically with depth after training.<n>The network preserves exactly the degrees of freedom relevant to the prediction task.
• Score: 16.36212521143563
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points. No existing work has defined a measurable RG order parameter, tested it under controlled variation of the input distribution, or made quantitative predictions that are empirically verified. We study the simplest architecture for which the analogy is tractable: a pure MLP residual stack trained on masked token prediction over synthetic Markov chain sequences with known spectral properties. We report three findings. (i) The effective rank of the residual stream decreases monotonically with depth after training, consistent with progressive integration of irrelevant degrees of freedom. (ii) This rank collapse is selective: it occurs for chains with short correlation length approximately 1 but is absent for chains with long correlation length approximately 7, measured at the position level to control for mean-pooling artifacts. The network preserves exactly the degrees of freedom relevant to the prediction task, the content of the RG relevance criterion. (iii) Inter-layer kernel drift is concentrated at one or two specific transitions, with the remainder of the network near a fixed point, consistent with a discrete fixed-point plateau. Together these findings constitute the first quantitative, position-level evidence that MLP residual networks implement a selective coarse-graining procedure governed by the spectral structure of the input distribution.

Related papers

• Scalable Policy Optimization for Networked Multi-Agent Reinforcement Learning with Continuous State-Action Spaces [16.43184149906767]
  We develop an algorithm for cooperative reinforcement learning in networked decision processes with continuous state and action spaces.<n>Each agent maintains a local actor over a graph neighborhood, and a localized temporal-difference critic evaluates a truncated actionvalue function.
  arXiv  Detail & Related papers  (2026-07-20T22:41:43Z)
• Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention [16.36212521143563]
  We treat the Transformer's attention mechanism as a perturbation of the trained residual-stack fixed point.<n>Testable predictions are made for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum.
  arXiv  Detail & Related papers  (2026-07-16T20:42:02Z)
• Sequential Spatiotemporal Magnetic-Field Reconstruction via Quantum Hamiltonian Learning with NV-Center Spin-1 Hamiltonians [0.0]
  We propose a quantum-Hamiltonian-learning-based sequential reconstruction framework for dynamic two-dimensional magnetic-field maps.<n>Local measurements are generated through nitrogen-vacancy spin dynamics governed by local magnetic-field values and a shared dipolar coupling parameter.<n>Results demonstrate feasibility for tested structured field class and identify coupling estimation as the main identifiability bottleneck.
  arXiv  Detail & Related papers  (2026-05-22T10:13:04Z)
• DeRegiME: Deep Regime Mixtures for Probabilistic Forecasting under Distribution Shift [10.952188121974613]
  DeRegiME is a direct multi-horizon probabilistic forecaster.<n>It separates latent uncertainty regimes from the underlying signal.<n>It assigns each forecast location to learned recurring regimes.
  arXiv  Detail & Related papers  (2026-05-19T01:04:21Z)
• Relaxation-Informed Training of Neural Network Surrogate Models [3.9722979176564763]
  ReLU neural networks trained surrogate models can be embedded exactly in mixed-integer linear programs (MILPs)<n>The tractability of the resulting MILP depends on structural properties of the network.<n>We show that the proposed regularizers can reduce MILP times by up to four orders of magnitude to an unregularized baseline.
  arXiv  Detail & Related papers  (2026-04-24T17:46:55Z)
• Verifying Closed-Loop Contractivity of Learning-Based Controllers via Partitioning [52.23804865017831]
  We address the problem of verifying closed-loop contraction in nonlinear control systems whose controller and contraction metric are both parameterized by neural networks.<n>We derive a tractable and scalable sufficient condition for closed-loop contractivity that reduces to checking that the dominant eigenvalue of a symmetric Metzler matrix is nonpositive.
  arXiv  Detail & Related papers  (2025-12-01T23:06:56Z)
• Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learning [56.87989363424]
  We show that a low-rank structure naturally emerges in the shifted successor measure.<n>We quantify the amount of shift needed for effective low-rank approximation and estimation.
  arXiv  Detail & Related papers  (2025-09-05T15:48:20Z)
• Unsupervised Deep Probabilistic Approach for Partial Point Cloud Registration [74.53755415380171]
  Deep point cloud registration methods face challenges to partial overlaps and rely on labeled data. We propose UDPReg, an unsupervised deep probabilistic registration framework for point clouds with partial overlaps. Our UDPReg achieves competitive performance on the 3DMatch/3DLoMatch and ModelNet/ModelLoNet benchmarks.
  arXiv  Detail & Related papers  (2023-03-23T14:18:06Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• The Sample Complexity of One-Hidden-Layer Neural Networks [57.6421258363243]
  We study a class of scalar-valued one-hidden-layer networks, and inputs bounded in Euclidean norm. We prove that controlling the spectral norm of the hidden layer weight matrix is insufficient to get uniform convergence guarantees. We analyze two important settings where a mere spectral norm control turns out to be sufficient.
  arXiv  Detail & Related papers  (2022-02-13T07:12:02Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.