As Language Models Scale, Low-order Linear Depth Dynamics Emerge

• URL: http://arxiv.org/abs/2603.12541v1
• Date: Fri, 13 Mar 2026 00:51:53 GMT
• Title: As Language Models Scale, Low-order Linear Depth Dynamics Emerge
• Abstract summary: Large language models are often viewed as high-dimensional nonlinear systems and treated as black boxes.<n>We show that transformer depth dynamics admit accurate low-order linear surrogates within context.
• Score: 15.919212303210351
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are often viewed as high-dimensional nonlinear systems and treated as black boxes. Here, we show that transformer depth dynamics admit accurate low-order linear surrogates within context. Across tasks including toxicity, irony, hate speech and sentiment, a 32-dimensional linear surrogate reproduces the layerwise sensitivity profile of GPT-2-large with near-perfect agreement, capturing how the final output shifts under additive injections at each layer. We then uncover a surprising scaling principle: for a fixed-order linear surrogate, agreement with the full model improves monotonically with model size across the GPT-2 family. This linear surrogate also enables principled multi-layer interventions that require less energy than standard heuristic schedules when applied to the full model. Together, our results reveal that as language models scale, low-order linear depth dynamics emerge within contexts, offering a systems-theoretic foundation for analyzing and controlling them.

Related papers

• Beyond Linear Dynamics: Neural Bilinear Dynamical Models for Time Series Forecasting [10.24195404755377]
  Time series in real-world applications are often generated by nonlinear dynamical systems.<n>We propose the Neural Bilinear Dynamical Model (NBDM), which models nonlinear system dynamics through a bilinear latent dynamical formulation.<n>Experiments on five real-world datasets demonstrate that NBDM consistently outperforms competitive baselines in both given-control and missing-control settings.
  arXiv  Detail & Related papers  (2026-08-05T05:54:17Z)
• Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training [56.323119575322146]
  Extra-Merge consistently outperforms standard merging baselines in experiments across GPT-2 and LLaMA families.<n>It yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon citepjordan2024muon.
  arXiv  Detail & Related papers  (2026-05-26T02:48:34Z)
• Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models [8.570762845568995]
  Diffusion models have achieved remarkable success in synthesizing complex static and temporal visuals.<n>Standard architectures-Free Guidance (CFG) relies on a globally uniform scalar.<n>This amplification traps models in a well-documented "detailartifact dilemma"<n>We propose Spatial Adaptive Multi Guidance (SAMG), a training-free and virtually zero-cost sampling algorithm.
  arXiv  Detail & Related papers  (2026-04-29T10:08:08Z)
• Linearization Explains Fine-Tuning in Large Language Models [13.157568306905885]
  We show that fine-tuning dynamics become equivalent to learning with the positive-definite neural tangent kernel (NTK)<n>When linearization is a good model, our findings reveal a strong correlation between the eigenvalue spectrum of the NTK and the performance of model adaptation.
  arXiv  Detail & Related papers  (2026-02-09T03:27:58Z)
• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction [87.33016661440202]
  Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality.<n>We propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions.<n> Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm.
  arXiv  Detail & Related papers  (2025-10-07T06:31:02Z)
• Random Matrix Theory for Deep Learning: Beyond Eigenvalues of Linear Models [51.85815025140659]
  Modern Machine Learning (ML) and Deep Neural Networks (DNNs) often operate on high-dimensional data.<n>In particular, the proportional regime where the data dimension, sample size, and number of model parameters are all large gives rise to novel and sometimes counterintuitive behaviors.<n>This paper extends traditional Random Matrix Theory (RMT) beyond eigenvalue-based analysis of linear models to address the challenges posed by nonlinear ML models.
  arXiv  Detail & Related papers  (2025-06-16T06:54:08Z)
• Equivalent Linear Mappings of Large Language Models [0.5076419064097734]
  We exploit a property of transformers where every operation can be expressed as $A(x) cdot x$, where $A(x)$ represents an input-dependent linear transform and $x$ preserves the linear pathway.<n>To expose this linear structure, we strategically detach components of the gradient with respect to an input sequence, freezing the $A(x)$ terms at their values computed during inference.<n>This detached Jacobian reconstructs the output with one linear operator per input token, which is shown for Qwen 3, Gemma 3 and Llama 3, up to Qwen 3 14B
  arXiv  Detail & Related papers  (2025-05-30T07:08:33Z)
• Weight-based Decomposition: A Case for Bilinear MLPs [0.0]
  Gated Linear Units (GLUs) have become a common building block in modern foundation models. Bilinear layers drop the non-linearity in the "gate" but still have comparable performance to other GLUs. We develop a method to decompose the bilinear tensor into a set of interacting eigenvectors.
  arXiv  Detail & Related papers  (2024-06-06T10:46:51Z)
• On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm [47.55215041326702]
  We discover an intriguing linear phenomenon in models that are from a common pretrained checkpoint and finetuned on different tasks, termed as Cross-Task Linearity (CTL) We show that if we linearly interpolate the weights of two finetuned models, the features in the weight-interpolated model are often approximately equal to the linearities of features in two finetuned models at each layer. We conjecture that in the pretraining-finetuning paradigm, neural networks approximately function as linear maps, mapping from the parameter space to the feature space.
  arXiv  Detail & Related papers  (2024-02-06T03:28:36Z)
• Layered Models can "Automatically" Regularize and Discover Low-Dimensional Structures via Feature Learning [6.109362130047454]
  We study a two-layer nonparametric regression model where the input undergoes a linear transformation followed by a nonlinear mapping to predict the output.<n>We show that the two-layer model can "automatically" induce regularization and facilitate feature learning.
  arXiv  Detail & Related papers  (2023-10-18T06:15:35Z)
• 2-D SSM: A General Spatial Layer for Visual Transformers [79.4957965474334]
  A central objective in computer vision is to design models with appropriate 2-D inductive bias. We leverage an expressive variation of the multidimensional State Space Model. Our approach introduces efficient parameterization, accelerated computation, and a suitable normalization scheme.
  arXiv  Detail & Related papers  (2023-06-11T09:41:37Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.