Dynamic Mode Decomposition along Depth in Vision Transformers

• URL: http://arxiv.org/abs/2605.07556v1
• Date: Fri, 08 May 2026 10:33:03 GMT
• Title: Dynamic Mode Decomposition along Depth in Vision Transformers
• Abstract summary: We ask whether ViT depth implements approximately textitautonomous linear dynamics.<n>We test this using Dynamic Mode Decomposition (DMD), which fits $K$ from selected, consecutive hidden-state pairs.<n>On four pretrained DINO ViTs, we study the regularization, rank, and calibration budget required for stable fitting.
• Score: 2.899294572150795
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent work has shown that contiguous vision transformer (ViT) blocks (a) can be replaced by a linear map and (b) organize into recurrent phases of computation. We ask whether these observations coincide: does ViT depth implement approximately \textit{autonomous linear} dynamics, admitting a single operator $K$ applied recurrently across a contiguous span? We test this using Dynamic Mode Decomposition (DMD), which fits $K$ from selected, consecutive hidden-state pairs and predicts $p$ steps ahead via $K^p$. On four pretrained DINO ViTs, we study the regularization, rank, and calibration budget required for stable fitting. For short spans ($p \leq 4$), $K^p$ tracks an unconstrained endpoint map to within $0.02$ cosine similarity on DINOv3-H/16+, while also recovering intermediate activations at each skipped block. At early cut starts, the fitted operators compress to rank $\ll d$ with minimal calibration data, and across tokens, \texttt{cls} is most amenable to linearization; both properties decay monotonically with depth. Yet this local fidelity does not transfer downstream. At the final hidden state, after propagating through the remaining blocks, an identity baseline becomes competitive.

Related papers

• High-Dimensional Nonparametric Change-Point Detection via Low-Rank Degree-Three Density Projection [2.1991623402625606]
  Distributional changes can be invisible to means and covariances yet appear in skewness, asymmetric interactions, or other third-order structure.<n>We develop a nonparametric change-point method that retains every degree-at-most-three coefficient of a density while avoiding direct density estimation.
  arXiv  Detail & Related papers  (2026-08-16T01:21:55Z)
• DeepLoop: Depth Scaling for Looped Transformers [91.7922038545625]
  Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds.<n>We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient.<n>The resulting method, textbfDeepLoop, keeps the Post-LN DeepNorm architecture and sets $= (2N)1/2$ and $=(8N)-1/2$ for unrolled depth.
  arXiv  Detail & Related papers  (2026-07-15T06:37:05Z)
• Why Do Few-Step Text Latents Fail When Image Latents Work? Non-Commitment at Sharp Categorical Readouts [1.1760370831826417]
  We show that a smooth, regularity-limited deterministic map cannot resolve a discrete branch choice before a sharp categorical readout.<n>In the overlapping regime of real text autoencoders, we prove that the posterior-mean terminal step flips tokens at the rate of the latent mass in an $O(s(t)$ tube around decision boundaries.
  arXiv  Detail & Related papers  (2026-06-29T14:20:57Z)
• Catching a Moving Subspace: Low-Rank Bandits Beyond Stationarity [13.385373310554327]
  We study piecewise-stationary low-rank linear contextual bandits with scalar feedback.<n>With single-play scalar rewards, the moving subspace is recoverable through quadratic functionals of rewards.<n>This is the first work to characterize the identification boundary and attain the intrinsic-rank dynamic-regret rate.
  arXiv  Detail & Related papers  (2026-05-18T22:01:48Z)
• Dimensional Criticality at Grokking Across MLPs and Transformers [2.652953665748039]
  Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems.<n>We introduce textbfTDU--OFC (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe.<n>Across Transformers trained on modular addition and generalizations trained on XOR, we discover a localized crossing of the diffusion baseline $D=1$.
  arXiv  Detail & Related papers  (2026-04-06T13:43:20Z)
• Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inference [1.7523718031184992]
  We identify a fundamental mechanism for this failure: textbfPremature Mode Collapse.<n>We propose textbfEfficient Piecewise Hybrid Adaptive Stability Control (EPH-ASC), an adaptive scheduling algorithm that monitors the stability of the inference process.
  arXiv  Detail & Related papers  (2026-01-30T14:47:18Z)
• Scaling Up Liquid-Resistance Liquid-Capacitance Networks for Efficient Sequence Modeling [50.994194925685434]
  LrcSSM is a $textitnon-linear$ recurrent model that processes long sequences as fast as today's linear state-space layers.<n>By forcing its Jacobian matrix to be diagonal, the full sequence can be solved in parallel.<n>LrcSSM offers a formal gradient-stability guarantee that other input-varying systems such as Liquid-S4 do not provide.
  arXiv  Detail & Related papers  (2025-05-27T20:02:59Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• Control of the von Neumann Entropy for an Open Two-Qubit System Using Coherent and Incoherent Drives [50.24983453990065]
  This article is devoted to developing an approach for manipulating the von Neumann entropy $S(rho(t))$ of an open two-qubit system with coherent control and incoherent control inducing time-dependent decoherence rates. The following goals are considered: (a) minimizing or maximizing the final entropy $S(rho(T))$; (b) steering $S(rho(T))$ to a given target value; (c) steering $S(rho(T))$ to a target value and satisfying the pointwise state constraint $S(
  arXiv  Detail & Related papers  (2024-05-10T10:01:10Z)
• Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPs [56.237917407785545]
  We consider the problem of learning an $varepsilon$-optimal policy in a general class of continuous-space Markov decision processes (MDPs) having smooth Bellman operators. Key to our solution is a novel projection technique based on ideas from harmonic analysis. Our result bridges the gap between two popular but conflicting perspectives on continuous-space MDPs.
  arXiv  Detail & Related papers  (2024-05-10T09:58:47Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Near-Linear Time and Fixed-Parameter Tractable Algorithms for Tensor Decompositions [51.19236668224547]
  We study low rank approximation of tensors, focusing on the tensor train and Tucker decompositions. For tensor train decomposition, we give a bicriteria $(1 + eps)$-approximation algorithm with a small bicriteria rank and $O(q cdot nnz(A))$ running time. In addition, we extend our algorithm to tensor networks with arbitrary graphs.
  arXiv  Detail & Related papers  (2022-07-15T11:55:09Z)
• Beyond Lazy Training for Over-parameterized Tensor Decomposition [69.4699995828506]
  We show that gradient descent on over-parametrized objective could go beyond the lazy training regime and utilize certain low-rank structure in the data. Our results show that gradient descent on over-parametrized objective could go beyond the lazy training regime and utilize certain low-rank structure in the data.
  arXiv  Detail & Related papers  (2020-10-22T00:32:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.