Influence Malleability in Linearized Attention: Dual Implications of Non-Convergent NTK Dynamics

• URL: http://arxiv.org/abs/2603.13085v1
• Date: Fri, 13 Mar 2026 15:33:34 GMT
• Title: Influence Malleability in Linearized Attention: Dual Implications of Non-Convergent NTK Dynamics
• Abstract summary: We show that linearized attention does not converge to its infinite-width NTK limit.<n>We also show that attention's power and vulnerability share a common origin in its departure from the kernel regime.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the theoretical foundations of attention mechanisms remains challenging due to their complex, non-linear dynamics. This work reveals a fundamental trade-off in the learning dynamics of linearized attention. Using a linearized attention mechanism with exact correspondence to a data-dependent Gram-induced kernel, both empirical and theoretical analysis through the Neural Tangent Kernel (NTK) framework shows that linearized attention does not converge to its infinite-width NTK limit, even at large widths. A spectral amplification result establishes this formally: the attention transformation cubes the Gram matrix's condition number, requiring width $m = Ω(κ^6)$ for convergence, a threshold that exceeds any practical width for natural image datasets. This non-convergence is characterized through influence malleability, the capacity to dynamically alter reliance on training examples. Attention exhibits 6--9$\times$ higher malleability than ReLU networks, with dual implications: its data-dependent kernel can reduce approximation error by aligning with task structure, but this same sensitivity increases susceptibility to adversarial manipulation of training data. These findings suggest that attention's power and vulnerability share a common origin in its departure from the kernel regime.

Related papers

• Correlation flow governs learning at criticality [9.341735544356167]
  Correlation propagation to infinite depth is possible only at a single critical point in the weight-bias variance plane.<n>We show that the end-to-end Jacobian vanishes algebraically with depth, and use this to prove that the NTK becomes exactly proportional to the output correlation at infinite depth.
  arXiv  Detail & Related papers  (2026-08-08T22:18:10Z)
• Exact Linear Attention [0.0]
  Exact Linear Attention (ELA) achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions.<n>ELA achieves up to 6x faster decoding speed and 75% reduction in KV cache memory usage compared to full attention.<n>We extend the linear attention principle to vision models, yielding YOLO-LAT, which attains up to 4.3x GPU inference speedup and 7.9x parameter reduction with competitive detection accuracy.
  arXiv  Detail & Related papers  (2026-05-13T08:06:48Z)
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors [53.891337639229285]
  We introduce attentionLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction connection.<n>Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding.
  arXiv  Detail & Related papers  (2026-01-25T19:21:25Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• Information-Theoretic Bounds and Task-Centric Learning Complexity for Real-World Dynamic Nonlinear Systems [0.6875312133832079]
  Dynamic nonlinear systems exhibit distortions arising from coupled static and dynamic effects.<n>This paper presents a theoretical framework grounded in structured decomposition, variance analysis, and task-centric complexity bounds.
  arXiv  Detail & Related papers  (2025-09-08T12:08:02Z)
• Analysis of Fourier Neural Operators via Effective Field Theory [11.824913874212802]
  We present a systematic effective field theory analysis of FNOs in an infinite dimensional function space.<n>We show that nonlinear activations inevitably couple frequency inputs to high frequency modes that are otherwise discarded by spectral truncation.<n>Our results quantify how nonlinearity enables neural operators to capture non-trivial features and explain why scale invariant activations and residual connections enhance feature learning in FNOs.
  arXiv  Detail & Related papers  (2025-07-29T14:10:46Z)
• A Random Matrix Analysis of In-context Memorization for Nonlinear Attention [18.90197287760915]
  We show that nonlinear Attention incurs higher memorization error than linear ridge regression on random inputs.<n>Our results reveal how nonlinearity and input structure interact with each other to govern the memorization performance of nonlinear Attention.
  arXiv  Detail & Related papers  (2025-06-23T13:56:43Z)
• Revealing Decurve Flows for Generalized Graph Propagation [108.80758541147418]
  This study addresses the limitations of the traditional analysis of message-passing, central to graph learning, by defining em textbfgeneralized propagation with directed and weighted graphs. We include a preliminary exploration of learned propagation patterns in datasets, a first in the field.
  arXiv  Detail & Related papers  (2024-02-13T14:13:17Z)
• Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systems [1.0878040851638]
  This paper delves into gradient descent-based learning algorithms, that display a linear structure in their parameter dynamics. We establish this apparent linearity arises due to weak correlations between the first and higher-order derivatives of the hypothesis function. Exploiting the relationship between linearity and weak correlations, we derive a bound on deviations from linearity observed during the training trajectory of gradient descent.
  arXiv  Detail & Related papers  (2024-01-08T16:44:23Z)
• SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biases [66.61789780666727]
  We show how the second-order continuity can be incorporated into GNNs while maintaining the equivariant property. We also offer theoretical insights into SEGNO, highlighting that it can learn a unique trajectory between adjacent states. Our model yields a significant improvement over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-08-25T07:15:58Z)
• Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
  convergence rate analysis of the mean field Langevin dynamics is presented. $p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
  arXiv  Detail & Related papers  (2022-01-25T17:13:56Z)
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks [51.1848572349154]
  neural network models that perfectly fit noisy data can generalize well to unseen test data. We consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk.
  arXiv  Detail & Related papers  (2021-08-25T22:01:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.