RoPE attention is an exact forward-pass gradient step with softmax intact

• URL: http://arxiv.org/abs/2609.06685v1
• Date: Sun, 06 Sep 2026 15:53:38 GMT
• Title: RoPE attention is an exact forward-pass gradient step with softmax intact
• Abstract summary: We derive an exact gradient-step representation of the RoPE-softmax forward pass.<n>We prove that a nonconstant finite-cache head cannot admit a globally exact affine query readout.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we construct a query-dependent effective matrix $ΔM_i$ satisfying $y_i = μ_i + u_i^\top ΔM_i$, where $μ_i$ is the uniform mean of the attended values and $u_i$ is the augmented query input. The construction applies the classical exponential divided difference $ρ= φ_1$ to retain the softmax exactly. Its positive coefficients give a unit gradient-step representation on a query-conditioned quadratic objective. The same function connects the RoPE generator to exact positional finite differences. We derive a tokenwise formula for the error of reusing one query's matrix and prove that a nonconstant finite-cache head cannot admit a globally exact affine query readout. Reconstruction checks and frozen-reuse calibration on one pretrained Qwen2.5-0.5B layer verify the representation and quantify the correction required when one query's matrix is reused.

Related papers

• Scalable estimation of VARMA models [4.301367153728694]
  Autoregressive moving-average (VARMA) models have been considered impractical beyond moderate dimensions.<n>We introduce an estimation that removes the bias of the series length.<n>This brings VARMA estimation, at a per-it cost independent of the series length, to the problem sizes where practitioners have far relied on VAR models.
  arXiv  Detail & Related papers  (2026-08-06T17:49:58Z)
• Finding Most Influential Sets [0.0]
  Dinkelbach's method yields an algorithm with $mathcalO(n)$ cost per iteration and finite termination.<n>We show that the method recovers exact MIS that were previously computationally inaccessible.
  arXiv  Detail & Related papers  (2026-06-04T09:24:26Z)
• Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization [12.92201130466754]
  CAQ-ZO forms one-grid-step Rademacher stencils $z pm r$ in $z = (x)$, maps endpoints back through $-1$, and updates in $z$.<n>Our theory proves the grid-spanarity, decomposes endpoint-rounding estimator residuals, and gives station bounds in which generic off-grid queries retain a $2/2$ residual channel while CAQ-ZO makes the query-time residual exactly zero.
  arXiv  Detail & Related papers  (2026-05-11T14:55:20Z)
• Provably Adaptive Linear Approximation for the Shapley Value and Beyond [73.0940890296463]
  A fundamental and long-standing challenge is their efficient approximation.<n>We develop a linear-space algorithm that requires $P(|hatboldsymbol-boldsymbol|_2geq)leq $ for all commonly used semi-values.<n>Our algorithm allows explicit minimization of the mean square error for each specific utility function.
  arXiv  Detail & Related papers  (2026-04-09T16:38:14Z)
• Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterization [85.91302339486673]
  We study robust Markov decision processes (RMDPs) with general policy parameterization under s-rectangular and non-rectangular uncertainty sets.<n>We prove novel Lipschitz and Lipschitz-smoothness properties for general policy parameterizations that extends to infinite state spaces.<n>We design a projected gradient descent algorithm for s-rectangular uncertainty and a Frank-Wolfe algorithm for non-rectangular uncertainty.
  arXiv  Detail & Related papers  (2026-02-11T21:44:20Z)
• Parameter-free Algorithms for the Stochastically Extended Adversarial Model [59.81852138768642]
  Existing approaches for the Extended Adversarial (SEA) model require prior knowledge of problem-specific parameters, such as the diameter of the domain $D$ and the Lipschitz constant of the loss functions $G$.<n>We develop parameter-free methods by leveraging the Optimistic Online Newton Step (OONS) algorithm to eliminate the need for these parameters.
  arXiv  Detail & Related papers  (2025-10-06T10:53:37Z)
• Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization [56.78271181959529]
  SubLoRA is a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function.<n>Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency.
  arXiv  Detail & Related papers  (2025-07-02T15:56:40Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Regression for matrix-valued data via Kronecker products factorization [0.5156484100374059]
  We propose an estimation algorithm, termed KRO-PRO-FAC, for estimating the parameters $beta_1k subset Rep times q_1$ and $beta_2k subset Rep times q$. Numerical studies on simulated and real data indicate that our procedure is competitive, in terms of both estimation error and predictive accuracy, compared to other existing methods.
  arXiv  Detail & Related papers  (2024-04-30T02:44:41Z)
• Conditional Matrix Flows for Gaussian Graphical Models [1.6435014180036467]
  We propose a general framework for variation inference matrix GG-Flow in which the benefits of frequent keyization and Bayesian inference are studied. As a train of the sparse for any $lambda$ and any $l_q$ (pse-) and for any $l_q$ (pse-) we have to (i) train the limit for any $lambda$ and any $l_q$ (pse-) and (like for the selection) the frequent solution.
  arXiv  Detail & Related papers  (2023-06-12T17:25:12Z)
• Bridging Discrete and Backpropagation: Straight-Through and Beyond [62.46558842476455]
  We propose a novel approach to approximate the gradient of parameters involved in generating discrete latent variables. We propose ReinMax, which achieves second-order accuracy by integrating Heun's method, a second-order numerical method for solving ODEs.
  arXiv  Detail & Related papers  (2023-04-17T20:59:49Z)
• Variational Inference for Bayesian Bridge Regression [0.0]
  We study the implementation of Automatic Differentiation Variational inference (ADVI) for Bayesian inference on regression models with bridge penalization. The bridge approach uses $ell_alpha$ norm, with $alpha in (0, +infty)$ to define a penalization on large values of the regression coefficients. We illustrate the approach on non-parametric regression models with B-splines, although the method works seamlessly for other choices of basis functions.
  arXiv  Detail & Related papers  (2022-05-19T12:29:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.