VORT: Adaptive Power-Law Memory for NLP Transformers

• URL: http://arxiv.org/abs/2605.08966v1
• Date: Sat, 09 May 2026 14:20:17 GMT
• Title: VORT: Adaptive Power-Law Memory for NLP Transformers
• Abstract summary: We introduce a memory architecture in which each ingested token is assigned a learnable fractional order _iin[,1]<n>Because the fractional weighted sum is non-Markovian, we approximate it through a sum-of-exponentials (SOE) decomposition computed by Gauss--Laguerre quadrature.<n>Retrieval is keyed and associative via a linear-attention accumulator with an exact O(KSd_d_v) -per-step recurrence.
• Score: 0.913755431537592
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard Transformers impose near-exponential decay on the influence of distant tokens, conflicting with the power-law structure of long-range dependencies in natural language. We introduce the \emph{Variable-Order Retention Transformer} (\VORT{}), a memory architecture in which each ingested token is assigned a learnable fractional order α_i\in[δ,1] that governs a Grünwald--Letnikov power-law retention kernel. Because the fractional weighted sum is non-Markovian, we approximate it through a sum-of-exponentials (SOE) decomposition computed by Gauss--Laguerre quadrature on a Laplace-type integral representation of the kernel weights. Each exponential component admits a one-step Markovian recurrence at O(Sd_v) per step, where S=O(\log(T/\varepsilon)) terms suffice for \varepsilon-uniform accuracy on horizon [1,T]. Retrieval is keyed and associative via a linear-attention accumulator with an exact O(KSd_φd_v) -per-step recurrence. Four results are established: (i) an SOE approximation theorem with geometric convergence rate from the analyticity of the integrand after a log-change of variables; (ii) a quantisation bound valid on [δ,1] with correct analysis near α=0; (iii) a direct L^2 energy argument (Proposition) showing that for α>1/2 any mixture with fixed minimum decay rate Λ>0 incurs L^2([1,T]) error at least N_α(T)-C(Λ)\to\infty, with the Λ-dependence made explicit; and (iv) linear convergence of a gradient plasticity rule under the Polyak--Łojasiewicz condition. Two synthetic experiments confirm the architectural advantage: a Zipf-distributed retrieval benchmark and an entity label-copy task with uniform lag distribution, the latter ruling out prior-matching as an explanation for the power-law kernel's advantage.

Related papers

• Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks [16.84694273405234]
  We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss.<n>Our central question is whether algorithmic stability can explain the statistical generalization of the estimator produced by the discontinuous STE training rule.
  arXiv  Detail & Related papers  (2026-09-06T07:12:16Z)
• An Exactness Barrier for ZX-Calculus Optimization of Synthesized Clifford+T Circuits [3.825239835665478]
  Gate and circuit optimization are usually studied separately.<n>We show both behaviours follow from a single bound.<n>This explains and predicts the size of the near-null optimization pipeline.
  arXiv  Detail & Related papers  (2026-08-24T04:53:20Z)
• A proximal subgradient method for nonconvex stochastic optimization under the Kurdyka-Łojasiewicz condition [39.146761527401424]
  This work introduces a proximal subgradient method for minimizing the sum of an expected cost.<n>We use the Kurdyka-Lojasiewicz (KL) property to guarantee convergence of the whole trajectory to a single stationary point.
  arXiv  Detail & Related papers  (2026-08-05T23:07:27Z)
• Geometric Classification of Biased Quantum Capacity via Harmonic Translation [0.20524609401792393]
  Under uniform locality, the maximal logical dimension under t-local phase errors equals Aq(n,2t+1)>Bq(n,2t+1)<n>For structured phase noise, exact correction is equivalent to independence in an additive Cayley graph.<n>For mixed Pauli noise, simultaneous protection in conjugate domains incurs an intrinsic rate penalty R = 1-(gamma_X+gamma_Z)/2.
  arXiv  Detail & Related papers  (2026-03-21T04:17:42Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Analysis of Hessian Scaling for Local and Global Costs in Variational Quantum Algorithm [0.42970700836450487]
  We quantify the entrywise resolvability of the Hessian for Variational Quantum Algorithms.<n>We show two distinct scaling regimes that govern the sample complexity required to resolve Hessian entries against shot noise.
  arXiv  Detail & Related papers  (2026-01-31T15:49:23Z)
• Linear combination of unitaries with exponential convergence [0.0]
  We present a general method for decomposing non-unitary operators into a linear combination of unitary operators.<n>When implemented in a quantum circuit, the subnormalisation of the resulting block encoding scales with the double logarithm of the inverse error.
  arXiv  Detail & Related papers  (2026-01-25T22:47:21Z)
• Transmutation based Quantum Simulation for Non-unitary Dynamics [35.35971148847751]
  We present a quantum algorithm for simulating dissipative diffusion dynamics generated by positive semidefinite operators of the form $A=Ldagger L$.<n>Our main tool is the Kannai transform, which represents the diffusion semigroup $e-TA$ as a Gaussian-weighted superposition of unitary wave propagators.
  arXiv  Detail & Related papers  (2026-01-07T05:47:22Z)
• Correlation Lengths for Stochastic Matrix Product States [0.8594140167290097]
  We prove the existence of thermodynamic limits of expectations of local observables.<n>In the homogeneous (random translation-invariant) case, for any error tolerance in probability, the two-point function decays exponentially in the distance between the two sites.<n>In the i.i.d. case, the exponential decay still holds with a deterministic rate, with the probability approaching one exponentially fast in the distance.
  arXiv  Detail & Related papers  (2025-10-08T21:17:43Z)
• On the Constant Depth Implementation of Pauli Exponentials [49.48516314472825]
  We decompose $Zotimes n$ exponentials of arbitrary length into circuits of constant depth using $mathcalO(n)$ ancillae and two-body XX and ZZ interactions.<n>We prove the correctness of our approach, after introducing novel rewrite rules for circuits which benefit from qubit recycling.
  arXiv  Detail & Related papers  (2024-08-15T17:09:08Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Convergence of Adam Under Relaxed Assumptions [72.24779199744954]
  We show that Adam converges to $epsilon$-stationary points with $O(epsilon-4)$ gradient complexity under far more realistic conditions. We also propose a variance-reduced version of Adam with an accelerated gradient complexity of $O(epsilon-3)$.
  arXiv  Detail & Related papers  (2023-04-27T06:27:37Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• Spectral Analysis of Product Formulas for Quantum Simulation [0.0]
  We show that the Trotter step size needed to estimate an energy eigenvalue within precision can be improved in scaling from $epsilon$ to $epsilon1/2$ for a large class of systems. Results partially generalize to diabatic processes, which remain in a narrow energy band separated from the rest of the spectrum by a gap.
  arXiv  Detail & Related papers  (2021-02-25T03:17:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.