Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy

• URL: http://arxiv.org/abs/2603.06875v2
• Date: Tue, 10 Mar 2026 23:55:26 GMT
• Title: Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy
• Abstract summary: We show that Langevin sampling from the corresponding distribution yields attention: a training-free sampler controlled by a single temperature.<n>Because the energy gradient equals the attention map, no score network, training loop, or learned model is required.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention heads retrieve: given a query, they return a softmax-weighted average of stored values. We show that this computation is one step of gradient descent on a classical energy function, and that Langevin sampling from the corresponding distribution yields stochastic attention: a training-free sampler controlled by a single temperature. Lowering the temperature gives exact retrieval; raising it gives open-ended generation. Because the energy gradient equals the attention map, no score network, training loop, or learned model is required. We derive a closed-form entropy inflection condition that identifies the retrieval-to-generation transition temperature for any memory geometry, with a scaling law $β^*\!\sim\!\sqrt{d}$ for random patterns. We validate on five domains (64 to 4,096 dimensions). On MNIST digit images, stochastic attention is $2.6{\times}$ more novel and $2.0{\times}$ more diverse than the best learned baseline (a VAE trained on the same patterns), while matching a Metropolis-corrected gold standard. On protein sequences from the Pfam RRM family, the generation regime achieves $6.9{\times}$ lower amino acid composition divergence than the VAE (KL $= 0.060$ vs.\ $0.416$) at matched novelty, demonstrating that the training-free score function preserves family-level fidelity that learned models lose. A denoising diffusion baseline (DDPM) fails across all memory sizes tested ($K = 100$ to $3{,}500$), producing samples indistinguishable from isotropic noise. The approach requires no architectural changes to the underlying attention mechanism.

Related papers

• Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training [51.18425726646089]
  We propose textscSURE, a unified latent-space framework for image and video diffusion models.<n>It learns reward distributions and directly uses their reliability to guide dense post-training.<n>textscSURE-REFL achieves the highest VBench quality, semantic, and total scores among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-06T14:55:42Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification [3.058685580689604]
  Hardlabel classification is usually trained with smooth surrogate losses, most prominently softmax crossentropy.<n>We isolate an mechanism by which this mismatch between smooth surrogate and discrete labels produces powerlaw learning curves.<n>We show that learning-rate schedules can improve the generalization error towards a $_g sim -1/2$ power law.
  arXiv  Detail & Related papers  (2026-05-21T11:26:32Z)
• Scale-Equivariant Generative Forecasting: Weight-Tied Dilated Convolutions, Wavelet Scattering Inputs, and Spectral-Consistency Training for Self-Similar Time Series [0.0]
  We give a precise definition of discrete scale equivariance for 1D causal networks.<n>We prove that dyadic dilation commutes with any dilated-convolution stack whose kernel weights are shared across levels.<n>On 30 years of S&P 500 daily log-returns, SE-WaveNet samples reproduce the empirical scaling-collapse diagnostic on the Allan-Variance top-25 universe.
  arXiv  Detail & Related papers  (2026-05-17T18:21:30Z)
• Wavelet Variance Equipartition as a Threshold for World-Model Quality and Quantum Kernel TN-Simulability [0.0]
  We identify the wavelet scaling exponent $$ as a critical diagnostic.<n>We establish $= 1/2$ as a sharp transition boundary for the classical simulability of amplitude-encoded quantum kernels.<n>We prove this variance scales strictly as $Var[X] = (d-2)$.
  arXiv  Detail & Related papers  (2026-05-12T05:41:12Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking [44.614763110719274]
  We study the phenomenon of grokking, i.e., delayed generalization.<n>We propose a novel framework that captures three key stages for the grokking behavior of 2-layer nonlinear networks.<n>Our study sheds lights on roles played by hypers such as weight decay, learning rate and sample sizes in grokking.
  arXiv  Detail & Related papers  (2025-09-25T20:08:09Z)
• GradPower: Powering Gradients for Faster Language Model Pre-Training [15.650619834236789]
  GradPower is a lightweight-transformation technique for accelerating language model pre-training.<n>It requires only a single-line code change and no modifications to the base's internal logic.<n>It consistently achieves lower terminal loss across diverse architectures.
  arXiv  Detail & Related papers  (2025-05-30T06:49:57Z)
• Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processes [43.857810191928166]
  We analyze the gap between the training and test errors when training a potentially over-parametrized model.<n>We have no dependence on either training time or reliance on mixing, nor a dependence on dimensionality, gradient norms, or any other properties of the loss or model.
  arXiv  Detail & Related papers  (2025-05-25T10:49:09Z)
• Learning to Scale Logits for Temperature-Conditional GFlowNets [77.36931187299896]
  We propose textitLogit-scaling GFlowNets (Logit-GFN), a novel architectural design that greatly accelerates the training of temperature-conditional GFlowNets. We find that the challenge is greatly reduced if a learned function of the temperature is used to scale the policy's logits directly.
  arXiv  Detail & Related papers  (2023-10-04T13:45:56Z)
• Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models [49.81937966106691]
  We develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach.
  arXiv  Detail & Related papers  (2023-06-15T16:30:08Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Bridging Discrete and Backpropagation: Straight-Through and Beyond [62.46558842476455]
  We propose a novel approach to approximate the gradient of parameters involved in generating discrete latent variables. We propose ReinMax, which achieves second-order accuracy by integrating Heun's method, a second-order numerical method for solving ODEs.
  arXiv  Detail & Related papers  (2023-04-17T20:59:49Z)
• Gradient Estimation for Binary Latent Variables via Gradient Variance Clipping [6.234350105794441]
  gradient estimation is often necessary for fitting generative models with discrete latent variables. DisARM and other estimators have potentially exploding variance near the boundary of the parameter space. We propose a new gradient estimator textitbitflip-1 that has lower variance at the boundaries of the parameter space.
  arXiv  Detail & Related papers  (2022-08-12T05:37:52Z)
• Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models? [46.01087792062936]
  We compare classes of estimators via the relative performance of the emphbest method in the class This allows us to rigorously quantify the tuning sensitivity of learning algorithms.
  arXiv  Detail & Related papers  (2021-08-26T16:01:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.