Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

• URL: http://arxiv.org/abs/2605.26355v1
• Date: Mon, 25 May 2026 22:04:31 GMT
• Title: Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
• Abstract summary: We identify two complementary inductive biases that standard attention lacks.<n>Salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone.<n>This superadditivity, observed across two independent training runs, is the central empirical finding.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard transformer attention computes pairwise token similarity but treats all tokens as equally salient and all positions as equally local, regardless of the informational structure of the input. We identify two complementary inductive biases that standard attention lacks: energy salience (which tokens concentrate informational energy, learned end-to-end without explicit frequency decomposition) and scale-selective locality (how far positional influence extends at each frequency, implemented via Morlet wavelet encoding). We address both with two simple components. Energy-Gated Attention (EGA) gates value aggregation by a learned energy estimate of key token embeddings, computed via a single linear projection; it selects what to attend to. Morlet Positional Encoding (MoPE) replaces fixed sinusoidal encodings with learned Gaussian-windowed wavelets that adapt the joint position-frequency localization to the corpus; it specifies where attention operates at each scale. On TinyShakespeare, EGA alone achieves +0.092 validation loss improvement over standard attention (+0.103 over Phase 1-3 baseline); MoPE alone is -0.032 (below baseline as a standalone encoding); but their combination achieves +0.119 -- more than the sum of parts. This superadditivity, observed across two independent training runs, is the central empirical finding: salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone. Ablations confirm that structured spectral priors (Morlet wavelet gates, scale-initialized heads, fixed sinusoidal PE) consistently underperform their unconstrained learned counterparts, while complementary learned components interact superadditively. All experiments are at small scale (<=6M parameters, character-level benchmarks, single seed); larger-scale multi-seed validation is the most important direction for future work.

Related papers

• Adaptive Anisotropic Attention for Axis-Structured Signals [5.616707402426108]
  Self-attention treats all token pairs as equally plausible before learning, an interaction-isotropic prior that can be mismatched to structured signals.<n>We introduce Adaptive Anisotropic Attention (AAA), which splits attention into two paths: a temporal path and a spatial path.<n>We show that both paths are necessary and that the weighted sum beats a hard choice of one path; most of the benefit comes from the gate learning a different temporal/spatial balance at each layer of the network.
  arXiv  Detail & Related papers  (2026-09-08T14:19:33Z)
• Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention [10.26086309142631]
  GPT attention measures token compatibility through dot-product similarity.<n>It does not explicitly model whether strong token features should reinforce or suppress one another.<n>We introduce Q-Interference, a fully classical quantum-inspired attention mechanism for autoregressive language modeling.
  arXiv  Detail & Related papers  (2026-08-18T02:38:17Z)
• Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding [0.0]
  We propose that the Morlet wavelet is the natural basis for positional encoding, and introduce Morlet Positional dimension (MoPE)<n>MoPE learns its own frequency and locality bandwidth from data.<n>The phase of MoPE recovers the RoPE rotation angle exactly; the amplitude adds a learned Gaussian kernel that standard encodings lack.
  arXiv  Detail & Related papers  (2026-05-31T14:26:26Z)
• Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention [0.0]
  In turbulent fluid dynamics, coherent structures carry a disproportionate fraction of total energy and govern all transport.<n>We propose that tokens play an analogous role in transformer attention: informationally dense positions concentrate spectral energy.<n>We show that Energy-Gated Attention (EGA) achieves value aggregation by the spectral energy of key token embeddings.
  arXiv  Detail & Related papers  (2026-05-21T00:21:14Z)
• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors [53.891337639229285]
  We introduce attentionLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction connection.<n>Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding.
  arXiv  Detail & Related papers  (2026-01-25T19:21:25Z)
• Focus What Matters: Matchability-Based Reweighting for Local Feature Matching [6.361840891399624]
  We propose a novel attention reweighting mechanism that simultaneously incorporates a learnable bias term into the attention logits.<n>Experiments conducted on three benchmark datasets validate the effectiveness of our method.
  arXiv  Detail & Related papers  (2025-05-04T15:50:28Z)
• Hadamard Attention Recurrent Transformer: A Strong Baseline for Stereo Matching Transformer [53.62241289871256]
  We present the Hadamard Attention Recurrent Stereo Transformer (HART)<n>HART includes a novel attention mechanism that incorporates the following components.<n>In reflective area, HART ranked 1st on the KITTI 2012 benchmark.
  arXiv  Detail & Related papers  (2025-01-02T02:51:16Z)
• CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction [77.8576094863446]
  We propose a new detextbfCoupled dutextbfAl-interactive lineatextbfR atttextbfEntion (CARE) mechanism. We first propose an asymmetrical feature decoupling strategy that asymmetrically decouples the learning process for local inductive bias and long-range dependencies. By adopting a decoupled learning way and fully exploiting complementarity across features, our method can achieve both high efficiency and accuracy.
  arXiv  Detail & Related papers  (2024-11-25T07:56:13Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• EcoFormer: Energy-Saving Attention with Linear Complexity [40.002608785252164]
  Transformer is a transformative framework that models sequential data. We propose a new binarization paradigm customized to high-dimensional softmax attention. We show that EcoFormer consistently achieves comparable performance with standard attentions.
  arXiv  Detail & Related papers  (2022-09-19T13:28:32Z)
• Rethinking Query-Key Pairwise Interactions in Vision Transformers [5.141895475956681]
  We propose key-only attention, which excludes query-key pairwise interactions and uses a compute-efficient saliency-gate to obtain attention weights. We develop a new self-attention model family, LinGlos, which reach state-of-the-art accuracies on the parameter-limited setting of ImageNet classification benchmark.
  arXiv  Detail & Related papers  (2022-07-01T03:36:49Z)
• Adaptive Split-Fusion Transformer [90.04885335911729]
  We propose an Adaptive Split-Fusion Transformer (ASF-former) to treat convolutional and attention branches differently with adaptive weights. Experiments on standard benchmarks, such as ImageNet-1K, show that our ASF-former outperforms its CNN, transformer counterparts, and hybrid pilots in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-26T10:00:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.