Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention

• URL: http://arxiv.org/abs/2605.21842v1
• Date: Thu, 21 May 2026 00:21:14 GMT
• Title: Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
• Abstract summary: In turbulent fluid dynamics, coherent structures carry a disproportionate fraction of total energy and govern all transport.<n>We propose that tokens play an analogous role in transformer attention: informationally dense positions concentrate spectral energy.<n>We show that Energy-Gated Attention (EGA) achieves value aggregation by the spectral energy of key token embeddings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard transformer attention computes pairwise similarity between queries and keys, treating all tokens as equally salient regardless of their intrinsic informational content. In turbulent fluid dynamics, coherent structures -- the energetically dominant, spatially organized patterns that persist amid background chaos -- carry a disproportionate fraction of total energy and govern all transport. We propose that tokens play an analogous role in transformer attention: informationally dense positions (morphological boundaries, syntactic heads, discourse markers) concentrate spectral energy and should attract proportionally more attention than background tokens (function words, repeated patterns, low-information filler). We propose Energy-Gated Attention (EGA): a simple modification that gates value aggregation by the spectral energy of key token embeddings, computed by a single learned linear projection that discovers the dominant spectral mode of the embedding field. On TinyShakespeare, EGA achieves +0.103 validation loss improvement with only 12,480 additional parameters (<0.26% overhead) and no measurable computational cost. The result is consistent on Penn Treebank (+0.101), demonstrating dataset independence. A systematic ablation across three wavelet families (fixed Morlet, Daubechies db2/db4, and a parametric Morlet) establishes that fixed structured bases are suboptimal -- the optimal energy direction is data-adaptive and non-sinusoidal -- while identifying learned wavelet packets as a promising open direction. The learned energy threshold converges to tau ~= 0.35 independently of initialization, corresponding to the fraction (~36%) of tokens carrying above-average spectral energy in English text, a stable linguistic property consistent with the fraction of content words in running English text.

Related papers

• How Hard Does It Think? Analyzing Step-Aware Reasoning Energy in LLM Chain-of-Thought Trajectories [51.16651846289446]
  We propose Step-Aware Reasoning Energy (SARE), a geometric framework that quantifies effort at the granularity of individual CoT steps.<n>We find that reasoning energy is highly non-uniform across step types, exhibiting phase-like transitions invisible to trajectory-level metrics.<n>SARE-based features match or outperform output-based confidence baselines in most settings.
  arXiv  Detail & Related papers  (2026-07-28T06:15:51Z)
• Chiaroscuro Attention: Spending Compute in the Dark [0.0]
  CHIAR-Former is a 4-layer hybrid transformer that routes each token to one of three operators.<n>A purpose-designed DCT+Attention-only variant achieves Val PPL 36.54 on WikiText-103.
  arXiv  Detail & Related papers  (2026-06-06T20:38:42Z)
• Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention [0.0]
  We identify two complementary inductive biases that standard attention lacks.<n>Salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone.<n>This superadditivity, observed across two independent training runs, is the central empirical finding.
  arXiv  Detail & Related papers  (2026-05-25T22:04:31Z)
• Lost in Tokenization: Fundamental Trade-offs in Graph Tokenization for Transformers [50.98108117044413]
  We show that the choice of a graph-to-token map is a fundamental component of transformer expressivity.<n>We examine three tokenizations that serve as building blocks for many existing graph tokenizations: spectral, random-walk, and adjacency tokenizations.
  arXiv  Detail & Related papers  (2026-05-21T13:32:20Z)
• Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning [11.322734738973603]
  Reinforcement-learning-based post-training has become a key approach for improving the reasoning ability of large language models.<n>This work studies their heterogeneity through attention entropy, which measures how concentrated or diffuse the contextual support is for each response token.
  arXiv  Detail & Related papers  (2026-05-08T12:31:28Z)
• Triple Spectral Fusion for Sensor-based Human Activity Recognition [86.21994396715074]
  This paper proposes a novel triple spectral fusion framework tailored for sensor-based human activity recognition (HAR)<n>We develop an adaptive complementary filtering technique for noise suppression and organize each IMU's sensors into posture and motion modality nodes.<n>An adaptive wavelet frequency selection approach is implemented to suppress context redundancy and shorten the length of features.
  arXiv  Detail & Related papers  (2026-05-04T15:42:58Z)
• A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification [0.8711791966428426]
  We propose a Boltzmann-machine-enhanced Transformer for DNA sequence classification.<n>The model introduces structured binary gating variables to represent latent connections and constrains them with query-key connections.<n>During training, we jointly optimize classification and energy losses, encouraging the model to achieve accurate prediction.
  arXiv  Detail & Related papers  (2026-03-27T14:32:37Z)
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors [53.891337639229285]
  We introduce attentionLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction connection.<n>Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding.
  arXiv  Detail & Related papers  (2026-01-25T19:21:25Z)
• Entropy-based Coarse and Compressed Semantic Speech Representation Learning [72.18542411704347]
  We propose an entropy-based dynamic aggregation framework for learning compressed semantic speech representations.<n> Experiments on ASR, speech-to-text translation, and voice conversion tasks demonstrate that the compressed representations perform on par with or better than dense token sequences.
  arXiv  Detail & Related papers  (2025-08-30T13:50:58Z)
• EnergyFormer: Energy Attention with Fourier Embedding for Hyperspectral Image Classification [3.105394345970172]
  High dimensionality and spectral variability of HSI data pose challenges for feature extraction and classification.<n>This paper presents EnergyFormer, a transformer-based framework designed to address these challenges.<n>EnergyFormer achieves exceptional overall accuracies of 99.28%, 98.63%, and 98.72%, respectively, outperforming state-of-the-art CNN, transformer, and Mamba-based models.
  arXiv  Detail & Related papers  (2025-03-11T10:03:35Z)
• Accelerating Transformers with Spectrum-Preserving Token Merging [43.463808781808645]
  PiToMe prioritizes the preservation of informative tokens using an additional metric termed the energy score. Experimental findings demonstrate that PiToMe saved from 40-60% FLOPs of the base models.
  arXiv  Detail & Related papers  (2024-05-25T09:37:01Z)
• CWF: Consolidating Weak Features in High-quality Mesh Simplification [50.634070540791555]
  We propose a smooth functional that simultaneously considers all of these requirements. The functional comprises a normal anisotropy term and a Centroidal Voronoi Tessellation (CVT) energy term.
  arXiv  Detail & Related papers  (2024-04-24T05:37:17Z)
• Gramformer: Learning Crowd Counting via Graph-Modulated Transformer [68.26599222077466]
  Gramformer is a graph-modulated transformer to enhance the network by adjusting the attention and input node features respectively. A feature-based encoding is proposed to discover the centrality positions or importance of nodes. Experiments on four challenging crowd counting datasets have validated the competitiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-01-08T13:01:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.