FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention

• URL: http://arxiv.org/abs/2607.07478v1
• Date: Wed, 08 Jul 2026 14:43:00 GMT
• Title: FourierQK: Spectral Preprocessing of Query-Key Projections Improves Transformer Attention
• Abstract summary: spectral preprocessing of Q/K projections improves transformer attention on character-level language modelling.<n>Results are verified by a shuffled-validation diagnostic against positional leakage.<n>This work defines an architectural boundary between bilateral spectral attention and causal attention at wordscale tokenisation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling. On TinyShakespeare: a fixed random spectral filter achieves val=1.031 (Delta=+0.443); a single learned frequency at paragraph scale achieves val=0.608 (Delta=+0.867); and four learned frequencies spanning paragraph to word scale achieve val=0.309 (Delta=+1.166), a 79% reduction over standard dot-product attention. The single-frequency result is confirmed across three random seeds (mean val=0.236, std=0.019). The four frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens/cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales. The gain is specific to spectral preprocessing: random orthogonal and non-orthogonal projections of Q/K produce no measurable improvement, suggesting the benefit comes from global frequency-domain mixing rather than metric distortion. All results are verified by a shuffled-validation diagnostic against positional leakage. Causal filters (Gaussian, Mexican Hat, Morlet) do not improve over standard attention at character-level tokenisation: the bilateral FFT kernel is structurally non-causal, coupling every position to future tokens. This defines an architectural boundary between bilateral spectral attention (this paper) and genuinely causal spectral attention at word-scale tokenisation (companion paper MorletQK). This work is architecturally distinct from FNet (Lee-Thorp et al., 2021), which replaces attention with Fourier mixing of token embeddings. Here, spectral preprocessing applies only to Q/K projections while the full attention score structure is preserved.

Related papers

• FLaG: Frequency-Domain Latent-attention Gated Pooling for Token Aggregation [6.298319903082888]
  Token aggregation converts token-level representations into fixed-dimensional sample representations, but most pooling methods operate only in the original token space.<n>We introduce Frequency-Domain Latent-attention Gated Pooling (FLaG), a plug-in aggregation module that re-expresses encoder outputs in the domain before final pooling.<n>FLaG represents nonredundant rFFT spectrum through real and imaginary components, summarizes spectral tokens with learnable latent queries, derives a sample-conditioned gate, and reconstructs modulated token representations for downstream aggregation.
  arXiv  Detail & Related papers  (2026-09-01T07:32:44Z)
• Circuit Depth Compression via Spectral Gap Amplification in Quantum Phase Estimation [1.2318267573115809]
  We show that quantum phase estimation can be significantly compressed in depth by preprocessing the input operator.<n>We prove that this compression is exact, bounded by log2(1/(4w Delta_lambda) + 1, and impossible for exactly spectra.<n>A net resource advantage occurs when 4w2 (2Delta_m - 1) > Delta_lambda log (1/epsilon)
  arXiv  Detail & Related papers  (2026-08-14T06:45:48Z)
• Overcoming Fourier Locking in Quantum Data Re-uploading Classifiers via Spectral Homotopy [0.0]
  Data re-uploading quantum circuits (DRUPQCs) are universal function approximators.<n>We show that the primary bottleneck in DRUPQCs is not structural misalignment but a failure mode we term Fourier locking.
  arXiv  Detail & Related papers  (2026-07-13T02:28:17Z)
• Graph Convolutional Attention: A Spectral Perspective on Graph Denoising and Diffusion [82.7007012514703]
  We show that under a denoising objective, linear attention is suboptimal and can only learn an average spectral denoising filter over the training distribution.<n>We introduce Spectral Attention, which directly utilizes the input graph spectrum and provably outperforms linear attention by a margin governed by the spectral diversity of the distribution.<n>For block models, GCA provably matches the idealized Spectral Attention mechanism.
  arXiv  Detail & Related papers  (2026-07-07T17:52:17Z)
• SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneity [57.84199526590779]
  Federated Learning (FL) is fundamentally challenged by statistical heterogeneity, where non-identically distributed (non-IID) data induces client drift that hampers global convergence.<n>In this work, we revisit client drift from a novel frequency-domain perspective and uncover a critical Spectral Bias of Drift.<n>Motivated by this, we propose SpecGradFilter, a unified Spectral Filtering Gradient Framework that tames heterogeneity by suppressing discordant low-frequency signals.
  arXiv  Detail & Related papers  (2026-07-05T09:23:35Z)
• SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference [0.0]
  We present SpiralFovea, a parameter-free, input-adaptive tokeniser.<n>Around content-driven hotspot anchors, multi-scale spiral rings produce = 78 patches that replace the standard 196-patch ViT grid at the input stage.<n>Across four canonical fine-grained benchmarks, SpiralFovea yields +1.7-2.1 pp accuracy with a 60% reduction in input tokens, an 84% reduction in self-attention FLOPs at every transformer layer, and 18-29% throughput gains over the matched static tokenisation baseline.
  arXiv  Detail & Related papers  (2026-07-01T11:09:07Z)
• Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention [0.0]
  We identify two complementary inductive biases that standard attention lacks.<n>Salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone.<n>This superadditivity, observed across two independent training runs, is the central empirical finding.
  arXiv  Detail & Related papers  (2026-05-25T22:04:31Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• The Extrapolation Cliff in On-Policy Distillation of Near-Deterministic Structured Outputs [52.709361620508595]
  ListOPD brings a student to in-domain with an 8B-SFT baseline at one-fifth the parameters.<n>On Amazon Fashion, three pre-registered tests--a fine-grid cliff interval test, and a small-clip cross-prediction--fall within their locked prediction windows--with the small-clip value matching the closed-form prediction below grid resolution.
  arXiv  Detail & Related papers  (2026-05-09T06:48:00Z)
• Long Range Frequency Tuning for QML [2.34764060945577]
  Trainable-frequency approaches theoretically reduce this to match the target spectrum size, requiring only as many encoding gates as frequencies in the target spectrum.<n>We show that gradient-based optimization can drive prefactors to arbitrary target values, but they exhibit limited trainability.<n>We propose grid-based initialization using ternary encodings, which generate dense integer frequency spectra.
  arXiv  Detail & Related papers  (2026-02-26T14:56:37Z)
• PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading [0.0]
  PlotChain is a deterministic, generator-based benchmark for evaluating large language models (MLLMs)<n>PlotChain contains 15 plot families with 450 rendered plots (30 per family)<n>Top models achieve 80.42% (Gemini 2.5 Pro), 79.84% (GPT-4.1), and 78.21% (Claude Sonnet 4.5) overall field-level pass rates, while GPT-4o trails at 61.59%.
  arXiv  Detail & Related papers  (2026-01-29T06:08:19Z)
• SpectralAR: Spectral Autoregressive Visual Generation [74.48368364895387]
  We propose a Spectral AutoRegressive visual generation framework, which realizes causality for visual sequences from the spectral perspective.<n>By considering different levels of detail in images, our SpectralAR achieves both sequence causality and token efficiency without bells and whistles.
  arXiv  Detail & Related papers  (2025-06-12T17:57:44Z)
• Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study [0.23020018305241333]
  Chirp-like patterns in EEG spectrograms are key biomarkers for seizure dynamics.<n>This study bridges this gap by fine-tuning a Vision Transformer (ViT) model and Low-Rank Adaptation (LoRA)<n>We generated 100000 spectrograms with chirp parameters, creating the first large-scale benchmark for chirp localization.
  arXiv  Detail & Related papers  (2025-03-24T14:27:07Z)
• Learning Temporal Resolution in Spectrogram for Audio Classification [40.80903296278466]
  This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution modeling for audio classification. Given a spectrogram calculated with a fixed hop size, DiffRes merges non-essential time frames while preserving important frames. Compared with previous methods using the fixed temporal resolution, the DiffRes-based method can achieve the equivalent or better classification accuracy with at least 25% computational cost reduction.
  arXiv  Detail & Related papers  (2022-10-04T16:18:50Z)
• Low-complexity deep learning frameworks for acoustic scene classification [64.22762153453175]
  We present low-complexity deep learning frameworks for acoustic scene classification (ASC) The proposed frameworks can be separated into four main steps: Front-end spectrogram extraction, online data augmentation, back-end classification, and late fusion of predicted probabilities. Our experiments conducted on DCASE 2022 Task 1 Development dataset have fullfiled the requirement of low-complexity and achieved the best classification accuracy of 60.1%.
  arXiv  Detail & Related papers  (2022-06-13T11:41:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.