Chiaroscuro Attention: Spending Compute in the Dark

• URL: http://arxiv.org/abs/2606.08327v1
• Date: Sat, 06 Jun 2026 20:38:42 GMT
• Title: Chiaroscuro Attention: Spending Compute in the Dark
• Abstract summary: CHIAR-Former is a 4-layer hybrid transformer that routes each token to one of three operators.<n>A purpose-designed DCT+Attention-only variant achieves Val PPL 36.54 on WikiText-103.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard transformers apply self-attention uniformly at every layer and token, regardless of whether the input requires dynamic cross-token interaction. We propose CHIAR-Former (Chiaroscuro Attention), a 4-layer hybrid transformer that routes each token to one of three operators - DCT spectral mixing, RBF kernel mixing, or full self-attention - based on per-token spectral entropy, a theoretically justified complexity signal. Through systematic ablation on WikiText-103, we discover routing collapse: the router consistently rejects RBF in favour of DCT and attention, revealing that spectral mixing and dynamic attention are complementary and sufficient. A purpose-designed DCT+Attention-only variant achieves Val PPL 36.54 on WikiText-103 - a 45% improvement over a full-attention baseline (PPL 66.62) at 62.5% fewer attention FLOPs. We extend evaluation to WikiText-2, IMDB sentiment classification, and synthetic ListOps operations, establishing a clear operating regime: CHIAR-Former excels on large-scale naturalistic text where token diversity supports spectral specialisation, while full attention retains an edge on small datasets and synthetic pattern-matching tasks. These findings - both the wins and the losses - together define when and why spectral routing earns its keep.

Related papers

• Contour-Guided Spectral Routing for Robust Real-Time Pedestrian Detection [0.0]
  Real-time pedestrian detection in driving scenes is constrained by three coupled failure modes.<n>We formulate a detector through a unified.<n>emphcontour-guided spectral routing view.
  arXiv  Detail & Related papers  (2026-09-13T08:38:54Z)
• WAVE-Stereo: Warp-Aligned Volume Encoding for Stereo Matching [0.0]
  WAVE-Stereo is built on a core insight: correlation volumes and feature warping provide complementary matching cues.<n>On five real-world benchmarks, WAVE-Stereo achieves 3.18% D1-all on KITTI 2015, 4.42% Bad-2.0 on Booster, and 66ms real-time inference, striking a favorable balance between accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-07-15T10:15:05Z)
• The Devil is in the Spectrum: Mitigating Representation Collapse in LLMs via Topologically Regularized Side-Path [10.000577340093434]
  The Topologically Regularized Side-Path (TRSP) is a non-invasive architectural intervention that achieves spectral balance.<n>TRSP employs a parameter-free Triangular Box mechanism, scaled by a lightweight, length-aware gate, to regularize the token interaction topology.<n>Experiments show significant improvements across general capabilities and long-context benchmarks.
  arXiv  Detail & Related papers  (2026-06-02T09:11:50Z)
• Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation [58.33732779555114]
  We argue that the key bottleneck is a representation-level failure caused by two coupled heterogeneities.<n>We propose Dynamic Spectral Denoising with Global-Context Attention for Multi-Behavior Recommendation (SpectraMB)<n>SpectraMB achieves the best results in most evaluation settings and exhibits improved robustness under noisy interactions.
  arXiv  Detail & Related papers  (2026-06-01T15:58:13Z)
• Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention [0.0]
  We identify two complementary inductive biases that standard attention lacks.<n>Salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone.<n>This superadditivity, observed across two independent training runs, is the central empirical finding.
  arXiv  Detail & Related papers  (2026-05-25T22:04:31Z)
• Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention [0.0]
  In turbulent fluid dynamics, coherent structures carry a disproportionate fraction of total energy and govern all transport.<n>We propose that tokens play an analogous role in transformer attention: informationally dense positions concentrate spectral energy.<n>We show that Energy-Gated Attention (EGA) achieves value aggregation by the spectral energy of key token embeddings.
  arXiv  Detail & Related papers  (2026-05-21T00:21:14Z)
• Triple Spectral Fusion for Sensor-based Human Activity Recognition [86.21994396715074]
  This paper proposes a novel triple spectral fusion framework tailored for sensor-based human activity recognition (HAR)<n>We develop an adaptive complementary filtering technique for noise suppression and organize each IMU's sensors into posture and motion modality nodes.<n>An adaptive wavelet frequency selection approach is implemented to suppress context redundancy and shorten the length of features.
  arXiv  Detail & Related papers  (2026-05-04T15:42:58Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat [0.0]
  When sparse attention is trained end-to-end, the model's Q/K/V projections co-adapt to whatever mask is imposed.<n>Differentiable soft gating converges to nearly the same perplexity whether the gate is learned or random.<n>Experts co-adapt to any router, but show that attention exhibits a structurally more severe form.
  arXiv  Detail & Related papers  (2026-02-11T15:06:44Z)
• LEFT: Learnable Fusion of Tri-view Tokens for Unsupervised Time Series Anomaly Detection [53.191369031661885]
  Unsupervised time series anomaly detection aims to build a model for identifying abnormal timestamps without assuming the availability of annotations.<n>We present Learnable Fusion of Tri-view Tokens (LEFT), a unified unsupervised TSAD framework that models anomalies as inconsistencies across complementary representations.<n>Experiments on real-world benchmarks show that LEFT yields the best detection accuracy against SOTA baselines, while achieving a 5x reduction on FLOPs and 8x speed-up for training.
  arXiv  Detail & Related papers  (2026-02-09T13:33:49Z)
• SEER: Spectral Entropy Encoding of Roles for Context-Aware Attention-Based Design Pattern Detection [0.0]
  This paper presents an upgraded version of our prior method Context Is All You Need for detecting Gang of Four (GoF) design patterns from source code.<n> SEER addresses these limitations with two principled additions: (i) a spectral-entropy role encoder that derives per-member role embeddings from the Laplacian spectrum of each class's interaction graph, and (ii) a time-weighted calling context that assigns empirically calibrated duration priors to method categories.<n>We evaluate SEER on PyDesignNet (1,832 files, 35,000 sequences, 23 GoF patterns) and observe consistent gains over our previous system
  arXiv  Detail & Related papers  (2026-01-19T19:13:40Z)
• Hadamard Attention Recurrent Transformer: A Strong Baseline for Stereo Matching Transformer [53.62241289871256]
  We present the Hadamard Attention Recurrent Stereo Transformer (HART)<n>HART includes a novel attention mechanism that incorporates the following components.<n>In reflective area, HART ranked 1st on the KITTI 2012 benchmark.
  arXiv  Detail & Related papers  (2025-01-02T02:51:16Z)
• Sharing Key Semantics in Transformer Makes Efficient Image Restoration [148.22790334216117]
  Self-attention mechanism, a cornerstone of Vision Transformers (ViTs) tends to encompass all global cues.<n>Small segments of a degraded image, particularly those closely aligned semantically, provide particularly relevant information to aid in the restoration process.<n>We propose boosting IR's performance by sharing the key semantics via Transformer for IR (ie, SemanIR) in this paper.
  arXiv  Detail & Related papers  (2024-05-30T12:45:34Z)
• G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation [49.421099172544196]
  We propose a novel semantic-guided feature imitation technique, which automatically performs soft matching between feature pairs across all pyramid levels. We also introduce contrastive distillation to effectively capture the information encoded in the relationship between different feature regions. Our method consistently outperforms the existing detection KD techniques, and works when (1) components in the framework are used separately and in conjunction.
  arXiv  Detail & Related papers  (2021-08-17T07:44:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.