Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat

• URL: http://arxiv.org/abs/2603.02227v1
• Date: Wed, 11 Feb 2026 15:06:44 GMT
• Title: Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
• Abstract summary: When sparse attention is trained end-to-end, the model's Q/K/V projections co-adapt to whatever mask is imposed.<n>Differentiable soft gating converges to nearly the same perplexity whether the gate is learned or random.<n>Experts co-adapt to any router, but show that attention exhibits a structurally more severe form.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can a transformer learn which attention entries matter during training? In principle, yes: attention distributions are highly concentrated, and a small gate network can identify the important entries post-hoc with near-perfect accuracy. In practice, barely. When sparse attention is trained end-to-end, the model's Q/K/V projections co-adapt to whatever mask is imposed, absorbing the routing signal until learned gates perform little better than frozen random gates. We call this routing absorption and present four independent lines of evidence for it in a controlled 31M-parameter transformer: (1) differentiable soft gating converges to nearly the same perplexity whether the gate is learned or random (48.73 +/- 0.60 vs. 49.83 +/- 0.04 over 3 seeds); (2) hard top-k gating receives exactly zero gradient through the mask; (3) a gate distilled onto co-adapted Q/K/V achieves high F1 against oracle masks but catastrophic perplexity when deployed (601.6 vs. 48.6 on mask-agnostic Q/K/V); and (4) stochastic mask randomization during training fails to prevent co-adaptation (78.2 ppl deployed dense vs. 37.3 baseline). We connect routing absorption to the same phenomenon in Mixture-of-Experts, where random routing matches learned routing because experts co-adapt to any router, but show that attention exhibits a structurally more severe form: shared Q/K/V parameters enable cross-layer compensation pathways absent in MoE, where experts are self-contained modules. The implication is that end-to-end sparse attention methods employing per-query token-level gating face absorption pressure proportional to the parameter asymmetry between the gate and the model, and that post-hoc approaches, which decouple representation learning from sparsification, sidestep this entirely.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models [18.625239232550932]
  We find that across five injection designs, every gated pathway becomes behaviourally closed, with accuracy invariant to ablating the pathway at inference even when the gate parameter would nominally pass 30-45% of the signal.<n>We attribute this suppression phenomenon to two regimes, a dead-gradient regime formalised through the caption-invariance of image-derived signals, and a negative-utility regime in which the auxiliary signal actively hurts the loss.
  arXiv  Detail & Related papers  (2026-07-25T19:14:34Z)
• Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference [45.663066206650115]
  Split inference is often impractical, pushing inference on sensitive text to third-party providers.<n>We introduce ConjFormer, a transformer variant that is exactly $mathrmO(d)$-equivariant via a lightweight normalization change.
  arXiv  Detail & Related papers  (2026-06-15T09:31:24Z)
• Chiaroscuro Attention: Spending Compute in the Dark [0.0]
  CHIAR-Former is a 4-layer hybrid transformer that routes each token to one of three operators.<n>A purpose-designed DCT+Attention-only variant achieves Val PPL 36.54 on WikiText-103.
  arXiv  Detail & Related papers  (2026-06-06T20:38:42Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models [0.0]
  This paper localizes the policy routing mechanism in alignment-trained language models.<n>An intermediate-layer attention gate reads detected content and triggers deeper heads that boost the signal toward refusal.
  arXiv  Detail & Related papers  (2026-04-06T03:20:37Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• Directional Routing in Transformers [0.0]
  directional routing is a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router.<n>We trace the resulting circuits through mechanistic interpretability.<n> routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
  arXiv  Detail & Related papers  (2026-03-16T07:28:22Z)
• Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds [6.908972852063454]
  Mixture-of-Experts models rely on learned routers to assign tokens to experts, yet standard softmax gating provides no principled mechanism to control the tradeoff between sparsity and utilization.<n>We propose Grassmannian MoE, a routing framework that operates on the Grassmannian manifold of subspaces.
  arXiv  Detail & Related papers  (2026-02-19T20:03:23Z)
• Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers [28.800065959523696]
  Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning.<n>We propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions.
  arXiv  Detail & Related papers  (2026-02-13T03:59:09Z)
• What Layers When: Learning to Skip Compute in LLMs with Residual Gates [66.23658560048241]
  GateSkip is a residual-stream gating mechanism that enables token-wise layer skipping in decoder-only LMs.<n>Each Attention/MLP branch is equipped with a sigmoid-linear gate that condenses the branch's output before it re-enters the residual stream.
  arXiv  Detail & Related papers  (2025-10-13T16:31:50Z)
• A fidelity-driven approach to quantum circuit partitioning via weighted hypergraphs for noise-resilient computation [0.0]
  This paper introduces Fidelipart, a novel framework that transforms quantum circuits into a fidelity-aware hypergraph.<n>We evaluate Fidelipart against BQSKit's QuickPartitioner on 6-qubit/22-gate, 10-qubit/55-gate, and 24-qubit/88-gate benchmarks.<n>Results demonstrate Fidelipart's superior performance. It achieved SWAP gate reductions ranging from 77.3% to 100% and up to a 52.2% decrease in cut qubits.
  arXiv  Detail & Related papers  (2025-06-07T17:13:51Z)
• VSA: Faster Video Diffusion with Trainable Sparse Attention [38.37291040904089]
  Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D attention, even though most of the attention mass concentrates on a small subset of positions.<n>We turn this observation into VSA, a trainable, hardware-efficient sparse attention that replaces full attention at emphboth training and inference.
  arXiv  Detail & Related papers  (2025-05-19T17:30:13Z)
• SCHEME: Scalable Channel Mixer for Vision Transformers [52.605868919281086]
  Vision Transformers have achieved impressive performance in many computation tasks.<n>We show that the dense connections can be replaced with a sparse block diagonal structure that supports larger expansion ratios.<n>We also propose the use of a lightweight, parameter-free, channel covariance attention mechanism as a parallel branch during training.
  arXiv  Detail & Related papers  (2023-12-01T08:22:34Z)
• Random quantum circuits anti-concentrate in log depth [118.18170052022323]
  We study the number of gates needed for the distribution over measurement outcomes for typical circuit instances to be anti-concentrated. Our definition of anti-concentration is that the expected collision probability is only a constant factor larger than if the distribution were uniform. In both the case where the gates are nearest-neighbor on a 1D ring and the case where gates are long-range, we show $O(n log(n)) gates are also sufficient.
  arXiv  Detail & Related papers  (2020-11-24T18:44:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.