When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models

• URL: http://arxiv.org/abs/2603.20997v1
• Date: Sun, 22 Mar 2026 01:04:57 GMT
• Title: When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models
• Abstract summary: We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We identify a routing paradox in hybrid recurrent-attention architectures: content-based routing - deciding which tokens deserve expensive attention - requires exactly the pairwise computation that routing is designed to avoid. Through 20+ controlled experiments across three tasks (a synthetic diagnostic, the Zoology MQAR benchmark, and HotpotQA), we map the routing landscape exhaustively. One layer of softmax attention creates a latent ~34-dimensional subspace enabling 98.4% routing precision; zero layers yield 1.2%. This subspace is invisible to cosine similarity, destroyed by random projections (98.4% to 2.6%), and cannot be created by contrastive pretraining - proving attention's role is writing pairwise match results into representations, not merely computing them. Twelve alternative mechanisms all cluster at 15-29%. Non-learned indices (Bloom filter: 90.9%; BM25 on HotpotQA: 82.7%) bypass the bottleneck entirely. The result is a sharp two-regime hierarchy with an empty middle ground. These findings provide the mechanistic explanation for the empirical observation that recurrent models fail at associative recall, and reframe attention as a representation constructor rather than merely a computation mechanism.

Related papers

• CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing [90.63350360476294]
  We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm.<n>We show that the routing decision can be read directly off the structure of the proxy attention map.<n>We formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts.
  arXiv  Detail & Related papers  (2026-09-01T22:49:46Z)
• From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers [16.546400103214715]
  Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution.<n>We propose RaDiCal, a training-free framework that uses normalized attention entropy to decide when saliency can be trusted.<n>It cuts FLOPs by 39--45% and delivers 1.28--1.45$times$ measured speedups across two VLM architectures without dataset-specific retuning.
  arXiv  Detail & Related papers  (2026-09-01T03:45:21Z)
• Mechanism-level routing failure in LLMs over Lean-verified algebraic structures [3.8048003898069975]
  We study structural routing failure in large language models (LLMs) over a formally verified algebraic corpus.<n>Our central finding is a mechanism-level routing ceiling.<n>A cross-model dissociation in Llama is notable.
  arXiv  Detail & Related papers  (2026-07-05T22:45:49Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Cascade-Aware Multi-Agent Routing: Spatio-Temporal Sidecars and Geometry-Switching [0.0]
  A common architectural pattern in advanced AI reasoning systems is the symbolic graph network.<n>Current schedulers optimize and fitness but are geometry-blind.<n>In tree-like delegation, a single failure can cascade exponentially.<n>In dense cyclic graphs, failures tend to self-limit.
  arXiv  Detail & Related papers  (2026-03-17T20:10:16Z)
• The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers [0.0]
  We show that layers in language models perform binary routing of continuous signals.<n>We find that specific neurons implement a consensus architecture that are 93-98% mutually exclusive.<n>We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization.
  arXiv  Detail & Related papers  (2026-03-11T17:14:57Z)
• Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers [0.0]
  We introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt.<n>We show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity.<n>A logistic regression trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification.
  arXiv  Detail & Related papers  (2026-03-11T12:45:53Z)
• SEER: Spectral Entropy Encoding of Roles for Context-Aware Attention-Based Design Pattern Detection [0.0]
  This paper presents an upgraded version of our prior method Context Is All You Need for detecting Gang of Four (GoF) design patterns from source code.<n> SEER addresses these limitations with two principled additions: (i) a spectral-entropy role encoder that derives per-member role embeddings from the Laplacian spectrum of each class's interaction graph, and (ii) a time-weighted calling context that assigns empirically calibrated duration priors to method categories.<n>We evaluate SEER on PyDesignNet (1,832 files, 35,000 sequences, 23 GoF patterns) and observe consistent gains over our previous system
  arXiv  Detail & Related papers  (2026-01-19T19:13:40Z)
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning [49.16227597771663]
  D2Pruner is a framework that combines debiased importance with a structural pruning mechanism.<n>It reduces FLOPs by 74.2% while retaining 99.2% of its original performance.<n>It marks a significant advancement with up to 63. 53% improvement over existing methods.
  arXiv  Detail & Related papers  (2025-12-22T14:42:31Z)
• DeepPrune: Parallel Scaling without Inter-trace Redundancy [53.62015294143274]
  Over 80% of parallel reasoning traces yield identical final answers, representing substantial wasted computation.<n>We propose DeepPrune, a novel framework that enables efficient parallel scaling through dynamic pruning.<n>Our work establishes a new standard for efficient parallel reasoning, making high-performance reasoning more efficient.
  arXiv  Detail & Related papers  (2025-10-09T17:24:54Z)
• Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation [84.00166854547241]
  Diffusion Transformers (DiTs) are essential for video generation but suffer from significant latency due to the quadratic complexity of attention.<n>We propose SVG2, a training-free framework that maximizes identification accuracy and computation minimizes waste.
  arXiv  Detail & Related papers  (2025-05-24T21:30:29Z)
• Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction [52.14200610448542]
  A transformer has a quadratic complexity, leading to high inference costs and latency for long sequences.<n>We propose a simple, novel, and effective procedure for correcting this distributional shift.<n>Our method can maintain approximately 98.5% sparsity over full quadratic attention, making our model 32 times faster than Flash Attention 2 when processing 1M token prefills.
  arXiv  Detail & Related papers  (2025-05-16T13:48:33Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.