Directional Routing in Transformers

• URL: http://arxiv.org/abs/2603.14923v1
• Date: Mon, 16 Mar 2026 07:28:22 GMT
• Title: Directional Routing in Transformers
• Abstract summary: directional routing is a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router.<n>We trace the resulting circuits through mechanistic interpretability.<n> routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce directional routing, a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router, at 3.9% parameter cost. We train a 433M-parameter model alongside an identical baseline in a single run, then trace the resulting circuits through mechanistic interpretability. Routing becomes the model's dominant computational pathway. Disabling it collapses factual recall to near-zero probability across all 8 test prompts and drops induction accuracy from 93.4% to 0.0%. Knocking out individual attention heads has negligible effect: the primary mover head's removal actually increases target probability, and induction heads retain 98.6% accuracy without their strongest member. The coordination mechanism is irreplaceable; the components it coordinates are not. The model also self-organizes, without explicit pressure, into two regimes: domain-adaptive routing in early layers and fixed syntactic pruning in late layers, where the least-varying layer is the most critical (+42.6 PPL when disabled). Routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World [0.0]
  LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents.<n>Yet agentic routers are evaluated like single-turn routers, by replaying logged trajectories and substituting another model's recorded outputs.<n>We test this assumption with branching rollouts: we fork live SWE-bench agent trajectories at controlled points, rebuild the environment, continue each fork with a different model, and compare against same-model control forks that isolate sampling and replay noise.
  arXiv  Detail & Related papers  (2026-08-08T17:07:11Z)
• Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models [3.747647072414422]
  Mixture-of-Experts (MoE) models decouple total parameters from per-token compute, but deployment still requires storing every expert.<n>Recent theory shows that pruning experts with the smallest router-norm changes during fine-tuning can preserve accuracy, but assumes full fine-tuning.<n>We test whether lightweight adaptation can recover this signal.
  arXiv  Detail & Related papers  (2026-08-08T03:36:41Z)
• Grounding latent algorithm routing in transformer reasoning [2.3977651066739374]
  We study whether transformers can organize episode-level adaptation around different inductive-bias families.<n>We introduce ROUTEBENCH, a diagnostic benchmark whose regimes favor global shrinkage, sparsity, robustness, and locality.<n>Across dense decoder-only transformers trained from scratch at 44M-612M parameters, a 306M model closes 80.9 percent of the oracle-routing gap and achieves route F1 of 84.1.
  arXiv  Detail & Related papers  (2026-07-27T14:07:12Z)
• EntroRouter: Learning Efficient Model Routing via Entropy Regulation [45.88530258803968]
  We propose a single-round routing framework that treats entropy regulation as a core objective.<n>We demonstrate that Entro retains 98.3% of the strongest expert's accuracy while reducing computational costs by 48.25%.
  arXiv  Detail & Related papers  (2026-06-28T14:39:06Z)
• Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference [0.0]
  We propose Meta-Attention, a framework that dynamically routes each token to the most appropriate attention strategy.<n>Unlike prior routing approaches that use deterministic or prior-free learned routing, the Meta-Controller treats per-token mechanism selection as posterior inference.<n>This design produces principled routing uncertainty estimates that govern the soft-to-hard routing transition, mitigates routing collapse without ad hoc load-balancing losses, and yields better compute-performance trade-offs.
  arXiv  Detail & Related papers  (2026-05-27T12:21:28Z)
• The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers [0.0]
  We show that layers in language models perform binary routing of continuous signals.<n>We find that specific neurons implement a consensus architecture that are 93-98% mutually exclusive.<n>We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization.
  arXiv  Detail & Related papers  (2026-03-11T17:14:57Z)
• Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers [28.800065959523696]
  Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning.<n>We propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions.
  arXiv  Detail & Related papers  (2026-02-13T03:59:09Z)
• Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat [0.0]
  When sparse attention is trained end-to-end, the model's Q/K/V projections co-adapt to whatever mask is imposed.<n>Differentiable soft gating converges to nearly the same perplexity whether the gate is learned or random.<n>Experts co-adapt to any router, but show that attention exhibits a structurally more severe form.
  arXiv  Detail & Related papers  (2026-02-11T15:06:44Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• TraceRouter: Robust Safety for Large Foundation Models via Path-Level Intervention [44.64827167753535]
  harmful semantics act as distributed, cross-layer circuits, rendering localized interventions brittle and detrimental to utility.<n>We propose textbfTrace, a path-level framework that traces and disconnects the causal propagation circuits of illicit semantics.<n>Trace significantly outperforms state-of-the-art baselines, achieving a superior trade-off between adversarial robustness and general utility.
  arXiv  Detail & Related papers  (2026-01-29T15:58:12Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performance [87.19164603145056]
  We propose LoRAT, a method that unveils the power of large ViT model for tracking within laboratory-level resources. The essence of our work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency. We design an anchor-free head solely based on to adapt PETR, enabling better performance with less computational overhead.
  arXiv  Detail & Related papers  (2024-03-08T11:41:48Z)
• Retrosynthetic Planning with Dual Value Networks [107.97218669277913]
  We propose a novel online training algorithm, called Planning with Dual Value Networks (PDVN) PDVN alternates between the planning phase and updating phase to predict the synthesizability and cost of molecules. On the widely-used USPTO dataset, our PDVN algorithm improves the search success rate of existing multi-step planners.
  arXiv  Detail & Related papers  (2023-01-31T16:43:53Z)
• ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention [48.697458429460184]
  Two factors, information bottleneck sensitivity and inconsistency between different attention topologies, could affect the performance of the Sparse Transformer. This paper proposes a well-designed model named ERNIE-Sparse. It consists of two distinctive parts: (i) Hierarchical Sparse Transformer (HST) to sequentially unify local and global information, and (ii) Self-Attention Regularization (SAR) to minimize the distance for transformers with different attention topologies.
  arXiv  Detail & Related papers  (2022-03-23T08:47:01Z)
• Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention [4.925939498983408]
  Multi-head self-attention forms the core of Transformer networks. We propose a dynamic pruning method, which exploits the temporal stability of data across tokens to reduce inference cost. Our experiments show that we can reduce 80% of operations while maintaining the original 98.4% accuracy.
  arXiv  Detail & Related papers  (2022-03-20T20:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.