Grounding latent algorithm routing in transformer reasoning

• URL: http://arxiv.org/abs/2607.24471v1
• Date: Mon, 27 Jul 2026 14:07:12 GMT
• Title: Grounding latent algorithm routing in transformer reasoning
• Abstract summary: We study whether transformers can organize episode-level adaptation around different inductive-bias families.<n>We introduce ROUTEBENCH, a diagnostic benchmark whose regimes favor global shrinkage, sparsity, robustness, and locality.<n>Across dense decoder-only transformers trained from scratch at 44M-612M parameters, a 306M model closes 80.9 percent of the oracle-routing gap and achieves route F1 of 84.1.
• Score: 2.3977651066739374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central question in the in-context learning literature is whether transformers can organize episode-level adaptation around different inductive-bias families. We study this question in a controlled setting through latent algorithm routing: route-like behavior in which the solver-family preference changes with the latent data-generating regime while prompt form is held fixed, remains stable under nuisance perturbations, and is selectively influenced by targeted activation interventions without large losses in answer quality. We introduce ROUTEBENCH, a diagnostic benchmark whose regimes differentially favor global shrinkage, sparsity, robustness, and locality, operationalized by ridge-like, lasso-like, Huber-like, and kNN-like family representatives. Across dense decoder-only transformers trained from scratch at 44M-612M parameters, a 306M model closes 80.9 percent of the oracle-routing gap and achieves route F1 of 84.1. The effect remains substantial under natural-language renderings, shuffled supports, lexical paraphrases, and a unified four-way routing setting. Stronger adaptive alternatives, including an input-conditioned soft mixture and an unsupervised Gumbel router, narrow the gap but remain below the 306M and 612M models on route F1 and OOD performance. Probe controls and matched activation-patching controls further show that route-relevant internal directions are decodable and functionally involved in solver-family-consistent output behavior. These results provide controlled evidence that dense transformers trained on ROUTEBENCH can develop route-like internal variables, but they do not establish universal routing in pretrained language models or unrestricted natural-language reasoning.

Related papers

• RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models [13.298401993417611]
  We introduce a router-agnostic representation engineering framework for MoE language models.<n>RARE projects arbitrary behavioral perturbations onto the null space of the router matrix, thereby removing router-visible components.<n>On harmfulness steering, RARE reaches an average attack success rate of 53.3% while retaining 67.8% MMLU accuracy.
  arXiv  Detail & Related papers  (2026-08-21T15:45:36Z)
• Meta-Attention: Bayesian Per-Token Routing for Efficient Transformer Inference [0.0]
  We propose Meta-Attention, a framework that dynamically routes each token to the most appropriate attention strategy.<n>Unlike prior routing approaches that use deterministic or prior-free learned routing, the Meta-Controller treats per-token mechanism selection as posterior inference.<n>This design produces principled routing uncertainty estimates that govern the soft-to-hard routing transition, mitigates routing collapse without ad hoc load-balancing losses, and yields better compute-performance trade-offs.
  arXiv  Detail & Related papers  (2026-05-27T12:21:28Z)
• How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models [0.0]
  This paper localizes the policy routing mechanism in alignment-trained language models.<n>An intermediate-layer attention gate reads detected content and triggers deeper heads that boost the signal toward refusal.
  arXiv  Detail & Related papers  (2026-04-06T03:20:37Z)
• Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States [10.639846833695806]
  We quantify internal density via activation sparsity, (C2) domain-keyword attention, and (C3) output stability via predictive entropy and semantic variation.<n>We present RIDE as a diagnostic probe for calibrating routing design and uncertainty estimation.
  arXiv  Detail & Related papers  (2026-03-31T03:19:36Z)
• Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation [0.0]
  We test a strong-only hypothesis in two controlled settings.<n>In Experiment 1, we use a cellular automaton with a pure XOR transition rule.<n>In Experiment 2, we study symbolic operator chains over integers with one operator pair held out.
  arXiv  Detail & Related papers  (2026-03-17T18:02:28Z)
• Directional Routing in Transformers [0.0]
  directional routing is a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router.<n>We trace the resulting circuits through mechanistic interpretability.<n> routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
  arXiv  Detail & Related papers  (2026-03-16T07:28:22Z)
• Adaptive Linear Path Model-Based Diffusion [52.84663832658799]
  We introduce Linear Path Model-Based Diffusion (LP-MBD), which replaces the variance-preserving schedule with a flow-matching linear probability path.<n>We also propose Adaptive LP-MBD (ALP-MBD), which leverages reinforcement learning to adjust diffusion steps and noise levels according to task complexity and environmental conditions.
  arXiv  Detail & Related papers  (2026-02-02T21:33:03Z)
• Error Amplification Limits ANN-to-SNN Conversion in Continuous Control [64.99656514469972]
  Spiking Neural Networks (SNNs) can achieve competitive performance by converting already existing well-trained Artificial Neural Networks (ANNs)<n>Existing conversion methods perform poorly in continuous control, where suitable baselines are largely absent.<n>We propose Cross-Step Residual Potential Initialization (CRPI), a lightweight training-free mechanism that carries over residual membrane potentials across decision steps to suppress temporally correlated errors.
  arXiv  Detail & Related papers  (2026-01-29T14:28:00Z)
• RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steering [62.63376387138257]
  We propose a plug-and-play intervention framework that adaptively steers large language models (LLMs) reasoning in activation space.<n>RISER constructs a library of reusable reasoning vectors and employs a lightweight Router to dynamically compose them for each input.<n>The Router is optimized via reinforcement learning under task-level rewards, activating latent cognitive primitives in an emergent and compositional manner.
  arXiv  Detail & Related papers  (2026-01-14T08:04:33Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking [51.154226183713405]
  We propose Inner Thinking Transformer, which reimagines layer computations as implicit thinking steps.<n>ITT achieves 96.5% performance of a 466M Transformer using only 162M parameters, reduces training data by 43.2%, and outperforms Transformer/Loop variants in 11 benchmarks.
  arXiv  Detail & Related papers  (2025-02-19T16:02:23Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.