Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States

• URL: http://arxiv.org/abs/2603.29206v1
• Date: Tue, 31 Mar 2026 03:19:36 GMT
• Title: Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States
• Abstract summary: We quantify internal density via activation sparsity, (C2) domain-keyword attention, and (C3) output stability via predictive entropy and semantic variation.<n>We present RIDE as a diagnostic probe for calibrating routing design and uncertainty estimation.
• Score: 10.639846833695806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Routing is widely used to scale large language models, from Mixture-of-Experts gating to multi-model/tool selection. A common belief is that routing to a task ``expert'' activates sparser internal computation and thus yields more certain and stable outputs (the Sparsity--Certainty Hypothesis). We test this belief by injecting routing-style meta prompts as a textual proxy for routing signals in front of frozen instruction-tuned LLMs. We quantify (C1) internal density via activation sparsity, (C2) domain-keyword attention, and (C3) output stability via predictive entropy and semantic variation. On a RouterEval subset with three instruction-tuned models (Qwen3-8B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.2), meta prompts consistently densify early/middle-layer representations rather than increasing sparsity; natural-language expert instructions are often stronger than structured tags. Attention responses are heterogeneous: Qwen/Llama reduce keyword attention, while Mistral reinforces it. Finally, the densification--stability link is weak and appears only in Qwen, with near-zero correlations in Llama and Mistral. We present RIDE as a diagnostic probe for calibrating routing design and uncertainty estimation.

Related papers

• Mixture-of-Expert Blocks Contain Strong Hallucination Detection Signals [2.2302915692528367]
  We introduce InnerExpert, the first method to leverage MoE-specific signals for per-token hallucination detection.<n>Our results show that InnerExpert outperforms existing methods across five datasets and two MoE architectures.
  arXiv  Detail & Related papers  (2026-08-18T12:00:20Z)
• StructRL: Structured Action-Space Exploration for Flow-Based VLAs [81.43458629514139]
  Flow-based Vision-Language-Action models are now widely used for continuous robotic manipulation.<n>Online reinforcement learning is emerging as a key technique for adapting them to new tasks.<n>We show that effective robot exploration calls for structured noise: temporally smooth and scaled differently across action groups.
  arXiv  Detail & Related papers  (2026-08-15T09:31:29Z)
• CausalGate: Causal Importance Distillation for Transformer Module Pruning [0.0]
  CausalGate is an intervention-guided framework for compute-efficient runtime inference.<n>It measures semantic damage via the Kullback-Leibler divergence of the final logit distribution.<n>It consistently outperforms prominent dynamic routing and layer-skipping baselines.
  arXiv  Detail & Related papers  (2026-07-21T20:19:59Z)
• How Much Static Structure Do Code Agents Need? A Study of Deterministic Anchoring [10.106526629957655]
  LLM-based code agents navigate repositories through keyword search but miss the structural relationships that define how software actually works.<n>We investigate whether lightweight static analysis can provide deterministic anchors for these agents.<n>Our study identifies what we call the deterministic anchoring effect: static structure helps less by making agents "smarter" and more by making their navigation disciplined and reproducible.
  arXiv  Detail & Related papers  (2026-06-25T12:50:01Z)
• Route Before Retrieve: Activating Latent Routing Abilities of LLMs for RAG vs. Long-Context Selection [57.3886742625188]
  Pre-Route is a proactive routing framework that performs structured reasoning before answering.<n>Our study shows three key findings: (i) LLMs possess latent routing ability that can be reliably elicited with guidelines; (ii) linear probes reveal that structured prompts sharpen the separability of the "optimal routing" in representation space; and (iii) distillation transfers this reasoning structure to smaller models for lightweight deployment.
  arXiv  Detail & Related papers  (2026-05-11T09:10:55Z)
• Sparse Visual Thought Circuits in Vision-Language Models [2.5754366051855837]
  We develop a causal pipeline to localize and test sparse visual thought circuits in Qwen3-VL-8B.<n>We train SAEs at this layer, construct task-selective sets via an explicit rule, and perform inference time scaling and ablation while quantifying accuracy and drift.
  arXiv  Detail & Related papers  (2026-03-26T06:24:36Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Listen to the Layers: Mitigating Hallucinations with Inter-Layer Disagreement [0.24443539255794253]
  Pretrained Large Language Models (LLMs) are prone to generating fluent yet factually incorrect text-a phenomenon known as hallucinations.<n>We propose a novel, training-free decoding algorithm that mitigates hallucinations at inference time by listening to these signals in the middle layers.
  arXiv  Detail & Related papers  (2026-02-10T07:32:37Z)
• RankSteer: Activation Steering for Pointwise LLM Ranking [13.718395381871751]
  Large language models (LLMs) have recently shown strong performance as zero-shot rankers, yet their effectiveness is highly sensitive to prompt formulation.<n>We propose RankSteer, a post-hoc activation steering framework for zero-shot pointwise LLM ranking.<n> Experiments on TREC DL 20 and multiple BEIR benchmarks show that RankSteer consistently improves ranking quality using only a small number of anchor queries.
  arXiv  Detail & Related papers  (2026-02-03T11:49:00Z)
• TraceRouter: Robust Safety for Large Foundation Models via Path-Level Intervention [44.64827167753535]
  harmful semantics act as distributed, cross-layer circuits, rendering localized interventions brittle and detrimental to utility.<n>We propose textbfTrace, a path-level framework that traces and disconnects the causal propagation circuits of illicit semantics.<n>Trace significantly outperforms state-of-the-art baselines, achieving a superior trade-off between adversarial robustness and general utility.
  arXiv  Detail & Related papers  (2026-01-29T15:58:12Z)
• RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steering [62.63376387138257]
  We propose a plug-and-play intervention framework that adaptively steers large language models (LLMs) reasoning in activation space.<n>RISER constructs a library of reusable reasoning vectors and employs a lightweight Router to dynamically compose them for each input.<n>The Router is optimized via reinforcement learning under task-level rewards, activating latent cognitive primitives in an emergent and compositional manner.
  arXiv  Detail & Related papers  (2026-01-14T08:04:33Z)
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models [86.88657425848547]
  Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning.<n>We explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks.<n>Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosts performance by over 10% relative to instruction-tuned baselines.
  arXiv  Detail & Related papers  (2025-05-15T17:58:33Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.