Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference

• URL: http://arxiv.org/abs/2609.08189v1
• Date: Tue, 08 Sep 2026 03:20:07 GMT
• Title: Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference
• Abstract summary: Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens.<n>Existing methods treat each routing decision as a local operation conditioned solely on the current hidden state.<n>We propose History-Aware Routing (HeRo), a dynamic routing framework that maintains an explicit routing state across model depth.
• Score: 13.935681015472262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, however, treat each routing decision as a local operation conditioned solely on the current hidden state which is a formulation that overlooks the sequential, path-dependent nature of routing across depth: earlier decisions shape the representations seen by downstream routers, and the layer-usage objective couples all decisions jointly. We propose History-Aware Routing (HeRo), a dynamic routing framework that resolves this mismatch by introducing a router memory mechanism to maintain an explicit routing state across model depth. The memory is constructed via linear attention, incrementally aggregating preceding routing scores and their induced residual updates into a compact history representation. At each routed layer, the router conditions jointly on this accumulated state and the current hidden representation to select the executed branch. Instantiated for token-wise FFN routing, HeRo trains only lightweight routers and adapters on a frozen backbone, requiring no modification to pretrained parameters. Across Llama 3.1-8B, Llama 2-7B, and Llama 2-13B, HeRo consistently achieves the highest aggregate performance retention among ten baselines. On Llama 3.1-8B, it bypasses 26.87% of model parameters while achieving 100.24% of dense model performance across seven benchmarks, and retains 97.01% while bypassing 38.82% of model parameters under a tighter computation budget. Ablation studies confirm that removing routing history consistently degrades performance, most notably on multistep reasoning and code generation, validating that explicit routing memory enables more accurate and adaptive dynamic routing than solely conditioning on hidden state.

Related papers

• LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference [0.0]
  LayerRoute is a parameter-efficient method for adaptive transformer layer-skipping.<n>It combines per-layer hard-gated routing with joint LoRA fine-tuning.<n>It trains in under 7 minutes on a single A100 and adds negligible overhead.
  arXiv  Detail & Related papers  (2026-09-12T03:30:59Z)
• SWRouter: Similarity-Contractive Window Routing for Multi-Turn Large Language Model Conversations [88.71160380389277]
  We propose SW routers, a Similarity-Contractive Window Router for multi-turn large language model routing.<n> Experiments on multi-turn dialogue benchmarks demonstrate that SW consistently surpasses strong baselines.<n>Our results highlight that multi-turn large language model routing requires a joint design of context construction and evaluation.
  arXiv  Detail & Related papers  (2026-09-10T11:45:55Z)
• RouteRelay: Event-Triggered Cross-Layer Route Reuse for Efficient Dynamic Sparse Attention [10.543370808241159]
  RouteRelay is a router-agnostic method that reuses only route metadata across depth.<n>We show that RouteRelay retains at least 99.99% route recall while rerouting 25.0%, 55.4%, and 78.2% of rows under low, moderate, and high cross-layer drift.
  arXiv  Detail & Related papers  (2026-09-07T10:23:06Z)
• Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts [0.0]
  We show that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems.<n>A matched-rank comparison shows that residual representations are often easier to predict across layers, while router-control states preserve the model's expert choices much more faithfully.
  arXiv  Detail & Related papers  (2026-09-02T10:17:12Z)
• CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing [90.63350360476294]
  We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm.<n>We show that the routing decision can be read directly off the structure of the proxy attention map.<n>We formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts.
  arXiv  Detail & Related papers  (2026-09-01T22:49:46Z)
• TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI [8.291830023781403]
  We present TRACE, a task-level routing framework that aligns routing with the unit of supervision.<n>By leveraging delayed task feedback, TRACE- learns routing policies that adapt to the workload.<n>It consistently improves the accuracy-matched trade-off, achieving non-latency frontier points.
  arXiv  Detail & Related papers  (2026-07-24T16:29:06Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routing [41.435534676485936]
  RouteJudge is an online pairwise preference evaluation framework for LLM routing systems.<n>Multiple routing strategies independently recommend model models under the same model pool and budget constraints.<n>The resulting user preferences are attributed back to the routing strategies behind the compared responses.
  arXiv  Detail & Related papers  (2026-06-17T07:35:10Z)
• Learning Agent Routing From Early Experience [55.88886987958933]
  Boundary is a training-free routing framework that uses early behavioral experience and rubric-guided reasoning to decide whether to answer a query with direct inference or escalate to an agent.<n>We show that Boundary reduces inference time by 60.6% compared to the agent while improving performance by 28.6% over direct LLM inference.
  arXiv  Detail & Related papers  (2026-05-08T03:18:40Z)
• Dr.LLM: Dynamic Layer Routing in LLMs [55.11953638340419]
  Dr.LLM is a retrofittable framework that equips pretrained models with lightweight per-layer routers deciding to skip, execute, or repeat a block.<n>On ARC (logic) and DART (math), Dr.LLM improves accuracy by up to +3.4%p while saving 5 layers per example on average.
  arXiv  Detail & Related papers  (2025-10-14T17:51:26Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.