Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey

• URL: http://arxiv.org/abs/2603.04445v1
• Date: Mon, 23 Feb 2026 21:57:27 GMT
• Title: Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
• Abstract summary: Large language models (LLMs) with diverse capabilities, costs, and domains have created a critical need for intelligent model selection at inference time.<n> Dynamic routing systems that adaptively select models based on query characteristics have emerged as a solution to this challenge.<n>We provide a systematic analysis of state-of-the-art multi-LLM routing and cascading approaches.
• Score: 4.273936276295959
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid growth of large language models (LLMs) with diverse capabilities, costs, and domains has created a critical need for intelligent model selection at inference time. While smaller models suffice for routine queries, complex tasks demand more capable models. However, static model deployment does not account for the complexity and domain of incoming queries, leading to suboptimal performance and increased costs. Dynamic routing systems that adaptively select models based on query characteristics have emerged as a solution to this challenge. We provide a systematic analysis of state-of-the-art multi-LLM routing and cascading approaches. In contrast to mixture-of-experts architectures, which route within a single model, we study routing across multiple independently trained LLMs. We cover diverse routing paradigms, including query difficulty, human preferences, clustering, uncertainty quantification, reinforcement learning, multimodality, and cascading. For each paradigm, we analyze representative methods and examine key trade-offs. Beyond taxonomy, we introduce a conceptual framework that characterizes routing systems along three dimensions: when decisions are made, what information is used, and how they are computed. This perspective highlights that practical systems are often compositional, integrating multiple paradigms under operational constraints. Our analysis demonstrates that effective multi-LLM routing requires balancing competing objectives. Choosing the optimal routing strategy depends on deployment and computational constraints. Well-designed routing systems can outperform even the most powerful individual models by strategically leveraging specialized capabilities across models while maximizing efficiency gains. Meanwhile, open challenges remain in developing routing mechanisms that generalize across diverse architectures, modalities, and applications.

Related papers

• Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning [87.18590662029432]
  We propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models.<n>Our draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query.<n>These ratings enable fine-grained, instance-level textbfProactive Routing and substantially accelerate inference without compromising overall performance.
  arXiv  Detail & Related papers  (2026-06-29T12:30:24Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts [56.02203242609604]
  Large language models (LLMs) fine-tuned with lightweight adapters achieve strong performance across diverse tasks.<n>Fusing independently trained models with different strengths has shown promise for multi-task learning through three main strategies.<n>We empirically evaluate their trade-offs, addressing two key questions: What are the advantages of going beyond uniform ensembling or merging? and does the flexibility of routing justify its complexity?
  arXiv  Detail & Related papers  (2026-03-03T21:44:11Z)
• CONCUR: A Framework for Continual Constrained and Unconstrained Routing [79.85419373937765]
  AI tasks differ in complexity and are best addressed with different computation strategies.<n>Most prior methods build the routing framework by training a single model across all strategies.<n>We propose CONCUR, a continual routing framework that supports both constrained and unconstrained routing.
  arXiv  Detail & Related papers  (2025-12-10T07:30:13Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inference [37.57624773333661]
  MoMA (Mixture of Models and Agents) is a framework that integrates both large language models (LLMs) and agent-based routing.<n>We present a training dataset to profile the capabilities of various LLMs under different routing model structures.<n>During inference, queries are dynamically routed to the LLM with the best cost-performance efficiency.
  arXiv  Detail & Related papers  (2025-09-09T10:15:42Z)
• RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory [57.449129198822476]
  RCR is a role-aware context routing framework for multi-agent large language model (LLM) systems.<n>It dynamically selects semantically relevant memory subsets for each agent based on its role and task stage.<n>A lightweight scoring policy guides memory selection, and agent outputs are integrated into a shared memory store.
  arXiv  Detail & Related papers  (2025-08-06T21:59:34Z)
• Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques [14.892995952768352]
  Language Models (LMs) have excelled at tasks like text generation, summarization, and question answering.<n>Their inference remains computationally expensive and energy intensive in settings with limited hardware, power, or bandwidth.<n>Recent approaches have introduced multi LLM intelligent model selection strategies that dynamically allocate computational resources based on query complexity.
  arXiv  Detail & Related papers  (2025-06-06T23:13:08Z)
• A Unified Approach to Routing and Cascading for LLMs [5.653106385738822]
  Large language models (LLMs) embedded in various agentic systems have increased the potential of model selection strategies to improve the cost-performance tradeoff.<n>Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found.<n>We derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy.<n>We propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy.
  arXiv  Detail & Related papers  (2024-10-14T10:00:49Z)
• Towards Robust Multi-Modal Reasoning via Model Selection [7.6621866737827045]
  LLM serves as the "brain" of the agent, orchestrating multiple tools for collaborative multi-step task solving. We propose the $textitM3$ framework as a plug-in with negligible runtime overhead at test-time. Our experiments reveal that our framework enables dynamic model selection, considering both user inputs and subtask dependencies.
  arXiv  Detail & Related papers  (2023-10-12T16:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.