Is Escalation Worth It? A Decision-Theoretic Characterization of LLM Cascades

• URL: http://arxiv.org/abs/2605.06350v1
• Date: Thu, 07 May 2026 14:32:27 GMT
• Title: Is Escalation Worth It? A Decision-Theoretic Characterization of LLM Cascades
• Abstract summary: We develop a decision-theoretic framework grounded in constrained optimization and duality.<n>We validate the framework on five benchmarks across eight models from five providers.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model cascades, in which a cheap LLM defers to an expensive one on low-confidence queries, are widely used to navigate the cost-quality tradeoff at deployment. Existing approaches largely treat the deferral threshold as an empirical hyperparameter, with limited guidance on the geometry of the resulting cost-quality frontier over a model pool. We develop a decision-theoretic framework grounded in constrained optimization and duality. For a two-model cascade, we establish piecewise concavity of the cost-quality frontier on decreasing-benefit regions of the confidence support, with reciprocal shadow prices linking the budget- and quality-constrained formulations. Given a pool of $k$ models, we characterize the frontier achievable by deterministic two-model threshold cascades as the pointwise envelope over $\binom{k}{2}$ pairwise cascades, with switching points where the optimal pair changes. For $k$-model cascades, we derive first-order conditions in which a single shadow price equalizes marginal quality-per-cost across stage boundaries. We validate the framework on five benchmarks (MATH, MMLU, TriviaQA, SimpleQA, LiveCodeBench) across eight models from five providers. Within the deterministic threshold-cascade class, full fixed chains underperform the pairwise envelope, and optimized subsequence cascades do not deliver practically meaningful held-out gains over it. A lightweight pre-generation router exceeds the best cascade policy on four of five datasets, mainly because it avoids the cheap model's generation cost on queries sent directly to a larger model rather than because of a stronger routing signal. These results suggest that cascade performance is limited primarily by structural cost, since cascades pay the cheap model before any escalation decision, rather than by a shortage of intermediate stages.

Related papers

• RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning [6.614808404204836]
  We propose a quality-estimator-free framework that formulates cascade routing as a Markov decision process.<n>It uses trajectory returns and advantages to directly optimize the performance-cost objective.
  arXiv  Detail & Related papers  (2026-08-16T15:47:04Z)
• Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference [6.094431019524038]
  Large language model (LLM) cascades reduce inference cost by routing easy queries to a small model and deferring hard queries to a larger one.<n>We introduce textbfConformal Cascade (CC), a multi-tier inference framework that uses conformal prediction set size as the deferral rule.
  arXiv  Detail & Related papers  (2026-07-27T19:19:58Z)
• Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving [69.5155152169752]
  We propose a two-stage cascaded solution for large language models (LLMs)<n>Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model.<n>Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model.<n>On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT)<n>It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.
  arXiv  Detail & Related papers  (2026-06-25T18:29:24Z)
• RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment [57.588738943463646]
  Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT)<n> deploying them at scale remains prohibitively expensive.<n>We propose textbfRouteLMT, an in-model router that predicts the expected gain by probing the small translators prompt-token representation.
  arXiv  Detail & Related papers  (2026-04-24T13:02:45Z)
• CascadeDebate: Multi-Agent Deliberation for Cost-Aware LLM Cascades [3.2562960801091094]
  CascadeDebate is a multi-agent system that coordinates models with human experts to balance accuracy, cost, and abstention under uncertainty.<n>Our architecture alternates single-model inference with selective multi-agent deliberation across model scales, culminating in human experts as the final fallback.<n>Across five benchmarks spanning science, medicine, and general knowledge, CascadeDebate outperforms strong single-model cascades and standalone multi-agent systems by up to 26.75 percent.
  arXiv  Detail & Related papers  (2026-04-14T04:26:39Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• Budget-Aware Agentic Routing via Boundary-Guided Training [24.0709108941881]
  Budget-Aware Agentic Routing selects between a cheap and an expensive model at each step to optimize the cost-success frontier.<n> Boundary-Guided Training builds a difficulty taxonomy to anchor learning under sparse rewards.<n>Experiment results show that our method improves the efficiency frontier, matching strong routing baselines at substantially lower cost.
  arXiv  Detail & Related papers  (2026-02-04T07:39:27Z)
• C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning [24.65381108650337]
  Large language models (LLMs) have achieved impressive results on complex reasoning tasks, but their high inference cost remains a major barrier to real-world deployment.<n>Existing cascade methods rely on supervised training with labeled data, offer no theoretical generalization guarantees, and provide limited control over test-time computational cost.<n>We introduce C3PO, a self-supervised framework for optimizing LLM cascades under probabilistic cost constraints.
  arXiv  Detail & Related papers  (2025-11-10T18:50:27Z)
• Agreement-Based Cascading for Efficient Inference [29.378710032811227]
  Agreement-Based Cascading (ABC) is a simple, effective adaptive inference technique.<n>ABC builds a cascade of models of increasing size/complexity, and uses agreement between ensembles of models at each level of the cascade as a basis for data-dependent routing.<n>We show that ABC can reliably act as a drop-in replacement for existing models and surpass the best single model it aims to replace in terms of both efficiency and accuracy.
  arXiv  Detail & Related papers  (2024-07-02T15:14:12Z)
• Faster Cascades via Speculative Decoding [66.16909847419198]
  Cascades and speculative decoding are approaches to improving language models' inference efficiency. We propose new speculative cascading techniques that implement their deferral rule through speculative execution. We show that our approach yields better cost quality trade-offs than cascading and speculative decoding baselines.
  arXiv  Detail & Related papers  (2024-05-29T16:55:08Z)
• On Optimal Caching and Model Multiplexing for Large Model Inference [66.50550915522551]
  Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. We study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing.
  arXiv  Detail & Related papers  (2023-06-03T05:01:51Z)
• $\texttt{FedBC}$: Calibrating Global and Local Models via Federated Learning Beyond Consensus [66.62731854746856]
  In federated learning (FL), the objective of collaboratively learning a global model through aggregation of model updates across devices tends to oppose the goal of personalization via local information. In this work, we calibrate this tradeoff in a quantitative manner through a multi-criterion-based optimization. We demonstrate that $texttFedBC$ balances the global and local model test accuracy metrics across a suite datasets.
  arXiv  Detail & Related papers  (2022-06-22T02:42:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.