Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference

• URL: http://arxiv.org/abs/2609.07786v1
• Date: Mon, 07 Sep 2026 17:31:58 GMT
• Title: Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
• Abstract summary: Large language model cascades answer easy requests with a small model and escalate selected requests to a larger model.<n>This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one.<n>We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference.
• Score: 18.473413242504503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong. This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one. We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference. We show that this signed conditional gain is the Bayes-optimal routing score under a fixed escalation budget. SRR requires only the small model's output statistics at deployment and adds a lightweight two-head router. We evaluate SRR with Qwen3-4B and Qwen3-8B on TBD examples from MMLU, HellaSwag, and ARC-Challenge. Across the accuracy-compute curve, SRR reaches an area of TBD, compared with TBD for a learned small-model error predictor and TBD for entropy routing. These results show that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.

Related papers

• Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inference [5.997400745120167]
  We propose a token-aware progressive model routing method with adaptive edge-cloud collaboration for efficient multimodal inference.<n>Pro-Theater employs a two-stage progressive decision mechanism. First, a lightweight prompt pre-scorer module performs rapid pre-screening before token generation begins.<n>Second, a token-aware verifier reads the sampling probability distribution of each token the small model generates, estimating the model's confidence in its own output to determine, per request, whether the answer ships or escalates.
  arXiv  Detail & Related papers  (2026-08-28T16:45:58Z)
• A Systematic Sample Size Analysis of ML-Based Path Loss Prediction for LPWAN [1.411614392022118]
  Low Power Wide Area Networks like LoRa are increasingly deployed for smart city applications.<n>Traditional (empirical) propagation models often exhibit limited accuracy in these scenarios.<n>We investigate machine learning models for LoRa path loss prediction, systematically analyzing how prediction accuracy scales with training set size.
  arXiv  Detail & Related papers  (2026-08-11T15:48:54Z)
• Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning [87.18590662029432]
  We propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models.<n>Our draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query.<n>These ratings enable fine-grained, instance-level textbfProactive Routing and substantially accelerate inference without compromising overall performance.
  arXiv  Detail & Related papers  (2026-06-29T12:30:24Z)
• Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving [69.5155152169752]
  We propose a two-stage cascaded solution for large language models (LLMs)<n>Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model.<n>Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model.<n>On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT)<n>It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.
  arXiv  Detail & Related papers  (2026-06-25T18:29:24Z)
• From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing [79.19755531338872]
  Existing routing methods treat a model's single response to a query as its capability label for training.<n>We show that this assumption introduces systematic noise into routing supervision, making learned routing policies less reliable.<n>We propose DARS (Distribution-Aware Routing Supervision), a framework that constructs routing supervision from a distributional view of model behavior.
  arXiv  Detail & Related papers  (2026-06-05T05:42:00Z)
• RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment [57.588738943463646]
  Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT)<n> deploying them at scale remains prohibitively expensive.<n>We propose textbfRouteLMT, an in-model router that predicts the expected gain by probing the small translators prompt-token representation.
  arXiv  Detail & Related papers  (2026-04-24T13:02:45Z)
• Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Models [7.704706624419061]
  Retrieval-Augmented Generation grounds Large Language Models (LLMs) to mitigate factual hallucinations.<n>We propose Tiny-Critic RAG, a parameter-efficient Small Language Model (SLM) via Low-Rank Adaptation (LoRA)<n>Tiny-Critic RAG achieves routing accuracy comparable to GPT-4o-mini while reducing latency by an order of magnitude.
  arXiv  Detail & Related papers  (2026-03-01T00:16:31Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Probing Model Signal-Awareness via Prediction-Preserving Input Minimization [67.62847721118142]
  We evaluate models' ability to capture the correct vulnerability signals to produce their predictions. We measure the signal awareness of models using a new metric we propose- Signal-aware Recall (SAR) The results show a sharp drop in the model's Recall from the high 90s to sub-60s with the new metric.
  arXiv  Detail & Related papers  (2020-11-25T20:05:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.