Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

• URL: http://arxiv.org/abs/2607.18253v1
• Date: Wed, 13 May 2026 20:29:09 GMT
• Title: Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
• Abstract summary: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost.<n>Current query routers are largely latency-agressive and do not consider the generation latency experienced by queries at model instances.<n>We propose a lightweight latency estimator that simulates batch processing in the serving framework and estimates the time-to-first-token of queries.
• Score: 14.56862716394492
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost. However, current query routers are largely latency-agnostic and do not consider the generation latency experienced by queries at model instances. In practice, latency is often controlled by load-balancing policies such as round-robin or join-the-shortest-queue, which do not account for model accuracy or inference cost. Incorporating query latency into routing is challenging as it depends not only on the query's prompt length, but also on the current prefill and decode workload at the model instance and the scheduling and batching policy of the serving framework. We design a lightweight latency estimator that simulates autoregressive token batch processing in the serving framework and estimates the time-to-first-token (TTFT) of queries. We incorporate this latency estimator into a latency-aware router that jointly optimizes latency, accuracy, and cost when assigning queries to model instances. Our experimental results indicate that this joint optimization yields up to 40% improvement in accuracy--cost utility while maintaining the same latencies as standard load-balancing approaches.

Related papers

• End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services [13.04369901641143]
  Large language model (LLM)-powered agentic AI services increasingly demand low-latency inference.<n>We develop a cross-slot inference model that captures transmission, prefill, redistribution-level decoding, and key-value cache evolution for each diverse request.<n>We propose the LYREO approach that transforms the long-term load-balancing constraint via Lyapunov optimization and employs reward with sequencebased return prediction.
  arXiv  Detail & Related papers  (2026-09-15T13:50:54Z)
• INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration [18.376319491516266]
  We introduce INFRAMIND, a framework that makes the entire multi-agent stack infrastructure-aware.<n>An infra-aware executor then observes per-model queue depths, cache utilization, and response latencies at each agent step to decide which model to call.<n>A budget-aware scheduler further reorders each model's queue so that urgent requests are served first.
  arXiv  Detail & Related papers  (2026-06-09T20:50:12Z)
• ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models [7.869130026927]
  ConsRoute is a lightweight, semantic-aware, and adaptive routing framework for large language models.<n>We show that ConsRoute achieves near-cloud performance (>=95%) while reducing end-to-end latency and inference cost by nearly 40%.
  arXiv  Detail & Related papers  (2026-03-22T13:54:12Z)
• RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents [91.0187958746262]
  RouteMoA is an efficient mixture-of-agents framework with dynamic routing.<n>It employs a lightweight scorer to perform initial screening by predicting coarse-grained performance from the query.<n>It refines these scores through lightweight self- and cross-assessment based on existing model outputs, providing posterior correction without additional inference.
  arXiv  Detail & Related papers  (2026-01-26T04:22:22Z)
• Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation [55.47971671635531]
  Large Language Models (LLMs) have shown remarkable performance on general Question Answering (QA)<n>Retrieval-Augmented Generation (RAG) addresses this limitation by enriching LLMs with external knowledge.<n>Existing systems primarily rely on unstructured documents, while largely overlooking relational databases.
  arXiv  Detail & Related papers  (2025-09-30T22:19:44Z)
• Intra-request branch orchestration for efficient LLM reasoning [52.68946975865865]
  Large Language Models (LLMs) increasingly rely on inference-time reasoning algorithms to improve accuracy on complex tasks.<n>Prior work has largely focused on reducing token usage, often at the expense of accuracy, while overlooking other latency factors.<n>We present DUCHESS, an LLM serving system that reduces cost and latency without sacrificing accuracy through intra-request branch orchestration guided by predictions.
  arXiv  Detail & Related papers  (2025-09-29T15:52:08Z)
• Latency and Token-Aware Test-Time Compute [3.573250939705335]
  Inference-time scaling can improve large language model (LLM) performance by generating multiple candidate responses and selecting among them.<n>We formulate inference-time scaling as a problem of dynamic compute allocation and method selection.<n>Our framework explicitly incorporates both token cost and wall-clock latency, the latter being critical for user experience and particularly for agentic models.
  arXiv  Detail & Related papers  (2025-09-11T21:35:19Z)
• MixLLM: Dynamic Routing in Mixed Large Language Models [57.309520357563215]
  Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency.<n>We develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment.
  arXiv  Detail & Related papers  (2025-02-09T02:26:15Z)
• Budget-aware Query Tuning: An AutoML Perspective [14.561951257365953]
  Modern database systems rely on cost-based querys to come up with good execution plans for input queries. We show that by varying the costunit values one can obtain query plans that significantly outperform the default query plans.
  arXiv  Detail & Related papers  (2024-03-29T20:19:36Z)
• Keep Your Friends Close: Leveraging Affinity Groups to Accelerate AI Inference Workflows [0.1874930567916036]
  We propose a novel affinity grouping mechanism that makes it easier for developers to express application-specific data access correlations.<n> Experiments confirm the limitations of standard techniques, while showing that the proposed mechanism is able to maintain significantly lower latency as workload and scale-out increase.
  arXiv  Detail & Related papers  (2023-11-30T16:02:04Z)
• Multi-mode Transformer Transducer with Stochastic Future Context [53.005638503544866]
  Multi-mode speech recognition models can process longer future context to achieve higher accuracy and when a latency budget is not flexible, the model can still achieve reliable accuracy. We show that a Multi-mode ASR model rivals, if not surpasses, a set of competitive streaming baselines trained with different latency budgets.
  arXiv  Detail & Related papers  (2021-06-17T18:42:11Z)
• Towards Streaming Perception [70.68520310095155]
  We present an approach that coherently integrates latency and accuracy into a single metric for real-time online perception. The key insight behind this metric is to jointly evaluate the output of the entire perception stack at every time instant. We focus on the illustrative tasks of object detection and instance segmentation in urban video streams, and contribute a novel dataset with high-quality and temporally-dense annotations.
  arXiv  Detail & Related papers  (2020-05-21T01:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.