Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live

• URL: http://arxiv.org/abs/2511.02230v1
• Date: Tue, 04 Nov 2025 03:43:05 GMT
• Title: Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
• Abstract summary: Continuum is a serving system to optimize job completion time for multi-turn agent workloads.<n>By predicting tool call durations in agentic, Continuum selectively pins the KV cache in GPU memory with a time-to-live value based on total turn number.<n>Our evaluation on real-world agentic workloads with Llama-3.1 8B/70B models shows that Continuum significantly improves the average job completion times.
• Score: 30.099614426825834
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic LLM applications interleave LLM generation requests with tool calls. These tool calls break the continuity of the workflow by creating pauses between LLM requests, bringing many challenges for the serving system, especially under multi-turn scenarios. Each pause potentially causes KV cache eviction and extra waiting time before entering the continuous batch for the following LLM request. Since these pauses happen for each call, this problem becomes increasingly severe as turn number grow for agentic programs. Previous works either fail to incorporate information from the tool call, evicting KV cache that leads to repetitive prefill or loading, or ignore the continuity of a multi-turn program, creating waiting time between turns that increases per-request latency. We present Continuum, a serving system to optimize job completion time for multi-turn agent workloads by combining tool-aware KV cache timeout with program-level scheduling. By predicting tool call durations in agentic workflows, Continuum selectively pins the KV cache in GPU memory with a time-to-live value based on total turn number. When combined with program-level first-come-first-serve, Continuum prevents scheduling bubbles, preserves multi-turn continuity, and optimizes for throughput for complex agentic workflows. By modeling the variability of tool call and agent program continuity, Continuum outperforms state-of-the-art baselines. Our evaluation on real-world agentic workloads (SWE-Bench and BFCL) with Llama-3.1 8B/70B models shows that Continuum significantly improves the average job completion times, and remains performant across different hardware setups and DRAM offloading schemes. Preview code is available at: https://github.com/Hanchenli/vllm-continuum

Related papers

• Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows [25.368121854431035]
  tail-risk-aware turn release scheduling method that jointly decides which ready turn to release next.<n>We present a tail-risk-aware turn release scheduling method that jointly decides which ready turn to release next.
  arXiv  Detail & Related papers  (2026-09-10T01:35:10Z)
• FRAMEWORKERS: A Dynamic Multi-Agent Framework for AI-Generated Video Production [55.71860113185132]
  We introduce FRAMEWORKERS, a task-centric and workspace-grounded multi-agent framework for open-ended video production.<n>A central Director formulates video creation as dynamic task management, continuously editing a Task Stack to determine which subtask to execute next and which sub-agent to invoke.<n>An Assistant serves as the execution layer, grounding each selected task in a shared Workspace, retrieving the required assets and context, invoking the assigned sub-agent, and persisting the resulting artifacts.<n>Experiments show that FRAMEWORKERS outperforms strong planners in routing accuracy, recovers reliably from runtime failures, generalizes to unseen sub
  arXiv  Detail & Related papers  (2026-08-30T14:28:50Z)
• PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents [79.28372522980153]
  PeakBench is a benchmark of executable multi-tool dependency annotations and measured resource profiles.<n>We show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints.<n>PeakBench is a useful testbed for diagnosing resource-aware agent behavior.
  arXiv  Detail & Related papers  (2026-08-25T12:56:08Z)
• PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs [0.8460698440162889]
  We present PersistentKV, a block-table decode attention engine and page-aware scheduling study.<n>Per PersistentKV maps work by KV-head group, executes directly over native page tables, and adds a compact workqueue schedule.<n>We show that five held-out seeds improve mean wall decode-token throughput by 1.04x to 1.08x on bimodal, uniform, and Zipf-like workloads.
  arXiv  Detail & Related papers  (2026-06-25T06:56:43Z)
• AGENTSERVESIM: A Hardware-aware Simulator for Multi-Turn LLM Agent Serving [16.735743806437487]
  We present AGENTSERVESIM, a hardware-aware simulator for multi-turn LLM agent serving.<n>We show that AGENTSERVESIM reproduces real-system behavior within 6% error across key performance metrics while running entirely on commodity CPUs.
  arXiv  Detail & Related papers  (2026-06-08T15:20:23Z)
• Harnessing Generalist Agents for Contextualized Time Series [67.28481985993467]
  Time series are often embedded in rich contexts that are essential for holistic modeling.<n>In this work, we introduce TimeClaw, an agentic harness framework for time series.<n>TimeClaw integrates executable temporal tools for grounded and auditable analysis, experience-driven capability evolution for creating reusable analytical routines, and episodic multimodal memory for retrieving relevant reasoning traces.
  arXiv  Detail & Related papers  (2026-06-03T20:20:34Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents [2.1447191409339244]
  TVCACHE is a stateful tool-value cache for LLM agent post-training.<n>Hit rates of up to 70% and reduces median tool call execution time by up to 6.9X.
  arXiv  Detail & Related papers  (2026-02-11T16:13:44Z)
• PLA-Serve: A Prefill-Length-Aware LLM Serving System [33.313531352453346]
  PLA-Serve identifies and disaggregates requests with different prompt lengths to reduce TTFT latency.<n>We observe that prompt-length variations lead to distinct bottlenecks, motivating an adaptive scheduling strategy.<n> PLA-Serve reduces prefill latency by over 30% compared to vanilla SG under prefill**-Lang**decode disaggregation.
  arXiv  Detail & Related papers  (2026-01-04T18:14:24Z)
• DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing [15.376910065679994]
  DuetServe is a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU.<n>DuetServe improves total throughput by up to 1.3x while maintaining low generation latency compared to state-of-the-art frameworks.
  arXiv  Detail & Related papers  (2025-11-06T20:18:34Z)
• Semantic-Aware Scheduling for GPU Clusters with Large Language Models [60.14838697778884]
  We propose SchedMate, a framework that bridges the semantic gap between schedulers and jobs they manage.<n>SchedMate extracts deep insights from overlooked, unstructured data sources: source code, runtime logs, and historical jobs.<n>We show SchedMate reduces average job completion times by up to 1.91x, substantially enhancing the scheduling performance.
  arXiv  Detail & Related papers  (2025-10-02T02:01:02Z)
• Real-Time Execution of Action Chunking Flow Policies [49.1574468325115]
  This paper presents a novel inference-time algorithm that enables asynchronous execution of action interacting systems.<n>It is applicable to any diffusion- or VLA-based systems executing out of the box with no re-training.<n>Results show that RTC is fast, performant, and uniquely robust to inference manipulation.
  arXiv  Detail & Related papers  (2025-06-09T01:01:59Z)
• Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference [4.7730970530715835]
  Large Language Models have revolutionized natural language processing, yet serving them efficiently in data centers remains challenging.<n>We introduce QLLM, a novel inference system designed for Mixture of Experts (MoE) models.<n>QLLM enables expert-level preemption, deferring BE job execution while minimizing LS time-to-first-token (TTFT)
  arXiv  Detail & Related papers  (2025-03-12T11:56:01Z)
• Autellix: An Efficient Serving Engine for LLM Agents as General Programs [59.673243129044465]
  Large language model (LLM) applications are evolving beyond simple chatbots into dynamic, general-purpose agentic programs.<n>Existing LLM serving systems ignore dependencies between programs and calls, missing significant opportunities for optimization.<n>We introduce Autellix, an LLM serving system that treats programs as first-class citizens to minimize their end-to-end latencies.
  arXiv  Detail & Related papers  (2025-02-19T18:59:30Z)
• Multi-Bin Batching for Increasing LLM Inference Throughput [19.652542432683234]
  Large language models (LL) grow in popularity improving the efficiency of their systems.<n> requests is a critical step in scheduling jobs on servers.<n> requests often have varying generation lengths, causing resource underutilization.<n>We formalize this problem from a queueing-theoretic perspective, and aim to design a throughput control policy.
  arXiv  Detail & Related papers  (2024-12-03T03:16:12Z)
• RelayAttention for Efficient Large Language Model Serving with Long System Prompts [59.50256661158862]
  This paper aims to improve the efficiency of LLM services that involve long system prompts. handling these system prompts requires heavily redundant memory accesses in existing causal attention algorithms. We propose RelayAttention, an attention algorithm that allows reading hidden states from DRAM exactly once for a batch of input tokens.
  arXiv  Detail & Related papers  (2024-02-22T18:58:28Z)
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory [93.20588235940453]
  In this paper, we introduce a training-free memory-based method, InfLLM. InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention. Even when the sequence length is scaled to $1,024$K, InfLLM still effectively captures long-distance dependencies.
  arXiv  Detail & Related papers  (2024-02-07T06:50:42Z)
• Fast Distributed Inference Serving for Large Language Models [12.703624317418237]
  We present FastServe, a distributed inference serving system for large language models (LLMs) FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. We build a system prototype of FastServe and experimental results show that compared to the state-of-the-art solution vLLM, FastServe improves the throughput by up to 31.4x and 17.9x under the same average and tail latency requirements, respectively.
  arXiv  Detail & Related papers  (2023-05-10T06:17:50Z)
• Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints [52.58352707495122]
  We present a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination. We validate our results over a wide range of simulations on two distinct domains: multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
  arXiv  Detail & Related papers  (2020-05-27T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.