MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

• URL: http://arxiv.org/abs/2609.15359v1
• Date: Mon, 14 Sep 2026 10:43:42 GMT
• Title: MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
• Abstract summary: Large language model (LLM) applications on personal devices impose massive, bursty workloads on cloud serving infrastructure.<n>We propose MAPS, a Memory-Aware Predictive Scheduling framework tailored for disaggregated LLM serving.<n>We show that MAPS significantly outperforms three state-of-the-art systems, reducing average end-to-end latency by 42.6 and tail latency by up to 84.8.
• Score: 22.18337792694688
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The surge of large language model (LLM) applications on personal devices imposes massive, bursty workloads on cloud serving infrastructure. While prefill-decode disaggregation improves throughput and scalability, memory-bound decode instances often suffer from persistent load imbalance, as output lengths are unknown when requests arrive at the cloud. To address this, we propose MAPS, a Memory-Aware Predictive Scheduling framework tailored for disaggregated LLM serving. MAPS performs device-assisted speculative output length prediction overlapped with cloud-side prefilling, incurring negligible latency overhead. To handle generation uncertainty, MAPS applies uncertainty-aware calibration to derive output-length upper bounds with target coverage, enabling safe scheduling decisions. Building on these bounds, MAPS employs a hierarchical global-local scheduling strategy to mitigate inter-decoder queue buildup and intra-decoder head-of-line blocking. Extensive experiments on two real-world workloads and two LLMs show that MAPS significantly outperforms three state-of-the-art systems, reducing average end-to-end latency by 42.6 and tail latency by up to 84.8.

Related papers

• End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services [13.04369901641143]
  Large language model (LLM)-powered agentic AI services increasingly demand low-latency inference.<n>We develop a cross-slot inference model that captures transmission, prefill, redistribution-level decoding, and key-value cache evolution for each diverse request.<n>We propose the LYREO approach that transforms the long-term load-balancing constraint via Lyapunov optimization and employs reward with sequencebased return prediction.
  arXiv  Detail & Related papers  (2026-09-15T13:50:54Z)
• DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks [54.32016216994156]
  We introduce the Causal Latent World Model (CLWM), which employs DINOv3 features as generative targets to disentangle interaction semantics from visual noise.<n>To overcome memory scaling, CLWM features a Dual-State Test-Time Training (TTT) Memory that guarantees a strict $mathcalO(1)$ footprint for long-horizon tasks.<n>We present EmbodiChain, an online framework that establishes the Efficiency Law by injecting an infinite flow of physics-grounded trajectories during training.
  arXiv  Detail & Related papers  (2026-04-13T03:19:36Z)
• SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning [104.01865949020304]
  Agentic multimodal large language models (MLLMs) achieve remarkable reasoning capabilities through iterative visual tool invocation.<n>However, the cascaded perception, reasoning, and tool-calling loops introduce significant sequential overhead.<n>This overhead, termed agentic depth, incurs prohibitive latency and seriously limits system-level throughput.<n>We propose SpecEyes, an agentic-level speculative acceleration framework that breaks this sequential bottleneck.
  arXiv  Detail & Related papers  (2026-03-24T17:45:47Z)
• HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network [50.33808558714122]
  Large language models' (LLMs) inference at the edge can facilitate prompt service responsiveness while protecting user privacy.<n>We propose HALO, a novel framework that can boost the distributed LLM inference in lossy edge network.<n> Experimental results from a Raspberry Pi cluster demonstrate that HALO achieves a 3.41x end-to-end speedup for LLaMA-series LLMs under unreliable network conditions.
  arXiv  Detail & Related papers  (2026-01-16T07:37:23Z)
• WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching [9.305759319664483]
  A growing number of inference requests are initiated from edge devices and computed on centralized GPU clusters.<n>We propose WISP, an efficient and SLO-aware distributed LLM inference system that consists of an intelligent speculation controller, a verification time estimator, and a verification batch scheduler.<n>Extensive numerical results show that WISP improves system capacity by up to 2.1x and 4.1x, and increases system goodput by up to 1.94x and 3.7x, compared to centralized serving and SLED, respectively.
  arXiv  Detail & Related papers  (2026-01-15T16:46:01Z)
• PLA-Serve: A Prefill-Length-Aware LLM Serving System [33.313531352453346]
  PLA-Serve identifies and disaggregates requests with different prompt lengths to reduce TTFT latency.<n>We observe that prompt-length variations lead to distinct bottlenecks, motivating an adaptive scheduling strategy.<n> PLA-Serve reduces prefill latency by over 30% compared to vanilla SG under prefill**-Lang**decode disaggregation.
  arXiv  Detail & Related papers  (2026-01-04T18:14:24Z)
• Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms [33.64527903547734]
  Harli improves the finetune throughput by 46.2% on average (up to over state-of-the-art serving systems)<n>Harli improves the finetune throughput by 46.2% on average (up to over state-of-the-art serving systems)
  arXiv  Detail & Related papers  (2025-11-13T05:58:52Z)
• Semantic-Aware Scheduling for GPU Clusters with Large Language Models [60.14838697778884]
  We propose SchedMate, a framework that bridges the semantic gap between schedulers and jobs they manage.<n>SchedMate extracts deep insights from overlooked, unstructured data sources: source code, runtime logs, and historical jobs.<n>We show SchedMate reduces average job completion times by up to 1.91x, substantially enhancing the scheduling performance.
  arXiv  Detail & Related papers  (2025-10-02T02:01:02Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• Progressive Mixed-Precision Decoding for Efficient LLM Inference [49.05448842542558]
  We introduce Progressive Mixed-Precision Decoding (PMPD) to address the memory-boundedness of decoding.<n>PMPD achieves 1.4$-$12.2$times$ speedup in matrix-vector multiplications over fp16 models.<n>Our approach delivers a throughput gain of 3.8$-$8.0$times$ over fp16 models and up to 1.54$times$ over uniform quantization approaches.
  arXiv  Detail & Related papers  (2024-10-17T11:46:33Z)
• ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving [61.35068981176018]
  ConServe is a large language model (LLM) serving system that achieves high throughput and strong online latency guarantees.<n>We show that ConServe delivers an average of 2.2$times$ higher throughput and reduces online serving tail latency by 2.9$times$ on average compared to state-of-the-art systems.
  arXiv  Detail & Related papers  (2024-10-02T04:12:13Z)
• Fast Distributed Inference Serving for Large Language Models [12.703624317418237]
  We present FastServe, a distributed inference serving system for large language models (LLMs) FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. We build a system prototype of FastServe and experimental results show that compared to the state-of-the-art solution vLLM, FastServe improves the throughput by up to 31.4x and 17.9x under the same average and tail latency requirements, respectively.
  arXiv  Detail & Related papers  (2023-05-10T06:17:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.