ML Inference Scheduling with Predictable Latency

• URL: http://arxiv.org/abs/2512.18725v2
• Date: Wed, 24 Dec 2025 04:31:46 GMT
• Title: ML Inference Scheduling with Predictable Latency
• Abstract summary: We evaluate the potential limitations of existing interference prediction approaches.<n>We show that coarse-grained methods can lead to noticeable deviations in prediction accuracy and that static models degrade considerably under changing workloads.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) inference serving systems can schedule requests to improve GPU utilization and to meet service level objectives (SLOs) or deadlines. However, improving GPU utilization may compromise latency-sensitive scheduling, as concurrent tasks contend for GPU resources and thereby introduce interference. Given that interference effects introduce unpredictability in scheduling, neglecting them may compromise SLO or deadline satisfaction. Nevertheless, existing interference prediction approaches remain limited in several respects, which may restrict their usefulness for scheduling. First, they are often coarse-grained, which ignores runtime co-location dynamics and thus restricts their accuracy in interference prediction. Second, they tend to use a static prediction model, which may not effectively cope with different workload characteristics. In this paper, we evaluate the potential limitations of existing interference prediction approaches, finding that coarse-grained methods can lead to noticeable deviations in prediction accuracy and that static models degrade considerably under changing workloads.

Related papers

• SegDiff: Segmented Trajectory Diffusion for Consistent and Adaptive Robot Manipulation [65.81476183180527]
  We introduce SegDiff, a closed-loop visuomotor policy that integrates the strengths of both paradigms.<n>SegDiff decomposes demonstrations into motion segments between keyposes and learns to predict the continuous trajectory from the current state to the next keypose.<n>SegDiff demonstrates significant performance gains over existing approaches across various simulated and real-world scenarios.
  arXiv  Detail & Related papers  (2026-07-13T02:48:03Z)
• LAMP: Lane-Aligned Motion Primitives for Feasible Trajectory Prediction [24.838775533427977]
  We propose LAMP (Lane-Aligned Motion Primitives), a topology-aware forecasting framework that anchors multimodal prediction to structured motion primitives.<n>Experiments on the Argoverse 2 dataset demonstrate that LAMP achieves prediction accuracy comparable to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-25T06:52:17Z)
• Strait: Perceiving Priority and Interference in ML Inference Serving [0.0]
  We present emphStrait, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization.<n>Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on low-priority tasks.
  arXiv  Detail & Related papers  (2026-04-30T17:55:28Z)
• Adaptively Robust LLM Inference Optimization under Prediction Uncertainty [9.541681114575812]
  We study the problem of optimizing Large Language Model (LLM) inference scheduling to minimize total latency.<n>A key challenge in LLM inference scheduling is that while the prompt length is known upon arrival, the output length, which critically impacts memory usage and processing time, is unknown.<n>We propose algorithms that leverage machine learning to predict output lengths, assuming the prediction provides an interval classification (min-max range) for each request.
  arXiv  Detail & Related papers  (2025-08-20T08:55:26Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networks [56.37880529653111]
  The demand for large computation model (LAIM) services is driving a paradigm shift from traditional cloud-based inference to edge-based inference for low-latency, privacy-preserving applications.<n>In this paper, we investigate the LAIM-inference scheme, where a pre-trained LAIM is pruned and partitioned into on-device and on-server sub-models for deployment.
  arXiv  Detail & Related papers  (2025-05-14T08:18:55Z)
• Network-Aware Scheduling for Remote Gate Execution in Quantum Data Centers [8.528068737844364]
  We evaluate two entanglement scheduling strategies-static and dynamic-and analyze their performance.<n>We show that dynamic scheduling consistently outperforms static scheduling in scenarios with high entanglement parallelism.
  arXiv  Detail & Related papers  (2025-04-28T18:22:22Z)
• Interference-Aware Edge Runtime Prediction with Conformal Matrix Completion [10.776912158818437]
  Accurately estimating workload runtime is a longstanding goal in computer systems.<n>We develop a matrix factorization-inspired method that generates accurate interference-aware predictions with tight provably-guaranteed uncertainty bounds.<n>We validate our method on a novel WebAssembly runtime dataset collected from 24 unique devices, achieving a prediction error of 5.2% -- 2x better than a naive application of existing methods.
  arXiv  Detail & Related papers  (2025-03-09T03:41:32Z)
• Error-quantified Conformal Inference for Time Series [55.11926160774831]
  Uncertainty quantification in time series prediction is challenging due to the temporal dependence and distribution shift on sequential data.<n>We propose itError-quantified Conformal Inference (ECI) by smoothing the quantile loss function.<n>ECI can achieve valid miscoverage control and output tighter prediction sets than other baselines.
  arXiv  Detail & Related papers  (2025-02-02T15:02:36Z)
• Physics-guided Active Sample Reweighting for Urban Flow Prediction [75.24539704456791]
  Urban flow prediction is a nuanced-temporal modeling that estimates the throughput of transportation services like buses, taxis and ride-driven models. Some recent prediction solutions bring remedies with the notion of physics-guided machine learning (PGML) We develop a atized physics-guided network (PN), and propose a data-aware framework Physics-guided Active Sample Reweighting (P-GASR)
  arXiv  Detail & Related papers  (2024-07-18T15:44:23Z)
• Addressing Prediction Delays in Time Series Forecasting: A Continuous GRU Approach with Derivative Regularization [27.047129496488292]
  Predictions succeeding ground-truth observations are not practically meaningful although their MSEs can be low. We introduce a continuous-time gated recurrent unit (GRU) based on the neural ordinary differential equation (NODE) which can supervise explicit time-derivatives. Our method outperforms in metrics such as MSE, Dynamic Time Warping (DTW) and Time Distortion Index (TDI)
  arXiv  Detail & Related papers  (2024-06-29T05:36:04Z)
• Loss Shaping Constraints for Long-Term Time Series Forecasting [79.3533114027664]
  We present a Constrained Learning approach for long-term time series forecasting that respects a user-defined upper bound on the loss at each time-step. We propose a practical Primal-Dual algorithm to tackle it, and aims to demonstrate that it exhibits competitive average performance in time series benchmarks, while shaping the errors across the predicted window.
  arXiv  Detail & Related papers  (2024-02-14T18:20:44Z)
• Performative Time-Series Forecasting [64.03865043422597]
  We formalize performative time-series forecasting (PeTS) from a machine-learning perspective.<n>We propose a novel approach, Feature Performative-Shifting (FPS), which leverages the concept of delayed response to anticipate distribution shifts.<n>We conduct comprehensive experiments using multiple time-series models on COVID-19 and traffic forecasting tasks.
  arXiv  Detail & Related papers  (2023-10-09T18:34:29Z)
• Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction [60.60223171143206]
  Trajectory prediction is a crucial undertaking in understanding entity movement or human behavior from observed sequences. Current methods often assume that the observed sequences are complete while ignoring the potential for missing values. This paper presents a unified framework, the Graph-based Conditional Variational Recurrent Neural Network (GC-VRNN), which can perform trajectory imputation and prediction simultaneously.
  arXiv  Detail & Related papers  (2023-03-28T14:27:27Z)
• Data-driven method for real-time prediction and uncertainty quantification of fatigue failure under stochastic loading using artificial neural networks and Gaussian process regression [0.0]
  Methods for early failure prediction are essential for engineering, military, and civil applications. Uncertainty (UQ) is of major importance for real-time decision-making purposes.
  arXiv  Detail & Related papers  (2021-03-11T13:57:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.