Load Testing for Machine Learning Model Serving Systems at Scale

• URL: http://arxiv.org/abs/2606.22013v1
• Date: Sat, 20 Jun 2026 12:34:27 GMT
• Title: Load Testing for Machine Learning Model Serving Systems at Scale
• Abstract summary: Machine learning (ML) model serving has become a dominant consumer of GPU infrastructure, yet capacity planning in these systems remains largely ad hoc.<n>This paper presents sys, an industrial load testing framework for ML serving systems that systematically estimates serving capacity through an adaptive, feedback-driven search strategy.
• Score: 3.2375553784320394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) model serving has become a dominant consumer of GPU infrastructure, yet capacity planning in these systems remains largely ad hoc. Under-provisioning leads to service-level objective (SLO) violations and production incidents, while over-provisioning results in substantial resource waste. This paper presents \sys, an industrial load testing framework for ML serving systems that systematically estimates serving capacity through an adaptive, feedback-driven search strategy. The approach leverages real-time performance signals, incorporating dampening, spike tolerance, and convergence detection to efficiently identify maximum sustainable throughput under SLO constraints. We evaluate \sys through a longitudinal analysis of 14 industrial case studies spanning four ML architecture classes: recommendation, ranking, vision, and NLP. This study demonstrates that systematic load testing leads to substantial improvements in GPU resource efficiency and operational reliability. Prior to adopting \sys, a significant fraction of model launches were under-provisioned, resulting in recurring incidents; these issues were substantially reduced after deployment. Our results show that ML-specific design decisions are critical to accurate capacity estimation: workload calibration using recorded traffic reduces estimation error from approximately 30\% to 2--6\%, while proper warmup handling yields a 22.2\% improvement in accuracy. Further analysis reveals key factors influencing prediction error, including model size and co-location effects. This paper distills six lessons and derive architectural guidelines for ML load testing, offering actionable insights for building reliable and efficient ML serving systems.

Related papers

• ScaleSense: Cost-Intelligent Scaling Framework via Learned Resource Estimation in Alibaba AnalyticDB [59.708423132520515]
  Cloud-native serverless data warehouses achieve fine-grained elasticity by decoupling storage from compute.<n>Yet determining the optimal resource allocation for highly heterogeneous ad-hoc queries remains a formidable industrial challenge.<n>We propose ScaleSense, a proactive, query-level resource scaling framework.
  arXiv  Detail & Related papers  (2026-08-08T06:10:35Z)
• On Predicting the Post-training Potential of Pre-trained LLMs [60.07459271263409]
  We introduce a new task of predicting post-training potential - forecasting a base model's performance before post-training.<n>We propose RuDE, a unified framework that bypasses the generation gap of base models by leveraging response discrimination.<n>Experiments demonstrate a correlation greater than 90% with post-training performance.
  arXiv  Detail & Related papers  (2026-05-12T11:33:49Z)
• Quantized Inference for OneRec-V2 [28.8042155916846]
  We develop a FP8 post training quantization framework and integrate it into an optimized inference infrastructure.<n>The proposed joint optimization achieves a 49% reduction in end-to-end inference latency and a 92% increase in throughput.<n>These results suggest that as recommender systems evolve toward the paradigms of large language models, algorithm-level and system-level optimization techniques can be effectively adapted to large-scale recommendation workloads.
  arXiv  Detail & Related papers  (2026-03-12T03:13:08Z)
• Robustness Evaluation of Machine Learning Models for Fault Classification and Localization In Power System Protection [5.539105299550525]
  This work introduces a unified framework for evaluating the robustness of machine learning models in power system protection.<n>High-fidelity EMT simulations are used to model realistic degradation scenarios, including sensor outages, reduced sampling rates, and transient communication losses.<n>Results show that FC remains highly stable under most degradation types but drops by about 13% under single-phase loss, while FL is more sensitive overall, with voltage loss increasing localization error by over 150%.
  arXiv  Detail & Related papers  (2025-12-17T12:38:53Z)
• Enhancing reliability in AI inference services: An empirical study on real production incidents [6.549475714716768]
  We present one of the first provider-internal, practice-based analysis of large language model (LLM) inference incidents.<n>We developed a taxonomy and methodology grounded in a year of operational experience, validating it on 156 high-severity incidents.<n>This study demonstrates how systematic, empirically grounded analysis of inference operations can drive more reliable and cost-efficient LLM serving at scale.
  arXiv  Detail & Related papers  (2025-10-17T23:16:29Z)
• Redundancy Analysis and Mitigation for Machine Learning-Based Process Monitoring of Additive Manufacturing [3.414636048610798]
  Excessive redundancy leads to increased equipment costs, compromised model performance, and high computational requirements.<n>This paper defines redundancy in ML-based AM process monitoring and categorizes it into sample-level, feature-level, and model-level redundancy.<n>A comprehensive multi-level redundancy mitigation (MLRM) framework is proposed, incorporating advanced methods such as data registration, downscaling, cross-modality knowledge transfer, and model pruning.
  arXiv  Detail & Related papers  (2025-04-30T05:04:53Z)
• Scaling Laws for Predicting Downstream Performance in LLMs [75.28559015477137]
  This work focuses on the pre-training loss as a more computation-efficient metric for performance estimation.<n>We present FLP-M, a fundamental approach for performance prediction that addresses the practical need to integrate datasets from multiple sources during pre-training.
  arXiv  Detail & Related papers  (2024-10-11T04:57:48Z)
• Applying Fine-Tuned LLMs for Reducing Data Needs in Load Profile Analysis [9.679453060210978]
  This paper presents a novel method for utilizing fine-tuned Large Language Models (LLMs) to minimize data requirements in load profile analysis. A two-stage fine-tuning strategy is proposed to adapt a pre-trained LLM for missing data restoration tasks. We demonstrate the effectiveness of the fine-tuned model in accurately restoring missing data, achieving comparable performance to state-of-the-art models such as BERT-PIN.
  arXiv  Detail & Related papers  (2024-06-02T23:18:11Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• Weak Supervision Performance Evaluation via Partial Identification [46.73061437177238]
  Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels. We present a novel method to address this challenge by framing model evaluation as a partial identification problem. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques.
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• Federated Learning with Unreliable Clients: Performance Analysis and Mechanism Design [76.29738151117583]
  Federated Learning (FL) has become a promising tool for training effective machine learning models among distributed clients. However, low quality models could be uploaded to the aggregator server by unreliable clients, leading to a degradation or even a collapse of training. We model these unreliable behaviors of clients and propose a defensive mechanism to mitigate such a security risk.
  arXiv  Detail & Related papers  (2021-05-10T08:02:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.