Scalable AI Inference: Performance Analysis and Optimization of AI Model Serving

• URL: http://arxiv.org/abs/2604.20420v1
• Date: Wed, 22 Apr 2026 10:39:14 GMT
• Title: Scalable AI Inference: Performance Analysis and Optimization of AI Model Serving
• Abstract summary: This study investigates the performance and optimization of a BentoML-based AI inference system for scalable model serving developed in collaboration with graphworks.ai.<n>The study examines how latency and throughput scale under varying workloads, how optimizations at the runtime, service, and deployment levels affect response time, and how deployment in a single-node K3s cluster influences resilience during disruptions.
• Score: 0.9167082845109437
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI research often emphasizes model design and algorithmic performance, while deployment and inference remain comparatively underexplored despite being critical for real-world use. This study addresses that gap by investigating the performance and optimization of a BentoML-based AI inference system for scalable model serving developed in collaboration with graphworks.ai. The evaluation first establishes baseline performance under three realistic workload scenarios. To ensure a fair and reproducible assessment, a pre-trained RoBERTa sentiment analysis model is used throughout the experiments. The system is subjected to traffic patterns following gamma and exponential distributions in order to emulate real-world usage conditions, including steady, bursty, and high-intensity workloads. Key performance metrics, such as latency percentiles and throughput, are collected and analyzed to identify bottlenecks in the inference pipeline. Based on the baseline results, optimization strategies are introduced at multiple levels of the serving stack to improve efficiency and scalability. The optimized system is then reevaluated under the same workload conditions, and the results are compared with the baseline using statistical analysis to quantify the impact of the applied improvements. The findings demonstrate practical strategies for achieving efficient and scalable AI inference with BentoML. The study examines how latency and throughput scale under varying workloads, how optimizations at the runtime, service, and deployment levels affect response time, and how deployment in a single-node K3s cluster influences resilience during disruptions.

Related papers

• Implicit Neural Representation-Based Continuous Single Image Super Resolution: An Empirical Study [50.15623093332659]
  Implicit neural representation (INR) has become the standard approach for arbitrary-scale image super-resolution (ASSR)<n>We compare existing techniques across diverse settings and present aggregated performance results on multiple image quality metrics.<n>We examine a new loss function that penalizes intensity variations while preserving edges, textures, and finer details during training.
  arXiv  Detail & Related papers  (2026-01-25T07:09:20Z)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs [53.749193998004166]
  Curriculum learning plays a crucial role in enhancing the training efficiency of large language models.<n>We propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead.
  arXiv  Detail & Related papers  (2025-10-01T15:41:27Z)
• XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning [26.063477716451512]
  We introduce XQC: a well-motivated, sample-efficient deep actor-critic algorithm built upon soft actor-critic.<n>We achieve state-of-the-art sample efficiency across 55 proprioception and 15 vision-based continuous control tasks.
  arXiv  Detail & Related papers  (2025-09-29T17:58:53Z)
• Tuning for Trustworthiness -- Balancing Performance and Explanation Consistency in Neural Network Optimization [49.567092222782435]
  We introduce the novel concept of XAI consistency, defined as the agreement among different feature attribution methods.<n>We create a multi-objective optimization framework that balances predictive performance with explanation.<n>Our research provides a foundation for future investigations into whether models from the trade-off zone-balancing performance loss and XAI consistency-exhibit greater robustness.
  arXiv  Detail & Related papers  (2025-05-12T13:19:14Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining [55.262510814326035]
  Existing reweighting strategies primarily focus on group-level data importance.<n>We introduce novel algorithms for dynamic, instance-level data reweighting.<n>Our framework allows us to devise reweighting strategies deprioritizing redundant or uninformative data.
  arXiv  Detail & Related papers  (2025-02-10T17:57:15Z)
• Asymptotically Fair Participation in Machine Learning Models: an Optimal Control Perspective [21.962258178900065]
  The performance of state-of-the-art machine learning models often deteriorates when testing on demographics that are under-represented in the training dataset. We aim to address the problem of achieving skewedally fair participation via optimal control formulation. We apply an efficient implementation of Pontryagin's maximum principle to estimate the optimal control solution.
  arXiv  Detail & Related papers  (2023-11-16T22:28:38Z)
• Data Augmentation through Expert-guided Symmetry Detection to Improve Performance in Offline Reinforcement Learning [0.0]
  offline estimation of the dynamical model of a Markov Decision Process (MDP) is a non-trivial task. Recent works showed that an expert-guided pipeline relying on Density Estimation methods effectively detects this structure in deterministic environments. We show that the former results lead to a performance improvement when solving the learned MDP and then applying the optimized policy in the real environment.
  arXiv  Detail & Related papers  (2021-12-18T14:32:32Z)
• On Effective Scheduling of Model-based Reinforcement Learning [53.027698625496015]
  We propose a framework named AutoMBPO to automatically schedule the real data ratio. In this paper, we first theoretically analyze the role of real data in policy training, which suggests that gradually increasing the ratio of real data yields better performance.
  arXiv  Detail & Related papers  (2021-11-16T15:24:59Z)
• IOHanalyzer: Detailed Performance Analyses for Iterative Optimization Heuristics [3.967483941966979]
  IOHanalyzer is a new user-friendly tool for the analysis, comparison, and visualization of performance data of IOHs. IOHanalyzer provides detailed statistics about fixed-target running times and about fixed-budget performance of the benchmarked algorithms. IOHanalyzer can directly process performance data from the main benchmarking platforms.
  arXiv  Detail & Related papers  (2020-07-08T08:20:19Z)
• Dynamic Federated Learning [57.14673504239551]
  Federated learning has emerged as an umbrella term for centralized coordination strategies in multi-agent environments. We consider a federated learning model where at every iteration, a random subset of available agents perform local updates based on their data. Under a non-stationary random walk model on the true minimizer for the aggregate optimization problem, we establish that the performance of the architecture is determined by three factors, namely, the data variability at each agent, the model variability across all agents, and a tracking term that is inversely proportional to the learning rate of the algorithm.
  arXiv  Detail & Related papers  (2020-02-20T15:00:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.