Scalability Optimization in Cloud-Based AI Inference Services: Strategies for Real-Time Load Balancing and Automated Scaling

• URL: http://arxiv.org/abs/2504.15296v1
• Date: Wed, 16 Apr 2025 04:00:04 GMT
• Title: Scalability Optimization in Cloud-Based AI Inference Services: Strategies for Real-Time Load Balancing and Automated Scaling
• Authors: Yihong Jin, Ze Yang,
• Abstract summary: This study proposes a comprehensive scalability optimization framework for cloud AI inference services.<n>The proposed model is a hybrid approach that combines reinforcement learning for adaptive load distribution and deep neural networks for accurate demand forecasting.<n> Experimental results demonstrate that the proposed model enhances load balancing efficiency by 35 and reduces response delay by 28.
• Score: 1.3689475854650441
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: The rapid expansion of AI inference services in the cloud necessitates a robust scalability solution to manage dynamic workloads and maintain high performance. This study proposes a comprehensive scalability optimization framework for cloud AI inference services, focusing on real-time load balancing and autoscaling strategies. The proposed model is a hybrid approach that combines reinforcement learning for adaptive load distribution and deep neural networks for accurate demand forecasting. This multi-layered approach enables the system to anticipate workload fluctuations and proactively adjust resources, ensuring maximum resource utilisation and minimising latency. Furthermore, the incorporation of a decentralised decision-making process within the model serves to enhance fault tolerance and reduce response time in scaling operations. Experimental results demonstrate that the proposed model enhances load balancing efficiency by 35\ and reduces response delay by 28\, thereby exhibiting a substantial optimization effect in comparison with conventional scalability solutions.

Related papers

• Efficient Federated Split Learning for Large Language Models over Communication Networks [14.461758448289908]
  Fine-tuning pre-trained large language models (LLM) in a distributed manner poses significant challenges on resource-constrained edge devices. We propose FedsLLM, a novel framework that integrates split federated learning with parameter-efficient fine-tuning techniques.
  arXiv  Detail & Related papers  (2025-04-20T16:16:54Z)
• Global-Decision-Focused Neural ODEs for Proactive Grid Resilience Management [50.34345101758248]
  We propose predict-all-then-optimize-globally (PATOG), a framework that integrates outage prediction with globally optimized interventions.<n>Our approach ensures spatially and temporally coherent decision-making, improving both predictive accuracy and operational efficiency.<n>Experiments on synthetic and real-world datasets demonstrate significant improvements in outage prediction consistency and grid resilience.
  arXiv  Detail & Related papers  (2025-02-25T16:15:35Z)
• Reinforcement Learning-Based Adaptive Load Balancing for Dynamic Cloud Environments [0.0]
  We propose a novel adaptive load balancing framework using Reinforcement Learning (RL) to address these challenges. Our framework is designed to dynamically reallocate tasks to minimize latency and ensure balanced resource usage across servers. Experimental results show that the proposed RL-based load balancer outperforms traditional algorithms in terms of response time, resource utilization, and adaptability to changing workloads.
  arXiv  Detail & Related papers  (2024-09-07T19:40:48Z)
• Optimization Hyper-parameter Laws for Large Language Models [52.49860340549727]
  We present Opt-Laws, a framework that captures the relationship between hyper- parameters and training outcomes.<n>Our validation across diverse model sizes and data scales demonstrates Opt-Laws' ability to accurately predict training loss.<n>This approach significantly reduces computational costs while enhancing overall model performance.
  arXiv  Detail & Related papers  (2024-09-07T09:37:19Z)
• Neural Horizon Model Predictive Control -- Increasing Computational Efficiency with Neural Networks [0.0]
  We propose a proposed machine-learning supported approach to model predictive control. We propose approximating part of the problem horizon, while maintaining safety guarantees. The proposed MPC scheme can be applied to a wide range of applications, including those requiring a rapid control response.
  arXiv  Detail & Related papers  (2024-08-19T08:13:37Z)
• Switchable Decision: Dynamic Neural Generation Networks [98.61113699324429]
  We propose a switchable decision to accelerate inference by dynamically assigning resources for each data instance. Our method benefits from less cost during inference while keeping the same accuracy.
  arXiv  Detail & Related papers  (2024-05-07T17:44:54Z)
• OptScaler: A Collaborative Framework for Robust Autoscaling in the Cloud [10.97507717758812]
  We propose OptScaler, a collaborative autoscaling framework that integrates proactive and reactive modules through an optimization module.<n> Numerical results have demonstrated the superiority of our workload prediction model and the collaborative framework.
  arXiv  Detail & Related papers  (2023-10-26T04:38:48Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Predictive GAN-powered Multi-Objective Optimization for Hybrid Federated Split Learning [56.125720497163684]
  We propose a hybrid federated split learning framework in wireless networks. We design a parallel computing scheme for model splitting without label sharing, and theoretically analyze the influence of the delayed gradient caused by the scheme on the convergence speed.
  arXiv  Detail & Related papers  (2022-09-02T10:29:56Z)
• A Meta Reinforcement Learning Approach for Predictive Autoscaling in the Cloud [10.970391043991363]
  We propose an end-to-end predictive meta model-based RL algorithm, aiming to optimally allocate resource to maintain a stable CPU utilization level. Our algorithm not only ensures the predictability and accuracy of the scaling strategy, but also enables the scaling decisions to adapt to the changing workloads with high sample efficiency.
  arXiv  Detail & Related papers  (2022-05-31T13:54:04Z)
• Automatically Learning Compact Quality-aware Surrogates for Optimization Problems [55.94450542785096]
  Solving optimization problems with unknown parameters requires learning a predictive model to predict the values of the unknown parameters and then solving the problem using these values. Recent work has shown that including the optimization problem as a layer in a complex training model pipeline results in predictions of iteration of unobserved decision making. We show that we can improve solution quality by learning a low-dimensional surrogate model of a large optimization problem.
  arXiv  Detail & Related papers  (2020-06-18T19:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.