Artificial Intelligence for Cost-Aware Resource Prediction in Big Data Pipelines

• URL: http://arxiv.org/abs/2510.05127v1
• Date: Tue, 30 Sep 2025 20:01:12 GMT
• Title: Artificial Intelligence for Cost-Aware Resource Prediction in Big Data Pipelines
• Authors: Harshit Goyal,
• Abstract summary: Over-provisioning leads to unnecessary costs, while under-provisioning risks performance degradation and SLA violations.<n>This work presents an artificial intelligence approach to predict resource utilization in big data pipelines using Random Forest regression.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Efficient resource allocation is a key challenge in modern cloud computing. Over-provisioning leads to unnecessary costs, while under-provisioning risks performance degradation and SLA violations. This work presents an artificial intelligence approach to predict resource utilization in big data pipelines using Random Forest regression. We preprocess the Google Borg cluster traces to clean, transform, and extract relevant features (CPU, memory, usage distributions). The model achieves high predictive accuracy (R Square = 0.99, MAE = 0.0048, RMSE = 0.137), capturing non-linear relationships between workload characteristics and resource utilization. Error analysis reveals impressive performance on small-to-medium jobs, with higher variance in rare large-scale jobs. These results demonstrate the potential of AI-driven prediction for cost-aware autoscaling in cloud environments, reducing unnecessary provisioning while safeguarding service quality.

Related papers

• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios [76.85739138203014]
  We present SpecFormer, a novel architecture that accelerates unidirectional and attention mechanisms.<n>We demonstrate that SpecFormer achieves lower training demands and reduced computational costs.
  arXiv  Detail & Related papers  (2025-11-25T14:20:08Z)
• Towards a Proactive Autoscaling Framework for Data Stream Processing at the Edge using GRU and Transfer Learning [0.0]
  We show how a GRU neural network forecasts the upstream load using real-world and synthetic DSP datasets.<n>A transfer learning framework integrates the predictive model into an online stream processing system.<n>The lightweight GRU model for load predictions recorded up to 1.3% SMAPE value on a real-world data set.
  arXiv  Detail & Related papers  (2025-07-19T12:47:50Z)
• Collaborative Prediction: To Join or To Disjoin Datasets [5.9697789282446605]
  We study the problem of developing practical algorithms that select appropriate dataset to minimize population loss.<n>By leveraging an oracle inequality and data-driven estimators, the algorithm reduces population loss with high probability.
  arXiv  Detail & Related papers  (2025-06-12T20:25:07Z)
• Network Resource Optimization for ML-Based UAV Condition Monitoring with Vibration Analysis [54.550658461477106]
  Condition Monitoring (CM) uses Machine Learning (ML) models to identify abnormal and adverse conditions.<n>This work explores the optimization of network resources for ML-based UAV CM frameworks.<n>By leveraging dimensionality reduction techniques, there is a 99.9% reduction in network resource consumption.
  arXiv  Detail & Related papers  (2025-02-21T14:36:12Z)
• Value-Based Deep RL Scales Predictably [100.21834069400023]
  We show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior.<n>We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym.
  arXiv  Detail & Related papers  (2025-02-06T18:59:47Z)
• Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels [57.46832672991433]
  We propose a novel equation discovery method based on Kernel learning and BAyesian Spike-and-Slab priors (KBASS) We use kernel regression to estimate the target function, which is flexible, expressive, and more robust to data sparsity and noises. We develop an expectation-propagation expectation-maximization algorithm for efficient posterior inference and function estimation.
  arXiv  Detail & Related papers  (2023-10-09T03:55:09Z)
• TransPath: Learning Heuristics For Grid-Based Pathfinding via Transformers [64.88759709443819]
  We suggest learning the instance-dependent proxies that are supposed to notably increase the efficiency of the search. The first proxy we suggest to learn is the correction factor, i.e. the ratio between the instance independent cost-to-go estimate and the perfect one. The second proxy is the path probability, which indicates how likely the grid cell is lying on the shortest path.
  arXiv  Detail & Related papers  (2022-12-22T14:26:11Z)
• Privacy Adhering Machine Un-learning in NLP [66.17039929803933]
  In real world industry use Machine Learning to build models on user data. Such mandates require effort both in terms of data as well as model retraining. continuous removal of data and model retraining steps do not scale. We propose textitMachine Unlearning to tackle this challenge.
  arXiv  Detail & Related papers  (2022-12-19T16:06:45Z)
• Optimal Resource Allocation for Serverless Queries [8.59568779761598]
  Prior work focused on predicting peak allocation while ignoring aggressive trade-offs between resource allocation and run-time. We introduce a system for optimal resource allocation that can predict performance with aggressive trade-offs, for both new and past observed queries.
  arXiv  Detail & Related papers  (2021-07-19T02:55:48Z)
• Cost-effective Machine Learning Inference Offload for Edge Computing [0.3149883354098941]
  This paper proposes a novel offloading mechanism by leveraging installed-base on-premises (edge) computational resources. The proposed mechanism allows the edge devices to offload heavy and compute-intensive workloads to edge nodes instead of using remote cloud.
  arXiv  Detail & Related papers  (2020-12-07T21:11:02Z)
• A Predictive Autoscaler for Elastic Batch Jobs [8.354712625979776]
  Large batch jobs such as Deep Learning, HPC and Spark require far more computational resources and higher cost than conventional online service. We propose a predictive autoscaler to provide an elastic interface for the customers and overprovision instances.
  arXiv  Detail & Related papers  (2020-10-10T17:35:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.