Governing Cloud Data Pipelines with Agentic AI

• URL: http://arxiv.org/abs/2512.23737v1
• Date: Wed, 24 Dec 2025 19:30:32 GMT
• Title: Governing Cloud Data Pipelines with Agentic AI
• Authors: Aswathnarayan Muthukrishnan Kirubakaran, Adithya Parthasarathy, Nitin Saksena, Ram Sekhar Bodala, Akshay Deshpande, Suhas Malempati, Shiva Carimireddy, Abhirup Mazumder,
• Abstract summary: Agentic Cloud Data Engineering is a policy-aware control architecture that integrates bounded AI agents into the governance and control plane of cloud data pipelines.<n>We show that Agentic Cloud Data Engineering platform reduces mean pipeline recovery time by up to 45%, lowers operational cost by approximately 25%, and decreases manual intervention events by over 70% compared to static orchestration.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cloud data pipelines increasingly operate under dynamic workloads, evolving schemas, cost constraints, and strict governance requirements. Despite advances in cloud-native orchestration frameworks, most production pipelines rely on static configurations and reactive operational practices, resulting in prolonged recovery times, inefficient resource utilization, and high manual overhead. This paper presents Agentic Cloud Data Engineering, a policy-aware control architecture that integrates bounded AI agents into the governance and control plane of cloud data pipelines. In Agentic Cloud Data Engineering platform, specialized agents analyze pipeline telemetry and metadata, reason over declarative cost and compliance policies, and propose constrained operational actions such as adaptive resource reconfiguration, schema reconciliation, and automated failure recovery. All agent actions are validated against governance policies to ensure predictable and auditable behavior. We evaluate Agentic Cloud Data Engineering platform using representative batch and streaming analytics workloads constructed from public enterprise-style datasets. Experimental results show that Agentic Cloud Data Engineering platform reduces mean pipeline recovery time by up to 45%, lowers operational cost by approximately 25%, and decreases manual intervention events by over 70% compared to static orchestration, while maintaining data freshness and policy compliance. These results demonstrate that policy-bounded agentic control provides an effective and practical approach for governing cloud data pipelines in enterprise environments.

Related papers

• Cognitive Platform Engineering for Autonomous Cloud Operations [0.14658400971135652]
  This paper introduces Cognitive Platform Engineering, a next-generation paradigm that integrates sensing, reasoning, and autonomous action directly into the platform lifecycle.<n>A prototype implementation built with Terraform, Open Policy Agent, and ML-based anomaly detection demonstrates improvements in mean time to resolution, resource efficiency, and compliance.
  arXiv  Detail & Related papers  (2026-01-24T18:17:49Z)
• Big Data Workload Profiling for Energy-Aware Cloud Resource Management [0.0]
  This paper presents a workload aware and energy efficient scheduling framework.<n>It profiles utilization, memory demand, and storage IO behavior to guide virtual machine placement decisions.<n>Results demonstrate consistent energy savings of 15 to 20 percent compared to a baseline scheduler.
  arXiv  Detail & Related papers  (2026-01-17T06:50:51Z)
• Analyzing and Internalizing Complex Policy Documents for LLM Agents [53.14898416858099]
  Large Language Model (LLM)-based agentic systems rely on in-context policy documents encoding diverse business rules.<n>This motivates developing internalization methods that embed policy documents into model priors while preserving performance.<n>We introduce CC-Gen, an agentic benchmark generator with Controllable Complexity across four levels.
  arXiv  Detail & Related papers  (2025-10-13T16:30:07Z)
• Autonomous Data Agents: A New Opportunity for Smart Data [50.02229219403014]
  Report argues that DataAgents represent a paradigm shift toward autonomous data-to-knowledge systems.<n>DataAgents transform complex and unstructured data into coherent and actionable knowledge.<n>We first examine why the convergence of agentic AI and data-to-knowledge systems has emerged as a critical trend.
  arXiv  Detail & Related papers  (2025-09-23T06:46:41Z)
• Large Language Model-Empowered Decision Transformer for UAV-Enabled Data Collection [71.84636717632206]
  Unmanned aerial vehicles (UAVs) for reliable and energy-efficient data collection from spatially distributed devices holds great promise in supporting Internet of Things (IoT) applications.<n>We propose a joint language model (LLM) to learn effective UAV control policies.<n>LLM-CRDT outperforms benchmark online and offline methods, achieving up to 36.7% higher energy efficiency than current state-of-the-art DT approaches.
  arXiv  Detail & Related papers  (2025-09-17T13:05:08Z)
• Design and Evaluation of a Scalable Data Pipeline for AI-Driven Air Quality Monitoring in Low-Resource Settings [0.4681310436826459]
  This paper presents the design, implementation, and evaluation of the AirQo data pipeline.<n>It is built using open-source technologies such as Apache Airflow, Apache Kafka, and Google BigQuery.<n>We demonstrate the pipeline's ability to ingest, transform, and distribute millions of air quality measurements monthly from over 400 monitoring devices.
  arXiv  Detail & Related papers  (2025-08-20T06:19:27Z)
• Building Scalable AI-Powered Applications with Cloud Databases: Architectures, Best Practices and Performance Considerations [0.0]
  The rapid adoption of AI-powered applications demands high-performance, scalable, and efficient cloud database solutions.<n>This paper explores how cloud-native databases enable AI-driven applications by leveraging purpose-built technologies.<n>Performance benchmarks, scalability considerations, and cost-efficient strategies are evaluated to guide the design of AI-enabled applications.
  arXiv  Detail & Related papers  (2025-04-26T04:17:46Z)
• Automated Planning for Optimal Data Pipeline Instantiation [10.501636306956385]
  We model the problem of optimal data pipeline deployment as planning with action costs.<n>We propose strategies aiming to minimize total execution time.<n> Experimental results indicate that the strategies can outperform the baseline deployment.
  arXiv  Detail & Related papers  (2025-03-16T19:43:12Z)
• Deep Reinforcement Learning for Job Scheduling and Resource Management in Cloud Computing: An Algorithm-Level Review [10.015735252600793]
  Deep Reinforcement Learning (DRL) has emerged as a promising solution to these challenges.<n>DRL enables systems to learn and adapt policies based on continuous observations of the environment.<n>This survey provides a comprehensive review of DRL-based algorithms for job scheduling and resource management in cloud computing.
  arXiv  Detail & Related papers  (2025-01-02T02:08:00Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows [58.762959061522736]
  offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. We build upon recent works on learning policies in latent action spaces and use a special form of Normalizing Flows for constructing a generative model. We evaluate our method on various locomotion and navigation tasks, demonstrating that our approach outperforms recently proposed algorithms.
  arXiv  Detail & Related papers  (2022-11-20T21:57:10Z)
• Scalable Discovery and Continuous Inventory of Personal Data at Rest in Cloud Native Systems [0.0]
  Cloud native systems are processing large amounts of personal data through numerous and possibly multi-paradigmatic data stores. From a privacy engineering perspective, a core challenge is to keep track of all exact locations, where personal data is being stored. We present Teiresias, comprising i) a workflow pattern for scalable discovery of personal data at rest, and ii) a cloud native system architecture and open source prototype implementation of said workflow pattern.
  arXiv  Detail & Related papers  (2022-09-09T10:45:34Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.