Agentic Self-Healing for Data and AI Pipelines: An Affordable Vendor-Agnostic Architecture using Open-Source Software

• URL: http://arxiv.org/abs/2608.01955v1
• Date: Mon, 03 Aug 2026 09:26:30 GMT
• Title: Agentic Self-Healing for Data and AI Pipelines: An Affordable Vendor-Agnostic Architecture using Open-Source Software
• Abstract summary: This paper compares existing off-the-shelf solutions for AI-assisted pipeline monitoring, root-cause analysis, and automated remediation.<n>We find that the required ingredients for self-healing pipelines already exist, but they are fragmented across vendor-specific platforms, observability tools, incident systems, and open-source components.<n>We propose an affordable, vendor-agnostic reference architecture for agentic self-healing pipelines using open-source and low-cost tools.
• Score: 1.6612854433953734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern organizations rely on data, machine learning, and software delivery pipelines to move data, train models, deploy applications, refresh dashboards, and support business-critical decisions. However, these pipelines often fail because of data quality issues, schema changes, upstream source changes, infrastructure problems, orchestration failures, and model workflow issues. Existing ZeroOps, observability, and AI operations platforms can help teams detect incidents, investigate root causes, and in some cases recommend or execute fixes. However, many of these solutions are expensive, vendor-specific, or difficult for smaller teams to adapt across different tools and environments. This paper first compares existing off-the-shelf solutions for AI-assisted pipeline monitoring, root-cause analysis, and automated remediation, including their strengths, limitations, and practical trade-offs. Based on this comparison, we find that the main gap is architectural rather than technological: the required ingredients for self-healing pipelines already exist, but they are fragmented across vendor-specific platforms, observability tools, incident systems, and open-source components. We therefore propose an affordable, vendor-agnostic reference architecture for agentic self-healing pipelines using open-source and low-cost tools. The proposed architecture combines monitoring, pipeline metadata, incident history, deterministic policy checks, AI-assisted diagnosis, approval workflows, and controlled remediation actions to help teams detect, diagnose, repair, verify, and learn from pipeline issues with less manual effort. The goal is to provide a practical reference architecture that can be adapted across data engineering, machine learning operations, and software delivery environments.

Related papers

• Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling [1.6639892342953113]
  We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design in an industry-realistic tool-calling setting.<n>To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development.<n>We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures.
  arXiv  Detail & Related papers  (2026-08-25T15:33:50Z)
• QwenPaw-Data: Bridging Facts, Methodology, and Execution for Autonomous Enterprise Data Analytics [45.48758200829718]
  We introduce QwenPaw-Data, an agentic data system designed for enterprise intelligent data analysis.<n>QwenPaw-Data consolidates heterogeneous assets into reusable, governable, and evolvable analysis assets, then turns natural-language requests into end-to-end analytical spanning data understanding, retrieval, analysis, report generation, and decision support.
  arXiv  Detail & Related papers  (2026-07-13T02:38:00Z)
• VADAOrchestra: Neurosymbolic Orchestration of Adaptive Reasoning Workflows [47.70653885737167]
  Decision-making in real-world settings rarely follows a fixed script. Instead, it unfolds as a dynamic reasoning process.<n>We introduce VADAOrchestra, a neurosymbolic framework that models complex as evolving reasoning processes.
  arXiv  Detail & Related papers  (2026-06-21T13:10:48Z)
• The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration [58.61149924275458]
  Tool use enables large language models to access external information, invoke software systems, and act in digital environments beyond what can be solved from model parameters alone.<n>As agent systems evolve, the central problem has shifted from isolated invocation to multi-tool orchestration over long trajectories with intermediate state, execution feedback, changing environments, and practical constraints such as safety, cost, and verifiability.<n>We organize the literature around six core dimensions: inference-time planning and execution, training and trajectory construction, safety and control, efficiency under resource constraints, capability completeness in open environments, and benchmark design and evaluation.
  arXiv  Detail & Related papers  (2026-03-24T07:05:05Z)
• kRAIG: A Natural Language-Driven Agent for Automated DataOps Pipeline Generation [0.5833117322405447]
  We introduce kRAIG, an AI agent that translates natural language specifications into production-ready KubeReason Pipelines (KFP)<n>To resolve ambiguity in user intent, we propose ReQuesAct, an interaction framework that explicitly clarifies intent prior to pipeline synthesis.<n>Our framework achieves a 3x improvement in extraction and loading success and a 25 percent increase in transformation accuracy compared to state-of-the-art agentic baselines.
  arXiv  Detail & Related papers  (2026-03-19T17:53:00Z)
• Agentic Diagnostic Reasoning over Telecom and Datacenter Infrastructure [0.0]
  We introduce an agentic diagnostic framework for root cause analysis.<n>Instead of embedding causal logic into the application, the agent autonomously navigates the infrastructure model.<n>This work lays the foundation for autonomous change impact mitigation.
  arXiv  Detail & Related papers  (2026-01-12T09:13:04Z)
• Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization [69.36509281190662]
  Adapting production-level computer vision tools to bespoke scientific datasets is a critical "last mile" bottleneck.<n>We consider using AI agents to automate this manual coding, and focus on the open question of optimal agent design.<n>We demonstrate that a simple agent framework consistently generates adaptation code that outperforms human-expert solutions.
  arXiv  Detail & Related papers  (2025-12-02T18:42:26Z)
• Design and Evaluation of a Scalable Data Pipeline for AI-Driven Air Quality Monitoring in Low-Resource Settings [0.4681310436826459]
  This paper presents the design, implementation, and evaluation of the AirQo data pipeline.<n>It is built using open-source technologies such as Apache Airflow, Apache Kafka, and Google BigQuery.<n>We demonstrate the pipeline's ability to ingest, transform, and distribute millions of air quality measurements monthly from over 400 monitoring devices.
  arXiv  Detail & Related papers  (2025-08-20T06:19:27Z)
• RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models [46.476439550746136]
  Large language model (LLM) applications in cloud root cause analysis (RCA) have been actively explored recently. We present RCAgent, a tool-augmented LLM autonomous agent framework for practical and privacy-aware industrial RCA usage. Running on an internally deployed model rather than GPT families, RCAgent is capable of free-form data collection and comprehensive analysis with tools.
  arXiv  Detail & Related papers  (2023-10-25T03:53:31Z)
• Trusted Provenance of Automated, Collaborative and Adaptive Data Processing Pipelines [2.186901738997927]
  We provide a solution architecture and a proof of concept implementation of a service, called Provenance Holder. Provenance Holder enables provenance of collaborative, adaptive data processing pipelines in a trusted manner.
  arXiv  Detail & Related papers  (2023-10-17T17:52:27Z)
• PyRCA: A Library for Metric-based Root Cause Analysis [66.72542200701807]
  PyRCA is an open-source machine learning library of Root Cause Analysis (RCA) for Artificial Intelligence for IT Operations (AIOps) It provides a holistic framework to uncover the complicated metric causal dependencies and automatically locate root causes of incidents.
  arXiv  Detail & Related papers  (2023-06-20T09:55:10Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• MLCask: Efficient Management of Component Evolution in Collaborative Data Analytics Pipelines [29.999324319722508]
  We address two main challenges that arise during the deployment of machine learning pipelines, and address them with the design of versioning for an end-to-end analytics system MLCask. We define and accelerate the metric-driven merge operation by pruning the pipeline search tree using reusable history records and pipeline compatibility information. The effectiveness of MLCask is evaluated through an extensive study over several real-world deployment cases.
  arXiv  Detail & Related papers  (2020-10-17T13:34:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.