Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA

• URL: http://arxiv.org/abs/2602.06709v1
• Date: Fri, 06 Feb 2026 13:55:48 GMT
• Title: Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA
• Abstract summary: Large language models (LLMs) have shown promising results for automated failure management by previous work.<n>We evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP.<n>We provided the system with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures.<n>The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it.
• Score: 40.11960515464488
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: CI/CD pipeline failure management is time-consuming when performed manually. Automating this process is non-trivial because the information required for effective failure management is unstructured and cannot be automatically processed by traditional programs. With their ability to process unstructured data, large language models (LLMs) have shown promising results for automated failure management by previous work. Following these studies, we evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP HANA. We evaluated the ability of the LLM-based system to identify the error location and to propose exact solutions that contain no unnecessary actions. To support the LLM in generating exact solutions, we provided it with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures. We conducted an ablation study to determine which type of domain knowledge contributed most to solution accuracy. The results show that data from historical failures contributed the most to the system's accuracy, enabling it to produce exact solutions in 92.1% of cases in our dataset. The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it. In conclusion, our findings indicate that LLMs, when provided with data from historical failures, represent a promising approach for automating CI/CD pipeline failure management.

Related papers

• How Far Can Root Cause Analysis Go on Real-World Telemetry Data? [0.8594140167290097]
  We show that classical causal discovery methods and existing LLM-based multi-agent systems fail to reliably identify root causes.<n>We present a Structured Multi-Agent RCA pipeline that substantially outperforms existing LLM-based and classical baselines.
  arXiv  Detail & Related papers  (2026-07-15T07:54:46Z)
• Tracing Agentic Failure from the Flow of Success [7.071269758592172]
  We argue that a practical failure attribution model should be lightweight and trainable without step-level supervision on failure data.<n>We propose OAT, which casts this problem as one-class learning with neural controlled differential equations.<n>With training on only 100 successful trajectories, OAT is 200--5000 $times$ faster than prompting-based baselines.
  arXiv  Detail & Related papers  (2026-07-14T13:16:14Z)
• Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents [80.03824805228719]
  A major challenge in developing computer-use agents is collecting large-scale, high-quality trajectories.<n>We propose a data-centric paradigm that turns failed trajectories into agent improvements.<n>We validate this approach with the state-of-the-art OpenCUA-72B model on the OSWorld benchmark.
  arXiv  Detail & Related papers  (2026-06-30T07:44:37Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis [68.28714988482703]
  Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs)<n>We present a study revealing that general-domain PRMs struggle to supervise data analysis agents.<n>We introduce DataPRM, a novel environment-aware generative process reward model.
  arXiv  Detail & Related papers  (2026-04-27T09:00:30Z)
• An End-to-End Framework for Building Large Language Models for Software Operations [9.453014847107069]
  OpsLLM is a domain-specific large language model for software operations.<n>It supports both knowledge-based question answering (QA) and root cause analysis (RCA)<n>We will open-source three versions of OpsLLM with 7B, 14B and 32B parameters, along with a 15K fine-tuning dataset.
  arXiv  Detail & Related papers  (2026-04-06T02:40:18Z)
• Graph-Based Self-Healing Tool Routing for Cost-Efficient LLM Agents [0.0]
  Self-Healing Router is a fault-tolerant orchestration architecture.<n>It treats most agent control-flow decisions as routing rather than reasoning.<n>Every failure is either a logged reroute or an explicit escalation, never a silent skip.
  arXiv  Detail & Related papers  (2026-03-02T07:21:15Z)
• An Agentic System for Schema Aware NL2SQL Generation [0.0]
  Large Language Models (LLMs) raise concerns regarding computational overhead, data privacy, and real-world deployability in resource-constrained environments.<n>We propose a schema based agentic system that strategically employs Small Language Models (SLMs) as primary agents, complemented by a selective LLM fallback mechanism.<n>The system achieves an average cost per query of 0.0085 compared to 0.094 for LLM-only systems, achieving near-zero operational costs for locally executed queries.
  arXiv  Detail & Related papers  (2026-02-25T02:58:02Z)
• DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows [22.16698382751559]
  Large language models (LLMs) have emerged as a promising solution for automating data governance by translating user intent into code.<n>Existing benchmarks for automated data science often emphasize snippet-level coding or high-level analytics.<n>We introduce DataGovBench, a benchmark featuring 150 diverse tasks grounded in real-world scenarios, built on data from actual cases.
  arXiv  Detail & Related papers  (2025-12-04T03:25:12Z)
• Specification-Guided Repair of Arithmetic Errors in Dafny Programs using LLMs [79.74676890436174]
  We present an APR tool for Dafny that uses formal specifications as oracles for fault localization and repair.<n>We localize faults through a series of steps, which include using Hoare logic to determine the state of each statement within the program.<n>Our tool achieves 89.6% fault localization coverage and GPT-4o mini yields the highest repair success rate of 74.18%.
  arXiv  Detail & Related papers  (2025-07-04T15:36:12Z)
• Scaling LLM Planning: NL2FLOW for Parametric Problem Generation and Rigorous Evaluation [0.913755431537592]
  This work introduces NL2Flow, a fully automated pipeline for generating and evaluating workflow planning problems.<n> NL2Flow generates problems parametrically in a structured intermediate representation, translating them into both natural language and formal PDDL.<n>I evaluate several open-source, instruct-tuned LLMs on a dataset of 2296 low-difficulty problems generated by NL2Flow.
  arXiv  Detail & Related papers  (2025-07-03T03:02:49Z)
• LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation [15.666174528504806]
  LogSage is an end-to-end framework for root cause analysis (RCA) and automated remediation of CI/CD failures.<n>It employs a token-efficient log preprocessing pipeline to filter noise and extract critical errors, then performs structured diagnostic prompting for accurate RCA.<n>On a benchmark of 367 CI/CD failures, LogSage achieves over 98% precision, near-perfect recall, and an F1 improvement of more than 38% points in the RCA stage.
  arXiv  Detail & Related papers  (2025-06-04T08:22:56Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)
• Discovering Knowledge Deficiencies of Language Models on Massive Knowledge Base [30.705524808195268]
  Large language models (LLMs) possess impressive linguistic capabilities but often fail to faithfully retain factual knowledge.<n>We propose error ascent (SEA), a scalable and efficient framework for discovering knowledge deficiencies (errors) in closed-weight LLMs.<n>SEA uncovers 40.7x more knowledge errors than Automated Capability Discovery and 26.7% more than AutoBencher.
  arXiv  Detail & Related papers  (2025-03-30T08:33:56Z)
• Confident or Seek Stronger: Exploring Uncertainty-Based On-device LLM Routing From Benchmarking to Generalization [61.02719787737867]
  Large language models (LLMs) are increasingly deployed and democratized on edge devices.<n>One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger LLMs when resulting in low-confidence responses on SLM.<n>We conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 1500+ settings.
  arXiv  Detail & Related papers  (2025-02-06T18:59:11Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning [56.887047551101574]
  We present DS-Agent, a novel framework that harnesses large language models (LLMs) agent and case-based reasoning (CBR) In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle. In the deployment stage, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm, significantly reducing the demand on foundational capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-02-27T12:26:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.