LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation

• URL: http://arxiv.org/abs/2506.03691v2
• Date: Mon, 06 Oct 2025 05:05:37 GMT
• Title: LogSage: An LLM-Based Framework for CI/CD Failure Detection and Remediation with Industrial Validation
• Abstract summary: LogSage is an end-to-end framework for root cause analysis (RCA) and automated remediation of CI/CD failures.<n>It employs a token-efficient log preprocessing pipeline to filter noise and extract critical errors, then performs structured diagnostic prompting for accurate RCA.<n>On a benchmark of 367 CI/CD failures, LogSage achieves over 98% precision, near-perfect recall, and an F1 improvement of more than 38% points in the RCA stage.
• Score: 15.666174528504806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous Integration and Deployment (CI/CD) pipelines are critical to modern software engineering, yet diagnosing and resolving their failures remains complex and labor-intensive. We present LogSage, the first end-to-end LLM-powered framework for root cause analysis (RCA) and automated remediation of CI/CD failures. LogSage employs a token-efficient log preprocessing pipeline to filter noise and extract critical errors, then performs structured diagnostic prompting for accurate RCA. For solution generation, it leverages retrieval-augmented generation (RAG) to reuse historical fixes and invokes automation fixes via LLM tool-calling. On a newly curated benchmark of 367 GitHub CI/CD failures, LogSage achieves over 98\% precision, near-perfect recall, and an F1 improvement of more than 38\% points in the RCA stage, compared with recent LLM-based baselines. In a year-long industrial deployment at ByteDance, it processed over 1.07M executions, with end-to-end precision exceeding 80\%. These results demonstrate that LogSage provides a scalable and practical solution for automating CI/CD failure management in real-world DevOps workflows.

Related papers

• Log-Insight: Automating Microservice Incident Diagnosis via Neuro-Symbolic Log Analysis [4.172239714799967]
  We present Log-Insight, an automated incident-diagnosis system deployed in production at Huawei.<n>Our six-stage pipeline reduces millions of raw events by 1,000-7,000x while preserving statistically significant failure signals.<n>Log-Insight achieves MRR = 0.790, returning the correct root cause within the top-3 hypotheses in over 90% of runs in under a minute of latency.
  arXiv  Detail & Related papers  (2026-07-09T14:25:31Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generation [55.947962672433675]
  ChipMATE is the first self-trained multi-agent framework for RTL generation.<n>Inspired by industrial practice, ChipMATE pairs a Verilog agent with a Python reference-model agent that mutually verify each other's outputs.<n>ChipMATE achieves 75.0% and 80.1% pass@1 on VerilogEval V2 with 4B and 9B base models.
  arXiv  Detail & Related papers  (2026-05-13T01:04:21Z)
• Multi-Agent Systems for Root Cause Analysis in Microservices [58.95503998127531]
  LATS-RCA is a multi-agent framework for root cause analysis in MSS.<n> Reflection scores are used to guide the search toward the most likely root cause.<n>We evaluate LATS-RCA on the open-source industrial MSS, Light-OAuth2.
  arXiv  Detail & Related papers  (2026-05-05T08:39:42Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• Graph-Based Self-Healing Tool Routing for Cost-Efficient LLM Agents [0.0]
  Self-Healing Router is a fault-tolerant orchestration architecture.<n>It treats most agent control-flow decisions as routing rather than reasoning.<n>Every failure is either a logged reroute or an explicit escalation, never a silent skip.
  arXiv  Detail & Related papers  (2026-03-02T07:21:15Z)
• Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA [40.11960515464488]
  Large language models (LLMs) have shown promising results for automated failure management by previous work.<n>We evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP.<n>We provided the system with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures.<n>The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it.
  arXiv  Detail & Related papers  (2026-02-06T13:55:48Z)
• LogSieve: Task-Aware CI Log Reduction for Sustainable LLM-Based Analysis [0.0]
  We present LogSieve, a lightweight, RCA-aware and semantics-aware log reduction technique.<n>We evaluate it on CI logs from 20 open-source Android projects using GitHub Actions.<n>It achieves an average 42% reduction in lines and 40% reduction in tokens with minimal semantic loss.
  arXiv  Detail & Related papers  (2026-01-28T00:49:50Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• LogPilot: Intent-aware and Scalable Alert Diagnosis for Large-scale Online Service Systems [41.55191803277989]
  LogPilot is an intent-aware framework powered by Large Language Models (LLMs) for automated log-based alert diagnosis.<n>It reconstructs each request's execution into atemporal log chain, clusters similar chains to identify recurring execution patterns, and provides representative samples to the LLMs for diagnosis.<n> evaluated on real-world alerts from Volcano Engine Cloud, LogPilot improves the usefulness of root cause summarization by 50.34% and exact localization accuracy by 54.79% over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-09-30T07:11:28Z)
• InferLog: Accelerating LLM Inference for Online Log Parsing via ICL-oriented Prefix Caching [23.53991589506265]
  We present InferLog, the first inference optimization method for online log parsing.<n>InferLog accelerates inference by designing (1) A Prefix-aware ICL Refinement policy to refine the examples and permutation of in-context learning to improve the prefix caching efficiency.
  arXiv  Detail & Related papers  (2025-07-11T12:21:29Z)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems [50.29939179830491]
  Failure attribution in LLM multi-agent systems remains underexplored and labor-intensive.<n>We develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons.<n>The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps.
  arXiv  Detail & Related papers  (2025-04-30T23:09:44Z)
• AnomalyGen: An Automated Semantic Log Sequence Generation Framework with LLM for Anomaly Detection [25.83270938475311]
  AnomalyGen is the first automated log synthesis framework specifically designed for anomaly detection.<n>Our framework integrates enhanced program analysis with Chain-of-Thought reasoning (CoT reasoning) to enable iterative log generation and anomaly annotation.<n>When augmenting benchmark datasets with synthesized logs, we observe maximum F1-score improvements of 3.7%.
  arXiv  Detail & Related papers  (2025-04-16T16:54:38Z)
• LogLSHD: Fast Log Parsing with Locality-Sensitive Hashing and Dynamic Time Warping [2.415288727960745]
  Large-scale software systems generate vast volumes of system logs that are essential for monitoring, diagnosing, and performance optimization.<n>LogLSHD demonstrates exceptional efficiency in parsing time, significantly outperforming state-of-the-art methods.<n>For example, compared to Drain, LogLSHD reduces the average parsing time by 73% while increasing the average parsing accuracy by 15% on the LogHub 2.0 benchmark.
  arXiv  Detail & Related papers  (2025-04-02T23:08:04Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models.<n>Our framework incorporates two complementary strategies: internal TTC and external TTC.<n>We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal [55.13854171147104]
  Large Language Models (LLMs) have revolutionized various domains, including natural language processing, data analysis, and software development.<n>We present Dynamic Action Re-Sampling (DARS), a novel inference time compute scaling approach for coding agents.<n>We evaluate our approach on SWE-Bench Lite benchmark, demonstrating that this scaling strategy achieves a pass@k score of 55% with Claude 3.5 Sonnet V2.
  arXiv  Detail & Related papers  (2025-03-18T14:02:59Z)
• LLM2: Let Large Language Models Harness System 2 Reasoning [65.89293674479907]
  Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs.<n>We introduce LLM2, a novel framework that combines an LLM with a process-based verifier.<n>LLMs2 is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs.
  arXiv  Detail & Related papers  (2024-12-29T06:32:36Z)
• EDA-Aware RTL Generation with Large Language Models [0.7831852829409273]
  Large Language Models (LLMs) have become increasingly popular for generating RTL code.<n> producing error-free RTL code in a zero-shot setting remains highly challenging for even state-of-the-art LLMs.<n>We introduce AIvril2, a self-verifying, LLM-agnostic agentic framework aimed at enhancing RTL code generation through iterative corrections of both syntax and functional errors.
  arXiv  Detail & Related papers  (2024-11-21T00:37:51Z)
• LogParser-LLM: Advancing Efficient Log Parsing with Large Language Models [19.657278472819588]
  We introduce Log-LLM, a novel log integrated with LLM capabilities. We address the intricate challenge of parsing granularity, proposing a new metric to allow users to calibrate granularity to their specific needs. Our method's efficacy is empirically demonstrated through evaluations on the Loghub-2k and the large-scale LogPub benchmark.
  arXiv  Detail & Related papers  (2024-08-25T05:34:24Z)
• HELP: Hierarchical Embeddings-based Log Parsing [0.25112747242081457]
  Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. Existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies.
  arXiv  Detail & Related papers  (2024-08-15T17:54:31Z)
• Face It Yourselves: An LLM-Based Two-Stage Strategy to Localize Configuration Errors via Logs [29.736051629726145]
  configuration errors pose significant challenges for maintainers and new end-users. Given that logs are easily accessible to most end-users, we conduct a preliminary study to outline the challenges and opportunities of utilizing logs in localizing configuration errors. Based on the insights gained from the preliminary study, we propose an LLM-based two-stage strategy for end-users to localize the root-cause configuration properties based on logs.
  arXiv  Detail & Related papers  (2024-03-31T10:47:38Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning [56.887047551101574]
  We present DS-Agent, a novel framework that harnesses large language models (LLMs) agent and case-based reasoning (CBR) In the development stage, DS-Agent follows the CBR framework to structure an automatic iteration pipeline, which can flexibly capitalize on the expert knowledge from Kaggle. In the deployment stage, DS-Agent implements a low-resource deployment stage with a simplified CBR paradigm, significantly reducing the demand on foundational capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-02-27T12:26:07Z)
• LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection [73.69399219776315]
  We propose a unified Transformer-based framework for Log anomaly detection (LogFormer) to improve the generalization ability across different domains. Specifically, our model is first pre-trained on the source domain to obtain shared semantic knowledge of log data. Then, we transfer such knowledge to the target domain via shared parameters.
  arXiv  Detail & Related papers  (2024-01-09T12:55:21Z)
• Self-Supervised Log Parsing [59.04636530383049]
  Large-scale software systems generate massive volumes of semi-structured log records. Existing approaches rely on log-specifics or manual rule extraction. We propose NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling.
  arXiv  Detail & Related papers  (2020-03-17T19:25:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.