eARCO: Efficient Automated Root Cause Analysis with Prompt Optimization

• URL: http://arxiv.org/abs/2504.11505v1
• Date: Tue, 15 Apr 2025 08:10:32 GMT
• Title: eARCO: Efficient Automated Root Cause Analysis with Prompt Optimization
• Authors: Drishti Goel, Raghav Magazine, Supriyo Ghosh, Akshay Nambi, Prathamesh Deshpande, Xuchao Zhang, Chetan Bansal, Saravan Rajmohan,
• Abstract summary: Root cause analysis (RCA) for incidents in large-scale cloud systems is a complex, knowledge-intensive task.<n>Recent advancements in Large-Language Models (LLMs) have proven to be effective in solving different stages of the incident management lifecycle.<n>We leverage 'PromptWizard', a state-of-the-art prompt optimization technique, to automatically identify the best optimized prompt instruction.
• Score: 15.299667843493491
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Root cause analysis (RCA) for incidents in large-scale cloud systems is a complex, knowledge-intensive task that often requires significant manual effort from on-call engineers (OCEs). Improving RCA is vital for accelerating the incident resolution process and reducing service downtime and manual efforts. Recent advancements in Large-Language Models (LLMs) have proven to be effective in solving different stages of the incident management lifecycle including RCA. However, existing LLM-based RCA recommendations typically leverage default finetuning or retrieval augmented generation (RAG) methods with static, manually designed prompts, which lead to sub-optimal recommendations. In this work, we leverage 'PromptWizard', a state-of-the-art prompt optimization technique, to automatically identify the best optimized prompt instruction that is combined with semantically similar historical examples for querying underlying LLMs during inference. Moreover, by utilizing more than 180K historical incident data from Microsoft, we developed cost-effective finetuned small language models (SLMs) for RCA recommendation generation and demonstrate the power of prompt optimization on such domain-adapted models. Our extensive experimental results show that prompt optimization can improve the accuracy of RCA recommendations by 21% and 13% on 3K test incidents over RAG-based LLMs and finetuned SLMs, respectively. Lastly, our human evaluation with incident owners have demonstrated the efficacy of prompt optimization on RCA recommendation tasks. These findings underscore the advantages of incorporating prompt optimization into AI for Operations (AIOps) systems, delivering substantial gains without increasing computational overhead.

Related papers

• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner [31.033131727230277]
  Large reasoning models (LRMs) have recently shown promise in solving complex math problems when optimized with Reinforcement Learning (RL)<n>We propose a novel intrinsic signal-driven generative process evaluation mechanism operating at the thought level to address major bottlenecks in RL-based training.<n>Experiments on 1.5B and 7B parameter LRMs demonstrate that our method achieves higher problem-solving accuracy with significantly fewer training samples than outcome-only reward baselines.
  arXiv  Detail & Related papers  (2025-07-31T07:54:58Z)
• Preference Optimization for Combinatorial Optimization Problems [54.87466279363487]
  Reinforcement Learning (RL) has emerged as a powerful tool for neural optimization, enabling models learns that solve complex problems without requiring expert knowledge.<n>Despite significant progress, existing RL approaches face challenges such as diminishing reward signals and inefficient exploration in vast action spaces.<n>We propose Preference Optimization, a novel method that transforms quantitative reward signals into qualitative preference signals via statistical comparison modeling.
  arXiv  Detail & Related papers  (2025-05-13T16:47:00Z)
• CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction [0.2548904650574671]
  Retrieval Augmented Generation (RAG) has emerged as a powerful application of Large Language Models (LLMs) This research contributes to enhancing the reliability and trustworthiness of AI-generated content in information retrieval and summarization tasks.
  arXiv  Detail & Related papers  (2025-04-22T06:41:25Z)
• Review, Refine, Repeat: Understanding Iterative Decoding of AI Agents with Dynamic Evaluation and Selection [71.92083784393418]
  Inference-time methods such as Best-of-N (BON) sampling offer a simple yet effective alternative to improve performance.<n>We propose Iterative Agent Decoding (IAD) which combines iterative refinement with dynamic candidate evaluation and selection guided by a verifier.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Self-Routing RAG: Binding Selective Retrieval with Knowledge Verbalization [97.72503890388866]
  We propose Self-Routing RAG (SR-RAG), a novel framework that binds selective retrieval with knowledge verbalization.<n>SR-RAG enables an LLM to dynamically decide between external retrieval and verbalizing its own parametric knowledge.<n>We introduce dynamic knowledge source inference via nearest neighbor search to improve the accuracy of knowledge source decision.
  arXiv  Detail & Related papers  (2025-04-01T17:59:30Z)
• DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal [55.13854171147104]
  Large Language Models (LLMs) have revolutionized various domains, including natural language processing, data analysis, and software development.<n>We present Dynamic Action Re-Sampling (DARS), a novel inference time compute scaling approach for coding agents.<n>We evaluate our approach on SWE-Bench Lite benchmark, demonstrating that this scaling strategy achieves a pass@k score of 55% with Claude 3.5 Sonnet V2.
  arXiv  Detail & Related papers  (2025-03-18T14:02:59Z)
• RAG-Gym: Systematic Optimization of Language Agents for Retrieval-Augmented Generation [43.50113345998687]
  We introduce RAG-Gym, a comprehensive platform that explores three optimization dimensions: (1) prompt engineering, (2) actor tuning, and (3) critic training.<n>For prompt engineering, we propose Re$2$Search, a novel agent incorporating reflection reasoning that significantly outperforms standard prompts.<n>In actor tuning, we evaluate three popular post-training algorithms with fine-grained process supervision and identify direct preference optimization as the most effective.
  arXiv  Detail & Related papers  (2025-02-19T18:56:03Z)
• RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization [53.63439735067081]
  Large language models (LLMs) have achieved impressive performance but face high computational costs and latency.<n>Retrieval-augmented generation (RAG) helps by integrating external knowledge, but imperfect retrieval can introduce distracting noise that misleads SLMs.<n>We propose RoseRAG, a robust RAG framework for SLMs via Margin-aware Preference Optimization.
  arXiv  Detail & Related papers  (2025-02-16T04:56:53Z)
• The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exit [46.37267466656765]
  This paper presents an optimization framework that combines Retrieval-Augmented Generation (RAG) with an innovative multi-head early exit architecture.<n>Our experiments demonstrate how this architecture effectively decreases time without sacrificing the accuracy needed for reliable recommendation delivery.
  arXiv  Detail & Related papers  (2025-01-04T03:26:46Z)
• Discovering Preference Optimization Algorithms with and for Large Language Models [50.843710797024805]
  offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. We perform objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Experiments demonstrate the state-of-the-art performance of DiscoPOP, a novel algorithm that adaptively blends logistic and exponential losses.
  arXiv  Detail & Related papers  (2024-06-12T16:58:41Z)
• DNS-Rec: Data-aware Neural Architecture Search for Recommender Systems [79.76519917171261]
  This paper addresses the computational overhead and resource inefficiency prevalent in Sequential Recommender Systems (SRSs)<n>We introduce an innovative approach combining pruning methods with advanced model designs.<n>Our principal contribution is the development of a Data-aware Neural Architecture Search for Recommender System (DNS-Rec)
  arXiv  Detail & Related papers  (2024-02-01T07:22:52Z)
• PerfRL: A Small Language Model Framework for Efficient Code Optimization [14.18092813639534]
  In this paper, we introduce PerfRL, an innovative framework designed to tackle the problem of code optimization.<n>Our framework leverages the capabilities of small language models (SLMs) and reinforcement learning (RL)<n>Our approach achieves similar or better results compared to state-of-the-art models using shorter training times and smaller pre-trained models.
  arXiv  Detail & Related papers  (2023-12-09T19:50:23Z)
• Prompt-Tuning Decision Transformer with Preference Ranking [83.76329715043205]
  We propose the Prompt-Tuning DT algorithm to address challenges by using trajectory segments as prompts to guide RL agents in acquiring environmental information. Our approach involves randomly sampling a Gaussian distribution to fine-tune the elements of the prompt trajectory and using preference ranking function to find the optimization direction. Our work contributes to the advancement of prompt-tuning approaches in RL, providing a promising direction for optimizing large RL agents for specific preference tasks.
  arXiv  Detail & Related papers  (2023-05-16T17:49:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.