ORCA-bench: How Ready Are Language Model Agents for Oncall?

• URL: http://arxiv.org/abs/2607.28545v2
• Date: Wed, 05 Aug 2026 12:21:27 GMT
• Title: ORCA-bench: How Ready Are Language Model Agents for Oncall?
• Abstract summary: ORCA-bench is a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting.<n>The best RCA Accuracy is 25.3% on Medium-difficulty tasks.<n>The weakest model hallucinates an implausible root cause in 40% of incident reports.
• Score: 14.379965421387586
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $κ_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/orca-bench.

Related papers

• Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures [11.365690419023425]
  Existing root-cause attribution methods suffer from low diagnostic accuracy as execution traces grow larger.<n>Continual Search nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence.<n>MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks.
  arXiv  Detail & Related papers  (2026-09-11T19:29:21Z)
• ARGUS: MCP-Grounded Root Cause Analysis for Kubernetes Incidents [0.9685837672183747]
  ARGUS connects a commercial to live observability data through MCP servers covering Prometheus state, metrics, logs, and NATS messaging.<n>ARGUS named the correct root cause in all ten scenarios with an aggregate MCP success ratio of 0.91.<n>Our central finding is a diagnostic/prescriptive asymmetry: ARGUS reliably identifies what went wrong, but is perceived as less reliable or trustworthy at specifying what to do next.
  arXiv  Detail & Related papers  (2026-08-24T10:48:21Z)
• OpenCodeReview: Determinism over Non-Determinism for Cost-Effective Agent-Based Code Review [14.707064801983172]
  We introduce OpenCodeReview, built on deterministic engineering for uncertain agents.<n>On AACR-Bench (200 real-world PRs, 10 languages, 1,505 expert-verified comments), OpenCodeReview outperforms mainstream coding agents.
  arXiv  Detail & Related papers  (2026-08-10T08:43:16Z)
• LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent [60.27239351179265]
  LOOPSBENCH is a benchmark for loop engineering in coding agent evaluation.<n>It comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains.<n>Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations.
  arXiv  Detail & Related papers  (2026-07-31T20:18:25Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• The Prover Is the Judge: Verified Security Software from AI Coding Agents in Ada/SPARK [0.0]
  In a verifier-driven loop, AI agents wrote and verified bare-metal security software in Ada/SPARK spanning classical and post-quantum cryptography, TLS 1.3, IKEv2, X.509, and a Matrix client.<n>GNATprove discharged 49,280 proof obligations, established functional correctness for selected primitives, and proved the absence of run-time errors for the rest.<n>We report where each layer caught faults and draw the central lesson: what an agent can be trusted to establish is bounded by the strength of its feedback.
  arXiv  Detail & Related papers  (2026-07-15T20:09:05Z)
• Log-Insight: Automating Microservice Incident Diagnosis via Neuro-Symbolic Log Analysis [4.172239714799967]
  We present Log-Insight, an automated incident-diagnosis system deployed in production at Huawei.<n>Our six-stage pipeline reduces millions of raw events by 1,000-7,000x while preserving statistically significant failure signals.<n>Log-Insight achieves MRR = 0.790, returning the correct root cause within the top-3 hypotheses in over 90% of runs in under a minute of latency.
  arXiv  Detail & Related papers  (2026-07-09T14:25:31Z)
• Multi-Agent Systems for Root Cause Analysis in Microservices [58.95503998127531]
  LATS-RCA is a multi-agent framework for root cause analysis in MSS.<n> Reflection scores are used to guide the search toward the most likely root cause.<n>We evaluate LATS-RCA on the open-source industrial MSS, Light-OAuth2.
  arXiv  Detail & Related papers  (2026-05-05T08:39:42Z)
• Synthesizing Multi-Agent Harnesses for Vulnerability Discovery [8.518689779459974]
  LLM agents have begun to find real security vulnerabilities that human auditors and automated fuzzers missed for decades.<n>In practice the work is split among several agents, wired together by a harness: the program that fixes which roles exist, how they pass information, which tools each may call, and how retries are coordinated.<n>AgentFlow addresses both limitations with a typed graph DSL whose search space jointly covers agent roles, prompts, tools, communication topology, and coordination protocol.
  arXiv  Detail & Related papers  (2026-04-22T17:27:40Z)
• SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
  SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
  arXiv  Detail & Related papers  (2026-04-22T17:08:19Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• MALCDF: A Distributed Multi-Agent LLM Framework for Real-Time Cyber [0.0]
  Four large language model (LLM) agents-Detection, Intelligence, Response, and Analysis-work together in real time.<n>Agents communicate over a Secure Communication Layer (SCL) with encrypted, ontology-aligned messages, and produce audit-friendly outputs.<n>MALCDF reaches 90.0% detection accuracy, 85.7% F1-score, and 9.1% false-positive rate, with 6.8s average per-event latency.
  arXiv  Detail & Related papers  (2025-12-16T19:08:12Z)
• COCA: Generative Root Cause Analysis for Distributed Systems with Code Knowledge [33.88776152946711]
  We propose COCA, a code knowledge enhanced root cause analysis approach for issue reports.<n>Based on the data within issue reports, COCA intelligently extracts relevant code snippets and reconstructs execution paths.<n>Our evaluation on datasets from five real-world distributed systems demonstrates that COCA significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2025-03-29T11:56:48Z)
• Automatic Root Cause Analysis via Large Language Models for Cloud Incidents [51.94361026233668]
  We introduce RCACopilot, an on-call system empowered by a large language model for automating root cause analysis of cloud incidents. RCACopilot matches incoming incidents to corresponding incident handlers based on their alert types, aggregates the critical runtime diagnostic information, predicts the incident's root cause category, and provides an explanatory narrative. We evaluate RCACopilot using a real-world dataset consisting of a year's worth of incidents from Microsoft.
  arXiv  Detail & Related papers  (2023-05-25T06:44:50Z)
• WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition [59.975078145303605]
  We propose a novel framework, namely WNARS, using hybrid CTC-attention AED models and weighted finite-state transducers. On the AISHELL-1 task, our WNARS achieves a character error rate of 5.22% with 640ms latency, to the best of our knowledge, which is the state-of-the-art performance for online ASR.
  arXiv  Detail & Related papers  (2021-04-08T07:56:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.