Related papers: CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

URL: http://arxiv.org/abs/2608.12002v1
Date: Wed, 12 Aug 2026 12:37:02 GMT
Title: CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations
Abstract summary: We introduce CTBench, a benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer.<n>CTBench focuses on root cause analysis and path restoration.<n> Experiments show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis.
Score: 16.752373220559733
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.

Related papers

A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline [4.751533658796929]
We present an empirical study of general-purpose coding agents on a fly optogenetics data-to-discovery pipeline.<n>We assess agents on tasks substantially larger than existing benchmarks, datasets orders of magnitude bigger, and evaluation criteria grounded in domain expert standards.<n>We show that agents can solve several individual pipeline stages, suggesting stage-level automation is tractable.
arXiv Detail & Related papers (2026-06-05T15:38:18Z)
NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration [0.669087470775851]
We introduce NetAgentBench, a benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization.<n>This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors.
arXiv Detail & Related papers (2026-04-03T05:11:05Z)
Dynamic analysis enhances issue resolution [53.50448142467294]
DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
arXiv Detail & Related papers (2026-03-23T14:48:54Z)
AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems [0.0]
We present AgentTrace, a lightweight causal tracing framework for post-hoc failure diagnosis in deployed multi-agent AI systems.<n>AgentTrace reconstructs causal graphs from execution logs, traces backward from error manifestations, and ranks candidate root causes using interpretable structural and positional signals.<n>Our results suggest that causal tracing provides a practical foundation for improving the reliability and trustworthiness of agentic systems in the wild.
arXiv Detail & Related papers (2026-03-16T00:46:44Z)
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
arXiv Detail & Related papers (2026-03-15T16:13:58Z)
TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces [32.4073751390339]
We propose TraceSIR, a framework for structured analysis and reporting of agentic execution traces.<n>TraceSIR coordinates three specialized agents: StructureAgent, InsightAgent, and ReportAgent.<n>Experiments show that TraceSIR consistently produces coherent, informative, and actionable reports.
arXiv Detail & Related papers (2026-02-28T12:33:24Z)
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
arXiv Detail & Related papers (2025-10-13T22:22:28Z)
AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering [51.07491603393163]
tAgent is a framework that formulates multi-agent QA as a knowledge-graph-guided routing problem supervised by empirical performance signals.<n>By leveraging soft supervision and weighted aggregation of agent outputs, Agent learns principled collaboration schemes that capture the complementary strengths of diverse agents.
arXiv Detail & Related papers (2025-10-06T23:20:49Z)
Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
arXiv Detail & Related papers (2025-10-06T05:03:57Z)
Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
arXiv Detail & Related papers (2025-10-01T07:59:03Z)
Exploring LLM-based Agents for Root Cause Analysis [17.053079105858497]
Root cause analysis (RCA) is a critical part of the incident management process. Large Language Models (LLMs) have been used to perform RCA, but are not able to collect additional diagnostic information. We present an evaluation of a ReAct agent equipped with retrieval tools, on an out-of-distribution dataset of production incidents collected at Microsoft.
arXiv Detail & Related papers (2024-03-07T00:44:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.