NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration

• URL: http://arxiv.org/abs/2604.09678v1
• Date: Fri, 03 Apr 2026 05:11:05 GMT
• Title: NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration
• Abstract summary: We introduce NetAgentBench, a benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization.<n>This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors.
• Score: 0.669087470775851
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As agentic network management gains popularity, there is a critical need for evaluation frameworks that transcend static, one-shot testing. To address this, we introduce NetAgentBench, a dynamic benchmark that evaluates agent interactions through a Finite State Machine (FSM) formalization guaranteeing determinism, correctness, and bounded execution. This provides the networking landscape with a rigorous foundation to measure complex, multi-turn operational behaviors. Our empirical evaluation of four state-of-the-art LLM agents through diverse network configuration tasks reveals stark deficiencies: while agents can solve basic tasks, they suffer severe exploration meltdowns and coherence collapse during expert-level configurations. Ultimately, NetAgentBench demonstrates that systematically evaluating multi-turn behavioral stability is an indispensable step toward realizing trustworthy, fully autonomous networks.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration [14.126948395559987]
  We present NetConfArena, an executable benchmark for evaluating large language model (LLM) agents in closed-loop network configuration.<n>The benchmark relies on an LLM-assisted, emulation-grounded pipeline, which converts human-oriented network materials into reusable parameterized task templates.<n>We evaluate representative LLM agents on 480 task instances instantiated from 96 protocol-focused task templates, and show that failures are not limited to command errors.
  arXiv  Detail & Related papers  (2026-08-24T12:26:37Z)
• AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities [55.23913617185508]
  Agent is an open-source, lightweight, and infrastructure for evaluating Large Language Models (LLMs)-based agents.<n>It organizes the evaluation process around three independent components, namely Benchmark, Harness and Environment.<n>It features a fault-tolerant runtime and comprehensive trajectory analysis tools to transparently diagnose failure modes like reward-hacking.
  arXiv  Detail & Related papers  (2026-07-15T11:11:17Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions [78.49000936275773]
  We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions.<n>We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation.<n>We propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking.
  arXiv  Detail & Related papers  (2026-05-25T11:09:22Z)
• Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI [45.21562889170875]
  We propose a systematic evaluation paradigm that characterizes agent trustworthiness over a scenario manifold.<n>The framework integrates four complementary components: static cognitive and policy analysis, (ii) interactive sandbox simulation, (iii) social-ethical alignment assessment, and (iv) a distribution-aware representative sampling engine.
  arXiv  Detail & Related papers  (2026-03-16T08:51:33Z)
• AgentTrace: A Structured Logging Framework for Agent System Observability [0.0]
  AgentTrace is a dynamic observability and telemetry framework designed to fill this gap.<n>Unlike traditional logging systems, AgentTrace emphasizes continuous, introspectable trace capture.<n>Our research highlights how AgentTrace can enable more reliable agent deployment, fine-grained risk analysis, and informed trust calibration.
  arXiv  Detail & Related papers  (2026-02-07T04:04:59Z)
• Multi-Agent Collaborative Intrusion Detection for Low-Altitude Economy IoT: An LLM-Enhanced Agentic AI Framework [60.72591149679355]
  The rapid expansion of low-altitude economy Internet of Things (LAE-IoT) networks has created unprecedented security challenges.<n>Traditional intrusion detection systems fail to tackle the unique characteristics of aerial IoT environments.<n>We introduce a large language model (LLM)-enabled agentic AI framework for enhancing intrusion detection in LAE-IoT networks.
  arXiv  Detail & Related papers  (2026-01-25T12:47:25Z)
• AgentCompass: Towards Reliable Evaluation of Agentic Workflows in Production [4.031479494871582]
  We present Agent, the first evaluation framework designed specifically for post-deployment monitoring and reasoning of agentic pipeline.<n>Agent achieves state-of-the-art results on key metrics, while uncovering critical issues missed in human annotations.
  arXiv  Detail & Related papers  (2025-09-18T05:59:04Z)
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
  We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
  arXiv  Detail & Related papers  (2025-07-08T16:18:54Z)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites.<n>We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions.<n>Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents [74.16170899755281]
  We introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents.<n>AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit.<n>This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront.
  arXiv  Detail & Related papers  (2024-01-24T01:51:00Z)
• Reputation-driven Decision-making in Networks of Stochastic Agents [0.0]
  We propose a Markov Decision Process-derived framework, called RepNet-MDP, tailored to domains in which agent reputation is a key driver of the interactions between agents. In a series of experiments, RepNet agents are shown to be able to adapt their own behavior to the past behavior and reliability of the remaining agents of the network.
  arXiv  Detail & Related papers  (2020-08-26T20:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.