Can AI Agents Detect and Repair Artifact Drift in Network Experiments?

• URL: http://arxiv.org/abs/2609.09849v1
• Date: Wed, 09 Sep 2026 07:58:24 GMT
• Title: Can AI Agents Detect and Repair Artifact Drift in Network Experiments?
• Abstract summary: We call this property artifact integrity.<n>The benchmark contains 52 instances with inconsistencies ranging from direct contradictions to unstated relations spread across several artifacts.<n>We evaluate 23 agent configurations across three general-purpose AI agent runtimes using deterministic scoring.
• Score: 29.12202496097423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, AI agents have evolved into capable assistants that carry out multi-step tasks in digital environments. The network systems community is beginning to explore these capabilities in operational and experimental settings. However, an agent operating in network systems should not be judged solely by whether it completes the immediate task. The experiment record it modifies must also remain trustworthy. We call this property artifact integrity: the record's claims must remain supported by the available evidence, confined to the scope established by that evidence, and traceable through the artifacts that encode their support. To make this property measurable, we introduce NetArtifactBench, which tests whether AI agents can repair inconsistent records derived from public network-system artifacts while preserving claims that remain supported. The benchmark contains 52 instances with injected inconsistencies ranging from direct contradictions to unstated relations spread across several artifacts. We evaluate 23 agent configurations across three general-purpose AI agent runtimes using deterministic scoring. The average contract pass rate is 65.3 % across 5,980 outputs, but no agent runtime exceeds 30 % when repair requires recovering implicit relations and propagating changes across artifacts. These results reveal a sharp boundary between local correction and complete record-level repair. Therefore, we argue that artifact integrity should become a first-class design and evaluation requirement for AI agents operating on network systems.

Related papers

• Can AI Agents Deliver Verifiable Network-Wide Outcomes Across Authority Boundaries? [31.85609823320721]
  We present EvidenceNet, a runtime assurance layer for deciding whether coordinated agent operations have achieved an operator's network intent.<n> Experiments on live routing networks show that post-change state checks recognize successful outcomes that configuration-action records alone cannot establish.
  arXiv  Detail & Related papers  (2026-09-09T13:51:15Z)
• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agents [29.81912218748482]
  Large language model (LLM) agents automate penetration testing through an observation-action loop.<n>Current defenses rely heavily on static, isolated artifacts planted in the environment prior to an attack.<n>AgentSnare dynamically unfolds a decoy environment to continually steer the penetration agent away from the real target.
  arXiv  Detail & Related papers  (2026-07-29T14:56:31Z)
• AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation [27.0859707906163]
  AgentTether is a run-time repair framework that automates post-run diagnosis and guided recovery without modifying the underlying agent or environment.<n>It localizes failure-critical subtrajectories by combining an offline normal-behavior model with a run-local graph detector.<n>It can be deployed as an offline diagnostic-and-guidance tool or as an online repair layer.
  arXiv  Detail & Related papers  (2026-07-07T13:40:31Z)
• Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories [51.22051230894794]
  Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable.<n>We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through expert review.<n>We propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path.
  arXiv  Detail & Related papers  (2026-06-01T10:50:26Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Re-Evaluating EVMBench: Are AI Agents Ready for Smart Contract Security? [10.248746359119625]
  EVMbench is the first large-scale benchmark for AI agents on smart contract security.<n>Its results have fueled expectations that fully automated AI auditing is within reach.<n>These findings challenge the narrative that fully automated AI auditing is imminent.
  arXiv  Detail & Related papers  (2026-03-11T14:07:16Z)
• RIVA: Leveraging LLM Agents for Reliable Configuration Drift Detection [3.494935876363005]
  Existing agentic systems implicitly assume that the tools they invoke always return correct outputs, making them vulnerable to erroneous tool responses.<n>We introduce RIVA, a novel multi-agent system that performs robust IaC verification even when tools produce incorrect or misleading outputs.<n>Our results show that cross-validation of diverse tool calls enables more reliable autonomous infrastructure verification in production cloud environments.
  arXiv  Detail & Related papers  (2026-03-02T19:28:27Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.