Anchor: Mitigating Artifact Drift in Agent Benchmark Generation

• URL: http://arxiv.org/abs/2605.26321v1
• Date: Mon, 25 May 2026 20:44:17 GMT
• Title: Anchor: Mitigating Artifact Drift in Agent Benchmark Generation
• Abstract summary: We introduce Anchor, a task-generation pipeline that formalizes domain experts' specifications into constraint optimization programs.<n>With Anchor, altering parameters yields new tasks with controlled difficulty and known optimal solutions.<n>We apply Anchor to produce ERP-Bench: a benchmark of 300 long-horizon tasks spanning procurement and manufacturing in a production-grade ERP system.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents are beginning to complete valuable, long-horizon business operations tasks, but training and evaluation environments for enterprise work still struggle to balance realism, verifiability, and scale. Environment and task creation frequently suffers from a failure mode we call artifact drift: when instructions, environments, oracles, and verifiers are created by loosely coupled processes, they frequently disagree on what a task requires, producing environments that are unsolvable, reward-hackable, or inconsistent. We introduce Anchor, a task-generation pipeline that formalizes domain experts' specifications of business workflows into constraint optimization programs. From a single parametric specification, the pipeline jointly produces a natural-language instruction, environment configuration, solver-certified ground-truth solution, and state-based verifier. With Anchor, altering parameters yields new tasks with controlled difficulty and known optimal solutions, producing harness-agnostic environments whose rewards depend solely on end-state business correctness. We apply Anchor to produce ERP-Bench: a benchmark of 300 long-horizon tasks spanning procurement and manufacturing workflows in a production-grade ERP system. We find that generation parameters predict realized difficulty, and that frontier models satisfy explicit task constraints in 26.1% of trials but reach a fully optimal solution in only 17.4% of trials. Overall, we show that Anchor and ERP-Bench offer a concrete recipe for building auditable evaluation environments for economically valuable agent work. We release the task generator and ERP-Bench dataset at erpbench.ai

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science [39.146761527401424]
  GRACE-DS is a Reward-guided Agent Correction Environment in Data Science for pre-deployment evaluation of LLM-powered AutoML agents.<n>It exposes agents to realistic workflow stages, from planning and data inspection through feature engineering, model development, validation, and code repair to final submission.<n>These results establish GRACE-DS as a robust platform for assessing the capacity of LLM-based AutoML agents to execute machine learning under production-like conditions.
  arXiv  Detail & Related papers  (2026-06-14T19:58:06Z)
• TAPE: Tool-Guided Adaptive Planning and Constrained Execution in Language Model Agents [16.59223734824801]
  We propose Tool-guided Adaptive Planning with constrained Execution (TAPE)<n>TAPE planning capability by aggregating multiple plans into a graph and employing an external solver to identify a feasible path.<n>During execution, TAPE employs constrained decoding to reduce sampling noise, while adaptively re-planning whenever environmental feedback deviates from the intended state.
  arXiv  Detail & Related papers  (2026-02-23T09:19:56Z)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
  Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
  arXiv  Detail & Related papers  (2026-01-17T01:29:30Z)
• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem [90.17610617854247]
  We introduce the Agentic Learning Ecosystem (ALE), a foundational infrastructure that optimize the production pipeline for agentic model.<n>ALE consists of three components: ROLL, a post-training framework for weight optimization; ROCK, a sandbox environment manager for trajectory generation; and iFlow CLI, an agent framework for efficient context engineering.<n>We release ROME, an open-source agent grounded by ALE and trained on over one million trajectories.
  arXiv  Detail & Related papers  (2025-12-31T14:03:39Z)
• AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning [71.4322853508083]
  Conducting reinforcement learning in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents.<n>Previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth.<n>We propose a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks.
  arXiv  Detail & Related papers  (2025-12-28T09:43:11Z)
• DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle [41.576417987200074]
  Real-world enterprise data intelligence encompass data engineering that turns raw sources into analytical-ready tables and data analysis that convert those tables into decision-oriented insights.<n>We introduce DAComp, a benchmark of 210 tasks that mirrors these complex capabilities.
  arXiv  Detail & Related papers  (2025-12-03T23:21:28Z)
• Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents [71.85020581835042]
  Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck.<n>Existing benchmarks assess only end-to-end build/test success, obscuring where and why agents succeed or fail.<n>We introduce Enconda-bench, which provides process-level trajectory assessment of fine-grained agent capabilities during environment setup-planning.
  arXiv  Detail & Related papers  (2025-10-29T16:59:07Z)
• SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environments [2.184775414778289]
  We introduce setupbench, a benchmark that isolates the environment-bootstrap skill.<n>Our tasks span seven language ecosystems, five database engines, and multi-service orchestration scenarios.<n>We find low success rates across task categories, with particular challenges in repository setup (38.9-57.4%) and local database configuration (20.0-53.3%)
  arXiv  Detail & Related papers  (2025-07-11T22:45:07Z)
• Uncovering Systemic and Environment Errors in Autonomous Systems Using Differential Testing [9.625308787676286]
  We introduce AIProbe, a novel black-box testing technique that applies differential testing to attribute undesirable agent behaviors.<n>AIProbe identifies whether failures are due to errors in the agent's model or policy, or due to unsolvable task conditions.<n>Our evaluation shows that AIProbe significantly outperforms state-of-the-art techniques in detecting both total and unique errors.
  arXiv  Detail & Related papers  (2025-07-05T02:50:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.