DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents

• URL: http://arxiv.org/abs/2609.14637v2
• Date: Tue, 15 Sep 2026 10:27:42 GMT
• Title: DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents
• Abstract summary: DynSTEER is a dynamic stage-wise framework for agent trajectory evaluation.<n>It segments rollouts at key execution nodes and adapts its multi-tier review strategy based on stage-level results.<n>It compiles a path-tolerant milestone graph from public task views to preserve diverse legal paths without reference leakage.
• Score: 29.982261116765162
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model agents are increasingly deployed for long-horizon task execution, raising a central granularity question for trajectory evaluation: whole-trajectory verification is too coarse to capture concrete failures and their associated evidence in long trajectories, while atomic-step scoring is too fine-grained, noise-sensitive, and computationally expensive. This granularity gap makes a single-reference trajectory paradigm inadequate for assessing the rich space of valid agent execution paths and delays timely feedback and early stopping in long-horizon tasks. To address these issues, we propose DynSTEER, a dynamic stage-wise framework for agent trajectory evaluation. DynSTEER bridges the granularity gap through stage-wise dynamic evaluation that segments rollouts at key execution nodes and adapts its multi-tier review strategy based on stage-level results; it compiles a path-tolerant milestone graph from public task views to preserve diverse legal paths without reference leakage; and it supports terminating unrecoverable agent executions to curb resource waste. Experiments show that DynSTEER improves evaluation discriminability by 85.2\% over native evaluation, separates all model pairs with statistical significance, and saves 45.41\% of execution steps on failed rollouts.

Related papers

• CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents [66.9575676266637]
  Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments.<n>In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution.<n>We develop CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization.
  arXiv  Detail & Related papers  (2026-08-31T01:59:51Z)
• Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems [47.66236392009794]
  Large Language Model(LLM)-based agents have demonstrated exceptional performance across a wide range of complex interactive tasks.<n>We propose textitTrajectory Graph Copilot, a novel framework that acts as a copilot'' for LLM agents by diagnosing potential action errors before they are executed.
  arXiv  Detail & Related papers  (2026-07-29T20:14:43Z)
• Speculative Rollback Correction for Quality-Diverse Web Agent Imitation [43.603850569331975]
  Training interactive web agents through imitation learning from expert trajectories has emerged as a highly effective approach.<n>We propose Speculative Rollback Correction (SRC), a branch-level imitation framework for resettable agent environments.
  arXiv  Detail & Related papers  (2026-06-10T08:56:27Z)
• StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents [67.03593791535786]
  Reinforcement Learning (RL) has become a promising approach for improving GUI Agents in long-horizon, digital environments.<n>To mitigate this issue, recent studies introduce Process Reward Models (PRMs)<n>PRMs provide finer-grained training feedback through global milestone verification or local step-level evaluation.<n>We propose StainFlow, an entity-stain-flow process reward model for GUI Agents.
  arXiv  Detail & Related papers  (2026-06-05T08:17:28Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• Efficient Agent Evaluation via Diversity-Guided User Simulation [7.723545220477047]
  DIVERT is a snapshot-based, coverage-guided user simulation framework for systematic exploration of agent-user interactions.<n>It captures the full agent-environment state at critical decision points and resumes execution from these snapshots.<n>By focusing evaluation on semantically diverse and underexplored trajectories, DIVERT improves both efficiency and coverage.
  arXiv  Detail & Related papers  (2026-04-23T09:41:21Z)
• The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break [69.91172974854422]
  HORIZON is a cross-domain diagnostic benchmark for analyzing long-horizon failure behaviors in large language model (LLM) agents.<n>We evaluate state-of-the-art (SOTA) agents from multiple model families to study horizon-dependent degradation patterns.<n>Our findings offer an initial methodological step toward systematic, cross-domain analysis of long-horizon agent failures.
  arXiv  Detail & Related papers  (2026-04-13T19:11:42Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• Agentic Spatio-Temporal Grounding via Collaborative Reasoning [80.83158605034465]
  Temporal Video Grounding aims to retrieve thetemporal tube of a target object or person in a video given a text query.<n>We propose the Agentic Spatio-Temporal Grounder (ASTG) framework for the task of STVG towards an open-world and training-free scenario.<n>Specifically, two specialized agents SRA (Spatial Reasoning Agent) and TRA (Temporal Reasoning Agent) constructed leveraging on modern Multimoal Large Language Models (MLLMs)<n>Experiments on popular benchmarks demonstrate the superiority of the proposed approach where it outperforms existing weakly-supervised and zero-shot approaches by a margin
  arXiv  Detail & Related papers  (2026-02-10T10:16:27Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• AgentRx: Diagnosing AI Agent Failures from Execution Trajectories [9.61742219198197]
  We release a benchmark of 115 failed trajectories spanning structured API, incident management, and open-ended web/file tasks.<n>Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy.<n>We present AGENTRX, an automated domain-agnostic diagnostic framework that pinpoints the critical failure step in a failed agent trajectory.
  arXiv  Detail & Related papers  (2026-02-02T18:54:07Z)
• Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents [22.781523439717223]
  A proper evaluation of an agent's performance must go beyond the final answer to also assess the problem-solving trajectory.<n>We introduce TRACE, a framework for the multi-dimensional evaluation of tool-augmented LLM agent performance.<n>Our results confirm that TRACE accurately evaluates these complex behaviors in a scalable and cost-effective manner.
  arXiv  Detail & Related papers  (2025-10-03T09:19:15Z)
• CORE: Full-Path Evaluation of LLM Agents Beyond Final State [2.0391237204597368]
  Existing agentic benchmarks often reduce evaluation to a binary judgment of the final state.<n>We propose a framework based on deterministic finite automata that encodes tasks as sets of valid tool-use paths.<n>We introduce CORE, a suite of five metrics, namely Path Correctness, Path Correctness - Kendall's tau Composite, Prefix Criticality, Harmful-Call Rate, and Efficiency.
  arXiv  Detail & Related papers  (2025-09-25T10:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.