Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation

• URL: http://arxiv.org/abs/2608.20797v1
• Date: Fri, 21 Aug 2026 07:16:36 GMT
• Title: Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation
• Abstract summary: We introduce CRATE, a novel two-stage VLM-as-judge framework for automated mobile agent evaluation.<n>We further extend CRATE to CRATE-S for operational safety assessment.<n>CRATE achieves an F1-score of 0.833 on AndroidWorld, while CRATE-S reaches an F1-score of 0.697 on MobileRisk.
• Score: 23.628754736526975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating language-guided mobile agents has recently shifted from rule-based to model-based approaches to achieve scalable and automated assessments. However, existing holistic evaluation paradigms process entire trajectories at once, leading to substantial context overload. Moreover, they primarily focus on task completion while overlooking operational safety. To address these limitations, we introduce CRATE, a novel two-stage VLM-as-judge framework for automated mobile agent evaluation that is compatible with both open- and closed-source models. Leveraging a step-level consequence reasoning mechanism, CRATE independently extracts task-relevant visual clues and infers action-conditioned state changes at each step. The resulting step-level textual evidence is then synthesized through trajectory-level aggregation to deliver an evidence-grounded evaluation of task completion. Building upon this evaluation scheme, we further extend CRATE to CRATE-S for operational safety assessment. Extensive experiments validate the effectiveness and robustness of both CRATE and CRATE-S. Powered by Qwen2.5-VL-72B-Instruct, CRATE achieves an F1-score of 0.833 on AndroidWorld (outperforming SPA-Bench by 20%), while CRATE-S reaches an F1-score of 0.697 on MobileRisk, demonstrating strong alignment with benchmark ground truths. Code is available at https://anonymous.4open.science/r/CRATE-D580.

Related papers

• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model [54.35791816657227]
  We introduce textbfVLA-ATTC', a framework that endows VLA models with adaptive test-time compute.<n>VLA-ATTC employs an uncertainty-based cognitive clutch'' to dynamically transition from reflexive execution to a TTC deliberation phase.<n>On the LIBERO-LONG benchmark, VLA-ATTC reduces the failure rate of the SOTA model PI0.5 by over 50%.
  arXiv  Detail & Related papers  (2026-05-02T02:13:11Z)
• SAGE: A Service Agent Graph-guided Evaluation Benchmark [27.342044311161654]
  We propose SAGE (Service Agent Graph-guided Evaluation), a universal benchmark for automated, dual-axis assessment.<n>SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance.<n>We also observe Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures.
  arXiv  Detail & Related papers  (2026-04-10T12:55:23Z)
• From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation [32.41847293364159]
  PRIMO R1 is a 7B framework that transforms video MLLMs into active "Critics"<n>We leverage outcome-based Reinforcement Learning to incentivize explicit Chain-of-Thought generation for progress estimation.<n>We establish state-of-the-art performance on RoboFail benchmark with 67.0% accuracy, surpassing closed-source models like OpenAI o1 by 6.2%.
  arXiv  Detail & Related papers  (2026-03-16T17:53:28Z)
• How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs [49.61011897610774]
  How2Everything is a framework to evaluate and improve goal-conditioned procedure generation.<n>Our framework includes How2Mine, which mines 351K procedures from 980K web pages across 14 topics.<n>How2Score is an evaluation protocol that uses an LLM judge to detect whether a generation contains any critical failure that would prevent achieving the goal.
  arXiv  Detail & Related papers  (2026-02-09T15:47:14Z)
• A2Eval: Agentic and Automated Evaluation for Embodied Brain [26.357063836707223]
  Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks.<n>Agentic Automatic Evaluation (A2Eval) is the first agentic framework that automates benchmark curation and evaluation through two collaborative agents.<n> Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup.
  arXiv  Detail & Related papers  (2026-02-02T04:55:27Z)
• Trustworthy Evaluation of Robotic Manipulation: A New Benchmark and AutoEval Methods [30.612032540735402]
  We propose a solution that combines the Eval-Actions benchmark and the AutoEval architecture.<n>This dataset is structured around three core supervision signals: Expert Grading (EG), Rank-Guided preferences (RG), and Chain-of-Thought (CoT)<n>Experiments show AutoEval achieves Spearman's Rank Correlation Coefficients (SRCC) of 0.81 and 0.84 under the EG and RG protocols, respectively.
  arXiv  Detail & Related papers  (2026-01-26T17:47:42Z)
• VIPER: Process-aware Evaluation for Generative Video Reasoning [64.86465792516658]
  We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning.<n>Our experiments reveal that state-of-the-art video models achieve only about 20% POC@1.0 and exhibit a significant outcome-hacking.
  arXiv  Detail & Related papers  (2025-12-31T16:31:59Z)
• CORE: Full-Path Evaluation of LLM Agents Beyond Final State [2.0391237204597368]
  Existing agentic benchmarks often reduce evaluation to a binary judgment of the final state.<n>We propose a framework based on deterministic finite automata that encodes tasks as sets of valid tool-use paths.<n>We introduce CORE, a suite of five metrics, namely Path Correctness, Path Correctness - Kendall's tau Composite, Prefix Criticality, Harmful-Call Rate, and Efficiency.
  arXiv  Detail & Related papers  (2025-09-25T10:49:35Z)
• RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
  RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
  arXiv  Detail & Related papers  (2025-05-28T14:55:33Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domains [19.492393243160244]
  Large Language Models (LLMs) are increasingly deployed in highly specialized vertical domains.<n>Existing evaluations for vertical domains typically rely on the labor-intensive construction of static single-turn datasets.<n>We propose TestAgent, a framework for automatic benchmarking and exploratory dynamic evaluation in vertical domains.
  arXiv  Detail & Related papers  (2024-10-15T11:20:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.