A2Eval: Agentic and Automated Evaluation for Embodied Brain

• URL: http://arxiv.org/abs/2602.01640v1
• Date: Mon, 02 Feb 2026 04:55:27 GMT
• Title: A2Eval: Agentic and Automated Evaluation for Embodied Brain
• Abstract summary: Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks.<n>Agentic Automatic Evaluation (A2Eval) is the first agentic framework that automates benchmark curation and evaluation through two collaborative agents.<n> Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup.
• Score: 26.357063836707223
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon.

Related papers

• Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation [141.29138788636013]
  We develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents.<n>We conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks.<n>We introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T16:26:20Z)
• Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation [23.628754736526975]
  We introduce CRATE, a novel two-stage VLM-as-judge framework for automated mobile agent evaluation.<n>We further extend CRATE to CRATE-S for operational safety assessment.<n>CRATE achieves an F1-score of 0.833 on AndroidWorld, while CRATE-S reaches an F1-score of 0.697 on MobileRisk.
  arXiv  Detail & Related papers  (2026-08-21T07:16:36Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators [17.44053709945519]
  Self-improving agents are state-of-the-art (SOTA) on agentic coding benchmarks and have recently been extended to general domains.<n>We introduce the Red Queen Godel Machine (RQGM), an evolutionary framework for self-improvement under non-stationary utilities.<n>We show that the RQGM improves test pass rate over the prior SOTA by adding a complementary agent-as-a-judge code-review signal.<n>We then turn to scientific paper writing and reviewing, and Olympiad-level proof writing and grading, where the RQGM improves performance over prior self-improving agents
  arXiv  Detail & Related papers  (2026-06-24T18:38:26Z)
• AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility [104.46861849039357]
  Agent systems are advancing quickly across domains, but their evaluation remains fragmented.<n>The root problem is the lack of an open, agent-agnostic assessment interface.<n>We advocate Agentified Agent Assessment (AAA), where evaluation is performed by judge agents and all participants interact through standardized protocols.
  arXiv  Detail & Related papers  (2026-06-11T17:23:54Z)
• WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agents [53.84430233754606]
  We introduce WMAttack, an automated attack-search framework for adversarial evaluation of world-model agents.<n> WMAttack formulates robustness evaluation as a finite-budget search over attack configurations.<n>We show that WMAttack consistently discovers stronger attacks than the evaluated baselines.
  arXiv  Detail & Related papers  (2026-05-22T04:31:09Z)
• An Empirical Study of Automating Agent Evaluation [14.239299198848764]
  We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline.<n>EvalAgent encodes evaluation domain expertise as evaluation skills.<n>We show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches.
  arXiv  Detail & Related papers  (2026-05-12T01:06:34Z)
• Efficient Benchmarking of AI Agents [0.0]
  We study whether small task subsets can preserve agent rankings at substantially lower cost.<n>We find that absolute score prediction degrades under scaffold-driven distribution shift.<n>We propose an optimization-free protocol: evaluate new agents only on tasks with intermediate historical pass rates.
  arXiv  Detail & Related papers  (2026-03-24T22:17:11Z)
• ReLE: A Scalable System and Structured Benchmark for Diagnosing Capability Anisotropy in Chinese LLMs [37.23311145049677]
  We present ReLE, a scalable system designed to diagnose Capability Anisotropy.<n>We evaluate 304 models across a Domain $times$ Capability Symbolic matrix comprising 207,843 samples.
  arXiv  Detail & Related papers  (2026-01-24T09:57:59Z)
• AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment [12.9569411072262]
  AutoBench is a fully automated and self-sustaining framework for evaluating Large Language Models (LLMs)<n>This paper provides a rigorous scientific validation of the AutoBench methodology, originally developed as an open-source project by eZecute S.R.L.
  arXiv  Detail & Related papers  (2025-10-26T09:20:39Z)
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains [97.5573252172065]
  We train a family of Automatic Reasoning Evaluators (FARE) with a simple iterative rejection-sampling supervised finetuning approach.<n>FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators.<n>As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH.
  arXiv  Detail & Related papers  (2025-10-20T17:52:06Z)
• NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimation [58.30936615525824]
  We present NAIPv2, a debiased and efficient framework for paper quality estimation.<n> NAIPv2 employs pairwise learning within domain-year groups to reduce inconsistencies in reviewer ratings.<n>It is trained on pairwise comparisons but enabling efficient pointwise prediction at deployment.
  arXiv  Detail & Related papers  (2025-09-29T17:59:23Z)
• Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance? [2.010294990327175]
  Current AI evaluation practices depend heavily on established benchmarks.<n>This research addresses the urgent need to quantify this "benchmark-regulation gap"<n>Our findings reveal a profound misalignment: the evaluation ecosystem dedicates the vast majority of its focus to a narrow set of behavioral propensities.
  arXiv  Detail & Related papers  (2025-08-07T15:03:39Z)
• The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models [53.12387628636912]
  We propose an automatic evaluation framework that is validated against human annotations.<n>This approach was originally developed for the TREC Question Answering (QA) Track in 2003.<n>We observe strong agreement at the run level between scores derived from fully automatic nugget evaluation and human-based variants.
  arXiv  Detail & Related papers  (2025-04-21T12:55:06Z)
• AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents [5.995751996623217]
  We propose AutoEval, an evaluation framework which tests mobile agents without any manual effort.<n>Our approach designs a UI state change representation which can be used to automatically generate task reward signals.<n>We also evaluate state-of-the-art mobile agents using our framework, providing insights into their performance and limitations.
  arXiv  Detail & Related papers  (2025-03-04T08:44:30Z)
• Early-Exit and Instant Confidence Translation Quality Estimation [46.13074343863971]
  We tackle two connected challenges: (1) reducing the cost of quality estimation at scale, and (2) developing an inexpensive uncertainty estimation method for quality estimation.<n>To address the latter, we introduce Instant Confidence COMET, an uncertainty-aware quality estimation model that matches the performance of previous approaches at a fraction of their costs.<n>We extend this to Early-Exit COMET, a quality estimation model that can compute quality scores and associated confidences already at early model layers, allowing us to early-exit computations and reduce evaluation costs.
  arXiv  Detail & Related papers  (2025-02-20T10:27:13Z)
• Autonomous Evaluation and Refinement of Digital Agents [57.12281122337407]
  We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control. We validate the performance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics.
  arXiv  Detail & Related papers  (2024-04-09T17:25:47Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework [91.94389491920309]
  Textual adversarial attacks can discover models' weaknesses by adding semantic-preserved but misleading perturbations to the inputs. The existing practice of robustness evaluation may exhibit issues of incomprehensive evaluation, impractical evaluation protocol, and invalid adversarial samples. We set up a unified automatic robustness evaluation framework, shifting towards model-centric evaluation to exploit the advantages of adversarial attacks.
  arXiv  Detail & Related papers  (2023-05-29T14:55:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.