vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.13966v1
• Date: Sat, 14 Mar 2026 14:38:53 GMT
• Title: vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
• Abstract summary: VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository.<n>We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution.<n>A complete evaluation requires only two commands: vla eval serve and vla eval run.
• Score: 58.633451339058986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Language Action VLA models are typically evaluated using per benchmark scripts maintained independently by each model repository, leading to duplicated code, dependency conflicts, and underspecified protocols. We present vla eval, an open source evaluation harness that decouples model inference from benchmark execution through a WebSocket msgpack protocol with Docker based environment isolation. Models integrate once by implementing a single predict() method; benchmarks integrate once via a four method interface; the full cross evaluation matrix works automatically. A complete evaluation requires only two commands: vla eval serve and vla eval run. The framework supports 13 simulation benchmarks and six model servers. Parallel evaluation via episode sharding and batch inference achieves a 47x throughput improvement, completing 2000 LIBERO episodes in about 18 minutes. Using this infrastructure, we conduct a reproducibility audit of a published VLA model across three benchmarks, finding that all three closely reproduce published values while uncovering undocumented requirements ambiguous termination semantics and hidden normalization statistics that can silently distort results. We additionally release a VLA leaderboard aggregating 657 published results across 17 benchmarks. Framework, evaluation configs, and all reproduction results are publicly available.

Related papers

• A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation [8.788947389563083]
  OmniEvaluator connects inference engines and curated evaluation libraries at a higher level.<n>Every run is recorded as an artifact capturing the full configuration for exact reproduction.<n>A built-in verifier, small enough to run on CPU, keeps its score stable across engines.
  arXiv  Detail & Related papers  (2026-09-01T14:37:57Z)
• Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models [74.1961755473754]
  Existing evaluation methods rely on rigid pipelines that overlook specific user needs and provide numerical results without clear explanations.<n>We propose the Evaluation Agent framework, which employs human-like strategies for efficient, dynamic, multi-round evaluations.<n>We show that Evaluation Agent reduces evaluation time to 10% of traditional methods while delivering comparable results.
  arXiv  Detail & Related papers  (2026-08-10T14:42:10Z)
• What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations) [1.0499611180329804]
  We compare two modalities, ChatGPT's chat UI and OpenAI's API, with and without web search enabled.<n>For instance, chat UI responses were less accurate than API responses on both benchmarks with search disabled.<n>We argue that AI safety evaluations should systematically account for modality, multi-run consistency, search conditions, and response-level behaviors.
  arXiv  Detail & Related papers  (2026-08-06T15:58:32Z)
• The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality [70.45240108873001]
  The FACTS Leaderboard is an online leaderboard suite that comprehensively evaluates the ability of language models to generate factually accurate text.<n>The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards.
  arXiv  Detail & Related papers  (2025-12-11T16:35:14Z)
• fev-bench: A Realistic Benchmark for Time Series Forecasting [19.931138737002215]
  Existing benchmarks often have narrow domain coverage or overlook important real-world settings.<n>We propose fevbench, a benchmark comprising 100 forecasting tasks across seven domains.<n> fev-bench employs principled aggregation methods with bootstrapped confidence intervals to report model performance.
  arXiv  Detail & Related papers  (2025-09-30T16:17:18Z)
• A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench [18.149327897427234]
  We present clembench, which has been in continuous development since 2023 and has in its latest release been optimized for ease of general use.<n>We describe how it can be used to benchmark one's own models (using a provided set of benchmark game instances in English)
  arXiv  Detail & Related papers  (2025-07-11T11:16:01Z)
• SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits [31.98028879922584]
  We introduce SummExecEdit, a novel pipeline and benchmark to assess models on their ability to both detect factual errors and provide accurate explanations.<n>The top-performing model, Claude3-Opus, achieves a joint detection and explanation score of only 0.49 in our benchmark.<n>We identify four primary types of explanation errors, with 45.4% of them involving a focus on completely unrelated parts of the summary.
  arXiv  Detail & Related papers  (2024-12-17T23:26:44Z)
• VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation [16.889939234103153]
  We propose to variabilize benchmarks and evaluate language models dynamically. Specifically, we extract variables from each test case and define a value range for each variable. For each evaluation, we sample new values from these value ranges to create unique test cases, thus ensuring a fresh evaluation each time.
  arXiv  Detail & Related papers  (2024-06-25T16:13:53Z)
• Perception Test: A Diagnostic Benchmark for Multimodal Video Models [78.64546291816117]
  We propose a novel multimodal video benchmark to evaluate the perception and reasoning skills of pre-trained multimodal models. The Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  arXiv  Detail & Related papers  (2023-05-23T07:54:37Z)
• COLO: A Contrastive Learning based Re-ranking Framework for One-Stage Summarization [84.70895015194188]
  We propose a Contrastive Learning based re-ranking framework for one-stage summarization called COLO. COLO boosts the extractive and abstractive results of one-stage systems on CNN/DailyMail benchmark to 44.58 and 46.33 ROUGE-1 score.
  arXiv  Detail & Related papers  (2022-09-29T06:11:21Z)
• SummEval: Re-evaluating Summarization Evaluation [169.622515287256]
  We re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion. We benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics. We assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset.
  arXiv  Detail & Related papers  (2020-07-24T16:25:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.