Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis

• URL: http://arxiv.org/abs/2606.08881v2
• Date: Thu, 11 Jun 2026 21:42:22 GMT
• Title: Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis
• Abstract summary: Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation.<n>We present a standardized real-world benchmark for evaluating representative VLA and imitation learning policies on the low-cost SO-101 robotic platform.
• Score: 9.340058165047717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted in simulation or on expensive robotic platforms, leaving their robustness on affordable real-world robots largely unexplored. We present a standardized real-world benchmark for evaluating representative VLA and imitation learning policies on the low-cost SO-101 robotic platform. The benchmark comprises four representative manipulation tasks together with unified evaluation protocols, enabling systematic comparison under embodiment uncertainty. Using real-world teleoperated demonstrations, we fine-tune and evaluate $π_{0.5}$, SmolVLA, Wall-X, and ACT directly on the physical platform. Beyond conventional task success rates, the benchmark incorporates a structured failure taxonomy, semantic- and execution-level failure decomposition, and recovery-aware evaluation metrics to characterize policy robustness. Experimental results show that stronger pretrained VLA policies generally outperform the imitation learning baseline, although performance remains highly task-dependent under low-cost robotic deployment conditions. Execution instability emerges as the dominant failure source, while recovery capability varies substantially across architectures. These results highlight the importance of failure and recovery analysis beyond binary task success and establish SO-101 as a practical benchmark for evaluating embodied AI systems under realistic low-cost robotic deployment conditions.

Related papers

• R2S-Eval: Robot Evaluation with Real-to-Sim Calibration via Vision-Language Models [12.45799943639499]
  R2S-Eval is an evaluation pipeline that combines real-to-sim calibration with vision-language model (VLM) preference evaluation.<n>It produces reliable and stable policy conclusions, achieves agreement with human preferences, substantially reduces repeated hardware-operation effort, and reveals behavior-quality differences that are not captured by binary success labels.
  arXiv  Detail & Related papers  (2026-09-03T02:08:16Z)
• RoboSynChallenge: Mastering Real-World Dexterity via Generalizing Synthesized Manipulation Skills [90.48218237655708]
  RoboSynChallenge aims to foster the development of broadly capable, data-efficient, and adaptable manipulation systems.<n>Challenge integrates large-scale synthetic data generation with standardized real-world robotic evaluation.
  arXiv  Detail & Related papers  (2026-08-12T04:29:00Z)
• GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation [73.02743604295246]
  WMBench is a benchmark constructed from real-robot teleoperation data and matched policy rollouts.<n>We analyze 7 video world models, 4 action representation schemes, and over 324,000 simulated policy rollouts paired with real robot executions.<n>Our experiments deliver three core insights: evaluator quality is dominated by long-horizon, action-faithful rollout consistency rather than short-term visual realism.
  arXiv  Detail & Related papers  (2026-07-02T17:02:43Z)
• RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models [90.39703013636868]
  RePO-VLA is a recovery-driven policy optimization framework.<n>It assigns distinct roles to success, recovery, and failure trajectories.<n>It raises adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
  arXiv  Detail & Related papers  (2026-05-10T08:24:05Z)
• Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models [28.981226513192535]
  Recent Vision-Language-Action (VLA) models report impressive success rates on standard robotic benchmarks.<n>Recent evidence suggests a systematic misalignment between standard benchmark success and true embodied reasoning.<n>We introduce BeTTER, a diagnostic Benchmark for Testing True Embodied Reasoning in robotic policies.
  arXiv  Detail & Related papers  (2026-04-20T09:25:30Z)
• ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation [66.02142169323521]
  Vision-Language-Action models and world models have recently emerged as promising paradigms for general-purpose robotic intelligence.<n>Existing benchmarks are largely simulator-centric, which provide controllability but fail to capture the reality gap caused by perception noise.<n>We introduce ManipArena, a standardized evaluation framework designed to bridge simulation and real-world execution.
  arXiv  Detail & Related papers  (2026-03-30T15:06:41Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models [42.89413870143421]
  Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations.<n>Current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity.<n>We propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework.
  arXiv  Detail & Related papers  (2025-11-19T16:52:23Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• RobotArena $\infty$: Scalable Robot Benchmarking via Real-to-Sim Translation [47.79800816696372]
  Real-world testing of manipulation policies is labor-intensive at scale, and difficult to reproduce.<n>Existing simulation benchmarks are similarly limited, as they train and test policies within the same synthetic domains.<n>In this paper, we introduce a new benchmarking framework that overcomes these challenges by shifting VLA evaluation into large-scale simulated augmented environments.
  arXiv  Detail & Related papers  (2025-10-27T17:41:38Z)
• From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation [35.79160868966466]
  We propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning.<n>Our approach combines a hierarchical data pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals.<n>We show that FSD achieves 40.6% success rate in SimplerEnv and 72% success rate across 8 real-world tasks, outperforming the strongest baseline by 30%.
  arXiv  Detail & Related papers  (2025-05-13T13:20:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.