RoboTwin-Phys: Do WAMs and VLAs Understand the Physical World?
Abstract Overview
RoboTwin-Phys introduces physical-condition diversity as an explicit evaluation axis for robot manipulation, extending the 50-task RoboTwin-2.0 platform without altering the task set itself. The benchmark continuously varies 13 episode-level physical attributes within physically plausible, task-aware ranges and retains only instances verified as feasible by an expert planner. The authors also release over 5,000 expert demonstrations annotated with ground-truth 13-dimensional physical parameters to support physical-attribute estimation, condition-aware modeling, and physics-conditioned policy learning. Evaluations of representative world-action models (WAMs) and vision-language-action models (VLAs) show that robustness under standard visual and layout randomization does not reliably transfer to variation in physical conditions.
Novelty
RoboTwin-Phys treats physical-condition diversity—encompassing mass, friction, center of mass, damping, and sensing configuration—as a first-class benchmark dimension for bimanual manipulation. It combines this protocol with per-episode ground-truth physical annotations and task-aware validity calibration, contrasting with prior benchmarks that keep underlying physics nominal while randomizing appearance and layout.
Results
Across five evaluated WAM and VLA models, success rates drop sharply in the Physical Random setting compared to Clean and Official Random baselines. For example, Fast-WAM drops from 91.78% in Official Random to 44.24% in Physical Random, while Galaxea-VLA drops from 92.80% to 37.82%. Task-level analyses show structured performance patterns across model families, with contact- and dynamics-heavy tasks exhibiting consistent vulnerability to physical variation.
Key Points
- RoboTwin-Phys continuously samples 13 physical attributes at the episode level, applying task-specific calibrated ranges and expert planner feasibility checks to ensure valid benchmark instances.
- The accompanying dataset provides over 5,000 expert demonstrations with ground-truth 13-dimensional physical parameters while maintaining compatibility with the standard RoboTwin data format.
- Evaluations across representative WAMs and VLAs demonstrate that strong performance under visual and layout randomization degrades substantially under physical-condition variation.