Robots Need More than VLA and World Models

• URL: http://arxiv.org/abs/2606.06556v1
• Date: Thu, 04 Jun 2026 10:43:14 GMT
• Title: Robots Need More than VLA and World Models
• Abstract summary: Generalist robot intelligence is often framed as a policy-scaling problem.<n>In this paper we argue that this framing is incomplete.<n>We identify four missing components for the next generation of robotics.
• Score: 38.16463528269755
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Generalist robot intelligence is often framed as a policy-scaling problem: collect more robot demonstrations, train larger Vision-Language-Action (VLA) models, and expect broader generalisation. In this position paper, we argue that this framing is incomplete. The central bottleneck is not only policy learning, but the absence of mechanisms that convert the world's abundant unstructured behavioural data into grounded robot supervision. Human motion, internet video, simulation rollouts, and interactive demonstrations contain rich information about tasks, goals, contacts, failures, and physical constraints, yet most of this information is not directly usable by robot policies because it lacks embodiment-specific action labels, task semantics, and reward structure. We identify four missing components for the next generation of robotics: data interfaces for autolabelling unstructured behaviour, embodiment interfaces for retargeting human motion to robot actions, world-model interfaces for physics-grounded 3D reasoning, and reward interfaces for inferring task progress and success from video and language. We survey recent progress in robot foundation models, cross-embodiment datasets, learning from video, world models, and reward modelling, and propose a research agenda for building robotics systems that can learn not only from robot demonstrations, but from the broader physical world.

Related papers

• H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models [56.60098789696351]
  H2R-Bench is a benchmark for evaluating cross-embodiment human-to-robot manipulation video generation.<n>Each benchmark instance contains a human demonstration video, target embodiment constraints, and source-grounded annotations.<n>We benchmark eleven state-of-the-art video generation models across six manipulation families and two robot embodiments.
  arXiv  Detail & Related papers  (2026-08-13T10:14:33Z)
• OSCAR: Omni-Embodiment Action-Conditioned World Model for Robotics [6.134835623350618]
  We present a precise action-conditioned video world model that generalizes across different robot embodiments and enables robot policy evaluation.<n>Our model achieves significant improvement on action following, appearance quality, and motion consistency, compared to existing baselines.<n>We further deploy OSCAR to evaluate robot policies from RoboArena.
  arXiv  Detail & Related papers  (2026-06-03T05:16:41Z)
• Large Video Planner Enables Generalizable Robot Control [117.49024534548319]
  General-purpose robots require decision-making models that generalize across diverse tasks and environments.<n>Recent works build robot foundation models by extending multimodal large language models (LMs) with action outputs, creating vision--action (VLA) systems.<n>We explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models.
  arXiv  Detail & Related papers  (2025-12-17T18:35:54Z)
• RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video [56.9581053843815]
  We introduce RobotSeg, a foundation model for robot segmentation in image and video.<n>It addresses the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations.<n>It achieves state-of-the-art performance on both images and videos.
  arXiv  Detail & Related papers  (2025-11-28T07:51:02Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Action-Free Reasoning for Policy Generalization [23.34099331171177]
  Reasoning through Action-free Data (RAD) learns from both robot demonstration data and action-free human video data.<n>RAD enables effective transfer across the embodiment gap, allowing robots to perform tasks seen only in action-free data.<n>We will release a new dataset of 3,377 human-hand demonstrations with reasoning annotations compatible with the Bridge V2 benchmark.
  arXiv  Detail & Related papers  (2025-02-06T02:43:23Z)
• $π_0$: A Vision-Language-Action Flow Model for General Robot Control [77.32743739202543]
  We propose a novel flow matching architecture built on top of a pre-trained vision-language model (VLM) to inherit Internet-scale semantic knowledge. We evaluate our model in terms of its ability to perform tasks in zero shot after pre-training, follow language instructions from people, and its ability to acquire new skills via fine-tuning.
  arXiv  Detail & Related papers  (2024-10-31T17:22:30Z)
• GRAPPA: Generalizing and Adapting Robot Policies via Online Agentic Guidance [15.774237279917594]
  We propose an agentic framework for robot self-guidance and self-improvement.<n>Our framework iteratively grounds a base robot policy to relevant objects in the environment.<n>We demonstrate that our approach can effectively guide manipulation policies to achieve significantly higher success rates.
  arXiv  Detail & Related papers  (2024-10-09T02:00:37Z)
• Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts [21.249837293326497]
  Generalizable reward function is central to reinforcement learning and planning for robots. This paper transfers video-language models with robust generalization into a language-conditioned reward function. Our model shows outstanding generalization to new environments and new instructions for robot planning and reinforcement learning.
  arXiv  Detail & Related papers  (2024-07-20T13:22:59Z)
• RT-1: Robotics Transformer for Real-World Control at Scale [98.09428483862165]
  We present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.
  arXiv  Detail & Related papers  (2022-12-13T18:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.