World Models for Robotic Manipulation: A Survey

• URL: http://arxiv.org/abs/2606.00113v1
• Date: Wed, 27 May 2026 05:32:17 GMT
• Title: World Models for Robotic Manipulation: A Survey
• Abstract summary: We survey world models for robotic manipulation through three questions.<n>We operationally define a world model as an action-conditioned predictive system.<n>We characterize infrastructure roles including synthetic experience generation, candidate filtering, search-based evaluation, learned environments, and outcome verification.
• Score: 30.864774708923147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robotic manipulation depends on the ability to anticipate how actions reshape objects, contacts, and scene geometry before execution. Learned world models provide this capability by predicting task-relevant future evolution under robot intervention, yet the term now spans latent dynamics models, action-conditioned video generators, three- and four-dimensional scene predictors, physics-informed simulators, and predictive modules inside vision-language-action systems. This breadth has fragmented the literature and obscured the design choices that matter for manipulation. We survey world models for robotic manipulation through three questions: what future representation is predicted, how prediction is connected to action, and when prediction is used in the robot-learning pipeline. We operationally define a world model as an action-conditioned predictive system and distinguish it from perception modules, inverse models, policies, rewards, and value functions. We then organize existing work into five representation families, develop a functional taxonomy that separates integrated prediction-action models from explicit predictive planners, and characterize infrastructure roles including synthetic experience generation, candidate filtering, search-based evaluation, learned environments, and outcome verification. We further map these roles across pretraining, post-training, and inference adaptation, review 34 manipulation datasets, and synthesize evaluation protocols for predictive fidelity, task performance, and simulator reliability. This survey shows that world models are evolving from task-specific dynamics predictors into predictive infrastructure for robot learning, while exposing open challenges in contact modeling, hallucination control, action alignment, and benchmarking under closed-loop use.

Related papers

• World-Action Models for Robot Learning and Control: A Survey [41.45329550202786]
  Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs)<n>This survey provides a robotics-oriented review of WAMs.<n>We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy.<n>We discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference
  arXiv  Detail & Related papers  (2026-09-13T16:35:56Z)
• Predicting Consequences and Reinforcing Navigation Policies with Latent World Models [31.865100130203317]
  We propose a compatibility prediction Latent World Model (LWM) for robot navigation.<n>Our key insight is that spatial proximity correlates with latent feature similarity, enabling action consequences to be evaluated directly in latent space.<n>Our approach significantly outperforms prior world model and imitation learning methods in prediction accuracy, policy learning, and real-world navigation performance.
  arXiv  Detail & Related papers  (2026-08-23T11:58:42Z)
• From World Models to World Action Models: A Concise Tutorial for Robotics [3.9330868250536457]
  This tutorial presents a design-space view of world models as action-conditioned predictive models.<n>The goal of this tutorial is to clarify the conceptual scope of world (action) models and provide a structured taxonomy for embodied prediction and control.
  arXiv  Detail & Related papers  (2026-07-01T11:56:54Z)
• Nano World Models: A Minimalist Implementation of Future Video Prediction [52.36220976002044]
  Nano World Models is a minimalist for future video prediction centered around diffusion forcing.<n>By releasing code, configurations, evaluation scripts, and pretrained checkpoints, Nano World Models aims to provide a compact yet experimental substrate for open, reproducible, and scientific world-model research.
  arXiv  Detail & Related papers  (2026-05-17T22:46:44Z)
• HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models [58.191567345416836]
  World Action Models (WAMs) have emerged as a promising paradigm for robot control by modeling physical dynamics.<n>HarmoWAM fully leverages a world model to unify predictive and reactive control, enabling both general transit and precise manipulation.<n>We construct three training-unseen test environments across six real-world robotic tasks, covering variations in background, position, and object semantics.
  arXiv  Detail & Related papers  (2026-05-11T17:59:56Z)
• Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond [209.35045331678043]
  We introduce a "levels x laws" taxonomy organized along two axes.<n>The first defines three capability levels: L1 Predictor, which learns one-step local transition operators; L2 Simulator, which composes them into multi-step, action-conditioned rollouts that respect domain laws; and L3 Evolver, which autonomously revises its own model when predictions fail against new evidence.<n>We synthesize over 400 works and summarize more than 100 representative systems spanning model-based reinforcement learning, video generation, web and GUI agents, multi-agent social simulation, and AI-driven scientific discovery.
  arXiv  Detail & Related papers  (2026-04-24T17:48:47Z)
• Causal World Modeling for Robot Control [56.31803788587547]
  Video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics.<n>We introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously.<n>We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations.
  arXiv  Detail & Related papers  (2026-01-29T17:07:43Z)
• A Step Toward World Models: A Survey on Robotic Manipulation [58.8419978790227]
  We look at approaches that exhibit the core capabilities of world models through a review of methods in robotic manipulation.<n>We analyze their roles across perception, prediction, and control, identify key challenges and solutions, and distill the core components, capabilities, and functions that a fully realized world model should possess.
  arXiv  Detail & Related papers  (2025-10-31T00:57:24Z)
• Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Models [22.826115023573205]
  We infuse the predictive nature of human manipulation strategies into robot imitation learning.<n>We train a diffusion model to predict future states and compute robot actions that achieve the predicted states.<n>Our framework consistently outperforms state-of-the-art state-to-action mapping policies.
  arXiv  Detail & Related papers  (2025-03-30T01:25:35Z)
• RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing [38.97168020979433]
  We introduce an approach that combines visual and tactile sensing for robotic manipulation by learning a neural, tactile-informed dynamics model. Our proposed framework, RoboPack, employs a recurrent graph neural network to estimate object states. We demonstrate our approach on a real robot equipped with a compliant Soft-Bubble tactile sensor on non-prehensile manipulation and dense packing tasks.
  arXiv  Detail & Related papers  (2024-07-01T16:08:37Z)
• A Control-Centric Benchmark for Video Prediction [69.22614362800692]
  We propose a benchmark for action-conditioned video prediction in the form of a control benchmark. Our benchmark includes simulated environments with 11 task categories and 310 task instance definitions. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling.
  arXiv  Detail & Related papers  (2023-04-26T17:59:45Z)
• Factored World Models for Zero-Shot Generalization in Robotic Manipulation [7.258229016768018]
  We learn to generalize over robotic pick-and-place tasks using object-factored world models. We use a residual stack of graph neural networks that receive action information at multiple levels in both their node and edge neural networks. We show that an ensemble of our models can be used to plan for tasks involving up to 12 pick and place actions using search.
  arXiv  Detail & Related papers  (2022-02-10T21:26:11Z)
• Scene Transformer: A unified multi-task model for behavior prediction and planning [42.758178896204036]
  We formulate a model for predicting the behavior of all agents jointly in real-world driving environments. Inspired by recent language modeling approaches, we use a masking strategy as the query to our model. We evaluate our approach on autonomous driving datasets for behavior prediction, and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-06-15T20:20:44Z)
• Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approach [34.70843462687529]
  We provide a self-contained tutorial on a conditional variational autoencoder approach to human behavior prediction. The goals of this tutorial paper are to review and build a taxonomy of state-of-the-art methods in human behavior prediction.
  arXiv  Detail & Related papers  (2020-08-10T03:18:27Z)
• Learning Predictive Models From Observation and Interaction [137.77887825854768]
  Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works. However, learning a model that captures the dynamics of complex skills represents a major challenge. We propose a method to augment the training set with observational data of other agents, such as humans.
  arXiv  Detail & Related papers  (2019-12-30T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.