Grounded World Model for Semantically Generalizable Planning

• URL: http://arxiv.org/abs/2604.11751v1
• Date: Mon, 13 Apr 2026 17:25:41 GMT
• Title: Grounded World Model for Semantically Generalizable Planning
• Abstract summary: We learn a Grounded World Model (GWM) in a vision-language-aligned latent space.<n>Each proposed action is scored based on how close its future outcome is to the task instruction.<n>On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set.
• Score: 94.53923128709965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate.

Related papers

• TrAct: Bridging Robot Control and Visual Prediction with Visual Tracks [22.702826122425268]
  We propose TrAct, a world-model-based robot decision-making framework that uses visual tracks as an intermediate interface between control and prediction.<n>TrAct consists of three components: a Vision-Language-Action-and-Track model (VLAT) that jointly predicts candidate actions and corresponding visual tracks from the current observation and language instruction; a track-conditioned world model (TWM) that predicts future visual outcomes conditioned on the proposed tracks; and a vision-language reward model (VLAC) that scores the predicted outcomes.
  arXiv  Detail & Related papers  (2026-08-25T06:00:01Z)
• DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning [19.23819338884138]
  DREAMSTEER is a deployment-time steering framework for pretrained vision-language-action (VLA) policies.<n>Itleverages a latent world model and a value model to steer pretrained VLA policies.<n>It improves task success rate from 23.75% to 66.25% and instruction-following accuracy from 38.75% to 56.25% over the base VLA policy.
  arXiv  Detail & Related papers  (2026-07-03T02:08:50Z)
• FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation [14.178611843420683]
  We introduce FOCA, a future-oriented conditioning framework for data-efficient VLA adaptation.<n> FOCA combines explicit prediction of task-grounded future interaction embeddings with implicit alignment to future goal observations.<n>Experiments demonstrate FOCA achieves 95.7% success with 20 demonstrations on LIBERO, improves 7-12% on RoboCasa, and delivers up to 26% absolute gains on real robots.
  arXiv  Detail & Related papers  (2026-06-18T18:54:51Z)
• World Pilot: Steering Vision-Language-Action Models with World-Action Priors [21.867210989881016]
  Vision-Language-Action (VLA) models inherit semantic grounding from large-scale pretraining.<n>World Pilot augments the policy with priors from a World-Action Model (WAM)<n>World Pilot attains a state-of-the-art Total success rate of 84.7% on the LIBERO-Plus zero-shot OOD benchmark.
  arXiv  Detail & Related papers  (2026-06-10T17:59:08Z)
• Universal Pose Pretraining for Generalizable Vision-Language-Action Policies [83.39008378156647]
  Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency.<n>We propose Pose-VLA, a decoupled paradigm that separates VLA training into a pre-training phase for extracting universal 3D spatial priors.<n>Our framework follows a two-stage pre-training pipeline, establishing fundamental spatial grounding via poses followed by motion alignment.
  arXiv  Detail & Related papers  (2026-02-23T11:00:08Z)
• Reshaping Action Error Distributions for Reliable Vision-Language-Action Models [69.38615670891038]
  In robotic manipulation, vision-language-action (VLA) models have emerged as a promising paradigm for learning generalizable and scalable robot policies.<n>We focus on continuous-action VLA models and move beyond conventional MSE-based regression by reshaping action error distributions during training.<n>We evaluate our approaches across standard, few-shot, and noisy settings on multiple representative VLA architectures.
  arXiv  Detail & Related papers  (2026-02-04T05:37:09Z)
• Robotic VLA Benefits from Joint Learning with Motion Image Diffusion [114.60268819583017]
  Vision-Language-Action (VLA) models have achieved remarkable progress in robotic manipulation by mapping multimodal observations and instructions directly to actions.<n>We propose joint learning with motion image diffusion, a novel strategy that enhances VLA models with motion reasoning capabilities.<n> Experiments in both simulation and real-world environments demonstrate that joint learning with motion image diffusion improves the success rate of pi-series VLAs to 97.5%.
  arXiv  Detail & Related papers  (2025-12-19T19:07:53Z)
• Planning with Reasoning using Vision Language World Model [30.40687339662431]
  We introduce the Vision Language World Model (VLWM), a foundation model trained for language-based world modeling on natural videos.<n>Given visual observations, the VLWM first infers the overall goal achievements then predicts a trajectory composed of interleaved actions and world state changes.<n>The VLWM achieves state-of-the-art Visual Planning for Assistance (VPA) performance on both benchmark evaluations and our proposed PlannerArena human evaluations.
  arXiv  Detail & Related papers  (2025-09-02T18:18:57Z)
• LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation [45.02469804709771]
  We propose LaDi-WM, a world model that predicts the latent space of future states using diffusion modeling.<n>We show that LaDi-WM significantly enhances policy performance by 27.9% on the LIBERO-LONG benchmark and 20% on the real-world scenario.
  arXiv  Detail & Related papers  (2025-05-13T04:42:14Z)
• CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation [100.25567121604382]
  Vision-Language-Action (VLA) models have improved robotic manipulation in terms of language-guided task execution and generalization to unseen scenarios.<n>We present a new advanced VLA architecture derived from Vision-Language-Models (VLM)<n>We show that our model not only significantly surpasses existing VLAs in task performance and but also exhibits remarkable adaptation to new robots and generalization to unseen objects and backgrounds.
  arXiv  Detail & Related papers  (2024-11-29T12:06:03Z)
• Goal-Conditioned End-to-End Visuomotor Control for Versatile Skill Primitives [89.34229413345541]
  We propose a conditioning scheme which avoids pitfalls by learning the controller and its conditioning in an end-to-end manner. Our model predicts complex action sequences based directly on a dynamic image representation of the robot motion. We report significant improvements in task success over representative MPC and IL baselines.
  arXiv  Detail & Related papers  (2020-03-19T15:04:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.