VisualPatchWorld: Code World Models as Latent Structured Representations for Planning

• URL: http://arxiv.org/abs/2607.25236v1
• Date: Tue, 28 Jul 2026 03:23:47 GMT
• Title: VisualPatchWorld: Code World Models as Latent Structured Representations for Planning
• Abstract summary: We introduce VisualPatchWorld (VPW), which represents world dynamics as code.<n>VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces.<n>VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points.
• Score: 8.39933163242427
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Different research lines use the term world model in different ways, yet they share a common aim: to capture how the world evolves under action in a form that supports perception, simulation, and planning. Two prominent realizations are neural predictors that learn dynamics in continuous vector spaces, and hand-built physics engines that expose explicit state and physical laws. Neural predictors scale from data but leave the form of the dynamics implicit; physics engines are inspectable and editable but difficult to construct at scale. We introduce VisualPatchWorld (VPW), which represents world dynamics as code. VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces by minimizing multi-step prediction error. The resulting programs can be rolled forward like a simulator, inspected in source form, and used inside model-predictive control; image-derived scene graphs can supply the live state at replan time. Across comparisons with prior code-based world models, VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points. The largest gains arise when choosing the correct qualitative dynamics is essential. Under the same planner, the induced models approach ground-truth engine success on navigation and grasp-rich control; a residual gap remains for contact-rich pushing, and checking a shortlist of promising plans in the engine closes most of that gap. These results establish a practical route toward automatically constructed code world models that are useful for planning. Code is available at https://github.com/HKBU-KnowComp/VisualPatchWorld/.

Related papers

• Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution [51.519325778407016]
  World models let robots imagine possible futures, but exploiting this capability for real-time control is bottlenecked by a representation misalignment.<n>We present Hydra, a discrete World Action Model that closes this gap by moving the planner, both the sampler and the evaluator, inside the model.<n> Hydra outperforms state-of-the-art world models in goal-directed planning, while matching or exceeding the closed-loop execution capabilities of leading reactive foundation policies.
  arXiv  Detail & Related papers  (2026-08-29T01:58:19Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• PatchWorld: Gradient-Free Optimization of Executable World Models [69.05094454437402]
  We introduce PatchWorld, a gradient-free framework that turns offline trajectories into executable Python world models.<n>Across seven AgentGym environments, PatchWorld-Simple achieves the highest code-based planning score.<n>We find that a human-specified residual-memory bias improves surface observation fidelity but weakens decision utility.
  arXiv  Detail & Related papers  (2026-05-29T06:11:14Z)
• Nano World Models: A Minimalist Implementation of Future Video Prediction [52.36220976002044]
  Nano World Models is a minimalist for future video prediction centered around diffusion forcing.<n>By releasing code, configurations, evaluation scripts, and pretrained checkpoints, Nano World Models aims to provide a compact yet experimental substrate for open, reproducible, and scientific world-model research.
  arXiv  Detail & Related papers  (2026-05-17T22:46:44Z)
• Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond [209.35045331678043]
  We introduce a "levels x laws" taxonomy organized along two axes.<n>The first defines three capability levels: L1 Predictor, which learns one-step local transition operators; L2 Simulator, which composes them into multi-step, action-conditioned rollouts that respect domain laws; and L3 Evolver, which autonomously revises its own model when predictions fail against new evidence.<n>We synthesize over 400 works and summarize more than 100 representative systems spanning model-based reinforcement learning, video generation, web and GUI agents, multi-agent social simulation, and AI-driven scientific discovery.
  arXiv  Detail & Related papers  (2026-04-24T17:48:47Z)
• WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Models [32.71152441572375]
  Vision-language models (VLMs) and generative world models are opening new opportunities for embodied navigation.<n>We present WorldMAP, a framework that converts world-model-generated futures into persistent semantic-spatial structure and planning-derived supervision.
  arXiv  Detail & Related papers  (2026-04-09T08:21:58Z)
• DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning [44.543763428623976]
  We present DriveDreamer-Policy, a unified driving world-action model that integrates depth generation, future video generation, and motion planning.<n>The proposed model produces more coherent imagined futures and more informed driving actions, while maintaining modularity and latency controllable.
  arXiv  Detail & Related papers  (2026-04-02T08:33:18Z)
• Parallel Stochastic Gradient-Based Planning for World Models [39.699893143984916]
  We propose a robust and highly parallelizable planner that leverages the differentiability of the learned world model for efficient optimization.<n>Our method treats states as optimization variables ("virtual states") with soft dynamics constraints, enabling parallel and easier optimization.<n>Our planner, which we call GRASP (GradAxed Planner), can be viewed as a valid version of a non-condensed or collocation-based optimal controller.
  arXiv  Detail & Related papers  (2026-01-31T02:57:47Z)
• From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction [57.56072009935036]
  We introduce a new driving paradigm named Policy World Model (PWM)<n>PWM integrates world modeling and trajectory planning within a unified architecture.<n>Our method matches or exceeds state-of-the-art approaches that rely on multi-view and multi-modal inputs.
  arXiv  Detail & Related papers  (2025-10-22T14:57:51Z)
• WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning [52.36434784963598]
  We introduce WorldPrediction, a video-based benchmark for evaluating world modeling and procedural planning capabilities of different AI models.<n>We show that current frontier models barely achieve 57% accuracy on WorldPrediction-WM and 38% on WorldPrediction-PP whereas humans are able to solve both tasks perfectly.
  arXiv  Detail & Related papers  (2025-06-04T18:22:40Z)
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning [38.749045283035365]
  DINO World Model (DINO-WM) is a new method to model visual dynamics without reconstructing the visual world.<n>We demonstrate that DINO-WM achieves zero-shot behavioral solutions at test time on six environments.
  arXiv  Detail & Related papers  (2024-11-07T18:54:37Z)
• Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving [15.100104512786107]
  Drive-OccWorld adapts a visioncentric- 4D forecasting world model to end-to-end planning for autonomous driving.<n>We propose injecting flexible action conditions, such as velocity, steering angle, trajectory, and commands, into the world model to enable controllable generation.<n>Our method can generate plausible and controllable 4D occupancy, paving the way for advancements in driving world generation and end-to-end planning.
  arXiv  Detail & Related papers  (2024-08-26T11:53:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.