Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning

• URL: http://arxiv.org/abs/2609.03565v1
• Date: Thu, 03 Sep 2026 09:11:13 GMT
• Title: Toward Physically Grounded JEPA World Models for Goal-Conditioned Robotic Planning
• Abstract summary: We introduce an end-to-end JEPA world model that augments latent prediction with inverse dynamics (IDM) and state alignment (SA)<n>Our model attains the highest success rates on TwoRoom (100%), PushT (98%), and OGBench-Cube (87%), while performing comparably to LeWorldModel on Reacher.
• Score: 17.909802735976466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Action-conditioned JEPA world models enable planning toward visually specified goals without reconstructing future pixels, yet latent prediction alone does not explicitly encourage the learned representations to retain information relevant to robotic control. We introduce an end-to-end JEPA world model that augments latent prediction with inverse dynamics (IDM) and state alignment (SA). While inverse dynamics discourages latent collapse and makes latent transitions informative of the actions that produced them, state alignment grounds consecutive representations in their associated physical configuration and motion. Across four benchmark tasks, our model attains the highest success rates on TwoRoom (100%), PushT (98%), and OGBench-Cube (87%), while performing comparably to LeWorldModel on Reacher. Our ablation further shows that adding state alignment consistently improves planning success over IDM alone across all four tasks. Although LeWorldModel, our primary baseline, attains higher average straightening on OGBench-Cube, transition-subspace analysis shows that its transition energy is concentrated in a substantially lower-dimensional subspace. Our state-aligned model exhibits a higher effective transition dimension than LeWorldModel and improves planning over IDM alone, supporting state alignment as an effective complement to inverse dynamics for robotic planning.

Related papers

• DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation [19.179682667653328]
  Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control.<n>We introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations.<n>By exploiting complementary global scene and gripper-centric information, DUET-DINO enables latent planning over the full 7-DoF action space.
  arXiv  Detail & Related papers  (2026-09-09T17:41:38Z)
• Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization [32.65135632960651]
  Joint-Embedding Predictive Architecture (JEPA) world models learn a compact latent representation of the world that supports prediction and planning.<n>We introduce SemiGroup-JEPA, which extends the LeWorldModel framework by supplying the parameter governing the physics to the temporal model.<n>We design dynamical tasks under different gravitational fields that, despite obeying the same physical law, exhibit qualitatively different dynamics.
  arXiv  Detail & Related papers  (2026-09-09T17:08:13Z)
• GaussianDream++: Efficient 3D Gaussian World Modeling for Robotic Manipulation [55.47130289185285]
  Vision-Language-Action (VLA) policies have advanced language-conditioned robotic manipulation.<n>We present textbfMethodname, a compact, policy-native extension that inserts textbfWorld State Tokens and textbfWorld Prediction Tokens directly into the VLA backbone.
  arXiv  Detail & Related papers  (2026-08-26T11:41:55Z)
• SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space [12.198811616220835]
  World Action Models (WAMs) couple action generation with prediction of future states.<n>We propose SG-WAM, a self-guided framework that learns geometry-aware action-conditioned dynamics directly in the policy-derived representation space.
  arXiv  Detail & Related papers  (2026-08-02T17:25:08Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• HaM-World: Soft-Hamiltonian World Models with Selective Memory for Planning [2.28917458387011]
  HaM-World is a structured world model that decomposes the latent state into a canonical (q, p) subspace and a context subspace c.<n>On four DeepMind Control Suite tasks, HaM-World reaches the highest Avg. AUC (117.9, +9.5%), reduces long-horizon rollout error to 45% of a strong baseline model, and wins 11/12 k in 3,5,7 MSE cells.
  arXiv  Detail & Related papers  (2026-05-07T09:58:18Z)
• World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry [82.93104394404781]
  General-purpose world models promise scalable policy evaluation, optimization, and planning.<n>We propose World Action Verifier (WAV), a framework that enables world models to identify their own prediction errors and self-improve.
  arXiv  Detail & Related papers  (2026-04-02T12:48:36Z)
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination [55.982504915794514]
  We propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination.<n>SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines.
  arXiv  Detail & Related papers  (2026-02-25T06:58:06Z)
• TC-IDM: Grounding Video Generation for Executable Zero-shot Robot Motion [38.88993836311246]
  Generative world models offer a promising alternative for general-purpose embodied AI.<n>We propose the Tool-Centric Inverse Dynamics Model (TC-IDM) to bridge the gap between visual planning and physical control.<n>In real-world evaluations, the world model with TC-IDM achieves an average success rate of 61.11 percent, with 77.7 percent on simple tasks and 38.46 percent on zero-shot deformable object tasks.
  arXiv  Detail & Related papers  (2026-01-26T10:06:56Z)
• From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction [57.56072009935036]
  We introduce a new driving paradigm named Policy World Model (PWM)<n>PWM integrates world modeling and trajectory planning within a unified architecture.<n>Our method matches or exceeds state-of-the-art approaches that rely on multi-view and multi-modal inputs.
  arXiv  Detail & Related papers  (2025-10-22T14:57:51Z)
• Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models [28.777224599594717]
  Implicit Residual World Model focuses on modeling the current state and evolution of the world.<n> IR-WM achieves top performance in both 4D occupancy forecasting and trajectory planning.
  arXiv  Detail & Related papers  (2025-10-19T06:45:37Z)
• WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning [52.36434784963598]
  We introduce WorldPrediction, a video-based benchmark for evaluating world modeling and procedural planning capabilities of different AI models.<n>We show that current frontier models barely achieve 57% accuracy on WorldPrediction-WM and 38% on WorldPrediction-PP whereas humans are able to solve both tasks perfectly.
  arXiv  Detail & Related papers  (2025-06-04T18:22:40Z)
• LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation [45.02469804709771]
  We propose LaDi-WM, a world model that predicts the latent space of future states using diffusion modeling.<n>We show that LaDi-WM significantly enhances policy performance by 27.9% on the LIBERO-LONG benchmark and 20% on the real-world scenario.
  arXiv  Detail & Related papers  (2025-05-13T04:42:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.