PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models

• URL: http://arxiv.org/abs/2608.05720v1
• Date: Thu, 06 Aug 2026 08:02:15 GMT
• Title: PhyLatent: Learning Dynamics-Relevant Representations for JEPA World Models
• Abstract summary: PhyLatent is a dynamics-relevant training objective for JointEmbedding Predictive Architecture (JEPA) world models.<n>We identify three failure modes in JEPA world models: physical invariance collapse, physical identifiability collapse, and counterfactual dynamics collapse.<n>On OGBench-Cube, PhyLatent reduces the three failure rates from 15.60%, 6.71%, and 8.41% to 7.53%, 0.95%, and 4.62%, respectively, and improves model predictive control (MPC) success from 70.0% to 78.1%.
• Score: 12.761667671058767
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose PhyLatent, a dynamics-relevant training objective for JointEmbedding Predictive Architecture (JEPA) world models. Our key observation is that preventing global latent collapse does not ensure that a representation preserves physical states and action consequences. We identify three failure modes in JEPA world models: physical invariance collapse, physical identifiability collapse, and counterfactual dynamics collapse. PhyLatent addresses them through three training pathways: physical invariance, physical identifiability, and counterfactual dynamics, implemented with physical state grounding, future representation alignment, static visual invariance, counterfactual branch separation, and latent denoising. On OGBench-Cube, PhyLatent reduces the three failure rates from 15.60%, 6.71%, and 8.41% to 7.53%, 0.95%, and 4.62%, respectively, and improves model predictive control (MPC) success from 70.0% to 78.1%. With the same architecture and planner, it further improves success from 81.0% to 98.0% on TwoRooms and remains competitive on Reacher and PushT. These results show that global non-collapse alone is insufficient for learning a reliable JEPA worldmodel state space.

Related papers

• Energy-Structured Latent World Models with Neural Time Fields for Physically Constistent Open-World Motion Planning [25.402498594599013]
  We propose a novel Energy-Structured Latent World Model (ELWM)<n>Our key idea is to structure the ELWM latent state to explicitly carry energy and momentum, ensuring strictly causal transitions.<n>We further implement this for motion planning by constructing Physics-Conditioned Neural Time Fields (PC-NTF)<n>Compared to generic latent models, PC-NTF reduces 0.8-s motion-prediction NRMSE from 0.36 to 0.29.
  arXiv  Detail & Related papers  (2026-08-10T17:31:18Z)
• Is Forward Prediction Enough? Physical State Grounding for JEPA World Models [18.174985055477666]
  PSG-JEPA shapes its latent space with two complementary grounding objectives beyond forward prediction.<n>We conduct experiments at three levels: (1) latent identifiability via probing, (2) goal-conditioned planning on frozen latents, and (3) policy learning in simulation and on a real robot.
  arXiv  Detail & Related papers  (2026-08-07T04:44:16Z)
• AquaJEPA: Action-Conditioned Multimodal Predictive Representations for Underwater Robot Dynamics [0.0]
  AquaJEPA fuses an RGB camera, forward-looking sonar, and proprioception with explicit sensor validity.<n>It predicts a future latent target conditioned on eight-thruster commands.<n>It significantly outperforms three action-conditioned predictive baselines in paired final error.
  arXiv  Detail & Related papers  (2026-07-31T13:11:11Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL [0.8546394944138487]
  We formalize text-based world modeling as a steerable transition-dynamics problem decomposed into initial state.<n>We curate 239,403 grounded state-action trajectories spanning nine open-source environments and twelve frontier model families.<n>We introduce a plug-and-play GRPO training framework with deterministic state checks, and perform zero-shot transfer ablations on three OOD environments.
  arXiv  Detail & Related papers  (2026-05-07T00:40:32Z)
• RecoverFormer: End-to-End Contact-Aware Recovery for Humanoid Robots [5.584060970507507]
  We present RECOVERFORMER, a fully end-to-end humanoid recovery policy.<n>It learns when and how to switch among recovery behaviors.<n>The architecture combines a causal transformer over a 50-step observation history with two novel heads.
  arXiv  Detail & Related papers  (2026-04-24T17:52:06Z)
• Enhancing Policy Learning with World-Action Model [4.941630596191806]
  World-Action Model (WAM) is an action-regularized world model that jointly reasons over future visual observations and the actions that drive state transitions.<n>WAM incorporates an inverse dynamics objective into DreamerV2 that predicts actions from latent state transitions.<n>We evaluate WAM on enhancing policy learning across eight manipulation tasks from the CALVIN benchmark.
  arXiv  Detail & Related papers  (2026-03-30T19:56:05Z)
• CtrlAttack: A Unified Attack on World-Model Control in Diffusion Models [92.04182855442254]
  We analyze the vulnerability of I2V models, find that temporal control mechanisms constitute a new attack surface, and reveal the challenge of modeling them uniformly.<n>We propose a trajectory-control attack, called CtrlAttack, to interfere with state evolution during the generation process.<n> Experimental results show that even under low-dimensional and strongly regularized perturbation constraints, our method can still significantly disrupt temporal consistency.
  arXiv  Detail & Related papers  (2026-03-13T08:05:50Z)
• When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models [54.08784776767683]
  We present Physical-Conditioned World Model Attack (PhysCond-WMA), the first white-box world model attack that perturbs physical-condition channels.<n>PhysCond-WMA induces semantic, logic, or decision-level distortion while preserving perceptual fidelity.
  arXiv  Detail & Related papers  (2026-02-21T07:22:37Z)
• Aligning Agentic World Models via Knowledgeable Experience Learning [68.85843641222186]
  We introduce WorldMind, a framework that constructs a symbolic World Knowledge Repository by synthesizing environmental feedback.<n>WorldMind achieves superior performance compared to baselines with remarkable cross-model and cross-environment transferability.
  arXiv  Detail & Related papers  (2026-01-19T17:33:31Z)
• Internalizing World Models via Self-Play Finetuning for Agentic RL [65.96875390986655]
  Large Language Models (LLMs) as agents often struggle in out-of-distribution (OOD) scenarios.<n>We show how to encode this world model by decomposing it into two components: state representation and transition modeling.<n>We introduce SPA, a simple reinforcement learning framework that cold-starts the policy via a Self-Play supervised finetuning stage to learn the world model.
  arXiv  Detail & Related papers  (2025-10-16T18:03:39Z)
• FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning [66.56240101249803]
  We study how hardening benign clients can affect the global model (and the malicious clients) We propose a trigger reverse engineering based defense and show that our method can achieve improvement with guarantee robustness. Our results on eight competing SOTA defense methods show the empirical superiority of our method on both single-shot and continuous FL backdoor attacks.
  arXiv  Detail & Related papers  (2022-10-23T22:24:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.