Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding

• URL: http://arxiv.org/abs/2606.31232v1
• Date: Tue, 30 Jun 2026 07:08:24 GMT
• Title: Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding
• Abstract summary: Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions.<n>We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments latent forward prediction with a Latent Difference Action Decoder (LDAD)<n>LDAD reconstructs the executed action from the latent displacement between consecutive observations.
• Score: 25.77502065015808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations. We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments latent forward prediction with a Latent Difference Action Decoder (LDAD). Unlike inverse decoders that infer actions from concatenated endpoint embeddings, LDAD reconstructs the executed action from the latent displacement between consecutive observations. This displacement-level supervision directly regularizes transition geometry: adjacent embeddings cannot collapse without losing action information, and different actions are encouraged to induce distinguishable latent changes for rollout-based planning. Delta-JEPA uses only latent prediction and action reconstruction, avoiding pixel reconstruction and distribution-matching regularizers. Across four visual continuous-control tasks, Delta-JEPA improves planning over JEPA-based and representation-learning world model baselines. Ablations show that displacement-based action decoding is consistently more effective than endpoint concatenation, and action-sensitivity analyses show clearer action-conditioned latent responses. These results indicate that supervising latent differences is a simple and effective mechanism for collapse-resistant and action-sensitive world model learning.

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• ActSWM: Action-Sensitive World Models for Long-Horizon Planning in Open-World Games [6.8149466180197]
  We propose ActSWM, an action-sensitive latent world model grounded in a transition-separation principle.<n>Across step-drift analysis, closed-loop Minecraft planning, and cross-game local action recovery, ActSWM preserves larger action-dependent rollout gaps than existing baselines.
  arXiv  Detail & Related papers  (2026-07-29T09:55:54Z)
• DWM: Separating World Effects from Actions in Latent World Models [34.35433044964186]
  We introduce DWM (Decomposed World Model), a supervision-level framework that operationalizes this decomposition.<n>DWM augments the predictor of a latent world model with an auxiliary world head, regularized by a normalized world-contrastive objective to be action-invariant.<n>We construct W-variants of three standard control benchmarks -- PushT-W, Reacher-W, and TwoRoom-W -- each instantiating a distinct action-invariant dynamic.
  arXiv  Detail & Related papers  (2026-07-21T05:13:26Z)
• Causally Debiased Latent Action Model for Embodied Action Conditioned World Models [47.281442841934876]
  Action-conditioned world models (ACWMs) aim to simulate future observations conditioned on embodied actions.<n>Existing LAMs are typically trained with reconstruction-only objectives.<n>We propose CD-LAM, a causally debiased framework for LAM-based ACWMs.
  arXiv  Detail & Related papers  (2026-07-10T08:20:27Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• Stealthy World Model Manipulation via Data Poisoning [6.257560917160816]
  We propose SWAAP, the first two-stage data poisoning framework for learned world models.<n>In the first stage, SWAAP identifies a harmful target world model that induces low-return behavior under planning.<n>In the second stage, SWAAP realizes this target through stealth-constrained gradient matching.
  arXiv  Detail & Related papers  (2026-06-17T05:24:18Z)
• Latent Action Reparameterization for Efficient Agent Inference [56.42014061367112]
  We propose a framework that learns a compact latent action space in which each latent action corresponds to a multi-step semantic behavior.<n>Unlike hand-crafted macros or hierarchical controllers, latent actions are learned from agent trajectories and integrated directly into the model.
  arXiv  Detail & Related papers  (2026-05-18T16:07:44Z)
• SCAR: Self-Supervised Continuous Action Representation Learning [36.917304453471864]
  We propose SCAR, a joint inverse-forward dynamics framework for learning unified action representations across embodiments from visual transitions.<n>Built on a pretrained generative backbone, SCAR uses an inverse dynamics model (IDM) to infer latent actions from latent observation pairs and a forward dynamics model (FDM) to predict future dynamics conditioned on them.<n>Experiments on the Procgen and Robotwin dataset show that the learned unified latent action representation serves as a stronger conditioning interface for world modeling than embodiment-specific raw actions.
  arXiv  Detail & Related papers  (2026-05-13T16:23:11Z)
• ALAM: Algebraically Consistent Latent Action Model for Vision-Language-Action Models [31.2829405167292]
  ALAM (Algebraic Latent Action Model) turns temporal relations in action-free video into structural supervision.<n>ALAM learns latent transitions grounded by reconstruction while being regularized by composition and reversal consistency.<n>For downstream VLA learning, we freeze the pretrained encoder and use its latent transition sequences as auxiliary generative targets.
  arXiv  Detail & Related papers  (2026-05-11T16:37:07Z)
• AnchorRefine: Synergy-Manipulation Based on Trajectory Anchor and Residual Refinement for Vision-Language-Action Models [60.04879435087352]
  Vision-language-action (VLA) policies generate actions within a single unified space.<n>We propose AnchorRefine, a hierarchical framework that factorizes VLA action modeling into trajectory anchor and residual refinement.<n> Experiments on LIBERO, CALVIN, and real-robot tasks demonstrate that AnchorRefine consistently improves both regression-based and diffusion-based VLA backbones.
  arXiv  Detail & Related papers  (2026-04-20T04:25:24Z)
• \ extsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning [63.73629127832652]
  We introduce TD-JEPA, which leverages TD-based latent-predictive representations into unsupervised RL.<n> TD-JEPA trains explicit state and task encoders, a policy-conditioned multi-step predictor, and a set of parameterized policies directly in latent space.<n> Empirically, TD-JEPA matches or outperforms state-of-the-art baselines on locomotion, navigation, and manipulation tasks across 13 datasets.
  arXiv  Detail & Related papers  (2025-10-01T10:21:18Z)
• Latent Diffusion Planning for Imitation Learning [78.56207566743154]
  Latent Diffusion Planning (LDP) is a modular approach consisting of a planner and inverse dynamics model.<n>By separating planning from action prediction, LDP can benefit from the denser supervision signals of suboptimal and action-free data.<n>On simulated visual robotic manipulation tasks, LDP outperforms state-of-the-art imitation learning approaches.
  arXiv  Detail & Related papers  (2025-04-23T17:53:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.