JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics

• URL: http://arxiv.org/abs/2608.24044v3
• Date: Fri, 28 Aug 2026 10:39:50 GMT
• Title: JEPA-x: Cross-Predictive Physics Grounding for Forecastable Latent Dynamics
• Abstract summary: In self-predictive models, the encoder and predictor are optimized jointly and can co-adapt to latent transitions.<n>We introduce the cross-predictive JEPA (JEPA-x), which grounds latent dynamics in privileged physical trajectories.
• Score: 9.006010556484968
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Latent world models plan by predicting how candidate actions advance learned latent dynamics. In self-predictive models, however, the encoder and predictor are optimized jointly and can co-adapt to latent transitions that are easy to predict but weakly constrained by the physical evolution of the scene. We introduce the cross-predictive JEPA (JEPA-x), which grounds latent dynamics in privileged physical trajectories. JEPA-x treats visual observations and physical states as corresponding views of the same action-conditioned trajectory, advances both through a shared predictor, and matches each prediction to the future representations of both modalities. This encourages the action-conditioned predictor to learn a common transition rule across the two views. Privileged physical state is used only during training, leaving a visual-only model at deployment. Empirical results show that JEPA-x reduces the rollout drift of a newly fitted predictor from $0.361$ to $0.104$ and increases mean control success from $53.6\%$ to $78.2\%$ on a multi-task suite spanning six evaluation subfamilies. We additionally show that direct physical-state regression improves decodability without improving forecastability or control, indicating that the benefit comes from shaping latent dynamics rather than merely encoding physical variables.

Related papers

• Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization [32.65135632960651]
  Joint-Embedding Predictive Architecture (JEPA) world models learn a compact latent representation of the world that supports prediction and planning.<n>We introduce SemiGroup-JEPA, which extends the LeWorldModel framework by supplying the parameter governing the physics to the temporal model.<n>We design dynamical tasks under different gravitational fields that, despite obeying the same physical law, exhibit qualitatively different dynamics.
  arXiv  Detail & Related papers  (2026-09-09T17:08:13Z)
• Flow-JEPA: Flow Matching for Robust Latent Dynamics in JEPA World Models [18.035166689862205]
  Joint-Embedding Predictive Architectures (JEPAs) have shown strong potential for learning compact predictive representations.<n>We propose Flow-JEPA, a conditional flow matching dynamics model that jointly generates a sequence of future latent states conditioned on the current observation and actions.
  arXiv  Detail & Related papers  (2026-08-29T03:40:06Z)
• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• SLIM-0.5B: Learning Action-Grounded Predictive Latents for Robot Manipulation [35.54319311306262]
  We present SLIM, a compact latent interaction policy.<n>It learns representations through self-supervised masked trajectory prediction.<n>SLIM matches or exceeds representative large-scale VLA and world-action-model baselines with fewer parameters.
  arXiv  Detail & Related papers  (2026-08-10T15:58:39Z)
• JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling [23.76460420882974]
  We introduce JEPA-WAM, a latent WAM built in a pretrained V-JEPA space.<n>It couples latent transition prediction with continuous action generation through a shared predictor.<n>On LIBERO-Plus, JEPA-WAM achieves 79.2%, the best result without large-scale robot-policy pretraining.
  arXiv  Detail & Related papers  (2026-08-10T09:57:54Z)
• Is Forward Prediction Enough? Physical State Grounding for JEPA World Models [18.174985055477666]
  PSG-JEPA shapes its latent space with two complementary grounding objectives beyond forward prediction.<n>We conduct experiments at three levels: (1) latent identifiability via probing, (2) goal-conditioned planning on frozen latents, and (3) policy learning in simulation and on a real robot.
  arXiv  Detail & Related papers  (2026-08-07T04:44:16Z)
• On the Identifiability of Controlled World Models [34.31084112465058]
  This paper presents a joint identifiability condition for controlled world models with Gaussian latent states.<n>We prove that when this condition holds, minimizing the LeJEPA-style predictive objective can recover both latent states and controlled dynamics.<n>We also characterize an attainable amplification of counterfactual prediction error that scales inversely with the weakest conditional action-excitation margin.
  arXiv  Detail & Related papers  (2026-07-24T15:49:12Z)
• MoP-JEPA: Hard-Assigned Predictor Mixtures for Stochastic JEPA World Models [14.76812870029721]
  MoP-JEPA produces a finite candidate set in one pass.<n>On held-out OGBench transitions, graph search with single-output predictors succeeds on $0.02$--$0.09$ of queries, whereas MoP-JEPA reaches $0.85$.
  arXiv  Detail & Related papers  (2026-07-06T15:47:37Z)
• Predictive Objectives Discard Exogenous Control-Relevant Features: A Controlled Mechanistic Study [0.0]
  Joint-embedding predictive (JEPA) objectives learn representations by predicting future latents.<n>This occurs because the objective optimize temporal predictability rather than control-relevance.<n>We show that as little as 2% of reward-labeled transitions recovers the feature, the effect holds across two environments.
  arXiv  Detail & Related papers  (2026-06-29T10:00:50Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Models [58.191567345416836]
  World Action Models (WAMs) have emerged as a promising paradigm for robot control by modeling physical dynamics.<n>HarmoWAM fully leverages a world model to unify predictive and reactive control, enabling both general transit and precise manipulation.<n>We construct three training-unseen test environments across six real-world robotic tasks, covering variations in background, position, and object semantics.
  arXiv  Detail & Related papers  (2026-05-11T17:59:56Z)
• LeapTS: Rethinking Time Series Forecasting as Adaptive Multi-Horizon Scheduling [74.94985663101906]
  We present LeapTS, a novel framework that reformulates time series forecasting as a dynamic scheduling process over the prediction horizon.<n>We show that LeapTS improves overall forecasting performance by at least 7.4% while achieving a 2.6$times$ to 5.3$times$ inference speedup over representative Transformer-based models.
  arXiv  Detail & Related papers  (2026-05-11T09:54:02Z)
• Out-of-distribution transfer of PDE foundation models to material dynamics under extreme loading [86.6550968435969]
  Most PDE foundation models are pretrained and fine-tuned on fluid-centric benchmarks.<n>We benchmark out-of-distribution transfer on two discontinuity-dominated regimes in which shocks, evolving interfaces, and fracture produce highly non-smooth fields.<n>We evaluate two open-source PDE foundation models, POSEIDON and MORPH, and compare fine-tuning from pretrained weights against training from scratch across training-set sizes to quantify sample efficiency under distribution shift.
  arXiv  Detail & Related papers  (2026-03-04T18:19:35Z)
• Self-Correcting VLA: Online Action Refinement via Sparse World Imagination [55.982504915794514]
  We propose Self-Correcting VLA (SC-VLA), which achieve self-improvement by intrinsically guiding action refinement through sparse imagination.<n>SC-VLA achieve state-of-the-art performance, yielding the highest task throughput with 16% fewer steps and a 9% higher success rate than the best-performing baselines.
  arXiv  Detail & Related papers  (2026-02-25T06:58:06Z)
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning [63.73629127832652]
  We introduce TD-JEPA, which leverages TD-based latent-predictive representations into unsupervised RL.<n> TD-JEPA trains explicit state and task encoders, a policy-conditioned multi-step predictor, and a set of parameterized policies directly in latent space.<n> Empirically, TD-JEPA matches or outperforms state-of-the-art baselines on locomotion, navigation, and manipulation tasks across 13 datasets.
  arXiv  Detail & Related papers  (2025-10-01T10:21:18Z)
• Panoptic Segmentation Forecasting [71.75275164959953]
  Our goal is to forecast the near future given a set of recent observations. We think this ability to forecast, i.e., to anticipate, is integral for the success of autonomous agents. We develop a two-component model: one component learns the dynamics of the background stuff by anticipating odometry, the other one anticipates the dynamics of detected things.
  arXiv  Detail & Related papers  (2021-04-08T17:59:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.