What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations

• URL: http://arxiv.org/abs/2607.27017v3
• Date: Fri, 31 Jul 2026 14:52:09 GMT
• Title: What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations
• Abstract summary: POKEWORLD is an interactive environment whose visually identical objects hide mass, drag, and contact stiffness.<n>A certificate-gated protocol first certifies each parameter as recoverable from raw observations, then measures whether it enters the latent.<n>On RH20T, an input-target factorial across scaling curves reproduces both mechanisms across two robots and 4,258 episodes.
• Score: 18.546826658944806
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A central premise of latent world models is that predicting the future forces a representation to internalize the physics of its environment. Which physical quantities does a trained latent actually contain, and what decides this? We answer with controlled interventions in POKEWORLD, an interactive environment whose visually identical objects hide mass, drag, and contact stiffness. A certificate-gated protocol first certifies each parameter as recoverable from raw observations, then measures whether it enters the latent, so a null result can be attributed to the objective rather than to the environment. The resulting identifiability map has two organizing mechanisms and one frontier. Inputs limit what can be known, while prediction targets decide what is retained. Stiffness enters the latent only when touch is forecast ($R^2=0.50$, compared with $-0.02$ when the same signal is merely fused into the input), and under single-step prediction a vision-only latent discards even perfectly visible object state. Drag marks the frontier. It carries a recoverability certificate of 0.89 yet plateaus near 0.13 under every deterministic prediction objective we test, while a supervised head on the same trunk reaches 0.45. Parameters whose readout is slow and ratio-type under the sensed coordinates fall outside what these objectives acquire. On RH20T, an input-target factorial across scaling curves reproduces both mechanisms across two robots and 4,258 episodes. Every arm missing information or prediction pressure stays flat over a fivefold data range, and only the full multimodal objective forecasts force beyond a persistence baseline, with held-out gains that grow with scale. Objective structure determines which physical parameters a latent acquires, and additional data improves only the parameters it already acquires.

Related papers

• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• Why Does the Future Branch? Identifiable Closure Tests for Stochastic Physical World Models [0.0]
  ClosurePairs crosses compatible microstates with repeated disturbances.<n>It recovers source attribution at unchanged likelihood.<n>It reduces equal-budget decomposition error in a nonlinear interaction benchmark.
  arXiv  Detail & Related papers  (2026-08-01T11:07:24Z)
• From Surface Forecasting to Observability Forecasting: A Latent World Model for Cloud-Aware EO Monitoring [0.0]
  We adapt LeWorldModel, a joint-embedding predictive architecture world model, to cloud-aware Earth Observation sequences.<n>The resulting model has 18.0M trainable parameters and is trained from scratch on 23,904 training episodes.<n>LeWorldModel consistently outperforms persistence on the full frozen-bundle observability benchmark.
  arXiv  Detail & Related papers  (2026-07-15T09:52:22Z)
• UWM-JEPA: Predictive World Models That Imagine in Belief Space [0.2864713389096699]
  We introduce the Unitary World Model JEPA, a JEPA world model with a density-matrix latent on a joint system-environment space and a learned unitary predictor.<n>The construction preserves the joint-state spectrum exactly during rollout, so the predictor itself cannot dissipate the represented uncertainty.<n>For JEPA world models to imagine under partial observability, latent geometry and predictor dynamics matter, not frozen context-encoding capacity alone.
  arXiv  Detail & Related papers  (2026-05-25T00:28:51Z)
• Emergent Semantic Representations in World Models through Physical Interaction without Linguistic Supervision [0.0]
  We train a VAE-based world model on random embodied exploration.<n>We find that its latent space develops spatial structure semantic that mirrors physical geometry.<n>These findings establish physical world geometry as the organizing principle of world model representations.
  arXiv  Detail & Related papers  (2026-05-22T03:31:47Z)
• Quantifying Uncertainty in Motion Prediction with Variational Bayesian Mixture [17.78048571619575]
  Safety and robustness are crucial factors in developing trustworthy autonomous vehicles. We propose the Sequential Neural Variational Agent (SeNeVA), a generative model that describes the distribution of future trajectories for a single moving object. Our approach can distinguish Out-of-Distribution data while quantifying uncertainty and achieving competitive performance.
  arXiv  Detail & Related papers  (2024-04-04T20:04:12Z)
• Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties [100.19685489335828]
  This work proposes a novel dataset and benchmark, termed Physion++, to rigorously evaluate visual physical prediction in artificial systems. We test scenarios where accurate prediction relies on estimates of properties such as mass, friction, elasticity, and deformability. We evaluate the performance of a number of state-of-the-art prediction models that span a variety of levels of learning vs. built-in knowledge, and compare that performance to a set of human predictions.
  arXiv  Detail & Related papers  (2023-06-27T17:59:33Z)
• Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation [51.809741427975105]
  This work aims to assess how well a model performs under distribution shifts without using labels. We use the nuclear norm that has been shown to be effective in characterizing both properties. We show that the nuclear norm is more accurate and robust in accuracy than existing methods.
  arXiv  Detail & Related papers  (2023-02-02T13:30:48Z)
• End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation [86.41437210485932]
  We aim at advancing zero-shot HOI detection to detect both seen and unseen HOIs simultaneously. We propose a novel end-to-end zero-shot HOI Detection framework via vision-language knowledge distillation. Our method outperforms the previous SOTA by 8.92% on unseen mAP and 10.18% on overall mAP.
  arXiv  Detail & Related papers  (2022-04-01T07:27:19Z)
• Neuro-Symbolic Entropy Regularization [78.16196949641079]
  In structured prediction, the goal is to jointly predict many output variables that together encode a structured object. One approach -- entropy regularization -- posits that decision boundaries should lie in low-probability regions. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object.
  arXiv  Detail & Related papers  (2022-01-25T06:23:10Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Learning Transferable Push Manipulation Skills in Novel Contexts [3.1981440103815717]
  We learn a parametric internal model for push interactions that enables a robot to predict the outcome of a physical interaction even in novel contexts. We train on 2 objects for a total of 24,000 pushes in various conditions, and test on 6 objects for a total of 14,400 predicted push outcomes. Our results show that both biased and unbiased predictors can reliably produce predictions in line with the outcomes of a carefully tuned physics simulator.
  arXiv  Detail & Related papers  (2020-07-29T11:48:56Z)
• Causal Discovery in Physical Systems from Videos [123.79211190669821]
  Causal discovery is at the core of human cognition. We consider the task of causal discovery from videos in an end-to-end fashion without supervision on the ground-truth graph structure.
  arXiv  Detail & Related papers  (2020-07-01T17:29:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.