Physical Object Understanding with a Physically Controllable World Model

• URL: http://arxiv.org/abs/2606.00439v1
• Date: Sat, 30 May 2026 00:10:32 GMT
• Title: Physical Object Understanding with a Physically Controllable World Model
• Abstract summary: A central challenge in visual intelligence is learning the physical structure of scenes from raw videos.<n>We introduce a new class of probabilistic world models that support estimation of the probability of any visual variable.<n>We show that our world model can manipulate objects in 3D, enabling applications such as Visual Jenga.
• Score: 11.085294403458972
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central challenge in visual intelligence is learning the physical structure of scenes from raw videos: how regions form objects and the laws that govern their interactions. Solving these tasks requires world models capable of inferring distributional states of the world from partial observations - capabilities that current architectures do not provide. We introduce a new class of probabilistic world models that support estimation of the probability of any visual variable, such as appearance and dynamics, conditioned on any other variables. Here, we identify that these models can be trained efficiently with autoregressive sequence modeling, yielding world models from which rich object understanding emerges. First, we demonstrate that our model captures the physical laws governing how objects move by generating multiple plausible future states of the world through sequential inference. Then, by analyzing motion correlations across these futures, we extract objects and articulated object subparts. Having discovered these objects, we show that our world model can manipulate them in 3D. Finally, we demonstrate how physical relationships between objects can be computed from the world model, enabling applications such as Visual Jenga.

Related papers

• Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models [36.26785019314161]
  Deform360 is a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactiles.<n>We evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models.
  arXiv  Detail & Related papers  (2026-07-06T17:59:18Z)
• Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects [4.870609209341298]
  We develop a novel model that learns action-conditional dynamics of rigid objects in 3D.<n>We evaluate our model in model-predictive control for non-prehensile manipulation in simulation.
  arXiv  Detail & Related papers  (2026-06-01T09:12:07Z)
• WorldString: Actionable World Representation [29.267308971639682]
  We propose WorldString, a neural architecture capable of modeling the state manifold of real-world objects by learning directly from point clouds or RGB-D video streams.<n>WorldString acts as a foundational building block for physical world models; thus, we name it WorldString. Sweetly, its fully differentiable structure seamlessly enables future integration with policy learning and neural dynamics.
  arXiv  Detail & Related papers  (2026-05-18T17:58:51Z)
• Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling [51.40150411616207]
  We introduce Latent Particle World Model (LPWM), a self-supervised object-centric world model scaled to real-world multi-object datasets.<n>LPWM autonomously discovers keypoints, bounding boxes, and object masks directly from video data.<n>Our architecture is trained end-to-end purely from videos and supports flexible conditioning on actions, language, and image goals.
  arXiv  Detail & Related papers  (2026-03-04T19:36:08Z)
• ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos [48.24897274501108]
  We introduce a 3D object-centric dynamics model that predicts future 6-DoF poses and trajectories of rigid objects from short egocentric video sequences.<n>Unlike conventional world or dynamics models that operate in pixel or latent space, ObjectForesight represents the world explicitly in 3D at the object level.<n>We show that ObjectForesight achieves significant gains in accuracy, geometric consistency, and generalization to unseen objects and scenes.
  arXiv  Detail & Related papers  (2026-01-08T18:58:08Z)
• WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World [100.68103378427567]
  Generative world models are reshaping embodied AI, enabling agents to synthesize realistic 4D driving environments that look convincing but often fail physically or behaviorally.<n>We introduce WorldLens, a full-spectrum benchmark evaluating how well a model builds, understands, and behaves within its generated world.<n>We further construct WorldLens-26K, a large-scale dataset of human-annotated videos with numerical scores and textual rationales, and develop WorldLens-Agent.
  arXiv  Detail & Related papers  (2025-12-11T18:59:58Z)
• Simulating the Visual World with Artificial Intelligence: A Roadmap [48.64639618440864]
  Video generation is shifting from generating visually appealing clips to building virtual environments that support interaction and maintain physical plausibility.<n>This survey provides a systematic overview of this evolution, conceptualizing modern video foundation models as the combination of two core components.<n>We trace the progression of video generation through four generations, culminating in a video generation model that embodies intrinsic physical plausibility.
  arXiv  Detail & Related papers  (2025-11-11T18:59:50Z)
• Dyn-O: Building Structured World Models with Object-Centric Representations [42.65409148846005]
  We introduce Dyn-O, an enhanced structured world model built upon object-centric representations.<n>Compared to prior work in object-centric representations, Dyn-O improves in both learning representations and modeling dynamics.<n>We find that our method can learn object-centric world models directly from pixel observations, outperforming DreamerV3 in rollout prediction accuracy.
  arXiv  Detail & Related papers  (2025-07-04T05:06:15Z)
• SlotPi: Physics-informed Object-centric Reasoning Models [37.32107835829927]
  We introduce SlotPi, a physics-informed object-centric reasoning model.<n>Our experiments highlight the model's strengths in tasks such as prediction and Visual Question Answering (VQA) on benchmark and fluid datasets.<n>We have created a real-world dataset encompassing object interactions, fluid dynamics, and fluid-object interactions, on which we validated our model's capabilities.
  arXiv  Detail & Related papers  (2025-06-12T14:53:36Z)
• Learning 3D Persistent Embodied World Models [84.40585374179037]
  We introduce a new persistent embodied world model with an explicit memory of previously generated content.<n>During generation time, our video diffusion model predicts RGB-D video of the future observations of the agent.<n>This generation is then aggregated into a persistent 3D map of the environment.
  arXiv  Detail & Related papers  (2025-05-05T17:59:17Z)
• Visual Grounding of Learned Physical Models [66.04898704928517]
  Humans intuitively recognize objects' physical properties and predict their motion, even when the objects are engaged in complicated interactions. We present a neural model that simultaneously reasons about physics and makes future predictions based on visual and dynamics priors. Experiments show that our model can infer the physical properties within a few observations, which allows the model to quickly adapt to unseen scenarios and make accurate predictions into the future.
  arXiv  Detail & Related papers  (2020-04-28T17:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.