Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects

• URL: http://arxiv.org/abs/2606.01950v1
• Date: Mon, 01 Jun 2026 09:12:07 GMT
• Title: Learning Action-Conditional and Object-Centric Gaussian Splatting World Models for Rigid Objects
• Abstract summary: We develop a novel model that learns action-conditional dynamics of rigid objects in 3D.<n>We evaluate our model in model-predictive control for non-prehensile manipulation in simulation.
• Score: 4.870609209341298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World models enable intelligent agents to predict the consequences of their actions on the environment. In this paper, we propose Multi Rigid Object Gaussian World Model (MRO-GWM), a novel model that learns action-conditional dynamics of rigid objects in 3D. By representing the scene by object-centric Gaussians, we can represent arbitrary object shapes and multi-object scenes. We develop a novel spatio-temporal transformer architecture that predicts future rigid body motion from a history of object Gaussians and future actions. Objects are represented by their Gaussians in a canonical frame, which allows for describing object motion as rigid body transformation. Our model is trained on reconstructions from multiple viewpoints, which requires the model to handle partial observations of objects due to occlusions. We analyze prediction performance of our approach on synthetic datasets composed of typical household objects with multi-object dynamics and interactions by a robot end effector. We also evaluate our model in model-predictive control for non-prehensile manipulation in simulation.

Related papers

• 4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting [72.02343855552051]
  Current world action models (WAMs) typically operate on 2D visual data.<n>We leverage an explicit 4D Gaussian Splatting (4DGS) representation that separately models dynamic objects and the static background of a scene.<n>4DGS-WAM lifts 2D observations into a persistent 4D representation so that previously observed static content can be reused during future prediction.
  arXiv  Detail & Related papers  (2026-08-26T16:16:57Z)
• Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control [86.64250947574749]
  Reconstructing articulated 3D objects is important for animation, gaming, and robotic simulations.<n>Recent neural networks can estimate the articulated structure of 3D objects, but their generalization remains limited by the scarcity of annotated data.<n>We introduce Instruct-Particulate, a model that takes a 3D mesh together with a target kinematic specification.
  arXiv  Detail & Related papers  (2026-06-12T17:59:36Z)
• Physical Object Understanding with a Physically Controllable World Model [11.085294403458972]
  A central challenge in visual intelligence is learning the physical structure of scenes from raw videos.<n>We introduce a new class of probabilistic world models that support estimation of the probability of any visual variable.<n>We show that our world model can manipulate objects in 3D, enabling applications such as Visual Jenga.
  arXiv  Detail & Related papers  (2026-05-30T00:10:32Z)
• NeuROK: Generative 4D Neural Object Kinematics [35.76495786203886]
  We learn a data-driven kinematic state parameterization for object-centric physical systems.<n>We learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset.
  arXiv  Detail & Related papers  (2026-05-28T17:59:53Z)
• Articulated 3D Scene Graphs for Open-World Mobile Manipulation [55.97942733699124]
  We present MoMa-SG, a framework for building semantic-kinematic 3D scene graphs of articulated scenes.<n>We estimate articulation models using a novel unified twist estimation formulation.<n>We also introduce the novel Arti4D-Semantic dataset.
  arXiv  Detail & Related papers  (2026-02-18T10:40:35Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• ObjectForesight: Predicting Future 3D Object Trajectories from Human Videos [48.24897274501108]
  We introduce a 3D object-centric dynamics model that predicts future 6-DoF poses and trajectories of rigid objects from short egocentric video sequences.<n>Unlike conventional world or dynamics models that operate in pixel or latent space, ObjectForesight represents the world explicitly in 3D at the object level.<n>We show that ObjectForesight achieves significant gains in accuracy, geometric consistency, and generalization to unseen objects and scenes.
  arXiv  Detail & Related papers  (2026-01-08T18:58:08Z)
• Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos [28.473117601626956]
  We develop a neural dynamics framework that combines object particles and spatial grids in a hybrid representation.<n>We demonstrate that our model learns the dynamics of diverse objects from sparse-view RGB-D recordings of robot-object interactions.<n>Our approach outperforms state-of-the-art learning-based and physics-based simulators, particularly in scenarios with limited camera views.
  arXiv  Detail & Related papers  (2025-06-18T17:59:38Z)
• PartRM: Modeling Part-Level Dynamics with Large Cross-State Reconstruction Model [23.768571323272152]
  PartRM is a novel 4D reconstruction framework that simultaneously models appearance, geometry, and part-level motion from multi-view images of a static object.<n>We introduce the PartDrag-4D dataset, providing multi-view observations of part-level dynamics across over 20,000 states.<n> Experimental results show that PartRM establishes a new state-of-the-art in part-level motion learning and can be applied in manipulation tasks in robotics.
  arXiv  Detail & Related papers  (2025-03-25T17:59:58Z)
• GAMMA: Generalizable Articulation Modeling and Manipulation for Articulated Objects [53.965581080954905]
  We propose a novel framework of Generalizable Articulation Modeling and Manipulating for Articulated Objects (GAMMA) GAMMA learns both articulation modeling and grasp pose affordance from diverse articulated objects with different categories. Results show that GAMMA significantly outperforms SOTA articulation modeling and manipulation algorithms in unseen and cross-category articulated objects.
  arXiv  Detail & Related papers  (2023-09-28T08:57:14Z)
• 3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators [24.181604511269096]
  We propose an action-conditioned dynamics model that predicts scene changes caused by object and agent interactions in a viewpoint-in 3D neural scene representation space. In this space, objects do not interfere with one another and their appearance persists over time and across viewpoints. We show our model generalizes well its predictions across varying number and appearances of interacting objects as well as across camera viewpoints.
  arXiv  Detail & Related papers  (2020-11-12T16:15:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.