Learning a Particle Dynamics Model with Real-world Videos

• URL: http://arxiv.org/abs/2605.23845v1
• Date: Fri, 22 May 2026 16:56:27 GMT
• Title: Learning a Particle Dynamics Model with Real-world Videos
• Abstract summary: We introduce a novel framework for training neural object dynamics models directly from unlabeled real-world videos.<n>The model is trained via rendering supervision, enabling learning from real-world videos without requiring particle-level labeled states.<n>To enable this study, we also present a real-world dataset consisting of about 500 videos capturing diverse object interactions.
• Score: 3.8381345540147414
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data-driven learning approaches for physics simulation, sometimes referred to as world models, have emerged as promising alternatives to traditional physics simulators due to their differentiable nature. Prior work has demonstrated impressive results in predicting the motions of rigid and non-rigid objects in complex scenes involving multiple interacting bodies. However, these models are typically trained in simulated environments because obtaining perfect state information such as complete scene point clouds and point correspondences over time is challenging in real-world settings. This reliance on synthetic data can limit their applicability when the sim-to-real gap is large. In this work, we aim to overcome these limitations by introducing a novel framework for training neural object dynamics models directly from unlabeled real-world videos. Specifically, we propose to learn a particle-based dynamics model compatible with a Gaussian splatting framework, which operates on dense particles derived from Gaussians (i.e., particles with scales and rotations) and predicts their position and rotation changes over time. The model is trained via rendering supervision, enabling learning from real-world videos without requiring particle-level labeled states. Our model operates directly on dense Gaussians without relying on heuristic subsampling anchor points. To enable this study, we also present a real-world dataset consisting of about 500 videos capturing diverse object interactions.

Related papers

• PhysiFormer: Learning to Simulate Mechanics in World Space [56.16644631495]
  We present PhysiFormer, a diffusion transformer for physically-plausible 3D object motion.<n>It represents objects as 3D meshes expressed in world coordinates.<n>It substantially outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency.
  arXiv  Detail & Related papers  (2026-06-25T17:59:00Z)
• Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics [23.400128774539812]
  We propose a novel framework to overcome the data-scarcity bottleneck in HOI generation.<n>Our method demonstrates enhanced generalization to unseen objects and the capability of long-horizon generation.
  arXiv  Detail & Related papers  (2026-06-22T03:32:35Z)
• Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling [51.40150411616207]
  We introduce Latent Particle World Model (LPWM), a self-supervised object-centric world model scaled to real-world multi-object datasets.<n>LPWM autonomously discovers keypoints, bounding boxes, and object masks directly from video data.<n>Our architecture is trained end-to-end purely from videos and supports flexible conditioning on actions, language, and image goals.
  arXiv  Detail & Related papers  (2026-03-04T19:36:08Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goals [15.286299359279509]
  Goal Force allows users to define goals via explicit force vectors and intermediate dynamics.<n>We train a video generation model on a curated dataset of synthetic causal primitives.<n>Our results suggest that by grounding video generation in fundamental physical interactions, models can emerge as implicit neural physics simulators.
  arXiv  Detail & Related papers  (2026-01-09T15:23:36Z)
• What Happens Next? Anticipating Future Motion by Generating Point Trajectories [76.16266402727643]
  We consider the problem of forecasting motion from a single image, predicting how objects in the world are likely to move.<n>We formulate this task as conditional generation of dense trajectory grids with a model that closely follows the architecture of modern video generators.<n>This approach captures scene-wide dynamics and uncertainty, yielding more accurate and diverse predictions than prior regressors and generators.
  arXiv  Detail & Related papers  (2025-09-25T21:03:56Z)
• Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos [28.473117601626956]
  We develop a neural dynamics framework that combines object particles and spatial grids in a hybrid representation.<n>We demonstrate that our model learns the dynamics of diverse objects from sparse-view RGB-D recordings of robot-object interactions.<n>Our approach outperforms state-of-the-art learning-based and physics-based simulators, particularly in scenarios with limited camera views.
  arXiv  Detail & Related papers  (2025-06-18T17:59:38Z)
• Learning 3D-Gaussian Simulators from RGB Videos [20.250137125726265]
  3DGSim is a learned 3D simulator that learns physical interactions from multi-view RGB videos.<n>It unifies 3D scene reconstruction, particle dynamics prediction and video synthesis into an end-to-end trained framework.
  arXiv  Detail & Related papers  (2025-03-31T12:33:59Z)
• GausSim: Foreseeing Reality by Gaussian Simulator for Elastic Objects [55.02281855589641]
  GausSim is a novel neural network-based simulator designed to capture the dynamic behaviors of real-world elastic objects represented through Gaussian kernels.<n>We leverage continuum mechanics and treat each kernel as a Center of Mass System (CMS) that represents continuous piece of matter.<n>In addition, GausSim incorporates explicit physics constraints, such as mass and momentum conservation, ensuring interpretable results and robust, physically plausible simulations.
  arXiv  Detail & Related papers  (2024-12-23T18:58:17Z)
• Towards Complex Dynamic Physics System Simulation with Graph Neural ODEs [75.7104463046767]
  This paper proposes a novel learning based simulation model that characterizes the varying spatial and temporal dependencies in particle systems. We empirically evaluate GNSTODE's simulation performance on two real-world particle systems, Gravity and Coulomb.
  arXiv  Detail & Related papers  (2023-05-21T03:51:03Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.