Video-Driven Graph Network-Based Simulators

• URL: http://arxiv.org/abs/2409.15344v1
• Date: Tue, 10 Sep 2024 07:04:48 GMT
• Title: Video-Driven Graph Network-Based Simulators
• Authors: Franciszek Szewczyk, Gilles Louppe, Matthia Sabatelli,
• Abstract summary: This paper presents a method that can infer a system's physical properties from a short video. The learned representation is then used within a Graph Network-based Simulator to emulate the trajectories of physical systems. We demonstrate that the video-derived encodings effectively capture the physical properties of the system and showcase a linear dependence between some of the encodings and the system's motion.
• Score: 7.687678490751104
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lifelike visualizations in design, cinematography, and gaming rely on precise physics simulations, typically requiring extensive computational resources and detailed physical input. This paper presents a method that can infer a system's physical properties from a short video, eliminating the need for explicit parameter input, provided it is close to the training condition. The learned representation is then used within a Graph Network-based Simulator to emulate the trajectories of physical systems. We demonstrate that the video-derived encodings effectively capture the physical properties of the system and showcase a linear dependence between some of the encodings and the system's motion.

Related papers

• PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation [29.831214435147583]
  We present PhysGen, a novel image-to-video generation method. It produces a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process.
  arXiv  Detail & Related papers  (2024-09-27T17:59:57Z)
• Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video [58.043569985784806]
  We introduce latent intuitive physics, a transfer learning framework for physics simulation. It can infer hidden properties of fluids from a single 3D video and simulate the observed fluid in novel scenes. We validate our model in three ways: (i) novel scene simulation with the learned visual-world physics, (ii) future prediction of the observed fluid dynamics, and (iii) supervised particle simulation.
  arXiv  Detail & Related papers  (2024-06-18T16:37:44Z)
• DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians with Video Diffusion Priors [75.83647027123119]
  We propose to learn the physical properties of a material field with video diffusion priors. We then utilize a physics-based Material-Point-Method simulator to generate 4D content with realistic motions.
  arXiv  Detail & Related papers  (2024-06-03T16:05:25Z)
• Learning 3D Particle-based Simulators from RGB-D Videos [15.683877597215494]
  We propose a method for learning simulators directly from observations. Visual Particle Dynamics (VPD) jointly learns a latent particle-based representation of 3D scenes. Unlike existing 2D video prediction models, VPD's 3D structure enables scene editing and long-term predictions.
  arXiv  Detail & Related papers  (2023-12-08T20:45:34Z)
• NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos [82.74918564737591]
  We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches.
  arXiv  Detail & Related papers  (2022-10-22T04:57:55Z)
• Distilling Governing Laws and Source Input for Dynamical Systems from Videos [13.084113582897965]
  Distilling interpretable physical laws from videos has led to expanded interest in the computer vision community. This paper introduces an end-to-end unsupervised deep learning framework to uncover the explicit governing equations of dynamics presented by moving object(s) based on recorded videos.
  arXiv  Detail & Related papers  (2022-05-03T05:40:01Z)
• Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language [92.7638697243969]
  We propose a unified framework that can jointly learn visual concepts and infer physics models of objects from videos and language. This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine.
  arXiv  Detail & Related papers  (2021-10-28T17:59:13Z)
• gradSim: Differentiable simulation for system identification and visuomotor control [66.37288629125996]
  We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering. Our unified graph enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision.
  arXiv  Detail & Related papers  (2021-04-06T16:32:01Z)
• Non-linear State-space Model Identification from Video Data using Deep Encoders [0.0]
  We propose a novel non-linear state-space identification method starting from high-dimensional input and output data. An encoder function, represented by a neural network, is introduced to learn a reconstructability map to estimate the model states from past inputs and outputs. We apply the proposed method to a video stream of a simulated environment of a controllable ball in a unit box.
  arXiv  Detail & Related papers  (2020-12-14T17:14:46Z)
• Learning to Identify Physical Parameters from Video Using Differentiable Physics [2.15242029196761]
  We propose a differentiable physics engine within an action-conditional video representation network to learn a physical latent representation. We demonstrate that our network can learn to encode images and identify physical properties like mass and friction from videos and action sequences.
  arXiv  Detail & Related papers  (2020-09-17T13:36:57Z)
• Learning to Simulate Complex Physics with Graph Networks [68.43901833812448]
  We present a machine learning framework and model implementation that can learn to simulate a wide variety of challenging physical domains. Our framework---which we term "Graph Network-based Simulators" (GNS)--represents the state of a physical system with particles, expressed as nodes in a graph, and computes dynamics via learned message-passing. Our results show that our model can generalize from single-timestep predictions with thousands of particles during training, to different initial conditions, thousands of timesteps, and at least an order of magnitude more particles at test time.
  arXiv  Detail & Related papers  (2020-02-21T16:44:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.