Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators

• URL: http://arxiv.org/abs/2605.20978v1
• Date: Wed, 20 May 2026 10:08:13 GMT
• Title: Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators
• Abstract summary: We introduce Point Cloud temporal cloud sequence to adapt a learned simulator to unseen physical properties during inference.<n>We demonstrate that PEACH is capable of accurate zero-shot sim-to-real transfer on a challenging, dynamic scene.
• Score: 18.946877621144193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Graph Network Simulators (GNSs) have emerged as powerful surrogates for complex physics-based simulation, offering inherent differentiability and orders-of-magnitude speedups over traditional solvers. However, GNSs typically assume access to the underlying material parameters, such as stiffness or viscosity, severely limiting their utility in realistic experimental settings. While recent meta-learning approaches address the parameter dependency by inferring properties from mesh trajectories, reconstructing a mesh from an observed scene is challenging. In this work, we introduce Point Cloud Encoding for Accurate Context Handling (PEACH), a novel framework that applies in-context learning on point clouds to adapt a learned simulator to unseen physical properties during inference. Our approach relies on a novel spatio-temporal point cloud sequence encoder, as well as two forms of auxiliary supervision to help improve simulation fidelity. We demonstrate that PEACH is capable of accurate zero-shot sim-to-real transfer on a challenging, dynamic scene. Experiments on simulation scenes show that PEACH even outperforms mesh-based baselines on prediction accuracy, while being much more practical for real-world deployment.

Related papers

• Context-aware Learned Mesh-based Simulation via Trajectory-Level Meta-Learning [20.72669976554826]
  Learned Graph Network Simulators (GNSs) offer a promising alternative to traditional mesh-based physics simulators.<n>Their speed and inherent differentiability make them well suited for applications that require fast and accurate simulations, such as robotic manipulation or manufacturing optimization.<n>We utilize movement primitives to directly predict fast, stable and accurate simulations from a single model call.
  arXiv  Detail & Related papers  (2025-11-07T13:34:02Z)
• MaNGO - Adaptable Graph Network Simulators via Meta-Learning [25.80703650406406]
  Graph Network Simulators (GNSs) offer faster inference but suffer from two key limitations.<n>They must be retrained from scratch for even minor variations in physical parameters.<n>This is inefficient, as simulations with varying parameters often share a common underlying latent structure.<n>We propose a novel architecture that generates a latent representation by encoding graph trajectories using conditional neural processes.
  arXiv  Detail & Related papers  (2025-10-07T12:44:24Z)
• GausSim: Foreseeing Reality by Gaussian Simulator for Elastic Objects [55.02281855589641]
  GausSim is a novel neural network-based simulator designed to capture the dynamic behaviors of real-world elastic objects represented through Gaussian kernels.<n>We leverage continuum mechanics and treat each kernel as a Center of Mass System (CMS) that represents continuous piece of matter.<n>In addition, GausSim incorporates explicit physics constraints, such as mass and momentum conservation, ensuring interpretable results and robust, physically plausible simulations.
  arXiv  Detail & Related papers  (2024-12-23T18:58:17Z)
• Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning [49.91297276176978]
  We propose a novel intrinsic textbfPointGST (textbfGraph textbfSpectral textbfTuning) method for point cloud.<n>Point GST freezes the pre-trained model and introduces a lightweight, trainable Point Cloud Spectral Adapter (PCSA) for fine-tuning parameters in the spectral domain.<n>Experiments on challenging point cloud datasets across various tasks demonstrate that Point GST not only outperforms its fully fine-tuning counterpart but also significantly reduces trainable parameters.
  arXiv  Detail & Related papers  (2024-10-10T17:00:04Z)
• Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks [93.38375271826202]
  We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. We first build a simulator by integrating Gaussian splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.
  arXiv  Detail & Related papers  (2024-06-21T13:48:37Z)
• Latent Task-Specific Graph Network Simulators [16.881339139068018]
  Graph Network Simulators (GNSs) pose an efficient alternative to traditional physics-based simulators. We frame mesh-based simulation as a meta-learning problem and use a recent Bayesian meta-learning method to improve GNSs adaptability to new scenarios. We validate the effectiveness of our approach through various experiments, performing on par with or better than established baseline methods.
  arXiv  Detail & Related papers  (2023-11-09T10:30:51Z)
• DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation [81.11585774044848]
  We present DeepSimHO, a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization.
  arXiv  Detail & Related papers  (2023-10-11T05:34:36Z)
• Grounding Graph Network Simulators using Physical Sensor Observations [12.017054986629846]
  We integrate sensory information to ground Graph Network Simulators on real world observations. We predict the mesh state of deformable objects by utilizing point cloud data.
  arXiv  Detail & Related papers  (2023-02-23T09:06:42Z)
• Learning to Simulate Realistic LiDARs [66.7519667383175]
  We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor. We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces. We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
  arXiv  Detail & Related papers  (2022-09-22T13:12:54Z)
• DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects [19.435386865144586]
  We show how to use recent neural network architectures to perform simulation parameter inference from real point clouds.<n>We propose to leverage differentiable point cloud sampling and differentiable simulation to significantly reduce the time to achieve the alignment.<n> Experiments with highly deformable objects show that our method can achieve comparable or better alignment with real object behavior.
  arXiv  Detail & Related papers  (2022-04-07T00:45:26Z)
• Learning to Simulate Complex Physics with Graph Networks [68.43901833812448]
  We present a machine learning framework and model implementation that can learn to simulate a wide variety of challenging physical domains. Our framework---which we term "Graph Network-based Simulators" (GNS)--represents the state of a physical system with particles, expressed as nodes in a graph, and computes dynamics via learned message-passing. Our results show that our model can generalize from single-timestep predictions with thousands of particles during training, to different initial conditions, thousands of timesteps, and at least an order of magnitude more particles at test time.
  arXiv  Detail & Related papers  (2020-02-21T16:44:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.