EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting

• URL: http://arxiv.org/abs/2605.16022v1
• Date: Fri, 15 May 2026 14:56:54 GMT
• Title: EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting
• Abstract summary: In robot-assisted minimally invasive surgery, dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes.<n>We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting.
• Score: 14.379676968984013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In robot-assisted minimally invasive surgery, high-fidelity dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes. However, existing methods primarily focus on visual reconstruction, lacking physics-based descriptions of the scene required for realistic simulation. We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting. Our approach utilizes 4D Gaussian Splatting (4DGS) integrated with pre-trained segmentation and depth estimation to represent deformable tissues and tools. To achieve automatic inference of physical properties, we introduce an object-wise material field that initializes material parameters via MLLM and refines them through a differentiable Material Point Method (MPM) under joint supervision from rendered images and optical flow. Validated on both open-source and in-house datasets, our framework achieves superior simulation fidelity and physical accuracy compared to state-of-the-art methods, underscoring its potential to advance robot-assisted surgical applications.

Related papers

• FastPhysGS: Accelerating Physics-based Dynamic 3DGS Simulation via Interior Completion and Adaptive Optimization [56.17833729527066]
  We propose FastPhysGS, a framework for physics-based dynamic 3DGS simulation.<n>FastPhysGS achieves high-fidelity physical simulation in 1 minute using only 7 GB runtime memory.
  arXiv  Detail & Related papers  (2026-02-02T07:00:42Z)
• Mirage2Matter: A Physically Grounded Gaussian World Model from Video [87.9732484393686]
  We present Simulate Anything, a graphics-driven world modeling and simulation framework.<n>Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS)<n>We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target.
  arXiv  Detail & Related papers  (2026-01-24T07:43:57Z)
• Efficient 3D Scene Reconstruction and Simulation from Sparse Endoscopic Views [0.7614628596146601]
  We propose a Gaussian Splatting-based framework to reconstruct interactive surgical scenes from endoscopic data.<n>A key challenge in this data-driven simulation paradigm is the restricted movement of endoscopic cameras.<n>We show that our method can efficiently reconstruct and simulate surgical scenes from sparse endoscopic views.
  arXiv  Detail & Related papers  (2025-09-21T10:40:36Z)
• MASIV: Toward Material-Agnostic System Identification from Videos [76.36666848173141]
  MASIV is a vision-based framework for material-agnostic system identification.<n>It employs learnable neural models, inferring object dynamics without assuming a scene-specific material prior.<n>It achieves state-of-the-art performance in geometric accuracy, rendering quality, and generalization ability.
  arXiv  Detail & Related papers  (2025-08-01T23:23:45Z)
• GausSim: Foreseeing Reality by Gaussian Simulator for Elastic Objects [55.02281855589641]
  GausSim is a novel neural network-based simulator designed to capture the dynamic behaviors of real-world elastic objects represented through Gaussian kernels.<n>We leverage continuum mechanics and treat each kernel as a Center of Mass System (CMS) that represents continuous piece of matter.<n>In addition, GausSim incorporates explicit physics constraints, such as mass and momentum conservation, ensuring interpretable results and robust, physically plausible simulations.
  arXiv  Detail & Related papers  (2024-12-23T18:58:17Z)
• Realistic Surgical Simulation from Monocular Videos [49.114370617718244]
  This paper tackles the challenge of automatically performing realistic surgical simulations from readily available surgical videos.<n>We propose SurgiSim, a novel automatic simulation system to overcome these limitations.<n> Experiments on various surgical scenarios and interactions demonstrate SurgiSim's ability to perform realistic simulation of soft tissues.
  arXiv  Detail & Related papers  (2024-12-03T10:32:41Z)
• PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation [9.306758077479472]
  PhysFlow is a novel approach that leverages multi-modal foundation models and video diffusion to achieve enhanced 4D dynamic scene simulation.<n>This integrated framework enables accurate prediction and realistic simulation of dynamic interactions in real-world scenarios.
  arXiv  Detail & Related papers  (2024-11-21T18:55:23Z)
• Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting [32.846428862045634]
  We present Sim Anything, a physics-based approach that endows static 3D objects with interactive dynamics.<n>Inspired by human visual reasoning, we propose MLLM-based Physical Property Perception.<n>We also simulate objects in an open-world scene with particles sampled via the Physical-Geometric Adaptive Sampling.
  arXiv  Detail & Related papers  (2024-11-19T12:52:21Z)
• Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics [48.99021224773799]
  We propose the Neural Material Adaptor (NeuMA), which integrates existing physical laws with learned corrections. We also propose Particle-GS, a particle-driven 3D Gaussian Splatting variant that bridges simulation and observed images.
  arXiv  Detail & Related papers  (2024-10-10T17:43:36Z)
• SimEndoGS: Efficient Data-driven Scene Simulation using Robotic Surgery Videos via Physics-embedded 3D Gaussians [19.590481146949685]
  We introduce 3D Gaussian as a learnable representation for surgical scene, which is learned from stereo endoscopic video. We apply the Material Point Method, which is integrated with physical properties, to the 3D Gaussians to achieve realistic scene deformations. Results show that it can reconstruct and simulate surgical scenes from endoscopic videos efficiently-taking only a few minutes to reconstruct the surgical scene.
  arXiv  Detail & Related papers  (2024-05-02T02:34:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.