Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenes

• URL: http://arxiv.org/abs/2501.01101v1
• Date: Thu, 02 Jan 2025 06:50:25 GMT
• Title: Deformable Gaussian Splatting for Efficient and High-Fidelity Reconstruction of Surgical Scenes
• Authors: Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh, Shing Shin Cheng, Hesheng Wang,
• Abstract summary: EH-SurGS is an efficient and high-fidelity reconstruction algorithm for deformable surgical scenes. We present an adaptive motion hierarchy strategy that distinguishes between static and deformable regions within the surgical scene.
• Score: 11.151168064680444
• License:
• Abstract: Efficient and high-fidelity reconstruction of deformable surgical scenes is a critical yet challenging task. Building on recent advancements in 3D Gaussian splatting, current methods have seen significant improvements in both reconstruction quality and rendering speed. However, two major limitations remain: (1) difficulty in handling irreversible dynamic changes, such as tissue shearing, which are common in surgical scenes; and (2) the lack of hierarchical modeling for surgical scene deformation, which reduces rendering speed. To address these challenges, we introduce EH-SurGS, an efficient and high-fidelity reconstruction algorithm for deformable surgical scenes. We propose a deformation modeling approach that incorporates the life cycle of 3D Gaussians, effectively capturing both regular and irreversible deformations, thus enhancing reconstruction quality. Additionally, we present an adaptive motion hierarchy strategy that distinguishes between static and deformable regions within the surgical scene. This strategy reduces the number of 3D Gaussians passing through the deformation field, thereby improving rendering speed. Extensive experiments demonstrate that our method surpasses existing state-of-the-art approaches in both reconstruction quality and rendering speed. Ablation studies further validate the effectiveness and necessity of our proposed components. We will open-source our code upon acceptance of the paper.

Related papers

• SurgicalGS: Dynamic 3D Gaussian Splatting for Accurate Robotic-Assisted Surgical Scene Reconstruction [18.074890506856114]
  We present SurgicalGS, a dynamic 3D Gaussian Splatting framework specifically designed for surgical scene reconstruction with improved geometric accuracy. Our approach first initialises a Gaussian point cloud using depth priors, employing binary motion masks to identify pixels with significant depth variations and fusing point clouds from depth maps across frames for initialisation. We use the Flexible Deformation Model to represent dynamic scene and introduce a normalised depth regularisation loss along with an unsupervised depth smoothness constraint to ensure more accurate geometric reconstruction.
  arXiv  Detail & Related papers  (2024-10-11T22:46:46Z)
• Deep intra-operative illumination calibration of hyperspectral cameras [73.08443963791343]
  Hyperspectral imaging (HSI) is emerging as a promising novel imaging modality with various potential surgical applications. We show that dynamically changing lighting conditions in the operating room dramatically affect the performance of HSI applications. We propose a novel learning-based approach to automatically recalibrating hyperspectral images during surgery.
  arXiv  Detail & Related papers  (2024-09-11T08:30:03Z)
• A Review of 3D Reconstruction Techniques for Deformable Tissues in Robotic Surgery [8.909938295090827]
  NeRF-based techniques have recently garnered attention for the ability to reconstruct scenes implicitly. On the other hand, 3D-GS represents scenes explicitly using 3D Gaussians and projects them onto a 2D plane as a replacement for the complex volume rendering in NeRF. This work explores and reviews state-of-the-art (SOTA) approaches, discussing their innovations and implementation principles.
  arXiv  Detail & Related papers  (2024-08-08T12:51:23Z)
• SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction [17.126895638077574]
  Dynamic reconstruction of deformable tissues in endoscopic video is a key technology for robot-assisted surgery. NeRFs struggle to capture intricate details of objects in the scene. Our network outperforms existing method on many aspects, including rendering quality, rendering speed and GPU usage.
  arXiv  Detail & Related papers  (2024-07-06T09:31:30Z)
• HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction [13.012536387221669]
  Robot-assisted minimally invasive surgery benefits from enhancing dynamic scene reconstruction, as it improves surgical outcomes. NeRF have been effective in scene reconstruction, but their slow inference speeds and lengthy training durations limit their applicability. 3D Gaussian Splatting (3D-GS) based methods have emerged as a recent trend, offering rapid inference capabilities and superior 3D quality. In this paper, we propose HFGS, a novel approach for deformable endoscopic reconstruction that addresses these challenges from spatial and temporal frequency perspectives.
  arXiv  Detail & Related papers  (2024-05-28T06:48:02Z)
• Deform3DGS: Flexible Deformation for Fast Surgical Scene Reconstruction with Gaussian Splatting [20.147880388740287]
  This work presents a novel fast reconstruction framework, termed Deform3DGS, for deformable tissues during endoscopic surgery. We introduce 3D Gaussian Splatting, an emerging technology in real-time 3D rendering, into surgical scenes by integrating a point cloud. We also propose a novel flexible deformation modeling scheme (FDM) to learn tissue deformation dynamics at the level of individual Gaussians.
  arXiv  Detail & Related papers  (2024-05-28T05:14:57Z)
• Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction [89.53963284958037]
  We propose a novel motion-aware enhancement framework for dynamic scene reconstruction. Specifically, we first establish a correspondence between 3D Gaussian movements and pixel-level flow. For the prevalent deformation-based paradigm that presents a harder optimization problem, a transient-aware deformation auxiliary module is proposed.
  arXiv  Detail & Related papers  (2024-03-18T03:46:26Z)
• Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane [58.871015937204255]
  We introduce Fast Orthogonal Plane (plane) for the reconstruction of deformable tissues. We conceptualize surgical procedures as 4D volumes, and break them down into static and dynamic fields comprised of neural planes. This factorization iscretizes four-dimensional space, leading to a decreased memory usage and faster optimization.
  arXiv  Detail & Related papers  (2023-12-23T13:27:50Z)
• Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues [52.886545681833596]
  LerPlane is a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields. LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling.
  arXiv  Detail & Related papers  (2023-05-31T14:38:35Z)
• NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction [95.37644907940857]
  We propose a fast neural surface reconstruction approach, called NeuS2. NeuS2 achieves two orders of magnitude improvement in terms of acceleration without compromising reconstruction quality. We extend our method for fast training of dynamic scenes, with a proposed incremental training strategy and a novel global transformation prediction component.
  arXiv  Detail & Related papers  (2022-12-10T07:19:43Z)
• Limited-angle tomographic reconstruction of dense layered objects by dynamical machine learning [68.9515120904028]
  Limited-angle tomography of strongly scattering quasi-transparent objects is a challenging, highly ill-posed problem. Regularizing priors are necessary to reduce artifacts by improving the condition of such problems. We devised a recurrent neural network (RNN) architecture with a novel split-convolutional gated recurrent unit (SC-GRU) as the building block.
  arXiv  Detail & Related papers  (2020-07-21T11:48:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.