Registering Neural 4D Gaussians for Endoscopic Surgery

• URL: http://arxiv.org/abs/2407.20213v1
• Date: Mon, 29 Jul 2024 17:42:45 GMT
• Title: Registering Neural 4D Gaussians for Endoscopic Surgery
• Authors: Yiming Huang, Beilei Cui, Ikemura Kei, Jiekai Zhang, Long Bai, Hongliang Ren,
• Abstract summary: We propose a novel strategy for dynamic surgical neural scene registration. By incorporating both spatial and temporal information for correspondence matching, our approach achieves superior performance. The proposed method has the potential to improve surgical planning and training, ultimately leading to better patient outcomes.
• Score: 6.270137275503944
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The recent advance in neural rendering has enabled the ability to reconstruct high-quality 4D scenes using neural networks. Although 4D neural reconstruction is popular, registration for such representations remains a challenging task, especially for dynamic scene registration in surgical planning and simulation. In this paper, we propose a novel strategy for dynamic surgical neural scene registration. We first utilize 4D Gaussian Splatting to represent the surgical scene and capture both static and dynamic scenes effectively. Then, a spatial aware feature aggregation method, Spatially Weight Cluttering (SWC) is proposed to accurately align the feature between surgical scenes, enabling precise and realistic surgical simulations. Lastly, we present a novel strategy of deformable scene registration to register two dynamic scenes. By incorporating both spatial and temporal information for correspondence matching, our approach achieves superior performance compared to existing registration methods for implicit neural representation. The proposed method has the potential to improve surgical planning and training, ultimately leading to better patient outcomes.

Related papers

• Real-Time Spatio-Temporal Reconstruction of Dynamic Endoscopic Scenes with 4D Gaussian Splatting [1.7947477507955865]
  This paper presents ST-Endo4DGS, a novel framework that models the dynamics of dynamic endoscopic scenes. This approach enables precise representation of deformable tissue, capturing spatial and temporal correlations in real time.
  arXiv  Detail & Related papers  (2024-11-02T11:24:27Z)
• Intraoperative Registration by Cross-Modal Inverse Neural Rendering [61.687068931599846]
  We present a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. We tested our method on retrospective patients' data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration.
  arXiv  Detail & Related papers  (2024-09-18T13:40:59Z)
• SurgicalGaussian: Deformable 3D Gaussians for High-Fidelity Surgical Scene Reconstruction [17.126895638077574]
  Dynamic reconstruction of deformable tissues in endoscopic video is a key technology for robot-assisted surgery. NeRFs struggle to capture intricate details of objects in the scene. Our network outperforms existing method on many aspects, including rendering quality, rendering speed and GPU usage.
  arXiv  Detail & Related papers  (2024-07-06T09:31:30Z)
• SimEndoGS: Efficient Data-driven Scene Simulation using Robotic Surgery Videos via Physics-embedded 3D Gaussians [19.590481146949685]
  We introduce 3D Gaussian as a learnable representation for surgical scene, which is learned from stereo endoscopic video. We apply the Material Point Method, which is integrated with physical properties, to the 3D Gaussians to achieve realistic scene deformations. Results show that it can reconstruct and simulate surgical scenes from endoscopic videos efficiently-taking only a few minutes to reconstruct the surgical scene.
  arXiv  Detail & Related papers  (2024-05-02T02:34:19Z)
• Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane [58.871015937204255]
  We introduce Fast Orthogonal Plane (plane) for the reconstruction of deformable tissues. We conceptualize surgical procedures as 4D volumes, and break them down into static and dynamic fields comprised of neural planes. This factorization iscretizes four-dimensional space, leading to a decreased memory usage and faster optimization.
  arXiv  Detail & Related papers  (2023-12-23T13:27:50Z)
• DNS SLAM: Dense Neural Semantic-Informed SLAM [92.39687553022605]
  DNS SLAM is a novel neural RGB-D semantic SLAM approach featuring a hybrid representation. Our method integrates multi-view geometry constraints with image-based feature extraction to improve appearance details. Our experimental results achieve state-of-the-art performance on both synthetic data and real-world data tracking.
  arXiv  Detail & Related papers  (2023-11-30T21:34:44Z)
• Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues [52.886545681833596]
  LerPlane is a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields. LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling.
  arXiv  Detail & Related papers  (2023-05-31T14:38:35Z)
• NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fields [99.57774680640581]
  We present an efficient framework capable of fast reconstruction, compact modeling, and streamable rendering. We propose to decompose the 4D space according to temporal characteristics. Points in the 4D space are associated with probabilities belonging to three categories: static, deforming, and new areas.
  arXiv  Detail & Related papers  (2022-10-28T07:11:05Z)
• Neural Adaptive SCEne Tracing [24.781844909539686]
  We present NAScenT, the first neural rendering method based on directly training a hybrid explicit-implicit neural representation. NAScenT is capable of reconstructing challenging scenes including both large, sparsely populated volumes like UAV captured outdoor environments.
  arXiv  Detail & Related papers  (2022-02-28T10:27:23Z)
• Neural Radiance Flow for 4D View Synthesis and Video Processing [59.9116932930108]
  We present a method to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images. Key to our approach is the use of a neural implicit representation that learns to capture the 3D occupancy, radiance, and dynamics of the scene.
  arXiv  Detail & Related papers  (2020-12-17T17:54:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.