EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos

• URL: http://arxiv.org/abs/2307.11307v2
• Date: Mon, 4 Sep 2023 03:55:03 GMT
• Title: EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos
• Authors: Ruyi Zha, Xuelian Cheng, Hongdong Li, Mehrtash Harandi, Zongyuan Ge
• Abstract summary: Reconstructing soft tissues from stereo endoscope videos is an essential prerequisite for many medical applications. Previous methods struggle to produce high-quality geometry and appearance due to their inadequate representations of 3D scenes. We propose a novel neural-field-based method, called EndoSurf, which effectively learns to represent a deforming surface from an RGBD sequence.
• Score: 72.59573904930419
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing soft tissues from stereo endoscope videos is an essential prerequisite for many medical applications. Previous methods struggle to produce high-quality geometry and appearance due to their inadequate representations of 3D scenes. To address this issue, we propose a novel neural-field-based method, called EndoSurf, which effectively learns to represent a deforming surface from an RGBD sequence. In EndoSurf, we model surface dynamics, shape, and texture with three neural fields. First, 3D points are transformed from the observed space to the canonical space using the deformation field. The signed distance function (SDF) field and radiance field then predict their SDFs and colors, respectively, with which RGBD images can be synthesized via differentiable volume rendering. We constrain the learned shape by tailoring multiple regularization strategies and disentangling geometry and appearance. Experiments on public endoscope datasets demonstrate that EndoSurf significantly outperforms existing solutions, particularly in reconstructing high-fidelity shapes. Code is available at https://github.com/Ruyi-Zha/endosurf.git.

Related papers

• DynoSurf: Neural Deformation-based Temporally Consistent Dynamic Surface Reconstruction [93.18586302123633]
  This paper explores the problem of reconstructing temporally consistent surfaces from a 3D point cloud sequence without correspondence. We propose DynoSurf, an unsupervised learning framework integrating a template surface representation with a learnable deformation field. Experimental results demonstrate the significant superiority of DynoSurf over current state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-18T08:58:48Z)
• EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting [20.848027172010358]
  We present EndoGS, applying Gaussian Splatting for deformable endoscopic tissue reconstruction. Our approach incorporates deformation fields to handle dynamic scenes, depth-guided supervision with spatial-temporal weight masks, and surface-aligned regularization terms. As a result, EndoGS reconstructs and renders high-quality deformable endoscopic tissues from a single-viewpoint video, estimated depth maps, and labeled tool masks.
  arXiv  Detail & Related papers  (2024-01-21T16:14:04Z)
• Nuvo: Neural UV Mapping for Unruly 3D Representations [61.87715912587394]
  Existing UV mapping algorithms operate on geometry produced by state-of-the-art 3D reconstruction and generation techniques. We present a UV mapping method designed to operate on geometry produced by 3D reconstruction and generation techniques.
  arXiv  Detail & Related papers  (2023-12-11T18:58:38Z)
• NeuSD: Surface Completion with Multi-View Text-to-Image Diffusion [56.98287481620215]
  We present a novel method for 3D surface reconstruction from multiple images where only a part of the object of interest is captured. Our approach builds on two recent developments: surface reconstruction using neural radiance fields for the reconstruction of the visible parts of the surface, and guidance of pre-trained 2D diffusion models in the form of Score Distillation Sampling (SDS) to complete the shape in unobserved regions in a plausible manner.
  arXiv  Detail & Related papers  (2023-12-07T19:30:55Z)
• DynamicSurf: Dynamic Neural RGB-D Surface Reconstruction with an Optimizable Feature Grid [7.702806654565181]
  DynamicSurf is a model-free neural implicit surface reconstruction method for high-fidelity 3D modelling of non-rigid surfaces from monocular RGB-D video. We learn a neural deformation field that maps a canonical representation of the surface geometry to the current frame. We demonstrate it can optimize sequences of varying frames with $6$ speedup over pure-based approaches.
  arXiv  Detail & Related papers  (2023-11-14T13:39:01Z)
• NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes [56.31855837632735]
  We propose a compact and flexible architecture that enables easy 3D surface reconstruction from any NeRF-driven approach. Our final 3D mesh is physically accurate and can be rendered in real time on an array of devices.
  arXiv  Detail & Related papers  (2023-03-16T16:06:03Z)
• DiffusionSDF: Conditional Generative Modeling of Signed Distance Functions [42.015077094731815]
  DiffusionSDF is a generative model for shape completion, single-view reconstruction, and reconstruction of real-scanned point clouds. We use neural signed distance functions (SDFs) as our 3D representation to parameterize the geometry of various signals (e.g., point clouds, 2D images) through neural networks.
  arXiv  Detail & Related papers  (2022-11-24T18:59:01Z)
• NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos [82.74918564737591]
  We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches.
  arXiv  Detail & Related papers  (2022-10-22T04:57:55Z)
• Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis [4.597864989500202]
  We propose a new method for reconstructing implicit 3D human models from sparse multi-view RGB videos. Our method defines the neural scene representation on the mesh surface points and signed distances from the surface of a human body mesh.
  arXiv  Detail & Related papers  (2022-01-05T16:25:32Z)
• Dynamic Reconstruction of Deformable Soft-tissue with Stereo Scope in Minimal Invasive Surgery [24.411005883017832]
  In minimal invasive surgery, it is important to rebuild and visualize the latest deformed shape of soft-tissue surfaces. This paper proposes an innovative Simultaneous localization and Mapping (SLAM) algorithm for deformable dense reconstruction of surfaces. In-vivo experiments with publicly available datasets demonstrate that the 3D models can be incrementally built for different soft-tissues.
  arXiv  Detail & Related papers  (2020-03-22T16:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.