High-fidelity Endoscopic Image Synthesis by Utilizing Depth-guided Neural Surfaces

• URL: http://arxiv.org/abs/2404.13437v2
• Date: Wed, 30 Oct 2024 13:31:07 GMT
• Title: High-fidelity Endoscopic Image Synthesis by Utilizing Depth-guided Neural Surfaces
• Authors: Baoru Huang, Yida Wang, Anh Nguyen, Daniel Elson, Francisco Vasconcelos, Danail Stoyanov,
• Abstract summary: We introduce a novel method for colon section reconstruction by leveraging NeuS applied to endoscopic images, supplemented by a single frame of depth map. Our approach demonstrates exceptional accuracy in completely rendering colon sections, even capturing unseen portions of the surface. This breakthrough opens avenues for achieving stable and consistently scaled reconstructions, promising enhanced quality in cancer screening procedures and treatment interventions.
• Score: 18.948630080040576
• License:
• Abstract: In surgical oncology, screening colonoscopy plays a pivotal role in providing diagnostic assistance, such as biopsy, and facilitating surgical navigation, particularly in polyp detection. Computer-assisted endoscopic surgery has recently gained attention and amalgamated various 3D computer vision techniques, including camera localization, depth estimation, surface reconstruction, etc. Neural Radiance Fields (NeRFs) and Neural Implicit Surfaces (NeuS) have emerged as promising methodologies for deriving accurate 3D surface models from sets of registered images, addressing the limitations of existing colon reconstruction approaches stemming from constrained camera movement. However, the inadequate tissue texture representation and confused scale problem in monocular colonoscopic image reconstruction still impede the progress of the final rendering results. In this paper, we introduce a novel method for colon section reconstruction by leveraging NeuS applied to endoscopic images, supplemented by a single frame of depth map. Notably, we pioneered the exploration of utilizing only one frame depth map in photorealistic reconstruction and neural rendering applications while this single depth map can be easily obtainable from other monocular depth estimation networks with an object scale. Through rigorous experimentation and validation on phantom imagery, our approach demonstrates exceptional accuracy in completely rendering colon sections, even capturing unseen portions of the surface. This breakthrough opens avenues for achieving stable and consistently scaled reconstructions, promising enhanced quality in cancer screening procedures and treatment interventions.

Related papers

• Hybrid NeRF-Stereo Vision: Pioneering Depth Estimation and 3D Reconstruction in Endoscopy [11.798218793025974]
  We introduce an innovative pipeline using Neural Radiance Fields (NeRF) for 3D reconstruction. Our approach utilizes a preliminary NeRF reconstruction that yields a coarse model, then creates a binocular scene within the reconstructed environment. High-fidelity depth maps are generated from monocular endoscopic video of a realistic cranial phantom.
  arXiv  Detail & Related papers  (2024-10-05T05:26:21Z)
• Intraoperative Registration by Cross-Modal Inverse Neural Rendering [61.687068931599846]
  We present a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. We tested our method on retrospective patients' data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration.
  arXiv  Detail & Related papers  (2024-09-18T13:40:59Z)
• ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation [67.22294293695255]
  We propose a novel reconstruction pipeline with a bi-directional adaptation architecture named ToDER to get precise depth estimations. Experimental results demonstrate that our approach can precisely predict depth maps in both realistic and synthetic colonoscopy videos.
  arXiv  Detail & Related papers  (2024-07-23T14:24:26Z)
• A Quantitative Evaluation of Dense 3D Reconstruction of Sinus Anatomy from Monocular Endoscopic Video [8.32570164101507]
  We perform a quantitative analysis of a self-supervised approach for sinus reconstruction using endoscopic sequences and optical tracking. Our results show that the generated reconstructions are in high agreement with the anatomy, yielding an average point-to-mesh error of 0.91 mm. We identify that pose and depth estimation inaccuracies contribute equally to this error and that locally consistent sequences with shorter trajectories generate more accurate reconstructions.
  arXiv  Detail & Related papers  (2023-10-22T17:11:40Z)
• Towards Abdominal 3-D Scene Rendering from Laparoscopy Surgical Videos using NeRFs [0.7106122418396085]
  We present a comprehensive examination of NeRFs in the context of laparoscopy surgical videos. NeRFs have recently gained attention thanks to their ability to generate photorealistic images from a 3-D static scene. Although our experimental results are promising, the proposed approach encounters substantial challenges.
  arXiv  Detail & Related papers  (2023-10-18T01:06:19Z)
• MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction [72.05649682685197]
  State-of-the-art neural implicit methods allow for high-quality reconstructions of simple scenes from many input views. This is caused primarily by the inherent ambiguity in the RGB reconstruction loss that does not provide enough constraints. Motivated by recent advances in the area of monocular geometry prediction, we explore the utility these cues provide for improving neural implicit surface reconstruction.
  arXiv  Detail & Related papers  (2022-06-01T17:58:15Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Adversarial Domain Feature Adaptation for Bronchoscopic Depth Estimation [111.89519571205778]
  In this work, we propose an alternative domain-adaptive approach to depth estimation. Our novel two-step structure first trains a depth estimation network with labeled synthetic images in a supervised manner. The results of our experiments show that the proposed method improves the network's performance on real images by a considerable margin.
  arXiv  Detail & Related papers  (2021-09-24T08:11:34Z)
• Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images [13.996932179049978]
  We propose SADepth, a new self-supervised depth estimation method based on Generative Adversarial Networks. It consists of an encoder-decoder generator and a discriminator to incorporate geometry constraints during training. Experiments on two public datasets show that SADepth outperforms recent state-of-the-art unsupervised methods by a large margin.
  arXiv  Detail & Related papers  (2021-07-09T19:40:20Z)
• SERV-CT: A disparity dataset from CT for validation of endoscopic 3D reconstruction [8.448866668577946]
  We present a stereo-endoscopic reconstruction validation dataset based on CT (SERV-CT) The SERV-CT dataset provides an easy to use stereoscopic validation for surgical applications with smooth reference disparities and depths with coverage over the majority of the endoscopic images.
  arXiv  Detail & Related papers  (2020-12-22T01:28:30Z)
• Tattoo tomography: Freehand 3D photoacoustic image reconstruction with an optical pattern [49.240017254888336]
  Photoacoustic tomography (PAT) is a novel imaging technique that can resolve both morphological and functional tissue properties. A current drawback is the limited field-of-view provided by the conventionally applied 2D probes. We present a novel approach to 3D reconstruction of PAT data that does not require an external tracking system.
  arXiv  Detail & Related papers  (2020-11-10T09:27:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.