3D endoscopic depth estimation using 3D surface-aware constraints

• URL: http://arxiv.org/abs/2203.02131v1
• Date: Fri, 4 Mar 2022 04:47:20 GMT
• Title: 3D endoscopic depth estimation using 3D surface-aware constraints
• Authors: Shang Zhao, Ce Wang, Qiyuan Wang, Yanzhe Liu, S Kevin Zhou
• Abstract summary: We show that depth estimation can be reformed from a 3D surface perspective. We propose a loss function for depth estimation that integrates the surface-aware constraints. Camera parameters are incorporated into the training pipeline to increase the control and transparency of the depth estimation.
• Score: 16.161276518580262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robotic-assisted surgery allows surgeons to conduct precise surgical operations with stereo vision and flexible motor control. However, the lack of 3D spatial perception limits situational awareness during procedures and hinders mastering surgical skills in the narrow abdominal space. Depth estimation, as a representative perception task, is typically defined as an image reconstruction problem. In this work, we show that depth estimation can be reformed from a 3D surface perspective. We propose a loss function for depth estimation that integrates the surface-aware constraints, leading to a faster and better convergence with the valid information from spatial information. In addition, camera parameters are incorporated into the training pipeline to increase the control and transparency of the depth estimation. We also integrate a specularity removal module to recover more buried image information. Quantitative experimental results on endoscopic datasets and user studies with medical professionals demonstrate the effectiveness of our method.

Related papers

• Depth to Anatomy: Learning Internal Organ Locations from Surface Depth Images [2.2821382245867414]
  We propose a learning-based framework that directly predicts the 3D locations and shapes of multiple internal organs from single 2D depth images of the body surface.<n>Our method accurately localizes a diverse set of anatomical structures, including bones and soft tissues, without requiring explicit surface reconstruction.
  arXiv  Detail & Related papers  (2026-01-26T08:33:11Z)
• EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model [2.8913847481700667]
  3D reconstruction of endoscopic surgery scenes plays a vital role in enhancing scene perception, enabling AR visualization, and supporting context-aware decision-making in image-guided surgery.<n>In intrinsic calibration is hindered by sterility constraints and the use of specialized endoscopes with continuous zoom and telescope rotation.<n>In this paper, we integrate intrinsic parameter estimation into a self-supervised monocular depth estimation framework by adapting the Depth Anything V2 (DA2) model for joint depth, pose, and intrinsics prediction.<n>Our method is validated on the SCARED and C3VD public datasets, demonstrating superior performance compared to recent state-of-the
  arXiv  Detail & Related papers  (2025-10-25T16:39:04Z)
• Accelerating 3D Photoacoustic Computed Tomography with End-to-End Physics-Aware Neural Operators [74.65171736966131]
  Photoacoustic computed tomography (PACT) combines optical contrast with ultrasonic resolution, achieving deep-tissue imaging beyond the optical diffusion limit.<n>Current implementations require dense transducer arrays and prolonged acquisition times, limiting clinical translation.<n>We introduce Pano, an end-to-end physics-aware model that directly learns the inverse acoustic mapping from sensor measurements to volumetric reconstructions.
  arXiv  Detail & Related papers  (2025-09-11T23:12:55Z)
• Spatial-Aware Self-Supervision for Medical 3D Imaging with Multi-Granularity Observable Tasks [4.097364225798782]
  We propose a method consisting of three sub-tasks to capture the spatially relevant semantics in medical 3D imaging.<n>Their design adheres to observable principles to ensure interpretability, and minimize the performance loss caused thereby as much as possible.
  arXiv  Detail & Related papers  (2025-09-07T08:16:37Z)
• Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting [64.64738535860351]
  We present a scalable pipeline that converts single-view images into comprehensive, scale- and appearance-realistic 3D representations.<n>Our method bridges the gap between the vast repository of imagery and the increasing demand for spatial scene understanding.<n>By automatically generating authentic, scale-aware 3D data from images, we significantly reduce data collection costs and open new avenues for advancing spatial intelligence.
  arXiv  Detail & Related papers  (2025-07-24T14:53:26Z)
• SurgicalGS: Dynamic 3D Gaussian Splatting for Accurate Robotic-Assisted Surgical Scene Reconstruction [18.074890506856114]
  We present SurgicalGS, a dynamic 3D Gaussian Splatting framework specifically designed for surgical scene reconstruction with improved geometric accuracy. Our approach first initialises a Gaussian point cloud using depth priors, employing binary motion masks to identify pixels with significant depth variations and fusing point clouds from depth maps across frames for initialisation. We use the Flexible Deformation Model to represent dynamic scene and introduce a normalised depth regularisation loss along with an unsupervised depth smoothness constraint to ensure more accurate geometric reconstruction.
  arXiv  Detail & Related papers  (2024-10-11T22:46:46Z)
• SLAM assisted 3D tracking system for laparoscopic surgery [22.36252790404779]
  This work proposes a real-time monocular 3D tracking algorithm for post-registration tasks. Experiments from in-vivo and ex-vivo tests demonstrate that the proposed 3D tracking system provides robust 3D tracking.
  arXiv  Detail & Related papers  (2024-09-18T04:00:54Z)
• Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy [2.906891207990726]
  We introduce a novel fine-tuning strategy for the Depth Anything Model. We integrate it with an intrinsic-based unsupervised monocular depth estimation framework. Our results show that our method achieves state-of-the-art performance while minimizing the number of trainable parameters.
  arXiv  Detail & Related papers  (2024-09-12T03:04:43Z)
• Brain3D: Generating 3D Objects from fMRI [76.41771117405973]
  We design a novel 3D object representation learning method, Brain3D, that takes as input the fMRI data of a subject. We show that our model captures the distinct functionalities of each region of human vision system. Preliminary evaluations indicate that Brain3D can successfully identify the disordered brain regions in simulated scenarios.
  arXiv  Detail & Related papers  (2024-05-24T06:06:11Z)
• High-fidelity Endoscopic Image Synthesis by Utilizing Depth-guided Neural Surfaces [18.948630080040576]
  We introduce a novel method for colon section reconstruction by leveraging NeuS applied to endoscopic images, supplemented by a single frame of depth map. Our approach demonstrates exceptional accuracy in completely rendering colon sections, even capturing unseen portions of the surface. This breakthrough opens avenues for achieving stable and consistently scaled reconstructions, promising enhanced quality in cancer screening procedures and treatment interventions.
  arXiv  Detail & Related papers  (2024-04-20T18:06:26Z)
• Creating a Digital Twin of Spinal Surgery: A Proof of Concept [68.37190859183663]
  Surgery digitalization is the process of creating a virtual replica of real-world surgery. We present a proof of concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal surgery. We employ five RGB-D cameras for dynamic 3D reconstruction of the surgeon, a high-end camera for 3D reconstruction of the anatomy, an infrared stereo camera for surgical instrument tracking, and a laser scanner for 3D reconstruction of the operating room and data fusion.
  arXiv  Detail & Related papers  (2024-03-25T13:09:40Z)
• Dense 3D Reconstruction Through Lidar: A Comparative Study on Ex-vivo Porcine Tissue [16.786601606755013]
  Researchers are actively investigating depth sensing and 3D reconstruction for vision-based surgical assistance. It remains difficult to achieve real-time, accurate, and robust 3D representations of the abdominal cavity for minimally invasive surgery. This work uses quantitative testing on fresh ex-vivo porcine tissue to thoroughly characterize the quality with which a 3D laser-based time-of-flight sensor can perform anatomical surface reconstruction.
  arXiv  Detail & Related papers  (2024-01-19T14:14:26Z)
• Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments [66.74633676595889]
  We present a multi-camera capture setup consisting of static and head-mounted cameras. Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre. Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments.
  arXiv  Detail & Related papers  (2023-05-05T13:42:19Z)
• Self-Supervised Depth Completion for Active Stereo [55.79929735390945]
  Active stereo systems are widely used in the robotics industry due to their low cost and high quality depth maps. These depth sensors suffer from stereo artefacts and do not provide dense depth estimates. We present the first self-supervised depth completion method for active stereo systems that predicts accurate dense depth maps.
  arXiv  Detail & Related papers  (2021-10-07T07:33:52Z)
• Adversarial Domain Feature Adaptation for Bronchoscopic Depth Estimation [111.89519571205778]
  In this work, we propose an alternative domain-adaptive approach to depth estimation. Our novel two-step structure first trains a depth estimation network with labeled synthetic images in a supervised manner. The results of our experiments show that the proposed method improves the network's performance on real images by a considerable margin.
  arXiv  Detail & Related papers  (2021-09-24T08:11:34Z)
• 3-Dimensional Deep Learning with Spatial Erasing for Unsupervised Anomaly Segmentation in Brain MRI [55.97060983868787]
  We investigate whether using increased spatial context by using MRI volumes combined with spatial erasing leads to improved unsupervised anomaly segmentation performance. We compare 2D variational autoencoder (VAE) to their 3D counterpart, propose 3D input erasing, and systemically study the impact of the data set size on the performance. Our best performing 3D VAE with input erasing leads to an average DICE score of 31.40% compared to 25.76% for the 2D VAE.
  arXiv  Detail & Related papers  (2021-09-14T09:17:27Z)
• Self-Supervised Generative Adversarial Network for Depth Estimation in Laparoscopic Images [13.996932179049978]
  We propose SADepth, a new self-supervised depth estimation method based on Generative Adversarial Networks. It consists of an encoder-decoder generator and a discriminator to incorporate geometry constraints during training. Experiments on two public datasets show that SADepth outperforms recent state-of-the-art unsupervised methods by a large margin.
  arXiv  Detail & Related papers  (2021-07-09T19:40:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.