Online 3D reconstruction and dense tracking in endoscopic videos

• URL: http://arxiv.org/abs/2409.06037v1
• Date: Mon, 9 Sep 2024 19:58:42 GMT
• Title: Online 3D reconstruction and dense tracking in endoscopic videos
• Authors: Michel Hayoz, Christopher Hahne, Thomas Kurmann, Max Allan, Guido Beldi, Daniel Candinas, ablo Márquez-Neila, Raphael Sznitman,
• Abstract summary: 3D scene reconstruction from stereo endoscopic video data is crucial for advancing surgical interventions. We present an online framework for online, dense 3D scene reconstruction and tracking, aimed at enhancing surgical scene understanding and assisting interventions.
• Score: 5.667206318889122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D scene reconstruction from stereo endoscopic video data is crucial for advancing surgical interventions. In this work, we present an online framework for online, dense 3D scene reconstruction and tracking, aimed at enhancing surgical scene understanding and assisting interventions. Our method dynamically extends a canonical scene representation using Gaussian splatting, while modeling tissue deformations through a sparse set of control points. We introduce an efficient online fitting algorithm that optimizes the scene parameters, enabling consistent tracking and accurate reconstruction. Through experiments on the StereoMIS dataset, we demonstrate the effectiveness of our approach, outperforming state-of-the-art tracking methods and achieving comparable performance to offline reconstruction techniques. Our work enables various downstream applications thus contributing to advancing the capabilities of surgical assistance systems.

Related papers

• SALVE: A 3D Reconstruction Benchmark of Wounds from Consumer-grade Videos [20.69257610322339]
  This paper presents a study on 3D wound reconstruction from consumer-grade videos. We introduce the SALVE dataset, comprising video recordings of realistic wound phantoms captured with different cameras. We assess the accuracy and precision of state-of-the-art methods for 3D reconstruction, ranging from traditional photogrammetry pipelines to advanced neural rendering approaches.
  arXiv  Detail & Related papers  (2024-07-29T02:34:51Z)
• FLex: Joint Pose and Dynamic Radiance Fields Optimization for Stereo Endoscopic Videos [79.50191812646125]
  Reconstruction of endoscopic scenes is an important asset for various medical applications, from post-surgery analysis to educational training. We adress the challenging setup of a moving endoscope within a highly dynamic environment of deforming tissue. We propose an implicit scene separation into multiple overlapping 4D neural radiance fields (NeRFs) and a progressive optimization scheme jointly optimizing for reconstruction and camera poses from scratch. This improves the ease-of-use and allows to scale reconstruction capabilities in time to process surgical videos of 5,000 frames and more; an improvement of more than ten times compared to the state of the art while being agnostic to external tracking information
  arXiv  Detail & Related papers  (2024-03-18T19:13:02Z)
• Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction [89.53963284958037]
  We propose a novel motion-aware enhancement framework for dynamic scene reconstruction. Specifically, we first establish a correspondence between 3D Gaussian movements and pixel-level flow. For the prevalent deformation-based paradigm that presents a harder optimization problem, a transient-aware deformation auxiliary module is proposed.
  arXiv  Detail & Related papers  (2024-03-18T03:46:26Z)
• Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Classification, Reconstruction, and Tracking [21.133420628173067]
  We present a novel surgical perception framework, Semantic-SuPer. It integrates geometric and semantic information to facilitate data association, 3D reconstruction, and tracking of endoscopic scenes.
  arXiv  Detail & Related papers  (2022-10-29T19:33:21Z)
• Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery [18.150476919815382]
  Reconstruction of the soft tissues in robotic surgery from endoscopic stereo videos is important for many applications. Previous works on this task mainly rely on SLAM-based approaches, which struggle to handle complex surgical scenes. Inspired by recent progress in neural rendering, we present a novel framework for deformable tissue reconstruction.
  arXiv  Detail & Related papers  (2022-06-30T13:06:27Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks [77.56526918859345]
  We present a novel framework that brings the 3D motion task from controlled environments to in-the-wild scenarios. It is capable of body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure.
  arXiv  Detail & Related papers  (2021-12-19T07:52:05Z)
• Stereo Dense Scene Reconstruction and Accurate Laparoscope Localization for Learning-Based Navigation in Robot-Assisted Surgery [37.14020061063255]
  The computation of anatomical information and laparoscope position is a fundamental block of robot-assisted surgical navigation in Minimally Invasive Surgery (MIS) We propose a learning-driven framework, in which an image-guided laparoscopic localization with 3D reconstructions of complex anatomical structures is hereby achieved.
  arXiv  Detail & Related papers  (2021-10-08T06:12:18Z)
• Adversarial Domain Feature Adaptation for Bronchoscopic Depth Estimation [111.89519571205778]
  In this work, we propose an alternative domain-adaptive approach to depth estimation. Our novel two-step structure first trains a depth estimation network with labeled synthetic images in a supervised manner. The results of our experiments show that the proposed method improves the network's performance on real images by a considerable margin.
  arXiv  Detail & Related papers  (2021-09-24T08:11:34Z)
• E-DSSR: Efficient Dynamic Surgical Scene Reconstruction with Transformer-based Stereoscopic Depth Perception [15.927060244702686]
  We present an efficient reconstruction pipeline for highly dynamic surgical scenes that runs at 28 fps. Specifically, we design a transformer-based stereoscopic depth perception for efficient depth estimation. We evaluate the proposed pipeline on two datasets, the public Hamlyn Centre Endoscopic Video dataset and our in-house DaVinci robotic surgery dataset.
  arXiv  Detail & Related papers  (2021-07-01T05:57:41Z)
• SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion [86.77318031029404]
  We propose a framework that performs scene reconstruction and semantic scene completion jointly in an incremental and real-time manner. Our framework relies on a novel neural architecture designed to process occupancy maps and leverages voxel states to accurately and efficiently fuse semantic completion with the 3D global model.
  arXiv  Detail & Related papers  (2020-10-26T15:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.