Tracking-Aware Deformation Field Estimation for Non-rigid 3D Reconstruction in Robotic Surgeries

• URL: http://arxiv.org/abs/2503.02558v1
• Date: Tue, 04 Mar 2025 12:33:17 GMT
• Title: Tracking-Aware Deformation Field Estimation for Non-rigid 3D Reconstruction in Robotic Surgeries
• Authors: Zeqing Wang, Han Fang, Yihong Xu, Yutong Ban,
• Abstract summary: It is still safety critical to be aware of even the least tissue deformation during instrument-tissue interactions.<n>We propose Tracking-Aware Deformation Field (TADF), a novel framework which reconstructs the 3D mesh along with the 3D tissue deformation simultaneously.
• Score: 15.231519282512567
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Minimally invasive procedures have been advanced rapidly by the robotic laparoscopic surgery. The latter greatly assists surgeons in sophisticated and precise operations with reduced invasiveness. Nevertheless, it is still safety critical to be aware of even the least tissue deformation during instrument-tissue interactions, especially in 3D space. To address this, recent works rely on NeRF to render 2D videos from different perspectives and eliminate occlusions. However, most of the methods fail to predict the accurate 3D shapes and associated deformation estimates robustly. Differently, we propose Tracking-Aware Deformation Field (TADF), a novel framework which reconstructs the 3D mesh along with the 3D tissue deformation simultaneously. It first tracks the key points of soft tissue by a foundation vision model, providing an accurate 2D deformation field. Then, the 2D deformation field is smoothly incorporated with a neural implicit reconstruction network to obtain tissue deformation in the 3D space. Finally, we experimentally demonstrate that the proposed method provides more accurate deformation estimation compared with other 3D neural reconstruction methods in two public datasets.

Related papers

• Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models [65.90387371072413]
  We introduce Difix3D+, a novel pipeline designed to enhance 3D reconstruction and novel-view synthesis.<n>At the core of our approach is Difix, a single-step image diffusion model trained to enhance and remove artifacts in rendered novel views.
  arXiv  Detail & Related papers  (2025-03-03T17:58:33Z)
• MedTet: An Online Motion Model for 4D Heart Reconstruction [59.74234226055964]
  We present a novel approach to reconstruction of 3D cardiac motion from sparse intraoperative data.<n>Existing methods can accurately reconstruct 3D organ geometries from full 3D volumetric imaging.<n>We propose a versatile framework for reconstructing 3D motion from such partial data.
  arXiv  Detail & Related papers  (2024-12-03T17:18:33Z)
• DuoLift-GAN:Reconstructing CT from Single-view and Biplanar X-Rays with Generative Adversarial Networks [1.3812010983144802]
  We introduce DuoLift Generative Adversarial Networks (DuoLift-GAN), a novel architecture with dual branches that independently elevate 2D images and their features into 3D representations. These 3D outputs are merged into a unified 3D feature map and decoded into a complete 3D chest volume, enabling richer 3D information capture.
  arXiv  Detail & Related papers  (2024-11-12T17:11:18Z)
• Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering [5.617649111108429]
  We present DiffPose, a self-supervised approach that leverages patient-specific simulation and differentiable physics-based rendering to achieve accurate 2D/3D registration without relying on manually labeled data. DiffPose achieves sub-millimeter accuracy across surgical datasets at intraoperative speeds, improving upon existing unsupervised methods by an order of magnitude and even outperforming supervised baselines.
  arXiv  Detail & Related papers  (2023-12-11T13:05:54Z)
• Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling [83.76377808476039]
  We propose a new modeling method for human pose deformations and design an accompanying diffusion-based motion prior. Inspired by the field of non-rigid structure-from-motion, we divide the task of reconstructing 3D human skeletons in motion into the estimation of a 3D reference skeleton. A mixed spatial-temporal NRSfMformer is used to simultaneously estimate the 3D reference skeleton and the skeleton deformation of each frame from 2D observations sequence.
  arXiv  Detail & Related papers  (2023-08-18T16:41:57Z)
• CNN-based real-time 2D-3D deformable registration from a single X-ray projection [2.1198879079315573]
  This paper presents a method for real-time 2D-3D non-rigid registration using a single fluoroscopic image. A dataset composed of displacement fields and 2D projections of the anatomy is generated from a preoperative scan. A neural network is trained to recover the unknown 3D displacement field from a single projection image.
  arXiv  Detail & Related papers  (2022-12-15T09:57:19Z)
• Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian [58.704089101826774]
  We present a 3D-aware image deformation method with minimal restrictions on shape category and deformation type. We take a supervised learning-based approach to predict the shape Laplacian of the underlying volume of a 3D reconstruction represented as a point cloud. In the experiments, we present our results of deforming 2D character and clothed human images.
  arXiv  Detail & Related papers  (2022-03-29T04:57:18Z)
• 3D Reconstruction of Curvilinear Structures with Stereo Matching DeepConvolutional Neural Networks [52.710012864395246]
  We propose a fully automated pipeline for both detection and matching of curvilinear structures in stereo pairs. We mainly focus on 3D reconstruction of dislocations from stereo pairs of TEM images.
  arXiv  Detail & Related papers  (2021-10-14T23:05:47Z)
• Revisiting 3D Context Modeling with Supervised Pre-training for Universal Lesion Detection in CT Slices [48.85784310158493]
  We propose a Modified Pseudo-3D Feature Pyramid Network (MP3D FPN) to efficiently extract 3D context enhanced 2D features for universal lesion detection in CT slices. With the novel pre-training method, the proposed MP3D FPN achieves state-of-the-art detection performance on the DeepLesion dataset. The proposed 3D pre-trained weights can potentially be used to boost the performance of other 3D medical image analysis tasks.
  arXiv  Detail & Related papers  (2020-12-16T07:11:16Z)
• Dynamic Reconstruction of Deformable Soft-tissue with Stereo Scope in Minimal Invasive Surgery [24.411005883017832]
  In minimal invasive surgery, it is important to rebuild and visualize the latest deformed shape of soft-tissue surfaces. This paper proposes an innovative Simultaneous localization and Mapping (SLAM) algorithm for deformable dense reconstruction of surfaces. In-vivo experiments with publicly available datasets demonstrate that the 3D models can be incrementally built for different soft-tissues.
  arXiv  Detail & Related papers  (2020-03-22T16:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.