Markerless Augmented Reality Registration for Surgical Guidance: A Multi-Anatomy Clinical Accuracy Study

• URL: http://arxiv.org/abs/2511.02086v1
• Date: Mon, 03 Nov 2025 21:58:33 GMT
• Title: Markerless Augmented Reality Registration for Surgical Guidance: A Multi-Anatomy Clinical Accuracy Study
• Authors: Yue Yang, Fabian Necker, Christoph Leuze, Michelle Chen, Andrey Finegersh, Jake Lee, Vasu Divi, Bruce Daniel, Brian Hargreaves, Jie Ying Wu, Fred M Baik,
• Abstract summary: We develop and clinically evaluate a depth-only, markerless augmented reality (AR) registration pipeline on a head-mounted display.<n>We assess accuracy across small or low-curvature anatomies in real-life operative settings.
• Score: 6.154533735564864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Purpose: In this paper, we develop and clinically evaluate a depth-only, markerless augmented reality (AR) registration pipeline on a head-mounted display, and assess accuracy across small or low-curvature anatomies in real-life operative settings. Methods: On HoloLens 2, we align Articulated HAnd Tracking (AHAT) depth to Computed Tomography (CT)-derived skin meshes via (i) depth-bias correction, (ii) brief human-in-the-loop initialization, (iii) global and local registration. We validated the surface-tracing error metric by comparing "skin-to-bone" relative distances to CT ground truth on leg and foot models, using an AR-tracked tool. We then performed seven intraoperative target trials (feet x2, ear x3, leg x2) during the initial stage of fibula free-flap harvest and mandibular reconstruction surgery, and collected 500+ data per trial. Results: Preclinical validation showed tight agreement between AR-traced and CT distances (leg: median |Delta d| 0.78 mm, RMSE 0.97 mm; feet: 0.80 mm, 1.20 mm). Clinically, per-point error had a median of 3.9 mm. Median errors by anatomy were 3.2 mm (feet), 4.3 mm (ear), and 5.3 mm (lower leg), with 5 mm coverage 92-95%, 84-90%, and 72-86%, respectively. Feet vs. lower leg differed significantly (Delta median ~1.1 mm; p < 0.001). Conclusion: A depth-only, markerless AR pipeline on HMDs achieved ~3-4 mm median error across feet, ear, and lower leg in live surgical settings without fiducials, approaching typical clinical error thresholds for moderate-risk tasks. Human-guided initialization plus global-to-local registration enabled accurate alignment on small or low-curvature targets, improving the clinical readiness of markerless AR guidance.

Related papers

• Automated glenoid bone loss measurement and segmentation in CT scans for pre-operative planning in shoulder instability [4.618498494409548]
  Reliable measurement of glenoid bone loss is essential for operative planning in shoulder instability.<n>We developed and validated a fully automated deep learning pipeline for measuring glenoid bone loss on three-dimensional computed tomography (CT) scans.
  arXiv  Detail & Related papers  (2025-11-18T03:12:22Z)
• Surf2CT: Cascaded 3D Flow Matching Models for Torso 3D CT Synthesis from Skin Surface [13.161605581865357]
  Surf2CT is a framework that synthesizes full 3D computed tomography (CT) volumes of the human torso from external surface scans and demographic data.<n>We trained our model on a combined dataset of 3,198 torso CT scans sourced from Massachusetts General Hospital (MGH) and the AutoPET challenge.
  arXiv  Detail & Related papers  (2025-05-28T16:01:36Z)
• A weakly-supervised deep learning model for fast localisation and delineation of the skeleton, internal organs, and spinal canal on Whole-Body Diffusion-Weighted MRI (WB-DWI) [0.0]
  We developed an automated deep-learning pipeline based on a 3D patch-based Residual U-Net architecture.<n>We employed a multi-centre WB-DWI dataset comprising 532 scans from patients with Advanced Prostate Cancer (APC) or Multiple Myeloma (MM)<n>Relative median ADC differences between automated and manual full-body delineations were below 10%.<n>The model was 12x faster than the atlas-based registration algorithm.
  arXiv  Detail & Related papers  (2025-03-26T17:03:46Z)
• Registration of Longitudinal Spine CTs for Monitoring Lesion Growth [32.449176315776114]
  We present a novel method to automatically align longitudinal spine CTs. Our method follows a two-step pipeline where vertebrae are first automatically localized, labeled and 3D surfaces are generated. We tested our approach on 37 vertebrae, from 5 patients, with baseline CTs and 3, 6, and 12 months follow-ups leading to 111 registrations.
  arXiv  Detail & Related papers  (2024-02-14T17:43:50Z)
• Weakly-Supervised Detection of Bone Lesions in CT [48.34559062736031]
  The skeletal region is one of the common sites of metastatic spread of cancer in the breast and prostate. We developed a pipeline to detect bone lesions in CT volumes via a proxy segmentation task. Our method detected bone lesions in CT with a precision of 96.7% and recall of 47.3% despite the use of incomplete and partial training data.
  arXiv  Detail & Related papers  (2024-01-31T21:05:34Z)
• Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery [52.63271687382495]
  We present an approach that automatically solves the registration problem for lumbar spinal fusion surgery in a radiation-free manner. A deep neural network was trained to segment the lumbar spine and simultaneously predict its orientation, yielding an initial pose for preoperative models. An intuitive surgical guidance is provided thanks to the integration into an augmented reality based navigation system.
  arXiv  Detail & Related papers  (2023-08-05T16:26:41Z)
• Accurate Fine-Grained Segmentation of Human Anatomy in Radiographs via Volumetric Pseudo-Labeling [66.75096111651062]
  We created a large-scale dataset of 10,021 thoracic CTs with 157 labels. We applied an ensemble of 3D anatomy segmentation models to extract anatomical pseudo-labels. Our resulting segmentation models demonstrated remarkable performance on CXR.
  arXiv  Detail & Related papers  (2023-06-06T18:01:08Z)
• Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments [64.59698930334012]
  We present a multi-camera capture setup consisting of static and head-mounted cameras.<n>Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre.<n>Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments.
  arXiv  Detail & Related papers  (2023-05-05T13:42:19Z)
• Corneal endothelium assessment in specular microscopy images with Fuchs' dystrophy via deep regression of signed distance maps [48.498376125522114]
  This paper proposes a UNet-based segmentation approach that requires minimal post-processing. It achieves reliable CE morphometric assessment and guttae identification across all degrees of Fuchs' dystrophy.
  arXiv  Detail & Related papers  (2022-10-13T15:34:20Z)
• Building Brains: Subvolume Recombination for Data Augmentation in Large Vessel Occlusion Detection [56.67577446132946]
  A large training data set is required for a standard deep learning-based model to learn this strategy from data. We propose an augmentation method that generates artificial training samples by recombining vessel tree segmentations of the hemispheres from different patients. In line with the augmentation scheme, we use a 3D-DenseNet fed with task-specific input, fostering a side-by-side comparison between the hemispheres.
  arXiv  Detail & Related papers  (2022-05-05T10:31:57Z)
• Systematic Clinical Evaluation of A Deep Learning Method for Medical Image Segmentation: Radiosurgery Application [48.89674088331313]
  We systematically evaluate a Deep Learning (DL) method in a 3D medical image segmentation task. Our method is integrated into the radiosurgery treatment process and directly impacts the clinical workflow.
  arXiv  Detail & Related papers  (2021-08-21T16:15:40Z)
• Controlling False Positive/Negative Rates for Deep-Learning-Based Prostate Cancer Detection on Multiparametric MR images [58.85481248101611]
  We propose a novel PCa detection network that incorporates a lesion-level cost-sensitive loss and an additional slice-level loss based on a lesion-to-slice mapping function. Our experiments based on 290 clinical patients concludes that 1) The lesion-level FNR was effectively reduced from 0.19 to 0.10 and the lesion-level FPR was reduced from 1.03 to 0.66 by changing the lesion-level cost.
  arXiv  Detail & Related papers  (2021-06-04T09:51:27Z)
• Segmentation-free Estimation of Aortic Diameters from MRI Using Deep Learning [2.231365407061881]
  We propose a supervised deep learning method for the direct estimation of aortic diameters. Our approach makes use of a 3D+2D convolutional neural network (CNN) that takes as input a 3D scan and outputs the aortic diameter at a given location. Overall, the 3D+2D CNN achieved a mean absolute error between 2.2-2.4 mm depending on the considered aortic location.
  arXiv  Detail & Related papers  (2020-09-09T18:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.