EasyREG: Easy Depth-Based Markerless Registration and Tracking using Augmented Reality Device for Surgical Guidance

• URL: http://arxiv.org/abs/2504.09498v1
• Date: Sun, 13 Apr 2025 09:48:33 GMT
• Title: EasyREG: Easy Depth-Based Markerless Registration and Tracking using Augmented Reality Device for Surgical Guidance
• Authors: Yue Yang, Christoph Leuze, Brian Hargreaves, Bruce Daniel, Fred Baik,
• Abstract summary: We present a markerless framework that relies only on the depth sensor of AR devices.<n>The registration module integrates depth sensor error correction, a human-in-the-loop region filtering technique, and a robust global alignment.<n>The tracking module employs a fast and robust registration algorithm that uses the initial pose from the registration module to estimate the target pose in real-time.
• Score: 10.156036566483888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The use of Augmented Reality (AR) devices for surgical guidance has gained increasing traction in the medical field. Traditional registration methods often rely on external fiducial markers to achieve high accuracy and real-time performance. However, these markers introduce cumbersome calibration procedures and can be challenging to deploy in clinical settings. While commercial solutions have attempted real-time markerless tracking using the native RGB cameras of AR devices, their accuracy remains questionable for medical guidance, primarily due to occlusions and significant outliers between the live sensor data and the preoperative target anatomy point cloud derived from MRI or CT scans. In this work, we present a markerless framework that relies only on the depth sensor of AR devices and consists of two modules: a registration module for high-precision, outlier-robust target anatomy localization, and a tracking module for real-time pose estimation. The registration module integrates depth sensor error correction, a human-in-the-loop region filtering technique, and a robust global alignment with curvature-aware feature sampling, followed by local ICP refinement, for markerless alignment of preoperative models with patient anatomy. The tracking module employs a fast and robust registration algorithm that uses the initial pose from the registration module to estimate the target pose in real-time. We comprehensively evaluated the performance of both modules through simulation and real-world measurements. The results indicate that our markerless system achieves superior performance for registration and comparable performance for tracking to industrial solutions. The two-module design makes our system a one-stop solution for surgical procedures where the target anatomy moves or stays static during surgery.

Related papers

• Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection [50.388465935739376]
  Liver registration by overlaying preoperative 3D models onto intraoperative 2D frames can assist surgeons in perceiving the spatial anatomy of the liver clearly for a higher surgical success rate. Existing registration methods rely heavily on anatomical landmark-based, which encounter two major limitations. We propose a landmark-free preoperative-to-intraoperative registration framework utilizing effective self-supervised learning.
  arXiv  Detail & Related papers  (2025-04-21T14:55:57Z)
• EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidance [79.66329903007869]
  We present EchoWorld, a motion-aware world modeling framework for probe guidance. It encodes anatomical knowledge and motion-induced visual dynamics. It is trained on more than one million ultrasound images from over 200 routine scans.
  arXiv  Detail & Related papers  (2025-04-17T16:19:05Z)
• Real-time guidewire tracking and segmentation in intraoperative x-ray [52.51797358201872]
  We propose a two-stage deep learning framework for real-time guidewire segmentation and tracking. In the first stage, a Yolov5 detector is trained, using the original X-ray images as well as synthetic ones, to output the bounding boxes of possible target guidewires. In the second stage, a novel and efficient network is proposed to segment the guidewire in each detected bounding box.
  arXiv  Detail & Related papers  (2024-04-12T20:39:19Z)
• Data-Driven Goal Recognition in Transhumeral Prostheses Using Process Mining Techniques [7.95507524742396]
  Active prostheses utilize real-valued, continuous sensor data to recognize patient target poses, or goals, and proactively move the artificial limb. Previous studies have examined how well the data collected in stationary poses, without considering the time steps, can help discriminate the goals. Our approach involves transforming the data into discrete events and training an existing process mining-based goal recognition system.
  arXiv  Detail & Related papers  (2023-09-15T02:03:59Z)
• DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion [10.439802168557513]
  Motion capture from a limited number of body-worn sensors has important applications in health, human performance, and entertainment. Recent work has focused on accurately reconstructing whole-body motion from a specific sensor configuration using six IMUs. We propose a single diffusion model, DiffusionPoser, which reconstructs human motion in real-time from an arbitrary combination of sensors.
  arXiv  Detail & Related papers  (2023-08-31T12:36:50Z)
• Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery [52.63271687382495]
  We present an approach that automatically solves the registration problem for lumbar spinal fusion surgery in a radiation-free manner. A deep neural network was trained to segment the lumbar spine and simultaneously predict its orientation, yielding an initial pose for preoperative models. An intuitive surgical guidance is provided thanks to the integration into an augmented reality based navigation system.
  arXiv  Detail & Related papers  (2023-08-05T16:26:41Z)
• Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments [64.59698930334012]
  We present a multi-camera capture setup consisting of static and head-mounted cameras.<n>Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre.<n>Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments.
  arXiv  Detail & Related papers  (2023-05-05T13:42:19Z)
• Robust Landmark-based Stent Tracking in X-ray Fluoroscopy [10.917460255497227]
  We propose an end-to-end deep learning framework for single stent tracking. It consists of three hierarchical modules: U-Net based landmark detection, ResNet based stent proposal and feature extraction. Experiments show that our method performs significantly better in detection compared with the state-of-the-art point-based tracking models.
  arXiv  Detail & Related papers  (2022-07-20T14:20:03Z)
• Real-time landmark detection for precise endoscopic submucosal dissection via shape-aware relation network [51.44506007844284]
  We propose a shape-aware relation network for accurate and real-time landmark detection in endoscopic submucosal dissection surgery. We first devise an algorithm to automatically generate relation keypoint heatmaps, which intuitively represent the prior knowledge of spatial relations among landmarks. We then develop two complementary regularization schemes to progressively incorporate the prior knowledge into the training process.
  arXiv  Detail & Related papers  (2021-11-08T07:57:30Z)
• Occlusion-robust Visual Markerless Bone Tracking for Computer-Assisted Orthopaedic Surgery [41.681134859412246]
  We propose a RGB-D sensing-based markerless tracking method that is robust against occlusion. By using a high-quality commercial RGB-D camera, our proposed visual tracking method achieves an accuracy of 1-2 degress and 2-4 mm on a model knee.
  arXiv  Detail & Related papers  (2021-08-24T09:49:08Z)
• Human Leg Motion Tracking by Fusing IMUs and RGB Camera Data Using Extended Kalman Filter [4.189643331553922]
  IMU-based systems, as well as Marker-based motion tracking systems, are the most popular methods to track movement due to their low cost of implementation and lightweight. This paper proposes a quaternion-based Extended Kalman filter approach to recover the human leg segments motions with a set of IMU sensors data fused with camera-marker system data.
  arXiv  Detail & Related papers  (2020-11-01T17:54:53Z)
• Searching for Efficient Architecture for Instrument Segmentation in Robotic Surgery [58.63306322525082]
  Most applications rely on accurate real-time segmentation of high-resolution surgical images. We design a light-weight and highly-efficient deep residual architecture which is tuned to perform real-time inference of high-resolution images.
  arXiv  Detail & Related papers  (2020-07-08T21:38:29Z)
• Deep Soft Procrustes for Markerless Volumetric Sensor Alignment [81.13055566952221]
  In this work, we improve markerless data-driven correspondence estimation to achieve more robust multi-sensor spatial alignment. We incorporate geometric constraints in an end-to-end manner into a typical segmentation based model and bridge the intermediate dense classification task with the targeted pose estimation one. Our model is experimentally shown to achieve similar results with marker-based methods and outperform the markerless ones, while also being robust to the pose variations of the calibration structure.
  arXiv  Detail & Related papers  (2020-03-23T10:51:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.