Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization

• URL: http://arxiv.org/abs/2603.11404v2
• Date: Fri, 13 Mar 2026 01:46:22 GMT
• Title: Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization
• Abstract summary: We propose a versatile tracking pipeline that jointly estimates surgical instrument pose and joint configurations.<n>The proposed framework further generalizes to joint angle-free and bi-manual tracking settings, making it suitable for both vision feedback control and online surgery video calibration.
• Score: 21.436806983625885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate and efficient tracking of surgical instruments is fundamental for Robot-Assisted Minimally Invasive Surgery. Although vision-based robot pose estimation has enabled markerless calibration without tedious physical setups, reliable tool tracking for surgical robots still remains challenging due to partial visibility and specialized articulation design of surgical instruments. Previous works in the field are usually prone to unreliable feature detections under degraded visual quality and data scarcity, whereas rendering-based methods often struggle with computational costs and suboptimal convergence. In this work, we incorporate CMA-ES, an evolutionary optimization strategy, into a versatile tracking pipeline that jointly estimates surgical instrument pose and joint configurations. Using batch rendering to efficiently evaluate multiple pose candidates in parallel, the method significantly reduces inference time and improves convergence robustness. The proposed framework further generalizes to joint angle-free and bi-manual tracking settings, making it suitable for both vision feedback control and online surgery video calibration. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method significantly outperforms prior approaches in both accuracy and runtime.

Related papers

• Invertible Neural Network Adapter for One-Step Flow Matching in Robot Manipulation [48.77928097573051]
  This paper presents an invertible neural network adapter for general robotic manipulation.<n>Built upon a flow-matching formulation, the proposed adapter constrains the action generation trajectory within an invertible latent space.<n>Experiments are conducted across a diverse set of simulation benchmarks and real-world robotic platforms to evaluate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2026-06-17T15:35:27Z)
• A Rapid Instrument Exchange System for Humanoid Robots in Minimally Invasive Surgery [9.137620295313711]
  This paper proposes an immersive teleoperated rapid instrument exchange system.<n>The system utilizes a low-latency mechanism based on single-axis compliant docking and environmental constraint release.
  arXiv  Detail & Related papers  (2026-04-03T04:05:00Z)
• Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality [40.023882327049684]
  Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments.<n>Most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments.<n>This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation.
  arXiv  Detail & Related papers  (2026-03-09T05:37:49Z)
• On-the-fly hand-eye calibration for the da Vinci surgical robot [35.18343759591773]
  We propose a calibration framework to produce accurate tool localization results through computing the hand-eye transformation matrix on-the-fly.<n>Results show a significant reduction in tool localization errors under the proposed calibration framework.
  arXiv  Detail & Related papers  (2026-01-21T10:58:28Z)
• Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection [21.460727996614704]
  MIS robots have long kinematic chains and partial visibility of their degrees of freedom in the camera.<n>We propose a novel framework that unifies the detection of geometric primitives through a shared encoding.<n>This architecture detects both keypoints and edges in a single inference and is trained on large-scale synthetic data with projective labeling.
  arXiv  Detail & Related papers  (2025-10-03T22:03:28Z)
• Efficient 3D Scene Reconstruction and Simulation from Sparse Endoscopic Views [0.7614628596146601]
  We propose a Gaussian Splatting-based framework to reconstruct interactive surgical scenes from endoscopic data.<n>A key challenge in this data-driven simulation paradigm is the restricted movement of endoscopic cameras.<n>We show that our method can efficiently reconstruct and simulate surgical scenes from sparse endoscopic views.
  arXiv  Detail & Related papers  (2025-09-21T10:40:36Z)
• Tracking Everything in Robotic-Assisted Surgery [39.62251870446397]
  We present an annotated surgical tracking dataset for benchmarking tracking methods for surgical scenarios.<n>We evaluate state-of-the-art (SOTA) TAP-based algorithms on this dataset and reveal their limitations in challenging surgical scenarios.<n>We propose a new tracking method, namely SurgMotion, to solve the challenges and further improve the tracking performance.
  arXiv  Detail & Related papers  (2024-09-29T23:06:57Z)
• Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane [58.871015937204255]
  We introduce Fast Orthogonal Plane (plane) for the reconstruction of deformable tissues. We conceptualize surgical procedures as 4D volumes, and break them down into static and dynamic fields comprised of neural planes. This factorization iscretizes four-dimensional space, leading to a decreased memory usage and faster optimization.
  arXiv  Detail & Related papers  (2023-12-23T13:27:50Z)
• Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries [29.201385352740555]
  We propose a novel visual-kinematics graph learning framework to accurately segment the instrument tip given various surgical procedures. Specifically, a graph learning framework is proposed to encode relational features of instrument parts from both image and kinematics. A cross-modal contrastive loss is designed to incorporate robust geometric prior from kinematics to image for tip segmentation.
  arXiv  Detail & Related papers  (2023-09-02T14:52:58Z)
• Neural LerPlane Representations for Fast 4D Reconstruction of Deformable Tissues [52.886545681833596]
  LerPlane is a novel method for fast and accurate reconstruction of surgical scenes under a single-viewpoint setting. LerPlane treats surgical procedures as 4D volumes and factorizes them into explicit 2D planes of static and dynamic fields. LerPlane shares static fields, significantly reducing the workload of dynamic tissue modeling.
  arXiv  Detail & Related papers  (2023-05-31T14:38:35Z)
• Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments [64.59698930334012]
  We present a multi-camera capture setup consisting of static and head-mounted cameras.<n>Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre.<n>Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments.
  arXiv  Detail & Related papers  (2023-05-05T13:42:19Z)
• EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration [49.90228618894857]
  We introduce a new approach to hand-eye calibration called EasyHeC, which is markerless, white-box, and delivers superior accuracy and robustness. We propose to use two key technologies: differentiable rendering-based camera pose optimization and consistency-based joint space exploration. Our evaluation demonstrates superior performance in synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-02T03:49:54Z)
• Real-time landmark detection for precise endoscopic submucosal dissection via shape-aware relation network [51.44506007844284]
  We propose a shape-aware relation network for accurate and real-time landmark detection in endoscopic submucosal dissection surgery. We first devise an algorithm to automatically generate relation keypoint heatmaps, which intuitively represent the prior knowledge of spatial relations among landmarks. We then develop two complementary regularization schemes to progressively incorporate the prior knowledge into the training process.
  arXiv  Detail & Related papers  (2021-11-08T07:57:30Z)
• Searching for Efficient Architecture for Instrument Segmentation in Robotic Surgery [58.63306322525082]
  Most applications rely on accurate real-time segmentation of high-resolution surgical images. We design a light-weight and highly-efficient deep residual architecture which is tuned to perform real-time inference of high-resolution images.
  arXiv  Detail & Related papers  (2020-07-08T21:38:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.