Streamlining stereo differentiable rendering for marker-free real-time tracking of surgical robots

• URL: http://arxiv.org/abs/2607.12604v1
• Date: Tue, 14 Jul 2026 10:28:28 GMT
• Title: Streamlining stereo differentiable rendering for marker-free real-time tracking of surgical robots
• Abstract summary: We evaluate stereo differentiable rendering for marker-free, real-time robot pose tracking.<n>We achieve real-time 1080p tracking at 30 fps (up from 14 fps for vanilla roboreg), matching the camera frame rate.
• Score: 7.312849670879312
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Purpose: Marker-based tracking of surgical robots is occlusion-prone in cluttered operating rooms. We evaluate stereo differentiable rendering for marker-free, real-time robot pose tracking, potentially improving safety, reducing setup time, and enabling multi-robot interaction. Methods: We extend the markerless pose estimation framework roboreg to online dynamic tracking via (i) sequential optimisation that propagates pose estimates across frames with motion-adaptive hyperparameter tuning, and (ii) CUDA stream parallelisation of segmentation and optimisation, combined with CUDA-graph accelerated segmentation. We evaluate on 38 unobstructed and 5 occluded displacement sequences with static start/end ground-truth calibrations and dynamic marker-based reference tracking. Results: We achieve real-time 1080p tracking at 30 fps (up from 14 fps for vanilla roboreg), matching the camera frame rate. Accuracy reaches 1.7 cm / 0.6 deg against static ground truth and 1.2 cm mean 3D error over 27,460 frames against the marker-based reference (1.53 cm over 1,242 occluded frames). Our method outperforms FoundationPose by 11% in dynamic estimation (63% under occlusion) and 250% in static estimation, with 6x faster inference. Conclusions: Stereo differentiable rendering enables real-time, high-resolution marker-free surgical robot tracking, on par with marker-based approaches and surpassing foundation-model baselines.

Related papers

• Event-Based Motion Estimation via Oriented Distance Fields [55.23416502666468]
  Event-based motion estimation is central to tasks that demand high temporal resolution and robustness to fast motion.<n>We propose Oriented Distance Field Motion Estimation (ODF Motion Estimation), which replaces this optimization with a single averaging step over a precomputed field of event distance vectors.<n>On public and self-collected datasets, ODF motion estimation reaches sub-pixel accuracy at the lowest latency among compared methods.
  arXiv  Detail & Related papers  (2026-08-25T08:27:42Z)
• RRTrack: Robust and Recoverable Object 6D Pose Tracking for Dynamic Scenes [55.15267980622371]
  RRTrack is an efficient, recoverable object 6D pose tracker.<n>It enables robust tracking through fast motion and target disappearance--reappearance.<n>Experiments show RRTrack improves equal-subset mean ADD-S AR by 66.3% and ADD-S AUC by 65.7% over FoundationPose.
  arXiv  Detail & Related papers  (2026-07-26T14:05:46Z)
• Time-to-Collision Based Dynamic Obstacle Avoidance Using Pretrained Vision Models for Robots in Unstructured Environments [12.499778437707759]
  We present a vision-based dynamic obstacle avoidance method that operates entirely on real-world data.<n>Our approach avoids the sim-to-real transfer problem inherent in simulation-trained policies.<n>It detects at least one frame with TTC less than 1 second for 20 out of 22 unique physical obstacles present in our test sequences.
  arXiv  Detail & Related papers  (2026-07-08T19:41:19Z)
• RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics [53.053660003572965]
  We propose RoboTracer, a 3D-aware VLM that first achieves both 3D spatial referring and measuring.<n>RoboTracer advances multi-step metric-grounded reasoning via reinforcement fine-tuning.<n>We present TraceSpatial-Bench, a challenging benchmark to evaluate spatial tracing.
  arXiv  Detail & Related papers  (2025-12-15T18:52:43Z)
• Multi-View 3D Point Tracking [67.21282192436031]
  We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views.<n>Our model directly predicts 3D correspondences using a practical number of cameras.<n>We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks.
  arXiv  Detail & Related papers  (2025-08-28T17:58:20Z)
• SpatialTrackerV2: 3D Point Tracking Made Easy [73.0350898700048]
  SpatialTrackerV2 is a feed-forward 3D point tracking method for monocular videos.<n>It decomposes world-space 3D motion into scene geometry, camera ego-motion, and pixel-wise object motion.<n>By learning geometry and motion jointly from such heterogeneous data, SpatialTrackerV2 outperforms existing 3D tracking methods by 30%.
  arXiv  Detail & Related papers  (2025-07-16T17:59:03Z)
• Towards Accurate State Estimation: Kalman Filter Incorporating Motion Dynamics for 3D Multi-Object Tracking [11.111388829965103]
  This work addresses the critical lack of precision in state estimation in the Kalman filter for 3D multi-object tracking (MOT)<n>Existing literature commonly relies on constant motion models for estimating the states of objects, neglecting the complex motion dynamics unique to each object.<n>This work introduces a novel formulation of the Kalman filter that incorporates motion dynamics, allowing the motion model to adaptively adjust according to changes in the object's movement.
  arXiv  Detail & Related papers  (2025-05-12T06:09:32Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• An Improved End-to-End Multi-Target Tracking Method Based on Transformer Self-Attention [24.17627001939523]
  This study proposes an improved end-to-end multi-target tracking algorithm. It adapts to multi-view multi-scale scenes based on the self-attentive mechanism of the transformer's encoder-decoder structure.
  arXiv  Detail & Related papers  (2022-11-11T04:58:46Z)
• MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video [75.23812405203778]
  Recent solutions have been introduced to estimate 3D human pose from 2D keypoint sequence by considering body joints among all frames globally to learn-temporal correlation. We propose Mix Mix, which has temporal transformer block to separately model the temporal motion of each joint and a transformer block inter-joint spatial correlation. In addition, the network output is extended from the central frame to entire frames of input video, improving the coherence between the input and output benchmarks.
  arXiv  Detail & Related papers  (2022-03-02T04:20:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.