Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives

• URL: http://arxiv.org/abs/2403.04971v1
• Date: Fri, 8 Mar 2024 00:57:03 GMT
• Title: Robust Surgical Tool Tracking with Pixel-based Probabilities for Projected Geometric Primitives
• Authors: Christopher D'Ambrosia, Florian Richter, Zih-Yun Chiu, Nikhil Shinde, Fei Liu, Henrik I. Christensen, Michael C. Yip
• Abstract summary: Controlling robotic manipulators via visual feedback requires a known coordinate frame transformation between the robot and the camera. Uncertainties in mechanical systems as well as camera calibration create errors in this coordinate frame transformation. We estimate the camera-to-base transform and joint angle measurement errors for surgical robotic tools using an image based insertion-shaft detection algorithm and probabilistic models.
• Score: 28.857732667640068
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Controlling robotic manipulators via visual feedback requires a known coordinate frame transformation between the robot and the camera. Uncertainties in mechanical systems as well as camera calibration create errors in this coordinate frame transformation. These errors result in poor localization of robotic manipulators and create a significant challenge for applications that rely on precise interactions between manipulators and the environment. In this work, we estimate the camera-to-base transform and joint angle measurement errors for surgical robotic tools using an image based insertion-shaft detection algorithm and probabilistic models. We apply our proposed approach in both a structured environment as well as an unstructured environment and measure to demonstrate the efficacy of our methods.

Related papers

• TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics [53.442362491589726]
  We present TIGeR (Tool-Integrated Geometric Reasoning), a novel framework that transforms Vision-Language Models (VLMs) into geometric computers.<n>Rather than attempting to internalize complex geometric operations within neural networks, TIGeR empowers models to recognize geometric reasoning requirements.<n>We show that TIGeR achieves SOTA performance on geometric reasoning benchmarks while demonstrating centimeter-level precision in real-world robotic manipulation tasks.
  arXiv  Detail & Related papers  (2025-10-08T16:20:23Z)
• Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection [21.460727996614704]
  MIS robots have long kinematic chains and partial visibility of their degrees of freedom in the camera.<n>We propose a novel framework that unifies the detection of geometric primitives through a shared encoding.<n>This architecture detects both keypoints and edges in a single inference and is trained on large-scale synthetic data with projective labeling.
  arXiv  Detail & Related papers  (2025-10-03T22:03:28Z)
• Learning Causal Structure Distributions for Robust Planning [53.753366558072806]
  We find that learning the functional relationships while accounting for the uncertainty about the structural information leads to more robust dynamics models.<n>This in contrast with common model-learning methods that ignore the causal structure and fail to leverage the sparsity of interactions in robotic systems.<n>We show that our model can be used to learn the dynamics of a robot, which together with a sampling-based planner can be used to perform new tasks in novel environments.
  arXiv  Detail & Related papers  (2025-08-08T22:43:17Z)
• Verification of Visual Controllers via Compositional Geometric Transformations [49.81690518952909]
  We introduce a novel verification framework for perception-based controllers that can generate outer-approximations of reachable sets.<n>We provide theoretical guarantees on the soundness of our method and demonstrate its effectiveness across benchmark control environments.
  arXiv  Detail & Related papers  (2025-07-06T20:22:58Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• ARC-Calib: Autonomous Markerless Camera-to-Robot Calibration via Exploratory Robot Motions [15.004750210002152]
  ARC-Calib is a model-based markerless camera-to-robot calibration framework. It is fully autonomous and generalizable across diverse robots.
  arXiv  Detail & Related papers  (2025-03-18T20:03:32Z)
• Kalib: Markerless Hand-Eye Calibration with Keypoint Tracking [52.4190876409222]
  Hand-eye calibration involves estimating the transformation between the camera and the robot. Recent advancements in deep learning offer markerless techniques, but they present challenges. We propose Kalib, an automatic and universal markerless hand-eye calibration pipeline.
  arXiv  Detail & Related papers  (2024-08-20T06:03:40Z)
• 3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects [13.58353565350936]
  We contribute methodology to jointly estimate the geometry and pose of objects grasped by a robot. Our method transforms the estimated geometry into the robot's coordinate frame. We empirically evaluate our approach on a robot manipulator holding a diverse set of real-world objects.
  arXiv  Detail & Related papers  (2024-07-14T21:02:55Z)
• VICAN: Very Efficient Calibration Algorithm for Large Camera Networks [49.17165360280794]
  We introduce a novel methodology that extends Pose Graph Optimization techniques. We consider the bipartite graph encompassing cameras, object poses evolving dynamically, and camera-object relative transformations at each time step. Our framework retains compatibility with traditional PGO solvers, but its efficacy benefits from a custom-tailored optimization scheme.
  arXiv  Detail & Related papers  (2024-03-25T17:47:03Z)
• Robot Hand-Eye Calibration using Structure-from-Motion [9.64487611393378]
  We propose a new flexible method for hand-eye calibration. We show that the solution can be obtained in linear form. We conduct a large number of experiments which validate the quality of the method by comparing it with existing ones.
  arXiv  Detail & Related papers  (2023-11-20T14:41:44Z)
• Online estimation of the hand-eye transformation from surgical scenes [11.797350284719803]
  We present a neural network-based solution that estimates the transformation from a sequence of images and kinematic data. The proposed algorithm shows that the calibration procedure can be simplified by using deep learning techniques.
  arXiv  Detail & Related papers  (2023-06-04T04:55:02Z)
• EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration [49.90228618894857]
  We introduce a new approach to hand-eye calibration called EasyHeC, which is markerless, white-box, and delivers superior accuracy and robustness. We propose to use two key technologies: differentiable rendering-based camera pose optimization and consistency-based joint space exploration. Our evaluation demonstrates superior performance in synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-02T03:49:54Z)
• Online Learning of Wheel Odometry Correction for Mobile Robots with Attention-based Neural Network [63.8376359764052]
  Modern robotic platforms need a reliable localization system to operate daily beside humans. Simple pose estimation algorithms based on filtered wheel and inertial odometry often fail in the presence of abrupt kinematic changes and wheel slips. We propose an innovative online learning approach for wheel odometry correction, paving the way for a robust multi-source localization system.
  arXiv  Detail & Related papers  (2023-03-21T10:30:31Z)
• A Distance-Geometric Method for Recovering Robot Joint Angles From an RGB Image [7.971699294672282]
  We present a novel method for retrieving the joint angles of a robot manipulator using only a single RGB image of its current configuration. Our approach, based on a distance-geometric representation of the configuration space, exploits the knowledge of a robot's kinematic model.
  arXiv  Detail & Related papers  (2023-01-05T12:57:45Z)
• A Stitching Algorithm for Automated Surface Inspection of Rotationally Symmetric Components [0.0]
  This paper presents a novel approach to stitching surface images of rotationally symmetric parts. It uses a feature-based stitching approach to create a distortion-free and true-to-life image from a video file. The developed process enables, for example, condition monitoring without having to view many individual images.
  arXiv  Detail & Related papers  (2020-12-01T07:03:45Z)
• Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimation [89.82169646672872]
  We propose an unsupervised vision-based system to estimate the joint configurations of the robot arm from a sequence of RGB or RGB-D images without knowing the model a priori. We combine a classical geometric formulation with deep learning and extend the use of epipolar multi-rigid-body constraints to solve this task.
  arXiv  Detail & Related papers  (2020-11-30T20:46:48Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.