Probabilistic Task Parameterization of Tool-Tissue Interaction via Sparse Landmarks Tracking in Robotic Surgery

• URL: http://arxiv.org/abs/2504.11495v1
• Date: Mon, 14 Apr 2025 21:28:48 GMT
• Title: Probabilistic Task Parameterization of Tool-Tissue Interaction via Sparse Landmarks Tracking in Robotic Surgery
• Authors: Yiting Wang, Yunxin Fan, Fei Liu,
• Abstract summary: Models of tool-tissue interactions in robotic surgery require precise tracking of deformable tissues and integration of surgical domain knowledge.<n>We propose a framework combining keypoint tracking and probabilistic modeling that propagates expert-annotated landmarks across endoscopic frames.
• Score: 5.075735148466963
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate modeling of tool-tissue interactions in robotic surgery requires precise tracking of deformable tissues and integration of surgical domain knowledge. Traditional methods rely on labor-intensive annotations or rigid assumptions, limiting flexibility. We propose a framework combining sparse keypoint tracking and probabilistic modeling that propagates expert-annotated landmarks across endoscopic frames, even with large tissue deformations. Clustered tissue keypoints enable dynamic local transformation construction via PCA, and tool poses, tracked similarly, are expressed relative to these frames. Embedding these into a Task-Parameterized Gaussian Mixture Model (TP-GMM) integrates data-driven observations with labeled clinical expertise, effectively predicting relative tool-tissue poses and enhancing visual understanding of robotic surgical motions directly from video data.

Related papers

• Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection [50.388465935739376]
  Liver registration by overlaying preoperative 3D models onto intraoperative 2D frames can assist surgeons in perceiving the spatial anatomy of the liver clearly for a higher surgical success rate. Existing registration methods rely heavily on anatomical landmark-based, which encounter two major limitations. We propose a landmark-free preoperative-to-intraoperative registration framework utilizing effective self-supervised learning.
  arXiv  Detail & Related papers  (2025-04-21T14:55:57Z)
• Monocular pose estimation of articulated surgical instruments in open surgery [0.873811641236639]
  This work presents a novel approach to monocular 6D pose estimation of surgical instruments in open surgery, addressing challenges such as object articulations, symmetries, and lack of annotated real-world data. The proposed approach consists of three main components: (1) synthetic data generation using 3D modeling of surgical tools with articulation rigging; (2) a tailored pose estimation framework combining object detection with pose estimation and a hybrid geometric fusion strategy; and (3) a training strategy that utilizes both synthetic and real unannotated data, employing domain adaptation on real video data using automatically generated pseudo-labels.
  arXiv  Detail & Related papers  (2024-07-16T19:47:35Z)
• Instrument-tissue Interaction Detection Framework for Surgical Video Understanding [31.822025965225016]
  We present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance.
  arXiv  Detail & Related papers  (2024-03-30T11:21:11Z)
• Self-trained Panoptic Segmentation [0.0]
  Panoptic segmentation is an important computer vision task which combines semantic and instance segmentation. Recent advancements in self-supervised learning approaches have shown great potential in leveraging synthetic and unlabelled data to generate pseudo-labels. The aim of this work is to develop a framework to perform embedding-based self-supervised panoptic segmentation using self-training in a synthetic-to-real domain adaptation problem setting.
  arXiv  Detail & Related papers  (2023-11-17T17:06:59Z)
• Domain Adaptive Sim-to-Real Segmentation of Oropharyngeal Organs Towards Robot-assisted Intubation [15.795665057836636]
  This work introduces a virtual dataset generated by the Open Framework Architecture framework to overcome the limited availability of actual endoscopic images. We also propose a domain adaptive Sim-to-Real method for oropharyngeal organ image segmentation, which employs an image blending strategy. Experimental results demonstrate the superior performance of the proposed approach with domain adaptive models.
  arXiv  Detail & Related papers  (2023-05-19T14:08:15Z)
• Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments [64.59698930334012]
  We present a multi-camera capture setup consisting of static and head-mounted cameras.<n>Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre.<n>Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments.
  arXiv  Detail & Related papers  (2023-05-05T13:42:19Z)
• Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures [70.69948035469467]
  We take advantage of the latest computer vision methodologies for generating 3D graphs from camera views. We then introduce the Multimodal Semantic Graph Scene (MSSG) which aims at providing unified symbolic and semantic representation of surgical procedures.
  arXiv  Detail & Related papers  (2021-06-09T14:35:44Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)
• Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment [60.31418655784291]
  We propose a novel unsupervised domain adaptation framework which can simultaneously transfer multi-modality knowledge, i.e., both kinematic and visual data, from simulator to real robot. It remedies the domain gap with enhanced transferable features by using temporal cues in videos, and inherent correlations in multi-modal towards recognizing gesture. Results show that our approach recovers the performance with great improvement gains, up to 12.91% in ACC and 20.16% in F1score without using any annotations in real robot.
  arXiv  Detail & Related papers  (2021-03-06T09:10:03Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)
• Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery [10.562627972607892]
  We show that it may be possible to use robot kinematic data coupled with laparoscopic images to alleviate the labelling problem. We propose a new deep learning based model for parallel processing of both laparoscopic and simulation images.
  arXiv  Detail & Related papers  (2020-07-17T16:33:33Z)
• A Trainable Optimal Transport Embedding for Feature Aggregation and its Relationship to Attention [96.77554122595578]
  We introduce a parametrized representation of fixed size, which embeds and then aggregates elements from a given input set according to the optimal transport plan between the set and a trainable reference. Our approach scales to large datasets and allows end-to-end training of the reference, while also providing a simple unsupervised learning mechanism with small computational cost.
  arXiv  Detail & Related papers  (2020-06-22T08:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.