Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness

• URL: http://arxiv.org/abs/2407.05448v1
• Date: Sun, 7 Jul 2024 17:17:52 GMT
• Title: Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness
• Authors: Idris Hamoud, Alexandros Karargyris, Aidean Sharghi, Omid Mohareri, Nicolas Padoy,
• Abstract summary: In the Operating Room, semantic segmentation is at the core of creating robots aware of clinical surroundings. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras.
• Score: 44.15562068190958
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Semantic segmentation and activity classification are key components to creating intelligent surgical systems able to understand and assist clinical workflow. In the Operating Room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras. Contrary to other self-supervised approaches, where handcrafted pretext tasks are focused on 2D image features, our proposed task consists of predicting the relative 3D distance of image patches by exploiting the depth maps. Learning 3D spatial context generates discriminative features for our downstream tasks. Our approach is evaluated on two tasks and datasets containing multi-view data captured from clinical scenarios. We demonstrate a noteworthy improvement of performance on both tasks, specifically on low-regime data where utility of self-supervised learning is the highest.

Related papers

• Generalized Label-Efficient 3D Scene Parsing via Hierarchical Feature Aligned Pre-Training and Region-Aware Fine-tuning [55.517000360348725]
  This work presents a framework for dealing with 3D scene understanding when the labeled scenes are quite limited. To extract knowledge for novel categories from the pre-trained vision-language models, we propose a hierarchical feature-aligned pre-training and knowledge distillation strategy. Experiments with both indoor and outdoor scenes demonstrated the effectiveness of our approach in both data-efficient learning and open-world few-shot learning.
  arXiv  Detail & Related papers  (2023-12-01T15:47:04Z)
• Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge [62.981429762309226]
  The ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually. We propose to improve phrase grounding models' ability on localizing the active objects by: learning the role of objects undergoing change and extracting them accurately from the instructions. We evaluate our framework on Ego4D and Epic-Kitchens datasets.
  arXiv  Detail & Related papers  (2023-10-23T16:14:05Z)
• A Point in the Right Direction: Vector Prediction for Spatially-aware Self-supervised Volumetric Representation Learning [12.369884719068228]
  VectorPOSE promotes better spatial understanding with two novel pretext tasks: Vector Prediction and Boundary-Focused Reconstruction. We evaluate VectorPOSE on three 3D medical image segmentation tasks, showing that it often outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-11-15T22:10:50Z)
• Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation [87.1188556802942]
  We present an approach for encoding visual task relationships to improve model performance in an Unsupervised Domain Adaptation (UDA) setting. We propose a novel Cross-Task Relation Layer (CTRL), which encodes task dependencies between the semantic and depth predictions. Furthermore, we propose an Iterative Self-Learning (ISL) training scheme, which exploits semantic pseudo-labels to provide extra supervision on the target domain.
  arXiv  Detail & Related papers  (2021-05-17T13:42:09Z)
• 3D Spatial Recognition without Spatially Labeled 3D [127.6254240158249]
  We introduce WyPR, a Weakly-supervised framework for Point cloud Recognition. We show that WyPR can detect and segment objects in point cloud data without access to any spatial labels at training time.
  arXiv  Detail & Related papers  (2021-05-13T17:58:07Z)
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation [4.127128889779478]
  This work focuses on performing better or comparable to the existing learning-based solutions for visual navigation for autonomous agents. We propose a method to encode vital scene semantics into a semantically informed, top-down egocentric map representation. We conduct experiments on 3-D reconstructed indoor PointGoal visual navigation and demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2021-03-21T12:01:23Z)
• Embodied Visual Active Learning for Semantic Segmentation [33.02424587900808]
  We study the task of embodied visual active learning, where an agent is set to explore a 3d environment with the goal to acquire visual scene understanding. We develop a battery of agents - both learnt and pre-specified - and with different levels of knowledge of the environment. We extensively evaluate the proposed models using the Matterport3D simulator and show that a fully learnt method outperforms comparable pre-specified counterparts.
  arXiv  Detail & Related papers  (2020-12-17T11:02:34Z)
• Improving Point Cloud Semantic Segmentation by Learning 3D Object Detection [102.62963605429508]
  Point cloud semantic segmentation plays an essential role in autonomous driving. Current 3D semantic segmentation networks focus on convolutional architectures that perform great for well represented classes. We propose a novel Aware 3D Semantic Detection (DASS) framework that explicitly leverages localization features from an auxiliary 3D object detection task.
  arXiv  Detail & Related papers  (2020-09-22T14:17:40Z)
• Taskology: Utilizing Task Relations at Scale [28.09712466727001]
  We show that we can leverage the inherent relationships among collections of tasks, as they are trained jointly. explicitly utilizing the relationships between tasks allows improving their performance while dramatically reducing the need for labeled data. We demonstrate our framework on subsets of the following collection of tasks: depth and normal prediction, semantic segmentation, 3D motion and ego-motion estimation, and object tracking and 3D detection in point clouds.
  arXiv  Detail & Related papers  (2020-05-14T22:53:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.