Disentangling spatio-temporal knowledge for weakly supervised object detection and segmentation in surgical video

• URL: http://arxiv.org/abs/2407.15794v4
• Date: Fri, 1 Nov 2024 14:19:14 GMT
• Title: Disentangling spatio-temporal knowledge for weakly supervised object detection and segmentation in surgical video
• Authors: Guiqiu Liao, Matjaz Jogan, Sai Koushik, Eric Eaton, Daniel A. Hashimoto,
• Abstract summary: This paper introduces Video Spatio-Temporal Disment Networks (VDST-Net) to disentangle information using semi-decoupled temporal knowledge distillation to predict high-quality class activation maps (CAMs) We demonstrate the efficacy of our framework on a public reference dataset and on a more challenging surgical video dataset where objects are, on average, present in less than 60% of annotated frames.
• Score: 10.287675722826028
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Weakly supervised video object segmentation (WSVOS) enables the identification of segmentation maps without requiring an extensive training dataset of object masks, relying instead on coarse video labels indicating object presence. Current state-of-the-art methods either require multiple independent stages of processing that employ motion cues or, in the case of end-to-end trainable networks, lack in segmentation accuracy, in part due to the difficulty of learning segmentation maps from videos with transient object presence. This limits the application of WSVOS for semantic annotation of surgical videos where multiple surgical tools frequently move in and out of the field of view, a problem that is more difficult than typically encountered in WSVOS. This paper introduces Video Spatio-Temporal Disentanglement Networks (VDST-Net), a framework to disentangle spatiotemporal information using semi-decoupled knowledge distillation to predict high-quality class activation maps (CAMs). A teacher network designed to resolve temporal conflicts when specifics about object location and timing in the video are not provided works with a student network that integrates information over time by leveraging temporal dependencies. We demonstrate the efficacy of our framework on a public reference dataset and on a more challenging surgical video dataset where objects are, on average, present in less than 60\% of annotated frames. Our method outperforms state-of-the-art techniques and generates superior segmentation masks under video-level weak supervision.

Related papers

• Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts [57.01985221057047]
  This paper introduces a novel method that learnstemporal prompt embeddings for weakly supervised video anomaly detection and localization (WSVADL) based on pre-trained vision-language models (VLMs) Our method achieves state-of-theart performance on three public benchmarks for the WSVADL task.
  arXiv  Detail & Related papers  (2024-08-12T03:31:29Z)
• Training-Free Robust Interactive Video Object Segmentation [82.05906654403684]
  We propose a training-free prompt tracking framework for interactive video object segmentation (I-PT) We jointly adopt sparse points and boxes tracking, filtering out unstable points and capturing object-wise information. Our framework has demonstrated robust zero-shot video segmentation results on popular VOS datasets.
  arXiv  Detail & Related papers  (2024-06-08T14:25:57Z)
• Robotic Scene Segmentation with Memory Network for Runtime Surgical Context Inference [8.600278838838163]
  Space Time Correspondence Network (STCN) is a memory network that performs binary segmentation and minimizes the effects of class imbalance. We show that STCN achieves superior segmentation performance for objects that are difficult to segment, such as needle and thread. We also demonstrate that segmentation and context inference can be performed at runtime without compromising performance.
  arXiv  Detail & Related papers  (2023-08-24T13:44:55Z)
• TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering [27.52568444236988]
  We propose an unsupervised approach for learning action classes from untrimmed video sequences. In particular, we propose a temporal embedding network that combines relative time prediction, feature reconstruction, and sequence-to-sequence learning. Based on the identified clusters, we decode the video into coherent temporal segments that correspond to semantically meaningful action classes.
  arXiv  Detail & Related papers  (2023-03-09T10:46:23Z)
• Video Salient Object Detection via Contrastive Features and Attention Modules [106.33219760012048]
  We propose a network with attention modules to learn contrastive features for video salient object detection. A co-attention formulation is utilized to combine the low-level and high-level features. We show that the proposed method requires less computation, and performs favorably against the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-11-03T17:40:32Z)
• Generating Masks from Boxes by Mining Spatio-Temporal Consistencies in Videos [159.02703673838639]
  We introduce a method for generating segmentation masks from per-frame bounding box annotations in videos. We use our resulting accurate masks for weakly supervised training of video object segmentation (VOS) networks. The additional data provides substantially better generalization performance leading to state-of-the-art results in both the VOS and more challenging tracking domain.
  arXiv  Detail & Related papers  (2021-01-06T18:56:24Z)
• DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping [72.84991726271024]
  We describe an unsupervised method to detect and segment portions of images of live scenes that are seen moving as a coherent whole. Our method first partitions the motion field by minimizing the mutual information between segments. It uses the segments to learn object models that can be used for detection in a static image.
  arXiv  Detail & Related papers  (2020-08-16T22:05:13Z)
• ZSTAD: Zero-Shot Temporal Activity Detection [107.63759089583382]
  We propose a novel task setting called zero-shot temporal activity detection (ZSTAD), where activities that have never been seen in training can still be detected. We design an end-to-end deep network based on R-C3D as the architecture for this solution. Experiments on both the THUMOS14 and the Charades datasets show promising performance in terms of detecting unseen activities.
  arXiv  Detail & Related papers  (2020-03-12T02:40:36Z)
• Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences [25.299599341774204]
  This paper proposes an approach for the unsupervised learning of actions in untrimmed video sequences based on a joint visual-temporal embedding space. We show that the proposed approach is able to provide a meaningful visual and temporal embedding out of the visual cues present in contiguous video frames.
  arXiv  Detail & Related papers  (2020-01-29T22:51:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.