First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network

• URL: http://arxiv.org/abs/2412.16503v1
• Date: Sat, 21 Dec 2024 06:36:25 GMT
• Title: First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network
• Abstract summary: Video polyp segmentation plays a critical role in gastrointestinal cancer screening.<n>For the first time, we reduce the annotation cost to just a single frame per polyp video, regardless of the video's length.<n>PSDNet employs two distinct types of teachers: the propagative teacher and the semantic teacher.
• Score: 15.213681514145346
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic video polyp segmentation plays a critical role in gastrointestinal cancer screening, but the cost of frameby-frame annotations is prohibitively high. While sparse-frame supervised methods have reduced this burden proportionately, the cost remains overwhelming for long-duration videos and large-scale datasets. In this paper, we, for the first time, reduce the annotation cost to just a single frame per polyp video, regardless of the video's length. To this end, we introduce a new task, First-Frame Supervised Video Polyp Segmentation (FSVPS), and propose a novel Propagative and Semantic Dual-Teacher Network (PSDNet). Specifically, PSDNet adopts a teacher-student framework but employs two distinct types of teachers: the propagative teacher and the semantic teacher. The propagative teacher is a universal object tracker that propagates the first-frame annotation to subsequent frames as pseudo labels. However, tracking errors may accumulate over time, gradually degrading the pseudo labels and misguiding the student model. To address this, we introduce the semantic teacher, an exponential moving average of the student model, which produces more stable and time-invariant pseudo labels. PSDNet merges the pseudo labels from both teachers using a carefully-designed back-propagation strategy. This strategy assesses the quality of the pseudo labels by tracking them backward to the first frame. High-quality pseudo labels are more likely to spatially align with the firstframe annotation after this backward tracking, ensuring more accurate teacher-to-student knowledge transfer and improved segmentation performance. Benchmarking on SUN-SEG, the largest VPS dataset, demonstrates the competitive performance of PSDNet compared to fully-supervised approaches, and its superiority over sparse-frame supervised state-of-the-arts with a minimum improvement of 4.5% in Dice score.

Related papers

• SA-VIS: Sparse frame Annotations for training Video Instance Segmentation [58.2561806729996]
  Recent online video instance segmentation (VIS) methods have achieved impressive results, thus becoming the preferred approach to segment instances in videos.<n>However, a training setup of VIS is expensive in the sense of compute as well as dense annotations required.<n>We argue that the effective modeling of the instances and their evolution in videos do not require densely annotated frames.<n>This simple low-compute module provides tremendous learning capability in using sparse video frame labels for end-to-end training.
  arXiv  Detail & Related papers  (2026-06-18T12:03:04Z)
• CoDTS: Enhancing Sparsely Supervised Collaborative Perception with a Dual Teacher-Student Framework [15.538850922083652]
  We propose an end-to-end Collaborative perception Dual Teacher-Student framework (CoDTS) It employs adaptive complementary learning to produce both high-quality and high-quantity pseudo labels. CoDTS effectively ensures an optimal balance of pseudo labels in both quality and quantity.
  arXiv  Detail & Related papers  (2024-12-11T12:34:37Z)
• Semi-supervised Video Semantic Segmentation Using Unreliable Pseudo Labels for PVUW2024 [12.274092278786966]
  We adopt semi-supervised video semantic segmentation method based on unreliable pseudo labels. Our method achieves the mIoU scores of 63.71% and 67.83% on development test and final test respectively. We obtain the 1st place in the Video Scene Parsing in the Wild Challenge at CVPR 2024.
  arXiv  Detail & Related papers  (2024-06-02T01:37:26Z)
• Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding [70.31050639330603]
  Video paragraph grounding aims at localizing multiple sentences with semantic relations and temporal order from an untrimmed video. Existing VPG approaches are heavily reliant on a considerable number of temporal labels that are laborious and time-consuming to acquire. We introduce and explore Weakly-Supervised Video paragraph Grounding (WSVPG) to eliminate the need of temporal annotations.
  arXiv  Detail & Related papers  (2024-03-18T04:30:31Z)
• Switching Temporary Teachers for Semi-Supervised Semantic Segmentation [45.20519672287495]
  The teacher-student framework, prevalent in semi-supervised semantic segmentation, mainly employs the exponential moving average (EMA) to update a single teacher's weights based on the student's. This paper introduces Dual Teacher, a simple yet effective approach that employs dual temporary teachers aiming to alleviate the coupling problem for the student.
  arXiv  Detail & Related papers  (2023-10-28T08:49:16Z)
• Two-shot Video Object Segmentation [35.48207692959968]
  We train a video object segmentation model on sparsely annotated videos. We generate pseudo labels for unlabeled frames and optimize the model on the combination of labeled and pseudo-labeled data. For the first time, we present a general way to train VOS models on two-shot VOS datasets.
  arXiv  Detail & Related papers  (2023-03-21T17:59:56Z)
• Semi-Supervised Semantic Segmentation via Gentle Teaching Assistant [72.4512562104361]
  We argue that the unlabeled data with pseudo labels can facilitate the learning of representative features in the feature extractor. Motivated by this consideration, we propose a novel framework, Gentle Teaching Assistant (GTA-Seg) to disentangle the effects of pseudo labels on feature extractor and mask predictor.
  arXiv  Detail & Related papers  (2023-01-18T07:11:24Z)
• Self-supervised and Weakly Supervised Contrastive Learning for Frame-wise Action Representations [26.09611987412578]
  We introduce a new framework of contrastive action representation learning (CARL) to learn frame-wise action representation in a self-supervised or weakly-supervised manner. Specifically, we introduce a simple but effective video encoder that considers both spatial and temporal context. Our method outperforms previous state-of-the-art by a large margin for downstream fine-grained action classification and even faster inference.
  arXiv  Detail & Related papers  (2022-12-06T16:42:22Z)
• Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning [59.64695628433855]
  We propose a novel cross-modality weakly supervised method for 3D segmentation, incorporating complementary information from unlabeled images. Basically, we design a dual-branch network equipped with an active labeling strategy, to maximize the power of tiny parts of labels. Our method even outperforms the state-of-the-art fully supervised competitors with less than 1% actively selected annotations.
  arXiv  Detail & Related papers  (2022-09-16T07:59:04Z)
• Label Matching Semi-Supervised Object Detection [85.99282969977541]
  Semi-supervised object detection has made significant progress with the development of mean teacher driven self-training. Label mismatch problem is not yet fully explored in the previous works, leading to severe confirmation bias during self-training. We propose a simple yet effective LabelMatch framework from two different yet complementary perspectives.
  arXiv  Detail & Related papers  (2022-06-14T05:59:41Z)
• Video Polyp Segmentation: A Deep Learning Perspective [97.70996418522748]
  We present the first comprehensive video polyp segmentation (VPS) study. We first introduce a high-quality per-frame annotated VPS dataset, named SUN-SEG, which includes 158,690 frames from the famous SUN dataset. We also design a simple but efficient baseline, dubbed PNS+, consisting of a global encoder, a local encoder, and normalized self-attention blocks.
  arXiv  Detail & Related papers  (2022-03-27T12:40:10Z)
• Unsupervised Domain Adaptation for Video Semantic Segmentation [91.30558794056054]
  Unsupervised Domain Adaptation for semantic segmentation has gained immense popularity since it can transfer knowledge from simulation to real. In this work, we present a new video extension of this task, namely Unsupervised Domain Adaptation for Video Semantic approaches. We show that our proposals significantly outperform previous image-based UDA methods both on image-level (mIoU) and video-level (VPQ) evaluation metrics.
  arXiv  Detail & Related papers  (2021-07-23T07:18:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.