Can Unsupervised Segmentation Reduce Annotation Costs for Video Semantic Segmentation?

• URL: http://arxiv.org/abs/2603.27697v1
• Date: Sun, 29 Mar 2026 13:51:56 GMT
• Title: Can Unsupervised Segmentation Reduce Annotation Costs for Video Semantic Segmentation?
• Abstract summary: Deep neural networks for video semantic segmentation require a large number of fine-grained pixel-level annotations to achieve the best results.<n>This paper investigates approaches to reduce the annotation cost required for video segmentation datasets by utilising such resources.
• Score: 21.229795661475826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Present-day deep neural networks for video semantic segmentation require a large number of fine-grained pixel-level annotations to achieve the best possible results. Obtaining such annotations, however, is very expensive. On the other hand, raw, unannotated video frames are practically free to obtain. Similarly, coarse annotations, which do not require precise boundaries, are also much cheaper. This paper investigates approaches to reduce the annotation cost required for video segmentation datasets by utilising such resources. We show that using state-of-the-art segmentation foundation models, Segment Anything Model (SAM) and Segment Anything Model 2 (SAM 2), we can utilise both unannotated frames as well as coarse annotations to alleviate the effort required for manual annotation of video segmentation datasets by automating mask generation. Our investigation suggests that if used appropriately, we can reduce the need for annotation by a third with similar performance for video semantic segmentation. More significantly, our analysis suggests that the variety of frames in the dataset is more important than the number of frames for obtaining the best performance.

Related papers

• Parameter-free Video Segmentation for Vision and Language Understanding [55.20132267309382]
  We propose an algorithm for segmenting videos into contiguous chunks, based on the minimum description length principle.<n>The algorithm is entirely parameter-free, given feature vectors, not requiring a set threshold or the number or size of chunks to be specified.
  arXiv  Detail & Related papers  (2025-03-03T05:54:37Z)
• PM-VIS+: High-Performance Video Instance Segmentation without Video Annotation [15.9587266448337]
  Video instance segmentation requires detecting, segmenting, and tracking objects in videos. This paper introduces a method that eliminates video annotations by utilizing image datasets.
  arXiv  Detail & Related papers  (2024-06-28T05:22:39Z)
• Appearance-Based Refinement for Object-Centric Motion Segmentation [85.2426540999329]
  We introduce an appearance-based refinement method that leverages temporal consistency in video streams to correct inaccurate flow-based proposals. Our approach involves a sequence-level selection mechanism that identifies accurate flow-predicted masks as exemplars. Its performance is evaluated on multiple video segmentation benchmarks, including DAVIS, YouTube, SegTrackv2, and FBMS-59.
  arXiv  Detail & Related papers  (2023-12-18T18:59:51Z)
• How to Efficiently Annotate Images for Best-Performing Deep Learning Based Segmentation Models: An Empirical Study with Weak and Noisy Annotations and Segment Anything Model [16.745318743249864]
  Deep neural networks (DNNs) have demonstrated exceptional performance across various image segmentation tasks.<n>To mitigate this challenge, alternative approaches such as using weak labels or less precise (noisy) annotations can be employed.<n>Noisy and weak labels are significantly quicker to generate, allowing for more annotated images within the same time frame.
  arXiv  Detail & Related papers  (2023-12-17T04:26:42Z)
• Learning Semantic Segmentation with Query Points Supervision on Aerial Images [57.09251327650334]
  We present a weakly supervised learning algorithm to train semantic segmentation algorithms. Our proposed approach performs accurate semantic segmentation and improves efficiency by significantly reducing the cost and time required for manual annotation.
  arXiv  Detail & Related papers  (2023-09-11T14:32:04Z)
• Tracking Anything with Decoupled Video Segmentation [87.07258378407289]
  We develop a decoupled video segmentation approach (DEVA) It is composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks.
  arXiv  Detail & Related papers  (2023-09-07T17:59:41Z)
• Learning Referring Video Object Segmentation from Weak Annotation [78.45828085350936]
  Referring video object segmentation (RVOS) is a task that aims to segment the target object in all video frames based on a sentence describing the object. We propose a new annotation scheme that reduces the annotation effort by 8 times, while providing sufficient supervision for RVOS. Our scheme only requires a mask for the frame where the object first appears and bounding boxes for the rest of the frames.
  arXiv  Detail & Related papers  (2023-08-04T06:50:52Z)
• XMem++: Production-level Video Segmentation From Few Annotated Frames [32.68978079571079]
  We introduce a novel semi-supervised video object segmentation (SSVOS) model, XMem++, that improves existing memory-based models. Our method can extract highly consistent results while keeping the required number of frame annotations low. We demonstrate SOTA performance on challenging (partial and multi-class) segmentation scenarios as well as long videos.
  arXiv  Detail & Related papers  (2023-07-29T11:18:23Z)
• Weakly Supervised Instance Segmentation for Videos with Temporal Mask Consistency [28.352140544936198]
  Weakly supervised instance segmentation reduces the cost of annotations required to train models. We show that these issues can be better addressed by training with weakly labeled videos instead of images. We are the first to explore the use of these video signals to tackle weakly supervised instance segmentation.
  arXiv  Detail & Related papers  (2021-03-23T23:20:46Z)
• Temporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation [96.67525775629444]
  Action segmentation refers to inferring boundaries of semantically consistent visual concepts in videos. We present a fully automatic and unsupervised approach for segmenting actions in a video that does not require any training. Our proposal is an effective temporally-weighted hierarchical clustering algorithm that can group semantically consistent frames of the video.
  arXiv  Detail & Related papers  (2021-03-20T23:30:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.