FlowCut: Unsupervised Video Instance Segmentation via Temporal Mask Matching

• URL: http://arxiv.org/abs/2505.13174v1
• Date: Mon, 19 May 2025 14:30:33 GMT
• Title: FlowCut: Unsupervised Video Instance Segmentation via Temporal Mask Matching
• Authors: Alp Eren Sari, Paolo Favaro,
• Abstract summary: FlowCut is a method for unsupervised video instance segmentation consisting of a three-stage framework.<n>In the first stage, we generate pseudo-instance masks by exploiting the affinities of features from both images and optical flows.<n>In the second stage, we construct short video segments containing high-quality, consistent pseudo-instance masks by temporally matching them across the frames.<n>In the third stage, we use the YouTubeVIS-2021 video dataset to extract our training instance segmentation set, and then train a video segmentation model.
• Score: 19.401125268811015
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We propose FlowCut, a simple and capable method for unsupervised video instance segmentation consisting of a three-stage framework to construct a high-quality video dataset with pseudo labels. To our knowledge, our work is the first attempt to curate a video dataset with pseudo-labels for unsupervised video instance segmentation. In the first stage, we generate pseudo-instance masks by exploiting the affinities of features from both images and optical flows. In the second stage, we construct short video segments containing high-quality, consistent pseudo-instance masks by temporally matching them across the frames. In the third stage, we use the YouTubeVIS-2021 video dataset to extract our training instance segmentation set, and then train a video segmentation model. FlowCut achieves state-of-the-art performance on the YouTubeVIS-2019, YouTubeVIS-2021, DAVIS-2017, and DAVIS-2017 Motion benchmarks.

Related papers

• PM-VIS+: High-Performance Video Instance Segmentation without Video Annotation [15.9587266448337]
  Video instance segmentation requires detecting, segmenting, and tracking objects in videos. This paper introduces a method that eliminates video annotations by utilizing image datasets.
  arXiv  Detail & Related papers  (2024-06-28T05:22:39Z)
• UVIS: Unsupervised Video Instance Segmentation [65.46196594721545]
  Videocaption instance segmentation requires classifying, segmenting, and tracking every object across video frames. We propose UVIS, a novel Unsupervised Video Instance (UVIS) framework that can perform video instance segmentation without any video annotations or dense label-based pretraining. Our framework consists of three essential steps: frame-level pseudo-label generation, transformer-based VIS model training, and query-based tracking.
  arXiv  Detail & Related papers  (2024-06-11T03:05:50Z)
• VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation [87.13210748484217]
  VideoCutLER is a simple method for unsupervised multi-instance video segmentation without using motion-based learning signals like optical flow or training on natural videos. We show the first competitive unsupervised learning results on the challenging YouTubeVIS 2019 benchmark, achieving 50.7% APvideo50. VideoCutLER can also serve as a strong pretrained model for supervised video instance segmentation tasks, exceeding DINO by 15.9% on YouTubeVIS 2019 in terms of APvideo.
  arXiv  Detail & Related papers  (2023-08-28T17:10:12Z)
• Robust Online Video Instance Segmentation with Track Queries [15.834703258232002]
  We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos.
  arXiv  Detail & Related papers  (2022-11-16T18:50:14Z)
• TubeFormer-DeepLab: Video Mask Transformer [98.47947102154217]
  We present TubeFormer-DeepLab, the first attempt to tackle multiple core video segmentation tasks in a unified manner. TubeFormer-DeepLab directly predicts video tubes with task-specific labels.
  arXiv  Detail & Related papers  (2022-05-30T18:10:33Z)
• Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion [92.80981308407098]
  We propose an approach that combines the strengths of motion-based and appearance-based segmentation. We propose to supervise an image segmentation network, tasking it with predicting regions that are likely to contain simple motion patterns. In the unsupervised video segmentation mode, the network is trained on a collection of unlabelled videos, using the learning process itself as an algorithm to segment these videos.
  arXiv  Detail & Related papers  (2022-05-16T17:55:34Z)
• Tag-Based Attention Guided Bottom-Up Approach for Video Instance Segmentation [83.13610762450703]
  Video instance is a fundamental computer vision task that deals with segmenting and tracking object instances across a video sequence. We introduce a simple end-to-end train bottomable-up approach to achieve instance mask predictions at the pixel-level granularity, instead of the typical region-proposals-based approach. Our method provides competitive results on YouTube-VIS and DAVIS-19 datasets, and has minimum run-time compared to other contemporary state-of-the-art performance methods.
  arXiv  Detail & Related papers  (2022-04-22T15:32:46Z)
• Spatial Feature Calibration and Temporal Fusion for Effective One-stage Video Instance Segmentation [16.692219644392253]
  We propose a one-stage video instance segmentation framework by spatial calibration and temporal fusion, namely STMask. Experiments on the YouTube-VIS valid set show that the proposed STMask with ResNet-50/-101 backbone obtains 33.5 % / 36.8 % mask AP, while achieving 28.6 / 23.4 FPS on video instance segmentation.
  arXiv  Detail & Related papers  (2021-04-06T09:26:58Z)
• End-to-End Video Instance Segmentation with Transformers [84.17794705045333]
  Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy.
  arXiv  Detail & Related papers  (2020-11-30T02:03:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.