UVIS: Unsupervised Video Instance Segmentation

• URL: http://arxiv.org/abs/2406.06908v1
• Date: Tue, 11 Jun 2024 03:05:50 GMT
• Title: UVIS: Unsupervised Video Instance Segmentation
• Authors: Shuaiyi Huang, Saksham Suri, Kamal Gupta, Sai Saketh Rambhatla, Ser-nam Lim, Abhinav Shrivastava,
• Abstract summary: Videocaption instance segmentation requires classifying, segmenting, and tracking every object across video frames. We propose UVIS, a novel Unsupervised Video Instance (UVIS) framework that can perform video instance segmentation without any video annotations or dense label-based pretraining. Our framework consists of three essential steps: frame-level pseudo-label generation, transformer-based VIS model training, and query-based tracking.
• Score: 65.46196594721545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video instance segmentation requires classifying, segmenting, and tracking every object across video frames. Unlike existing approaches that rely on masks, boxes, or category labels, we propose UVIS, a novel Unsupervised Video Instance Segmentation (UVIS) framework that can perform video instance segmentation without any video annotations or dense label-based pretraining. Our key insight comes from leveraging the dense shape prior from the self-supervised vision foundation model DINO and the openset recognition ability from the image-caption supervised vision-language model CLIP. Our UVIS framework consists of three essential steps: frame-level pseudo-label generation, transformer-based VIS model training, and query-based tracking. To improve the quality of VIS predictions in the unsupervised setup, we introduce a dual-memory design. This design includes a semantic memory bank for generating accurate pseudo-labels and a tracking memory bank for maintaining temporal consistency in object tracks. We evaluate our approach on three standard VIS benchmarks, namely YoutubeVIS-2019, YoutubeVIS-2021, and Occluded VIS. Our UVIS achieves 21.1 AP on YoutubeVIS-2019 without any video annotations or dense pretraining, demonstrating the potential of our unsupervised VIS framework.

Related papers

• DVIS++: Improved Decoupled Framework for Universal Video Segmentation [30.703276476607545]
  We present OV-DVIS++, the first open-vocabulary universal video segmentation framework. By integrating CLIP with DVIS++, we present OV-DVIS++, the first open-vocabulary universal video segmentation framework.
  arXiv  Detail & Related papers  (2023-12-20T03:01:33Z)
• NOVIS: A Case for End-to-End Near-Online Video Instance Segmentation [22.200700685751826]
  Video Instance (VIS) community operated under the common belief that offline methods are generally superior to a frame by frame online processing. We present a detailed analysis on different processing paradigms and a new end-to-end Video Instance method. Our NOVIS represents the first near-online VIS approach which avoids any handcrafted trackings.
  arXiv  Detail & Related papers  (2023-08-29T12:51:04Z)
• CTVIS: Consistent Training for Online Video Instance Segmentation [62.957370691452844]
  Discrimination of instance embeddings plays a vital role in associating instances across time for online video instance segmentation (VIS) Recent online VIS methods leverage CIs sourced from one reference frame only, which we argue is insufficient for learning highly discriminative embeddings. We propose a simple yet effective training strategy, called Consistent Training for Online VIS (CTVIS), which devotes to aligning the training and inference pipelines.
  arXiv  Detail & Related papers  (2023-07-24T08:44:25Z)
• Towards Open-Vocabulary Video Instance Segmentation [61.469232166803465]
  Video Instance aims at segmenting and categorizing objects in videos from a closed set of training categories. We introduce the novel task of Open-Vocabulary Video Instance, which aims to simultaneously segment, track, and classify objects in videos from open-set categories. To benchmark Open-Vocabulary VIS, we collect a Large-Vocabulary Video Instance dataset (LV-VIS), that contains well-annotated objects from 1,196 diverse categories.
  arXiv  Detail & Related papers  (2023-04-04T11:25:23Z)
• BoxVIS: Video Instance Segmentation with Box Annotations [15.082477136581153]
  We adapt the state-of-the-art pixel-supervised VIS models to a box-supervised VIS baseline and observe slight performance degradation. We propose a box-center guided spatial-temporal pairwise affinity loss to predict instance masks for better spatial and temporal consistency. It exhibits comparable instance mask prediction performance and better generalization ability than state-of-the-art pixel-supervised VIS models by using only 16% of their annotation time and cost.
  arXiv  Detail & Related papers  (2023-03-26T04:04:58Z)
• A Generalized Framework for Video Instance Segmentation [49.41441806931224]
  The handling of long videos with complex and occluded sequences has emerged as a new challenge in the video instance segmentation (VIS) community. We propose a Generalized framework for VIS, namely GenVIS, that achieves state-of-the-art performance on challenging benchmarks. We evaluate our approach on popular VIS benchmarks, achieving state-of-the-art results on YouTube-VIS 2019/2021/2022 and Occluded VIS (OVIS)
  arXiv  Detail & Related papers  (2022-11-16T11:17:19Z)
• MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Training [84.81566912372328]
  MinVIS is a minimal video instance segmentation framework. It achieves state-of-the-art VIS performance with neither video-based architectures nor training procedures.
  arXiv  Detail & Related papers  (2022-08-03T17:50:42Z)
• Crossover Learning for Fast Online Video Instance Segmentation [53.5613957875507]
  We present a novel crossover learning scheme that uses the instance feature in the current frame to pixel-wisely localize the same instance in other frames. To our knowledge, CrossVIS achieves state-of-the-art performance among all online VIS methods and shows a decent trade-off between latency and accuracy.
  arXiv  Detail & Related papers  (2021-04-13T06:47:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.