One-shot Training for Video Object Segmentation

• URL: http://arxiv.org/abs/2405.14010v1
• Date: Wed, 22 May 2024 21:37:08 GMT
• Title: One-shot Training for Video Object Segmentation
• Authors: Baiyu Chen, Sixian Chan, Xiaoqin Zhang,
• Abstract summary: Video Object (VOS) aims to track objects across frames in a video and segment them based on the initial annotated frame of the target objects. Previous VOS works typically rely on fully annotated videos for training. We propose a general one-shot training framework for VOS, requiring only a single labeled frame per training video.
• Score: 11.52321103793505
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video Object Segmentation (VOS) aims to track objects across frames in a video and segment them based on the initial annotated frame of the target objects. Previous VOS works typically rely on fully annotated videos for training. However, acquiring fully annotated training videos for VOS is labor-intensive and time-consuming. Meanwhile, self-supervised VOS methods have attempted to build VOS systems through correspondence learning and label propagation. Still, the absence of mask priors harms their robustness to complex scenarios, and the label propagation paradigm makes them impractical in terms of efficiency. To address these issues, we propose, for the first time, a general one-shot training framework for VOS, requiring only a single labeled frame per training video and applicable to a majority of state-of-the-art VOS networks. Specifically, our algorithm consists of: i) Inferring object masks time-forward based on the initial labeled frame. ii) Reconstructing the initial object mask time-backward using the masks from step i). Through this bi-directional training, a satisfactory VOS network can be obtained. Notably, our approach is extremely simple and can be employed end-to-end. Finally, our approach uses a single labeled frame of YouTube-VOS and DAVIS datasets to achieve comparable results to those trained on fully labeled datasets. The code will be released.

Related papers

• Learning the What and How of Annotation in Video Object Segmentation [11.012995995497029]
  Video Object (VOS) is crucial for several applications, from video editing to video data generation. Traditional way of annotating objects requires humans to draw detailed segmentation masks on the target objects at each video frame. We propose EVA-VOS, a human-in-the-loop annotation framework for video object segmentation.
  arXiv  Detail & Related papers  (2023-11-08T00:56:31Z)
• Learning Referring Video Object Segmentation from Weak Annotation [78.45828085350936]
  Referring video object segmentation (RVOS) is a task that aims to segment the target object in all video frames based on a sentence describing the object. We propose a new annotation scheme that reduces the annotation effort by 8 times, while providing sufficient supervision for RVOS. Our scheme only requires a mask for the frame where the object first appears and bounding boxes for the rest of the frames.
  arXiv  Detail & Related papers  (2023-08-04T06:50:52Z)
• Boosting Video Object Segmentation via Space-time Correspondence Learning [48.8275459383339]
  Current solutions for video object segmentation (VOS) typically follow a matching-based regime. We devise a correspondence-aware training framework, which boosts matching-based VOS solutions by explicitly encouraging robust correspondence matching. Our algorithm provides solid performance gains on four widely used benchmarks.
  arXiv  Detail & Related papers  (2023-04-13T01:34:44Z)
• Two-shot Video Object Segmentation [35.48207692959968]
  We train a video object segmentation model on sparsely annotated videos. We generate pseudo labels for unlabeled frames and optimize the model on the combination of labeled and pseudo-labeled data. For the first time, we present a general way to train VOS models on two-shot VOS datasets.
  arXiv  Detail & Related papers  (2023-03-21T17:59:56Z)
• FlowVOS: Weakly-Supervised Visual Warping for Detail-Preserving and Temporally Consistent Single-Shot Video Object Segmentation [4.3171602814387136]
  We introduce a new foreground-targeted visual warping approach that learns flow fields from VOS data. We train a flow module to capture detailed motion between frames using two weakly-supervised losses. Our approach produces segmentations with high detail and temporal consistency.
  arXiv  Detail & Related papers  (2021-11-20T16:17:10Z)
• Generating Masks from Boxes by Mining Spatio-Temporal Consistencies in Videos [159.02703673838639]
  We introduce a method for generating segmentation masks from per-frame bounding box annotations in videos. We use our resulting accurate masks for weakly supervised training of video object segmentation (VOS) networks. The additional data provides substantially better generalization performance leading to state-of-the-art results in both the VOS and more challenging tracking domain.
  arXiv  Detail & Related papers  (2021-01-06T18:56:24Z)
• Reducing the Annotation Effort for Video Object Segmentation Datasets [50.893073670389164]
  densely labeling every frame with pixel masks does not scale to large datasets. We use a deep convolutional network to automatically create pseudo-labels on a pixel level from much cheaper bounding box annotations. We obtain the new TAO-VOS benchmark, which we make publicly available at www.vision.rwth-aachen.de/page/taovos.
  arXiv  Detail & Related papers  (2020-11-02T17:34:45Z)
• Learning Video Object Segmentation from Unlabeled Videos [158.18207922363783]
  We propose a new method for video object segmentation (VOS) that addresses object pattern learning from unlabeled videos. We introduce a unified unsupervised/weakly supervised learning framework, called MuG, that comprehensively captures properties of VOS at multiple granularities.
  arXiv  Detail & Related papers  (2020-03-10T22:12:15Z)
• Learning Fast and Robust Target Models for Video Object Segmentation [83.3382606349118]
  Video object segmentation (VOS) is a highly challenging problem since the initial mask, defining the target object, is only given at test-time. Most previous approaches fine-tune segmentation networks on the first frame, resulting in impractical frame-rates and risk of overfitting. We propose a novel VOS architecture consisting of two network components.
  arXiv  Detail & Related papers  (2020-02-27T21:58:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.