PM-VIS+: High-Performance Video Instance Segmentation without Video Annotation

• URL: http://arxiv.org/abs/2406.19665v1
• Date: Fri, 28 Jun 2024 05:22:39 GMT
• Title: PM-VIS+: High-Performance Video Instance Segmentation without Video Annotation
• Authors: Zhangjing Yang, Dun Liu, Xin Wang, Zhe Li, Barathwaj Anandan, Yi Wu,
• Abstract summary: Video instance segmentation requires detecting, segmenting, and tracking objects in videos. This paper introduces a method that eliminates video annotations by utilizing image datasets.
• Score: 15.9587266448337
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Video instance segmentation requires detecting, segmenting, and tracking objects in videos, typically relying on costly video annotations. This paper introduces a method that eliminates video annotations by utilizing image datasets. The PM-VIS algorithm is adapted to handle both bounding box and instance-level pixel annotations dynamically. We introduce ImageNet-bbox to supplement missing categories in video datasets and propose the PM-VIS+ algorithm to adjust supervision based on annotation types. To enhance accuracy, we use pseudo masks and semi-supervised optimization techniques on unannotated video data. This method achieves high video instance segmentation performance without manual video annotations, offering a cost-effective solution and new perspectives for video instance segmentation applications. The code will be available in https://github.com/ldknight/PM-VIS-plus

Related papers

• Rethinking Video Segmentation with Masked Video Consistency: Did the Model Learn as Intended? [22.191260650245443]
  Video segmentation aims at partitioning video sequences into meaningful segments based on objects or regions of interest within frames. Current video segmentation models are often derived from image segmentation techniques, which struggle to cope with small-scale or class-imbalanced video datasets. We propose a training strategy Masked Video Consistency, which enhances spatial and temporal feature aggregation.
  arXiv  Detail & Related papers  (2024-08-20T08:08:32Z)
• Point-VOS: Pointing Up Video Object Segmentation [16.359861197595986]
  Current state-of-the-art Video Object (VOS) methods rely on dense per-object mask annotations both during training and testing. We propose a novel Point-VOS task with a sparse-temporally point-wise annotation scheme that substantially reduces the effort. We show that our data can be used to improve models that connect vision and language, by evaluating it on the Video Narrative Grounding (VNG) task.
  arXiv  Detail & Related papers  (2024-02-08T18:52:23Z)
• Learning the What and How of Annotation in Video Object Segmentation [11.012995995497029]
  Video Object (VOS) is crucial for several applications, from video editing to video data generation. Traditional way of annotating objects requires humans to draw detailed segmentation masks on the target objects at each video frame. We propose EVA-VOS, a human-in-the-loop annotation framework for video object segmentation.
  arXiv  Detail & Related papers  (2023-11-08T00:56:31Z)
• Tracking Anything with Decoupled Video Segmentation [87.07258378407289]
  We develop a decoupled video segmentation approach (DEVA) It is composed of task-specific image-level segmentation and class/task-agnostic bi-directional temporal propagation. We show that this decoupled formulation compares favorably to end-to-end approaches in several data-scarce tasks.
  arXiv  Detail & Related papers  (2023-09-07T17:59:41Z)
• Learning Referring Video Object Segmentation from Weak Annotation [78.45828085350936]
  Referring video object segmentation (RVOS) is a task that aims to segment the target object in all video frames based on a sentence describing the object. We propose a new annotation scheme that reduces the annotation effort by 8 times, while providing sufficient supervision for RVOS. Our scheme only requires a mask for the frame where the object first appears and bounding boxes for the rest of the frames.
  arXiv  Detail & Related papers  (2023-08-04T06:50:52Z)
• Robust Online Video Instance Segmentation with Track Queries [15.834703258232002]
  We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos.
  arXiv  Detail & Related papers  (2022-11-16T18:50:14Z)
• Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion [92.80981308407098]
  We propose an approach that combines the strengths of motion-based and appearance-based segmentation. We propose to supervise an image segmentation network, tasking it with predicting regions that are likely to contain simple motion patterns. In the unsupervised video segmentation mode, the network is trained on a collection of unlabelled videos, using the learning process itself as an algorithm to segment these videos.
  arXiv  Detail & Related papers  (2022-05-16T17:55:34Z)
• HODOR: High-level Object Descriptors for Object Re-segmentation in Video Learned from Static Images [123.65233334380251]
  We propose HODOR: a novel method that effectively leveraging annotated static images for understanding object appearance and scene context. As a result, HODOR achieves state-of-the-art performance on the DAVIS and YouTube-VOS benchmarks. Without any architectural modification, HODOR can also learn from video context around single annotated video frames.
  arXiv  Detail & Related papers  (2021-12-16T18:59:53Z)
• Generating Masks from Boxes by Mining Spatio-Temporal Consistencies in Videos [159.02703673838639]
  We introduce a method for generating segmentation masks from per-frame bounding box annotations in videos. We use our resulting accurate masks for weakly supervised training of video object segmentation (VOS) networks. The additional data provides substantially better generalization performance leading to state-of-the-art results in both the VOS and more challenging tracking domain.
  arXiv  Detail & Related papers  (2021-01-06T18:56:24Z)
• Video Panoptic Segmentation [117.08520543864054]
  We propose and explore a new video extension of this task, called video panoptic segmentation. To invigorate research on this new task, we present two types of video panoptic datasets. We propose a novel video panoptic segmentation network (VPSNet) which jointly predicts object classes, bounding boxes, masks, instance id tracking, and semantic segmentation in video frames.
  arXiv  Detail & Related papers  (2020-06-19T19:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.