SDI-Paste: Synthetic Dynamic Instance Copy-Paste for Video Instance Segmentation

• URL: http://arxiv.org/abs/2410.13565v1
• Date: Wed, 16 Oct 2024 12:11:34 GMT
• Title: SDI-Paste: Synthetic Dynamic Instance Copy-Paste for Video Instance Segmentation
• Authors: Sahir Shrestha, Weihao Li, Gao Zhu, Nick Barnes,
• Abstract summary: We leverage the recent growth in video fidelity of generative models to explore effective ways of incorporating synthetically generated objects into existing video datasets to artificially expand object instance pools. We name our video data augmentation pipeline Synthetic Dynamic Instance Copy-Paste, and test it on the complex task of Video Instance detection, segmentation and tracking of object instances across a video sequence.
• Score: 26.258313321256097
• License:
• Abstract: Data augmentation methods such as Copy-Paste have been studied as effective ways to expand training datasets while incurring minimal costs. While such methods have been extensively implemented for image level tasks, we found no scalable implementation of Copy-Paste built specifically for video tasks. In this paper, we leverage the recent growth in video fidelity of generative models to explore effective ways of incorporating synthetically generated objects into existing video datasets to artificially expand object instance pools. We first procure synthetic video sequences featuring objects that morph dynamically with time. Our carefully devised pipeline automatically segments then copy-pastes these dynamic instances across the frames of any target background video sequence. We name our video data augmentation pipeline Synthetic Dynamic Instance Copy-Paste, and test it on the complex task of Video Instance Segmentation which combines detection, segmentation and tracking of object instances across a video sequence. Extensive experiments on the popular Youtube-VIS 2021 dataset using two separate popular networks as baselines achieve strong gains of +2.9 AP (6.5%) and +2.1 AP (4.9%). We make our code and models publicly available.

Related papers

• X-Paste: Revisiting Scalable Copy-Paste for Instance Segmentation using CLIP and StableDiffusion [137.84635386962395]
  Copy-Paste is a simple and effective data augmentation strategy for instance segmentation. We revisit Copy-Paste at scale with the power of newly emerged zero-shot recognition models. X-Paste provides impressive improvements over the strong baseline CenterNet2 with Swin-L as the backbone.
  arXiv  Detail & Related papers  (2022-12-07T18:59:59Z)
• Robust Online Video Instance Segmentation with Track Queries [15.834703258232002]
  We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos.
  arXiv  Detail & Related papers  (2022-11-16T18:50:14Z)
• Segmenting Moving Objects via an Object-Centric Layered Representation [100.26138772664811]
  We introduce an object-centric segmentation model with a depth-ordered layer representation. We introduce a scalable pipeline for generating synthetic training data with multiple objects. We evaluate the model on standard video segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-07-05T17:59:43Z)
• Human Instance Segmentation and Tracking via Data Association and Single-stage Detector [17.46922710432633]
  Human video instance segmentation plays an important role in computer understanding of human activities. Most current VIS methods are based on Mask-RCNN framework. We develop a new method for human video instance segmentation based on single-stage detector.
  arXiv  Detail & Related papers  (2022-03-31T11:36:09Z)
• 1st Place Solution for YouTubeVOS Challenge 2021:Video Instance Segmentation [0.39146761527401414]
  Video Instance (VIS) is a multi-task problem performing detection, segmentation, and tracking simultaneously. We propose two modules, named Temporally Correlated Instance (TCIS) and Bidirectional Tracking (BiTrack) By combining these techniques with a bag of tricks, the network performance is significantly boosted compared to the baseline.
  arXiv  Detail & Related papers  (2021-06-12T00:20:38Z)
• CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation [67.17625278621134]
  Video instance segmentation is a complex task in which we need to detect, segment, and track each object for any given video. Previous approaches only utilize single-frame features for the detection, segmentation, and tracking of objects. We propose a novel comprehensive feature aggregation approach (CompFeat) to refine features at both frame-level and object-level with temporal and spatial context information.
  arXiv  Detail & Related papers  (2020-12-07T00:31:42Z)
• End-to-End Video Instance Segmentation with Transformers [84.17794705045333]
  Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy.
  arXiv  Detail & Related papers  (2020-11-30T02:03:50Z)
• Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching [67.02962970820505]
  We introduce "tracking-by-detection" into Video Object (VOS) We propose a new temporal aggregation network and a novel dynamic time-evolving template matching mechanism to achieve significantly improved performance. We achieve new state-of-the-art performance on the DAVIS benchmark without complicated bells and whistles in both speed and accuracy, with a speed of 0.14 second per frame and J&F measure of 75.9% respectively.
  arXiv  Detail & Related papers  (2020-07-11T05:44:16Z)
• Video Panoptic Segmentation [117.08520543864054]
  We propose and explore a new video extension of this task, called video panoptic segmentation. To invigorate research on this new task, we present two types of video panoptic datasets. We propose a novel video panoptic segmentation network (VPSNet) which jointly predicts object classes, bounding boxes, masks, instance id tracking, and semantic segmentation in video frames.
  arXiv  Detail & Related papers  (2020-06-19T19:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.