Robust Online Video Instance Segmentation with Track Queries

• URL: http://arxiv.org/abs/2211.09108v1
• Date: Wed, 16 Nov 2022 18:50:14 GMT
• Title: Robust Online Video Instance Segmentation with Track Queries
• Authors: Zitong Zhan, Daniel McKee, Svetlana Lazebnik
• Abstract summary: We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos.
• Score: 15.834703258232002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, transformer-based methods have achieved impressive results on Video Instance Segmentation (VIS). However, most of these top-performing methods run in an offline manner by processing the entire video clip at once to predict instance mask volumes. This makes them incapable of handling the long videos that appear in challenging new video instance segmentation datasets like UVO and OVIS. We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark and considerably outperforms them on UVO and OVIS. This method, called Robust Online Video Segmentation (ROVIS), augments the Mask2Former image instance segmentation model with track queries, a lightweight mechanism for carrying track information from frame to frame, originally introduced by the TrackFormer method for multi-object tracking. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos.

Related papers

• NOVIS: A Case for End-to-End Near-Online Video Instance Segmentation [22.200700685751826]
  Video Instance (VIS) community operated under the common belief that offline methods are generally superior to a frame by frame online processing. We present a detailed analysis on different processing paradigms and a new end-to-end Video Instance method. Our NOVIS represents the first near-online VIS approach which avoids any handcrafted trackings.
  arXiv  Detail & Related papers  (2023-08-29T12:51:04Z)
• VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation [87.13210748484217]
  VideoCutLER is a simple method for unsupervised multi-instance video segmentation without using motion-based learning signals like optical flow or training on natural videos. We show the first competitive unsupervised learning results on the challenging YouTubeVIS 2019 benchmark, achieving 50.7% APvideo50. VideoCutLER can also serve as a strong pretrained model for supervised video instance segmentation tasks, exceeding DINO by 15.9% on YouTubeVIS 2019 in terms of APvideo.
  arXiv  Detail & Related papers  (2023-08-28T17:10:12Z)
• Two-Level Temporal Relation Model for Online Video Instance Segmentation [3.9349485816629888]
  We propose an online method that is on par with the performance of the offline counterparts. We introduce a message-passing graph neural network that encodes objects and relates them through time. Our model achieves trained end-to-end, state-of-the-art performance on the YouTube-VIS dataset.
  arXiv  Detail & Related papers  (2022-10-30T10:01:01Z)
• DeVIS: Making Deformable Transformers Work for Video Instance Segmentation [4.3012765978447565]
  Video Instance (VIS) jointly tackles multi-object detection, tracking, and segmentation in video sequences. Transformers recently allowed to cast the entire VIS task as a single set-prediction problem. Deformable attention provides a more efficient alternative but its application to the temporal domain or the segmentation task have not yet been explored.
  arXiv  Detail & Related papers  (2022-07-22T14:27:45Z)
• SiamMask: A Framework for Fast Online Object Tracking and Segmentation [96.61632757952292]
  SiamMask is a framework to perform both visual object tracking and video object segmentation, in real-time, with the same simple method. We show that it is possible to extend the framework to handle multiple object tracking and segmentation by simply re-using the multi-task model. It yields real-time state-of-the-art results on visual-object tracking benchmarks, while at the same time demonstrating competitive performance at a high speed for video object segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-07-05T14:47:17Z)
• Efficient Video Instance Segmentation via Tracklet Query and Proposal [62.897552852894854]
  Video Instance aims to simultaneously classify, segment, and track multiple object instances in videos. Most clip-level methods are neither end-to-end learnable nor real-time. This paper proposes EfficientVIS, a fully end-to-end framework with efficient training and inference.
  arXiv  Detail & Related papers  (2022-03-03T17:00:11Z)
• Improving Video Instance Segmentation via Temporal Pyramid Routing [61.10753640148878]
  Video Instance (VIS) is a new and inherently multi-task problem, which aims to detect, segment and track each instance in a video sequence. We propose a Temporal Pyramid Routing (TPR) strategy to conditionally align and conduct pixel-level aggregation from a feature pyramid pair of two adjacent frames. Our approach is a plug-and-play module and can be easily applied to existing instance segmentation methods.
  arXiv  Detail & Related papers  (2021-07-28T03:57:12Z)
• End-to-End Video Instance Segmentation with Transformers [84.17794705045333]
  Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy.
  arXiv  Detail & Related papers  (2020-11-30T02:03:50Z)
• Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching [67.02962970820505]
  We introduce "tracking-by-detection" into Video Object (VOS) We propose a new temporal aggregation network and a novel dynamic time-evolving template matching mechanism to achieve significantly improved performance. We achieve new state-of-the-art performance on the DAVIS benchmark without complicated bells and whistles in both speed and accuracy, with a speed of 0.14 second per frame and J&F measure of 75.9% respectively.
  arXiv  Detail & Related papers  (2020-07-11T05:44:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.