An End-to-End Trainable Video Panoptic Segmentation Method usingTransformers

• URL: http://arxiv.org/abs/2110.04009v1
• Date: Fri, 8 Oct 2021 10:13:37 GMT
• Title: An End-to-End Trainable Video Panoptic Segmentation Method usingTransformers
• Authors: Jeongwon Ryu, Kwangjin Yoon
• Abstract summary: We present an algorithm to tackle a video panoptic segmentation problem, a newly emerging area of research. Our proposed video panoptic segmentation algorithm uses the transformer and it can be trained in end-to-end with an input of multiple video frames. The method archived 57.81% on the KITTI-STEP dataset and 31.8% on the MOTChallenge-STEP dataset.
• Score: 0.11714813224840924
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we present an algorithm to tackle a video panoptic segmentation problem, a newly emerging area of research. The video panoptic segmentation is a task that unifies the typical task of panoptic segmentation and multi-object tracking. In other words, it requires generating the instance tracking IDs along with panoptic segmentation results across video sequences. Our proposed video panoptic segmentation algorithm uses the transformer and it can be trained in end-to-end with an input of multiple video frames. We test our method on the STEP dataset and report its performance with recently proposed STQ metric. The method archived 57.81\% on the KITTI-STEP dataset and 31.8\% on the MOTChallenge-STEP dataset.

Related papers

• 3rd Place Solution for PVUW Challenge 2023: Video Panoptic Segmentation [10.04177400017471]
  We propose a robust integrated video panoptic segmentation solution. In our solution, we represent both semantic and instance targets as a set of queries. We then combine these queries with video features extracted by neural networks to predict segmentation masks.
  arXiv  Detail & Related papers  (2023-06-11T19:44:40Z)
• Video Segmentation Learning Using Cascade Residual Convolutional Neural Network [0.0]
  We propose a novel deep learning video segmentation approach that incorporates residual information into the foreground detection learning process. Experiments conducted on the Change Detection 2014 and on the private dataset PetrobrasROUTES from Petrobras support the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2022-12-20T16:56:54Z)
• Tag-Based Attention Guided Bottom-Up Approach for Video Instance Segmentation [83.13610762450703]
  Video instance is a fundamental computer vision task that deals with segmenting and tracking object instances across a video sequence. We introduce a simple end-to-end train bottomable-up approach to achieve instance mask predictions at the pixel-level granularity, instead of the typical region-proposals-based approach. Our method provides competitive results on YouTube-VIS and DAVIS-19 datasets, and has minimum run-time compared to other contemporary state-of-the-art performance methods.
  arXiv  Detail & Related papers  (2022-04-22T15:32:46Z)
• Improving Video Instance Segmentation via Temporal Pyramid Routing [61.10753640148878]
  Video Instance (VIS) is a new and inherently multi-task problem, which aims to detect, segment and track each instance in a video sequence. We propose a Temporal Pyramid Routing (TPR) strategy to conditionally align and conduct pixel-level aggregation from a feature pyramid pair of two adjacent frames. Our approach is a plug-and-play module and can be easily applied to existing instance segmentation methods.
  arXiv  Detail & Related papers  (2021-07-28T03:57:12Z)
• Merging Tasks for Video Panoptic Segmentation [0.0]
  Video panoptic segmentation (VPS) is a recently introduced computer vision task that requires classifying and tracking every pixel in a given video. To understand video panoptic segmentation, first, earlier introduced constituent tasks that focus on semantics and tracking separately will be researched. Two data-driven approaches which do not require training on a tailored dataset will be selected to solve it.
  arXiv  Detail & Related papers  (2021-07-10T08:46:42Z)
• A Survey on Deep Learning Technique for Video Segmentation [147.0767454918527]
  Video segmentation plays a critical role in a broad range of practical applications. Deep learning based approaches have been dedicated to video segmentation and delivered compelling performance.
  arXiv  Detail & Related papers  (2021-07-02T15:51:07Z)
• STEP: Segmenting and Tracking Every Pixel [107.23184053133636]
  We present a new benchmark: Segmenting and Tracking Every Pixel (STEP) Our work is the first that targets this task in a real-world setting that requires dense interpretation in both spatial and temporal domains. For measuring the performance, we propose a novel evaluation metric and Tracking Quality (STQ)
  arXiv  Detail & Related papers  (2021-02-23T18:43:02Z)
• End-to-End Video Instance Segmentation with Transformers [84.17794705045333]
  Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy.
  arXiv  Detail & Related papers  (2020-11-30T02:03:50Z)
• Video Panoptic Segmentation [117.08520543864054]
  We propose and explore a new video extension of this task, called video panoptic segmentation. To invigorate research on this new task, we present two types of video panoptic datasets. We propose a novel video panoptic segmentation network (VPSNet) which jointly predicts object classes, bounding boxes, masks, instance id tracking, and semantic segmentation in video frames.
  arXiv  Detail & Related papers  (2020-06-19T19:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.