Learning Video Instance Segmentation with Recurrent Graph Neural Networks

• URL: http://arxiv.org/abs/2012.03911v1
• Date: Mon, 7 Dec 2020 18:41:35 GMT
• Title: Learning Video Instance Segmentation with Recurrent Graph Neural Networks
• Authors: Joakim Johnander, Emil Brissman, Martin Danelljan, Michael Felsberg
• Abstract summary: We propose a novel learning formulation, where the entire video instance segmentation problem is modelled jointly. We fit a flexible model to our formulation that, with the help of a graph neural network, processes all available new information in each frame. Our approach, operating at over 25 FPS, outperforms previous video real-time methods.
• Score: 39.06202374530647
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Most existing approaches to video instance segmentation comprise multiple modules that are heuristically combined to produce the final output. Formulating a purely learning-based method instead, which models both the temporal aspect as well as a generic track management required to solve the video instance segmentation task, is a highly challenging problem. In this work, we propose a novel learning formulation, where the entire video instance segmentation problem is modelled jointly. We fit a flexible model to our formulation that, with the help of a graph neural network, processes all available new information in each frame. Past information is considered and processed via a recurrent connection. We demonstrate the effectiveness of the proposed approach in comprehensive experiments. Our approach, operating at over 25 FPS, outperforms previous video real-time methods. We further conduct detailed ablative experiments that validate the different aspects of our approach.

Related papers

• Rethinking Video Segmentation with Masked Video Consistency: Did the Model Learn as Intended? [22.191260650245443]
  Video segmentation aims at partitioning video sequences into meaningful segments based on objects or regions of interest within frames. Current video segmentation models are often derived from image segmentation techniques, which struggle to cope with small-scale or class-imbalanced video datasets. We propose a training strategy Masked Video Consistency, which enhances spatial and temporal feature aggregation.
  arXiv  Detail & Related papers  (2024-08-20T08:08:32Z)
• Appearance-Based Refinement for Object-Centric Motion Segmentation [85.2426540999329]
  We introduce an appearance-based refinement method that leverages temporal consistency in video streams to correct inaccurate flow-based proposals. Our approach involves a sequence-level selection mechanism that identifies accurate flow-predicted masks as exemplars. Its performance is evaluated on multiple video segmentation benchmarks, including DAVIS, YouTube, SegTrackv2, and FBMS-59.
  arXiv  Detail & Related papers  (2023-12-18T18:59:51Z)
• Learning from One Continuous Video Stream [70.30084026960819]
  We introduce a framework for online learning from a single continuous video stream. This poses great challenges given the high correlation between consecutive video frames. We employ pixel-to-pixel modelling as a practical and flexible way to switch between pre-training and single-stream evaluation.
  arXiv  Detail & Related papers  (2023-12-01T14:03:30Z)
• Learning to Associate Every Segment for Video Panoptic Segmentation [123.03617367709303]
  We learn coarse segment-level matching and fine pixel-level matching together. We show that our per-frame computation model can achieve new state-of-the-art results on Cityscapes-VPS and VIPER datasets.
  arXiv  Detail & Related papers  (2021-06-17T13:06:24Z)
• PGT: A Progressive Method for Training Models on Long Videos [45.935259079953255]
  Main-stream method is to split a raw video into clips, leading to incomplete temporal information flow. Inspired by natural language processing techniques dealing with long sentences, we propose to treat videos as serial fragments satisfying Markov property. We empirically demonstrate that it yields significant performance improvements on different models and datasets.
  arXiv  Detail & Related papers  (2021-03-21T06:15:20Z)
• Unsupervised Learning Consensus Model for Dynamic Texture Videos Segmentation [12.462608802359936]
  We present an effective unsupervised learning consensus model for the segmentation of dynamic texture (ULCM) In the proposed model, the set of values of the requantized local binary patterns (LBP) histogram around the pixel to be classified are used as features. Experiments conducted on the challenging SynthDB dataset show that ULCM is significantly faster, easier to code, simple and has limited parameters.
  arXiv  Detail & Related papers  (2020-06-29T16:40:59Z)
• Scene-Adaptive Video Frame Interpolation via Meta-Learning [54.87696619177496]
  We propose to adapt the model to each video by making use of additional information that is readily available at test time. We obtain significant performance gains with only a single gradient update without any additional parameters.
  arXiv  Detail & Related papers  (2020-04-02T02:46:44Z)
• Non-Adversarial Video Synthesis with Learned Priors [53.26777815740381]
  We focus on the problem of generating videos from latent noise vectors, without any reference input frames. We develop a novel approach that jointly optimize the input latent space, the weights of a recurrent neural network and a generator through non-adversarial learning. Our approach generates superior quality videos compared to the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-21T02:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.