Video Object Segmentation in Panoptic Wild Scenes

• URL: http://arxiv.org/abs/2305.04470v2
• Date: Sun, 25 Jun 2023 06:39:30 GMT
• Title: Video Object Segmentation in Panoptic Wild Scenes
• Abstract summary: We introduce semi-supervised video object segmentation (VOS) to panoptic wild scenes. We present a large-scale benchmark as well as a baseline method for it. Experimental results show that VIPOSeg can not only boost the performance of VOS models by panoptic training but also evaluate them comprehensively in panoptic scenes.
• Score: 31.701108453289162
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce semi-supervised video object segmentation (VOS) to panoptic wild scenes and present a large-scale benchmark as well as a baseline method for it. Previous benchmarks for VOS with sparse annotations are not sufficient to train or evaluate a model that needs to process all possible objects in real-world scenarios. Our new benchmark (VIPOSeg) contains exhaustive object annotations and covers various real-world object categories which are carefully divided into subsets of thing/stuff and seen/unseen classes for comprehensive evaluation. Considering the challenges in panoptic VOS, we propose a strong baseline method named panoptic object association with transformers (PAOT), which uses panoptic identification to associate objects with a pyramid architecture on multiple scales. Experimental results show that VIPOSeg can not only boost the performance of VOS models by panoptic training but also evaluate them comprehensively in panoptic scenes. Previous methods for classic VOS still need to improve in performance and efficiency when dealing with panoptic scenes, while our PAOT achieves SOTA performance with good efficiency on VIPOSeg and previous VOS benchmarks. PAOT also ranks 1st in the VOT2022 challenge. Our dataset is available at https://github.com/yoxu515/VIPOSeg-Benchmark.

Related papers

• Robust Promptable Video Object Segmentation [67.1533741758339]
  This paper offers the first comprehensive study on robust PVOS (RobustPVOS)<n>We first construct a new, comprehensive benchmark with two real-world evaluation datasets of 351 video clips and more than 2,500 object masks.<n>We present a new RobustPVOS method, dubbed Memory-object-conditioned Gated-rank Adaptation (MoGA)
  arXiv  Detail & Related papers  (2026-05-12T11:55:31Z)
• Show Me When and Where: Towards Referring Video Object Segmentation in the Wild [98.87931411432106]
  In this work, we introduce a new setting towards in-the-wild RVOS.<n>Our new benchmark challenges RVOS methods to show not only where but also when objects appear in videos.<n>Our YoURVOS dataset offers an imperative benchmark, which will push forward the advancement of RVOS methods for practical applications.
  arXiv  Detail & Related papers  (2026-03-15T09:30:33Z)
• Object segmentation in the wild with foundation models: application to vision assisted neuro-prostheses for upper limbs [2.7554193753662015]
  We investigate whether foundation models, trained on a large number and variety of objects, can perform object segmentation without fine-tuning on specific images containing everyday objects.<n>We propose a method for generating prompts based on gaze fixations to guide the Segment Anything Model (SAM) in our segmentation scenario.<n> Evaluation results of our approach show an improvement of the IoU segmentation quality metric by up to 0.51 points on real-world challenging data of Grasping-in-the-Wild corpus.
  arXiv  Detail & Related papers  (2025-07-24T15:40:44Z)
• Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples [61.66967790884943]
  Referring video object segmentation (RVOS) relies on sufficient data for a given scene. In more realistic scenarios, only minimal annotations are available for a new scene. We propose a model with a newly designed cross-modal affinity (CMA) module based on a Transformer architecture. CMA module builds multimodal affinity with a few samples, thus quickly learning new semantic information, and enabling the model to adapt to different scenarios.
  arXiv  Detail & Related papers  (2023-09-05T08:34:23Z)
• MOSE: A New Dataset for Video Object Segmentation in Complex Scenes [106.64327718262764]
  Video object segmentation (VOS) aims at segmenting a particular object throughout the entire video clip sequence. The state-of-the-art VOS methods have achieved excellent performance (e.g., 90+% J&F) on existing datasets. We collect a new VOS dataset called coMplex video Object SEgmentation (MOSE) to study the tracking and segmenting objects in complex environments.
  arXiv  Detail & Related papers  (2023-02-03T17:20:03Z)
• Region Aware Video Object Segmentation with Deep Motion Modeling [56.95836951559529]
  Region Aware Video Object (RAVOS) is a method that predicts regions of interest for efficient object segmentation and memory storage. For efficient segmentation, object features are extracted according to the ROIs, and an object decoder is designed for object-level segmentation. For efficient memory storage, we propose motion path memory to filter out redundant context by memorizing the features within the motion path of objects between two frames.
  arXiv  Detail & Related papers  (2022-07-21T01:44:40Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)
• Slot-VPS: Object-centric Representation Learning for Video Panoptic Segmentation [29.454785969084384]
  Video Panoptic (VPS) aims at assigning a class label to each pixel, uniquely segmenting and identifying all object instances consistently across all frames. We present Slot-VPS, the first end-to-end framework for this task. We encode all panoptic entities in a video, including instances and background semantics, with a unified representation called panoptic slots. The coherent-temporal object's information is retrieved and encoded into the panoptic slots by proposed the Video Panoptic Retriever, enabling it to localize, segment, differentiate, and associate objects in a unified manner.
  arXiv  Detail & Related papers  (2021-12-16T15:12:22Z)
• ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation [31.078913193966585]
  We present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. On the individual sub-tasks, ViP-DeepLab achieves state-of-the-art results, outperforming previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian.
  arXiv  Detail & Related papers  (2020-12-09T19:00:35Z)
• Make One-Shot Video Object Segmentation Efficient Again [7.7415390727490445]
  Video object segmentation (VOS) describes the task of segmenting a set of objects in each frame of a video. e-OSVOS decouples the object detection task and predicts only local segmentation masks by applying a modified version of Mask R-CNN. e-OSVOS provides state-of-the-art results on DAVIS 2016, DAVIS 2017, and YouTube-VOS for one-shot fine-tuning methods.
  arXiv  Detail & Related papers  (2020-12-03T12:21:23Z)
• Video Panoptic Segmentation [117.08520543864054]
  We propose and explore a new video extension of this task, called video panoptic segmentation. To invigorate research on this new task, we present two types of video panoptic datasets. We propose a novel video panoptic segmentation network (VPSNet) which jointly predicts object classes, bounding boxes, masks, instance id tracking, and semantic segmentation in video frames.
  arXiv  Detail & Related papers  (2020-06-19T19:35:47Z)
• Learning Video Object Segmentation from Unlabeled Videos [158.18207922363783]
  We propose a new method for video object segmentation (VOS) that addresses object pattern learning from unlabeled videos. We introduce a unified unsupervised/weakly supervised learning framework, called MuG, that comprehensively captures properties of VOS at multiple granularities.
  arXiv  Detail & Related papers  (2020-03-10T22:12:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.