Learning Spatial-Semantic Features for Robust Video Object Segmentation

• URL: http://arxiv.org/abs/2407.07760v1
• Date: Wed, 10 Jul 2024 15:36:00 GMT
• Title: Learning Spatial-Semantic Features for Robust Video Object Segmentation
• Authors: Xin Li, Deshui Miao, Zhenyu He, Yaowei Wang, Huchuan Lu, Ming-Hsuan Yang,
• Abstract summary: We propose a robust video object segmentation framework equipped with spatial-semantic features and discriminative object queries. We show that the proposed method set a new state-of-the-art performance on multiple datasets.
• Score: 108.045326229865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tracking and segmenting multiple similar objects with complex or separate parts in long-term videos is inherently challenging due to the ambiguity of target parts and identity confusion caused by occlusion, background clutter, and long-term variations. In this paper, we propose a robust video object segmentation framework equipped with spatial-semantic features and discriminative object queries to address the above issues. Specifically, we construct a spatial-semantic network comprising a semantic embedding block and spatial dependencies modeling block to associate the pretrained ViT features with global semantic features and local spatial features, providing a comprehensive target representation. In addition, we develop a masked cross-attention module to generate object queries that focus on the most discriminative parts of target objects during query propagation, alleviating noise accumulation and ensuring effective long-term query propagation. The experimental results show that the proposed method set a new state-of-the-art performance on multiple datasets, including the DAVIS2017 test (89.1%), YoutubeVOS 2019 (88.5%), MOSE (75.1%), LVOS test (73.0%), and LVOS val (75.1%), which demonstrate the effectiveness and generalization capacity of the proposed method. We will make all source code and trained models publicly available.

Related papers

• Discriminative Spatial-Semantic VOS Solution: 1st Place Solution for 6th LSVOS [68.47681139026666]
  Video object segmentation (VOS) is a crucial task in computer vision. Current VOS methods struggle with complex scenes and prolonged object motions. This report introduces a discriminative spatial-temporal VOS model.
  arXiv  Detail & Related papers  (2024-08-29T10:47:17Z)
• 1st Place Solution for MOSE Track in CVPR 2024 PVUW Workshop: Complex Video Object Segmentation [72.54357831350762]
  We propose a semantic embedding video object segmentation model and use the salient features of objects as query representations. We trained our model on a large-scale video object segmentation dataset. Our model achieves first place (textbf84.45%) in the test set of Complex Video Object Challenge.
  arXiv  Detail & Related papers  (2024-06-07T03:13:46Z)
• Spatial-Temporal Multi-level Association for Video Object Segmentation [89.32226483171047]
  This paper proposes spatial-temporal multi-level association, which jointly associates reference frame, test frame, and object features. Specifically, we construct a spatial-temporal multi-level feature association module to learn better target-aware features.
  arXiv  Detail & Related papers  (2024-04-09T12:44:34Z)
• Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation [52.11279360934703]
  Current prevailing Video Object (VOS) methods usually perform dense matching between the current and reference frames after extracting features. We propose a unified VOS framework, coined as JointFormer, for joint modeling of the three elements of feature, correspondence, and a compressed memory.
  arXiv  Detail & Related papers  (2023-08-25T17:30:08Z)
• Tackling Background Distraction in Video Object Segmentation [7.187425003801958]
  A video object segmentation (VOS) aims to densely track certain objects in videos. One of the main challenges in this task is the existence of background distractors that appear similar to the target objects. We propose three novel strategies to suppress such distractors. Our model achieves a comparable performance to contemporary state-of-the-art approaches, even with real-time performance.
  arXiv  Detail & Related papers  (2022-07-14T14:25:19Z)
• Addressing Multiple Salient Object Detection via Dual-Space Long-Range Dependencies [3.8824028205733017]
  Salient object detection plays an important role in many downstream tasks. We propose a network architecture incorporating non-local feature information in both the spatial and channel spaces. We show that our approach accurately locates multiple salient regions even in complex scenarios.
  arXiv  Detail & Related papers  (2021-11-04T23:16:53Z)
• Sparse Spatial Attention Network for Semantic Segmentation [11.746833714322156]
  The spatial attention mechanism captures long-range dependencies by aggregating global contextual information to each query location. We present a sparse spatial attention network (SSANet) to improve the efficiency of the spatial attention mechanism without sacrificing the performance.
  arXiv  Detail & Related papers  (2021-09-04T18:41:05Z)
• Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation [79.6596425920849]
  This paper addresses the task of unsupervised video multi-object segmentation. We introduce a novel approach for more accurate and efficient unseen-temporal segmentation. We evaluate the proposed approach on DAVIS$_17$ and YouTube-VIS, and the results demonstrate that it outperforms state-of-the-art methods both in segmentation accuracy and inference speed.
  arXiv  Detail & Related papers  (2021-04-10T14:39:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.