Discriminative Spatial-Semantic VOS Solution: 1st Place Solution for 6th LSVOS

• URL: http://arxiv.org/abs/2408.16431v1
• Date: Thu, 29 Aug 2024 10:47:17 GMT
• Title: Discriminative Spatial-Semantic VOS Solution: 1st Place Solution for 6th LSVOS
• Authors: Deshui Miao, Yameng Gu, Xin Li, Zhenyu He, Yaowei Wang, Ming-Hsuan Yang,
• Abstract summary: Video object segmentation (VOS) is a crucial task in computer vision. Current VOS methods struggle with complex scenes and prolonged object motions. This report introduces a discriminative spatial-temporal VOS model.
• Score: 68.47681139026666
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Video object segmentation (VOS) is a crucial task in computer vision, but current VOS methods struggle with complex scenes and prolonged object motions. To address these challenges, the MOSE dataset aims to enhance object recognition and differentiation in complex environments, while the LVOS dataset focuses on segmenting objects exhibiting long-term, intricate movements. This report introduces a discriminative spatial-temporal VOS model that utilizes discriminative object features as query representations. The semantic understanding of spatial-semantic modules enables it to recognize object parts, while salient features highlight more distinctive object characteristics. Our model, trained on extensive VOS datasets, achieved first place (\textbf{80.90\%} $\mathcal{J \& F}$) on the test set of the 6th LSVOS challenge in the VOS Track, demonstrating its effectiveness in tackling the aforementioned challenges. The code will be available at \href{https://github.com/yahooo-m/VOS-Solution}{code}.

Related papers

• Learning Spatial-Semantic Features for Robust Video Object Segmentation [108.045326229865]
  We propose a robust video object segmentation framework equipped with spatial-semantic features and discriminative object queries. We show that the proposed method set a new state-of-the-art performance on multiple datasets.
  arXiv  Detail & Related papers  (2024-07-10T15:36:00Z)
• Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs [38.02017186215372]
  Integration of Large Language Models (LLMs) into visual domain tasks, resulting in visual-LLMs (V-LLMs), has enabled exceptional performance in vision-language tasks. However, existing V-LLMs demonstrate weak spatial reasoning and localization awareness. We explore how image-space coordinate based instruction fine-tuning objectives could inject spatial awareness into V-LLMs.
  arXiv  Detail & Related papers  (2024-04-11T03:09:34Z)
• Detect2Interact: Localizing Object Key Field in Visual Question Answering (VQA) with LLMs [5.891295920078768]
  We introduce an advanced approach for fine-grained object visual key field detection. First, we use the segment anything model (SAM) to generate detailed spatial maps of objects in images. Next, we use Vision Studio to extract semantic object descriptions. Third, we employ GPT-4's common sense knowledge, bridging the gap between an object's semantics and its spatial map.
  arXiv  Detail & Related papers  (2024-04-01T14:53:36Z)
• DOZE: A Dataset for Open-Vocabulary Zero-Shot Object Navigation in Dynamic Environments [28.23284296418962]
  Zero-Shot Object Navigation (ZSON) requires agents to autonomously locate and approach unseen objects in unfamiliar environments. Existing datasets for developing ZSON algorithms lack consideration of dynamic obstacles, object diversity, and scene texts. We propose a dataset for Open-Vocabulary Zero-Shot Object Navigation in Dynamic Environments (DOZE) DOZE comprises ten high-fidelity 3D scenes with over 18k tasks, aiming to mimic complex, dynamic real-world scenarios.
  arXiv  Detail & Related papers  (2024-02-29T10:03:57Z)
• MOSE: A New Dataset for Video Object Segmentation in Complex Scenes [106.64327718262764]
  Video object segmentation (VOS) aims at segmenting a particular object throughout the entire video clip sequence. The state-of-the-art VOS methods have achieved excellent performance (e.g., 90+% J&F) on existing datasets. We collect a new VOS dataset called coMplex video Object SEgmentation (MOSE) to study the tracking and segmenting objects in complex environments.
  arXiv  Detail & Related papers  (2023-02-03T17:20:03Z)
• DQnet: Cross-Model Detail Querying for Camouflaged Object Detection [54.82390534024954]
  A convolutional neural network (CNN) for camouflaged object detection tends to activate local discriminative regions while ignoring complete object extent. In this paper, we argue that partial activation is caused by the intrinsic characteristics of CNN. In order to obtain feature maps that could activate full object extent, a novel framework termed Cross-Model Detail Querying network (DQnet) is proposed.
  arXiv  Detail & Related papers  (2022-12-16T06:23:58Z)
• Scalable Video Object Segmentation with Identification Mechanism [125.4229430216776]
  This paper explores the challenges of achieving scalable and effective multi-object modeling for semi-supervised Video Object (VOS) We present two innovative approaches, Associating Objects with Transformers (AOT) and Associating Objects with Scalable Transformers (AOST) Our approaches surpass the state-of-the-art competitors and display exceptional efficiency and scalability consistently across all six benchmarks.
  arXiv  Detail & Related papers  (2022-03-22T03:33:27Z)
• Learning Video Object Segmentation from Unlabeled Videos [158.18207922363783]
  We propose a new method for video object segmentation (VOS) that addresses object pattern learning from unlabeled videos. We introduce a unified unsupervised/weakly supervised learning framework, called MuG, that comprehensively captures properties of VOS at multiple granularities.
  arXiv  Detail & Related papers  (2020-03-10T22:12:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.