Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos

• URL: http://arxiv.org/abs/2501.04001v2
• Date: Thu, 13 Feb 2025 18:14:33 GMT
• Title: Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos
• Authors: Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, Ming-Hsuan Yang,
• Abstract summary: Sa2VA is a unified model for grounded understanding of both images and videos. It supports a wide range of image and video tasks, including referring segmentation and conversation. We show that Sa2VA achieves state-of-the-art across multiple tasks, particularly in referring video object segmentation.
• Score: 110.3379755761583
• License:
• Abstract: This work presents Sa2VA, the first unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models, which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal one-shot instruction tuning. Sa2VA combines SAM-2, a foundation video segmentation model, with LLaVA, an advanced vision-language model, and unifies text, image, and video into a shared LLM token space. Using the LLM, Sa2VA generates instruction tokens that guide SAM-2 in producing precise masks, enabling a grounded, multi-modal understanding of both static and dynamic visual content. Additionally, we introduce Ref-SAV, an auto-labeled dataset containing over 72k object expressions in complex video scenes, designed to boost model performance. We also manually validate 2k video objects in the Ref-SAV datasets to benchmark referring video object segmentation in complex environments. Experiments show that Sa2VA achieves state-of-the-art across multiple tasks, particularly in referring video object segmentation, highlighting its potential for complex real-world applications.

Related papers

• MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation [21.43947114468122]
  Referring video object segmentation (RVOS) aims to segment objects in a video according to textual descriptions. The Segment Anything Model 2 (SAM 2) has shown great effectiveness across various video segmentation tasks. We propose a novel RVOS framework, termed MPG-SAM 2, to address these challenges.
  arXiv  Detail & Related papers  (2025-01-23T13:53:33Z)
• Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM [28.64108439552772]
  We introduce a large-scale synthetic dataset created from proprietary models. We also explore a dynamic visual token compression architecture that strikes a balance between computational efficiency and performance. Our proposed model achieves state-of-the-art results across various video tasks and shows impressive generalization.
  arXiv  Detail & Related papers  (2024-12-12T18:20:41Z)
• TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models [52.590072198551944]
  Recent advances in multimodal Large Language Models (LLMs) have shown great success in understanding multi-modal contents. For video understanding tasks, training-based video LLMs are difficult to build due to the scarcity of high-quality, curated video-text paired data. In this work, we explore the limitations of the existing compression strategies for building a training-free video LLM.
  arXiv  Detail & Related papers  (2024-11-17T13:08:29Z)
• VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos [58.765796160750504]
  VideoGLaMM is a new model for fine-grained pixel-level grounding in videos based on user-provided textual inputs. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. Experimental results show that our model consistently outperforms existing approaches across all three tasks.
  arXiv  Detail & Related papers  (2024-11-07T17:59:27Z)
• Semantic Alignment for Multimodal Large Language Models [72.10272479476161]
  We introduce Semantic Alignment for Multi-modal large language models (SAM) By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis. By involving the bidirectional semantic guidance between different images in the visual-token extraction process, SAM aims to enhance the preservation of linking information for coherent analysis.
  arXiv  Detail & Related papers  (2024-08-23T06:48:46Z)
• Video Object Segmentation via SAM 2: The 4th Solution for LSVOS Challenge VOS Track [28.52754012142431]
  Segment Anything Model 2 (SAM 2) is a foundation model towards solving promptable visual segmentation in images and videos. SAM 2 builds a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Without fine-tuning on the training set, SAM 2 achieved 75.79 J&F on the test set and ranked 4th place for 6th LSVOS Challenge VOS Track.
  arXiv  Detail & Related papers  (2024-08-19T16:13:14Z)
• Video-LLaVA: Learning United Visual Representation by Alignment Before Projection [27.04277811443469]
  Video-LLaVA learns from a mixed dataset of images and videos, mutually enhancing each other. Video-LLaVA superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4 image benchmark toolkits.
  arXiv  Detail & Related papers  (2023-11-16T10:59:44Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation [124.02278735049235]
  VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. We evaluate various baseline methods with and without large-scale VidL pre-training. The significant gap between our best model and human performance calls for future study for advanced VidL models.
  arXiv  Detail & Related papers  (2021-06-08T18:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.