GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation

• URL: http://arxiv.org/abs/2406.12834v2
• Date: Sun, 23 Jun 2024 19:10:29 GMT
• Title: GroPrompt: Efficient Grounded Prompting and Adaptation for Referring Video Object Segmentation
• Abstract summary: Referring Video Object (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. We propose Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences. With the proposed TAP-CL, our GroPrompt framework can generate temporal-consistent yet text-aware position prompts.
• Score: 41.67544072483324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring Video Object Segmentation (RVOS) aims to segment the object referred to by the query sentence throughout the entire video. Most existing methods require end-to-end training with dense mask annotations, which could be computation-consuming and less scalable. In this work, we aim to efficiently adapt foundation segmentation models for addressing RVOS from weak supervision with the proposed Grounded Prompting (GroPrompt) framework. More specifically, we propose Text-Aware Prompt Contrastive Learning (TAP-CL) to enhance the association between the position prompts and the referring sentences with only box supervisions, including Text-Contrastive Prompt Learning (TextCon) and Modality-Contrastive Prompt Learning (ModalCon) at frame level and video level, respectively. With the proposed TAP-CL, our GroPrompt framework can generate temporal-consistent yet text-aware position prompts describing locations and movements for the referred object from the video. The experimental results in the standard RVOS benchmarks (Ref-YouTube-VOS, Ref-DAVIS17, A2D-Sentences, and JHMDB-Sentences) demonstrate the competitive performance of our proposed GroPrompt framework given only bounding box weak supervisions.

Related papers

• GA2-CLIP: Generic Attribute Anchor for Efficient Prompt Tuningin Video-Language Models [34.002791706686345]
  Visual and textual soft prompt tuning can improve the adaptability of Vision-Language Models (VLMs) in downstream tasks.<n>Existing methods attempt to mitigate this effect by regularizing the gap between hand-crafted prompts and soft prompts.<n>We propose a plug-and-play coupling prompt learning framework to optimize the performance of V-L models in video tasks.
  arXiv  Detail & Related papers  (2025-11-27T05:36:47Z)
• Temporal Prompting Matters: Rethinking Referring Video Object Segmentation [64.82333675385802]
  Referring Video Object (RVOS) aims to segment the object referred to by the query sentence in the video.<n>Most existing methods require end-to-end training with dense mask annotations.<n>We propose a Temporal Prompt Generation and Selection (Tenet) framework to address the referring and video factors.
  arXiv  Detail & Related papers  (2025-10-08T17:59:57Z)
• Unleashing Hierarchical Reasoning: An LLM-Driven Framework for Training-Free Referring Video Object Segmentation [17.238084264485988]
  Referring Video Object (RVOS) aims to segment an object of interest throughout a video based on a language description.<n>bftextPARSE-VOS is a training-free framework powered by Large Language Models (LLMs)<n>bftextPARSE-VOS achieved state-of-the-art performance on three major benchmarks: Ref-YouTube-VOS, Ref-DAVIS17, and MeViS.
  arXiv  Detail & Related papers  (2025-09-06T15:46:23Z)
• Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation [115.74044261016554]
  Video instance segmentation (VIS) has gained significant attention for its capability in tracking and segmenting object instances across video frames.<n>Most of the existing VIS approaches unrealistically assume that the categories of object instances remain fixed over time.<n>We develop a novel Hierarchical Visual Prompt Learning model that overcomes catastrophic forgetting of previous categories from both frame-level and video-level perspectives.
  arXiv  Detail & Related papers  (2025-08-12T03:49:08Z)
• ReferDINO: Referring Video Object Segmentation with Visual Grounding Foundations [33.74746234704817]
  Referring video object segmentation (RVOS) aims to segment target objects throughout a video based on a text description. We present textbfReferDINO, an end-to-end RVOS model that inherits strong vision-language understanding from the pretrained visual grounding foundation models.
  arXiv  Detail & Related papers  (2025-01-24T16:24:15Z)
• SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation [35.063881868130075]
  This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. We propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin.
  arXiv  Detail & Related papers  (2023-05-26T15:13:44Z)
• Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos [57.830865926459914]
  We propose a vision-language learning framework for untrimmed videos, which automatically detects informative events. Instead of coarse-level video-language alignments, we present two dual pretext tasks to encourage fine-grained segment-level alignments. Our framework is easily to tasks covering visually-grounded language understanding and generation.
  arXiv  Detail & Related papers  (2023-03-11T11:00:16Z)
• Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection [67.64272825961395]
  We present Relation Prompt (RePro) for Open-vocabulary Video Visual Relation Detection (Open-VidVRD) RePro addresses the two technical challenges of Open-VidVRD: 1) the prompt tokens should respect the two different semantic roles of subject and object, and 2) the tuning should account for the diverse predicate-temporal motion patterns of the subject-object compositions.
  arXiv  Detail & Related papers  (2023-02-01T06:20:54Z)
• HierVL: Learning Hierarchical Video-Language Embeddings [108.77600799637172]
  HierVL is a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA.
  arXiv  Detail & Related papers  (2023-01-05T21:53:19Z)
• Fine-grained Semantic Alignment Network for Weakly Supervised Temporal Language Grounding [148.46348699343991]
  Temporal language grounding aims to localize a video segment in an untrimmed video based on a natural language description. Most of the existing weakly supervised methods generate a candidate segment set and learn cross-modal alignment through a MIL-based framework. We propose a novel candidate-free framework: Fine-grained Semantic Alignment Network (FSAN), for weakly supervised TLG.
  arXiv  Detail & Related papers  (2022-10-21T13:10:27Z)
• The Second Place Solution for The 4th Large-scale Video Object Segmentation Challenge--Track 3: Referring Video Object Segmentation [18.630453674396534]
  ReferFormer aims to segment object instances in a given video referred by a language expression in all video frames. This work proposes several tricks to boost further, including cyclical learning rates, semi-supervised approach, and test-time augmentation inference. The improved ReferFormer ranks 2nd place on CVPR2022 Referring Youtube-VOS Challenge.
  arXiv  Detail & Related papers  (2022-06-24T02:15:06Z)
• Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding [78.71529237748018]
  Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance. We propose a commonsense-aware cross-modal alignment framework, which incorporates commonsense-guided visual and text representations into a complementary common space.
  arXiv  Detail & Related papers  (2022-04-04T13:07:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.