GLaVE-Cap: Global-Local Aligned Video Captioning with Vision Expert Integration

• URL: http://arxiv.org/abs/2509.11360v1
• Date: Sun, 14 Sep 2025 17:25:55 GMT
• Title: GLaVE-Cap: Global-Local Aligned Video Captioning with Vision Expert Integration
• Authors: Wan Xu, Feng Zhu, Yihan Zeng, Yuanfan Guo, Ming Liu, Hang Xu, Wangmeng Zuo,
• Abstract summary: We propose GLaVE-Cap, a Global-Local aligned framework with Vision Expert integration for Captioning.<n>We construct GLaVE-Bench, a comprehensive video captioning benchmark featuring 5X more queries per video than existing benchmarks.<n>We also provide a training dataset GLaVE-1.2M containing 16K high-quality fine-grained video captions and 1.2M related question-answer pairs.
• Score: 57.5390432800788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video detailed captioning aims to generate comprehensive video descriptions to facilitate video understanding. Recently, most efforts in the video detailed captioning community have been made towards a local-to-global paradigm, which first generates local captions from video clips and then summarizes them into a global caption. However, we find this paradigm leads to less detailed and contextual-inconsistent captions, which can be attributed to (1) no mechanism to ensure fine-grained captions, and (2) weak interaction between local and global captions. To remedy the above two issues, we propose GLaVE-Cap, a Global-Local aligned framework with Vision Expert integration for Captioning, which consists of two core modules: TrackFusion enables comprehensive local caption generation, by leveraging vision experts to acquire cross-frame visual prompts, coupled with a dual-stream structure; while CaptionBridge establishes a local-global interaction, by using global context to guide local captioning, and adaptively summarizing local captions into a coherent global caption. Besides, we construct GLaVE-Bench, a comprehensive video captioning benchmark featuring 5X more queries per video than existing benchmarks, covering diverse visual dimensions to facilitate reliable evaluation. We further provide a training dataset GLaVE-1.2M containing 16K high-quality fine-grained video captions and 1.2M related question-answer pairs. Extensive experiments on four benchmarks show that our GLaVE-Cap achieves state-of-the-art performance. Besides, the ablation studies and student model analyses further validate the effectiveness of the proposed modules and the contribution of GLaVE-1.2M to the video understanding community. The source code, model weights, benchmark, and dataset will be open-sourced.

Related papers

• IF-VidCap: Can Video Caption Models Follow Instructions? [44.2412700621584]
  We introduce IF-VidCap, a new benchmark for evaluating controllable video captioning.<n>IF-VidCap incorporates a systematic framework that assesses captions on two dimensions: format correctness and content correctness.
  arXiv  Detail & Related papers  (2025-10-21T15:25:08Z)
• SGCap: Decoding Semantic Group for Zero-shot Video Captioning [14.484825416367338]
  Zero-shot video captioning aims to generate sentences for describing videos without training the model on video-text pairs.<n>We propose a Semantic Group Captioning (SGCap) method for zero-shot video captioning.
  arXiv  Detail & Related papers  (2025-08-02T09:05:45Z)
• Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos [53.723410664944566]
  We present Perceive Anything Model (PAM), a framework for comprehensive region-level visual understanding in images and videos.<n>Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation.<n>A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features into multi-modal tokens.
  arXiv  Detail & Related papers  (2025-06-05T17:51:39Z)
• Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation [118.5096631571738]
  We present Any2Caption, a novel framework for controllable video generation under any condition.<n>By leveraging modern multimodal large language models (MLLMs), Any2Caption interprets diverse inputs--text, images, videos, and specialized cues such as region, motion, and camera poses--into dense, structured captions.<n> Comprehensive evaluations demonstrate significant improvements of our system in controllability and video quality across various aspects of existing video generation models.
  arXiv  Detail & Related papers  (2025-03-31T17:59:01Z)
• The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning [89.64905703368255]
  We propose a novel progressive multi-granularity textual prompting strategy for zero-shot video captioning.<n>Our approach constructs three distinct memory banks, encompassing noun phrases, scene graphs of noun phrases, and entire sentences.
  arXiv  Detail & Related papers  (2025-03-31T03:00:19Z)
• GL-RG: Global-Local Representation Granularity for Video Captioning [52.56883051799501]
  We propose a GL-RG framework for video captioning, namely a textbfGlobal-textbfLocal textbfRepresentation textbfGranularity. Our GL-RG demonstrates three advantages over the prior efforts: 1) we explicitly exploit extensive visual representations from different video ranges to improve linguistic expression; 2) we devise a novel global-local encoder to produce rich semantic vocabulary to obtain a descriptive granularity of video contents across frames; and 3) we develop an incremental training strategy which organizes model learning in an incremental fashion to incur an optimal captioning
  arXiv  Detail & Related papers  (2022-05-22T02:00:09Z)
• Hierarchical Modular Network for Video Captioning [162.70349114104107]
  We propose a hierarchical modular network to bridge video representations and linguistic semantics from three levels before generating captions. The proposed method performs favorably against the state-of-the-art models on the two widely-used benchmarks: MSVD 104.0% and MSR-VTT 51.5% in CIDEr score.
  arXiv  Detail & Related papers  (2021-11-24T13:07:05Z)
• CLIP4Caption: CLIP for Video Caption [9.470254059503862]
  We propose a CLIP4Caption framework that improves video captioning based on a CLIP-enhanced video-text matching network (VTM) This framework is taking full advantage of the information from both vision and language and enforcing the model to learn strongly text-correlated video features for text generation.
  arXiv  Detail & Related papers  (2021-10-13T10:17:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.