Addressing the ID-Matching Challenge in Long Video Captioning

• URL: http://arxiv.org/abs/2510.06973v1
• Date: Wed, 08 Oct 2025 12:59:21 GMT
• Title: Addressing the ID-Matching Challenge in Long Video Captioning
• Abstract summary: Key challenge in long video captioning is accurately recognizing the same individuals who appear in different frames.<n>We propose a novel video captioning method called Recognizing Identities for Captioning Effectively (RICE)
• Score: 16.41663075626347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating captions for long and complex videos is both critical and challenging, with significant implications for the growing fields of text-to-video generation and multi-modal understanding. One key challenge in long video captioning is accurately recognizing the same individuals who appear in different frames, which we refer to as the ID-Matching problem. Few prior works have focused on this important issue. Those that have, usually suffer from limited generalization and depend on point-wise matching, which limits their overall effectiveness. In this paper, unlike previous approaches, we build upon LVLMs to leverage their powerful priors. We aim to unlock the inherent ID-Matching capabilities within LVLMs themselves to enhance the ID-Matching performance of captions. Specifically, we first introduce a new benchmark for assessing the ID-Matching capabilities of video captions. Using this benchmark, we investigate LVLMs containing GPT-4o, revealing key insights that the performance of ID-Matching can be improved through two methods: 1) enhancing the usage of image information and 2) increasing the quantity of information of individual descriptions. Based on these insights, we propose a novel video captioning method called Recognizing Identities for Captioning Effectively (RICE). Extensive experiments including assessments of caption quality and ID-Matching performance, demonstrate the superiority of our approach. Notably, when implemented on GPT-4o, our RICE improves the precision of ID-Matching from 50% to 90% and improves the recall of ID-Matching from 15% to 80% compared to baseline. RICE makes it possible to continuously track different individuals in the captions of long videos.

Related papers

• Beyond Pedestrians: Caption-Guided CLIP Framework for High-Difficulty Video-based Person Re-Identification [0.0]
  We propose a caption-guided CLIP framework that leverages textual descriptions and learnable tokens.<n>We evaluate our approach on two standard datasets and two newly constructed high-difficulty datasets.<n> Experimental results demonstrate that our method outperforms current state-of-art approaches.
  arXiv  Detail & Related papers  (2026-04-09T02:55:51Z)
• Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization [30.445325065182868]
  auxiliary captions are often used to enhance video understanding, bridging the gap between the modalities.<n>We propose $textbfCaRe-DPO, a retrieval framework that directly optimize caption generation using retrieval relevance scores.<n>We show that CaRe-DPO significantly enhances retrieval performance by effectively leveraging auxiliary knowledge to generate fine-grained captions for retrieval.
  arXiv  Detail & Related papers  (2025-09-20T07:36:53Z)
• Identity-Preserving Text-to-Video Generation via Training-Free Prompt, Image, and Guidance Enhancement [58.85593321752693]
  Identity-preserving text-to-video (IPT2V) generation creates videos faithful to both a reference subject image and a text prompt.<n>We introduce a Training-Free Prompt, Image, and Guidance Enhancement framework that bridges the semantic gap between the video description and the reference image.<n>We win first place in the ACM Multimedia 2025 Identity-Preserving Video Generation Challenge.
  arXiv  Detail & Related papers  (2025-09-01T11:03:13Z)
• VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understanding [22.400847202448478]
  Long video understanding presents a significant challenge to large language models (MLs)<n>VisualSubtitleation(VSI) integrates subtitles, semantic timestamps, and scene boundaries into a unified multimodal search process.<n>The proposed method captures the visual information of video frames as well as the complementary textual information through a dual-stream search mechanism.
  arXiv  Detail & Related papers  (2025-08-09T07:38:48Z)
• Expertized Caption Auto-Enhancement for Video-Text Retrieval [10.250004732070494]
  This paper proposes an automatic caption enhancement method that improves expression quality and mitigates empiricism in augmented captions through self-learning.<n>Our method is entirely data-driven, which not only dispenses with heavy data collection and computation workload but also improves self-adaptability.<n>Our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo.
  arXiv  Detail & Related papers  (2025-02-05T04:51:46Z)
• Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage [50.84150600032693]
  Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations.<n>We propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions.<n>Our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V.
  arXiv  Detail & Related papers  (2024-12-20T01:37:22Z)
• AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark [89.73538448786405]
  We propose AuroraCap, a video captioner based on a large multimodal model.<n>We implement the token merging strategy, reducing the number of input visual tokens.<n>AuroraCap shows superior performance on various video and image captioning benchmarks.
  arXiv  Detail & Related papers  (2024-10-04T00:13:54Z)
• Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach [56.610806615527885]
  A key challenge in text-video retrieval (TVR) is the information asymmetry between video and text.<n>This paper introduces a data-centric framework to bridge this gap by enriching textual representations to better match the richness of video content.<n>We propose a query selection mechanism that identifies the most relevant and diverse queries, reducing computational cost while improving accuracy.
  arXiv  Detail & Related papers  (2024-08-14T01:24:09Z)
• IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model [52.697180472760635]
  This paper explores the potential of character identities memory and recognition across multiple visual scenarios. We propose visual instruction tuning with ID reference and develop an ID-Aware Large Vision-Language Model, IDA-VLM. Our research introduces a novel benchmark MM-ID, to examine LVLMs on instance IDs memory and recognition across four dimensions.
  arXiv  Detail & Related papers  (2024-07-10T12:11:59Z)
• MICap: A Unified Model for Identity-aware Movie Descriptions [16.287294191608893]
  We present a new single stage approach that can seamlessly switch between id-aware caption generation or FITB when given a caption with blanks. Our model, Movie-Identity Captioner (MICap), uses a shared auto-regressive decoder that benefits from training with FITB and full-caption generation objectives.
  arXiv  Detail & Related papers  (2024-05-19T08:54:12Z)
• Identity-Aware Multi-Sentence Video Description [105.13845996039277]
  We introduce an auxiliary task of Fill-in the Identity, that aims to predict persons' IDs consistently within a set of clips. One of the key components is a gender-aware textual representation as well an additional gender prediction objective in the main model. Experiments show that our proposed Fill-in the Identity model is superior to several baselines and recent works.
  arXiv  Detail & Related papers  (2020-08-22T09:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.