Expertized Caption Auto-Enhancement for Video-Text Retrieval

• URL: http://arxiv.org/abs/2502.02885v1
• Date: Wed, 05 Feb 2025 04:51:46 GMT
• Title: Expertized Caption Auto-Enhancement for Video-Text Retrieval
• Authors: Junxiang Chen, Baoyao yang, Wenbin Yao,
• Abstract summary: This paper proposes an automatic caption enhancement method that enhances expression quality and mitigates empiricism in augmented captions through self-learning. Our method is entirely data-driven, which not only dispenses with heavy data collection and workload but also improves self-adaptability. Our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo.
• Score: 8.65473421564069
• License:
• Abstract: The burgeoning field of video-text retrieval has witnessed significant advancements with the advent of deep learning. However, the challenge of matching text and video persists due to inadequate textual descriptions of videos. The substantial information gap between the two modalities hinders a comprehensive understanding of videos, resulting in ambiguous retrieval results. While rewriting methods based on large language models have been proposed to broaden text expressions, carefully crafted prompts are essential to ensure the reasonableness and completeness of the rewritten texts. This paper proposes an automatic caption enhancement method that enhances expression quality and mitigates empiricism in augmented captions through self-learning. Additionally, an expertized caption selection mechanism is designed and introduced to customize augmented captions for each video, facilitating video-text matching. Our method is entirely data-driven, which not only dispenses with heavy data collection and computation workload but also improves self-adaptability by circumventing lexicon dependence and introducing personalized matching. The superiority of our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo.

Related papers

• GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval [56.610806615527885]
  This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video. By adaptively segmenting videos into short clips and employing zero-shot captioning, GQE enriches the training dataset with comprehensive scene descriptions. GQE achieves state-of-the-art performance on several benchmarks, including MSR-VTT, MSVD, LSMDC, and VATEX.
  arXiv  Detail & Related papers  (2024-08-14T01:24:09Z)
• Towards Holistic Language-video Representation: the language model-enhanced MSR-Video to Text Dataset [4.452729255042396]
  A more robust and holistic language-video representation is the key to pushing video understanding forward. The current plain and simple text descriptions and the visual-only focus for the language-video tasks result in a limited capacity in real-world natural language video retrieval tasks. This paper introduces a method to automatically enhance video-language datasets, making them more modality and context-aware.
  arXiv  Detail & Related papers  (2024-06-19T20:16:17Z)
• SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval [11.548061962976321]
  We propose a novel Syntax-Hierarchy-Enhanced text-video retrieval method (SHE-Net) First, to facilitate a more fine-grained integration of visual content, we employ the text syntax hierarchy, which reveals the grammatical structure of text descriptions. Second, to further enhance the multi-modal interaction and alignment, we also utilize the syntax hierarchy to guide the similarity calculation.
  arXiv  Detail & Related papers  (2024-04-22T10:23:59Z)
• Unifying Latent and Lexicon Representations for Effective Video-Text Retrieval [87.69394953339238]
  We propose the UNIFY framework, which learns lexicon representations to capture fine-grained semantics in video-text retrieval. We show our framework largely outperforms previous video-text retrieval methods, with 4.8% and 8.2% Recall@1 improvement on MSR-VTT and DiDeMo respectively.
  arXiv  Detail & Related papers  (2024-02-26T17:36:50Z)
• Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling [79.49128866877922]
  Video-Teller is a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions.
  arXiv  Detail & Related papers  (2023-10-08T03:35:27Z)
• Does Video Summarization Require Videos? Quantifying the Effectiveness of Language in Video Summarization [37.09662541127891]
  Video summarization remains a huge challenge in computer vision due to the size of the input videos to be summarized. We propose an efficient, language-only video summarizer that achieves competitive accuracy with high data efficiency.
  arXiv  Detail & Related papers  (2023-09-18T00:08:49Z)
• In-Style: Bridging Text and Uncurated Videos with Style Transfer for Text-Video Retrieval [72.98185525653504]
  We propose a new setting, text-video retrieval with uncurated & unpaired data, that during training utilizes only text queries together with uncurated web videos. To improve generalization, we show that one model can be trained with multiple text styles. We evaluate our model on retrieval performance over multiple datasets to demonstrate the advantages of our style transfer framework.
  arXiv  Detail & Related papers  (2023-09-16T08:48:21Z)
• CelebV-Text: A Large-Scale Facial Text-Video Dataset [91.22496444328151]
  CelebV-Text is a large-scale, diverse, and high-quality dataset of facial text-video pairs. CelebV-Text comprises 70,000 in-the-wild face video clips with diverse visual content, each paired with 20 texts generated using the proposed semi-automatic text generation strategy. The superiority of CelebV-Text over other datasets is demonstrated via comprehensive statistical analysis of the videos, texts, and text-video relevance.
  arXiv  Detail & Related papers  (2023-03-26T13:06:35Z)
• Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision [54.73758942064708]
  We teach machines to understand visuals and natural language by learning the mapping between sentences and noisy video snippets without explicit annotations. For training and evaluation, we contribute a new dataset ApartmenTour' that contains a large number of online videos and subtitles.
  arXiv  Detail & Related papers  (2020-11-19T03:43:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.