DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval

• URL: http://arxiv.org/abs/2506.08887v1
• Date: Tue, 10 Jun 2025 15:16:40 GMT
• Title: DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval
• Authors: Leqi Shen, Guoqiang Gong, Tianxiang Hao, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, Jungong Han, Guiguang Ding,
• Abstract summary: Three key discrepancies emerge in the transfer from image-level to video-level: vision, language, and alignment.<n>We propose Discrepancy Reduction in Vision, Language, and Alignment (DiscoVLA), which simultaneously mitigates all three discrepancies.<n> Specifically, we introduce Image-Video Features Fusion to integrate image-level and video-level features, effectively tackling both vision and language discrepancies.<n>To mitigate alignment discrepancies, we propose Image-to-Video Alignment Distillation, which leverages image-level alignment knowledge to enhance video-level alignment.
• Score: 71.24147642498876
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The parameter-efficient adaptation of the image-text pretraining model CLIP for video-text retrieval is a prominent area of research. While CLIP is focused on image-level vision-language matching, video-text retrieval demands comprehensive understanding at the video level. Three key discrepancies emerge in the transfer from image-level to video-level: vision, language, and alignment. However, existing methods mainly focus on vision while neglecting language and alignment. In this paper, we propose Discrepancy Reduction in Vision, Language, and Alignment (DiscoVLA), which simultaneously mitigates all three discrepancies. Specifically, we introduce Image-Video Features Fusion to integrate image-level and video-level features, effectively tackling both vision and language discrepancies. Additionally, we generate pseudo image captions to learn fine-grained image-level alignment. To mitigate alignment discrepancies, we propose Image-to-Video Alignment Distillation, which leverages image-level alignment knowledge to enhance video-level alignment. Extensive experiments demonstrate the superiority of our DiscoVLA. In particular, on MSRVTT with CLIP (ViT-B/16), DiscoVLA outperforms previous methods by 1.5% in R@1, reaching a final score of 50.5% R@1. The code is available at https://github.com/LunarShen/DsicoVLA.

Related papers

• OmniCaptioner: One Captioner to Rule Them All [33.98387155732322]
  We propose a versatile visual captioning framework for generating fine-grained textual descriptions.<n>By converting low-level pixel information into semantically rich textual representations, our framework bridges the gap between visual and textual modalities.<n>We believe the versatility and adaptability of OmniCaptioner can offer a new perspective for bridging the gap between language and visual modalities.
  arXiv  Detail & Related papers  (2025-04-09T17:58:58Z)
• The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning [89.64905703368255]
  We propose a novel progressive multi-granularity textual prompting strategy for zero-shot video captioning.<n>Our approach constructs three distinct memory banks, encompassing noun phrases, scene graphs of noun phrases, and entire sentences.
  arXiv  Detail & Related papers  (2025-03-31T03:00:19Z)
• TULIP: Towards Unified Language-Image Pretraining [60.99500935831526]
  We introduce T, an open-source, drop-in replacement for existing CLIP-like models.<n>Our method leverages generative data augmentation, enhanced image-image and text-text contrastive learning, and image/text reconstruction regularization to learn fine-grained visual features.<n>Our approach, scaling to over 1B parameters, outperforms existing state-of-the-art (SOTA) models across benchmarks.
  arXiv  Detail & Related papers  (2025-03-19T17:58:57Z)
• Contrastive Vision-Language Alignment Makes Efficient Instruction Learner [31.281236193979165]
  We study the task of extending the large language model (LLM) into a vision-language instruction-following model. Existing methods typically train a visual adapter to align the representation between a pre-trained vision transformer (ViT) and the LLM by a generative image captioning loss. We propose CG-VLM that applies Contrastive and Generative alignment objectives to effectively align the representation of ViT and LLM.
  arXiv  Detail & Related papers  (2023-11-29T03:29:46Z)
• Exploring Part-Informed Visual-Language Learning for Person Re-Identification [52.92511980835272]
  We propose Part-Informed Visual-language Learning ($pi$-VL) to enhance fine-grained visual features with part-informed language supervisions for ReID tasks.<n>$pi$-VL introduces a human parsing-guided prompt tuning strategy and a hierarchical visual-language alignment paradigm to ensure within-part feature semantic consistency.<n>As a plug-and-play and inference-free solution, our $pi$-VL achieves performance comparable to or better than state-of-the-art methods on four commonly used ReID benchmarks.
  arXiv  Detail & Related papers  (2023-08-04T23:13:49Z)
• VicTR: Video-conditioned Text Representations for Activity Recognition [73.09929391614266]
  We argue that better video-VLMs can be designed by focusing more on augmenting text, rather than visual information. We introduce Video-conditioned Text Representations (VicTR), a form of text embeddings optimized w.r.t. visual embeddings. Our model can further make use of freely-available semantic information, in the form of visually-grounded auxiliary text.
  arXiv  Detail & Related papers  (2023-04-05T16:30:36Z)
• VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning [19.73126931526359]
  Video paragraph captioning aims to generate a multi-sentence description of an untrimmed video with several temporal event locations in coherent storytelling. We first propose a visual-linguistic (VL) feature. In the proposed VL feature, the scene is modeled by three modalities including (i) a global visual environment; (ii) local visual main agents; (iii) linguistic scene elements. We then introduce an autoregressive Transformer-in-Transformer (TinT) to simultaneously capture the semantic coherence of intra- and inter-event contents within a video.
  arXiv  Detail & Related papers  (2022-11-28T07:39:20Z)
• OmniVL:One Foundation Model for Image-Language and Video-Language Tasks [117.57580168859512]
  We present OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer. We introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label (e.g., image classification), video-label (e.g., video action recognition) data together.
  arXiv  Detail & Related papers  (2022-09-15T17:59:59Z)
• VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts [2.0434814235659555]
  Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. We propose to enhance CLIP via Visual-guided Texts, named VT-CLIP. In few-shot settings, we evaluate our VT-CLIP on 11 well-known classification datasets to demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2021-12-04T18:34:24Z)
• HANet: Hierarchical Alignment Networks for Video-Text Retrieval [15.91922397215452]
  Video-text retrieval is an important yet challenging task in vision-language understanding. Most current works simply measure the video-text similarity based on video-level and text-level embeddings. We propose a Hierarchical Alignment Network (HANet) to align different level representations for video-text matching.
  arXiv  Detail & Related papers  (2021-07-26T09:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.