EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval

• URL: http://arxiv.org/abs/2603.25267v3
• Date: Wed, 01 Apr 2026 15:02:25 GMT
• Title: EagleNet: Energy-Aware Fine-Grained Relationship Learning Network for Text-Video Retrieval
• Abstract summary: We introduce Energy-Aware Fine-Grained Relationship Learning Network (EagleNet)<n>EagleNet generates accurate and context-aware enriched text embeddings.<n>Experiments have demonstrated the superiority of EagleNet across MSRVTT, DiDeMo, MSVD, and VATEX.
• Score: 59.858268313337355
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-video retrieval tasks have seen significant improvements due to the recent development of large-scale vision-language pre-trained models. Traditional methods primarily focus on video representations or cross-modal alignment, while recent works shift toward enriching text expressiveness to better match the rich semantics in videos. However, these methods use only interactions between text and frames/video, and ignore rich interactions among the internal frames within a video, so the final expanded text cannot capture frame contextual information, leading to disparities between text and video. In response, we introduce Energy-Aware Fine-Grained Relationship Learning Network (EagleNet) to generate accurate and context-aware enriched text embeddings. Specifically, the proposed Fine-Grained Relationship Learning mechanism (FRL) first constructs a text-frame graph by the generated text candidates and frames, then learns relationships among texts and frames, which are finally used to aggregate text candidates into an enriched text embedding that incorporates frame contextual information. To further improve fine-grained relationship learning in FRL, we design Energy-Aware Matching (EAM) to model the energy of text-frame interactions and thus accurately capture the distribution of real text-video pairs. Moreover, for more effective cross-modal alignment and stable training, we replace the conventional softmax-based contrastive loss with the sigmoid loss. Extensive experiments have demonstrated the superiority of EagleNet across MSRVTT, DiDeMo, MSVD, and VATEX. Codes are available at https://github.com/draym28/EagleNet.

Related papers

• Rethinking Video-Language Model from the Language Input Perspective [57.766724144002346]
  Video-Language Models (VLMs) have become a significant yet challenging technology to bridge the gap between videos and texts.<n>In this paper, we propose a novel plug-and-play framework for various VLM-based methods to fully bridge videos and texts.
  arXiv  Detail & Related papers  (2026-05-27T03:47:05Z)
• Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval [90.72791786676753]
  Video-ColBERT introduces a simple and efficient mechanism for fine-grained similarity assessment between queries and videos.<n>We find that this interaction and training paradigm leads to strong individual, yet compatible, representations for encoding video content.<n>These representations lead to increases in performance on common text-to-video retrieval benchmarks compared to other bi-encoder methods.
  arXiv  Detail & Related papers  (2025-03-24T17:51:29Z)
• VTD-CLIP: Video-to-Text Discretization via Prompting CLIP [44.51452778561945]
  Vision-language models bridge visual and linguistic understanding and have proven to be powerful for video recognition tasks.<n>Existing approaches rely primarily on parameter-efficient fine-tuning of image-text pre-trained models.<n>We propose a video-to-text discretization framework to address limited interpretability and poor generalization due to inadequate temporal modeling.
  arXiv  Detail & Related papers  (2025-03-24T07:27:19Z)
• NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality [52.08735848128973]
  We study the capability of Video-Language (VidL) models in understanding compositions between objects, attributes, actions and their relations. We propose a training method called NAVERO which utilizes video-text data augmented with negative texts to enhance composition understanding.
  arXiv  Detail & Related papers  (2024-08-18T15:27:06Z)
• SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval [11.548061962976321]
  We propose a novel Syntax-Hierarchy-Enhanced text-video retrieval method (SHE-Net)<n>First, to facilitate a more fine-grained integration of visual content, we employ the text syntax hierarchy, which reveals the grammatical structure of text descriptions.<n>Second, to further enhance the multi-modal interaction and alignment, we also utilize the syntax hierarchy to guide the similarity calculation.
  arXiv  Detail & Related papers  (2024-04-22T10:23:59Z)
• Weakly Supervised Video Representation Learning with Unaligned Text for Sequential Videos [39.42509966219001]
  This paper studies weakly supervised sequential video understanding where the accurate time-level text-video alignment is not provided. We use a transformer to aggregate frame-level features for video representation and use a pre-trained text encoder to encode the texts corresponding to each action and the whole video. Experiments on video sequence verification and text-to-video matching show that our method outperforms baselines by a large margin.
  arXiv  Detail & Related papers  (2023-03-22T08:13:25Z)
• Temporal Perceiving Video-Language Pre-training [112.1790287726804]
  This work introduces a novel text-video localization pre-text task to enable fine-grained temporal and semantic alignment. Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description. Our method connects the fine-grained frame representations with the word representations and implicitly distinguishes representations of different instances in the single modality.
  arXiv  Detail & Related papers  (2023-01-18T12:15:47Z)
• Contrastive Video-Language Learning with Fine-grained Frame Sampling [54.542962813921214]
  FineCo is an approach to better learn video and language representations with a fine-grained contrastive objective operating on video frames. It helps distil a video by selecting the frames that are semantically equivalent to the text, improving cross-modal correspondence.
  arXiv  Detail & Related papers  (2022-10-10T22:48:08Z)
• X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval [26.581384985173116]
  In text-video retrieval, the objective is to learn a cross-modal similarity function between a text and a video. We propose a cross-modal attention model called X-Pool that reasons between a text and the frames of a video.
  arXiv  Detail & Related papers  (2022-03-28T20:47:37Z)
• Align and Prompt: Video-and-Language Pre-training with Entity Prompts [111.23364631136339]
  Video-and-language pre-training has shown promising improvements on various downstream tasks. We propose Align and Prompt: an efficient and effective video-and-language pre-training framework with better cross-modal alignment. Our code and pre-trained models will be released.
  arXiv  Detail & Related papers  (2021-12-17T15:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.