MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?

• URL: http://arxiv.org/abs/2609.02565v1
• Date: Wed, 02 Sep 2026 13:16:06 GMT
• Title: MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?
• Abstract summary: Text-video retrieval requires representations that can distinguish videos with similar scenes, actions, and temporal patterns.<n>Recent multimodal large language models have been adapted as embedding models, but they often represent each input using a single token from the final layer.<n>We propose MARS, a multi-layer and multi-slot embedding framework for text-video retrieval.
• Score: 2.0824228840987447
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text-video retrieval requires representations that can distinguish videos with similar scenes, actions, and temporal patterns. Recent multimodal large language models have been adapted as embedding models, but they often represent each input using a single token from the final layer. This can compress diverse video-text cues into a single vector and limit fine-grained retrieval. To address this limitation, we propose MARS, a multi-layer and multi-slot embedding framework for text-video retrieval. MARS constructs multiple adaptive representation slots by combining hidden states from different decoder layers, compares corresponding text and video slots, and aggregates their similarities for retrieval. To better handle confusing candidates, we further introduce a hard-negative-aware slot specialization objective that encourages the slots to capture discriminative matching cues. Experiments on four text-video retrieval benchmarks show that MARS achieves state-of-the-art results in both direct similarity-based retrieval and reranking settings. Ablation studies and analyses demonstrate that multi-layer fusion, multiple slots, and hard-negative-aware slot specialization provide complementary gains. Code is available at https://github.com/sejong-rcv/MARS.

Related papers

• TC-MGC: Text-Conditioned Multi-Grained Contrastive Learning for Text-Video Retrieval [1.8434042562191815]
  We propose a Text-Conditioned Multi-Grained Contrast framework, dubbed TC-MGC.<n>Our model employs a language-video attention block to generate aggregated frame and video representations conditioned on the word's and text's attention weights over frames.<n> Empirically, TC-MGC achieves competitive results on multiple text-video retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-04-07T03:33:14Z)
• Unified Coarse-to-Fine Alignment for Video-Text Retrieval [71.85966033484597]
  We propose a Unified Coarse-to-fine Alignment model, dubbed UCoFiA. Our model captures the cross-modal similarity information at different granularity levels. We apply the Sinkhorn-Knopp algorithm to normalize the similarities of each level before summing them.
  arXiv  Detail & Related papers  (2023-09-18T19:04:37Z)
• Video-Text Retrieval by Supervised Sparse Multi-Grained Learning [22.17732989393653]
  We present a novel multi-grained sparse learning framework, S3MA, to learn an sparse space shared between the video and the text for video-text retrieval. With the text data at hand, we learn and update the shared sparse space in a supervised manner using the proposed similarity and alignment losses. Benefiting from the learned shared sparse space and multi-grained similarities, experiments on several video-text retrieval benchmarks demonstrate the superiority of S3MA over existing methods.
  arXiv  Detail & Related papers  (2023-02-19T04:03:22Z)
• Tagging before Alignment: Integrating Multi-Modal Tags for Video-Text Retrieval [23.418120617544545]
  Vision-language alignment learning for video-text retrieval arouses a lot of attention in recent years. In this paper, we integrate multi-modal information in an explicit manner by tagging, and use the tags as the anchors for better video-text alignment. To strengthen the interaction between video and text, we build a joint cross-modal encoder with the triplet input of [vision, tag, text] and perform two additional supervised tasks.
  arXiv  Detail & Related papers  (2023-01-30T03:53:19Z)
• UATVR: Uncertainty-Adaptive Text-Video Retrieval [90.8952122146241]
  A common practice is to transfer text-video pairs to the same embedding space and craft cross-modal interactions with certain entities. We propose an Uncertainty-language Text-Video Retrieval approach, termed UATVR, which models each look-up as a distribution matching procedure.
  arXiv  Detail & Related papers  (2023-01-16T08:43:17Z)
• RaP: Redundancy-aware Video-language Pre-training for Text-Video Retrieval [61.77760317554826]
  We propose Redundancy-aware Video-language Pre-training. We design a redundancy measurement of video patches and text tokens by calculating the cross-modal minimum dis-similarity. We evaluate our method on four benchmark datasets, MSRVTT, MSVD, DiDeMo, and LSMDC.
  arXiv  Detail & Related papers  (2022-10-13T10:11:41Z)
• Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval [70.30052749168013]
  Multi-channel video-language retrieval require models to understand information from different channels. contrastive multimodal models are shown to be highly effective at aligning entities in images/videos and text. There is not a clear way to quickly adapt these two lines to multi-channel video-language retrieval with limited data and resources.
  arXiv  Detail & Related papers  (2022-06-05T01:43:52Z)
• BridgeFormer: Bridging Video-text Retrieval with Multiple Choice Questions [38.843518809230524]
  We introduce a novel pretext task dubbed Multiple Choice Questions (MCQ) A module BridgeFormer is trained to answer the "questions" constructed by the text features via resorting to the video features. In the form of questions and answers, the semantic associations between local video-text features can be properly established. Our method outperforms state-of-the-art methods on the popular text-to-video retrieval task in five datasets.
  arXiv  Detail & Related papers  (2022-01-13T09:33:54Z)
• Video Corpus Moment Retrieval with Contrastive Learning [56.249924768243375]
  Video corpus moment retrieval (VCMR) is to retrieve a temporal moment that semantically corresponds to a given text query. We propose a Retrieval and Localization Network with Contrastive Learning (ReLoCLNet) for VCMR. Experimental results show that ReLoCLNet encodes text and video separately for efficiency, its retrieval accuracy is comparable with baselines adopting cross-modal interaction learning.
  arXiv  Detail & Related papers  (2021-05-13T12:54:39Z)
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training [75.55823420847759]
  We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions.
  arXiv  Detail & Related papers  (2020-05-01T03:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.