ViLL-E: Video LLM Embeddings for Retrieval

• URL: http://arxiv.org/abs/2604.12148v1
• Date: Mon, 13 Apr 2026 23:54:58 GMT
• Title: ViLL-E: Video LLM Embeddings for Retrieval
• Abstract summary: ViLL-E (Video-LLM-Embed) is a unified VideoLLM architecture endowed with a novel embedding generation mechanism.<n>We train this model with a three-stage training methodology combining generative and contrastive learning.<n>Our model significantly improves temporal localization (on avg. 7% over other VideoLLMs) and video retrieval (up to 4% over dual encoder models)
• Score: 55.66444095205989
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Large Language Models (VideoLLMs) excel at video understanding tasks where outputs are textual, such as Video Question Answering and Video Captioning. However, they underperform specialized embedding-based models in Retrieval tasks, such as Text-toVideo Retrieval and Moment Retrieval. We introduce ViLL-E (Video-LLM-Embed), a unified VideoLLM architecture endowed with a novel embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. We train this model with a three-stage training methodology combining generative and contrastive learning: initial large-scale pre-training with video-caption pairs; followed by continual training on a smaller, detailed-caption dataset; and concluding with task-specific fine-tuning on a novel multi-task dataset covering Video QA, Temporal Localization, Video Retrieval, and Video-Text Matching. Our model significantly improves temporal localization (on avg. 7% over other VideoLLMs) and video retrieval (up to 4% over dual encoder models), achieving performance comparable to state-of-the-art specialized embedding models while remaining competitive on VideoQA tasks. Furthermore, our joint contrastive-generative training unlocks new zero-shot capabilities, significantly outperforming state-of-the-art methods in composed video retrieval (+5% over SotA) and retrieval from long text (+2% over SotA).

Related papers

• VideoWeave: A Data-Centric Approach for Efficient Video Understanding [54.5804686337209]
  We present VideoWeave, a simple yet effective approach to improve data efficiency by constructing synthetic long-context training samples.<n>VideoWeave reorganizes available video-text pairs to expand temporal diversity within fixed compute.<n>Our results highlight that reorganizing training data, rather than altering architectures, may offer a simple and scalable path for training video-language models.
  arXiv  Detail & Related papers  (2026-01-09T20:55:26Z)
• Unleashing Hour-Scale Video Training for Long Video-Language Understanding [61.717205915329664]
  We present VideoMarathon, a large-scale hour-long video instruction-following dataset.<n>This dataset includes around 9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60 minutes per video.<n>We propose Hour-LLaVA, a powerful and efficient Video-LMM for hour-scale video-language modeling.
  arXiv  Detail & Related papers  (2025-06-05T17:59:04Z)
• Text-Conditioned Resampler For Long Form Video Understanding [94.81955667020867]
  We present a text-conditioned video resampler (TCR) module that uses a pre-trained visual encoder and large language model (LLM) TCR can process more than 100 frames at a time with plain attention and without optimised implementations.
  arXiv  Detail & Related papers  (2023-12-19T06:42:47Z)
• Analyzing Zero-Shot Abilities of Vision-Language Models on Video Understanding Tasks [6.925770576386087]
  We propose a detailed study on the generalization abilities of image-text models when evaluated on video understanding tasks in a zero-shot setting. Our experiments show that image-text models exhibit impressive performance on video AR, video RT and video MC. These findings shed a light on the benefits of adapting foundational image-text models to an array of video tasks while avoiding the costly pretraining step.
  arXiv  Detail & Related papers  (2023-10-07T20:57:54Z)
• VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blending [78.1399386935455]
  Large-scale image-text contrastive pre-training models, such as CLIP, have been demonstrated to effectively learn high-quality multimodal representations. We propose a novel video-text pre-training method dubbed VLAB: Video Language pre-training by feature generativearity and Blending. VLAB transfers CLIP representations to video pre-training tasks and develops unified video multimodal models for a wide range of video-text tasks.
  arXiv  Detail & Related papers  (2023-05-22T15:54:22Z)
• VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation [43.90887811621963]
  We propose a new two-stage pre-training framework for video-to-text generation tasks such as video captioning and question answering. A generative encoder-decoder model is first jointly pre-trained on massive image-language data to learn fundamental concepts. As a result, our VideoOFA model achieves new state-the-art performance on four Video Captioning benchmarks.
  arXiv  Detail & Related papers  (2023-05-04T23:27:21Z)
• Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning [39.80936685227549]
  Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. We introduce a Long-Form VIdeo-LAnguage pre-training model (VILA) and train it on a large-scale long-form video and paragraph dataset. We fine-tune the model on seven downstream long-form video-language understanding tasks, achieve new state-of-the-art performances.
  arXiv  Detail & Related papers  (2022-10-12T09:08:27Z)
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation [124.02278735049235]
  VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. We evaluate various baseline methods with and without large-scale VidL pre-training. The significant gap between our best model and human performance calls for future study for advanced VidL models.
  arXiv  Detail & Related papers  (2021-06-08T18:34:21Z)
• Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval [80.7397409377659]
  We propose an end-to-end trainable model that is designed to take advantage of both large-scale image and video captioning datasets. Our model is flexible and can be trained on both image and video text datasets, either independently or in conjunction. We show that this approach yields state-of-the-art results on standard downstream video-retrieval benchmarks.
  arXiv  Detail & Related papers  (2021-04-01T17:48:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.