Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model

• URL: http://arxiv.org/abs/2507.05513v1
• Date: Mon, 07 Jul 2025 22:20:04 GMT
• Title: Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model
• Authors: Mengyao Xu, Gabriel Moreira, Ronay Ak, Radek Osmulski, Yauhen Babakhin, Zhiding Yu, Benedikt Schifferer, Even Oldridge,
• Abstract summary: llama-nemoretriever-colembed is a unified text-image retrieval model that delivers state-of-the-art performance across multiple benchmarks.<n>The 3B model achieves state of the art performance, scoring NDCG@5 91.0 on ViDoRe V1 and 63.5 on ViDoRe V2, placing first on both leaderboards as of June 27, 2025.
• Score: 20.055106781946417
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Motivated by the growing demand for retrieval systems that operate across modalities, we introduce llama-nemoretriever-colembed, a unified text-image retrieval model that delivers state-of-the-art performance across multiple benchmarks. We release two model variants, 1B and 3B. The 3B model achieves state of the art performance, scoring NDCG@5 91.0 on ViDoRe V1 and 63.5 on ViDoRe V2, placing first on both leaderboards as of June 27, 2025. Our approach leverages the NVIDIA Eagle2 Vision-Language model (VLM), modifies its architecture by replacing causal attention with bidirectional attention, and integrates a ColBERT-style late interaction mechanism to enable fine-grained multimodal retrieval in a shared embedding space. While this mechanism delivers superior retrieval accuracy, it introduces trade-offs in storage and efficiency. We provide a comprehensive analysis of these trade-offs. Additionally, we adopt a two-stage training strategy to enhance the model's retrieval capabilities.

Related papers

• UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? [50.92401586025528]
  Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear.<n>We introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks.
  arXiv  Detail & Related papers  (2026-03-03T18:36:16Z)
• Xray-Visual Models: Scaling Vision models on Industry Scale Data [40.21391133092764]
  We present Xray-Visual, a unified vision model architecture for large-scale image and video understanding trained on industry-scale social media data.<n>Our model leverages over 15 billion curated image-text pairs and 10 billion video-hashtag pairs from Facebook and Instagram.<n>Xray-Visual establishes new benchmarks for scalable, multimodal vision models, while maintaining superior accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2026-02-18T22:22:44Z)
• Nemotron ColEmbed V2: Top-Performing Late Interaction embedding models for Visual Document Retrieval [19.23621110865551]
  Motivated by the growing demand for visual document retrieval, we introduce Nemotron ColEmbed V2, a family of models that achieve state-of-the-art performance on the ViDoRe benchmarks.<n>We describe the main techniques used across data processing, training, and post-training to build our top-performing models.
  arXiv  Detail & Related papers  (2026-02-03T20:26:44Z)
• Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model [62.889356203346985]
  We propose DUal-STream diffusion (DUST), a world-model augmented VLA framework that handles the modality conflict.<n>DUST achieves up to 6% gains over a standard VLA baseline and implicit world-modeling methods.<n>On real-world tasks with the Franka Research 3, DUST outperforms baselines in success rate by 13%.
  arXiv  Detail & Related papers  (2025-10-31T16:32:12Z)
• Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model [28.559525134847828]
  We present UniPic2-SD3.5M-Kontext, which achieves state-of-the-art image generation and editing while extending seamlessly into a unified multimodal framework.<n>Our approach begins with architectural modifications to SD3.5-Medium and large-scale pre-training on high-quality data.<n>UniPic2-SD3.5M-Kontext demonstrates stronger image generation and editing capabilities than models with significantly larger generation parameters.
  arXiv  Detail & Related papers  (2025-09-04T17:00:17Z)
• Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation [66.73899356886652]
  We build an image tokenizer directly atop pre-trained vision foundation models.<n>Our proposed image tokenizer, VFMTok, achieves substantial improvements in image reconstruction and generation quality.<n>It further boosts autoregressive (AR) generation -- achieving a gFID of 2.07 on ImageNet benchmarks.
  arXiv  Detail & Related papers  (2025-07-11T09:32:45Z)
• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• Towards Visuospatial Cognition via Hierarchical Fusion of Visual Experts [4.454997649515497]
  We introduce ViCA2 (Visuospatial Cognitive Assistant 2), a novel MLLM designed to enhance spatial reasoning.<n>ViCA2 features a dual vision architecture integrating SigLIP for semantics and Hiera for spatial structure, coupled with a token ratio control mechanism for efficiency.<n>We also developed ViCA322K, a new large-scale cognition dataset with over 322,000 spatially grounded question-answer pairs.
  arXiv  Detail & Related papers  (2025-05-18T10:57:33Z)
• DSU-Net:An Improved U-Net Model Based on DINOv2 and SAM2 with Multi-scale Cross-model Feature Enhancement [7.9006143460465355]
  This paper proposes a multi-scale feature collabora-tion framework guided by DINOv2 for SAM2, with core innovations in three aspects.<n>It surpasses existing state-of-the-art meth-ods in downstream tasks such as camouflage target detection and salient ob-ject detection, without requiring costly training processes.
  arXiv  Detail & Related papers  (2025-03-27T06:08:24Z)
• Multi-Head Attention Driven Dynamic Visual-Semantic Embedding for Enhanced Image-Text Matching [0.8611782340880084]
  This study proposes an innovative visual semantic embedding model, Multi-Headed Consensus-Aware Visual-Semantic Embedding (MH-CVSE)<n>This model introduces a multi-head self-attention mechanism based on the consensus-aware visual semantic embedding model (CVSE) to capture information in multiple subspaces in parallel.<n>In terms of loss function design, the MH-CVSE model adopts a dynamic weight adjustment strategy to dynamically adjust the weight according to the loss value itself.
  arXiv  Detail & Related papers  (2024-12-26T11:46:22Z)
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling [128.24325909395188]
  We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0.<n>InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet.<n>We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems.
  arXiv  Detail & Related papers  (2024-12-06T18:57:08Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrieval [7.233106731197739]
  We propose a Multi-teacher Cross-modality Alignment Distillation (MCAD) technique to integrate the advantages of single- and dual-stream models. We implement a lightweight CLIP model on Snapdragon/Dimensity chips with only $sim$100M running memory and $sim$8.0ms search latency.
  arXiv  Detail & Related papers  (2023-10-30T15:38:43Z)
• IRGen: Generative Modeling for Image Retrieval [82.62022344988993]
  In this paper, we present a novel methodology, reframing image retrieval as a variant of generative modeling. We develop our model, dubbed IRGen, to address the technical challenge of converting an image into a concise sequence of semantic units. Our model achieves state-of-the-art performance on three widely-used image retrieval benchmarks and two million-scale datasets.
  arXiv  Detail & Related papers  (2023-03-17T17:07:36Z)
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers [115.90778814368703]
  Our objective is language-based search of large-scale image and video datasets. For this task, the approach that consists of independently mapping text and vision to a joint embedding space, a.k.a. dual encoders, is attractive as retrieval scales. An alternative approach of using vision-text transformers with cross-attention gives considerable improvements in accuracy over the joint embeddings.
  arXiv  Detail & Related papers  (2021-03-30T17:57:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.