RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models

• URL: http://arxiv.org/abs/2609.16847v1
• Date: Tue, 15 Sep 2026 08:39:55 GMT
• Title: RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models
• Abstract summary: Region-level retrieval plays a crucial role in realworld applications such as e-commerce product search and RAG.<n>We present RegRet, an LMM-based framework that enhances the regional representations without compromising global performance.<n>At its core, RegRet integrates a Region-Aware to capture detailed regional features while balancing them with the global background context.
• Score: 42.63747293653087
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Region-level retrieval aims to align user-specified image regions with relevant regions or textual descriptions, playing a crucial role in realworld applications such as e-commerce product search and RAG. Although recent Large Multimodal Models (LMMs) have made significant strides in multimodal retrieval, they primarily focus on global-level tasks and struggle to capture effective region-level representations. To bridge this gap, we present RegRet, an LMM-based Region-level Retrieval framework that enhances the regional representations without compromising overall global retrieval performance. At its core, RegRet integrates a Region-Aware Encoder to capture detailed regional features while balancing them with the global background context. To further enhance the fine-grained understanding and discriminability of representations, we design a multi-stage training pipeline that includes detailed localized captioning and regional contrastive learning tasks. In addition, considering the absence of region-level contrastive training data and the limited diversity of evaluation tasks in current benchmarks, we introduce the REGMB benchmark. It comprises 225k contrastive pairs, covering four multimodal retrieval tasks. Extensive experiments validate the effectiveness of our approach. RegRet outperforms strong baselines in the zero-shot setting. Further training with contrastive learning leads to an average improvement of more than 20\% on both REGMB and public benchmarks, while achieving comparable or better results on global-level retrieval tasks.

Related papers

• AINet: Anchor Instances Learning for Regional Heterogeneity in Whole Slide Image [61.54860340942449]
  We introduce a novel concept of anchor instance (AI), a compact subset of instances that are representative within their regions (local) and discriminative at the bag (global) level.<n>These AIs act as semantic references to guide interactions across regions, correcting non-discriminative patterns while preserving regional diversity.<n>We develop a concise yet effective framework, AINet, which employs a simple predictor and surpasses state-of-the-art methods with substantially fewer FLOPs and parameters.
  arXiv  Detail & Related papers  (2026-02-21T09:36:27Z)
• RegionReasoner: Region-Grounded Multi-Round Visual Reasoning [69.75509909581133]
  RegionReasoner is a reinforcement learning framework for visual reasoning.<n>It enforces grounded reasoning by requiring each reasoning trace to explicitly cite the corresponding reference bounding boxes.<n>RegionReasoner is optimized with structured rewards combining grounding fidelity and global-local semantic alignment.
  arXiv  Detail & Related papers  (2026-02-03T16:52:16Z)
• RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks [33.68425991692674]
  Region Index (RCI) is the first model-based score to quantify a dataset's reliance on global versus local visual information.<n>When applying RCI to 13 widely used multimodal benchmarks, we observed that most of them favor localized reasoning.
  arXiv  Detail & Related papers  (2025-09-28T06:26:11Z)
• Region-based Cluster Discrimination for Visual Representation Learning [30.79223671093668]
  Region-Aware Cluster Discrimination (RICE) is a novel method that enhances region-level visual and OCR capabilities.<n>RICE consistently outperforms previous methods on tasks, including segmentation, dense detection, and visual perception.
  arXiv  Detail & Related papers  (2025-07-26T17:47:09Z)
• Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning [50.88504784466931]
  Multi-task dense prediction involves semantic segmentation, depth estimation, and surface normal estimation. Existing solutions typically rely on learning global image representations for global cross-task image matching. Our proposal involves modeling region-wise representations using Gaussian Distributions.
  arXiv  Detail & Related papers  (2024-03-15T12:41:30Z)
• RegionGPT: Towards Region Understanding Vision Language Model [88.42271128373191]
  RegionGPT (short as RGPT) is a novel framework designed for complex region-level captioning and understanding. We develop an automated region caption data generation pipeline, enriching the training set with detailed region-level captions. We demonstrate that a universal RGPT model can be effectively applied and significantly enhancing performance across a range of region-level tasks.
  arXiv  Detail & Related papers  (2024-03-04T18:58:08Z)
• R-MAE: Regions Meet Masked Autoencoders [113.73147144125385]
  We explore regions as a potential visual analogue of words for self-supervised image representation learning. Inspired by Masked Autoencoding (MAE), a generative pre-training baseline, we propose masked region autoencoding to learn from groups of pixels or regions.
  arXiv  Detail & Related papers  (2023-06-08T17:56:46Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.