Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering

• URL: http://arxiv.org/abs/2605.01827v1
• Date: Sun, 03 May 2026 11:34:36 GMT
• Title: Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering
• Abstract summary: We introduce Contextual Latent Steering (CSteer), a training-free approach for guiding general LMMs to refer multiple regions contextually.<n>CSteer starts with pre-computing contextual vectors that implicitly represent visual referring behaviors.<n> Experimental results indicate that general LMMs with CSteer outperform tailored referring LMMs in most cases.
• Score: 51.54461480047943
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Multimodal Models (LMMs) have recently demonstrated their proficiency in holistic visual comprehension. However, most of them struggle to tackle region-level perception guided by visual prompts, especially for cases where multiple regions are referred simultaneously, or scenarios where global contexts are necessary for precise visual referring. We introduce Contextual Latent Steering (CSteer), a training-free approach for guiding general LMMs to refer multiple regions contextually, without expensive fine-tuning or architectural modifications. CSteer starts with pre-computing contextual vectors that implicitly represent visual referring behaviors, such as differentiation among regions and attention to global contexts, followed by representation editing during inference time. Experimental results on multiple datasets indicate that general LMMs with CSteer outperform tailored referring LMMs in most cases, suggesting a promising solution in training-free, and setting new state-of-the-art for this field. Code is available at https://github.com/xing0047/csteer.git.

Related papers

• Multi-Granularity Context-Enhanced RAG over Multimodal Knowledge Graphs [47.75183060175264]
  Retrieval-augmented generation (RAG) is widely used to mitigate hallucination issues in large language models (LLMs) and multimodal large language models (MLLMs)
  arXiv  Detail & Related papers  (2026-08-26T16:38:02Z)
• Global Logic and Local Search: Dual-Stream Multimodal In-Context Learning for Verifiable Industrial Anomaly Detection [27.04976556268606]
  We present Global Logic and Local Search (GLLS), a training-free framework for reference-guided multimodal in-context verification.<n>GLLS uses a Part-Aware Visual-Logical Atlas to organize normal references and structured specifications in the inference context.<n> Experiments on MMAD-QA and additional anomaly detection datasets show consistent gains over matched and general-purpose baselines.
  arXiv  Detail & Related papers  (2026-07-04T11:05:57Z)
• Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation [52.8308168727975]
  Seg-Agent is a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning.<n>Our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement.<n>Various-LangSeg is a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks.
  arXiv  Detail & Related papers  (2026-05-13T03:36:44Z)
• Multi-Grained Vision-Language Alignment for Domain Generalized Person Re-Identification [15.307492395180658]
  Generalized person Re-identification (DG Re-ID) is a challenging task, where models are trained on source domains but tested on unseen target domains.<n>Recently, Vision-Language Models (VLMs) present outstanding generalization capabilities in various visual applications.<n>We propose a CLIP-based multi-grained vision-language alignment framework in this work.
  arXiv  Detail & Related papers  (2026-03-14T16:33:12Z)
• Test-Time Computing for Referring Multimodal Large Language Models [143.49848714354698]
  We propose ControlMLLM++, a novel test-time adaptation framework.<n>It injects learnable visual prompts into frozen multimodal large language models.
  arXiv  Detail & Related papers  (2026-02-23T04:42:10Z)
• EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models [80.00303150568696]
  We propose a novel Multimodal Large Language Models (MLLM) that empowers comprehension of arbitrary referring visual prompts with less training efforts than existing approaches. Our approach embeds referring visual prompts as spatial concepts conveying specific spatial areas comprehensible to the MLLM. We also propose a Geometry-Agnostic Learning paradigm (GAL) to further disentangle the MLLM's region-level comprehension with the specific formats of referring visual prompts.
  arXiv  Detail & Related papers  (2024-09-25T08:22:00Z)
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning [40.972648044298374]
  Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. They often lack interpretability and struggle with complex visual inputs. We introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs. We propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts.
  arXiv  Detail & Related papers  (2024-03-25T17:59:23Z)
• Compositional Chain-of-Thought Prompting for Large Multimodal Models [46.721769077885966]
  Compositional Chain-of-Thought (CCoT) is a novel zero-shot Chain-of-Thought prompting method. We first generate an SG using the Large Language Model (LLM) and then use that SG in the prompt to produce a response. We find that the proposed CCoT approach not only improves LMM performance but also improves the performance of several popular LMMs on general multimodal benchmarks.
  arXiv  Detail & Related papers  (2023-11-27T22:23:27Z)
• Dynamic Context-guided Capsule Network for Multimodal Machine Translation [131.37130887834667]
  Multimodal machine translation (MMT) mainly focuses on enhancing text-only translation with visual features. We propose a novel Dynamic Context-guided Capsule Network (DCCN) for MMT. Experimental results on the Multi30K dataset of English-to-German and English-to-French translation demonstrate the superiority of DCCN.
  arXiv  Detail & Related papers  (2020-09-04T06:18:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.