VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs

• URL: http://arxiv.org/abs/2608.30705v1
• Date: Mon, 31 Aug 2026 12:43:42 GMT
• Title: VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs
• Abstract summary: We propose emphVisLens (Visual Focus via Logit Lens), a Visual Search method built on the logit lens.<n>VisLens matches or exceeds prior baselines while delivering a substantial latency advantage.
• Score: 50.62479086957651
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into two families: (1) Training-free methods based on attention or confidence scores are accurate but slow, since they require multiple MLLM queries per example. (2) Reinforcement Learning (RL) trained tool-use models are faster at inference but opaque, since their tool calls remain uncontrollable and hard to interpret. To overcome this, we propose \emph{VisLens} (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head. VisLens further uses a lightweight tuned-lens that maps early hidden states into the final hidden state space, so visual tokens can be read out from early layers. These tokens are matched to target words in the query to generate a crop of the relevant region, which is fed back in alongside the original image to produce the final answer. The whole process, from decoding to the final answer, completes in a single forward pass without repeated queries. VisLens matches or exceeds prior baselines while delivering a substantial latency advantage, running $8.5$--$9.9\times$ faster than Thyme and up to $22.2\times$ faster than training-free multi-pass search methods.

Related papers

• Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models [56.11961584000622]
  Multimodal large language models (MLLMs) often fail in fine-grained visual reasoning.<n>We propose LatEnt Noise maSk (Lens), a question-conditioned visual evidence purification framework.<n>Lens scores which visual tokens support the current question and preserve them during decoding.
  arXiv  Detail & Related papers  (2026-06-29T11:44:09Z)
• DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection [39.153744982595036]
  We propose Detection Prompt Optimization (DetPO), a gradient-free test-time optimization approach.<n>Our proposed approach yields consistent improvements across generalist MLLMs on Roboflow20-VL and LVIS.
  arXiv  Detail & Related papers  (2026-03-24T17:26:55Z)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models [9.530137749236617]
  Multimodal large language models (MLLMs) project visual tokens into the embedding space of language models.<n>We introduce a two-fold analytical framework featuring a novel probing tool, $textLenEmbeds$, to conduct a fine-grained analysis.<n>We uncover a pronounced semantic sparsity at the input level: visual tokens consistently partition into sink, dead, and alive categories.
  arXiv  Detail & Related papers  (2026-02-28T07:13:36Z)
• Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing [19.577278316436807]
  Large Language Models (LLMs) are limited by the context window size.<n>We propose a novel method that leverages the LLMs's own attention information to enable accurate retrieval.<n>InfiniRetri achieves 100% accuracy in the Needle-In-a-Haystack(NIH) test over 1M tokens using a 0.5B parameter model.
  arXiv  Detail & Related papers  (2025-02-18T15:45:36Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• ST$^3$: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming [14.937905258757635]
  $textbfST3$ is a framework designed to accelerate MLLM inference without retraining.<n>$textbfST3$ can be seamlessly integrated into existing pre-trained MLLMs.
  arXiv  Detail & Related papers  (2024-12-28T10:17:29Z)
• AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning [19.68349294206012]
  Large language models (LLMs) have enabled the creation of multi-modal LLMs that exhibit strong comprehension of visual data such as images and videos.<n>We propose a training-free adaptive inference method for multi-modal LLMs that can accommodate a broad range of efficiency requirements.
  arXiv  Detail & Related papers  (2024-12-04T11:47:57Z)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings [66.04061083611863]
  Excessive use of visual tokens in existing Multimoal Large Language Models (MLLMs) often exhibits obvious redundancy and brings in prohibitively expensive computation.<n>We propose a simple yet effective method to improve the efficiency of MLLMs, termed dynamic visual-token exit (DyVTE)<n>DyVTE uses lightweight hyper-networks to perceive the text token status and decide the removal of all visual tokens after a certain layer.
  arXiv  Detail & Related papers  (2024-11-29T11:24:23Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• VILA: On Pre-training for Visual Language Models [74.08039416548209]
  We study the design options for VLM pre-training through step-by-step controllable comparisons. We build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-12T18:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.