Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models

• URL: http://arxiv.org/abs/2509.17588v1
• Date: Mon, 22 Sep 2025 11:12:12 GMT
• Title: Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models
• Abstract summary: Large Vision-Language Models (LVLMs) answer visual questions by transferring information from images to text through a series of attention heads.<n>We propose head attribution, a technique inspired by component attribution methods, to identify consistent patterns among attention heads that play a key role in information transfer.
• Score: 12.062173567776453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) answer visual questions by transferring information from images to text through a series of attention heads. While this image-to-text information flow is central to visual question answering, its underlying mechanism remains difficult to interpret due to the simultaneous operation of numerous attention heads. To address this challenge, we propose head attribution, a technique inspired by component attribution methods, to identify consistent patterns among attention heads that play a key role in information transfer. Using head attribution, we investigate how LVLMs rely on specific attention heads to identify and answer questions about the main object in an image. Our analysis reveals that a distinct subset of attention heads facilitates the image-to-text information flow. Remarkably, we find that the selection of these heads is governed by the semantic content of the input image rather than its visual appearance. We further examine the flow of information at the token level and discover that (1) text information first propagates to role-related tokens and the final token before receiving image information, and (2) image information is embedded in both object-related and background tokens. Our work provides evidence that image-to-text information flow follows a structured process, and that analysis at the attention-head level offers a promising direction toward understanding the mechanisms of LVLMs.

Related papers

• Leveraging Text-to-Image Diffusion Models for Unsupervised Visual Object Tracking [53.08526613328243]
  Unsupervised visual object tracking is a challenging task that requires following arbitrary targets in videos without training on ground-truth annotations.<n>We build on text-to-image diffusion models to exploit the rich semantic knowledge encoded in pretrained models.<n>Our method Diff-Tracking is composed of two main components: an initial prompt learner and an online prompt updater.
  arXiv  Detail & Related papers  (2026-05-26T12:26:56Z)
• VIPA: Visual Informative Part Attention for Referring Image Segmentation [21.08745617831414]
  Referring Image (RIS) aims to segment a target object described by a natural language expression.<n>We propose a novel Visual Informative Part Attention (VIPA) framework for referring image segmentation.<n>Our framework enables the network's attention to robustly align with the fine-grained regions of interest.
  arXiv  Detail & Related papers  (2026-02-16T14:36:50Z)
• TeSG: Textual Semantic Guidance for Infrared and Visible Image Fusion [55.34830989105704]
  Infrared and visible image fusion (IVF) aims to combine complementary information from both image modalities.<n>We introduce textual semantics at two levels: the mask semantic level and the text semantic level.<n>We propose Textual Semantic Guidance for infrared and visible image fusion, which guides the image synthesis process.
  arXiv  Detail & Related papers  (2025-06-20T03:53:07Z)
• How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads [3.6152232645741025]
  We identify the heads responsible for recognizing text from images, which we term the Optical Character Recognition Head (OCR Head)<n>Our findings regarding these heads are as follows: (1) Less Sparse: Unlike previous retrieval heads, a large number of heads are activated to extract textual information from images.<n>We validate our findings in downstream tasks by applying Chain-of-Thought (CoT) to both OCR and conventional retrieval heads and by masking these heads.
  arXiv  Detail & Related papers  (2025-05-21T10:53:41Z)
• Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models [36.18155629835474]
  Vision-language models (VLMs) excel at extracting and reasoning about information from images.<n>Yet, their capacity to leverage internal knowledge about specific entities remains underexplored.<n>This work investigates the disparity in model performance when answering factual questions about an entity described in text versus depicted in an image.
  arXiv  Detail & Related papers  (2024-12-18T18:22:30Z)
• What's in the Image? A Deep-Dive into the Vision of Vision Language Models [20.669971132114195]
  Vision-Language Models (VLMs) have recently demonstrated remarkable capabilities in comprehending complex visual content. In this paper, we conduct a thorough empirical analysis, focusing on attention modules across layers. We reveal several key insights about how these models process visual data.
  arXiv  Detail & Related papers  (2024-11-26T14:59:06Z)
• See then Tell: Enhancing Key Information Extraction with Vision Grounding [32.445618057103324]
  We introduce STNet (See then Tell Net), a novel end-to-end model designed to deliver precise answers with relevant vision grounding.<n>To enhance the model's seeing capabilities, we collect extensive structured table recognition datasets.<n>Our approach demonstrates substantial advancements in KIE performance, achieving state-of-the-art results on publicly available datasets.
  arXiv  Detail & Related papers  (2024-09-29T06:21:05Z)
• Revolutionizing Text-to-Image Retrieval as Autoregressive Token-to-Voken Generation [90.71613903956451]
  Text-to-image retrieval is a fundamental task in multimedia processing. We propose an autoregressive voken generation method, named AVG. We show that AVG achieves superior results in both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2024-07-24T13:39:51Z)
• Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning [35.47078178526536]
  Recent advancements in pre-trained large-scale language-image models have ushered in a new era of visual comprehension. This paper tackles two well-known issues within the realm of visual analytics: (1) the efficient exploration of large-scale image datasets and identification of potential data biases within them; (2) the evaluation of image captions and steering of their generation process.
  arXiv  Detail & Related papers  (2023-11-02T06:21:35Z)
• Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? [50.29862466940209]
  We introduce InfoSeek, a visual question answering dataset tailored for information-seeking questions. We analyze various pre-trained visual question answering models and gain insights into their characteristics. We show that accurate visual entity recognition can be used to improve performance on InfoSeek by retrieving relevant documents.
  arXiv  Detail & Related papers  (2023-02-23T00:33:54Z)
• SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning [61.57887011165744]
  multimodal Transformers have made great progress in the task of Visual Commonsense Reasoning. We propose a Scene Graph Enhanced Image-Text Learning framework to incorporate visual scene graphs in commonsense reasoning.
  arXiv  Detail & Related papers  (2021-12-16T03:16:30Z)
• Rethinking of the Image Salient Object Detection: Object-level Semantic Saliency Re-ranking First, Pixel-wise Saliency Refinement Latter [62.26677215668959]
  We propose a lightweight, weakly supervised deep network to coarsely locate semantically salient regions. We then fuse multiple off-the-shelf deep models on these semantically salient regions as the pixel-wise saliency refinement. Our method is simple yet effective, which is the first attempt to consider the salient object detection mainly as an object-level semantic re-ranking problem.
  arXiv  Detail & Related papers  (2020-08-10T07:12:43Z)
• TextCaps: a Dataset for Image Captioning with Reading Comprehension [56.89608505010651]
  Text is omnipresent in human environments and frequently critical to understand our surroundings. To study how to comprehend text in the context of an image we collect a novel dataset, TextCaps, with 145k captions for 28k images. Our dataset challenges a model to recognize text, relate it to its visual context, and decide what part of the text to copy or paraphrase.
  arXiv  Detail & Related papers  (2020-03-24T02:38:35Z)
• Fine-grained Image Classification and Retrieval by Combining Visual and Locally Pooled Textual Features [8.317191999275536]
  In particular, the mere presence of text provides strong guiding content that should be employed to tackle a diversity of computer vision tasks. In this paper, we address the problem of fine-grained classification and image retrieval by leveraging textual information along with visual cues to comprehend the existing intrinsic relation between the two modalities.
  arXiv  Detail & Related papers  (2020-01-14T12:06:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.